Patent Description:
In recent years, KV-SSDs have been used in an increasing variety of applications. Therefore, enhancing IO performance for KV-SSDs to enhance throughput and to reduce latency may be beneficial.

However, enhanced IO performance may be difficult when KV-SSDs encounter a mixture of both large object IOs and small object IOs because KV-SSD processing may address both large object IOs and small object IOs in the same manner.

As a result, KV-SSD processing may favor large object IOs, which results in small object IOs having higher latency. Alternatively, KV-SSD processing may favor small object IOs, which results in large object IOs having lower throughput.

<CIT> discloses: Ensuring reproducibility in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit ('GPU') servers, including: identifying, by a unified management plane, one or more transformations applied to a dataset by the artificial intelligence infrastructure, wherein applying the one or more transformations to the dataset causes the artificial intelligence infrastructure to generate a transformed dataset; storing, within the one or more storage systems, information describing the dataset, the one or more transformations applied to the dataset, and the transformed dataset; identifying, by the unified management plane, one or more machine learning models executed by the artificial intelligence infrastructure using the transformed dataset as input; and storing, within the one or more storage systems, information describing one or more machine learning models executed using the transformed dataset as input.

Auxiliary embodiments are defined in the dependent claims.

Non-limiting and non-exhaustive embodiments of the present embodiments are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

Corresponding reference characters indicate corresponding components throughout the drawings. Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity, and have not necessarily been drawn to scale. For example, the dimensions of some of the elements, layers, and regions in the figures may be exaggerated relative to other elements, layers, and regions to help to improve clarity and understanding of various embodiments. Also, common but well-understood elements and parts not related to the description of the embodiments might not be shown in order to facilitate a less obstructed view of these various embodiments and to make the description clear.

Features of the present disclosure and methods of accomplishing the same may be understood more readily by reference to the detailed description of embodiments and the accompanying drawings. Hereinafter, embodiments will be described in more detail with reference to the accompanying drawings. The described embodiments, however, may be embodied in various different forms, and should not be construed as being limited to only the illustrated embodiments herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the present disclosure to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary to those having ordinary skill in the art for a complete understanding of the aspects and features of the present disclosure may not be described.

Unless otherwise noted, like reference numerals denote like elements throughout the attached drawings and the written description, and thus, descriptions thereof will not be repeated. Further, parts not related to the description of the embodiments might not be shown to make the description clear.

In the detailed description, for the purposes of explanation, numerous specific details are set forth to provide a thorough understanding of various embodiments. It is apparent, however, that various embodiments may be practiced without these specific details or with one or more equivalent arrangements. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring various embodiments.

It will be understood that, although the terms "first," "second," "third," etc., may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms.

It will be further understood that the terms "comprises," "comprising," "have," "having," "includes," and "including," when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

"About" or "approximately," as used herein, is inclusive of the stated value and means within an acceptable range of deviation for the particular value as determined by one of ordinary skill in the art, considering the measurement in question and the error associated with measurement of the particular quantity (e.g., the limitations of the measurement system).

The electronic or electric devices and/or any other relevant devices or components according to embodiments of the present disclosure described herein may be implemented using any suitable hardware, firmware (e.g. an application-specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, the various components of these devices may be formed on one integrated circuit (IC) chip or on separate IC chips. Further, the various components of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on one substrate.

Further, the various components of these devices may be a process or thread, running on one or more processors, in one or more computing devices, executing computer program instructions and interacting with other system components for performing the various functionalities described herein. The computer program instructions are stored in a memory which may be implemented in a computing device using a standard memory device, such as, for example, a random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer readable media such as, for example, a CD-ROM, flash drive, or the like. Also, a person of skill in the art should recognize that the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed across one or more other computing devices without departing from the scope of the embodiments of the present disclosure.

<FIG> is a block diagram depicting a network according to one or more embodiments of the present disclosure.

Referring to <FIG>, a network <NUM> according to one or more embodiments of the present disclosure follows a NVMeoF (Non-Volatile Memory Express over Fabrics) architecture. In one or more embodiments, the network <NUM> includes one or more hosts <NUM>, a network switch <NUM> (e.g., a NVMeoF switch), one or more targets <NUM> (e.g., a NVMe target), and one or more target storage devices <NUM> (e.g., NVMe-based Key-Value SSD or KV-SSD).

With reference to <FIG>, one or more hosts <NUM> may generate one or more IO requests according to data processing applications or software. The IO requests may include IO operations such as "put", "get", "delete", etc., for object-based storage (e.g., for key-value storage). The IO requests includes different IO sizes (e.g., small object IOs and/or large object IOs). The one or more hosts <NUM> forward or transmit one or more IO requests to the network switch <NUM>, such as a multi-port network bridge that connects devices together on a computer network. The network switch <NUM> may use hardware addresses to process and forward data. Therefore, the network switch <NUM> transmits the one or more IO requests to a suitable target <NUM> from among one or more targets <NUM>.

A target <NUM> from among the one or more targets <NUM> may include a dispatcher <NUM>, a memory cache (e.g., a dynamic random access memory (DRAM) cache), and a log device <NUM> (e.g., a low latency log device). The dispatcher <NUM> may be a KV IO dispatcher for receiving and dispatching one or more IOs from the network switch <NUM> to portions of the memory cache. The memory cache may include one or more zones <NUM>, which may be formed by partitions in the memory cache, for receiving and handling small object IOs from the dispatcher <NUM>.

The zone(s) <NUM> may include a flush buffer <NUM> that flushes object IOs to the target storage device <NUM> (to a KV-SSD) to complete an IO. The zone(s) <NUM> may also include a log buffer <NUM> that receives the one or more small object IOs dispatched from the dispatcher <NUM>. For example, at least two zones may each include a log buffer that receives one or more respective small object IOs dispatched from the dispatcher <NUM>. The dispatcher <NUM> may dispatch one or more IOs (referred to as sets) to corresponding log buffers in multiple zones.

The log buffer <NUM> may log or store object IOs received from the dispatcher <NUM> in the log device <NUM>. As an example, the log device <NUM> may be a non-volatile dual in-line memory module (NVDIMM) or a low latency SSD for crash recovery purposes.

The log device <NUM> may maintain a log in persistent memory that can be checked after a system crash to determine whether respective IOs are completed. If one or more IOs are not completed, the system can determine what additional steps, if any, may be suitable to complete any uncompleted IOs based on the log. The log device <NUM> may suitably use less storage capacity than a target storage device <NUM> (e.g., to save on cost).

Depending on the size of the object IOs received from the network switch <NUM>, the target <NUM> may determine a path for the object IO. For example, the target <NUM> may include a first path (e.g., a normal path) <NUM> through which one or more large object IOs received by the target <NUM> may pass directly to the target storage device <NUM> without being received by the log buffer <NUM>, such that the object IOs may be stored in the log device <NUM>. Large object IOs may be object IOs that are at or above a threshold size, wherein the threshold size may be adjustable in one or more embodiments.

The target <NUM> may also include a second path (e.g., a grouping path) <NUM> through which one or more small object IOs are received by the dispatcher <NUM>. The small object IOs may be object IOs below the threshold size. On the second path <NUM>, the dispatcher <NUM> may dispatch or transmit the one or more small object IOs to a corresponding zone <NUM> (e.g., to the log buffer <NUM> of the zone <NUM>). The one or more small object IOs on the second path <NUM> may be concatenated in the log buffer <NUM> to form an object group <NUM>, or clip, according to a grouping schema according to one or more embodiments of the present disclosure.

Accordingly, as described above, object IOs may be screened by an adjustable size threshold, and large object IOs may follow a first path <NUM> while small object IOs follow a second path <NUM> according to an object grouping schema, the details of which are described in the cross-referenced co-pending <CIT>, entitled "GROUPING KEY VALUE OBJECT IOs TO IMPROVE IO.

For example, in one or more embodiments, the log buffer <NUM> on the second path <NUM> may receive small object IOs until a condition or trigger is reached (e.g., until an object group size surpasses a threshold or maximum size, until a maximum number of small object IOs have been formed into an object group, and/or until a timeout window has lapsed, wherein the threshold, maximum number, and/or timeout window are adjustable in one or more embodiments). The timeout window refers to a time period during which small object IOs are received by, or within, the log buffer <NUM>. For example, the time period may be set according to latency and throughput suitability of the small object IOs according to an object group and/or zone.

After a condition or trigger for the log buffer <NUM> is satisfied, the log buffer <NUM> may log the one or more small object IOs that have been concatenated into an object group <NUM> to the log device <NUM>. The log buffer <NUM> may change roles to function as the flush buffer after logging the small object IOs to the log device <NUM>. In this manner, the flush buffer <NUM> may have stored therein an object group <NUM>, or clip, including small object IOs. The flush buffer may include multiple object groups <NUM>. In this case, multiple log buffers may change roles to function as flush buffers (e.g., thereby forming a single flush buffer that has a greater size). In other words, according to one or more embodiments, there is one buffer that may be partitioned as a plurality of buffers or buffer sections. Typically, one of the partitioned buffers operates as a log buffer, and there could be one or more flush buffers. The small object IOs are stored in the log buffer until the log buffer is full, after which the log buffer may change function to that of the flush buffer.

Upon reaching a certain condition, such as a threshold flush buffer size and/or a threshold idle period (e.g., based on an elapsed time of object IOs being present in the flush buffer), the flush buffer may flush the contents of the flush buffer (e.g., may flush the object group <NUM>) to the target storage device <NUM>. Then, the contents of the flush buffer may be marked for deletion to purge the contents of the flush buffer to thereby free up space in the memory cache when the flush buffer is emptied and the object IOs are completed.

After the flush buffer is emptied, the flush buffer may be available to change roles to function as a log buffer <NUM>, thereby fulfilling a single log-flush cycle. In one or more embodiments, multiple log-flush cycles for different portions of the memory cache may occur concurrently (e.g., simultaneously or substantially simultaneously). Also, the threshold flush buffer size and/or threshold idle period may be adjustable (e.g., by an algorithm, application programing interface, configuration file, or any other suitable method) known to those skilled in the art.

Although the term "object group" is used throughout the specification, if a single small object IO is received and no other small object IOs are received by the log buffer <NUM> within the receiving period, then concatenation may not occur and the "object group" may include a single small object IO.

Accordingly, an object group, or clip, including one or more small object IOs is formed in the log buffer <NUM>. The concatenated object group <NUM> may be treated as a single large object IO (due to concatenation) for the purpose of writing small object IOs to a storage device, such as the log device <NUM>, and to the target storage device <NUM>, such as the KV-SSD. In this manner, average latency and throughput for small object IOs may be enhanced because the small object IOs are treated as a single large object IO.

Although the target storage device <NUM> is depicted in <FIG> as separate from the target <NUM>, the classification of the target storage device <NUM> is not limited thereto, and the target storage device <NUM> could be considered part of the target <NUM> in other embodiments. Further, while three hosts <NUM>, two targets <NUM>, and two target storage devices <NUM> are illustrated in <FIG>, any number of hosts, targets, and target storage devices may be used with suitable modifications to the inputs and outputs of each host <NUM>, target <NUM>, and target storage device <NUM>. Accordingly, the network switch <NUM> may transmit IO requests to multiple targets <NUM> associated with one or more target storage devices <NUM>. As a result, the NVMeoF architecture may connect multiple KV-SSDs (e.g. thousands or more KV-SSDs housed in dedicated storage servers) to hosts.

<FIG> are flow charts illustrating methods of updating parameters for the object grouping schema according embodiments of the present disclosure.

Referring to <FIG>, the system and method of the present disclosure includes an object storage application <NUM>, one or more hosts <NUM>, a target <NUM>, a database <NUM>, a training circuit <NUM>, and an inferencing circuit <NUM>.

In one or more embodiments, the object storage application <NUM> interacts with one or more hosts <NUM> to generate and transmit IOs. In one or more embodiments, the one or more hosts <NUM> run a host-side software program that interacts with an external second software program based on a mode of operation. For example, in a pass-through mode, the external software program determines which path data should take to reach a particular target (e.g., which target among one or more targets), and, in a nonpass-through mode, the host-side software program determines which path data should take to reach a particular target (e.g., the target <NUM>).

In one or more embodiments, a target <NUM> tracks parameter data <NUM> associated with the object grouping schema described with respect to <FIG>. In one or more embodiments, the parameter data <NUM> includes tracking parameters <NUM>, input parameters <NUM>, and performance parameters <NUM>.

In one or more embodiments, the tracking parameters <NUM> may include data indicating when input parameters <NUM> and performance parameters <NUM> are collected, for example, a time-stamp or any other suitable parameter for determining when data collection occurred. Tracking parameters <NUM> may also include the number of operating CPU cores and/or threads available as determined by a user or specified at start up in a first software module (e.g., a Key Value Storage Application of the target) <NUM> for the object grouping schema. Multiple zones may follow the object grouping schema and each zone may have a single thread working in that zone. Each thread runs on a particular CPU core, and therefore, an increase in the cores or threads assigned to work on a zone (assuming zones are available) may improve the performance of the object grouping schema for high frequency small object IOs. Accordingly, the core or thread count is relevant to an assessment of performance parameters such as input/output operations per second ("IOPS") for determination of an inferencing model (e.g., a golden model or a reference model that is used for inferring input parameters) as described in more detail below.

In one or more embodiments, the input parameters <NUM> may include parameters for the object grouping schema that may be adjusted (e.g., via a remote procedure call (an "RPC call")) while the first software module <NUM> is online. For example, input parameters <NUM> include object group size and flash out time (e.g., maximum latency time permitted for IOs). Flash out time may be measured from the time period beginning when a small object IO is received by the log buffer and ending when the small object IO is flushed to the target storage device. Input parameters <NUM> may also include the object size being screened for the first and second paths. Input parameters <NUM> may also include the number of zones allocated for processing small object IOs via the log buffer. For example, it may be desirable to have a higher number of zones for processing when small object IO frequency is high and it may be desirable to have a lower number of zones for processing when small object IO frequency is low. Although several input parameters <NUM> are discussed, all or less than all of the discussed input parameters <NUM> may be recorded, stored, and/or adjusted in one or more embodiments, for example, in one embodiment flash out time and object group size is adjusted and in another embodiment number of zones may be adjusted in addition to the flash out time and object group size.

In one or more embodiments, the performance parameters <NUM> may include parameters indicating the performance of the object grouping schema such as IOPS (i.e., IOs completed per second) and/or latency. Latency refers to the time an object takes to return. In other words, the time period beginning when an object IO is received by a device and ending when a response is sent back to a requesting device.

Generally, it is desirable to improve or optimize performance parameters <NUM> (e.g., maximizing IOPS and/or reducing average latency) using the object grouping schema. However, IOPS and/or latency may vary as IOs transmitted to the target <NUM> change throughout the day. For example, if the input parameters <NUM> are fixed, then changes to the IO frequency, mixture, and/or size may result in suboptimal input parameters for the object grouping schema. Therefore, changes in the inbound IOs may result in a decrease in IOPS because object group size and flash out time are not updated. For example, at the peak hours, the system may experience a high volume of IO operation. Accordingly, the performance of the object grouping schema may degrade if suitable values for the input parameters <NUM> such as object group size and flash out time are not set in a timely manner. Therefore, aspects of embodiments of the present disclosure are directed toward a method of optimizing or improving performance parameters <NUM> by adjusting input parameters <NUM>.

In one or more embodiments, parameter data <NUM> is collected according to a data capture process for analysis. The data capture process includes storing parameter data <NUM> on a database (e.g., a time series database) <NUM>. The data may be transmitted by running the first software module <NUM> (e.g., the Key Value Storage Application) and a second software module <NUM> (e.g., an Agent) that runs on the target node of the target <NUM>. The first software module <NUM> stores and reads parameter data <NUM> while the second software module <NUM> polls the first software module <NUM> for health statistics from the storage application to be stored on the database <NUM>.

The data capture process may occur when object group IOs are completed as desired. For example, the data capture process may occur when every object group is flushed to the target storage device, when every other object group is flushed to the target storage device, or any other suitable sampling (e.g., any number) of object groups are flushed to the target storage device. Sampling data over time enables the database <NUM> to capture additional parameter data <NUM> throughout the day.

A training circuit <NUM> extracts extracts parameter data <NUM> from the database <NUM>. The training circuit <NUM> may extract all or substantially all data relating to the target <NUM> from the database <NUM>. The training circuit <NUM> may receive or extract data from the database <NUM> at set intervals, for example, time-based intervals (e.g., once a day, twice a day, three times a day, etc.) as desired. In one or more embodiments, the training circuit <NUM> receives or extracts data at off-times or nonpeak hours when the system that the training circuit <NUM> is a part of has more resources to perform pre-processing (<NUM>), model training and validation (<NUM>), model checks (<NUM>), and/or inferencing model updates (<NUM>). The training circuit <NUM> is separate from the target <NUM> as illustrated in <FIG>, however, in one or more embodiments the training circuit <NUM> is integrated with the target <NUM> as illustrated in <FIG>. Accordingly, the training circuit <NUM> in the embodiments of <FIG> may be separate from the target <NUM> to avoid burdening the target <NUM> with pre-processing (<NUM>), model training and validation (<NUM>), model checks (<NUM>), and/or inferencing model updates (<NUM>). In the embodiment of <FIG>, the target <NUM> may include, but is not limited to, a device optimized or configured for training to function as the training circuit <NUM>. In one or more embodiments, the training circuit <NUM> may be implemented in any suitable number and types of chips that are known to those skilled in the art. By way of example, the training circuit <NUM> may include any suitable software and hardware, for example, the training circuit <NUM> may include an FPGA, GPU, and/or any other suitable chips or components of a chip, and/or one or more discrete analog or digital elements, where one or more of these components and/or functions may be integrated in a single integrated circuit (IC) chip.

As illustrated in <FIG>, the parameter data <NUM> received or extracted from the database <NUM> is pre-processed (<NUM>) by the training circuit <NUM>. Pre-processing (<NUM>) the extracted data may include removing bad data (e.g., data entries that reduce or negatively impact a predictive ability of a training model). For example, data entries that reduce or negatively impact the predictive ability of the training model may include missing or unexpected values (or values outside an expected range). In a non-limiting example, in one or more embodiments, a single data entry may include a timestamp, object group size, flash out time, thread count, and IOPS. If any of these parameters are not present in the extracted data, then pre-processing (<NUM>) may remove the entire data entry. As another example, if any of these parameters have an unexpected value, then the entire data entry may be removed. For example, if the parameters are outside an expected range or the parameters are a negative value, then pre-processing <NUM> may remove the entire data entry. In one or more embodiments, the pre-processed data entry that is removed prior to training is also removed from the database <NUM>.

After pre-processing (<NUM>) the data, the training circuit <NUM> trains a training model (<NUM>). The training model is based on all or substantially all data extracted from the database <NUM> including any new and historical data stored on the database <NUM>. The inclusion of historical data in addition to new data over multiple training cycles results in an incremental learning approach. Therefore, in one or more embodiments, the historical data may capture at least the data on which a current inferencing model is based.

The training circuit <NUM> performs cross-validation (<NUM>) on the training model. For example, cross-validation is performed on a data sample (e.g., from the database <NUM>) where the data sample includes a training set that is used to train the training model and a validation set that evaluates the predictive ability of the training model. In one or more embodiments, cross-validation (<NUM>) is performed by k-fold cross validation. For example, the k-fold cross validation method according to one or more embodiments divides a data sample into K sets where one set is used for validation and (K-<NUM>) set(s) are used for training the model. This process may be repeated K times where the K results are averaged out. Although the k-fold cross-validation method is described with reference to cross-validation, any other suitable method of cross-validation may be used.

The training circuit <NUM> performs a model check (<NUM>) on the training model to determine if the accuracy or error reduction (e.g., a degree of error reduction) of the training model is better than the accuracy or error reduction of an inferencing model. For example, the training model may be evaluated based on error reduction which refers to how concentrated data points are. In one or more embodiments, error reduction is measured using an error handling functions, such as Mean Squared Error (MSE), Root Mean Squared Error (RMSE), R^<NUM>, and/or etc. For example, in one or more embodiments, the RMSE value for each k-fold validation (e.g., training/test split) is gathered and averaged out for comparison to the inferencing model. Although specific error handling functions are described for performing the model check, in one or more embodiments, other suitable error handling functions may be used.

If the accuracy or error reduction of the training model is greater than or better than the accuracy or error reduction of the inferencing model, then the training circuit <NUM> updates the inferencing model (<NUM>) based on the training model. In one or more embodiments, the training circuit <NUM> replaces the inferencing model with the training model to create a new inferencing model. Conversely, if the accuracy or error reduction of training model is less than or worse than the accuracy or error reduction of the inferencing model, then the training model is rejected and the inferencing model remains unchanged. Accordingly, the inferencing model may be incrementally updated (<NUM>) over time to become more robust based on historical data stored on the database <NUM>.

The target <NUM> performs an IOPS and/or latency check (<NUM>) and determines that the performance parameters <NUM> is outside of an expected range. The target <NUM> continuously monitors IOPS for the target to check whether IOPS fall outside a range of about <NUM>% to about <NUM>% of an expected IOPS value. When one or more performance parameters <NUM> (e.g., IOPS and/or latency) are outside the expected range, the inferencing model is used by an inferencing circuit <NUM> to infer (<NUM>) suitable input parameters (e.g., object group size, flash out time, etc.). The inferencing circuit <NUM> performs a parameter check (<NUM>) by comparing the inferred input parameters to the input parameters <NUM> of the target <NUM>. If the inferred input parameters and the input parameters <NUM> of the target <NUM> are the same then no action is taken. If the inferred input parameters are different from the input parameters <NUM> of the target <NUM>, then the inferencing circuit <NUM> updates (e.g., via an RPC call) the input parameters <NUM> of the target <NUM>.

The inferencing circuit <NUM> is separate from the target <NUM> to avoid burdening the target <NUM> with inferencing as illustrated in the embodiments of <FIG> and <FIG>, and in one or more embodiments, the inferencing circuit <NUM> is a part of the target <NUM> as illustrated in the embodiments of <FIG> and <FIG>. Although the inferencing circuit <NUM> is depicted in the illustrated embodiments as separate from the training circuit <NUM>, in one or more embodiments, the inferencing circuit <NUM> may be a part of the same device as the training circuit <NUM> as illustrated in the embodiment of <FIG>. In one or more embodiments, the inferencing circuit <NUM> may be implemented in any suitable number and types of chips that are known to those skilled in the art. By way of example, the inferencing circuit <NUM> may include any suitable software and/or hardware including, for example, an FPGA and/or GPU based accelerator for fast inferencing, and/or any other suitable chips or components of a chip, and/or one or more discrete analog or digital elements, where one or more of these components and/or functions may be integrated in a single chip.

Claim 1:
A computer-implemented method of optimizing or improving parameters for grouping differently sized object input/outputs, IOs, for key-value solid state drives, the method performed by a system having a target device (<NUM>), a training circuit (<NUM>) and an inferencing circuit (<NUM>), the method comprising:
receiving, by the target device (<NUM>), differently sized object IOs for the target device (<NUM>);
grouping, by the target device (<NUM>), the IOs using a first plurality of input parameters;
associating, by the target device (<NUM>), a tracking parameter (<NUM>) with the first plurality of input parameters and with a performance parameter (<NUM>) corresponding to the first plurality of input parameters;
storing, by the target device (<NUM>), a first data entry comprising the tracking parameter (<NUM>), the first plurality of input parameters, and the performance parameter (<NUM>) in a database (<NUM>);
extracting, by the training circuit (<NUM>), a plurality of data entries which include the first data entry from the database (<NUM>);
training (<NUM>), by the training circuit (<NUM>), a training model using one or more of the plurality of data entries;
cross-validating (<NUM>), by the training circuit (<NUM>), the training model to determine error reduction of the training model;
performing (<NUM>), by the training circuit (<NUM>), a model check to compare the error reduction of the training model to error reduction of an inferencing model;
if the error reduction of the training model is greater than the error reduction of the inferencing model, then updating (<NUM>), by the training circuit (<NUM>), the inferencing model based on the training model;
using (<NUM>), by the inferencing circuit (<NUM>), the updated inferencing model to infer suitable input parameters as a response to the performance parameter (<NUM>) being outside of an expected range (<NUM>);
performing (<NUM>), by the inferencing circuit (<NUM>), a parameter check by comparing the inferred input parameters to the input parameters (<NUM>) of the target device (<NUM>); and
if the inferred input parameters are different from the input parameters (<NUM>) of the target device (<NUM>), then updating, by the inferencing circuit (<NUM>), the input parameters (<NUM>) of the target device (<NUM>).