Systems and methods for checking whether training data to be inputted into a training phase of a ML model is Independent and Identically Distributed data (IID data), and taking action based on that determination. One example of the present disclosure provides a method implemented by an edge node operating in a distributed swarm learning blockchain network. The method includes receiving a smart contract including a definition of conforming data and executing the smart contract including the definition of conforming data. The method further includes receiving one or more batches of training data for training a ML model. The method further includes checking whether each batch of training data conforms to the agreed-upon definition of conforming data, tagging and isolating non-conforming batches of training data, and inputting conforming batches of training data into a training phase of the machine learning model. The conforming batches of training data are IID data.

BACKGROUND

Machine learning (ML) generally involves a computer-implemented process that builds a model using sample data (e.g., training data) in order to make predictions or decisions without being explicitly programmed to do so. ML processes are used in a wide variety of applications, particularly where it is difficult or unfeasible to develop conventional algorithms to perform various computing tasks.

Blockchain is a tamper-proof, decentralized ledger that establishes a level of trust for the exchange of value without the use of intermediaries. A blockchain can be used to record and provide proof of any transaction, and is updated every time a transaction occurs.

A particular type of ML process, called supervised machine learning, uses labeled datasets to train algorithms to classify data or predict outcomes. The process for setting up the supervised machine learning generally involves (a) centralizing a large data repository, (b) acquiring a ground truth for these data, i.e., the reality or correct answer that is being modeled with the supervised machine learning algorithm, and (c) employing the ground truth to train the ML model for the classification task. However, this framework poses significant practical challenges, including data privacy and security challenges that come with creating a large central data repository for training the ML models.

DETAILED DESCRIPTION

Federated learning or collaborative learning is a type of ML process that trains an ML model across multiple decentralized devices holding local data samples. In some examples, the decentralized devices may not exchange their data sets. This approach stands in contrast to traditional centralized ML techniques where all local datasets are uploaded to one server, as well as in contrast to more classical decentralized approaches which often assume that local data samples are identically distributed. Particularly, federated learning enables multiple devices to build a common, robust ML model without sharing data, thus allowing to address critical issues such as data privacy, data security, data access rights, and access to heterogeneous data. Its applications are spread over a number of industries including but not limited to defense, telecommunications, Internet of Things (IoT), and pharmaceutics.

However, privacy concerns remain in federated learning. For example, the sources that provide data for federated learning may be unreliable. The sources may be vulnerable to network issues since they commonly rely on less powerful communication media (i.e., Wi-Fi) or battery-powered systems (i.e., smartphones and IoT devices) compared to traditional centralized ML where nodes are typically data centers that have powerful computational capabilities and are connected to one another with fast networks.

Distributed or decentralized ML can refer to ML model building across multiple nodes using data locally available to each of the nodes. The local model parameters learned from each local model can be merged to derive a global model, where the resulting global model can be redistributed to all the nodes for another iteration, i.e., localized data trains the global model. This can be repeated until the desired level of accuracy with the global model is achieved.

Model training in general involves separating available data into training datasets and validation datasets, where after running a training iteration using the training dataset, a model can be evaluated on its performance/accuracy by performing on data it has never seen, i.e., the validation dataset. The degree of error or loss resulting from this evaluation is referred to as validation loss. Validation loss can be an important aspect of ML for implementing training features. For example, validation loss can be used to avoid overfitting a model on training data by creating an early stopping criterion in which training is halted once the validation loss reaches a minimum value. As another example, validation loss can be used in an adaptive synchronization setting, where the length of a synchronization interval is modulated based on the progress of validation loss values across multiple iterations (i.e., modulating the synchronization frequency).

However, in a distributed ML environment, where data is not maintained in a central/single location, validation is not possible at any single participating node during training. Thus the model at any single participating node cannot be evaluated as to its accuracy during training.

Therefore, according to some embodiments, prior to each training iteration, respective local datasets at each participating node are divided into training datasets and validation datasets. Then, a local training iteration commences in batches using the training datasets. At the end of each training batch, a node is designated as a merge leader for that batch. The merge leader merges the parameters from each of the participating nodes (including itself) to build a global model. The merged parameters then can be shared with the rest of the participating nodes. The merged parameters are subsequently applied to each local model at each of the participating nodes. The updated local models are then evaluated using the previously identified validation datasets. Each participating node then shares its respective local validation loss value with the leader. The merge leader merges or averages the local validation loss values to arrive at a global validation loss value, which can then be shared with the rest of the nodes. Accordingly, a global validation loss value can be derived based on the universe of participating nodes, and this global validation loss value can be used by each of the participating nodes to determine if training can stop or if further training may be needed.

This parameter merging process can be repeated until the network is able to converge the global model to a desired accuracy level. As part of determining whether the global model is able to achieve a desired accuracy level, the validation loss may be calculated. As discussed above, validation may be performed locally, the local validation loss of each node can be shared, and an average of the local validation loss can be calculated to derive a global validation loss. This global validation loss may be shared with each node so that each node may determine how well its local model has performed or has been trained from a network or system-wide (i.e., global) perspective.

In particular, upon determining that a quorum of nodes in a swarm learning network are ready to merge their respective model parameters, a merge leader is elected. In Swarm Learning (SL) each node possessing local training data trains a common ML model without sharing the local training data to any other node or entity in the swarm blockchain network. This is accomplished by sharing parameters (weights) derived from training the common ML model using the local training data. In this way, the learned insights regarding raw data (instead of the raw data itself) can be shared amongst participants or collaborating peers/nodes, which aids in protecting data against privacy breaches. Moreover, Swarm Learning as described herein leverages blockchain technology to allow for decentralized control, monetization, and ensuring trust and security amongst individual nodes. Additional information describing Swarm Learning is described in greater detail in U.S. patent application Ser. No. 17/205,632 filed on Mar. 18, 2021 and published as U.S. Patent Application Publication No. US 2021/0398017, the contents of which are incorporated by reference herein.

The assumption of “independent and identically distributed data” (IID data) is common in Machine Learning algorithms. “Independent” means that samples taken from individual random variables are independent of each other. A distribution is “identical” if the samples come from the same random variable.

As industry demand trends towards Swarm Learning and Decentralized Learning as well as Federated Learning, various technical problems arise. How to deal with a technical problem operationally is a significant issue. As referred to above, it is a common technology assumption that many techniques work well provided the data honors IID. For example, in a Decentralized Learning context such as Swarm Learning, there is a significant assumption made about the training data that is fed into the training phase, specifically, that such data is IID. However in conventional techniques the burden lies on the customer to ensure that the data is IID.

Accordingly a technical problem with conventional techniques is that there is no provision to detect IID—or non-IID—within Machine Learning systems such as Swarm Learning. Without such provision it can be difficult for a distributed customer to ensure that the data fed into the training phase is IID. This can lead to spurious weights being exchanged in the batches of training, which can adversely affect the learning. Spurious weights in one example are parameters that do not conform to IID and therefore can lower the trustworthiness of the model.

In more detail, when the IID assumption of a system does not hold, data drift can result, thereby causing issues such as the ML algorithms in the system becoming less trustworthy, with less reliable data used in distributed trainings. For example, a drift can occur on input data streams or predicted output data, or a concept shift can occur in a relationship between input data streams and outputs, over a period of time. In another example drift might be due to bias in the data collection phase. In any event non-IID data can cause drift.

Accordingly IID requirements of data if not met can alter the model output and can be a leakage which can affect certain global assumptions of decentralized models. One example of a global assumption of decentralized models may be that the data input into the training phase at edge nodes is IID data. Training a network behavior model from a different node, without each node abiding by an agreement of IID data for training data, can cause less reliable or less trustworthy model outputs to be exchanged.

The present disclosure provides technical solutions that address the above technical problems. For example, in a Swarm Learning framework the disclosed technology provides a data consistency check or an explicit data configuration check in the SL framework, as a configuration exchanged before the training phase. This can isolate data batches that do not conform to the expected IID norm and thereby keep these non-conforming data batches away from training. The isolated data can be used later for data science admin introspection, in which a human or data admin analyzes the isolated data, and may even later correct the non-conforming data and place the corrected back into the pipeline for IID checking, or include the corrected data in the training set as qualified IID data. Data isolation logs and data sheets can be kept. Accordingly the disclosure uses decentralized configuration checks to conform to Global IID requirements. Human Introspection for example might include statistical analysis or data analysis to investigate how the data ended up as non-conforming in the first place. For example, this inquiry might look into whether there were any data entry errors, or measurement errors by sensors, etc. After the non-conforming data has been suitably treated by making any needed corrections, the admin can plan to re-include the data for training batches.

Swarm Learning has an enterprise enablement framework with licensing and control for nodes joining for privacy preserving Decentralized Learning. Examples of the disclosed technology provide localized IID drift detection capability before the training stage. This can be achieved in multiple ways. One non-limiting example is for structured data. Structured data can be measured such as being bounded within the Min/Max of a univariate variable or even checking the integrity of multivariate variables by checking the correlation or covariance shifts. Another non-limiting example is for unstructured data such as images: it may be difficult or quite complex to filter the image at the source. In such cases according to the disclosed technology there can be knobs or levers of control provided for aspects such as batch training times, layer weights, etc., which may be indicators of IID data.

These additional steps can make ML algorithms more conforming and with more reliable data used in distributed trainings. By virtue of the technical solutions described herein, IID conformance can be increased, which can increase data trustworthiness, particularly when using SL. This can have technical effects such as increasing control, monitoring, and visibility of ML algorithms which can result in more adoption and trustworthiness. IID requirements of data can be met which can reduce alteration of the model output and reduce the leakage that can affect the global assumption of decentralized models. Model output trustworthiness can be increased via explicit IID data conformance and configuration.

As an example, consider a system that uses one or more machine learning models, in which system behavior is used to model the system reliability, degradation, availability, conformance, policy, etc. For example, a robotics system can exchange monitoring parameters operating within set IID thresholds, and detected non-IID conformance can flag an anomaly and then take action to isolate the non-conforming data or pause the system, etc. In the case of a network where non-conforming IID data is detected, the network can then be isolated based on the analyzed system behavior. The present disclosure is of course not limited to these examples.

The present disclosure according to an example provides an edge node operating in a distributed swarm learning blockchain network. The edge node comprises at least one processor and a memory unit operatively connected to the at least one processor. The memory unit includes instructions that, when executed, cause the at least one processor to receive a smart contract including a definition of conforming data and execute the smart contract including the definition of conforming data, to thereby agree upon the definition of conforming data set out in the smart contract. The instructions further cause the at least one processor to receive one or more batches of training data for training a machine learning model. The instructions further cause the at least one processor to check whether each batch of training data conforms to the agreed-upon definition of conforming data, to determine conforming batches of training data and non-conforming batches of training data, tag and isolate the non-conforming batches of training data, and input the conforming batches of training data into a training phase of the machine learning model. The conforming batches of training data are IID data. The non-conforming batches of training data may be listed in a log, or may be discarded. The trained weights or parameters may be shared with the network; more particularly, the memory unit may include instructions that when executed further cause the at least one processor to share with other nodes in the network trained weights or parameters derived from training the local version of the machine learning model using the conforming batches of training data.

As noted above, the disclosed technology may be applied at participant nodes under control of a blockchain network such as a Swarm Learning network. These nodes can be referred to as “edge” systems as they may be placed at or near the boundary where the real world (e.g., user computing devices, IoT devices, or other user-accessible network devices) interacts with large information technology infrastructure. For example, autonomous ground vehicles currently include more than one computing device that can communicate with fixed server assets. More broadly, edge devices such as IoT devices in various context such as consumer electronics, appliances, or drones, are increasingly equipped with computational and network capacity. Another example includes real-time traffic management in smart cities, which divert their data to a data center. However, as described herein, these edge devices may be decentralized for efficiency and scale to perform collaborative machine learning as nodes in a blockchain network.

FIG.1illustrates an example system of decentralized IID checking network using blockchain, according to an example implementation of the disclosure. Illustrative system100comprises decentralized IID checking network110with a plurality of nodes10in a cluster or group of nodes at a location (illustrated as first node10A, second node10B, third node10C, fourth node10D, fifth node10E, sixth node10F, seventh node10G). The decentralized IID checking network110may be a Swarm Learning network.

The plurality of nodes10in the cluster in decentralized IID checking network110(also referred to as a blockchain network110) may comprise any number, configuration, and connections between nodes10. As such, the arrangement of nodes10shown inFIG.1is for illustrative purposes only. Node10may be a fixed or mobile device. Examples of further details of node10will now be described. While only one of nodes10is illustrated in detail in the figures, each of nodes10may be configured in the manner illustrated. Node10may include one or more processors20(interchangeably referred to herein as processors20, processor(s)20, or processor20for convenience), one or more storage devices40, or other components.

Distributed ledger42may include a series of blocks of data that reference at least another block, such as a previous block. In this manner, the blocks of data may be chained together as distributed ledger42. For example, in a distributed currency context, a plurality of exchanges may exist to transfer a user's currency into a digital or virtual currency. Once the digital or virtual currency is assigned to a digital wallet of a first user, the first user may transfer the value of the digital or virtual currency to a digital wallet of a second user in exchange for goods or services. The digital or virtual currency network may be secured by edge devices or servers (e.g., miners) that are rewarded new digital or virtual currency for verifying this and other transactions occurring on the network. After verification, the transaction from the digital wallet of the first user to the digital wallet of the second user may be recorded in distributed ledger42, where a portion of distributed ledger42may be stored on each of the edge devices or servers.

In some implementations, distributed ledger42may provide a blockchain with a built-in fully fledged Turing-complete programming language that can be used to create “contracts” that can be used to encode arbitrary state transition functions. Distributed ledger42may correspond with a protocol for building decentralized applications using an abstract foundational layer. The abstract foundational layer may include a blockchain with a built-in Turing-complete programming language, allowing various decentralized systems to write smart contracts and decentralized applications that can communicate with other decentralized systems via the platform. Each system can create its own arbitrary rules for ownership, transaction formats, and state transition functions. Smart contracts or blocks can contain one or more values (e.g., state) and be encrypted until they are unlocked by meeting conditions of the system's protocol.

Distributed ledger42may store the blocks that indicate a state of node10relating to its machine learning during an iteration. Thus, distributed ledger42may store an immutable record of the state transitions of node10. In this manner, distributed ledger42may store a current and historic state of model in model data store44.

Model data store44may be memory storage (e.g., data store) for storing locally trained ML models at node10based on locally accessible data, as described herein, and then updated based on model parameters learned at other participant nodes10. As noted elsewhere herein, the nature of model data store44will be based on the particular implementation of the node10itself. For instance, model data store44may include trained parameters relating: to self-driving vehicle features such as sensor information as it relates object detection, dryer appliance relating to drying times and controls, network configuration features for network configurations, security features relating to network security such as intrusion detection, and/or other context-based models.

ML algorithms stored in model store44include the general class of ML algorithms that operate on IID. That includes many statistical and classical ML algorithms in use by verticals, such as regression-based, Decision Tree (DT), Support Vector Machine (SVM), etc. Training methods can include, but are not limited to, standard batch training.

Rules46may include smart contracts or computer-readable rules that configure nodes to behave in certain ways in relation to decentralized machine learning and enable decentralized control. For example, rules46may specify deterministic state transitions, when and how to elect a voted leader node, when to initiate an iteration of machine learning, whether to permit a node to enroll in an iteration, a number of nodes required to agree to a consensus decision, a percentage of voting participant nodes required to agree to a consensus decision, and/or other actions that node10may take for decentralized machine learning.

SL framework with IID check48is a Swarm Learning framework, platform, architecture, application, or the like. The SL framework48will be further described below in connection withFIG.2B. The SL framework48can be downloaded and installed on the respective nodes10during which the configuration of the network110, finalized during an initialization and onboarding step as further described below, is also supplied. Afterwards, the SL framework48boots up and initiates the connection of a node10to the network110, which is essentially a blockchain overlay on the underlying network connection between the nodes10. The boot-up process is an ordered process in which the set of participant nodes10designated as peer-discover nodes10(during the initialization phase) are booted up first, followed by the rest of the nodes10in the network.

In general a smart contract is a program stored on a blockchain that contains a set of rules by which the parties or participants to the smart contract agree to interact with each other. The program runs when predetermined condition(s) are met. Accordingly, smart contracts are typically used to automate the execution of an agreement so that all participants can be immediately certain of the outcome or the predetermined condition(s), without any intermediary's involvement or time loss. Smart contracts can also automate a workflow, triggering the next action when conditions are met. In examples of the disclosed technology a smart contract can include a definition of conforming data, e.g., IID data that all participating nodes agree to, as described in more detail herein.

As will be described in more detail below in connection withFIG.2B, the SL framework48may be implemented by swarm learning architecture230which includes an IID configuration file232and an IID check engine238. These structural elements in the SL framework48facilitate rules or smart contracts that define a definition of conforming data, e.g., IID data, that may be stored in the distributed ledger42or the rules46ofFIG.1. The SL framework48may require nodes10to agree with the rules or definitions of conforming data, e.g., IID data, in order to be connected to the network110or treated as a participating node10. If a node10fails to execute the smart contract agreeing to the definition of IID then that node cannot join the decentralized training. The SL framework48may also require participating nodes10to check for conforming data, e.g., IID data, before such data is inputted into the training phase of a ML model stored for example in model store44, as described further below in connection withFIG.2A.

Accordingly, examples of the disclosed technology provide a data consistency check or an explicit data configuration check in the SL framework, as a configuration exchanged before the training phase. This check can isolate data batches that do not conform to the expected IID norm, meaning that these non-conforming data batches are kept away from training and may be recorded in a data log. Thus in examples the disclosure uses decentralized configuration checks, to conform to Global IID requirements.

Processors20may obtain other data accessible locally to node10but not necessarily accessible to other nodes10. Such locally accessible data may include, for example, private data that should not be shared with other devices, but model parameters that are learned from the private data can be shared. Processors20may be programmed by one or more computer program instructions. For example, processors20may be programmed to execute application layer22, machine learning framework24(illustrated and also referred to as ML framework24), interface layer28, or other instructions to perform various operations, each of which are described in greater detail herein. As used herein, for convenience, the various instructions will be described as performing an operation, when, in fact, the various instructions program processors20(and therefore node10) to perform the operation.

Application layer22may execute applications on the node10. For instance, application layer22may include a blockchain agent (not illustrated) that programs node10to participate in a decentralized machine learning across blockchain network110as described herein. In examples each node10may be programmed with the same blockchain agent, thereby ensuring that each acts according to the same set of decentralized IID checking rules, such as those which may be encoded using rules46. For example, the blockchain agent may program each node10to perform IID checking using an agreed-upon definition of IID data as set out in an executed smart contract according to the process further described below in connection withFIG.2A. Application layer22may execute machine learning through the ML framework24.

ML framework24may train a model based on data accessible locally at node10. This data may undergo IID checking in accordance with the disclosed technology. For example, ML framework24may generate model parameters from sensor data, data aggregated from nodes10or other sources, data that is licensed for sources, and/or other devices or data sources to which the node10has access. The data may include private data that is owned by the particular node10and not visible to other devices. In an implementation, the ML framework24may use the TensorFlow™ machine learning framework, although other frameworks may be used as well.

Application layer22may use interface layer28to interact with and participate in the blockchain network110for decentralized machine learning across multiple participant nodes10. Interface layer28may communicate with other nodes using blockchain by, for example, broadcasting blockchain transactions and writing blocks to the distributed ledger42based on those transactions. Application layer22may use the distributed ledger42to coordinate parallel IID agreement and checking during an iteration with other participant nodes10in accordance with rules46.

Interface layer28may share the one or more parameters and inferences with the other participant nodes10. Interface layer28may include a messaging interface used to communicate via a network with other participant nodes10. The messaging interface may be configured as a Secure Hypertext Transmission Protocol (“HTTPS”) micro server. Other types of messaging interfaces may be used as well. Interface layer28may use a blockchain API to make calls for blockchain functions based on a blockchain specification. Examples of blockchain functions include, but are not limited to, reading and writing blockchain transactions and reading and writing blockchain blocks to the distributed ledger42.

As noted above, the network110can be a network such as a Swarm Learning network. Swarm Learning can involve various stages or phases of operation including, but not limited to: initialization and onboarding; installation and configuration; and integration and training. Initialization and onboarding can refer to a process (that can be an offline process) that involves multiple entities interested in swarm-based ML to come together and formulate the operational and legal requirements of the decentralized system. This includes aspects such as but not limited to data (parameter) sharing agreements, arrangements to ensure node visibility across organizational boundaries of the entities, a consensus on the expected outcomes from the model training process. Values of configurable parameters provided by a Swarm Learning network, such as the peer-discovery nodes supplied during boot up and the synchronization frequency among nodes, are also finalized at this stage. Moreover, the common (global) model to be trained and the reward system (if applicable) can be agreed upon.

As noted above, once the initialization and onboarding phase is complete, all nodes10ofFIG.1may download and install the Swarm Learning framework48onto their respective machines. The Swarm Learning framework48may then boot up, and each node's connection to the swarm learning/swarm-based blockchain network can be initiated. As used herein, the term Swarm Learning framework48can refer to a blockchain overlay on an underlying network of connections between nodes10. The boot up process can be an ordered process in which the set of nodes designated as peer-discovery nodes (during the initialization phase) are booted up first, followed by the rest of the nodes10in the Swarm Learning network110.

With regard to the integration and training phase, the Swarm Learning framework48can provide a set of APIs that enable fast integration with multiple frameworks. These APIs can be incorporated into an existing code base for the Swarm Learning framework48to quickly transform a stand-alone ML node into a swarm learning participant. It should be understood that participant and node may be used interchangeably in describing various examples.

At a high level, model training in accordance with various examples may be described in terms of enrollment, IID agreement, IID checking, local model training, parameter sharing, parameter merging, and stopping criterion check.FIG.2Aillustrates operations that can be performed by the Swarm Learning framework48embodied by, e.g., SL architecture230in accordance with an example of the disclosed technology.

At200, enrollment occurs. That is, each node10in the Swarm Learning network110may enroll or register itself in a swarm learning contract or smart contract. This means that each node10may execute a swarm learning contract or smart contract. In one example, this can be a one-time process. In other examples, enrollment or registration may be performed after some time as a type of verification process. Each node10can subsequently record its relevant attributes in the swarm learning or smart contract, e.g., the uniform resource locator (URL) from which its own set of trained parameters can be downloaded by other nodes.

At202, an IID agreement occurs. That is, each participating node10in the swarm learning network110may execute a smart contract that includes a definition of conforming data, e.g., a definition of IID data. Executing a smart contract means that each participating node10may enroll in or agree to a smart contract that includes a definition of conforming data. In this way each participating node10that has executed a smart contract having a definition of conforming data has agreed to the definition of conforming data. Conforming data refers to data that is in compliance with a definition or set of rules or criteria. Such rules or criteria could be, for example, IID or others. Indicia of conforming data may comprise markers or factors that alone or together tend to indicate whether data is conforming data. A definition of conforming data refers to how conforming data is defined in the smart contract. In the example ofFIGS.2A and2Bconforming data is IID data but conforming data is not necessarily limited to IID data. In other examples Steps202and204ofFIG.2Aare conforming data agreeing and checking steps in which (Step202) a smart contract that is executed by a participating node10may include a definition of conforming data and (Step204) a conforming data check is run.

The definition of conforming data may be formed or provided by a data scientist, for example. This definition is subsequently used in examples of the disclosed technology to determine whether potential training data qualifies as conforming data, e.g., IID data, to be used in one or more ML models44. Accordingly at202a participating node10executes a smart contract that includes a definition of conforming data, e.g., IID data, in order to agree to the definition. The definition of IID may be a set definition as described. In examples it enables a conformance check and is given as an input that is configurable by a human operator or data science admin. It can be a simple rule or use an algorithm output to have as a bound check. Notably the definition of IID removes ambiguity around unknown or unexpected data properties. The IID agreement and checking provides a conformance check to filter out non-IID data. The definition of IID may include various IID criteria, or measures to be performed to implement IID conformance (e.g., “all values of X falling in a range from A to B is IID”). A histogram could be used as a filter to perform IID checking.

At204, an IID check occurs. More specifically, data from a potential training set or batch is checked against the definition of conforming data agreed to in202. Conforming data can be for example IID data. Accordingly the SL framework48may require participating nodes10to agree with the rules or definitions of conforming data, e.g., IID data, in order for that data to be used during an upcoming training phase.

An IID check in202can occur in various ways. As noted above one non-limiting example is for structured data. Structured data can be measured such as being bounded within the Min/Max of a univariate variable or even checking the integrity of multivariate variables by checking the correlation or covariance shifts. Another non-limiting example is for unstructured data such as images; it may be difficult or quite complex to filter the image at source. In such cases according to the disclosed technology there can be knobs or controls provided for aspects like batch training times, layer weights, etc., which might be indicators of IID data. As an example, if the training data includes height and weight of a person, then IID data might be a minimum height of 6 feet and a minimum weight of 200 pounds.

Accordingly, the IID check of204can check whether each set of training data conforms to the definition of conforming data agreed upon in202. The IID check204can also include marking/tagging and isolating non-conforming sets of training data, and inputting conforming sets of training data into a training phase of the machine learning model, as described in more detail below in connection withFIGS.3and5. The non-conforming sets or batches or portions of batches of training data may be listed in a log, or may be discarded. (It is noted that in this disclosure “set” of training data and “batch” of training data are being used interchangeably.)

Therefore the disclosed technology in examples provides a data consistency check or an explicit data configuration check in the SL framework48, as a configuration exchanged before the training phase. This can isolate data batches that do not conform to the expected IID norm and thereby keep these non-conforming data batches away from training. Thus the disclosed technology uses decentralized configuration checks to conform to Global IID requirements.

At206, local model training occurs, where each node proceeds to train a local copy of the global or common model in an iterative fashion over multiple rounds that can be referred to as epochs. Due to the IID agreement at202and the IID checking at204, the training done at206can be done with training datasets that satisfy the definition of IID data as agreed to in a smart contract during the IID agreement202. Training data may include, but is not limited to, numerical data.

During each epoch, each node10trains its local model using one or more data batches for some given number of iterations. The data batches to be used in the training are data batches that have passed the IID check at204using the agreed-upon IID definition at202. A further check to determine if parameters can be merged may be performed at208. That check can determine if the threshold number of iterations has been reached and/or whether a threshold number of participating nodes10are ready to share their respective parameters. These thresholds can be specified during the initialization phase. After the threshold number of iterations has been reached, the parameter values of each node10are exported to a file, which can then be uploaded to a shared file system for other nodes10to access. Each node10may signal the other nodes10that it is ready to share its parameters.

The merge leader may then merge the downloaded parameter files (from each swarm learning network node10). Appropriate merge mechanisms or algorithms may be used, e.g., one or more of mean merging, weighted mean merging, median merging, etc. The merge leader may combine the parameter values from all of the nodes10to create a new file with the merged parameters, and signals to the other nodes10that a new file is available. At214, each node10may obtain the merged parameters (represented in the new file) from the merge leader via the swarm API. At216, each node10may update its local version of the common model with the merged parameters. By virtue of the features of the disclosed technology including the IID agreeing and checking, a situation in which spurious weights are exchanged in the batches of training due to non-IID data being included in training sets, which can adversely affect the learning, can be avoided.

At218, a check can be performed to determine if a stopping criterion has been reached. That is, each of the nodes10evaluate the model with the updated parameter values using their local data to calculate various validation metrics. The values obtained from this operation are shared using a smart contract state variable. As each node completes this step, it signals to the Swarm Learning network110that the update and validation step is complete. In the interim, the merge leader may keep checking for an update complete signal from each node10. When it discovers that all merge participants have signaled completion, the merge leader merges the local validation metric numbers to calculate global metric numbers. This updating of the model can be thought of as a synchronization step. If the policy decided during initialization supports monetization during model building, the rewards corresponding to the contributions by each of the participants are calculated and dispensed at this point. Afterwards, the current state of the Swarm Learning network110is compared against a stopping criterion, and if it is found to be met, the Swarm Learning process ends. Otherwise, the steps of local model training with IID data, parameter sharing, parameter merging, and stopping criterion check are repeated until the criterion is fulfilled.

FIG.2Billustrates swarm learning architecture230in accordance with examples of the present disclosure. The swarm learning architecture230may be implemented by swarm learning framework48ofFIG.1. This swarm learning architecture230may include general configuration file231, IID configuration file232, and local ML models233A,233B, . . . ,233N at each node10. These local ML models233A-233N may be maintained and trained at nodes making up the swarm learning network110, e.g., edge nodes10, described above that make up blockchain network110. The local ML models233A-233N may also be stored in model store44ofFIG.1and IID agreeing and checking as described herein may be performed on training data before the training data is inputted into ML models233A-233N.

The swarm learning architecture230may include swarm learning component236which may include an IID check engine238, an API layer240, a control layer242, a data layer244, and a monetization layer246. The swarm learning component236may operate (as noted above) in a blockchain context to ensure data privacy where a blockchain platform248operates on top of a ML platform250(that is distributed amongst nodes10of a swarm learning network). The sharing of parameters and validation loss values can be performed using a blockchain ledger252, which may be an example of distributed ledger42.

It should be noted that the components or elements of swarm learning architecture230can be modular so that the technologies used in implementing them can be replaced, adjusted, adapted, etc. based on requirements. The entire framework is designed to run on both commodity and high-end machines, supporting a heterogeneous set of infrastructure in the Swarm Learning network110. It can be deployed within and across data centers, and has built-in support for a fault-tolerant network, where nodes10can exit and re-enter the Swarm Learning network110dynamically without derailing or stalling the model building process. In other words, blockchain platform252is used as an infrastructure component for implementing a swarm learning ledger (or blackboard) which can encompass the decentralized control logic for ML model building, key sharing, and parameter sharing logic. Edge nodes10(where ML models233A,233B . . . ,233N are trained) may themselves have all the infrastructure components and control logic used for controlling/managing swarm learning.

Swarm learning, in one example, can be implemented as an API library240available for multiple popular frameworks such as TensorFlow, Keras, and the like. These APIs provide an interface that is similar to the training APIs in the native frameworks familiar to data scientists. Calling these APIs automatically inserts the required “hooks” for swarm learning so that nodes10seamlessly exchange parameters at the end of each model training epoch, and subsequently continue the training after resetting the local models to the globally merged parameters.

Responsibility for keeping the Swarm Learning network110in a globally consistent state lies with the control layer242, which is implemented using blockchain technology. The control layer242ensures that all operations and the corresponding state transitions are performed in an atomic manner. Both state and supported operations are encapsulated in a blockchain smart contract. The state comprises information such as the current epoch, the current members or participants of the Swarm Learning network110, along with their IP addresses and ports, and the URIs for parameter files. The set of supported operations includes logic to elect a merge leader of the Swarm Learning network110toward the end of each epoch, fault-tolerance, and self-healing mechanisms, along with signaling among nodes for commencement and completion of various phases.

Data layer244controls the reliable and secure sharing of model parameters and validation loss values across the Swarm Learning network110. Like control layer242, data layer244is able to support different file-sharing mechanisms, such as hypertext transfer protocol secure (HTTPS) over transport layer security (TLS), interplanetary file system (IPFS), and so on. Data layer244may be controlled through the supported operations invoked by control layer242, where information about this layer may also be maintained.

FIG.3is an example computing component that may be used to implement various IID agreement and checking functions of a node in accordance with one example of the disclosed technology. Computing component300may be integrated in the SL framework236or may be separate from the SL framework236and may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. In the example implementation ofFIG.3, the computing component300includes a hardware processor302, and machine-readable storage medium304. In some examples, computing component300may be an embodiment of processor20of node or edge node10(FIG.1), and the node10may be operating in a distributed swarm learning blockchain network. In some examples, computing component300may be implemented, e.g., as the IID check engine238ofFIG.2B.

Hardware processor302may be one or more central processing units (CPUs), semiconductor-based microprocessors, and/or other hardware devices suitable for retrieval and execution of instructions stored in machine-readable storage medium304. Hardware processor302may fetch, decode, and execute instructions, such as instructions306-316, to control processes or operations for agreeing to a definition of conforming data and checking whether data is conforming, in order to determine whether the data qualifies as training data for an ML model44or233A-N. As an alternative or in addition to retrieving and executing instructions, hardware processor302may include one or more electronic circuits that include electronic components for performing the functionality of one or more instructions, such as a field programmable gate array (FPGA), application specific integrated circuit (ASIC), or other electronic circuits.

Hardware processor302may execute instruction306to execute a smart contract that sets out a definition or criteria of conforming data based on the definition or criteria of IID stored in the IID configuration file232. Accordingly, the node10agrees upon the definition of conforming data set out in the smart contract. By virtue of this agreement and smart contract process, IID conformance can be enforced. If the node10does not execute the smart contract then the node10may not participate in the parameter sharing of the SL network.

Hardware processor302may execute instruction308to receive sets or batches of training data for training a machine learning model44or233A-N. Training data can be in various forms. One non-limiting example is structured data from different verticals such as IoT, System Monitoring, Clinical Data, Health, etc.

Hardware processor302may execute instruction310to check whether each set or batch of training data conforms to the agreed-upon definition or criteria of conforming data. The IID check can take on various forms. For example it can be a data consistency check or an explicit data configuration check as a configuration exchanged before the training phase. Therefore the network110is configured such that each node10can agree to the definition of IID data and perform the IID checking before the data is included as training data in an ML model44or233A-N. One non-limiting example is for structured data. Structured data can be measured such as being bounded within the Min/Max of a univariate variable or even checking the integrity of multivariate variables by checking the correlation or covariance shifts. Another non-limiting example is for unstructured data such as images; it may be difficult or quite complex to filter the image at source. In such cases there can be knobs or controls provided for aspects like batch training times, layer weights, etc., which might be indicators of IID data.

Hardware processor302may execute instruction312to tag and isolate non-conforming sets or batches of training data. In this way non-conforming data, including either batches of data or portions of batches, can be tagged or marked and kept away from training. The isolated data can be kept in a log or on a data sheet. The non-conforming batches or sets of training data can be input into the check at a later time. Trained weights can be shared with the network, i.e., trained weights derived from training the local version of the machine learning model using the conforming batches of training data can be shared with other nodes in the network.

Hardware processor302may execute instruction314to discard the non-conforming batches of training data.

Hardware processor302may execute instruction316to output conforming sets of training data to be input into a training phase of the machine learning model44or233A-N, wherein the conforming sets of training data are IID data. Accordingly the conforming batches or sets of training data can be used to train a local version of a ML model44or233A-N at the training node. In distributed or decentralized ML networks, training of a ML model at, e.g., an edge node, may entail training an instance or version of a common, global model using training data at the edge node10. The training data may be a training data subset of local data at the edge node10.

FIG.4is an example computing component400that may be used to embody the IID check engine238ofFIG.2B, in order to perform various IID agreement and checking functions of a node10in accordance with an example of the disclosed technology. Computing component400may be integrated in the SL framework236or may be separate from the SL framework236and may be, for example, a server computer, a controller, or any other similar computing component capable of processing data. In the example implementation ofFIG.4, the computing component400includes a hardware processor402, and machine-readable storage medium404. In some examples, computing component400may be an embodiment of processor20of node or edge node10(FIG.1), and the node10may be operating in a distributed swarm learning blockchain network.

Hardware processor402may be one or more central processing units (CPUs), semiconductor-based microprocessors, and/or other hardware devices suitable for retrieval and execution of instructions stored in machine-readable storage medium404. Hardware processor402may fetch, decode, and execute instructions, such as but not limited to instructions502-520ofFIG.5, to control processes or operations for agreeing to a definition of conforming data and checking whether data is conforming, in order to determine whether the data qualifies as training data for a ML model44or233A-N. As an alternative or in addition to retrieving and executing instructions, hardware processor402may include one or more electronic circuits that include electronic components for performing the functionality of one or more instructions, such as a field programmable gate array (FPGA), application specific integrated circuit (ASIC), or other electronic circuits.

The remaining components ofFIG.4will now be described in conjunction of the method ofFIG.5which illustrates a method500for performing IID agreement and checking functions of a node10in accordance with an example of the disclosed technology. Step502may be performed for example by conformance configuration module406and includes receiving a smart contract that sets out a definition or criteria of conforming data based on the definition or criteria of IID stored in the IID configuration file232or408. The definition or criteria for IID may be created by, e.g., a data scientist or human admin who then incorporates it into the smart contract. Accordingly, the node10agrees upon the definition of conforming data set out in the smart contract. By virtue of this agreement, IID conformance can be enforced. If the node10does not execute the smart contract then the node10may not participate in the parameter sharing of the SL network.

Step504, which also may be performed by the conformance configuration module406, determines whether the node10has agreed to the smart contract which includes the definition or criteria for IID. If NO then in Step506the node10is designated as a node10that does not participate in IID enforcement. If YES then in Step508, which may be performed by the receiving module410, the node10receives a batch of training data for training a machine learning model44or233A-N. Training data can be in various forms. One non-limiting example is structured data from different verticals such as IoT, System Monitoring, Clinical Data, Health, etc.

In Step510, which may be performed by detect module412, the node10checks the batch of training data against the IID definition or criteria set out in the smart contract, in order to check whether the batch of training data conforms to the agreed-upon definition or criteria of conforming data. One non-limiting example is for structured data. Structured data can be measured like being bounded within the Min/Max of a univariate variable or even checking the integrity of multivariate variables by checking the correlation or covariance shifts. Another non-limiting example is for unstructured data such as images; it may be difficult or quite complex to filter the image at source. In such cases there can be knobs provided for aspects like batch training times, layer weights, etc., which might be indicators of IID data.

In Step512, which also may be performed by the detect module412, the node10determines whether the batch of training data qualifies as IID. If YES then in Step514, which may be performed by output module414, the training data is inputted into a training phase of a ML. Accordingly the conforming batches or sets of training data can be used to train a local version of a ML model44or233A-N at the training node. In distributed or decentralized ML networks, training of a ML model at, e.g., an edge node, may entail training an instance or version of a common, global model using training data at the node10. The training data may be a training data subset of local data at the node10.

If the inquiry at Step512is NO then in Step516the data is tagged as non-conforming. In Step518non-conforming data is isolated and kept away from training. Steps516and518may be performed by tag and isolate module416. Data that is tagged and/or isolated as non-conforming may be logged into a log or data sheet stored in non-IID database418. In Step520, which may be performed by discard module420, non-conforming data is discarded and outputted to discard database422.

FIG.6depicts a block diagram of an example computer system600in which various of the examples described herein may be implemented, including but not limited to node10, SL framework with IID check48, swarm learning component236, IID check engine238, and computing components300and400. The computer system600includes a bus602or other communication mechanism for communicating information, one or more hardware processors604coupled with bus602for processing information. Hardware processor(s)604may be, for example, one or more general purpose microprocessors.

The computer system600also includes a main memory606, such as a random access memory (RAM), cache and/or other dynamic storage devices, coupled to bus602for storing information and instructions to be executed by processor604. Main memory606also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor604. Such instructions, when stored in storage media accessible to processor604, render computer system600into a special-purpose machine that is customized to perform the operations specified in the instructions.

The computer system600further includes a read only memory (ROM)608or other static storage device coupled to bus602for storing static information and instructions for processor604. A storage device610, such as a magnetic disk, optical disk, or USB thumb drive (Flash drive), is provided and coupled to bus602for storing information and instructions.

The computer system600may be coupled via bus602to a display612, such as a liquid crystal display (LCD) (or touch screen), for displaying information to a computer user. An input device614, including alphanumeric and other keys, is coupled to bus602for communicating information and command selections to processor604. Another type of user input device is cursor control616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor604and for controlling cursor movement on display612. In some examples, the same direction information and command selections as cursor control may be implemented via receiving touches on a touch screen without a cursor.

The computer system600also includes a communication interface618coupled to bus602. Communication interface618provides a two-way data communication coupling to one or more network links that are connected to one or more local networks. For example, communication interface618may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface618may be a local area network (LAN) card to provide a data communication connection to a compatible LAN (or WAN component to communicated with a WAN). Wireless links may also be implemented. In any such implementation, communication interface618sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

The computer system600can send messages and receive data, including program code, through the network(s), network link and communication interface618. In the Internet example, a server might transmit a requested code for an application program through the Internet, the ISP, the local network and the communication interface618.