DEVICE, METHOD, AND SYSTEM FOR WEIGHTED KNOWLEDGE TRANSFER

Aspects relate to a privacy preserving public machine learning model that achieves high performance while maintaining data privacy. Further aspects relate to a weighted knowledge transfer device including a feature determination unit to generate a public knowledge transfer dataset and a private knowledge transfer dataset; a data selection unit to generate, based on a similarity calculation of the public knowledge transfer dataset and the private knowledge transfer dataset, a public training dataset and a similarity weight vector; a machine learning model management unit to generate, by processing the public training dataset with a set of machine learning models trained based on the private knowledge transfer dataset, a public label vector that indicates labels for the set of public features; and a knowledge transfer unit to generate a public machine learning model based on the weight vector, the public training dataset, and the public label vector.

TECHNICAL FIELD

The present disclosure generally relates to machine learning techniques, and more particularly relates to a weighted knowledge transfer technique for creating a privacy preserving machine learning model.

BACKGROUND ART

In recent years, machine-learning approaches for data analysis have been widely explored for recognizing patterns which, in turn, allow for extraction of useful information and insights. Machine learning techniques include algorithms that may be trained to make generalizations based on training data with known outcomes. Once trained, these machine learning algorithms may then be applied to predict the outcome in cases of unknown outcome.

Machine-learning approaches, which may include neural networks, hidden Markov models, belief networks, support vector machines, and the like, are ideally suited for domains characterized by the existence of large amounts of data, noisy patterns and the absence of general theories, and have been applied to a variety of fields including healthcare, finance, and insurance.

Some machine learning applications involve the use of training data that is sensitive, such as the medical histories of patients in a clinical trial. A machine learning model trained on such training data may inadvertently and implicitly store some of this sensitive information, such that careful analysis of the trained model may lead to privacy risks in which sensitive information is obtained by unauthorized actors.

Accordingly, in view of such challenges, methods have been proposed for protecting the privacy of training data. For example, Papernot et al. (Non-patent Document 1) discloses “To address this problem, we demonstrate a generally applicable approach to providing strong privacy guarantees for training data: Private Aggregation of Teacher Ensembles (PATE). The approach combines, in a black-box fashion, multiple models trained with disjoint datasets, such as records from different subsets of users. Because they rely directly on sensitive data, these models are not published, but instead used as “teachers” for a “student” model. The student learns to predict an output chosen by noisy voting among all of the teachers, and cannot directly access an individual teacher or the underlying data or parameters.

The student's privacy properties can be understood both intuitively (since no single teacher and thus no single dataset dictates the student's training) and formally, in terms of differential privacy. These properties hold even if an adversary can not only query the student but also inspect its internal workings. Compared with previous work, the approach imposes only weak assumptions on how teachers are trained: it applies to any model, including non-convex models like DNNs. We achieve state-of-the-art privacy/utility trade-offs on MNIST and SVHN thanks to an improved privacy analysis and semi-supervised learning.”

CITATION LIST

Non Patent Literature

NPL 1: Papernot et al. “Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data.” International Conference on Learning Representations 2017. 2017.

SUMMARY OF INVENTION

Technical Problem

Non-Patent Document 1 discloses a technique in which a “student” machine learning model is trained not directly on sensitive training data, but instead based on multiple “teacher” models that are each trained on a portion of the sensitive training data. Since these teacher models are not made public, and the student model is not dependent on one single student model or one single data set, information regarding the sensitive training data cannot be extracted from the teacher model by an unauthorized actor. In this way, privacy of the sensitive training data can be facilitated.

The technique disclosed in Non-Patent Document 1, however, lacks the ability to transfer knowledge from a private model to a public model where great variability exists between the private data set used to train the private model and the public data set used to train the public model. For instance, Non-Patent Document 1 does not disclose techniques for generating a knowledge transfer dataset that can be used in the creation of a privacy preserving machine learning model. As a result, Non-Patent Document 1 does not provide a technique for conveying the characteristics of features in private datasets that can be used in the creation of a privacy preserving machine learning model.

Accordingly, it is an object of the present disclosure to provide a device, method, and system for performing a weighted knowledge transfer from a private machine learning model to a public machine learning model in order to create a privacy preserving public machine learning model that achieves high performance while maintaining data privacy.

Solution to Problem

One representative example of the present disclosure relates to a weighted knowledge transfer device including a data selection unit configured to generate, based on a similarity calculation of a public dataset and a private dataset: a subset of the public dataset that achieves a similarity threshold with respect to the private dataset, and a similarity weight vector that indicates weights of a set of public features included in the subset of the public dataset; a machine learning model management unit configured to generate, by processing the subset of the public dataset with a set of machine learning models trained based on the private dataset, a public label vector that indicates labels for the set of public features; and a knowledge transfer unit configured to generate a public machine learning model based on the weight vector, the subset of the public dataset, and the public label vector.

Advantageous Effects of Invention

According to the present disclosure it is possible to provide a device, method, and system for performing a weighted knowledge transfer from a private machine learning model to a public machine learning model in order to create a privacy preserving public machine learning model that achieves high performance while maintaining data privacy.

Problems, configurations, and effects other than those described above will be made clear by the following description in the embodiments for carrying out the invention.

DESCRIPTION OF EMBODIMENT(S)

Hereinafter, embodiments of the present invention will be described with reference to the Figures. It should be noted that the embodiments described herein are not intended to limit the invention according to the claims, and it is to be understood that each of the elements and combinations thereof described with respect to the embodiments are not strictly necessary to implement the aspects of the present invention.

Various aspects are disclosed in the following description and related drawings. Alternate aspects may be devised without departing from the scope of the disclosure. Additionally, well-known elements of the disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the disclosure.

Further, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that various actions described herein can be performed by specific circuits (e.g., an application specific integrated circuit (ASIC)), by program instructions being executed by one or more processors, or by a combination of both. Additionally, the sequence of actions described herein can be considered to be embodied entirely within any form of computer readable storage medium having stored therein a corresponding set of computer instructions that upon execution would cause an associated processor to perform the functionality described herein. Thus, the various aspects of the disclosure may be embodied in a number of different forms, all of which have been contemplated to be within the scope of the claimed subject matter.

In domains such as healthcare, finance, and insurance, for example, protecting the privacy of personal data is of great importance. Unintended data leaks could have detrimental consequences both for those companies that store the data as well as the individuals from whom the data was collected. As the number of companies offering machine learning services (personal health prediction, financial risk prediction) increase, the need for protecting the privacy of personal data also increases.

In some cases, machine learning applications may involve the use of training data that is sensitive, such as the medical histories of patients in a clinical trial. Those individuals that contribute their personal data for use in training machine learning models do so in good faith, believing that machine learning models that can be accessed by third parties will not expose any of their personal information.

However, malicious actors can hack machine learning models and obtain the personal information of those individuals who contributed the private data that was used to train said machine learning models.

In view of such challenges, privacy preserving configurations can be deployed to prevent leaking private data to malicious actors through the machine learning model. However, using healthcare data as an example, due to the varied nature of private and public data sources, it is challenging to create a privacy preserving machine learning model that achieves similar predictive performance to machine learning models that have been directly trained on private data.

For instance, in order to create a user-facing, privacy preserving, public machine learning model that achieves comparable performance to a private machine learning model, knowledge needs to be transferred from the private model to the public model. Conventional methods, however, lack the ability to transfer knowledge from a private model to a public model where great variability exists between a private data set used to train the private model and a public data set used to train the public model. For instance, conventional methods do not disclose techniques for generating a knowledge transfer dataset that can be used in the creation of a privacy preserving machine learning model. As a result, conventional methods do not provide techniques for conveying the characteristics of features in private datasets that can be used in the creation of a privacy preserving machine learning model.

Accordingly, aspects of the present disclosure are directed to addressing the above challenges by creating a public dataset that resembles a private dataset, selecting a set of target features between the private dataset and the public dataset to create a knowledge transfer dataset, allocating weights to convey characteristics found in the private dataset via the knowledge transfer dataset, and performing a weighted knowledge transfer from a private to a public model based on the knowledge transfer dataset and the allocated weights.

Additional aspects of the disclosure relate to determining a partitioning scheme of public data and parameter configurations for a set of machine learning models to optimize knowledge transfer performance. Additional aspects of the disclosure relate to determining a set of thresholds to select a set of public training data from a public knowledge transfer dataset, training a plurality of machine learning models on selected subsets of public data, and selecting thresholds and parameter configurations to optimize the knowledge transfer capacity and the performance of the public model.

In this way, according to the present disclosure, it is possible to provide a device, method, and system for performing a weighted knowledge transfer from a private machine learning model to a public machine learning model in order to create a privacy preserving public machine learning model that achieves high performance while maintaining data privacy.

Turning now to the Figures,FIG.1depicts a high-level block diagram of a computer system300for implementing various embodiments of the present disclosure, according to embodiments. The mechanisms and apparatus of the various embodiments disclosed herein apply equally to any appropriate computing system. The major components of the computer system300include one or more processors302, a memory304, a terminal interface312, a storage interface314, an I/O (Input/Output) device interface316, and a network interface318, all of which are communicatively coupled, directly or indirectly, for inter-component communication via a memory bus306, an I/O bus308, bus interface unit309, and an I/O bus interface unit310.

The computer system300may contain one or more general-purpose programmable central processing units (CPUs)302A and302B, herein generically referred to as the processor302. In embodiments, the computer system300may contain multiple processors; however, in certain embodiments, the computer system300may alternatively be a single CPU system. Each processor302executes instructions stored in the memory304and may include one or more levels of on-board cache.

In embodiments, the memory304may include a random-access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing or encoding data and programs. In certain embodiments, the memory304represents the entire virtual memory of the computer system300, and may also include the virtual memory of other computer systems coupled to the computer system300or connected via a network. The memory304can be conceptually viewed as a single monolithic entity, but in other embodiments the memory304is a more complex arrangement, such as a hierarchy of caches and other memory devices. For example, memory may exist in multiple levels of caches, and these caches may be further divided by function, so that one cache holds instructions while another holds non-instruction data, which is used by the processor or processors. Memory may be further distributed and associated with different CPUs or sets of CPUs, as is known in any of various so-called non-uniform memory access (NUMA) computer architectures.

The memory304may store all or a portion of the various programs, modules and data structures for processing data transfers as discussed herein. For instance, the memory304can store a weighted knowledge transfer application350. In embodiments, the weighted knowledge transfer application350may include instructions or statements that execute on the processor302or instructions or statements that are interpreted by instructions or statements that execute on the processor302to carry out the functions as further described below.

In certain embodiments, the weighted knowledge transfer application350is implemented in hardware via semiconductor devices, chips, logical gates, circuits, circuit cards, and/or other physical hardware devices in lieu of, or in addition to, a processor-based system. In embodiments, the weighted knowledge transfer application350may include data in addition to instructions or statements. In certain embodiments, a camera, sensor, or other data input device (not shown) may be provided in direct communication with the bus interface unit309, the processor302, or other hardware of the computer system300. In such a configuration, the need for the processor302to access the memory304and the latent factor identification application may be reduced.

The computer system300may include a bus interface unit309to handle communications among the processor302, the memory304, a display system324, and the I/O bus interface unit310. The I/O bus interface unit310may be coupled with the I/O bus308for transferring data to and from the various I/O units. The I/O bus interface unit310communicates with multiple I/O interface units312,314,316, and318, which are also known as I/O processors (IOPs) or I/O adapters (IOAs), through the I/O bus308. The display system324may include a display controller, a display memory, or both. The display controller may provide video, audio, or both types of data to a display device326. Further, the computer system300may include one or more sensors or other devices configured to collect and provide data to the processor302.

As examples, the computer system300may include biometric sensors (e.g., to collect heart rate data, stress level data), environmental sensors (e.g., to collect humidity data, temperature data, pressure data), motion sensors (e.g., to collect acceleration data, movement data), or the like. Other types of sensors are also possible. The display memory may be a dedicated memory for buffering video data. The display system324may be coupled with a display device326, such as a standalone display screen, computer monitor, television, or a tablet or handheld device display.

In one embodiment, the display device326may include one or more speakers for rendering audio. Alternatively, one or more speakers for rendering audio may be coupled with an I/O interface unit. In alternate embodiments, one or more of the functions provided by the display system324may be on board an integrated circuit that also includes the processor302. In addition, one or more of the functions provided by the bus interface unit309may be on board an integrated circuit that also includes the processor302.

The I/O interface units support communication with a variety of storage and I/O devices. For example, the terminal interface unit312supports the attachment of one or more user I/O devices320, which may include user output devices (such as a video display device, speaker, and/or television set) and user input devices (such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device). Auser may manipulate the user input devices using a user interface in order to provide input data and commands to the user I/O device320and the computer system300, and may receive output data via the user output devices. For example, a user interface may be presented via the user I/O device320, such as displayed on a display device, played via a speaker, or printed via a printer.

The storage interface314supports the attachment of one or more disk drives or direct access storage devices322(which are typically rotating magnetic disk drive storage devices, although they could alternatively be other storage devices, including arrays of disk drives configured to appear as a single large storage device to a host computer, or solid-state drives, such as flash memory). In some embodiments, the storage device322may be implemented via any type of secondary storage device. The contents of the memory304, or any portion thereof, may be stored to and retrieved from the storage device322as needed. The I/O device interface316provides an interface to any of various other I/O devices or devices of other types, such as printers or fax machines. The network interface318provides one or more communication paths from the computer system300to other digital devices and computer systems; these communication paths may include, for example, one or more networks330.

Although the computer system300shown inFIG.1illustrates a particular bus structure providing a direct communication path among the processors302, the memory304, the bus interface309, the display system324, and the I/O bus interface unit310, in alternative embodiments the computer system300may include different buses or communication paths, which may be arranged in any of various forms, such as point-to-point links in hierarchical, star or web configurations, multiple hierarchical buses, parallel and redundant paths, or any other appropriate type of configuration. Furthermore, while the I/O bus interface unit310and the I/O bus308are shown as single respective units, the computer system300may, in fact, contain multiple I/O bus interface units310and/or multiple I/O buses308. While multiple I/O interface units are shown which separate the I/O bus308from various communications paths running to the various I/O devices, in other embodiments, some or all of the I/O devices are connected directly to one or more system I/O buses.

Next, an example configuration of a weighted knowledge transfer system according to embodiments of the present disclosure will be described with reference toFIG.2.

FIG.2illustrates an example configuration of a weighted knowledge transfer system100, according to embodiments. As illustrated inFIG.2, the weighted knowledge transfer system100primarily includes a private device101, a weighted knowledge transfer device104, and a public device106. The private device101, the weighted knowledge transfer device104, and the public device106may be communicably connected via a communication network such as a local area network (LAN) or the Internet.

The private device101is a storage device configured to store a private dataset102and a private machine learning model103trained on the private dataset102. For example, the private device101may include a collection of hard disk drives, solid state drives, or cloud-based storage repositories configured to store the private dataset102and the private machine learning model103.

The private dataset102may include a collection of data that contains confidential information. For example, the private dataset102may include information regarding medical records, financial transactions, or personal data (names, addresses, passwords, bank account information) for one or more individuals, businesses, or other organizations.

The private machine learning model103may include a machine learning model that has been trained using the private dataset102. For example, the machine learning model may be a neural network that has been trained to predict health risks for patients based on the private dataset102.

The private device101may be maintained in a private network of an individual, business, or other organization. For example, the private device101may belong to a hospital. In embodiments, the private device101may be connected to a weighted knowledge transfer device104via interface110. The private device101may be insulated from the public device106by the weighted knowledge transfer device104(that is, the private device101may be inaccessible from the public device106). Accordingly, users113accessing the public machine learning model107through the interface unit109of the public device106cannot retrieve the private dataset102through malicious acts (e.g., hacking) because the public machine learning model107has been trained solely using the public dataset108.

The public device106is a storage device configured to store a public dataset108and a public machine learning model107trained on the public dataset102. For example, the public device106may include a collection of hard disk drives, solid state drives, or cloud-based storage repositories configured to store the private dataset108and the public machine learning model107.

The public dataset108may include a collection of data that contains public information. For example, the public dataset108may include information regarding medical records or financial transactions that are not associated with any particular individual or entity.

The public machine learning model107may include a machine learning model that has been created using the weighted knowledge transfer unit105based on the public dataset108and the private dataset102. For example, the machine learning model may be a neural network that has been trained to predict the occurrence of health risks based on the presence of particular health factors included in the public dataset108.

In embodiments, the public machine learning model107may be accessible to users113via an interface unit109. For example, the interface unit109may include a server module configured to provide access to the public machine learning model107as a service (e.g., via a subscription-based software application or the like). Users may access the public machine learning model107via the interface unit109to obtain insights provided by the public machine learning model107.

The weighted knowledge transfer device104is a storage device configured to store one or more functional units used to perform the weighted knowledge transfer process according to the present disclosure. As illustrated inFIG.2, the weighted knowledge transfer device104may include a weighted knowledge transfer unit105. The weighted knowledge transfer unit105is a functional unit configured to perform a weighted knowledge transfer from the private machine learning model103to the public machine learning model107in order to create a privacy preserving public machine learning model that achieves high performance while maintaining data privacy.

It should be noted thatFIG.2illustrates a simplified configuration of the weighted knowledge transfer system100, and the weighted knowledge transfer system100is not limited to the configuration illustrated inFIG.2. For instance, in addition to the weighted knowledge transfer unit105, the weighted knowledge transfer device104may include a feature determination unit, a data selection unit, a partition unit, and a random noise generator as illustrated inFIG.3,FIG.4, andFIG.5.

The weighted knowledge transfer unit105may access the private dataset102through an interface110, and access the public dataset108through an interface112to create the public machine learning model107through the interface111. As the details of the weighted knowledge transfer unit105will be described later, the description thereof will be omitted here.

The weighted knowledge transfer system100according to the present disclosure may be applied to a variety of domains. Below, an example will be considered in which the weighted knowledge transfer system100is applied to a healthcare domain.

In embodiments, the private device101may be a server deployed within a private network managed by a healthcare facility (e.g., an entity subject to the Health Insurance Portability and Accountability Act). The private dataset102may include electronic health records of patients in the care of the healthcare facility. These electronic health records may contain personal information that should not be accessible to unauthorized entities or shared without the consent of the patients. The private machine learning model103may be trained to predict risks such as hospital readmission risks or mortality risks based on the private data set102. The predictions made by the private machine learning model103may be used by healthcare professionals to take appropriate actions to improve the well-being of patients.

In this case, the weighted knowledge transfer unit105stored on the weighted knowledge transfer device104may access the private dataset102containing the electronic health records via the interface110, and access a public dataset108containing publicly available healthcare information (e.g., a Medical Information Mart for Intensive Care dataset) via the interface112. As will be described later, the weighted knowledge transfer unit105uses the public dataset108and the private dataset102to generate the public machine learning model107. As described herein, the public machine learning model107has comparable performance to the private machine learning model103, but is trained on the public dataset108such that sensitive information present in the private dataset102is not accessible to unauthorized users even in the event of a cyberattack.

The weighted knowledge transfer system100illustrated inFIG.2makes it possible to perform a weighted knowledge transfer from a private machine learning model to a public machine learning model in order to create a privacy preserving public machine learning model that achieves high performance while maintaining data privacy. The weighted knowledge transfer system100may provide benefits associated with data privacy and machine learning model performance.

Next, an example logical configuration of the weighted knowledge transfer device will be described with reference toFIG.3.

FIG.3illustrates an example logical configuration of a weighted knowledge transfer device104, according to embodiments. The weighted knowledge transfer device104may be used to generate a privacy preserving public machine learning model that achieves high performance while maintaining data privacy by performing a weighted knowledge transfer from a private machine learning model to a public machine learning model.

First, feature determination unit203analyzes the private dataset102and the public dataset108to determine a set of target features. The set of target features may include a collection of features that are shared between the private dataset102and the public dataset108. In embodiments, the feature determination unit203may determine the set of target features by using one or more of a variety of statistical analysis techniques with respect to the private data to ascertain features that are shared between the private dataset102and the public dataset108. In embodiments, the feature determination unit203may determine the set of target features by using natural language processing based methods.

Subsequently, the feature determination unit203may output a private knowledge transfer dataset204and a public knowledge transfer dataset205that both include the set of target features. The private knowledge transfer dataset204and the public knowledge transfer dataset205are subsets of the private dataset102and the public dataset108, respectively, that both contain the set of target features determined by the feature determination unit.

Next, data selection unit206inputs the private knowledge transfer dataset204and the public knowledge transfer dataset205, and generates, based on a similarity calculation of the private knowledge transfer dataset204and the public knowledge transfer dataset205, a public training dataset207which is a subset of the public knowledge transfer dataset205that achieves a similarity threshold with respect to the private knowledge transfer dataset204, and a similarity weight vector208that indicates weights of a set of public features included in the public training dataset207.

In order to facilitate an efficient knowledge transfer it is desirable for the feature space coverage of the public knowledge transfer dataset205to be substantially equivalent to the feature space of the private knowledge transfer dataset204. Accordingly, for the feature space A and B′ of the private knowledge transfer dataset204and the public knowledge transfer dataset205, respectively, the data selection unit206generates a public training dataset207that approximates the feature space coverage of the private knowledge transfer dataset204.

As described herein, in embodiments, the data selection unit206may generate the public training dataset207and the similarity weight vector208based on a similarity calculation of the private knowledge transfer dataset204and the public knowledge transfer dataset205. For example, the data selection unit206may calculate the similarity of each feature in the public knowledge transfer dataset205with respect to each feature in the private knowledge transfer dataset204. Here, the similarity may be calculated using a distance calculation including the Euclidean, Manhattan, Chebyshev, or Mahalanobis methods. Each feature in the public knowledge transfer dataset205may be annotated with a calculated similarity score which can be normalized to fall within the range of 0 to 1. These scores are then attached to each feature in the public knowledge transfer dataset205and output as the weight vector208. Based on a threshold which can either be set by the user or determined in a feedback loop from the weighted knowledge transfer unit209, the set of features (e.g., the set of public features) to be included in the public training dataset207may be determined.

In embodiments, the similarity between the private knowledge transfer dataset204and the public knowledge transfer dataset205may be determined by statistical methods such as propensity scoring matching methods and clustering methods such as k-means clustering. Further, in embodiments, the similarity between the private knowledge transfer dataset204and the public knowledge transfer dataset205may be determined with information theoretic means such as the Kullback-Leibler divergence method or various measures of entropy.

In embodiments, the similarity between the private knowledge transfer dataset204and the public knowledge transfer dataset205may be determined using machine learning model-based similarity. As an example, a new private machine learning model may be created using the private knowledge transfer unit204. This newly created machine learning model may be used to perform a particular prediction task (e.g., classifying patients into various risk groups). The machine learning model may calculate the probability of a particular set of features (e.g., patient) based on data from the private knowledge transfer dataset204, assign a probability of the set of features belonging to each of a number of groups, and choose the final group label based on a statistical decision-making method.

The trained private machine learning models created using the private knowledge transfer dataset204implicitly contains knowledge about the private knowledge transfer dataset204which is encoded in the model structure, such as the internal weights of a neural network or the node parameters of a decision tree model. This knowledge is used to measure the similarity of features in the public knowledge transfer dataset205by instructing the trained private machine learning model to make predictions using those features. Each set of features in the public knowledge transfer dataset205may be assigned a probability of belonging to each group of a set of groups. Based on these output probabilities the similarity of a sample in the public knowledge transfer dataset205to the private knowledge transfer dataset204can be inferred with statistical decision-making methods such as measuring the entropy of the output probability distribution and categorizing low entropy samples to be similar to the private knowledge transfer dataset204whereas high entropy samples are categorized as dissimilar to the private knowledge transfer dataset204. As described herein, the entropy measures can be normalized to a range of 0 to 1 and converted to a weight vector208that is output by the data selection unit206.

In embodiments, the similarity between the private knowledge transfer dataset204and the public knowledge transfer dataset205may be determined by labeling sets of features in the private knowledge transfer dataset204as belonging to the private knowledge transfer dataset204, and labeling sets of features in the public knowledge transfer dataset205as belonging to the public knowledge transfer dataset205. Subsequently, the private knowledge transfer dataset204and the public knowledge transfer dataset205may be merged into a single dataset, and a discriminator model may be used to distinguish the data by calculating the likelihood of a particular set of features to either belong to the private knowledge transfer dataset204or the public knowledge transfer dataset205.

Once the discriminator model has been trained, the trained discriminator model may be used to process the public knowledge transfer dataset205and output for each set of features a probability of this set of features belonging to the private knowledge transfer dataset204. Sets of features with a probability that achieves a probability threshold may be selected for inclusion in the public training dataset207. The calculated probabilities may be output as the weight vector208by the data selection unit206. Here, the probability threshold can either be set by a user or determined in a feedback loop from the weighted knowledge transfer unit209.

Further, in embodiments, the data selection unit206may generate the public training dataset207and the similarity weight vector208using techniques such as generative adversarial networks. A generator network may be trained to generate sets of generated features similar to the features of the private knowledge transfer dataset204. A discriminator network may be trained to distinguish between the sets of generated features and a set of private features included in the private knowledge transfer dataset204(e.g., real features). After training, the discriminator network may be used to evaluate the set of generated features generated by the trained generator network to calculate a probability of a set of generated features belonging to the private knowledge transfer dataset204. Subsequently, the discriminator network may select, as the public training dataset207, a subset of the set of generated features that are associated with a probability of belonging to the private knowledge transfer dataset204that exceeds a first probability threshold. The calculated probabilities may be output as the weight vector208by the data selection unit206.

The weighted knowledge transfer unit105may use the private knowledge transfer dataset204, the public knowledge transfer dataset205, the public training dataset207, and the weight vector208to create a public machine learning model107. As described herein, access to the public machine learning model107may be provided to users113via the interface unit109. As the details of the weighted knowledge transfer unit105will be described later, the description thereof will be omitted here.

Next, an example of selecting a set of target features for use in creation of the privacy preserving public machine learning model will be described with reference toFIG.4.

FIG.4illustrates an example of selecting a set of target features for use in creation of the privacy preserving public machine learning model, according to embodiments. As described herein, the feature determination unit203according to the present disclosure analyzes the private dataset102and the public dataset108to determine a set of target features that are shared between the private dataset102and the public dataset108. Subsequently, the feature determination unit203may output a private knowledge transfer dataset204and a public knowledge transfer dataset205that both include the set of target features.

Below, an example of determining the set of target features for outputting the private knowledge transfer dataset204and the public knowledge transfer dataset205will be described in the context of a healthcare application.

In embodiments, as illustrated inFIG.4, the private dataset102may include a description table that lists a set of private features402and their respective units of measurement403(e.g., information available in the electronic health records of a healthcare system). Similarly, the public dataset108may include a description table that lists a set of public features408and their respective units of measurement409. In embodiments, the feature extraction unit203may be configured to generate these description tables from unstructured data included in the private dataset102and the public dataset108respectively to facilitate feature determination.

For each feature of the set of private features402in the data description table of the private dataset102, the feature determination unit203may transmit a query405to ascertain the presence or absence of this feature in the set of public features408in the description table of the public dataset108, and subsequently receive a response406indicating the presence or absence of this feature in the set of public features408in the description table of the public dataset108. Here, the comparison between the set of private features402and the set of public features408may be performed using a natural language processing unit to parse the semantic or syntactic similarity between particular features.

In the example illustrated inFIG.4, for example, the set of private features402may include a feature of “albumin” that is measured in units of “g/dL”. The feature determination unit203may determine based on the response406that the feature of “albumin” is available in the set of public features408, but depending on the record is measured in difference scales of “g/dL” and “mg/dL.” In embodiments, the feature determination unit203may determine that the feature of “albumin” in the set of public features408that is measured in units of “g/dL” achieves a higher similarity with respect to the feature of “albumin” in the set of private features402that is measured in units of “g/dL,” and determine this feature of “albumin” that is measured in units of “g/dL” as a target feature that can be used in the knowledge transfer process.

In the case that units of measurement cannot be found for the features stored in the data description tables, statistical methods based on distance or similarity measures may be used to determine if a feature available in both the private dataset102and the public dataset108can be used in the knowledge transfer process. As an example, the feature determination unit203may acquire probability density functions410for a particular feature (e.g., “albumin”) based on measurement frequency counts in the public dataset108and the private dataset102.

Subsequently, the feature determination unit203may use a determination threshold based on the Kullback-Leibler divergence, for instance, to determine if the particular feature is measured on the same scale in both the private dataset102and the public dataset108. For instance, in the case that the distributions do not satisfy the determination threshold, as illustrated in graph411, the feature determination unit203may determine that the particular feature is measured on different scales between the private dataset102and the public dataset108, and exclude it as a target feature. In contrast, in the case that the distributions satisfy the determination threshold, as illustrated in graph412, the feature determination unit203may determine that the particular feature is measured on the same scale between the private dataset102and the public dataset108, and include it as a target feature. In certain embodiments, the feature determination unit203may be configured to perform unit conversions on particular features to facilitate comparison of the set of private features402and the set of public features408.

In this way, a set of target features for use in creation of the privacy preserving public machine learning model can be determined.

Next, an example logical configuration of a weighted knowledge transfer unit for creating the privacy preserving public machine learning model will be described with reference toFIG.5.

FIG.5illustrates an example logical configuration of a weighted knowledge transfer unit for creating the privacy preserving public machine learning model, according to embodiments. As described herein, the weighted knowledge transfer unit uses the private knowledge transfer dataset204, the public training dataset207, and the weight vector208as inputs to generate a privacy preserving public machine learning model107that can be accessed by users113via an interface unit109.

Partition unit502divides the private knowledge transfer dataset204into a plurality of partitions503. Here, a partition refers to a portion of the private knowledge transfer dataset that includes a mutually exclusive set of private features with respect to the other partitions of the plurality of partitions503. In this way, data corresponding to one set of features (e.g., one patient in a healthcare context) is not distributed among multiple partitions but randomly assigned and maintained exclusively in one partition.

Next, machine learning model management unit504trains a set of machine learning models using the plurality of partitions503in order to generate a set of trained private machine learning models507. In embodiments, the machine learning model management unit504trains each of the set of machine learning models based on a separate portion of the plurality of partitions503, such that no data from other partitions is used. In this way, the performance of each machine learning model is optimized using only the data included in one partition of the plurality of partitions503.

Next, machine learning model management unit504generates, by processing the public training dataset207with the set of trained private machine learning models507, a public label vector511that indicates labels for the set of public features included in the public training dataset207. More particularly, the machine learning model management unit504receives the public knowledge transfer dataset207as input, and uses the set of trained private machine learning models507to perform a machine learning task (e.g., a prediction task, a classification task, a detection task) on each set of features in the public training dataset207. As an example, in a healthcare context, the machine learning model management unit504may use the set of trained private machine learning models507to predict a risk group label for each set of features (e.g., patient) included in the public training dataset207. For each of a set of possible output labels, each of the set of trained private machine learning models507assigns a probability to each label for each set of features in the public training dataset207. Next, using a statistical decision-making method such as maximum likelihood, each trained machine learning model assigns a label to each set of features.

As each of the sets of features in the public training dataset207are processed by each trained machine learning model of the set of trained private machine learning models507, each set of features is assigned a plurality of labels508. Accordingly, the machine learning model management unit504aggregates the label counts for each set of features and adds random noise from a random noise generator510. Next, the machine learning model management unit504selects the label having the majority of counts as the final output label for this set of features of the public training dataset207. The addition of random noise reduces the likelihood of a tie in which multiple candidate labels have the same count number. However, in the event of a tie in which multiple candidate labels have the same count number, a label may be chosen at random as the final output label. By performing this labeling process for each set of features in the public training dataset207, the machine learning model management unit504can generate the public label vector511that indicates labels for each of the set of public features included in the public training dataset207.

Next, the machine learning unit512uses the public training dataset207, the similarity weight vector208, and the public label vector511to create and train the public machine learning model107. As described herein, the public machine learning model107has comparable performance to a private machine learning model trained on private datasets, but is trained on the public training dataset207such that sensitive information present in the private datasets is not accessible to unauthorized users even in the event of a cyberattack.

In embodiments, the machine learning unit512may utilize a mapping function that adjusts the priority of sets of features in the public training dataset207based on their corresponding weights in the similarity weight vector208. For instance, the public machine learning model107being trained by the machine learning unit512may use the sets of features in the public training dataset207as input and the labels in the public label vector511as the prediction target. The performance of the public machine learning model107is optimized by minimizing a loss function. Further, as part of the training process, the weights of the sets of features of the public training dataset207in the similarity weight vector208may be adjusted through a feedback loop between the machine learning unit512and the data selection unit206.

Upon completion of the training process, the public machine learning model107created by the machine learning unit512may be made publicly accessible as a service to users113via an interface unit109.

The weighted knowledge transfer unit configuration illustrated inFIG.5makes it possible to perform a weighted knowledge transfer from a private machine learning model to a public machine learning model in order to create a privacy preserving public machine learning model that achieves high performance while maintaining data privacy. The weighted knowledge transfer configuration illustrated inFIG.5may provide benefits associated with data privacy and machine learning model performance.

Next, the knowledge transfer dataset creation process according to the present disclosure will be described with reference toFIG.6.

FIG.6illustrates a flowchart of a knowledge transfer dataset creation process600, according to embodiments. The knowledge transfer dataset creation process600is a process for creating the knowledge transfer datasets (e.g., the private knowledge transfer dataset204and the public knowledge transfer dataset205illustrated inFIG.3) according to the present disclosure, and may be performed by the feature determination unit (for example, the feature determination unit203illustrated inFIG.3).

First, at Step S601, the feature determination unit acquires a private dataset and a public dataset. In embodiments, the feature determination unit may acquire the private dataset by requesting access to it via a secure connection between the weighted knowledge transfer device and a private device (e.g., owned by a hospital, business, individual, or other organization). In embodiments, the feature determination unit may acquire the public dataset by accessing a public data repository. In certain embodiments, the private dataset and the public dataset may be selected by an administrator of the weighted knowledge transfer device.

Next, at Step S602, the feature determination unit determines a set of target features by analyzing the private dataset and the public dataset. As described herein, the set of target features may include a collection of features that are shared between the private dataset and the public dataset. In embodiments, the feature determination unit may determine the set of target features by using one or more of a variety of statistical analysis techniques with respect to the private data to ascertain features that are shared between the private dataset and the public dataset. Further, in embodiments, the feature determination unit203may determine the set of target features by using natural language processing based methods.

Next, at Step S603, in the case that the feature determination unit was able to determine a set of target features (e.g., a set of shared features was present in both the public dataset and the private dataset), the knowledge transfer dataset creation process600proceeds to Step S604. In contrast, in the case that the feature determination unit was not able to determine a set of target features (e.g., a set of shared features was not present in both the public dataset and the private dataset), the knowledge transfer dataset creation process600returns to Step S601to acquire different or additional private and public data.

Next, at Step S604, the feature determination unit may create a private knowledge transfer dataset and a public knowledge transfer dataset that both include the set of target features. For instance, the feature determination unit may extract a set of public features from the public dataset as the public knowledge transfer dataset and extract a set of private features from the public dataset as the private knowledge transfer dataset, where the set of public features and the set of private features substantially correspond to one another.

As described herein, the knowledge transfer dataset creation process600described above with reference toFIG.6allows for the creation of knowledge transfer datasets to be used in the weighted knowledge transfer process.

Next, a similarity weighting process according to the present disclosure will be described with reference toFIG.7.

FIG.7illustrates a flowchart of a similarity weighting process700, according to embodiments. The similarity weighting process700is a process for generating a similarity weighting vector (for example, the similarity weighting vector208illustrated inFIG.3) for a set of public features included in a public training dataset, and may be performed by the data selection unit (for example, the data selection unit206illustrated inFIG.3).

First, at Step S701, the data selection unit receives the private knowledge transfer dataset and the public knowledge transfer dataset. Here, the data selection unit may receive transmission of the private knowledge transfer dataset and the public knowledge transfer dataset from the feature extraction unit, or may access a designated storage address where the private knowledge transfer dataset and the public knowledge transfer dataset have been stored.

Next, at Step S702, the data selection unit determines a similarity calculation method for calculating the degree of similarity between the public knowledge transfer dataset and the private knowledge transfer dataset. As described herein, the similarity calculation may be selected from a variety of similarity calculation techniques including Euclidean, Manhattan, Chebyshev, or Mahalanobis distance calculations, statistical methods such as propensity scoring matching methods and clustering methods such as k-means clustering machine learning, model-based similarity, discriminator networks, generative adversarial networks, or the like. In embodiments, the data selection unit may determine the similarity calculation method by using a machine learning model trained to predict which of a number of given similarity calculation techniques is most likely to achieve the highest accuracy with respect to the nature of the public knowledge transfer dataset and the private knowledge transfer dataset. In embodiments, the data selection unit may determine the similarity calculation method based using a lookup table that ranks the performance of each of a number of given similarity calculation techniques based on the nature of the public knowledge transfer dataset and the private knowledge transfer dataset.

Next, at Step S703, the data selection unit utilizes the similarity calculation method determined in Step S702to calculate the similarity of each feature in the public knowledge transfer dataset with respect to each feature in the private knowledge transfer dataset. In embodiments, the calculated similarity may be expressed as a similarity weight value between 0 and 1, where greater values indicate a higher degree of similarity.

Next, at Step S704, the data selection unit attaches the similarity weight values calculated in Step S703to the corresponding features in the public knowledge transfer dataset.

Next, at Step S705, the data selection unit may confirm whether or not a similarity change request has been received from the machine learning unit as part of the feedback loop of training the public machine learning model. The similarity change request, may, for example, be a request from the machine learning unit to increase or decrease the similarity weight of a particular feature or type of feature of the set of public features of the public knowledge transfer dataset. In the case that a similarity change request has been received, the similarity weighting process700may return to Step S702. In the case that a similarity change request has not been received, the similarity weighting process700may proceed to Step S706.

Next, at Step S706, the data selection unit may confirm whether or not a filter request has been received from the machine learning unit as part of the feedback loop of training the public machine learning model. The filter request may, for example, be a request from the machine learning unit to delete or exclude a particular feature or type of feature from the set of public features of the public knowledge transfer dataset. In the case that a filter request has been received, the similarity weighting process700may proceed to Step S707. In the case that a filter change request has not been received, the similarity weighting process700may proceed to Step S709.

Next, at Step S707, the data selection unit may filter the set of public features included in the public knowledge transfer dataset based on the filter request received at Step S706. For example, the data selection unit may delete, from the set of public features, those features specified in the filter request received at Step S706.

Next, at Step S708, the data selection unit may select, from the set of public features included in the public knowledge transfer dataset, those public features that are associated with a similarity weight above a similarity threshold as the public training dataset. Here, the similarity threshold can either be set by the user or determined in a feedback loop from the weighted knowledge transfer unit.

Next, at Step S709, the data selection unit may output the public training dataset selected in Step S708together with a similarity weight vector indicating the weights of the set of public features included in the public training dataset. In embodiments, the data selection unit may output the public training dataset and the similarity weight vector to the weighted knowledge transfer unit (for example, the weighted knowledge transfer unit105illustrated inFIG.3). As described herein, the weighted knowledge transfer unit may use the private knowledge transfer dataset, the public knowledge transfer dataset, the public training dataset, and the weight vector to create a public machine learning model.

As described herein, the similarity weighting process700described above with reference toFIG.7allows for the creation of the public training dataset and the weight vector used to create the public machine learning model.

Next, a weighted knowledge transfer process according to the present disclosure will be described with reference toFIG.8.

FIG.8illustrates a flowchart of a weighted knowledge transfer process800, according to embodiments. The weighted knowledge transfer process800is a process for training a public machine learning model (for example, the public machine learning model107illustrated inFIG.3) that achieves comparable performance to a private machine learning model, and may be performed by the weighted knowledge transfer unit (for example, the weighted knowledge transfer unit105illustrated inFIG.3).

First, at Step S801, the weighted knowledge transfer unit acquires the private knowledge transfer dataset and the public knowledge transfer dataset. Here, the weighted knowledge transfer unit may receive transmission of the private knowledge transfer dataset and the public knowledge transfer dataset from the feature extraction unit, or may access a designated storage address where the private knowledge transfer dataset and the public knowledge transfer dataset have been stored.

Next, at Step S802, the weighted knowledge transfer unit may divide the private knowledge transfer dataset into a plurality of partitions. As described herein, here, a partition refers to a portion of the private knowledge transfer dataset that includes a mutually exclusive set of private features with respect to the other partitions of the plurality of partitions503. In this way, data corresponding to one set of features (e.g., one patient in a healthcare context) is not distributed among multiple partitions but randomly assigned and maintained exclusively in one partition.

Next, at Step S803, the weighted knowledge transfer unit trains a set of machine learning models using the plurality of partitions in order to generate a set of trained private machine learning models. In embodiments, the weighted knowledge transfer unit trains each of the set of machine learning models based on a separate portion of the plurality of partitions, such that no data from other partitions is used. In this way, the performance of each machine learning model is optimized using only the data included in one partition of the plurality of partitions.

Next, at Step S804, the weighted knowledge transfer unit generates, by processing the public training dataset knowledge transfer dataset with the set of trained private machine learning models trained in Step S803, a set of labels for the set of public features included in the public training dataset.

Next, at Step S805, the weighted knowledge transfer unit generates the public label vector by aggregating the set of labels generated in Step S804for each set of features, adding random noise from a random noise generator, and selecting the label that has the majority of counts as the final output label for each set of features of the public training dataset.

Next, at Step S806, the weighted knowledge transfer unit uses the public training dataset, the similarity weight vector, and the public label vector generated in Step S805to create and train the public machine learning model. As described herein, the public machine learning model has comparable performance to a private machine learning model trained on private datasets, but is trained on the public training dataset such that sensitive information present in the private datasets is not accessible to unauthorized users even in the event of a cyberattack.

Next, at Step S807, the weighted knowledge transfer unit confirms whether or not there is a request for further optimization of the weighted knowledge transfer. For example, the weighted knowledge transfer unit may verify whether or not there is feedback from previously performed weighted knowledge transfer processes that may be used to further optimize the current weighted knowledge transfer. As another example, the weighted knowledge transfer unit may prompt a user for additional instructions or data that may be used to further optimize the weighted knowledge transfer. In the case that the weighted knowledge transfer unit determines that further optimization is possible, the weighted knowledge transfer process800returns to Step S801. In the case that the weighted knowledge transfer unit determines that further optimization is not possible, the weighted knowledge transfer process800proceeds to Step S808.

Next, at Step S808, the weighted knowledge transfer unit may provide access to the public machine learning model trained in Step S806. For example, as described herein, the weighted knowledge transfer unit may configure the public machine learning model to be accessed by users as a network-based service via an interface unit (e.g., software application).

The weighted knowledge transfer process800illustrated inFIG.8makes it possible to perform a weighted knowledge transfer from a private machine learning model to a public machine learning model in order to create a privacy preserving public machine learning model that achieves high performance while maintaining data privacy. The weighted knowledge transfer process800illustrated inFIG.8may provide benefits associated with data privacy and machine learning model performance.

Next, an example logical configuration of a weighted knowledge transfer unit according to a second embodiment of the present disclosure will be described with reference toFIG.9.

FIG.9illustrates an example logical configuration of a weighted knowledge transfer device900according to a second embodiment of the present disclosure. The weighted knowledge transfer device900according to the second embodiment of the present disclosure relates to performing a weighted knowledge transfer from a private machine learning model to a public machine learning model by optimizing a knowledge transfer capacity. As illustrated inFIG.9, the weighted knowledge transfer device900according to the second embodiment of the present disclosure primarily includes a partitioning optimization unit925, a data selection unit950, a control unit975, and a machine learning unit980. The weighted knowledge transfer device900according to the second embodiment may be implemented using a similar system configuration as that of the previously described embodiment.

In the following, those aspects of the weighted knowledge transfer device900that differ from those of the previously described embodiment will be primarily described, and the description of redundant elements will be omitted.

First, the weighted knowledge transfer device900receives a private dataset910and a public dataset920. Here, the private dataset910may include a collection of data that contains confidential information. For example, the private dataset910may include information regarding medical records, financial transactions, or personal data (names, addresses, passwords, bank account information) for one or more individuals, businesses, or other organizations (e.g., the private dataset910may correspond to the private dataset102of the previous embodiment).

The public dataset920may include a collection of data that contains public information. For example, the public dataset920may include information regarding medical records or financial transactions that are not associated with any particular individual or entity (e.g., the public dataset920may correspond to the public dataset108of the previous embodiment). In other embodiments, the private dataset910and the public dataset920may correspond to the private knowledge transfer dataset204and the public knowledge transfer dataset205of the previously described embodiment.

Next, in the partitioning optimization unit925, the partition unit930divides the private dataset910into a plurality of partitions932(e.g., a first plurality of partitions). Next, the model optimization unit935trains and optimizes a set of private machine learning models using the plurality of partitions932in order to generate a set of trained private machine learning models937. The model optimization unit935evaluates the performance of each of the set of trained private machine learning models937with respect to each partition of the plurality of partitions932for a variety of model parameter configurations, and stores the results in the configuration database940.

The configuration selection unit945then determines a partition configuration and a set of model parameters that maximize a predetermined performance metric, such as Area Under the Curve of the Receiver Operating Characteristic (AUROC), precision, recall, or the like. The set of model parameters selected by the configuration selection unit945may be applied to the set of trained private machine learning models937. The set of trained private machine learning models937is then communicated to the data selection unit950.

As the logical configuration of the partitioning optimization unit925will be described in detail later, the description thereof will be omitted here.

The data selection unit950receives the public dataset920, and uses the set of trained private machine learning models937to process the public dataset920, thereby attaching a group of labels and weights to the public dataset920. The aggregation unit960aggregates the group of labels and weights as a processed public dataset970. In embodiments, the processed public dataset970may be selected by adjusting filtering thresholds for the attached weights. As one example, the processed public data970may be determined by assigning the public dataset920to separate partitions according to weight thresholds (for example, thresholds determined based on the group of labels and weights), training a plurality of public machine learning models, and selecting the optimal threshold and model parameters according to an evaluation metric.

The machine learning unit980uses the processed public dataset970to train a public machine learning model985. This process may be controlled by the control unit975, which establishes a feedback loop from the public machine learning model985to the partitioning optimization unit925and the data selection unit950to optimize the transfer capacity of the weighted knowledge transfer. The public machine learning model985may be used to provide a variety of machine-learning based services to users995over the interface unit990.

In this way, the weighted knowledge transfer device900makes it possible to perform a weighted knowledge transfer from a private machine learning model to a public machine learning model by optimizing the knowledge transfer capacity.

Next, an example logical configuration of the partitioning optimization unit for partitioning the private dataset and generating the set of machine learning models will be described with respect toFIG.10.

FIG.10illustrates an example logical configuration of the partitioning optimization unit925for partitioning the private dataset910and generating the set of machine learning models937, according to the second embodiment of the present disclosure.

As described herein, aspects of the disclosure relate to dividing the private dataset910into a plurality of partitions932, and training a set of machine learning models using the plurality of partitions932in order to generate a set of trained private machine learning models937. Here, a partition refers to a portion of the private dataset910that includes a mutually exclusive set of private features with respect to the other partitions of the plurality of partitions932. In this way, data corresponding to one set of features (e.g., one patient in a healthcare context) is not distributed among multiple partitions but randomly assigned and maintained exclusively in one partition.

As illustrated inFIG.10, first, the private dataset910is input to the partitioning optimization unit925. In the partitioning optimization unit925, the partition unit930divides the private dataset910into a plurality of partitions932(e.g., a first plurality of partitions) based on a set of partition constraints. Here, the set of partition constraints may include limitations, restrictions, or conditions that define how the private dataset910is to be distributed. As an example, the set of partition constraints may indicate that data corresponding to one set of features (e.g., one patient in a healthcare context) may not be divided between multiple partitions, but must be assigned to a single partition. Further, the partition unit930generates a set of external test data1003. The set of external test data1003may be used to evaluate the performance of the set of trained private machine learning models937.

Next, the model optimization unit935trains a set of machine learning models using the plurality of partitions932in order to generate a set of trained private machine learning models937. In embodiments, the model optimization unit935trains each of the set of machine learning models based on a separate portion of the plurality of partitions, such that no data from other partitions is used. In this way, the performance of each machine learning model is optimized using only the data included in one partition of the plurality of partitions. Further, the model optimization unit935may evaluate the performance of each of the trained private machine learning models937with respect to the data included in other partitions of the plurality of partitions932(e.g., each of the trained private machine learning models937is evaluated with respect to data included in the partitions other than the partition on which it was trained).

Next, the evaluation unit1007may receive the model configuration parameters and the performance results of the evaluation of the trained private machine learning models937performed by the model optimization unit935, and further evaluate the trained private machine learning models937with respect to the external test data1003. The results of the performance evaluation may be stored in the configuration database940.

Next, the configuration selection unit945selects the partition configuration and associated model parameters that achieve the highest performance according to some evaluation metric such as Area Under the Curve of the Receiver Operating Characteristic (AUROC), and applies them to the plurality of partitions932and the set of trained private machine learning models937, respectively.

In this way, trained private machine learning models937for use in the weighted knowledge transfer process can be generated.

Next, a flowchart for a trained machine learning model generation process will be described with reference toFIG.11.

FIG.11illustrates a flowchart of a trained machine learning model generation process1100, according to the second embodiment of the present disclosure. The trained machine learning model generation process1100is a process for generating the trained private machine learning models (for example, the trained private machine learning models937illustrated inFIG.9andFIG.10) using a plurality of partitions, and may be performed by the various function units of the weighted knowledge transfer device according to the second embodiment of the present disclosure.

First, at Step S1102, the partition unit (for example, the partition unit930illustrated inFIG.9andFIG.10) determines a number of partitions into which to divide the private dataset (for example, the private knowledge transfer dataset910illustrated inFIG.9andFIG.10). In embodiments, the number of partitions may be determined based on a user input. In embodiments, the number of partitions may be determined automatically based on the nature (e.g., size, number of feature sets, etc.) of the private knowledge transfer dataset.

Next, at Step S1103, the partition unit determines whether or not to create a set of external test data for use in evaluating the performance of the set of trained private machine learning models. In embodiments, the determination of whether or not to create a set of external test data may be performed based on instructions received from a user, or a pre-set performance goal criteria.

In the case that it is determined to create the set of external test data, the trained machine learning model generation process1100proceeds to Step S1104. In the case that it is determined not to create the set of external test data, the trained machine learning model generation process1100proceeds to Step S1105.

At Step S1104, the partition unit assigns a subset of the private dataset as the external test data. In embodiments, the partition unit may select a random subset of the private dataset for use as the external test data, and designate it as a separate external partition.

Next, at Step S1105, the partition unit randomly shuffles the private dataset based on the set of partitioning constrains. As described herein, the set of partition constraints include limitations, restrictions, or conditions that define how the private dataset is to be distributed. As an example, the set of partition constraints may indicate that data corresponding to one set of features (e.g., one patient in a healthcare context) may not be divided between multiple partitions, but must be assigned to a single partition. Accordingly, here, the partition unit may randomly shuffle the private dataset while satisfying the set of partitioning constraints.

Next, at Step S1106, the partitioning unit assigns the private dataset shuffled at Step S1006into separate partitions.

Next, at Step S1107, the model optimization unit (for example, the model optimization unit935illustrated inFIG.9andFIG.10) trains a set of machine learning models using the plurality of partitions in order to generate a set of trained private machine learning models. In embodiments, the model optimization unit trains each of the set of machine learning models based on a separate portion of the plurality of partitions, such that no data from other partitions is used. In this way, the performance of each machine learning model is optimized using only the data included in one partition of the plurality of partitions.

Next, at Step S1108, the model optimization unit evaluates the performance of each of the trained private machine learning models with respect to the data included in other partitions of the plurality of partitions. That is, each of the trained private machine learning models is evaluated with respect to data included in the partitions other than the partition on which it was trained.

Next, at Step S1109, the evaluation unit (for example, the evaluation unit1007illustrated inFIG.10) determines the performance of each of the trained private machine learning models. In embodiments, the evaluation unit may evaluate the trained machined learning models with respect to the external test data created at Step S1104and S1105, and determine the performance of each of the trained private machine learning models based on the evaluation with respect to the external test data and the performance results of the evaluation of the trained private machine learning models performed by the model optimization unit.

Next, at Step S1110, the evaluation unit stores the results of the performance evaluation determined at Step S1110in the configuration database (for example, the configuration database940illustrated inFIG.9andFIG.10).

Next, at Step S1111, the evaluation unit determines if the data should be re-shuffled according to constraints and re-distributed among partitions to execute another evaluation cycle. In embodiments, this determination can be performed based on a user input that specifies the evaluation of a fixed number of evaluation cycles. In other embodiments, the evaluation unit can automatically determine a stopping condition by comparing current performance results to data in the configuration database and using statistical decision making logic to determine whether or not to initiate the next evaluation cycle loop.

Next, at Step S1112, the configuration selection unit (for example, the configuration selection unit945illustrated inFIG.9andFIG.10) analyzes the results of the performance valuation stored in the configuration database, and determines the partition configuration and associated model parameters that achieve the highest performance. Here, the configuration selection unit may determine the partition configuration and associated model parameters that achieve the highest performance using some evaluation metric such as Area Under the Curve of the Receiver Operating Characteristic (AUROC),), and apply them to the plurality of partitions932and the set of trained private machine learning models937, respectively.

In this way, according to the trained machine learning model generation process1100, trained private machine learning models for use in the weighted knowledge transfer process can be generated.

Next, the logical configuration of the weighted knowledge transfer device for data selection and optimization of a weighted knowledge transfer will be described with reference toFIG.12.

FIG.12illustrates the logical configuration of the weighted knowledge transfer device according to the second embodiment for data selection and optimization of a weighted knowledge transfer.

First, as illustrated inFIG.12, the partitioning optimization unit925receives a private dataset910. The partitioning optimization unit925divides the private dataset910into a plurality of partitions (e.g., a first plurality of partitions, such as the plurality of partitions932illustrated inFIG.9andFIG.10; not illustrated inFIG.12), and generates a set of external test data1003. The partitioning optimization unit925trains and optimizes a set of private machine learning models using the plurality of partitions in order to generate a set of trained private machine learning models937.

Next, the public dataset920is processed by the set of trained private machine learning models and their outputs (e.g., group of labels and weights) are aggregated by the aggregation unit960to produce the processed public dataset970. The processed public dataset970is input to the threshold partitioning unit1210. The threshold partitioning unit1210divides the processed public dataset970data into a second plurality of partitions1220according to a set of thresholds determined based on the group of labels and weights. This set of thresholds can be used to filter the processed public dataset970according to the weights assigned thereto. Here, the partitioning of the processed public dataset970may be performed based on the data thresholding optimization process described later.

Next, the model optimization unit935trains a set of public models using the second plurality of partitions1220to generate a set of trained public machine learning models1230. Here, the model optimization unit935may train the set of public models such that each of the public models are trained on a different partition of the plurality of partitions1220.

The evaluation unit1007evaluates the performance of the set of trained public machine learning models1230using the external test data1003and records the results in the configuration database940.

The configuration selection unit945selects the thresholds, model, and model parameters that achieve the highest performance in the weighted knowledge transfer, and applies them. In embodiments, one or more of the models among the set of trained public machine learning models1230may be selected for deployment to users via an interface unit (not shown inFIG.12).

The control unit975is used to combine the data partition process and data selection process with the threshold-based optimization process. In this way, weighted knowledge transfer performance can be maximized by simultaneously optimizing and determining a partition scheme for a public dataset public data, parameter configurations for sets of private and public machine learning models, and weights and weighting thresholds for selecting a the processed public dataset.

Next, a flowchart of a data thresholding optimization process will be described with reference toFIG.13.

FIG.13illustrates a flowchart of a data thresholding optimization process1300, according to the second embodiment of the present disclosure. The data thresholding optimization process1300is a process for determining the set of thresholds to be used in partitioning the processed public data, and may be performed by the various functional units of the weighted knowledge transfer device according to the second embodiment.

First, at Step S1301, the partitioning optimization unit (for example, the partitioning optimization unit925illustrated inFIG.9,FIG.10, andFIG.12) divides the private dataset into a plurality of partitions, and generates a set of external test data. The partitioning optimization unit925trains and optimizes a set of private machine learning models using the plurality of partitions in order to generate a set of trained private machine learning models.

Next, at Step S1302, the public dataset is processed by the set of trained private machine learning models and their outputs (e.g., a group of labels and weights) are aggregated by the aggregation unit960to produce the processed public dataset970.

Next, at Step S1303, the threshold partitioning unit (for example, the threshold partitioning unit1210illustrated inFIG.12) determines a set of thresholds for filtering the processed public dataset according to the weights assigned thereto. In embodiments, the set of thresholds may be determined based on the weights generated by processing the public dataset with the set of trained private machine learning models in Step S1302. In other embodiments, the set of thresholds may be initially set by a user or administrator of the weighted knowledge transfer system, and updated by subsequent steps of the data thresholding optimization process1300.

Next, at Step1304, the threshold partitioning unit divides the processed public dataset data into a second plurality of partitions according to the set of thresholds which are used to filter the processed public dataset according to the weights assigned thereto.

Next, at Step S1305, the model optimization unit (for example, the model optimization unit935illustrated inFIG.10andFIG.12) trains a set of public models using the second plurality of partitions created at Step S1304to generate a set of trained public machine learning models. Here, the model optimization unit may train the set of public models such that each of the public models are trained on a different partition of the plurality of partitions.

Next, at Step S1306, the evaluation unit (for example, the evaluation unit1007illustrated inFIG.10andFIG.12) evaluates the performance of the set of trained public machine learning models using the external test data and records the results in the configuration database.

Next, at Step S1307, the configuration selection unit (for example, the configuration selection unit945illustrated inFIG.10andFIG.12) selects the weight thresholds, model, and model parameters that achieve the highest performance in the weighted knowledge transfer.

Next, at Step S1308, the control unit (for example, the control unit975illustrated inFIG.10andFIG.12) determines whether or not to update the set of thresholds. In embodiments, the control unit may perform the determination of whether or not to update the set of thresholds based on the performance of the set of trained public machine learning models. For instance, in the event that the set of trained public machine learning models fail to achieve a designated performance criterion, the control unit may update the set of thresholds to facilitate the creation of data partitions predicted to provide increased performance.

In the case that the control unit determines to update the set of thresholds, the data thresholding optimization process1300may return to Step S1303. In the case that the control unit determines not to update the set of thresholds, the data thresholding optimization process may proceed to Step S1309.

Next, at Step S1309, the control unit determines whether or not to update the plurality of partitions. For example, the control unit may divide the private dataset based on new partition constraints created based on the performance of the set of trained public machine learning models.

In this way, according to the weighted knowledge transfer device900according to the second embodiment of the present disclosure, machine learning models can be trained using optimal data partitions, allowing for knowledge transfer capacity to be maximized. The weighted knowledge transfer device900may be associated with additional performance and efficiency benefits with respect to the weighted knowledge transfer device according to the previous embodiment.

A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

REFERENCE SIGNS LIST