Patent ID: 12217875

DETAILED DESCRIPTION

An exemplary approach is to augment minority class data by learning to generate variations of the minority class records that are still likely to be in that minority class. Data records consist of a set of features, and for each feature, we learn a model that can predict the likely values of that feature given the other features in the record. If the feature model says that the feature could have had a value other than the value it does have, we create a new record that is a copy of the original record but with the feature set to the predicted value. Embodiments applied to high-dimensional spaces surmount or get around the “curse of dimensionality” by not looking for pairs of “nearby” records and instead changing one or more features of a record to ensure that it is nearby or to create another that is “nearby.”

As an example, imagine building a model to classify expensive cars from inexpensive ones. The record for each car has a feature “color” that can take the values “red” or “blue”, a feature “seat material” that can take the values “leather”, “cloth”, or “vinyl”, and many other features. We want to do classification, but while we have many records of inexpensive cars, we don't have many examples of expensive cars. We want our algorithm to create new instances of expensive cars.

Given a blue expensive car with cloth material, the algorithm could learn that it could copy this car record and add a new record where the material was leather (because presumably there are some expensive blue cars with leather interior). The algorithm could also learn that it could copy the original record and add a new record where the color of the car was red. But since vinyl material is not associated with our set of expensive cars, the model would not be able to copy this record and replace it with one with vinyl material.

A model is a function that maps data records to labels. In an exemplary embodiment, the data records are medical records, and the labels are “overdose” and “no overdose.” Data records with labels are used to train the model. After training, this model is used to predict the likely labels of data records without labels. For example, we may have the medical records of a group of people as well as data on whether each had an overdose on opioids. These records can be fed into a model, which then learns to predict whether some new patient (based on her medical records) will overdose. As a further concrete case, one machine may read the medical records of an individual, pass those records to the model, and receive back a prediction. It may then use that prediction to determine whether the person should receive a letter warning them of the dangers of opioids. A model can also be thought of as a function that maps data to a decision or choice. For example, the decision could be whether a person is likely to overdose on opioids.

Medical records (or electronic health records) are high-dimensional because of the number of different features that can determine or be measured with respect to health. Medical records typically include 10 to 1000 dimensions.

In an exemplary embodiment, the model function is a software program, package, or suite of software applications. The software includes computer instructions, and is stored in computer-readable media. The function of the model is performed by one or more computer processors executing the computer instructions. The input medical records preferably are data records stored in computer-readable media.

We now explain the algorithm in detail as shown inFIG.1.

Step115is to collect all of the records in the minority class. Each record consists of a set of features where each feature has a value. Each record in the minority class, in an embodiment, is stored in a computer-readable media.

Step120is then to learn a model that predicts the value of each feature given all of the other values of the features for that record. This model is learned over those minority class records collected in step115. For example, if there are four features in the records from step115, then step120will learn four models, and the model for feature f is represented with the notation Mf. The model Mf(r,v) takes a record r and gives the probability of feature f having value (or class or classification) v. These feature models can be learned (or trained) using, for example, logistic regression. In an embodiment in which every feature is a binary classification, a binary logistic regression model can be used. In an embodiment in which a feature may have more than two classifications, a multinomial logistic regression can be used. In an alternative embodiment, a feature with categorical or multiple classes (multi-class) can be represented as binary values using “one-hot” encoding. An exemplary logistic regression model for use with embodiments can be found in the software library scikit-learn at https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html, which is hereby incorporated by reference.

After the feature models are learned, the method loops through the minority class records and makes a copy of each record where a feature model says that its feature is likely to have some other value. Step125loops over the minority class records, and considers each record r one at a time. Step130loops over the features for record r and considers each feature f one at a time. Step135loops over each possible value for feature f and considers each value v one at a time.

Step140computes whether the current value for record r for feature f is equal to v, and if it is not, it computes whether the probability of feature f having value v is greater than a threshold t, represented by Mf(r, v)>t. If both of these conditions are satisfied, step145adds a new record to the set of augmented records. This new record is a copy of record r where feature f is set to have value v. After all of the records have been processed, the algorithm stops in step165. In an embodiment, the value of t is 0.2. In alternative embodiments, the value may range between any of these ranges: 0.175-0.25, 0.2-0.3, 0.2-0.4, 0.3-0.4, 0.2-0.5, 0.3-0.5, 0.4-0.5, 0.2-0.6, 0.3-0.6, 0.4-0.6, 0.5-0.6, 0.2-0.7, 0.3-0.7, 0.4-0.7, 0.5-0.7, 0.6-0.7, 0.175-0.5, and 0.175-0.7.

The result is a set of augmented records that can be used along with the minority class records during training. These augmented records have the label of the minority class. The augmented records, in an embodiment, are stored in a computer-readable media.

In an embodiment relevant to the study of opioid overdoses, the overdose would be the class or classification, will this person overdose or not? The features are the aspects of the person the model uses to classify whether the person will overdose, such as age and whether that person is currently taking opioids.

We looked at processed binary medical-data records to determine who would overdose in the next year. Each record had 73 binary features. For training, there were 133,728 records, where 744 of those were positive. For testing, there were 2,828 records, where 17 were positive. “Positive” means that an overdose occurred in the following year.

When we used our feature prediction model, it added 7,923 positive records to the training set, giving 8,667 positive training records. The augmented set of training records (including the synthetic positive records) were fed to the training model. We call this condition “real+pred” and the condition where the prediction algorithm was not used is called “only_real.” We also consider two conditions for evaluation.a. Balanced means the positive records in the test set were oversampled so there were as many positive as negative. This makes the accuracy meaningful.b. Raw means like it was with 17 positive records, where the system can always predict “negative” and have an accuracy of 99%.

The results are shown in table 1 below. F1 is the F1 Score and AUC is area under the curve (receiver operating characteristic).

TABLE 1RawBalancedAccuracyF1AUCAccuracyF1AUConly_real.9940.765.4990.765real + pred.985.157.795.612.378.795

We see that the “real+pred” condition does much better on everything other than raw accuracy. Raw accuracy doesn't mean much here because the “only_real” condition classified all records as being negative (not overdose). This is also why F1 is 0 for “only_real.”

FIG.2illustrates an example of a general computer environment200useful in the context of the environments ofFIG.1, in accordance with an implementation of the disclosed subject matter. The general computer environment200includes a computation resource202capable of implementing the processes described herein. It will be appreciated that other devices can alternatively used that include more components, or fewer components, than those illustrated inFIG.2.

The illustrated operating environment200is only one example of a suitable operating environment, and the example described with reference toFIG.2is not intended to suggest any limitation as to the scope of use or functionality of the implementations of this disclosure. Other computing systems, architectures, environments, and/or configurations can be suitable for implementation and/or application of the subject matter disclosed herein.

The computation resource202includes one or more processors or processing units204, a system memory206, and a bus208that couples various system components including the system memory206to processor(s)204and other elements in the environment200. The bus208represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port and a processor or local bus using any of a variety of bus architectures, and can be compatible with SCSI (small computer system interconnect), or other conventional bus architectures and protocols. Bus208may include logical and physical interconnection, including a network or WAN connection, to operationally couple one or more system components located remotely to other system components coupled to Bus208.

The computation resource202may include computers or processors operating in parallel, including architectures in which some of the parallel computers or processors are remote and/or virtual. Computation resource202may be implemented on one or more servers that perform the software functions in a hosted Software As A Service (SAAS) environment.

The system memory206includes nonvolatile read-only memory (ROM)210and random access memory (RAM)212, which may or may not include volatile memory elements. A basic input/output system (BIOS)214, containing the elementary routines that help to transfer information between elements within computation resource202and with external items, typically invoked into operating memory during start-up, is stored in ROM210. System memory206may contain non-volatile or volatile memory components located remotely and coupled to computation resource202by conventional logical and/or physical interconnections, including a network or WAN connection.

The computation resource202further can include a non-volatile read/write memory216, represented inFIG.2as a hard disk drive, coupled to bus208via a data media interface217(e.g., a SCSI, ATA, or other type of interface); a magnetic disk drive (not shown) for reading from, and/or writing to, a removable magnetic disk220and an optical disk drive (not shown) for reading from, and/or writing to, a removable optical disk226such as a CD, DVD, or other optical media. Non-volatile read/write memory216may include one or more non-volatile memory components located remotely and coupled to computation resource202by conventional logical and/or physical interconnections, including a network or WAN connection.

The non-volatile read/write memory216and associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the computation resource202. Although the exemplary environment200is described herein as employing a non-volatile read/write memory216, a removable magnetic disk220and a removable optical disk226, it will be appreciated by those skilled in the art that other types of computer-readable media which can store data that is accessible by a computer, such as magnetic cassettes, FLASH memory cards, random access memories (RAMs), read only memories (ROM), and the like, can also be used in the exemplary operating environment.

A number of program modules can be stored via the non-volatile read/write memory216, magnetic disk220, optical disk226, ROM210, or RAM212, including an operating system230, one or more application programs232, other program modules234and program data236. Examples of computer operating systems conventionally employed include LINUX,® Windows® and MacOS® operating systems, and others, for example, providing capability for supporting application programs232using, for example, code modules written in the C++® computer programming language or an interpreted language such as Python.

A user can enter commands and information into computation resource202through input devices such as input media238(e.g., keyboard/keypad, tactile input or pointing device, mouse, foot-operated switching apparatus, joystick, touchscreen or touchpad, microphone, antenna etc.). Such input devices238are coupled to the processing unit204through a conventional input/output interface242that is, in turn, coupled to the system bus. A monitor250or other type of display device is also coupled to the system bus208via an interface, such as a video adapter252. One or more remote input devices or display devices may be coupled to computation resource202by conventional logical and/or physical interconnections, including a network or WAN connection.

The computation resource202can include capability for operating in a networked environment using logical connections to one or more remote computers, such as a remote computer260. The remote computer260can be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computation resource202. In a networked environment, program modules depicted relative to the computation resource202, or portions thereof, can be stored in a remote memory storage device such as can be associated with the remote computer260. By way of example, remote application programs262reside on a memory device of the remote computer260. The logical connections represented inFIG.2can include interface capabilities, a storage area network (SAN, not illustrated inFIG.2), local area network (LAN)272and/or a wide area network (WAN)274, but can also include other networks.

Such networking environments are commonplace in modern computer systems, and in association with intranets and the Internet. In certain implementations, the computation resource202executes an Internet Web browser program (which can optionally be integrated into the operating system230), such as the “Internet Explorer®” Web browser manufactured and distributed by the Microsoft Corporation of Redmond, Washington.

When used in a LAN-coupled environment, the computation resource202communicates with or through the local area network272via a network interface or adapter276. When used in a WAN-coupled environment, the computation resource202typically includes interfaces, such as a modem278, or other apparatus, for establishing communications with or through the WAN274, such as the Internet. The modem278, which can be internal or external, is coupled to the system bus208via a serial port interface.

In a networked environment, program modules depicted relative to the computation resource202, or portions thereof, can be stored in remote memory apparatus. It will be appreciated that the network connections shown are exemplary, and other means of establishing a communications link between various computer systems and elements can be used.

A user of a computer can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer260, which can be a personal computer, a server, a router, a network PC, a peer device or other common network node, a laptop computer, notebook computer, palm top computer, network computer, smart phone, tablet, or other mobile device, or any processor-controlled device capable of performing the steps, methods, and functions described herein. Typically, a remote computer260includes many or all of the elements described above relative to the computer200ofFIG.2.

Embodiments described herein can be implemented on an Amazon g2.2×large GPU machine, an Amazon c4.8×large, or a multi-core CPU machine. The computer system may be implemented using other computer architectures (for example, a client/server type architecture, a mainframe system with terminals, an ASP model, a peer to peer model, and the like) and other networks (for example, a local area network, the internet, a telephone network, a wireless network, a mobile phone network, and the like), and those other implementations are within the scope of the invention since the invention is not limited to any particular computer architecture or network.

The computation resource202typically includes at least some form of computer-readable media. Computer-readable media can be any available media that can be accessed by the computation resource202. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

Computer storage media include volatile and nonvolatile, removable and non-removable media, implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules or other data. The term “computer storage media” includes, but is not limited to, RAM, ROM, EEPROM, FLASH memory or other memory technology, CD, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media which can be used to store computer-intelligible information and which can be accessed by the computation resource202.

Communication media typically embodies computer-readable instructions, data structures, program modules. By way of example, and not limitation, communication media include wired media, such as wired network or direct-wired connections, and wireless media, such as acoustic, RF, infrared and other wireless media. The scope of the term computer-readable media includes combinations of any of the above.

More specifically, in the computer-readable program implementation, the programs can be structured in an object-orientation using an object-oriented language such as Java, Smalltalk or C++, an interpreted language such as Python, and the programs can be structured in a procedural-orientation using a procedural language such as COBOL or C. The software components communicate in any of a number of means that are well-known to those skilled in the art, such as application program interfaces (API) or interprocess communication techniques such as remote procedure call (RPC), common object request broker architecture (CORBA), Component Object Model (COM), Distributed Component Object Model (DCOM), Distributed System Object Model (DSOM) and Remote Method Invocation (RMI). The components execute on as few as one computer as in general computer environment200inFIG.2, or on at least as many computers as there are components.

With reference toFIGS.1and2, the steps of algorithm100are preferably performed by a Feature Predictor software application program, package, or suite of software applications that realize the model Mfand/or, in an embodiment, generative model320. The Feature Predictor software preferably is stored with other Application Programs232in non-volatile read/write memory216and includes instructions that, when executed by the one or more processing units204, cause the computation resource202to perform the steps of algorithm100and generate one or more augmented records having the label of the training class and store the one or more augmented records in data files with other program data236in non-volatile read/write memory216. Algorithm100operates on minority class records that have been stored in date files with other program data236in non-volatile read/write memory216. In an embodiment, the one or more augmented records having the label of the training class combined with the minority class records in a training data file that is stored with other program data236in non-volatile read/write memory216. Predictive model370, explanatory model370, and certainty model350are similarly realized in a software application program, package, or suite of software applications.

The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible with the spirit and scope of the invention, including as defined in the claim and their equivalents. The described embodiments illustrate the scope of the claim but do not restrict the scope of the claims.