SYSTEMS AND METHODS FOR PREDICTING HIGH FREQUENCY AND LOW FREQUENCY PATIENT PARAMETERS

Systems and methods are disclosed for predicting values for health parameters traditionally sampled at higher frequencies (such as vital signs) and at lower frequencies (such as lab results). The higher frequency parameter predictions can take the form of a time series of predicted vital sign values and the lower frequency parameter predictions can be single predicted values based on statistical measures of past collected input features. Suitable statistical measures can include the maximum, minimum and median values.

BACKGROUND

Vital signs (vitals) are a group of clinical signs that measures the status of a human being's vital functions. These measurements are made to assess one's physical health, and often point to the existence of one or more physical conditions when one or more of these vitals are not within the normal accepted ranges for a person with the same age, weight and gender. They can be used to assess if someone is in good health.

Temperature, heart rate, blood pressure and respiration rate are the generally accepted group of vitals. Other measurements can be added to vitals such oxygen saturation (SpO2), white blood cell count (WBC) and others for their frequency of measurement and health status assessment.

It is particularly important to be able to predict the time dynamics of vitals as they possess a great deal of information on the health status of a patient. Hospital patients have their vitals continuously monitored to make sure that in the case these degrade, patients can be admitted to intensive care unit (ICU) and proper intervention can be administered by clinical personnel. Vitals information can be combined to create new scores that measure the overall state of a patient. An example is SIRS (Systemic Inflammatory Response Syndrome), where temperature, heart rate and respiration are being used.

The measurement of body temperature is used to get an indication of core body temperature which is maintained and regulated by the body itself in order to sustain its correct functioning. Temperature changes depending on which part of the body is measured, rectal temperature is one-degree Fahrenheit higher than the oral one. It also normally varies during the day to respond to the amount of activity carried out. The average body temperature is 98.6 F (37C), but it is not uncommon to find individuals that have temperature one-degree F. lower or higher than the average.

Heart rate or pulse is the number of heart beats per minute. Its measurement is important as well the strength and dynamics of its time evolution to indicate a problem with heart function. The heart rate normally varies with age and physical conditions. It also varies during the day and night. A typical heart rate for an adult varies between 60 bpm and 100 bpm.

Blood pressure is recorded as two separate values: the systolic pressure and the diastolic or resting pressure. The systolic is higher than the diastolic. Mean arterial blood pressure is an average blood pressure taken during a single cardiac event. The normal range is between 70 and 100 mmHg. A deviation from this range can have a negative impact on the body health status.

The respiratory rate for a human body is measured by counting how many times the chest rises when a person is at rest. The typical respiration rate range is 12 to 18 bpm, except for elderly a respiration rate above 20 is not uncommon. Respiration rates may increase with medical conditions such as fever and illness.

Oxygen saturation (SpO2) is the fraction of oxygen-saturated hemoglobin relative to total hemoglobin (unsaturated+saturated) in the blood. Its normal range is 95 to 100%.

White blood cells in the blood have the task of protecting the body from infection and external substances. The normal white cell count is usually between 4×109/L and 1.1×1010/L. A high count of WBC denotes that the body is fighting disease, while a low count denotes that the immune system is weak. White blood cell count is typically collected at a lower frequency than the vital signs discussed above. In some cases it can be considered a low-frequency patient parameter.

MIMIC-III (‘Medical Information Mart for Intensive Care’) is a clinical database that contains information on ICU patients at established Boston hospitals with more than 60,000 admissions. The data was collected from hospital databases and specifically from the tables representing chart measurements, laboratory measurements, drugs, fluids, microbiology, and cumulative fluids. The patient data from the hospital databases is time-stamped and contains physiological signals and measurements, vital signs, as well as a comprehensive set of clinical data representing such quantitative data as medications taken (amounts, times, and routes), laboratory tests, measurements, and outcomes, feeding and diagnostic assessments. Each admission is characterized by a unique number called admission id. Patients are also identified by their patient ids.

DETAILED DESCRIPTION

It is to be understood that the present disclosure includes examples of the subject technology and does not limit the scope of the appended claims. Various aspects of the subject technology will now be disclosed according to particular but non-limiting examples. Various embodiments described in the present disclosure may be carried out in different ways and variations, and in accordance with a desired application or implementation.

Vital Sign Prediction

Described herein are systems and methods for predicting future values of vital signs using predictive models. The future values of any given vital sign are predicted based on times series of values of multiple vital signs.

For example, the goal of a vitals predictive model may be to predict the vitals value of a patient in a hospital setting or outside the hospital, for example at home with a portable vitals monitor, for 12 hours where a prediction is output for every half an hour interval of that 12 hour period for a total of 24 values. In other models other total time periods with outputs of different spacings may be used without departing from the scope of this disclosure. For example, 6-24 hours' worth of vital sign predictions can be generated, with outputs for intervals ranging from 6-90 minutes in length, though the interval is preferably less than 60 minutes in length. As one specific additional non-limiting example, the model may output values for an 18 hour period using 20 minute intervals.

The use of a 12 hour space, divided into half-hour (or smaller) periods is non-standard, but it allows a more detailed picture of time evolution of the predicted vital sign. The problem of predicting such values can be satisfied by treating the data set as a statistical problem of the regression type, applied to time series. The following table shows an example time setup for a 12 hour prediction model.

This particular setup lends itself well to high frequency output as it reconstructs a discrete time function of the machine learning output from which a clinician or another software system can extract the maximum, minimum and other statistical measures.

In some implementations, the model uses can use 5 vital signs (heart rate, blood pressure, respiration rate, SpO2, and body temperature) that have the power to predict themselves by sampling each vital 24 times during the arc of 12 hours. In some implementations, the model may take into account white blood cell counts sampled at the same or a lesser frequency. The predictions are made so as to follow the feature samples. So for instance, if the samples start at time t=0.5 hours and they end at time t=12, the predictions start at time 12.5 and end at time t=24. As indicated above, other time periods with other sampling intervals may also be used. For example, home health vitals monitors may output values at a lower frequency, such as once every hour, once every two hours, once every three hours, or even less frequently.

For categorical features, a value of 1 is used for positive and zero for negative. All non-categorical features were normalized to speed up the convergence of the predictive engine.

The availability of additional features in some cases leads to an enhanced prediction performance. This assumes that the added features carry dependence on the output variable. This is not always the case and when the added features do not carry such dependence, the neural net tends to overfit. For this reason, 2 approaches can be used: 1) use the Lasso regressor to trim the number of features and 2) heuristically try all combinations of features and find which combination yields the best performance. In the case of vital sign prediction, we offer an example that supports our previous statements. This example pertains to the prediction of SpO2. When we predict SpO2 using the following features: Mean BP, Temperature, Heart Rate, Respiration Rate, SpO2 and WBC with a specific machine learning model we obtain an MAE of 1.4, while if we use only Spo2 as predictive feature, we obtain an MAE of 1.1. This is due to the aforementioned problem of overfitting where one or more feature in the first case do not exhibit dependence on the output variable.

One example model uses 6 features: Temperature, Mean BP, Respiration Rate, Heart Rate, SP02, WBC and is generated using a predictive regressor model written in Python, though other programming languages could also be used. Training data was obtained from the chart table of the MIMIC3 database discussed above. Chart times were used to located the data values in time. One example of the vitals predictive model used a Multi-Layer Perceptron machine learning library called SKLEARN.NEURAL_NETWORK.MLPREGRESSOR. For the example model, every input and output node is connected to a particular time in the evolution of the model features and labels, which is believed to be unique in the health care setting.

In other implementations, other forms of regression based machine learning models may be used, including:

Logistic Regression

Polynomial Regression

Stepwise Regression

Ridge Regression

Lasso Regression

Support Vector Regression

It is understood that other predictive algorithms or methodologies may be utilized without departing from the scope of the disclosure.

In one specific non-limiting example, the vital sign predictive model is implemented as a MLP neural net with 3 hidden layers, with 100, 50 and 10 nodes, respectively trained using stochastic gradient descent (SGD) or ADAM with adaptive learning rate. The loss objective is to minimize the mean absolute error (MAE). In alternative implementations, the loss objective is to minimize the mean squared error (MSE).

Experimental Results

Using the above model, in a random test of 15% of the total available data, we computed the mean absolute error (MAE) for all predicted vital labels.

The total number of rows sampled was over 7 million and the training and test sizes were 85% and 15%, respectively of the total number of samples. Each row contains 24 separate records for a total of 168 million data points.

Temperature Results

The temperature MAE is 0.626. The table below shows the specific MAE for different ranges of temperatures:

Heart Rate Results

The heart rate MAE is 6.64. The table below shows the specific MAE for different ranges:

Respiration Rate Results

The respiration rate MAE is 3.05. The table below shows the specific MAE for different ranges:

Mean BP Results

The mean BP MAE is 7.83. The table below shows the specific MAE for different ranges:

The Sp02 MAE is 1.04 in %. The table below shows the specific MAE for different ranges:

WBC Results

The WBC MAE is 0.849. The table below shows the specific MAE for different ranges:

Conclusions

As demonstrated by the above data, a predictive model as described above (i.e., a predictive matrix comprised of 5 vitals and white blood cell count values sampled with an interval of half an hour (or less frequently for WBC) for 12 hours giving a time series of 24 data points) can successfully predict 5 vital measurements: HR, RR, Temperature, Mean BP, and SpO2, and WBC, using neural network technology. The neural net predicts a time series for 12 hours following the last sample predictive data for a total of 24 data points, each sampled half an hour apart with mean MAEs that are low compared to the typical standard deviations of the vital measurements for a wide variety of intervals. As indicated above, other neural networks using only the five above-mentioned vitals can output time series for just the five vitals.

In some implementations, an aggregate vitals score can be calculated using the output time series to provide a holistic prediction of the patients health. For example, in one implementation a vitals score or V-Score can be calculated by summing the absolute differences between predicted values for a vital at a given point in time and a measure of normal for that vital for a patient at rest. For vitals whose “normal” takes the form of the range, in various implementations, the value of normal used in the absolute value calculation may be, for example, the minimum value of the range for predicted values that fall below the normal range and the maximum value of the range for predicted values that fall above the normal range, or the middle of the normal range for all values. In some implementations, each of the vital sign differences may have a different weight attached to it. For example each vital may be weighted by the inverse of its normal standard deviation. Example weights include:w(Heart Rate)=1/10w(Respiration Rate)=3w(Temperature in C)=0.4w(Spo2)=1.5w(MeanBp)=15

In models which output white blood cell counts, the difference between the predicted white blood cell count and the normal white blood cell count can also be included in the vital score. The white blood cell count difference can also be similarly weighted. For example, its weight may be set equal to 3.5. Each of the aforementioned weights is illustrative in nature and may change, for example, based on the demographics of the patient, or even on past historical data associated with the given patient.

In other implementations, instead of the vital score being calculated based on the sum of the weighted absolute differences, the vital score may be calculated based on the sum of changes in the absolute differences between a predicted measure of a vital sign or patient parameter and a normal value for that vital sign or patient parameter. This vital score tracks and emphasizes information about a trend in the patient's health parameters with respect to their normal levels rather than emphasizing the absolute patient health parameter values.

Hemoglobin Prediction

Clinical data measurements can be roughly divided into two classes: high-frequency and low-frequency. Examples of high-frequency measurements are body vitals: heart rate, temperature, respiration rate, spo2 and mean blood pressure. Low-frequency measurements are typically lab measurements that have frequencies of the order of 1 day or more, such as levels of bicarbonate, chloride, bun, anion, and many others. The application of machine learning predictive algorithms that require low-frequency features has to deal with 2 fundament problems: missing data and frequency variability. To remedy these problems, we propose the adoption of statistical measures and the use of data imputation methods. An example of these techniques is provided when hemoglobin is predicted using neural network technology. Such techniques, while described with respect to hemoglobin can be applied to other low frequency features, too.

Missing Data

There are 3 types of missing data: A) Missing Completely at Random (MCAR), B) Missing at Random (MAR), and C) Missing not at Random (MNAR).

MCAR data occurs when the missing data is independent of any observable or unobservable variable. MAR data occurs when the missingness looks random but in fact it is not random when all variables (observable and unobservable) are accounted for. Finally, MNAR data is neither MCAR or MAR. MNAR is the case when the value is missing due to its very value.

In clinical data measurements missing data is probably due to the fact that doctors may not deem certain clinical tests necessary given their knowledge of the patient's conditions and their expectation that the data is negative. This supports the case for MNAR. The timing of the missingness may be assigned to MCAR or MNAR. Data imputation methods can be applied to model the missing data as a function of the same feature in question in the form of point statistics (SINGLE FEATURE IMPUTATION) or as a function of all other features (MULTIVARIATE IMPUTATION).

Data Imputation Methods

Statistical Substitution

This imputation technique consists of replacing any missing value with e mean or median of that variable for all other cases. This is a simple case of univariate analysis that is simple to implement and that has the problem of not accounting for any correlations between the modelled feature and the remaining features. To carry out this type of imputation, the Python package sklearn.impute.SimpleImputer can be employed.

Iterative (regression) imputation is a type of multivariate imputation here each feature with missing values is modelled as a function of other features, and where the output of the regression is used for imputation. The regression is carried in an iterated round-robin fashion: at each step until the maximum number of rounds is achieved. To carry out this type of imputation, the Python package sklearn.impute.IterativeImputer can be employed.

The matrix factorization problem that we are trying to solve can be formulated as an optimization problem:

The feature values fut, where the subscript u refers to an admission id and its chart time and the subscript i to a clinical feature, are being approximated by a matrix Xu*Yi, obtained by the multiplication of 2 factor matrices: Xu and Yi. The sparse matrix whose values (when present) are given by futhas therefore dimension M×N. M is the total number of rows where each row corresponds to a given unique combination of admission id and chart time, while N is the total number of columns where each column corresponds to a specific clinical feature or group. The 2 factor matrices Xu and Yi have dimensions M×K and K×N, respectively, where K is the number of factors. Typically, the number of factors vary from 10 to 100 depending on the data, its sparseness and computation cost. To avoid overfitting a regularization term is added to the quadratic term proportionally to the constant parameter A typically less than unity.

Both the number of factors K and the parameter A can be optimized by minimizing the square loss on a test set. In practice, this may, though not necessarily, only be done for the parameter λ in the case of very large datasets and a grid of a few factor values is used.

The objective function for matrix factorization is non-convex (because of the quadratic Xu*Yi term). Stochastic gradient descent can be used to find approximate solutions to this optimization problem, however it is too slow with very large datasets. If the set of variables Xu is fixed and treated as constants, then the objective is a convex function of Yi and vice versa.

Hence, to solve the optimization problem Yi is fixed and Xu is optimized, then Xu is fixed and Yi is optimized. This is repeated until convergence. This approach is known as ALS (Alternating Least Squares). The ALS algorithm works as follows:

Initialize Xu and Yi with small random weights:

For N in Nepochs:

For u in patients:

For i in Features:

Repeat until it converges below a given tolerance level.

To carry matrix factorization, the Python package called sklearn. decomposition.nmf (non-negative matrix factorization) can be used with the regularization term set to zero. For this, we have to ensure that the feature values comprising the predictive matrix are not negative, which is typically the case for clinical measurements.

A neural net predictive experiment was carried out where we have used a series of low and high frequency clinical data to predict hemoglobin with the help of data imputation.

To carry matrix factorization, the Python package called sklearn. decomposition.nmf (non-negative matrix factorization) can be used with the regularization term set to zero. For this, we have to ensure that the feature values comprising the predictive matrix are not negative, which is typically the case for clinical measurements.

A neural net predictive experiment was carried out where we have used a series of low and high frequency clinical data to predict hemoglobin with the help of data imputation.

This is the list of 29 input features selected as input features:anionbicarbonatebilirubinbunchloridecreatininediastolicgcsglucosehearthematocrithemoglobinInrLactatemagnesiummeanbpo2satphphosphateplateletspotassiumptpttrespirationsodiumspo2systolicTempWbc

These features have a wide range of frequencies and availability. The lab features will be most likely affected by the problem of missing data but even vital data may not totally exempted from it. For this reason, instead of sampling every half an hour, it is better to sample statistical measures for a certain period of time. In our case, we introduce the variable t for time in hours. In some implementations, feature values are collected for a period of 12 hours, from t=0.0 to t=12, where t=0 is the first time of the data collection for the analysis. In some implementations, fewer than 12 hours or more than 12 hours of data can be used to make a prediction. Then, in some implementations, a prediction is made for hemoglobin for a time no later than 24 after the last data point used in the prediction. In some other implementations, predictions can be made for times more than 24 hours after the time associated with the last collected data points.

The collected values of the predictive features can be used to compute the minimum value, the median and the maximum value of each feature over the data collection period. Using maximum, minimum and median values in lieu of a complete time series of data points across the same time period produces a lot fewer missing values which renders data imputation more effective. Furthermore, the median is used in lieu of the average because it is robust to outliers, which are not infrequent in healthcare data.

In the case that there is only one value present for a given feature, e.g., a lab value that is measured only a limited number of times, and in some cases only once, per day, maximum, median and minimum values are all set to that same value. If there is no value for a given feature, a data imputation method is used based on data for the patient collected prior to the given data collection period.

Hemoglobin is an iron protein in red-blood cells that in the human being that transport oxygen. Hemoglobin carries oxygen from the lungs to the entire body. A typical range of hemoglobin for a healthy individual is 12 to 20 grams of hemoglobin for every 100 ml of blood. A low hemoglobin count is generally defined as less than 13.5 grams of hemoglobin per deciliter of blood for men and less than 12 grams per deciliter for women. A condition of very low hemoglobin count is called anemia. High levels of hemoglobin over 17.5 grams for every ml of blood can be imputed to serious health conditions as heart failure such as. The first three most serious conditions associated with high hemoglobin count are coronary atherosclerosis, aortic valve disorder and subendocardial infarction.

Hence, the prediction of hemoglobin levels is of paramount importance as it is an indicator of the body healthiness.

The Computer Programs

For each of the three different types of imputation discussed above, a different software program is used.

With respect to the matrix factorization method, typically matrix factorization is applied to homogenous datasets with missing data. However, in this case, and in healthcare more generally, the features are often not homogeneous. For instance, maximum temperature and the minimum temperature can be considered as 2 separate and different features although they both refer to temperature. MINMAX normalization can be applied before matrix factorization imputation, leading to two beneficial results: A) negative values are removed because the NMF model works only on non-negative matrices: B) values are scaled from zero to 1 so the values are of the same order of magnitude:

Hemoglobin Model Building Using Simple Imputation Code

These are steps taken to implement a hemoglobin predictive model using simple imputation for missing data:1) We divide the matrix feature X and the output array Y into training and test sets with a 80%, 20% split done with fixed seed.2) We impute the missing numpy nans in the training and test matrix sets (X_train and X_test) using the Simplelmputer with strategy median.3) We apply the package preprocessing.MinMaxScaler to scale the X_train matrix from zero to one. Separately with the same maximum and minimum imputed from X_train we apply to X_test. Min max scaling. The formula for minmax scaling is new_value=(Old Value-Min)/(Max-Min)4) We apply the sklearn MLP (MultiLayer Perceptron) to the X_train and y_train as follows: MLPRegressor(hidden_layer_sizes=(150,50,10),activation=‘relu’,solver=‘sgd’,learning_rate=‘adaptive’,max_iter=500,learning_rate_init=0.00001,alpha=0.01,random_state=42)5) Finally, we compute the MAE of the fitted model on the X_test.

Hemoglobin Model Building Using Matrix Factorization Imputation Code

These are steps taken to implement a hemoglobin predictive model using matrix factorization imputation for missing data:1) We divide the matrix feature X and the output array Y into training and test sets with a 80%, 20% split done with fixed seed.2) We apply the package preprocessing.MinMaxScaler to scale the X_train matrix from zero to one. Separately with the same maximum and minimum imputed from X_train we apply to X_test. Min max scaling. The formula for minmax scaling is new_value=(Old Value-Min)/(Max-Min) We use the MinMaxScaler first before applying the matrix factorization method because we want neighboring features to have values of the same order.3) We model the matrix as csr sparse matrix.4) We apply the package NMF(n_components=50,alpha=0.1, random_state=0) to both X_train and X_test separately. Possible negative values in the X_test were set to zero before the application of the package.5) We reconstruct the approximate matrix by multiplying the 2 factor matrices W and H.6) We apply the sklearn MLP (MultiLayer Perceptron) to the X_train and y_train as follows: MLPRegressor(hidden_layer_sizes=(150,50,10),activation=‘relu’,solver=‘sgd’,learning_rate=‘adaptive’,max_iter=500,learning_rate_init=0.00001,alpha=0.01,random_state=42)7) Finally, we compute the MAE of the fitted model on the X_test

Hemoglobin Model Building with Iterative Imputation (Regression) Code

These are steps taken to implement a hemoglobin predictive model using iterative imputation for missing data:1) We divide the matrix feature X and the output array Y into training and test sets with a 80%, 20% split done with fixed seed.2) We impute the missing numpy nans in the training and test matrix sets (X_train and X_test) using the IterativeImputer with max_iter=10,n_nearest_features=3, random_state=0.3) We apply the package preprocessing.MinMaxScaler to scale the X_train matrix from zero to one. Separately with the same maximum and minimum imputed from X_train we apply to X_test. Min max scaling. The formula for minmax scaling is new_value=(Old Value-Min)/(Max-Min)4) We apply the sklearn MLP (MultiLayer Perceptron) to the X_train and y_train as follows: MLPRegressor(hidden_layer_sizes=(150,50,10),activation=‘relu’,solver=‘sgd’,learning_rate=‘adaptive’,max_iter=500,learning_rate_init=0.00001,alpha=0.01,random_state=42)5) Finally, we compute the MAE of the fitted model on the X_test.

Please note that the K nearest-neighbor strategy can be also used to do data imputation, however we found the memory requirement for matrices of the size of the order and above (1M by 100)) impractical.

Experimental Results

The best main MAE and range MAE's was obtained for the Iteration (Regression) Imputation model. These are the final results:

However, the resulting MAE for matrix factorization produce results within 1% of the iterative method and therefore it should be considered a competitive method to the use of iterative regression.

Conclusions

Three different data imputation methods along with neural net technology were employed to predict the level of hemoglobin in MIMIC patients 24 hours in advance. The best imputation method is the iterative imputation method, although the other 2 methods (median and NMF) yielded similar results.

In some implementations, the vital sign and lower-frequency health parameter (e.g., hemoglobin) predictions are made in the context of monitoring patients in an Intensive Care Unit of a hospital in order to detect potential negative health trends in advance. In some implementations, the predictions are made in the context of monitoring patients involved in clinical trials to obtain advance warning of potential adverse events. In some implementations, the predictions are used in the context of immunotherapy patient monitoring where potential for rapid decline in health would otherwise require a patient to remain in close proximity to a health care center, where the availability of predictive information may make increase a time window for such a patient to seek medical attention if there condition is predicted to worsen. In some implementations, the predictions are made in the context of monitoring patients recently released from a hospital, potentially allowing for earlier discharges given the availability of early warnings of health declines.

FIG. 1is an example block diagram of a system200afor predicting patient parameters (e.g., vitals or hemoglobin (or other lower frequency parameter) using machine learning according to some implementations. System200aincludes an input device201and an output device202coupled to a client204. The client204includes a processor206and a memory208storing an application210. The client204also includes a communications module212connected to network214. System200aalso includes a server216which further includes a communications module218, a processor220and a memory222. The server216also includes a model training system224. The model training system224includes a feature selector226, a model trainer228and one or more training models230. The server216also includes one or more patient parameter prediction models232, which are shown in dotted lines to indicate that the training models230, which were output during the training performed in the machine learning process, can be one or more patient parameter prediction models, such as the one or more patient parameter prediction models232.

As shown inFIG. 1, the system200aincludes an input device201. The input device201receives user input and provides the user input to client204. The input device201may include a keyboard, mouse, microphone, stylus, and/or any other device or mechanism used to input user data or commands to an application on a client, such as client204. In some implementations, the input device201may include haptic, tactile or voice recognition interfaces to receive the user input, such as on a small-format device.

The system200aalso includes a client204. The client204communicates via the network214with the server216. The client204receives input from the input device201. The client204can be, for example, a large-format computing device, a small-format computing device (e.g., a smartphone or tablet), a medical data device (e.g., a small or large-format device used in a healthcare setting to collect, manage or generate patient diagnostic data or patient record data), or any other similar device having appropriate processor, memory, and communications capabilities. The client204may be configured to receive, transmit, and store data associated with predicting patient parameters for a patient at various amounts of time into the future.

As further shown inFIG. 1, the client204includes a processor206and a memory208. The processor206operates to execute computer-readable instructions and/or data stored in memory208and transmit the computer-readable instructions and/or data via the communications module212. The memory208may store computer-readable instructions and/or data associated with predicting a patient's parameters (e.g., vitals or low frequency parameter, such as hemoglobin level) for a specified amount of time into the future. The prediction may be a time series of values over that specified amount of time, or an individual value prediction at that period of time in the future. For example, the memory208may include a database of patient data, such as patient records database115. The memory208includes an application210. The application210may be, for example, an application to receive user input or patient data for use in determining predicted patient parameters for a given patient as discussed above. In some implementations, the application210may receive user input or patient data for use in determining one or more patient parameters for a given patient at a specified amount of time into the future (time series or individual value). The application210may include textual and graphical user interfaces to receive patient data as input and display output including predicted patient parameters for a given patient at one or more amounts of time into the future. The application210may include a number of configurable settings associated with triggering alerts or user notifications when one or more of the particular patient's parameters falls below or above a threshold. Additionally, or alternatively, the application210may output an indication, in a graphical user interface, identifying the amount of time in the future at which a parameter for a given patient is expected to exceed or fall below the applicable threshold value(s). In some implementations, the application210may output a list of patients for whom any predicted patient parameter is predicted to fall outside a designated safe range at one or more times in the future.

As shown inFIG. 1, the client204includes a communications module212. The communications module212transmits the computer-readable instructions and/or patient data stored on or received by the client204via network214. The network214connects the client204to the server216. The network214can include, for example, any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. Further, the network214can include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, and the like.

As further shown inFIG. 1, the server216operates to receive, store and process the computer-readable instructions and/or patient data generated and received by client204. In some implementations, the server216may receive patient data directly from one or more patient monitoring devices or an electronic medical records server. The server216can be any device having an appropriate processor, memory, and communications capability for hosting a machine learning process. In certain aspects, the server216can be located on-premises with client204, or the server216may be located remotely from client204, for example in a cloud computing facility or remote data center. The server216includes a communications module218to receive the computer-readable instructions and/or patient data transmitted via network214. The server216also includes one or more processors220configured to execute instructions that when executed cause the processors to determine predicted patient parameters for a given patient at a specified (or unspecified) amount of time into the future. The server216also includes a memory222configured to store the computer-readable instructions and/or patient data associated with predicting health parameters for a given patient at a specified (or unspecified) amount of time into the future. For example, the memory222may store one or more models, such as the vital sign or low-frequency health parameter prediction models232generated during the training of a machine learning process which have been trained to output patient parameters for patients at various amounts of time into the future. In some implementations, the memory222may store one or more machine learning algorithms that will be used to generate one or more training models. In some implementations, the memory222may store patient data that is received from client204and is used as a training dataset in the machine learning process in order to train a patient parameter prediction model. In some implementations, the memory222may store one or more trained prediction models that are used to predict vital signs or a low frequency parameter such as hemoglobin level.

As shown inFIG. 1, the server216includes a model training system224. The model training system224functions in a machine learning process to receive patient data as training input and processes the patient data to train one or more training models. The model training system224includes a feature selector226, a model trainer228, and one or more training models230. In some implementations, the training models230that are generated and output as a result of the machine learning process are configured on server216as standalone components on server216. For example, the patient parameter prediction models232are configured on server216to process patient data and output a patient's parameter(s) for specified amounts of time into the future. In some implementations, the patient parameter prediction models232are stored in memory222on server216.

The model training system224is configured to implement a machine learning process which will receive patient data as training input and generate a training model that can be subsequently used to predict patient parameters at specified amounts of time into the future. The components of the machine learning process operate to receive patient data as training input, select unique subsets of features within the patient data, use a machine learning algorithm to train a model based on the subset of features in the training input and generate a training model that may be output and used for future predictions based on a variety of received patient data.

As shown inFIG. 1, the model training system224includes a feature selector226. The feature selector226operates in the machine learning process to receive patient data and select a subset of features from the patient data which will be provided as training inputs to a machine learning algorithm. In some implementations, the feature selector226may select a subset of features corresponding to a given patient parameter such that the machine learning algorithm will be trained to predict such parameter based on the selected subset of features. In other implementations, the feature processor226may select different subsets of features which do not correspond to patient data commonly used to determine parameter in question. By using a variety of training inputs, the machine learning process will generate a trained model that is able to predict a patient's parameter value from a wide variety of disparate patient data.

During the machine learning process, the feature selector226provides the selected subset of features to the model trainer228as inputs to a machine learning algorithm to generate one or more training models. A wide variety of machine learning algorithms may selected for use including algorithms such as support vector regression, ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS), ordinal regression, Poisson regression, fast forest quantile regression, Bayesian linear regression, neural network regression, decision forest regression, boosted decision tree regression, artificial neural networks (ANN), Bayesian statistics, case-based reasoning, Gaussian process regression, inductive logic programming, learning automata, learning vector quantization, informal fuzzy networks, conditional random fields, genetic algorithms (GA), Information Theory, support vector machine (SVM), Averaged One-Dependence Estimators (AODE), Group method of data handling (GMDH), instance-based learning, lazy learning, and Maximum Information Spanning Trees (MIST).

The model trainer228evaluates the machine learning algorithm's prediction performance based on patterns in the received subset of features processed as training inputs and generates one or more new training models230. The generated training models, e.g., patient parameter prediction models232, are then capable of receiving patient data outside of the machine learning process in which they were trained and generated to output predicted parameter values at specified amounts of time into the future for a given patient.

As further shown inFIG. 1, the patient parameter prediction models232that were generated as a result of performing the machine learning process, may receive patient data and process the patient data to output predicted parameter values to the processor220. For example, the patient parameter prediction models232, that were produced in the machine learning process, may be subsequently be included in an artificial intelligence system or application configured to receive patient data as prediction inputs and process the data to output parameter value predictions for a patient at specified amounts of time into the future. In some implementations, the processor220may store the predicted parameter value output from the prediction model232in memory222. In some implementations, the memory222may store instructions to adjust or transform the received patient data based on the parameter input requirements of the prediction model. For example, the feature selector226may normalize values or impute missing values. In other implementations, the outputted patient parameter predictions may be forwarded to communications module218for transmission to the client204via network214. Once received by the client204, the outputted prediction may be transmitted to output device202, such as a monitor, printer, portable hard drive or other storage device. In some implementations, the output device202may include specialized clinical diagnostic or laboratory equipment that is configured to interface with client204and may display the predicted parameter values in conjunction with the diagnostic or laboratory data for which the specialized clinical diagnostic or laboratory equipment is normally configured to output.

FIG. 2is a block diagram illustrating an example computer system600with which the client204, server216, and server202ofFIGS. 1 and 2can be implemented. In certain aspects, the computer system600may be implemented using hardware or a combination of software and hardware, either in a dedicated server, or integrated into another entity, or distributed across multiple entities.

Computer system600(e.g., client204, server216, and server202) includes a bus608or other communication mechanism for communicating information, and a processor602(e.g., processors206and220) coupled with bus608for processing information. According to one aspect, the computer system600can be a cloud computing server of an IaaS that is able to support PaaS and SaaS services. According to one aspect, the computer system600is implemented as one or more special-purpose computing devices. The special-purpose computing device may be hard-wired to perform the disclosed techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be large-format computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques. By way of example, the computer system600may be implemented with one or more processors602. Processor602may be a general-purpose microprocessor, a microcontroller, a Digital Signal Processor (DSP), an ASIC, a FPGA, a Programmable Logic Device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable entity that can perform calculations or other manipulations of information.

Computer system600can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them stored in an included memory (e.g., memory208or222), such as a Random Access Memory (RAM), a flash memory, a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable PROM (EPROM), registers, a hard disk, a removable disk, a CD-ROM, a DVD, or any other suitable storage device, coupled to bus608for storing information and instructions to be executed by processors208or220. The processor602and the memory604can be supplemented by, or incorporated in, special purpose logic circuitry. Expansion memory may also be provided and connected to computer system600through input/output module610, which may include, for example, a SIMM (Single In-Line Memory Module) card interface. Such expansion memory may provide extra storage space for computer system600, or may also store applications or other information for computer system600. Specifically, expansion memory may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory may be provided as a security module for computer system600, and may be programmed with instructions that permit secure use of computer system600. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

Computer system600further includes a data storage device606such as a magnetic disk or optical disk, coupled to bus608for storing information and instructions. Computer system600may be coupled via input/output module610to various devices (e.g., device614or device616. The input/output module610can be any input/output module. Example input/output modules610include data ports such as USB ports. In addition, input/output module610may be provided in communication with processor602, so as to enable near area communication of computer system600with other devices. The input/output module602may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used. The input/output module610is configured to connect to a communications module612. Example communications modules (e.g., communications module612include networking interface cards, such as Ethernet cards and modems).

The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. The communication network (e.g., communication network214) can include, for example, any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. Further, the communication network can include, but is not limited to, for example, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, or the like. The communications modules can be, for example, modems or Ethernet cards.

For example, in certain aspects, communications module612can provide a two-way data communication coupling to a network link that is connected to a local network. Wireless links and wireless communication may also be implemented. Wireless communication may be provided under various modes or protocols, such as GSM (Global System for Mobile Communications), Short Message Service (SMS), Enhanced Messaging Service (EMS), or Multimedia Messaging Service (MMS), CDMA (Code Division Multiple Access), Time division multiple access (TDMA), Personal Digital Cellular (PDC), Wideband CDMA, General Packet Radio Service (GPRS), or LTE (Long-Term Evolution), among others. Such communication may occur, for example, through a radio-frequency transceiver. In addition, short-range communication may occur, such as using a BLUETOOTH, WI-FI, or other such transceiver.

In any such implementation, communications module612sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information. The network link typically provides data communication through one or more networks to other data devices. For example, the network link of the communications module612may provide a connection through local network to a host computer or to data equipment operated by an Internet Service Provider (ISP). The ISP in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet”. The local network and Internet both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link and through communications module612, which carry the digital data to and from computer system600, are example forms of transmission media.

Computer system600can send messages and receive data, including program code, through the network(s), the network link and communications module612. In the Internet example, a server might transmit a requested code for an application program through Internet, the ISP, the local network and communications module612. The received code may be executed by processor602as it is received, and/or stored in data storage606for later execution.

In certain aspects, the input/output module610is configured to connect to a plurality of devices, such as an input device614(e.g., input device201) and/or an output device616(e.g., output device202). Example input devices614include a keyboard and a pointing device, e.g., a mouse or a trackball, by which a user can provide input to the computer system600. Other kinds of input devices614can be used to provide for interaction with a user as well, such as a tactile input device, visual input device, audio input device, or brain-computer interface device. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, tactile, or brain wave input. Example output devices616include display devices, such as a LED (light emitting diode), CRT (cathode ray tube), LCD (liquid crystal display) screen, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, for displaying information to the user. The output device616may comprise appropriate circuitry for driving the output device616to present graphical and other information to a user.

According to one aspect of the present disclosure, the client204and servers216can be implemented using a computer system600in response to processor602executing one or more sequences of one or more instructions contained in memory604. Such instructions may be read into memory604from another machine-readable medium, such as data storage device606. Execution of the sequences of instructions contained in main memory604causes processor602to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in memory604. Processor602may process the executable instructions and/or data structures by remotely accessing the computer program product, for example by downloading the executable instructions and/or data structures from a remote server through communications module612(e.g., as in a cloud-computing environment). In alternative aspects, hard-wired circuitry may be used in place of or in combination with software instructions to implement various aspects of the present disclosure. Thus, aspects of the present disclosure are not limited to any specific combination of hardware circuitry and software.

Various aspects of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. For example, some aspects of the subject matter described in this specification may be performed on a cloud-computing environment. Accordingly, in certain aspects a user of systems and methods as disclosed herein may perform at least some of the steps by accessing a cloud server through a network connection. Further, data files, circuit diagrams, performance specifications and the like resulting from the disclosure may be stored in a database server in the cloud-computing environment, or may be downloaded to a private storage device from the cloud-computing environment.