Patent Description:
Computing systems can be used to collect, store, manage, and interpret generated data. Using such systems, computer users can monitor the data.

It may be useful to use data collected by a system in predictive ways. For example, the data may be used to predict when and/or the probability that certain events will occur.

Previously, to implement such predictive functionality, data would be provided to a specialized machine learning system that had a known specialized functionality. Thus, a system would collect data and provide such data to the machine learning system that would provide the appropriate prediction function that could then be used by the original system to perform predictive functionality. However, this required using a large number of very specialized machine learning systems, one for each type of machine learning predictive functionality.

<CIT> describes a computer-implemented method including receiving, in a system of one or more computers, training data for predictive modeling, the training data including a plurality of categories; determining, by the system, one or more attributes of the training data; identifying, by the system in a mapping of attributes to types of predictive models, a type of predictive model that is mapped to at least one of the one or more attributes; obtaining a utility function for the predictive model of the identified type, the utility function specifying importance of the plurality of categories relative to each other; and generating, based on the training data and the utility function, a predictive model of the identified type.

It is the object of the present invention to provide an improved method and system for applying machine learning prediction to data.

Training prediction models and applying machine learning prediction to data is illustrated herein. A prediction instance comprising a set of data and metadata associated with the set of data identifying a prediction type is obtained. The data and metadata are used to determine an entity to train a prediction model using the prediction type. As a result, a trained prediction model is obtained from the entity.

Predicting and reacting to future events has typically involved specialized specific information being used in making predictions.

"Machine learning" (ML) can learn from examples of previously gathered data ("training data"), creating a prediction model, and subsequently predicting based on new data and the prediction model. In the embodiments illustrated herein, ML is able to analyze previously generated data and provide "insights" without the user requesting such insights. A machine learning system can run in the background and use a notification subsystem to let the user know of a given condition - or even initiate workflows following predefined rules to initiate favorable actions.

One example of machine learning-based prediction is inventory forecasting in which a machine learning algorithm, based on inventory levels over time for a product and potentially other information in an inventory control system, can predict future inventory levels of a product. Another example is payment time prediction for sales invoices in which a machine learning algorithm using training data on past sales invoices from a payment system (e.g., amount, payment terms, customer identifiers, and number of late days in payment) may predict the probability of a customer paying late on a new sales invoice.

While the above examples illustrate how machine learning might be used in an enterprise, machine learning algorithms may be used for other purposes, such as predicting traffic flows, weather patterns, event attendance, disease outbreaks or virtually any other predictive analysis.

Some embodiments illustrated herein can be implemented in a fashion that simplifies a base system by reducing the complexity of an interface between a base system and specialized machine learning components. In particular, embodiments illustrated herein implement a subsystem that can analyze data and metadata from a base system and determine machine learning processing that should be performed on the data and provide the data to the appropriate components for performing the machine learning processing. In contrast, a previous base system would necessarily have more complex interfaces with machine learning systems, where the base systems would need to have logic built into the base system itself for sending the data to the appropriate machine learning components.

Additionally, or alternatively, embodiments may be implemented where prediction can be integrated into the workflows of end users. For example, it may be useful to automate the purchase of new inventory on the basis of inventory level predictions. This can automatically be performed based on the results of predictive analysis.

Additionally, or alternatively, embodiments may be implemented where predictions can be integrated into the context of end users. For example, a purchasing agent will not typically use predictive analysis themselves to create a purchase invoice, but rather will wait for direction from others to create the purchase invoice based on the other predictive analysis. However, some embodiments described herein can provide direction to end users directly, e.g., provide a contextually relevant suggestion, without additional direction, to perform various actions based simply on the results of a predictive analysis.

Some embodiments of the invention illustrated herein enable machine learning by associating metadata with data, such as tabular data in tables and columns in a data storage and/or processing system. In some embodiments, this may be done by actually adding metadata to tables in table cells as illustrated in <FIG>, which shows a table <NUM> which includes table metadata <NUM> applicable to the entire table <NUM> and column metadata <NUM> applicable to individual columns in the table <NUM>. Table metadata <NUM> in the table <NUM> (or in some embodiments in a side-structure associated with the table) describes one or more prediction types that should be used for data in the table <NUM>. For example, in <FIG>, the table metadata <NUM> identifies that a regression analysis should be used for the data in the table <NUM> to create a prediction model and for prediction tasks. In the illustrated example, the metadata can be metadata that supports prediction types that work on tabular data. Such prediction types may include time series forecasting (i.e., when future values in a time series is being predicted), regression (i.e., when a continuous value is being predicted), classification (two-class and/or multi-class classification) (i.e., when the data are being used to predict a category), anomaly detection (i.e., identifying unusual data points), and/or clustering (i.e., grouping data in a relevant way).

Column metadata <NUM> describes the role of columns as identifiers, features, or labels. Examples of each of these are illustrated in <FIG>. In this example, an identifier identifies a specific instance. In machine learning, a feature is quantifiable property of a thing being observed. A label in supervised machine learning is that which is to be predicted. In the machine learning examples illustrated herein, a prediction instance can be used for predictions. A prediction instance, as used herein, can include one or more of training data, machine learning metadata (including table metadata and/or column metadata as described below), a trained prediction model, and/or a record for prediction (i.e., a record that is missing one or more labels).

Initially, a prediction instance will often only include training data. The training data is data previously collected. This data will include features and labels.

Machine learning metadata can be added to the prediction instance as illustrated below. The machine learning metadata can define the type of analysis, i.e., a prediction type, to be performed on the training data, as well as identifying features and labels in the training data. Training can be performed using the training data, according to the metadata, to create a prediction model that helps to identify patterns in the data, that can be used for predictions. A prediction instance having a trained model associated with it is referred to herein as a trained prediction instance.

Some prediction instances can have records for prediction added to them. These may be records that are missing one or more labels. The prediction model can be applied to predict the one or more labels. Alternatively, in clustering or classification analysis, records for prediction may be complete as the analysis is typically performed on complete records. An example is now illustrated.

<FIG> illustrates the table <NUM> with table metadata <NUM> describing that data in the table <NUM> is used for the machine learning type "regression". In this case, a prediction model is trained using past training data taken from columns <NUM>, <NUM><NUM> and <NUM> (the features). "Late Days" (the label) in column <NUM>, for a new record added to the table that includes all data but the label, will be predicted based on the observed features: "Cust. Late Days", "Total Amount", and "Payment Terms" in the columns <NUM>, <NUM> and <NUM> for a particular sales invoice identified by an identifier as illustrated in column <NUM>. Thus, for example, for the new invoice (not shown) with an identifier <NUM>, "Late Days", <NUM> can be predicted based on knowing that the "Cust. Late Days" is <NUM>, the "Total Amount" is <NUM>, and "Payment Terms" is COD, and using training data from previous invoices, including previous "Cust. Late Days", "Total Amount", and "Payment Terms" and previous labels, such as "Late Days" in other records (e.g., records having identifiers <NUM>-<NUM>) in the present example.

Given the table metadata identifying a prediction type, embodiments of the invention can automatically train a model based on existing data and labels and predict labels for new data. For example, data and metadata can be provided to a machine learning service <NUM> (see <FIG>) which can automatically use the data <NUM> as training data and the metadata <NUM> (which may include the table metadata <NUM> and column metadata <NUM>) to determine a prediction type to apply to the data <NUM>. The data <NUM> may be historical data stored in a data store. Such data may include data marked as features and data marked as labels by column metadata.

Furthermore, because machine learning is declarative, machine learning may run in the background and/or as a batch, enabling proactive notification of predictions and inclusions in workflows. This enables a system that requires less direct user interaction and creates a more efficient system. Indeed, user interaction can affect a system's performance as the system pauses for interrupts and uses hardware intensive user interface inputs. Rather, the system is more efficient as it is able to perform actions autonomously without the need for user direction to identify certain conditions or predictions.

Further, as illustrated below, embodiments can optimize resource usage by selecting lower cost resources when appropriate. For example, if a prediction analysis, i.e., evaluation of a prediction type, is simple (such that it can be computed by a simple program in a local system), the computer program will not use external machine learning resources, but rather allow the local system to perform the analysis. An example of this would be classification or regression based on decision trees or time series forecasting based on a naive model. In general, if a linear execution time (in terms of input data size) program can be devised to perform the prediction, the prediction could be performed locally.

In some embodiments, based on the knowledge of features, embodiments of the invention may optimize machine learning resource usage by executing acts for preprocessing data locally rather than sending the data to a machine learning service <NUM>. For example, in time series forecasting, embodiments of the invention may, at the local system <NUM>, and particularly at the ML optimization subsystem <NUM> as discussed in more detail below, decide that a time series is white noise and thus prevent the data <NUM> and metadata <NUM> from being sent to the machine learning service <NUM> where analysis would be unproductive, thus saving remote machine learning resources at the machine learning service <NUM>. Another example is to not perform training for classification or regression if the training data size is below a threshold. In general, if it can be determined that the data is too noisy (as in the first example) or there is too little data, data should not be sent.

As noted above, embodiments are able to provide users with insights based on machine learning proactively, i.e., without the user explicitly requesting these insights. This is supported through, for example, enabling machine learning by metadata associated tables and columns, background processing and context and workflow integration enabled by metadata descriptions, optimization of machine learning resource usage based on metadata, etc..

Some embodiments of the invention may implement various components as illustrated in <FIG> illustrates an architecture diagram of machine learning in a system <NUM>. One such system may be Dynamics NAV available from Microsoft Corporation of Redmond, Washington. The subsystems below the dotted line are part of the system <NUM> (except as noted below), the items above the dotted line are external to the system <NUM>, in the illustrated example.

These components illustrated in <FIG> include an ML subsystem <NUM> included in the system <NUM>. Note that while the ML subsystem <NUM> is shown in the system <NUM>, it should be appreciated that the ML subsystem may be implemented in some embodiments as a separate system.

The ML subsystem <NUM> illustrated includes an ML prediction subsystem <NUM>, an ML optimization subsystem <NUM>, an ML notification subsystem <NUM> and an ML workflow subsystem <NUM>. Details of these are now illustrated.

The ML prediction subsystem <NUM> uses metadata annotations associated with tables. <FIG> shows an example of such an annotation in which a table has been annotated to be used in regression prediction type. A prediction type may then be instantiated and used by the developer in training a prediction model. Consider the following code with reference to <FIG> and <FIG>:
<IMG>
<IMG>.

Here, as illustrated at <NUM> in <FIG>, this table (InvoiceLateDaysPrediction) is annotated to be used in machine learning through table metadata <NUM> on the table <NUM>. This is illustrated by the table metadata: "MachineLearningType=Regression" and illustrated by the code above and in the table metadata <NUM> in <FIG>.

In some embodiments, this table metadata <NUM> may be omitted in which case the type is deduced from the types of the label and features. The column metadata <NUM> specifies that it is late days that needs to be predicted by the column metadata including: "MachineLearningRole=Label" based on data in the remaining columns as indicated by the metadata "MachineLeamingRole=Feature".

<FIG> illustrates creating a prediction instance at <NUM> and invoking prediction training at <NUM>. Consider the following additional code which shows programmatically how these two acts are performed. <IMG>
<IMG>.

Based on the table metadata <NUM>, column metadata <NUM>, and training data including previous records in data (with records also containing values for the label), the machine learning prediction subsystem <NUM> will choose an appropriate prediction model and train the chosen model (either locally or using the machine learning service <NUM>), making it available to the system <NUM> for future predictions of records not containing values for the label.

The ML Optimization Subsystem <NUM> detects if training may be run locally (on the server) to not use external ML prediction resources. An example would be to calculate the autocorrelation function (ACF) to detect that a time series is white noise and thus that advanced prediction at the machine learning service <NUM> is neither necessary nor useful. The calculation ACF is cheap and subsequently a prediction may be a simple mean of historical values. Thus, as illustrated at <NUM> in <FIG>, training may occur locally (e.g., at the ML subsystem <NUM> or the base system <NUM> of the system <NUM>).

If training is not performed locally, training the model will be performed at the machine learning service <NUM> as illustrated at <NUM>.

In either case, the trained model (or an identification of the trained model) can be returned to the system <NUM> where it can be used to update the prediction instance with the trained model as illustrated at <NUM> and used for subsequent predictions.

This drastically simplifies the task of programming predictions for the programmer since the programmer can program in terms of familiar abstractions (e.g., tables and metadata/properties) rather than machine learning abstractions (e.g., experiments, training, statistics). This means that the programmer can use predictions directly in their tailored solutions.

<FIG> illustrates a flowchart for simple prediction. <FIG> illustrates inserting a record into a prediction instance for training (see act <NUM> and <NUM>). This may be done by adding a record missing a label (for the illustrated example) to the trained prediction instance created by the acts shown in <FIG>. This is illustrated by the following code. // Insert record without label for prediction
PaymentTime. INIT;
PaymentTime. "Invoice No." := '<NUM>';
PaymentTime. Late Days" := <NUM>;
PaymentTime. "Total Amount" := <NUM>;
PaymentTime. "Payment Terms" := 'COD';
PaymentTime.

Prediction is then initiated on the prediction instance including the record without the label as illustrated by the following code:
PaymentTimePrediction. SETCALLBACK(CODEUNIT:: "Payment Time
Training",'Callback');
PaymentTimePrediction. PREDICT(PaymentTime);
END;.

A determination may be made as to whether the prediction should be performed locally or by the machine learning service <NUM> as described above. The prediction is either performed locally at the system <NUM> as illustrated at <NUM> or as illustrated at <NUM> by the machine learning service <NUM>. In either case, the inserted record is updated with the predicted label as illustrated at <NUM>.

The ML Notification Subsystem <NUM> uses predicted values to identify situations in which users should be notified in the context of their work. An example would be in the context of a sales invoice in which the user is notified by the means of a non-blocking message box that the customer might pay late. In an alternative example, a user may be notified while they are creating a purchase invoice that is based on sales forecasts and inventory that a user should restock one or more items and add certain items to the purchase invoice. An example of this is illustrated in <FIG>.

In particular, <FIG> illustrates a user interface <NUM>. A user is in the context of creating a purchase invoice from a distributor in the user interface <NUM> of a base application or extension <NUM> (see <FIG>). Sales forecasting can be performed by the machine learning service <NUM>, which identifies probable sales for lamps available from the distributor. The ML notification subsystem <NUM> can compare a current stock of lamps to the probable sales and identify that additional lamps are needed to be able to fulfill the orders. The ML notification subsystem <NUM> can notify a notification subsystem <NUM> of the base system <NUM> which can then display the notification <NUM> in the invoice <NUM> being created by the user at the base application or extension <NUM>. The notification <NUM> may include a selectable link that allows a purchase line requesting lamps to be added to the invoice <NUM>.

Note that in some embodiments, an action may be taken automatically instead of prompting the user. For example, the purchase line may be added automatically to the invoice without user interaction. Illustratively, The ML workflow subsystem <NUM> uses predicted values to produce events that are reacted to by a workflow subsystem <NUM> in the base subsystem <NUM>. For example, the prediction of a product going out of stock may start an automated workflow that creates a purchase invoice to reorder the product or adds a line to a purchase invoice. Thus for example, the purchase invoice <NUM> illustrated in <FIG> may be created automatically for the user or a line may be added to the purchase invoice <NUM> when evaluation of a prediction instance identifies that based on a sales forecast and inventory predictions that a product should be restocked.

<FIG> illustrates an example where a user is in the context of viewing inventory information in a user interface <NUM>. In the example illustrated in <FIG>, a user is presented with an inventory forecast <NUM>. The inventory forecast <NUM> may be provided by the machine learning service <NUM> to the ML notification subsystem <NUM>, which provides the forecast <NUM> to the notification subsystem <NUM>. The notification subsystem <NUM> can display the inventory forecast <NUM> in the user interface <NUM>. Here, various prompts can be provided to the user or the user has various user interface options that the user can initiate sua sponte. For example, a user can be prompted with a prompt <NUM> asking whether the user would like automated updated forecast data according to some schedule. If the user responds in the affirmative, a workflow item can be added to the ML workflow subsystem <NUM> to provide automated forecast data according to the schedule. Alternatively or additionally, the user could manually specify parameters that would cause a workflow item to the ML workflow subsystem <NUM>.

Thus, a user could manually initiate other work in an inline process, inline with receiving forecast data. In particular, embodiments can allow a user to manually initiate tasks from the same user interface that provides the machine learning prediction. Thus, for example, a user can view the forecast <NUM>. The user may be able to right click on the forecast <NUM> and be presented with a number of options. Such options may include options that allow the user to see different predictions that resulted in the displayed prediction (in this example, the forecast <NUM>). Thus, in the illustrated example, a user can initiate displaying an inventory forecast and/or a sales forecast. Alternatively, the user may be able to initiate actions based on the forecast <NUM>. For example, as illustrated, a user may be able to create an invoice to purchase inventory.

Referring now to <FIG>, a method <NUM> is illustrated. The method <NUM> may be practiced in a computing environment. The method includes acts for applying machine learning prediction types to data.

The method includes obtaining a prediction instance comprising a set of data and metadata associated with the set of data, the metadata including a prediction type (act <NUM>). For example, in the illustrated example, the data may include invoice numbers, average days late on invoices, total amount of a particular invoice, payment terms, and potentially late days for a particular invoice. In other examples, other enterprise data, or other data generally may be obtained. The data may include or be associated with table joins or other operations to obtain the data. The metadata may include an identification of a machine learning type (e.g., regression) identification of identifiers, identification of features and/or identification of labels in the data <NUM>.

The method <NUM> further includes based on the data and metadata determining an entity to train a prediction model using the prediction type (act <NUM>). For example, the system <NUM> may analyze the data <NUM> and metadata <NUM> and determine whether to send the data to the machine learning service <NUM> to train the model or to have a model trained locally. Sometimes the data <NUM> and metadata <NUM> will not be sent to the machine learning service <NUM> because analysis can be conducted locally and/or the machine learning service <NUM> would not provide meaningful analysis.

The method <NUM> further includes, as a result, obtaining a trained prediction model from the entity (act <NUM>). Thus, for example, a prediction model can be obtained from the machine learning service <NUM> or from the machine learning prediction subsystem <NUM>.

The method <NUM> may be practiced where the entity is the remote machine learning service. As illustrated in <FIG> (at <NUM>) and <NUM> (at <NUM>) training and prediction are performed by the remote machine learning service <NUM>. Further, reference is now made to <FIG>, which illustrates a method for determining to provide the data and metadata to the remote machine learning service. <FIG> illustrates analyzing the data and metadata (act <NUM>). <FIG> further illustrates identifying that training the prediction instance with the data exceeds a predetermined complexity threshold, for example in terms of size of training data, (act <NUM>). The method <NUM> further includes, as a result, providing the data and meta data to a machine learning service (act <NUM>).

As noted above, the metadata identifies a machine learning prediction type. In this case, the data and the metadata are provided to the machine learning service which applies a generic example of the machine learning prediction type identified. Thus in the illustrated example, the metadata <NUM> identifies a regression prediction type which then causes the data <NUM> to be applied to a generic regression prediction type at the machine learning service <NUM>.

In some embodiments the method further includes monitoring user contextual information and proactively applying the prediction model to the prediction instance when contextually relevant and providing contextually relevant suggestions based on the results of applying the prediction model to the prediction instance. Such context may be point of execution in an application, user interface screen, user physical location, user role, role in and state of workflow, state of application data, etc. Predictions may be automatically made. Embodiments can match predictions to context. Based on the context and prediction match, suggestions may be provided to the user. For example, as illustrated above, if a user is in a purchase invoice user interface context and a prediction is made that additional inventory of an item is needed, the user can be prompted to add the item to the purchase invoice. For example, <FIG> illustrates a method <NUM> for proactively applying a prediction model including determining a user context (act <NUM>). The method <NUM> further includes determining contexts applicable to a prediction model (act <NUM>). The method <NUM> further includes determining that a user context matches a context applicable to a prediction model (act <NUM>).

In some embodiments the method <NUM> further includes monitoring current conditions and automatically performing a function based on application of the prediction model to the prediction instance. For example, as illustrated above, invoices could be automatically generated when new inventory is needed. For example, <FIG> illustrates a method <NUM> for monitoring current conditions and automatically performing a function based on application of the prediction model to the prediction instance. The method <NUM> includes determining a condition (act <NUM>). The method <NUM> further includes determining that the condition matches a condition applicable to a prediction model (act <NUM>). The method <NUM> further includes, as a result, performing a set of tasks (act <NUM>).

The method <NUM> may be practiced where the entity a local system. For example, as illustrated above, the ML optimization subsystem <NUM> may determine that a time series is white noise and that analysis by the machine learning service <NUM> would not be useful. <FIG>, illustrates a method for determining to provide the data and metadata to the local system. <FIG> illustrates analyzing the data and metadata (act <NUM>). <FIG> further illustrates identifying that training the prediction instance with the data is below a predetermined complexity threshold or would not benefit from training by an external service (act <NUM>).

The method <NUM> may be practiced where the metadata is included in a data table with the data, such as is illustrated in <FIG>. Alternatively, the method <NUM> may be practiced where the metadata is included in a side structure separate from a table with the data.

Physical computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage (such as CDs, DVDs, etc), magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry or desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

Claim 1:
A computer system comprising:
one or more processors; and
one or more computer-readable media having stored thereon instructions that are executable by the one or more processors to configure the computer system to apply machine learning prediction to data, including instructions that are executable to configure the computer system to perform at least the following:
obtain a prediction instance comprising a set of training data (<NUM>) and metadata (<NUM>) associated with the set of training data, the metadata including a prediction type based on the training data and metadata, determining whether training the prediction instance with the set of training data exceeds a predetermined complexity threshold, including determining that training the prediction instance with the set of training data does not exceed the predetermined complexity threshold if:
according to the prediction type, a program with linear execution time in terms of size of the training data can be devised to train the prediction model; or
a time series of the set of training data is white noise; or
a size of the training data is below a threshold;
if a time series in the set of training data is white noise or if a size of the training data is below a threshold, not performing training;
if a program with linear execution time in terms of size of the training data can be devised to train the prediction model, training the prediction model locally at the computer system; and
only if it has been determined that the predetermined complexity threshold is exceeded, sending the set of training data and metadata to an entity being a machine learning service (<NUM>)
remote from the computer system and as a result, obtaining a trained prediction model from the entity.