Dual model incremental learning

In an approach to efficient model adjustment utilizing a dual model system, one or more computer processors create a subset of a dataset utilizing a trained primary model; create a secondary model based on the created subset of the dataset; calculate a confidence of a case utilizing the trained primary model, wherein the confidence is a robustness indicator of a model indicating a capacity of the model to meet or exceed performance when applied to the dataset; responsive to the calculated confidence not exceeding a confidence threshold, generate a score of the case utilizing the created secondary model; responsive to an incorrect classification, update the created subset of the dataset with the case; retrain the secondary model utilizing the updated subset of the dataset.

BACKGROUND

The present invention relates generally to the field of machine learning and more particularly to incremental learning.

Incremental learning is a method of machine learning in which input data is continuously used to extend knowledge of an existing model (i.e., to further train the model). Incremental learning represents a dynamic technique of supervised learning and unsupervised learning that can be applied when training data becomes available gradually over time or its size is out of system memory limits. Algorithms that can facilitate incremental learning are known as incremental machine learning algorithms. Many traditional machine learning algorithms inherently support incremental learning. Other algorithms can be adapted to facilitate incremental learning. Examples of incremental algorithms include decision trees, decision rules, artificial neural networks, and incremental SVM. The aim of incremental learning is for the learning model to adapt to new data without forgetting its existing knowledge, it does not retrain the model. Some incremental learners have built-in some parameter or assumption that controls the relevancy of old data, while others, called stable incremental machine learning algorithms, learn representations of the training data that are not even partially forgotten over time. Incremental algorithms are frequently applied to data streams or big data, addressing issues in data availability and resource scarcity respectively. Stock trend prediction and user profiling are some examples of data streams where new data becomes continuously available. Applying incremental learning to big data aims to produce faster classification or forecasting times.

SUMMARY

Embodiments of the present invention disclose a computer-implemented method, a computer program product, and a system for efficient model adjustment utilizing a dual model system. The computer-implemented method includes one or more computer processers creating a subset of a dataset utilizing a trained primary model. The one or more computer processors create a secondary model based on the created subset of the dataset. The one or more computer processors calculate a confidence of a case utilizing the trained primary model, wherein the confidence is a robustness indicator of a model indicating a capacity of the model to meet or exceed performance when applied to the dataset. The one or more computer processors responsive to the calculated confidence not exceeding a confidence threshold, generate a score of the case utilizing the created secondary model. The one or more computer processors responsive to an incorrect classification, update the created subset of the dataset with the case. The one or more computer processors retrain the secondary model utilizing the updated subset of the dataset.

DETAILED DESCRIPTION

Robotics, system monitoring, and user-modeling in real time require adaptive systems that can capture the information from new coming data, change when necessary with their environments. Traditionally, batch learning is utilized when a whole training set is available at the beginning of the learning process. When new data is added to database, the existing model built on past data may or may not stay accurate. Incremental learning is a method of machine learning, in which input data is continuously used to extend the existing model's knowledge (i.e., to further train the model) Like batch learning, incremental learning aims at minimizing the generalization error but with a growing training set. Incremental learning is a two-step procedure that is applied at an arrival of new observations. A system initiates a complete relearning of all the parameters using the training data observed so far when a new example fails to be correctly classified and evaluates if the prediction for the new example is correct. Then, if the prediction for the new example is not correct, the system adds new parameters to be learnt with the whole network (for instance, a neuron is created, if the classifier is a neural network). A significant drawback for said incremental learning system is that the cost for a complete re-learning process is very large under a big data concept.

Embodiments of the present invention allow provide a fast, low cost model adjustment and adaptation system. Embodiments of the present invention train a primary model, detects, and collects cases that have low confidence, not necessarily wrongly classified. Embodiments of the present invention expand the low confidence cases using model information to form an auxiliary dataset. Embodiments of the present invention create a second model, referred to as the auxiliary model, based on the auxiliary data for fast adjustment. Embodiments of the present invention recognize that system efficiency is increased due to the utilization of the auxiliary data which is substantially smaller than the original training set. Embodiments of the present invention rebuild the primary model utilizing the entire data set only when a significant amount of low confidence cases are cumulated. Embodiments of the present invention utilizes a collaboration of dual models to generate better predictions when compared with a single predictive model. Embodiments of the present invention endow a more efficient framework for incremental learning. Embodiments of the present invention recognize that system efficiency is gained by reducing a size of a training dataset and corpus. Embodiments of the present invention recognize reducing a training corpus, reduces training time by subsequent models. For example, reducing a training corpus by a third, reduces training time at least a third. Implementation of embodiments of the invention may take a variety of forms, and exemplary implementation details are discussed subsequently with reference to the Figures.

FIG.1is a functional block diagram illustrating a computational environment, generally designated100, in accordance with one embodiment of the present invention. The term “computational” as used in this specification describes a computer system that includes multiple, physically, distinct devices that operate together as a single computer system.FIG.1provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by those skilled in the art without departing from the scope of the invention as recited by the claims.

Computational environment100includes server computer120interconnected over network102. Network102can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN), such as the Internet, or a combination of the three, and can include wired, wireless, or fiber optic connections. Network102can include one or more wired and/or wireless networks that are capable of receiving and transmitting data, voice, and/or video signals, including multimedia signals that include voice, data, and video information. In general, network102can be any combination of connections and protocols that will support communications between server computer120, and other computing devices (not shown) within computational environment100. In various embodiments, network102operates locally via wired, wireless, or optical connections and can be any combination of connections and protocols (e.g., personal area network (PAN), near field communication (NFC), laser, infrared, ultrasonic, etc.).

Server computer120can be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, server computer120can represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment. In another embodiment, server computer120can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with other computing devices (not shown) within computational environment100via network102. In another embodiment, server computer120represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single pool of seamless resources when accessed within computational environment100. In the depicted embodiment, server computer120includes database122and training program150, scoring program250, and incremental training program350. In other embodiments, server computer120may contain other applications, databases, programs, etc. which have not been depicted in computational environment100. Server computer120may include internal and external hardware components, as depicted and described in further detail with respect toFIG.5.

Database122is a repository for data used by training program150, scoring program250, and incremental training program350. In the depicted embodiment, database122resides on server computer120. In another embodiment, database122may reside elsewhere within computational environment100provided training program150, scoring program250, and incremental training program350have access to database122. A database is an organized collection of data. Database122can be implemented with any type of storage device capable of storing data and configuration files that can be accessed and utilized by training program150, scoring program250, and incremental training program350, such as a database server, a hard disk drive, or a flash memory. In an embodiment, database122stores data used by training program150, scoring program250, and incremental training program350, such as one or more examples, sets of training data, data structures, and/or variables used to fit the parameters of a specified model. The contained data may comprise of input vector pairs with associated output vectors. In an embodiment, database122may contain one or more sets of one or more instances of unclassified or classified (e.g., labelled) data, hereinafter referred to as training statements. In another embodiment, the training data contains an array of training statements organized in labelled training sets. For example, a plurality of training sets include “positive” and “negative” labels paired with associated training statements (e.g., words, sentences, etc.). In an embodiment, each training set includes a label and an associated array or set of training statements which can be utilized to train one or more models. In an embodiment, database122contains unprocessed training data. In an alternative embodiment, database122contains natural language processed (NLP) (e.g., section filtering, sentence splitting, sentence tokenizer, part of speech (POS) tagging, tf-idf, etc.) feature sets. In a further embodiment, database122contains vectorized (i.e., one-hot encoding, word embedded, dimension reduced, etc.) training sets, associated training statements, and labels. In an embodiment, database122contains a primary corpus (e.g., dataset), utilized to train a primary model and an auxiliary corpus (e.g., dataset) utilized to a train a secondary (e.g., subset of a dataset (e.g., primary dataset)) model.

Primary model152and secondary model154are representative of a plurality of models capable of utilizing incremental learning algorithms such as trees, decision rules, artificial neural networks, and incremental SVM. In the depicted embodiment, primary model152and secondary model are representative of a plurality of decision trees. Primary model152is created and trained with an entire corpus or a primary dataset. Secondary model154is created and trained with an auxiliary dataset based on a very limited number of auxiliary (e.g., short-term, “special cases”, etc.) cases, allowing fast adjustment and inclusion a new case without a retraining of the primary model. The training of primary model152and secondary model154is depicted and described in further detail with respect toFIG.2.

Training program150, scoring program250, and incremental training program350are programs for efficient model adjustment utilizing a dual model system. In various embodiments, training program150, scoring program250, and incremental training program350may implement the following steps: create a subset of a dataset utilizing a trained primary model; create a secondary model based on the created subset of the dataset; calculate a confidence of a case utilizing the trained primary model, wherein the confidence is a robustness indicator of a model indicating a capacity of the model to meet or exceed performance when applied to the dataset; responsive to the calculated confidence not exceeding a confidence threshold, generate a score of the case utilizing the created secondary model; responsive to an incorrect classification, update the created subset of the dataset with the case; retrain the secondary model utilizing the updated subset of the dataset. In the depicted embodiment, training program150, scoring program250, and incremental training program350are standalone software program. In another embodiment, the functionality of training program150, scoring program250, and incremental training program350, or any combination programs thereof, may be integrated into a single software program. In some embodiments, training program150, scoring program250, and incremental training program350each may be located on separate computing devices (not depicted) but can still communicate over network102. In various embodiments, client versions of training program150, scoring program250, and incremental training program350reside on any other computing device (not depicted) within computational environment100. training program150, scoring program250, and incremental training program350are depicted and described in further detail with respect toFIG.2,FIG.3, andFIG.4.

The present invention may contain various accessible data sources, such as database122, that may include personal storage devices, data, content, or information the user wishes not to be processed. Processing refers to any, automated or unautomated, operation or set of operations such as collection, recording, organization, structuring, storage, adaptation, alteration, retrieval, consultation, use, disclosure by transmission, dissemination, or otherwise making available, combination, restriction, erasure, or destruction performed on personal data. Training program150, scoring program250, and incremental training program350provide informed consent, with notice of the collection of personal data, allowing the user to opt in or opt out of processing personal data. Consent can take several forms. Opt-in consent can impose on the user to take an affirmative action before the personal data is processed. Alternatively, opt-out consent can impose on the user to take an affirmative action to prevent the processing of personal data before the data is processed. Training program150, scoring program250, and incremental training program350enable the authorized and secure processing of user information, such as tracking information, as well as personal data, such as personally identifying information or sensitive personal information. Training program150, scoring program250, and incremental training program350provide information regarding the personal data and the nature (e.g., type, scope, purpose, duration, etc.) of the processing. Training program150, scoring program250, and incremental training program350provide the user with copies of stored personal data. Training program150, scoring program250, and incremental training program350allow the correction or completion of incorrect or incomplete personal data. Training program150, scoring program250, and incremental training program350allow the immediate deletion of personal data.

FIG.2is a flowchart depicting operational steps of a training program for training a plurality of models, in accordance with an embodiment of the present invention.

Training program150creates a primary model (step202). In an embodiment, training program150creates, initializes, and trains primary model152utilizing an entire corpus or training set (e.g., primary dataset). In the depicted embodiment, primary model152is a random forest model trained with an entire corpus, dataset, and/or training set. In an alternative embodiment, primary model152can be any model or machine learning algorithm capable of incremental training principles and practices. In an embodiment, training program150initializes primary model152with randomly generated weights. In an alternative embodiment, training program150initializes primary model152with weights calculated from a preprocessing function such as tf-idf. In yet another embodiment, training program150performs supervised training with the labeled vectorized data. For example, training program150feeds input/output pairs into primary model152, allowing training program150to make inferences between the problem data and the solution data (i.e., label). In an embodiment, training program150creates a plurality of auxiliary models (e.g., tertiary, quaternary, etc.), each trained with a dataset that is a subset of a larger training set and each associated with a distinct confidence threshold.

Primary model152produces two sets of outputs: a confidence threshold (δ) based on one or more case predictions (e.g., test sets or historical cases) and a set of derived features (e.g., attributes, etc.) based on the one or more predictions. In the depicted embodiment, training program150utilizes out-of-bag (OOB) predictions to generate the confidence threshold. Training program150utilizes confidence as a robustness indicator of a model indicating the capacity of the model to meet or exceed the same performance when it is applied to a new data set containing same characteristics as the training dataset. In an embodiment, training program150computes the confidence threshold from a calculated cumulative probability distribution. For example, in order to collect 2.3% of the training data to form the auxiliary dataset, training program150utilizes a 66.7% confidence value as the confidence threshold utilizing a calculated probability distribution.

Training program150creates auxiliary dataset (step204). In an embodiment, responsive to primary model152being fed with cases (e.g., inputs), primary model152generates a confidence value for each case. Training program150, then, collects and stores identified low confidence cases (e.g., cases that do not exceed a confidence threshold (e.g., less than 50%, etc.)) into an auxiliary dataset, independent from the original training corpus. In an embodiment, “special cases” below a confidence threshold (δ) are referred to as minor pattern cases. Said minor pattern cases are collected and expanded with outputted derived features to form an auxiliary dataset which is considerably smaller than the original training corpus.

Training program150creates secondary model with auxiliary dataset (step206). Secondary model154is trained, solely, utilizing the created auxiliary dataset. In various embodiments, training program150creates and trains secondary model154utilizing the procedures, algorithms, and methods discussed in step202. Training program150trains secondary model154with the created auxiliary dataset containing low confidence cases (e.g., training and test sets). Training secondary model154is substantially more resource and cost efficient when compared to the training of primary model152utilizing the entire corpus. In the depicted embodiment, secondary model154is a random forest model. In an alternative embodiment, secondary model154can be any model or machine learning algorithm capable of incremental training principles and methods.

FIG.3is a flowchart depicting operational steps of a scoring program for scoring a plurality of models, in accordance with an embodiment of the present invention.

Scoring program250calculates a confidence value of a case utilizing the primary model (step302). Scoring program250feeds a new case (e.g., input) into primary model152to obtain a confidence value for the new case. In the depicted embodiment, scoring program250inputs a new case into a random forest model (i.e., primary model152), and generates an associated confidence value corresponding to the new case. In an embodiment, the new case is a new training example utilized to update a plurality of models based on an associated confidence value and threshold. In various embodiments, scoring program250stores the generated confidence value in database122.

If the confidence exceeds a confidence threshold (“yes” branch, decision block304), then training program150reports a score (step308). Training program150reports the score generated from the primary model as described in step308.

If the confidence does not exceed a confidence threshold (“no” branch, decision block304), then scoring program250inputs the case into the secondary model (step306). Scoring program250utilizes the confidence threshold described and generated in steps202and204, outputted by primary model152. In an embodiment, scoring program250determines which threshold (e.g., accuracy, reliability, storage, etc.) to utilize and adjusts the confidence threshold value (e.g., a 20% training set delta, etc.). In an embodiment, a user specifies the confidence threshold value. In other embodiment, the model, system, and/or production server requirements/purpose dictates the confidence threshold value. In the depicted embodiment, scoring program250inputs the case into another random forest (the secondary model152or auxiliary model).

Scoring program250reports a score (step308). In various embodiments, scoring program250reports a score generated from the inputting of a new case into primary model152and/or secondary model154. In decision block304, if the calculated confidence value is higher than a confidence threshold then scoring program250reports the score (e.g., present the output, incorporate the score into a plurality of datasets and corpuses, trigger retraining of a plurality of models, pass score to next function (e.g., step, etc.) etc.) outputted by primary model152. For example, scoring program250reports a score is generated by a primary random forest model that exceeded a confidence threshold. In decision block304, if the calculated confidence value is lower than a confidence threshold then scoring program250reports the score outputted by secondary model154. For example, scoring program250reports a score is generated by a secondary random forest model that did not exceed a confidence threshold. In various embodiments, scoring program250stores a score generated from a primary or secondary model in database122. In an embodiment, scoring program250transmits one or more scores to incremental training program350.

FIG.4is a flowchart depicting operational steps of an incremental training for efficient model adjustment utilizing a multiple model system, in accordance with an embodiment of the present invention.

Incremental training program350retrieves a confidence score (step402). In an embodiment, incremental training program350retrieves one or more scores stored within database122. In another embodiment, incremental training program150receives one or more scores from scoring program250.

If the confidence score is based on the primary model (“yes” branch, decision block404), then incremental training program350determines if the score is correct (decision block406). If the score is incorrect (“no” branch, decision block406), then incremental training program350updates the primary dataset (step410). In an embodiment, incremental training program350compares a generated or calculated score and compares the score with an expected output. In this embodiment, the score is a confidence value, accuracy value, or a classification. In an embodiment, if the score is a confidence value and the value does not match (e.g., exceeding a threshold or not within a predetermined range of values) an expect output confidence then the score is determined as incorrect. In another embodiment, if the score is a classification and the classification is incorrect (e.g., a classification should be a dog but program150classifies a cat, etc.), then the score is determined as incorrect. Incremental training program350updates primary dataset (step410). In an embodiment, where primary model152reported a score that is not correct (e.g., wrongly classified, etc.), then the corresponding case is labelled, considered, and/or a long-term training case. In this embodiment, long-term training cases are stored and/or logged into one or more training corpuses (e.g., database122, primary corpus, primary dataset, etc.) and/or sets. In various embodiments, the retraining of primary model152is initiated by a plurality of triggers including, but not limited to, fixed training intervals, validation/testing thresholds, accuracy thresholds, and/or threshold controlling a number of stored (e.g., accumulation) long-term training cases. For example, incremental training program350initiates retraining of primary model152when long-term training cases comprises (e.g., accumulates) over 25% of training cases in a primary corpus.

If the confidence score is based on the secondary model (“no” branch, decision block404), then incremental training program350determines if the score is correct (decision block408). If the score is incorrect (“no” branch, decision block408), then incremental training program350updates the auxiliary dataset (step412). In an embodiment, incremental training program350compares a generated or calculated score and compares the score with an expected output. In this embodiment, the score is a confidence value, accuracy value, or a classification. In an embodiment, if the score is a confidence value and the value does not match an expect output confidence then the score is determined as incorrect. In another embodiment, if the score is a classification and the classification is incorrect, then the score is determined as incorrect.

Incremental training program350updates auxiliary dataset (step412). In an embodiment, where secondary model152reported a score that is not correct (e.g., wrongly classified, etc.), then the corresponding case is labelled, considered, and/or a short-term training case. In a further embodiment, training program150adds said short-term training case into an auxiliary dataset and expands said dataset. Short-term training (e.g., secondary model154, auxiliary model, etc.) is performed more frequently when compared to the long-term training (e.g., primary model152). In an embodiment, short-term training updates only the auxiliary model (e.g., secondary model154, the secondary random forest in the continuing example, etc.). Since secondary model154(e.g., auxiliary model) is built on very limited number of auxiliary cases, as well as the short-term training cases, updating said model requires a much lower computational cost when compared to updating a whole system (primary model152and secondary model154).

Incremental training program350retrains secondary model (step414). Incremental training program350logs relevant data into database122and retrains a plurality of models utilizing an adjusted corpus containing training and testing sets. In an embodiment, the retraining of secondary model154is triggered by an identification of a new short-term training case. Responsive to a retraining, training program150may deploy said retrained model to a plurality of environments such as production, testing, auxiliary environments and servers.

Server computer120each include communications fabric504, which provides communications between cache503, memory502, persistent storage505, communications unit507, and input/output (I/O) interface(s)506. Communications fabric504can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications, and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric504can be implemented with one or more buses or a crossbar switch.

Memory502and persistent storage505are computer readable storage media. In this embodiment, memory502includes random access memory (RAM). In general, memory502can include any suitable volatile or non-volatile computer readable storage media. Cache503is a fast memory that enhances the performance of computer processor(s)501by holding recently accessed data, and data near accessed data, from memory502.

Training program150, scoring program250, and incremental training program350each may be stored in persistent storage505and in memory502for execution by one or more of the respective computer processor(s)501via cache503. In an embodiment, persistent storage505includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage505can include a solid-state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

Communications unit507, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit507includes one or more network interface cards. Communications unit507may provide communications through the use of either or both physical and wireless communications links. Training program150, scoring program250, and incremental training program350each may be downloaded to persistent storage505through communications unit507.

I/O interface(s)506allows for input and output of data with other devices that may be connected to server computer120. For example, I/O interface(s)506may provide a connection to external device(s)508, such as a keyboard, a keypad, a touch screen, and/or some other suitable input device. External devices508can also include portable computer readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention, e.g., training program150, scoring program250, and incremental training program350, can be stored on such portable computer readable storage media and can be loaded onto persistent storage505via I/O interface(s)506. I/O interface(s)506also connect to a display509.

Display509provides a mechanism to display data to a user and may be, for example, a computer monitor.