Patent Document ID: 9311599
Application ID: 13544779

Base Claim:
1. A method for evaluating predictive models in classification systems, the method comprising: causing, using a hardware processor, an input region to be presented to a user, wherein the input region receives an instance from the user; in response to receiving the instance from the user, retrieving, using the hardware processor, a classification assigned by a predictive model for the received instance and a confidence value associated with the classification, wherein the classification has been stored in association with the instance from the user in response to determining that the classification exceeds a threshold value, and wherein the threshold value is set based at least in part on a penalty associated with misclassification of the instance; determining, using the hardware processor, whether the received instance has been incorrectly classified by the predictive model; in response to determining that the received instance has been incorrectly classified by the predictive model, (i) causing, using the hardware processor, the received instance and a corrected classification provided by the user to be used as training data to update the predictive model, and (ii) determining, using the hardware processor, a reward associated with the incorrect classification made by the predictive model, wherein the reward is directly proportional to the confidence value associated with the classification of the received instance; updating, using the hardware processor, the predictive model using the received instance as training data; and causing, using the hardware processor, an indication of the reward to be provided to the user.

---

Claim 3:
3. The method of claim 1 , wherein the input region includes task information relating to the predictive model, the method further comprising: receiving, from an administrator user, a selected predictive model from a plurality of predictive models; and determining the task information associated with the selected predictive model to display in the input region.