Patent ID: 11861531
Assignee: INTEGRAL AD SCIENCE, INC.
Field: IT methods for management (Electrical engineering)
Classification: CPC G | IPC G

Claim 10:
11. A system for data acquisition for construction of classification models, the system comprising:
a processor that:
receives a budget and a cost structure for constructing a classification model using a data set that includes positive and negative examples of a class of interest, wherein the classification model is set to a guided learning mode that receives selected instances of the class of interest that satisfy at least one criterion from the data set and wherein the cost structure includes a cost of a search performed by a human reviewer;
determines a rate of change in the estimated performance of the classification model as a function of the cost of at least one of the plurality of human reviewers performing an additional search in the guided learning mode, wherein the rate of change in the estimated performance of the classification model is determined based on a slope of the learning curve;
in response to determining that the rate of change in the estimated performance as a function of the cost of the at least one of the plurality of human reviewers performing the additional search in the guided learning mode is lower than a first predetermined threshold, switches the classification model from the guided learning mode to an active learning mode, which includes machine learning, that receives labelled instances from portions of the data set selected as being useful to the classification model until the estimated performance of the classification model is greater than a second predetermined threshold or the budget for constructing the classifier has been exhausted, wherein the plurality of human reviewers include human reviewers that are available to label one or more instances of the class of interest during training of the classification model using the active learning mode, and wherein the training of the classification model using the active learning mode is discontinued in response to determining that the rate of change in the estimated performance as a function of a cost of labeling performed by the plurality of human reviewers is lower than a third predetermined threshold.