Patent Document ID: 9082084
Application ID: 13931109

Base Claim:
1. A computer-implemented method for facilitating an automatic machine-learning process for an online social network, the method comprising: during operation of the online social network, automatically collecting labeled training events; snapshotting raw entity data associated with subjects from the collected training events; generating features for each training event using the snapshotted entity data and the current entity data to produce training examples; consolidating the training examples from one or more time periods to produce a consolidated set of training examples; determining one or more contradictory user actions upon a set of entity data in the consolidated set of training examples, wherein a contradictory user action involves a same user responding differently to the same entity data over different time intervals; resolving the contradictory user actions for the consolidated set of training examples; splitting the training examples into a training set and a test set; using a machine-learning technique to train a set of models and select the best model based on one or more evaluation metrics using the training set; evaluating the performance of the best model on the test set; and if the performance of the best model satisfies a performance criterion, using the best model to predict responses for the online social network.

---

Claim 5:
5. The computer-implemented method of claim 1 , wherein evaluating the best model involves using quantitative and qualitative measures; and wherein using a qualitative measure includes using a crowd-sourcing evaluation technique that comprises: sending a selected set of training examples and associated responses that were produced by the best model to a crowd-sourcing module that solicits the opinions of an additional set of users about the accuracy of the responses; and comparing the predictions of the best model with predictions of the additional set of users on the selected training examples.