Patent ID: 8572011

Claim:
A method performed by data processing apparatus, the method comprising: training an outcome estimation model using both regression error measures and ranking error measures, the regression error measures specifying error measures associated with training outcomes for respective data records, the ranking error measures specifying error measures associated with training outcomes for respective record pairs, each data record being a data record from a data set of data records, and each record pair being a pair of data records from the data set; receiving a request for one or more content items; in response to the request: for each content item in a set of content items, computing a predicted outcome using the outcome estimation model and feature values associated with the content item; computing selection scores for the content items using the predicted outcome; and selecting one or more content items based on the selection scores; and providing data that cause presentation of the one or more content items at a client device, wherein training the outcome estimation model comprises: initializing feature weights of the outcome estimation model, each feature weight representing a relative importance of a feature value for predicting an outcome; selecting, as training data with which the feature weights are to be updated, one of a data record and a record pair, each data record being associated with a set of feature values and a reference outcome, the data record being selected with a first probability and the record pair being selected with a second probability; updating the feature weights of the outcome estimation model using the selected training data; determining whether a stop condition has been met; in response to determining that the stop condition has not been met, repeating the selecting, the updating, and the determining; and in response to determining that the stop condition has been met, determining that the outcome estimation model is trained.