Patent ID: 7827060

Claim:
A method implemented by one or more processors of a computer system, comprising: obtaining, using one or more processors of the computer system, ratings associated with a first group of advertisements, where the ratings, which include a quality of the first group of advertisements, are received from human raters; observing, using one or more processors of the computer system, multiple different user actions associated with user selection of advertisements of the first group of advertisements; deriving, using one or more processors of the computer system, a probability model using the observed user actions and the obtained ratings, where the probability model includes a probability function that specifies a probability that an advertisement, of a second group of advertisements, is of a certain quality as a function of multiple different types of user actions, where deriving the probability model comprises: using at least one of logistic regression, regression trees or boosted stumps to generate the probability model; using, by one or more processors of the computer system, the probability model to estimate quality scores associated with the second group of advertisements; calculating, using one or more processors of the computer system, a combination of the estimated quality scores and click through rates associated with advertisements in the second group of advertisements; filtering, using one or more processors of the computer system, the second group of advertisements based on a comparison of the combination of the estimated quality scores and click through rates to a threshold to generate a subset of advertisements from the second group of advertisements; and providing, to a user, the subset of advertisements from the second group of advertisements.