Patent ID: 8065184

Claim:
A method implemented by one or more processors of a computer system, comprising: obtaining, using one or more processors of the computer system, ratings associated with a set of advertisements, where the ratings, which include a measure of quality of the set of advertisements, are received by manual input from human raters; logging, using one or more processors of the computer system, first user actions associated with user selection of advertisements of the set of advertisements, the first user actions representing a measure of user satisfaction with regard to the selected advertisements; logging, using one or more processors of the computer system, second user actions associated with user selection of an unrated advertisement of a set of unrated advertisements; calculating, by one or more processors of the computer system, a probability that the unrated advertisement is of a certain measure of quality based on the logged second user actions and on a probability generated by a probability model that operates based on the logged first user actions and the obtained ratings.