Patent ID: 8688716

Claim:
A computer-implemented method comprising: accessing a plurality of videos, each of the plurality of videos associated with a quality distribution, the quality distribution indicating probabilities that the video belongs to particular quality levels based at least in part on historical user input designating a preferred video from a pairwise comparison of two videos; determining a ranking of the plurality of videos, based in part on the quality distribution of each of the videos; calculating a plurality of costs for video pairwise comparisons, based at least in part on an expected cost of incorrect rankings and a probability of the incorrect rankings; receiving a request for a pair of videos to compare from a requestor; selecting a video comparison based at least in part on the costs for video pairwise comparisons for video comparisons; providing the selected video comparison to the requestor; receiving a comparison result relative to a pair of videos; and updating the quality distribution associated with each video of the pair of videos to reflect the comparison result.