Patent ID: 9430533
Filing Date: 2016-08-30
CPC Classification: G06F

Claim Text:
1. A method that is executed by a computer processor on a computing device, the method comprising: receiving a first search result list returned by a first ranker system responsive to a search query; receiving a second search result list returned by a second ranker system responsive to the search query; employing a first classifier to predict whether a search preference judgment of the first search result list and the second search result list indicates a quality difference between the first search result list and the second search result list; when the first search result list and the second search result list for the search query are predicted to have the quality difference, determining a confidence level for whether the search preference judgment indicates one of the first search result list or the second search result list is preferred; inserting the search query, the first search result list, and the second search result list in a first set of search queries and search result list pairs to be manually judged for search preference judgments when: transmitting the first set of the search queries and the search result list pairs to one or more annotator computing devices for manual judgment.