Patent Document ID: 9223868
Application ID: 13308770

Base Claim:
1. A method performed by a system of one or more computers, the method comprising: comparing, by the system, a first scoring algorithm for scoring electronic documents responsive to search queries with a second scoring algorithm for scoring the electronic documents responsive to search queries, wherein the comparing includes: obtaining a first profile that characterizes historical interaction, by a first plurality of users, with electronic documents in first result sets of electronic documents, the electronic documents in each first result set scored by a search engine in accordance with the first scoring algorithm and ranked by the search engine in each first result set in accordance with said scoring, obtaining a second profile that characterizes historical interaction, by a second plurality of users, with electronic documents in second result sets of electronic documents, the electronic documents in each second result set scored by the search engine in accordance with the second scoring algorithm and ranked by the search engine in each second result set in accordance with said scoring, determining that one or more metrics of the user interaction characterized in the first profile collectively indicate a higher level of user satisfaction with the first result sets of electronic documents than a level of user satisfaction with the second result sets indicated by the one or more metrics of the user interaction characterized in the second profile, and in response to the determination, outputting an indication that the first scoring algorithm is better than the second scoring algorithm, wherein the one or more metrics of the user interaction characterized in the first profile comprise a total number of respective selections of one or more electronic documents from each of the first result sets of electronic documents, wherein the one or more metrics of the user interaction characterized in the second profile comprise a total number of respective selections of one or more electronic documents from each of the second result sets of electronic documents, and wherein the higher level of user satisfaction is indicated by a lower total number of respective selections of the one or more electronic documents from each of the first result sets of electronic documents than the total number of respective selections of the one or more electronic documents from each of the second result sets of electronic documents.

---

Claim 3:
3. The method of claim 1 , wherein: the one or more metrics of the user interaction characterized in the first profile further comprise a first ratio of long duration selections of electronic documents from the first result sets of electronic documents to short duration selections of electronic documents from the first result sets of electronic documents; the one or more metrics of the user interaction characterized in the second profile further comprise a second ratio of long duration selections of electronic documents from the second result sets of electronic documents to short duration selections of electronic documents from the second result sets of electronic documents; and the higher level of user satisfaction is further indicated by the first ratio being greater than the second ratio.