Patent ID: 7363296

Claim:
A method for rating information retrieval ranking algorithms, said method comprising: receiving a query; submitting the received query to a search engine for execution on an index file, said index file storing a plurality of documents; receiving a list of index documents and a plurality of attributes from the search engine in response to the submitted query, said plurality of attributes including data describing the received list of index documents; storing a portion of the received list of index documents and the received plurality of attributes associated therewith in a subindex file; executing the received query on the subindex file to obtain a list of subindex documents; ranking, by a ranking algorithm, the obtained list of subindex documents using the attributes associated therewith wherein the ranking algorithm has a plurality of parameters associated therewith; comparing the list of ranked subindex documents with a list of subjectively ranked documents to generate a score for the ranking algorithm, said generated score representing a degree of correlation between the list of ranked subindex documents and the list of subjectively ranked documents; varying at least one of the plurality of parameters to create another ranking algorithm; ranking, by the created another ranking algorithm, the obtained list of subindex documents to obtain another list of ranked subindex documents; comparing the another list of ranked subindex documents ranked by the another ranking algorithm with the list of subjectively ranked documents corresponding thereto to generate another score for the another ranking algorithm, said generated another score representing a degree of correlation between the another list of subindex documents ranked by the another ranking algorithm and the list of subjectively ranked documents; storing in an output file the generated score, information describing the ranking algorithm, the another generated score, information describing the another ranking algorithm, and information relating a difference between the ranking algorithm and the another ranking algorithm with a difference between the generated score and the another generated score; comparing the generated score and the another generated score to identify an optimal score representing the higher degree of correlation; and applying an optimizing algorithm to the output file to identify one of the ranking algorithm and the another ranking algorithm as an optimal ranking algorithm as a function of the identified optimal score, wherein the optimal ranking algorithm having the optimal score, said optimal ranking algorithm being defined by the plurality of parameters and a plurality of weights associated with the plurality of parameters, wherein the optimal score is a maximum possible score for any ranking algorithm defined by said plurality of parameters and said plurality of weights associated with said plurality of parameters ranking said obtained list of subindex documents.