Patent Document ID: 8069179
Application ID: 12108531

Base Claim:
1. A system comprising: a processor; one or more memories communicatively coupled to the processor, the one or more memories having stored instructions that, when executed at the processor, causes the processor to implement a component that at least one of trains or evaluates ranking techniques by at least one of employing or obtaining relative preference judgments, the component configured to retrieve a set of documents from a storage device, combine the set of documents with a judgment task received via an interface to form a comparative selection panel and present the comparative selection panel for evaluation by an assessor, the component requests the assessor to compare documents in the set of documents with each other and make a selection as to which document included in the set of documents most satisfies the judgment task, the component produces a ranked comparative assessment of the set of documents based at least in part on the selection elicited from the assessor, the component automatically determining whether to present one or more of a plurality of interfaces to the assessor based on the judgment task, the plurality of interfaces including an absolute judgment panel, a binary selection panel, and a graded preference panel, and wherein the absolute judgment panel presents a single document for the assessor to evaluate, the absolute judgment panel requests the assessor to select at least one ranking criteria, the ranking criteria selected by the assessor being associated with the single document, the binary selection panel presents at least two documents from the set of documents for review by the assessor, the assessor selects one of the at least two documents as being binarily preferable over another of the at least two documents, and the graded preference panel presents at least two documents from the set of documents for review by the assessor, the assessor identifies one of the at least two documents as being preferable, the one of the at least two documents identified as preferable being retained as one of the at least two documents for a next judgment.

---

Claim 2:
2. The system of claim 1 , wherein the component dynamically chooses another set of documents based at least in part on a previous selection made by the assessor as to which document included in the set of documents most satisfied the judgment task.