Patent Document ID: 8838606
Application ID: 13921015
Patent Flag: 1

Claim One:
1. An active learning system for processing documents in connection with conducting e-discovery in a legal proceeding, the system comprising: a processor being configured to: generate or receive a document information profile for one or more documents in a document collection, each document information profile corresponding to a particular document and representing features of that document; select a first document from the document collection to present to a human reviewer; receive a user coding decision associated with the first document; calculate a classifier based on the received user coding decision associated with the first document and the document information profile for the first document; compute a first set of scores for a subset of documents in the collection by applying the classifier to the document information profile for each document in the subset for which the first set of scores are computed; determine a first plurality of rankings for documents in the document collection by choosing among a plurality of rankings, wherein at least two of the following rankings are considered in determining which to choose: rankings determined by the first set of scores, relevance rankings derived from user input for the documents in the document collection, and rankings determined by a combination of the first set of scores and the relevance rankings; select a further document from the document collection to present to a human reviewer based on the first plurality of rankings; receive a user coding decision associated with the further document; incrementally update the classifier based on the received user coding decision associated with the further document and the document information profile for the further document; compute a second set of scores for a subset of documents in the collection by applying the updated classifier to the document information profile for each document in the subset for which the second set of scores are computed; determine a second plurality of rankings for documents in the document collection by choosing among a plurality of rankings, wherein at least two of the following rankings are considered in determining which to choose: rankings determined by the second set of scores, relevance rankings derived from user input for the documents in the document collection, and rankings determined by a combination of the second set of scores and the relevance rankings; calculate an estimate of the number of documents in the document collection that are relevant to the legal proceeding by fitting scores computed for documents for which user coding decisions were received to a standard distribution; estimate recall based on the calculated estimate of the number of documents in the document collection that are relevant to the legal proceeding; in response to determining that the estimated recall is acceptable, classify a set of documents in the document collection as relevant or non-relevant to the legal proceeding using computed scores or user coding decisions; and in response to determining that the estimated recall is not acceptable, repeat the steps of selecting a further document, receiving a user coding decision associated with the further document, updating the classifier, computing a second set of scores, determining a second plurality of rankings, calculating an estimate of the number of documents in the document collection that are relevant to the legal proceeding, and estimating recall.