Patent Document ID: 8914376
Application ID: 13933560

Base Claim:
1. An electronic document analysis method receiving N electronic documents pertaining to a case encompassing a set of issues including at least one issue and establishing relevance of at least the N electronic documents to at least one individual issue in the set of issues, the method performed with a processor, the method comprising, for at least one individual issue from among said set of issues: i. receiving an output of a categorization process applied to documents in at least control subsets of said at least N electronic documents, said output including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication; ii. seeking an input as to whether or not to initiate a new iteration I; if a new iteration is initiated, perform steps iii-x; and if a new iteration is not initiated, go to step xi; iii. selecting m electronic documents from among a subset of the N electronic documents that are not in the control set and that were not used in previous rounds for training the classifier; iv. receiving an output of a categorization process applied to the m electronic documents; v. adding the m electronic documents to an existing training subset and building a text classifier simulating said categorization process using said output for all documents in said training subset of documents; vi. evaluating said text classifier's quality using said output for documents in said control subset; vii. selecting a cut-off point for binarizing said rankings of said documents in said control subset; viii. using said cut-off point, computing and storing at least one quality criterion characterizing said binarizing of said rankings of said documents in said control subset, thereby to define a quality of performance indication of a current iteration I; ix. displaying a comparison of the quality of performance indication of the current iteration I to quality of performance indications of previous iterations; x. returning to step ii; and xi. generating a computer display of said output of said categorization process received in step i as most recently performed, including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication.

---

Claim 24:
24. A method according to claim 1 and also comprising: executing a plurality of learning iterations each characterized by precision and recall, only until a diminishing returns criterion is true, including: executing at least one learning iteration; computing the diminishing returns criterion; and subsequently executing at least one additional learning iteration only if the diminishing returns criterion is not true, wherein said diminishing returns criterion returns a true value if and only if a non-decreasing function of one of the precision and the recall is approaching a steady state, said non-decreasing function comprises an F-measure, and said diminishing returns criterion is computed by using a linear regression to compute a linear function estimating an F-measure obtained in previous iterations as a function of a log of a corresponding iteration number, generating a prediction of at least one F-measure at least one future iteration by finding a value along the linear function corresponding to a log of said future iteration, comparing said prediction to a currently known F-measure, and returning true if the prediction is close to said currently known F-measure to a predetermined degree.