Patent Document ID: 9524281
Application ID: 13477730

Base Claim:
1. A method, in a data processing system, for evaluating the performance of a text analysis engine, the method comprising: receiving a plurality of pre-annotated reference documents; receiving a set of annotation types associated with the pre-annotated reference documents; analyzing annotation contexts of reference annotations in the plurality of pre-annotated reference documents using the set of annotation types; identifying similar annotation contexts between the reference annotations and the set of annotation types; responsive to identifying the similar annotation contexts, clustering the similar annotation contexts thereby forming a plurality of reference annotation clusters; computing a set of reference content heterogeneity scores based on the number of reference annotation clusters for each annotation type in the set of annotation types; computing an integral reference content rate for the set of annotation types; and outputting the integral reference content rate to a user.

---

Claim 5:
5. The method of claim 1 , wherein the integral reference content rate for the set of annotation types is computed using the following equation: ContentRate = ∑ n = 1 N ⁢ ⁢ _ ⁢ ⁢ types ⁢ 1 N_types ⁢ CH ⁡ ( T n ) , wherein N_types is the number of annotations types and wherein T n (n=1, N_types) are the plurality of annotations types.