Patent Document ID: 9721002
Application ID: 14093227

Base Claim:
1. One or more tangible, non-transitory, computer-readable media storing computer-executable instructions for causing a computing system programmed thereby to perform a method comprising: for each of multiple different named entity recognition (“NER”) services, receiving extraction results generated for a document corpus according to an entity type schema, the extraction results comprising an annotation set comprising a plurality of annotations, each annotation comprising an entity type or subtype; determining one or more hypotheses, each hypothesis comprising an annotation set comprising a plurality of annotations, each annotation comprising a type or subtype; for each of one or more hypotheses, determining whether the annotation set associated with the hypothesis is type-consistent with the annotation set associated with a NER service of the multiple NER services by determining whether the hypothesis annotation set and the NER annotation set have the same number of annotations within an annotation area and all of those annotations form pairs of type-consistent annotations; ranking each of the one or more hypotheses based at least in part on a difference in type-consistency between the one or more hypotheses, the ranking for a hypothesis being based at least in part on the agreement between the hypothesis annotation set and the annotation set of each of a plurality of the multiple NER services; and returning to a requesting user or a requesting application a top n number of hypotheses based on the ranking, where n is a positive integer value less than or equal to the number of the one or more hypotheses.

---

Claim 15:
15. The one or more computer-readable media of claim 1 , wherein the annotation set of a NER service of the multiple NER services is type-consistent with the annotation set of a hypothesis if each of the annotations in the hypothesis is type-consistent with a corresponding annotation in the annotation set of the NER service.