Natural language processing with dynamic pipelines

Natural language processing is provided. A computer processor, selects a pipeline based on an artifact that includes unstructured data, the pipeline identifying a first algorithm of a first set of algorithms of a first human language technology (HLT) component and a second algorithm of a second set of algorithms of a second HLT component; applies the first algorithm based on the artifact to generate a first cluster space associated with the artifact; amends an evidence chain associated with the artifact in response to applying the first algorithm, wherein the evidence chain includes one or more probabilistic findings of truth corresponding to the artifact; standardizes a first ontology of the first cluster space; applies the second algorithm based on the artifact to generate a second cluster space that is associated with the artifact; and identifies a set of information of one or more corpora that is relevant to the artifact.

FIELD OF THE INVENTION

The present disclosure relates generally to the field of natural language processing, and more particularly to natural language processing with dynamic pipelines.

BACKGROUND OF THE INVENTION

Natural language processing is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, natural language processing is related to the area of human—computer interaction. Many challenges in natural language processing involve natural language understanding—that is, enabling computers to derive meaning from human or natural language input.

Unstructured Information Management Architecture (UIMA) is an open, industrial-strength, scalable and extensible platform that can be used to create analytic applications or search for programs that process text or other unstructured information to find the latent meaning, relationships, and relevant facts buried within. UIMA is a software architecture which specifies component interfaces, design patterns and development roles for creating, describing, discovering, composing, and deploying analysis capabilities for text, audio, video, or other unstructured information.

SUMMARY

A method, system, and computer program product for natural language processing is provided. A computer processor, selects a pipeline based on an artifact that includes unstructured data, the pipeline identifying a first algorithm of a first set of algorithms of a first human language technology (HLT) component and a second algorithm of a second set of algorithms of a second HLT component; applies the first algorithm based on the artifact to generate a first cluster space associated with the artifact; amends an evidence chain associated with the artifact in response to applying the first algorithm, wherein the evidence chain includes one or more probabilistic findings of truth corresponding to the artifact; standardizes a first ontology of the first cluster space; applies the second algorithm based on the artifact to generate a second cluster space that is associated with the artifact; and identifies a set of information of one or more corpora that is relevant to the artifact.

DETAILED DESCRIPTION

Embodiments of the present disclosure recognize that a natural language processing (NLP) system can automatically parse, tag, and extract knowledge from unstructured text. Further recognized is that an NLP system may be limited to identifying facts from unambiguous text. Further recognized is that an NLP system may be limited to determining relationships expressed explicitly.

Embodiments of the present disclosure provide an NLP system that identifies facts (e.g., business names, locations, dates) from ambiguous or vague text (e.g., natural language search queries). Further provided is an NLP system that determines relationships expressed indirectly, (e.g., relationships between entities within unstructured data). Further provided is an NLP system with dynamic analytic pipelines linking various human language technology (HLT) components. Further provided is that each HLT component can include a variety of algorithms. Further provided is an NLP system that mediates ontologies between HLT components and among resident and external corpora.

The present disclosure will now be described in detail with reference to the Figures.

FIG. 1is a functional block diagram illustrating a data processing environment, generally designated100, in accordance with one embodiment of the present disclosure.

Data processing environment100includes server computer102and client device130, both interconnected over network120.

Network120can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and can include wired, wireless, or fiber optic connections. In general, network120can be any combination of connections and protocols that will support communications between server computer102and client device130.

In various embodiments of the present disclosure, client device130can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with server computer102via network120. Client device130includes application user interface (UI)132, which executes locally on client device130and has the capability to provide a user interface and receive user interactions. The user interactions can include a query, which client device130can send to server computer102.

Server computer102may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with client device130via network120. In other embodiments, server computer102represents a computing system utilizing clustered computers and components to act as a single pool of seamless resources. In general, computing system102is representative of any programmable electronic device or combination of programmable electronic devices capable of accessing and/or executing UI132, natural language processing (NLP) program104, primary analysis engine106, deep analysis engine108, UIMA components110, data store112, or any combination thereof, and capable of communicating with other computing devices (e.g., client device130) via a network (e.g., network120). Server computer102may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 4. Server computer102includes NLP program104, Unstructured Information Management Architecture (UIMA) components110, and data store112. NLP program104includes primary analysis engine106and deep analysis engine108.

NLP program104operates to perform natural language processing. NLP program104includes primary analysis engine106and deep analysis engine108. In one embodiment, primary analysis engine106and deep analysis engine108are each components or sub-routines of NLP program104. In other embodiments, either or both of primary analysis engine106and deep analysis engine108may be programs independent from NLP program104, provided that each of primary analysis engine106and deep analysis engine108can access one another, UIMA components110, data store112, and client device130. In other embodiments, each of primary analysis engine106and deep analysis engine108are collective references to a group of sub-components, in which case any functionality attributed to either directly is instead performed by one or more of the sub-components of the respective group. In one embodiment, NLP program104resides on server computer102. In other embodiments, NLP program104may reside on another server computer or another computing device, provided that NLP program104is accessible to and can access UIMA components110, data store112, and client device130.

Primary analysis engine106operates to perform primary evidence retrieval. Primary evidence retrieval provides support for the formation of a conclusion or determination of a probabilistic finding. In one embodiment, primary evidence retrieval gathers evidence for the finding based on an artifact. Primary analysis engine106includes one or more human language technology (HLT) components, each of which includes one or more algorithms. A dynamic pipeline is a pipeline, which is a data structure that links a first HLT component to a second HLT component. A dynamic pipeline can identify one or more algorithms of each of the first and second HLT components. Each dynamic pipeline is associated with one or more cluster spaces, which are each a topological representation of unstructured data. Primary analysis engine106determines one or more algorithms for a dynamic pipeline. Each cluster space includes one or more clusters of data, each of which is a group of unstructured data that shares similarities with one another. Primary analysis engine106selects a dynamic pipeline that is populated with the determined one or more algorithms, or, alternatively, populates the dynamic pipeline with the determined one or more algorithms. Primary analysis engine106applies the algorithms of the dynamic pipeline to an artifact in order to generate a cluster space associated with the dynamic pipeline. Primary analysis engine106includes a mediating ontology, by which primary analysis engine106standardizes communications among algorithms to enable communication across disparate input and output formats. For example, a first algorithm may output a cluster space that represents data according to a first ontology, which may be an ontology that is incompatible for input to a second algorithm. Primary analysis engine106can standardize the output of the first algorithm to a mediating ontology to enable compatibility as input to the second algorithm, thereby enabling the application of both the first algorithm and the second algorithm to the same cluster space. Primary analysis engine106amends an evidence chain for truth maintenance. Primary analysis engine106is depicted and described in more detail, particularly with respect toFIG. 2andFIG. 3.

Deep analysis engine108operates to perform deep evidence retrieval. Deep evidence retrieval provides support for the formation of a conclusion or determination of a probabilistic finding. In one embodiment, deep evidence retrieval gathers evidence for the finding based on one or more cluster spaces of an artifact and one or more corpora. Deep analysis engine108accesses one or more corpora, each of which (i.e., each corpus) is a body of data. Deep analysis engine108can mediate the ontologies of the one or more corpora. Deep analysis engine108determines a list (or, e.g., a set) of information from the corpora that is relevant to an artifact. Deep analysis engine108generates a summary report based, at least in part, on the list of information. Deep analysis engine108is depicted and discussed in more detail, particularly with respect toFIG. 2andFIG. 4.

In some embodiments, each corpus is a body of data residing in a database. In one embodiment, a corpus resides in a database that resides in computer system102. In various embodiments, one or more corpora reside on server computer102(e.g., within data store112), on another server computer, on another computing device, or on any combination thereof, provided that NLP program104can access the corpora. In one example, the one or more corpora may include a data set from the linguistic data consortium. In another example, deep analysis engine108may generate one or more corpora based on at least one Common Analysis Structure (CAS), which is a data structure that can hold an artifact. A CAS can have multiple views, each with a representation of the artifact.

UIMA components110provide a UIMA framework and shared components that support the operations of NLP program104. In one embodiment, UIMA components110reside on server computer102. In other embodiments, UIMA components110may reside on another server computer or another computing device, provided that UIMA components110is accessible to and can access NLP program104and client device130.

Data store112is a repository that may be written and read by NLP program104and UIMA components110. In various embodiments, data may be stored to data store112including, for example, one or more corpora, summary reports, queries, artifacts, CASes, or dynamic pipelines. In some embodiments, data store112may be written and read by outside programs and entities to, e.g., populate the database with corpora. In one embodiment, data store112resides on server computer102. In other embodiments, data store112may reside on another server, another computing device, or client device130, provided that data store112is accessible to NLP program104and MIA components110.

FIG. 2is a functional block diagram of NLP program104. NLP program104includes CAS226, which includes artifact220. NLP program104further includes primary analysis engine106, which includes filtering component202, CAS multiplier204, dynamic pipelines206aand206b(collectively referred to as dynamic pipelines206), anomaly analysis component208, and relational analysis component210. NLP program104further includes deep analysis engine108, which includes deep evidence retrieval212and report generator216. In other embodiments, CAS226is independent of but accessible to NLP program104. For example, CAS226may reside in data store112or in memory (e.g., RAM) of server computer102, where CAS226is accessible to NLP program104.

CAS226is a data structure generated by NLP program104. CAS226includes artifact220, which is the subject of analysis by NLP program104. CAS226can include one or more views, each of which includes a representation of artifact220. In various embodiments, artifact220includes text, audio, video, or any combination thereof. For example, artifact220may include a document containing unstructured text. Alternatively, artifact220may include an audio-video stream with subtitles, in which case artifact220includes text, audio, and video content. Artifact220is associated with an evidence chain. An evidence chain may be a data structure that includes one or more values representing probabilistic findings of truth. Such values may be determined by primary analysis engine106(e.g., based on a cluster space resulting from an algorithm of an HLT component), filtering component202, anomaly analysis component208, relational analysis component210), deep analysis engine108, deep evidence retrieval212, or any combination thereof. A probabilistic finding of truth is a conclusion of a probability that an asserted finding is true or untrue. For example, an algorithm may assert an initial finding based on an algorithm, such as the initial finding that an artifact has a relevance that exceeds a pre-determined threshold with respect to a particular item of information. The algorithm may make a finding of probabilistic conclusion of truth by testing the asserted initial finding, which is included in the evidence chain for the artifact. In another embodiment, the evidence chain includes provenance information for each of one or more findings. Provenance information may identify the algorithm that generated the finding, the basis on which algorithm made the finding, a source of evidence, or any combination thereof. In one embodiment, artifact220can be associated with one or more dynamic pipelines (e.g., dynamic pipelines206). A dynamic pipeline is a pipeline that can dynamically link one or more algorithms of each of a first and second HLT component. Each of dynamic pipelines206may be associated with CAS226, artifact220, or both.

Filtering component202operates to identify a knowledge domain. A knowledge domain is a field of interrelated information. Filtering component202may identify a knowledge domain based on artifact220. For example, filtering component202may determine that unstructured data of artifact220contains the word “bears.” Filtering component202may identify a knowledge domain for artifact202by determining that the unstructured data relates to animals, rather than, for example, an athletic team. NLP program104may reduce or eliminate from consideration of information outside of the knowledge domain identified by filtering component202by other analyses by NLP program104(e.g., by other HLT components), thereby reducing the risk of a falsely positive conclusion. Filtering component202may include one or more algorithms (i.e., a set of algorithms) that function to identify a knowledge domain based on unstructured data of artifact202. One or more of such algorithms may be linked by a dynamic pipeline. Filtering component202applies one or more algorithms to CAS226in order to identify a knowledge domain. In other embodiments, primary analysis engine106applies filtering component202by applying one or more algorithms of filtering component202. For example, primary analysis engine106applies filtering component202to CAS226by applying one or more algorithms of filtering component202to CAS226in order to identify a knowledge domain based on artifact220. In some embodiments, filtering component202can include any algorithm that identifies a knowledge domain. In one such embodiment, filtering component202may add or remove algorithms based on user specifications. For example, filtering component202may include a set of algorithms that includes a first algorithm and a second algorithm, and may add a third algorithm to the set of algorithms in response to user specifications. In another embodiment, filtering component202may utilize machine learning to modify or refine one or more such algorithms. In one embodiment, filtering component202is an HLT component of primary analysis engine106.

CAS multiplier204operates to generate views of CAS226. A view is a unique representation of an artifact of a CAS resulting from the application of an operation, algorithm, or analysis engine to the artifact. For example, artifact220may be an audio file, in which case CAS multiplier204may generate a view of artifact220that includes a transcript by applying an analysis engine that determines the transcript of the audio file. In one embodiment, CAS multiplier204generates alternate views of CAS226in order to enable analysis of alternate representations of an artifact. In one example, CAS multiplier204generates alternate views of CAS226by applying one or more analysis engines of UIMA components110. In one embodiment, CAS multiplier204is a component of primary analysis engine106. In another embodiment, CAS multiplier204is a component of UIMA components110that is available to and executable by NLP program106.

Anomaly analysis component208operates to identify novel unstructured data. In one embodiment, anomaly analysis component280includes one or more algorithms (i.e., a set of algorithms) that identify novel portions of unstructured data of the unstructured data of artifact220. A portion of unstructured data may be novel if the portion does not conform to an expected pattern. In various examples, anomaly analysis component208may identify novel unstructured data based on unsupervised anomaly detection, supervised anomaly detection, or semi-supervised anomaly detection, based on statistical analysis to identify statistical outliers, based on cluster analysis to determine clusters formed by the portions of unstructured data, or based on any combination thereof. In one embodiment, anomaly analysis component208includes one or more algorithms that function to identify novel portions of unstructured data. For example, anomaly analysis component208applies one or more algorithms to CAS226in order to identify novel portions of unstructured data based on artifact220. In other embodiments, primary analysis engine106applies anomaly analysis component208by applying one or more algorithms of anomaly analysis component208. For example, primary analysis engine106applies an algorithm of anomaly analysis component208to artifact220to generate a cluster space associated with artifact220in order to identify novel portions of unstructured data. In some embodiments, anomaly analysis component208can include any algorithm that identifies novel portions of unstructured data. In one such embodiment, anomaly analysis component208may add or remove algorithms based on user specifications. For example, anomaly analysis component208may include a set of algorithms including a first algorithm and a second algorithm, and may add a third algorithm to the set of algorithm in response to user specifications. In another embodiment, anomaly analysis component208may utilize machine learning to modify or refine one or more such algorithms. In one embodiment, anomaly analysis component208is an HLT component of primary analysis engine106.

Relational analysis component210operates to identify relationships of unstructured data. In one embodiment, relational analysis component210includes one or more algorithms (i.e., a set of algorithms) that identify relationships between and among portions of unstructured data of artifact220. For example, for an artifact that includes the text “City Grill is a restaurant” in unstructured data, relational analysis component210may determine an associative relationship between “City Grill” and “restaurant.” In another example, relational analysis component210may determine an employment relationship for an artifact that includes the text “John Doe is the head chef at City Grill” between “John Doe” and “City Grill.” Relational analysis component210may apply one or more algorithms to CAS226in order to identify relationships of the unstructured data of artifact220. In other embodiments, primary analysis engine106applies relational analysis component210by applying one or algorithms of relational analysis component210. For example, primary analysis engine106applies an algorithm of relational analysis component210to CAS226that generates a cluster space associated with artifact220in order to identify relationships of unstructured data of artifact220. In some embodiments, relational analysis component210can include any algorithm that identifies relationships of unstructured data. In one such embodiment, relational analysis component210may add or remove algorithms based on user specifications. For example, relational analysis component210may include a set of algorithms including a first algorithm and a second algorithm, and may add a third algorithm to the set of algorithm in response to user specifications. In another embodiment, relational analysis component210may utilize machine learning to modify or refine one or more such algorithms. In one embodiment, relational analysis component210is an HLT component of primary analysis engine106.

Each of dynamic pipelines206represents one or more dynamic analytic pipelines. In one embodiment, each of dynamic pipelines206links a first HLT component and a second HLT component by identifying one or more algorithms of the first HLT component and one or more algorithms of the second HLT component. For example, a first dynamic pipeline identifies at least one algorithm of the one or more algorithms of a first HLT component, and also at least one algorithm of the one or more algorithms of a second HLT component. Each of dynamic pipelines206may identify algorithms based on a determination by primary analysis engine106(see step310).

In some embodiments, each of dynamic pipelines206identifies an ordering of algorithms of HLT components. In one such embodiment, the order is an order in which the identified algorithms are to be applied. For example, a first dynamic pipeline identifies an order in which primary analysis engine106is to apply the one or more identified algorithms.

Each algorithm generates output (e.g., a cluster space) based on input (e.g., unstructured data of artifact220). In one embodiment, the input and output ontologies of each algorithm varies with respect to other algorithms. For example, the output generated by a first algorithm may follow an ontology that is different from the ontology followed by an input of a second algorithm. In one embodiment, dynamic pipelines206include ontology mediation functionality to standardize the input ontology and output ontology of each algorithm. Dynamic pipelines206may utilize ontology mediation to map the ontology of one algorithm to another algorithm, thereby enabling compatibility between the algorithms that follow disparate ontologies. For example, the word “chef” may be characterized as an “Employment Position” by an ontology of a first algorithm and as a “Job Title” by an ontology of a second algorithm, in which case dynamic pipelines206may utilize ontology mediation to map “Employment Position” to “Job Title” in order to enable compatibility between the first and second algorithms. In one embodiment, dynamic pipelines206maps ontologies based on pre-determined equivalencies between fields of various ontologies. In another embodiment, dynamic pipelines206maps ontologies by performing unsupervised clustering based on the ontologies in order to identify fields to merge and fields that have no equivalency between the ontologies. Alternatively, dynamic pipelines206may utilize ontology mediation to map the ontology of each algorithm to a common representation, such as a resource description framework (RDF) representation.

Deep evidence retrieval212operates to identify relevant information from one or more corpora. In one embodiment, deep evidence retrieval212identifies relevant information from one or more corpora based on, at least in part, artifact220, CAS226(including the one or more cluster spaces associated therewith), and the one or more corpora. For example, deep evidence retrieval212analyzes CAS226and one or more corpora in order to identify a list (or, e.g., a set) of information of the one or more corpora related to CAS226. In one embodiment, deep evidence retrieval212is a component of deep analysis engine108.

Report generator216operates to generate a summary report. In one embodiment, report generator216generates a summary report based on, at least in part, a list of relevant information. For example, report generator216generates a summary report based on a list of relevant information, which is generated by deep evidence retrieval212, and an evidence chain, which is associated with artifact220. In one embodiment, report generator216is a component of deep analysis engine108.

FIG. 2is discussed in more detail in connection with the discussions accompanyingFIG. 3andFIG. 4.

FIG. 3is a flowchart depicting the operational steps of primary analysis engine106of NLP program104.

In step302, primary analysis engine106receives artifact220. In one embodiment, primary analysis engine106receives a query from client device130that identifies artifact220.

In step304, primary analysis engine106encapsulates artifact220in CAS226. Primary analysis engine106may encapsulate artifact220in CAS226by generating CAS226based on artifact220. In one embodiment, CAS226and each HLT component utilize a consistent ontology. In some embodiments, the input and output ontologies of CAS226is consistent with an input and output ontologies of UIMA. In one such embodiment, UIMA components110include an implementation of UIMA, thereby enabling primary analysis engine106to utilize any of various analysis engines or other capabilities of UIMA components110in combination with CAS226.

In step306, primary analysis engine106applies filtering component202to CAS226. In one embodiment, primary analysis engine106applies one or more algorithms of filtering component202to CAS226in order to identify a knowledge domain. In one embodiment, primary analysis engine106associates the identified knowledge domain with artifact220. For example, one or more algorithms of filtering component202identify a knowledge domain by comparing unstructured data of artifact220to data from one or more knowledge domains to determine a knowledge domain to which the unstructured data belongs. In various embodiments, primary analysis engine106applies one or more algorithms of filtering component202that are identified by, for example, dynamic pipelines206, CAS226, primary analysis engine106, or user specifications.

In some embodiments, primary analysis engine106applies CAS multiplier204in order to generate additional views of CAS226. In one such embodiment, CAS multiplier204generates additional views of CAS206by generating one or more alternative representations of artifact220. In various embodiments, CAS multiplier204generates additional views by applying one or more annotators or analysis engines of UIMA components110. Primary analysis engine106may store additional views resulting from CAS multiplier204to CAS226. Alternatively, primary analysis engine106may store additional views resulting from CAS multiplier204to data store112, and may associate the additional views with CAS226.

In step308, primary analysis engine106amends an evidence chain based on each filtering algorithm applied. In one embodiment, primary analysis engine106amends the evidence chain associated with artifact220by adding a probabilistic finding of truth generated by each applied filtering algorithm to the evidence chain. In another embodiment, primary analysis engine106amends the evidence chain to reflect providence information corresponding to each such algorithm applied.

In step310, primary analysis engine106identifies algorithms for dynamic pipelines206. Identifying algorithms for dynamic pipelines206may include determining an ordering of a plurality of algorithms. In one embodiment, primary analysis engine106identifies algorithms for each of dynamic pipelines206based on a cluster space associated with artifact220. The cluster space may be an initial cluster space generated by primary analysis engine106based on artifact220(or, e.g., based on unstructured data of artifact220). Alternatively, the cluster space may be a cluster space generated by an algorithm of an HLT component based on artifact220(or, e.g., based on unstructured data of artifact220). Primary analysis engine106may identify algorithms based on the cluster space by applying semi-supervised machine learning. The machine learning may, for example, evaluate the cluster space associated with artifact220based on a statistical F-score, cluster quality, search efficiency, search quality, a rand index, or a combination thereof, and may identify algorithms based on the evaluation of the cluster space.

In some embodiments, primary analysis engine106identifies algorithms for dynamic pipelines206based on user specifications. Primary analysis engine106may receive the user specifications from, e.g., client device130. For example, the user specifications may include a list specifying one or more algorithms and an ordering of the algorithms for one or more of dynamic pipelines206, in which case primary analysis engine106identifies the algorithms and ordering specified by the user specifications.

In some embodiments, primary analysis engine106identifies algorithms for one or more dynamic pipelines for each view of CAS226. Each dynamic pipeline for each view of CAS226may link the same algorithms, or may link different algorithms relative to one another. Primary analysis engine106may identify algorithms for more than one dynamic pipeline, including identifying every possible combination and permutation of the one or more algorithms of a first HLT component and a second HLT component. Alternatively, primary analysis engine106may identify less than all of such possible combinations and permutations. For example, primary analysis engine106may determine one or more algorithms based on semi-supervised machine learning, as previously discussed.

In step312, primary analysis engine106populates dynamic pipelines. In one embodiment, primary analysis engine106populates dynamic pipelines of dynamic pipelines206based on the identified algorithms (see step310). For example, primary analysis engine106populates dynamic pipelines of dynamic pipelines206in order to link the identified algorithms. The number of dynamic pipelines that primary analysis engine106populates may depend upon the number of dynamic pipelines for which primary analysis engine106identified algorithms (see step310).

In step318, primary analysis engine106applies each algorithm identified by each of dynamic pipelines206. In one embodiment, primary analysis engine106applies each identified algorithm to the one or more cluster spaces associated with artifact220. For example, primary analysis engine106modifies a cluster space based on the results of each algorithm applied. Alternatively, each algorithm generates a cluster space that is associated with artifact220. In one embodiment, each of dynamic pipelines206identifies an order in which to apply one or more algorithms. In such embodiment, primary analysis engine106applies each of the one or more algorithms in sequence, according to the identified order. Alternatively, primary analysis engine106may apply each algorithm in parallel. In various embodiments, each of dynamic pipelines206identifies one or more algorithms of anomaly analysis component208, one or more algorithms of relational analysis component210, or any combination thereof.

In some embodiments, primary analysis engine106applies algorithms of anomaly analysis component208and algorithms of relational analysis component210in parallel. For example, both anomaly analysis component208and relational analysis component210may share simultaneous access to the one or more cluster spaces associated with artifact220, thereby allowing primary analysis engine106to execute an algorithm of each component in parallel.

In step320, primary analysis engine106mediates the ontology of each applied algorithm. Primary analysis engine106may mediate the ontology of each applied algorithm by modifying each cluster space associated with CAS226resulting from an applied algorithm (see step318). In one embodiment, primary analysis engine106mediates the ontology of each such algorithm of anomaly analysis component208and relational analysis component210applied to CAS226. For example, dynamic pipelines206of primary analysis engine106mediate the ontology of each such applied algorithm of anomaly analysis component208and relational analysis component210. In one embodiment, the ontology of each such applied algorithm is mediated by standardizing the ontology to RDF representation. In another embodiment, the ontology of each such applied algorithm is mediated by standardizing the ontology to the ontology of the algorithm that is next in order, as identified by a dynamic pipeline of dynamic pipelines206.

In step322, primary analysis engine106amends the evidence chain associated with artifact220based on each dynamic pipeline. Primary analysis engine106may amend the evidence chain based on each cluster space resulting from the application of each algorithm identified by each dynamic pipeline associated with artifact220. In one embodiment, primary analysis engine106amends the evidence chain to reflect probabilistic findings of truth of each such algorithm applied. In another embodiment, primary analysis engine106amends the evidence chain to reflect providence information corresponding to each such algorithm applied.

In step324, primary analysis engine106sends CAS226to deep analysis engine108.

FIG. 4is a flowchart depicting the operational steps of deep analysis engine108of NLP program104.

In step402, deep analysis engine108receives a Common Analysis Structure (CAS). In one embodiment, deep evidence retrieval212of deep analysis engine108receives CAS226from primary analysis engine106. In another embodiment, deep evidence retrieval212receives CAS226by retrieving CAS226. For example, deep evidence retrieval212receives a reference to CAS226and accesses storage (or, e.g., memory). Responsive to the reference, deep evidence retrieval212retrieves CAS226. In one embodiment, CAS226includes artifact220, which is associated with an evidence chain. In another embodiment, CAS226includes artifact220, which is associated with one or more cluster spaces generated by primary analysis engine106(e.g., resulting from the application of an algorithm of an HLT component of primary analysis engine106). For example, the one or more cluster spaces associated with artifact220are modified by the results of one or more algorithms of anomaly analysis component208and the results of one or more algorithms of relational analysis component210.

In step404, deep analysis engine108receives one or more corpora. In one embodiment, deep evidence retrieval212of deep analysis engine108receives one or more corpora of data store112. For example, deep evidence retrieval212receives one or more corpora by accessing (or, e.g., retrieving) one or more corpora of data store112.

In step406, deep analysis engine108mediates the ontology of the corpora. In one embodiment, deep analysis engine108mediates the ontology of each of the one or more received corpora. For example, deep evidence retrieval212of deep analysis engine108mediates the ontology of each of the one or more corpora. In one embodiment, the ontology of each received corpora is mediated by standardizing the ontology to RDF representation.

In step408, deep analysis engine108analyzes one or more cluster spaces and one or more corpora. In one embodiment, deep evidence retrieval212of deep analysis engine108analyzes the one or more cluster spaces associated with artifact220and the received corpora. In some embodiments, deep evidence retrieval212analyzes the one or more cluster spaces and corpora in order to identify information of the corpora with a degree of relevance to the one or more cluster spaces of artifact220that exceeds a pre-determined threshold. In various embodiments, deep evidence retrieval212applies one or more deep evidence algorithms, search engines, other linguistic tools, or any combination thereof in order to analyze the one or more cluster spaces and corpora. In various examples, deep evidence retrieval212may analyze the one or more cluster spaces and corpora based on a variety of algorithms, including natural language search engines, word-sense disambiguation systems, lexical answer type coercion systems, or any combination thereof.

In some embodiments, deep evidence retrieval212can include one or more algorithms that each function to analyze the one or more cluster spaces and corpora to identify relevant information. In one such embodiment, deep evidence retrieval212may add or remove algorithms in response to user specifications. In another embodiment, deep evidence retrieval212may utilize machine learning to modify or refine one or more such algorithms.

In step410, deep analysis engine108determines a list of related information. The list of related information includes information of one or more corpora identified by deep analysis engine108as related to CAS226(see step408). For example, the information may be related to CAS226based on the one or more cluster spaces associated with artifact220. An inference may be a conclusion or determination of a probabilistic finding based on a plurality of other conclusions or determination of probabilistic findings.

In some embodiments, deep evidence retrieval212determines a confidence score for each information item identified by the list of relevant information. The confidence score may represent a degree of certainty that the information item has a relevance that exceeds a pre-determined threshold with respect to artifact220. Deep analysis engine108may determine the confidence score using probability density functions. Alternatively, deep analysis engine108determines the confidence score utilizing statistical and analytical techniques including logistics regression, decision trees, clustering, neural networks, or any combination thereof. In another embodiment, deep evidence retrieval212also compares the information item to the one or more cluster spaces associated with artifact220, information of one or more corpora, or any combination thereof. In another such embodiment, deep evidence retrieval212utilizes one or more machine experts in order to determine a confidence score for each information item. For example, deep evidence retrieval212may generate a confidence score utilizing a machine expert that comprises an inference engine and a knowledge base.

In some embodiments, deep evidence retrieval212determines a list of related information based on a machine learning sub-system that includes machine-generated models. In one such embodiment, deep analysis engine108refines the machine-generated models using a joint inference model that utilizes client specifications from client device130and unclassified training materials.

In step412, deep analysis engine108amends an evidence chain. In one embodiment, deep evidence retrieval212of deep analysis engine108amends the evidence chain associated with artifact220by adding to the evidence chain a probabilistic finding of truth generated by each filtering algorithm applied. For example, deep evidence retrieval212may amend the evidence chain based on the results of the one or more deep evidence algorithms applied by deep analysis engine108. In various embodiments, the evidence chain includes probabilistic findings of truth, provenance information, or a combination thereof, which, in one such example, correspond to information items of one or more corpora.

In step414, deep analysis engine108determines inferences. In one embodiment, the determined inferences identify one or more information items from the received corpora that are related to artifact220. For example, the determined inferences may be determinations of the existence of a relationship (e.g., a semantic relationship) between artifact220and an information item identified by the list of related information. Deep analysis engine108may determine the inferences based on a confidence score for each information item. Alternatively, deep evidence retrieval212of deep analysis engine108determines inferences based on artifact220, the evidence chain associated with artifact220, the one or more cluster spaces associated with artifact220, the one or more corpora, or any combination thereof. In another embodiment, deep evidence retrieval212modifies the list of related information based on the one or more determined inferences. For example, deep evidence retrieval212may modify the list or related information to identify some or all of the determined inferences.

In step416, deep analysis engine108generates a summary report. In one embodiment, report generator216of deep analysis engine108generates a summary report based on the determined inferences. In one embodiment, the summary report includes a list of information of the one or more corpora that is related to artifact220. For example, the list of information may include one or more information items ranked in order of the confidence score for each information item. In another embodiment, report generator216generates a summary report based on the one or more cluster spaces associated with artifact220. For example, the summary report may identify one or more clusters associated with artifact220. In some embodiments, the summary report includes a confidence score for each information item. In some embodiments, the summary report includes truth maintenance information for each information item. In one such embodiment, deep analysis engine108generates the truth maintenance information based on the evidence chain associated with artifact220. In various embodiments, the truth maintenance information corresponding to an information item includes probabilistic findings of truth, provenance information, or a combination thereof.

In step418, deep analysis engine108sends the summary report. In one embodiment, report generator216of deep analysis engine108sends the summary report to, for example, client device130. In another embodiment, report generator216sends a notification corresponding to the summary report to client device130. In yet another embodiment, report generator216sends the summary report to a database. For example, report generator216may store the summary report to a database (e.g., data store112).

FIG. 5depicts a block diagram of respective components of server computer102and client device130in accordance with an illustrative embodiment of the present disclosure. It should be appreciated thatFIG. 5provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

The media used by persistent storage508may also be removable. For example, a removable hard drive may be used for persistent storage508. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage508.

Communications unit510, in these examples, provides for communications with other data processing systems or devices, including resources of client device130and server computer102, respectively. In these examples, communications unit510includes one or more network interface cards. Communications unit510may provide communications through the use of either or both physical and wireless communications links. NLP program104, UIMA components110, and data store112may be downloaded to persistent storage508through communications unit510.

I/O interface(s)512allows for input and output of data with other devices that may be connected to server computer102and client device130, respectively. For example, I/O interface(s)512may provide a connection to external devices518such as a keyboard, a keypad, a touch screen, and/or some other suitable input device. External devices518can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present disclosure, e.g., NLP program104, UIMA components110, and data store112, can be stored on such portable computer-readable storage media and can be loaded onto persistent storage508via I/O interface(s)512. I/O interface(s)512also connect to a display520.

The programs described herein are identified based upon the application for which they are implemented in a specific embodiment of the disclosure. However, it should be appreciated that any particular program nomenclature herein is used merely for convenience, and thus the disclosure should not be limited to use solely in any specific application identified and/or implied by such nomenclature.