KNOWLEDGE INDUCTION USING CORPUS EXPANSION

A method, a computer program product, and a computer system induce knowledge from a knowledge graph. The method includes receiving a request indicative of a domain. The method includes determining a corpus corresponding to the domain and determining a quality of the corpus in generating the knowledge graph relative to a quality threshold. If the quality threshold is not met, the method includes determining a candidate expansion corpus to incorporate further data therefrom into the corpus relative to an expansion threshold. If the expansion threshold is met, the method includes generating an expanded corpus by expanding the corpus with the further data. The method includes generating the knowledge graph based on the expanded corpus from which the knowledge is induced and generating a response to the request based on the knowledge graph.

BACKGROUND

The exemplary embodiments relate generally to knowledge graphs, and more particularly to inducing knowledge from available corpora when an original corpus of a knowledge graph requires expansion.

A system may develop a knowledge graph by acquiring and integrating various types of information such as for a specific domain into an ontology such that a reasoner may derive new knowledge or information. The information may be embodied as a corpus of documents or other data in which to develop the knowledge graph. The knowledge graph may model a knowledge domain from different sources in a manual manner or an automated manner such as subject-matter experts, data interlinking, machine learning algorithms, etc. For example, the knowledge graph may be used in providing information from a search query. A user may provide an input into a search field and the search engine may utilize the knowledge graph to ascertain information regarding the search query. In a particular example, when the search query is an individual, the search engine may use the knowledge graph to ascertain personal information, professional information, educational information, etc. and determine which aspects to include in a result of the search query (e.g., based on a threshold of relevance to the search query).

Domain-specific knowledge graphs may either be not publicly available or not reusable in commercial applications for a variety of reasons. For example, as the knowledge graph may be created based on proprietary information, the knowledge graph itself may remain proprietary to the developer of that knowledge graph. Although knowledge graphs may utilize various mechanisms for development, creating the domain-specific knowledge graphs utilizing manual approaches may take a huge effort. For example, manually creating the knowledge graphs may require domain experts who must provide a substantial effort. With regard to a conventional knowledge graph such as the Universal Medical Language System (UMLS), the development of the UMLS took over a decade with contributions from a large number of doctors.

When domain-specific knowledge graphs utilize automated approaches, the knowledge graph may be developed through automatic knowledge induction. However, conventional automatic knowledge induction approaches rely on sufficient and repeated evidences. The evidences may derive from corpora of data. However, some of these corpora lack the evidence on which to induce the information for the knowledge graph. Those skilled in the art will understand that knowledge induction from a corpus may be difficult if the corpus is too small and/or if the corpus is not of a sufficient quality (e.g., the corpus is a chat log). For example, a system may be configured with a knowledge graph based on public troubleshooting documents. However, such a system may utilize a relatively small corpus size (e.g., approximately 4,000 documents occupying 6 MB). Such a corpus size may be relatively small compared to other systems that utilize a much larger corpus size (e.g., another system may use approximately 55,000 documents occupying 75 MB while a further system may use approximately 212,000 documents occupying 768 MB). The corpus being used by the system may also not use fully natural language such as being formatted in lists, commands, logs, etc. Some relevant terms in the corpus for the system may also have very low frequencies where products may appear only once in a single issue. Such a system may not be configured to properly induce knowledge from the knowledge graph.

Conventional approaches for knowledge induction used for knowledge graphs and use of various corpora may utilize a plurality of different mechanisms. However, the conventional approaches often focus on only one aspect of knowledge induction for knowledge graphs and incorporation of corpora without considering the overall process and relative success or relevance of inducing certain knowledge in a meaningful way. For example, U.S. Pat. No. 10,229,188 describes a method to expand a question answer corpus using a quality analysis system where a candidate answer having a rank above pre-specified requirements is added to the corpus. In another example, U.S. Pat. No. 8,892,550 describes improving performance of information retrieval by adding new information like paraphrases that are found in the sources to increase the semantic redundancy through searching for similar content related to existing data, automatically retrieving the content, extracting units of text, and determining their relevance and/or relatedness. However, in these conventional approaches, there is no determination of whether the corpus requires the candidate answer, no identification of a candidate corpus to which the candidate answer is to be added, and no identification of documents in the candidate corpus.

In a further example, U.S. Pat. No. 7,805,288 describes a method for corpus expansion by expanding new sample seeds based on existing sample seeds for classification based applications. This approach uses existing sample seeds as keywords to search Web documents for collecting candidate new sample seeds for a specific class. However, this approach does not expand corpora for knowledge induction as well as any mechanism to detect a need for expansion of the corpus, to identify appropriate candidate corpora, and to further identify appropriate candidate documents in the candidate corpora.

Although conventional approaches have been developed in association with knowledge graphs and corpora, these conventional approaches utilize a relatively straightforward mechanism that does not consider whether a corpus requires expansion, determining which corpus requires expansion, determining documents or sources that are to be used in expansion of a candidate corpus, etc. from which a knowledge graph may induce knowledge based on a domain-specific corpus.

SUMMARY

The exemplary embodiments disclose a method, a computer program product, and a computer system for inducing knowledge from a knowledge graph. The method comprises receiving a request, the request being indicative of a domain. The method comprises determining a corpus corresponding to the domain, the corpus including data related to the domain. The method comprises determining a quality of the corpus in generating the knowledge graph in which to induce knowledge relative to a quality threshold. As a result of the quality of the corpus not satisfying the quality threshold, the method comprises determining a candidate expansion corpus to incorporate further data therefrom into the corpus relative to an expansion threshold. As a result of the candidate expansion corpus satisfying the expansion threshold, the method comprises generating an expanded corpus by expanding the corpus with the further data. The method comprises generating the knowledge graph based on the expanded corpus from which the knowledge is induced. The method comprises generating a response to the request based on the knowledge graph.

In a preferred embodiment, the method further comprises determining candidate terms from the corpus, the candidate terms being selected based on the domain, an analysis of the request, or a combination thereof.

In a preferred embodiment, the method further comprises determining a corpus quality score for each of the candidate terms, the corpus quality score being indicative of a relation of the candidate terms across the corpus, wherein the quality threshold is a configurable percentage of the corpus quality scores satisfying a minimum threshold.

In a preferred embodiment, determining the candidate expansion corpus further comprises taking a sample of data from the candidate expansion corpus, calculating the corpus quality score for each of the candidate terms in the candidate expansion corpus, and determining whether the candidate expansion corpus satisfying the expansion threshold, the expansion threshold being indicative of a similarity metric between the corpus and the candidate expansion corpus.

In a preferred embodiment, generating the expanded corpus comprises determining a first set of data associated with a seed category in the candidate expansion corpus, determining a second set of data associated with a seed document in the candidate expansion corpus, and determining a third set of data associated with an interaction between the first and second sets.

In a preferred embodiment, generating the expanded corpus comprises extracting domain specific terminology from data of the corpus, scoring each of the domain specific terminology based on relation objects, ranking the domain specific terminology, selecting ones of the domain specific terminology based on the ranking, and determining the further data in the candidate expansion corpus based on the select ones of the domain specific terminology.

In a preferred embodiment, generating the expanded corpus is an automatic domain specific corpus creation, an entity lookup-based automatic domain specific corpus expansion using knowledge base relations, or a combination thereof.

The drawings are not necessarily to scale. The drawings are merely schematic representations, not intended to portray specific parameters of the exemplary embodiments. The drawings are intended to depict only typical exemplary embodiments. In the drawings, like numbering represents like elements.

DETAILED DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Detailed embodiments of the claimed structures and methods are disclosed herein; however, it can be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be embodied in various forms. The exemplary embodiments are only illustrative and may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to be covered by the exemplary embodiments to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

In the interest of not obscuring the presentation of the exemplary embodiments, in the following detailed description, some processing steps or operations that are known in the art may have been combined together for presentation and for illustration purposes and in some instances may have not been described in detail. In other instances, some processing steps or operations that are known in the art may not be described at all. It should be understood that the following description is focused on the distinctive features or elements according to the various exemplary embodiments.

The exemplary embodiments are directed to a method, computer program product, and system for corpus expansion techniques that focus on documents with consideration of domain entities as well as additionally focusing on a strength of the relations among the entities. That is, the exemplary embodiments are configured to analyze the documents that contain text that may be used to facilitate extracting relations among the entities. Through an analysis of available corpora of documents, corpora may be selected to incorporate into a corpus from which a knowledge graph may be created. As will be described in greater detail herein, the exemplary embodiments may automatically induce knowledge from a corpus of domain specific documents by extracting candidate terms from an original corpus to determine if the original corpus requires expansion based on a corpus quality score of the candidate terms. From the available corpora, the exemplary embodiments may select and determine the available corpora to use for the knowledge inductions and then expanding the domain specific original corpus using the selected available corpora through expansion mechanisms. Key benefits of the exemplary embodiments may include a more comprehensive approach to knowledge induction to generate a knowledge graph having more meaningful connections between information by expanding a corpus that requires corpus expansion. Detailed implementation of the exemplary embodiments follows.

In utilizing a knowledge graph, the information from which the knowledge graph is based may define how well an application that utilizes the knowledge graph to generate responses, for example, to information requests. Knowledge induction in creating the knowledge graph may generate the artifacts needed by downstream applications with minimum manual effort (e.g., virtual assistants for information technology support). As the scale in which knowledge graphs increase, automatic knowledge induction processes become more essential for creating the knowledge graphs at scale. With data in the public domain changing at increasing rates (e.g., an information repository may experience 10 million new entries over a six month period), automatic knowledge induction may enable knowledge graphs to remain in sync with ever changing knowledge. Accordingly, the corpus from which a knowledge graph is created may be of paramount importance. The exemplary embodiments are therefore configured to measure a quality of an original corpus from which a knowledge graph may be created and expanding the original corpus using available corpora of documents to increase the quality of the original corpus when a minimum quality threshold is not satisfied. In this manner, the exemplary embodiments may avoid the original corpus not being of a sufficient quality for automatic knowledge induction operations.

The exemplary embodiments are described with regard to knowledge graphs and corpora associated with knowledge graphs. However, the exemplary embodiments may be utilized for any basis on which information may be interconnected and used for a variety of purposes such that a quality of the information meets a minimum standard from which to draw conclusions. Accordingly, the mechanisms provided by the exemplary embodiments may be utilized and/or modified for use in inducing information.

FIG. 1depicts a knowledge induction system100, in accordance with the exemplary embodiments. According to the exemplary embodiments, the knowledge induction system100may include one or more smart devices110, one or more corpus repositories120, a knowledge server130, and one or more data repositories140, which may all be interconnected via a network108. While programming and data of the exemplary embodiments may be stored and accessed remotely across several servers via the network108, programming and data of the exemplary embodiments may alternatively or additionally be stored locally on as few as one physical computing device or amongst other computing devices than those depicted.

In the exemplary embodiments, the network108may be a communication channel capable of transferring data between connected devices. Accordingly, the components of the knowledge induction system100may represent network components or network devices interconnected via the network108. In the exemplary embodiments, the network108may be the Internet, representing a worldwide collection of networks and gateways to support communications between devices connected to the Internet. Moreover, the network108may utilize various types of connections such as wired, wireless, fiber optic, etc. which may be implemented as an intranet network, a local area network (LAN), a wide area network (WAN), or a combination thereof. In further embodiments, the network108may be a Bluetooth network, a WiFi network, or a combination thereof. In yet further embodiments, the network108may be a telecommunications network used to facilitate telephone calls between two or more parties comprising a landline network, a wireless network, a closed network, a satellite network, or a combination thereof. In general, the network108may represent any combination of connections and protocols that will support communications between connected devices. For example, the network108may also represent direct or indirect wired or wireless connections between the components of the knowledge induction system100that do not utilize the network108.

In the exemplary embodiments, the smart device110may include a service client112and may be an enterprise server, a laptop computer, a notebook, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a server, a personal digital assistant (PDA), a rotary phone, a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, an Internet of Things (IoT) device, or any other electronic device or computing system capable of receiving and sending data to and from other computing devices. While the smart device110is shown as a single device, in other embodiments, the smart device110may be comprised of a cluster or plurality of computing devices, in a modular manner, etc., working together or working independently. The smart device110is described in greater detail as a hardware implementation with reference toFIG. 3, as part of a cloud implementation with reference toFIG. 4, and/or as utilizing functional abstraction layers for processing with reference toFIG. 5.

In the exemplary embodiments, the service client112may act as a client in a client-server relationship and may be a software, hardware, and/or firmware based application capable of allowing a user to request data and receive a response to the request via the network108. In embodiments, the service client112may operate as a user interface allowing the user to submit a request for data and present the requested data to the user as well as interact with one or more components of the knowledge induction system100, and utilize various wired and/or wireless connection protocols for data transmission and exchange associated with data that is determined based on fusion operations, including Bluetooth, 2.4 gHz and 5 gHz internet, near-field communication, Z-Wave, Zigbee, etc.

A user may require selected data for a variety of reasons and may utilize a service for which the selected data is requested. For example, the service client112may be a browser or application in which a search may be requested. The user may enter a search request in an available field and transmit the search request to a network component configured to return a response to the request. In generating the request, the other components of the knowledge induction system100(e.g., the knowledge server130) may be utilized.

In the exemplary embodiments, the corpus repository120may include one or more corpora122and may be an enterprise server, a laptop computer, a notebook, a tablet computer, a netbook computer, a PC, a desktop computer, a server, a PDA, a rotary phone, a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, an IoT device, or any other electronic device or computing system capable of storing, receiving, and sending data to and from other computing devices. While the corpus repository120is shown as a single device, in other embodiments, the corpus repository120may be comprised of a cluster or plurality of electronic devices, in a modular manner, etc., working together or working independently. While the corpus repository120is also shown as a separate component, in other embodiments, the corpus repository120may be incorporated with one or more of the other components of the knowledge induction system100. For example, the corpus repository120may be incorporated in the knowledge server130. Thus, access to the corpus repository120by the knowledge server130may be performed locally. The corpus repository120is described in greater detail as a hardware implementation with reference toFIG. 3, as part of a cloud implementation with reference toFIG. 4, and/or as utilizing functional abstraction layers for processing with reference toFIG. 5.

Each of the corpora122may be domain specific and utilized to create a knowledge graph from inducing knowledge therefrom that may be used in responding to the request. The corpora122may be a collection of data or information that are associated with a domain, a topic, etc. For example, the corpora122may be a set of documents that pertain to a given domain. Thus, for a given request, the appropriate corpus from the corpora122may be determined as an original corpus from which subsequent operations may be performed (e.g., to measure a quality of the original corpus to generate a knowledge graph that is used in responding to the request).

In the exemplary embodiments, the data repository140may include one or more document sources142and may be an enterprise server, a laptop computer, a notebook, a tablet computer, a netbook computer, a PC, a desktop computer, a server, a PDA, a rotary phone, a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, an IoT device, or any other electronic device or computing system capable of storing, receiving, and sending data to and from other computing devices. While the data repository140is shown as a single device, in other embodiments, the data repository140may be comprised of a cluster or plurality of electronic devices, in a modular manner, etc., working together or working independently. While the data repository140is also shown as a separate component, in other embodiments, the data repository140may be incorporated with one or more of the other components of the knowledge induction system100. For example, the data repository140may be incorporated in the knowledge server130. Thus, access to the data repository140by the knowledge server130may be performed locally. The data repository140is described in greater detail as a hardware implementation with reference toFIG. 3, as part of a cloud implementation with reference toFIG. 4, and/or as utilizing functional abstraction layers for processing with reference toFIG. 5.

Each data repository140may represent a source from which documents for a variety of domains or topics may be available. For example, a first data repository140may be an online encyclopedia that includes data for a variety of different topics. In another example, a second data repository140may be an online medical resource that includes data for medically related topics. Each data repository140may hold a plurality of document sources142that represent individual documents from which to draw information related to the domain in which the data repository140is directed.

In the exemplary embodiments, the knowledge server130may include a candidate program132, a scoring program134, an expansion program136, and an inducing program138, and act as a server in a client-server relationship with the service client112as well as be in a communicative relationship with the corpus repository120and the data repository140. The knowledge server130may be an enterprise server, a laptop computer, a notebook, a tablet computer, a netbook computer, a PC, a desktop computer, a server, a PDA, a rotary phone, a touchtone phone, a smart phone, a mobile phone, a virtual device, a thin client, an IoT device, or any other electronic device or computing system capable of receiving and sending data to and from other computing devices. While the knowledge server130is shown as a single device, in other embodiments, the knowledge server130may be comprised of a cluster or plurality of computing devices, working together or working independently. While the knowledge server130is also shown as a separate component, in other embodiments, the operations and features of the knowledge server130may be incorporated with one or more of the other components of the knowledge induction system100. For example, the operations and features of the knowledge server130may be incorporated in the smart device110, particularly the smart device110of the user who is requesting the data. The knowledge server130is described in greater detail as a hardware implementation with reference toFIG. 3, as part of a cloud implementation with reference toFIG. 4, and/or as utilizing functional abstraction layers for processing with reference toFIG. 5.

The knowledge server130is configured to, in response to a request for data, generate a response using a knowledge graph that is created based on one of the corpora122associated with a topic of the request. In creating the knowledge graph, the knowledge server130may determine a quality of the corpus122and determine whether the corpus122has a sufficient quality (e.g., satisfying a minimum threshold as will be described in detail below). As a result of the corpus122having a sufficient quality, the knowledge server130may generate the knowledge graph and generate the response using the knowledge graph. However, as a result of the corpus122having an insufficient quality, the knowledge server130may expand the corpus122into a modified form of the corpus122that includes information from the data repositories140. In expanding the corpus122, the knowledge server130may determine which of the data repositories140and/or the document sources142therein to use to expand the corpus122in a meaningful manner. Using the corpus122(e.g., in the original form or in the modified form), the knowledge server130may generate the knowledge graph and induce knowledge therefrom to generate the response for the request.

The exemplary embodiments are described with regard to receiving a request for data and generating a response for the request where the response is generated based on a knowledge graph from a corpus122having a sufficient quality (e.g., in the original form or the modified form where the original form is expanded). However, the exemplary embodiments being utilized as a request and response format where the knowledge graph is generated for the request is only exemplary. The knowledge server130may be configured to determine a quality of the corpora122for various domains at a variety of other times and generate corresponding knowledge graphs. For example, in preparation to provide a response, the knowledge server130may associate the corpora122to various domains and topics and determine a quality of the corpora122for a selected domain. Based on a result of the quality determination, the knowledge server130may preliminarily expand the corpora122when a quality does not meet a minimum threshold for a given domain. Accordingly, when a request is subsequently received for the given domain, the knowledge server130may have already determined that the corpora122meets the minimum threshold (e.g., the corpora122in its original form meets the minimum threshold or the corpora122has been expanded to meet the minimum threshold). Accordingly, the knowledge server130may reduce processing requirements and proceed with generating the knowledge graph and induce knowledge therefrom in generating the response.

In the exemplary embodiments, the candidate program132may be a software, hardware, and/or firmware application configured to determine candidates used in determining a corpus122to be used for the knowledge graph. For example, the candidate program132may be configured to determine a candidate corpus122for which subsequent operations may be performed. In determining the candidate corpus122, the candidate program132may analyze the request and identify one or more corpora122that may be utilized in creating the appropriate knowledge graph. For example, the request may be parsed to determine keywords. The corpora122may be associated with keywords such that corpora122having matching keywords to the keywords of the request are identified as candidate corpora122. Once the candidate corpora122are determined, the knowledge server130may perform subsequent operations.

The candidate program132may also be configured to extract candidate terms from a selected corpus122to be used in creating the knowledge graph for the response. The candidate terms may be selected based on the request to determine whether the candidate terms in the corpus122are representative of a quality of the corpus122. For example, based on a parsing of the request, one of the corpora122may be selected (e.g., each corpus122may be associated with certain keywords where the request including these keywords may be indicative of utilizing the corpus122). The request may also be indicative of select portions of the corpus122that is to be considered to meaningfully generate the response. The select portions may be associated with terms from which the candidate program132may determine the candidate terms.

In the exemplary embodiments, the scoring program134may be a software, hardware, and/or firmware application configured to determine a corpus quality score for each of the candidate terms. An overall analysis of the corpus quality scores of the candidate terms may be used to determine whether the corpus122requires expansion to meet the minimum threshold for quality. As will be described in further detail below, the corpus quality score may be determined based on a relation type across the selected corpus122and/or based on terms and relations across the selected corpus122. The corpus quality scores on an individual basis and/or a holistic basis may be used by the scoring program134to determine whether the corpus122in its original form meets the minimum threshold for quality or whether the corpus122requires expansion into a modified form to meet the minimum threshold.

In the exemplary embodiments, the expansion program136may be a software, hardware, and/or firmware application configured to expand the corpus122as a result of determining that the selected corpus122does not meet the minimum threshold for quality in its original form. In expanding the selected corpus122, the expansion program136may determine the data repositories140to use for the expansion. As will be described in further detail below, the expansion program136may take a sample of the document sources142and calculate a corpus quality score for the candidate terms in the data repository140. Based on the corpus quality score, the expansion program136may select the data repository140when a cross-document reference ratio meets a selection threshold.

The expansion program136may further be configured to expand the selected corpus122utilizing an expansion mechanism. As will be described in further detail below, the expansion program136may expand the selected corpus122using an automatic corpus expansion process. For example, the expansion program136may select seed documents among the document sources142of the data repository140. The expansion program136may determine a first set of documents associated with a seed category and descendant sub-categories up to a first depth and a second set of documents from a seed document and associated documents up to a second depth. The expansion program136may determine a third set as an interaction of the first and second set of documents to output documents corresponding to the third set which are used for the expansion. In another example, the expansion program136may utilize all documents of the selected corpus122in its original form. The expansion program136may process the documents of the selected corpus122and extract domain terminology in a manner substantially similar to determining the candidate terms. The expansion program136may perform a search (e.g., a text search( ) on each terminology and collect the top results up to a selected limit. The expansion program136may determine relation objects in each entity and count select ones that match with terms in the domain terminology from which a score may be assigned. After all searches, the expansion program136may calculate a cumulative score of each entity and rank the entities from which the top entities from the ranked list are used as a basis to expand the selected corpus122.

In the exemplary embodiments, the inducing program138may be a software, hardware, and/or firmware application configured to induce knowledge from the knowledge graph that is created based on the corpus122that is to be used. For example, the corpus122may be in its original form as a result of the quality meeting the minimum threshold. In another example, the corpus122may be in a modified form as a result of expanding the corpus122to meet the minimum threshold. The corpus122in its original form or the modified form may be used to create the knowledge graph from which the inducing program138may induce knowledge using any mechanism as would be understood by one skilled in the art. The knowledge server130may subsequently generate a response to the request using the available information including direct information and induced information.

FIG. 2depicts an exemplary flowchart of a method200illustrating the operations of the knowledge server130of the knowledge induction system100in in inducing knowledge from a knowledge graph, in accordance with the exemplary embodiments. The method200may relate to operations that are performed by the candidate program132, the scoring program134, the expansion program136, and the inducing program138to determine a corpus122to be used in creating the knowledge graph and inducing knowledge therefrom. The method200will be described from the perspective of the knowledge server130.

The knowledge server130may receive a request that utilizes a knowledge graph to generate the response (step202). As described above, a user utilizing the smart device110may use the service client112to enter a request for data. For example, the service client112may be a browser, a proprietary application, etc. in which the user may enter a request in a field. The request may be a string of characters including one or more words as a phrase, a sentence, a question, etc. In a specific implementation, the user may enter a request to receive information for an individual. The request may further request an event that occurred for the individual (e.g., “When did individual X discover event Y?”). The service client112may package the request and transmit the request to the knowledge server130.

The knowledge server130may determine one of the corpora122to be used for the knowledge graph in its original form (step204). For example, the candidate program132may determine one or more candidate corpora122to be used in creating the knowledge graph based on keyword matching where keywords extracted from the request are matched to keywords respectively associated with the corpora122.

The knowledge server130may extract candidate terms from the determined corpus122in its original form (step206). For example, the candidate program132may extract the candidate terms from a select one of the candidate corpora122. The candidate terms that are extracted may be representative of determining a quality of the candidate corpora122in creating the knowledge graph in generating the response. For example, the corpora122may include a plurality of documents with a wide range of different types of information. The candidate terms may be selected based on the request and extracted in the corpus122. Those skilled in the art will understand the various different techniques that may be utilized to determine the candidate terms and extracting those candidate terms in the corpus122. For example, when the request relates to when an individual had an event occur, the candidate terms may include the individual's name, an identity of the event, etc. Such candidate terms may be direct candidate terms derived directly from the request. The candidate program132may also elaborate to select and extract indirect candidate terms based on the direct candidate terms. For example, the individual's name may be associated with further individuals who were involved in the selected event. The selected event may be associated with further events that may have occurred prior to, concurrent with, and subsequent to the event indicated in the request. Other techniques such as distributional similarity and word vectors may be used to further select indirect candidate terms.

The knowledge server130may determine a corpus quality score of the candidate terms in the determined corpus122in its original form (step208). The objective of the corpus quality score is to have an indicative measure on how good the corpus is for inducing a knowledge graph from it or if it needs further expansion. For example, the scoring program134may be configured to determine the corpus quality scores. In determining the corpus quality scores, the knowledge server130may determine whether the selected corpus122requires expansion to create the knowledge graph. That is, the knowledge server130may determine a quality of the corpus122in creating the knowledge graph.

The knowledge server130may perform an initial knowledge induction on the corpus122in its original form through determining the corpus quality scores of the candidate terms. The knowledge server130may utilize a variety of variables in determining the corpus quality scores. For example, the inputs to determine the corpus quality scores may include a number N of terms that are extracted (e.g., the number of candidate terms) and all the relation instances that are extracted, R, which consists of k subsets, each subset corresponding to a given relation type.

The knowledge server130may be configured to utilize different mechanisms in calculating the corpus quality scores. For example, the knowledge server130may utilize a mechanism based on relation types across the corpus122. In this mechanism, the knowledge server130may consider R consists of k subsets where each subset corresponds to a given relation type. In this manner, the knowledge server130may calculate the corpus quality score as a ratio between size of the R and of N by N. For example, for each relation type Ri, the knowledge server130may compute si=|Ri|/(N×N). The knowledge server130may also set a configurable as the percentage of relation types out of all relation types that are lower than a minimum threshold for the corpus quality score to satisfy. For example, an administrator of the knowledge server130may set the configurable percentage. The configurable percentage may indicate a relative strength of quality of the relations in the corpus122. Accordingly, when the relation types Ri do not meet the configurable percentage, the knowledge server130may conclude that the corpus expansion on the corpus122in its original form is required. That is, as a result of the configurable percentage of relation types Rihaving a low score sithat is below the minimum threshold, the knowledge server130may indicate that the knowledge server130is to expand the corpus122in its original form using subsequent operations.

In another example, the knowledge server130may utilize a mechanism based on terms and relations across the corpus122. In this mechanism, the knowledge server130may again refer to the variable inputs N and R. According to this mechanism, for each term tin the terminology (e.g., the candidate terms), the knowledge server130may determine a number of relation instances that each term t participates. In calculating the corpus quality scores, the knowledge server130may determine a ratio between all relations that a term t is part of relative to R. For example, for each term t, the knowledge server130may compute s=|all relations that t is a part|/|R|. In a substantially similar manner as the other mechanism described above, the knowledge server130may set a configurable percentage as a minimum threshold for the corpus quality score to satisfy. When the terms t do not meet the configurable percentage, the knowledge server130may conclude that the corpus expansion on the corpus122in its original form is required.

In computing the corpus quality score, the knowledge server130may create ground truth data from reference documents having the same domain. The knowledge server130may use the ground truth data for comparison purposes to determine how the corpus quality scores may satisfy the minimum threshold.

The knowledge server130may output the corpus quality scores and the overall determination of whether the corpus122in its original form requires expansion. In the manner described above, the knowledge server130may thereby set the threshold for triggering the corpus expansion based on user defined parameters (e.g., the configurable percentage), learned information from previous satisfactory corpora, or a combination thereof.

The knowledge server130may determine whether a threshold percentage of the candidate terms have a threshold score (decision210). As a result of the threshold percentage of the candidate terms having the threshold score (decision210, “YES” branch), the knowledge server130may generate the knowledge graph using the determined corpus122in its original form (step212). The knowledge server130may induce knowledge from the knowledge graph (step214) and generate the response to the request based on the available knowledge (step216). For example, the inducing program138may induce knowledge from the knowledge graph that has been created.

As a result of the threshold percentage of the candidate terms not having the threshold score (decision210, “NO” branch), the knowledge server130may determine the available corpora to expand the determined corpus122(step218). For example, the expansion program136may determine the manner in which to perform the corpus expansion. In an aspect of determining the manner in which to perform the corpus expansion, the knowledge server130may determine select corpora from the data repositories140in which to perform the corpus expansion in a meaningful way. The knowledge server130may input the determined corpus122that has been determined to require corpus expansion. The knowledge server130may also analyze available candidate expansion corpora among the data repositories140. As described above, the data repositories140may include open domain sources, specific domain sources, etc. In a particular implementation, the documents in the expansion corpus that is to be used in the corpus expansion may be interlinked.

In selecting the corpora of the data repositories140for the corpus expansion, the knowledge server130may obtain domain specific terminology from the target corpus in the data repositories140. The knowledge server130may utilize any available tool for terminology extraction. For example, the knowledge server130may use a domain specific terminology extraction mechanism by boosting frequency metrics. In another example, the knowledge server130may use an automatic extraction of domain specific terminology from a large corpus. The terminology extracted in this manner for the corpus expansion will be referred hereinafter as “domain terminology”.

The knowledge server130may select the corpora in the data repositories140by considering each candidate expansion corpus. For each candidate expansion corpus, the knowledge server130may take a sample of the documents in the document sources142of a selected one of the data repositories140that is a candidate. The sample of the documents may be random or predetermined to properly represent the candidate data repository140. The knowledge server130may determine the corpus quality score for the same candidate terms used in the determined corpus122using a substantially similar process as the corpus quality score determination described above. Based on the configurable percentage of candidate terms satisfying the minimum score threshold, the knowledge server130may determine whether the candidate expansion corpus is an appropriate candidate to use in the corpus expansion. For example, if the corpus quality scores fall below the configurable percentage (e.g., for the first mechanism described above), the knowledge server130may skip this candidate expansion corpus. In computing the corpus quality scores for the candidate expansion corpus, the knowledge server130may ignore the domain. The knowledge server130may also determine a cross-document reference (e.g., interlinks) ratio among the documents in the sample. If the ratio is smaller than a predetermined threshold (e.g., the configurable percentage for the second mechanism described above), the knowledge server130may skip the candidate expansion corpus. The knowledge server130may then run terminology extraction on the sample. The terminology extracted at this stage will be referred hereinafter as the “expansion corpus terminology”. The knowledge server130may determine a semantic and/or topic similarity between the terminology of the determined corpus122and the expansion corpus terminology. In setting an expansion threshold for the similarity in terminologies, the knowledge server130may determine whether the candidate expansion corpus satisfies the expansion threshold for selection in the corpus expansion of the determined corpus122.

The knowledge server130may expand the determined corpus122in its original form to a modified form (step220). For example, the expansion program136may be configured to perform the corpus expansion using the candidate expansion corpus determined previously. In performing the corpus expansion, the knowledge server130may utilize an automatic corpus expansion based on a variety of mechanisms. As will be described below, according to an exemplary mechanism, the knowledge server130may perform an automatic domain specific corpus creation from the candidate expansion corpus with minimal input. According to a further exemplary mechanism, the knowledge server130may perform an entity lookup-based automatic domain specific corpus expansion using knowledge base relations.

According to the first mechanism in which the knowledge server130performs an automatic domain specific corpus creation from the candidate expansion corpus, the knowledge server130may utilize inputs including one or more representative seed documents for the domain (e.g., a specific document in the document source142of the data repository140) and one or more seed categories of the domain (e.g., a higher level or general category of the domain). Categories may be any high level organization of documents into subsets of smaller document collections. In performing the corpus expansion, the knowledge server130may normalize the documents of the expansion corpus if this process is required. For example, seed documents from a first data repository140may be normalized to a second data repository140. Those skilled in the art will understand that there are a variety of different manners of normalizing the documents such as utilizing a linking tool. The knowledge server130may determine a first set of article links that are associated with a seed category and descendent sub-categories up to a first depth n. The knowledge server130may determine a second set of document links from a seed document and then from further documents linked to the seed document and linked documents up to a second depth m. The depths n and m may be specified, for example, by a user such as an administrator of the knowledge server130. The greater the values of n and/or m corresponds to a larger domain specific corpus and may require more time to detect and fetch the respective documents and/or links. The knowledge server130may determine a third set of documents as an interaction between the first and second sets. The knowledge server130may then output documents corresponding to the links established in the third set of documents.

According to the second mechanism in which the knowledge server130performs an entity lookup-based automatic domain specific corpus expansion using knowledge base relations, the knowledge server130may utilize inputs including the documents of the determined corpus122. In performing the corpus expansion, the knowledge server130may process the documents of the determined corpus122and extract the domain terminology as described above. The knowledge server130may perform a text search on each extract phrase and collect the top results r where the search term is greater than the ranked list of r entities. The knowledge server130may determine the relation objects in each entity and count those that match with terms in the domain terminology where the relations establish relations rd. The knowledge server130may assigned each search result with a score where the score is based on the relations rd and a similarity. For example, the knowledge server130may compute the score as rd+(string similarity (phrase, entity label)*1/rank). After all the searches, the knowledge server130may determine a cumulative score of each entity appearing in the results and rank the entities based on the cumulative score. The knowledge server130may then select the top s entities from the ranked list. The values of r and s may be specified, for example, by a user such as the administrator of the knowledge server130. When r is smaller, only the top search results may be considered. The value of s may determine a size of the expanded corpus. The knowledge server130may utilize a query service to get corresponding documents in the data repository140for each entity in the list.

Through one or more of the above mechanisms, the knowledge server130may perform the corpus expansion by expanding the determined corpus122in its original form with the determined documents satisfying a threshold to expand the determined corpus122in a meaningful manner. Accordingly, in performing the corpus expansion, the knowledge server130may generate a modified corpus122which is an expanded form of the determined corpus122.

The knowledge server130may then generate the knowledge graph using the modified corpus122in a modified form relative to the original form (step212). The knowledge server130may induce knowledge from the knowledge graph (step214) and generate the response to the request based on the available knowledge (step216). For example, the inducing program138may induce knowledge from the knowledge graph that has been created.

The method200may include additional iterations of the above operations when more than one candidate corpus122is determined for use in generating the response. For example, a request for data may combine generally disparate domains. In such a request, the knowledge server130may require corresponding corpora122that are associated with each of the different domains. For each corpora122that is determined to be used, the knowledge server130may perform the above operations for information used to generate the response by determining a quality of each corpus122and expanding those corpus122that do not meet the minimum threshold for quality.

Through iterations of the method200, the corpora122may be expanded such that subsequent uses of any of the corpora122may not require expansion for the knowledge graph to induce knowledge therefrom. Accordingly, in an exemplary implementation, for a given application or type of query, the method200may be a one-time process and may not be required for each user query. In this manner, the method200may be performed as a pre-processing step before knowledge induction.

The exemplary embodiments are configured to determine whether a knowledge graph may be created based on a quality determination of a base corpus used to create the knowledge graph. The exemplary embodiments may perform an initial determination as to a quality of the base corpus. The base corpus having a minimum threshold quality may be used to generate the knowledge graph. However, the base corpus not satisfying the minimum threshold quality may require corpus expansion. The exemplary embodiments may perform the corpus expansion by selecting appropriate expansion corpora that expands the base corpus in a meaningful manner. Upon determining the appropriate expansion corpora, the exemplary embodiments may expand the base corpus using an automatic corpus expansion mechanism. Upon expanding the base corpus, the exemplary embodiments may generate the knowledge graph using the expanded corpus from which knowledge may be induced. Based on the knowledge graph and the induced knowledge, the exemplary embodiments may handle requests for data using this data.

FIG. 3depicts a block diagram of devices within the knowledge induction system100ofFIG. 1, in accordance with the exemplary embodiments. It should be appreciated thatFIG. 3provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

Devices used herein may include one or more processors02, one or more computer-readable RAMs04, one or more computer-readable ROMs06, one or more computer readable storage media08, device drivers12, read/write drive or interface14, network adapter or interface16, all interconnected over a communications fabric18. Communications fabric18may be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system.

One or more operating systems10, and one or more application programs11are stored on one or more of the computer readable storage media08for execution by one or more of the processors02via one or more of the respective RAMs04(which typically include cache memory). In the illustrated embodiment, each of the computer readable storage media08may be a magnetic disk storage device of an internal hard drive, CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk, a semiconductor storage device such as RAM, ROM, EPROM, flash memory or any other computer-readable tangible storage device that can store a computer program and digital information.

Devices used herein may also include a R/W drive or interface14to read from and write to one or more portable computer readable storage media26. Application programs11on said devices may be stored on one or more of the portable computer readable storage media26, read via the respective R/W drive or interface14and loaded into the respective computer readable storage media08.

Devices used herein may also include a display screen20, a keyboard or keypad22, and a computer mouse or touchpad24. Device drivers12interface to display screen20for imaging, to keyboard or keypad22, to computer mouse or touchpad24, and/or to display screen20for pressure sensing of alphanumeric character entry and user selections. The device drivers12, R/W drive or interface14and network adapter or interface16may comprise hardware and software (stored on computer readable storage media08and/or ROM06).

The programs described herein are identified based upon the application for which they are implemented in a specific one of the exemplary embodiments. However, it should be appreciated that any particular program nomenclature herein is used merely for convenience, and thus the exemplary embodiments should not be limited to use solely in any specific application identified and/or implied by such nomenclature.

Based on the foregoing, a computer system, method, and computer program product have been disclosed. However, numerous modifications and substitutions can be made without deviating from the scope of the exemplary embodiments. Therefore, the exemplary embodiments have been disclosed by way of example and not limitation.

It is to be understood that although this disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, the exemplary embodiments are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Hardware and software layer60include hardware and software components. Examples of hardware components include: mainframes61; RISC (Reduced Instruction Set Computer) architecture based servers62; servers63; blade servers64; storage devices65; and networks and networking components66. In some embodiments, software components include network application server software67and database software68.