Sub-question result merging in question and answer (QA) systems

An input question enhancement mechanism is provided for merging sub-question results in Question and Answer (QA) systems in order to answer an input question submitted from a user. A plurality of sub-questions associated with the input question is identified. Responsive to submitting the set of questions to a QA pipeline of the QA system, a set of answers and a set of passages related to the set of answers corresponding to each of the plurality of sub-questions are received. A mapping is applied to the set of passages to form a consolidated set of passages. The set of answers is then merged into the mapping so as to generate a consolidated set of answers. A factoid question response report is generated with answers from the consolidated set of answers and passages from the consolidated set of passages. The factoid question response report is then provided to the user.

BACKGROUND

The present application relates generally to an improved data processing apparatus and method and more specifically to mechanisms for merging sub-question results in Question and Answer (QA) systems in order to answer an input question.

With the increased usage of computing networks, such as the Internet, humans are currently inundated and overwhelmed with the amount of information available to them from various structured and unstructured sources. However, information gaps abound as users try to piece together what they can find that they believe to be relevant during searches for information on various subjects. To assist with such searches, recent research has been directed to generating Question and Answer (QA) systems which may take an input question, analyze it, and return results indicative of the most probable answer to the input question. QA systems provide automated mechanisms for searching through large sets of sources of content, e.g., electronic documents, and analyze them with regard to an input question to determine an answer to the question and a confidence measure as to how accurate an answer is for answering the input question.

Examples, of QA systems are Siri® from Apple®, Cortana® from Microsoft®, and question answering pipeline of the IBM Watson™ cognitive system available from International Business Machines (IBM®) Corporation of Armonk, N.Y. The IBM Watson™ system is an application of advanced natural language processing, information retrieval, knowledge representation and reasoning, and machine learning technologies to the field of open domain question answering. The IBM Watson™ system is built on IBM's DeepQA™ technology used for hypothesis generation, massive evidence gathering, analysis, and scoring. DeepQA™ takes an input question, analyzes it, decomposes the question into constituent parts, generates one or more hypothesis based on the decomposed question and results of a primary search of answer sources, performs hypothesis and evidence scoring based on a retrieval of evidence from evidence sources, performs synthesis of the one or more hypothesis, and based on trained models, performs a final merging and ranking to output an answer to the input question along with a confidence measure.

SUMMARY

In one illustrative embodiment, a method, in a data processing system, is provided to implement an input question enhancement mechanism for merging sub-question results in Question and Answer (QA) systems in order to answer an input question submitted from a user. The illustrative embodiment identifies a plurality of sub-questions associated with the input question. The illustrative embodiment receives a set of answers and a set of passages related to the set of answers corresponding to each of the plurality of sub-questions in response to submitting the set of questions to a QA pipeline of the QA system. The illustrative embodiment applies a mapping to the set of passages to form a consolidated set of passages. The illustrative embodiment merges the set of answers into the mapping so as to generate a consolidated set of answers. The illustrative embodiment generates a factoid question response report with answers from the consolidated set of answers and passages from the consolidated set of passages. The illustrative embodiment provides the factoid question response report to the user.

DETAILED DESCRIPTION

In the question-answering task, overly broad questions may be difficult to answer. For example, an input question like “Who are Dennis Rodman's friends?” would be easy to answer for a Question and Answer (QA) system to answer if a corpus of reference documents comprised a phrase that states “Dennis Rodman's friends are Michael Jordan and Moses Malone.” However, the QA system might struggle with differently phrased reference documents rich with pragmatic implications, such as “Dennis Rodman traveled to North Korea to visit Kim Jong-Un several times this year.”

In order to address the issue of dealing with differently phrased reference documents rich with pragmatic implications in order to answer such input questions, the illustrative embodiments provide mechanisms for domain adaptation techniques that may be used to map broad references within a corpus of reference documents to a set of specific questions, where the correct answers to the specific questions are assumed to be correct answers for to the input question. That is, the illustrative embodiment employs a set of sub-questions that are identified as being related to the input question. The QA system then operates to answer each sub-question separately and merge the results, i.e. the answer to each sub-question into a single report so the most relevant information is easily accessible. By performing such operations, the illustrative embodiments improve the answering capabilities of QA systems such that differently phrased reference documents rich with pragmatic implications from the corpus of reference documents provide an accurate answer to the input question.

FIGS.1-3are directed to describing an example cognitive system for merging sub-question results in order to answer an input question (also referred to as a request) submitted to a Question and Answer (QA) system, which implements a request processing pipeline, such as a Question Answering (QA) pipeline (also referred to as a Question/Answer pipeline or Question and Answer pipeline) for example, request processing methodology, and request processing computer program product with which the mechanisms of the illustrative embodiments are implemented. These requests may be provided as structure or unstructured request messages, natural language questions, or any other suitable format for requesting an operation to be performed by the cognitive system. As described in more detail hereafter, the particular application that is implemented in the cognitive system of the present invention is an application for dealing with differently phrased reference documents rich with pragmatic implications in order to answer a specific input question.

It should be appreciated that the cognitive system, while shown as having a single request processing pipeline in the examples hereafter, may in fact have multiple request processing pipelines. Each request processing pipeline may be separately trained and/or configured to process requests associated with different domains or be configured to perform the same or different analysis on input requests (or questions in implementations using a QA pipeline), depending on the desired implementation. For example, in some cases, a first request processing pipeline may be trained to operate on input requests directed to historical figures. In other cases, for example, the request processing pipelines may be configured to provide different types of cognitive functions or support different types of applications, such as one request processing pipeline being used for answering question directed to sports figures, medical diagnoses, legal matters, etc.

Moreover, each request processing pipeline may have its own associated corpus or corpora that they ingest and operate on, e.g., one corpus for historical domain related documents, another corpus for sports domain related documents, yet another corpus for medical domain related documents, and an additional corpus for law domain related documents, in the above examples. In some cases, the request processing pipelines may each operate on the same domain of input questions but may have different configurations, e.g., different annotators or differently trained annotators, such that different analysis and potential answers are generated. The cognitive system may provide additional logic for routing input questions to the appropriate request processing pipeline, such as based on a determined domain of the input request, combining and evaluating final results generated by the processing performed by multiple request processing pipelines, and other control and interaction logic that facilitates the utilization of multiple request processing pipelines.

As noted above, one type of request processing pipeline with which the mechanisms of the illustrative embodiments may be utilized is a Question Answering (QA) pipeline. The description of example embodiments of the present invention hereafter will utilize a QA pipeline as an example of a request processing pipeline that may be augmented to include mechanisms in accordance with one or more illustrative embodiments. It should be appreciated that while the present invention will be described in the context of the cognitive system implementing one or more QA pipelines that operate on an input question, the illustrative embodiments are not limited to such. Rather, the mechanisms of the illustrative embodiments may operate on requests that are not posed as “questions” but are formatted as requests for the cognitive system to perform cognitive operations on a specified set of input data using the associated corpus or corpora and the specific configuration information used to configure the cognitive system. For example, rather than asking a natural language question of “Who are Dennis Rodman's friends?”, the cognitive system may instead receive a request of “identify friends of Dennis Rodman,” or the like. It should be appreciated that the mechanisms of the QA system pipeline may operate on requests in a similar manner to that of input natural language questions with minor modifications. In fact, in some cases, a request may be converted to a natural language question for processing by the QA system pipelines if desired for the particular implementation.

As will be discussed in greater detail hereafter, the illustrative embodiments may be integrated in, augment, and extend the functionality of these QA pipeline, or request processing pipeline, mechanisms of a cognitive system with regard to merging sub-question results in Question and Answer (QA) systems in order to answer an input question. That is, the illustrative embodiment employs a set of sub-questions that are identified as being related to the input question received as a request from a user. The QA system then operates to answer each sub-question separately and merge the results, i.e. the answer to each sub-question into a single report so the most relevant information is easily accessible. By performing such operations, the illustrative embodiments improve the answering capabilities of QA systems such that differently phrased reference documents rich with pragmatic implications from the corpus of reference documents provide an accurate answer to the input question.

Thus, it is important to first have an understanding of how cognitive systems and question and answer creation in a cognitive system implementing a QA pipeline is implemented before describing how the mechanisms of the illustrative embodiments are integrated in and augment such cognitive systems and request processing pipeline, or QA pipeline, mechanisms. It should be appreciated that the mechanisms described inFIGS.1-3are only examples and are not intended to state or imply any limitation with regard to the type of cognitive system mechanisms with which the illustrative embodiments are implemented. Many modifications to the example cognitive system shown inFIGS.1-3may be implemented in various embodiments of the present invention without departing from the spirit and scope of the present invention.

As an overview, a cognitive system is a specialized computer system, or set of computer systems, configured with hardware and/or software logic (in combination with hardware logic upon which the software executes) to emulate human cognitive functions. These cognitive systems apply human-like characteristics to conveying and manipulating ideas which, when combined with the inherent strengths of digital computing, can solve problems with high accuracy and resilience on a large scale. A cognitive system performs one or more computer-implemented cognitive operations that approximate a human thought process as well as enable people and machines to interact in a more natural manner so as to extend and magnify human expertise and cognition. A cognitive system comprises artificial intelligence logic, such as natural language processing (NLP) based logic, for example, and machine learning logic, which may be provided as specialized hardware, software executed on hardware, or any combination of specialized hardware and software executed on hardware. The logic of the cognitive system implements the cognitive operation(s), examples of which include, but are not limited to, question answering, identification of related concepts within different portions of content in a corpus, intelligent search algorithms, such as Internet web page searches, for example, medical diagnostic and treatment recommendations, and other types of recommendation generation, e.g., items of interest to a particular user, potential new contact recommendations, or the like.

IBM Watson™ is an example of one such cognitive system which can process human readable language and identify inferences between text passages with human-like high accuracy at speeds far faster than human beings and on a larger scale. In general, such cognitive systems are able to perform the following functions:Navigate the complexities of human language and understanding,Ingest and process vast amounts of structured and unstructured data,Generate and evaluate hypothesis,Weigh and evaluate responses that are based only on relevant evidence,Provide situation-specific advice, insights, and guidance,Improve knowledge and learn with each iteration and interaction through machine learning processes,Enable decision making at the point of impact (contextual guidance),Scale in proportion to the task,Extend and magnify human expertise and cognition,Identify resonating, human-like attributes and traits from natural language,Deduce various language specific or agnostic attributes from natural language,High degree of relevant recollection from data points (images, text, voice) (memorization and recall),Predict and sense with situational awareness that mimic human cognition based on experiences, and/orAnswer questions based on natural language and specific evidence.

In one aspect, cognitive systems provide mechanisms for answering questions posed to these cognitive systems using a Question Answering pipeline or system (QA system) and/or process requests which may or may not be posed as natural language questions. The QA pipeline or system is an artificial intelligence application executing on data processing hardware that answers questions pertaining to a given subject-matter domain presented in natural language. The QA pipeline receives inputs from various sources including input over a network, a corpus of electronic documents or other data, data from a content creator, information from one or more content users, and other such inputs from other possible sources of input. Data storage devices store the corpus of data. A content creator creates content in a document for use as part of a corpus of data with the QA pipeline. The document may include any file, text, article, or source of data for use in the QA system. For example, a QA pipeline accesses a body of knowledge about the domain, or subject matter area, e.g., financial domain, medical domain, legal domain, etc., where the body of knowledge (knowledgebase) can be organized in a variety of configurations, e.g., a structured repository of domain-specific information, such as ontologies, or unstructured data related to the domain, or a collection of natural language documents about the domain.

Content users input questions to cognitive system which implements the QA pipeline. The QA pipeline then answers the input questions using the content in the corpus of data by evaluating documents, sections of documents, portions of data in the corpus, or the like. When a process evaluates a given section of a document for semantic content, the process can use a variety of conventions to query such document from the QA pipeline, e.g., sending the query to the QA pipeline as a well-formed question which is then interpreted by the QA pipeline and a response is provided containing one or more answers to the question. Semantic content is content based on the relation between signifiers, such as words, phrases, signs, and symbols, and what they stand for, their denotation, or connotation. In other words, semantic content is content that interprets an expression, such as by using Natural Language Processing.

As will be described in greater detail hereafter, the QA pipeline receives an input question, parses the question to extract the major features of the question, uses the extracted features to formulate queries, and then applies those queries to the corpus of data. Based on the application of the queries to the corpus of data, the QA pipeline generates a set of hypotheses, or candidate answers to the input question, by looking across the corpus of data for portions of the corpus of data that have some potential for containing a valuable response to the input question. The QA pipeline then performs deep analysis on the language of the input question and the language used in each of the portions of the corpus of data found during the application of the queries using a variety of reasoning algorithms. There may be hundreds or even thousands of reasoning algorithms applied, each of which performs different analysis, e.g., comparisons, natural language analysis, lexical analysis, or the like, and generates a score. For example, some reasoning algorithms may look at the matching of terms and synonyms within the language of the input question and the found portions of the corpus of data. Other reasoning algorithms may look at temporal or spatial features in the language, while others may evaluate the source of the portion of the corpus of data and evaluate its veracity.

The scores obtained from the various reasoning algorithms indicate the extent to which the potential response is inferred by the input question based on the specific area of focus of that reasoning algorithm. Each resulting score is then weighted against a statistical model. The statistical model captures how well the reasoning algorithm performed at establishing the inference between two similar passages for a particular domain during the training period of the QA pipeline. The statistical model is used to summarize a level of confidence that the QA pipeline has regarding the evidence that the potential response, i.e. candidate answer, is inferred by the question. This process is repeated for each of the candidate answers until the QA pipeline identifies candidate answers that surface as being significantly stronger than others and thus, generates a final answer, or ranked set of answers, for the input question.

As mentioned above, QA pipeline mechanisms operate by accessing information from a corpus of data or information (also referred to as a corpus of content), analyzing it, and then generating answer results based on the analysis of this data. Accessing information from a corpus of data typically includes: a database query that answers questions about what is in a collection of structured records, and a search that delivers a collection of document links in response to a query against a collection of unstructured data (text, markup language, etc.). Conventional question answering systems are capable of generating answers based on the corpus of data and the input question, verifying answers to a collection of questions for the corpus of data, correcting errors in digital text using a corpus of data, and selecting answers to questions from a pool of potential answers, i.e. candidate answers.

Content creators, such as article authors, electronic document creators, web page authors, document database creators, and the like, determine use cases for products, solutions, and services described in such content before writing their content. Consequently, the content creators know what questions the content is intended to answer in a particular topic addressed by the content. Categorizing the questions, such as in terms of roles, type of information, tasks, or the like, associated with the question, in each document of a corpus of data allows the QA pipeline to more quickly and efficiently identify documents containing content related to a specific query. The content may also answer other questions that the content creator did not contemplate that may be useful to content users. The questions and answers may be verified by the content creator to be contained in the content for a given document. These capabilities contribute to improved accuracy, system performance, machine learning, and confidence of the QA pipeline. Content creators, automated tools, or the like, annotate or otherwise generate metadata for providing information useable by the QA pipeline to identify these question and answer attributes of the content.

Operating on such content, the QA pipeline generates answers for input questions using a plurality of intensive analysis mechanisms which evaluate the content to identify the most probable answers, i.e. candidate answers, for the input question. The most probable answers are output as a ranked listing of candidate answers ranked according to their relative scores or confidence measures calculated during evaluation of the candidate answers, as a single final answer having a highest ranking score or confidence measure, or which is a best match to the input question, or a combination of ranked listing and final answer.

FIG.1depicts a schematic diagram of one illustrative embodiment of a cognitive system100implementing a request processing pipeline108, which in some embodiments may be a question answering (QA) pipeline, in a computer network102. For purposes of the present description, it will be assumed that the request processing pipeline108is implemented as a QA pipeline that operates on structured and/or unstructured requests in the form of input questions. One example of a question processing operation which may be used in conjunction with the principles described herein is described in U.S. Patent Application Publication No. 2011/0125734, which is herein incorporated by reference in its entirety. The cognitive system100is implemented on one or more computing devices104A-D (comprising one or more processors and one or more memories, and potentially any other computing device elements generally known in the art including buses, storage devices, communication interfaces, and the like) connected to the computer network102. For purposes of illustration only,FIG.1depicts the cognitive system100being implemented on computing device104A only, but as noted above the cognitive system100may be distributed across multiple computing devices, such as a plurality of computing devices104A-D. The network102includes multiple computing devices104A-D, which may operate as server computing devices, and110-112which may operate as client computing devices, in communication with each other and with other devices or components via one or more wired and/or wireless data communication links, where each communication link comprises one or more of wires, routers, switches, transmitters, receivers, or the like. In some illustrative embodiments, the cognitive system100and network102enables question processing and answer generation (QA) functionality for one or more cognitive system users via their respective computing devices110-112. In other embodiments, the cognitive system100and network102may provide other types of cognitive operations including, but not limited to, request processing and cognitive response generation which may take many different forms depending upon the desired implementation, e.g., cognitive information retrieval, training/instruction of users, cognitive evaluation of data, or the like. Other embodiments of the cognitive system100may be used with components, systems, sub-systems, and/or devices other than those that are depicted herein.

The cognitive system100is configured to implement a request processing pipeline108that receive inputs from various sources. The requests may be posed in the form of a natural language question, natural language request for information, natural language request for the performance of a cognitive operation, or the like. For example, the cognitive system100receives input from the network102, a corpus or corpora of electronic documents106, cognitive system users, and/or other data and other possible sources of input. In one embodiment, some or all of the inputs to the cognitive system100are routed through the network102. The various computing devices104A-D on the network102include access points for content creators and cognitive system users. Some of the computing devices104A-D include devices for a database storing the corpus or corpora of data106(which is shown as a separate entity inFIG.1for illustrative purposes only). Portions of the corpus or corpora of data106may also be provided on one or more other network attached storage devices, in one or more databases, or other computing devices not explicitly shown inFIG.1. The network102includes local network connections and remote connections in various embodiments, such that the cognitive system100may operate in environments of any size, including local and global, e.g., the Internet.

In one embodiment, the content creator creates content in a document of the corpus or corpora of data106for use as part of a corpus of data with the cognitive system100. The document includes any file, text, article, or source of data for use in the cognitive system100. Cognitive system users access the cognitive system100via a network connection or an Internet connection to the network102, and input questions/requests to the cognitive system100that are answered/processed based on the content in the corpus or corpora of data106. In one embodiment, the questions/requests are formed using natural language. The cognitive system100parses and interprets the question/request via a pipeline108, and provides a response to the cognitive system user, e.g., cognitive system user110, containing one or more answers to the question posed, response to the request, results of processing the request, or the like. In some embodiments, the cognitive system100provides a response to users in a ranked list of candidate answers/responses while in other illustrative embodiments, the cognitive system100provides a single final answer/response or a combination of a final answer/response and ranked listing of other candidate answers/responses.

The cognitive system100implements the pipeline108which comprises a plurality of stages for processing an input question/request based on information obtained from the corpus or corpora of data106. The pipeline108generates answers/responses for the input question or request based on the processing of the input question/request and the corpus or corpora of data106. The pipeline108will be described in greater detail hereafter with regard toFIG.3.

In some illustrative embodiments, the cognitive system100may be the IBM Watson™ cognitive system available from International Business Machines Corporation of Armonk, N.Y., which is augmented with the mechanisms of the illustrative embodiments described hereafter. As outlined previously, a pipeline of the IBM Watson™ cognitive system receives an input question or request which it then parses to extract the major features of the question/request, which in turn are then used to formulate queries that are applied to the corpus or corpora of data106. Based on the application of the queries to the corpus or corpora of data106, a set of hypotheses, or candidate answers/responses to the input question/request, are generated by looking across the corpus or corpora of data106for portions of the corpus or corpora of data106(hereafter referred to simply as the corpus106) that have some potential for containing a valuable response to the input question/response (hereafter assumed to be an input question). The pipeline108of the IBM Watson™ cognitive system then performs deep analysis on the language of the input question and the language used in each of the portions of the corpus106found during the application of the queries using a variety of reasoning algorithms.

The scores obtained from the various reasoning algorithms are then weighted against a statistical model that summarizes a level of confidence that the pipeline108of the IBM Watson™ cognitive system100, in this example, has regarding the evidence that the potential candidate answer is inferred by the question. This process is be repeated for each of the candidate answers to generate ranked listing of candidate answers which may then be presented to the user that submitted the input question, e.g., a user of client computing device110, or from which a final answer is selected and presented to the user. More information about the pipeline108of the IBM Watson™ cognitive system100may be obtained, for example, from the IBM Corporation website, IBM Redbooks, and the like. For example, information about the pipeline of the IBM Watson™ cognitive system can be found in Yuan et al., “Watson and Healthcare,” IBM developerWorks, 2011 and “The Era of Cognitive Systems: An Inside Look at IBM Watson and How it Works” by Rob High, IBM Redbooks, 2012.

As noted above, while the input to the cognitive system100from a client device may be posed in the form of a natural language question, the illustrative embodiments are not limited to such. Rather, the input question may in fact be formatted or structured as any suitable type of request which may be parsed and analyzed using structured and/or unstructured input analysis, including but not limited to the natural language parsing and analysis mechanisms of a cognitive system such as IBM Watson™, to determine the basis upon which to perform cognitive analysis and providing a result of the cognitive analysis.

In the context of the present invention, cognitive system100may provide a cognitive functionality for merging sub-question results in Question and Answer (QA) systems in order to answer an input question. For example, depending upon the particular implementation, the knowledge base expansion based operations may comprise, responsive to receiving an input question identifying a set of sub-questions related to the input question. Then, the QA system operates to answer the input question and each of the set of sub-questions separately using differently phrased reference documents rich with pragmatic implications thereby adapting the domain to which the questions are posed. The QA system then merges the results, i.e. the answer to the input question and each sub-question into a single report so the most relevant information is easily accessible. By performing such operations, the illustrative embodiments improve the answering capabilities of QA systems such that differently phrased reference documents rich with pragmatic implications from the corpus of reference documents provide an accurate answer to the input question.

As shown inFIG.1, the cognitive system100is further augmented, in accordance with the mechanisms of the illustrative embodiments, to include logic implemented in specialized hardware, software executed on hardware, or any combination of specialized hardware and software executed on hardware, for implementing input question enhancement mechanism120. Input question enhancement mechanism120receives input question122from a user. Sub-question identification engine124receives parsing information from a question and topic analysis stage of request processing pipeline108, the parsing information identifying major features from the input question and a classification of the major features according to types, e.g., names, dates, or any of a plethora of other defined topics.

Utilizing this parsing information, sub-question identification engine124identifies a list of sub-questions from corpus of sub-questions126related to input question122based on the parsing information, thereby forming a set of questions to be posed to the corpus of data/information140in order to generate one or more hypotheses. The set of questions are generated in any known or later developed query language, such as the Structure Query Language (SQL), or the like. The set of questions are applied to one or more databases storing information about the electronic texts, documents, articles, websites, and the like, that make up the corpora of data/information140. That is, these various sources themselves, different collections of sources, and the like, represent a different corpus142within the corpora140. There may be different corpora142defined for different collections of documents based on various criteria depending upon the particular implementation. For example, different corpora may be established for different topics, subject matter categories, sources of information, or the like.

In conjunction with a hypothesis generation stage of request processing pipeline108, each question of the set of questions is submitted to request processing pipeline108and result processing engine128stores a set of answers provided by request processing pipeline108in answer data structure130and a set of related passages related to the set of answers in passage data structure132. Initially, answer data structure130and passage data structure132are empty. As result processing engine128adds an associated passage to passage data structure132, result processing engine128determines whether the passage already exists in passage data structure132. If result processing engine128determines that the associated passage already exists in passage data structure132, then result processing engine128set that passage in passage data structure132to have the higher score between itself and the answer as well as any other answer associated with the passage. If result processing engine128determines that the associated passage fails to exist in passage data structure132, result processing engine128adds the associated passage to passage data structure132. Once all passages are added to passage data structure132, result processing engine128sorts passage data structure132by score and generates a map m[x][y] that relates the question form the set of questions (x) and the passage associated with the answer to that question (y) of original passages to merged passage index134.

In another operation, as result processing engine128adds each answer to answer data structure130, result processing engine128updates each passage index for each answer occurrence of the answer to m[ai][pi], where ai is the index of the answer in answer data structure130and pi is the original passage index for this answer in merged passage index134. If result processing engine128determines that an answer to a question from the set of questions already exists in answer data structure130, result processing engine128determines whether a score associated with that answer being added is higher than the answer already existing in answer data structure130. The score associated with each answer is provided by a hypothesis and evidence scoring stage and a synthesis stage of request processing pipeline108. If result processing engine128determines that a score associated with the answer being added is higher, then result processing engine128sets the existing answer's score in answer data structure130to the higher score. If result processing engine128determines that the answer's score being added fails to be higher, then result processing engine128leaves the score of the existing answer as is. If result processing engine128determines that an answer to a question from the set of questions fails to exist in answer data structure130, result processing engine128adds the answer with its score to answer data structure130. Once all answers are added to answer data structure130, result processing engine128sorts answer data structure130by score. Result processing engine128then generates a factoid question response report with answers from answer data structure130and passages from passage data structure132. Result processing engine128then provides the factoid question response report to a confidence merging and ranking stage of request processing pipeline108so as to replace the normal operations provided by the confidence merging and ranking stage and thus, be utilized in a final set of candidate answers and confidence scores stage of request processing pipeline108.

As noted above, the mechanisms of the illustrative embodiments are rooted in the computer technology arts and are implemented using logic present in such computing or data processing systems. These computing or data processing systems are specifically configured, either through hardware, software, or a combination of hardware and software, to implement the various operations described above. As such,FIG.2is provided as an example of one type of data processing system in which aspects of the present invention may be implemented. Many other types of data processing systems may be likewise configured to specifically implement the mechanisms of the illustrative embodiments.

FIG.2is a block diagram of an example data processing system in which aspects of the illustrative embodiments are implemented. Data processing system200is an example of a computer, such as server104or client110inFIG.1, in which computer usable code or instructions implementing the processes for illustrative embodiments of the present invention are located. In one illustrative embodiment,FIG.2represents a server computing device, such as a server104, which, which implements a cognitive system100and QA system pipeline108augmented to include the additional mechanisms of the illustrative embodiments described hereafter.

As a server, data processing system200may be, for example, an IBM® eServer™ System p® computer system, running the Advanced Interactive Executive (AIX®) operating system or the LINUX® operating system. Data processing system200may be a symmetric multiprocessor (SMP) system including a plurality of processors in processing unit206. Alternatively, a single processor system may be employed.

Instructions for the operating system, the object-oriented programming system, and applications or programs are located on storage devices, such as HDD226, and are loaded into main memory208for execution by processing unit206. The processes for illustrative embodiments of the present invention are performed by processing unit206using computer usable program code, which is located in a memory such as, for example, main memory208, ROM224, or in one or more peripheral devices226and230, for example.

A bus system, such as bus238or bus240as shown inFIG.2, is comprised of one or more buses. Of course, the bus system may be implemented using any type of communication fabric or architecture that provides for a transfer of data between different components or devices attached to the fabric or architecture. A communication unit, such as modem222or network adapter212ofFIG.2, includes one or more devices used to transmit and receive data. A memory may be, for example, main memory208, ROM224, or a cache such as found in NB/MCH202inFIG.2.

FIG.3illustrates an example of a cognitive system processing pipeline which, in the depicted example, is a question and answer (QA) system pipeline used to process an input question in accordance with one illustrative embodiment. As noted above, the cognitive systems with which the illustrative embodiments may be utilized are not limited to QA systems and thus, not limited to the use of a QA system pipeline.FIG.3is provided only as one example of the processing structure that may be implemented to process a natural language input requesting the operation of a cognitive system to present a response or result to the natural language input.

The QA system pipeline ofFIG.3may be implemented, for example, as QA pipeline108of cognitive system100inFIG.1. It should be appreciated that the stages of the QA pipeline shown inFIG.3are implemented as one or more software engines, components, or the like, which are configured with logic for implementing the functionality attributed to the particular stage. Each stage is implemented using one or more of such software engines, components or the like. The software engines, components, etc. are executed on one or more processors of one or more data processing systems or devices and utilize or operate on data stored in one or more data storage devices, memories, or the like, on one or more of the data processing systems. The QA pipeline ofFIG.3is augmented, for example, in one or more of the stages to implement the improved mechanism of the illustrative embodiments described hereafter, additional stages may be provided to implement the improved mechanism, or separate logic from the pipeline300may be provided for interfacing with the pipeline300and implementing the improved functionality and operations of the illustrative embodiments.

As shown inFIG.3, the QA pipeline300comprises a plurality of stages310-380through which the cognitive system operates to analyze an input question and generate a final response. In an initial question input stage310, the QA pipeline300receives an input question that is presented in a natural language format. That is, a user inputs, via a user interface, an input question for which the user wishes to obtain an answer, e.g., “Who are Washington's closest advisors?” In response to receiving the input question, the next stage of the QA pipeline300, i.e. the question and topic analysis stage320, parses the input question using natural language processing (NLP) techniques to extract major features from the input question, and classify the major features according to types, e.g., names, dates, or any of a plethora of other defined topics. For example, in the example question above, the term “who” may be associated with a topic for “persons” indicating that the identity of a person is being sought, “Washington” may be identified as a proper name of a person with which the question is associated, “closest” may be identified as a word indicative of proximity or relationship, and “advisors” may be indicative of a noun or other language topic.

In addition, the extracted major features include key words and phrases classified into question characteristics, such as the focus of the question, the lexical answer type (LAT) of the question, and the like. As referred to herein, a lexical answer type (LAT) is a word in, or a word inferred from, the input question that indicates the type of the answer, independent of assigning semantics to that word. For example, in the question “What maneuver was invented in the 1500s to speed up the game and involves two pieces of the same color?”, the LAT is the string “maneuver.” The focus of a question is the part of the question that, if replaced by the answer, makes the question a standalone statement. For example, in the question “What drug has been shown to relieve the symptoms of ADD with relatively few side effects?,” the focus is “drug” since if this word were replaced with the answer, e.g., the answer “Adderall” can be used to replace the term “drug” to generate the sentence “Adderall has been shown to relieve the symptoms of ADD with relatively few side effects.” The focus often, but not always, contains the LAT. On the other hand, in many cases it is not possible to infer a meaningful LAT from the focus.

Referring again toFIG.3, the identified major features are then used during the question decomposition stage330to decompose the question into one or more queries that are applied to the corpora of data/information345in order to generate one or more hypotheses. The queries are generated in any known or later developed query language, such as the Structure Query Language (SQL), or the like. The queries are applied to one or more databases storing information about the electronic texts, documents, articles, websites, and the like, that make up the corpora of data/information345. That is, these various sources themselves, different collections of sources, and the like, represent a different corpus347within the corpora345. There may be different corpora347defined for different collections of documents based on various criteria depending upon the particular implementation. For example, different corpora may be established for different topics, subject matter categories, sources of information, or the like. As one example, a first corpus may be associated with healthcare documents while a second corpus may be associated with financial documents. Alternatively, one corpus may be documents published by the U.S. Department of Energy while another corpus may be IBM Redbooks documents. Any collection of content having some similar attribute may be considered to be a corpus347within the corpora345.

The queries are applied to one or more databases storing information about the electronic texts, documents, articles, websites, and the like, that make up the corpus of data/information, e.g., the corpus of data106inFIG.1. The queries are applied to the corpus of data/information at the hypothesis generation stage340to generate results identifying potential hypotheses for answering the input question, which can then be evaluated. That is, the application of the queries results in the extraction of portions of the corpus of data/information matching the criteria of the particular query. These portions of the corpus are then analyzed and used, during the hypothesis generation stage340, to generate hypotheses for answering the input question. These hypotheses are also referred to herein as “candidate answers” for the input question. For any input question, at this stage340, there may be hundreds of hypotheses or candidate answers generated that may need to be evaluated.

The QA pipeline300, in stage350, then performs a deep analysis and comparison of the language of the input question and the language of each hypothesis or “candidate answer,” as well as performs evidence scoring to evaluate the likelihood that the particular hypothesis is a correct answer for the input question. As mentioned above, this involves using a plurality of reasoning algorithms, each performing a separate type of analysis of the language of the input question and/or content of the corpus that provides evidence in support of, or not in support of, the hypothesis. Each reasoning algorithm generates a score based on the analysis it performs which indicates a measure of relevance of the individual portions of the corpus of data/information extracted by application of the queries as well as a measure of the correctness of the corresponding hypothesis, i.e. a measure of confidence in the hypothesis. There are various ways of generating such scores depending upon the particular analysis being performed. In generally, however, these algorithms look for particular terms, phrases, or patterns of text that are indicative of terms, phrases, or patterns of interest and determine a degree of matching with higher degrees of matching being given relatively higher scores than lower degrees of matching.

Thus, for example, an algorithm may be configured to look for the exact term from an input question or synonyms to that term in the input question, e.g., the exact term or synonyms for the term “movie,” and generate a score based on a frequency of use of these exact terms or synonyms. In such a case, exact matches will be given the highest scores, while synonyms may be given lower scores based on a relative ranking of the synonyms as may be specified by a subject matter expert (person with knowledge of the particular domain and terminology used) or automatically determined from frequency of use of the synonym in the corpus corresponding to the domain. Thus, for example, an exact match of the term “movie” in content of the corpus (also referred to as evidence, or evidence passages) is given a highest score. A synonym of movie, such as “motion picture” may be given a lower score but still higher than a synonym of the type “film” or “moving picture show.” Instances of the exact matches and synonyms for each evidence passage may be compiled and used in a quantitative function to generate a score for the degree of matching of the evidence passage to the input question.

Thus, for example, a hypothesis or candidate answer to the input question of “What was the first movie?” is “The Horse in Motion.” If the evidence passage contains the statements “The first motion picture ever made was ‘The Horse in Motion’ in 1878 by Eadweard Muybridge. It was a movie of a horse running,” and the algorithm is looking for exact matches or synonyms to the focus of the input question, i.e. “movie,” then an exact match of “movie” is found in the second sentence of the evidence passage and a highly scored synonym to “movie,” i.e. “motion picture,” is found in the first sentence of the evidence passage. This may be combined with further analysis of the evidence passage to identify that the text of the candidate answer is present in the evidence passage as well, i.e. “The Horse in Motion.” These factors may be combined to give this evidence passage a relatively high score as supporting evidence for the candidate answer “The Horse in Motion” being a correct answer.

It should be appreciated that this is just one simple example of how scoring can be performed. Many other algorithms of various complexities may be used to generate scores for candidate answers and evidence without departing from the spirit and scope of the present invention.

In the synthesis stage360, the large number of scores generated by the various reasoning algorithms is synthesized into confidence scores or confidence measures for the various hypotheses. This process involves applying weights to the various scores, where the weights have been determined through training of the statistical model employed by the QA pipeline300and/or dynamically updated. For example, the weights for scores generated by algorithms that identify exactly matching terms and synonym may be set relatively higher than other algorithms that are evaluating publication dates for evidence passages. The weights themselves may be specified by subject matter experts or learned through machine learning processes that evaluate the significance of characteristics evidence passages and their relative importance to overall candidate answer generation.

The weighted scores are processed in accordance with a statistical model generated through training of the QA pipeline300that identifies a manner by which these scores may be combined to generate a confidence score or measure for the individual hypotheses or candidate answers. This confidence score or measure summarizes the level of confidence that the QA pipeline300has about the evidence that the candidate answer is inferred by the input question, i.e. that the candidate answer is the correct answer for the input question.

The resulting confidence scores or measures are processed by a final confidence merging and ranking stage370which compares the confidence scores and measures to each other, compares them against predetermined thresholds, or performs any other analysis on the confidence scores to determine which hypotheses/candidate answers are the most likely to be the correct answer to the input question. The hypotheses/candidate answers are ranked according to these comparisons to generate a ranked listing of hypotheses/candidate answers (hereafter simply referred to as “candidate answers”). From the ranked listing of candidate answers, at stage380, a final answer and confidence score, or final set of candidate answers and confidence scores, are generated and output to the submitter of the original input question via a graphical user interface or other mechanism for outputting information.

As shown inFIG.3, in accordance with one illustrative embodiment, QA pipeline300is further augmented, in accordance with the mechanisms of the illustrative embodiments, to include logic implemented in specialized hardware, software executed on hardware, or any combination of specialized hardware and software executed on hardware, for implementing input question enhancement mechanism390. As QA pipeline300receives input question310from a user, sub-question identification engine391receives parsing information from a question and topic analysis stage320, the parsing information identifying major features from the input question and a classification of the major features according to types, e.g., names, dates, or any of a plethora of other defined topics.

Utilizing this parsing information, sub-question identification engine391identifies a list of sub-questions from corpus of sub-questions392related to input question130based on the parsing information, thereby forming a set of questions to be posed to the corpus of data/information345in order to generate one or more hypotheses. The set of questions are generated in any known or later developed query language, such as the Structure Query Language (SQL), or the like. The set of questions are applied to one or more databases storing information about the electronic texts, documents, articles, websites, and the like, that make up the corpora of data/information345.

In conjunction with hypothesis generation stage340, each question of the set of questions is applied to the corpus of data/information to generate results identifying potential hypotheses for answering the input question. That is, the application of the queries results in the extraction of portions of the corpus of data/information matching the criteria of the particular query. These portions of the corpus are then analyzed and used, during the hypothesis generation stage340, to generate hypotheses for answering the question. These hypotheses are also referred to herein as “candidate answers” for the question. The hypothesis generation stage340provides these candidate answers and supporting passages to result processing engine393. Result processing engine393stores a set of answers provided by hypothesis generation stage340in answer data structure394and stores the set of related passages related to the set of answers in passage data structure395. Initially, answer data structure394and passage data structure395are empty. As result processing engine393adds an associated passage to passage data structure395, result processing engine393determines whether the passage already exists in passage data structure395. If result processing engine393determines that the associated passage already exists in passage data structure395, then result processing engine393sets that passage in passage data structure395to have the higher score between itself and the answer as well as any other answer associated with the passage. If result processing engine393determines that the associated passage fails to exist in passage data structure395, result processing engine393adds the associated passage to passage data structure395and sets a score between the associated passage and the answer associated with the passage to an initial value. Once all passages are analyzed, result processing engine393sorts passage data structure395by score and generates a map m[x][y] that relates the question form the set of questions (x) and the passage associated with the answer to that question (y) of original passages in merged passage index396.

In another operation, as result processing engine393adds each answer to answer data structure394, result processing engine393updates each passage index for each answer occurrence of the answer to m[ai][pi], where ai is the index of the answer in answer data structure394and pi is the original passage index for this answer in merged passage index396. If result processing engine393determines that an answer to a question from the set of questions already exists in answer data structure394, result processing engine393determines whether an assigned score associated with that answer being added is higher than the answer already existing in answer data structure394. The score associated with each answer is provided by the hypothesis and evidence scoring stage350and the synthesis stage360of QA pipeline300. If result processing engine393determines that a score associated with the answer being added is higher, then result processing engine393sets the existing answer's score in answer data structure394to the higher score. If result processing engine393determines that the answer's score being added fails to be higher, then result processing engine393leaves the score of the existing answer as is. If result processing engine393determines that an answer to a question from the set of questions fails to exist in answer data structure394, result processing engine393adds the answer with its score to answer data structure394. Once all answers are added to answer data structure394, result processing engine393sorts answer data structure394by score. Result processing engine393then generates a factoid question response report with answers from answer data structure394and passages from passage data structure395. Result processing engine393then provides the factoid question response report to the confidence merging and ranking stage370so as to replace the normal operations provided by the confidence merging and ranking stage370and thus, be utilized in a final set of candidate answers and confidence scores stage380to produce a final answer and confidence score, or final set of candidate answers and confidence scores, which are output to the submitter of the original input question via a graphical user interface or other mechanism for outputting information.

The following provides one example of the operation performed by input question enhancement mechanism390. Responsive to receiving an input question130of “Who are Emanuel Macron's closest advisors?”, sub-question identification engine391identifies from corpus of sub-questions392though domain adaptation efforts a series of more-specific sub-questions, where the correct answer to each sub-question is considered a valid answer to input question130. The set of sub-questions being “Who is the Prime Minister of France?”, “What cabinet ministers were appointed by Emanuel Macron?”, and “Who served as a senior member of Emanuel Macron's campaigns?” Each of these sub-questions as well as possibly input question130is applied through hypothesis generation stage340. For each question, hypothesis generation stage340provides an answer and related passages where the answer scheme is of the form {answer,score,[{passage,sentence,token} . . . ]}′ and the passage scheme is of the form ‘ {passage, score} ’.

Thus, responsive to the question “Who is the Prime Minister of France?”, the answers are {Édouard Charles Philippe, 0.9, [{0, 0, 0}]}′ and {Nicolas Hulot, 0.4, [{0, 1, 3}]}′ and the associated passages are {Edouard Charles Philippe is the French prime minister. He met with Nicolas Hulot., 0.7}′. Responsive to the question “What cabinet ministers were appointed by Emanuel Macron?”, the answers are {Bruno Le Maire, 0.85, [{0, 0, 2}]}′, {Francois Bayrou, 0.80, [{0, 0, 4}]}′, {Nicolas Hulot, 0.75, [{1, 0, 0}]}′, and {Edouard Charles Philippe, 0.1, [{1, 0, 0}]}′ and the associated passages are {Macron appointed Bruno Le Maire and Francois Bayrou as his economy minister and justice minister, respectively., 0.85}′, {Nicolas Hulot dramatically quit Macron's cabinet on live radio., 0.6}′, {Édouard Charles Philippe is the French prime minister. He met with Nicolas Hulot., 0.2}′. Responsive to the question “Who served as a senior member of Emanuel Macron's campaigns?”, the answers are {Alexis Kohler, 0.5, [{0, 0, 0}]}′ and {Elyśee, 0.1, [{0, 0, 4}]}′ and the associated passage is “{Alexis Kohler: Secretary-general at the Elyśee, he has worked closely with Mr. Macron for several years and has long acted as his special adviser., 0.3}”.

Responsive to receiving these results, results processing engine393merges and sorts the passages as described previously such that passage data structure395comprises ‘{Macron appointed Bruno Le Maire and Francois Bayrou as his economy minister and justice minister, respectively., 0.85}’, ‘{Édouard Charles Philippe is the French prime minister. He met with Nicolas Hulot., 0.7},’ ‘{Nicolas Hulot dramatically quit Macron's cabinet on live radio., 0.6}’, and ‘{Alexis Kohler: Secretary-general at the Elyśee, he has worked closely with Mr. Macron for several years and has long acted as his special adviser., 0.3}’. Note that there is one passage ‘Édouard Charles Philippe is the French prime minister. He met with Nicolas Hulot.’ that appears in response to two sub-questions. Result processing engine393then generates a map m[x][y] that relates the question form the set of questions (x) and the passage associated with the answer to that question (y) of original passages in merged passage index396. The map ends up with the following values:m[0][0]=1m[1][0]=0m[1][1]=2m[1][2]=1m[2][0]=3
Notice that ‘m[0][0]’ and ‘m[1][2]’ map to the same passage—the duplicated passage mentioned above.

Now, result processing engine393merges the answers, taking care to update the indexes of the answer occurrences using the map m, the result being:{Édouard Charles Philippe, 0.9, [}1, 0, 0}]},{Bruno Le Maire, 0.85, [{0, 0, 2}]},{Franøois Bayrou, 0.80, [{0, 0, 4}]},{Nicolas Hulot, 0.75, [{2, 0, 0},{1, 1, 3}]},{Alexis Kohler, 0.5, [{3, 0, 0}]},{Édouard Charles Philippe, 0.1, [{2, 0, 0}]}{Elysée, 0.1, [{3, 0, 4}]}
Notice that the two instances of Nicolas Hulot from different sub-questions are merged. Result processing engine393then provides this factoid question response report to the confidence merging and ranking stage370so as to replace the normal operations provided by the confidence merging and ranking stage370and thus, be utilized in a final set of candidate answers and confidence scores stage380to produce a final answer and confidence score, or final set of candidate answers and confidence scores, which are output to the submitter of the original input question via a graphical user interface or other mechanism for outputting information.

FIG.4depicts one example the operation performed by an input question enhancement mechanism in identifying a set of sub-questions related to an input question in accordance with an illustrative embodiment. As the operation begins, the input question enhancement mechanism receives parsed information associated with an input question (step402) the parsed information identifying major features from the input question and a classification of the major features according to types, e.g., names, dates, or any of a plethora of other defined topics. The input question enhancement mechanism identifies a list of sub-questions from a corpus of sub-questions related to the input question based on the parsed information (step404) thereby forming a set of questions to be posed to a corpus of data/information in order to generate one or more hypotheses. The input question enhancement mechanism then applies each of these set of questions to the corpora of data/information (step406) through a hypothesis generation stage of a QA system to which the input question enhancement mechanism is associated. The operation terminates thereafter.

FIGS.5A and5Bdepict one example the operation performed by an input question enhancement mechanism in analyzing answers and passages obtained as a result processing the set of sub-questions related to an input question in accordance with an illustrative embodiment. As a result of applying each of the set of questions results in the extraction of portions of the corpus of data/information matching the criteria of the particular query, these portions of the corpus are then analyzed and used to generate hypotheses for answering the question, i.e. “candidate answers”. Upon receiving each of these candidate answers and supporting passages, the input question enhancement mechanism stores a set of answers in an answer data structure and stores the set of related passages related to the set of answers in a passage data structure. Initially, the answer data structure and the passage data structure are empty. Thus, as the operation begins, the input question enhancement mechanism determines whether the passage already exists in the passage data structure (step502). If at step502the input question enhancement mechanism determines that the associated passage already exists in the passage data structure, then the input question enhancement mechanism sets that passage in the passage data structure to have the higher score between itself and the answer as well as any other answer associated with the passage (step504). If at step502the input question enhancement mechanism determines that the associated passage fails to exist in the passage data structure, the input question enhancement mechanism adds the associated passage to the passage data structure (step506).

From steps504and506, the input question enhancement mechanism then determines whether there is another passage to process (step508). If at step508the input question enhancement mechanism determines that there is another passage to process, the operation returns to step502. If at step508the input question enhancement mechanism determines that there is not another passage to process, the input question enhancement mechanism sorts the passage data structure by score (step510) and generates a map m[x][y] that relates the question form the set of questions (x) and the passage associated with the answer to that question (y) of original passages in a merged passage index (step512).

The input question enhancement mechanism then tors to the answers associated with the results. As the input question enhancement mechanism adds each answer to the answer data structure, the input question enhancement mechanism updates each passage index for each answer occurrence of the answer to m[ai][pi], where ai is the index of the answer in the answer data structure and pi is the original passage index for this answer in the merged passage index (step514). The input question enhancement mechanism then determines whether an answer to a question from the set of questions already exists in the answer data structure (step516). If at step516the input question enhancement mechanism determines that an answer to a question from the set of questions fails to exist in the answer data structure, the input question enhancement mechanism adds the answer to the answer data structure (step518). If at step516the input question enhancement mechanism determines that an answer to a question from the set of questions already exists in the answer data structure, the input question enhancement mechanism determines whether a score associated with that answer being added is higher than the answer already existing in the answer data structure (step520). If at step520the input question enhancement mechanism determines that a score associated with the answer being added is higher, then the input question enhancement mechanism sets the existing answer's score in the answer data structure to the higher score (step522). If at step520the input question enhancement mechanism determines that the answer's score being added fails to be higher, then the input question enhancement mechanism leaves the score of the existing answer's as is (step524).

From steps518,522, and524, the input question enhancement mechanism then determines whether there is another answer to process (step526). If at step526the input question enhancement mechanism determines that there is another answer to process, the operation returns to step514. If at step526the input question enhancement mechanism determines that there is not another answer to process, the input question enhancement mechanism sorts the answer data structure by score (step528). The input question enhancement mechanism then generates a factoid question response report with answers from the answer data structure and passages from the passage data structure (step530). The input question enhancement mechanism then provides the factoid question response report to be utilized as a final set of candidate answers and confidence scores (step532), which are output to the submitter of the original input question via a graphical user interface or other mechanism for outputting information. The operation terminates thereafter.

Thus, the illustrative embodiments provide mechanisms for employing a set of sub-questions that are identified as being related to the input question. The QA system then operates to answer each sub-question separately and merge the results, i.e. the answer to each sub-question into a single report so the most relevant information is easily accessible. By performing such operations, the illustrative embodiments improve the answering capabilities of QA systems such that differently phrased reference documents rich with pragmatic implications from the corpus of reference documents provide an accurate answer to the input question.