Cognitive question answering pipeline blending

An answer to a question may selected from answers from a set of answering pipelines. Question answer data can be generated for a question, using a first answering pipeline. Another set of question answer data can be generated for the second question, using the second answering pipeline. The question answer data can include answers and confidence values for each answer. Using a weighting formula and a blending profile for the first answering pipeline, a vote weight can be determined for an answer with the highest confidence value. The same weighting formula and a second blending profile may be used to determine a vote weight for another answer with the highest confidence value. An answer to the question may be selected from the answers, based on the overall highest vote weight.

BACKGROUND

The present relates to computer systems, and more specifically, to question answering (QA) systems.

Recent research has been directed to developing question answering (QA) systems designed to receive input questions, analyze them, and return applicable answers. These systems may rely on natural language processing, automated reasoning, machine learning, and other advanced techniques. Using these techniques, QA systems may provide mechanisms for searching large sources of content and analyzing the content with regard to a given input question in order to determine an answer to the question. In some QA systems this may take the form of hypothesis generation, scoring, and ranking in order to determine a final set of one or more output answers. An example of a QA system is IBM's WATSON system.

SUMMARY

Embodiments of the present disclosure may be directed toward a method that begins when a first set of question answer data is generated. The data may be generated using a first answering pipeline, and the question answer data may include a first set of answers and a set of first pipeline confidence values for each answer. A second set of question answer data for the question may be generated using a second answering pipeline. The second set of question answer data may include a second set of answers and a set of second pipeline confidence values for each answer in the second set of answers. Using a weighting formula and a first blending profile, a first vote weight for an answer in the first set of question answer data that was assigned a first pipeline highest confidence value by the first answering pipeline. Using the same weighting formula and a second blending profile, a second vote weight for an answer in the second set of question answer data that was assigned a second pipeline highest confidence value by the second answer pipeline may be determined. The first and second blending profiles may be associated with the first and second answering pipelines, respectively. The answer with an overall highest vote weight may then be selected from among the answers in the first set of question answer data and in the second set of question answer data. The answer selected may be selected as the first answer to the question.

Embodiments of the present disclosure may be directed toward a computer readable storage medium with program instructions stored thereon, and one or more processors configured to execute the program instructions to perform a method. The method may begin when a first set of question answer data is generated. The data may be generated using a first answering pipeline, and the question answer data may include a first set of answers and a set of first pipeline confidence values for each answer. A second set of question answer data for the question may be generated using a second answering pipeline. The second set of question answer data may include a second set of answers and a set of second pipeline confidence values for each answer in the second set of answers. Using a weighting formula and a first blending profile, a first vote weight for an answer in the first set of question answer data that was assigned a first pipeline highest confidence value by the first answering pipeline. Using the same weighting formula and a second blending profile, a second vote weight for an answer in the second set of question answer data that was assigned a second pipeline highest confidence value by the second answer pipeline may be determined. The first and second blending profiles may be associated with the first and second answering pipelines, respectively. The answer with an overall highest vote weight may then be selected from among the answers in the first set of question answer data and in the second set of question answer data. The answer selected may be selected as the first answer to the question.

Embodiments of the present disclosure may be directed toward a computer program product with a computer readable storage medium, where the computer readable storage medium may have program instructions embodied therewith, and it is not a transitory signal per se. The program instructions may be executed by a computer processing circuit to cause the circuit to perform a method. The method may begin when a first set of question answer data is generated. The data may be generated using a first answering pipeline, and the question answer data may include a first set of answers and a set of first pipeline confidence values for each answer. A second set of question answer data for the question may be generated using a second answering pipeline. The second set of question answer data may include a second set of answers and a set of second pipeline confidence values for each answer in the second set of answers. Using a weighting formula and a first blending profile, a first vote weight for an answer in the first set of question answer data that was assigned a first pipeline highest confidence value by the first answering pipeline. Using the same weighting formula and a second blending profile, a second vote weight for an answer in the second set of question answer data that was assigned a second pipeline highest confidence value by the second answer pipeline may be determined. The first and second blending profiles may be associated with the first and second answering pipelines, respectively. The answer with an overall highest vote weight may then be selected from among the answers in the first set of question answer data and in the second set of question answer data. The answer selected may be selected as the first answer to the question.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to computer systems, more particular aspects relate to question answering (QA) systems. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.

In cognitive computing, a monolithic question and answer (QA) system containing multiple answering pipelines may be decomposed into separate cognitive services. For example, answering pipelines for different types of questions may be created as separate cognitive services, in order to facilitate reuse in different combinations within different cognitive solutions. Additionally, answering pipelines for the same type of questions may be created by separate development teams and operate over separate data sources. The combination of the pipeline capabilities may exceed the capabilities of each individual pipeline. In some cases, a cognitive solution developer may be expected to utilize a set of answering pipeline cognitive services in order to develop a situation-specific solution (e.g., for a particular client).

In some instances, a cognitive solution developer may use a question classifier service. A question classifier service may be a system trained to distinguish questions of various types. For example, a QA system may be trained to answer both questions that require a factoid answer (e.g., “What color is the sky?”) as well as questions that require a descriptive passage answer (e.g., “Why is the sky blue?”). In using a question classifier, the QA system may first filter incoming questions through the question classifier, and, based on a resultant classification, invoke the appropriate answering pipeline cognitive service.

At times, the question classifier may not be able to achieve the desired accuracy in distinguishing the intent of the question, due to, for example, linguistic ambiguity or the brevity of a question. Thus, in many cases, the question classifier is unable to return a specific class for the question type, but rather it may return a list of likely classes, with confidence values attached to each class. The confidence values may indicate the particular confidence of the question classifier that the question is of the indicated class. In some cases, the cognitive solution developer may simply choose to invoke an answering pipeline associated with the highest confidence question class, but this may be less accurate than desired because the selected answering pipeline may not produce the best answers.

The cognitive solution developer may also experience further challenge when faced with multiple answering pipeline cognitive services that are intended to answer the same types of questions, but they may have different data sources or may have been trained differently. In some cases, the developer may simply choose the answer or answers from the answering pipeline that has the highest confidence top answer or answers, however this may also be less accurate than desired because the selected answering pipeline may not produce the best answers.

Finally, the cognitive solution developer may run each of the applicable answering pipelines individually, receive the list of the highest confidence answers from each, and merge the lists into one list, that may be sorted by, for example, the highest confidence value assigned to each (by the different answering pipelines). However, the confidence values from the different answering pipelines may be incomparable types due to being generated by distinct cognitive models, and thus using them interchangeably may result in inaccuracies.

In embodiments, the initial “conversion” may occur through a calibration of the different answering pipelines, using a probabilistic weighted voting mechanism. A system may first receive a set of test questions to be processed by each answering pipeline, in order to generate a set of answer data for each question. The set of test questions may be processed by a first answering pipeline to obtain a set of answers, and their confidence values, for each test question. The processing of each test question may generate a set of answers and compare them with an answer key entry associated with the test question to ascertain which of the answers are correct for the test question. Using this first set of answers, along with their confidence and correctness values, the system may generate a set of metadata for the first answering pipeline to reflect various levels of accuracy and confidence probabilities.

In embodiments, the same set of test questions may be processed by a second answering pipeline. A second set of answer data, including the set of answers as well as their confidence and correctness values, may be generated. Using this second set of answer data, the system may generate another set of metadata for the second answering pipeline to reflect various levels of accuracy and confidence probabilities. The metadata from the first and second pipelines may be stored in profiles, referred to herein as a “first blending profile” and a “second blending profile”, respectively.

In embodiments, a system may receive a question for processing by a plurality of question answering pipelines. In response, the system may generate a first set of question answer data using the first pipeline. The question answer data may include a set of answers and a set of confidence values. These confidence values (“first pipeline confidence values”) may be values assigned to each answer that indicate the pipeline's confidence in the accuracy of the answer. The system may also generate a set of question answer data for the question using the second answering pipeline. Like the first set of question answer data, the second set of question answer data may include a second set of answers as well as a set of second pipeline confidence values for each answer in the second set of answers. In embodiments, the system may then transmit these blending profiles to a user for use in answering a question. This user may be a cognitive solution developer, a user client device, an external system, or another user. In embodiments, the system may then store the profiles, transmit them to an external system, or communicate them to the same or another system, as is appropriate for the various QA systems.

Using the first blending profile and a weighting formula, the system may determine a vote weight for each answer in the set of question answer data. Similarly, using the second blending profile and the same weighting formula, the system may determine a vote weight for each answer in the second set of question answer data. The system can then compare the vote weights assigned to each answer (from each of the pipelines), and select, from the combined sets, an answer with the highest vote weight. This answer can be considered the best or highest confidence answer to the question. In embodiments, the system can return a single answer with a vote weight attached, top answers with vote weights attached, or another result as determined by user configured settings, a system administrator, or in another way.

As discussed above, aspects of the disclosure may relate to QA systems. Accordingly, an understanding of the embodiments of the present invention may be aided by describing embodiments of these QA systems and the environments in which these systems may operate.FIGS. 1-3depict example natural language processing systems, for example QA systems, which may be integrated with, expanded upon, or otherwise modified, according to embodiments of the present disclosure.

FIG. 1depicts a block diagram of an example computing environment100for use with a question answering system, according to embodiments of the present disclosure. In some embodiments, the computing environment100may include one or more remote devices102,112and one or more host devices122. Remote devices102,112and host device122may be distant from each other and communicate over a network150in which the host device122comprises a central hub from which remote devices102,112can establish a communication connection. Alternatively, the host device and remote devices may be configured in any other suitable relationship (e.g., in a peer-to-peer or other relationship).

In some embodiments, the network150can be implemented by any number of any suitable communications media (e.g., wide area network (WAN), local area network (LAN), Internet, Intranet, etc.). Alternatively, remote devices102,112and host devices122may be local to each other, and communicate via any appropriate local communication medium (e.g., local area network (LAN), hardwire, wireless link, Intranet, etc.). In some embodiments, the network150can be implemented within a cloud computing environment, or using one or more cloud computing services. Consistent with various embodiments, a cloud computing environment may include a network-based, distributed data processing system that provides one or more cloud computing services. Further, a cloud computing environment may include many computers, hundreds or thousands of them or more, disposed within one or more data centers and configured to share resources over the network150.

In some embodiments, host device122may include a question answering system130having a search module134and an answer module132. The search module may be implemented by a conventional or other search engine, and may be distributed across multiple computer systems. The search module134may be configured to search one or more databases or other computer systems for content that is related to a question input by a user at a remote device102,112.

In some embodiments, remote devices102,112may enable users to submit questions (e.g., search requests or other user queries) to host devices122to retrieve search results. For example, the remote devices102,112may include a query module110,120(e.g., in the form of a web browser or any other suitable software module) and present a graphical user interface or other interface (e.g., command line prompts, menu screens, etc.) to solicit queries from users for submission to one or more host devices122and to display answers/results obtained from the host devices122in relation to such user queries.

Consistent with various embodiments, host device122and remote devices102,112may be computer systems, and may each be equipped with a display or monitor. The computer systems may include at least one processor106,116,126; memories108,118,128; internal or external network interface or communications devices104,114,124(e.g., modem, network cards, etc.); optional input devices (e.g., a keyboard, mouse, or other input device); and any commercially available or custom software (e.g., browser software, communications software, server software, natural language processing software, search engine and/or web crawling software, filter modules for filtering content based upon predefined criteria, etc.). In some embodiments, the computer systems may include servers, desktops, laptops, and hand-held devices. In addition, the answer module132may include one or more modules or units to perform the various functions of embodiments as described below (e.g., receiving an input question, assigning the input question to a question category, determining a set of candidate answers, comparing confidence scores and user feedback to confidence criteria, etc.), and may be implemented by any combination of any quantity of software and/or hardware modules or units.

FIG. 2depicts a block diagram of an example question answering system usable to generate answers to one or more input questions, according to embodiments of the present disclosure. Aspects ofFIG. 2are directed toward an exemplary system architecture200, including a question answering system212to generate answers to user queries (e.g., input questions). In some embodiments, one or more users can send requests for information to QA system212using a remote device (such as remote devices102,112ofFIG. 1). Such a remote device may include a client application208which may itself involve one or more entities operable to generate information that is then dispatched to QA system212via network215. QA system212may be able to perform methods and techniques for responding to the requests sent by the client application208. In some embodiments, the information received at QA system212may correspond to input questions received from users, where the input questions may be expressed in a free form and in natural language. In embodiments, and as described herein, multiple QA systems like QA system212may exist, the answers from which may be blended as described herein.

A question (similarly referred to herein as a user query) may be one or more words that form a search term or request for data, information, or knowledge. A question may be expressed in the form of one or more keywords. Questions may include various selection criteria and search terms. A question may be composed of complex linguistic features in addition to keywords. However, a keyword-based search for answers may also be possible. In some embodiments, using restricted syntax for questions posed by users may be enabled. The use of restricted syntax may result in a variety of alternative expressions that assist users in better stating their needs.

Consistent with various embodiments, client application208may operate on a variety of devices. Such devices may include, but are not limited to, mobile and handheld devices (e.g., laptops, mobile phones, personal or enterprise digital assistants, and the like), personal computers, servers, or other computer systems that access the services and functionality provided by QA system212. In some embodiments, client application208may include one or more components, such as a mobile client210. Mobile client210, acting as an agent of client application210, may dispatch user query requests to QA system212.

Consistent with various embodiments, client application208may also include a search module202, either as part of mobile client210or separately, that may perform several functions, including some or all of the above functions of mobile client210listed above. For example, in some embodiments, search module202may dispatch requests for information to QA system212. In some embodiments, search module202may be a client application to QA system212. Search module202may send requests for answers to QA system212. Search module202may be installed on a personal computer, a server, or other computer system.

In some embodiments, search module202may include a search graphical user interface (GUI)204and session manager206. In such situations, users may be able to enter questions in search GUI204. In some embodiments, search GUI204may be a search box or other GUI component, the content of which can represent a question to be submitted to QA system212. Users may authenticate to QA system212via session manager206. In some embodiments, session manager206may keep track of user activity across sessions of interaction with the QA system212. Session manager206may also keep track of what questions are submitted within the lifecycle of a session of a user. For example, session manager206may retain a succession of questions posed by a user during a session. In some embodiments, answers produced by QA system212in response to questions posed throughout the course of a user session may also be retained. Information for sessions managed by session manager206may be shared between computer systems and devices.

In some embodiments, client applications208and QA system212may be communicatively coupled through network215, e.g., the Internet, intranet, or other public or private computer network. In some embodiments, QA system212and client application208may communicate by using Hypertext Transfer Protocol (HTTP) or Representational State Transfer (REST) calls. In some embodiments, QA system212may reside on a server node. Client application208may establish server-client communication with QA system212or vice versa. In some embodiments, the network215can be implemented within a cloud computing environment, or using one or more cloud computing services.

Consistent with various embodiments, QA system212may respond to the requests for information sent by client applications208(e.g., questions posed by users). QA system212may generate answers to the received questions. In some embodiments, QA system212may include a question analyzer214, data sources224, and answer generator228. Question analyzer214may be a computer module that analyzes the received questions. Question analyzer214may perform various methods and techniques for analyzing the questions syntactically and semantically. In some embodiments, question analyzer214can parse received questions. Question analyzer214may include various modules to perform analyses of received questions. For example, computer modules that question analyzer214may encompass include, but are not limited to, a tokenizer216, part-of-speech (POS) tagger218, semantic relationship identifier220, and syntactic relationship identifier222. In embodiments, question analyzer214may include a question classifier, as described herein, in order to identify a type or class to which the question belongs.

Consistent with various embodiments, tokenizer216may be a computer module that performs lexical analysis. Tokenizer216can convert a sequence of characters into a sequence of tokens. A token may be a string of characters typed by a user and categorized as a meaningful symbol. Further, in some embodiments, tokenizer316can identify word boundaries in an input question and break the question or any text into its component parts such as words, multiword tokens, numbers, and punctuation marks. In some embodiments, tokenizer216can receive a string of characters, identify the lexemes in the string, and categorize them into tokens.

Consistent with various embodiments, POS tagger218may be a computer module that marks up a word in a text to correspond to a particular part of speech. POS tagger218can read a question or other text in natural language and assign a part of speech to each word or other token. POS tagger218can determine the part of speech to which a word corresponds based on the definition of the word and the context of the word. The context of a word may be based on its relationship with adjacent and related words in a phrase, sentence, question, or paragraph. In some embodiments, the context of a word may be dependent on one or more previously posed questions. Examples of parts of speech that may be assigned to words include, but are not limited to, nouns, verbs, adjectives, adverbs, and the like. Examples of other part of speech categories that POS tagger218may assign include, but are not limited to, comparative or superlative adverbs, wh-adverbs, conjunctions, determiners, negative particles, possessive markers, prepositions, wh-pronouns, and the like. In some embodiments, POS tagger218may tag or otherwise annotate tokens of a question with part of speech categories. In some embodiments, POS tagger218may tag tokens or words of a question to be parsed by QA system212.

Consistent with various embodiments, semantic relationship identifier220may be a computer module that can identify semantic relationships of recognized entities (e.g., words, phrases, etc.) in questions posed by users. In some embodiments, semantic relationship identifier220may determine functional dependencies between entities and other semantic relationships.

Consistent with various embodiments, syntactic relationship identifier222may be a computer module that can identify syntactic relationships in a question composed of tokens posed by users to QA system212. Syntactic relationship identifier222can determine the grammatical structure of sentences, for example, which groups of words are associated as “phrases” and which word is the subject or object of a verb. Syntactic relationship identifier222may conform to formal grammar.

In some embodiments, question analyzer214may be a computer module that can parse a received user query and generate a corresponding data structure of the user query. For example, in response to receiving a question at QA system212, question analyzer214may output the parsed question as a data structure. In some embodiments, the parsed question may be represented in the form of a parse tree or other graph structure. To generate the parsed question, question analyzer214may trigger computer modules216-222. Additionally, in some embodiments, question analyzer214may use external computer systems for dedicated tasks that are part of the question parsing process.

Consistent with various embodiments, the output of question analyzer214may be used by QA system212to perform a search of one or more data sources224to retrieve information to answer a question posed by a user. In some embodiments, data sources224may include data warehouses, information corpora, data models, and document repositories. In some embodiments, the data source224may include an information corpus226. The information corpus226may enable data storage and retrieval. In some embodiments, the information corpus226may be a storage mechanism that houses a standardized, consistent, clean and integrated form of data. The data may be sourced from various operational systems. Data stored in the information corpus226may be structured in a way to specifically address reporting and analytic requirements. In some embodiments, the information corpus may be a relational database. In some example embodiments, data sources224may include one or more document repositories.

In some embodiments, answer generator228may be a computer module that generates answers to posed questions. Examples of answers generated by answer generator228may include, but are not limited to, answers in the form of natural language sentences; reports, charts, or other analytic representation; raw data; web pages; and the like.

Consistent with various embodiments, answer generator228may include query processor230, visualization processor232, and feedback handler234. When information in a data source224matching a parsed question is located, a technical query associated with the pattern can be executed by query processor230. Based on data retrieved by a technical query executed by query processor230, visualization processor232may be able to render visualization of the retrieved data, where the visualization represents the answer. In some embodiments, visualization processor232may render various analytics to represent the answer including, but not limited to, images, charts, tables, dashboards, maps, and the like. In some embodiments, visualization processor232may present the answer to the user.

In some embodiments, feedback handler234may be a computer module that processes feedback from users on answers generated by answer generator228. In some embodiments, users may be engaged in dialog with the QA system212to evaluate the relevance of received answers. Answer generator228may produce a list of answers (e.g., candidate answers) corresponding to a question submitted by a user. The user may rank each answer according to its relevance to the question. In some embodiments, the feedback of users on generated answers may be used for future question answering sessions.

In embodiments, a set of test questions may be used with the QA system112, and the generated answers may be considered the set of test question data. Each QA system in a set of QA systems may be calibrated as described herein in order to allow for a relevant comparison between the generated and scored answers from each of their respective answer generators228.

The various components of the exemplary question answering system described above may be used to implement various aspects of the present disclosure. For example, the client application208could be used to receive an input question from a user. The question analyzer214could, in some embodiments, be used to analyze the input question to determine to which question category the input question should be assigned. Further, the query processor230or the answer generator228could, in some embodiments, be used to determine a set of candidate answers and calculate confidence scores for the candidate answers.

FIG. 3depicts a block diagram of an example high level logical architecture of a QA system, consistent with embodiments of the present disclosure. Aspects ofFIG. 3may be directed toward components and modules for use with a QA system300. In some embodiments, host device301and remote device302may be embodied by host device122and remote device102ofFIG. 1, respectively. In some embodiments, the question analysis module304, located on host device301, may receive a natural language question (e.g., an input question) from a remote device302, and can analyze the question to produce information about the question based on the question's content and context. This may be accomplished, for example, by using components216-222ofFIG. 2. The information produced by question analysis module304may include, for example, the semantic type of the expected answer. In addition the question analysis module304may assign a question category to the input question and provide this information to the information source quality control module314. As used herein question categories may refer to any suitable groupings of input questions wherein a determination as to the appropriate category for a given question is made at least in part based on an analysis of the content of the question itself. In some embodiments, a single given question may be included in multiple question categories.

Next, the candidate generation module306may formulate queries from the output of the question analysis module304and then pass these queries on to search module308which may consult various resources such as the internet or one or more knowledge resources, e.g., databases or corpora, to retrieve documents that are relevant to answering the user question. As used herein, documents may refer to various types of written, printed, or electronic matter (including passages, web-pages, database tuples, etc.) that provide information or evidence. As shown inFIG. 3, the search module308may consult core information source310. As used herein, a core information source may refer to any document or group of documents that is used by a relevant QA system to identify candidate answers to user questions. The candidate generation module306may extract, from the search results obtained by search module308, potential (candidate) answers to the question, which it may then score (e.g., with confidence scores) and rank. A final set of candidate answers, based on a comparison of various confidence scores associated with the candidate answers, may then be sent from the candidate generation module306to remote device302for presentation to the user. In addition, this information about candidate answers and confidence scores may also be sent to information source quality control module314. A user may respond, via remote device302, to the candidate answers (for example, by indicating that none of the provided candidate answers are accurate or relevant) through user feedback module312. The user feedback module312may then provide this feedback to the information source quality control module314.

In some embodiments, the information source quality control module314may compile and analyze information that it receives during the course of normal operations of question and answering system300. This received information (e.g., information from question analysis module304, candidate generation module306, and user feedback module312) may be usable by the information source quality control module314to determine whether one or more new information sources should be ingested. When the information source quality control module314determines that a new information source having certain characteristics is needed (e.g., an information source that is associated with a specific question category), it may instruct an ingestion module316accordingly. Based on these instructions, ingestion module316may search one or more remote sources, such as remote corpora318, in an attempt to locate one or more suitable new information sources. In some embodiments, once discovered, these new information sources may be ingested by ingestion module316and become newly ingested information source320. This information source may in turn be analyzed by training module322. This training analysis may take the form of obtaining training candidate answers to training questions using the newly ingested information source320and then reviewing the quality of these training answers. As used herein, training questions may refer to predetermined questions that are used by a QA system for either (1) reviewing or determining the quality or characteristics of an information source used to identify training candidate answers to these questions, (2) creating or refining machine learning models and routing paths usable by the QA system, or both. In some embodiments, once a threshold level of confidence in the new information source is met, it may be combined with core information source310and used to answer new input questions as they are received from users.

The various components and modules of the exemplary high level logical architecture for a QA system described above may be used to implement various aspects of the present disclosure. For example, the question analysis module304may, in some embodiments, be used to obtain input questions and assign these input questions to appropriate question categories. Further, the candidate generation module306and search module308may together, in some embodiments, be used to perform searches of core information source310, generate candidate answers, calculate confidence scores associated with these candidate answer, and provide these candidate answers to one or more users. Further, the information source quality control module314may, in some embodiments, be used to analyze confidence scores and determine whether the confidence scores fail to meet one or more confidence criteria. Further, ingestion module316may, in some embodiments, be used to ingest new information sources (in response to an indication from the information source quality control module314that a confidence criterion has not been satisfied).

FIG. 4depicts a diagram of an illustrative embodiment of a system400for generating metadata for a set of QA pipelines, according to embodiments of the present disclosure. The system400may be executed over a computer processor system, or over a series of processors including those connected locally, on the cloud, or in another way. A set of test questions402may be received by the system for preprocessing404. In embodiments, the test questions402may be a set of one or more test questions generated by the system, by a subject matter expert, or in another way. Each question in the set of test questions402may have an associated entry in an answer key, which indicates the correct or most correct answers to the particular question.

As part of preprocessing404, the system may send the questions, per406, to each of the answering pipelines412a,412b, and412c. In some embodiments, a user or other process within the system may assign the question to a particular answering pipeline or set of answering pipelines. In other embodiments, the test question may be sent without previous sorting, to each of the answering pipelines412. The question may be processed by each answering pipeline412, and the answering pipeline412can return a set of answer data. This answer data can include a set of answers to the question as well as a confidence value for each answer. The confidence value may indicate the answer pipeline's estimate of the likelihood that the particular answer is a suitable answer to the test question.

The system can then determine an answer rating408for each answer, based on the returned set of test answer data from each answering pipeline412. In embodiments, the answer rating can be a determination of whether or not the answer is considered correct, based on a comparison of the returned answer with an answer key entry. The content of the answer key may have previously been created by a subject matter expert. In embodiments, the same answer key can be used for the accuracy determination of each pipeline.

Using the confidence values and answer ratings of the answers, the system can then generate metadata410to create a blending profile414a-cfor each of the answering pipelines412. The metadata calculated for each profile414can include an “answer accuracy value”, an “answer confidence table”, and a “correct answer confidence table”.

The first metadata value to be generated for the blending profile for a particular answering pipelines (e.g., answering pipeline412a), may be the answer accuracy value. The answer accuracy value may indicate the probability that the particular answering pipeline (for example, answering pipelines412a) will produce the correct answer to a question it receives.

The second metadata value set to be generated for the blending profile for a particular answering pipelines (e.g., answering pipeline412a), may be the answer confidence table. The answer confidence table may comprise values that indicate a probability that the answering pipelines (e.g., answering pipeline412a) will produce an answer that has a confidence of at least a particular confidence value.

The third metadata value set to be generated for the blending profile for a particular answering pipeline (e.g., answering pipeline412a) may be the correct answer confidence table. This table can indicate the probability that the answering pipeline will produce a correct answer that has a confidence of at least a particular confidence value.

Metadata can be generated for each answering pipeline412a,412b, and412c, in order to create a blending profile for each pipelines,414a,414b, and414c, respectively. The blending profiles for each profile can then be transmitted to a user, to another system, to a cognitive solution developer, or to a system administrator for use in answering non-test questions within a QA system or set of QA systems. In embodiments, blending profile can be created for any number of answering pipelines, in order to facilitate for the comparison of resultant answers from each.

The following example may be carried out over the system400as described inFIG. 4. In an embodiment, let “N” denote the number of test set questions for a given answering pipeline, and let “K” denote an answering pipeline parameter that indicates the maximum number of answers the pipeline is configured to return in response to any question. A cognitive solution developer or other user may be expected to execute the answering pipeline on the N questions to obtain “T” answers. The value of T is at most N*K, but it may be less if the answering pipeline returns fewer than K answers for any of the questions.

The answering pipeline may also provide a confidence value “C” for each of the T answers. The value space of C is the range 0 to 1, and the value assigned to C indicates the answer pipeline's estimate of the likelihood that the answer is a suitable answer to the test question.

The cognitive solution developer or other user may then engage the service of a subject matter expert (SME). The SME may rate the degree of suitability of each answer for each test question, and thus further refine and validate the results of the particular answering pipeline. In other embodiments, a prepared answer key may be used. The answer key can be the same for all pipelines used, and contain an answer or set of answers considered to be correct.

The system can then calculate certain metadata for each answering pipeline, prior to the use of the pipeline in [non-test] question answering. To assist in calculating the metadata, an internal value referred to herein as the correct answer count or “CAC” may be determined. In embodiments, the CAC may be determined by first reducing the value space of the answering ratings, based on a configurable threshold, to a simple binary scale with “1” indicating a correct answer and “0” indicating an incorrect answer. Thus, the CAC may be the count of how many of the T answers have a 1 rating (i.e., how many of the returned answers are correct).

The first metadata value calculated, the answer accuracy value, as described herein, may indicate the probability that the answering pipeline will produce a correct answer to a question. The answer accuracy value, or “AVV”, may be calculated as the ratio CAC/T (i.e., correct answers CAC over total answer T), or the number of the T answers with a “1” rating divided by the number of total number of answers T.

The second metadata value set to be generated, the answer confidence table, as described herein, may indicate the probability that the answering pipeline will produce an answer that has a confidence of a certain confidence value. The answer confidence table, or “ACT”, may be computed using a configurable step value. In embodiments, this value can default to a step value of .01. For each confidence value, or “CV”, from 0 to 1, according to the step value, a number of correct answers, or “NCA”, can be calculated to reflect the number of the T answers that have a confidence value C of at least CV. The ACT location CV may then be associated with NCA/T, denoted ACT[CV]=NCA/T. Thus, each value ACT[CV] (in the ACT) may indicate the probability that the answering pipeline will produce an answer that has a confidence of at least CV.

The third metadata value set to be generated, the correct answer confidence table, as described herein, may contain a set of values which each indicate that the answering pipeline will produce an answer that has a confidence of at least a particular confidence value, when the answering pipeline produces a correct answer. The correct answer confidence table, or “CACT”, may be computed using a configurable step value. In embodiments, this value can default to a step value of .01. For each confidence value CV from 0 to 1 according to the step value, the invention may calculate a number, “NCCA” of the T answers that have a 1 rating and a confidence value C or at least CV. Then, the ACT location CV may be associated with NCCA/CAC, denoted ACT[CV]=NCCA/CAC. Thus, each value CACT[CV] may indicate the probability that the answering pipeline will produce an answer that has a confidence of at least CV when the answering pipeline produces a correct answer.

FIG. 5depicts a flow diagram of a method500for calibrating a set of QA pipelines, according to embodiments of the present disclosure. In embodiments, this calibration may occur via the creation of metadata for blending profiles, as described inFIG. 4. In embodiments, the method500may begin when a first set of test answer data is generated for a first answering pipeline, per501. The test answer data may be the output of a QA system, for example, the QA system described inFIG. 3, and may comprise a set of answers to a processed question as well as one or more confidence values associated with the particular answer. The answer data may include other data or metadata, as is relevant to the particular QA system processing the question and generating the test answer data.

The system may generate a first blending profile using the first set of test answer data, per502. The blending profile may include metadata generated by the system using the first set of test answer data, including the answers and the confidence value. In the generation of the first blending profile, the system may compare, with an answer key, the answers in the test answer data.

A second set of test answer data may be generated by use of the second answering pipeline, per503. Using the set of test answer data for the second answering pipeline, the system may generate a second blending profile, per504. The second blending profile may comprise metadata values including, for example, those described inFIG. 4, such as the answer accuracy value, an answer confidence table, and the correct answer confidence table.

Test answer data and blending profiles may be generated for a number of answering pipelines, as is appropriate to the particular problem, solution, or QA environment. The blending profiles may then be stored or transmitted for use in answering a question or set of non-test questions.

FIG. 6depicts a diagram of an illustrative embodiment of a system600for blending answers from a set of answering pipelines, according to embodiments of the present disclosure. In embodiments, a question602may be provided to the answering pipelines604for processing. The result of the processing may be a set of question answer data, including, for example an answer or set of answers (“A”) and a confidence value for each answer (“C”). The system may obtain blending profiles606for each answering pipeline. In embodiments, the elements described inFIG. 6including the answering pipelines604and the blending profiles606may be analogous to those described inFIG. 4, including for example, the answering pipelines412and blending profiles414. The question602may be processed in a similar manner as a question from the set of test questions402of system400inFIG. 4. However, the question inFIG. 6may be an actual (i.e., non-test) question submitted by a user, and the blending profiles can comprise the metadata created based on the processes described inFIGS. 4 and 5.

In embodiments, the “blender”608may be a computer system for blending the answers from numerous pipelines604. The blender608may receive the question answer data from each of the pipelines604, along with the blending profiles606(e.g., blending profiles created via generation of metadata value sets as described inFIGS. 4 and 5). The blender608may then calculate a vote weight for each answer from each of the pipelines610, using the metadata in the blending profile606. In one embodiment, blender608is configured to only retain from each answering pipeline the J answers with the highest confidence value, for some configurable value J.

In some embodiments, the vote weight can be calculated for each answer. A vote weight of each answer may represent the probability that it is a correct answer given that the answering pipeline assigned it a confidence of at least C′. The vote weight may be calculated in some embodiments using the formula: CACT[C′]*AVV/ACT[C′], as depicted at610a. Each of these values may be determined using the metadata calculated for each answering pipeline blending profile. As described herein, CACT[C′] may be a particular confidence “C′” location on the correct answer confidence table. AVV may be the answer accuracy value, calculated as described herein; and ACT[C] may be a particular location on the answer confidence table keyed using the particular confidence “C′”. In embodiments, C′ may be the answer's confidence (as provided by the answering pipelines604) truncated to the nearest confidence step value. For example, if the answering pipeline provided answer confidence C was 0.926 and the step value was .01, then C′ could be 0.92. The generated metadata enables the formula to calculate an answer's vote weight based on the probability that its confidence is at least C′ (assuming the answer is correct), multiplied by the probability of the answer being correct, and then divided by the probability that its confidence is at least C′.

In embodiments, the blender608may then select a top answer or list of top answers based on the calculated vote weights,612. In one embodiment, blender608may combine the answers from all the answering pipelines by sorting the answers from greatest to least vote weight and the selecting the top “K” answers (where “K” is based on, for example, a configurable setting) from the beginning of the sorted list. In another embodiment, blender608selects the answer with the highest vote weight from all answer pipeline lists, adds it to the output list, and then removes it from the answer pipeline list. This sequence may be iterated “K” times to select the top “K” answers. The blender608can then output, to a user or other system, the “K” top answers614. The answer may be output by the blender608in the form depicted at616to including the answer “A” and the vote weight “VW”. In this way, the answers output by each of the answering pipelines604may be compared amongst one another, regardless of the processing used by each particular answering pipeline604.

FIG. 7depicts a flow diagram of a method for blending answers from a set of QA pipelines, according to embodiments of the present disclosure. At701, a system may generate a first set of question answer data using a first answering pipeline. This question answer data may result from the processing of a question or query by the first answering pipeline. The question answer data may include a set of one or more answers to the question, as determined by the first answering pipeline. The question answer data may also include a confidence value for each answer generated by the first answering pipeline. As described herein, the confidence value may indicate the answering pipeline's estimate of the likelihood that the answer is a suitable answer to the question. The system may then generate a second set of question answer data using a second answering pipeline, per702. This process may mirror that of the generation of the question answer data for the first answering pipeline, as described at701. In embodiments, the process of question answer data generation may be repeated for each pipeline, as determined by the number of different pipelines being utilized by the QA blending system.

The QA blending system can then determine a vote weight for the answers in the first set of question answer data, per703. The system can determine the vote weight using a weighting formula, for example the weighting formula described inFIG. 6, and the first blending profile. The system can then determine a vote weight for each of the answers in the second set of question answer data, using the same weighting formula as used for the first set of question answer data and the second blending profile, per704. The system can then select an answer or set of answers with the highest vote weight or highest vote weights from among all the answer data, per705. This answer or set of answers can be presented or returned to a user, per706. In embodiments, the answer or set of answers can each be presented with its corresponding vote weight or their corresponding vote weights.

In embodiments, a question classifier may be used in conjunction with the QA blending system described herein. The question classifier may be integrated with the system's overall processing of the question, in order to provide a more precise confidence value based on a type or class of question.

In embodiments, the question classifier may be integrated with the QA system by generating a question classifier confidence value for each question in the set of test questions for each pipeline. For example, question classifier confidence values A1, B1, and C1, may be calculated for each question A, B, and C in the set of test questions, in relation to the first answering pipeline. For each question, the question classifier confidence value can be combined with the confidence value of each answer from the pipeline, such as by multiplying them, to produce a revised confidence value for the answer in the test answer data for the pipeline. This process can be repeated for the second answering pipeline (and any additional answering pipelines). In this way, the revised confidence value for each answer takes into account the type of question being answered and the extent to which the QA system's question classifier has been trained to prefer the answers from a particular answering pipeline for a particular question type.

In other embodiments, the system may generate a first question classifier confidence value for the question submitted to the system. The question classifier could determine a confidence that the question is associated with one or more question types. In embodiments, each question type could be associated with a distinct answering pipeline. Thus, the question classifier confidence value produced by the question classifier could indicate a determined level of appropriateness that the particular answering pipeline is the correct pipelines to be used in answering the question. In embodiments, the data contained within the question answer data set may be modified based on the use of a question classifier.

For example, the first answering pipeline may generate a set of question answer data, in response to receiving a question. The question classifier confidence value may be determined for the question, as described above. The question classifier confidence value may then be combined with the confidence value in the question answer data generated by the first answering pipeline, such as by multiplying them, to produce a revised confidence value in the question answer data. Similarly, the second answering pipeline may combine a generated question classifier confidence value with the standard confidence value. This new value (question classifier confidence value combined with second answering pipeline answering confidence value) may replace the confidence value in the generation of the question answer data set. In this way, the question classifier may impact the determined confidence of an answer, when used with the disclosed QA system blending.

In other embodiments, the vote weight as described, for example, atFIG. 6, may be impacted by the question classifier. As noted, the vote weight for a particular answer in a particular pipeline may be calculated by multiplying the CACT probability/ACT probability ratio by the particular AAV. When using a question classifier, the product of the aforementioned may be further multiplied by the confidence value obtained from the question classifier for the particular question type associated with a particular answering pipeline. This process may be repeated for each of the pipelines, in order to impact the vote weights of each of the answers from each of the pipelines.

In some embodiments, a weighting formula is used to determine a vote weight of an answer in a set of question answer data that was assigned the highest confidence value by a first answering pipeline by: obtaining a first CACT probability by searching a first CACT for a CACT probability, the CACT probability associated with a greatest confidence value that does not exceed the confidence value of the answer, the confidence value of the answer assigned by the first answering pipeline; obtaining a first ACT probability by searching a first ACT for a ACT probability, the first ACT probability associated with a greatest confidence value that does not exceed the confidence value of the answer; calculating a ratio of the first CACT probability divided by the first ACT probability; calculating a product of the ratio multiplied by an AAV; and assigning the product as the vote weight.

FIG. 10depicts the representative major components of an example computer system1000that may be used, in accordance with embodiments of the present disclosure. It is appreciated that individual components may vary in complexity, number, type, and\or configuration. The particular examples disclosed are for example purposes only and are not necessarily the only such variations. The computer system1000may comprise a processor1010, memory1020, an input/output interface (herein I/O or I/O interface)1030, and a main bus1040. The main bus1040may provide communication pathways for the other components of the computer system1000. In some embodiments, the main bus1040may connect to other components such as a specialized digital signal processor (not depicted).

The processor1010of the computer system1000may be comprised of one or more cores1012A,1012B,1012C,1012D (collectively1012). The processor1010may additionally include one or more memory buffers or caches (not depicted) that provide temporary storage of instructions and data for the cores1012. The cores1012may perform instructions on input provided from the caches or from the memory1020and output the result to caches or the memory. The cores1012may be comprised of one or more circuits configured to perform one or methods consistent with embodiments of the present disclosure. In some embodiments, the computer system1000may contain multiple processors1010. In some embodiments, the computer system1000may be a single processor1010with a singular core1012.

The memory1020of the computer system1001may include a memory controller1022. In some embodiments, the memory1020may comprise a random-access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing data and programs. In embodiments, the blending profiles may be recalled from memory for use by the system in pipeline blending. In some embodiments, the memory may be in the form of modules (e.g., dual in-line memory modules). The memory controller1022may communicate with the processor1010, facilitating storage and retrieval of information in the memory1020. The memory controller1022may communicate with the I/O interface1030, facilitating storage and retrieval of input or output in the memory1020.

The I/O interface1030may comprise an I/O bus1050, a terminal interface1052, a storage interface1054, an I/O device interface1056, and a network interface1058. The I/O interface1030may connect the main bus1040to the I/O bus1050. The I/O interface1030may direct instructions and data from the processor1010and memory1020to the various interfaces of the I/O bus1050. The I/O interface1030may also direct instructions and data from the various interfaces of the I/O bus1050to the processor1010and memory1020. The various interfaces may include the terminal interface1052, the storage interface1054, the I/O device interface1056, and the network interface1058. In some embodiments, the various interfaces may include a subset of the aforementioned interfaces (e.g., an embedded computer system in an industrial application may not include the terminal interface1052and the storage interface1054).

Logic modules throughout the computer system1000—including but not limited to the memory1020, the processor1010, and the I/O interface1030—may communicate failures and changes to one or more components to a hypervisor or operating system (not depicted). The hypervisor or the operating system may allocate the various resources available in the computer system1000and track the location of data in memory1020and of processes assigned to various cores1012. In embodiments that combine or rearrange elements, aspects and capabilities of the logic modules may be combined or redistributed. These variations would be apparent to one skilled in the art.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows: