Exploiting answer key modification history for training a question and answering system

A system and a computer program product are provided for evaluating question-answer pairs in an answer key by generating a predicted answer to a test question based on the answer key modification history for comparison matching against a generated answer that is generated in response to the test question, and then comparing the predicted answer and generated answer to determine an accuracy score match indication therebetween so as to present an indication that the answer key may have a problem if there is a match between the predicted answer and generated answer.

BACKGROUND OF THE INVENTION

In the field of artificially intelligent computer systems capable of answering questions posed in natural language, cognitive question answering (QA) systems (such as the IBM Watson™ artificially intelligent computer system or and other natural language question answering systems) process questions posed in natural language to determine answers and associated confidence scores based on knowledge acquired by the QA system. To train such QA systems, users may generate question-answer (QA) pairs, also known as answer keys, from fact statements for submission to the QA system, but answers and/or questions in the training QA pairs can be incorrect or inaccurate, particularly when new data is ingested into the system corpus, making it difficult to keep the answer key current. The time and resources needed to conduct the fact checking needed to verify and correct training QA pairs is large and also error prone due to the manual operation of the fact checking. Such inaccuracies, if not corrected with cumbersome and time intensive answer correction processing tasks, can result in answer-key problems—such as semantically correct answers missing from the answer key, syntactically correct answers missing from the answer key, and/or inadequate QA pairs—which in turn impair the accuracy of the QA system. At the same time, efforts to keep the answer key current should not impose a penalty for correct answers simply because the answer key is dated. As a result, the existing solutions for efficiently generating and verifying accurate question and/or answer keys for training QA pairs are extremely difficult at a practical level.

SUMMARY

Broadly speaking, selected embodiments of the present disclosure provide a system, method, and apparatus for identifying question set answer key problems by using the cognitive power of the information handling system to exploit historical changes in the question set answer key to identify potential problems and/or updates for the answer key, thereby providing more accurate training data for a question answering (QA) system as it ingests new material and/or assistance with maintaining accurate question set answer keys. To identify potential question set answer key problems, the information handling system generates an answer in response to a test question from an answer key and evaluates the answer for correctness. For any generated answer that is scored as incorrect by the answer key, the information handling system retrieves answers and metadata associated with the answer key over time to generate a predicted answer to the test question. In selected embodiments, the answers and metadata are retrieved by collecting the change set of answer key from all previous answers to the question along with associated metadata to detect a pattern of changes in the answer key along with the associated metadata to compute the predicted answer. In other embodiments, the retrieved answers and metadata may include the previous answers from the answer key along with metadata identifying the author for each QA pair which may be used to identify a trust or reliability measure for each generated answer. However generated, the predicted answer is then compared to the generated answer to generate an accuracy score which may indicate a prediction of likelihood that the generated answer is correct. Based on the comparative analysis, the generated answer, though initially scored by the answer key as “incorrect,” may be highlighted to the user for use in augmenting the answer key as another correct answer if the comparison indicates that there is a match between the generated answer and predicted answer. In addition or in the alternative, the predicted answer may be highlighted to the user for use in augmenting the answer key as another correct answer. In selected embodiments, the test question may be scored as correct and flagged with the generated answer and/or predicted answer for review and possible update of the answer key. In this way, the domain expert or system knowledge expert can use the accuracy score to review, evaluate, and accept proposed updates to the answer key based on returned answers that match with predicted answers derived from the historical answers and metadata associated with the answer key. Alternatively, the domain expert or system knowledge expert can use the accuracy score to identify a mismatch between the generated answer and the predictive answer when the accuracy score indicates an uncertainty associated with the mismatch.

DETAILED DESCRIPTION

A method, system, apparatus, and a computer program product are provided for exploiting an answer key's modification history to improve the training of a QA system. As disclosed, question-answer pairs in an answer key are evaluated against predicted answers derived from the historical answers and metadata associated with the answer key by generating answers from test questions submitted to the QA system and comparing generated answers that are scored incorrect by the answer key to the predicted answers, with any matches from the comparison being used to flag the test question with the generated answer and/or predicted answer for domain expert review and evaluation as a possible update for the answer key. In disclosing the claimed method and system for exploiting the answer key modification history and metadata by using specific rules, rather than humans, to evaluate generated answers against predicted answers to find matches indicating that the generated answer should be evaluated for updating the answer key, there is disclosed an improvement in computer-related technology that previously could only be produced by humans, if at all.

The present invention may be a system, a method, and/or a computer program product. In addition, selected aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and/or hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of computer program product embodied in a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention. Thus embodied, the disclosed system, a method, and/or a computer program product is operative to improve the functionality and operation of a cognitive question answering (QA) systems by efficiently providing answer keys with more accurate training QA pairs.

FIG. 1depicts a schematic diagram100of one illustrative embodiment of a question/answer (QA) system101connected across a computer network12to a first computing system17that uses an answer key analysis engine16to identify and correct answer key problems. The QA system101may include one or more QA system pipelines101A,101B, each of which includes a knowledge manager computing device104(comprising one or more processors and one or more memories, and potentially any other computing device elements generally known in the art including buses, storage devices, communication interfaces, and the like) for processing questions10received over the network12from the first computing device17along with questions received over the network102from one or more users at computing devices (e.g.,110,120,130). Over the networks12,102, the computing devices and QA system101communicate with each other and with other devices or components via one or more wired and/or wireless data communication links, where each communication link may comprise one or more of wires, routers, switches, transmitters, receivers, or the like. In this networked arrangement, the QA system101and networks12,102may enable question/answer (QA) generation functionality for one or more content users. Other embodiments of QA system101may be used with components, systems, sub-systems, and/or devices other than those that are depicted herein.

In the QA system101, the knowledge manager104may be configured to receive inputs from various sources. For example, knowledge manager104may receive input from the networks12,102, one or more knowledge bases or corpora106of electronic documents107, semantic data108, or other data, content users, and other possible sources of input. In selected embodiments, the knowledge base106may include structured, semi-structured, and/or unstructured content in a plurality of documents that are contained in one or more large knowledge databases or corpora. The various computing devices (e.g.,110,120,130) on the network102may include access points for content creators and content users. Some of the computing devices may include devices for a database storing the corpus of data as the body of information used by the knowledge manager104to generate answers to cases. The networks12,102may each include local network connections and remote connections in various embodiments, such that knowledge manager104may operate in environments of any size, including local and global, e.g., the Internet. Additionally, knowledge manager104serves as a front-end system that can make available a variety of knowledge extracted from or represented in documents, network-accessible sources and/or structured data sources. In this manner, some processes populate the knowledge manager with the knowledge manager also including input interfaces to receive knowledge requests and respond accordingly.

In one embodiment, the content creator creates content in an electronic document107for use as part of a corpora106of data with knowledge manager104. The corpora106may include any structured and unstructured documents, including but not limited to any file, text, article, or source of data (e.g., scholarly articles, dictionary definitions, encyclopedia references, and the like) for use in knowledge manager104. Content users may access knowledge manager104via a network connection or an Internet connection to the network102, and may input questions to knowledge manager104that may be answered by the content in the corpus of data. As further described below, when a process evaluates a given section of a document for semantic content, the process can use a variety of conventions to query it from the knowledge manager. One convention is to send a well-formed question1. Semantic content is content based on the relation between signifiers, such as words, phrases, signs, and symbols, and what they stand for, their denotation, or connotation. In other words, semantic content is content that interprets an expression, such as by using Natural Language (NL) Processing. In one embodiment, the process sends well-formed questions1(e.g., natural language questions, etc.) to the knowledge manager104. Knowledge manager104may interpret the question and provide a response to the content user containing one or more answers2to the question1, alone or in combination with supporting evidence and confidence metrics for each answer. In some embodiments, knowledge manager104may provide a response to users in a ranked list of answers2.

In some illustrative embodiments, QA system101may be the IBM Watson™ QA system available from International Business Machines Corporation of Armonk, N.Y., which is augmented with the mechanisms of the illustrative embodiments described hereafter to implement Watson Discovery Advisor functionality. The IBM Watson™ knowledge manager system may receive an input question1which it then parses to extract the major features of the question, that in turn are then used to formulate queries that are applied to the corpus of data stored in the knowledge base106. Based on the application of the queries to the corpus of data, a set of hypotheses, or candidate answers to the input question, are generated by looking across the corpus of data for portions of the corpus of data that have some potential for containing a valuable response to the input question.

In particular, a received question1may be processed by the IBM Watson™ QA system101which performs deep analysis on the language of the input question1and the language used in each of the portions of the corpus of data found during the application of the queries using a variety of reasoning algorithms. There may be hundreds or even thousands of reasoning algorithms applied, each of which performs different analysis, e.g., comparisons, and generates a score. For example, some reasoning algorithms may look at the matching of terms and synonyms within the language of the input question and the found portions of the corpus of data. Other reasoning algorithms may look at temporal or spatial features in the language, while others may evaluate the source of the portion of the corpus of data and evaluate its veracity.

The scores obtained from the various reasoning algorithms indicate the extent to which the potential response is inferred by the input question based on the specific area of focus of that reasoning algorithm. Each resulting score is then weighted against a statistical model. The statistical model captures how well the reasoning algorithm performed at establishing the inference between two similar passages for a particular domain during the training period of the IBM Watson™ QA system. The statistical model may then be used to summarize a level of confidence that the IBM Watson™ QA system has regarding the evidence that the potential response, i.e., candidate answer, is inferred by the question. This process may be repeated for each of the candidate answers until the IBM Watson™ QA system identifies candidate answers that surface as being significantly stronger than others and thus, generates a final answer, or ranked set of answers, for the input question. The QA system101then generates an output response or answer2with the final answer and associated confidence and supporting evidence. More information about the IBM Watson™ QA system may be obtained, for example, from the IBM Corporation website, IBM Redbooks, and the like. For example, information about the IBM Watson™ QA system can be found in Yuan et al., “Watson and Healthcare,” IBM developerWorks, 2011 and “The Era of Cognitive Systems: An Inside Look at IBM Watson and How it Works” by Rob High, IBM Redbooks, 2012.

In addition to providing answers to questions, QA system101is connected to a first computing system17which identifies and corrects answer key problems in a QA pair list19which may be used to train the QA system101. Though shown as being connected across a network12, the QA system101may be directly connected to the first computing system17or indirectly connected via the computer network102. Alternatively, the functionality described herein with reference to the first computing system17may be embodied in or integrated with the QA system101. To train the QA system101, the first computing system17may be configured to store the QA pair list19of question-answer pairs (e.g.,19a,19c) in a database18, alone or in combination with associated answer key metadata information (e.g.,19b,19d). However, the answers and/or question in the QA pair list19can be incorrect or inaccurate, particularly when new data is ingested into the system corpus, making it difficult to keep the answer key current (e.g., when the question or answer information becomes stale).

To ensure the accuracy of the QA pair list19, the first computing system17may be configured to assist with updating the answer key19by exploiting historical changes to the answer key to identify answers Ai13from the QA system101and/or to generate predicted answers Ai′ that should be evaluated for updating the answer key19. In selected embodiments, the answer key19is updated by using an answer key analysis engine16to (1) submit a test question10from the answer key19to the QA system101, (2) receive one or more answers Ai13generated and returned by the QA system101, (3) evaluating the received answers Ai13against the answer key19to identify incorrect answers Ai, (4) collect previous answers and metadata associated with the test question from the answer key19, (5) generate a predicted answer Ai′ to the test question from the previous answers and metadata associated, alone or combination with an optional answer tolerance or range and/or accuracy score, for comparison with the received answer Ai, and (6) flag the test question in the answer key for possible update with the predicted answer and/or with the generated answer if the comparison indicates a match between the generated answer and predicted answer. To obtain the one or more answers Ai13, the first computing system17may be configured to process each question-answer pair (e.g.,19a,19c) by passing each question Qi10to the QA system101. In response to a received question Qi10, the knowledge manager104and/or QA information handling system105generates answers A1, A2, . . . An13which are sent back to the first computing system17. To compare the first answer key answer from the QA pair list (e.g.,19a) to the returned answer(s) Ai13, the answer key analysis engine16may use natural language processing (NLP) or any known similarity calculation engine (e.g., Alchemy API™) to determine the similarity level between the first answer key answer19aand the returned answer(s)13for purposes of determining whether the returned answer is correct and should be further evaluated for updating the answer key19with information from the returned answer(s)13.

In selected embodiments, the answer key analysis engine16may use natural language (NL) processing to compare the first answer key answer19ato the returned answers Ai13based on one or more specified similarity metric values. For each returned answer(s) Ai13which is incorrect, the answer key analysis engine16may use the answer key modification history for the first answer key answer19ato find previous answers to the test question along with associated metadata information19b, and to compute therefrom a predicted answer Ai to the test question, alone or combination with an optional answer tolerance or range and/or accuracy score.

To provide a specific training session example, suppose the answer key19currently has the following edit or answer key history for a specific question:

In this training session example, if the QA system101produces a generated answer13(e.g., 323.64 million) for a test question, the answer key analysis engine16would determine that the generated answer is incorrect based a comparison with the current answer (e.g., 317.68 million) in the answer key. But by applying NLP techniques to evaluate the answer history, the answer key analysis engine16may determine that the question has a degree of temporal relevance so as to allow some latitude when determining the correctness of the answer13generated by QA system101by looking back through the entries in the answer key to determine that it is regularly and somewhat predictably changing. With such historical answers and metadata, the answer key analysis engine16could then compute a predicted answer Ai, such as by doing some basic fitting to determine an expected current value based on the past data to check whether a generated answer Ai is within a relatively likely range of correctness for the current value. Continuing with the training session example, the answer key analysis engine16may analyze the answer history to arrive at a prediction Ai′ of the correct current answer (or range of answers) for this question, such as by performing a statistical analysis of the time series to arrive at an interval of values within which the answer could be expected to fall. For example, a linear model might be applied to compute a predicted answer Ai′ that should be “325.34 million” plus or minus 10%. In this way, the system could avoid erroneously penalizing scorers which are producing good answers that simply suffer from a mismatch between the date of the corpus and the age of the information in the answer key.

If the predicted answer Ai′ matches with the generated answer(s) Ai from the QA system101, then the answer key analysis engine16generates a review message20for the QA system101indicating that there may be a problem with the first answer key answer and that the returned answer(s)13and/or predicted answers Ai may be equivalent or acceptable for inclusion in the first answer key answer. In addition, the returned answer(s)13may be highlighted to the user at display14for use in augmenting the first answer key answer19aas another correct answer. In addition or in the alternative, the predicted answer(s) Ai′ may be highlighted to the user at display14for use in augmenting the first answer key answer19aas another correct answer. In selected embodiments, the answer key analysis engine16may use the predicted answer Ai′ to compute an accuracy score for the generated answer Ai13which is in the review message20to the QA system101. Using the accuracy score, the domain expert or system knowledge expert can review and evaluate proposed updates to the answer key based on returned answers.

Types of information handling systems that can utilize QA system101range from small handheld devices, such as handheld computer/mobile telephone110to large mainframe systems, such as mainframe computer170. Examples of handheld computer110include personal digital assistants (PDAs), personal entertainment devices, such as MP3 players, portable televisions, and compact disc players. Other examples of information handling systems include pen, or tablet, computer120, laptop, or notebook, computer130, personal computer system150, and server160. As shown, the various information handling systems can be networked together using computer network102. Types of computer network102that can be used to interconnect the various information handling systems include Local Area Networks (LANs), Wireless Local Area Networks (WLANs), the Internet, the Public Switched Telephone Network (PSTN), other wireless networks, and any other network topology that can be used to interconnect the information handling systems. Many of the information handling systems include nonvolatile data stores, such as hard drives and/or nonvolatile memory. Some of the information handling systems may use separate nonvolatile data stores (e.g., server160utilizes nonvolatile data store165, and mainframe computer170utilizes nonvolatile data store175). The nonvolatile data store can be a component that is external to the various information handling systems or can be internal to one of the information handling systems. An illustrative example of an information handling system showing an exemplary processor and various components commonly accessed by the processor is shown inFIG. 2.

FIG. 2illustrates information handling system200, more particularly, a processor and common components, which is a simplified example of a computer system capable of performing the computing operations described herein. Information handling system200includes one or more processors210coupled to processor interface bus212. Processor interface bus212connects processors210to Northbridge215, which is also known as the Memory Controller Hub (MCH). Northbridge215connects to system memory220and provides a means for processor(s)210to access the system memory. In the system memory220, a variety of programs may be stored in one or more memory device, including an answer key evaluation engine module221which may be invoked to compare one or more generated answers, passages and sentences returned from a computing system (e.g., an IBM Watson™ QA system) to specified answers in the answer key using a semantic analysis, and then to identify potential answer key updates by evaluating the generated answers against predicted answers that are derived from answer key history and metadata, such as by computing a predicted answer from the answer key modification history and inserting the predicted answer and/or generated answer into the answer key if there is a match therebetween. Graphics controller225also connects to Northbridge215. In one embodiment, PCI Express bus218connects Northbridge215to graphics controller225. Graphics controller225connects to display device230, such as a computer monitor.

Northbridge215and Southbridge235connect to each other using bus219. In one embodiment, the bus is a Direct Media Interface (DMI) bus that transfers data at high speeds in each direction between Northbridge215and Southbridge235. In another embodiment, a Peripheral Component Interconnect (PCI) bus connects the Northbridge and the Southbridge. Southbridge235, also known as the I/O Controller Hub (ICH) is a chip that generally implements capabilities that operate at slower speeds than the capabilities provided by the Northbridge. Southbridge235typically provides various busses used to connect various components. These busses include, for example, PCI and PCI Express busses, an ISA bus, a System Management Bus (SMBus or SMB), and/or a Low Pin Count (LPC) bus. The LPC bus often connects low-bandwidth devices, such as boot ROM296and “legacy” I/O devices (using a “super I/O” chip). The “legacy” I/O devices (298) can include, for example, serial and parallel ports, keyboard, mouse, and/or a floppy disk controller. Other components often included in Southbridge235include a Direct Memory Access (DMA) controller, a Programmable Interrupt Controller (PIC), and a storage device controller, which connects Southbridge235to nonvolatile storage device285, such as a hard disk drive, using bus284.

ExpressCard255is a slot that connects hot-pluggable devices to the information handling system. ExpressCard255supports both PCI Express and USB connectivity as it connects to Southbridge235using both the Universal Serial Bus (USB) the PCI Express bus. Southbridge235includes USB Controller240that provides USB connectivity to devices that connect to the USB. These devices include webcam (camera)250, infrared (IR) receiver248, keyboard and trackpad244, and Bluetooth device246, which provides for wireless personal area networks (PANs). USB Controller240also provides USB connectivity to other miscellaneous USB connected devices242, such as a mouse, removable nonvolatile storage device245, modems, network cards, ISDN connectors, fax, printers, USB hubs, and many other types of USB connected devices. While removable nonvolatile storage device245is shown as a USB-connected device, removable nonvolatile storage device245could be connected using a different interface, such as a Firewire interface, etc.

Wireless Local Area Network (LAN) device275connects to Southbridge235via the PCI or PCI Express bus272. LAN device275typically implements one of the IEEE 802.11 standards for over-the-air modulation techniques to wireless communicate between information handling system200and another computer system or device. Extensible Firmware Interface (EFI) manager280connects to Southbridge235via Serial Peripheral Interface (SPI) bus278and is used to interface between an operating system and platform firmware. Optical storage device290connects to Southbridge235using Serial ATA (SATA) bus288. Serial ATA adapters and devices communicate over a high-speed serial link. The Serial ATA bus also connects Southbridge235to other forms of storage devices, such as hard disk drives. Audio circuitry260, such as a sound card, connects to Southbridge235via bus258. Audio circuitry260also provides functionality such as audio line-in and optical digital audio in port262, optical digital output and headphone jack264, internal speakers266, and internal microphone268. Ethernet controller270connects to Southbridge235using a bus, such as the PCI or PCI Express bus. Ethernet controller270connects information handling system200to a computer network, such as a Local Area Network (LAN), the Internet, and other public and private computer networks.

To provide additional details for an improved understanding of selected embodiments of the present disclosure, reference is now made toFIG. 3which depicts a simplified flow chart300showing the logic for identifying and correcting answer key problems by exploiting historical changes in the question set answer key to identify potential updates to the answer key with answers returned from the QA system, such as QA system101shown inFIG. 1, which match predicted answers derived from the answer key history and metadata. The processing shown inFIG. 3may be performed by a cognitive system, such as the first computing system17, QA system101, or other natural language question answering system and/or computing system, method, or module. Wherever implemented, the disclosed answer key evaluation scheme identifies answer key problems in question-answer pairs by comparing a returned answer from a QA system to a predicted answer that is derived from the answer key modification history (e.g., answer history and associated metadata) to find other correct answers. The comparison processing may include determining a similarity level between the answer key answer and one or more generated answers returned in response to a test question, indicating if the answer key answer is correct or incorrect based on a similarity level with the answer key, and generating a predictive answer to the test question based on analysis of changes to correct answers to the test question over a history of time, as given in the answer key, comparing the predictive answer to the generated answer(s) to form an accuracy score, and performing an action to update the answer key with the predictive answer and/or generated answer based on the accuracy score. With the disclosed answer key evaluation scheme, an information handling system can be configured to identify and correct answer key problems in a training set of QA pairs, thereby improving the accuracy of training for the QA system.

FIG. 3processing commences at301whereupon, at step302, a question/answer key is obtained. For example, a previously generated answer key may be retrieved from memory storage, or a received statement may be decomposed into one or more sets of questions and answer pairs, such as the QA pair list19stored in a database18, alone or in combination with associated answer key metadata (e.g., information identifying the authorship and creation data for each QA pair). The processing at step302may be performed at the first computing system17, the QA system101or other NLP question answering system.

At step303, the evaluation processing of each answer Ai in the QA key begins by submitting the corresponding question Qi from the QA key to a computing system. For example, a first computing system (e.g., computing system17) may send the corresponding questions of the question and answer pair sets, e.g., via a network (e.g., a network12including, but not limited to Intranet, Internet, Wireless communication network, Wired communication network, Satellite communication network, etc.) to a second computing system. The second computing system can process a natural language text and can provide answers to questions expressed in natural language. The second computing system may be embodied, but is not limited to, an artificial intelligent computer, Watson™ computer (trademark of International Business Machine, Inc. Armonk, N.Y.), etc. In selected embodiments, the first computing system and the second computing system are two different computing systems. In other embodiments, the first computing system and the second computing system are a single computing system. An artificial intelligent computer refers to an intelligent computing system adapted to perform a question answering (QA), natural language processing, data retrieval, logical reasoning, machine learning, etc.

At step304, after processing the corresponding question(s) Qi at the computing system (e.g., QA system101), one or more candidate responses or answers Ai to the questions (e.g., A1, A2, . . . An) are received. For each question Qi with corresponding answer Ai provided in the answer key, the method300passes the question Qi to the QA system and obtains back a corresponding set of answers A1, . . . , An.

Upon receiving the returned answers Ai generated in response to a test question Qi from the QA key, a counter value “i” may be initialized (step305) so that each answer Ai may be evaluated for correctness with an answer key matching process (step306). Through the counter incrementation process, the top ranked answer Ai may be evaluated first in a first iterative loop, followed by evaluation of additional answers returned by the QA system. In selected embodiments, each returned answer Ai may be compared to the corresponding answer in the QA key answer using natural language processing techniques to evaluate any desired comparison metric, including but not limited to lexical, syntactic, semantic, entropy, or ngram similarity metric, word overlap, etc. For example, a returned answer Ai may be decomposed by a natural language processor15in the first computing system17which is configured to identify at least one clause in the received statement, and is configured to separate the received statement according to the identified at least one clause. As will be appreciated, decomposing a statement is a process which takes a given statement as a “complex” fact and identifies at least one clause within the statement. As described herein, a Natural Language Processing (NLP) routine may be used to process the received questions and/or generate a computed answer with associated evidence and confidence measures, where “NLP” refers to the field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. In this context, NLP is related to the area of human-computer interaction and natural language understanding by computer systems that enable computer systems to derive meaning from human or natural language input. In selected embodiments, the natural language processor15is configured to decompose the received statement, e.g., by performing one or more of: (1) identifying syntactic labels (e.g., “this,” “that,” etc.) in the received statement in which the syntactic labels are indicators for breaking a statement into two or more statements; (2) identifying a conjunction(s) (e.g., “and,” “but,” etc.) that connects two parallel statements; (3) identifying configuration information of the received statement that includes, but is not limited to: (a) a syntactic contour that shows a main clause and a subordinate clause characterize a same entity, and (b) an indicative of a nested statement that focuses on one of several underspecified entities which do not share their facts; (4) identifying pre-modifiers (e.g., an adjective(s) in front of a named entity) and/or post-modifiers (e.g., an adjective(s) behind of a named entity) in the received statement which separate the received statement into one element and a statement; (5) identifying relative qualifiers (e.g., “the first,” “only,” “the westernmost,” etc.) which need to be completed by information from elsewhere. The natural language processor15is configured to separate the received statement according to one or more of: the identified at least one clause, the identified at least one syntactic label, the identified at least one conjunction, the identified configuration information, the identified at least one pre-modifier or post-modifier, and the identified at least one relative qualifier. In selected embodiments, the evaluation of the returned answer Ai at306may be computed by the first computing system17which applies a semantic or similarity analysis of the returned answer Ai and the answer key answer to compute a first similarity metric which is evaluated against an accuracy threshold to determine if the returned answer is correct.

If the returned answer matches the answer key (affirmative outcome to step306), the answer Ai is marked as correct for training purposes (step307), in which case the counter value “i” is incremented (step308) so that the next answer Ai may be evaluated for correctness with an answer key matching process (step306). However, if the returned answer does not match the answer key (negative outcome to step306), then the methodology300evaluates the returned answer Ai (which has identified as “incorrect”) for possible inclusion in the answer key by exploiting the answer key modification history by first assembling the answer key history and metadata (step309). Instead of thinking of answer keys as flat, stateless, one-dimensional things, the processing at step309exploits the wealth of information that is associated with each answer key answer in the form of file metadata, such as edit history, publication date, authorship, etc. Thus, step309may collect from the change set for the answer key the set of all previous answers to the test question along with associated metadata.

Using the assembled answer key history and metadata, a predictive answer Ai′ is computed at step310. For example, a first computing system (e.g., computing system17) may generate a prediction of the likely current answer and/or range of answers to the test question based on the pattern of changes in the answer key along with the associated metadata. In selected embodiments, any suitable technique may be used to identify patterns from the answer-key change history for use in computing the predictive answer Ai′, including but not limited to identifying patterns of numerical change (e.g., linear growth, exponential growth) in the answer. In addition, the processing at step310may identify characteristic temporal patterns for answers where certain classes of questions (e.g., “Who is the governor of Virginia?”) have answers change on a temporal basis (e.g., every four years) which could provide an indication that the generated answer is not incorrect if the answer key data metadata indicates that the answer key is due for an update (e.g., the answer key is over four years old for this question).

Given a predictive answer Ai′, the returned answer Ai is evaluated for matching at step311. For example, if the predictive answer Ai′ generated at step310includes a range of correct answers, then the match processing at step311may determine if the returned answer Ai falls within that range. If there is no match (negative outcome to decision311), then the returned answer—having previously been identified as being incorrect—is not a candidate for further review and evaluation as an answer key update candidate, in which case methodology300determines if all the returned answers have been evaluated (step313) then a match is indicated.

On the other hand, if there is a match between the predicted and returned answers (affirmative outcome to decision311), then the returned answer Ai and/or the predictive answer Ai′ may be scored as “correct” or otherwise denoted for additional review and possible inclusion in the answer key without imposing a training process penalty for the returned answer which was identified as incorrect (step312). To this end, the test question may be flagged along with the returned answer Ai for presentation to the domain expert with an indication that the returned answer should be evaluated for possible answer key inclusion or updating. As will be appreciated, each returned answer Ai that is sent for review may include associated information generated by the QA system, such as the passage from the corpus from which the answer was generated and/or accuracy score, to facilitate that review. In addition or in the alternative, the test question may be flagged with the predictive answer Ai′ so that the domain expert is notified to evaluate the predictive answer for a possible answer key update. As a result of step312, a new answer key may be generated when the domain expert reviews the flagged test question and any highlighted returned answer Ai or predictive answer Ai′, and then generates a new version of the key via this curation.

Once the returned answer Ai and/or predictive answer Ai′ is presented at step312, the methodology300determines if all the returned answers have been evaluated (step313). If not (negative outcome to decision step313), then counter value “i” is incremented (step314) so that the next answer Ai may be evaluated for correctness with an answer key matching process (step306). But once all returned answer Ai have been evaluated (affirmative outcome to decision step313), the process ends (step315), at which point the answer key evaluation process300may await reactivation by the domain expert or according to a predetermined or periodic activation schedule. Alternatively, the information generated by the methodology300may be fed to a machine learning process which acts on the training data to evaluate and update the answer key, as appropriate.

In general, the answer prediction processing at steps310-312may employ any desired analytic technique which uses past correct answer key answers to a question in order evaluate the current correctness of a generated answer, with particular benefit in the training and test of a deep QA system with frequently updated source data and less frequently updated answer keys.

Selected embodiments of the present disclosure are described with reference to evaluating answer key answers for possible update with respect to one or more generated answers returned by a QA system by using historical answer key answers to calculate or project a projected answer that is compared with and scored against the generated answer(s) to help identify generated answers which should be evaluated for review and possible inclusion as an answer key update. However, it will be appreciated that the present disclosure may be also be applied to evaluate answer key questions in terms of any desired answer key metadata to identify generated questions returned by a question-answer system for inclusion in the answer key, even when such generated answers are scored as incorrect when initially compared to the answer key. In such embodiments, answer key metadata corresponding to a test question that is used to generate an answer from the QA system is used to compare and evaluate the generated answer in terms of computed similarity metric values for purposes of identifying a question-answer pair in the answer key as potentially including an answer key problem and also flagging the generated answer as a candidate answer for updating the answer key.

By now, it will be appreciated that there is disclosed herein a system, method, apparatus, and computer program product for evaluating question-answer pairs in an answer key at a first information handling system having a processor and a memory by exploiting an answer key modification history for the answer key. As disclosed, the system, method, apparatus, and computer program product submit a test question from the answer key to a system capable of answering questions, and then subsequently receive a first generated answer from the system in response to the test question. If there is not a match between the first generated answer and a first answer from the answer key corresponding to the test question, the first information handling system generates a predicted answer to the test question based on the answer key modification history. In selected embodiments, the information handling system generates the predicted answer by analyzing a plurality of correct answers and/or metadata associated with the test question over time, and then computes the predicted answer from the plurality of correct answers and/or metadata. In other embodiments, the predicted answer is generated by applying a statistical analysis of answers in the answer key modification history to define the predicted answer with an interval of values for an accurate answer. In addition, the first information handling system generates an accuracy score based on how closely the predicted answer matches the first generated answer. Finally, the information handling system presents an indication that the answer key may have a problem if the accuracy score indicates there is a match between the predicted answer and first generated answer. In selected embodiments, the indication is presented by flagging the test question with the first generated answer and the accuracy score indication that the first generated answer should be evaluated for updating the answer key. In other embodiments, the indication is presented by flagging the test question with the predicted answer and the accuracy score indication that the predicted answer should be evaluated for updating the answer key. In other embodiments, the indication is presented by identifying a mismatch between the first generated answer and the predicted answer when the accuracy score indication indicates an uncertainty associated with the mismatch. In selected embodiments, the first information handling system applies natural language processing (NLP) to generate the predicted answer, and then applies a lexical, syntactic, semantic, entropy, or ngram similarity metric to compare the predicted answer and first generated answer.