Latent token representations for passage and answer scoring in question answering systems

The present invention may receive the question, a plurality of the candidate answers, and a plurality of documents associated with the plurality of candidate answers in the natural language. Then the present invention may tokenize the question, the plurality of the candidate answers, and the plurality of the documents into a corresponding n-gram sequence. The present invention may map n-gram elements from the tokenized question to the n-gram elements of the plurality of the tokenized candidate answers and the plurality of the tokenized documents using the latent token representation technique. The present invention may score the plurality of tokenized candidate answers based on the latent token representation technique. Then, the present invention may determine the precise answer based on the plurality of scored candidate answers.

BACKGROUND

The present invention relates, generally, to the field of computing, and more particularly to question answering (QA) in a natural language.

QA may be a computer system that combines information retrieval and natural language processing (NLP) capabilities and enables the location of answers to a specific question from an unstructured collection of natural language documents. Typically, a QA system extracts pertinent information from the question and generates a set of candidate answers while each set has a confidence score. QA may determine the best answer using a correlation between the confidence score and the candidate answer. For example, answers with high associated confidence scores may be rated a better quality.

SUMMARY

According to one embodiment, a method, computer system, and computer program product for determining a precise answer to a question using a latent token representation technique is provided. The present invention may receive the question, a plurality of the candidate answers, and a plurality of documents associated with the plurality of candidate answers in the natural language. Then the present invention may tokenize the question, the plurality of the candidate answers, and the plurality of the documents into a corresponding n-gram sequence. The present invention may map n-gram elements from the tokenized question to the n-gram elements of the plurality of the tokenized candidate answers and the plurality of the tokenized documents using the latent token representation technique. The present invention may score the plurality of tokenized candidate answers based on the latent token representation technique. Then, the present invention may determine the precise answer based on the plurality of scored candidate answers.

DETAILED DESCRIPTION

Embodiments of the present invention relate to the field of computing, and more particularly to question answering (QA) in a natural language. The following described exemplary embodiments provide a system, method, and program product to, among other things, improve question answering quality by receiving a question and corresponding candidate answers within underlying documents that support the candidate answers, and, by using latent token representations, score the answers to determine the precise answer based on the determined confidence score. Therefore, the present embodiment has the capacity to improve the technical field of NLP by using a scoring method based on latent token representation of the underlying passage that supports the candidate answer. By using the scoring method based on Latent Token Representation of the underlying passage, the QA system may provide precise answers to questions where important information is missing either in a question or in an answer and otherwise would have not been chosen as a precise answer.

As previously described, QA may be a computer system that combines information retrieval and natural language processing (NLP) capabilities and enables the location of answers to a specific question from an unstructured collection of natural language documents. Typically, QA system extracts pertinent information from the question and generates a set of candidate answers while each set has a confidence score. QA may determine the best answer using a correlation between the confidence score and the candidate answer. For example, answers with high associated confidence scores may be rated a better quality.

IBM Watson® (IBM Watson and all IBM Watson-based trademarks and logos are trademarks or registered trademarks of International Business Machines Corporation and/or its affiliates) may be utilized as a QA system. IBM Watson® is a knowledge base service designed for question answering that may have the capabilities to apply advanced natural language processing, information retrieval, knowledge representation, automated reasoning, and machine learning technologies to the field of open domain question answering. Typically, the precise answer to a question is based on different techniques that are used to analyze natural language, identify sources, find and generate candidate answers, find and score evidence, merge and determine a confident score for the candidate answers, and pick the candidate answer with a highest confidence score as the precise answer.

In QA systems, the title, passage, or metadata of the document used to determine a candidate answer may contain a critical piece of information that may affect the confidence score. For example, in the question “Who is the Secretary General of NATO?”, a document entitled “NATO” may be located with a passage that reads “The current Secretary General is Jens Stoltenberg”. Unfortunately, this passage does not contain the word “NATO”, which may cause the passage to be scored unreasonably low and, therefore, not chosen by a QA system as a precise answer. As such, it may be advantageous to, among other things, implement a system that receives a question coupled with candidate answers related to the documents or passages, determine a confidence score of each candidate answer using latent token representations, and choose the precise answer based on a confidence score.

According to one embodiment, an answer scoring program may receive a question and candidate answers coupled with the documents where the candidate answers may be derived from the documents or passages within the documents, and, using latent token representations, determine the confidence score of each candidate answer by analyzing metadata of the documents. A precise answer may then be chosen based on a highest confidence score.

The following described exemplary embodiments provide a system, method, and program product to receive a question and candidate answers coupled with the documents from a QA application, and, using a latent token representation method, the embodiment may determine the confidence score of each candidate answer by analyzing metadata and the content of the documents to choose the precise answer based on a highest confidence score.

Referring toFIG. 1, an exemplary networked computer environment100is depicted, according to at least one embodiment. The networked computer environment100may include client computing device102and a server112interconnected via a communication network114. According to at least one implementation, the networked computer environment100may include a plurality of client computing devices102and servers112, of which only one of each is shown for illustrative brevity.

Client computing device102may include a processor104and a data storage device106that is enabled to host and run a software program108, an answer scoring program110A, a candidate answers118, and communicate with the server112via the communication network114, in accordance with one embodiment of the invention. Client computing device102may be, for example, a mobile device, a telephone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running a program and accessing a network. Candidate answers118may be a data output received from QA application116in response to a question. According to at least one embodiment, the candidate answers118may include a question, candidate answers to a question, and documents corresponding to the candidate answers. The documents may be a text, a title, a passage or metadata associated with the candidate answer. In another embodiment, candidate answers118may be stored in database116. As will be discussed with reference toFIG. 3, the client computing device102may include internal components302aand external components304a, respectively.

The server computer112may be a laptop computer, netbook computer, personal computer (PC), a desktop computer, or any programmable electronic device or any network of programmable electronic devices capable of hosting and running a question answering application116, an answer scoring program110B, and a database116and communicating with the client computing device102via the communication network114, in accordance with embodiments of the invention. Question answering application116may be a program that enables identifying candidate answers to a specific question from an unstructured collection of natural language documents, such as IBM Watson® and storing the data in candidate answers118. In another embodiment, the answer scoring program110B may be integrated into the question answering application116. As will be discussed with reference toFIG. 3, the server computer112may include internal components302band external components304b, respectively. The server112may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server112may also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud.

According to the present embodiment, the answer scoring program110A,110B may be a program capable of determining a precise answer by analyzing the corresponding to the candidate answer documents determined by the QA application116. The answer scoring method is explained in further detail below with respect toFIG. 2.

Referring now toFIG. 2, an operational flowchart illustrating an answer scoring process200is depicted according to at least one embodiment. At202, the answer scoring program110A,110B receives a question in a natural language. According to at least one embodiment, the answer scoring program110A,110B may receive the question as a text string from QA application116. In another embodiment, the answer scoring program110A,110B may access the candidate answers118and load the question, or may ask a user of the client computing device102.

Next, at204, answer scoring program110A,110B receives candidate answers and one or more documents corresponding to candidate answers. According to at least one embodiment, answer scoring program110A,110B may receive the one or more documents from QA application116and extract a metadata, or a passage that corresponds to the candidate answer. In another embodiment, answer scoring program110A,110B may submit the question to the QA application and receive the candidate answers and the corresponding documents from the QA application116. In further embodiment, the answer scoring program110A,110B may access the candidate answers118and load the documents. According to at least one embodiment, the one or more documents may be text files in a natural language or one or more passages corresponding to each of the candidate answers determined by the QA application116while analyzing the question.

Then, at206, answer scoring program110A,110B generates n-grams from the question. According to at least one embodiment, answer scoring program110A,110B may generate a gappy n-grams from the question. Typically, n-gram is a sequence of adjacent elements or tokens from a string that may be a word, a letter, or a symbol. Gappy bigrams, or skipping bigrams, are word pairs which allow gaps between the n-gram elements. For example, if answer scoring program110A,110B tokenizes a question “Who is a secretary general of NATO?” using a gappy bigram, answer scoring program110A,110B may in addition to tokens “Who is”, “is a”, “a secretary”, “secretary general”, “general of”, and “of NATO?” add tokens with a gap between the elements such as “who, secretary”, “secretary, NATO?”, etc.

Next, at208, answer scoring program110A,110B maps the n-grams to each document associated with candidate answers using latent token representation method. According to at least one embodiment, answer scoring program110A,110B may tokenize the document data using the same n-gram technique as the tokenization of the question (such as using a bigram tokenization), and count the successful mappings of the question n-grams to the document data, and the number of skipped tokens according to the following latent token representation method:

1. Let L represent a set of latent tokens;

2. Let P represent the document in which the candidate answer appears;

4. For each n-gram N in the question Q:a. Let F=false;b. Let S=0;c. Represent N as a queue NQ;d. Represent P as a queue PQ;e. While PQ is not empty:i. Let T be the first element of PQ;ii. Let M be the first element of NQ;iii. If M is equivalent to T:1. Dequeue the first token from NQ;2. Dequeue the first token from PQ;3. Set variable F=true;iv. Else if M is a member of L:1. Dequeue the first token from NQ;2. Set variable F=true;v. Else if F==true:1. Add a penalty to S corresponding to T;2. Dequeue the first token from PQ;vi. If NQ is empty1. Break;f. While NQ is not empty AND the first element of NQ is a member of L:i. Dequeue the first token from NQ;g. If NQ is empty:i. Add to R a weight representing S and the length of N;

For example, in the question “Who is the secretary general of NATO?”, the candidate answer is “Jens Stoltenberg is the current secretary general”, and the title of the corresponding document is “Leadership of NATO”. Without analyzing the document, the phrase “of NATO” is not appearing in the candidate answer, therefore this candidate answer would be assigned a low confidence score. In addition, if there is an additional candidate answer, such as “Ban Ki-Moon is the current secretary general”, this candidate answer would be assigned the same confidence score, which may lead to returning an incorrect precise answer.

To continue the previous example, the latent token representation method may map the tokenized question Q=[“secretary, general, of, nato”] to the passage P—“Jens Stoltenberg is the secretary general who is currently serving” from the document having a tokenized title L=[“leadership, of, nato”] as described in the following iterations:

1. Token T is “Jens Stoltenberg”. It is not equivalent to the first token of NQ (“secretary”). The method is not dequeening the element from the queue yet (that is, F==false), therefore, the method does not add a penalty. The queue is not empty, so the method proceeds to the next iteration.

2. Token T is “is”. It is not equivalent to the first token of NQ (“secretary”). The method is not dequeening an element from the queue yet (that is, F==false), therefore, the method does not add a penalty. The queue is not empty, so the method proceeds to the next iteration.

3. Token T is “secretary”. It is equivalent to the first token of NQ (“secretary”), so the method dequeening the first element from the NQ and PQ and sets F=true. The queue is not empty, so the method proceeds to the next iteration.

4. Token T is “general”. It is equivalent to the first token of NQ (“of”), so the method dequeening the first element from the NQ and PQ and sets F=true (although it is already true). The queue is not empty, so the method proceeds to the next iteration.

5. Token T is “who”. It is not equivalent to the first token of NQ (“of”), but it is a member of L, the set of latent tokens, therefore the method dequeening the first element from NQ and sets F=true (although it is already true). The queue is not empty, so the method proceeds to the next iteration.

6. Token T is still “who”. It is not equivalent to the first token of NQ (“nato”), but it is a member of L, our set of latent tokens, so the method dequeening the first element from NQ and sets F=true (although it is already true). The queue is now empty, so we break and return R (that represents a perfect match).

Then, at210, the answer scoring program110A,110B scores each candidate answer based on latent token representation output. According to at least one embodiment, answer scoring program110A,110B may score each candidate answer based on values returned by the latent token representation method performed in step208, such as R, S, T and F. For example, the score may be a numerical value such as a weighted average of the values R, S, T and F. In another embodiment, answer scoring program110A,110B may use a simple Jaccard-style n-gram scorer that compares strings or sets of n-grams.

Next, at212, the answer scoring program110A,110B determines the precise answer. According to at least one embodiment, the answer scoring program110A,110B may determine the precise answer based on the highest score of each candidate answer by picking the highest score candidate answer as the precise answer. In another embodiment, answer scoring program110A,110B may choose the candidate answer as a precise answer based on finding a perfect match only. In further embodiment, the answer scoring program110A,110B may prompt a user to select a high scoring answer from a list if there are a number of answers with scores above a predefined threshold value.

It may be appreciated thatFIG. 2provides only an illustration of one implementation and does not imply any limitations with regard to how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

The data processing system302,304is representative of any electronic device capable of executing machine-readable program instructions. The data processing system302,304may be representative of a smart phone, a computer system, PDA, or other electronic devices. Examples of computing systems, environments, and/or configurations that may represented by the data processing system302,304include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

The client computing device102and the server112may include respective sets of internal components302a,band external components304a,billustrated inFIG. 3. Each of the sets of internal components302include one or more processors320, one or more computer-readable RAMs322, and one or more computer-readable ROMs324on one or more buses326, and one or more operating systems328and one or more computer-readable tangible storage devices330. The one or more operating systems328, the software program108and the answer scoring program110A in the client computing device102, and the answer scoring program110B in the server112are stored on one or more of the respective computer-readable tangible storage devices330for execution by one or more of the respective processors320via one or more of the respective RAMs322(which typically include cache memory). In the embodiment illustrated inFIG. 3, each of the computer-readable tangible storage devices330is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices330is a semiconductor storage device such as ROM324, EPROM, flash memory or any other computer-readable tangible storage device that can store a computer program and digital information.

Each set of internal components302a,balso includes a R/W drive or interface332to read from and write to one or more portable computer-readable tangible storage devices338such as a CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk or semiconductor storage device. A software program, such as the cognitive screen protection program110A,110B, can be stored on one or more of the respective portable computer-readable tangible storage devices338, read via the respective R/W drive or interface332, and loaded into the respective hard drive330.

Each set of internal components302a,balso includes network adapters or interfaces336such as a TCP/IP adapter cards, wireless Wi-Fi interface cards, or 3G or 4G wireless interface cards or other wired or wireless communication links. The software program108and the answer scoring program110A in the client computing device102and the answer scoring program110B in the server112can be downloaded to the client computing device102and the server112from an external computer via a network (for example, the Internet, a local area network or other, wide area network) and respective network adapters or interfaces336. From the network adapters or interfaces336, the software program108and the answer scoring program110A in the client computing device102and the answer scoring program110B in the server112are loaded into the respective hard drive330. The network may comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.

Each of the sets of external components304a,bcan include a computer display monitor344, a keyboard342, and a computer mouse334. External components304a,bcan also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each of the sets of internal components302a,balso includes device drivers340to interface to computer display monitor344, keyboard342, and computer mouse334. The device drivers340, R/W drive or interface332, and network adapter or interface336comprise hardware and software (stored in storage device330and/or ROM324).

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Workloads layer90provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation91; software development and lifecycle management92; virtual classroom education delivery93; data analytics processing94; transaction processing95; and candidate answer scoring96. Candidate answer scoring96may relate to analyzing the documents corresponding to candidate answers using a latent token representation method, and determining the precise answer based on a score derived from the latent token representation method.