Generating dual sequence inferences using a neural network model

A computer-implemented method for dual sequence inference using a neural network model includes generating a codependent representation based on a first input representation of a first sequence and a second input representation of a second sequence using an encoder of the neural network model and generating an inference based on the codependent representation using a decoder of the neural network model. The neural network model includes a plurality of model parameters learned according to a machine learning process. The encoder includes a plurality of coattention layers arranged sequentially, each coattention layer being configured to receive a pair of layer input representations and generate one or more summary representations, and an output layer configured to receive the one or more summary representations from a last layer among the plurality of coattention layers and generate the codependent representation.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to neural network models and more particularly to neural network models for dual sequence inference.

BACKGROUND

Neural networks have demonstrated great promise as a technique for automatically analyzing real-world information with human-like accuracy. In general, neural network models receive input information and make predictions based on the input information. For example, a neural network classifier may predict a class of the input information among a predetermined set of classes. Whereas other approaches to analyzing real-world information may involve hard coded processes, statistical analysis, and/or the like, neural networks learn to make predictions gradually, by a process of trial and error, using a machine learning process. A given neural network model may be trained using a large number of training examples, proceeding iteratively until the neural network model begins to consistently make similar inferences from the training examples that a human might make. Neural network models have been shown to outperform and/or have the potential to outperform other computing techniques in a number of applications. Indeed, some applications have even been identified in which neural networking models exceed human-level performance.

DETAILED DESCRIPTION

Question answering (QA) is one class of problems to which neural networks may be been applied. In QA applications, a QA model receives a sequence of text representing a document and a sequence of text representing a question. The goal of the QA model is to accurately predict a portion of the document (e.g., a span of text in the document) that answers the question. To illustrate, suppose a document provided to a QA model includes the text “Some believe that the Golden State Warriors team of 2017 is one of the greatest teams in NBA history,” and further suppose that a question provided to the QA model includes the text “Which team is considered to be one of the greatest teams in NBA history?” The ground truth answer to the question is the span of text in the document that reads “the Golden State Warriors team of 2017.” Accordingly, the QA model should identify the span of text in the document that matches the ground truth answer. At the very least, the QA model should identify an overlapping span of text that is close in meaning to the ground truth answer (e.g., “Golden State Warriors”).

QA models are applicable to a variety of technologies, including search engines, digital personal assistants, chatbots, and/or the like. Some QA models may be designed for general-purpose applications (e.g., capable of answering a wide variety of question and/or document types, question and/or document lengths, answer lengths, and/or the like). Others may be tailored for specialized applications.

The performance of QA models may be compared or benchmarked by testing different models on a shared dataset, such as, for example, the Stanford Question Answering Dataset (SQuAD). The accuracy of each model may be measured by evaluating one or more metrics, such as exact match accuracy (e.g., the percentage of trials where the predicted answer exactly matches the ground truth answer), F1 score accuracy (which assesses the amount of overlap between the predicted answer and the ground truth answer), and/or the like. State of art QA models achieve less than or equal to 72.3% exact match accuracy and less than or equal to 80.7% F1 score accuracy on SQuAD, or when ensembled, less than or equal to 76.9% exact match accuracy and less than or equal to 84.0% F1 score accuracy.

Accordingly, it is desirable to develop QA models that achieve higher accuracy than current state of art QA models. It is also desirable to develop techniques for training QA models faster and/or with less training data. More generally, it is desirable to developed improved neural network models that generate inferences based on a pair of input sequences, referred to herein as dual sequence inference. Although some dual sequence inference models receive text input sequences, such as the QA models described above, it is to be understood that the dual sequence inference models may operate on a wide variety of types of input sequences, including but not limited to text sequences, audio sequences, image sequences (e.g., video), and/or the like.

FIG. 1is a simplified diagram of a system100for dual sequence inference according to some embodiments. According to some embodiments, system100may receive a first sequence102and a second sequence104and generate an inference106(and/or multiple inferences). In QA applications, sequence102may correspond to a text sequence representing a document, sequence104may correspond to a text sequence representing a question, and inference106may correspond to a span of text in the document representing an answer to the question. However, it is to be understood that QA is merely one example, and that system100may be used in a wide variety of applications, including non-QA applications such as textual entailment (TE) applications.

As depicted inFIG. 1, system100includes a controller110. In some embodiments, controller110may include a processor120(e.g., one or more hardware processors). Although processor120may include one or more general purpose central processing units (CPUs), processor120may additionally or alternately include at least one processor that provides accelerated performance when evaluating neural network models. For example, processor120may include a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a tensor processing unit (TPU), a digital signal processor (DSP), a single-instruction multiple-data (SIMD) processor, and/or the like. Generally, such processors may accelerate various computing tasks associated with evaluating neural network models (e.g., training, prediction, preprocessing, and/or the like) by an order of magnitude or more in comparison to a general purpose CPU.

Controller110may further include a memory130(e.g., one or more non-transitory memories). Memory130may include various types of short-term and/or long-term storage modules including cache memory, static random access memory (SRAM), dynamic random access memory (DRAM), non-volatile memory (NVM), flash memory, solid state drives (SSD), hard disk drives (HDD), optical storage media, magnetic tape, and/or the like. In some embodiments, memory130may store instructions that are executable by processor120to cause processor120to perform operations corresponding to processes disclosed herein and described in more detail below.

Processor120and/or memory130may be arranged in any suitable physical arrangement. In some embodiments, processor120and/or memory130may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor120and/or memory130may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor120and/or memory130may be located in one or more data centers and/or cloud computing facilities.

In some embodiments, memory130may store a model140that is evaluated by processor120during dual sequence inference. Model140may include a plurality of neural network layers. Examples of neural network layers include densely connected layers, convolutional layers, recurrent layers, pooling layers, dropout layers, and/or the like. In some embodiments, model140may include at least one hidden layer that is not directly connected to either an input or an output of the neural network. Model140may further include a plurality of model parameters (e.g., weights and/or biases) that are learned according to a machine learning process. Examples of machine learning processes include supervised learning, reinforcement learning, unsupervised learning, and/or the like. Embodiments of model140are described in further detail below with reference toFIGS. 2-7.

Model140may be stored in memory130using any number of files and/or data structures. As depicted inFIG. 1, model140includes a model description file142that defines a computational graph of model140(e.g., a sequence of neural network layers) and a model parameters file144that stores parameters of model140(e.g., weights and/or biases). In general, model description file142and/or model parameters file144may be store information associated with model140in any suitable format, including but not limited to structured, unstructured, serialized, and/or database formats.

FIG. 2is a simplified diagram of a model200for dual sequence inference according to some embodiments. According to some embodiments consistent withFIG. 1, model200may be used to implement model140. In some embodiments, model200may receive a first sequence202and a second sequence204and generate an inference206(and/or multiple inferences). In embodiments consistent withFIG. 1, sequences202and204and inference206may generally correspond to sequences102and104and inference106, respectively.

Model200may include a first input stage212and a second input stage214that receive sequences202and204, respectively. Input stage212generates an input representation216of sequence202, and input stage214generates an input representation218of sequence204. In some embodiments, input representations216and/or218may correspond to vector representations of sequences202and/or204, respectively. For example, when sequences202and/or204correspond to text sequences, input stages212and/or214may generate the corresponding vector representations by (1) tokenizing the text sequences and (2) embedding the tokenized text sequences in a vector space. Tokenizing the text sequences may include identifying tokens within the text sequences, where examples of tokens include characters, character n-grams, words, word n-grams, lemmas, phrases (e.g., noun phrases), sentences, paragraphs, and/or the like. Embedding the tokenized text sequences may include mapping each token to a vector representation in a multidimensional vector space. For example, a token corresponding to a word may be mapped to a 300-dimensional vector representation of the word using pre-trained GloVe vectors.

Model200may further include an encoder stage220that receives input representations216and218and generates a codependent representation222of sequences202and/or204that depends on each of sequences202and204. For example, in QA applications, where sequence202corresponds to a document and sequence204corresponds to a question, codependent representation222may depend on both the document and the question. This is in contrast to input stages212and214, which analyze the document and the question independently of one another. In this regard, encoder stage220may harness the context that the question provides when analyzing the document and/or vice versa. In some embodiments, encoder stage220may include a deep coattention encoder, embodiments of which are described in greater detail below with reference toFIGS. 3-4.

Model200may further include a decoder stage230that receives codependent representation222and generates inference206. In QA applications, decoder stage230may include a dynamic decoder that iteratively predicts a span in sequence202that contains the answer to the question corresponding to second sequence204. For example, the dynamic decoder may output a pair of pointers corresponding to the start and end of the predicted span. The iterative process may terminate when the prediction converges (e.g., when a change in the prediction between consecutive iterations is below a threshold). Embodiments of dynamic decoders are described in further detail in “Dynamic Coattention Networks for Question Answering,” inICLR,2017, to Xiong et al., which is herein incorporated by reference in its entirety.

According to some embodiments, model200may correspond to a computational graph, in which case input stages212and/or214, encoder stage220, and/or decoder stage230may correspond to collections of nodes in the computational graph. Consistent with such embodiments, various representations used by model200, such as input representations216and/or218, codependent representation222, and/or any intermediate representations used by model200, may correspond to real-valued tensors (e.g., scalars, vectors, multidimensional arrays, and/or the like) that are passed along edges of the computational graph. Moreover, each node of the computation graph may perform one or more tensor operations, e.g., transforming one or more input representations of the node into one or more output representations of the node. Examples of tensor operations performed at various nodes may include matrix multiplication, n-dimensional convolution, normalization, element-wise operations, and/or the like.

FIGS. 3A and 3Bare simplified diagrams of a deep coattention encoder300according to some embodiments. According to some embodiments consistent withFIGS. 1-2, deep coattention encoder300may be used to implement encoder stage220. Consistent with such embodiments, deep coattention encoder300may receive an input representation302of a first sequence and an input representation304of a second sequence and may generate a codependent representation306that depends on each of the first and second sequences. In some embodiments, input representations302and/or304may be generated by respective input stages, such as input stages212and/or214.

Deep coattention encoder300may include a plurality of coattention layers310a-narranged sequentially (e.g., in a pipelined fashion). Each of coattention layer310a-ngenerates a respective first summary representation312a-ncorresponding to the first sequence and a respective second summary representation314a-ncorresponding to the second sequence based on a pair of layer input representations. In the case of the first layer in the sequence (i.e., coattention layer310a), the pair of layer input representations corresponds to input representations302and304. In the case of other layers in the sequence (i.e., coattention layers310b-n), the pair of layer input representations corresponds to summary representations312a-nand314a-ngenerated by a preceding layer in the sequence. In the case of the last layer in the sequence (i.e., coattention layer310n), either of summary representations312nand/or314nmay be omitted and/or optional. For example, as depicted inFIG. 3A, summary representation314nis optional.

In comparison to encoders that include a single coattention layer, deep coattention encoder300may be capable of generating a richer codependent representation306that contains more relevant information associated with the first and second input sequences. For example, deep coattention encoder300may include more trainable model parameters than single-layer coattention encoders. Moreover, whereas a single-layer coattention encoder may allow each sequence to attend to the other sequence, deep coattention encoder300may allow each sequence to attend to itself as well as to the other sequence. Consequently, deep coattention encoder300may be capable of achieving higher accuracy than single-layer coattention encoders in dual sequence inference problems, such as QA problems.

FIG. 3Bdepicts a coattention layer310f, which may be used to implement one or more of coattention layers310a-ndepicted inFIG. 3A. As depicted inFIG. 3B, coattention layer310fincludes a pair of encoding sub-layers322and324that each receive a respective layer input representation312eand314eand generate a respective encoded representation E1and E2. In some embodiments, encoding sub-layers322and/or324may include one or more recurrent neural network (RNN) layers. In general, an RNN layer injects sequence-related information (e.g., temporal information) into the transformed representation. For example, the RNN layer may include a sequence of simple RNN cells, long short-term memory (LSTM) cells, gated recurrent units (GRUs), and/or the like. In some examples, the RNN layer may be bi-directional, e.g., a bi-directional LSTM (Bi-LSTM) layer. Additionally or alternately, encoding sub-layers322and/or324may include a feed-forward neural network layer, and/or may perform any other suitable transformation or set of transformation on layer input representations312eand314e. In some embodiments, encoding sub-layers322and/or324may include one or more nonlinear activation functions (e.g., rectified linear units (ReLU), sigmoid, hypertangent (tanh), softmax, and/or the like).

In illustrative embodiments, encoded representations E1and E2may correspond to real-valued tensors determined according to the following equations:
E1=encoding1(L1)∈h×m(1)
E2=encoding2(L2)∈h×n(2)
where L1and L2denote the respective layer input representations; m and n denote the length of the first and second sequences, respectively; h denotes the number of dimensions of the encoded representations; and encoding1(X) and encoding2(X) denote respective encoding operations (e.g., RNN operations, bi-LSTM operations, feed-forward operations, and/or the like) applied to an input X.

Coattention layer310fmay further include an affinity node331that determines a set of affinity scores corresponding to each pair of items in in encoded representations E1and E2. In general, an affinity score may be large for a related pair of items and small for an unrelated pair of items. For example, when the words “dog,” and “tree” appear in the first sequence and the word “puppy” appears in the second sequence, the pairing (“dog”, “puppy”) is likely to receive a high affinity scores because the words refer to the same type of animal, whereas the pairing (“tree”, “puppy”) is likely to receive a low affinity score because they are unrelated concepts. In illustrative embodiments, the set of affinity scores may be determined according to the following equation:
A=(E1)TE2∈m×n(3)
where A denotes an affinity matrix containing the set of affinity scores and XTdenotes the transpose of the matrix X.

Coattention layer310fmay further include a pair of summary nodes332and333that generate summary representations S1and S2, respectively, based on the affinity scores and the encoded representations E1and E2. In illustrative embodiments, summary representations S1and S2may correspond to real-valued tensors determined according to the following equations:
S1=E2activation1(AT)∈h×m(4)
S2=E1activation2(A)∈h×n(5)
where activation1(X) and activation2(X) denote respective activation operations over the matrix X (e.g., linear, softmax, sigmoid, tanh, ReLU, ELU, and/or the like).

Coattention layer310fmay further include a context nodes334that generates context representation312f(C1) based on the affinity scores and summary representations S2. In illustrative embodiments, context representation C1may correspond to a real-valued tensor determined according to the following equation:
C1=S2activation3(AT)∈h×m(6)
The activation operations used by context node334may or may not be the same as the activation operations used by summary nodes332and/or333.

Returning toFIG. 3A, deep coattention encoder300may additionally include an output layer350that receives summary representations312nand/or314nfrom the last layer among the plurality of coattention layers310a-nand generates codependent representation306. In some embodiments, output layer350may include a neural network layer, such as and RNN, feed-forward neural network, and/or the like.

In some embodiments, deep coattention encoder300may include a plurality of model parameters learned according to a machine learning process, such as a supervised learning process, a reinforcement learning process, an unsupervised learning process, and/or the like. However, there are various challenges associated with training the model parameters of deep neural network models, such as a model that includes deep coattention encoder300. For example, one approach to training deep neural network models is to iteratively update the model parameters over a set of training data based on the gradient of a learning objective. However, deep neural networks may train slowly, or not at all, due to the degradation of the gradients (e.g., vanishing and/or exploding gradients) at layers far from the output of the neural network model. Accordingly, one challenge associated with deep coattention encoder300is to train model parameters associated with layers and/or sub-layers distant from output layer350(e.g., coattention layers310aand/or310b).

To address this challenge, deep coattention encoder300may include one or more residual connections360. Residual connections360bypass one or more layers (and/or sub-layers and/or nodes) of deep coattention encoder300, thereby reducing the effective distance between deep layers of the network (e.g., coattention layers310aand/or310b) and output layer350. In general, residual connections360may bypass any number of layers, sub-layers, and/or nodes. As depicted inFIG. 3B, the source end of residual connections360may correspond to any or each of encoded representations E1and/or E2, summary representations S1and/or S2, and/or context representation C1.

In some embodiments, residual connections360may be combined with other inputs at a destination layer. For example, residual connections360may be concatenated at the destination. Consistent with such embodiments, the size of the inputs to the destination layer may be increased by the use of residual connections360. To the extent that the increase in input size may be undesirable, various techniques may be applied to reduce the size concatenated input. For example, a pooling layer (e.g., max pooling, average pooling, and/or the like), a feed-forward neural network, and/or the like may be used to the reduce the size of the concatenated input. Additionally or alternately, residual connections360and/or other inputs may be combined by techniques other than concatenation, such as summation.

FIG. 4is a simplified diagram of a deep coattention encoder400of a QA model according to some embodiments. In some embodiments consistent withFIGS. 1-3B, deep coattention encoder400may be used to implement deep coattention encoder300. Deep coattention encoder400receives first sequence corresponding to a document of m words and a second sequence corresponding to a question of n words. The document and question are processed by respective input encoders, which generally correspond to input stages212and/or214. As depicted inFIG. 4, the input encoders generate input representation of the document LDand an input representation of a question LQaccording to the following equations:
LD=concat(embGloVe(wD),embchar(wD),embcoVe(wD))∈m×e(7)
LQ=concat(embGloVe(wQ),embchar(wQ),embcoVe(wQ))∈n×e(8)
where wD=[w1D, w2D. . . wmD] denotes the set of words in the document, wQ=[w1Q, w2Q. . . wnQ] denotes the set of words in the question, embGloVe(Iv) denotes the GloVe embeddings of a set of words, embchar(w) denotes the character embeddings of a set of words, embcoVe(w) denotes a context vector embedding of a set of words, concat(A,B,C) denotes a concatenation between matrices A, B, and C along a feature dimension, m denotes the number of words in the document, n denotes the number of words in the question, and e denotes the total number of dimensions of the word embeddings, character embeddings, and context vector embeddings. In some embodiments, the context vector embeddings are generated by a context vector encoder, such as a two-layer BiLSTM encoder, pretrained on a text corpus, such as the WMT machine translation corpus.

Deep coattention encoder400includes a first coattention layer410, which generally corresponds to coattention layer310aof deep coattention encoder300. The input representations of the document LDand the question LQare received by respective bi-LSTM encoders412and414of first coattention layer410. In some embodiments consistent withFIG. 3B, bi-LSTM encoders412and414may correspond to encoding sub-layers322and324, respectively. Bi-LSTM encoders412and414generate encoded representations E1Dand E1Qof the document and the question, respectively, according to the following equations:
E1D=bi-LSTM1(LD)∈h×(m+1)(9)
E1Q=tanh(W bi-LSTM1(LQ)+b)∈h×(n+1)(10)
where h denotes the number of dimensions of the encoded representation, W and b denote weights and biases, respectively, of a feed-forward neural network layer, and tanh(x) denotes the hypertangent activation function. A sentinel word is added to the input representation to prevent deep coattention encoder400from focusing on a particular part of the input representation, so the number of words in the encoded representation of the document and question is (m+1) and (n+1), respectively.

The encoded representations E1Dand E1Qof the document and the question are received by a coattention sub-layer416, which generally corresponds to nodes331-335as depicted inFIG. 3B. Based on encoded representations E1Dand E1Q, coattention sub-layer416determines an affinity matrix A between the document and the question according to the following equation:
A=(E1D)TE1Q∈(m+1)×(n+1)(11)

As discussed previously, the affinity matrix A contains an affinity score for each pair of words in E1Dand E1Q.

Based on affinity matrix A, coattention sub-layer416determines document an question summary representations S1Dand S1Q, respectively, according to the following equations:
S1D=E1Qsoftmax(AT)∈h×(m+1)(12)
S1Q=E1Dsoftmax(A)∈h×(n+1)(13)
where softmax(X) denotes the softmax operation over the matrix X that normalizes X column-wise.

Based on affinity matrix A and summary representations S1Dand S1Q, coattention sub-layer416determines document context representation C1Daccording to the following equation:
C1D=S1Qsoftmax(AT)∈h×m(14)
The sentinel word is removed, such that the number of words in the document context representations C1Dis m rather than m+1. In some embodiments consistent withFIGS. 3A-3B, C1Dmay correspond to context representation312a.

Deep coattention encoder400further includes a second coattention layer420that generally corresponds to coattention layers310band/or310nof deep coattention encoder300. As depicted inFIG. 4, second coattention layer420includes bi-LSTM encoders422and424and a coattention sub-layer426, which generally operate in a similar manner to bi-LSTM encoders412and414and coattention sub-layer416of first coattention layer410, as described above. In particular, bi-LSTM encoders422and424generate encoded representations E2Dand E2Qof the document and the question, respectively. In some embodiments, the size of representations E2Dand/or E2Qmay be the same as and/or different from the size of representations E1Dand/or E1Q. In illustrative embodiments, the size of representations E2Dand/or E2Qmay be double the size of representations E1Dand/or E1Q(e.g., E2D∈2h×mand/or E2Q∈2h×n). Based on encoded representations E2Dand E2Q, coattention sub-layer426generates and outputs one or more of a summary representation of the document S2Dand/or a coattention context of the document C2D.

An output encoder430receives the output representations from the preceding layers and generates a codependent representation U of the document according to the following equation:
U=bi-LSTM(concat(E1D;E2D;S1D;S2D;C1D;C2D))∈2h×n(17)
As indicated above, output encoder430receives various representations of the document (e.g., E1D, E2D, S1D, and C1D) from bi-LSTM encoder412, coattention sub-layer416, and BiLSTM encoder422, in addition to representations of the document from coattention sub-layer426(e.g., S2D, and C2D). The representations received from earlier layers of deep coattention encoder400correspond to residual connections, such as residual connection360, that bypass one or more layers and/or sub-layers of the network. In general, the use of residual connections may facilitate training of deep coattention encoder400by addressing gradient degradation issues.

FIG. 5is a simplified diagram of a training configuration500with a mixed learning objective according to some embodiments. As depicted inFIG. 5, training configuration500is used to train a model510. In some embodiments consistent withFIGS. 1-4, model510may be used to implement model200. In some embodiments, training configuration500may be used to reduce the amount of time and/or training data used to train model510. In some embodiments, model510may include a deep coattention encoder, such as deep coattention encoders300and/or400. However, it is to be understood that training configuration500may be used for training a wide variety of model types, including non-QA models and/or models without deep coattention encoders.

According to some embodiments, training configuration500may be used to train a plurality of model parameters of model510. During training, a large number of training examples (e.g., pairs of input sequences for dual sequence inference applications) are provided to model510. The inferences generated by model510are compared to a ground truth answers for each of the examples using a learning objective520, which determines a loss and/or reward associated with a given inference based on the ground truth answer.

The output of learning objective520(e.g., the loss and/or reward) is provided to an optimizer530to update the model parameters of model510. For example, optimizer530may determine the gradient of the objective with respect to the model parameters and adjust the model parameters using backpropagation. In some embodiments, optimizer530may include a gradient descent optimizer (e.g., stochastic gradient descent (SGD) optimizer), an ADAM optimizer, an Adagrad optimizer, an RMSprop optimizer, and/or the like. Various parameters may be supplied to optimizer530(e.g., a learning rate, a decay parameter, and/or the like) depending on the type of optimizer used.

In some embodiments, model510may iteratively generate a series of inferences for a given pair of input sequences. For example, model510may include a coattention encoder, such as deep coattention encoder300, that generates a codependent representation of the pair of input sequences and a dynamic decoder that iteratively generates inferences based on the codependent representation until the inferences converge (e.g., when the inferences change by less than a threshold amount during consecutive iterations).

In some embodiments, learning objective520may determine the loss and/or reward associated with a given series of inferences generated by model510using a supervised learning objective540. In some embodiments, supervised learning objective540may determine loss and/or reward by evaluating a differentiable objective function, such as the cross-entropy loss function. In QA applications, where each inference corresponds to a span in a document defined by a start position and an end position, the cross-entropy loss may be defined as follows:
lossce(Θ)=−Σt[logptstart(s|st-1,et-1;Θ)+logptend(e|st-1,et-1;Θ)]  (18)
where lossce(Θ) is the cross-entropy loss for a given set of model parameters Θ; ptstart∈mand ptend∈mare the distributions of the start and end positions, respectively, estimated by the dynamic decoder at decoder time step t; s and e are the ground truth start and end positions, respectively; and st-1and et-1are the estimates of the start and end positions at the previous decoder time step. Because the cross-entropy loss function is differentiable with respect to the model parameters, it is generally straightforward for optimizer530to determine the gradient and update the parameters at each training step by back propagation.

Although supervised learning objective540may provide a useful starting point for assessing the accuracy of the inferences generated by model510, this approach may on occasion produce undesirable results. For example, supervised learning objective540may punish and/or reward certain inferences in a non-intuitive or unwarranted manner. In QA applications, supervised learning objective540may correspond to the “exact match” accuracy metric discussed previously. In this regard, supervised learning objective540may determine loss and/or rewards in a binary manner, in which inferences are regarded as being correct when they exactly correspond to the ground truth answer and incorrect otherwise. However, the exact match metric does not provide a notion of being “close” to the correct answer; each inference is regarded as either right or wrong, with no in-between.

Other evaluation metrics, such as the F1 score, are non-binary. In general, non-binary evaluation metrics account for the fact that some inferences may be regarded as being at least partially correct, even if they do not exactly match the ground truth answer. For example, the F1 score partially rewards inferences that overlap with, but do not exactly match, the ground truth answer. In this regard, non-binary evaluation metrics, such as the F1 score, may provide a more nuanced comparison between the inferences and the ground truth than binary evaluation metrics, such as the exact match metric.

Accordingly, learning objective520may include a reinforcement learning objective550based on a non-binary evaluation metric, such as the F1 score. In some embodiments, reinforcement learning objective550may use the non-binary evaluation metric to define a loss and/or reward function for a reinforcement learning process. For example, reinforcement learning objective550may evaluate to the negative of the expected reward over trajectories T given a set of model parameters, where each of the trajectories T corresponds to a sequence of start and end positions at each decoder time step. In illustrative embodiments, reinforcement learning objective550may be evaluated as follows:
baseline=F1(ans(sT,eT),ans(s,e))  (19)
R=F1(ans(ŝT,êT),ans(s,e))−baseline  (20)
lossrl(Θ)=−{circumflex over (τ)}˜pτ[R]  (21)
where F1denotes the F1 word overlap scoring function; baseline denotes the baseline F1 score; ans(x, y) denotes the answer span retrieved from the document based on a given start position x and end position y; s and e are the ground truth start and end positions, respectively; sTand eTare the baseline start and end positions, respectively, at the last decoder time step T; R is the reinforcement learning reward function; ŝTand êTare the start an end positions, respectively, of the sampled trajectory {circumflex over (τ)} at the last decoder time step T; lossrl(Θ) is the reinforcement learning loss for a given set of model parameters Θ; and pτis the probability distribution of trajectories τ.

In some embodiments, reinforcement learning objective550may include a greedy prediction module551that determines sTand eT(the start and end positions of the baseline, as defined in Equation 19) in a greedy fashion without a teacher forcing on the start position. Reinforcement learning objective550may further include a first evaluator552that evaluates Equation 19 to determine the baseline F1 score based on sTand eT. In some embodiments, reinforcement learning objective550may include a sampled policy prediction module523that determines ŝTand êTand a second evaluator554that determines the policy F1 score based on ŝTand êT. The policy F1 score corresponds to the first term, F1(ans(ŝT, êT), ans(s,e)), of Equation 20. Reinforcement learning objective550further includes a self-critic module555that subtracts the baseline F1 score from the policy F1 score to obtain the reinforcement learning loss defined by Equation 21.

In some embodiments, learning objective520may include a task combination module560to combine supervised learning objective540and reinforcement learning objective550. In some embodiments, combining supervised learning objective540and reinforcement learning objective550(as opposed to using one or the other) may accelerate the training of model510. More specifically, the use of supervised learning objective540may accelerate policy learning according to reinforcement learning objective550by pruning the space of candidate trajectories. For example, in QA applications, the use of reinforcement learning objective550(without supervised learning objective540) may result in slow training due to the large space of potential answers, documents, and/or questions.

In illustrative embodiments, learning objective520may include a task combination module560that combines supervised learning objective540and reinforcement learning objective550using homoscedastic uncertainty as task-dependent weightings according to the following equation:

Unlike the cross-entropy loss function, the reinforcement learning loss function used by reinforcement learning objective550may not be differentiable. Accordingly, optimizer530may use estimation techniques to determine the gradient of associated with reinforcement learning objective550. According to some embodiments, the gradient associated with reinforcement learning objective550may be approximated using a single Monte-Carlo sample τ drawn from the probability distribution praccording to the following equation:
∇Θlossrl(Θ)≈−R∇Θ(ΣtT(logptstart(ŝt;Θ)+logptend(êT;Θ)))  (23)
where all terms are as previously defined. Based on the approximated gradient of reinforcement learning objective550with respect to the model parameters Θ, optimizer530may proceed to update the parameters of model510based on the combination of supervised learning objective540and reinforcement learning objective550.

FIG. 6is a simplified diagram of a method600for dual sequence inference according to some embodiments. According to some embodiments consistent withFIGS. 1-5, method600may be performed using a processor, such as processor120. In some embodiments, method600may be performed by evaluating a neural network model, such as model140and/or200. In some embodiments, the neural network model may include a plurality of model parameters learned according to a machine learning process.

At a process610, a codependent representation is generated based on a first sequence and a second sequence. In some embodiments, the codependent representation may be generated by an encoder stage of the neural network model. In illustrative embodiments, the encoder stage may be implemented using a deep coattention encoder, such as deep coattention encoder300and/or400. In some embodiments the first and second sequence may correspond to text sequences, audio sequences, image sequences (e.g., video), and/or the like. In QA applications, the first sequence may correspond to a document and the second sequence may correspond to a question.

At a process620, an inference is generated based on the codependent representation. In some embodiments, the inference may be generated using a decoder stage of the model, such as decoder stage230. In some embodiments, the decoder model may include a dynamic decoder model that iteratively generates a series of inferences based on the codependent representation. In QA applications, the inference may identify a span of text in the document that answers the question.

FIG. 7is a simplified diagram of a method700for training a neural network model using a mixed learning objective according to some embodiments. According to some embodiments consistent withFIGS. 1-6, method700may be used to train a neural network model, such as model140and/or200. During training, the model may be configured in a training configuration, such as training configuration500. In some examples, method700may be performed iteratively over a large number of training examples to gradually train the neural network model.

At a process710, a series of inferences is generated using the neural network model. In some embodiments, the series of inferences may be generated based on a training example that includes a first training sequence and a second training sequence. In some embodiments, the series of inferences may be generated according to method600, in which an encoder stage of the neural network model generates a codependent representation based on the first and second training sequences. Consistent with such embodiments, the series of inferences may correspond to a series of inferences generated by a dynamic decoder based on the codependent representation.

At a process720, a mixed learning objective is evaluated based on the series of inferences. In some embodiments, the mixed learning objective may correspond to learning objective520. Consistent with such embodiments, the mixed learning objective may include a supervised learning objective, such as supervised learning objective540, and a reinforcement learning objective, such as reinforcement learning objective550. Whereas the supervised learning objective may determine a loss and/or reward independently at each decoder step (e.g., independently at each of the series of inferences), the reinforcement learning objective may determine an expected loss and/or reward over an entire trajectory (e.g., collectively over the series of inferences). In some examples, the reinforcement learning objective may determine the expected loss and/or reward using a non-binary evaluation metric, such as the F1 evaluation metric.

At a process730, the parameters of the neural network model are updated based on the mixed learning objective. In some embodiments, the model parameters may be updated using an optimizer, such as optimizer530. In some embodiments, the parameters may be updated by determining a gradient of the mixed learning objective with respect to the model parameters and updating the parameters based on the gradient. The gradient of differentiable components of the mixed learning objective, such as the supervised learning objective component, may be determined by back propagation. To the extent that the component of the mixed learning objective associated with the reinforcement learning objective may not be differentiable, the gradient may be estimated, e.g., using Monte Carlo techniques.

FIGS. 8A-8Dare simplified diagrams of an experimental evaluation of a QA model according to some embodiments. The QA model being evaluated includes a deep coattention encoder, configured as depicted inFIG. 4, and is trained on the Stanford Question Answering Dataset (SQuAD) using a mixed learning objective, with a training configuration as depicted inFIG. 5.

FIG. 8Adepicts a table810that compares the accuracy of a QA model that includes the deep coattention encoder (top row) to the accuracy of QA models that do not include the deep coattention encoder (other rows). The training and testing is performed on SQuAD. As indicated in the table, the QA model that includes the deep coattention encoder achieves the highest accuracy across all metrics, including 75.1% exact match accuracy (Test EM), 83.1% F1 score accuracy (Test F1), 78.9% ensemble exact match accuracy (Ens Test F1), and 86.0% ensemble F1 accuracy (Ens Test F1).

FIG. 8Bdepicts a series of plots822,824, and826that compare the accuracy of a QA model that includes a deep coattention encoder to a model that includes a single-layer coattention encoder. In particular, plot822plots F1 score accuracy as a function of question type (e.g., “who,” “what,” “where,” “when,” “why,” “how,” “which,” and “other”). As indicated, the QA model that includes the deep coattention encoder outperforms the model that includes the single-layer coattention encoder for every question type. Plot824plots F1 score accuracy as a function of the number of words in the question. As indicated, the QA model that includes the deep coattention encoder outperforms the QA model that includes the single-layer coattention encoder for almost every length of question. Plot826plots F1 score accuracy as a function of the number of words in the answer. As indicated, the QA model that includes the deep coattention encoder outperforms the QA model that includes the single-layer coattention encoder for almost every length of answer.

FIG. 8Cdepicts a table830with results of an ablation study performed on the QA model. The ablation study isolates the contribution of various features (e.g., a deep coattention encoder, as depicted inFIG. 4, and/or a training configuration with a mixed learning objective, as depicted inFIG. 5) to the overall improvement in accuracy of the QA model. The top row corresponds to a QA model with a deep coattention encoder trained with a mixed learning objective. The second row corresponds to a QA model with a single-layer coattention encoder trained with a mixed learning objective. As indicated, the deep coattention layer is responsible for a 1.4% improvement in exact match accuracy and a 1.6% improvement in F1 score accuracy. Meanwhile, the mixed learning objective is responsible for an 0.7% improvement in exact match accuracy and a 1.0% improvement in F1 score accuracy.

FIG. 8Ddepicts a pair of plots842and844that compare training curves for a QA model with a deep coattention encoder with a mixed learning objective (solid line) and without a mixed learning objective (dotted line). Plot842displays the entire training curves, and plot844displays a close-up view of the training curves at early training stages. As indicated, the use of a mixed learning objective during training significantly increases the training rate of the QA model compared to training without the mixed learning objective.