Year: 2018

Abstract:
Text embedding representing natural language documents in a semantic vector space can be used for document retrieval using nearest neighbor lookup. In order to study the feasibility of neural models specialized for retrieval in a semantically meaningful way, we suggest the use of the Stanford Question Answering Dataset (SQuAD) in an open-domain question answering context, where the first task is to find paragraphs useful for answering a given question. First, we compare the quality of various text-embedding methods on the performance of retrieval and give an extensive empirical comparison on the performance of various non-augmented base embedding with, and without IDF weighting. Our main results are that by training deep residual neural models specifically for retrieval purposes can yield significant gains when it is used to augment existing embeddings. We also establish that deeper models are superior to this task. The best base baseline embeddings augmented by our learned neural approach improves the top-1 recall of the system by 14% in terms of the question side, and by 8% in terms of the paragraph side.

heading: 1 INTRODUCTION
text: The goal of open domain question answering is to answer questions posed in natural language using an unstructured collection of natural language documents such as Wikipedia. Given the recent successes of increasingly sophisticated, neural attention based question answering models such as Yu et al. (2018), it is natural to break the task of answering a question into two subtasks as suggested in Chen et al. (2017):
• Retrieval: retrieval of the paragraph most likely to contain all the information to answer the question correctly.
• Extraction: Utilizing one of the above question-answering models to extract the answer to the question from the retrieved paragraphs.
In our case, we use the collection of all SQuAD paragraphs as our knowledge base and try to answer the questions without knowing which paragraphs they correspond to. We do not benchmark the quality of the extraction phase, but study the quality of text retrieval methods, and the feasibility of learning specialized neural models text retrieval purposes. Due to the complexity of natural languages, a good text embedding that represents the natural language documents in a semantic vector space is critical for the performance of the retrieval model. We constrain our attention to approaches that use a nearest neighbor look-up over a database of embeddings using some embedding method.
We describe the implementation details utilizing the pre-trained embeddings to create semantically more meaningful text embedding improved by utilizing an extra knowledge base, and hence improving the quality of a target NLP retrieval task. Our solution is based on refining existing text embeddings trained on huge text corpora in an unsupervised manner. Given those embeddings, we learn residual neural models to improve their retrieval performance. For this purpose, we utilize the triplet learning methodology with hard negative mining.
The first question studied here, is whether having advanced, and refined embedding may provide higher recall for retrieval in the external knowledge base. Therefore we benchmark most popular available embedding models, and compare their performance on the retrieval task. First, we start
with a review of recent advances in text embeddings in section 2. In section 3 we describe our proposed approach. The core idea is to augment precomputed text embedding models with an extra deep residual model for retrieval purposes. We present an empirical study of the proposed method in comparison with our baselines which utilize the most popular word embedding models. We report the results in section 4. Finally, we conclude the paper with some future work in section 5.

heading: 2 RELATED WORK
text: There are various type of applications of word embedding in the literature that was well covered by Perone et al. (2018). The influential Word2Vec by Mikolov et al. (2013) is one of the first popular approaches of word embedding based on neural networks that was built on top of guiding work by Bengio et al. (2003) on the neural language model for distributed word representations. This type of implementation is able to conserve semantic relationships between words and their context or in other terms surrounding neighboring words. Two different approaches are proposed in [28] to compute word representations. One of the approaches is called Skip-gram that predicts surrounding neighboring words given a target word and the other approach is called Continuous Bag-of-Words that predicts target word using a bag-of-words context. As a following study, Global Vectors (GloVe) by Pennington et al. (2014), points to reduce some limitations of Word2Vec by focusing on the global context instead of surrounding neighboring words for learning the representations. The global context is calculated by utilizing the word co-occurrences in a corpus. During this calculation, a count-based approach is functioned unlike the prediction-based method in Word2Vec. On the other hand, FastText, by Bojanowski et al. (2017), is also announced recently. It has the same principles like others that focus on extracting word embedding from large corpus. FastText is very similar to Word2Vec except handling each word as a formation of character n-grams. That formation accelerate FastText to learn representation more efficiently.
Nonetheless, the important question still exists on extracting high-quality and more meaningful representations: How to seize the semantic, syntactic and the different meanings in different context. This is also the point where our journey is getting started. Embedding from Language Models (ELMo),by Peters et al. (2018), was newly proposed in order to tackle that question. ELMo extracts representations from a bi-directional Long Short Term Memory (LSTM),by Hochreiter & Schmidhuber (1997), that is trained with a language model (LM) objective on a very large text dataset. ELMo representations are a function of the internal layers of the bi-directional Language Model (biLM) that outputs good and diverse representations about the words/token (a CNN over characters). ELMo is also incorporating character n-grams like in FastText but there are some constitutional differences between ELMo and its predecessors. The reason why we focused on ELMo is that it can generate more refined and detailed representations of the words due to utilizing internal representations of the LSTM network. As we mentioned before, ELMo is built on top of a LM and each word/token representation is a function of the entire input which makes it different than the other methods and also eliminates the others’ restrictions where each word/token is a mean of multiple contexts.
There are some other embedding methods which are not used as a baseline in the field; therefore, we did not get into them in detail. Skip-Thought Vectors, by Kiros et al. (2015), uses encoderdecoder RNNs that predicts the nearby sentence of a given sentence. The other one is InferSent, by Conneau et al. (2017), that utilizes a supervised training for the sentence embeddings, unlike as in Skip-Thought. They used BiLSTM encoders to generate embeddings. p-mean, by Rücklé et al. (2018), is built to tackle the unfairness of the Skip-Through that is taking the mean of the word/token embeddings with different dimensions. There are some other more methods such as Le & Mikolov (2014)’s Doc2Vec/Paragraph2Vec, Hill et al. (2016)’s fastSent, and Pagliardini et al. (2018)’s Sent2Vec.
Furthermore, we also include the newly-announced Universal Sentence Encoder by Cer et al. (2018) in our comparisons. Universal Sentence Encoder is built on top of two different models. Initially, the Transformer-based attentive encoder model, by Vaswani et al. (2017), was implemented to get high accuracy and the other model takes the benefit of deep averaging network (DAN), by Iyyer et al. (2015). As a result, averages of words/token embeddings and bi-grams can be provided as input to a DAN where the sentence embeddings are getting computed. According to the authors, their sentence level embeddings exceeds the performance of transfer learning using word/token embeddings alone.
Last, but not least, distance metric learning is designed to amend the representation of the data in a way that retains the related points close to each other while separating different points on the vector space as stated by Lowe (1995), Cao et al. (2013), and Xing et al. (2002). Instead of utilizing a standard distance metric learning, a non-linear embedding of the data using deep networks has shown an important improvements by learning representations using triplet loss by Hadsell et al. (2006), Chopra et al. (2005), contrastive loss by Weinberger & Saul (2009), Chechik et al. (2010), angular loss by Wang et al. (2017), and n-pair loss by Sohn (2016) for some of influential studies presented by Taigman et al. (2014), Sun et al. (2014), Schroff et al. (2015), and Wang et al. (2014).
After providing brief review of the latest trends in the field, we describe the details of our approach and experimental results in the following sections.

heading: 3 PROPOSED APPROACH

heading: 3.1 EMBEDDING MODEL
text: After examining the latest advanced work in word embedding, and since ELMo outperforms the other approaches for all important NLP tasks, we included the pre-trained ELMo contextual word representations into our benchmark to find out the embedding model for the retrieval task that produces the best result. The pre-trained ELMO model used the raw 1 Billion Word Benchmark dataset Chelba et al. (2013), and the vocabulary of 79347 tokens.
By having all the requirements set, it was time to compute ELMo representations for the SQuAD dev set using the pre-trained ELMo model. Before explaining the implementation strategies, we would like to mention the details of ELMo representation calculations. As they showed in their study, a L-layer bILM computes a set of 2L + 1 representations for each token ti.
Li = {xLMi , −→ h LMi,j , ←− h LMi,j |j = 1, ..., L} = { ←→ h LMi,j |j = 0, ..., L}
where xLMi , a context-indepedent token representation, is computed via a CNN over characters. Therefore hLMk,0 is representing the token layer. These representations are then transmitted to L layers of forward and backward LSTMs (or BiDirectional LSTMs-biLSTMs). Each LSTM layer provides a context-dependent vector representation ←→ h LMi,j where j = 1, ..., L. While the top layer LSTM results, −→ h LMi,j , presents the prediction of the next token ti+1, likewise, ←− h LMi,j represents the prediction of the previous token ti−1 by using a Softmax layer. The general approach would be a weighting of all biLM layers to get ELMo representations of the corpus. For our task, we generated the representations for the corpus, not only considering the results from the weighting of all biLM layers (ELMo), but also other individual layers such as token layer, biLSTM layer 1 and biLSTM layer 2.
As a result, we build the tensors based on tokens instead of documents; but documents can also be sliced from these tensors. To this end, we created a mapping dictionary for keeping the document’s index and length (number of tokens). In this way a token is represented as the tensor of a shape [1, 3, 1, 1024]. To improve the traceability, the tensors were reshaped by swapping axis to become the tensor of a shape [1, 3, 1024], then we stacked all these computed tokens to end up with the tensor of a shape [416909, 3, 1024] where 416909 is the number of tokens in the corpus (in other words, 416909 represents the 12637 documents in the corpus). Therefore, any document could be extracted from this tensor using the previously-created mapping dictionary. For example, let us assume that doc1 has an index of 126 and a length of 236, so doc1 can be extracted by slicing the tensor using such a format [126 : 126 + 236, :, :]. In terms of math expression, we ended up building a tensor Ti,j,d where i is representing the total documents in the corpus, j is representing the layers, and d is a dimension space of a vector. T is based on tokens instead of documents, but documents can also be sliced from that tensor.
To achieve a good embedding, the main question was about identifying the most critical components that contain the most valuable information for the document. Since the new tensor of a shape [12637, 3, 1024] has 3 components, in order to extract the most valuable slice(s), we defined a new weight tensor W1,j,1 where the j had a same dimension as in the tensor T. For our experiment, we set the j to 3 as the same dimension value in which the pre-trained ELMo word representations were used. The elements of the vector j of the tensor W was symbolized as a, b, and c, where a+ b+ c = 1. In order to find the right combination within the elements of the weight tensor W for the best embedding matrixM , we calculated the following function:
M ′ = ||mean(T ∗W;θ = 1)||
whereM ′ = {M ′1,M ′2, . . . ,M ′n}, W ′ = {W′1,W ′ 2, . . . ,W ′ n}, n is the number of documents in the set and θ represents the axis argument of the function. Last, but not least, all candidate embedding was normalized by L2-Norm. With this setup, we were able to create the candidate embedding matrices represented by M ′ for further experiments to get the ultimate best embedding E. Each embedding in the embedding matrix is symbolized by f(x) ∈ Rd. Therefore, it was able to embed a document x into a d-dimensional Euclidean space.
In order to capture the best combination of W′ (in other words, finding the best a, b, and c values), we calculated the pair-wise distances between questions and paragraphs embedding represented Q′ and P ′ respectively that were sliced from M ′. The performance of finding the correct questionparagraph embedding pair is measured by recall@k. The reason why we used the recall@k is that we wanted to compute the hitting performance of the true paragraph (answers) to the particular question whether it is in the top-k closest paragraphs or not. We used the ranks of 1, 2, 3, 5, 10, and, 50. In order to calculate the recall values, we had to deal with 20M question-paragraph embedding pairs ( 10K questions x 2K answers) in the dev dataset. We primarily aimed to look at the Receiver Operating Characteristic (ROC) curve and using Area Under Curve (AUC) as the target metric for tuning the embedding. Therefore, we sorted the retrieved samples by their distance (the paragraphs to the question) and then we computed recall and precision at each threshold. After having the precision numbers from the computations, we observed that our precision numbers became very low values because of having only 10k correct pairs and a high amount of negative paragraphs. We decided to switch to other methods called Precision-Recall curve and Average Precision.
Additionally, we took one step further. We wanted to inject the inverse document frequency (IDF) weights of each token into the embedding M to observe whether it creates any positive impact on the representation or not. Since the IDF is calculated using the following function: IDF (t) = loge( #ofdocuments #ofdocs.w/term ), we calculated the IDF weights from each tokenized documents.Then we multiplied newly-computed token IDF weights with the original token embedding extracted from the pre-trained model to have the final injected token embedding.
Similar to both the ELMo and extension of ELMo with IDF weights steps, we followed the same pipeline for the GloVe since it is one of the very important baselines for this type of research. We specially utilized the ”glove-840B-300d” pre-trained word vectors where it was trained on using the common crawl within 840B tokens, 2.2M vocab, cased, 300d vectors. We created the GloVe representation of our corpus and also extended it with IDF weights.
Fully Connected Residual Retrieval Network (FCRR).
Convolutional Residual Retrieval Network (ConvRR).

heading: 3.2 RETRIEVAL MODEL
text: The retrieval model aims to improve the recall@1 score by selecting the correct pair among all possible options. While pursuing this goal, it is developing embedding accuracy by minimizing the distance from the questions to the true paragraphs after embedding both questions and paragraphs to the semantic space.
The model architectures we designed to implement our idea are presented in Figure 2 and Figure 3. Question embedding Qb (b is representing the batch of question embedding) were provided as inputs to the models. The output layer was wrapped with He et al. (2016)’s residual layer by scaling the previous layer’s output with a constant scalar factor sf . Finally, the ultimate output from the residual layer was also normalized by L2. The normalized question embedding and the corresponding paragraph embedding are processed by the model that is optimized using a loss function L . For this reason, we proposed and also utilized several loss functions. The main idea of all of this was to move the true q ∈ Qb, p ∈ P b pairs closer to each other while keeping the wrong pairs further from each other. Using a constant margin value m would be also helpful to create an enhanced effect on this goal.
The first function we defined was a quadratic regression conditional loss. With this Lquadratic regression conditionally, we aimed to use a conditional margin.
m′(m, d) = { m, if d > m. 0, otherwise.
(1)
Lquadratic regression conditional = [||q − p||2 −m]+ +m′(m, ||q − p||2)
The latter loss function we utilized was the triplet loss. With this setup, we not only consider the positive pairs but also negative pairs in the batch. Our aim was that a particular question qanchor would be a question close in proximity to a paragraph ppositive as the answer to the same question than to any paragraph pnegative as they are answers to other questions. The key point of the Ltriplet was building the correct triplet structure which should not meet the restraint of the following equation easily ||qanchor − ppositive||2 + m < ||qanchor − pnegative||2. Therefore, for each anchor, we took the
positive ppositive in such a way argmaxppositive ||qanchor − ppositive|| 2 and likewise the hardest negative pnegative in such a way that argminpnegative ||qanchor − pnegative|| 2 to form a triplet. This triplet selection strategy is called hard triplets mining.
Ltriplet = [||qanchor − ppositive||2 − ||qanchor − pnegative||2 +m]+
During all our experiments, we have utilized Kingma & Ba (2014)’s ADAM optimizer with a learning rate of 10−3. For the sake of equal comparison, we have fixed the seed of randomization. We have also observed that a weight decay 10−3 and/or a dropout 10−1 is the optimum value for all types of model architectures to tackle over-fitting. In addition to that, the batch size b of questionparagraph embedding pairs has been set to 512 during the training. We trained the networks with a range of between 50 to 200 iterations.
As specifically, we designed our FCRR model as a stack of multi-layers. During the observation of the experiments, the simple but the efficient configuration was emerged as 2-layers. Likewise, we used 1024 filters, 5 as a length of the 1D convolution window, and 2 as a length of the convolution in our ConvRR model. Last, but not least, the scaling factor is set to 1 for both models.

heading: 4 EXPERIMENT
text: We did an empirical study of the proposed question-answering method in comparison with major state-of-the-art methods in terms of both text embedding and question answering. In the subsequent sections, we are going to describe details for the dataset, the model comparison and the results.

heading: 4.1 SQUAD DATASET
text: For the empirical study of the performance of text embedding and question answering, we use the Stanford Question Answering Dataset (SQuAD) Rajpurkar et al. (2016). The SQuAD is a benchmark dataset, consisting of questions posed by crowd-workers on a set of Wikipedia articles, where the answer to every question is a segment of text from the corresponding reading passage.

heading: 4.2 EMBEDDING COMPARISON
text: We wanted to observe trade-off between Precision and Recall and AP value to compare performances of each candidate embeddingM ′. recall@k was calculated for each embedding matrixM ′ by using grid search over the a, b, c components. See Table 2.
The best component combination emerged as a = 1, b = 0, c = 0 that provides the best setting to represent the corpus. In other words, token layer is the most useful layer. Finally, question and paragraph embedding are represented Q = {q1, q2, . . . , qx} and P = {p1,p2, . . . ,p(n − x)},
where x represents the number of questions within the corpus. In other terms [0 : 10570, 1024] belongs to questions while [10570 :, 1024] belongs to paragraphs for the dev dataset.
We can now start to compare the results of the recall@k of the pair-wise embedding that are derived from different models for the dev corpus in order to measure the embedding quality. The recall@k tables and the Precision-Recall curve from each of the embedding structure are presented in the section below.
Table 3: recall@ks of GloVe.
recall@k # of docs %
1 3246 30.7 % 2 4318 40.8 % 5 5713 54.0 % 10 6747 63.8 % 20 7602 71.9 % 50 8524 80.6 %
Table 4: recall@ks of Extension of GloVe with IDF weights.
recall@k # of docs %
1 3688 34.8 % 2 4819 45.5 % 5 6293 59.5 % 10 7251 68.5 % 20 8076 76.4 % 50 8919 84.3 %
Figure 5: P-R Curve for extension of GloVe with IDF weights.
Table 5: recall@ks of ELMo.
recall@k # of docs %
1 4391 41.5 % 2 5508 52.1 % 5 6793 64.2 % 10 7684 72.6 % 20 8474 80.1 % 50 9271 87.7 %
Figure 6: P-R Curve for ELMo.
recall@k # of docs %
1 4755 44.9 % 2 5933 56.3 % 5 7208 68.1 % 10 8040 76.0 % 20 8806 83.3 % 50 9492 89.8 %
The average-precision scores of each embedding structure is listed in Table 7.
By following the same pipeline, we applied the defined best model to the train set. As an initial step, we randomly selected 5K question embedding and corresponding paragraph embedding from the corpus as our validation set. The remaining part of the set is defined to be used as a training data. The results from the best model (Extension of ELMo w/ IDF weights) and baseline model (Only ELMo) are presented in Table 8.
As presented in Table 8, we were able to improve the representation performance of the embeddings increased by 4% on a recall@1 and 4.7% on an average recall for our task by injecting the IDF weights into the ELMo token embedding.

heading: 4.3 MODEL FOR RETRIEVAL
text: After having the best possible results as a new baseline, we aimed to improve the representation quality of the question embedding. In order to train our models for the retrieval task, we decided to merge the SQuAD dev and train datasets to create the larger corpus. By executing that step, we have a total of 98169 questions and 20963 paragraphs. We randomly selected 5K question embedding and corresponding paragraph embedding from the corpus as our validation set for the recall task and similarly 10K as our validation set for the loss calculation task. The remaining part of the set is defined as the training data.
The intuition was that the correct pairs should be closer in proximity to one another so that closest paragraph embedding to the question embedding would be correct in an ideal scenario. With this goal; 1) the question embedding is fine-tuned by our retrieval models so that the closest paragraph embedding would be the true pair for the corresponding question embedding, 2) similarly, the paragraph embedding is also fine-tuned by our retrieval model so that the closest question embedding would be the true pair for the corresponding paragraph embedding, and, finally 3) on top of previous steps, we further fine-tuned question embedding using the combination of improved question and paragraph embedding stated at step 1 & 2.
As shown in Table 9 and Table 10, we were able to improve the question and paragraph embedding such a way that the retrieval performance for the question side increased by 9.36%± 0.11% on a recall@1 result and 11.60%± 0.12% on an average recall compared to our new baseline that is an extension of ELMo w/ IDF. In addition to that, if we compare our best results with the original ELMo baseline results, we can observe that we also improved the representation performance increased by 13.56%± 0.15% on a recall@1 and 16.10%± 0.08% on an average recall. Similarly, the retrieval performance for the paragraph side increased by 8.85%± 0.03% on a recall@1 result and 16.00%± 0.07% on an average recall compared to our new baseline that is an extension of ELMo w/ IDF. In addition to that, if we compare our best results with the original ELMo baseline results, we can observe that we also improved the representation performance increased by 7.95%± 0.08% on a recall@1 and 20.20%± 0.12% on an average recall On the other hand, Lquadratic regression conditional didn’t show any improved performance compared to Ltriplet. Last, but not least, during our experiments, we also wanted to compare the quality of our embedding with the embedding derived from USE. Although our best model with the Ltriplet was able to improve the representation performance of the USE document embedding for the retrieval task, it was still far from achieving our state-of-the-art fine-tuned result.

heading: 5 CONCLUSION
text: We developed a new question answering framework for retrieval answers from a knowledge base. The critical part of the framework is text embedding. The state-of-the-art embedding uses the output of the embedding model as the representation. We developed a representation selection method for determining the optimal combination of representations from a multi-layered language model. In addition, we designed deep learning based retrieval models, which further improved the performance of question answering by optimizing the distance from the source to the target. The empirical study using the SQuAD benchmark dataset shows a significant performance gain in terms of the recall. In the future, we plan to apply the proposed framework for other information retrieval and ranking tasks. We also want to improve the performance of the current retrieval models by applying and developing new loss functions.

heading: ACKNOWLEDGMENTS
text: This study used the Google Cloud Computing Platform (GCP) which is supported by Google AI research grant.

Review summaries:
This paper proposes to reformulate the QA task in SQUAD as a retrieval task, i.e., using question as query and paragraphs as candidate results to be ranked.  Authors makes some modifications to elmo model to create better word embedding for the ranking task. Authors have mentioned and are aware of open domain QA methodologies (e.g., DrQA).