Text Embeddings for Retrieval from a Large Knowledge Base

1 INTRODUCTION . The goal of open domain question answering is to answer questions posed in natural language using an unstructured collection of natural language documents such as Wikipedia . Given the recent successes of increasingly sophisticated , neural attention based question answering models such as Yu et al . ( 2018 ) , it is natural to break the task of answering a question into two subtasks as suggested in Chen et al . ( 2017 ) : • Retrieval : retrieval of the paragraph most likely to contain all the information to answer the question correctly . • Extraction : Utilizing one of the above question-answering models to extract the answer to the question from the retrieved paragraphs . In our case , we use the collection of all SQuAD paragraphs as our knowledge base and try to answer the questions without knowing which paragraphs they correspond to . We do not benchmark the quality of the extraction phase , but study the quality of text retrieval methods , and the feasibility of learning specialized neural models text retrieval purposes . Due to the complexity of natural languages , a good text embedding that represents the natural language documents in a semantic vector space is critical for the performance of the retrieval model . We constrain our attention to approaches that use a nearest neighbor look-up over a database of embeddings using some embedding method . We describe the implementation details utilizing the pre-trained embeddings to create semantically more meaningful text embedding improved by utilizing an extra knowledge base , and hence improving the quality of a target NLP retrieval task . Our solution is based on refining existing text embeddings trained on huge text corpora in an unsupervised manner . Given those embeddings , we learn residual neural models to improve their retrieval performance . For this purpose , we utilize the triplet learning methodology with hard negative mining . The first question studied here , is whether having advanced , and refined embedding may provide higher recall for retrieval in the external knowledge base . Therefore we benchmark most popular available embedding models , and compare their performance on the retrieval task . First , we start with a review of recent advances in text embeddings in section 2 . In section 3 we describe our proposed approach . The core idea is to augment precomputed text embedding models with an extra deep residual model for retrieval purposes . We present an empirical study of the proposed method in comparison with our baselines which utilize the most popular word embedding models . We report the results in section 4 . Finally , we conclude the paper with some future work in section 5 . 2 RELATED WORK . There are various type of applications of word embedding in the literature that was well covered by Perone et al . ( 2018 ) . The influential Word2Vec by Mikolov et al . ( 2013 ) is one of the first popular approaches of word embedding based on neural networks that was built on top of guiding work by Bengio et al . ( 2003 ) on the neural language model for distributed word representations . This type of implementation is able to conserve semantic relationships between words and their context or in other terms surrounding neighboring words . Two different approaches are proposed in [ 28 ] to compute word representations . One of the approaches is called Skip-gram that predicts surrounding neighboring words given a target word and the other approach is called Continuous Bag-of-Words that predicts target word using a bag-of-words context . As a following study , Global Vectors ( GloVe ) by Pennington et al . ( 2014 ) , points to reduce some limitations of Word2Vec by focusing on the global context instead of surrounding neighboring words for learning the representations . The global context is calculated by utilizing the word co-occurrences in a corpus . During this calculation , a count-based approach is functioned unlike the prediction-based method in Word2Vec . On the other hand , FastText , by Bojanowski et al . ( 2017 ) , is also announced recently . It has the same principles like others that focus on extracting word embedding from large corpus . FastText is very similar to Word2Vec except handling each word as a formation of character n-grams . That formation accelerate FastText to learn representation more efficiently . Nonetheless , the important question still exists on extracting high-quality and more meaningful representations : How to seize the semantic , syntactic and the different meanings in different context . This is also the point where our journey is getting started . Embedding from Language Models ( ELMo ) , by Peters et al . ( 2018 ) , was newly proposed in order to tackle that question . ELMo extracts representations from a bi-directional Long Short Term Memory ( LSTM ) , by Hochreiter & Schmidhuber ( 1997 ) , that is trained with a language model ( LM ) objective on a very large text dataset . ELMo representations are a function of the internal layers of the bi-directional Language Model ( biLM ) that outputs good and diverse representations about the words/token ( a CNN over characters ) . ELMo is also incorporating character n-grams like in FastText but there are some constitutional differences between ELMo and its predecessors . The reason why we focused on ELMo is that it can generate more refined and detailed representations of the words due to utilizing internal representations of the LSTM network . As we mentioned before , ELMo is built on top of a LM and each word/token representation is a function of the entire input which makes it different than the other methods and also eliminates the others ’ restrictions where each word/token is a mean of multiple contexts . There are some other embedding methods which are not used as a baseline in the field ; therefore , we did not get into them in detail . Skip-Thought Vectors , by Kiros et al . ( 2015 ) , uses encoderdecoder RNNs that predicts the nearby sentence of a given sentence . The other one is InferSent , by Conneau et al . ( 2017 ) , that utilizes a supervised training for the sentence embeddings , unlike as in Skip-Thought . They used BiLSTM encoders to generate embeddings . p-mean , by Rücklé et al . ( 2018 ) , is built to tackle the unfairness of the Skip-Through that is taking the mean of the word/token embeddings with different dimensions . There are some other more methods such as Le & Mikolov ( 2014 ) ’ s Doc2Vec/Paragraph2Vec , Hill et al . ( 2016 ) ’ s fastSent , and Pagliardini et al . ( 2018 ) ’ s Sent2Vec . Furthermore , we also include the newly-announced Universal Sentence Encoder by Cer et al . ( 2018 ) in our comparisons . Universal Sentence Encoder is built on top of two different models . Initially , the Transformer-based attentive encoder model , by Vaswani et al . ( 2017 ) , was implemented to get high accuracy and the other model takes the benefit of deep averaging network ( DAN ) , by Iyyer et al . ( 2015 ) . As a result , averages of words/token embeddings and bi-grams can be provided as input to a DAN where the sentence embeddings are getting computed . According to the authors , their sentence level embeddings exceeds the performance of transfer learning using word/token embeddings alone . Last , but not least , distance metric learning is designed to amend the representation of the data in a way that retains the related points close to each other while separating different points on the vector space as stated by Lowe ( 1995 ) , Cao et al . ( 2013 ) , and Xing et al . ( 2002 ) . Instead of utilizing a standard distance metric learning , a non-linear embedding of the data using deep networks has shown an important improvements by learning representations using triplet loss by Hadsell et al . ( 2006 ) , Chopra et al . ( 2005 ) , contrastive loss by Weinberger & Saul ( 2009 ) , Chechik et al . ( 2010 ) , angular loss by Wang et al . ( 2017 ) , and n-pair loss by Sohn ( 2016 ) for some of influential studies presented by Taigman et al . ( 2014 ) , Sun et al . ( 2014 ) , Schroff et al . ( 2015 ) , and Wang et al . ( 2014 ) . After providing brief review of the latest trends in the field , we describe the details of our approach and experimental results in the following sections . 3 PROPOSED APPROACH . 3.1 EMBEDDING MODEL . After examining the latest advanced work in word embedding , and since ELMo outperforms the other approaches for all important NLP tasks , we included the pre-trained ELMo contextual word representations into our benchmark to find out the embedding model for the retrieval task that produces the best result . The pre-trained ELMO model used the raw 1 Billion Word Benchmark dataset Chelba et al . ( 2013 ) , and the vocabulary of 79347 tokens . By having all the requirements set , it was time to compute ELMo representations for the SQuAD dev set using the pre-trained ELMo model . Before explaining the implementation strategies , we would like to mention the details of ELMo representation calculations . As they showed in their study , a L-layer bILM computes a set of 2L + 1 representations for each token ti . Li = { xLMi , −→ h LMi , j , ←− h LMi , j |j = 1 , ... , L } = { ←→ h LMi , j |j = 0 , ... , L } where xLMi , a context-indepedent token representation , is computed via a CNN over characters . Therefore hLMk,0 is representing the token layer . These representations are then transmitted to L layers of forward and backward LSTMs ( or BiDirectional LSTMs-biLSTMs ) . Each LSTM layer provides a context-dependent vector representation ←→ h LMi , j where j = 1 , ... , L. While the top layer LSTM results , −→ h LMi , j , presents the prediction of the next token ti+1 , likewise , ←− h LMi , j represents the prediction of the previous token ti−1 by using a Softmax layer . The general approach would be a weighting of all biLM layers to get ELMo representations of the corpus . For our task , we generated the representations for the corpus , not only considering the results from the weighting of all biLM layers ( ELMo ) , but also other individual layers such as token layer , biLSTM layer 1 and biLSTM layer 2 . As a result , we build the tensors based on tokens instead of documents ; but documents can also be sliced from these tensors . To this end , we created a mapping dictionary for keeping the document ’ s index and length ( number of tokens ) . In this way a token is represented as the tensor of a shape [ 1 , 3 , 1 , 1024 ] . To improve the traceability , the tensors were reshaped by swapping axis to become the tensor of a shape [ 1 , 3 , 1024 ] , then we stacked all these computed tokens to end up with the tensor of a shape [ 416909 , 3 , 1024 ] where 416909 is the number of tokens in the corpus ( in other words , 416909 represents the 12637 documents in the corpus ) . Therefore , any document could be extracted from this tensor using the previously-created mapping dictionary . For example , let us assume that doc1 has an index of 126 and a length of 236 , so doc1 can be extracted by slicing the tensor using such a format [ 126 : 126 + 236 , : , : ] . In terms of math expression , we ended up building a tensor Ti , j , d where i is representing the total documents in the corpus , j is representing the layers , and d is a dimension space of a vector . T is based on tokens instead of documents , but documents can also be sliced from that tensor . To achieve a good embedding , the main question was about identifying the most critical components that contain the most valuable information for the document . Since the new tensor of a shape [ 12637 , 3 , 1024 ] has 3 components , in order to extract the most valuable slice ( s ) , we defined a new weight tensor W1 , j,1 where the j had a same dimension as in the tensor T. For our experiment , we set the j to 3 as the same dimension value in which the pre-trained ELMo word representations were used . The elements of the vector j of the tensor W was symbolized as a , b , and c , where a+ b+ c = 1 . In order to find the right combination within the elements of the weight tensor W for the best embedding matrixM , we calculated the following function : M ′ = ||mean ( T ∗W ; θ = 1 ) || whereM ′ = { M ′1 , M ′2 , . . . , M ′n } , W ′ = { W′1 , W ′ 2 , . . . , W ′ n } , n is the number of documents in the set and θ represents the axis argument of the function . Last , but not least , all candidate embedding was normalized by L2-Norm . With this setup , we were able to create the candidate embedding matrices represented by M ′ for further experiments to get the ultimate best embedding E. Each embedding in the embedding matrix is symbolized by f ( x ) ∈ Rd . Therefore , it was able to embed a document x into a d-dimensional Euclidean space . In order to capture the best combination of W′ ( in other words , finding the best a , b , and c values ) , we calculated the pair-wise distances between questions and paragraphs embedding represented Q′ and P ′ respectively that were sliced from M ′ . The performance of finding the correct questionparagraph embedding pair is measured by recall @ k. The reason why we used the recall @ k is that we wanted to compute the hitting performance of the true paragraph ( answers ) to the particular question whether it is in the top-k closest paragraphs or not . We used the ranks of 1 , 2 , 3 , 5 , 10 , and , 50 . In order to calculate the recall values , we had to deal with 20M question-paragraph embedding pairs ( 10K questions x 2K answers ) in the dev dataset . We primarily aimed to look at the Receiver Operating Characteristic ( ROC ) curve and using Area Under Curve ( AUC ) as the target metric for tuning the embedding . Therefore , we sorted the retrieved samples by their distance ( the paragraphs to the question ) and then we computed recall and precision at each threshold . After having the precision numbers from the computations , we observed that our precision numbers became very low values because of having only 10k correct pairs and a high amount of negative paragraphs . We decided to switch to other methods called Precision-Recall curve and Average Precision . Additionally , we took one step further . We wanted to inject the inverse document frequency ( IDF ) weights of each token into the embedding M to observe whether it creates any positive impact on the representation or not . Since the IDF is calculated using the following function : IDF ( t ) = loge ( # ofdocuments # ofdocs.w/term ) , we calculated the IDF weights from each tokenized documents.Then we multiplied newly-computed token IDF weights with the original token embedding extracted from the pre-trained model to have the final injected token embedding . Similar to both the ELMo and extension of ELMo with IDF weights steps , we followed the same pipeline for the GloVe since it is one of the very important baselines for this type of research . We specially utilized the ” glove-840B-300d ” pre-trained word vectors where it was trained on using the common crawl within 840B tokens , 2.2M vocab , cased , 300d vectors . We created the GloVe representation of our corpus and also extended it with IDF weights . Fully Connected Residual Retrieval Network ( FCRR ) . Convolutional Residual Retrieval Network ( ConvRR ) .

Summary: This paper proposes to reformulate the QA task in SQUAD as a retrieval task, i.e., using question as query and paragraphs as candidate results to be ranked.  Authors makes some modifications to elmo model to create better word embedding for the ranking task. Authors have mentioned and are aware of open domain QA methodologies (e.g., DrQA).