A Comparative Study of Models for Answer Sentence Selection
Authors: Federico Rossetto, Alessio Gravina, Silvia Severini, Giuseppe Attardi (Università di Pisa, 2019)
Topic: Language Models

Abstract
Answer Sentence Selection is one of the steps typically involved in Question Answering. Question Answering is considered a hard task for natural language processing systems, since full solutions would require both natural language understanding and inference abilities. In this paper, we explore how the state of the art in answer selection has improved recently, comparing two of the best proposed models for tackling the problem: the Cross-attentive Convolutional Network and the BERT model. The experiments are carried out on two datasets, WikiQA and SelQA, both created for and used in open-domain question answering challenges. We also report on cross-domain experiments with the two datasets.

Introduction
Answer Sentence Selection is an important subtask of Question Answering, that aims at selecting the sentence containing the correct answer to a given question among a set of candidate sentences. Table 1 shows an example of a question and a list of its candidate answers, taken from the SelQA dataset (Jurczyk et al., 2016). The last column contains a binary value, representing whether the sentence contains the answer or not. Answer extraction involves natural language processing techniques for interpreting candidate sentences and establishing whether they relate to questions and contain an answer. More sophisticated methods of Answer Sentence Selection that go beyond Information Retrieval approaches involve, for example, tree edit models (Heilman and Smith, 2010) and semantic distances based on word embeddings (Wang et al., 2016). Recently, Deep Neural Networks have also been applied to this task (Rao et al., 2016), providing performance improvements with respect to previous techniques. The most common approaches exploit either recurrent or convolutional neural networks. These models are good at capturing contextual information from sentences, making them a nice fit for the problem of answer sentence selection. Research on this problem has benefited in the last few years by the development of better datasets for training systems on this task. These datasets include WikiQA (Yang et al., 2015) and SelQA (Jurczyk et al., 2016). The latter is notable for its larger size, that reaches more than 60,000 sentence-question pairs. This allows for the creation of deeper and more complex models, with less risk of overfitting. The state of the art model on the SelQA dataset (Jurczyk et al., 2016), up to 2018, was Cross-attentive Convolutional Network (Gravina et al., 2018), with a score of 0.906 MRR (Craswell, 2009). In this paper, we present further experiments with the Cross-attentive Convolutional Network model as well as experiments that exploit the BERT language model by Devlin et al. (2018). In the following sections, we survey relevant literature on the topic, we describe the datasets used in our experiments and present the models tested in our experiments. Finally, we describe the experiments conducted with these models and report the results achieved.

Conclusion
We compared the Cross-attentive Convolutional Network and several BERT-based models on the task of Answer Sentence Selection on two datasets. The experiments show that a BERT model, fine-tuned on an Answer Sentence Selection dataset, improves significantly the state of the art, with a gain of 5 to 9 points of MRR score on SelQA and WikiQA respectively. As a drawback, this approach takes a considerable amount of time to be trained even on GPUs. The BERT-Base model without fine-tuning achieves almost the same accuracy as the CACN with GloVe embeddings, which uses a much smaller number of parameters in the model. The CACN also requires less data to train. On the other hand, BERT is quite effective at leveraging the knowledge collected from large amounts of unlabeled text, and at transferring it across tasks. We also evaluated the abilities of CACN at transfer learning. BERT is a model that has been pre-trained on a large corpus, while CACN leverages the GloVe embeddings as a starting point for the training. We also exploited the WikiQA and SelQA datasets in a cross-domain experiment using CACN. We found that the model maintains a good score across domains, with a transfer score of about 83% from SelQA to WikiQA. We confirmed that the SelQA dataset is not easily solvable using simple word-occurrences methods like a logistic regression classifier on word count features. BERT models confirmed their superiority to previous state-of-the-art models for the task of Answer Sentence Selection. This was to be expected since they perform quite well also on the more complex task of Reading Comprehension, which requires not only to select a sentence but also to extract the answer from that sentence.