QUADRo: Dataset and Models for QUestion-Answer Database Retrieval
Authors: Stefano Campese
Ivano Lauriola
Alessandro Moschitti
Venue: EMNLP 2023 Findings (2023)
Keywords: question answering
semantic similarity
nlp application
question answering database
question answering resources
question ranking and retrieval
TL;DR: This paper proposes an effective paradigm for building Question Answering systems by using previously answered questions, achieving competitive performance with other QA methods, and providing data and models for future research.

Abstract: An effective approach to design automated Question Answering (QA) systems is to efficiently retrieve answers from pre-computed databases containing question/answer pairs. 
One of the main challenges to this design is the lack of training/testing data. Existing resources are limited in size and topics and either do not consider answers (question-question similarity only) or their quality in the annotation process. 
To fill this gap, we introduce a novel open-domain annotated resource to train and evaluate models for this task. The resource consists of 15,211 input questions. Each question is paired with 30 similar question/answer pairs, resulting in a total of 443,000 annotated examples. The binary label associated with each pair indicates the relevance with respect to the input question. 
Furthermore, we report extensive experimentation to test the quality and properties of our resource with respect to various key aspects of QA systems, including answer relevance, training strategies, and models input configuration.

QUADRo: Dataset and Models for QUestion-Answer Database Retrieval
Stefano Campese University of Trento
Ivano Lauriola Amazon Alexa AI
Alessandro Moschitti Amazon Alexa AI
Amazon Alexa AI
lauivano@mmazon. com
amosch@amazon.com com
stefano. campmeeeespseenii.....
Abstract
these approaches have recently shown impressive performance, their execution places substantial strain on system resources, making their adoption unpractical for industrial applications (Chen et al., 2023; Wang et al.. 2023). A third approach that started to become popular in the last years consists in retrieving the answer from DataBase of pre- generated question/answer pairs rather than lever- aging knowledge stored in the models' parameters or large corpora. These systems typically employ components: (i) a DataBase questions and their answers, (ii) a retrieval model to query the DataBase and to retrieve relevant content for an input question, and (iii) ranking model to select the final answer. Recent literature showed various benefits of these approaches, here named DataBase QA (DBQA), including fast inference and the abil- ity introduce new knowledge by simply adding new pairs without re-training models.
An effective approach to design automated Question Answering (QA) systems is effi- ciently retrieve answers from pre-computed datahases containing question/answer pairs. One of the main challenges this design is the lack of training/testing data, Existing resources are limited in size and topies and either do not consider answers (question-question similar- ity only) or their quality the annotation pro- cess. To fill this gap, introduce novel open-domain annotated resource to train and evaluate models for this task. The resource con- sists of 15,211 input questions. Each question is paired with 30 similar question/answer pairs, resulting total of 443,000 annotated exam- ples. The binary label associated with each pair indicates the relevance with respect to the in- put question. Furthermore, we report extensive experimentation test the quality and prop- erties of our resource with respect to various key aspects QA systems, including answer relevance, training strategies, and models input configuration.
One of the main issues with DBQA approaches the lack of large training data support the devel- opment of retrieval and ranking models. Only lim- ited resources exist in the litcrature and often mod- els are indeed trained on general tasks, including paraphrasing or semantic similarity (Lewis et al.. 2021). To fill this gap, we introduce novel large annotated dataset to train and evaluate models for question ranking and DBQA. Our resource consists of 15,211 open-domain input questions. For each input question, we annotated 30 relevant q/a pairs and we marked them as semantically equivalent or not (binary label) with respect to the input. total, the dataset comprises Rs 143, 000 annotated exam- ples. We selected q/a pairs using DBQA model trained on publicly available data. This allowed us to casily integrate labeled hard negatives, making the resource significantly complex and challenging. Differently from existing resources mainly based on question-question pairs, our dataset annotated both with respect to dnestion-questiie similarity and dnestion-answer correctness. To the best
1 Introduction
Recently, two main QA paradigms have gained more and more attention from both the industrial and research community: open- and closed-book. The former paradigm, also known retrieve-and read refers to set of models and techniques that, given input question, follow two-step process. First, they retrieve relevant content from large cor- pus of documents, e.g. from the web. Second, they extract the answer from the retrieved content (Yang et al., 2015; Nguyen et al., 2016; Thorne et al., 2018). The latter approach, closed-book, is rela- tively new and it consists of models that rely on knowledge acquired during training, and they gen- erate answer without accessing external corpora. Typically, these techniques are based on Seq2Seq models such as T5 (Raffel et al., 2020). Although
Work done as an intern at Amazon Alexa Al

---

our knowledge, this resource is the largest anno- tated dataset for training DBQA models. We per- formed extensive experiments, which provide base- lines and a comprehensive analysis of distinctive aspects, weaknesses, and benefits of the dataset.
query the database (e.g. BM25 or DPR) for finding an equivalent question, and (iii) an optional ranking component.
Early works on DBQA for forum or FAQ (Nakov et al., 2016; Shen et al., 2017: Hoogeveen et al., 2015) have pointed out that when answers are avail- able in the DataBase together with the questions, the resulting system can be very accurate. The main challenge to their general usage is the typ- ical specificity of the DB, also associated with limited availability of q/a pairs. Othman et al. (2019) introduced WEKOS, system able iden- tify semantically equivalent questions from a FAQ database. The model, based on k-means cluster- ing, Word-Embeddings (CBoW), and heuristics, was tested on a dataset based on Yahoo Answer, showing impressive performance compared to the existing systems. After the rise of Transformer models, Mass et al. (2020) proposed ble system that combines BM25 with BERT (De- vlin et al., 2019) for retrieval and ranking. The same authors also explored systems based on a GPT2 model to generate question and/or answer paraphrases to fine-tune the BERT models on low- resource FAQ datasets with promising results. Sim- ilarly, Sakata et al. (2019) proposed method based BERT and TSUBAKI, efficient retrieval chitecture based on BM25 (Shinzato et al., 2012), to retrieve from a database of FAQ the most simi- lar questions looking at query-question similarity. However, most of these works and practical applica- tions were confined to specific domains, including domain-specific FAQ or Community QA.
As second contribution of this paper, we use the annotated resource to train models for DBQA. Then, we build standard DBQA pipeline, named QUADRo (QUestion Answer Database Retrieval) based on these models and we test in various open domain QA settings. Our experiments aim to address a critical gap in the existing literature by providing consolidated evidence regarding some key aspects of DBQA end-to-end pipelines. For instance, despite the existing evidence support- ing the contribution of the answer to the retrieval task (Wang al., 2020), its utilization remains poorly explored in DBQA applications (Seonwoo al., 2022: Chowdhary et al., 2023).
ensem-
Our annotated dataset and our models will be available to the research community'.
2 Related Work
Question Similarity and Ranking Duplicate Question Detection (DQD) is well known prob- lem under the umbrella of Semantic-Textual Similarity (STS) tasks. It aims at identifying when two questions are semantically equivalent or not. Early approaches focused on the extraction and cre- ation of several types of lexical (Cai et al., 201 1), syntactic (Moschitti, 2006) and heuristic (Filice et al., 2017) features to measure the similarity be- tween two questions. Lately, translation- and topic- based modeling approaches have been explored, e.g., (Wu and Lan, 2016). DQD received a huge boost with the advent of embedding representations such as Word2Vec. Vec, Glove (Charlet and Damnati, 2017), and ELMO (Peters et al., 2018) to compute sentence-level embeddings for the two individual questions (Fadel et al., 2019).
Recently, effective open-domain DBQA pipelines have been shown. Lewis et al. (2021) assembled a DataBase of 65M q/a pairs auto- matically generated from the web. Then, they employed neural retrieval and ranker based on Transformer models to query the DataBase. The authors compared this end-to-end DBQA pipeline, named RePAQ, against various closed- and open-book QA strategies, showing interesting performance in terms of efficiency and competitive accuracy. However, there are some limitations of that system as (i) models are not trained on annotated data for question ranking due to lack of available resources and (ii) the DataBase is noisy (estimated 18% incorrect q/a pairs). As an extension of this approach, Seonwoo et al. (2022) proposed 2 steps retrieval based on BM25 and DPR executed sequence to improve the
More recently, pretrained Transformers set the state of the art for STS and DQD (Chandrasckaran and Mago, 2021). Peinelt al. (2020) proposed (BERT a Transformer model that takes the concate- nation of the two questions as input, providing a joint and contextualized representation of the pair.
DataBase QA is a known paradigm that typically relies on (i) a curated database (or collection) of questions and their answers, (ii) retrieval model to
'https://iithub.. cww. wwt.://....ccccceee//ea ataaaaamazaaiaaaaaaaaa science/ question-ranking

---

efficiency. However, most of the technical details are missing, including the DataBase used and training mechanisms. is worth noticing that these end-to-end systems rely solely on models trained for question similarity, without providing empirical evidence of the advantages gained from incorporating answers to enhance accuracy.
3 DataBase QA architecture
Inspired by previous work (Seonwoo et al., 2022; Lewis et al.. 2021), we consider retrieval- reranking DBQA architecture consisting of large- scale database of questions and correct answers, an efficient search engine to query the database, and an answer selector (ranker). In the remainder of the paper, we call this DBQA system QUADRo (QUestion Answer Database Retrieval).
Datasets and resources Various resources have been made available to train models for DQD and DBQA, One of the most popular resources is the QuoraQP dataset. This consists of 404,290 pairs of questions extracted from the Quora website, and annotated as having the same meaning or not. An extension of the original dataset was released by Wang et al. (2020), which consists of the original question pairs concatenated with answers to the sec- ond questions, extracted from the original Quora threads. Another popular resource the CQADup- Stack (Hoogeveen et al., 2015) dataset, originally released as benchmark for Multi-Domain Com- munity QA. It consists of questions coming from different threads sampled from twelve StackEx- change subforums. The dataset contains annota- tions for similar and related questions. For small portion of question pairs, there is an annotated an- swer. On average the dataset contains R 5.03% of duplicated questions. Next, WikiAnswers was built by clustering together questions classified as paraphrases by WikiAnswers users. The dataset consists of 30,370,994 clusters containing an aver- age of 25 questions per cluster. Unfortunately, the answers large paragraphs, which are not neces- sarily suitable for DQD. SemEvval-2016 Task 3 chal- lenge (Nakov al., 2016) introduced another fa- mous resource for DQD in community QA, where Question Question-External Comment, and Question-Question Similarity, are annotated. Although this dataset can be used in reranking setting, the amount of queries is limited. Moreover, all the data is extracted from specific domains.
Search Engine This is based on recent find- ings in neural retrieval, including Dense Passage Retrieval (Karpukhin al., 2020). It consists of siamese Bi-Encoder Transformer (Reimers and Gurevych, 2019) (also known as Sentence- Transformer) network. The first branch encodes the input question as [CLS] [EOS], whereas the second branch encodes question/answer pairs (g, a,) from the database as [CLS] qi [SEP] [EOS]. The cosine similarity between the represen- tations extracted from the two branches expresses the level of semantic similarity between the ques- tion and the q/a pair. Let ò : Σ+ + Rd be the Transformer function which maps a text (cither a question qi E Q, an answer a; E A, or a pair (q: as) E Q x A) into a d-dimensional embed- ding. The final score assigned to target question t E Q and an element of the database (gi, ..) is 8(0) T(0T88(a)))) where || denotes the 2-norm of a [+te)]||0 vector. When the user asks a new question, the bi- encoder computes the cosine distance between the question embedding and all q/a pairs and returns the most similar k pairs.
Ex
Answer Selector After the retrieval stage, an answer selector (or reranker) model re-ranks the pairs returned by the search engine and selects the final answer to be served to the user. For- mally, let R {(q;, at)] be the set of k re- turned pairs for given target question t. The answer selector r Q
+ E. assigns a score to each triplet, r(1, qb, a:). and returns the answer associated with highest ranked pair, i.e., arg max;=1...k r(t, qi, a;). Inspired by the Contex- tual Sentence Selector framework (Lauriola and Moschitti, 2021), which is the state of the art for open-domain answer selection, we former model encode and rank triplets. The in- put of the transformer is encoded as /CLSJ (SEP) a; [SEP] 9: [EOS]. Note that this allows to jointly modeling the semantic dependencies between the two questions, e.g., their similarity, and the rele-
In general, the resources above have some limi- tations: (i) the majority of datasets do not include answers, (ii) there are no guarantees on the qual- ity, and (iii) with the exception of SemEval, these resources are based on pairs of questions (e.g., Quo- raQP) rather than question ranking, preventing the possibility study search and ranking problems. Our annotated dataset instead enables research on large-scale retrieval of semantically equivalent questions, associated with correct answers, which are essential to build large-scale DBQA models.
Trans-

---

vance of the answer both questions.
Source
QA
Q
Q length
A length
Labeled
The DataBase The DataBase employed in this work consists of questions and their answers col- lected from various heterogeneous public high- quality annotated open ppen-domain QA datasets, in- cluding: GooAQ, WQA (Zhang et al., 2021), WikiAnswer (Fader al., 2014), CovidQA (Möller al., 2020), and HotporQA (Yang al., 2018). We extracted questions and the associated correct answer text span from these datasets, and ingested these q/a pairs into our DataBase.
GooAQ
3.1M
2.9M
9.1423
45.9 45.9218.9
WQA
391K
80.5K
7.513.2
24.8111.
WikiAnswer
2.3M
2.3M
9.1
60.311173
CovidQA
2K
1.9K
10.6
15.8g171
HotpotQQ
64K
64K
20.4
4.1424
Unlabeled
Quora Match
230K
170K
12.5=6.7
38, 28.8420.5
QuoraQP
219K
134K
9.6
25.1
EL.15
58.9K
58.7K
17.6
60.7
Total
6.3M
5.7M
Table I: Main statistics of QUADRo database, QA= q/a
Beyond annotated resources, we enhanced our database with various sets of artificially generated q/a pairs. First, we considered questions from Quo- raQP. Differently from the other datasets above, QuoraQP is designed for question duplication de- tection task and not QA. Thus it simply consists of question pairs. Some answers are available for a small fraction of questions, which were selected through a hcuristic approach based on the rank of users' content in Quora threads (Wang et al., 2020). To expand the QuoraQP, we collected the miss- ing answers using a similar approach described by Zhang al. (2021): given an input question, we queried a 2020 CommonCrawl snapshot us- ing BM25 and we selected the 200 most relevant documents. Then, documents are split into sen- tences and a state-of-the-art sentence selector (Lau- riola and Moschitti, 2021) is used to select the top- ranked passage as the answer. In addition, the score returned by the passage reranker model can be ef- fectively used to measure the likelihood of the an- swer being correct. We applied threshold to this score and accepted only the top 10% of q/a pairs. This guarantees higher level of answer quality in our database. We manually labeled 200 randomly selected answers from this unsupervised set as qual- ity assessment, observing an accuracy of 93.0%. Finally, we also ingested q/a pairs from ELI5 (Fan et al., 2019). This dataset consists of questions collected from three subreddits for which the an- swers have been ranked by the users' up-votes in the thread. Although this heuristic removed part of the noise of the dataset, to ensure the maximum quality we keep only the 50% of the q/a pairs with the highest sentence selector score. After a man- ual annotation, we estimate the 84.3% of accuracy. Our final DB contains 2 6.3 millions of English questions and their correct answers pairs. Further details and statistics are reported in Table 1. It
pairs, Q= unique questions.
is worth noticing that this DataBase can be com- bined with other Existing DataBases, including PAQ. However, we did not use those resources for two main reasons. First, PAQ authors stated that the resource contains considerable amount of noise and errors (only 82% correctness). Second, PAQ consists of generated questions that are, sometimes, unnatural. In the next section, we describe how we used this DataBase to build our annotated resource. Thus, we prefer data quality and correctness over possible coverage/recall.
Annotated Dataset
As Section 2 highlights, cxisting resources are not suited to train models for DBQA. In this section, we describe the annotation workflow, emphasizing the main features and key aspects.
4.1 Annotation workflow
We randomly selected 15,211 questions from our DataBase, and removed them from it to avoid bias for the next experiments. For each of the questions above, we ran the QUADRo pipeline described in Section 33 and retrieved k most similar questions and their answers. Based on the annotation budget and preliminary observations, we set 30 balance high diversification (15,211 diverse input questions) and recall (see Figure 1). We used Ama- zon Mechanical Turk (AM')) to annotate retrieved triplets (t, qi, ai), target question, retrieved ques- tion, and answer as correct/incorrect, where this binary label represents the semantic equivalence not, respectively, between and qi-
Given the complexity and partial ambiguity of the annotation task, we ensured the high quality
'We descrihe the configuration of the search engine used to collect q/a pairs in Section 5.

---

Split
QA
pos/neg QA
rect. We ran an independent annotation batch evaluate the performance of the annotators and quantify the benefits of including the answer into the annotation interface. Our manual evaluation of 200 annotations showed that adding the answer in the annotation workflow reduces the relative error rate for this task up to 45%, leading to absolute accuracy of 94%. In other words, the answer is precious source of information which, beyond the modeling part where previous work already showed the importance of the q/a relevance, can significantly help the human judgment and thus the quality of the dataset. To our knowledge, our is the first resource for question-question similarity ex- ploiting the answer context in order to perform accurate annotation.
Train
11711
28.9+10.3
6.1+7.9
11..
Dev. 1500
30
4.825.3 / 225.245.3
Test
2000
30
4.725.1 22..3
Table 2: Data splits. Q:Aqueries, QA:q/a pairs per query.
of our annotation by applying several procedures, some of which are completely novel. First, we used two distinct annotators for each triplet, and third one to resolve the tie cases, thus the final label is computed with the majority vote. Second, limited the annotation turkers with documented historical annotation activities, including master turkers with at least 95% approval rate and 100 ap- proved tasks. Third, we provide the following defi- nition of equivalence to the annotators: two ques- tions are equivalent iff they (i) have the same mean- ing AND (ii) share the same answers. While ex- pressing the same meaning is theoretically enough assess the semantic equivalence of the questions. in practice, there are several sources of ambiguity. e.g., well known that the meaning depends on the context, and the latter may be underspecified. The answer instead provides a strong context that can focus the interpretation of the annotator on the same meaning of both questions*. Finally, we intro- duced set of positive and negative control triplets that we used to filter bad annotations.
Given all precautions taken to improve the qual- ity of the data (MTurk filters, answer exposure, con- trol triplets), we observed an agreement between the first two annotators on 78% triplets (random labeling corresponds to 50% agreement). We also measured the Cohen's kappa inter-annotator metric (Artstein and Poesio, 2008) to be 0.875, indicat- ing good agreement quality. We do not have data showing the agreement between the annotators as the third annotator was considered only when there was no agreement between the first two annotators. The resulting dataset consists of 15,21 unique in- put questions and 443K annotated triplets. The answerability, i.e. the number of input queries with at least one positive match in the set of associated q/a pairs, is 75.4%. On average, each question has 5.8 out of 30 positive pairs. We split our dataset into training, development, and test. Details of the split are shown in Table 2.
Each annotation task consists of 7 triplets of which control triplets, one negative and one posi- tive. Positive control triplets are designed to be as clear as possible and sufficiently simple and nega- tive triplets are automatically defined by selecting two random questions. Answering incorrectly to at least one of the control pairs caused the rejec- tion of the annotation task. Moreover, if the same turker failed more than 10% of the total assigned HITs, all their HITs are discarded and the turker is blacklisted, precluding the exccution of further annotations. Guidelines, examples, and further de- tails are shown in the Appendix C. Some annotation anecdotes are shown in Appendix D.
5 Experiments
We ran various distinct sets of experiments to as- sess the quality and challenges of our dataset and to show the empirical performance of state-of-the-art models in DBQA tasks. First, we analyze the re- trieval and ranking tasks thanks to our new dataset. Then, we use this resource to fine-tune and evalu- ate retrieval and ranker models. Finally, com- pare our approach against strong Web-based open- domain QA systems. As we previously mentioned. our experiments aim to empirically clarify and re- port some design choices of DBQA systems includ- ing, for instance, the modeling of the answer.
4.2
Dataset analysis
One source of complexity was given by the fact that annotated triplets do not have 100% accurate answers as the internal accuracy of q/a pairs in the DataBase is 93%. Thus, we clarified with the annotators that they should use answers to help their judgment, but these are not necessarily cor- 'A few examples are reported in the appendix.

---

5.1 Model Training
We used our dataset to train and evaluate both, re- trieval and ranking models.
HUTTRate Rate
built the bi-encoder (de- scribed in Section 3) with RoBERTa continuously pre-trained on ~ 180M semantic text similarity pairs We considered the following two input configurations: QQ: the model encodes the input question, t. and the evaluation questions in the first and second transformer branches (with shared weights). QQA: the model encodes in the first branch, and the concatenation of the qi and o; in the second one. After this procedure, the model computes the cosine similarity between the two generated representations. We first fine-tuned the models on QuoraQP with artificially generated an- swers as previously described. Then, we fine-tuned the resulting model on our ranking dataset.
Retrieval Models:
60
Retrieval Ranker
10
15
20
25
30
K
Figure 1: Hit rate the retrieval module and the end- to-end system (QUADRo).
with QAQ configuration to be tested on our col- lected dataset. Figure 1 shows the Hit-rates at k defined as the portion of questions with at least one positive in the top k: returned pairs, of the sole retrieval component and the Hit-rate of the entire system, i.e., after the reranking performed by the answer selector, applied to the 30 retrieved (and annotated) answers.
Reranking Models: start from the state-of- the-art sentence selector model proposed by Lau- riola and Moschitti (2021). The model consists am Electra-base trained on ASNQ on triplets of the form [CLS] question [SEP] answer [SEP] context [EOS], where context refers to additional sentences that are relevant for the answer. The checkpoint then fine-tuned on QuoraQP and lately on our dataset. As in our case the a, can be considered as the context of both or qı or alterna- tively, qe may be considered the context of and az. Therefore, the use of a pre-trained model with con- text is promising. Similarly to the retrieval model training, we considered various configurations, in- cluding (i) QQ only using (t, ④)) pairs; (ii) QAQ corresponding to (t, ai q:). where the question acts context for the answer, (iii) QQA correspond- ing to (l, qi a.). where aj is the context, and (iv) QA encoding (t, a,), i.e., a standard sentence se- lector for QA. For both, retricval and reranker the model score ranks the q/a pairs with respect to each input question.
We note that: First, as we used the retrieval men- tioned above to select 30 question/answer pairs to be annotated in our dataset the plot exactly indi- cates the retrieval performance. Since the ranker operates just the 30 annotated candidates, can also evaluate its impact on an end-to-end sys- tem in an open-domain QA setting. This means that the accuracy of our DBQA system (Hit@) 1) in answering open domain questions sampled (and removed) from the DB is 46.6%, as indicated in the picture. Second, as expected, the larger the k, the higher is the probability of retrieving good answer. Using 30 candidates, the retrieval is able to find an answer for 75.4% of the questions. The reranker boosts the retrieval performance by 4% absolute (42.6 vs 46.6). Then, it consistently improves the retrieval. Note that configuration entirely based on the answer selector (i.c. without retrieval) is infeasible due to the complexity of the model as it needs to run Transformer for each stored pair (6.3M). Finally, there is considerable gap between the system accuracy (i.c.: Hit@ 1) and the potential accuracy (Hit@ 30). This suggests that dataset enables challenging future research. Also, there is significant amount of unanswerable queries, which opens other research challenges on how learning to not answer to improve system FI.
5.2 DBQA performance
As discussed in Section 4.1, we used an initial re- trieval model to select the q/a pairs that compose the question reranking dataset. Based on prelim- inary experiments conducted on existing bench- marks, we used Sentence-RoBERTY (base) model. The model is trained on QuoraQP with QQA con- figuration. Similarly, we trained an initial reranker
'See appendix A for further pre-training details.

---

5.3 Retrieval and reranker evaluation
Model
P@1
MAP
MRR
*S R--BBBRTaQQ4
39.1
39.1
50.4
We fine-tuned our models on the collected dataset described in 5.1 and evaluated them the test split. Details of the fine-tuning step are described in Appendix B. Table 3 reports the performance of (i) "S-RoBERTagqq4= the initial neural retrieval model used to select annotation data (and thus not trained on this resource), (ii) S-RoBERTay. our retrieval models fine-tuned on the collected resource, and (iii) Electra,: the fine-tuned rerank- ing models. As mentioned before, these models be applied to the top 30 candidates retrieved by "S-RoBERTagqq. which all annotated thus enabling their easy evaluation. The selection of RoBERTa for retricval and Electra for reranking was driven by a set of preliminary experiments de- scribed in Appendix B.
S-RoBERTagQ
43.4
41.6
52.9
S-RoBERTagg4
48 48.4+0.4
45.620.4
58.3g0.4
Electraqu
37.110.6
10.410.2
49.510.3
Electrage
50.0ga:2
47.7 47.720.3
59 59.540.2
Electragea
49.3+0.2
47. 47.63+0.1
59 29.2
Electrag Ag
50.8+0.2
48. 18.4
60. 20.1
QP Re-RoBERTaasse 43.5
41.8
54.4
QP- OP-RoBERTatare 45.6
43.5
56.0
Table 3:
Experiment results on the proposed dataset. (+)
This model is the one used to build dataset. QP- models are tfae-of-theeart cross encoders (Reimers and Gurevych, 2019).
5.4 Comparison with Web-based QA and Large Language Models
In the previous experiments, we (i) tuned retrieval and ranking models on task-specific annotated data and we (ii) empirically showed what the best con- figurations are to improve accuracy, corroborat- ing the findings presented in the existing literature and aligning with the state-of-ther DBQA. As further step, we evaluated the end-to- end DBQA pipeline updated with models tuned on our dataset, and we compared it against (i) pop- ular open-book Web-based QA system (WebQA), and (ii) Large Language Models (LLM).
We note that the Electra (reranker) generally out- performs S-RoBERTa (retrieval), it is cross- encoder, where single network encodes the whole triplet. Differently, S-RoBERTa uses two different encoders which provide two uncontextualized rep- resentations that are successively combined only in the final layer. Moreover, S-Roberta using the QQA configuration highly outperforms QQ (+5.0 P@ 1), while this gap disappears for Electra since its cross-encoder seems to be enough powerful to compensate for the lack of context, i.e., the answer. Concerning Electra models, QQA is outper- formed by QQ (-0.7 P@ I), mostly because the checkpoint that we used was trained by Lauriola and Moschitti (2021) on QA tasks, thus the model expects the answer to be close to the query. In- deed, QAQ, which is closer how the checkpoint was trained, improves the P@ 1 of QQ by 0.8. should also be stressed the fact that our dataset has been annotated with respect to equivalence and not question-ansn -nnswer correctness. Although the answers were shown to the annota- tors, they were biased on the question and used the answer just as support. Finally, we evaluated ex- isting ofate-of-theeard question de-duplication mod- els (Reimers and Gurevych, 2019) consisting a RoBERTa cross encoder trained on QuoraQp6 (see table 3). Not surprisingly, our models based on similar architectures achieve better performance thanks to the fine-tuning on the target data. We did not evaluate other existing solutions (e.g. Wang et al. (2020)) models are not publicly available.
in DQD and
WebQA consists of a search engine, BING, which finds a set of documents relevant to the input question, and state-of-the-an sentence selector, which chooses the most probably correct answer sentence, extracted from the retrieved documents. This is a typical web-based QA pipeline, the main difference with existing work is that we used BING, a complete and complex commercial search engine, instead of standard solutions based on BM25 or DPR (Zhang et al., 2022a,b). For LLMs, we used two popular models, Falcon 7B (ZXhang et al., 2023) and Vicuna 7B 1.5 (Chiang et al., 2023) to which we asked to generate proper answer for the given query exploiting their parametric knowledge. Our DBQA configuration consists in S- RoBERTa-QQA used as the retriever and Electra- base- QAQ as the reranker. For both, DBQA and WebQA, the answer selectors are applied on top of the top K candidate sentences from the search engines. After preliminary experiments, we set K 500. The retrieval corpus for QUADRo is the DataBase of 6.3M q/a pairs we described before, while, the WebQA corpus is defined as the retrieval capability of BING.
"Public checkpoints are available https: http//wwbbrrrr. htt//bocs/pretrainee.cmm cewpseeddddrsssecoerers.s.r. html.

---

LLM baselines, i.e. Falcon and Vicuna, were used in zero-shot setting. The models tried to gen- erate an answer given an input question through their parametric knowledge, without accessing ex- tra grounding information. The networks received a prompt to generate natural and correct answer given the input request, with few examples. The complete prompt is shown in the Appendix H.
Model
Quora
NQ
TriviaQA
WebQA
35.0
56.0
27.0
Falcon 7B
58.0
40.0
40.6
Vicuna 7B v1.5
25.0
40.0
21.0
QUADRo
58.0
50.5
29.3
our dataset
53.0
47.0
28.0
neural SE
39.0
37.5
29.3
sentence sel.
51.5
40.0
19.0
To remove biases, we evaluated them with three new sets of questions from open domain sources: Quora, Natural Questions (Kwiatkowski et al., 2019). and TriviaQA (Joshi et al., 2017), of size 200, 200, and 150, respectively. See Appendix F for further details regarding these datasets. The questions selected for this test are not in our DataBase in the training annotated the correctness of the returned answers.
answer relev.
50.0
45.0
25.3
Table 4: End-to-end accuracy of QUADRo and other baselines in 3 open domain QA settings. The best sults are highlighted in bold characters. "- x-x" means QUADRo without the feature "x".
We manually
erly understand the complexity of the questions generating appropriate answers.
The results are reported in Table 4. We note that: First, QUADRo highly outperforms WebQA on Quora 58% vs 35%. The main reason is that finding relevant documents for open domain ques- tions is always challenging, especially for ques- tions such Quora where the context is typically underspecified. In contrast, QUADRo can find questions that have similar shape and meaning, and whose answer is still valid for the target question. the same way, Quadro also outperforms Vicuna (+33%), while having the same performance on Falcon (58%). The main reason is that both LLMs rely only on their parametric knowledge, which can lead to hallucinations and incorrect answers.
5.5 Ablated end-to-end comparisons
For completeness, we an ablation study to eval- uate the impact of each component of our system, The last four rows of Table 4 show the end-to-end open-domain QA accuracy of QUADRo: (i) with- out fine-tume mheettnnng on our dataset, emphasizing the im- portance of the resource; (ii) when substituting the neural search engine with standard BM25; (iii) without the answer selector (neural search engine only); and (iv) with neural SE and selector trained on standard question/question similarity, without using the answer. Similarly the previous com- parison, the sentence selectors were applied on top 500 retrieved q/a pairs.
Second, the rationale above is confirmed by the better performance of WebQA on NQ. In this case, the questions that we used have always a relevant Wikipedia page by the construction of the NQ dataset. BING is enough powerful to find these pages so that the only real challenge is the answer selection step. QUADRo still produces good ac- curacy (50%), while Falcon and Vicuna are still not able to retrieve the correct information from their knowledge resulting a poor accuracy of 40%. Fi- nally, QUADRo also outperforms both Vicuna and WebQA on TriviQA questions. The latter are rather challenging questions, which are difficult to find on the web and complex to be answered and under- stand by LLM without grounding. QUADRo seems be able generalize its DB content enough well when operating the search step, or least better than what a standard search engine can do over web documents. Furthermore, it's worth highlighting that Falcon 7B seems to be strong enough to prop-
The results show that each element is extremely relevant, and the ensembling of various key tech- nologies makes QUADRo an effective DBQA sys- tem. For example, training on our dataset increases the accuracy by 5% on Quora, 3.5% on NQ, and 1.3% on TriviaQA. The latter small improvement is probably due to the fact that the questions of Tivi- aQA are rather specific, thus training on general data would impact less than on other datasets.
The usage of the neural search engine is essen- tial, our QUADRo using BM25 performs 19% ab- solute less than when using our neural ranker. This is rather intuitive as the major advantage of us- ing embeddings is their ability to generalize short text, such as the one constituting questions, while TFx IDF heuristics based on lexical features of BM25 largely suffer from sparse representations.
The answer selector component provides as ex- pected a good contribution, 7-10% absolute for all

---

datasets. Finally, the usage of the answer represen- tation as context is rather impactful, from 4% to 8%, This demonstrates that, when the target of the evaluation is answer correctness instead of ques- tion equivalence, models that take answer context into account are clearly superior.
iments show competitive performance with new data, we cannot quantify this possible issue. Finally, we observed that, despite the possibility of reading an answer, annotators tend to focus more on the query-question similarity and less on the query-answer relevance. A possible consequence is that models trained on triplets instead of query- question pairs may experience degradation in per- formance due to skewed labels. Notwithstanding this observation, models trained on query quers-question pairs work poorly in end-to-end QA evaluation (see Table 4).
6 Conclusions
End-to-end QA pipelines that find the answers in a precompiled DataBase of question/answer pairs (DBQA) have become popular in recent years due their efficiency, accuracy, and other benefits. One of the main issues of these approaches is the lack of large training/test data to develop compet- itive models. Existing resources are limited in di- mension, scope, or they do not consider the rel- evance of the answer. In order to fill this gap, we introduce novel annotated resource to train models for DBQA. Our dataset comprises 443,000 examples, making it one of the largest annotated resources for this task. Another novelty is the anno- tation mechanism that considers both, the similarity of the question with respect to the input and the rele- vance of the answer. Furthermore, our experiments report various key aspects of DBQA systems often neglected in the literature, including the contribu- tion of the answer in the retrieval and ranking tasks and the need for high-quality training data. We believe that our data and models, made available to the research community, will enable interesting fu- ture research in optimizing models for DBQA. For example, how to improve the selection of correct q/a pairs in the top pairs.
References
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Compula- tional linguistics, 34(4):555-555 -966.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Procesingg (EMNLP). Association for Computational Linguisties.
Li Cai, Guangyou Zhou, Kang Liu, and Jun Zhao. 2011. Learning the latent topics for question retrieval in community qa In Proceedings of 5th international joint conference on natural language processing, pages 273-281.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2011 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In P Proceedings of the 11th International Workshop on Semantic Evalua- tion (SemEval-2017) Association for Computational Linguistics.
7 Limitations
Dhivya Chandrasekaran and Vijay Mago. 2021. Evolu- tion of semantic similarity a survey, ACM Comput- ing Surveys (CSUR), 54(2):1-37.
The most glaring limit of QUADRo is that the pos- sibility of answering question strictly depends on the coverage of the database of question/answer pairs. Although the database can be enlarged, e.g. by incorporating q/a pairs from PAQ, covering even more questions, there still a conceptual limit that prevents the system from answering very infre- quent questions. Moreover, the database requires mechanisms for content refresh as some questions might change the answer over time.
Delphine Charlet and Geraldine Damnati. 2017. Sim- bow at semeval-2017 task 3: Soft-cosine semantic similarity between questions for community question answering. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 315-319.
Lingjiao Chen, Matei Zaharia, and James Zou, 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint 205.07230..
Concerning the retrieval model, largely known issue of Dense Retrieval systems regards possi- ble drop in performance subject to data domain shift (Wang al., 2021). Although (i) we train the models on various pre-training tasks and open- domain questions, and (ii) our end-to-end exper-
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph F, Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.

---

Maksimjeet Chowdhary, Sanyam Goyal, Venktesh Mukesh Mohania, and Vikram Goyal. 2023. Un- supervised question duplicate and related questions detection in e-learning platforms. In Proceedings of the Sixteetth ACM International Conference on Web Search and Data Mining, pages 1200-1203.
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun- Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Ku- mar, Balint Miklos, and Ray Kurzweil, 2017. Effi- cient natural language response suggestion for smart reply. arXiv preprint arXiv: 1aoo::705.00533
Christopher Hidey and Kathy McKeown. 2016. Identi- fying causal relations using parallel Wikipedia arti- cles. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Papers). pages 1424-1433, Berlin, Ger- many. Association for Computational Linguistics.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL.
William Coster and David Kauchak. 2011. Simple En- glish Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Techmologies, pages 665-669, Portland, Ore- gon, USA. Association for Computational Linguis- tics.
Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. Cqudupstack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Com- puting Symposium, ADCS '15 New York, NY, USA. Association for Computing Machinery.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code- SearchNet challenge: Evaluating the state of seman- tic code search. arXiv preprint 19090949..
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer, 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. arXiv preprint 17X.:::705003555
Ali Fadel, Ibraheem Tuffaha, and Mahmoud Al-Ayyoub. 2019. Tha3aroon at nsurl-2019 task 8: Seman- tic question similarity in arabic. arXiv preprint 1rX1v:1912.12514.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint MxXiv:2004049966.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open Question Answering Over Curated and Extracted Knowledge Bases. In KDD.
Mahnaz Koupaee and William Yang Wang. 2018. Wiki- how: A large scale text summarization dataset.
Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, Jason Weston, and Michael Auli. 2019. Elis: Long form question answering. arXiv preprint arXiv:1907.09190
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Simone Filice. Giovanni Da San Martino, and Alessan- dro Moschitti. 2017. Kelp semeval-2017 task 3: Learning pairwise patterns in community question answering. In Proceedings of the 1 11th International Workshop on Semantic Evaluation (S01EYal-2011), pages 326-333.
Ivano Lauriola and Alessandro Moschitti. 2021. An- swer sentence selection using local and global context in transformer models. In European Conference on Information Retrieval, pages 298-312. Springer.
Katja Filippova and Yasemin Altun. 2013. Overcom- ing the lack of parallel data in sentence compression. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1481-149) Seattle. Washington, USA. Association for Computational Linguistics.
Patrick Lewis, Yuxiang Wu, Linging Liu, Pasquale Min- ervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. Paq: 65 mil- lion probably-asked questions and what you can do with them. Transactions of the Association for Com- putational Linguistics, 9: 9:109.11115.
Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, and Zachary Lipton. 2019. Amazonqa: A review-based question answering task.
Tsung Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dol- lár. 2014. Microsoft coco: Common objects in con- text.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. 3:bertav3: Improving deberta using electra-style pre- training with disetenn-d embedding shar- ing.

---

Kyle Lo, Lucy Lu Wang, Mark Neumann. Rodney Kin- ney, and Dan S. Weld. 2020, S2ore: The semantic scholar open research corpus.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert, Sentence embeddings using siamese bert-networkk In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Yosi Mass, Boaz Carmeli, Haggai Roitman, and David Konopnicki. 2020. Unsupervised FAQ retrieval with question generation and BERT. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 807-812, Online. Asso- ciation for Computational Linguistics.
Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. Faq retrieval using query- question similarity and bert-based query-answer rele- vance. In Proceedings s of the 42nd Internationall ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1113-1116.
Timo Möller, Authony Reina. Raghavan Jayakumar, and Malte Pietsch. 2020. Covid-qa: question answering dataset for covid-19.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Alessandro Moschitti. 2006. Efficient convolution ker- nels for dependency and constituent syntactic trees. In European Conference on Machine Learning, pages 318 329. Springer.
Preslav Nakov, Lluís Marquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Frei- hat, Jim Glass, and Bilal Randeree. 2016. SemEval- 2016 task 3: Community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-20016 pages 525- 545, San Diego, California. Association for Com- putational Linguistics.
Yeon Seonwoo, Juhee Son, Jiho Jin, Sang-Woo Lee. Ji-Hoon Kim, Jung-Wo Ha, and Alice Oh. 2022. Two-step question retrieval for open-domain QA. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 1487-1492, Dublin, Ire- land. Association for Computational Linguistics.
Yikang Shen, Wenge Rong, Nan Jiang, Baolin Peng. Jie Tang, and Zhang Xiong. 2017. Word embed- ding based correlation model for question/answer matching. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1).
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details. just the summary! topic-aware convolutional neural networks for ex- treme summarization. ArXiv, aws//800..ee 087455
Keiji Shinzato, Tomohide Shibata, Daisuke Kawahara, and Sadao Kurohashi. 2012. Tsubaki: An open search engine infrastructure for developing informa- tion access methodology. Journal of information processing, 20( 20(1):216-227.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. In CoCo@ NIPs
Nouha Othman, Rim Faiz, and Kamel Smaili. 2019. Enhancing question retrieval in community question answering using word embeddings. Procedia Com- puter Science, 159:485-499 494.
James Thorne Andreas Vlachos. Christos Christodoulopoulls, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguisties.
Nicole Peinelt, Dong Nguyen, and Maria Liakata. 2020. tbert: Topic models and bert joining forces for se- mantic similarity detection. In ACI..
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke 7eulemoyer. 2018. Deep contextualized word repre- sentations.
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2021. Gpl: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. arXiv preprint 112.:2211..07577.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, (1()):545-5551)
Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. 2023. Tabi: An efficient multi-level inference sys- tem for large language models. In Proceedings of the Eighteenth European Conference on Computer Systems, pages 233-248.
Zizhen Wang, Yixing Fan, Jiafeng Guo, Liu Yang. Ruging Zhang, Yanyan Lan, Xueqi Cheng, Hui Jiang, and Xiaozhao Wang. 2020. Match?: A matching over matching model for similar question identifica- tion. In Proceedings of the 43rd International ACM
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page arXiv: 1506.5550.

---

SIGIR Conference on Research and Development in Information Retrieval, pages 559-568.
NLI (Bowman al., 2015), Altex (Hidey and McK- cown, 2016), AmazonQA (Gupta al., 2019). CNN Dailymail (See et al., 2017), Coco Cap- tions (Lin et al., 2014), CodeSearchNet (Husain et al., 2019), Eli5 (Fan et al., 2019), Fever (Thorne et al., 2018), Flickr30K (Young et al., 2014), GooAQ, Sentence Compression (Fillipovaa and Altun, 2013), SimpleWiki (Coster and Kauchak, 2011), 1), Specter (Cohan al., 2020), SQual (Ra- jpurkar et al., 2016), StuckExchange (Narayan et al., 2018), WikiHow (Koupaee and Wang, 2018), and Xsum (Narayan et al., 2018).
Guoshun Wu and Man Lan. 2016. Ecnu at semeval- 2016 task 3: Exploring traditional method and deep learning method for question retrieval and answer ranking in community question answering. In Pro- ceedings of the 10th International Workshop on Se- mantic Evaluation (SemEval-2016), pages 872-878.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain ques- tion amswering. In Proceedings of the 2015 con- ference on empirical methods in natural language processing, pages 20 2012-20
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Ben- gio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotga: A dataset for diverse, explainable multi-hop question answer- ing. arXiv preprint 1e..vv:809.09666
These datasets consist of pairs of semantically equivalent texts (e.g.: question and answer, title and abstract of a document, paraphrasing.... Overall, the pre-training data includes = 180M positive (i.e. semantically related) text pairs and R 17.5M existing hard-negatives' We consider simple pre- training task where the model predicts if two texts are semantically equivalent or not. We used the MultipeeNegativRRaniigg (Henderson et al., 2017) loss on top of the bi-encoder model combined with cosine similarity as the distance metric in order to make the model able to learn powerful embeddings for retrieval. We used batch size of 384 and a max sequence length of 256 tokens. We use the AdamW optimizer with learning rate of 2e 25-5
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Zeyu Zhang. Thuy Vu, Sunil Gandhi, Ankit Chadha, and Alessandro Moschitti. 2022a. Wdrass: web-scale dataset for document retrieval and answer sentence selection. In Proceedings of the 31st ACM Interna- tional Conference on Information & Knowledge Man- agement, CIKM '22. page 4707-4711, 1, New York, NY, USA. Association for Computing Machinery.
Zeyu Zhang, Thuy Vu, and Alessandro Moschitti. 2021. Joint models for answer verification in question an- swering systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 3252-3262, Online. Association for Computational Linguistics.
B Reproducibility details
We used the same strategy to fine-tune all models in the experiments shown in this paper. We used the development split of our proposed resource to select the optimal hyper-parameters configuration through grid search and to early stop the training when observing degradation of the validation loss for 2 consecutive epochs.
Zeyu Zhang, Thuy Vu, and Alessandro Moschitti. 2022b. In situ answer sentence selection at web-scale. arXiv preprint arXiv:2201. 220.09940..
The hyper-parameters evaluated for the re- trieval
Yoshua X ZXhang, Yann M Haxo, and Ying X Mat. 2023. Falcon Ilm: A new frontier in natural lan- guage processing. AC Investment Research Journal, 220(44).
RoBERTa) are: the learning rate [5e 6, le 5, 2e 5, 5e 5). the batch-size [64, 128, 256,384], 384], and the max sequence length
{128, 256). Concerning ranking models (Electra), we tuned the learning-rate (5e 6, 1e 5, 3e 5, 5e - 5) and the batch-size {64, 128, 1024) the max sequence length is set to 256 tokens.
A Pre-training
Starting from a public checkpoint of our search engine based on Sentence-RoBERTaa RoBERTa-base, we con- tinuously pre-trained it on a plethora of datasets for unsupervised STS tasks (paraphrasing, sen- tence similarity, question answering, and sum- marization..... These datasets include MS- MARCO (Nguyen al., 2016), Natural Ques- tions, The Semantic Scholar Open Research Cor- pus (Lo et al., 2020), PAQ (Lewis et al., 2021),
The selection of the initial checkpoint and archi- tecture for the retrieval (RoBERTa) and the rerank- ing (Electra) components was driven by a set of pre- liminary experiments, where evaluated multiple Transformer models. We observed, and reported on
HHadd negatives are provided for a small subset of pre- training datasets

---

Model
P
MAP
MRR
D Anecdotes
RoBERTaQ4Q
17.5+0.3
16.5+a.2
57.7+0.1
44
50.8
48.4+0.1
60.2
This section shows and discusses some examples of produced annotations, shown in Table 9. In the first example, Case A, the query and question are closer in terms of question-wording and are asking for the same thing making it easy to label them as duplicates even without taking the answer into account. By contrast, in Case B we can notice that query and question seem to be asking for different things. In this case, the answer played a key role for the final annotation label, since it answers both the query and the question. The assigned positive label is then correct.
Table 5: Electra and RoBERTa results comparison on the question/answer reranking dataset. The suffix QAQ indicates the input setting.
Model
Pearson corr.
Spearman corr.
S-RoBERTa 85.1
85.1
S S-Flectra
74.8
75.
S S-Deherta-Va
76.4
77.3
Table 6: S-RoBERTa vs S-Electra and S- S-Deberta- 33 performances comparison on the STShenchmark dataset. All models are base architectures.
Case C contains an annotation error. In this case, the query is contained in the question. While the question is asking for the name of the back and the front of a boat, the query is asking only for the name of the back of boat. Without considering the answer, an annotator might be prone to label these questions as non-duplicated, since they are asking for slightly different concepts. However, in end-to-end QA setting, the answer is correct for the input query (and the question), and thus we should consider the triplet as positive.
table 5, that Electra performs better than RoBERTa (+3333 P@ 1, + 1.9 MAP, and 2.5 MRR.) as cross- encoder in the ranking task measured set. We consider the QAQ configuration is it the one with the better performance. Both models have the same pre-training as described in the previous sections.
test
Differently, Sentence-RoBERTI showed bet- ter retrieval performance compared to Sentence- D3 (He et al., 2021) and Sentence-Electra measured on STSbenchmark (Cer et al., 2017), estabilished benchmark for these tasks.
In the last example, Case D, we report a wrong positive annotation. The query and the questions seem to be identical, but they are not. The query is asking how to replace the battery from a liftmaster remote control, while the question is asking how to replace the battery of a liftmaster remote keypad. In this case, the query and the question are referring to two distinct devices. Moreover, the provided answer is not correct with respect to the query. this case, the correct annotation should be negative.
Annotators training and guidelines
Annotators were asked if two input questions were equivalent or not. The definition of equivalence is: that two questions equivalent iff they have the same intent/meaning and share the same answers. A possible answer for the second question was provided to help the judgment.
We provided a set of guidelines with detailed and explained examples to train the annotators. Guide-
E Latency
lines consist of a clear description of the task and a set of positive and negative swery-guestion-answee triplets. The examples are meant to clarify when query-question pairs can be considered duplicated and how the answer can be used to help the judg- ment. Some examples used to train the annotators are reported in Table
We conducted a latency analysis on QUADRo, eval- uating the efficiency with respect to various key aspects, including the number of retrieved q/a pairs and the size of the database. In our tests, we used (i) a Nvidia A100 GPU, (ii) a S-RoBERTa (base) retrieval model with an embedding size of 768, and (iii) an Electra-base reranker.
Alongside the guidelines, we introduced a set of control triplets in order to guarantee high anno- tation quality. Control triplets are designed to be clear, simple, and easy be judged as positive or negative. Table reports subset of control triplets used during the annotation.
Figure show the latency of an end-to-end re- quest when varying the number of the retrieved q/a pairs while using database of x 6.3M q/a pairs. As you can see, the time scales linearly with the amount of retrieved data. To retrieve and rerank 500 q/a pairs QUADRo took only R 0.53s, 140ms to retrieve and rerank 50 pairs. According to Fig- ure we can notice that the majority of the time is
Annotators were rewarded with 0.15$ per anno- tation task.

---

Positive Examples
Negative Examples
Query: Can cat a dog get along?
Query: Who kill Brutus?
Question: Do cats like the company dogs and in other way around?
Question: Who Brutus kill?
Answer: Brutus was one the leaders the conspirucy that assassinated Julius Caesar
Answer: you are lucky, your cat and dog can become friends within a couple of hours. But that won't usually happen. It takes time for cats to adapt to the dogs and similarly for the dogs to learn how to behave around cats. Explanation: These questions are both asking if Cats and dogs can be friends. The Answer for the Question is also correct for the Query
Explanation: These questions are not asking for the same thing. Moreover, the Answer for the Question is not correct for the Query
Query: Can a person fall in love with another person while
Query: What is the best restaurant in LA
he/she is already in love?
Question: What is the best dish of the best restaurant in LA?
Question: Is possible for people to love 2 person at the same time?
Answer:The best dish of the best restaurant in LA is Lobster Rolls
Answer: It is possible to love and be intimate with more than one person time.
Explanation: Those questions not asking for the same thing. The query asks for restaurant while the Question asks for dish. Moreover, the Answer is correct for the Query
Explanation: These questions are both asking if loving people at the same time is possible. The Answer for the Question is correct for both Question and Query.
Table 7: Explained examples used during annotators training.
Query: What is the color of the sun?
Question: Which color the sun has? Answer: The sun has a temperature 5800 Kelvin, so
Retricval-Ranker
0.5
Ranker Retrieval
it appears white
Label: positive
0.4
Query: What is the food of Koalas?
Seconds
Question: What do Koalas cat?
0.3
Answer: Eucalyptus
Label: positive
0.2
Query: What is the coldest place in the world?
Question: What shape is watermelon?
0.1
Answer: The watermelons are round oval shaped
Label: negative
Query: How many humans there the world?
Question: What the color of the strawberry?
30
100
150
200
250
3000
350
400
450
500
Answer: Typically they red
Label: negative
# Caandidates
Table 8: Examples of control triplets used to discard annotations.
Figure 2: Latency of the end-to-end QUADRo system
while increasing the number of retrieved and reranked a/a pairs. Values are averaged over 200 executions.
spent in the ranking process rather than in retriev- ing the pairs from the database. The retrieval time is 77ms, and it does not scale the number of returned q/a pairs increases.
Datasets
We ran most of experiments collected dataset for question ranking. We divided our dataset into training, development, and test ques- tions as described in Table
Figure 3 shows the latency of the system while increasing the dimension of the DB. In the exper- iment, we set the number retrieved q/a pairs to 500. The retrieval time scales as the size of the database increases. Is worth mentioning that the retrieval process is in the magnitude order of mil- liseconds, confirming the efficiency of our system.
end-to-end QA evaluations on questions sampled from various open-domain sources, including QuoraQP and:
Additionally,
Natural Questions: NQ (K (Kwiatkowski et al., 2019) is popular open-domain QA dataset which consists of questions sampled from Google traffic. A Wikipedia page containing long and short answer associated with each question. Conse- quently, each question is virtually answerable by a web-based QA system.
We release the implementation and the code to replicate these experiments upon request. The ex- periments were based on PyTorch 1.9, Cuda 10.2, Python 3.8, and Transformers (by HuggingFace) 4.10.

---

Case
Examples
-10-2
Query: how old are oldest fossils of organisms?
8
Question: The oldest fossils date to how long ago?
Case A
Answer: The oldest documented fossil organ- isms date to roughly 3.8 billion years ago. shortly after the period of heavy asteroid hom- bardment in Earth's ss history.
7
Seconds
6
Label:
Query: where did the term bully pulpit come from
5
Case
Question: Who what is meant by the expres- sion Bully Pulpit?
4
Retrieval
Answer: This phrase. used by Theodore Roo- sevelt, refers the office of presidency. A pulpit older
3
is place preach from and "bully" adjective meaning excellent. Therefore bully pulpit great place to make speeches from Roosevelt appreciated the fact that people lis- tened to him when he spoke the President.
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
5.5
6.0
Database size millions
Figure 3: Latency of the end-to-end QUADRo system
Label:
while increasing the dimension of the DB. Values are averaged over 200 executions.
Query: What is name called back end boat?
Case C
Question: What the name for back and front a boat?
Answer: The front of a vessel he a ship or boat, is bow or stem. The back sterm.
H LLM zero-shot prompt for Q&A
Label:
We used the following prompt while generating answers for our experiments through Falcon and Vicuna.
Query: how how to replace battery in liftmaster remote control?
Case D
Question: how to change battery in liftmaster remote keypad?
"Answer the question below. The answer must be [well-formed and concise]. The only accepted format the following:
Answer: "Slide the battery cover down. The battery is located at the bottom of your keypad,', 'Disconnect the old battery and remove it. Re- member dispose of your old battery correctly''. 'Put the battery
Question: [here the question]
'Install the new battery. cover back in place.' Label: I
Answer: [here the answer]
Here you have some examples: Example I:
Table 9; Annotated triplets from our dataset.
Question: what is apple?
Answer: An apple, (Malus domestica), is domes- ticated tree and fruit of the rose family (Rosaceae), one of the most widely cultivated tree fruits. Ap- ples are predominantly grown for sale as fresh fruit, though apples are also used commercially for vine- gar, juice, jelly, applesauce, and apple butter and are canned as pie stock.
TriviaQA (Joshi et al., 2017) a QA dataset containing over 95,000 open-domain q/a pairs authored by trivia enthusiasts and indepen- dently gathered evidence documents. Trivia ques- tions are designed to be more challenging, complex, and compositional compared the other datasets,
TriviaQA:
Example 2:
Question: What is the largest airport in the world by travelers?
Metrics
Answer: Atlanta Cortsfield-Jacksne International Airport (ATL) is the larger airport in the world with 75,704 704, 760 total passengers. Dubai International Airport (DXB) the second busiest airport, fol- lowed by Tokyo International Airport (HND) which is the third. Ok, let's begin! Question: [input-question)"
We measure the performance of QA systems with Accuracy in providing correct answers, i.e., the percentage of correct responses, which also refers to Precision-at-1 (P@1) 1) in the context of rerank- ing. We also use standard metrics for ranking: Mean Average Precision (MAP), Mean Recipro- cal Rank (MRR), and Hit-rate@k,, k, which measure the percentage of questions that have least one correct answer (or correct question) in the top-k retrieved/ranker items.


Reviews:
[{"review_id": "6hNlhQc3e7", "reviewer": "Reviewer_VGzs", "summary": "", "questions": "1. Why and how do you select 15,211 questions from the big collection of 5.7M questions? What about their distribution over different sources?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their professional review.\n\n>The key structure of the proposed retrieval-reranking DBQA architecture QUADRo is inspired by previous work (Lewis et al. 2021; Seonwoo et al. 2022), which is stated in line 233.\n\nAlthough our retrieval-ranking DBQA architecture takes inspiration from previous work by Lewis et al. 2021 and Seonwoo et al. 2022, there are some key differences:\n\nFirst, those architectures were based on query-question similarity only, while our ranking and retrieval components use also the answer (of the retrieved question) to improve the similarity between query and question. \nSpecifically, we use the triplet, (Query, Question, Answer), which, especially in the cross-encoder transformer (used for the reranking), is a completely new model. Indeed, the neural cross-attention is able to combine information matched between the query and question with the query and answer and question and answer. This way, our model learn to rank questions also by matching common patterns between (query, answer) and (question, answer), i.e., using textual relations between pairs.\n\nSecond, both the models have been trained on our datasets, which was designed for question-question retrieval. This is different than datasets for standard question-question matching, since the model can learn how to rank questions from our dataset rather than how to classify pairs of questions to output if they are a good match or not.  We plan to extend the retrieval-ranking architecture in our future work.\n\nIn addition, we would like to clarify that the main contribution of our work is the resources that we release to train models for question/answer retrieval and ranking. Dataset, Database, and source code will be released with the paper to make the replication of our experiments easier and support the research in this field.\n\n> Why and how do you select 15,211 questions from the big collection of 5.7M questions? \n> What about their distribution over different sources?\n\nWe randomly sampled 15211 questions from two distinct sources: ~30% of the total from the QuoraQP dataset and ~70% from the WQA dataset. The datasets are widely used for question deduplication and for open domain question answering tasks, respectively. They are considered foundational research datasets. \n\nMany thanks to indicate typos and grammar errors: we will fix them for the camera ready", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A large annotated general dataset is created for Question Answering based on pre-compiled question-answer databases. It is expected to be a good resource for DBQA R&D in the future.\n\n2. Answers are proposed to be included in the annotated dataset and utilized in the QUADRo pipeline for DBQA. Extensive experiments demonstrate its effectiveness for candidate retrieval and answer ranking.", "reasons_to_reject": "1. The key structure of the proposed retrieval-reranking DBQA architecture QUADRo is inspired by previous work (Lewis et al. 2021; Seonwoo et al. 2022), which is stated in line 233.\n\n2. There are  not a few typos or grammar errors. Please refer to the \"Typos Grammar Style And Presentation Improvements\" section below.", "questions_to_authors": "1. Why and how do you select 15,211 questions from the big collection of 5.7M questions? What about their distribution over different sources?", "paper_topic_and_main_contributions": "The authors built a large annotated dataset for DBQA, i.e. finding answers from a precompiled question-answer database for a given question. They believe that answers are important for retrieval and ranking in DBQA, and therefore, consider not only question-question similarity but also question-answer similarity and correctness in the dataset construction and the proposed pipeline QUADRo. Extensive experiments verified the positive contribution of answers in candidate retrieval and answer ranking for DBQA and the need of high-quality training data in building an effective DBQA system.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos or grammar errors:\n\n1. in abstract, \"An effective approach to design ...\", \"either do not consider answers or their quality in the annotation process.\", \"train end evaluate\"\n2. line 031, \"from on large corpus ...\"\n3. line 059, \"without re-train models\"\n4. line 240, \"QUEstion\"\n5. line 304, \"using as a similar approach described by\"\n6. line 335, \"authos\"\n7. line 384, \"the same meaning both questions\"\n8. line 420, \"our is ...\"\n9. line 430, \"between the first annotators\"\n10. lines 560 and 635, \"enough powerful\"\n11. line 677, \"7-10% absolute ...\""}}, {"review_id": "lVdZ2KkhhM", "reviewer": "Reviewer_pEq3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments\n\n> despite using multiple data sources in QUADRo, most of the samples probably came from GooAQ and WikiAnswer (5.4M question-answer pairs out of 6.3M), making it not very heterogeneous.\n\nWe believe this is a good point: although the questions are from open domains and thus they are rather general by construction, GooAQ and WikiAnswer may have bias that are inherited by our DB. GooAQ and WikiAnswer represent the predominant part of the DB, but we also use other datasets that mitigate the bias and improve the heterogeneity, namely WQA, ELI5, QuoraQP, Quora Match, WikiAnswer, CovidQA and HotpotQA.\nIn the future, we will further improve the heterogeneity of our database adding more datasets from other resources (e.g., from CCQA dataset, about 130M q/a pairs).\n\n > to narrow choice of baseline models for analysis of a resource paper\n\nSecondly, we will add other baselines models in the camera ready of the paper (if accepted), including LLMs, e.g., (Vicuna 7B and Falcon 7B).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* dataset comprises of multiple sources with various question and answer length, and combinations of artificially generated question-answer pairs\n* authors made sure to reach high percentage of accuracy for generated samples (by human annotation).\n* annotation workflow well defined and executed to verify high quality human annotation.\n* empirical results showing that the dataset improves retrieval and ranking  compared to baselines, including to some degree on new domains (WebQA) and comparing to a commercial search engine (BING)", "reasons_to_reject": "* despite using multiple data sources in QUADRo, most of the samples probably came from GooAQ and WikiAnswer (5.4M question-answer pairs out of 6.3M), making it not very heterogeneous.\n* to narrow choice of baseline models for analysis of a resource paper", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new resource named QUADRo which is a dataset for evaluating QA systems which retrieves question-answer pairs given queries (DBQA). Compared to previous work the author incorporate the answers into the consideration of question-question similarity, combine several topical QA sets and resulting generated dataset is fast greater than previous works (443K).\nThe authors claim that this resource is the largest that is suitable for training database QA systems, that was also humanly annotated. \n\nAn additional contribution included in the paper is a pipeline built using the dataset to evaluate the contribution of annotation, the addition of fine-tuning of the models on the dataset and the contribution of including the answers for retrieval/ranking.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L56: grammar\nDataBase - casing? consider -> database or index\nTable 1: lenght ->length"}, "extra_text": {}}, {"review_id": "5Mkk6lyMet", "reviewer": "Reviewer_XhtM", "summary": "", "questions": "- In Section 3, in the part of the “search engine”, wouldn't it make more sense to take only pairs that exceed a minimum established similarity? what values of k were tested?\n\n- In Section 4.1, in how many cases was the third annotator required Although some percentages are mentioned, I would recommend reporting standard measures of inter-annotator agreement.\n\n- In the same Section 4.1, several strategies were followed to ensure the quality of the annotation, but how much did they really contribute to improving its final quality? Was there any evaluation of this?\n\n- In 4.2, the analysis was done on a sample of 200 instances, how was this size determined?\n\n\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments and suggestions. \n\n> The sections describing the experiments are not as clear as the first three sections, for example, several of the tables and figures were not explained in sufficient detail.\n\nThanks for the feedback. We  plan to invest the extra page for the camera ready version to improve the clarity of the experiments section and the captioning of figures and tables. \nIn addition, we will add more examples to emphasize the quality of the dataset.\n\nBelow we answer your questions:\n\n> In Section 3, in the part of the “search engine”, wouldn't it make more sense to take only pairs that exceed a minimum established similarity? what values of k were tested?\n\nThe retrieval model is less powerful and so less accurate than the ranker component. This means it can be less precise in identifying relevant q/a pairs. For this reason, a similarity threshold may filter out some best candidates that only the reranker can identify (as the top candidates). Additionally, using a threshold, in corner cases, can lead to have no candidates for an input query, which in some scenario may not be acceptable, e.g., when suggesting similar questions to users.\n\n\n > in Section 4.1, in how many cases was the third annotator required Although some percentages are mentioned, I would recommend reporting standard measures of inter-annotator agreement.\n \nIt was needed 22% of cases. In the camera ready, we will also add standard inter-annotator measures. Thanks for the great suggestion.\n\n> In the same Section 4.1, several strategies were followed to ensure the quality of the annotation, but how much did they really contribute to improving its final quality? Was there any evaluation of this?\n\nWe used multiple strategies to ensure high quality. Some of these were based on standard and established approaches, including adoption of master turkers (annotators with established high quality work history) with >95% acceptance rate on previous annotation tasks. Studying the quality of an annotated resource without these constraints can be expensive and it is outside the scope of the paper. Regarding other employed strategies, we estimated that control questions were able to filter 20-21% of annotation tasks. We will add this data to the paper to improve its completeness. \n\n > In 4.2, the analysis was done on a sample of 200 instances, how was this size determined?\n\nWe used a sample of 200 instances per annotation strategy (qq vs qqa) for 400 annotations in total. This number is sufficient as the result of a t-test showed that the distribution of errors (using our methods or not) are indeed different with high confidence, p<0.1. Moreover, evaluating the correctness of annotations is a time-consuming procedure that requires a significant amount of effort (annotation, fact verification, grammar check, analysis and understanding of the results, e.g., different outcome). \n\nRegarding reproducibility, we believe it is a key aspect of a resource paper such as ours. \nWe would like to highlight that we will release our source code, dataset and database to the scientific community. We will add a GitHub link in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Being a paper whose contribution is a new resource:\n\n- It clearly exposes the limitations of current resources and the needs of the task that are intended to be met with this new dataset.\n- The methodology used to build the recourse is clear and seems appropriate. In this regard, the new dataset is larger than previous similar ones, but above all it exploits the answer as context in order to improve the quality of the annotations. \n- Several experiments are presented to support its quality and relevance, as well as experiments that support its use in other QA tasks and subtasks.\n\nIn general, my main reason for supporting its acceptance is that it can contribute to improving and extending DBQA research.\n\n\n", "reasons_to_reject": "Without being an expert on the subject of the paper, I can't find any compelling reason to reject the work. However, I find two negative aspects:\n\n- The sections describing the experiments are not as clear as the first three sections, for example, several of the tables and figures were not explained in sufficient detail.\n- The paper makes strong criticisms about the quality of the previous datasets, but does not present strong evidence (quantitatively speaking) of the quality of the new resource. Several of the claims in this regard are made based on a manual analysis of only 200 instances (which is not a statistically significant sample).\n\n", "questions_to_authors": "- In Section 3, in the part of the “search engine”, wouldn't it make more sense to take only pairs that exceed a minimum established similarity? what values of k were tested?\n\n- In Section 4.1, in how many cases was the third annotator required Although some percentages are mentioned, I would recommend reporting standard measures of inter-annotator agreement.\n\n- In the same Section 4.1, several strategies were followed to ensure the quality of the annotation, but how much did they really contribute to improving its final quality? Was there any evaluation of this?\n\n- In 4.2, the analysis was done on a sample of 200 instances, how was this size determined?\n\n\n", "paper_topic_and_main_contributions": "This paper focuses on the task of question-answer database retrieval (DBQA), which given an input question consists in retrieving the answer from a database of question-answer pairs. The paper clearly introduces the task and describes its challenges, and accordingly, presents as main contribution a new large annotated dataset to train and evaluate models for DBQA. In addition, it reports an extensive experimentation to assess the quality and characteristics of the dataset, as well as to show its potential to be used in open-book QA and to fine-tune retrieval and ranker models.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper creates a large human-annotated dataset and models for "question-answer database retrieval", where given a question, retrieval takes place of the relevant answers from a pre-existing database with question-answer pairs. 

Pros:
- Introduces a valuable, solidly annotated data set that will be helpful for research in question-answer retrieval.
- The data set doesn't only allow for comparing new questions with a set of existing questions, but also for looking at the answers to those pre-existing questions in order to determine whether a given pre-existing (question+answer) pair could be relevant to the new question.
- Dataset and newly trained models are released.

Cons:
- Presentation could stand to be improved somewhat; details around the experiments aren't as clear as the other sections. 
- Some inter-annotator measures are missing.
As per the review thread, authors would be addressing these in the camera-ready.
Decision: Accept-Findings