Strong and Efficient Baselines for Open Domain Conversational Question Answering
Authors: Andrei Catalin Coman
Gianni Barlacchi
Adrià de Gispert
Venue: EMNLP 2023 Findings (2023)
Keywords: Open Domain Conversational Question Answering
TL;DR: We identify and address several limitations of the typical pipeline used in ODConvQA.

Abstract: Unlike the Open Domain Question Answering (ODQA) setting, the conversational (ODConvQA) domain has received limited attention when it comes to reevaluating baselines for both efficiency and effectiveness. In this paper, we study the State-of-the-Art (SotA) Dense Passage Retrieval (DPR) retriever and Fusion-in-Decoder (FiD) reader pipeline, and show that it significantly underperforms when applied to ODConvQA tasks due to various limitations. We then propose and evaluate strong yet simple and efficient baselines, by introducing a fast reranking component between the retriever and the reader, and by performing targeted finetuning steps. Experiments on two ODConvQA tasks, namely TopiOCQA and OR-QuAC, show that our method improves the SotA results, while reducing reader's latency by 60%. Finally, we provide new and valuable insights into the development of challenging baselines that serve as a reference for future, more intricate approaches, including those that leverage Large Language Models (LLMs).

Strong and Efficient Baselines for Open Domain Conversational Question Answering
Andrei C. Coman'
Gianni Barlacchi
Adrià de Gispert Amazon Alexa AI agispert@anne amazoo..cmmm
Idiap Research Institute, EPFL andrei.coman @ idiap.ch
Amazon Alexa Al gbarlac @ amazon.com
Abstract
Among the most promising approaches are those concerning the improvement of training strategies (Guu et al., 2020; Balachandran et al., 2021; ;; Qu et al., 2021), use of rerankers (Hu et al., 2019; Mao et al., 2021; Barlacchi et al., 2022: Iyer et al., 2021; Glass et al., 2022), question rewriting (Vakulenko et al., 2021; Del Tredici et al., 2021), reader to re- triever knowledge distillation (Izacard and Grave, 2021a), memory-cfficient pipeline (Izacard et al., 2020; Del Tredici et al., 2022), and leveraging struc- tured information (Min et al., 2019; Yu et al., 2022). Unlike the Open Domain Question Answering (ODQA) setting, a reassessment of the baselines in terms of both efficiency and effectiveness appears to be under-explored in the conversational (OD- ConvQA) domain. In this paper, focus the typical DPR retriever and FiD reader (DPR+FiD) pipeline, and show its limitations when applied to the ODConvQA setting. Despite its popularity, we find that this baseline significantly underper- forms when finetuned on downstream tasks. We show that simple improvements in the training, ar- chitecture, and inference setups of the DPR+FiD pipeline, provide strong and efficient baseline that exceeds the performance of SotA models two common ODConvQA datasets: TopiOCQA (Ad- lakha et al., 2022) and ORConvQA (OR-QuAC) (Qu et al., 2020).
Unlike the Open Domain Question Answer- ing (ODQA) setting, the conversational (OD- ConvQA) domain has received limited atten- tion when it comes to reevaluating baselines for both efficiency and effectiveness. In this paper, we study the State-of-the-Are (SotA) Dense Passage Retrieval (DPR) retriever and Fusion-in- -eccooe (FiD) reader pipeline, and show that significantly underperforms when applied to ODConvQA QA tasks due to various lim- itations. We then propose and evaluate strong yet simple and efficient baselines, by introduc- ing a fast reranking component between the retriever and the reader, and by performing tar- geted finetuning steps. Experiments on two ODConvQA tasks, namely TOPIOCQA and OR OR-QuAC, show that our method improves the SotA results, while reducing reader's latency by 60%. Finally, we provide new and valuable insights into the development of challenging basclines that serve as a reference for future. more intricate approaches, including those that leverage Large Language Models (LLMs).
1 Introduction
In an automated information-seeking conversation scenario between two parties, the human ques- tioner asks a series of questions and expects to receive relevant response from the answering system (Oddy, 1977). Current State-of-the-Art (SotA) shapes the answerer via two neural mod- els, the Dense Passage Retrieval (DPR) (Karpukhin et al., 2020) and the Fusion-in- iesiiion-nn-Deeeee in-Decoder (FiD) (Izac- ard and Grave, 2021b), which act as retriever and reader, respectively. Their success stems from the ability to overcome certain limitations of their sparse and extractive counterparts, such as not re- lying lexical retrieval heuristics extracting spans response (Chen al., 2017; Yang et al., 2019; Lee et al., 2019; McCallum et al., 2019; Guu et al., 2020; Lewis et al., 2020; Shen et al., 2023).
We point out several limitations of the pipeline, such as: 1) reader's susceptibility to noisy input, 2) retriever's 's reduced coverage, 3) retriever's 's lack of cross semantic encoding between the conversa- tion and the retrieved passages, and 4) reader's latency is heavily impacted by the number of input passages. To mitigate these, we propose and evalu- ate a simple and effective approach by including a fast reranking component between the retriever and the reader, and by performing targeted finetuning steps. The proposed Sttriveer-erranke-RReeee finetuning (R3FINE) strategy leads to baseline models with a better latency/performance trade-off.
WWrk was done during an intersship at Amazon Science.

---

Rug
loxt
Conversation History
Conversation Encoder
Passage Encoder
Encoder
100000
P1
Reranker
Ps
Encoder
000
Encoder
+000
Decoderr
Answer
Wikipedia
Passages Index
Encoder
+00
representations
DPR
FiD
Figure
The Retriever- Seaeeerr-Rraanee--eaaaae (R3) pipeline
These baselines, which are simple and easy to repli- cale, serve reference point for comparing new and more complex models, and determining their effectiveness. Our contributions the following:
tion history and subsequently passed to the FiD reader, which is based on the T5 (Raffel et al.. 2020) model. The newly created textual sequences of length S are then encoded in parallel via the Encoder component that outputs dense represen- tation ì = {hi...., .., h. As a final step, the dense representations the entire list are concatenated to form a single hn 1h22...., @h sequence that forms the input to the Decoder com- ponent responsible for generating the answer a.
We identify and address several limitations of the typical pipeline used in ODConvQA.
input passages
We propose the R3FINE strategy, which im- proves SotA results on two common datasets and reduces pipeline's latency by 60%.
We provide new and valuable insights for cre- ating simple and efficient baselines, which serve as reference point for future compari- son of new more complex approaches.
3 Strong Baseline Models
This work focuses on two main datasets.
TOPiOCQA (Adlakha et al., 2022) a large-scale open-domain informattio-sseeking conversational dataset that contains challenging phenomenon in the form of topic switching. OR-QuAC (Qu et al., 2020) leverages CANARD's (Elgohary et al., 2019) context-independent question rewrites of the QuAC (Choi et al., 2018) dataset, and adapts the open-domain setting. Further details regarding the datasets are provided in Appendix A.
2 End-to-End Baselines for ODConvQA
This section provides brief introduction to the pipeline on which this work focuses. Figure 1 shows the typical pipeline used within the OD- ConvQA setting, featuring additional reranker component. A conversation history is input to the DPR retriever. This module exploits a dual- encoder based on the BERT (Devlin et al., 2019) model. First, encodes the conversation his- tory via the Cnnneerrationnnnoooee component, which takes input the text of the conversation history C1, C Cj. and then it outputs a dense representation h.. Next, this representation is used perform dense search to retrieve the most rele- vant passages, i.e., text blocks that serve as basic retrieval units, from an external knowledge source (e.g., Wikipedia). The latter contains dense repre- sentations of the passages that have been encoded via the Passage Encoder Encoder component, which takes input a j-th passage with given text length N, i... Pji Pja.Pr..... P.a..... . .Pjy. and outputs dense represen- tation hy.. The dense search is performed via the Maximum inner-Produc Search (MIPS) function which outputs the value corresponding to h! hpy- Once the top-k relevant passages have been re- trieved, their text is appended to the conversa-
We outline number of limitations of the typical DPR+FiD pipeline, along with suggestions on how to mitigate them. While some of those intercon- nect at different levels the various efforts made in the ODQA domain (Balachandran et al., 2021; Yu et al., 2022), our goal is offer perspective on the ODConvQA setting.
3.1 Current Limitations and Bottlenecks
Reader's susceptibility to noisy input. Previous findings have shown that the FiD reader perfor- mance significantly improves when increasing the number of retrieved passages (Izacard and Grave, 2021b). While confirming this finding, in Table also present different perspective to it. We show that when the same reader model is provided with the relevant (i.e., gold) passage in input, the performance decreases as the number of retrieved passages increases. This suggests that there is a balance in presenting input to the reader: if the gold

---

TOPIOCQA
TOPIOCQA
OR-QuAC
w/o gold
w/ gold
top-k
w/o SR
w/ SR
w/o SR
w/ SR
top-k
EM
F1
EM
F1
1
24.66
42.64
29.27
56.64
19.3
37.6
38.3
65.3
10
62.45
75.62
57.14
72.27
10
29.8
52.4
35.8
61.5
50
77.41
84.69
65.22
73.53
50
33.0
55.1
35.9
59.5
1000
91.49
91.49
74.55
74.55
Table I: FiD reader performance (Exact Match and Fl scores) on the TOPIOCQA dev split, with/without the gold passage (w/o gold) in the top-k limit.
Table 2: Dev split retrieval coverage before/after the in- troduction of the SemanticRerandee (w/o SR) when larger number of passages is considered (50 vs 1000).
passage is present, i.e., the retriever could retrieve it, a small relevant list is best, but otherwise larger list is better.
Reader's latency is heavily impacted by the number of input passages. Figure 2 shows that the latency of the reader can be significantly duced by decreasing the number of input pas- sages. However, a trivial limitation to top-k con- siderably degrades the performance of the module, thus leading to an inevitable trade-off. The task of the SemanticReranke vant passages into the top-k list, and allowing for low l value to be set.
Retriever's reduced coverage. Current solutions impose a hard top-k limit on the number of pas- sages returned by the DPR retriever and assume that the relevant ones are present within this limit. Table shows that coverage is key during the re- trieval phase for the reader to perform well. To improve it, we suggest introducing simple and efficient Transformer-based (Vaswani et al., 2017) reranker component after the retriever. This com- ponent, shown in Figure and described in the next paragraph, is designed to reconsider a larger pool of passages returned by the DPR and to provide the
involves pushing rele-
55.0
1.00
52.3
0.90
50.0
47.5
0.80
45.0
0.70
40.0
FiD with reduced and improved list of passages. Since this module operates at the semantic level, we refer to it as the Semanticeeeanker.. Table 2 shows the potential coverage margins and the retrieval results obtained after the introduction of such a module, when a larger number of passages (50 vs 1000) considered.
0. 0.60
37.5
0.50
32.5
30.0
0.40
27.3
25.0
TopiOCOA score
OR-QuAC score
0.30
TopiOCOA latency
OR-QuAC latency
22.3
0.20
top-k
Figure 2: reader performance (Fl score and latency) on the TOPIOCQA dev split and OR- ORRQUAC test split, with varying top-k input passages. Latency is relative the top-50 (top-k top-50).
Retriever's lack of cross semantic encod- ing between the conversation and the re- trieved passages. The DPR retriever performs independent encoding of the passages via the Eneoder function. This means that not able to exploit the semantic relationship among them. This can be mitigated via the introduction of the previously mentioned SemanticReranke component. This new module is based on the Trans- formerEncoder and applies the following function:
3.2 A Strong and Efficient Baseline
Based on the findings above, we introduce the Retriever-Reranker-Real
finetuning (R3FINE) strategy, which can be used to design strong and efficient baselines for ODConvQA. First, we in- crease the number of passages returned by the DPR from the initial 50 to 1000. Then, we add the GemanticRerankee component, which corre- sponds single T Transfon former Encoder layer. We train/finetune the GemanticRerande with the Conversation Encoder Encooee while keeping the Passage Encoder frozen. Guided by the intu-
hes App.................... Peps W...Reaakee..... hp..........................hh hpp))
where each element of the input attends to both the conversation dense representation he and passages dense representations hpp- Reranking is performed over the new output sequence he, hp....., h.p via the previously mentioned MIPS function.
along

---

TOPIOCQA
OR-QuAC
w/o SR
w/ SR
w/SR
FT
w/o SR
w/ SR
w/ SR FT
top-k
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
19.3
37.6
28.1
50.4
30.7
52.4
14.7
23.0
13.9
25.7
16.6
28.9
10
29.8
52.4
33.2
57.3
35,8
59.0
19.0
28.2
19.4
30.0
21.6
32.9
50
33.0
55.1
33.9
56.2
35.2
56.1
22.0
29.9
22.1
30.2
23.5
32.2
Table 3: FiD reader performance (Exact Match and F1 scores) on the TOPIOCQA dev split and OR-QuAC test split (wlo SR), together with results obtained after further reader finetuning step with top-10 output by the SR (w/ SR FT). Underlined values indicate the results obtained by the DPR+FiD pipeline. Bold values indicate the results obtained after the introduction of the SR together with targeted finetuning steps.
before/after introduction
Semanticc
TOPIOCQA
OR-QuAC
Model
EM
F1
Model
EM
F1
BM25 + DPR Reader
13.6
25.0
DrQA (Chen et al., 2017)
6.3
BM25 + FiD
24.1
37.2
BERTserini (Yang et al., 2019)
26.0
DPR Retriever DPR Reader
21.0
43.4
ORConvQA (Qu et al., 2020)
29.4
DPR Retriever+ FiD
33.0
55.3
(Ours) DPR Retriever FiD
22.0
29.9
(Ours) DPR Retriever FiD
33.0
55.1
+ R3FINE (top-10)
21.6
32.9
+ R3FINE (0op-10)
35.8
59.0
Table 5: OR-QuAC test split baselines performance (Exact Match and Fl scores) comparison.
Table 4: TOPIOCQA dev split baselines performance (Exact Match and F1 scores) comparison between sparse/dense retrievers (BM25/DPR Retriever) and ex- tractive/generative readers (DPR Reader/FiD).
R3FINE achieves an FI score of 59 points on TOPiOCQA, and 32.9 on OR-QuAC, which are 3.9 and points higher than the best models pro- posed in the original papers. It is worth noting that these large improvements ple adjustments in the training, architecture, and inference setups of the well-established DPR+FiD baseline, and not via the introduction of new heav- ier and complex models.
ition that less but more relevant passages are ben- eficial to FiD as reported in Table 1, we finally perform an additional finetuning step by leverag- ing the new top-10 10 list of passages returned by the Semantic(Reranker
achieved with sim-
4 Experiments and Results
This section shows the impact that the introduc- tion of the SemantickRraanke module has on the pipeline, as well as the finetuning steps we fol- lowed to make the pipeline more efficient without compromising its performance.
To further support our R3FINE strategy, in Table 3 we present an ablation study which quantifies its impact on the DPR+FiD pipeline. We note that introducing the ways outperforms the DPR+FiD baseline (wlo SR), and at the same time it allows for a 5-fold input size reduction (top-10) while obtaining on-par or better results. In addition, a further finetuning step of the FiD (w/ SR + FT) outperforms the results ob- tained by the Semantic Reranker (w/ SR) by 1.7 and 2.9 Fl points on TOPIOCQA and OR-Que QR-QuAC, respectively. Further experiments and ablation stud- ies are provided in Appendix A.
(w/ SR) al-
Experimental Setup. As the starting point of our experiments, used the DPR and FiD models provided with the TOPIOCQA dataset. Currently, only the train and dev splits are made available for this dataset. We followed the same experimental setup and exploited TOPIOCQA's DPR module for both datasets. Unlike TOPIOCQA, OR-QU RR-QUAC is of extractive type, and for this reason trained the FiD module from scratch by following the same training configuration of TOPIOCQA.
Finally, in Figure 2 can also be observed that using top-10 instead of top-50 can reduce FiD's latency by 60% on average across the two datascts. We conducted latency measurement to evaluate the impact of the Semantic
End-to-End Results. Table 4 and Table com- pare our R3FINE strategy with previous baselines.

---

and its associated parameters, with detailed in- formation available in Appendix A. Given that the SemanticReranker consists of single Transformes Encoder layer, its parameters are negligible when compared to both the DPR and FiD. Moreover, the Rerankr accounts only for 0.34% of the overall latency of the FiD reader, adding additional 2.4ms per example on top of the 710ms taken by FiD. It is im- portant note that this impact is only consid- ered in relation to FiD, as the retrieval phase re- mains constant regardless of the inclusion of the SemanticReranker.
ficient ways to improve REALM. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 158-164, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Gianni Barlacchi, Ivano Lauriola. Alessandro Moschitti, Marco Del Tredici, Xiaoyu Shen, Thuy Vu, Bill Byme, and Adrià de Gispert. 2022. Focusqa: Open- domain question answering with context in focus. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5195-5208.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia answer open- domain questions. In ACL.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke 7ettle- moyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Com- putational Linguisties.
5 Conclusions
In this paper, we identified several limitations of the typical Depnse Passage Retrieval (DPR) retriever and Dusion-in-Decodde (FiD) reader pipeline when applied in an ODConvQ2 QA setting. We proposed and evaluated an improved approach by including a fast reranking component between these two mod- ules and by performing targeted finetuning steps. The proposed R3FINE strategy lead to better la- tency/performance trade-off. The new baseline has proven be both strong and efficient when com- pared to previous baselines, thus making it suitable for future comparisons of new approaches.
Marco Del Tredici, Gianni Barlacchi, Xiaoyu Shen, Weiwei Cheng, and Adríà de Gispert. 2021. Ques- tion rewriting for open-domain conversational qa: Best practices and limitations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2974-2976
Marco Del Tredici, Xiaoyu Shen, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. 2022. From rewriting to remembering: Common ground for conversational qa models. arXiv preprint arXiv:2204.033: 220.0200....900.
Limitations
The study presented in this work aimed to identify and address various limitations of the commonly used ODConvQA pipeline. While our approach may not be technically groundbreaking. the work's novelty lies in the presented findings to design strong and efficient basclines for ODConvQA. It should be noted that further research is needed to compare the performance of the proposed R3FINE strategy with other rerankers on non-conversational QA datasets, which would provide valuable in- sights into how effective the R3FINE approach is in different contexts.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ahmed Elgohary. Denis Peskov, and Jordan Boyd- Graber. 2019. Can you unpack that? learning to rewrite questions-in-comtext. In Proceedings of the 2019 Conference Empirical Methods in Natu- rul Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP) pages 5918-5924, Hong Kong, China. Association for Computational Linguistics.
References
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Sule- man, Harm de Vries, and Siva Reddy, 2022. Top-
Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik. Pengshan Cai, and Alfio Gliozzo. 2022. Re2G: Retrieve, rerank, generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2701-2715, Seattle, United States. Association for Computational Linguistics.
iOCQA: Open-domain conversational question an- swering with topic switching. Transactions of the Association for Computational Linguisties, 10:468- 483.
Vidhisha Balachandran, Ashish Vaswani, Yulia Tsvetkoy, and Niki Parmar. 2021. Simple and ef-

---

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat. and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
2021. Reader-guider passage reranking for open- domain question answering. In Findings of the Asso- ciation for Computational Linguistics: ACI-IJCNIP 2021, pages 344-350, Online. Association for Com- putational Linguistics.
Andrew McCallum, Manzil Zaheer, Rajarshi Das, and Shehzaad Dhuliawala. 2019. Multi-step retriever- reader interaction for scalable open-domain question answering. In ICLR.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dong- sheng Li. 2019. Retrieve, read, rerank: Towards end-to-end multi-document reading comprehension. In ACL.
Sewon Min, Danqi Chen, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2019. Knowledge guided text re- trieval and reading for open domain question answer- ing, ArXiv, abs/191 1..0666......
Srinivasan Iyer, Sewon Min, Yashar Mehdad, and Wen- tau Yih. 2021. RECONSIDER: Improved re-ranking using span-focused cross-attention for open domain question answering.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1280-1287, Online. Association for Computational Linguistics.
Robert N. Oddy, 1977. Information retrieval through man-machine dialogue. Journal of Documentation, 33:1-14.
Chen Qu, Liu Yang. Cen-Chier Chen, Minghui Qiu. W. Bruce Croft, and Mohit lyyer. 2020. Open- retrieval conversational question answering. Pro- ceedings of the 43rd International ACM SIGIR Con- Serence on Research and Development in Information Retrieval.
Gautier Izacard and Edouard Grave. 2021a la. Distilling knowledge from reader to retriever for question an- swering. In International Conference on Learning Representations.
Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized train- ing approach to dense passage retrieval for open- domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5835-5847, On- line. Association for Computational Linguisties.
Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Schastian Riedel, and Edouard Grave. 2020. A memory efficient baseline for open domain question answering. ArXiv, abs/2012.15156.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67,
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6788 (681, Online. Association for Computational Linguistics.
Xiaoyu Shen, Svitlana Vakulenko, Marco Del Tredici. Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. 2023. Neural ranking with weak supervision for open-domain question answering: A survey. In Find- ings of the A Association for Computational Linguistics: FACL. 2023, pages 1691-1705.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086-6096. Florence, Italy. Association for Computational Linguistics.
Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM international conference on web search and data mining, pages 355-363.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela, 2020. generation for knowledge- intensive nip tasks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9459- 9474. Curran Associates, Inc.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In NIPS.
Wei Yang, Yuging Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy J. Lin. 2019. End-to-end open-domain question answering with bertserini. In NAACL.
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen.

---

Donghan Yu, Chenguang Zhu, Yuwei Fang. Wenhao Yu, Shuohang Wang. Yichong Xu, Xiang Ren, Yim- ing Yang, and Michael Zeng. 2022. KG-FiD: Infus- ing knowledge graph in fusion-in-decoder for open- domain question answering. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 4961-4974, Dublin, Ireland. Association for Compu- tational Linguistics.
A.2 Reranker training and ablation study We tried different configurations of the Semantic to find the most efficient and effectivee onc. In addition to the decision of whether to finetune the DPR's Conversation Encoder together with the SemanticlRernneer we also tried varying the number of layers I of the Semanlic.eerankrr from 1 to 4 and changing its input, by choosing a combination from:
A
Appendix
This section provides additional information in sup- port of the work done within the paper.
c : of conversation's 's history dense repre- sentation
A.1
OR-QuAC conversion to TOPIOCQA's format and models
hegr P.y...., h use of conversation's 's history to- kens dense representation
To make the OR- RR-QuAC dataset compatible with TOPIOCQA's 's models, we applied the following steps:
Pl....,Pk: use of passages dense representa- tion
all answers of type CANNOTANSWER and NOTRECOVERED have been mapped to UNANSWERABE
Table 6 shows the average top-k results ob- tained on the TOPIOCQA and OR-QU- OR-QuAC dev split, where varies between 1, 3, 5, 10, 15, 20, 30, 50, 100, 250, 500, 750, and 1000. For TOPIOCQA, we report the presence of the gold passage within the top-k limit. For OR-QuAC we report the pres- ence of the gold answer within the top-k limit.
for the DPR's Cnnnverrsaiion.nncleee com- ponent, followed the TOPIOCQA's ALL- HISTORY conversation representation
for the DPR's PussageEncoder com- ponent, each passage title has been reduced from "passage_page_tti" titte [SEP]
Given that the MIPS function cannot be applied when Pl..... pk representations are used alone, i.e, without the conversation history, we applied lin- ear projection on top of the Semantic Rerankerr to obtain a score for each passage in input.
laargee__ae__uubtttte baassage_pag__tilles This is due to the fact that, compared TOPIOCQA, OR-QuAC does not provide the information about the section where particular passage is located within the page.
to
As far as training the SemanticRer Reranker is concerned, we trained it for 10 epochs when fine- tuned together with the Conversation Encoder. We instead trained it for 20 epochs when the Innversation.Encolee
was kept frozen and when Pı...., Pk representations were used alone. We leveraged the same objective function used for training the initial DPR. We used early stopping chose the best performing model on the dev set. We also used a linear learning rate decay throughout the training process, and Adam with a learning rate of 5e-5 and weight decay of le-2.
for the FiD component, each passage in- formation has been reduced from "title: sub-title:
context:"
"title:
context: ". This is done for the same reason mentioned in the previous point.
Each passage text in OR-QuAC's Wikipedia knowledge source has been mapped to cor- responding embedding via the TOPIOCQA's 's PassageEnccoeer component. We then per- formed the same retrieval step as the one done for TOPIOCQA. We exploited ToMiOCQA's DPR module for both datascts as the retrieval phase is very similar between the two. However, given that, unlike TOPIOCQA, OR-QU OR-QuAC is of extrac- tive type, we had to train the FiD module from scratch. We followed the same training configura- tion as the one used for TOPIOCQA.
Among the different combinations shown in Ta- ble 6, we considered the first entry as the best choice, i.c., the model with L 1, heg Pi. Pk: and CrossEncoder finetuning.
A.3 Retriever results
Table 7 shows the train split retrieval cov- erage before/after the introduction of the SemanticReranker (w/o SR) when a larger number of passages is considered (50 vs 1000).

---

L
hej
hegs.... heg
.l...... Pk
Cross Encoder
TOPIOCQA
OR-QuAC
78.85
71.51
78.88
71.48
78.84
71.32
77.17
69.83
77.35
70.18
4
77.30
70.45
67.07
64.18
Table 6: Average top-k results obtained the ToPIOCQA and OR-QU--II OR-QuAC dev split, with different configurations.
For both datasets, we report the presence of the gold passage within the top-k limit.
TOPIOCQA
OR-QuAC
top-k
w/o SR
w/ SR
w/o SR
w/ SR
1
24.66
42.64
29.27
56.64
TOPIOCQA
OR-QuAC
5
51.87
68.62
51.63
70.82
top-k
w/o SR
w/ SR
w/o SR
w/ SR
10
62.45
75.62
57.14
72.27
31.54
98.25
31.69
77.31
20
70.21
80.75
61.37
72.80
66.99
99.71
59.08
89.10
50
77.41
84.69
65.22
73.53
10
78.29
99.72
66.31
89.36
500
89.58
90.73
72.97
74.34
20
86.78
99.72
71.77
89.40
1000
91.49
91.49
74.55
74.55
50
93.55
99.72
77.35
89.40
Table 8: Dev split retrieval coverage before/after
500
99.36
99.72
87.15
89.41
troduction of the Semantic (w/o SR) when a larger number of passages is considered (50 vs 1000). For TOPIOCQA, we report the presence of the gold passage within the top-k limit. For OR-QuAC we port the presence of the gold answer within the top-k limit.
1000
99.72
99.72
89.41
89.41
Table 7: Train split retrieval coverage before/after the in- troduction of the Semantic/eeanker (w/o SR) when a larger number of passages is considered (50 vs 1000). For both datasets, we report the presence of the gold passage within the top-k limit.
OR-QuAC
top-k
w/o SR
w/ SR
Table shows the dev split retrieval cov- erage beforc/after the introduction of the SemanticReranker (w/o SR) when a larger number of passages is considered (50 1000). For TOPIOCQA, we report the presence of the gold passage within the top-k limit. For OR-QuAC report the presence of the gold answer within the top-k limit. Table 9 shows the OR-QuAC test split retricval coverage beforc/after the introduction of the Semantic (w/o SR) when a larger number of passages is considered (50 1000). We report the presence of the gold answer within the top-k limit.
26.82
48.75
5
46.29
64.71
10
51.50
66.47
20
55.63
67.31
50
59.85
68.10 10
500
67.87
69.48
1000
69.86
69.86
Table 9; OR-QuAC test split retrieval coverage be- fore/after the introduction of the Semantic (w/o SR) when larger number of passages consid- ered (50 vs 1000). We report the presence of the gold answer within the top-k limit.
A.4 Reader results
A.5 Reader is susceptible to noisy input
Table 10, Table 11, and Table 12 show the impact the introduction of the SemanticReranker has on the FiD reader. The input the FiD reader are either passages returned by the initial DPR retriever (w/o SR) or passages returned by the Semantic (w/ SR).
Table 13 shows the FiD reader performance the TOPIOCQA dev split, with/without the gold passage (w/o gold) in the top-k limit. This analysis limited to the ToP1OCQA dataset as the only one to provide information about the gold passage for the dev split.

---

TOPIOCQA
A.6 Further reader study
w/o SR
w/ SR
To better understand the impact that the introduc- tion of the GeeanntiRReannee has on the FiD reader, Table 14, Table 15, and Table 16 show the results obtained after taking a non-finetuned FiD and training it on the top-10 passages returned by the initial DPR retriever and the top-10 pas- On
top-k
EM
F1
EM
F1
19.3
37.6
28.1
50.4
5
27.0
49.6
32.2
56.5
10
29.8
52.4
33.2
57.3
20
31.3
54.0
33.4
56.5
50
33.0
55.1
33.9
56.2
sages returned by the Semantic/eran both datasets, we followed the same training con- figuration as the one used for TOPIOCQA.
Table 10: FiD reader performance (Exact Match and Fl scores) on the TOPIOCQA dev split before/after the introduction of the Gemanticceranker (w/o SR).
TOPIOCQA
w/o SR
W/R
OR-QuAC
top-k
EM
F1
EM
F1
w/o SR
w/ SR
19.2
37.8
29.5
50.9
top-k
EM
F1
EM
F1
5
27.8
49.9
34.0
56.9
13.2
22.2
12.2
25.9
10
30.4
52.0
34.3
57.0
5
16.4
26.3
16.4
28.7
10
18.0
27.6
18.0
29.4
Table 14: FiD reader performance (Exact Match and Fl scores) on the TOPIOCQA dev split after taking non-finetuned FiD and training it on the top-10 passages returned by the initial DPR retriever (wo SR) and on the top-10 10 passages returned by the Semontic (w/ SR).
20
19.2
28.2
18.9
29.2
50
19.6
27.7
19.3
27.7
Table 11: FiD reader performance (Exact Match and scores) on the OR-QuAC dev split before/after the introduction of the Semantic (w/o SR).
OR-QuAC
OR-QuAC
w/o SR
w/ SR
w/o SR
w/ SR
top-k
EM
F1
EM
F1
top-k
EM
F1
EM
F1
14.7
23.0
13.9
25.7
1
14.2
23.1
13.9
27.6
5
17.8
27.1
17.9
29.5
5
18.0
27.7
18.1
30.8
10
19.0
28.2
19.4
30.0
10
19.2
28.9
19.3
31.2
20
20.3
29.1
20.8
30.5
Table 15: FiD reader performance (Exact Match and scores) on the OR-QuAC dev split after taking non- finetuned FiD and training it on the top-10 returned by the initial DPR retriever (w/o SK) and on the top-10 returned by the GemandicRRranne (w/ SR).
50
22.0
29.9
22.1
30.2
Table 12: FiD reader performance (Exact Match and Fl scores) the OR-QuAC test split before/after the introduction of the SemanticReranker (w/o SR).
TOPIOCQA
OR-QuAC
w/o gold
w/ gold
w/o SR
w/ SR
top-k
EM
F1
EM
F1
top-k
EM
F1
EM
F1
19.3
37.6
38.3
65.5
1
15.5
23.9
15.8
28.3
5
27.0
49.6
36.3
62.5
5
19.6
29.1
19.4
31.7
10
29.8
52.4
35.8
61.5
10
20.9
29.7
20.6
31.9
20
31.3
54.0
36.2
60.8
50
33.0
55.1
35.9
59.5
Table 16: FiD reader performance (Exact Match and Fl scores) on the OR-QuAC test split after taking a non- finetuned FiD and training it the top-10 passages returned by the initial DPR retriever (wo SR) and on the top- 10 passages returned by the Semontic (w/ SR).
Table 13: FiD reader performance (Exact Match and Fl scores) on the TOPIOCQA dev split, with/without the gold passage (w/o gold) in the top-k limit.
Table 17, Table 18, and Table 19 show instead

---

the results obtained after taking an already fine- tuned FiD reader and further finetuning it on the top-10 passages returned by the initial DPR re- triever and on the top-10 passages returned by the Reranker. On both datasets, the amount of finetuning steps is equal to the one used for training the already finetuned FiD reader.
following the FiD's test_reader py script pro- vided with the TOPIOCQA dataset. We set the per_gpu_
size paramenter to and chose the value of the n_context parameter from 1, 5, 10, 20, 30, and 50, based on the num- ber of input passages, For cach value, we report the latency relative to the maximum context param- eter value, i.e., 50. We used CUDA events synchro- nization markers measure the elapsed time for the preprocessing and evaluation of TOPIOCQA's dev split and OR-Qui OR-QuAC's test split.
runs
TOPIOCQA
w/o SR
w/ SR
top-k
EM
F1
EM
F1
21.5
39.3
30.7
52.4
30.4
52.3
36.1
59.4
10
32.8
54.6
35.8
59.0
Table 17: FiD reader performance (Exact Match and scores) on the ToPIÔCQA dev split after taking an already finetuned FiD and further finetuning it on the top-10 returned by the initial DPR retriever (w/o SR) and the top-10 10 returned by the SemanliccRrannerr (w/ SR).
OR-QuA
w/o SR
w/ SR
top-k
EM
F1
EM
F1
15.2
23.8
15.4
29.0
5
18.9
28.4
18.4
31.2
10
19.9
29.4
20.1
32.0
Table 18: FiD reader performance (Exact Match and Fl scores) on the OR-Que OR-QuAC dev split after taking an already finetuned FiD and further finetuning it on the top-10 passages returned by the initial DPR retriever (w/o SR) and on the top-10 passages returned by Semanticceranker (w/ SR).
OR-QuAC
w/o SR
w/ SR
top-k
EM
F1
EM
F1
16.7
24.9
16.6
28.9
5
20.0
29.2
20.1
32.2
10
21.4
30.0
21.6
32.9
Table 19: FiD reader performance (Exact Match and Fl scores) on the OR-Que OR-QuAC test split after taking an already finetuned FiD and further finetuning it on the top-10 passages returned by the initial DPR retriever (w/o SR) and on the top-10 10 passages returned by the SemonticReronker (w/ SR).
A.7 Latency measurement
Latency measurement (see Figure 2) has been per- formed on the same NVIDIA V100 16GB GPU, by


Reviews:
[{"review_id": "S0RpC3Tffl", "reviewer": "Reviewer_UNK4", "summary": "", "questions": "A. In Table 4, I was wondering about the difference between the method DPR Retriever + FiD and (Ours) DPR Retriever + FiD. They have the same EM score while the latter has a lower F1 score.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful evaluation and supportive feedback on our approach, highlighting the paper's clarity, the method's simplicity and effectiveness, and the robust experimental evidence.\n\n__Reviewer__\n\n*\"Inconsistent results. On OR-QuAC dataset in Table 5, the EM score decreases from 22.0 to 21.6 while F1 increases from 29.9 to 32.9 after adding the SR. This is different from the performance on TOPIOCQA in Table 4, where both EM and F1 increase.\"*\n\n__Authors__\n\nTable 5 shows a slight decline in the EM performance of the \"+ R3FINE (top-10)\" model, which is attributed to its use of only 10 passages, in contrast to the \"(Ours) DPR Retriever + FiD\" baseline, which uses 50 passages. Nevertheless, Table 3's \"w/ SR + FT\" column indicates that the R3FINE model surpasses the baselines in both metrics when given access to the same number of passages (i.e., 50). For the OR-QuAC dataset, we opted for consistency in reporting results on both EM and F1 metrics, aligning with our approach for TopiOCQA. This decision has been made to establish a standardized reporting framework for future research, thereby enhancing the comparability of different models.\n\n__Reviewer__\n\n*\"In Table 4, I was wondering about the difference between the method DPR Retriever + FiD and (Ours) DPR Retriever + FiD. They have the same EM score while the latter has a lower F1 score.\"*\n\n__Authors__\n\nThe tag \"(Ours)\" indicates a model that leverages the pre-trained models from prior research, yet incorporates a distinct codebase and pipeline implementation compared to the original approach. This decision was primarily driven by the necessity to seamlessly integrate the SemanticReranker into the existing framework. The marginal reduction in the F1 score may be attributed to minor variances in the retrieved passages and their ranking. These nuances can marginally influence the answer generation process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and clear. The proposed method is simple and effective. The experiments conducted on two datasets support the main claims.", "reasons_to_reject": "Inconsistent results. On OR-QuAC dataset in Table 5, the EM score decreases from 22.0 to 21.6 while F1 increases from 29.9 to 32.9 after adding the SR. This is different from the performance on TOPIOCQA in Table 4, where both EM and F1 increase.", "questions_to_authors": "A. In Table 4, I was wondering about the difference between the method DPR Retriever + FiD and (Ours) DPR Retriever + FiD. They have the same EM score while the latter has a lower F1 score.", "paper_topic_and_main_contributions": "This paper points out four limitations of the existing pipeline and proposes Retriever-Reranker-Reader fine-tuning strategy. First, they increase the number of passages returned by the DPR from 50 to 1000. Then they add a Semantic Reranker component, which is fine-tuned with the Conversation Encoder. Their proposed method improves SOTA results on two common ODConVQA datasets: TopiOCQA and ORConvQA and reduces the pipeline's latency by 60%.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dmBmsFveao", "reviewer": "Reviewer_3qL2", "summary": "", "questions": "1. Lines 144-148: To clarify, are the \"w/\" and \"w/o gold\" results based on different evaluation data for the \"same\" model (rather than two models trained on different data)?\n2. Lines 251-254: Figure 2 does not discuss the latency that SemanticReranker brings. Please discuss the impact that the introduction of it has had.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the the reviewer for their thorough evaluation and the constructive remarks provided regarding the R3FINE method. We appreciate their acknowledgment of the method's effectiveness and efficiency, along with their recognition of the experimental validity and the potential impact of our contributions. Their meticulous analysis and helpful suggestions for improving the clarity of our presentation are greatly appreciated, and we are dedicated to promptly addressing these recommendations.\n\n__Reviewer__\n\n*”Lines 251-254: Figure 2 does not discuss the latency that SemanticReranker brings. Please discuss the impact that the introduction of it has had.”*\n\n__Authors__\n\nGiven that the SemanticReranker consists of a single TransformerEncoder layer, its parameters are negligible when compared to both the DPR and FiD. Moreover, the SemanticReranker accounts for 0.34% of the overall latency of the FiD reader, adding an additional 2.4ms per example on top of the 710ms taken by FiD. It is important to note that this impact is only considered in relation to FiD, as the retrieval phase remains constant regardless of the inclusion of the SemanticReranker. We will make sure to provide these additional details in the camera-ready.\n\n__Reviewer__\n\n*\"Lines 144-148: To clarify, are the \"w/\" and \"w/o gold\" results based on different evaluation data for the \"same\" model (rather than two models trained on different data)?\"*\n\n__Authors__\n\nIn Table 1, and its more comprehensive counterpart Table 13 (Appendix A.5), the goal is to show that the “same” FiD reader is susceptible to noisy input. The terms “w/” and “w/o gold” denote results that are indeed based on different evaluation data for the “same” model. To be more precise, the variation between these two settings is solely dependent on whether the gold passage is included within the top-k limit or not.\n\n__Reviewer__\n\n*\"Line 183: Is the \\tilde{h}_c different from h_c? If so, it is not obvious to me how different.\"*\n\n__Authors__\n\nGiven that the SemanticReranker leverages the TransformerEncoder layer, its output vectors represent a mixture of its input vectors. The inclusion of the notation serves the purpose of accentuating this distinction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method not only experimentally shows the problems of the DPR+FiD architecture but also improves the performance by adding a reranker. Furthermore, the effectiveness of the added components was demonstrated through an ablation study. This simple idea may be of interest to other researchers working on open-domain conversational QA.\n", "reasons_to_reject": "1. Although the proposed method significantly improves performance, this paper does not explain how the hyper-parameters are set. If they were tuned with evaluation data in a non-systematic manner, the advantages of the proposed method might be limited in other settings. However, the basic idea of the proposed method seems promising to me, and the lack of comprehensive experiments should not be the main reason for rejecting a short paper.\n", "questions_to_authors": "1. Lines 144-148: To clarify, are the \"w/\" and \"w/o gold\" results based on different evaluation data for the \"same\" model (rather than two models trained on different data)?\n2. Lines 251-254: Figure 2 does not discuss the latency that SemanticReranker brings. Please discuss the impact that the introduction of it has had.\n", "paper_topic_and_main_contributions": "This study sheds light on the limitations of the dense passage retrieval + fusion-in-decoder architecture in open-domain conversational QA and proposes a new approach to overcome these limitations. The problems include that the reader should have fewer inputs as long as the input contains gold passages for both performance and latency reasons, and that the output of the retriever does not semantically take the conversation history into account sufficiently. Their proposed method adds a lightweight model that reranks the passages output by the retriever, taking into account the conversation. The authors experimentally showed that the proposed method improves performance and reduces latency on two datasets.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Presentation\n\n- Lines 087-089: Figure 1 illustrates a proposed method, not a \"typical pipeline\", doesn't it?\n- Section 2: The symbols used here are complicated, and this paragraph should be reorganised in places. If I understand correctly, (a) the conversation history (line 095) may be written as c instead of c_1, ..., c_i, (b) a j-th passage (line 103) should be introduced with the symbol p_j, and the reference to its text length, N, and p_j_N (which makes confusion with p_1, ..., p_N) may be omitted in the text, and (c) a dense representation \\overline{h} can be explained without {h_1, ..., h_S}.\n- Figure 1: The notations here should follow the symbols in Section 2. In other words, c, S, h_1, ..., h_k, and a should be mentioned in the figure. Also, the Reranker should take as input the outputs of the conversation encoder in addition to p_1, ..., p_n. The output of the Reranker should be p_k instead of p_n.\n- Lines 149-151: (i.e., if the retriever could return the ideal answers)\n- Line 167: The term \"SemanticReranker\" should also be introduced here, as referenced in Table 2.\n- Line 183: Is the \\tilde{h}_c different from h_c? If so, it is not obvious to me how different.\n\nTypos\n\n- Line 546: the that the -> the\n- Line 583: 30 might be included here, I think.\n"}}, {"review_id": "kuHewWRq5O", "reviewer": "Reviewer_tKUu", "summary": "", "questions": "Q1A. Line 15 and line 187 says that the latency is about the decoder, while line 80 says that the latency is about the pipeline, which in my understanding, is retriever+reader. Which one is correct?\n\nQ1B. If the 60% reduction of latency is about the reader, then what is the overall reduction of latency and how does the reranker slow down the pipeline? If is about the pipeline, then what is cost of each component?\n\nQ2. What is the size of T5 used in your experiments? And what value is the length S? Are there any computational issues when experiments with k=50, especially during training, because k=50 means a very long input?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely thank the reviewer for their constructive input, which aligns with our efforts to enhance the robustness of our work. Their acknowledgment of the four limitations we pinpointed for the retriever-reader framework, the substantial experimental validation supporting our assertions, and our thorough exploration of alternative designs underscore the rigor and comprehensiveness of our research.\n\n__Reviewer__\n\n*\"Line 15 and line 187 says that the latency is about the decoder, while line 80 says that the latency is about the pipeline, which in my understanding, is retriever+reader. Which one is correct?\"*\n\n*\"If the 60% reduction of latency is about the reader, then what is the overall reduction of latency and how does the reranker slow down the pipeline? If is about the pipeline, then what is cost of each component?\"*\n\n__Authors__\n\nGiven that the SemanticReranker consists of a single TransformerEncoder layer, its parameters are negligible when compared to both the DPR and FiD. Moreover, the SemanticReranker accounts for 0.34% of the overall latency of the FiD reader, adding an additional 2.4ms per example on top of the 710ms taken by FiD. It is important to note that this impact is only considered in relation to FiD, as the retrieval phase remains constant regardless of the inclusion of the SemanticReranker. We will make sure to provide these additional details in the camera-ready.\n\n__Reviewer__\n\n*\"What is the size of T5 used in your experiments? And what value is the length S? Are there any computational issues when experiments with k=50, especially during training, because k=50 means a very long input?\"*\n\n__Authors__\n\nTo ensure complete compatibility with the conventional DPR+FiD pipeline, we follow the same experimental setup of the TopiOCQA dataset, where the FiD reader consists of a T5-base model. It is important to note that the FiD reader does not encode information from multiple passages in a cross-semantic way during the encoding phase. Instead, for efficiency reasons, each passage is encoded independently and in parallel with the conversation history. The cross-semantic connections between passages are only established during the decoding phase. A common FiD reader uses 50 passages, but our approach allows the reranker to work with 1000 passages and simultaneously condition on both passages and conversation. This leads to increased coverage (Table 2) and improved overall performance (Table 4 and 5). Given the limitations of the reader component, the use of a reranker appears to be necessary. Additionally, the proposed reranker helps to significantly reduce the latency of the reader without compromising its performance (Table 3 and Figure 2).\n\n__Reviewer__\n\n*“I cannot see any significant difference between the R3FINE (proposed in this paper) and Re2G [1]. […]”*\n\n__Authors__\n\nWe will diligently assess the missing reference highlighted by the reviewer, ensuring that proper recognition is accorded where it is due. We believe that the novelty of our work lies in the presented findings on how to design strong and efficient baselines for ConvQA. We argue that it is not limited to the simple addition of a reranker. In fact, we demonstrate that the typical pipeline used in the ODConvQA setting exhibits several limitations and that current baselines are not being utilized to their full potential, which can lead to unreliable conclusions when developing new models. To mitigate the problem, we propose simple improvements in training, architecture, and inference that lead to stronger and more efficient baselines that surpass current state-of-the-art models (Table 1 and 3 and Figure 2).\n\n__Reviewer__\n\n*”This paper distinguishes itself from open-domain QA (non-conversational). But I cannot find a strong reason why rerankers in ODQA differ from rerankers in ODConvQA.”*\n\n__Authors__\n\nIn this work, we decided to focus mainly on the Open Domain Conversational Question Answering (ODConvQA) domain as we identified a lack of exploration into the efficiency and effectiveness of the typical DPR+FiD pipeline within this setting. Our goal was to keep the paper concise and focused on a specific setting where the limitations play an important role. This implied the exclusion of experiments on ODQA and their relegation as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper provides four reasonable limitations on the retriever-reader framework.\n\n2. There are sufficient experiments to support all their claims and the effectiveness of their model. \n\n3. There are results on many alternative designs. \n", "reasons_to_reject": "1. I cannot see any significant difference between the R3FINE (proposed in this paper) and Re2G[1]. Re2G is evaluated on Wizard of Wikipedia, another open-domain conversational QA benchmark (not used in this paper), and shows the effectiveness of the retriever-reranker-reader framework on ODConvQA. Re2G also discussed the efficiency benefits brought by the reranker.\n\n2. Actually, many ODConvQA models use rerankers. Except for Re2G, the baseline model in this paper, ORConvQA, is also equipped with a reranker, although the reranker in ORConvQA does not aim to improve efficiency. However, this paper does not provide any discussion on this.\n\n3. This paper distinguishes itself from open-domain QA (non-conversational). But I cannot find a strong reason why rerankers in ODQA differ from rerankers in ODConvQA. \n\n[1] Glass, M.R., Rossiello, G., Chowdhury, M.F., Naik, A.R., Cai, P., & Gliozzo, A. (2022). Re2G: Retrieve, Rerank, Generate. ArXiv, abs/2207.06300.", "questions_to_authors": "Q1A. Line 15 and line 187 says that the latency is about the decoder, while line 80 says that the latency is about the pipeline, which in my understanding, is retriever+reader. Which one is correct?\n\nQ1B. If the 60% reduction of latency is about the reader, then what is the overall reduction of latency and how does the reranker slow down the pipeline? If is about the pipeline, then what is cost of each component?\n\nQ2. What is the size of T5 used in your experiments? And what value is the length S? Are there any computational issues when experiments with k=50, especially during training, because k=50 means a very long input?", "paper_topic_and_main_contributions": "The paper focuses on open-domain conversational question answering. It proposes adding a fast reranker between the retriever and the reader to balance speed and performance. It claims new SOTA results on two benchmarks and a 60% reduction in latency.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Glass, M.R., Rossiello, G., Chowdhury, M.F., Naik, A.R., Cai, P., & Gliozzo, A. (2022). Re2G: Retrieve, Rerank, Generate. ArXiv, abs/2207.06300.", "Typos_Grammar_Style_and_Presentation_Improvements": "L002: DPR and FiD are not abbreviations of their preceding words.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper points at weaknesses in the retriever-reader framework for open domain conversational QA, and suggests a retriever-reranker-reader approach that shows benefits on two benchmark datasets. 
While the approach is not very novel (there were previous works that suggested reranking in similar settings, e.g. as pointed by reviewer 3), the experiments are sound and demonstrate appealing results as agreed by all reviewers. 
Given the above, the paper may be accepted to findings.
Decision: Accept-Findings