Findings of the Association for Computational Linguistics: ACL 2023, pages 6019–6031 July 9-14, 2023 ©2023 Association for Computational Linguistics Phrase Retrieval for Open-Domain Conversational Question Answering with Conversational Dependency Modeling via Contrastive Learning
Year: 2023

Abstract:
Open-Domain Conversational Question Answering (ODConvQA) aims at answering questions through a multi-turn conversation based on a retriever-reader pipeline, which retrieves passages and then predicts answers with them. However, such a pipeline approach not only makes the reader vulnerable to the errors propagated from the retriever, but also demands additional effort to develop both the retriever and the reader, which further makes it slower since they are not runnable in parallel. In this work, we propose a method to directly predict answers with a phrase retrieval scheme for a sequence of words, reducing the conventional two distinct subtasks into a single one. Also, for the first time, we study its capability for ODConvQA tasks. However, simply adopting it is largely problematic, due to the dependencies between previous and current turns in a conversation. To address this problem, we further introduce a novel contrastive learning strategy, making sure to reflect previous turns when retrieving the phrase for the current context, by maximizing representational similarities of consecutive turns in a conversation while minimizing irrelevant conversational contexts. We validate our model on two ODConvQA datasets, whose experimental results show that it substantially outperforms the relevant baselines with the retriever-reader. Code is available at: https://github.com/ starsuzi/PRO-ConvQA.

heading: 1 Introduction
text: Conversational Question Answering (ConvQA) is the task of answering a sequence of questions that are posed during information-seeking conversations with users (Choi et al., 2018; Reddy et al., 2019; Zaib et al., 2022). This task has recently gained much attention since it is similar to how humans seek and follow the information that they want to find. To solve this problem, earlier ConvQA
∗ Corresponding author
work proposes to predict answers based on both the current question and the previous conversational histories, as well as the passage that is relevant to the ongoing conversation (Qu et al., 2019; Huang et al., 2019; Kim et al., 2021; Li et al., 2022a). However, this approach is highly suboptimal and might not be applicable to real-world scenarios, since it assumes that the gold passage, containing answers for the current question, is given to the ConvQA system; meanwhile, the gold passage is
6019
usually not available during the real conversation.
To address this limitation, some recent work (Qu et al., 2020; Anantha et al., 2021; Li et al., 2022c; Adlakha et al., 2022; Fang et al., 2022) proposes to extend the existing ConvQA task to an opendomain question answering setting with an assumption that the conversation-related passages are not given in advance; therefore, it is additionally required to access and utilize the query-relevant passages in a large corpus, for example, Wikipedia. Under this open-domain setting, most existing Open-Domain ConvQA (ODConvQA) work relies on the retriever-reader pipeline, where they first retrieve the passages, which are relevant to both the current question and conversational context, from a large corpus, and then predict answers based on information in the retrieved passages. This retrieverreader pipeline approach is illustrated in Figure 1.
However, despite their huge successes, such a pipeline approach consisting of two sub-modules has a few major drawbacks. First, since the reader is decomposed from the retriever, it is difficult to train the retriever-reader pipeline in an end-to-end manner, which results in an additional effort to develop both the retriever and the reader independently. Second, the error can be accumulated from the retriever to the reader, since the failure in finding the relevant passages for the current question negatively affects the reader in predicting answers, which is illustrated in Figure 1. Third, while the latency is an important factor when conversing with humans in the real-world scenarios, the retrieverreader pipeline might be less efficient, since these two modules are not runnable in parallel.
An alternative solution tackling the limitations above is to directly predict the phrase-level answers consisting of a set of words, which are predicted from a set of documents in a large corpus. While this approach appears challenging, recent work shows that it is indeed possible to directly retrieve phrases within a text corpus based on their representational similarities to the input question (Seo et al., 2019; Lee et al., 2021a,b). However, its capability of retrieving phrases has been studied only with single-turn-based short questions, and their applications to ODConvQA, additionally requiring contextualizing the multi-turn conversations as well as effectively representing the lengthy conversational histories, have not been explored.
To this end, in this work, we first formulate the open-domain ConvQA task, previously done
with the two-stage retriever-reader pipeline, as a direct phrase retrieval problem based on a single dense phrase retriever. However, in contrast to the single-turn open-domain question answering task that needs to understand only a single question, the target ODConvQA is more challenging since it has to comprehensively incorporate both the current question and the previous conversational histories in multi-turns. For example, as shown in Figure 1, in order to answer the question, “ What happened in 2003”, the model has to fully understand that the conversational context is related to the song, not the movie. While some work (Qu et al., 2020; Fang et al., 2022; Adlakha et al., 2022) proposes to feed an ODConvQA model the entire context consisting of the current question together with the conversational histories as an input, this naïve approach might be insufficient to solve the conversational dependency issue, which may lead to suboptimal performances in a phrase retrieval scheme.
In order to further address such a conversational dependency problem, we suggest to enforce the representation of the current conversational context to be similar to the representation of the previous context. Then, since two consecutive turns in a conversation are dependently represented in a similar embedding space, phrases that are relevant to both the current and previous conversational contexts are more likely to be retrieved, for the current question. To realize this objective, we maximize the representational similarities between the current conversational context and its previous contexts, while minimizing the representations between the current and its irrelevant contexts within the same batch via the contrastive learning loss, which is jointly trained with the dense phrase retriever. This is illustrated in Figure 1, where we force the representation of the current conversational turn to be similar to its previous turn. We refer to our proposed method as Phrase Retrieval for Open-domain Conversational Question Answering (PRO-ConvQA).
We validate our proposed PRO-ConvQA method on two standard ODConvQA datasets, namely ORQuAC (Qu et al., 2020) and TopiOCQA (Adlakha et al., 2022), against diverse ODConvQA baselines that rely on the retriever-reader pipeline. The experimental results show that our PRO-ConvQA significantly outperforms relevant baselines. Furthermore, a detailed analysis demonstrates the effectiveness of the proposed contrastive learning strategy and the efficiency of our phrase retrieval strategy.
Our contributions in this work are threefold:
• We formulate a challenging open-domain conversational question answering (ODConvQA) problem into a dense phrase retrieval problem for the first time, by simplifying the conventional two-stage pipeline approach to ODConvQA tasks consisting of the retriever and the reader into one single phrase retriever.
• We ensure that, when retrieving phrases, the representation for the current conversational context is similar to the representations for previous conversation histories, by modeling their conversational dependencies based on the contrastive learning strategy.
• We show that our PRO-ConvQA method achieves outstanding performances on two benchmark ODConvQA datasets against relevant baselines that use a pipeline approach.

heading: 2 Related Work
text: Conversational Question Answering ConvQA is similar to the reading comprehension task (Rajpurkar et al., 2016; Trischler et al., 2017) in that it also aims at correctly answering the question from the given reference passage (Choi et al., 2018; Reddy et al., 2019). However, ConvQA is a more difficult task than the reading comprehension task, since ConvQA has to answer questions interactively with users through multi-turns, which requires capturing all the contexts including previous conversational turns and the current question as well as its reference passage. To consider this unique characteristics, a line of research on ConvQA has focused on selecting only the queryrelevant conversation history (Huang et al., 2019; Qu et al., 2019; Chen et al., 2020; Qiu et al., 2021). However, recent work observed that a simple concatenation of the conversational histories outperforms the previous history selection approaches, thanks to the efficacy of the pre-trained language models (Vaswani et al., 2017) in contextualizing long texts (Kim et al., 2021). However, as the conversations often involve linguistic characteristics such as anaphora and ellipsis (Zaib et al., 2022), some work suggested to rewrite the ambiguous questions to explicitly model them (Kim et al., 2021; Vakulenko et al., 2021; Raposo et al., 2022). However, a naïve ConvQA setting assumes a fundamentally unrealistic setting, where the gold reference passages, containing answers corresponding to the questions, are already given.
Open-Domain ConvQA In order to address the unrealistic nature of the aforementioned ConvQA scenario, some recent work proposed to extend it to the open-retrieval scenario, which aims at retrieving relevant passages in response to the ongoing conversation and then uses them as reference passages, instead of using human-labeled passages. In this setting, effectively incorporating the conversational histories into the retrieval models is one of the main challenges, and several work (Lin et al., 2021; Yu et al., 2021; Mao et al., 2022; Wu et al., 2022) proposed improving the first-stage retrievers, which are trained with particular machine learning techniques such as knowledge distillation, data augmentation, and reinforcement learning. However, their main focus is only on the first-stage retrieval aiming at returning only the query-related candidate passages, without giving exact answers to the questions. Also, some methods, such as ConvDR (Yu et al., 2021) and ConvADR-QA (Fang et al., 2022), use additional questions, which are rewritten from original questions by humans, to improve a retrieval performance by distilling the knowledge from the rewritten queries to the original queries. However, manually-rewritten queries are usually not available, and annotating them requires significant costs; therefore, they are trainable only under specific circumstances. On the other hand, to provide exact answers for the question within the current conversation turn, some other work adapted a retriever-reader pipeline, which can additionally read the query-relevant passages retrieved from a large corpus (Qu et al., 2020; Li et al., 2022c; Adlakha et al., 2022; Fang et al., 2022). However, such a pipeline approach has critical drawbacks due to its structural limitation composed of two sub-modules, thereby requiring additional effort to independently train both the retriever and the reader, both of which are also not runnable in parallel during inference, as well as bounding the reader’s performance to the previous retrieval performance.
Dense Phrase Retrieval Instead of using a conventional pipeline approach, consisting of the retriever and the reader, we propose to directly predict answers for the ODConvQA task based on dense phrase retrieval. Following this line of previous researches, there exists some work that proposed to directly retrieve phrase-level answers from a large corpus; however, such work mainly focuses on non-conversational domains, such as question
answering and relation extraction tasks (Seo et al., 2019; Lee et al., 2021a,b). Specifically, the pioneering work (Seo et al., 2019) used both of the sparse and dense phrase representations for their retrieval. Afterwards, Lee et al. (2021a) improved the phrase retrieval model that uses only dense representations without using any sparse representations, resulting in improved performance while reducing the memory footprint. Motivated by its effectiveness and efficiency, several work recently proposed to use the dense phrase retrieval system in diverse open-retrieval problems (Lee et al., 2021b; Li et al., 2022b; Kim et al., 2022); however, their applicability to our target ODConvQA has been largely underexplored. Therefore, in this work, we adapt dense phrase retrieval to the ODConvQA task for the first time, and further propose to model conversational dependencies in phrase retrieval.

heading: 3 Method
text: In this section, we first define the Conversational Question Answering (ConvQA) task, and its extension to the open-domain setting: Open-Domain ConvQA (ODConvQA) in Section 3.1. Then, we introduce our dense phrase retrieval mechanism to effectively and efficiently solve the ODConvQA task, compared to the conventional retriever-reader pipeline approach, in Section 3.2. Last, we explain our novel conversational dependency modeling strategy via contrastive learning, in Section 3.3.

heading: 3.1 Preliminaries
text: In this subsection, we first provide general descriptions of the ConvQA and the ODConvQA tasks.
Conversational Question Answering Let qi be the question and ai be the answer for the i-th turn of the conversation. Also, let p∗i a reference passage, which contains the answer ai for the question qi. Then, given qi, the goal of the ConvQA task is to correctly predict the answer ai based on the reference passage p∗i and the previous conversation histories: {qi−1, ai−1, ..., q1, a1}. Here, for the simplicity of the notation, we denote the i-th conversational context as the concatenation of the current input question and the previous conversation histories, formally represented as follows:
Convi = {qi, qi−1, ai−1, ..., q1, a1}. (1)
Then, based on the notation of the conversational context Convi, we formulate the objective of the
ConvQA task with a scoring function f , as follows:
f(ai|Convi) = Mcqa(p∗i , Convi; θcqa), (2)
where Mcqa is a certain ConvQA model that predicts ai from p∗i based on Convi, which is parameterized by θcqa. However, this setting of providing the reference passage p∗i containing the exact answer ai is largely unrealistic, since such the gold passage is usually not available when conversing with users in the real-world scenario. Therefore, in this work, we consider the more challenging open-domain ConvQA scenario, where we should extract the answers within the query-related documents from a large corpus, such as Wikipedia.
Open-Domain ConvQA Unlike the ConvQA task that aims at extracting the answers from the gold passage p∗i , the ODConvQA task is required to search a collection of passages for the relevant passages and then extract answers from them. Therefore, the scoring function f of the ODConvQA task is formulated along with the certain passage pj from the large corpus P , as follows:
f(ai|Convi) = Modcqa(pj , Convi; θodcqa), with pj ∈ P, (3)
where Modcqa is an ODConvQA model parameterized by θodcqa, and P is a collection of passages.
Retriever-Reader To realize the scoring function in Equation 3 for ODConvQA, the retrieverreader pipeline approach is dominantly used, which first retrieves the top-K query-relevant passages and then reads a set of retrieved passages to answer the question based on them. Therefore, for this pipeline approach, the scoring function f is decomposed into two sub-components (i.e., retriever and reader), formally defined as follows:
f(ai|Convi) = Mretr(PK |Convi; θretr) ×Mread(ai|PK ; θread), (4)
where the first-stage retriever Mretr and the secondstage reader Mread are parameterized with θretr and θread, respectively. Also, PK indicates a set of top-K query-relevant passages, which are retrieved from the large corpus, PK ⊂ P , based on the retriever Mretr. However, such a retrieverreader pipeline is problematic for the following reasons. First, it is prone to error propagation from the retriever to the reader, since, if Mretr retrieves
irrelevant passages PK that do not contain the answer such that ai /∈ PK , the reader Mread fails to answer correctly. Second, it is inefficient, since Mread requires the Mretr’s output as the input; therefore, Mretr and Mread are not runnable in parallel. Last, it demands effort to construct both Mretr and Mread.

heading: 3.2 Dense Phrase Retrieval for ODConvQA
text: In order to address the aforementioned limitations of the retriever-reader pipeline for ODConvQA, in this work, we newly formulate the ODConvQA task as a dense phrase retrieval problem. In other words, we aim at directly retrieving the answer ai, consisting of a sequence of words (i.e., phrase), based on its representational similarity to the conversational context Convi via the dense phrase retriever (Lee et al., 2021a). Formally, the scoring function for our ODConvQA based on the phrase retrieval scheme is defined as follows:
f(ai|Convi) = EConvQ(Convi)⊤EA(ai), (5)
where EConvQ and EA are encoders that represent the conversational context Convi and the phraselevel answer ai, respectively. Also, ⊤ symbol denotes inner product between its left and right terms. We note that this phrase retrieval mechanism defined in Equation 5 is similarly understood as predicting the answer in the reading comprehension task (Rajpurkar et al., 2016; Seo et al., 2017). To be specific, in the reading comprehension task, we predict the start and end tokens of the answer ai located in the gold passage p∗i . Similarly, in the phrase retrieval task, we directly predict the start and end tokens of the answer which is located within one part of the entire total passages P; therefore, all words in all passages are sequentially pre-indexed and the goal is to find only the locations of the answer based on its similarity to the input context, e.g., Convi. Note that this phrase retrieval approach simplifies the conventional two-stage pipeline approach, commonly used for ODConvQA tasks, into the single direct answer retrieval, by removing the phrase reading done over the retrieved documents.
The training objective of the most information retrieval work (Karpukhin et al., 2020; Qu et al., 2021) is to rank the pair of the query and its relevant documents highest among all the other irrelevant pairs. Similar to this, our training objective with a
dense phrase retriever is formalized as follows:
Lneg = − log ef(a
+,Convi)
ef(a+,Convi) + N∑ k=1
ef(a−,Convi) , (6)
where, for the context Convi, a+ is the positive answer phrase and a− is the negative answer phrase. We describe how to construct the negative contextphrase pairs and additional details for training of the dense phrase retriever in the paragraph below.
Training Details In order to improve the performance of the dense phrase retriever, we adopt the existing strategies following Lee et al. (2021a). First of all, we construct the negative samples, used in Equation 6, based on in-batch and pre-batch sampling strategy. Specifically, for the B number of phrases in the batch, (B − 1) in-batch phrases are used for negative samples by excluding one positive phrase with regard to the certain conversation context. Also, given the preceding C number of batches, we can obtain the negative phrases for the current conversation context with a size of (B×C). In addition to negative sampling, we use the queryside fine-tuning scheme, which optimizes only the conversational question encoder, EConvQ, by maximizing the representational similarities between the correctly retrieved phrases and their corresponding conversational contexts after the phrase indexing. Last, to further improve predicting the start and end spans of the phrase retriever, we first train the reading comprehension model and then distill its knowledge, by minimizing the KL divergences of span predictions between the reading comprehension model and the phrase retriever. For more details, please refer to Lee et al. (2021a).

heading: 3.3 Conversational Dependency Modeling
text: While Equation 6 effectively discriminates positive answer phrases from negative answer phrases, relying on it is sub-optimal when solving the ODConvQA task, where each conversational turn shares a similar context with its previous turn. In other words, since information-seeking conversational questions are asked in a sequence, two consecutive contexts, Convi−1 and Convi, should have similar representations compared to the other turns from different conversations. Therefore, we further model such a conversational dependency by maximizing the similarity between the sequential turns while minimizing the similarity between the other
irrelevant turns via contrastive learning as follows:
Lturn = − log ef(Convi,Convi−1)
ef(Convi,Convi−1) + B−1∑ k=1 ef(Convi −,Convi−1)
,
(7) where Convi− comes from a collection of the irrelevant conversation turns within the batch. By optimizing the objective in Equation 7, the encoder EConvQ represents the current conversational turn Convi probably similar to its previous turn Convi−1; therefore, the retrieved phrase captures both the current and previous conversational contexts.
Overall Training objective We optimize the phrase retrieval loss from Equation 6 and conversational dependency loss from Equation 7 as follows:
L = λ1Lneg + λ2Lturn, (8)
where λ1 and λ2 are the weights for each loss term.

heading: 4 Experimental Setups
text: In this section, we explain datasets, metrics, models, and implementation details.

heading: 4.1 Datasets and Metrics
text: OR-QuAC OR-QuAC (Qu et al., 2020) is the benchmark ODConvQA dataset, which extends a popular ConvQA dataset, namely QuAC (Choi et al., 2018), to the open-retrieval setting. This dataset consists of 35,526 conversational turns for training, 3,430 for validation, and 5,571 for testing.
TopiOCQA TopiOCQA (Adlakha et al., 2022) is another ODConvQA dataset that considers the topic-switching problem across different conversational turns. This dataset contains 45,450 conversational turns and 2,514 turns for training and validation, respectively. Note that we use a validation set since the test set is not publicly open.
Evaluation Metrics We evaluate all models with F1-score and extact match (EM) following the standard protocol on the ODConvQA tasks (Qu et al., 2020; Adlakha et al., 2022). Also, for retrieval performances, we use the standard ranking metrics: Top-K accuracy, mean reciprocal rank (MRR), and Precision, following Lee et al. (2021b).

heading: 4.2 Baselines and Our Model
text: We introduce the baselines with a retriever-reader pipeline, which is dominantly adopted for ODConvQA. We do not compare against the incomparable
baselines that use the additional data, such as rewritten queries (Yu et al., 2021; Fang et al., 2022).
BM25 Retriever + DPR Reader This is one of the most widely used retriever-reader pipeline approaches that first retrieves query-relevant passages with a sparse retriever, BM25 (Robertson et al., 1994), and then reads top-k retrieved passages with a DPR reader (Karpukhin et al., 2020).
DPR Retriever + DPR Reader This pipeline uses a dense retriever for the first retrieval stage, DPR retriever (Karpukhin et al., 2020), which calculates the similarity between a query and passages on a latent space, instead of using a sparse retriever.
ORConvQA This model consists of a dense retriever and a reader with an additional re-ranker, which is trained with two phases (Qu et al., 2020): 1) retriever pre-training and 2) concurrent learning. Specifically, it first trains the retriever and generates dense passage representations. Then, the model further trains the retriever, reader, and reranker using the pre-trained retriever and generated passage representations.
PRO-ConvQA(Ours) This is our model that directly retrieves answers without passage reading, trained jointly with contrastive learning to further address a conversational dependency issue.

heading: 4.3 Implementation Details
text: We implement ODConvQA models using PyTorch (Paszke et al., 2019) and Transformers library (Wolf et al., 2020). For all the models, we use the 2018-12-20 Wikipedia snapshot having a collection of 16,766,529 passages. We exclude the questions with unanswerable answers, since we cannot find their answers with the corpus, which is not suitable for the goal of the open-retrieval problem. Furthermore, as our model answers questions extractively, we convert TopiOCQA with the gold answers in a free-form text to our extractive setting by considering the provided rationale as the gold answers, following the existing setting from Jeong et al. (2023). For training PRO-ConvQA, we set
Top-1 MRR Precision 0 5
10 15 20 25 30 35 40 45
BM25+DPR DPR+DPR ORConvQA PRO-ConvQA (ours)
Figure 2: Retrieval results on OR-QuAC, measured with Top-1 accuracy, MRR, and Precision. Note that we limit the number of total retrieved documents for MRR and Precision to 10.
the batch size (B) as 24 and the pre-batch size (C) as 2. Also, We train PRO-ConvQA with 3 epochs with a learning rate of 3e− 5 and further fine-tune a query encoder with 3 epochs. We set λ1 and λ2 as 4 and 1 for OR-QuAC and 2 and 1 for TopiOCQA, respectively. For computing resources, we use two GeForce RTX 3090 GPUs with 24GB memory. For retriever-reader baselines, we retrieve top-5 passages to train and evaluate the reader, following Qu et al. (2020). Also, due to the significant costs of evaluating retrieval models, we perform experiments with a single run.

heading: 5 Results and Discussion
text: In this section, we show the overall results and provide detailed analyses.
Main Results As Table 1 shows, our proposed PRO-ConvQA model significantly outperforms all baselines with a retriever-reader pipeline on two benchmark datasets. This implies that the twostage models might be susceptible to error propagation between the retrieval and reader stages, therefore ineffectively bounding the overall performances when a model fails to correctly retrieve reference passages during the first stage. However, our PRO-ConvQA is free from such a bottleneck problem, since it directly retrieves answer phrases, without requiring an additional reader.
Interestingly, a recent ORConvQA model shows largely inferior performances on the TopiOCQA dataset. Note that for TopiOCQA, target passages of two consecutive conversation turns sometimes have different topics, compared to the OR-QuAC dataset where all passages within the whole conversation share a single topic. Therefore, TopiOCQA follows a more realistic setting where a topic constantly changes during the conversation. However, note that ORConvQA is not trained in a truly end-to-end fashion, since it first retrieves passage embeddings from a pre-trained retriever,
and then uses the already encoded passage embeddings when concurrently training a retriever, reader, and re-ranker. Therefore, ORConvQA is vulnerable to such a topic-shifting situation, as the passage encoder and embedding are not updated during a concurrent training step. Meanwhile, our PRO-ConvQA is trained in an end-to-end fashion, thereby effectively learning to retrieve phrases.
Similarly, using BM25 as a first-stage retriever also shows a large performance gap between the two datasets. Note that BM25 lexically measures relevance between a conversational turn and a passage by counting their overlapping terms. Therefore, compared to the other dense-retrieval-based two-stage models, this unique characteristic of BM25 brings additional advantages on the ORQuAC dataset, where each conversational turn revolves around the same topic. More specifically, the conversational history, which is accumulated during each turn, becomes very relevant to the target retrieval passage as the conversation progresses. However, such a lexical comparison scheme fails to effectively retrieve the passages when a topic slightly changes for each conversation turn on TopiOCQA, since it cannot capture a semantic interrelationship between conversational turns and a passage. On the other hand, our PRO-ConvQA shows robust performances on both datasets by retrieving the phrases over the semantic representation space. We further analyze the strengths of the PRO-ConvQA in the following paragraphs.
Effectiveness on Retrieval Performance In order to validate whether a failure of the retriever works as a bottleneck in a two-stage pipeline, we measure retrieval performances in Figure 2. Compared to the PRO-ConvQA, the models based on the retriever-reader pipeline fail to correctly retrieve relevant reference passages, thus negatively leading to the degenerated overall performance. This result corroborates our hypothesis that there exists a bottleneck problem in the first retrieval
stage. Furthermore, this result demonstrates that our PRO-ConvQA also effectively retrieves the related passages at a phrase level, even though it is not directly designed to solve the conversational search task that aims at only retrieving the passages related to each conversational turn.
Efficiency on Inference Time In the real world, inference speed for returning answers to the given questions is crucially important. Thus, we report the runtime efficiency of our PRO-ConvQA against the other baselines in Table 2. Note that PROConvQA is highly efficient for searching answer phrases over the baselines with a retriever-reader pipeline. This is because retrieval and reader stages cannot be run in parallel, since the latter reader stage requires the retrieved passages as the input. On the other hand, our proposed PRO-ConvQA is simply composed of a single phrase retrieval stage with two decomposable encoders, as formulated in Equation 5. This decomposable feature enables maximum inner product search (MIPS), thus contributing to fast inference speed.
Ablation Studies To understand how each component in the PRO-ConvQA contributes to performance gains, we provide ablation studies in Table 3. As shown in Table 3, our contrastive learning for conversational dependency modeling and also query-side fine-tuning strategies positively contribute to the overall performance. Furthermore, the significant performance drops when removing each component indicate that there exists a complementary relation between the two components.
Zero-shot Performance In order to apply ODConvQA models in a real-world scenario, one may consider a zero-shot performance since highquality training data is not always available. Therefore, we show zero-shot performances, assuming that the target training data is only available for ORQuAC, but not for TopiOCQA. As Figure 3 shows, the proposed PRO-ConvQA outperforms the base-
line models by a large margin. This implies that such a zero-shot setting is challenging to the previous ODConvQA models, since they are trained and tested in a different topic-shifting setting; they are trained to assume that each turn shares the same topic within a conversation, but tested in a situation where the topic changes as the conversation proceeds. However, PRO-ConvQA is more robust than other baselines in a zero-shot setting, since its training objective aims at retrieving answers at a phrase-level, rather than a passage-level, which enables capturing topic shifts with more flexibility.
Efficient Transfer Learning Besides a zeroshot performance, transferability between different datasets is another important feature to consider in a real-world scenario. In particular, it would be efficient to reuse a dump of phrase embeddings and indexes even if the target data changes, with respect to the training effort and disk footprint for storing a large size of embeddings and indexes. As we have validated the effectiveness of fine-tuning a query encoder in Table 3, it would be more efficient if we could only update the query encoder to adapt to the newly given data, without re-training everything from scratch. To see this, we conduct an experiment in a transfer learning scenario, where a phrase retrieval model is trained on OR-QuAC, but the query-side encoder is further fine-tuned for TopiOCQA and tested on it. As Figure 3 shows, fine-tuning a query-side encoder further improves the performance when compared to the zero-shot model. This indicates that PRO-ConvQA can be efficiently adapted to diverse realistic settings, only compensating a little amount of costs for adaption.
Generative Reader While our PRO-ConvQA shows outstanding performances under the extractive reader setting, it is also possible to further combine PRO-ConvQA with a recent generative reader model, Fusion-in-Decoder (FiD) (Izacard and Grave, 2021). We conduct experiments with
the publicly available FiD model1, which is already trained on TopiOCQA, without any further training. As Figure 4 shows, our PRO-ConvQA consistently shows superior F1 and EM scores under the generative reader setting, compared to the DPR baseline. This is because PRO-ConvQA is superior in passage-level retrieval as shown in Figure 2, which further leads to accurately answering questions with correctly retrieved passages. Also, we believe that the performance would be further improved by additionally training a FiD model on the retrieved passages from PRO-ConvQA, instead of using an already trained one.

heading: 6 Conclusion
text: In this work, we pointed out the limitations of the retriever-reader pipeline approach to ODConvQA, which is prone to error propagation from the retriever, unable to run both sub-modules in parallel, and demanding effort to manage these two submodules, due to its decomposed structure. To address such issues, we formulated the ODConvQA task as a dense phrase retrieval problem, which makes it possible to directly retrieve the answer based on its representational similarity to the current conversational context. Furthermore, to model the conversational dependency between the current and its previous turns, we force their representations to be similar with contrastive learning, which leads to retrieving more related phrases to the conversational history as well as the current question. We validated our proposed PRO-ConvQA on ODConvQA benchmark datasets, showing its efficacy in effectiveness and efficiency.
Limitations
As shown in Table 3, the contrastive learning strategy to model the conversational dependencies between the current and previous conversational turns is a key element in our phrase retrieval-based OD-
1https://github.com/McGill-NLP/topiocqa
ConvQA task. However, when the current conversational topic is significantly shifted from the previous topic as the user may suddenly come up with new ideas, our contrastive learning strategy might be less effective. This is because modeling the conversational dependency is, in this case, no longer necessary. While we believe such situations are less frequent, one may further tackle this scenario of significant topic switching, for example, with history filtering, which we leave as future work.
Ethics Statement
We show clear advantages of our PRO-ConvQA framework for ODConvQA tasks compared to the retriever-reader approach in both effectiveness and efficiency perspectives. However, when given the conversational context from malicious users who ask for offensive and harmful content, our PROConvQA framework might become vulnerable to retrieving toxic phrases. Therefore, before deploying our PRO-ConvQA to real-world scenarios, we have to ensure the safety of the retrieved phrases.

heading: Acknowledgements
text: This work was supported by Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea government (No. 2018-0-00582, Prediction and augmentation of the credibility distribution via linguistic analysis and automated evidence document collection).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: See the ’Limitations’ section, after the conclusion.

heading: 3 A2. Did you discuss any potential risks of your work?
text: See the ’Ethics Statement’ section, after the conclusion.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: See the ’Abstract’ and ’1. Introduction’ sections.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? See ’4. Experimental Setups’.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: See ’4. Experimental Setups’.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No, but we followed their licenses.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No, but we followed their licenses.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. See ’4. Experimental Setups’.
C 3 Did you run computational experiments? See ’4. Experimental Setups’.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? See ’4. Experimental Setups’.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? See ’4. Experimental Setups’.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? See ’4. Experimental Setups’.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? See ’4. Experimental Setups’.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: While we believe such situations are less frequent, one may further tackle this scenario of significant topic switching, for example, with history filtering, which we leave as future work