Multi-Task Learning of Query Generation and Classification for Generative Conversational Question Rewriting
Authors: Sarawoot Kongyoung
Craig MacDonald
Iadh Ounis
Venue: EMNLP 2023 Findings (2023)
Keywords: multi-task learning
follow-up question identification
conversational question rewriting
text generation model
TL;DR: A Multi-Task Learning (MTL) approach that uses a text generation model for both question rewriting and classification.

Abstract: In conversational search settings, users ask questions and receive answers as part of a conversation. The ambiguity in the questions is a common challenge, which can be effectively addressed by leveraging contextual information from the conversation history. In this context, determining topic continuity and reformulating questions into well-defined queries are crucial tasks. Previous approaches have typically addressed these tasks either as a classification task in the case of topic continuity or as a text generation task for question reformulation. However, no prior work has combined both tasks to effectively identify ambiguous questions as part of a conversation. In this paper, we propose a Multi-Task Learning (MTL) approach that uses a text generation model for both question rewriting and classification. Our models, based on BART and T5, are trained to rewrite conversational questions and identify follow-up questions simultaneously. We evaluate our approach on multiple test sets and demonstrate that it outperforms single-task learning baselines on the three LIF test sets, with statistically significant improvements ranging from +3.5% to +10.5% in terms of F1 and Micro-F1 scores. We also show that our approach outperforms single-task question rewriting models in passage retrieval on a large OR-QuAC test set.

Multi-Task Learning of Query Generation and Classification for Generative Conversational Question Rewriting
Sarawoot Kongyoung', Craig Macdonald Iadh Ounis
University of Glasgow, UK
is. kongyoung 1 gla. ala.ac... ac. .ucuk
2 serrii..mcddonale. iadh. ounis) (omnis)@glaaa ac. uk
Abstract
System
qi: What does Sheldon have to do with the Israeli press?
proceeded parailel to publish a free dally newspaper to newspaper had co founded in 2005
conversational search settings, users ask questions and receive answers part of a con- versation. The ambiguity in the questions is common challenge, which can be effectively addressed by leveraging contextual information from the conversation history. In this context, determining topic continuity and reformulating questions into well-defined queries crucial tasks. Previous approaches have typically ad- dressed these tasks either classification task in the case of topic continuity or as text gener- ation task for question reformulation. However, no prior work has combined both tasks to effec- tively identify ambiguous questions as part of conversation. In this paper, we propose Multi- Task Learning (MTL) approach that generation model for both question rewriting and classification. Our models, based BART and T5, are trained to rewrite conversational questions and identify follow-up questions si- multaneously. We evaluate our approach on multiple test sets and demonstrate that it out- performs single-task learning baselines on three LIF test sets, with statistically significant improvements ranging from +3.5% to + +0.5% terms Fl and Micro-Fl scores. also show that approach outperforms single-task question rewriting models in passage retrieval on large OR-Qu OR-QuAC test set.
successful paper?
a2: The first edition the new newsppapin, Istant Hayom, was published July
43 : Was the owner of the paper?
Was Shelderr
of
a3: Iu hurd ctn to-faauddei
Israeli press?
64: What are his favorite outside of the media industry?
Figure I: An illustrative example of dialogue.
To address such ambiguities, number of approaches have been explored, such follow-up question identification (Bertomeu al., 2006; Kirschner and Bernardi, 2007, 2009; Kundu al., 2020) and conversational question rewriting (Lin et al., 2020; Mele et al., 2021; Ren et al., 2018; Vakulenko et al., 2021a,b; Voskarides et al., 2020; Yu et al., 2020). Indeed, the follow-up question identification approaches have been used in Conversational QA to enhance the interpretation of the user's intent and context. The main objective of these approaches is determine whether follow-up question is linked to the previous conversation history or not. If determined that the follow-up question is related to the previous conversation, the system can leverage the context provided by the conversation to generate more precise and relevant responses. In Figure 1, a follow-up question (g3) is classified as valid if it be linked to the previous conversation (gL, a1, Q2, 02), else it is classified as invalid (c.g.. Q4). For example, the three-way way attentive pooling network approach (Kundu et al., 2020) predicts this continuity by analysing the candidate question and conversation history, outperforming other models such as BiLSTM, CNN, and BERT (Devlin et al., 2019). In this paper, attentive pooling network as strongest baseline. On the other hand, conversational question rewriting approaches have been employed
text
1 Introduction
Conversational Question Answering (QA), which simulates human dialogues in information-seeking tasks, necessitates resolving ambiguities in the user queries based on the conversation history (Kundu et al., 2020). Figure 1 exemplifies the possible ambiguities that might arise from typical dialogue with user, highlighting the importance of address- ing these ambiguities based on the conversation history. For example, in the question q3. "Was he the owner of the paper?", the "he" and "the paper" referents are context-dependent, typically clarified by prior questions (q), Q2) and answers (a1, 02).
the three-way

---

Conversational QA enhance the accuracy of the retrieved information by reformulating the user's original question. These approaches usually involve modifying the original question or gener- ating new queries that better represent the user's intent by transforming concise conversational question into fully-grown, contextualised ad-hoc query. For example, in Figure 1, the question 4' "Was Sheldon the owner of the Israeli press?" is a rewrite of q3 "Was he the owner of the paper?", based on the conversation history (ς), 01; 92. 02). A T5 (Raffel et al., 2020) model can be fine-tuned to automatically reformulate the question by injecting information that exists in the context into a fully defined query (Lin et al., 2020). Previous work, such as (Lin et al., 2020; Mele et al., 2021; Ren et al., 2018; Vakulenko et al., 2021a,b; Voskarides al., 2020; Yu et al., 2020), focused on conversational question rewriting; however, they did not address the task of follow-up question identification. In this paper, to alleviate ambiguities in conversational QA, we investigate the combination of both the follow-up question identification task and the conversational question rewriting task into single framework, thereby also improving the effectiveness of both tasks.
1/0
Head
Li HEAD
Transformer
prompt:
au/s, [RP9]
XFP
MTL Discriminative Generative
tolliww/ahtr
Transformer
prompt:
e./s
(mh)
(G) Generative
Figure 2: Comparison of MTL models: (a) discrim- inative+generative model with separate heads for the classification and question rewriting tasks, and (b) a generative model with combined token generation for both tasks.
relevance can be enhanced. Indeed, by identifying connections between the user's questions, address- ing ambiguities, and leveraging the conversation' 's context, the system can refine its understanding of the user intent and can provide more precise and relevant responses. Our text generation models leverage the Multi-Task Learning of the conversa- tional question rewriting and classification tasks to identify whether a question a follow-up to the previous question and, accordingly, reformulate question using the dialogue context. To the best of our knowledge, no prior work has inherently combined both tasks to more effectively address ambiguity in conversational questions.
Recently, Multi-Task Learning (MTL) has emerged effective approach for simultane- ously learning numerous related tasks (Ide and Kawahara, 2021; Kongyoung et al., 2022). MTL can be used increase a system's performance on the text generation task by leveraging classification tasks. For example. Ide and Kawahara (2021) adopted an MTL approach using a text generation BART (Lewis al., 2019) model, which jointly learns classification task and text generation task by sharing the learner, and showed im- provement in an emotion-aware dialogue response generation model. Text generation models like T5 have also been effectively used for multi-task learning involving both text generation and clas- sification tasks, such as passage ranking/re-ranking and answer generation/extractoo (Kongyoung et al., 2022; Lee et al., 2022; Jiang et al., 2022). Given their effectiveness various tasks, argue that text generation models BART and T5 can be tailored for the follow-up question identification and conversational question rewriting tasks.
Our contributions are as follows: (1) We lever- age Multi Task Learning with a text generation model to effectively address the tasks of follow-up question identification and conversational question rewriting: (2) Using the recent LIF dataset (Kundu et al., 2020), we compare our models to two recent baselines from the literature, and show that Multi-Task Learning BART model yields the best Fl and Macro-F1 performance improvements over the strongest baseline, three-way attentive pooling, with statistically significant improvements rang- ing from 3.5% to 10.5% on all LIF test sets; (3) Our proposed Multi Task Learning T5 model sig- nificantly outperforms the single-task learning of question rewriting models for passage retrieval on the OR- OR-QuAC test set.
Multi-Task
MTL: Classification & Generation
We define the follow-up question identification and conversational question rewriting tasks
Our intuition is that by combining follow-up question identification and conversational question rewriting, the system's response accuracy and
Sections 2.1 & 2.2. An overview of the proposed text generation model follows in Section 2.3.

---

2.1 Follow-up Question Identification
generation. Such text generation approaches can be trained to generate meaningful textual response based on some input text. Moreover, like BERT, the pre-trained BART and T3 models can be fine-tume mee-tuned to perform a variety of downstream tasks.
Following Kundu et al. (2020), we consider the following inputs: list of previous ques- tions, list of their corresponding ground truth answer (respectively denoted as 92..9..... 12.............................. .ie and (a1, 02, a2.... ag] where the number of the previous question- guestion-answer answer pair in the conversation history), and a candidate follow-up question Gc- Given these inputs, the first task we address is to predict whether or not the candidate follow-up question qe is valid follow-up question. Figure 1 exemplifies the follow-up question identification task, showing history of length & = with the corresponding questions and answers. particular, as this task is a binary classification task, the aim a follow-up question identification approach is to classify a question qe as a valid follow-up question or as invalid.
In addition, the manner in which text genera- tion model is used in classification tasks can dif- fer, as they can be fine-tuned as discriminative generative models. In a discriminative setup, the model adapted for binary classification by adding a fully-connected layer with two output neurons (corresponding to each class) upon special [CLS] token in BERT, or the last token in BART. In con- trast, generative setup reframes NLP tasks as text generation tasks for instance, classification is per- formed by examining what text is generated and the corresponding likelihood.
An MTL Text Generation Approach: To adapt an MTL approach to a text generation model for jointly learning from both the classification and question rewriting tasks, the model can be used in either diserininaatieeeeeerattve or in gen- erative setup shown in Figure 2. A discrimi- native+generative MTL model makes predictions by applying a CLS head to create a score for a classification task and an LM head to generate the tokens for a question rewriting task (Ide and Kawa- hara, 2021) as shown in Figure 2 (a). In contrast, a generative MTL model makes predictions by gen- erating the first token for classification task and the follow-up tokens for query rewriting task shown in Figure 2 (b). Notably, while text gen- eration models like BART can function in either the discrimiinative-eenraaiiee or generative MTL sctups, the T5 model is exclusively applicable as a generative MTL model (Raffel et al., 2020).
2.2 Conversational Question Rewriting
Following Elgohary al. (2019), given conver- sation history Hg consisting of list of ques- tions and list of ground truth answer pairs, i.e H& Kg, a)), the task is to generate a rewrite d'm for the next question qm based on Hy- Be- cause Qm is part of the conversation, its meaning frequently includes references to parts of Hμ. A valid o'm should be self-containedd: i.c. correct answer to om without the history Hş is correct answer to 9m with the history Hg. Figure I exem- plifies the conversational question rewriting task, showing a history length k with the corre- sponding questions and answers. The question g3 omits the first question (replacing the pronoun "he" with Sheldon and replacing "the page" with Isracli press) ). Hence, to address this task, the system needs to resolve any omission by using history Hg Next, describe the MTL approaches com- bine the follow-up question identification with the conversational question rewriting.
In particular, when fine-tuning the T5 model for downstream task, a prefix text is required for example "translate English to German:" might be used for a translation task. Indeed, the text generation models have been shown to achieve state-of-the-art performances in classifi- cation (Lewis et al., 2019; Raffel et al., 2020), as well as in document re- re-ranking - by ranking based on the likelihood of generating particular token (Nogucira et al., 2020; dos Santos et al., 2020) (outperforming BERT models) and even in arithmetic tasks (Nogueira al., 2021). Hence, for the MTL. of both the follow-up question identification and question rewriting tasks we choose the MTL generative version of the BART
2.3 Model Overview
To tackle the tasks described in Sections 2.1 and 2.2, we propose classification and question rewriting models that leverage historical questions to identify whether candidate question de is follow-up to the previous question and to reformulate the current question qm- Our proposed method uses models including BART (Lewis et al., 2019) and T5 (Raffel et al., 2020), which are large pre-trained language models designed for text

---

and '5 models. However, for comparison purposes. we also deploy the discriminativeegenrraaii versions of the BART models our experiments.
LIF CANARD
LIF
OR-QuAC
Train
Dev
Test-1
Test-11
Test III
Test
Aquestions
62X19
4914
5,992
5,247
2,685
Avall Follow-ap
22,056
1,359
1,940
1,400
1.940
Ainvalid follow--p
40.783
3,355
4,052
3,807
745
More precisely, we deploy generative MTL mod- els capture the relation between the question (m/qm and the contextual information in the con- versation history, including the historical ques- tion(s) {qi, 92. -
Table Statisties of the used datasets
and the rewritten question. The loss function Egen measures the discrepancy between the generated tokens and the ground-truth tokens. To summarise, our MTL method involves fine-tuning the gener- ative model simultaneously generate sequence tokens for the follow-up question identification and conversational question rewriting.
9k J and the historical answer(s) {ai, d2.... } as shown in Figure 2 (b). In par- ticular, let (-n denotes a generative transfor- mation function by taking the input sequence as follows:
Gen( (onnpprompt :" de "[SEP]" H&)
(1)
In addition to the implementation of the gener- ative MTL model, we also incorporate discrim- inative+generative MTL model as shown in Fig- ure (a), and described in Appendix A.I.
Gen(.) = 11, u2,..., Wn
(2)
where "prompt:" prefix text, and "[SEP]" is a special token. The model is then fine-tuned to generate the target tokens length as shown in Equation (2), for the token wi namely "follow" whether the candidate
Experimental Setup
"shift" depending question is valid follow-up to the previous question or not, while the follow-up tokens ..., un are the output sequence for the target query. In particular, a prompt function, as shown in Equation (1), is employed to format and combine the question (4e) and the conversational history (q1-..Qk- 01...a)), creating well-structured input sequence for Gen(-). Subsequently, Gen(-) generates the contextual representation h. Once the contextual representation h is obtained, it is used by the Gen(-) decoder. This decoder takes the previously generated tokens as input and performs attention over h, enabling it to generate the subsequent token. Specifically, when given the tuple (ge. Qk. 01...0),, our training objective aims to minimise the following loss function:
We list our research questions in Section 3.1, introduce the used datasets in Section 3.2 and present our baselines Section 3.3
3.1 Research Questions
We address two main research questions:
RQI: Does the MTL approach of question rewriting and classification using text generation models outperform the single-task learning (STL) of text generation models and cxisting baselines for the follow-up question identification task?
RQ2: Does the MTL approach of question rewriting and classification using text generation models outperform the single-task learning (STL) of text generation models in the context of the conversational question rewriting and passage retrieval tasks?
M
C ggn
Σ log P(wa (h, w.))
3.2 Datasets
We experiment using the LIF dataset (Learning Identify Follow-Up) (Kundu et al., 2020) and the CANARD dataset (Context Abstraction: Necessary Additional Rewritten Discourse) (Elgohary et al., 2019), which are recent adaptation of the well- known QuAC Conversational QA dataset (Choi et al., 2018). For training the models to address both tasks, in the training and development sets, we integrate LIF and CANARD by picking only the candidate questions from the LIF dataset that exist in the CANARD dataset. To evaluate the models in the follow-up question identification task, we use the three test sets of the LIF dataset, namely Test-I, Test-II, and Test-III. In all three test sets,
where M is the number of tokens in the target sequence, which consists of the ground-truth follow-up question identification token w1 fol- lowed by the tokens in the manually rewritten question o/ In addition, 4/,, refers to the 7h token in g/n when 2 2 and the token we is the beginning of the sequence token (
). Our goal to train the model to generate subsequent tokens using the ground-truth follow-up question identification
We choose "follow" and "shift" as target tokens be- cause T5 tokenises sequences using the SentencePicce ap- proach (Kudo and Richardson, 2018), which splits the word "invalid" into two subwords.

---

the valid follow-up questions (label 1) are con- structed from the "should ask up question" instances in the QuAC dataset. For the invalid up questions (label = 0). Test-I combines the invalid instances from Test-Il & Test-III. For Test-II, questions with high similarity to the cur- rent passage are sampled from other conversations. On the other hand, for Test-Ill, the invalid follow- up questions are sampled from the non-follow-up questions of the same conversation in QuAC. For the question rewriting task, we use the test sets of the OR-QuAC dataset. We also aggregate pas- sage collection from the OR- ---ACC dataset (Qu et al., 2020), which is an aggregation of three exist- ing datasets consisting of QuAC, CANARD, and Wikipedia, to evaluate the passage retrieval perfor- mance of the conversational question rewriting task. This allows us assess our model's performance across both the conversational question answering and passage retrieval tasks. For further information about the used datasets, we also provide a summary of their statistics in Table 1.
only generate the rewritten question on. We com- pare them in terms of both conversational question rewriting and passage retrieval effectiveness.
Hyperparameter settings and evaluation metrics: We provide description of the hyper- parameter settings and the used evaluation metrics in Appendix A.2 and A.3, respectively.
4 Experimental Results
We now address RQs 1 & 2, and conclude with a qualitative analysis.
4.1 RQ1: Follow-up Identification
We investigate the performance of the baselines described in Section 3.3 in comparison our pro- posed MTL text generation models for follow-up question identification on all three test sets of the LIF dataset. Table 2 compares our proposed MTL. text generation models to the baselines on all three test sets of the LIF dataset.
Comparison of the MTL Models with the Baselines for Follow-up Question Identification: From the table, on three test sets of the LIF dataset, we see that the generative MTL BART classifier model achieves the highest recall, F1, and Macro- Fl, except MTL T5 in Test-III for Macro-F1. While the best precision scores on all three test sets are obtained by the three-way attentive pooling, discriminative+generaat MTL BART, and MTL T5, respectively. Within the table, on all three test sets, proposed generative MTL BART, significantly outperforms the baseline, BERT, three-way attentive pooling, STL BART (both discriminative and generative), and STL T5 in terms of Fl and Macro F1, except MTL T5 in Test- III for Macro-F1. These results suggest that the generative MTL BART classifier model exhibits strong overall performance and exceeds other models in terms of recall, Fl score, and Macro-F1 score across most test sets. This highlights its ability to accurately predict true positive instances (valid follow-up question) while maintaining a good balance between precision and recall.
3.3 Baselines and Implementation
Follow-up question identification task: We only include neural models existing rule-based and statistical machine learning models have been shown to be much less effective in the follow-up question identification task a previous study (Kundu et al., 2020). Indeed, as baselines, we choose the strongest baseline in the previous study (Kundu et al., 2020), BERT, as well as the tatte-of-theaar (SOTA) three-way attentive pooling model from the same study. For the threc- way attentive pooling model, we reproduce the model and its evaluation results provided by Kundu et al. (2020). We additionally compare our MTL of the generative BART and T5 models with the Single Taske Learning (STL) of T5 and BART. The STL models are only learned to predict whether a given question is a follow-up question. We addi- tionally compare our generative BART model with the discriminative generative version of BART, as described in Section 2.3.
baselines since the
Comparison of the MTL Models with STL: We observe that the generative MTL BART classifier model significantly outperforms the STL BART (both discriminative and generative) models in terms of F1 and Macro-F1 on all three test sets. This indicates that the MTL approach, which jointly trains the model on the follow-up question identification and conversational question
Conversational question rewriting task: We compare query rewriting methods with the following models: Raw: The user's original current question; Manual: The questions are written by hu- mans from the CANARD dataset. We also compare our proposed methods with the Single-Tase Learn- ing (STL) of BART and T5, which are learned to

---

Test-I
Test-Il
Test-III
Models
P
R
F1
Macro-F1
Macro-FI
P
R
Macro-FI
STL
BERT
70.7
79.5
74.9F
80.8f:
85.6
79.5
82.5f
86.4f
80.2
79.5
79.9f
64.2f
3-way AP
71.6
70.0
71.6
79.2
74.4
76.8
75.611
80.51
79.7
70.0
74.6
60.413
BARTídis)
69.7
79.4
74.2
80.3ft
85.7
79.4
82.51
86.4f
78.9
79.4
79.1
62.1f5
BART(gen)
71.0
79.7
75.1ft
81.0f
87.4
79.7
83.4
87.11
79.1
79.7
79.41
62.513
69.9
83.0
75.91
81.41
85.3
83.0
84.21
87.51
79.5
83.0
81.21
64.0f
MTL
BART (dis+gen)
71.5
84.6
77.5
82.6
86.3
84.6
85.4
88.5
80.6
84.6
82.5
66.4
BART (gen)
70.3
87.3
77.9
82.7
84.9
86.1
88.9
80.4
87.3
83.7
66.9
T5
71.3
83.5
76.97
82.2
85.4
81.6
83.5f
87.1f
84.6
77.6
80.91
68.9
Table 2: Results for follow-up question identification. f denotes a performance that is significantly worse than the MTL BART model (McNemar's test, < 0.05); $ denotes a performance that is significantly worse than the MTL. model (McNemar's 's test, p < 0.05). 3-way AP denotes Why Attentive Pooling. The highest value for each measure is highlighted.
rewriting tasks, provides a notable advantage over the STL approach for the BART classifier model. However, the MTL T5 model does not outperform the STL T5 model in terms of Fl and Macro-FI, but the two models are not significantly different. Comparison Between Our Generative MTL Models: We observe that the MTL BART classi- fier model significantly outperforms the MTL T5 model in terms of F1 on all three test sets, and also significantly outperforms MTL T5 on Macro- Fl score on Test-ll. Comparing the discrimina- tive+generative and generative MTL BART mod- els, we find that there little difference between the effectiveness of the two versions of the MLT BART models on both F1 and Macro-F1 scores on all three test sets.
BM25 retrieval model. The effectiveness of the monoT5 re-ranker for the same 1000 retrieved pas- sages is listed in the same row. The effectiveness of the manually rewritten questions can be seen an upper bound for the question rewriting methods. Comparison of the MTL Models with the Baselines for Conversational Question Rewrit- ing: In Table 3, we observe that the MTL T5 model achieves the highest ROUGE-1 score by sig- nificantly outperforming all STL baselines, demon- strating its superior performance in capturing the recall of the rewritten questions at the unigram level (individual words). On the other hand, the generative MTL BART model achieves the highest BLEU score by significantly outperforming all STL baselines, indicating its effectiveness in measuring the similarity between the generated texts and the reference texts (the manually rewritten questions). Moreover, we can observe that the generative MTL BART model outperforms the STL BART model in terms of both the ROUGE and BLEU scores. This indicates that the generative MTL BART model achieves a better performance in the conversational question rewriting task compared the STL BART model. Comparing the MTL T5 model to the STL T5 model, the MTL T5 model achieves a higher ROUGE-1 score, indicating a better performance in conversational question rewriting. However, both models have the same BLEU score.
Therefore. in response to RQ1, we find that our MTL text generation model, BART, jointly learned through both the follow-up question identification and conversational question rewriting tasks has the best overall effectiveness, yielding statistically sig- nificant improvements in terms of Fl and Macro-Fl over the baselines, on each of the three test sets of the LIF dataset.
4.2
RQ2: Question Rewriting
Next, we examine the effectiveness of the conver- sational question rewriting models including our proposed MTL text generation models, and those listed in Section 3.3 (STL BART, STL T5, and discriminnaivee+ennrriie MTL BART) on the test set of the OR- QU-QuAC dataset. Table 3 presents the effectiveness of various question reformulation models for conversational question rewriting, evaluated based on the ROUGE-1 recall and BLEU scores. Furthermore, the models' effectiveness for passage retrieval is evaluated when applied with the
Comparison of the MTL Models with Base- lines for Passage Retrieval: From Table 3, we ob- serve that our MTL T5 generative model achieves the highest MAP, MRR, and Recall 1000, and does significantly improve over the MTL BART model and all the STL models in both first stage retrieval and re-ranking, Comparing the MTL and
"We also conducted experiments with the DPH retrieval model, which yielded similar trends.

---

Question Rewriting
First Stage (BM25))
Re-ranker (mono'TS)
Models
ROUGE-1
BLEU
MAP
MRR
R 1000
NDCG
MAP
MRR
NDCG
Raw
62.82
36.01f
0,041011
0,0424t4
0,23351)
0.073311
0.078611
0,080914
0,1059:1
BART
72.91
48.15)
0,15171E
0.161713
0.557615
0.225711
0.243811
0.2580|2
0.3046
T5
73.22
45.86f
0.1720
0.1843
0.6055
0.2524g
0.2783
0.29571
0.3430
MIL
BART(dis+genn
73.12;
48.00F
0,15713
0.16851
0.57903
0.23488
0.2562;
0.2708;
0.31895
BART(gen)
73.56]
48.79
0,16463
0.1760
0.6006.
0.24400
0.2661
4
0.28231
0.3309:
74.12
45.86
0.2008
0.2150
0.6373
0.2829
0.3106
0.3302
0.3764
Manual
100.00
100.00
0.2486
0.2682
0.8012
0.3540
0,3811
0.4066
3295.0
Table 3: Comparison between the MTL models and the query rewriting baselines. f denotes a performance that is significantly worse than the MTI. BART model (paired I-test, 0.05); & denotes performance that is significantly worse than the MTL T3 model (paired t-test, p < 0.05).
STL models, we observe that both the MTL TS and BART models significantly outperform their corresponding STL models. Contrasting the perfor- mances of the discriminative-eeeraaiive with those of the generative MTL BART models, we find that there is little difference between the effectiveness of the two versions of the MLT BART model.
4.3 Analysis
In this section, we conduct a qualitative analysis to bolster support for our findings concerning the performance our MTL T5 model in comparison to the STL T5 model, as discussed in Section 4.2. The purpose of this qualitative analysis is to further validate our results and to shed additional lights on the advantages of the MTL approach in the follow- up question identification and conversational ques- tion rewriting tasks. First, we present exam- ple of dialogue that exemplifies the distinct advan- tages derived from the use of the MTL approach our model, specifically in conversational question rewriting, Next, we proceed to compare the differ- ences in NDCG scores between our proposed MTL model and the STL model for passage retrieval.
Comparison of our Proposed MTL Models with the Baselines for both tasks: Our MTL T5 model demonstrates its effectiveness in both tasks. It not only captures the recall of the rewritten questions at the unigram level but also enhances passage ranking, resulting in our MTL T5 model outperforming both the MTL BART model and all of the STL models, yielding statistically significant improvements both tasks. To illustrate these findings, we provide further qualitative analysis in Section 4.3.
Conversational Question Rewriting: To il- lustrate the advantages of the MTL approach in the follow-up question identification and conver- sational question rewriting tasks, we provide an example dialogue carefully selected based on the highest ROUGE-1 score achieved by our MTL T5 model. This example dialogue, shown in Figure 3, clearly showeases the benefits of employing the MTL strategy.
In answer to RQ2, we conclude that applying the Multi-Task Learning of question rewriting and classification to the T5 model improves the passage retrieval performance, yielding statistically signif- icant improvements over the MTL BART model and all the STL models.
This example consists of conversation history with two turns (Q1, A1, Q2, A2), the current ques- tion (Q3), the manually rewritten question (Q'3), the rewritten question of our MTL T5 model, and the rewritten question of the STL T5 model. The MTL T5 model successfully predicted the word "shift" indicating an invalid follow-up up question, Q3 deviates from the previous conversation's 's topic of album performance and instcad inquires about subsequent events. The MTL T5 model exhibits superior performance in predicting invalid follow- up questions, demonstrated by the model achieving the best Macro-Fl score on the Test-III LIF dataset (as described in Section 4.1). Notably, this test
Q1: How did the Disco Volante begin?
A1: Due to artwork delays the band members' many side-projects, It four years
before Disco Volante was released, October 1995.
Q2: How did they perform commercially?
A2: Not all critics were impressed with the
album, with The Washington Post describing "an album of cheesy synthesizers."
Q3: What followed this album?
Manual (Q'3): followed the Disco Volante album?
MTL TS: shift What followed the Disco Volante album?
STL T5: What followed the album of Cheesy Synthsynergics?
Figure 3: Comparison of question rewriting models.

---

Convorsation History:
fies candidate question valid follow-up to the previous question, thereby demonstrating its capa- bility to potentially aid in the correct resolution of the omitted entity (Odissi). On the other hand, Figure 4 (b), the candidate question "where did he go to school" would not have logically followed the previous question "did he have siblings". How- ever, the MTL T5 model predicted this candidate question valid follow-up question; hence this could lead the model incorrectly resolve the en- tity (Roy Acuff). As result, we can observe that the effectiveness of the follow-up question iden- tification task does influence the conversational question rewriting task performance.
QJ. at dl here du ii the se aeripr proceeded
publisht dally odl renoopapee
ansoquaee lacerr pais
founded in 2006
Q2: Was he ruccesstulatche paper?
A2 The tist acttion of the newrr ruspappr,, Israal Hayom, pucdiceet on kiy 2007
Cand question
Label
Was the swner the paper?
Valld
he changes the paper?
Inwalid
Det they put out and pooords butwoon 2005 7008/
Invelid
[a]
Hilssory:
Title: Shaldon Adetson
Description: teadiipess
W1:WWat does Sheldon have to do with pressh
hl groceeded with paralla plans publish dally nevsspaee tocor pete with tsrael, newspppprr had loundedd in 2006
q? Was he sucreecfulltt the peper?
A2: the first ecition of the rewspapet
publiined
2007.
Examples of question rewriting:
Question: the oweer Isrseli
paper?
Related Work
(6)
We discuss related work in Conversational QA. Conversational QA: Conversational QA, which involves the correct interpretation of questions within an ongoing dialogue, has been addressed by several datasets like QuAC (Choi et al., 2018), CoQA (Reddy et al., 2019), and TREC CAsT (Dalton et al., 2019, 2020, 2021; Owoicho et al., 2022). A common strategy concatenating previous questions with the current one for passage retrieval, yet this may be suboptimal for methods like BM25, which favour concise queries. Several studies (Aliannejadi et al., 2020; Mele et al., 2020; Ríssola et al., 2019; Sekulić et al., 2020) have offered alternatives to improve retrieval perfor- mance. Mele al. (2020) proposed detecting topic shifts through heuristics and dependency parsing, while Aliannejadi et al. (2020) and Ríssola et al. (2019) utilised BERT models to predict historical questions enhancing the current one's retrieval. Sekulić et al. (2020) furthered this by predicting related past questions and reformulating the current one using ALBERT. In contrast, our work focuses on using follow-up question identification to improve conversational question rewriting. Follow-up Question Identification: This task has seen diverse methods, ranging from rule-based solutions (Bertomeu et al., 2006; Kirschner and Bernardi, 2007) to statistical machine learning models like Logistic Regression (Kirschner and Bernardi, 2009). Kundu et al. (2020) proposed standout three-way attentive pooling network, iden- tfying follow-up questions related to the conversa- tion history and the associated answer passage, out- performing rule-based methods, logistic regression models, and neural setwork-based models such as
Figure 4: Examples of dialogue differences in terms
NDCG for queries in the OR- RR-QuAC query set. (a) higher NDCG for MTL T5 wrt STL. TS (b) higher NDCG for STL TS wrt MTL. T5.
set comprises sampled invalid follow-up questions from the same conversation, making this achieve- ment particularly noteworthy. This prediction helps the model to differentiate and choose the accurate entity "Disco Volante" instead of the misleading prediction "Cheesy Synthsynergics" made by the STL T5 model. This demonstrates the ability of the MTL T5 model better leverage and interpret the context of the conversation, leading to more accurate predictions and an improved performance.
Passage Retrieval: We also compare the dif- ferences in terms of NDCG scores when using a BM25 ranking model with our proposed MTL model in comparison to using it with the STL model in passage retrieval (MTL T5 vs. STL T5). Figure 4 shows two examples of dialogues selected using the difference in NDCG scores between MTL T5 and STL T5 on the OR-QuAC query set. Over- all, MTL T5 outperforms STL T5 for 781 questions, while the opposite was true for 497 questions. Fol- lowing Macdonald et al. (2021), we only consider differences larger than 0. 0.15 absolute NGCG when inspecting the effect of the MTL approach. Hence, this analysis demonstrates that proposed MTL T5 model exhibits superior performance over the STL T5 model in passage retrieval. To further illus- trate this point, we closely examine the predictions made by the MTL T5 model in Figure 4 (a). It is clear that the MTL T5 model successfully identi-

---

BiLSTM, CNN, and BERT (Devlin et al., 2019). In our study, we utilise both the three-way atten- tive pooling network and BERT as baselines, given their proven effectiveness.
flows, using a conversation-aware knowledge graph and novel relation-aware graph encoder. In contrast to these studies, our work focuses on addressing the ambiguity of the conversational question by reformulating the user's original question and classifying whether the conversation is still focused on the same topic to enhance the accuracy of the information retrieval system.
Conversational Question Rewriting: This task involves transforming conversational queries into context-independent queries suitable for information retrieval (IR) systems (Mele et al.. 2021). Approaches range from sequence-to- sequence models like LSTM and GRU (Ren et al., 2018), to rule and self-super learning methods fine-tuning GPT-2 (Yu et al., 2020), and Transformer++ trained on the CANARD dataset (Vakulenko et al., 2021a). Vakulenko et al. (2021b) compared various methods the CAsT 2019 and 2020 datasets, emphasising the effective- ness of Transformer++. Recently, Lin et al. (2020) presented T5 model for conversational question rewriting. T5 outperformed neural network-basee models such as LSTM, GPT-2, BERT, and UniLM on the CANARD and CAST 2019 datasets. In this paper, we propose T5 model for conversational question rewriting due to its overall effectiveness.
Conclusions
We proposed a method for Conversational QA. which learns predict the follow-up question and rewrites the conversational question simultane- ously. Our proposed MTL method makes use of text generation models including BART and T5 by generating the first token for a classification task and the follow-up tokens for a conversational question rewriting task. For the follow-up question identification task, our experiments on the LIF dataset showed that our proposed MTL BART model has the best effectiveness, yielding statis- tically significant improvements over the baselines. For the question rewriting task, our proposed MTL '5 model performed best both in terms first stage retrieval and re-ranking. Furthermore, we showed that the BART-based model can be employed for the follow-up question identification and the conversational question rewriting tasks in both dis- crininative+generativ and generative MTL setups.
MTL in Conversational QA: MTL has been applied in Conversational QA, primarily focussing on answer span prediction and auxiliary tasks (Kongyoung et al., 2020; Qu et al., 2019; Xu et al., 2019). Some studies have combined classification and text generation tasks to improve passage ranking and answer generation (Ide and Kawahara, 2021; Kongyoung et al., 2022; Lee al., 2022; Jiang al., 2022), often leveraging models such as T5 (Raffel al., 2020). Similarly, Jiang al. (2021) combined the classification of relation types and the generation of sentences to express these relation types. Moreover, Wang et al. (2020) aimed to improve the response selection in multi-party conversations using an auxiliary task, namely topic prediction task, which classifies whether follow-up question is relevant to the user. Our work also embraces the MTL paradigm, but unlike prior studies prioritising answer generation, response selection, and passage ranking, we focus on improving retrieval effectiveness by rewriting the user's question using classification task.
Limitations and Future Work
While our work has demonstrated promising results in leveraging multi-task learning for con- versational question rewriting, several limitations must be acknowledged. Firstly, our model does not currently incorporate user interaction or feedback. This means it may not always accurately capture user intent adjust evolving conversational dynamics. In addition, without user feedback, the model lacks a mechanism for continuous learning and improvement from real-world applications. Finally, the model does not currently ask clarifying questions (Owoicho et al., 2022; Aliannejadi et al., 2021) when faced with ambiguous queries. This could limit its effectiveness in certain complex or unclear conversational scenarios. In future work, we plan to address these limitations by integrating user interaction and feedback, and by developing a mechanism for the model to ask clarifying questions when necessary.
Other Approaches Addressing Ambiguity in Conversational Questions: Shao al. (2022) addressed the ambiguity of the conversational question by asking clarifying questions to the user. Similarly, Li et al. (2023) explored enhancing dialogue generation with conversational concept

---

References
Manuel Kirschner and Raffaella Bernardi. 2007. An empirical view on IQA follow-up questions. In Proc. SIGdial, pages 43-46.
Mohammad Aliannejadi, Manajit Chakraborty, Este- ban Andrés Ríssola, and Fabio Crestani. 2020. Har-
nessing evolution of multi-turn conversations for ef- fective answer retrieval. In Proc CHIR, pages 33-42.
Manuel Kirschner and Raffaella Bernardi. 2009. Ex- ploring topic continuation follow-up questions using machine learning. In Proc. NAACL, pages 13-18.
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue pora with clarifying questions. Proc. EMNLP, pages 4473-4484.
Sarawoot Kongyoung. Craig Macdonald, and Iadh Ou- nis. 2020. Multi-task learning using dynamic task weighting for conversational question answering. In Proc. SCAL, pages 17-26.
Núria Bertomeu, Hans Uszkoreit, Anette Frank, Hans- UIrich Krieger, and Brigitte Jörg. 2006. Contextual phenomena and thematic relations database qa dialogues: results from a wizard-of-oz experiment. In Proc. HLT-NAACL, pages 1-8.
Sarawoot Kongyoung, Craig Macdonald, and Iadh Ou- nis. 2022. monoQA: Multi-task learning of reranking and answer extraction for open-retriee open-retrieval conversa- tional question answering. In Proc. EMNLP, pages 7207-7218.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context. In Proc. EMNLP, page 2174-2184.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proc. EMNLP. pages 66-71.
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2019, TREC CAsT 2019: The conversational assistance track overview. Proc. TREC.
Souvik Kundu, Qian Lin, and Hwee Tou Ng. 2020. Learning to identify follow-up questions in conver- sational question answering. Proc. ACL, pages 959-968.
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020. CAsT 2020: The conversational assistance track overview. In Proc. TREC.
Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paran- jape. Christopher Manning, and Kyoung-Gi Woo, 2022. You only need one model for open-domain question answering. In Proc. EMNLP, pages 3047- 3060.
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. CAsT 2021: The conversational assistance track overview. In Proc. TREC.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proc. NAACL, pages 4171-4186.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. Proc. ACL, pages 7871-7880.
Cicero dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhi- heng Huang, and Bing Xiang. 2020. Beyond [CLS] through ranking by generation. In Proc. EMNLP, pages 1722-1727.
Siheng Li, Wangjie Jiang, Pengda Si, Cheng Yang, Qiu Yao, Jinchao Zhang, Jie Zhou, and Yujiu Yang. 2023. Enhancing dialogue generation with conversational concept flows. In Proc. EACL, pages 1514-1525.
Ahmed Elgohary, Denis Peskov, and Jordan Boyd- Graber. 2019. Can you unpack that? learning to rewrite questions-in-comeext. In Proc. EMNLP, pages 5918-5924.
Sheng-Chich Lin, theng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2020. Conversational question reformulation via sequence-tue sequence-to-ssqqeeee to-suuencc architectures and pretrained language models. arXiv preprint arXiv:2004.01
Tatsuya Ide and Daisuke Kawahara. 2021. Multi- task learning of generation and classification for emotion-awand dialogue response generation. In Proc. NAACL, pages 119-125.
Craig Macdonald and Nicola Tonellotto. 2020. Declar- alive experimentation in information retrieval using pyterrier. Proc. ICTIR, page 161-168.
Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not just classification: Recog- nizing implicit discourse relation on joint modeling of classification and generation. In Proc. EMNLP. pages 2418-2431.
Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2021. On single and multiple representations in dense passage retrieval. In Proc. IIR.
Zhengbao Jiang, Luyu Gao. Zhiruo Wang, Jun Araki, Haibo Ding, Jamie Callan, and Graham Neubig. 2022. Retrieval as attention: End-to-end learning of re- trieval and reading within a single transformer. In Proc. EMNLP, pages 2336-2349.
Ida Mele, Cristina Ioana Muntean, Franco Maria Nar- dini, Raffaele Perego, Nicola Tonellotto, and Ophir Frieder. 2020. Topic propagation in conversational search. In Proc. SIGIR, pages 2057-2060.

---

Ida Mele, Cristina Ioana Muntean, Franco Maria Nar- dini, Raffaele Perego, Nicola Tonellotto, and Ophir Frieder. 2021. Adaptive utterance rewriting for con- versational search. Information Processing & Man- agement, page 102682.
Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu. and Raviteja Anantha. 2021a. Question rewriting for conversational question answering. In Proc. WSDM, pages 355-363.
Svitlana Vakulenko, Nikos Voskarides, Zhucheng Tu, and Shayne Longpre. 2021b. A comparison of ques- tion rewriting methods for conversational passage retrieval. In Proc. E.IRR 2027, page 418-424.
Rodrigo Nogucira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with pretrained sequence-tor sequence model. In Proc. EMNLP, pages 708-718.
Rodrigo Nogucira, Zhiying Jiang, and Jimmy Lin. 2021. Investigating the limitations of the transform- ers with simple arithmetic tasks. arXiv preprint 1rXiv:2102.1 13019.
Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, and Maarten Rijke. 2020. Query reso- lution for conversational search with limited supervi- sion. In Proc. SIGIR, pages 921 921-930.
Weishi Wang, Steven C.H. Hoi, and Shafiq Joty. 2020. Response selection for multi-party conversations with dynamic topic tracking. In Proc. EMNLP, pages 6581 6591.
Paul Owoicho, Jeffrey Dalton, Mohammad Aliannejadi, Leif Azzopardi, and Johanne R. Trippas. 2022. CAsT 2022: Trec cast 2022 going beyond user ask and sys- tem retrieve with initiative and response generation. In Proc. TREC.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-an natural language processing, In Proc. EMNLP, pages 38-45.
Ronak Pradeep. Rodrigo Nogueira, and Jimmy Lin. 2021. The axpando-monoddus design pattern for text ranking with pretrained models. arXiv preprint arXiv:2 100.0007 05667.
Chen Qu, Liu Yang. Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit lyyer. 2020. Open-retrieval conver- sational question answering. In Proc. SIGIR, pages 539-548. 548.
Yichong Xu, Xiaodong Liu, Yelong Shen, Jingjing Liu, and Jianfeng Gao. 2019. Multi-task learning with sample we-ieinht for machine reading comprehen- sion. In Proc. NAACL, pages
Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W Bruce Croft, and Mohit lyyer. 2019. Attentive history selection for conversational ques- tion answering. In Proc. CIKM, pages 1391-1400.
Shi Yu, Jiahua Liu, Jinggin Yang, Chenyan Xiong, Paul Bennett, Jianfeng Gao, and Zhiyuan Liu. 2020. Few- shot generative conversational query rewriting. In Proc. SIGIR, page 1933-1936.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Proc. JMLR, pages 1-67.
A Appendix
Our code and data are publicly available at the following URL: https://github..
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. In Proc. ACL, pages 249-266.
terrierte.m/mtt__een
Gary Ren, Xiaochuan Ni, Manish Malik, and Qifa Ke. 2018. Conversational query understanding using se- quence to sequence modeling. Proc. WWW, pages 1715-1724.
A.1 A Diseriminative--eeeratte MTL Model
In addition to the implementation of the genera- tive MTL model, we also incorporate a discrimina- tive+generative MTL model to capture the relation- ship between the question qe/ qm and the contextual information in the conversation history, which in- cludes the historical question(s) g1, 42, ... and the historical answer(s) d1 a2; ag as illustrated in Figure 2(a). The input sequence of the discrim- inative+ MTL model the same the one shown in Equation (1). During the fine-tuning process, the LM head of the model refined generate the target tokens, with token sequence length denoted as n in Equation (2). In contrast to
Esteban A Ríssola, Manajit Chakraborty, Fabio Crestani, and Mohammad Aliannejadi. 2019. Predicting rel- evant conversation turns for improved retrieval in multi-turn conversational search. In Proc. TREC.
Sekulić, Mohammad Aliannejadi, Fabio Crestani. 2020. Extending the use of previous relevant utter- ances for response ranking in conversational search. Proc. TREC.
Taihua Shao, Fei Cai, Wanyu Chen, and Honghui Chen. 2022. Self-supervised clarification question genera- tion ambiguous multi-turn conversation. Inf. Sci., page 626-641.

---

the generative MTL model, this approach employs the tokens wi...., Wn encompass the entire out- put sequence for the target query. Simultaneously, the CLS head (see details in Section 2.3) of the model undergoes fine-tuning to predict either 1 or O, signifying whether the candidate question is valid follow-up the previous question or not.
ram These models are configured as follows:* the maximum sequence length is set to 512, the number of training epochs is set to 5, the batch size is set to 24, and we use Adam optimiser with learning rate of 0.00005.
'Initill experiments found this T5 model more effective than the original t5-basc across number of tasks. "Settings follow https://gite https:///gitbb... com/ amiiaalaa/mi httiiaillllltttings////
We compare our generative BART model to the discriminative-generatiee BART variant in Sec- tion 4.1 and Section 4.2, to conclude on their perfor- mance both the follow-up question identification and conversational question rewriting tasks.
A.3 Evaluation Metries
Since the follow-up question identification a bi- nary classification task, we evaluate performances using classical classification metrics, namely pre- cision, recall, F1 and Macro-Fi.. Indeed, follow- ing (Kundu et al., 2020), reporting Macro-Fl en- ables accuracy on topic shift detection to be mea- sured, while Fl focuses solely follow-up identi- fication as the positive class. We use McNemar's test measure statistically significant differences between the models' classification performances. For the evaluation of the conversational question rewriting performance, we adopt the ROUGE re- call calculated for unigrams (ROUGE- 1 recall) and BLEU metrics, following (Vakulenko et al., 2021 la; Lin et al., 2020; Elgohary et al., 2019). As for the passage retrieval evaluation, use Mean Average Precision (MAP), Mean Reciprocal Rank (MRR). Normalized Discounted Cumulative Gain (NDCG) and Recall 1000 as metrics. For each query, the top 1000 documents are considered. We use the paired t-test for testing significant differences. Passage Retrieval Pipeline: We use the PyTer- rier (Macdonald and Tonellotto, 2020) platform for indexing and retrieving passages. For pas- sage ranking, incorporate BM25 with the monoT5 (Pradeep al., 2021) re-ranker.
Model Training Details
Mathemarical sotting Souree code
Scction Appendix A.I
hetps:
torrierteam/mi _ee_____e_naasss
Computing infrastructure
NVIDIA RIX A6000 GPU
Training
4h
Inference time
500ms
Batch size
24
Number parameters
(MTL. TS) BART)
each model
metrics
Appendix
Hyper parameter Experiments
mumber train-
Appendix
ing and ovaluation runs
Bounds for each hyper- parameter
Numher epochs: 1-5
Hyperparameter
Appendix A.2
figurations for best- performing models
Number hyperparame- for scarch trials
The method of choosing
Highest question rewiting effectiveness (ROUGE- on validarion set
sepea-parameter values
Dataset
Dataser longuages
English Table
Number of examples
datascts
Explanation of any data were excluded, all pre processing sleps containing or link to downloudable version the data
Section 3.2
httpss/ /tiptt//gtthbb... con/ torrierteam/nt i _gan_elssa
Table Summary of Reproducibility Criteria.
A.2 Hyperparameter Settings
We implement the BERT, GPT-2, BART, and T5 models using the following PyTorrh models from HuggingFace (Wolf et al., 2020), namely bert-base, facebook/bar bbasee and
A.4 Reproducibility Criteria
Table summarises the reproducibility criteria questions for this paper.


Reviews:
[{"review_id": "1DUso2Q5VB", "reviewer": "Reviewer_PJ74", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Reasons to Reject 1: There are closely related works that need to be discussed. This paper focuses on the multi-topic scenario. But the topic is not the only dynamic state during a conversation, such as concepts and indents (e.g., [1][2]). There are works on tracing the dynamics of these states. A broader discussion is expected.*\n\nThank you for pointing out the work by Jiang et al., 2021 [1]. While it's true that they also use Multi-Task Learning involving both classification and text generation, their focus is on Implicit Discourse Relation Recognition (IDRR). Their model is trained to detect relation types and generate sentences that express these relations. In contrast, our work is focused on improving conversational search systems. We use models based on BART and T5 to rewrite conversational questions and identify follow-up questions simultaneously. These are substantially different applications and objectives.\n\nThe work of Shao et.al., 2021 [2] focused on addressing the ambiguity of the conversational question by *asking* clarifying questions to the user. In contrast, our work is focused on addressing the ambiguity of the conversational question by reformulating the user's original question to enhance the accuracy of the retrieval information system. Hence, this addresses a different user task.\n\nWe are happy to cite [1] & [2]. in our final version of the paper, but their omission from this stage do not detract from the merits of our paper. \n\n\n\n*Reasons to Reject 2: The experimental results section is a bit cluttered and difficult to read. Empty readme in anonymous repository.*\n\nWe appreciate the feedback on the organisation of the experimental results section. We plan to improve the readability and clarity of this section in the revised version using the extra page that is customarily made available. To make this feedback more actionable, clarifying your review with specific problems would be appreciated.\n\nAs for the code and data repository, we apologise for the delay. We have now improved our anonymous github repo to facilitate a more thorough understanding and replication of our proposed method. In particular, the training codes and training instructions have been provided.\n\n\n\n*Reasons to Reject 3: Shortcomings of ROUGE-1 and BLEU*\n\nWe appreciate your concern about the shortcomings of ROUGE-1 and BLEU. In our paper, we chose to use ROUGE-1 and BLEU because they are the most widely used metrics for question rewriting evaluation. They have been shown to be effective in many previous studies, and they are relatively easy to compute.\nWe think the suitability of more advanced measures (such as BERTScore and BLEURT) would require a separate study focussed on evaluation. In contrast, the scope of this paper is upon models rather than evaluation methods. In addition, we measure the downstream retrieval effectiveness of the reformulated queries (e.g. nDCG, MAP), which verifies the conclusions we observe from ROUGE-1 (Vakulenko et al., 2021a).\n\n\n\n*Questions For The Authors:In section 4 (Experimental Results) you refer to Test-I, II, III without explaining what is the difference between these tests.*\n\nIn the revised version, we will provide further clarification on the differences between the Test-I, II, and III sets in the LIF dataset. In all three, the valid follow-up questions (label = 1) are constructed from the “should ask follow-up question” instances in the QuAC dataset. On the other hand for Test-II, questions with high similarity to the current passage are sampled from other conversations; for Test-III, the invalid follow-up questions are sampled from non-follow-up questions from the same conversation in QuAC; Test-I combines invalid instances from  Test-II & Test-III. The training and the development sets follow the same procedure as Test-I.In particular, the training and development sets contain 126k and 5.8k instances each; Test-I, II & III all have 1940 positive instances, and respectively 745, 3307 and 4052 negative instances.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The topic of question rewriting/identification is well-introduced, although it would be great to include additional pointers to the similar task of clarification question identification and generation. There are some recent works on this topic (Shao et al., 2022) and even a shared task with the special track for CR identification organized at NeurIPS last year (Kiseleva et al., 2022).\n\n2. The authors perform a good comparison of the models using significance testing. The results look robust and show an improvement over the selected baselines. The computational budget and training details are reported.", "reasons_to_reject": "1. The idea of joint text generation and classification with T5 is quite straightforward and lacks novelty (see e.g., (Jiang et al., 2021) for similar experiments with implicit discourse relation classification/generation). Besides, all experiments are only for English.\n\n2. The experimental results section is a bit cluttered and difficult to read. Also, although the authors mention that their code and data are publicly available there are only empty README files in the corresponding repository in anonymous.4open.science (solved).\n\n3. The authors use ROUGE-1 and BLEU for the question rewriting evaluation. However, the shortcomings of these metrics are quite known (see e.g., the following ACL publications:(Post, 2018), (Sulem et al., 2018), (Mathur et al., 2020) and there exist some alternatives like BERTScore or BLEURT. I think they should be also included to make the study more robust/comprehensive.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper uses BART and T5 to learn simultaneously a generation task (question rewriting) and a classification tasks (whether a question is a follow-up to the previous one). The presented approach shows good performance compared to the single-task learning baselines on the LIF dataset (Kundu et al., 2020) and OR-QuAC.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In section 4 (Experimental Results) you refer to Test-I, II, III without explaining what is the difference between these tests."}, "extra_text": {"Missing_References": "Jiang, Feng, Yaxin Fan, Xiaomin Chu, Peifeng Li and Qiaoming Zhu. “Not Just Classification: Recognizing Implicit Discourse Relation on Joint Modeling of Classification and Generation.” Conference on Empirical Methods in Natural Language Processing (2021).\nShao, Taihua, Fei Cai, Wanyu Chen and Honghui Chen. “Self-supervised clarification question generation for ambiguous multi-turn conversation.” Inf. Sci. 587 (2021): 626-641."}}, {"review_id": "uiN2OfBxbN", "reviewer": "Reviewer_yEsQ", "summary": "", "questions": "Question A: I am curious about a more general setting: identify related turns in the conversation because I think a topic could be recalled. For example, for a conversation (q1, a1)(q2, a2)(q3, a3),   (q1, a1)(q3, a3) share the same topic while (q2, a2) is about another topic. Then would it be beneficial to find the recalled topic instead of just classifying whether it is a follow-up question.\n\nQuestion B: Some words say (e.g., Sec 3.2) LIF and CANARD are used for evaluation, while some others say (e.g., Abstract, Introduction, and Sec 4.2) LIF and OR-QuAC are used. Which one is correct?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Reasons to Reject 1: There are closely related works that need to be discussed.*\n\nThank you for the constructive feedback. We acknowledge the importance of a broader discussion of related works that focus on other dynamic states during conversations, such as concepts and intents. In our revised manuscript, we will consider expanding our discussion to cover these works and clarify how our paper complements or differs from them.\nFor example, while our paper uses Multi-Task Learning to focus on follow-up question identification and conversational question rewriting, Wang et al., 2020 aims to improve response selection in multi-party conversations by considering multiple topics. Similarly, Li et al., 2023, explores enhancing dialogue generation with conversational concept flows, using a conversation-aware knowledge graph and a novel relation-aware graph encoder.\n\n\nWhile these works provide valuable insights into other aspects of conversational dynamics, they do not serve as direct baselines for our study. This is because our focus is on follow-up question identification and conversational question rewriting, objectives that are distinct from the aims of the aforementioned papers. Consequently, while a broader discussion of these related works will provide valuable context (and we can add these as wider context in the next version of the paper, using the customary extra page), their absence in our study does not represent missing baselines, and hence do not detract from the validity or integrity of our results.\n\n\n\n\n*Reasons to Reject 2: The authors do not follow the writing instructions from EMNLP. The captions of the table are misplaced.*\n\n\nThanks for highlighting this. We'll make this 2-second fix in the next version of paper, but, with all due respect, this is not a reason for rejection.\n\n\n*Questions For The Authors 1:* \n\n*Question A: I am curious about a more general setting: identify related turns in the conversation because I think a topic could be recalled. For example, for a conversation (q1, a1)(q2, a2)(q3, a3), (q1, a1)(q3, a3) share the same topic while (q2, a2) is about another topic. Then would it be beneficial to find the recalled topic instead of just classifying whether it is a follow-up question.*\n\n\nThank you for raising this interesting point. Identifying related turns in a conversation based on recurring topics could certainly enhance the system's ability to generate more relevant and accurate responses. While our current work focuses on follow-up question identification and question rewriting, incorporating topic recall as an additional dynamic feature in the model could be beneficial. This approach could further refine the system's understanding of the conversation and thereby improve its responses. We appreciate the suggestion and will consider exploring this in future work.\n\n*Question B: Some words say (e.g., Sec 3.2) LIF and CANARD are used for evaluation, while some others say (e.g., Abstract, Introduction, and Sec 4.2) LIF and OR-QuAC are used. Which one is correct?*\n\nThank you for catching that inconsistency. To clarify, we experiment using the LIF dataset and the CANARD dataset, which are a recent adaptation of the well-known QuAC Conversational QA dataset. For the training and development sets, we integrate LIF and CANARD by picking only the candidate questions from the LIF dataset that exist in the CANARD dataset. We used the OR-QuAC dataset for evaluation, which is an aggregation of the QuAC, CANARD, and Wikipedia datasets. This allows us to assess our model's performance across conversational question answering and passage retrieval tasks, as detailed in Appendix A.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task discussed is well-defined and well-motivated.\n\n2. The proposed approach is simple and proven to be effective, according to comprehensive experiments.", "reasons_to_reject": "1. There are closely related works that need to be discussed. This paper focuses on the multi-topic scenario. But the topic is not the only dynamic state during a conversation, such as concepts and indents (e.g., [1][2]). There are works on tracing the dynamics of these states. A broader discussion is expected.\n\n2. ~~The authors do not follow the writing instructions from EMNLP. The captions of the table are misplaced.~~\n\n[1] Siheng Li, Wangjie Jiang, Pengda Si, Cheng Yang, Qiu Yao, Jinchao Zhang, Jie Zhou, and Yujiu Yang. 2023. Enhancing Dialogue Generation with Conversational Concept Flows.\n[2] Weishi Wang, Steven C.H. Hoi, and Shafiq Joty. 2020. Response Selection for Multi-Party Conversations with Dynamic Topic Tracking.", "questions_to_authors": "Question A: I am curious about a more general setting: identify related turns in the conversation because I think a topic could be recalled. For example, for a conversation (q1, a1)(q2, a2)(q3, a3),   (q1, a1)(q3, a3) share the same topic while (q2, a2) is about another topic. Then would it be beneficial to find the recalled topic instead of just classifying whether it is a follow-up question.\n\nQuestion B: Some words say (e.g., Sec 3.2) LIF and CANARD are used for evaluation, while some others say (e.g., Abstract, Introduction, and Sec 4.2) LIF and OR-QuAC are used. Which one is correct?", "paper_topic_and_main_contributions": "This study explores a multi-task learning approach for multi-topic question rewriting, which includes an additional task to identify follow-up questions. According to the authors, this is the first proposal of such a combination. A seq2seq model is used to predict the rewriting and the classification, and two alternative implementations are discussed. The authors conduct experiments on two datasets, one for each task, and report superior performance against recent baselines.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nzi7vcCfkm", "reviewer": "Reviewer_Y11v", "summary": "", "questions": "* Did you try experimenting w/ bigger models and try different retrieval and reranking methods to see how that affects the magnitude of the potential improvements from your MTL approach?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*While the results show stat-sig improvements, I still intuitively feel the improvement in results across a wider experiment setting (e.g. different retrieval and reranking methods, more QR models, bigger models etc) might not be significant.*\n\nWe appreciate the reviewer's suggestion to consider a broader experiment setting. However, it is worth mentioning that the models we used for evaluation are state-of-the-art and have already shown generalisability across various tasks and datasets. In the paper, we employed the OR-QuAC dataset, using a single but highly competitive retrieval and reranking method (BM25 + monoT5). \nDue to space constraints, we were unable to report additional results demonstrating the generalisability of our approach. In particular, in addition to the BM25 + monoT5 (extracted from Table 2 in the paper), the table below reports performances using a BERT reranker (Nogueira et al., 2020c) over the DPH weighting model. The results here illustrate that our approach generalises over the choice of retrieval pipeline.\n\n| Retriver+Reranker | Models |   MAP   |   MRR  |  NDCG  |\n|-------------------|:------:|:-------:|:------:|:------:|\n| BM25+monoT5       | STL T5 | 0.2783  | 0.2957 | 0.3430 |\n|                   | MTL T5 | 0.3106  | 0.3302 | 0.3764 |\n| DPH+BERT          | STL T5 | 0.2771  | 0.2949 | 0.3414 |\n|                   | MTL T5 | 0.3081  | 0.3286 | 0.3735 |\n\n\nThat said, we understand the value in diversifying the experiment setting, including different retrieval and reranking methods as well as other datasets. We believe that such extensions, while important, do not diminish the current contributions of our paper. We will consider this in our future work to provide a more comprehensive evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* It's an interesting find that combining follow up identification w/ question rewriting is improving the passage retrieval.", "reasons_to_reject": "* While the results show stat-sig improvements, I still intuitively feel the improvement in results across a wider experiment setting (e.g. different retrieval and reranking methods, more QR models, bigger models etc) might not be significant.", "questions_to_authors": "* Did you try experimenting w/ bigger models and try different retrieval and reranking methods to see how that affects the magnitude of the potential improvements from your MTL approach?", "paper_topic_and_main_contributions": "Authors propose a Multi-task Learning (MTL) for the problem of conversational QA where the model learns both to identify a context-dependent follow up question as well to rewrite one into a self-contained form by resolving references. Authors perform experiments using BART and T5 models, on a OR-QuAC test set authors show improvement over single-task learning (STL) for passage retrieval. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The following reference/s are missing from Conversational QA related work.\n* Raviteja Anantha,Svitlana Vakulenko,Zhucheng Tu,Shayne Longpre,Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 19th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’21). 520–534. https://aclanthology.org/2021.naacl-main.44."}}]

Author rebuttal:

Meta-review: This work considers the conversational search setting. Primarily, the work proposes an MTL approach which simultaneous considers topic continuity and question reformulation (instead of independently).

**Pros**: Reviewers tend to agree the task and motivations/hypotheses are well-defined with generally convincing experiments (among those conducted). Some reviewers also comment on interesting takeaways from the work.

**Cons**: After rebuttal, there is a shared sentiment among a few reviewers that the scope of the experiments is limited (1 language, missing perturbations among important variables like larger models and retrieval system, etc.). Authors do provide some new experiments, but they do not address all concerns. For one reviewer, this issue is compounded by substantial similarity to another MTL paper looking at classification/generation of implicit discourse relations. While this work is arguably very similar, and other MTL approaches abound in the current literature, the authors suggest the importance of the conversational setting in the current work. Besides these concerns on soundness and scope, reviewers are generally ambivalent - there is no champion.
Decision: Accept-Findings