Augmenting Ad-Hoc IR Dataset for Interactive Conversational Search
Authors: Pierre ERBACHER
Jian-Yun Nie
Philippe Preux
Laure Soulier
Venue: Accepted by TMLR ()

Abstract: A peculiarity of conversational search systems is that they involve mixed-initiatives such as system-generated query clarifying questions. Evaluating those systems at a large scale on the end task of IR is very challenging, requiring adequate datasets containing such interactions. However, current datasets only focus on either traditional ad-hoc IR tasks or query clarification tasks, the latter being usually seen as a reformulation task from the initial query.
Only few datasets are known to contain both document relevance judgments and the associated clarification interactions such as Qulac and ClariQ. Both are based on  the TREC Web Track 2009-12 collection but cover a very limited number of topics 
(237 topics), far from being enough for training and testing conversational IR models. 
To fill the gap, we propose a methodology to automatically build large-scale conversational IR datasets from ad-hoc IR datasets in order to facilitate explorations on conversational IR. 
Our methodology is based on two processes: 1) generating query clarification interactions through query clarification and answer generators, and 2) augmenting ad-hoc IR datasets with simulated interactions.
In this paper, we focus on MsMarco and augment it with query clarification and answer simulations. We perform a thorough evaluation showing the quality and the relevance of the generated interactions for each initial query. This paper shows the feasibility and utility of augmenting ad-hoc IR datasets for conversational IR.

Published in Transactions on Machine Learning Research (05/2024)
Augmenting Ad-Hoc IR Dataset for Interactive Conversa- tional Search
Pierre Erbacher
piere.ertacherriissr...... upme.f
Sorbonne Université
Jian-Yun Nic
nic umontreal.cr
Université de Montréal
Philippe Preux Inria, Université de Lille
phillppeeprreurdeee prw.r@inria..a
Laure Soulier Sorbonne Université
loure.soulier Wisir upmn..f
Reviewed OpenReview: https: https://openrevvaau
openreview. net/ forum? id= XXXX
Abstract
A peculiarity of conversational search systems is that they involve mixed-initiatives such as system-generated query clarifying questions. Exaluating those systemss at a large scale on the end task of IR is very challenging, requiring adequate datasets containing such interactions. However, current datasets only focus on either traditional ad-hoe IR tasks or query clarification tasks, the latter being usually seen a reformulation task from the initial query. Only a few datasets are known to contain both document relevance judgments and the associated clarification interactions such as Qulac and ClariQ. Both based the TREC Web Track 2009-12 collection, but cover very limited number of topics (237 topics), far from being enough for training and testing conversational IR models. To fill the gap, we propose a methodology to automatically build large-scale conversational IR datasets from ad-hoc dalasets in order to facilitate explorations on conversational IR. Our hodolggy based on two processes: 1) generating query clarification interactions through query clarification and answer generators, and 2) augmenting ad-hoc IR datasets with simulated interactions. In this paper, focus MsMarco and augment it with query clarification and answer simulations. We perform thorough evaluation showing the quality and the relevance of the generated interactions for each initial query. This paper shows the feasibility and utility of augmenting ad-hoc IR datasets for conversational IR.
1 Introduction
Conversational systems, including personal assistant systems and chatbots, becoming increasingly pop- ular for wide variety of tasks, including online information seeking. While more recent Large Language Models (LLM) like OpenAT's ChatGPT (Ouyang et al.) 2022) have demonstrated their ability to answer factual questions, they cannot. be considered as conversational search systems because they only generate the most likely answer text without referring explicitly to the sources and without guarantee of veracity, amplifying potential bias and false truth (called Stochastic Parrots) (Bender al. 2021). To overcome this limitation, conversational search systems must rely on information retrieval capabilities locate relevant purces//oocumetts (Shah & Bender 2022
Dalton et al. 2022 Zamani et. al. 2022 Anand et al. 2020:

---

Published in Transactions on Machine Learning Research (05/2024)
Bender et al. 2021 However, this does not mean that one can merely rely models such as LaMDA (Thoppilan et al.) 2022, WebGPT (Glaese et al.) 2022) or Sparrow (Shuster et al. 2022 to generate an answer conditioned to the information retrieved by an independent tool, because top retrieved documents may not contain relevant information, leading to untruthful or uninformative answers (Nakano et al. 2021
This underlines the importance of including retrieval capabilities in the evaluation of conversational search models as a whole, as suggested in (Dalton et al. 2020a)
Beyond providing natural language responses, key ability of conversational search systems is their (pro)active participation in the conversation with users to help clarify or refine their information need
(haah& & Bender 2022 Chu-Carroll & Brown 1997 Dalton et al.
2022
Zamani et al. 2022: Anand et al.
2020 Bender et al., 2021 Radlinski & Craswell 2017 "Trippas et al, 2020 Aliannejadi et al. 2019 Keyvan
& Huang 2022 Zamani et al. 2020b). While recent advances in information retrieval based on LLM have
significantly improved the performance of information retrieval (TR) models by reducing vocabulary mis-
matches between queries and documents (Formal ct al.) 2021 Reimers & Gurevych 2019a; Hofstätter et al., 2021), accurately understanding the user's intent remains a challenge, in particular when the information need is complex,
when the resulting query is ambiguous (Culpepper et al. 2018). As the user cannot browse through the list of documents in conversational search, the conversational system must actively participate in the conversation and ask clarifying questions to help the user to clarify or refine their
needs (Shah & Bender 2022
Dalton et al.
2022:
Zamani et al. 2022 Anand et al. 2020 Bender et al. 2021
Radlinski & Craswell 2017
Trippas et al., 2020 Aliannejadi et al. 2019 Keyvan & Huang 2022 Zamani
et al.] 2020b). This makes it particularly challenging to evaluate conversational search systems properly because the mixed initiatives of the user and the system can lead to many different directions.
Attempts have been made to evaluate conversational IR (Sekulić et al. 2021 Aliannejadi et al. 2019 Salle
ct al. 2021 Bi et al. 2021), but they mainly focus on evaluating the quality of the generation of clarifying questions using aligned datasets, such as Qulac (Aliannejadi et al.) 2019) and ClariQ (Aliannejadi et al. 2021) that contain pairs of query and clarifying qucstion. Other models Hai et al. 2023a) address the retrieval task in the context of conversation as in TREC CAST (Dalton el al. 2020b). Owoicho et al proposed to use GPT4 to simulate user feedback on topics from the TREC CAsT datasets (Owoicho et al. 2023
While the emerging trend highlights the need of designing ranking models laking into account mixed-initiative interactions, it also demonstrates the necessity to build large-scale IR datasets containing initiaiii interactions. The two datasets mentioned above, Qulac and ClariQ, have been built on the basis of queries of the TREC Web Track 2009-12 collection. However, they only contain at most 237 topics, limiting our ability to train and to test neural conversational retrieval models properly.
The critical need for large conversational IR datasets motivates us to carry out this study, which aims at IR. datasets, containing not only user's queries, but also user-system
creating large-scale interactions. Collecting such conversational data is challenging because of not only the high annotation cost, but also the inherent difficulties to ask clarifying questions: the ambiguity arising from user's query depends on the context, the collection, and also the user's patience, expertise, cooperation, or engagement. (Wang & 2021 Salle et al.) 2021 Zamani et al. 2020a)
In our work, we aim to leverage simulation techniques to automatically generate mixed-initiative interac- tions between user and system and propose a methodology to augment ad-hoc IR dalasets with such interactions. To do so, we design query clarification generator leveraging the ClariQ dataset as well as user simulation for user's response. We use them to generate mixed-initiative interactions on the MsMarco ad-hoc IR dataset. The resulting dataset is called MIMarco for Mixed Initiave MsMarco. Our contribution is threefold:
We propose a methodology to augment ad-hoc IR datasets to integrate mixed-initiative interactions;
We evaluate our dataset-building methodology, and particularly the quality of interac- tions generated for the ad-hoc IR dataset. MsMarco;
2

---

Published in Transactions on Machine Learning Research (05/2024)
We demonstrate the utility of simulated interactions for document ranking on augmented MsMarco. This result, can also be seen as an evaluation proxy of the usefulness and the relevance of the simulated mixed-initiative interactions within the ad-hoc IR dataset.
2
Related Work
2.1 Evaluating Conversational Search
Designing an adapted framework for evaluating conversational search systems is still challenging in the IR community. Indeed, conversational scarch involves both dialogue and IR tasks that should lead to mixed-initiative interactions to support and guide the user during his/ search (Dalton et al. 2022) A conversational search system should be therefore able to 1) generate questions to clarify/elion /elicit users' information needs, and 2) retrieve documents providing relevant information. There are two main strategies to train and evaluate these systems by either 1) leveraging existing datasets, often at the cost of not having all the dimensions of conversations, or 2) simulating interactions between the user and the system. We briefly review some typical attempts for different types of conversational systems.
On question answering (QA), the datasets have been extended from one-shot QA such as Squad (Rajpurkar et al. 2018), Quac (Choi et al.) 2018). ELI5 (Fan et al., 2019), or OpenQA (Chen et al. 2017 to con- versational Q&A such as coQA (Reddy et al., 2019). One can train/ answer generation systems, and possibly information retrieval systems using the collection of passages. Despite this interesting move, the datasets are insufficient for IR. because they usually focus on factual questions instead of complex or exploratory questions that characterize information ncods. The TREC CAsT dataset (Dalton et al. 2020b) extends the scope of questions and addresses different information facets within the conversation (a facet can be seen as a specific sub-category of the topic). However, the interactions provided are often limited to answer generation without proactive interactions engaging the system in a real support of search. Other datasets, such as CANARD (Elgohary et al. 2019). focus on query refinement or reformulation, without proactive interactions. Therefore, most. approaches focused on generating reformulated queries as input to ranking systems (Hai et al., 2023b). Last year, the TREC CAsT track (Dalton et al., 2020b) introduced mixed-initiative questions related to the proposed IR-oriented conversations, without providing the associ- ated users' responses. This dataset constitutes a first step toward exploring search, but does not dispose of a complete and coherent conversation.
conversational
Several initiatives have been taken to build query clarification datasets (Rahmani et al. 2023 For example, the MIMICS dataset (Zamani et al.) 2020a) contains large-scale open domain clarifying questions collected from real users on the Bing search engine. The query clarifications are associated with initial users' queries and other information such as clicks. However, this dataset does not provide document relevanco judgments or conversational interactions between the user and the system. To the best of our knowledge, only Qulac and ClariQ datasets contain both document relevance judgments and the associated mixed-initiative conver- sations. These datasets are built from the TREC Web Track 2009-12 collection, which provides annotated topics and facet pairs, associated with relevant documents. Users' responses have been collected through crowd-sourcing platforms, allowing to build a complete dataset of mixed-initiative interactions grounded in an ad-hoe IR dataset. However, collecting these interactions has been costly and the datasets remain small with only 237 topics and 762 topic facets. This is too limited for training and evaluating conversational retrieval systems.
Facing the lack of adequate datasets, a growing idea in the community is to rely on user simulation evaluate conversational search systems (Erbacher et al. 2022 Salle et al. 2021). User simulations by mimicking users' queries and feedback are cost-efficiet and allow for the evaluation of various strategies without direct data annotations. For example, Salle et al (aallc et al., 2021) evaluate their query clarification systems with a user sinulation aiming al generaling answers. Their user simulation relies on a BERT model fine-tuned to classify "Yes"/ /'No' answers to the clarifying questions. With a controllable parameter, the user simulation can also add words from the intent in the answer to simulate more or less cooperative feodback. Sckulić et al (Sekulié et al.] (022)) confirmed with an additional human judgment that user simulations can generate answers and give feedback with fluent and useful utterances. User simulation is also exploited to design
3

---

Published in Transactions on Machine Learning Research (05/2024)
evaluation frameworks for conversational recommender systems Kang et al.
2019 Gao et al. 2022 Wu
resulting in large synthetic dia logue interactions from ad- ad-hoc recommendation datasets (Kang et al. 2019 Gao et al., 2022 Wu et al. 2020 Zhou et al.) 2020 Fu et al, 2020 However, in the rocommendation context, we notice that conversations are generated under explicit search constraints over annolated features like price range, color, localion, movie genre, or brand, whatever the gencration approaches used either by using large language models (Asrí et al. 2016) or by following agenda (Schatzmann et al., 2007 Peng et al. 2018 Li al.) 2017; Kreyssig et al. 2018 Unfortunately, similar approaches cannot be used for complex and exploratory search tasks (Belkin & Croft, 1992). In open-domain IR, the facets underlying information needs are not necessarily discrete or easily identifiable, making it much harder to identify and annotate users' needs.
et al. 2020 Zhou et al., 2020: Fu et al.
2020),
2.2 Asking Clarifying Question
Asking clarifying questions is a conversational task that allows the user to be involved in the query disam- biguation process by interacting with the system. For open-domain conversational IR, first line of works to identify a clarifying question relies on ranking strategies applied to pool of predefined, human-generated candidates. In the pioneering work, Aliannejadi et al. (Aliannejadi el al.] 2019) propose ranker that iteratively selects a clarifying question at each conversation turn. Bi et al. (Bi et al. 2021) complete this approach with an intent detection based on negative feodback and a Maximal Marginal Relevanco-basod BERT. Hashemi et al. (Hashemi et al.) 2020) use a transformer architecture to retrieve useful clarifying questions using the query and the retrieved documents. However, leveraging a fixed question pool will limit the coverage of topies, and therefore hinder the effectiveness of the approach. To overcome this limitation, second line of works rather aims at generating clarifying questions. In (Salle et al.) 2021). Salle et al. use templates and facels collected from the Autosuggest Bing API to generate clarilying questions. Al each lurn in the conversation, they select a new facet to generate the question until the user's 's answer is positive. This inferred facet is then used to expand the initial query. Sekulić et al (Sekulić et al.) (021) propose to further improve the fluency by using a LLM to condition the clarifying questions generation on the initial query and a facet. For instance, the query 'Tell me about kiwi', conditioned to facets 'information fruit' or "biology birds' can generate questions like "Are you interested in kiwi fruit?' or 'Are you interested in the biology of kiwi birds?'. They rely on Clariq dataset to fine-tune GPT2, and found that generated questions are more natural and useful than template-based methods. They have extended this work by generating questions using facets extracted from retrieved documents (Sckulić et al.) 2022). Zamani et al. (Zamani et al. 2020a) propose to generate clarifying questions associated with multiple facets (clarifying panels) which are collected using query reformulation data. They investigate three methods to generate clarifying questions: templates, weak supervision, and maximum likelihood, and use reinforcement learning to maximize the diverse set of clarifying panel candidates.
(woicho et al. 2023) The literature review highlights that there is lack of adequate large-scale datasets containing mixed- inxettinitiative interactions for the IR task. Having in mind that collecting those datasets with human annotations would be costly, we believe that a possible alternative is to generate mixed-initiative interactions automatically from existing collections. We propose therefore a methodology to first generate clarifying questions and the associated answers, and second augment ad-hoc IR datasets. As already dis- cussed earlier, the definition of facets in information retrieval is not always obvious but seems critical for generating relevant clarifying questions. We will put a particular attention on this aspect in our methodology.
3 Simulated interactions
3.1 Problem definition
We introduce our methodology to generate automatically large-scale mikxed-inittativv-drive IR datasets. To do so, we propose to augment ad-hoc IR datasets with simulated user-system interactions, namely clarifying questions (for the system side) and the corresponding answers (for the user side). To provide a dataset useful for training mixed-inittaativo-rrintt neural ranking models and capturing similarity signals in the matching loss, it is important to provide a wide range of interactions, namely clarifying questions that give rise to either positive or negative answers. Having in mind that a topic might be complex ambiguous,
4

---

Published in Transactions on Machine Learning Research (05/2024)
we follow previous works (Sekulić et al. 2021 Zamani et al. 2020a Salle et al. 2021 leveraging facets to generate those clarifying questions. Extracting positive or negative facets around topic can be seen a proxy to constrain the generation of clarifying questions expecting 'yes' and 'no' answers. Moreover, to ensure the overall quality of the mixod-initiative interactions, we propose to introduce another constraint variable modeling the user's search intent. The pair of facet and intent variables allows lo generate positive and negative clarifying questions (thanks to the facet) by always keeping the answer generation coherent with the relevance judgments in the initial dataset (thanks to the intent). Said otherwise, sampling different facet-intent pairs from passages with known relevance judgment allows constituting dataset with positive and negative mixed-initiative interactions that reflect the search intent of the user. For the sake of simplicity, we only consider single-turn interactions, and discuss the extension to multi-turn interactions in Section D
Let us consider an ad-hoc IR dataset D = QP.Q. RJ, in which P is a collection of passages (or documents).
Q is a set of queries, and R a set of relevance judgments. R includes tuples (g. P. .P.) indicating relevant Pt+ C P and irrelevant passages Pj C P. for a query q Q. We assume P. n P. =. Our objective is to augment this dataset D with a mixed-initiative interaction set X = = {X1,..., ......,,................... X,). We note a miixed-initiation question, and a the associated answer. With this in mind, we design a datasot- methodology M + DU X....., } relying on two main steps: 1) extracting the (positive and negative) facets I related to each topic (if not available in the initial ad-hoc IR dataset) which is then used to constrain the clarifying question generation, and 2) generating mixed-initiative interactions given a query q and this facet f. Depending on the dataset, the sets of positive F+ and negative F facets associated with query q might be available need to be built. (Section 3.2 We also assume that the search intent int of the user is characterized by the relevant documents available in the initial datasct. Then we propose generate mixed-initiative interaction X; given query q and the constraint variables I and int. We rely on 1) a clarifying model CMM : f.f f oq aiming at generating a clarifying question eq around facet f given the query topic g. and 2) a user simulation US (eg, int, f) + a which infer the answer a in response to the clarifying question og given the relevance of the facet f and the user intent int.
interaction X, (q,cq, a) where q refers to initial query, eq clarifying
Below, we introduce the method to extract facets, as well as the components enabling to generate clarifying questions and answers. Then, we present the overall methodology to augment ad-hoc IR datasets with mixed-initiative interactions. Examples of expected
interactions are presented in Table [
3.2 Extracting Facet
Facets might be explicit or implicit depending on the dataset. For example, they are specified in TREC Web 2009-12 (Clarke et al. 2009), and accordingly, Qulac and ClariQ [Over] 200T)). If not explicitly specified, we propose to extract them from documents. Previous works have shown that query facets can be extracted from top-retrieved documents (Dou et al. 2016; Kong & Allan 2013). Inspired by the analysis provided by Sekulić et al. (Sekulié et al. 2022), extract top contextual keywords to represent facets, as suggested in (Sharma& & Li. 2019). The goal of the facet extraction is to provide additional keywords that can be used to later generate a clarifying question about various topics or subtopics. In this work, facets are a set of keywords providing additional context lo the query. We formulate it as a bijective function ((PP) : F that maps a set P of passages to a set of facets. Given a query q, we construct the sets F+ and F of positive and negative facets from respoctively relevant and irrelevant passage scts,, resp. P and P-- This allows us to keep the relevance of facets. To do so, for a passage p E (Ptt UPT), we extract as a facel ʃ E F the set of K words in the passage that are the most similar to the passage embedding (i.e., the embedding of the (CLS) token). To compute the similarity, we use a pre-trained Sentence-Ber (i.e., MiniLM-L6-x2 model) (Reimers & Gurevych 2019b) between each token embedding and the passage one.
3.3
Generating
Interactions
3.3.1
Generating clarifying questions
The goal of the clarifying model CM to ask relevant clarifying questions relating to an ambiguity in the meaning or the object of the query. In most of the proposed models (Zamani et al.] 2020a Sekulić et al.
5

---

Published in Transactions on Machine Learning Research (05/2024)
Simulated mixed-initiative interactions in MiMarco
Query: why did the volunteriaay enter ww1
System: do you want to know the effect of unrestricted submarine warfare on the us ?
Answer: yes
Intent: The effects of unrestricted submarine (U-Boat) warfare, economic ties to the Allies, the public
outrage to the sinking of the Lusitania, the Sussex torpedoing, and the Zimmerman Telegram caused this change, and the United Stales entered an Allied Alliance and a World War for the first time its history.
Facet: effects unrestricted submarine boat warfare
Query: find phone number for irs
System: are you looking for phone numbers for the ins internal revenue service? Answer: yes
Intent: Internal Revenue Service Phone Numbers. 800-829-1040 For individual and joint filers who need procedural or tax law information and/ or help to file their 1040-type IRS Tax Help Line for individual returns (including Individuals] Schedules C and E); and, general account information for Form 1040 Filers. Facet: internal revenue service phone numbers
Query: Ips laws definition
System: are you looking for a definition of a limited partnership?
Answer: no
Intent: The Court will not let you establish LPS conservatorship unless it finds beyond reasonable
doubt, that the mentally ill person, is gravely disabled. Gravely disabled means that, because of mental disorder, the person cannot take care of his/her basic, personal needs for food, clothing, or shelter. Facet: limited partnership business
Table Examples of simulated interactions belonging the MiMarco dataset. In the first ad second example, both the intent and the facet sampled from relevant passage. In the third example, the intent is sampled from a relevant passage but the clarifying question is referring to negative topic facet.
Aliannejadi et al 2019 this ambiguity is addressed by using the concept of facet. Therefore, the generation of clarifying questions eq is conditioned on the initial query and facet f:
2022 Sekulić et al. 2021 Salle al. 2021
p(cqlq, f) -.p(egllocsss
(1)
where di is the tth token in the sequence and q<s the previously decoded tokens. Our clarifying question generation is based on pre-trained sequence-to-sequence model which fine-tuned to generate clarifying question eq given the following input sequence:
Query: Facet: f
(2)
where Query: and Facet: : are special tokens.
In this paper, we limit the clarifying questions those that expect yes/no answers.
3.3.2 User Simulation
The goal of the user simulation US is mimic the user's answer response to a clarifying question given his/her intent. In the user simulation, we expect accurate answers to clarifying questions, giving useful feedback to help the system understand his/her intent. The intent is representation of the information need or of the goal behind the initial query. It is used to constrain the user simulation's 's answer towards this goal (Kang et al.) 2019 Gao et al. 2022: Wu et al. 2020 Zhou et al. 2020 Fu et al.) 2020 Erbacher et al. 2022). While sophisticated user simulations have been proposed to exhibit various types of behaviors like cooperativeness or patience (Salle et al.) 2021), we limit the clarifying question to ask the intent is about a facet and the answer of the user simulation to 'yes' or 'no' answer. This limited form of answer is motivated by two reasons: (1) despite the simplicity, correct answer of this form corresponds basic realistic interactions with users and is highly useful for the system to better identify the intent behind the
6

---

Published in Transactions on Machine Learning Research (05/2024)
Algorithm Offline methodology for building Mixed-Initiative IR dataset
Require: D (P. R}
X X+ 0
Set of mixed-initiative IR-oriented interactions
for q @ do
F+ +
Extract the positive facets Extract the negative facets
F ++ ((P) for SE ( do
ag f)
b Generate the clarifying question b Building the answer
if f then yes'
else
'no' end if X; (q, aq.a)
x X u X;
b Increment the interaction set
end for end for return DUX
query. (2) This simple form of question and answer is easier to generate and evaluate. As an initial attempt, we prefer to start with this simple setting.
More formally, the user simulation aims estimating the probability of answer {yes, no) given query q, search intent int, and clarifying question:
p(alg, int, og)
(3)
This is implemented as a
model that encodes the following input:
Query: q Intent: int Question: eq
(4)
and generates a 'yes' /'no' answer.
Intent modeling The user's ' intent corresponds to the user's information need and is only known by the user. While multiple intent representations can be adopted (such as a detailed description of the information need [Aliannejadi et al. 2019 2021), a vector representation (Erbacher et al. 2022) or constraints (Kang et al. 2019 Gao et al.) 2022 Wu et al. 2020 Zhou al. 2020 Fu al.] (0200). IR datasots usually do not have annolated intent associated with the query. However, relevant passages are known in IR dataset. In this paper,
sampled relevant passage p Pt and assimilate its content to the underlying intent int. Formally: int + p. We acknowledge that this choice relics on a strong hypothesis that passages annotated represent the user's search intent, we further discuss it in Section
3.4 Leveraging Mixed-Initiative Interactions to Adapt Ad-Hoc IR Datasets
Given an ad-hoe IR dataset D, our objective is augment with mixod-initiative conversations X. It is worth distinguishing the creation of training and testing datasets since they have different purposes. The training set requires including positive and negative interactions to allow the community to train properly mixed-initiaaiive IR-oriented neural models. As reminder, those sesitive/negattivn interactions are built on the basis of relevant and irrelevant documents determining positive and negative facets. Using the same heuristics to generate a lesting dalaset is not. suitable since it would imply to include relevance judgments as evidence sources of the clarifying question generation at the inference step. Therefore, we propose to design an online evaluation methodology, leveraging the clarifying model CMM and the user simulation US generate sexced-iiiiiaive interactions without introducing bias related relevance judgments. We present these two methodologies aiming at generating offline and online datasets in what follows.

---

Published in Transactions on Machine Learning Research (05/2024)
Passage Ranking
Green Arrow: Hore There Dragons the second collaction of Mike Grell's run on Green Arrow (Volume 2)...
Query
The Deep: Here Be Dragons an original graphic noval from Gestait Publishing written by ewerd-winning writer Tom Taylor...
"here there be dragons comic
First Stage Retriever
Intent
Dragone fantsay by is the first book...
dragons
american James
Facets Extractor
'no"
"Arr you referring to the green anow comics?
Clarifying question
US answer
Clarifying question
generator
Figure
Online evaluation pipeline to create lixxedinitiaaive interactions on a test ad-hoc IR. set.
3.4.1 Building an offline training dataset with relevance judgments
Our offline methodology aims at generating a wide range of positive and negative mixed-initiative inter- actions on the basis of an ad-hoc IR dataset. To do so, we use relevant/irrelevann documents to build positive/negative facets constraining the clarifying question generation. As a supplementary quality con- straint in the dataset supervision, we would like to ensure that answers fit with the relevance of the used documents. Said otherwise, the user simulation presented in Section 3.3.2 replaced by a simple heuristic matching answers with the relevance of facets
a = 'ee'' f noo' otherwise
(5)
We propose the 3-step pipeline presented in Algorithm [ Given a query g: 1) positive and negative facets, resp. F and F=, are extracted from relevant and non-relevant passage sets, resp. Pt and Py 2) a mixed- initiative interaction X; is issued for facet. f, gemerating the associated clarifying question eq (with C.) and associating answer with the facet relevance (Equation 5) 3) the interaction set X is incremented with this new interaction Xi, allowing to build mixed-initiative IR dataset by associating the interaction set X built over all queries with the initial ad-hoc IR dataset D.
3.4.2 Building a testing dataset for online evaluation without relevance judgments
Our online methodology aims at generating mixed-initiative interactions without relying on rele- vant/irrelevan documents. Instead, we leverage pseudo-relevance feedback by using SERPs of a first-stage ranking model as a proxy to extract query facets. Each facet conditions the generation the clarifying question and the answer. More particularly, the proposed pipeline generate online mixed-initiative inter- actions for a query q is presented Figure [ It is built on the following steps: 1) ranking documents using a first-stage ranker (in our case BM25), 2) extracting the set of facets on the basis of pseudo-relevant/ psseudo- irrelevant documents, and 3) generating the srixediinteractve interaction.
Depending on the evaluation needs, different choices can be made regarding facet extraction. One can extract a single facet. from the lop-retrieved document to perform a single retrieval step for query (the strategy used in our experiments). Other tasks or evaluation objectives would require generating multiple facets, and accordingly, multiple mixod-initiative interactions. This can be done by identifying top/flop documents obtained with the first-stage ranking as sonidoo-elleant/iireeee documents; each document conditioning the facet. extraction described in Section 3.2
8

---

Published in Transactions on Machine Learning Research (05/2024)
Train set
Test set
Number of documents
8M
8M
Number of query
500K
6980
Avg number of interactions per query
38.5
Avg length of clarifying questions
11.0
Percentage of positive answers
26.7%
Percentage of negative answers
73.3 %
Table 2: Statistics of the generated mixed-initiative IR. dataset MIMarco.
4
Assessing the Quality of the Dataset Generation Methodology
In this section, we evaluate our methodology, and particularly, the quality of simulated interactions. Please note that we focus augmenting the MsMarco dataset. but our methodology is generalizable to any ad-hoc IR datasets.
4.1
Evaluation protocol
4.1.1 Datasets
We focus here on the MsMarco 2021 passages dataset (Nguyen et al. 2016) which is an open-domain TR dataset containing 8.8M passages and more than 500K Query-Passage relovance pairs with approximately 1.1 relevant passages per query on average. MsMarco is commonly used to Irain and evaluate first-stage retriever and cross-encoder architectures (Thakur al.) 2021). We leverage the MsMarco passage dataset with mined Hard Negatives released by sentence-transformerrs [Reimers & Gurevych 2019b) to build our tuples (q, p+ P_). Hard Negatives are passages retrieved using a state-ol-the-arl retrieval method, which are more closely related to the query. They allow us to generate more relevant questions and answers.
To train the clarifying model C.M, we use the filtered version of the ClariQ datasct proposed in (Sckulić et al. 2021) that maps clarifying questions with facets. All clarifying questions in this dataset are built so as to expect 'yes'/ /'no' answers. This dataset provides 1756 supervised tuples of (query- fuurr--facet-ccarffyingg question) for 187 queries.
To train the user simulation US, we do not use the answers included in the ClariQ dataset for supervision since answers are verbose (sentences with detailed information). Therefore, we leverage half of the train set of the MsMarco dataset 250000 queries) to extract positive and negative facets as detailed in Section 3.2 and generate clarifying questions using the C.M model. The supervison label related to answers is inferred as proposed in the offline evaluation (see Equation 5
To build final dataset including training and testing sets, we respectively apply the offline evaluation methodology (Algorithm I on the other half of the training set (not used to train the user simulation) and the online evaluation methodology (Figure [ on the test set of the MsMarco dataset. For the offline evaluation, because the original dataset includes sparse annotations, i.c. some passages are actually relevant but not annotated as such, it might be possible that relevant documents are considered as irrelevant ones. This trend is however exhibited in the MsMarco train set which only includes one relevant document by query. Therefore, to ensure labeling consistency, we follow (Qu et al., 2021] and denoise hard-negative in the training set using a well-trained cross-encoder model that captures similarities between passages.
For the online evaluation, we choose to generate single mixod-initiative interaction based on the top- retrieved document to fit with our extrinsic evaluation task based IR. We will release, upon acceptance, the complete generated datasets as well as the clarifying model CM and the user simulation Us to allow the generation of additional interactions. Statistics of the obtained mixed-initiative IR dataset, called MIMarco, are presented in Table 2 Table depicts some examples of simulated conversations generated from MsMarco queries.
https://hnugiingffcccccm htte::/mmmenniitieeooooooooeeete

---

Published in Transactions on Machine Learning Research (05/2024)
4.1.2 Baselines and metrics
Evaluating clarifying questions with automatic metrics. We follow (Sekulié et al. 2021 and com- pare our clarilying model, denoted CM,, with 1) a lemplate-based approach (Template). The template follows predefined sequence concatenating facets: 'Are you looking for + Facet'. 2) CMw//of MM//FFacett the version of our CM model only conditioned on the query. This is in fact based on T5 model trained as machine translation model, which generates clarifying question from the query only.
We evaluate the ability of CM to generate clarifying questions using references provided in the ClariQ test set. We consider the METEOR metric (Banerjee & Lavie 2005) and the average cosine similarity between sentence embeddings (COSIM). METEOR. is commonly used to evaluate machine translation output considering unigram recall and precision. At the sentence level, this has a good correlation with human judgments (Banerjoe & Lavie, 2005). To compute the similarity score, we encode the questions using a well-trained MiniL.M-L.6-v2 (Reimers & Gurevych 2019b). We use I-Lest. to assess the significance of metric differences (***; p-value-c0.005).
To evaluate if the generated questions on MsMarco are similar to their relative passage, we also compute the mean cosine similarity between clarifying questions and their retrieved relevant and non-relevant passages. We encode the questions using MiniLM-L.6-v2 (Reimers & Gurevych 2019b).
Human evaluations on clarifying questions. To compare and better assess the quality of a generated clarifying question on MsMarco, we performed a human evaluation. Given the initial user query and the passage used to generate the question, we asked annotators to evaluate the quality of 200 sampled clarifying questions among the three models (Template, CMw/oFacte and our CM model). To do so, annotators are asked to select a preferred clarifying question from the three suggestions displayed in shuffled order for the following criteria:
Usefulness: Assess if question can help to better understand or refine the query by providing additional information or suggestions.
Naturalness: Assess the question fluency and readability.
Relevance: Assess whether a question specific or related to the information contained in a passage.
Each aunolator evaluated 20 different cases and, for each metric to identify the best model output. We recruited 10 evaluators. Each instance is evaluated by 2 annotators and we obtain Kappa metric equal to 0.324 showing fair agroement between evaluators. We also distinguished results for both positive and negative user's answers by sampling relevant and irrelevant facets.
Human evaluations on answers. A strong hypothesis in our method is that clarifying questions gener- aled with facets extracted from relevant passages lead lo positive answers while using irrelevant passages to generate negative facets intrinsically leads to negative answers. To validate this strong hypothesis, we have shown human evaluators different instances including a query 9, clarifying question eg, and the relevant passage p used to build the facet f. For each instance, we asked human evaluators to answer with 'yes' or 'no' clarifying questions. This human evaluation involves 10 human annotators for a total of 200 questions, with balanced relevant and non-rclevant facets used to generate the clarifying question. Each instance is annotated by 2 humans. We obtain a Kappa metric equal to 0.472 showing a moderate agreement bet evaluators. To validate our hypothesis, set human answers as the reference, and compare them with our auto-labeling method (namely, the user simulation US) to calculate the accuracy metric.
4.1.3 Implementation details
For both C.M and US, we used the pre-trained T5 checkpoint available on the Huggingface hub (Raffel al. 2020 Wolf et al.) 2019). To finetune these two models, we used teacher forcing (Williams & Zipser 1989) and a cross-entropy loss. For optimization, we AdaFactor (Shazeer & Stern 2018), weight decay, and
10

---

Published in Transactions on Machine Learning Research (05/2024)
METEOR
COSIM
Template CMwffot acet
0.338
0.643
0.326
0.608 *.*
CM
0.557
0.812
Table 3: Evaluation of different clarifying models on the test set of ClariQ dataset. Significance two-sided
t-test: *** indicates statistically significant difference between baselines and our CMM model (p<00.0015)
P+
P
cq+
0.675
0.721
0.450
oq-
0.521
0.450
0.685
Table 4: Mean cosine similarity between generated clarifying questions and their related passages the train set. The eqt, eq- denote respectively clarifying questions generated using positive and negative facets.
learning rate of 5.10- with batch size of 64. Keywords embeddings are computed using off-the-shelf pre-trained MiniLM-L6-x2 model (Reimers & Gurevych, 2019b). The number of extracted words is fixed to k = 5 for the overall experiments. For inference, we use mucleus sampling (p=0.95) for the CM and Us models.
4.2
Evaluation of the generated interactions
4.2.1
Automatic evaluation.
Table 3 reports the effectiveness of the clarifying model on the ClariQ test set. Results show that our model CMM significantly outperforms all baselines. The lower results obtained by the CMMwooFacet baseline highlight that simple machine translation model is less effective than templates using facet terms. Facets are useful to constrain the clarifying model, and seq-to-seq models based on large language models are more natural than template-based methods. Facets are extracted from a relevant or irrelevant passage and used to generate clarifying questions. Table reports the cosine similarity between embeddings of questions and respective passages. We observe that the similarity between clarifying questions and their related passages (in bold) is higher than that between the clarifying questions and the queries. This shows that the generated questions are not generic to the query but oriented toward the provided passages.
4.2.2 Human Evaluation
We report human evaluation of clarifying questions Table The CMw/ooaced fine-tuned without facet generates more natural questions than other baselines (preferred for 46.3% of the sample). The CM model fine-tuned with facet generates more useful and relevant questions, this model is considered as the more relevant by evaluators in 59.9% of the test sample. This shows that the retrieved facet in the generation helps generate more useful and relevant questions.
In the human evaluation of answers, we obtain an accuracy of 0.685 between human answers and automatic labeling of clarifying questions. There are multiple causes explaining the difference between human answers and auto-labeling. 1) Facet may not always capture correctly the information provided a passage, leading poor clarifying questions. 2) The CM model does not always generate a question oriented toward the provided facet and produce a reformulation of the initial query, therefore asking question not related to facet.
Concerning the accuracy, we acknowledge that this may be low. We observed that this discrepancy may arisc from questions labeled as non-natural. The main limitation comes from automatic facet. extraction. Some keywords may not be representative of the document topics. Regarding the kappa, the disagreement may come from various reasons; naturalness may play an important role in judgment as observed the table G Additionally, annotators must always choose a preferred question, even if none of the proposed questions are actually useful given the query if clarifying questions similar, thus adding noise to the agreement.
11

---

Published in Transactions on Machine Learning Research (05/2024)
Answer
Naturalness
Usefulness
Relevance
positive
0.044
0.086
0.120
Template
negative
0.073
0.095
0.146
total
0.119
0.181
0.267
positive
0.243
0.195
0.077
CMu/oFacet
negative
0.220
0.140
0.056
total
0.463
0.336
0.133
positive 0.206
0.213
0.297
CM
negative 0.211
0.268
0.301
total
0.417
0.481
0.599
Table 5: Results of the human evaluation on Msmarco-passage. natural questions, however not as relevant as C.M with facet.
The CMM without facet produces more
Examples for human evalutation
Query webster family definition
CQ1: Are you looking for Noah Webster (1758-1843) lexicographer?
CQ2: Would you like to know
about. Webster family definition? CQ3: Are you referring to the lexicographer Noah Webster (1758-1843)?
Passage Noah Webster (1758-1843)
a lexicographer and language reformer. He is often called the Father of American Scholarship and Education. In his lifetime, he was also a lawyer, schoolmaster, author, newspaper editor, and an outspoken politician.
Query: what is venous thromboembolism
CQ1: Would you like to know more about venous thromboembolism'
CQ2: Would you like to know more about venous thromboembolism?
CQ3: Are you looking for venous thromboembolism?
Passage: Venous thromboembolism (VTE) is the formation of blood clots in the vein. When clot forms in deep vein, usually in the leg, it is called deep vein thrombosis or DVT. If that clot breaks loose and travels to the lungs, it is called pulmonary embolism or PE. Together, DVT and PE are known as VTE- dangerous and ially deadly medical condition.
Table 6: Table showing examples clarifying questions shown to human evaluators. Evaluators must assess the most natural relevant and useful question.
5 Evaluation on IR Task
In this section, propose to assess indirectly the quality of the generated dataset. through IR task. Indeed, previous works (Qu et al. 2020 Zhou et al. 2020 Li et al. 2018 Fu et al., 2020 Jia et al, 2022) have already used extrinsic tasks to validate dataset. Therefore, we introduce a neural ranking model which estimates passage relevance scores based on the query and a mixed-initiative interaction. Our objective is twofold: 1) Applying this model to our generated dataset provides some insights on whether the clarifying question and the associated answer actually give useful feedback to better understand the underlying information need. The evaluation is based on the following assumption: if ranking model using the generated interactions outperforms the one without them, the interactions are deemed relevant and useful. 2) We provide first bascline for mixed-initiatin IR tasks.
5.1 Neural Ranking Model Leveraging
Interactions
We propose a simple model basod on a cross-encoder architecture which has been shown to be effective for IR task, especially when using large language models (Pradeep et al. 2021). Previous cross-encoder aims predicting the relevance of passage p given query 9 P(relevant 1).p). Our model estimates a score
12

---

Published in Transactions on Machine Learning Research (05/2024)
for passages based on the query, clarifying question, and user answer (q. oq, a), i.c.
p(relevant = 1pp q, og,
(6)
Following (Pradeep el al. 2021); the above score is transformed to the log-probability of predicting (decoding) the true/falie ffals tokens, i.e.
Sp logp(truclg. p, p....a)
Following Pradeep et. al. 2021], we use the MonoT5 model and integrate mixed-initiative interactions to estimate document scores. The input sequence a concatenation of query, document, question, and answer separated by special tokens:
Query: q Document: Question: eq Answer: a
5.2 Training details
We used pre-trained MonoT5 checkpoint available on the Huggingface hub (Raffel et al. 2020 Wolf et al. 2019). We fine-tune this model on our train set in 1 epoch, using our methodology with teacher forcing and a cross-entropy loss. We consider a maximum sequence length of 512 and a batch size of 128 sequences. In order to properly learn to contrast between relevant and non-relevant passages given a question, we use in-batch negative answers.
For optimization, we use AdaFactor (Shazeer & Stern 2018 weight decay, and learning rate of 10-4, The model fine-tuning lakes approximately 4 hours on 4 RTX 3080 (24 Go).
At test time, we perform a first-stage retrieval on the initial query using the pyserini (Lin et al. 2021 implementation of BM25. We then apply our model as a second-stage ranker with additional information. We set the number of retrieved documents 100.
5.3 Metrics and Baselines
We use classical metrics to evaluate the document ranking quality, namely the normalized discounted cumn- lative gain (NDCG) at rank 1, 3, and 10; and the Mean Reciprocal Rank (MRR) at rank 10.
To evaluate the potential of mixed-initiative dataset, we compare the performance of our model, noted BM25+ against the following approaches:
BM25. BM25 is well-known sparse first-stage retriever commonly used baseline Thakur et al. 2021
BM25 + RM3. RM3 is reeudo-reeevance feedback method for query expansion. expanded using expansion terms extracted from the top 10 retrieved documents. RM3 is competitive baseline and is still used for benchmarking IR models Thakur al. 2021 Adolphs al. 2022
The query is
sccond-stage ranker pro-trained on the original training set of MsMarco, i.e. only queries and relevance judgments. This model achieves slate-of-the-art, performance on the beir leaderboard (Takur et al. 2021) and is natural baseline as BM25+C uses the same second- stage pre-trained model before fine-tuning it on mixed-initiative interactions. We compare the MonoT5 performances with and without interaction. Interactions are added in the models context, following the same pattern as for CLART5 as shown in the equation S.
BM25 + MonoT5. MonoT5
5.4 Effectiveness of mixed-inttiative-orrntedd neural ranking
We present the results of our mixeediiiitaative-drien neural ranking model obtained on the online evaluation pipeline presented in Section 3.4 applied on the MsMarco test set (Table 7
Table highlights the fact that additional information helps BM25+ CLART5 improve significantly all re- trieval metrics on the mixee-initiaaiiv--nnneend MsMarco dataset. For example, CLARSS increases
13

---

Published in Transactions on Machine Learning Research (05/2024)
MRR
NDCG@1
NDCGG3
NDCG
BM25
U. 0.1840
0.105
0.16900*
0.228
BM25 + RM3
0. 0.1566
0.0807
0. 031366
0.2021
BM25 + M5 [without interaction)
0.3522
0.2398
0.3457
0.4034
BM25 Mo (with interaction)
0.3413
0.2302
0.3381
0
BM25 CLART5
0.3863
0.2788
0.3817
0.4327
Table 7: IR. effectiveness the MiMarco test set. ***; two-sided t-test w.r.t. p-value<0.005
BM25+CLARTS, with
the MRR@10 score by 0.034 point compared to BM25+ Further analysis of the results on MsMarco shows that for 33.0% of the queries, the relevant passage is not retrieved in the top-100 by BM25, leading MRR@ 100 to 0.0. For 25.6% the queries Monot5 and ClarT55 obtaín the same MRR@10. Out of all the queries, BM25+ achieves a superior MRR@10 for 30.3% of them, whereas it yields lower MRR@10 for 11.1% of the queries. Overall these results show that the feedback provided by the user simulation the clarifying question is relevant and useful. It helps increase the ranking of relevant passages. This result indirectly confirms that the simulated interactions indeed encode relevant information to the underlying search intents, which is what real users would provide in conversations. Therefore, the proposed simulations are reasonable.
6 Complementary Experiments
6.1 Extension to Multi- Turn Interactions
In the previous section, simulated one interaction given single query X (q, eg, a) for the online inference. However, multiple different facets can be extracted from retrieved passages. This means that sequences of interactions Xo..., can be inferred by sequentially selecting different facets. While a new pool of passages could be retrieved using the last interaction, we only consider here facets from passages retrieved with the initial query. Each 4t turn exploits the tt document the document list by the first-stage ranking to build a facet and generate clarifying question. Multi-turn interactions are therefore generated in non-arbitrary order.
Impact on the design of the neural ranking model. We propose to extend the model to multi-turn re-ranking using multiple clarification turns around the same query. We evaluate passages using multiple interactions around the same search intent. At each time step new score s is computed for the passages in the same ranking using single interaction. This score is computed using Equation g which predicts cumulative relevance scores all interactions, i.c. the sum of relevance scores till time T. This score is used as the ranking score of document following sequence of interactions X; (g, a.,,...,.,, oo', a'). eg¹ and at are the clarifying question and the answer generated at timestamp t.
T log prreeevant = 1/q, P, aa)
(9)
where ST is the score document at time As the ranking is updated between turns, we select facets from the top retrieved passage at each time step. We evaluate the retrieval performance at different lengths of interactions, from = 1 T 5. We also report ranking entropy (Shannon 1948) as measure of the system's 's confidence by measuring how the scores are distributed in the ranking. This entropy is maximized when the score distribution is uniform over the ranking.
Results Table S shows the ranking performance while increasing the number of clarifying turns for the MsMarco test set. We observe that incrcasing the number of turns does improve MRR, NDCG scores and while minimizing the Entropy, suggesting that additional feedback helps refine the user's information need further.
14

---

Published in Transactions on Machine Learning Research (05/2024)
MRR@10
NDCG@1
NDCG@3
NDCG@10
Entropy
BM25 + CLART5 T-1
0.3863
0.2788
0.3817
0.4327
2.951
BM25 + CLART5 T-2
0.44467
0.35186
0.43734
0.48038
2.303
BM25 + CLART5 T-3
0.48176
0.39828
0.47483
0.51089
2.163
BM25 + CLART5 T=4
0.50861
0.43266
0.50321
0.53183
2.06
BM25 + CLART5 T=5
0.52949
0.45946
0.52311
0.54814
1.99
Table & Effectiveness results on T interaction turns issued on the MsMarco test sets.
We depict qualitative examples of successful and unsuccessful clarification sequences in Table as well examples of failure cases in Table T0 We can in the first. example that additional interaction helps better refine passage scores leading to better MRR@ 100, while the entropy decreases. In the last interaction, the entropy is very low, meaning that the score distribution is dense on few passages. On the other hand, the second example is failure case where relevant passages are not even retrieved. The failure cases we observe, where the interaction turns deteriorate the ranking, shows that generated interactions are still not perfect. There is room for improvement in the future.
Multi-turn Conversations
mrr@100
Entropy
User: consolidate function
0.125
4.388
System: do you want to know what human resources functions the county provides?
0.14286
3.91
User:
System: are you looking for media files referenced to itunes?
0.16667
2.74
User: no
System: do you want to know how to consolidate data source?
0.2
2.61
User: no
System: do you want know how to use excel consolidate data?
0.5
0.09
User: yes
User: where was the banhaus built
0.04167
3.854
User: System: do you want know where the banhaus was founded?
0.25
3.52
System: do you want to know where the gropius sommerfeld house was built?
0.5
0.69
User: yes
System: do you want to know where the bauhaus art school was built?
1.0
0.000218
User: no
User: here there be dragons comic
0.16667
4.27
System: are you looking for a graphic novel called the deep ?
0.2
3.12
User: no
System: are you referring to the green arrow comics?
0.25
2.93
User: no
User: System: do you want know what lenox globe dragons means is?
0.33333
1.81
System: are you looking for a james owen american comic book
1.0
1.15
User: yes
Table 9: Examples successful cases. In this example, asking clarifying questions helps to refine the user's intent and better identify relevant passages. These examples show typical ambiguity the retrieved document. For example, the Bauhaus can refer to the rock band, the art school, the prize .... here there be dragons comic can also refer multiple comics with the same name.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Failure cases
mrr
Entropy
User: what are tor
0.0
3.776
System: do you want to know if tor is free?
0.0
3.612
User: no
System: do you want to know more about tors transoral robotic surgery?
0.0
3.586
User: no
System: do you want to know how to connect to computer network
0.0
3.567
User: no
System: are you looking for web browser for tor onion?
0.0
3.541
User: no
System: do you want to know about tor anonymity?
0.0
3.329
User: no
User: what is the fastest production sedan System: do you want to know what the fastest production sedan is?
1.0
3.484
0.1
3.463
User: yes
System: do you want to know what the fastest production sedan is?
0.1
3.416
User: yes
System: do you want to know what the fastest production sedan is?
0.090
3.346
User: yes
System: do you want to know what the fastest production sedan is?
0.083
3.251
User: yes
Table 10: Examples of failure cases. In the first example, none of the relevant passages are retrieved, asking clarifying questions and re-ranking passages do not improve scores. In the second example, the CM fails to capture passage's facet and generates the same questions. Additionally this tends to decrease the scores.
MRR@10
NDCG@1
NDCG@3
NDCG@10
BM25
0.2634
0.1648
0.2354
0.3055
BM25 + RM3
0.2483
0.1457
0.2177
0.2941
BM25 + MonoT5
0.4422
0.3314***
0.4190
0.4764
BM25 CLART5 T-1
0.4749
0.3725
0.4471
0.5012
BM25 +CLART5 T=2
0.54674
0.46002
0.51072
0.552
BM25 +CLART5 T-3
0.58267
0.50956
0.54204
0.57787
BM25 +CLART5 T-4
0.6047
0.53911
0.563
0.59291
BM25 +CLART5 T-5
0.62115
0.56286
0.57666
0.60469
Table 11: IR. effectiveness on the augmented version of the Natural Question test set (3452 queries). ***, two-sided t-test w.r.t. BM25+CLART5 T-1. with p-value-c0.005
6.2 Transferability of the Methology
To test the potential to transfer the generators trained on a dataset to other datasets, we apply the same clarifying model CM and user simulation US trained on MsMarco (as described in figure [ generate simulated interactions
new Natural Questions dataset [Kwiatkowski et al. 2019). Results presented in Table II The higher results obtained by method BM25+CLART5 w.r.t. other baselines suggest that the generated mixed-initiative interactions can benefit the neural ranking model. In other words, the generators trained on a datasel can be transferred to another dataset to create reasonable simulations. The experimental results on Natural Questions are consistent with those on MsMarco. This result particularly interesting, showing that our methodology can be usod inference of out-of-domain datascts. This opens the potential perspective of constructing generic simulators of mixed-initiative interactions for any ad-hoc IR dataset.
16

---

Published in Transactions on Machine Learning Research (05/2024)
of similar
0.62
0.52
0.48
6.45
644
Humber teteeddrr
0.24
0.75
0.74
0.00
0.50
0.43
0.30
0.00
0.75
0.95
Number of intraations
Figure Passages ranking similarity between interaction turn. Ranked Biased Overlap (RBO) metric (p-0.9). Interaction corresponds to the document ranking using Bm25 + MonoT5
6.3 Additional Analysis: Multi-turn Ranking similarity
In order to validate our results, we compute similarity metrics between ranking each turn in the con- versation. To measure the similarity between document rankings at different iterations, rely on the Rank-Biased Overlap (RBO) (Webber et al. 2010):
RBO(S,T,p) = (1-1) -2-p) p))
Σ
p
Add
(10)
where S and are two document rankings, d is the actual depth of the ranking. Aa expresses the agreement (the size of the intersection of the two rankings) at depth d: A. = [8.nN.al determines the weight given to the top ranked document. We set p = 0.9, which means that the 10 top documents weigh 85% of the score.
RBO measures the similarity between incomplete and non-conjoint rankings and also values more hcavily top ranked document. The more diverse the rankings, the lower the score. Figure shows the ranking similarity for each additional user feedback. We observe that the ranking seems to stabilize with the number of interactions: the similarity is higher between 4 and 5 interactions than between O and interactions.
7 Conclusion and discussion
There is critical need for adequate datasets with mixed-initiative interactions for conversational IR, but creating such a dataset very costly, In this paper, we proposed a method to augment ad-hoc IR datasets by simulating a simple form mixed-initiative interactions between user and a conversational IR system. This method generates automatically clarifying questions and answers from a large open-domain IR dataset, making it possible to experiment conversational IR approaches at large-scale. The proposed approach is generic and can be applied to any existing ad-hoc IR dataset. In the experiments, augmented the MsMarco dataset and evaluate the quality of the interactions with intrinsic and extrinsic tasks, relying on automatic metrics and human evaluations. The results show that, despite the simple form, the generated interactions are relevant to the search intents and useful for better document ranking. This is a first investigation on large-scale datasct augmentation for conversational IR. It demonstrates the fcasibility of the automatic construction of datasets. As first investigation, this study has several limitations that can be improved in the future.
17

---

Published in Transactions on Machine Learning Research (05/2024)
First, our investigation is limited to clarifying questions based single facet, often assimilated to questions of the type: "Are you referring to "Tacet""" However, real clarifying questions might also question about multiple topics in a single turn (ex: Are you interested to know about topicI, topic2 or topic3) or also be formulated as open-ended questions (c.g., 'What would you like to know about topic?"). These more complex questions are more difficult to generale and answer in simulations, but can polentially bring more information and be more natural in the conversation.
Second, the facet extraction relied on a few keywords and this can be improved. We observe that when passages are long and address multiple topics, the generated question may not represent the topic addressed in the passage.
Third, the user simulation has been limited to 'yes'/'no' answers. In a more sophisticated conversational search, the user might provide more and various information in the answer. Simulating more complex user's answers is a challenge for the future.
Finally, we also generated multi-turn interactions but did not consider the dependency between turns. In real conversational search, later turns may depend on previons ones. More reasonable simulations of multi-turn interactions should take the dependency into account.
Despite the limitations, the demonstration of fcasibility made in this paper to create large-scale conversational IR datasets opens the door for more investigations at large scale on the topic.
References
Leonard Adolphs, Michelle Chen Huebscher, Christian Buck, Sertan Girgin, Olivier Bachem, Massimiliano Ciaramita, and Thomas Hofmann. Decoding a neural retriever's latent space for query suggestion, 2022.
URL
wttp////g/bb////o/o////// 12084
Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. Asking clarifying questions open-domain information-sccking conversations. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, pp. 475 484, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361729, doi: 10.1145 335//33331113333111 381.331166. URL attpp:://bbi.orr/11..11413 33331131118433333555
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. Building and evaluating open-domain dialogue corpora with clarifying questions. In EMNLP, 2021.
Avishek Anand, Lawrence Cavedon, Matthias Hagen, Hideo Joho, Mark Sanderson, and Benno Stein. Conversational scarch report from dagstuhl seminar 19461. CoRR, abs/2005.08655 2020. URL
Layla El Asri, Jing Hc, and Kaheer Suleman. A sequencc-to-ssqqueee model for user simulation in spoken dialogue systems, 2016. URL
Satanjeev Banerjee and Alon Lavie. METEOR:
An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACI. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summorization, pp. 65 72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https://aclaatthooogg wrg.ey.org//05000000
Nicholas J. Belkin and W. Bruce Croft. Information filtering and information retrieval: Two sides of the same coin? Commun. ACM, 35(12):29-33) 38, dec 1992. ISSN 0001-0782, doi: https:///oi.coo org/ 10. 100011155518888 138811188889918811111
URL
Emily M. Bender, Timmit Gebru, Angelina McMillan-Major, and Shmargarel Shuntchell. On the dangers stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, pp. 610-623, New York, NY, USA, 2021.
Association for Computing Machinery. ISBN 9784453883097. doi: 10.1145/34421888 00.144/8442188.444552. URL https:
7/doi. http///...15//4421111 3445922
18

---

Published in Transactions on Machine Learning Research (05/2024)
Keping Bi, Qingyao Ai, and W. Bruce Croft. Asking clarifying questions based on negative feedback in conversational search. In Proceedings of the 2021 ACM STGIR International Conference on Theory of Information Retrieval, ICTIR "21, pp. Machinery. ISBN 9781450386111. doi: 10.144//3471158.3472222 URL https://doi http:::///oo.o//1..../
New York, NY, USA, 2021. Association for Computing
3471158. 3472232
Danqi Chen, Adamn Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1 16135/111/177117.1 URL https////aclaatioooooooo////////
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174-2184 Brussels, Belgium, Oetober-November 2018. Association for Computational Linguistics. doi: 10.18653, (0.18653/v1/DDDD //////1//D112//// 1241. URL battps::/acclamtiooogg.oo//////
1241
Jennifer Chu-Carroll and Michael K. Brown. Tracking initiative in collaborative dialogue interactions. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the Euro- pean Chapter of the Association for Compulational Linguistics, pp. 262 220, Madrid, Spain, July 1997. Association for Computational Linguistics. doi: 10.3115/9769000 99111/9999999996511. URL lttps:s//acaanthology org/P97-1034
Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. Overview of the TREC 2009 web track. In Ellen M. Voorhees and Lori P. Buckland (eds.), Proceedings of The Eighteenth Tert REtrieval Conference, TREC 2009, Gaithersburg, Maryland, USA, November 17-20, 2009, volume 500-278 of NIST Special Publica- tion. National Institute of Standards and Technology (NIST), 2009. URL http::/ttec. nist .ov gow.ggg//uuuuu gov/pubs/ trec18/papers Seep09 WEB09. OVERVIEW. pdf
J. Shane Culpepper, Fernando Diaz, and Mark D. Smucker. Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in lorne (swirl 2018). SIGIR Forum, 52 (1):34 90, aug 2018. ISSN 0163-5840. doi: 10. 1LL145/82744744.2777888 URL https://doi. htttp:://do..org10...r 3274784.3274788
Jeffrey Dalton, Chenyan Xiong, and Jamic Callan. Tree cast 2019: The conversational assistance track overview, 2020a. URL https://arxiv http::////rxi.ooo/abb//////////
13624
Jeffrey Dalton, Chenyan Xiong, Vaibhav Kumar, and Jamic Callan. CAsT-19: A Dataset for Conversational Information Seeking, pp. 1985-1988. Association for Computing Machinery, New York, NY, USA, 2020b. ISBN 4781450380164. URL htttps://do..or///
Jeffrey Dalton, Sophie Fischer, Paul Owoicho, Filip Radlinski, Federico Rossetto, Johanne R. Trippas, and Hamed Zamani. Conversational information seeking: Theory and application. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retricval, SIGIR '22, pp. 3455 3458, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. URL https://doi..og/11.... 11/////111115//47449999999 35537777777737777888888
doi:
Zhicheng Dou, Zhengbao Jiang, Sha Hu, Ji-Rong Wen, and Ruihua Song. Automatically mining facets for querics from their scarch results. IEEE Thans. on Knowl. and Data Eng., 28(2):385-33) 397, feb 2016. ISSN 1041-4347. doi: 10.1109/TKDE.20111 TKDE.2015.24777753 URL https://doi
Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. Can you unpack that? learning to rewrite crrstioos-in-conter In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (FMNLP- IJCNLP), pp. 5918 5924, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1119//11DDD110001 URL http::///aclaathooogyoorr////9
Pierre Erbacher, Ludovic Denoyer, and Laure Soulier. Interactive query clarification and refinement via user simulation. sigir, 2022. doi: 2201//AAAX...22..5555.. URL http::///arvvv.oo/aaaa/////s 15918
19

---

Published in Transactions on Machine Learning Research (05/2024)
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli ELI5: Long form question answering. In Proceedings of the 57th Annual Meeling of the Association for Compulational Linguistics, pp. 3558 3567, Florence, Italy, July 2019. Association for Computational Linguistics. doi:
10.18653 10.18653/v1/P19-11 /111/111/111//// 1346. URL https://aclanthoolog)) 1ttp://nnnooooooooo/1-----9991 134/
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 221, pp. 2288 2292, New York, NY, USA, 2021. Association URL httpss///ti.....ooo org/
for Computing Machinery. ISBN 9781450380379. doi: 10.1145/34048888
10. 3 3463098
Zuohui Fu, Yikun Xian, Yaxin Zhu, Yongleng Zhang, and Gerard de Melo. Cookie: A dataset for conver- sational recommendation over knowledge graphs in e-commerce, 2020. URL https://arxiv.r org/ abs//
0008.09237
Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recommender sys- tems. In Proceedings of the 31st ACM International Conference on Information amp; Knowledge Manage- ment, CIKM '22, pp. 540 550,550, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392365. doi: 10.1145 (0.1145/35118000 155553511 URL https://doi. http://6...0r/11111.115//1111111 11111/111111888. 1116//361188833572220
Amelia Glaese Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillnghaan Jonathan Uesato, Po- Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaune Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Tason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukeuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements, 2022. URL httts:///aaaat http///axxvvvrrraaaa/ 2209. 14375
Nam Le Hai, Thomas Gerald, Thibault Formal, Jian-Yun Nie, Benjamin Piwowarski, and Laure Soulier. Cosplade: Contextualizing SPLADE for conversational information retrieval. CoRR, 2as//2301.4400 2023a. doi: 40.48550/arXiv.. www.....0....4......... URL
48550/ 2301.04413
Nam Le Hai, Thomas Gerald, Thibault Formal, Jian- Yun Nie, Benjamin Piwowarski, and Laure Soulier. Cosplade: Contextualizing splade for conversational information retrieval, 2023b. URL https: h //arxiv.. org/ htsp/abb/333..0444
Helia Hashemi, Hamed Zamani, and W. Bruce Croft. Guided transformer: Leveraging multiple external sources for representation learning in conversational search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development. in Information Retrieval, SIGIR 220, pp- 1131-1140, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380161, doi: 10.1145/ 3397271.340161.. URL https://doi. org/ 10. 144113397771.34000611
Scbastian Hofstätter, Sheng-Chich Lin, Jheng- Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21 pp. 113-122, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379. doi: 1..115533444355.34
URL https://doi......
10. 111111111145/44488333 34628911
Meihuizi Jia, Ruixue Liu, Peiying Wang, Yang Song, Zexi Xi, Haobin Li, Xin Shen, Meng Chen, Jinhui Pang, and Xiaodong He. E-ConvRec: A large-scale conversational recommendation datasct for E-commerce customer service. In Proceedings of the Thirleenth Language Resources and Evaluation Conference, pp. 5787-5796, Marseille, France, June 2022. European Language Resources Association. URL https:// aclanthology org/2022 2022..1eec11.22
Dongveop Kang, Annsha Balakrishnan, Pararth Shah, Paul Crook, Y-Lan Boureau, and Jason Weston. Recommendation as a communication game: Sclf-supervised bot-play for goal-oriented dialogue. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternalional Joint Conference on Natural Language Processing (EMNIPP-JCCNLP) pp- 1951-1961, Hong
20

---

Published in Transactions on Machine Learning Research (05/2024)
Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DII URL httpps//acclaatboooo...../////.
Kimiya Keyvan and Jimmy Xiangji Huang. How to approach ambiguous queries in conversational search:
A survey of techniques, approaches, tools, and challenges. ACM Comput. Sure. 55(6), dec 2022. ISSN 0360-0300. doi: 10.1145/3534965 URL https://doi. http::////oo..o//////// 10. 11410..14/3334966
Weize Kong and James Allan. Extracting query facets from search results. In Proceedings of the 36th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '13, pp. 93 102, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450320344. doi: 10. 10.1145/2484028 2484097. URL https://doii org/10..//0 1145 /114522444028... 744444444892288444997
Florian Kreyssig, liigo Casanueva, Pawel Budzianowski, and Milica Gašić. Neural user simulation for corpus- based policy optimisation of spoken dialogue systems. In Proceedings of the 19th Annual STGdial Meeting on Discourse and Dialogue, pp. 60-69, Melbourne, Australia, July 2018. Association for Computational URL https: /aap:://allaanne aollaoooootoyyyooooooooooooooooa org/ /W1g-
Linguistics. doi: 11116633/1/WI//
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Tllia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural ques- tions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/tacl a 00276. URL. pettps://aclanthollne www.laanttoloooooo///11------- 0r11/19-0200011
Raymond Li, Samira Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp. 9748 9758, Red Hook, NY, USA, 2018. Curran Associates Inc.
Xinjun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. End-to-end task-completion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing ( (Volume Long Papers), pp. 733 733-743 743, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL latttss://acaanthologg.. htwp:/anttoolggooor//---00000000
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyscrini: A Python toolkit for reproducible information retrieval research with sparse and dense repre- sentations. In Proceedings of the 44th Arnual International ACM SIGIR Conference on Research and 2021.
Development in Information Retrieval (SIGIR 2021), pp.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2021. URL https:/ /ate:i///aaav//aaaa org/ wttpaa//......a 09332
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension datasct. CoRR, abs/1611.09220 2016. URL http://axivv.oogaaa//////aa
09268
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkinn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL http:////aavvv.o//aa/////00// org/abs//203.. 02155
Paul Over. The trec interactive track: an annotated bibliography. Information Processing Manage- ment, 36033386 2001. ISSN 0306-4573. doi: http:://da.oogg/o.oooo//////4000 URL https///ww..cmmmddree.t
sciencedirect. http:////..ooooorrrrrrr////////.
Interactivity
the Text Retrieval Conference (TREC).
21

---

Published in Transactions on Machine Learning Research (05/2024)
Paul Owoicho, Ivan Sekulic, Mohammad Aliannejadi, Jeffrey Dalton, and Fabio Crestani. Exploiting simu- lated user feedback for conversational search: Ranking, rewriting, and beyond. Tn Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, pp. 632-642, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394086. URL httpss///dii
doi: 10. 0114455553118.3555
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kai-Fai Wong. Deep Dyna-Q: Integrating planning for task-completion dialogue policy learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2182 2192, Melbourne, Australia, July 2018 Association for Computational Linguistics. doi: 10.18653/v1 1203. URL https://aclanthol.gg/
org/P18-120 1203
Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequeeee models. CoRR, abs/2101.056617 2021. URL. httpss://arxi org/ abs/2101. 05667
Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit. Tyyer. Open-retrieval conversational question answering. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, pp. 539-548, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164 doi: 10. 331144/3397277134000 10. 12 1144/3397221111 3401110
URL URT.ttts: httph::tt///// /rr//
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open- domain question answering. In Proceedings of the 2021 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, pp. 5885-5847, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1, /2021 naacl- URL https: //aclaantollgy.orgg//
-main. 466
Filip Radlinski and Nick Craswell. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR '17 pp. 117-126, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450346771. doi: 10.1145/
0020165.3020183. URL https: http:::////oo.or/1.... 10. 1144/1020161...... 3020183
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140):1-66 -7,, 2020. http://t: http://mlrrrrrgpperreeeee arg/papers//211200077 html
Hossein A. Rahmani, Xi Wang, Yue Feng, Qiang Zhang, Emine Yilmaz, and Aldo Lipani. A survey on asking clarification questions datasets in conversational systems. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2698 2716, Toronto, Canada, July 2023. Association for Com- putational Linguistics. doi: 10.18653/<1/2023.1 acl-long 152. URL https://aclaathhooogg.ooo
acl long. 152
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784 789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/w1/P188
URL https://aclannhooogg/ wtt.://atw.....o..or//////2222
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answer- ing challenge. Transactions of the Association for Computational Linguistics, 7:249 266, 2019. doi:
10.1162/taci 00266. URL https://acllantholggy htw.://aatttttooooooo0//-1-0-110 1rg/
Nils Reimers and Iryna Gurevych. Sentence-BERI Sentence embeddings using Siamese BERT-networks
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNLP--JCNNLP)) pp. 3982 3992, Hong
22

---

Published in Transactions on Machine Learning Research (05/2024)
Kong, China, November 2019a. Association for Computational Linguisties. doi: 10.18653/11/D19-1444 URL htttges//eclaathooooooorr////--/
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019b. URL httpp//aaxiv..oe wttp://rrggabab/........ 10084
Alexandre Salle, Shervin Malmasi, Oleg Rokhlenko, and Eugene Agichtein. Studying the effectiveness of con- versational search refinement through user simulation. In Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (eds.), Advances in Information Re- trieval, pp. 587-602, Cham, 2021. Springer International Publishing. ISBN 978-3-030-72113-8.
Jost. Schatzmann, Blaise Thomson, Karl Weilhammerr Hui Ye, and Steve Young. Agenda-based user simula- tion for bootstrapping a POMDP dialogue system. In Human Language Technologies 2007: The Confer- ence of the North American Chapter of the Association for Computational Linguisties; Companion Volume, Short Pupers, pp. 149 152, Rochester, New York, April 2007. Association for Computational Linguistics. URL https:///claatthol.s www.gy..rgg/ooo. 20032072
Ivan Sckulić, Mohammad Aliannejadi, and Fabio Crestani. Towards facet-driven generation of clarifying questions for conversational ional search. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '21, pp. 167 175, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450386111. doi: 10.1145/3471155.3472222
URL https://doi..oo
10. 11 /111/344111155 3472257
Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. Exploiting document-based features for clarifi- cation in conversational search. In Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvág, and Vinay Setty (eds.), ),, Advances in Information Retrieval, pp. 413 427, Cham, 2022. Springer International Publishing. ISBN 978-3-030-99736-66
Ivan Sckulié, Mohammad Aliannejadi, and Fabio Crestani. Evaluating mixed-initiative conversational scarch systems via user simulation. In Proceedings of the Fifleenth ACM International Conference Web Search and Data Mining, WSDM '22, pp. 888-896, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450891320, doi: 10. 011145//48856033499444
URL https://doi.org:::
3488560. 3498440
Chirag Shah and Emily M. Bender. Situating search. In ACM SIGIR Conference on Human Information Interaction and Retrieval, CHIIR '2, pp. 221-232, New York, NY, USA, 2022. Association for Computing URL https://doi.org::
Machinery. ISBN 9781450891863. doi: 10.1145 01114///396336.5558866
0198366.350584
Claude Elwood Shannon. A mathematical theory of commmication. The Bell System Technical Journal, 27:379 423, 1948. URL http://plan9.bell1 bell- 1illl-lab..rr hoom///mms/////////
pdf
Prafull Sharma and Yingbo Li. Selfssuperviied cantextual keyword and keyphrase retrieval with self-labelling, 2019. URL
0073.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause (cds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4596 4604. PMLR, 10 15 Jul 2018. URL htttps:/ .ml.iiinn.mm..mm...mmmm......... prrsss/80//saazerr/
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. Lan- guage models that seek for knowledge: Modular search amp; generation for dialogue and prompt comple- tion, 2022. URL
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A het- crogencous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021. URL httpp:///opereerii.eee/////u///p http:///..oorrrr///o////oooba//a
23

---

Published in Transactions on Machine Learning Research (05/2024)
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickeu, Pranesh Srinivasan, Laichee Man, Kathleen Meier- Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkunar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Areas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022. URL https: http///wwv...gaaaaaa/ee abgggbbb/200...... 08239
Johanne R. Trippas, Damiano Spina, Paul Thomas, Mark Sanderson, Hideo Joho, and Lawrence Cave- don. Towards a model for spoken conversational search. Information Processing Management, 57 (2):102162, 2020. ISSN 0306-4573, doi: https://cooi.orr/ 10. 1016//..m..j.mmmmmm http:/.....m...1121...... 2019.102162. URL https:
Zhenduo Wang and Qingyao Ai. Controlling the risk of conversational scarch via reinforcement learning. In Proceedings of the Web Conference 2021, WWW '21, pp. 1968 1977, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127. doi:
URL https:
7/doi. org/10 0111/14444440 34498933
William Webber, Alistair Moffat, and Justin Zobel. A similarity measure for indefinite rankings. ACM Trans. Inf. Syst., 28(4), nov 2010, ISSN 1046-8188. doi: 10.1145/1852100211522106, URL https://doi.
org/ 10. 1115/115211221 1852106
Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Newral Computation, 1(2):270-284) 1989. doi: 10.1162 aeeco.199.11..2770
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Touf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariana Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transformers: State-of-the-art natural language processing, 2019. URL https://arxiv.. org/abs/ 1910. 03771
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou, MIND: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Mecting of the Association for Computational Linguistics, pp. 3597-3606, Online, July 2020. Association for Computational Linguistics. doi:
URL
https://aclanthooogy/ or / 2020 acw...acc-aam.. acl-main 331
Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. Generating Clarifying Ques- tions for Information Retrieval, pp. 418-428. Association for Computing Machinery, New York, NY, USA, 2020a. ISBN 9781450370233. URL https://doi. .ow.oogg///// /01 1111//16/33642333 3380126
Hamed Zamani, Bhaskar Mitra, Everest Chen, Gord Lucck, Fernando Diaz, Paul N. Bennett, Nick Craswell, and Susan T. Dumais. Analyzing and learning from user interactions for search clarification. CoRR, 0066/ 2020b. URL https: httppss////axiv.//e wwwp/oggg.a....ooo//.///
Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. Conversational information seeking. CoRR, abs/2201 008008, 2022. URL htttss://arxiti http:/axxivv..ragbbaa/aa wttp/rg////bgrrooeeeeee 08808
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. Towards topic-guided con- versational recommender system. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4128-4188 Barcelona, Spain (Online), December 2020. International Committee on Com- putational Linguistics. doi: 10.18653/v /1tp://./////0////g//// m32n.336. URL hhttps:: htapp/////claathoogoo.oo.g. org.:logg..gg/.00....g
coling-maii main.365
24

---

Published in Transactions on Machine Learning Research (05/2024)
A Appendix
25


Reviews:
[{"review_id": "wVTanU2yQC", "reviewer": "Reviewer_RAMy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* Motivation is strong.\n* The evaluation framework is interesting and also covers extensions such as transfer to different dataset. \n\nWeakness:\n* The evaluation setting b/w single-turn vs. multi-turn interaction should be more clearly presented and discussed before we get to Section 6. Section 5.1, it assumes there’s only one clarifying question? This is a bit confusing as previously it assumed there could be multiple clarification questions per query? There could be multiple valid clarification question (one per facet?), how do you pick which facet to use for clarification question model? \n* The data setup is unnatural: The dataset has an average number of 38.5 interactions per query — I think this setting is very unlikely? This is largely because of the design decision of having yes/no answer only clarification question..  This deserves a bit more discussion.\n* Baselines are weak: \nFor evaluation on IR task: i think we should consider other baselines beyond BM25, such as dense retriever such as contriever or GTR? \nThe baselines for constructing clarification questions (template & CM w/o Facet) are both very weak — one is not even using full input. I am not sure this is very useful for readers. I am not sure this model is the main contribution of this paper -- very straightforward fine-tuning on supervised dataset w/ different prompt template than baselines?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies building an information retrieval system that can take initiative, specifically asking clarification question. They focus on building a large-scale datasets through simulation with LLM. Specifically, they fine-tune T5 checkpoint  to generate clarification question with ClariQ dataset. \nThey evaluate their approach by showing that simulated mixed-initiative interactions can help document ranking for MsMARCO, showing it outperforms baselines such as BM25 and its variants. \nOverall this is an interesting direction, but I found the paper a bit hard to comprehend, and need iterations to be published. Please see detailed comments below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here I provide some other comments: \n\nSection 4.2: I find the accuracy of 0.685 to be fairly low, making me a bit concerned about the dataset quality. Concrete examples would be helpful. \n\nSection 4.1.2: The Kappa of 0.324 is considered fairly low — what are some reasons for disagreements? \n\n\n\nTable 2: what’s the unit of average length of clarifying questions? \n\nOverall writings can be improved. Some concrete comments below: \n* The last bullet point on introduction, “augmented” does not make sense to readers at this point. \n* Please unify formatting, especially regarding single quote / double quote marks, these are used inconsistently throughout the paper. \n* I think it would be helpful of define a few key terms - such as “facet” and “intent” a bit clearly earlier in the paper. \n* At the end of Page 4 - “We assume P-q intersection P+q =? Is this missing something? \n\nAlgorithm 1: \nThis would be way more easier if you notate each component carefully in the figure caption. \n\nSection 2.1: incorrect citation - QuAC is a multi turn QA setting, similar to CoQA.\nThird paragraph — it would be helpful to clarify what’s the classification label space for these classification datasets.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XyIChuA5fd", "reviewer": "Reviewer_k81C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- Augmentation algorithm is completely human-free. This allows larger scale applications.\n- The evaluation is very extensive. Reviewer appreciate the effort to have human blind test which adds more confidence than just similarity measurements.\n- Additional evaluation on a different dataset to show the generalization capability of the method.\n\nWeakness\n- Baseline in 4.1.2 seems too weak. The CM w/o Facet baseline has no access to retrieved top documents, while still producing more fluence questions. A fair baseline would be to give the model at least some abstract of the retrieved documents.\n- Section 6.3's goal is not clear. It shows the trend of increased similarity of ranking which is almost expected with the kind of iterative narrowing of search. The Reviewer see little additional value in that.\n- Minor writing issues, could use more examples to help readers understand. See requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the context of IR dataset, the paper proposed a method to generate synthetic clarification questions and user interactions to make a dataset fit for interactive search training. The method involves the creation of a Clarification Model and a User Simulation. Given a query and a list of relevant + irrelevant documents retrieved, the method extract phrases as facets. Then a pretrained seq-to-seq deep model turn the facets into yes/no clarification questions (the Clarification Model) part. The User Simulation then give a binary answer to the clarification question based on the true intent; it's also implemented as a seq-to-seq model.\n\nThe paper mainly tests the methodology on the MsMarco dataset with two kinds of evaluations: directly on the quality of the generated clarification questions and simulated user responses, and indirectly by training a retrieval model on the augmented dataset and comparing it with baseline models. For direct evaluation, the paper showed similarity with reference golden questions from ClariQ, and also performed human blind evaluation. The result shows that model generated interactions are better than the baselines. For indirect evaluation, the paper finetuned a Mono-T5 pre-trained model to take the additional interactive data as input, and the model outperformed baselines, indicating that the augmentation of the dataset is effective in building better retrieval models.\n\nThe paper also performed additional evaluation by training a multi-turn retrieval model, which again showed stronger performance. The CM and US models are then moved to a new dataset, where they still show good performance, suggesting that the learning from MsMarco by the CM/US model can be reasonably transferred to a new dataset without retraining.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- There are different naming of the dataset, MsMarco and MiMarco. It's confusing and seems to be misspelling at the beginning, but later Reviewer realized that I in MiMarco might represent \"Interactive\". However a clarification of the name at the first mention (table 1)  is strongly recommended.\n- Page 7 last sentence of Section 3.3: \"... relies on a [strong hypothesis]  and we discuss it in Section 7\". It's better to briefly summarize the hypothesis here to save the readers a trip to Section 7.\n-  Section 5.4: \"BM25+CLART5 obtains a better MRR@10 while [11.1%] obtain a lower MRR@10\" is this a typo?\n- Consider improving the baseline in Section 4.1.2. See the weakness section.\n- Consider removing Section 6.3", "broader_impact_concerns": "No broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bs8qr3da8E", "reviewer": "Reviewer_2xjA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thanks to the authors for the hard work on this paper. \n\nStrengths\n-------------\n- Clear writing\n- Good experimental results\n- Big systems with a lot of thought-through designs\n\nWeaknesses\n----------------\n- Sec 3.4.1 (algorithm 1), you use gold answers for $a$ when the question is about facet $f$. However, during inference time, you won't know the ground truth answer $a$. As such, you have a mismatch between training and inference. Why not use the pipeline in Figure 1 during training as well? Or a mix of Figure 1 and Algorithm 1. By the way, it's awkward to have these two presented in different formats. Figure 1 and Algorithm 1 should be presented alongside one another in a comparable format.\n- Page 9: \"denoise hard-negative in the training set\" - can you please describe how this works and present some analysis? How important is this choice? Examples of what is left and what is removed?\n- Sec 5.1 - formula (6), you always condition on $p, q, cq, a$. Since the user answer is just \"yes\" or \"no\", doesn't it make sense to either (a) exclude $cq$ and $a$ when $a$ = \"no\" OR (b) actually include the numerical value of $p(a | q, int, cq) in the posterior somehow? After all, if the probability of \"yes\" is 0.5, that is very different from it being \"0.9\".\n- Sec 5.2 - re: maximum sequence length of 512. You are trying to cram a lot into a single instance with multiple back and forths etc. How often do you exceed 512? Is this an issue? Please include some analysis.\n- Sec 5.2 - \"we then apply our model as a second stage ranker\" - to clarify, is the \"model\" what's in Figure 1? Be specific here.\n- Page 14. Please reread this section and edit. Table 8 is missing a lot of values in the second column. Table 9 is not mentioned in the text.\n- I feel like a natural baseline is missing. Consider table 6. CLART5 outperforms MonoT5, but it has more information, so this is not surprising, nor is it a \"fair\" comparison given the information disparity. What we don't know is whether MonoT5 can use this extra information without further training. As such, can you just feed the inputs of CLART5 into MonoT5 and see how well it does?  The same can be done for models in table 10.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to augment IR datasets with simulated interactions. These are based on having access to known \"facets\", or extracting them, and then asking the user if the query is about one facet or another. The simulated users only say \"yes\" or \"no\" for multiple interactions. It is shown that extracting additional info from the users in this way raises retrieval scores.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "These are references to weaknesses above.\n\nCritical\n---------\n- Analysis of sequence length issues. If all of your samples are under 512, then no need.\n- MonoT5 but with inputs to CLART5 as baseline for Tables 6 and 10.\n- Careful editing of Page 14.\n\nNice to have\n----------------\n- Analysis of \"denoising hard-negatives\"\n- Additional experiments that try other versions of conditionality on $p, q, cq, a$, if my comment is convincing to you\n- Retraining with figure 1 instead of algorithm 1 to make training more in line with inference", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was no champion amongst the reviewers strongly in favor of either acceptance or rejection. The paper appears sound on a read-through, and the general sentiment across reviews was more weighed towards acceptance. Reading through the author replies, it seems to me that they have sufficiently addressed the main concerns of the sole dissenting voice amongst the reviewers, so that while there may be outstanding polemic issues which could be acknowledge in the camera ready, it is worth publishing. I recommend acceptance with minor revision. Should my recommendation be followed, I expect the authors to incorporate the feedback and the substance of their rebuttals and responses into their camera ready, and provide me with a list of changes made in order to obtain final approval.
Decision: Accept (poster)