An Analysis of Stopping Strategies in Conversational Search Systems

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-26 08:42:04

Abstract: Stopping strategies are a crucial aspect of conversational systems and user simulations, as they provide insight into when users end their interactions, which is vital for creating realistic simulations. While the Information Retrieval (IR) community has studied this topic extensively, little research has been done on stopping strategies in Conversational Search Systems (CSSs).

This is due to conversations' unique sequential and interactive nature, where traditional IR techniques struggle to accurately predict stopping points well and require new methods to be adapted from traditional IR techniques.

In this paper, we adapt Stopping Rules (SRs) from the IR community to the conversational setting, creating new SRs and identifying core features for each. 
We then analyze these features with several conversational datasets and aim to identify key features that predict stopping points in conversations between users and CSSs.

We found that models based on these features performed well in predicting stopping points and that textual statistical features, i.e., numbers of words, nouns, noun phrases and sentences users received from systems or outputted by users, always play a significant role in determining stopping points, with the number of outputted \textit{unique nouns} playing a particularly important role as an SR.

Our results provide a foundation for developing more realistic user models and simulators and guiding the design of more reliable evaluation measures for CSSs.

Main paper: 
 introduction: 
 In traditional search systems, queries are generally handled as independent entities, with the exception of ad hoc retrieval. This approach means that the response to a particular query is not influenced by the responses to previous queries. However, in Conversational Search Systems (CSSs), queries form sequences like conversations, making this independence assumption unrealistic. This dependency between queries makes the design of user simulators more challenging, as it is important to take into account features such as the recency effect, which refers to what happened in the last few turns near the end of a conversation, as it plays a crucial role in predicting user satisfaction [26] . Without realistic models of user-stopping behaviour, such an effect cannot be simulated, making the modelling of stopping behaviour essential to understanding user behaviour and developing user simulators. If these models don't incorporate realistic stopping points, it becomes challenging to simulate critical aspects like the recency effect, which significantly influences user satisfaction.In this paper, we focus on the stopping of conversations between users and a CSS. A CSS provides information to users via a conversational interface using natural language [27] . While the field of CSSs is well-established, with connections to Information Retrieval (IR) systems, chatbots, and dialogue systems, the evaluation of CSSs remains an open question [4, 17, 37] . Simulating a conversation is a promising approach for developing new evaluation methods, as it allows for low-cost and repeatable experimentation [5] . However, it is still a challenging task due to the complexity and uncertainty of real user interactions. One of the main challenges is determining the appropriate stopping point, as stopping too early can result in missing information, while stopping too late can lead to redundant or misleading information. As noted in Section 2 based on previous studies, the decision to stop often reflects the user's sentiment. In the IR community, several stopping Rrules (SRs) (or Stopping Strategies) have been proposed to depict stopping behaviours. SRs model users stopping behaviours according to given conditions. For example, one of the most classic SRs is Fixed Depth, which assumes users will stop their conversations once they are longer than a fixed length. These studies aim to quantify the feeling of "enough". For example, users would likely stop the conversation when they are frustrated as well as they are satisfied.Modelling and simulating stopping behaviour is of great importance in the field of CSS, and understanding it is crucial for evaluating search behaviour and performance [39, 40] . In this paper, we analyze SRs and features for each SR using four datasets detailed in Section 5: TopiOCQA, FAITHDIAL, TREC CAsT 2021 and EvalCran. For features based on the augmented four datasets, we run logistic regression to delve into weights and features for further explanation. The goal of this paper is to study SRs and their features in a conversational setting and predict the users' stopping points. We then analyze the weights of features on different datasets. We also identify interesting features by combining them separately and evaluate the performance of models based on these features. Our contributions can be summarized as follows:• An analysis of the performance of SRs on various datasets in a conversational setting. • An analysis of the features that contribute to predicting stopping points. • Identification of two distinct approaches to modeling stopping behaviors in conversations.Our results provide a foundation for developing more realistic user models and simulators, as well as guiding the design of more reliable evaluation measures for CSSs. 
related work 2.1 conversational search: 
 Conversational search is increasing in popularity due to the usefulness of searching on modern devices with small or no screens [27] . As defined by Anand et al. [4] , CSSs inherit properties from IR systems, chatbots and dialogue systems, such as access to information, statefulness, and interaction naturalness. Therefore, many studies in this field focus on user experience [3, 7] and performance [4, 24, 28, 49, 51] . Additionally, many studies also focus on the usage of information [2, 11] . Despite the many studies in this field, the evaluation of CSSs is still relatively underdeveloped [12, 30, 37] . This remains a key challenge for CSSs [4] . Traditional metrics, such as nDCG and MRR, and metrics from other fields, such as BLEU and ROUGE, are still popular in evaluating CSS in some studies [18, 30, 31, 43, 46] . However, recent studies have shown that these metrics may not accurately reflect the real user experience and preference [32, 38] .This study aims to contribute to the IR community by improving stopping prediction, which is essential for accurately simulating user behaviour.In IR, two popular methods for simulating users are cognitive and statistical approaches [23] . Cognitive approaches, which were first introduced by Belkin [6] , focus on characterizing users by their objectives, problems, and knowledge of the world. This approach has been further developed by many studies (see [22, 29, 35] ). On the other hand, statistical approaches analyze user behaviours and satisfaction [9, 13, 16, 19, 23] . These user predictive models are also commonly used as the basis for simulating users in IR.Early simulators in the field of conversational systems relied on statistical models [21, 48] . These simulators were built around a fixed corpus, and thus, had limitations when it came to the diversity of user intents [23] . To overcome these limitations, agendabased simulation became popular, as it allowed for more realistic responses and the ability to easily construct dialogue strategies [33, 36, 47] . Recently, there has been a trend towards using deep learning models for simulating users, such as training models with adversarial generative approaches [52] , reinforcement learning approaches [10] , and extracting abstract knowledge from data using inverse reinforcement learning [8] . Ideas from psychology communities, such as Priming Effects, are also considered as pathways to simulate user actions [25] .In this study, the primary focus is on the stopping of conversation, a crucial aspect that has not been adapted to recent trends. Various SRs will be analyzed and modified to enhance the prediction of conversation termination.Determining when to stop a conversation is a complex task as it is influenced by various features, including the user's cognitive state and decision-making processes [39] . Despite the difficulties of modelling stopping behaviours, several SRs have been proposed in the IR community to explain when users stop. Cooper [15] presented two rules:• the frustration point rule, which states that users will stop after receiving a certain number of non-relevant replies; and • the satisfaction stopping rule, which states that users will stop only after receiving a certain number of relevant replies.Kraft and Lee [34] introduced two additional rules:• the expected search length rule, which is based on the number of results read by users; and • the combination rule, which combines the rules of frustration and satisfaction, stating that users will stop if they are satisfied or disgusted by receiving too many non-relevant replies.Nickles [42] then proposed four cognitive rules:• the mental list rule, which assumes that users have a list of criteria that must be satisfied before stopping; • the representational stability rule, where a user continues to query until the underlying mental model begins to stabilize; • the difference threshold rule, a user sets an a priori difference level to gauge when there are no new replies, and; • the magnitude threshold rule, a user has to attain enough information before stopping.In this study, we aim to develop SRs that build upon those established in previous studies. In addition to the pre-existing SRs, we are introducing a range of indicators derived from lexical context. These indicators are significant as they have the potential to mirror user preferences within a conversation. [50]  
research questions: 
 In this paper, we aim to answer the following research questions:RQ1. How do different SRs perform on different datasets?To answer this question, we will evaluate the performance of various SRs using appropriate measures. This is crucial for understanding the reliability of these rules and the features associated with them in predicting stopping points.RQ2. What is the performance of models built on selected features? To build and evaluate models, we will select relevant features from SRes and augment datasets from completed conversations. We will also test the performance of models trained on one dataset on other datasets to demonstrate the robustness of these features and models.RQ3. How do the features influence the prediction of stopping? By analyzing different features and datasets, we will investigate if there are any unique or particularly important features that contribute to the stopping behaviours. This will provide insights into other aspects of stopping that can be further explored. 
stopping rules: 
 In this section, we propose several SRs. The first SR, which we also consider as a baseline, is the Fixed Depth.SR1 Fixed Depth Users will stop once the conversation is longer than a given length 𝑙, where 𝑙 is a feature.This SR is a basic approach in many simulations and its performance depends on the specific scenario [40] . For example, this SR will perform well in simulating conversations with a predefined number of turns. However, it may not be suitable for other settings.We also propose SR2-SR7, which are based on rules of frustration and satisfaction from previous studies.SR2 Similar Hits Users will stop once they get a response similar to a previous response. The feature of this SR is the number of similar (more than 0.5, TF-IDF) responses 𝑠. SR3 Continually Similar Users will stop once they get a series of similar responses. The feature of this SR is the number of series of similar (more than 0.5, TF-IDF) responses 𝑠 𝑐 .SR2 and SR3 focus on similar system's responses, which are based upon the difference threshold rule [42] . For these two SRs, the similarity between the current response and the previous response is calculated via TF-IDF. Getting similar responses means getting less new information, thereby frustrating the users and resulting in stopping. SR4-SR7 take another approach to judge frustration and satisfaction, which is based on the relevance of the responses. We introduce these SRs because the relevance of responses from the CSS has been proved that plays a significant role in satisfaction [26] . However, these SR and factors will only be applied in datasets where relevance labels are available.SR4 Total Non-relevant Users will stop once they receive more than a number of non-relevant responses. The feature of this SR is the number of non-relevant responses 𝑖. SR5 Continually Non-relevant Users will stop once they observe a series of non-relevant responses. The feature of this SR is the number of such series 𝑖𝑖. SR6 Total Relevant Users will stop once they receive more than a number of relevant responses. The feature of this SR is the number of relevant responses 𝑟 . SR7 Relevant-rate Users will stop once the relevant rate of replies is lower than a given number. The feature of this SR is the relevant-rate 𝑃 (𝑟𝑒𝑙).In SR4 and SR5, we focus on the number of non-relevant responses and assume that frustration accumulates with each non-relevant response received, leading to a stopping point [15] . Similarly, in SR6, we posit that users will feel satisfied when they receive a sufficient number of relevant responses. Lastly, with SR7, we consider the overall quality of the conversation, assuming that users will stop if they believe the quality is lower than their expectation.We then design SR8-SR10 to analyze conversations from a statistical perspective. These SRs are based on statistics calculated from the conversation logs. We assume that the number of words used by the replies reflects the effort users put into the conversation. We assume the same for the number of sentences, which may reflect the number of information clusters in the conversation. Note that when considering nouns (SR8.2, SR9.2 and SR10.2), we count the number of unique nouns/noun phrases as a measure of the amount of new information conveyed in the conversation as Clark and Sengul [14] point out the strong relationship between understanding new information and identifying new nouns/noun phrases. We used TextBlob and NLTK packages to locate nouns/noun phrases in this paper.In this study, we will utilize all of the SRs defined in this section to identify stopping points. Initially, each rule will be analyzed individually. Subsequently, indicators derived from these SRs will be employed as features in the development of models aimed at predicting stopping points. 
data: 
 In this study, we analyze four datasets to understand the stopping behaviours of users in conversational search systems. We choose four basic datasets in order to introduce more settings. For example, we use two datasets that include replies in free-form (TopiOCQA and FAITHDIAL), and two datasets that include replies with paragraphs. On average, the FAITHDIAL dataset has the shortest conversation depth, with an average of 4.5 turns, while the TopiOCQA dataset has the longest conversation depth, with an average of 13 turns. Replies from FAITHDIAL and TopiOCQA are written by the system as free-form answers.TopiOCQA (Topi). TopiOCQA is a conversational dataset that draws its content from Wikipedia [1] and is characterized by its focus on the complexities of topic switching. Each conversation turn includes a marker indicating the topic being discussed. Comprising of 3,920 interactions, the dataset includes free-form answers. Most of them are within one sentence. These conversations have a conversational depth of 13 turns on average, with a standard deviation of 3.3 and four distinct topics.. FAITHDIAL is a conversational dataset based on Wikipedia [20] that is designed to study the phenomenon of information seekers and bots misunderstanding each other. It includes 5,649 conversations, comprising 50,761 queries and responses. The bot typically responds to the user with a few sentences per turn. The average length of conversations in this dataset is 4.5, with a standard deviation of 0.5.The TREC Conversational Assistant Track (CAsT) 2021 dataset1 offers a collection of open-domain, information-centric conversational dialogues featuring 26 conversations between users and systems. TREC CAsT 2021 emphasizes the importance of considering the context of the dialogue and retrieving relevant information. The dataset includes information from sources such as the Washington Post2 (2012-2020), KILT Wikipedia (August 1st, 2019) [44] , and MS MACRO [41] . The system provides a passage in response to the user's question in each turn. The average conversation length in this dataset is 9.2, with a standard deviation of 1.7.EvalCran (Cran). This dataset, collected by Lipani et al. [37] , consists of 131 conversations based on the SQuAD dataset [45] . It features a user-system interaction where the user poses a query to the system, and the system responds with a paragraph. Users then evaluate the relevance of the paragraph to the query.In Table 1 , we show the properties of each dataset. In order to make the most of the features present in each dataset, we group them to create new datasets. For example, we merge TopiOCQA and FAITHDIAL to augment diversity and also merge EvalCran and TREC CAsT 2021. We merged TopiOCQA, and FAITHDIAL since both of them feature free-form as the response style, as well as TREC CAsT 2021 and EvalCran feature raw passages as the response. Another reason is the size of datasets, where TopiOCQA is similar to FAITHDIAL while TREC is similar to EvalCran. Additionally, we combine all four datasets into one to take advantage of all available data. In Table 1 , we also show the statistics of the merged datasets.To train the Logistic Regression (LR) models, we augmented the original datasets by producing unfinished conversations, i.e., negative samples. This is done by randomly cutting the conversations at different points. For each original conversation, there is one corresponding unfinished version for a balanced distribution. In the final dataset, 50% of the conversations are unfinished. Finally, we divide each dataset randomly into 7:3 train and test sets.The process of generating datasets in this study is 1) Merging datasets (e.g., TREC CAsT 2021 and EvalCran) into one dataset if needed; 2) Shuffling the dataset; 3) The dataset is split into a train set and a test set with a 7:3 ratio, and; 4) Augmenting each set by generating unfinished versions.To summarize, we have four original datasets: TopiOCQA, FAITH-DIAL, EvalCran, and TREC CAsT 2021. By grouping them, we create three mixed datasets: Topi+FD, TREC+Cran and ALL. Each dataset is then further split into train and test sets and augmented by unfinished conversations. 
experiments and findings: 
 In this study, we approach the task of understanding user behaviour in conversations as a binary classification problem. At each turn, we aim to predict whether the user will continue or stop the conversation. To make this prediction, we utilize as features the previously defined SRs. We employ Logistic Regression (LR), a method wellsuited for binary classification tasks, due to its efficiency and ability to illustrate the impact of individual features.To answer the research question RQ1, which asks about the performance of different SRs on different datasets, we study the performance of each SR by taking each one of them individually and use it to make predictions about whether users stop or not.To address RQ2 about the predictability of users' stopping behaviour combining all features and RQ3 about understanding the importance of each feature, we instead combine all the parameters and train an LR model and utilize the LR weights to analyse the impact of each SR on the prediction. The greater the weight, the greater the influence of the stopping rule. Positive weights indicate a tendency towards stopping, while negative weights suggest a tendency towards continuing.To evaluate the performance of the trained models, we use the metric Precision, Recall, F1-score and AUC (Area Under the Precision-Recall Curve). We use the F1-score as the main measure, and the results for the other measures are attached in the Appendix. In Figure 1 , we show the process of training LR models. From the raw conversations, we calculate the features. After a Min-Max normalization, these features are used to train the LR models. As concluded Section 5, the datasets will be augmented with generated unfinished conversations and keep 50% of conversations unfinished. For each conversation, each feature will be calculated.Because TREC CAsT 2021 and EvalCran are the only datasets with relevant labels on systems' replies, we trained two LR models for them. One with features related to relevance and one without.To answer RQ1, we use single features from SRs to train LR models. Table 2 shows the features of each SR we adjust to adapt different datasets.The F1-scores of each model on the corresponding test set are shown in Table 3 . The results indicate that the performance of SR varies depending on the dataset used. For instance, using the number of total sentences sent by users can yield high performance on TREC and Topi datasets, while on other datasets, the F1-score is lower. This highlights the limitations of traditional SRs, as the most classic rule, the fixed depth 𝑙, only achieves good performance on two datasets. The rules based on similarity are only effective on TREC and Cran, which may be due to the fact that replies on Topi and FD are generated by conversational systems rather than other humans. The rules based on relevance are also limited by the dataset, where only Cran and TREC have users' replies labelled. Overall, only one SR achieved good performance on each dataset. The feature 𝑛 𝑛𝑡𝑠 , the number of nouns sent by users, achieved scores higher than 0.7 on all datasets, with the exception of a score of 0.645 on Topi.To address RQ2, we trained LR models using all features on all seven datasets discussed in Section 5. We then applied all of the trained models on each dataset to evaluate their capacity and identify any biases resulting from training on a single dataset. After training the models, we analyzed the weights of each feature to gain insight into the models. We will begin by examining the weights compared with other features in all eight models to establish a general ranking of features for each dataset.In Table 4 , we display the scores of the LR models on the test sets. For the EvalCran and TREC CaST 2021 datasets, two models were trained: one without relevance features (Cran, TREC) and one with relevance labels (Cran + , TREC + ). The first observation is that all models achieved high scores on the corresponding test sets, indicating that combining all the features listed in Section 4 allows the models to effectively extract relevant information from the context to predict stopping points. Secondly, the scores of Cran are lower than those of Cran + as well as TREC vs TREC + , suggesting that features related to relevance play an important role in predicting stopping points and cannot be replaced by other features listed in Section 4.In Table 5 , we present the F1-scores of models that were trained and tested on different datasets. The columns indicate the source of the test sets, while the rows indicate the source of the training sets.Here we discover that four models obtained satisfactory F1-scores (higher than 0.7) in the majority of the test sets. This suggests that the weight distributions of these four models (All, Topi+FD, FD, and Cran) effectively predict stopping points in diverse settings. In Table 6 , we show the weights of models in Table 4 . For each column, the top 5 biggest weights in absolute value are marked in bold, and their orders are also labelled. One finding is that there are at least two distribution patterns of weights in the four highdiverse models. First is ALL and Topi+FD, where features related to the total received from the system and the total outputted by users play a leading role. Second is FD and Cran, where depth is the most significant contributor. Especially in FD, the weight of depth is far larger than the other features. While in ALL and Topi+FD, the depth is a minor feature. Another finding is that in all four original datasets' models, depth always plays the most important role. This means that the high value of the depth to predict stopping points works well only in specific contexts or that these datasets lack variety in the sampled conversations.Based on previous experiments, we can answer RQ3 by analyzing features with unique behaviours. One interesting property of such features is when they assume a negative weight, i.e., they promote the conversation to continue. Another potential direction is the changes in features' weights through the change of the dataset size. This is because when the dataset is small, some features may be important just because the LR models are overfitting on them. For example, predicting the stopping points in FD is relatively easy since most of the original conversations stop at turn 4 or 5. Hence depth will play a paramount role. By studying how feature importance changes when increasing the dataset size, which is proportional to the dataset diversity, we show which features tend to become more prominent. Our hypothesis is that when more data is added, features with better generalizability will rise.Before looking at what happens to the feature weights while changing the proportion of data used, we first look at how the performance of the LR models changes. In Table 7 , we show that there is not a significant change in the scores. Therefore, all models trained with a different proportion of data are comparable. In Figure 2 , we show how the features' weights change when changing the proportion of data used. We found that only three features change significantly when the size of the dataset increases, while other features do not change even after 70% of all data is used. We observe that the features that promote the conversation to continue are: the total number of received unique nouns, the number of unique nouns received at the last turn, the number of sentences received at the last turn and the number of similar replies. However, we notice that the magnitude of the features' weights increases with the increase of the data. For a LR model, this means that the separation of the space between stopping and continuing the conversation is becoming sharper and sharper. This means that without an appropriate normalization, we cannot tell if the features are becoming more important with the dataset size or because their overall magnitude is changing.To avoid this effect, we also look at how the features' weights change after having them normalized. To do that, we first take the absolute value of each feature, then divide them by the L2-norm of the weights and bias vector. The result of this normalization is shown in Figure 3 . After performing this normalization, we observe that the features that become more important are the total number of received sentences, words and unique nouns. The models tend to focus on three specific features when there is a larger sample of data. One notable finding is that the weight of the total number of unique nouns received has a negative and significant increase. Along with the increase in the number of total received sentences, we can deduce that users' stopping behaviour is depicted as a process of receiving information, with each sentence being treated as a unit of information. As users receive more information, they become more inclined to stop. Unique nouns represent novel information, and a larger amount of novel information prolongs the search process and ultimately delays the stopping point.In Figure 4 , we show the Kendall's Tau correlation coefficient of the weights between each dataset proportion vs the full dataset, as shown in Figure 3 . This shows that to achieve a Kendall's Tau of 0.85 we need around 50% of the dataset. That indicates that to get a similar distribution of weights as the full dataset, we need around 4,000 conversations. 
discussion and conclusion: 
 In this paper, we analyzed several SR and corresponding features. We first adapted SR to fit the conversational setting. Then we analyzed the performance of SR on four independent datasets. After SR, we further explored features from each SR. We run logistic regressions on different combinations of 4 datasets. Each model from this step is also tested in other combinations of datasets. We then analyzed the weights of features in each model and ran further LR based on different features. Finally, we revealed the special features by changing the amount of input data.Our results show:(1) While the baseline (depth, 𝑙) can achieve good performance, there are more suitable SRs for high-diverse scenarios. one focuses on features based on text statistics, and another focuses on depth. ( 7) Special features where weights will change through increasing diversity, which may also represent some properties of received information.Result 1 addresses RQ1, Result 2 addresses RQ2, and Results 3-7 collectively address RQ3.Our research reveals two distinct ways of depicting stopping behaviours. The first approach views stopping as the conclusion of an information-receiving process, where users reach their information limit and feel "enough. " Fresh information prolongs this limit, and textual statistical features such as the number of total sentences and unique nouns in received documents provide a reliable representation. This depiction is particularly effective in high-diverse settings. The second approach views stopping as a fixed-length, anticipated journey, where the depth of the conversation is the most important factor. This approach is suitable for single-regular settings.Our study presents substantial promise in augmenting user simulation and related functions. Specifically, this research can be assimilated into dialogue management systems, offering improved guidance of interactions. Furthermore, the implementation of our findings in agents may facilitate the extension of conversations through more adept selection of ensuing responses. Moreover, simulated users that incorporate our research can achieve more appropriate termination of conversations.In this study, we identified four limitations from which future works may benefit. First, more SR and features should be considered. For example, the time spent in the conversation is a potential feature for analyzing stopping points. However, in this paper, due to datasets' limitations, these features can not be explored. Second, the datasets are unbalanced. For example, TopiOCQA has 3,920 conversations, while EvalCran has only 131. This may result in a deficient analysis of SR and features in settings related to such datasets. Third, TF-IDF is an effective algorithm in calculating similarities between documents, but in some datasets, the system did not reply with original documents. For example, in FAITHDIAL, candidate documents will be rewritten. In this case, the results of TF-IDF may not be reliable. Finally, Large Language Models like 
 
 Appendix: https://www.treccast.ai/https://trec.nist.gov/data/wapost/ 


Decision: The work effectively utilizes the features proposed in other theory-driven works in a slightly different setting. The reviewers appreciate the theory in extending and modifying stopping behavior, especially with respect to the statistical and linguistic features of the conversation. 

There are several suggestions on how the paper can be improved. In particular, I would like to point to the potential of further discussion of some features and the generalizability of the work. Moreover, some discussions can be added on the implications of these stopping criteria on the future generations of conversational search systems would be appreciated. As mentioned by 3LJK, some of these systems have already expired and some of the assumptions on older models do not necessarily hold on the next generations. Therefore, the authors could improve their paper by having a discussion on these aspects.

Reviews: review: main_review: main_review: I think that the results provided here will be of some interest to ICTIR attendees, and to the conversations search system community at large, especially with respect to developing more accurate and realistic simulations of conversational search. This could also be important for developing measures for design and evaluation of conversational search systems.
main_review: main_review: Overall, this is a nice paper that does an insightful evaluation of stopping criteria that can be useful for later work on user simulations in conversational systems. The writing is easy to follow (although it might be a little bit more condensed). The tables and figures are mostly easy to follow as well. 

I do miss the theoretical aspects in this work. The paper reads like a typical analytical/experimentation paper that evaluates some features and their performance on different data sets. While this is absolutely valid and insightful to a specific degree, I would love to see a theoretical discussion on some new aspects, such as human factors, cognitive aspects, etc., of stopping criteria that would add something new to the discussion. For example, it was mentioned that relevance information can not be exchanged with other features. That sounds like a very interesting outcome, but it is not discussed (and many other aspects are not discussed either) in the paper. 

I miss the depth, meta-insights, or new ideas/thoughts a good ICTIR should contain. I am therefore against accepting the paper for ICITIR. Its core part is solid but limited in the insights provided, and the theoretical dimension is missing at all.
main_review: main_review: - **Readability and Understandability**: The paper is well-structured and detailed in its approach to adapting stopping rules from traditional information retrieval systems to conversational search systems. The authors provide a clear introduction to the problem, a comprehensive review of related work, and a detailed explanation of their methodology and experiments. Overall, the paper is well-written!

- **Relevance of the Problem**: The problem addressed by the paper is highly relevant. As conversational systems become increasingly prevalent, understanding when users choose to stop interacting with these systems is crucial for improving user experience and system efficiency. The focus on stopping strategies in conversational search systems fills a significant gap in the existing research, making it a timely and important study.

- **Originality of Methods**: The paper's approach to adapting and analyzing stopping rules in the context of conversational search systems is original. The adaptation of traditional stopping rules to a conversational setting and the introduction of new rules based on conversational features demonstrate innovative thinking. Moreover, the use of multiple datasets to validate the effectiveness of these rules adds depth to the research.

- **Interest of Results**: The results are interesting and provide valuable insights into the factors that influence stopping behavior in conversational search systems. The identification of key textual statistical features and the impact of unique nouns on stopping points are particularly noteworthy findings that contribute to our understanding of user behavior in conversational contexts.

- **Interest to the ICTIR Community**: This work is likely to be of significant interest to the ICTIR community. It addresses an important aspect of conversational search systems, an area of growing importance in information retrieval. The findings on stopping strategies have implications for the design of more effective conversational agents and can influence future research on user interaction with AI-driven systems. The methodological innovations and comprehensive analysis make this paper a valuable contribution to the field.
main_review: main_review: This is an interesting topic, and it's nice to see tests of the ideas with conversational corpora.
One problem with this flavour of work in general is that we're measuring current systems -- possibly even out of date, already -- instead of whatever's next. It would be nice to read some thoughts on whether we can use (or build) a theory to generalise to tomorrow's systems. For example, it might be that disfluencies are a problem, so a longer conversation means people are more likely to give up. If we fix the disfluencies, that correlation would go away. Are the criteria listed here generalisable? Are the models? (Certainly they seem very different from one corpus to the next.) 
Some smaller thoughts:
* abstract: does conversation really have a "unique sequential and interactive nature"? So does a series of e.g. conventional web searches, with clicks and other interactions.
* s1, para 1: "with the exception of ad-hoc retrieval": I think you mean to say the opposite, ad-hoc retrieval is more or less defined by handling queries independently.
* s1 and throughout: is "query" the best term in a CSS? Perhaps "utterance"?
* s1, para 2: "One of the main challenges...": this is a challenge for the searcher, not the system, unless the system aborts early.
 * s3, RQ3: "contribute to" should probably be "correlate with" unless we're making an explicitly causal claim.
* s6.2, para 5: "satisfactory F1 scores": how is "satisfactory" determined?
* s6 throughout: different names are used for the same data: e.g. "TF" amd "Topi+FD" in Tables 4 and 5.
* Table 5 is an interesting idea.