Reduce Human Labor On Evaluating Conversational Information Retrieval System: A Human-Machine Collaboration Approach
Authors: Chen Huang
Peixin Qin
Wenqiang Lei
Jiancheng Lv
Venue: EMNLP 2023 Main (2023)
Keywords: Interactive Evaluation
Human-Machine Collaboration
Conversational Information Retrieval

Abstract: Evaluating conversational information retrieval (CIR) systems is a challenging task that requires a significant amount of human labor for annotation. It is imperative to invest significant effort into researching more labor-effective methods for evaluating CIR systems. To touch upon this challenge, we take the first step to involve active testing in CIR evaluation and propose a novel method, called HomCoE. It strategically selects a few data for human annotation, then calibrates the evaluation results to eliminate evaluation biases. As such, it makes an accurate evaluation of the CIR system at low human labor. We experimentally reveal that it consumes less than 1\% of human labor and achieves a consistency rate of 95\%-99\% with human evaluation results. This emphasizes the superiority of our method over other baselines.

Reduce Human Labor On Evaluating Conversational Information
Retrieval System: A Human-Machine Collaboration Approach
Chen Huang, Peixin Qin, Wenqiang Lei! Jiancheng Lv
College of Computer Science, Sichuan University
[huangc.scu, qinpeixin.scu, wengianglei) @ @gmail.com
Abstract
Li al., 2022). For example, Braun et al. (2017); Choi et (2018) require the human to answer the questions in a pre-collected question set and use these answers as ground truth to evaluate the model- generated results. Li et al. (2022) requires the hu- man to validate and re-annotate the pre-collected conversation dataset to provide finer-grained labels for the input and characterize the system perfor- mance based on such labels. Moreover, Alianne- jadi et al. (2020) employs the human directly rate each response of the system on relevance and naturalness. While these methods show their ef- fectiveness in CIR evaluation, they still require traversing the entire dataset, resulting in signifi- cant human labor costs (Budzianowski et al., 2018; Dalton et al., 2020; Adlakha et al., 2022). Conse- quently, it is imperative to invest significant effort into researching more labor-effective methods for evaluating CIR systems.
Evaluating conversational information retrieval (CIR) systems is a challenging task that re- quires a significant amount of human labor for annotation. It is imperative to invest signifi- cant effort into researching more labor-effective methods for evaluating CIR systems. To touch upon this challenge, we take the first step to involve active testing in CTR evaluation and propose novel method called HumCoE. It strategically selects a few data for human an- notation and then calibrates the evaluation re- sults to eliminate evaluation biases. As such, it makes an accurate evaluation of the CIR system low human labor. We experimentally reveal that it consumes less than 1% of human labor and achieves a consistency rate of 95%-99% with human evaluation results. This empha- sizes the superiority of our method.
1 Introduction
To this end, we take the first step to involve active testing' for the labcl-effcctive CIR eval- uation, proposing novel method calledHumaa-- machine Collaborative Evaluation (HummCoE, for short). The idea is strategically select few data for the human to annotate the system outputs (c.g.. provide an answer to the question), and then cali- brate the evaluation results to eliminate evaluation biases. Specifically, given an affordable number of human interactions, HumCoE starts by select- ing subset of representative data and requesting human annotation scoring. In this process, the representative data measured based on the prob- ability inconsistency between CIR's prediction and a pscudo-label that is generated by a surrogate machine. After that, HumCoE assigns an impor- tance weight to each selected data and calibrates the evaluation results. As such, the sampling bias is eliminated, and hence the evaluation accuracy is
As evidenced by various studies (Zhang et al., 2018; Aliannejadi et al., 2021), the conversational infor- mation retrieval (CIR) system has shown its effec- tiveness in domains such as conversational question answering (Reddy et al., 2019) and conversational recommendation (Lei et al., 2020b). However, ac- curate evaluation of CIR systems continues to be rapidly evolving research area due the multi-turn nature user-system conversations (Zamani et al., 2022; Sckulić et al., 2022; Gao et al., 2021). It of- ten requires the user to constantly interact with the CIR system and then evaluate the system based on its real-time responses, which are labor-expensive and generally only feasible in an industry lab (Lab- hishetty and Zhai, 2021).
To alleviate this problem, current practices pre- pare a set of input data and employ humans to either manually annotate the answer to be compared with the system-generated results or directly score the corresponding system output (Larson et al., 2019;
It estimates model performance (i.c., CIR system, in our case) by selecting a small subset of data for human annotation (i.c., evaluation) Wvaluation/Kahman (aahman et al., 2020; Nguyen et al., 2018).
Correspondence to Wenqiang Lei.

---

largely improved.
ing, we offer literature reviews on CIR evaluation and active testing. For more discussion on human- machine evaluation, please refer to Appendix E.
We demonstrate the effectiveness of HumCoB on typical CIR tasks, including asking clarifying ques- tions and the conversational question answering. In our experiments, we assess the task success rate and quality of system responses using automatic evaluation metrics on pre-collected conversational datasets. Additionally, we employ the human to as- sess the relevance and naturalness scores of system responses. Our experiments indicate that HumCoE produces results that are extremely comparable to those of a numann-baser evaluation yet require very little labor. It consumes less than 1% of human labor and achieves a consistency rate of 95%-99% with human evaluation results on all tasks. Com- pared to other classical active testing methods, our approach achieves higher evaluation consistency rate by an average of 11.75%. We also explore our method in the context of ChatGPT. We notice that ChatGPT does not provide satisfactory evalu- ation consistency with human evaluation, with an average consistency rate of 88.24%. But, after in- troducing a small number of labor costs (0.68%) HumCoE improves the evaluation consistency by 9.59%. This highlights the importance of human- machine collaboration for efficient CIR evaluation. Evaluating any highly interactive process is chal- lenging. In this paper, we experimentally reveal our effectiveness and efficiency on the CIR evalua- tion task. We believe that our work could provide a landmark for future human-machine-bee evaluation research. To sum up, we claim the fol- lowing contributions.
CIR Evaluation. Given an evaluation met- ric, existing methods use user utterances and the system's responses to evaluate the CIR system. Such information is either obtained from real-time user-system conversations during online evalua- tion or contained in a pre-collected conversational dataset during offline evaluation (Zamani et al., 2022). Online evaluation (Zamani al., 2022; Aliannejadi al., 2019; Li al., 2019) employs real users to interact with the CIR system several times and dynamically obtains user utterances and the system's 's responses for evaluation (Ram et al., 2018; Park et al., 2020). However, a large number of interactions require a huge labor cost, which limits the feasibility and application scenarios of this method. A Alternatively, the offline evaluation method is economically inexpensive as it permits evaluations via static conversational datasets (Yang et al., 2018; Budzianowski et al., 2018; Hender- son et al., 2019). During the evaluation, the CIR system only receives rigid and fixed questions or responses, extracted from the static conversational datasets. However, let alone that such methods do not transfer well to the real-world scenario and of- ten lead unnatural conversations (Lipani et al., 2021; Sekulić al., 2022), constructing the dataset still requires a significant amount of human labor (Budzianowski et al., 2018; Dalton et al., 2020; Adlakha et al., 2022), especially when evaluating specific features of the system that are absent from its training data (Braun et al., 2017; Larson et al., 2019; Li et al., 2022). As a result, how to design labor-effectivg methods for CIR evaluation is still an unaddressed problem. In this paper, we propose an evaluation method based on buman-machine col- laboration. We experimentally show that it makes an accurate evaluation that extremely compara- ble to those of the human-based evaluation with low human labor.
CIR
We call attention to the challenge of designing labor-effective methods for accurately evaluating CIR systems.
For the first time, we introduce active testing to evaluate the CIR system and propose a novel method called HumCoE. It strategically selects representative data for human annotation and carefully calibrates the evaluation results.
We verify our effectiveness with empirical stud- ies, which demonstrate that our method enables accurate evaluation with minimal human effort. We set a landmark for future human-maching based CIR evaluation research.
Active Testing. Different from active learning, which targets selecting partial data to train a better model, active testing focuses on reliably estimating model performance by selecting less unlabeled test- ing data for human annotation (Nguyen et al., 2018: Li et al., 2020). By allocating data between humans and machines, active testing methods achieve col- laboration between humans and machines. Thus, data selection plays an important role in active test-
2 Related Works
As our focus is on utilizing both human and ma- chine in evaluating the CIR system via active test-

---

ing. Usually, hard-to-predict data with large model prediction uncertainties (Rahman al., 2020) or inconsistencies in model predictions (Nguyen et al., 2018) are always preferred. More broadly, Zhang et al. (2021a) has shown its evaluation effectiveness on the task of malevolence identification. It assigns dialogues with high model uncertainty to the hu- man while assigning the rest to the machine for evaluation. Inspired by the idea of active testing, we take the first step apply it to CIR evaluation by selecting partial hard data for human evaluation and carefully calibrating the results. According to our experiments, although human evaluation costs are reduced, existing methods (Zhang et al., 202 Rahman et al., 2020; Nguyen et al., 2018) still suf- fer low evaluation accuracy because the machine may be error-prone and the evaluation results are uncalibrated. This emphasizes the importance of evaluation calibration.
4 Methodology
There are two steps HumCoE: 1) It starts by selecting a subset of representative data D, and re- questing human annotation or scoring YD,, where the representative data D. are selected based on a surrogate machine. 2) After that, it assigns im- portance weight to cach selected data and calibrates the evaluation results.
Step 1: Surrogate Machine based Data Sam- pling. We aim to select representative data for human annotation. In this process, the represen- tative data is measured based on the probability of inconsistency between CIR's prediction and a pseudo-label that is generated by a surrogate ma- chine.
Before human annotation, we involve a sur- rogate machine g as a user proxy generate pseudo-labels. Taking the conversational question- answering task as an example, the surrogate ma- chine is required to answer the input questions. The output answers are stored as pseudo-labels. Formally, given the surrogate muchine-annotated dataset D1, we denote d; = {.a, 14) E D1 and i/ is list of proxy responses. Usually, surro- gate machine g is trained fine-tuned based on the same training data as f. Meanwhile, designing g with different architecture from f is always preferred to encourage response diversity between f and For example, one could set GPT-2 as f and BART as g to generate responses during the conversations.
3 Task Definition
Given an evaluation metric EM and a test conver- sational dataset D, the evaluation r is completed by calculating the metric value on the given dataset, described by the following equations.
annotated
EM, = EdGEDWErD EM 91) |D]
(1)
Here, d: { 3 represents the i-th user- CIR interaction that consists of multiple conver- sational turns. Specifically, I2 - -xx x2, .,...., at) is a list of responses of CIR f, where x/ is j-th CIR response in i-th user-CIR interaction. Similarly, i; (u,, u² .... ut²) is a list of user responses, and yt is list of user- annotated ground truth CIR re- sponses. Take the conversational question answer as an example (Peng et al., 2022), af refers to the system returned answer j-th user query ud in i-th user-CIR interaction. In this case, E. could refer the Rouge-L evaluate the word overlap be- tween the CIR response xi and ground truth y i.e., (M mn) = 1/k Σ, Rouge-L(x(, v²). EM could also refer to the Accuracy to evaluate the task success rate of CIR retrieving the correct answer, i..., EM 91) 1/k E, [(
Borrowing the idea of active testing methods (Zhang et al., 2021 a), we aim to select subset of representative data D, that are hard for the CIR system to properly handle for human annotation YD, The "hardness" in this paper is measured by the probability of the CIR system making mistakes on pseudo-labels YD,, rather than the model uncer- tainty (Zhang et al., 2021a). Namely, we prefer data with a high probability of inconsistency between CIR's 's predictions and pseudo labels. Technically, measure the hardness as follows.
((dc) 1 EM (d, 9:).
(2)
n/), where I(.) is an indicator. We denote 7 TD, EM,YD) as human evaluation, where the size of D is usually large to guarantee the statistical significance of the evaluation results.
Large (d:) means that the performance of CIR f on d, in terms of metric EM may not be satisfy- ing. Finally, given the maximum number of human interactions T. we sample subset D, with the size D.I = T according to probability g. The selected data are assigned to the human for evaluation and obtaining the ground truth labels.
Usually, It is a pre-collected dataset annotated hy humans. Ours are created automatically (cf. section 4).

---

Step 2: Evaluation Calibration. Using hard samples to evaluate CIR system inevitably leads to biased evaluation results. It tends to underes- timate the effectiveness of the system. To this end, HumCoF. assigns a weight to each sample and corrects its importance in the evaluation re- sults, thereby moderating the negative impact of hard samples the evaluation results. Formally, we follow previous work active learning (Far- quhar et al.. 2021) and design the weight uy of sample d; as follows.
MCM (Nguyen et al., 2018) selects hard samples whose CIR prediction confidences are very high, and whose CIR prediction results are inconsis- tent with those of the surrogate machine. For example, if the CIR predicts positive label with high probability while the surrogate machine as- signs negative pseudo label, it suggests that the surrogate machine's label may be incorrect and needs human annotation for correction.
These methods select hard samples for human evaluation like us but adopt different sampling strategies. Note that take the first step utilize active testing evaluate CIR. In our experiments, we re-implement other active testing methods to fit in the CIR scenario.
N-T T 1 - (g((c.) -))
(3)
where N is the size of the proxy-CIR conversation dataset Dr. In this case, we assign smaller weights to data with large (d.) and penalize the importance hard data. Different from existing work (Zhang et al., 202 we finally calibrate and yield the evaluation result as follows.
Implementation Details. Considering evaluat- ing any highly interactive process is challenging, follow previous works (Hwa, 2000; Kristjans- son al., 2004) and assume the time spent by the human evaluations on each sample de is constant Therefore, we use the number of human evalua- tions to estimate the human labor, i.e. the size of Dz. In our experiments, we limit the size of D, in (5, 10, 15, 20, 25, 30) to verify the effectiveness of different evaluation methods with minimal labor costs. Considering the randomness of data sam- pling, run the evaluation 100 times and report the mean estimation results F.
f(Do, E., Yo.) EdEED.IVVD Ed, vE)
(4)
[D_]
5 Experiments
Our method formulates a human-machine collab- oration to reduce the labor of human evaluation while retaining its evaluation accuracy. Therefore, the main experiments in section 5.1 are designed to assess if HumCoE and active testing based eval- uation methods could approximate the results of very few cost human
T Es,eSoed F(D,, EM,YD,,,..... (s Seed
(5)
the human evaluation labor. Finally, we further perform ablation studies on HumCoE in section 5.2 analyze its properties. Baselines. We utilize human evaluation to eval- uate a given CIR system with no limitations on hu- man labor costs. We compare our method to other active testing methods that are most relevant to us, including HMCEval (Zhang et al., 2021a), Active Top-K (Rahman al., 2020), and Most-Confident Mistake (MCM) (Nguyen et al., 2018).
Each time, the random seed is fixed and shared with different evaluation method. For implementation details, we refer readers to Appendix A.
Evaluation Metrics. We assess if HumCoE could approximate the results of the human eval- uation at very few cost human interactions. Therefore, we consider the following metrics.
Evaluation Consistency. We calculate the abso- lute value of the inconsistency between human evaluation r(D, F.M, YD) and a given evalua- tion method To. i.c., Ar |TD,, EM, YD) - 7 The lower the inconsistency. the greater the eval- uation accuracy of the given evaluation method. We further report the consistency ratio as 7 = 1 Ar
HMCEval (Zhang et al., 2021a). It is intended for assessing malevolence in dialogues, and it frames dialogue evaluation as a sample as- signment problem. To achieve this, HMCEval presents constrained optimization problem.
7(DEM)
Active Top-K (Rahman al., 2020) draws the top-k hard samples from the CIR's ' ranked uncer- tainty scores.
'Treating human labor equally on each data point is widely used and necessary in current research (Rahman et al., 2020; Aliannejadi and Trippas, 2022; Desmond et al., 2021).

---

Evaluation Stability. It measures the stability of HumCoE since the data sampling introduces randomness to a certain extent. We consider the variance Ty of multiple evaluation results and the squared error Te to highlight the evaluation stability in finer-grained way.
interchangeably as the surrogate machines. In par- ticular, we use GODEL-basc as the task model f and GODeL-large as the surrogate machine g. and vice versa. The predicted answers of the surrogate machine are utilized in Eq. 2 as pseudo-labels, while Rouge-L the role of EMM Since CoQA are fully annotated by humans, human evaluation can be easily and accurately computed. Based on the test data, we compared the answers generated by the GODEL- and GODEL-large on the test (6) data with the manually annotated answers, respec- tively. Human evaluation of the two models yields results of 68.56 and 75.09.
En Eege8oed (f(D., EM, Yo, (s:) 7)
Tv.
[Sedd
Engeseed (f(D,,EM 1a.) T EM,,
Seed
5.1 Main Results
We verify if HumCoB and other active testing meth- ods approximate the results of the human evalua- tion at a very few cost on human labor. We show HumCoF's effectiveness on typical CIR tasks, in- cluding the conversational question answer and clarifying question, suggested by a recent survey on CIR (Gao et al., 2022). We evaluate the task suc- cess rate well as the quality of system responses. Regarding clarifying questions, we consider two settings: the question is retrieved from pre-defined candidates or generated by a language model. We leave the implementation details in Appendix A.
Evaluation Results. As shown in Table 1, Hum- CoE consumes only less than 0.38% (i.e., 30 out of 7979) of human labor and achieves an average consistency rate of 99%41 with human evaluation, which verifies the effectiveness and efficiency of our method on the conversational question answer- ing task. We also found that the size of the parame- ters of the surrogate machine g and task model f has little impact on the evaluation results of Hum- CoE. Mcanwhile, compared to other active testing methods, our method achieves an average evalua- tion accuracy of 99. 99%17% and 99.64% on two task models for different labor costs. This translates into 46. 06.86% and 36.51 % performance gains over Active Top-K, K, 13.23% and 3.81 % performance gains over HMCEval, and 13.37% and 4.04% performance gains over MCM.
5.1.1 Task 1: Conversational QA
Task Description. Question answering is con- cerned with delivering relatively short pieces of information to answer the queries. Conversational question-answerrigg systems need to handle the more complex linguistic characteristics of conversa- tions, anaphoras or ellipses may occur in question sentences (Vakulenko et al., 2021). In the user- CIR conversations, the CIR system must respond to every question posed by the user.
Table I: the consistency ratio 7 (%) and the inconsistency Ar. '4' means the number of human interactions, and the number in the table is in the form of T(rr)
Evaluation Consistency on CoQA. We report
CIR/ /
HumCoFF (ours)
Active Top-K
HMCEval
MCM
Datasets & Metrics. Following previous works (Kim et al., 2022; Peng et al., 2022), we use the CoQA dataset (Reddy al., 2019). CoQA is es- sentially a collection of dialogues of questions and answers from crowded human annotations. Follow- ing (Peng et al., 2022: Bao et al., 2022), Rouge-L is used to assess the word overlap between the pre- diction and the ground truth answer.
99.60 p..69
56.96/9.51
85.91
85.92
10
(8.73
60.08 22.
(5.91
15
98.44:).0
40.05 (0.05
85.91
85 $5.91.
Godel-base
99.58 (..00)
62.93
85.91 65.94
25
99.20
53.21 (220))
85.98
85. a3.90
30
99.39 (.42)
(3.63
85.97/9.89
85.8 85.87
(9.84
53.27 (an.00)
95.82;2..0
95.78
10
99.28
53.27 (3.09)
95.71 95.71..00 95.82/3.14
Godel Godel-large
15
(9.72
74.58|usa0
95.79
(5.63
20
99.33
72.58/2009)
95.81
95. 95.54
25
99.91
63.39
95.50 95.86
30
19.75
61.70
95.86,3.1)
95.46,1413
Task Model in CIR. As the task model em- ployed in CIR, we use the recently proposed and open-soureed method GODEL (Peng et al., 2022) and evaluate the effectiveness of GODEL on CoQA. In our experiments, both GODEL-base and GODEL-large are evaluated on the test data, which consists of 7979 conversations. In Hum- CoE, GODEL-base and GODEL-large are used
Avg. 7" (%)
99.41
57.72
90.88
90.75
5.1.2 Task 2: Clarifying Question (Retrieved)
Task Description. With a mixed-initiativv conver- sational search, the CIR can take the lead and ask the user clarifying questions to clear up the ambigu- ity in the user's s query (Sekulić et al., 2022; Keyvan

---

and Huang, 2022). Formally, the user expresses their demand in the initial query form, which is subsequently provided to the CIR system. The user needs to generate an answer to the system's 's clari- fying question. The CIR system aims to elucidate the information need through a series of clarifying questions and return the correct answer.
Evaluation Results. Table 2 demonstrates how our HumCoE continues to achieve promising re- sults and improve over the baseline on the clarify- ing question. In particular, our HumCoE consumes only less than 0.68% of human labor (i.e., 30 out of 4411) and achieves an average consistency rate of 99.53% with human evaluation. It achieves eval- uation accuracy of 99.80% and 99.27% on two task models with different labor costs. Compared to other active testing methods, HumCoE enjoys 32.32% performance gains over Active Top-K (i.e., 37.89% on BERT and 26.762 76. on ERNIE), 1.62% performance gains over HMCEval (i.e., 2.37% on BERT and 0.89% on ERNIE), and 2.16% perfor- mance gains over MCM (i.e., 2.37% on BERT and 0.83% on ERNIE).
In accordance with earlier studies (Aliannejadi et al., 2020), we assume that the clarifying ques- tions are retrieved from a pre-defined question pool (see section 5.1.3 for model-genee gederated clarifying questions). In this task, we evaluate the task suc- cess rate (i.e., if it returns the correct answer at the end), and ignore the evaluation on question quality as the question is pre-defined'.
Datasets & Metrics. ClariQ (Aliannejadi et al., 2021, 2020) is employed to evaluate our methods. Given conversation with multiple turns, we evalu- ate if the CIR could select a candidate question that would best clarify the user's intent. Thus, it is a classification problem, and we utilize the accuracy metric to measure the task success rate.
5.1.3 Task 3: Clarifying Question (Generated) Task Description. The CIR system that generates clarifying questions is evaluated in this task. Here, we switch our attention to assessing the quality of the model- questions.
Datasets & Metries. Following earlier studies (Aliannejadi al., 2021, 2020). also consider ClariQ. To evaluate the quality of the generated questions, we consider two types of evaluation met- rics, i... automatic and user-based metrics. Specif- ically, Rouge-L is used as the automatic evaluation metric, where human-annown annotated questions act as the references. Regarding the user-based evalua- tion, follow previous work (Aliannejadi et al., 2020) and employ users assess the relevance and naturalness of each system response. On the one hand, the relevance score (0-5) measures if the clar- ifying question is relevant to the user's information needs. The naturalness score (0-5), on the other hand, measures if the clarifying question is natural in the context of the conversation.
Task Model in CIR. Following Aliannejadi et al. (2021), we use BERT-base as CIR model f. As for the surrogate machine g. we consider ERNIE- base for simplicity. Notably, the two models are also used interchangeably as f and g. The retrieved questions of the surrogate machine are utilized in Eq. 2 as pseudo-labels, and the accuracy metric serves as FMM Later, these two models are fine- tuned on ClariQ training data and then evaluated on the test data, which contains 4411 I conversations. Similarly, since ClariQ is fully numan-annotated the results of human evaluation on the BERT and ERNIE are 88.03% and 91. 91.79% respectively.
Table 2: Evaluation Consistency on ClariQ, where the clarifying questions retrieved.
Task Model in CIR. Following (Sekulić et al.. 2021), we alternate between using BART-base and GPT2-base as the task model and surrogate ma- chine g. These two models are fine-tuned on ClariQ training data. Here, we report the results of human
CIR:
#
HumCCoE (ours)
Active Top-K
HMCEval
MCM
99.76.0.21
90.88
97.52
97.52
99.69 99.69 (:277
56.80 05)
27.52
(2.82
15
99.93
51.12 (1
97.50 a7.
97.50 97.50
BERT
20
99.91 (o.0)
97. 20.40
97. 20.
25
99.75 (0.75
59.07 (s
(2.33)
97. p2..0
evaluation on the BART and GPT-2: the Rouge-L score on BART and GPT-2 are 37.57 and 48.46, respectively. Meanwhile, the relevance score and naturalness scores for GPT-2 are 3.09 and 3.51, while the scores for BART are 3.82 and 3.85, re- spectively.
(9.73
60.58
p2.32
p..00)
g7.01 (1.70)
43 (3 m9)
98.25 .x)
98.25 (111)
10
99.74
76.26 29)
98. 98.34
98. 34
15
90.56mp.00
79. 79.89
98. 98.42(1.15)
98. (..))
ERNIE
20
99.87 g0.17 (1.12)
76.26 79)
25
99.67 (0.30)
82. 12,80 79)
98.42
98. 98.52
30
(9.74(0.44)
76.26
98.47
98.5%
Avg. >" (%)
99.53
67.21
97.91
97.37
Automatic calculation on relevance and nat- uralness scores. The process of manual scoring explicitly write into a specific for-
"Wo also ignore the interaction efficiency as it could he casily computed without human involvement.
is difficult to

---

Table 3: Question quality evaluation on ClariQ, where the questions are model generated. We report the consistency ratio 7" (%) and inconsistency Ar. For consistency rate of ChatGPT, it achieves 89.32% relevance score and 99.46% on naturalness score when f=GPT-2. For BART, ChatGPT achieves 88 28.22% and 85.97% on those scores. After introducing small number of labor cost, HumCoE further improves the evaluation consistency.
CIR /
HumCoB (ours)
Active Top-K
HMCEval
MCM
CIR
HumCoE (ouns)
Active Top-K
HMCEsal
MCM
Claritying Question (Generated) with Offline Automatic Metric
99.57 16)
89, (9.97
94.01 (4.01
94.14
93.56 1y
(4.93
82 (2.07
82. (0 a0
10
98.86 ca)
81, 16
94, (4.01.2300)
93 93.88
10
94.99 eag
64.5
82. (5.15
83.41 or:
GPT-2
15
97.55 (uss
88. 68.32
93.77 93.85
BART
15
(5,23
7142
82.83
84.38
(Rouge-L)
20
96.75
79.82
93.59:2.4)
93.05/2m)
(Rouge-L.)
20
16.10
(7.79
89.04
87.27
25
95,26
81 (7.9))
93.80 (3.80
92.76
25
95.60
12.29
83 83.45
85 65.62, on
(3.43
85.01 85.01
93.69 (::37)
92.44/280
(2 013
76.25
83.55
y6.03 mi
Avg. (%)
96.90
84.26
93.83
93.34
Avg. r" (%)
95.05
71.21
82.85
84.59
Clurifying Question (Generuted) with Online Human-- based Metrics
5
99.06
96.55ge.33
99.06
98.81 m0.8
5
98.74
87.39%
89. 89.54/0.5m)
90.0%
98. 10
80 (..
99.21 (0.000
10
(u.62
85.40)
89. 89:79
90.77
GPT-2
15
97. 97.74jmmm
(2.61
98.96 p..9)
97.59 (7.
GPT-2
15
98.39 m8.39
80.43 so.s))
89.8 89.87
91.62,0
(Relevance)
(7.34
78. gquusss
99,04
96.88
(Naturalncss)
98.03 muor)
78.37 s0.799
89. 89.99
92.29
25
97. 97.63
86.21 86.211c.45)
98.86 (u:4)
96.20
25
98.18 p
80.55 yuass
89 (9.0))
93.00
30
96 81
86, 57.0. 450
98.73 98.73
30
98. (0
79. (.m1
89 (..)))
93, 74,0 en:
94.15 (5uu40
88.55 (u44)
88.0 (..0
99.75 (10))
89,97 yuss9
86 86.24
85.6 85.69
10
98.33 jumn
97, (4/c
88. 33
87. 9.
10
98. mu))
95. 16
86 86.08
85.51 (uso)
BART
97.9 97.99 on)
95. 32 5)
(1.47
BART (Naturalnces)
o0)
45,75 95.78
86. (6)36)
a0.49
(Rolevance)
20
98.19
97. 77.21.e
88.57 (8.57
87.2 87.2%
20
97.88
96.45 (6.45
86. 86.42
85. 14
(8.03
93.81
88.50 88.59
(6.97
(3.23
86.75jusx)
94
30
98.00
(4.44
88. g8.82
86.73
30
96.00
84.6 84.65 (6.87
Avg. (%)
98.07
90.29
93.77
92.33
Aug. r" (%)
98.60
88.60
88.16
88.57
mula. Namely, there is no pre-defined and specific formula like Rouge-L that takes generated sen- tence as input and directly outputs the relevance and naturalness scores. Therefore, it is challeng- ing to calculate Eq.2 for relevance and naturalness score evaluation as the EM is unknown. To this end, we design prompts for ChatGPT scoring and achieve automatic calculation of relevance and nat- uralness scores (see Appendix B for details). In this case, q(d) α 1 ChatGPT (di, 9:).
ChatGPT and human evaluation results is not sat- isfying (on average, 88.24%), and ChatGPT may be difficult to replace humans in achieving fully automated CIR evaluation. Interestingly, by intro- ducing a small number of labor costs (0.68% of human labor), HumCoE corrects the evaluation re- sults of ChatGPT and improves consistency with human evaluation results by 9.59% (see the Evalu- ation Results below).
Evaluation Results. Table summarizes the results of different evaluation methods on different evaluation metrics. Our method consumes only less than 0.68% of human labor (i.e., 30 out of 441) I) and achieves an average consistency rate of 95.98% on Rouge-L. with human evaluation (i.e., 96.90% on GPT-2 and 95.05% on BART), 98.07% on Rel- evant score (i.e., 97.88% on GPT-2 and 98.24% on BART), and 98.60% on Naturalness score (i.c.. 98.4 08.49% on GPT-2 and 98. 78% on BART). By in- troducing a small number of labor costs (0.68% of human labor), HumCoF corrects the evaluation results of ChatGPT and improves the consistency in terms of Relevant score (i.e., 8.56% improve- ment on GPT-2 and 10.02% on BART) and Natu- ralness score (i.c., 9.03% improvement GPT-2 and 12.73% on BART). This motivates the need of human-machine collaboration for CIR evalu- ation. Meanwhile, we observed that HumCoE's advantage on GPT-2's averaged relevance score is slightly reduced by 1.07% compared to HMCEval.
Given the results of ChatGPT on GPT-2, the rel- evance and naturalness scores are 3.42 and 3.14, which translate into an evaluation consistency of 89.32% and 89.46% with human evaluation. We also examine the inter-annotator reliability of the ChatGPT scores in finer-grained way. In par- ticular, we resort to Krippendorff's alpha (Krip- pendorff), which commonly used reliability coefficient developed to measure the agreement among different annotators. Given human evalua- tion, ChatGPT scores only achieve 0.321 on rele- vance and 0.284 on naturalness, implying the agree- ment between human evaluation and ChatGPT is relatively low. Similarly, regarding the BART, the relevance and naturalness scores from ChatGPT are 3.37 and 3.31, respectively. In this case, Chat- GPT scores achieve 88.22% and 85.97% in terms of evaluation accuracy, 0.405 on relevance, and 0.339 on naturalness in terms of Krippendorff's alpha. We highlight that the consistency between

---

100
35
HomCell Calibmation
95
Homoooll wi Kandom sampiig
90
25
raw Homoob
85
25
80
20
15
CAKEL OODE
70
10
60
0
10
15
30
15
20
30
10
20
25
20
25
5
20
15
20
30
GODEL-base
GODEL-large
GODEL-buse
GODEL-kryu
GODEL-base
GODEL-Aage
(a) Consistency
xd
(b) Evaluation Variance Ty
(c) Evaluation Squared Error Ta
Figure 1: Ablation studies on conversational question answering task. The sampling process enhances the stability HumCoE, while the alibration procedure focuses more on improving the accuracy.
The reason behind this is that the response of GPT- 2 overestimated by ChatGPT, as evidenced by the higher average relevance score (3.42) compared to human evaluation (3.09). This poses larger weights to selected data during the evaluation calibration, ultimately resulting in less precise evaluation re- sults. However, our method still achieves better results than HMCEval in most cases. On average, it enjoys 7.6 7.64% performance gains on Rouge-L, 4.30% on Relevance score, and 10.33% on Natural- ness score.
We examine the Evaluation Stability (i.e., Ty and Te) and Evaluation Consistency (i.e., 7") with or without surrogate machine based data sampling and evaluation calibration. Specifically, we remove sur- rogate machine based data sampling and replace it with random sampling, which we refer to as Hum- CoE w/ Random sample. IlumCoE w/o Calibration means calibration. We experimentally show that our surrogate machine-based sampling enhances the stability of HumCoE, while the calibration pro- cedure focuses more on improving the accuracy. Due to limited space, we present the results for conversational question answering in Fig. and pro- vide the rest in Appendix D. According to and in Fig. L(b) and Fig. I(c), randomly selecting partial data for human annotations and calibration involves higher variance and instability (i.e., large Ty and Te). Also, the calibration method enhances stabil- ity to some degree. These observations are in line with previous studies stating that the importance sampling is classic variance reduction technique (Elvira and Martino, 2021). Regarding the evalua- tion consistency in Ia), the use of calibration surrogate machine based data sampling tech- niques alone tends to reduce the consistency of evaluation results. But even with random sampling, calibration techniques still maintain good results (i.c., HumCoE w/ Random sample). This indicates that the calibration technique is more helpful in improving evaluation accuracy.
Summarization on three tasks. From the above experiments, we have two observations: 1) We ex- perimentally demonstrate that HumCoE produces evaluation results that are highly close those obtained through human evaluation while also sig- nificantly reducing the amount human labor re- quired. With less than 1 % of human labor, Hum- CoE achieves consistency rate between 95% and 99% when compared to human evaluation. This re- veals that the proposed method in this paper could effectively reduce human labor when evaluating CIR system. 2) Our approach performs better than other active testing baselines in terms of the average consistency rate for all tasks. Specifically, with less than 1% human labor, our approach achieves an av- erage consistency rate of 98.32%, which is 22.01 % higher than the Active Top-k method, 6.51% % higher than the HMCEval method, and 6. 6.73% higher than MCM. As mentioned carlier, our method experi- ences slight drop in performance when assessing the Relevance score for the GPT-2 model because ChatGPT overestimates the response of GPT-2. However, our approach still showcases its supe- riority over other baselines across numerous tasks and evaluation metrics.
Conclusion
Evaluating any highly interactive process is chal- lenging task when taking human labor into account. To touch upon this challenge, we take the first step utilize active testing evaluate the CIR system. We tentatively give a successful implementation explore the potential of human-machine collabo- rative evaluation the CIR scenario. Empirical experiments are conducted to testify to the effec-
5.2 Ablation studies
This section aims to perform ablation studies on HumCoE and investigate its properties in detail.

---

tiveness of our method. In consistent manner, our method produces results that are extremely com- parable to those of a human-based evaluation yet require very little human labor compared to other baselines. Our future work may design a more sophisticated active testing method and see the pos- sible outcomes of CIR evaluation.
2021 Conference on Empirical Methods in Natural Language Processing, pages
Mohammad Aliannejadi and Johanne R. Trippas. 2022. Conversational information seeking: Theory and eval- uation: Chiir 2022 half day tutorial. In Proceedings of the 2022 Conference on Human Information In- leraction and Retrieval, CHIIRR '22, page 365-366, New York, NY, USA. Association for Computing Machinery.
7 Limitations
Mohammad Aliannejadi. Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clari- fying questions in open-domain information-secking conversations. In Proceedings of the 42nd interna- tional acm sigir conference on research and develop- ment in information retrieval, pages 475-484.
As demonstrated in Section 5.1.3, some evaluation metrics in CIR cannot be directly computed using a predefined mathematical formula, such rele- vance and naturalness scores. This may restrict the application of HumCoE in evaluating CIR. In our experiments, we use ChatGPT to automatically simulate the human scoring process for these two metrics. Although our approach significantly ad- dresses the inconsistency issue between ChatGPT and human evaluations with minimal human ef- fort and provides evaluation results that are highly comparable to human evaluations, we observed slight reduction in the consistency of our method's evaluation results due to ChatGPT'ss overestimation when automatically computing the relevant scores of GPT-2. Therefore, when there is no predefined mathematical formula to directly compute an eval- uation metric, determining how to further enhance the evaluation accuracy of HumCoE with minimal human labor costs remains a challenge.
Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang. Wenquan Wu, Zhihua Wu, Zhen Guo, Hua Lu, Xinx- ian Huang. Xin Tian, Xinchao Xu, Yingzhan Lin, and Zu Niu. 2022. PLATO-XL: Exploring the large-scale pre-training of dialogue generation. In Findings of the Association for Computational Linguisties: AACI-IJCNI.P 2022, pages 107-1 18, Online only. Association for Computational Linguis- tics.
Daniel Braun, Adrian Hernandez Mendez. Florian Matthes, and Manfred Langen. 2017. Evaluating natural language understanding services for conver- sational question answering systems. In Proceedings of the I 18th annual SIGdial meeting on discourse and dialogue, pages 174-185.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašić. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for trik-triented dialogue modelling. arXiv preprint arXiv:1810.00 1810.00278.
Acknowledgement
This work was supported in part by the Na- tional Natural Science Foundation of China (No. 62272330); in part by the Fundamental Research Funds for the Central Universities (No. YJ202219).
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174-2188 2144, Brussels, Belgium. Association for Com- putational Linguistics.
References
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Sule- man, Harm de Vries, and Siva Reddy. 2022. Topi- ocqa: Open-domain conversational question answer- ing with topic switching. Transactions of the Associ- ation for Computational Linguistics, 10:468-483.
C. Chow. 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, (6:1:44446..
Jeffrey Dalton, Chenyan Xiong, Vaibhay Kumar, and Jamie Callan. 2020. Cast-19: A dataset for conver- sational information seeking. In Proceedings of the 43rd International ACM SIGIR Conference on Re- search and Development in Information Retrieval. pages 1985-1988.
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2020.
Convai3: Generating clarifying questions for open- domain dialogue systems (clariq). arXiv preprint arXiv:2009.11352
Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021. Building and evaluating open-domain dialogue cor- pora with clarifying questions. In Proceedings of the
Michael Desmond, Evelyn Duesterwald, Kristina Brimi- join, Michelle Brachman, and Qian Pan. 2021. Semi- automated data labeling. In Proceedings of the NewrlPS 2020 Competition and Demonstration Track,

---

volume 133 of Proceedings of Machine Learning Re- search, pages 156-169. PMLR.
conversations from unlabeled documents. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2362- 2378.
Víctor Elvira and Luca Martino. 2021. Ad- vances in importance sampling. arXiv preprint arXiv:2102.05007.
Jan-Chrin Chr--stiittphhhhoo Klie, Richard Eckart de Castilho, and Iryna Gurevych. 2020. From Zero to Hero: Human- In-The-Loop Entity Linking in Low Resource Do- mains. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 6982 6993, Online. Association for Computational Linguistics.
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. 1021. On statistical bias in active learning: How and when to fix it.
Chongming Gao, Wengiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey, Al Open, 2:100-126.
Klaus Krippendorff. Computing krippendorff's ' alpha- reliability. Computing, 1:25-2011.
Jianfeng Gao, Chenyan Xiong, Paul Bennett, and Nick Craswell. 2022. Neural approaches to conversational information retrieval.
Trausti Kristjansson, Aron Culotta, Paul Viola, and An- drew McCallum. 2004. Interactive information ex- traction with constrained conditional random fields. In AAAL, volume 4, pages 412 418.
Yves Grandvalet, Alain Rakotomamoniy, Joseph Keshet, and Stéphane Canu. 2008. Support vector machines with a reject option. In Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc.
Sahiti Labhishetty and Chengxiang Zhai 2021. An exploration of tester-based evaluation of user sim- ulators for comparing interactive retrieval systems. In Proceedings of the 44th International ACM SI- GIR Conference Research and Development in Information Retrieval, pages 1598-1602
Matthew Henderson, Pawel Budzianowski, Irigo Casanueva, Sam Coope. Daniela Gerz, Girish Ku- mar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su. Ivan Vulié, et al. 2019. A repository of conversa- tional datasets. arXiv preprint 1904. 06472.
Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lee, Parker Hill. Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evalua- tion dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing, pages 1311-1316.
Rebecca Hwa. 2000. Sample selection for statistical grammar induction. In 2000 Joint SIGDAT Confer- ence on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 45-52.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACL 2021-16th Conference of the European Chapter of the Associa- tion for Computational Linguistics, pages 874-880. Association for Computational Linguistics.
Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2020a. Conversational recommen- dation: Formulation, methods, and evaluation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR '20, page 2425-2428, New York, NY. USA. Association for Computing Machin- ery.
Vladimir Karpukhin. Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020b. Interactive path reasoning on graph for con- versational recommendation. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2073- 2083.
Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search? a survey of techniques, approaches, tools and challenges. ACM Computing Surveys.
Huihan Li, Tianyu Gao, Manan Goenka, and Danqi Chen. 2022. Ditch the gold standard: Re-evaluating conversational question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 8074-8085.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A Smith, and Daniel S Weld. 2021. Genie: leader- board for numan-in-the-loog evaluation of text gener- ation. Xiviv preprint arXiv:2
Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with opti- mized questions and multi-turn comparisons. arXiv preprint 790000307.
Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. 2022. Generating information-seeking

---

Pengshuai Li. Xinsong Zhang, Weijia Jia, and Wei Zhao. 2020. Active testing: An unbiased evaluation method for distantly supervised relation extraction. arXiv preprint arX1v:2010.08777.
Joao Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison- Burnh.. 2019. Chateval: A tool for chatbot evaluation. In Proceed- ings of the 2019 conference of the North American chapter of the association for computational linguis- tics (demonstrations)) pages 60-65.
Aldo Lipani, Ben Carterette, and Emine Yilmaz. 2021. How am i doing?: Evaluating conversational search systems offline. ACM Transactions on Information Systems, (-):1122.
Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards facet-driven generation of clarifying questions for conversational search. In Pro- ceedings of the 2021 I ACM SIGIR International Con- ference on Theory of Information Retrieval, pages 167-175.
Yang Liu and Yuexian Hou. 2023. Mining effective fea- tures using quantum entropy for humor recognition. arXiv preprint 230.023302..3716..
David Madras, Toni Pitassi, and Richard Zemel. 2018. Predict responsibly: Improving faimess and accu- racy by learning to defer. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2022. Evaluating mixed-initiative conversa- tional search systems via user simulation. In Proceed- ings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 888-894 188-896.
Hussein Mozannar and David Sontag. 2020. Consis- tent estimators for learning to defer to an expert. In Proceedings of the 37th International Conference on Machine Learning. ICML'20. Jrg..org.
Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 355-363.
Phuc Nguyen, Deva Ramanan, and Charless Fowlkes. 2018. Active testing: An efficient and robust frame- work for estimating accuracy. In International Con- ference on Machine Learning, pages 3759-3768. PMLR.
Zijie Wang. Dongjin Choi, Shenyu Xu, and Diyi Yang- 2021. Putting humans in the natural language pro- cessing loop: A survey. In Proceedings of the First Workshop on Bridging Human- Computer Interaction and Natural Language Processing, pages 47-52.
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. 2019. On the calibration of multiclass classification with rejection. In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Yijum Xiao and William Yang Wang. 2019. Quantifying uncertainties in natural language processing tasks. In Proceedings of the AAAT conference on artificial intelligence, volume 33, pages 7322 -3299.
Liu Yang. Minghui Qiu, Chen Qu, Jiafeng Guo. Yongfeng Zhang, W Bruce Croft. Jun Huang, and Haiqing Chen. 2018. Response ranking with deep matching networks and external knowledge in information-seeking conversation systems. In The 41 st international acm sigir conference on research & development in information retrieval, pages 245- 254.
Dookun Park, Hao Yuan, Dongmin Kim, Yinglei Zhang, Matsoukas Spyros, Young-Bum Kim, Ruhi Sarikaya, Edward Guo, Yuan Ling, Kevin Quinn, et al. 2020. Large-scale hybrid approach for predicting user sat- isfaction with conversational agents. arXiv preprint arXiv:2006.071133
Baolin Peng. Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. Godel: Large-scale pre-training for goal-directed dialog. arXiv.
Hamed Zamani, Johanne R Trippas. Jeff Dalton, and Filip Radlinski. 2022. Conversational information seeking: An introduction to conversational search, recommendation, and question answering.
Md Mustafizur Rahman, Mucahid Kutlu, Tamer EI- sayed, and Matthew Lease. 2020. Efficient test col- lection construction via active learning. In Proceed- ings of the 2020 ACM SIGIR on International Con- ference on Theory of Information Retrieval, pages 177-184.
Yangjun Zhang, Pengjie Ren, and Maarten de Rijke. 2021a. A for evaluating malevolence in dialogues. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, pages 1612-5623.
collaborative framework
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Racfer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, al. 2018 Conversational ai: The science behind the alexa prize. arXiv preprint 10T.00000033
Yangjun Zhang, Pengjie Ren, and Maarten de Rijke. 2021b. A human-machin collaborative framework for evaluating malevolence in dialogues. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tth Interna- tional Joint Conference Natural Language Pro- cessing, pages 5612-5623, Online. Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266

---

Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and Bruce Croft. 2018. Towards conversational search and recommendation: System ask, user respond. In Proceedings of the 27th aem international conference on information and knowledge munagement, pages 177-186.
Active Top-K (Rahman et al., 2020) draws the top-k hard samples from the CIR's ranked uncertainty scores. It has two variants, i.e., Human-Only Active Top-K K and Hybrid Active Top-K. The former uses only human-annotated data, while the latter incorporates both human- annotated data and surrogate machine-annotated data for CIR evaluation. In this case, HMCEval (Zhang et al., 2021a) is a Hybrid Active Top-K- based method.
Implementation Details
A.1 HumCoE Implementation
We use Eq.2 to compute the hardness value ((d:) for each sample d;. These hardness values are then normalized to a range of 0 to by di- viding each element by its sum, i.e., (dd)) q(d:)/ Ea, 4(d)). To avoid sampling data with excessively small g(d.), we add a clipping oper- ation by setting g(d:) min(g(d:), o). Here, a is a hyper-parameter, and in practice, a = Di), where D/ represents the uni- form sample probability. We observed that remov- ing excessively small sample weights can enhance evaluation stability. To implement the data sam- pling step, we use the WeightedRandomSampler class in Pytorch. Later, we limit the size of D, in {5, 10, 15, 20, 25,330 30} to verify the effectiveness of different evaluation methods with minimal labor costs. Considering the randomness of data sam- pling, we ran the evaluation 100 times and reported the mean estimation results.
HMCEval (Zhang al., 2021a). We have implemented HMCEval, originally designed to evaluate malevolence, to fit in the CIR scenario. Specifically, since we treat human labor for each interaction sample equally and machine evalu- ation is labor-free" the solution HMCEval suggests to assign samples with high prediction certainty to human and machine is responsible for evaluating the rest. To measure uncertainty, we initially used the maximum class probability (MCP), as suggested in the original HMCEval pa- per. However, MCP is intended for classification- based evaluation metrics (because identifying the malevolence in the dialog is a classification task itself), which does not apply to our first and third tasks which are generation ceneration-based. To this end, we resort to the entropy- -bsed method as an al- ternative to uncertainty modeling. In particular, inspired by (Xiao and Wang, 2019; Liu and Hou, 2023), calculate the average entropy of gener- ating each word (or tokens) as the uncertainty of the given model generating the whole response. Because the output dimension of the predictive word distribution is very large, it would be im- precise and not very useful to calculate the distri- bution entropy directly on this high-dimensional distribution. Therefore, we choose the top-5 val- ues from the predictive distribution and convert them into a 5-dimensional distribution using Soft- Max (We also use the same trick for MCM and Active Top-K). We then use this to calculate the word entropy.
set
A.2 Baseline Implementation
There is no previous work on active testing methods for CIR. We re-implement existing baselines as fol- lows. Since our method prefers hard samples, we compare methods that also select hard samples for human annotations, including HMCEval (Zhang et al., 20)1 MCM (Nguyen et al., 201 and Active Top-K K (Rahman et al., 2020). Given that the previous methods are not designed for CIR eval- uation, we re-implement them in the CIR setting, where the same entropy- based method is used.
Costt-Confident Mistake (MCM) (Nguyen et al., 2018) selects hard samples whose CIR prediction confidences are very high, and whose CIR predic- tion results are inconsistent with those of the sur- rogate machine. For example, if CIR believes that the current sample is predicted as being pos- itive with high probability, while the surrogate machine's pseudo-label is negative. The label from the surrogate machine is likely at fault, re- quiring human annotation for correction.
Task Setup & Task Model A.3
Implementation
In HumCoF, we train all surrogate machines using the same training data as the CIR model f. We
'Treating human labor equally on each data point is widely used and necessary in current research on Active testing (Nguyen et al., 2018: Rahman et al., 2020), CIR (Alianne- jadi and Trippas, 2022; Lei et al., 2020a), and human-machine interaction (Klie et al., 2020; Desmond et al,, 2021).

---

Table 4: Automatic calculation on relevance and naturalness score using ChatGPT scoring
Frangt Truplaat
1. [QUESTION] an thak about S1 Sifncet) I sur that Stqeest. [ANSWER] S Sjanower). ESCORE: ("Reisvence" 2. "Nouralees".
2. [QUESTION] san thak about Sifaeet) czy luut Siques [ANSWER] Staunwerr] (SCORE: '"Retonmnee" 2, "Nauralzee" eyh On horpln, is the heo, mas the (A (ANSW Wi)) in abons serronon hoord on the followring cnterta:
Releseno:: the JANSWER) rolonant the (QUESTION),
Naurattess tte (ANSWER] naturl.
eaampiee
1) [QUISTION] think about sa that Broohers dleannne [ANSWER] you looking specitic Beooks Beothers [SCOR(] ("Relanence": "Nauratnes":
2o (QUESTION] an think about home sale. eay that tell about cao. county miocuni (ANSWER) are you looking fir sale for home dellax, (SCIORE) Rebimenee" 4, "Natorinnaa"
Prount the scoes in ISEN fomes ac fullinoss
("Retavence" -choars "Nouralnes" :xtlorr) Pleas puvále souees and cuplais Jill
conducted all our experiments on an NVIDIA RTX A6000 GPU with 48G graphical memory.
(Sekulić et al., 2022; Keyvan and Huang, 2022). Formally, the user expresses their demand the initial query form, which is subsequently provided to the CIR system. The user needs to generate an answer to the system's clarifying question. The CIR system aims to elucidate the information need through series of clarifying questions and return the correct answer. Following (Aliannejadi et al., 2020), we assume that the clarifying questions are retrieved from a pre-defined question pool. In this task, we evaluate the task success rate (i.e., if it returns the correct answer at the end), and ignore the evaluation of question quality as the question is pre-defined. Regarding the task model in this task, follow the previous study (Aliannejadi et al., 2021), and utilize BERT-base and ERNIE-base as and g interchangeably in our experiments. These two models are fine-tuned on ClariQ (Aliannejadi et al., 2021, 2020) training data and then evaluated on its test data. Specifically, we fine-tune cach model on one epoch with -7 learning rate, 32 batch size, and Adam optimizer. Considering that it is classification problem, we utilize the accu- racy metric measure the task success rate. In this case, q(d,) x 1 Ace (d,, Dc). Note that the results of human evaluation on the BERT and ERNIE are 88.03% and 91.79%, respectively.
A.3.1 Conversational Question Answering
Conversational question-answerrig systems need to handle the more complex linguistic characteris- ties of conversations, anaphoras or ellipses may oc- cur question sentences ( (Vakulenko et al., 2021). user-CIR conversations, the CIR system must respond to every question posed by the user. As for the task model in HumCoE, we use the recently proposed and open-sourced method GODEL (Peng et al., 2022), which can be downloaded from Hug- gingface and use the given instruction finish a conversational question answering task. In Hum- CoE, we use GODEL-base as the task model and GODEL-large as the surrogate machine g. and vice versa. Following previous works (Kim et al., 2022; Peng et al., 2022), we use the CoQA dataset (Reddy et al., 2019) for conversational question- answering simulation. Based on the test data, we compared the answers generated by the GODEL- base and GODEL-large on the test data with the manually annotated answers, respectively. We use Rouge-L to assess the word overlap between the prediction and the ground truth answer. In this case, ((d)) & 1 - Rouge-L (d;, Di). Note that hu- man evaluation of the two models yields results of 68.56 and 75.09.
A.3.3 Clarifying Question (Generated)
In this task, the CIR system that generates clarify- ing questions is evaluated. Here, we shift the focus to evaluating the quality of the generated questions. Following earlier studies (Aliannejadi et al., 2021, 2020), we also consider ClariQ in our experiments. To evaluate the quality of the generated questions, consider two types of evaluation metrics, i.e., automatic and user-based metrics. Rouge-L is used as the automatic evaluation metric, while the user- based evaluation is conducted by the human provid- ing ground truth (cf. Appendix C) and the ChatGPT
A.3.2 Clarifying Question (Retrieved)
The CIR system needs have clear understand- ing of the underlying user needs. Since user's queries are often under-specified and vague, with a iixeed-iniiaatiee conversational search, the CIR can take the lead and ask the user clarifying ques- tions to clear up the ambiguity in the user's query
'GODEL-large: httpsss//huggggggaaacc/c//iiiiis GODEL-v1 Lar___laargeeeee2eess GODEL- https:// huggingface.commiccosstttoo

---

100
HauCCE Calinntas
95
35
#HooCCaE sampling
25
wne NooCalll
85
20
25
30
20
15
70
10
10
60
10
15
20
25
30
10
15
20
25
30
10
15
20
25
30
10
15
20
25
10
15
20
25
30
10
15
20
25
30
GODEL-base
GODEL--aage
GODEL-base
GODEL-lage
GODEL-base
GODEL-lage
(a1) Evaluation Consistency 1
(b1) Evaluation Variance T
(c1) Exallation Squared Error T
100
10
-HemCall wis Calibration
HommColl w/ Rkmdom samping
90
HomCoo
85
75
65
15
30
20
25
25
30
20
15 ERNEE
BERT
ERNIE
BERT
ERNIE
BERT
(a2) Evaluation Consisteney 2
(b2) Evaluation Variance F,
(c2) Evaluation Squared Errort,
100
1.4
HonCae wol Calibatiin
12
1.2
HomC wi Random saiping
90
HanCoE
0.8
80
0.6
0.4
0.4
65
0.2
0.2
10
15
20
25
30
20
25
30
10
15
20
25
30
10
15
20
25
30
5
10
15
20
25
30
10
15
20
25
30
GPT-2 (Rougs-L)
BART (Rouge-L)
GPT-2
(Rouge-L)
BART (Rouge-L)
GPT-2 (Rouge-L)
BART (Rouge-L)
(a3) Evaluation Consistency. ?
(b3) Evaluation Variance Ty
(c3) Evaluation Squared Error T2
100
0.30
0.35
HouC.EE Calisnaion
0.25
0.30
IlooCCal of wi Random sumpling
90
0.25
22M HonCoE
0.20
0.15
75
0.10
0.10
0.15
0.05
0.05
60
0.00
30
10
15
211
25
30
5
11
15
20
25
30
10
15
20
25
30
5
10
15
20
30
GPT-2 (relovance)
BART (relevence)
GPPT22 (relevance)
BART
(televance)
GPE-2 (relevance)
HART (relevance)
(a4) Evaluation Comsstency
(b4) Evaluation Variance F,
(cf) Evaluntion Squared Errore, F,
100
0.25
0.25
HomDaF wis Cxlibrtiion
HomColl Random campling
90
0.20
0.20
wow HomCoe
0.15
0.15
80
75
0.10
0.10
0.05
60
0.00
20
25
30
10
15
20
25
30
5
10
15
20
25
30
5
10
15
20
25
30
5
10
15
20
30
GPT-2 (naturainces)
BART (naturalness)
GPT-2
RART (nstumbhess) (b5) Evaluation Variance
(caturalness)
GPT-2 (nnturslness]
BART
(eaturalncss)
(a5) Evaluation Consistency e
(c5) Evaluation Squared Error
Figure 2: Ablation studies on all tasks and metrics.
completing the automatic evaluation (ef. Appendix B). In this case, (d4)) x 1 Rouge-L (d;; ÿx) and g(d:) x 1 ChatGPT (dę, ūt). Here, we report the results of human evaluation on the BART and GPT- 2: the Rouge-L scores on the BART and GPT-2 are 37.57 and 48.46, respectively. Meanwhile, the relevance score and naturalness score for GPT-2 are 3.09 and 3.51, respectively, while the scores
for BART are 3.82 and 3.85, respectively. As for the task model, we follow (Sekulié et al., 2021) and alternate between using BART-base and GPT2- base, which are also fine-tuned on ClariQ training. Specifically, we fine-tune each model for 5 epochs using an Adan optimizer, 5e learning rate, and a batch size of 32.

---

B Automatic calculation on relevance and naturalness score using ChatGPT
Table 5: Evaluation Consistency on TopiOCQA.
CIR
#
HumCoE (ours)
Active Top-K
HMCEEal
MCM
5
82.87 82.87g3.51 (3.31)
17, ps.05
a6.19
20.90 16
Tasks (such as conversational question-answeringg utilize automatic metrics (such as Rouge-I.) for evaluation. In terms of evaluating model-gencrated- based clarifying questions, relevance and natural- ness cannot be automatically calculated. To this end, resort ChatGPT calculate relevance and naturalness. As ChatGPT requires fee and has been banned in some countries, we use Mon- ica? for annotations, which is Chrome extension powered by ChatGPT API. As free users of Monica, we have a daily usage limit. Thus, we require Mon- ica to annotate multiple data at a time (precisely, 20 at a time). Specifically, to enhance the precision of ChatGPT scoring, we include two and scoring cases input. The prompt template for ChatGPT scoring can be found in Table 4.
77.73
ps.43
76. 16.19
23. 23.80 (0.15.00
15
82.68 (3.5))
32.58 (1..0)
76. cm
21. 30.
DRP
20
80.9 80.94, (3,000
24. (15.31)
76.00
23. 23:70 23.70(15.60)
79.81 (a.08)
39.09 (12.073
76.00
15..99
79.55 (a:10)
(1.58
26.00
23.60
5
90..18 (.26)
30.33 (00.19)
15,40 (1.80)
10
91,988 (0.47)
45.26
72.79
7. 7.70
90.89 (5.08)
50.28
3.80 (2.78
FID
(9.41
(5.35
(1.82
10.60, 10.60 (8.79)
90.62
(2.29
(2.85
5. 5.40 00)
30
90.61
64.11 (18.66)
(2.73)
9.20) a5)
D Appendix on ablation studies
We report the results of ablation studies in Fig. 2, which are consistent with the previous observations in Section 5.2. However, we notice some special cases in the results. For example, in Fig. 2(b2) and Fig. 2(c2), without calibration, the evaluation stability is worse than the case without surrogate machine based data sampling. This motivates more future work to study better and more stable active testing algorithms.
definitions
C User-evaluation on relevance and naturalness score (human annotations)
For the clarifying question task with model- generated questions, we follow previous work (Aliannejadi et al., 2020) and employ 3 annota- tors to assess the relevance and naturalness of each system response (i.e., GPT-2 and BART) the test data. The relevance score (0-5) measures if the clarifying question relevant the user's in- formation needs, while the naturalness score (0-5) measures if the clarifying question is natural in the context of the conversation.
However, from the overall experimental re- sults, we experimentally show that our surrogate machine based data sampling enhances the stabil- ity of HumCoE, while the calibration procedure fo- cuses more improving our accuracy. According to the results of evaluation consistency 7 combin- ing evaluation calibration and surrogate machine based data sampling methods can achieve better results in terms of the average consistency rate for all tasks. Specifically, our approach achieves an average consistency rate of 98.32%, which is 2.75% higher than the HumCoF w/ Random sample and 5.88% higher than the HumCoE w/o Calibra- tion. It also implies that calibration brings 3.13% performance gains over HumCoE w/o Calibration. Regarding the evaluation stability, our HumCoE also enjoys better results in terms of the average To and average Te for all tasks. In particular, our approach achieves an average Ta of 0.34, which translates into 1.24 higher than the HumCoE w/ Random sample and 0.24 higher than the HumCoF w/o Calibration. We also achieve an average of 0.35, which is 1.36 higher than the HumCoE w/ Random sample and 0.95 higher than the HumCoE w/o Calibration. These indicate that the surrogate machine based data sampling method brings higher evaluation stability than the random one.
In particular, we provide each annotator with the meanings and evaluation protocols of two scores, Three annotators independently annotate and eval- uate the system responses the test data, dis- cuss any disagreements, and revise the annotation scheme. Before annotating the test data, the three annotators complete the annotation exercises on 100 examples randomly selected from the training data. They continue this process until they achieve inter-annotator reliability of Krippendorff's alpha greater than 0.7 on these samples. Afterward, each annotator scores all the system responses. The final human evaluation score for each data is the mean of the scores given by the three annotators. This approach improves the robustness of the results.
hhttps: /ttpss///ooniaiiii

---

E Appendix on Related Work
Human-machine Evaluation. Human- machine evaluation has been under-explored in the NLP community, where the traditional focus has been on building a better model (Wang et al., 2021). They mainly build upon the human-in-the-Ble nario, where the values of automatic evaluation metries are provided for humans as evaluation sug- gestions (Khashabi et al., 2021; Sedoc et al., 2019). Such methods only reduce human cognitive labor yet still require humans to go through the whole dataset. We argue that they are machine-assisted evaluation methods rather than human-machine col- laboration methods for evaluation. Thus, current human-machine machine evaluation is out of our research scope.
Human-machine
sce-
Learning defer. One way to achieve human- machine evaluation is through learning to defer (Madras et al., 2018; Mozannar and Sontag, 2020), also known as rejection learning (Chow, 1970). This approach involves assigming tasks to either machines or humans to make decisions. When applied to the evaluation task, the main goal of learning to defer is determine which data should be given humans and which should be given to machines to ensure accurate evaluation results for the entire dataset. Similar to HMCEval (Zhang et al., 2021b), the most common method to as- sign data with high machine prediction uncertainty to humans and easier-to- -rredicct data to machines (Ni et al., 2019; Grandvalet et al., 2008) However, method differs from learning defer in that we focus on selecting a subset of data for human evaluation that can accurately estimate the evalua- tion results for the entire dataset, without requiring machines to evaluate the remaining data.
F Appendix on More Experiments
To provide more comprehensive understanding. we conduct additional experiments TopiOCQA (Adlakha et al., 2022), where the CIR model is DPR (Karpukhin et al., 2020) and the surrogate model is FiD (Izacard and Grave, 2021) (which are used interchangeably). Here, both models are fine-tuned on the TopiOCQA dataset. The results, shown in Table 5, demonstrate that our HumCoE approach continues to achieve promising results and outperforms the baselines.


Reviews:
[{"review_id": "B5qLCs2kQ7", "reviewer": "Reviewer_Hngp", "summary": "", "questions": "1. Could this method applicable in non-conversational IR task, e.g. traditional ad-hoc search, if not, what should we change or consider?\n\n2. The method is claimed as proposed for CIR system, while the used datasets Coqa are not retrieval-based. Is it conform to the definition?\n\n3. About the model using, why only use Bert and Bart rather than the dense retriever models?\n\n4. Why choose coqa and clariQ datasets only, rather than the conversational search datasets, such as CAsT, TopiOCQA and QReCC?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed feedback and suggestions. In the following, we clarify your concerns point by point. Considering some responses and questions are closely related, we answer them together.\n\n# 1.The differences between the proposed Human-Machine Collaboration annotation approach and the traditional human-annotated approach. (R1)\nWe appreciate your suggestions and will incorporate additional descriptions in the revised version. As for the annotation process, the traditional approach relies solely on humans to annotate the entire dataset for evaluation. In contrast, our Human-Machine Collaboration approach utilizes both human and machine efforts to collaboratively complete the evaluation task, reducing human labor. Specifically, human labor is reduced by only annotating a carefully selected small subset of the data for evaluation. Machines, in this case, are responsible for selecting which data to annotate and how to calibrate the evaluation results to eliminate biases.\n\n# 2.Motivation for CIR. (R2, Q1)\nWe appreciate the reviewer's recognition of the generality of our method. Ideally, our method does have a strong potential for generalization to Information Retrieval (IR) systems. Our motivation for evaluating CIR in this paper lies in two folds. \n\n1. We consider CIR as a special research area in IR. It is preferable to narrow the scope of our research paper to specific scenarios, such as CIR, in order to conduct a more in-depth analysis and provide a more comprehensive evaluation, rather than exploring general scenarios.\n2. Compared to IR, CIR evaluation requires more human involvement due to its multi-turn human-machine interactions, making it more labor-intensive for humans. Therefore, it would be more crucial and necessary to ease the human evaluation labor for CIR. \n\nTherefore, we made a successful attempt and the implementation on the CIR setting. Having demonstrated our effectiveness in CIR, we are certainly interested in applying our method to IR in the future :)\n\n# 3.Reason for the chosen datasets and the models. (R3, Q2, Q3, Q4)\nOur definition of CIR aligns with a recent CIR survey[1], which includes conversational QA (i.e., the CoQA dataset) and clarifying questions as common use cases. In these scenarios, We aim to retrieve the answers and questions from the databases, respectively. Particularly on the clarifying question, our experiments include both retrieval-based and generation-based settings, and the results show our effectiveness.  \n\nFor the sake of convenience in our experiments, we prioritized open-source datasets and models with available checkpoints in these two scenarios. Therefore, based on references [2-4], we used the GODEL model on CoQA for conversational QA evaluation, and BERT, BART, and GPT2 for clarifying question evaluation. To provide a more comprehensive understanding, we conducted additional experiments based on your suggestions. Due to time constraints, we present the results on TopiOCQA [5], where the CIR model is DPR [6] and the surrogate model is FiD [7] (which are used interchangeably). The results demonstrate that our HumCoE approach continues to achieve promising results and outperforms the baselines.\n\n| **CIR** | **#** | **HumCoE (ours)** | **Active Top-K** | **HMCEval** | **MCM** |\n|---------|-------|-------------------|------------------|-------------|---------|\n|   DPR   |     5 |        **82.87%** |           17.05% |    76.19%   |  20.90% |\n|         |    10 |        **77.73%** |           19.43% |  76.19%     |  23.80% |\n|         |    15 |        **82.68%** |           32.58% |      76.19% |  21.30% |\n|         |    20 |        **80.94%** |           24.43% |      76.00% |  23.70% |\n|         |    25 |        **79.81%** |           39.09% |      76.00% |  22.10% |\n|         |    30 |        **79.55%** |           32.58% |      76.00% |  23.60% |\n|   FID   |     5 |        **90.18%** |           30.33% |      72.70% |  15.40% |\n|         |    10 |        **91.98%** |           45.26% |      72.79% |   7.70% |\n|         |    15 |        **90.89%** |           50.28% |      72.78% |   3.80% |\n|         |    20 |        **89.41%** |           65.35% |      72.82% |  10.60% |\n|         |    25 |        **90.62%** |           72.29% |      72.85% |   5.40% |\n|         |    30 |        **90.61%** |           64.11% |      72.83% |   9.20% |\n\n**Reference**\n- [1] Neural Approaches to Conversational Information Retrieval, 2022\n- [2] GODEL: Large-Scale Pre-Training for Goal-Directed Dialog, 2022\n- [3] Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions, 2021\n- [4] Towards Facet-Driven Generation of Clarifying Questions for Conversational Search, 2021\n- [5] TopiOCQA: Open-domain Conversational Question Answering with Topic Switching, 2022\n- [6] Dense Passage Retrieval for Open-Domain Question Answering, 2020\n- [7] Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem is interesting and important, which can reduce human labor for the CIR that is considered as the next generation of search engine. The task definition and evaluation methods are clear.\n\n2. The proposed method is valid and promising, and the experimental results are effective and sufficient, showing the better results than existing studies. The chosen tasks are various enough.\n\n3. The paper structure is good and easy to follow. The implementation details are clear and the analysis are sufficient to draw the conclusion.", "reasons_to_reject": "1. The differences between the proposed Human-Machine Collaboration annotation approach and the traditional human annotated approach is unclear. I suggest to add more description and the statistic information by comparing both, which can obviously view the advantage of  Human-Machine Collaboration annotation approach.\n\n2. The motivation of the method design could be more clear, and explain why it is special for CIR system rather than the traditional IR system.\n\n3. The reason for the chosen datasets and the models are unclear, since they are not directly align to the CIR tasks.\n\nPlease also carefully answer the questions for the authors.", "questions_to_authors": "1. Could this method applicable in non-conversational IR task, e.g. traditional ad-hoc search, if not, what should we change or consider?\n\n2. The method is claimed as proposed for CIR system, while the used datasets Coqa are not retrieval-based. Is it conform to the definition?\n\n3. About the model using, why only use Bert and Bart rather than the dense retriever models?\n\n4. Why choose coqa and clariQ datasets only, rather than the conversational search datasets, such as CAsT, TopiOCQA and QReCC?", "paper_topic_and_main_contributions": "This work mainly addresses the problem of the automatically evaluation for conversational information retrieval, providing a landmark and more labor-effective methods\nfor future human-machine-based CIR evaluation research.\n\nThe main contributions are:\n\n1. Introduce active testing to evaluate CIR system and propose a novel method, called HumCoE, based on selecting representative data for human annotation and carefully calibrating the evaluation results, based on surrogate machine.\n\n2. Verify the effectiveness with empirical studies. The results show that it makes an accurate evaluation at low human labor.\n\nTo sum up, this paper tries to address an important and interesting problem, which is valuable for the CIR research community development. The proposed method is promising, and the experiments seems supportive.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "06gTNl0XmW", "reviewer": "Reviewer_sq7x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and suggestions. In Section 4, we made an effort to provide detailed explanations of our method. However, based on your feedback, we understand that we may not have been able to convey all the details to every reviewer. We hope that our subsequent responses will address your concerns and provide a sufficient understanding of our approach. If any part of our description is unclear, please don't hesitate to ask us for clarification.\n\n# 1.More explanation on methodology, especially on surrogate model (R1)\nOur approach consists of two main steps: data sampling and evaluation calibration. In the data sampling step, we utilize a surrogate model as a user proxy to generate pseudo-labels that aid in selecting representative data for human evaluation. \n\nTo illustrate, let's consider a scenario where we have ten conversational QA data. In the human evaluation process, the human first annotates the ground truth answers and assesses the accuracy of the system's predicted answers. However, due to resource constraints, we now assume that we can only afford to evaluate three of them. To ensure the evaluation results accurately reflect the whole dataset, we aim to select three representative data points that are hard for the CIR system. To achieve this, we employ the surrogate model to generate pseudo answers for all ten data points, allowing us to estimate the probabilities of the CIR system making mistakes without requiring ground truth answers from humans. Based on these probabilities, we sample three representative data and then require for human evaluation. As for the evaluation calibration, to mitigate the potential bias introduced by evaluating only a subset of the data, we incorporate a data weight (i.e., Eq. 3) to calibrate the evaluation results. This helps reduce the impact of hard data on the overall evaluation. Our experiments demonstrate that this simple yet effective method yields promising results.\n\n\n# 2.More explanation on metrics (R2)\nInspired by [1, 2], we assess the performance from two aspects: evaluation consistency and stability (cf. Line 347-366, page 4). Recalling the ten-data example above, evaluation consistency assesses whether the evaluation results, obtained from evaluating three selected data, can accurately represent the evaluation results from the entire dataset. Since our method and baselines involve data sampling, which introduces randomness, we run experiments multiple times and use the averaged evaluation results to measure consistency (i.e., Eq. 5). We further consider measuring the stability of the evaluation results across different runs. We use two metrics, namely variance and squared error of inconsistency of each run, to achieve this. This allows us to identify potential sources of instability.\n\n- [1] Active Testing: An Efﬁcient and Robust Framework for Estimating Accuracy, 2018\n- [2] Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation, 2022\n\n# 3.Sensitivity to the choice of surrogate model (R3)\nThanks for your suggestions. In our paper, we conducted preliminary experiments on the sensitivity of our method to different surrogate models, as shown in Tables 1, 2, and 3. Specifically, we tested our approach using different CIR models and surrogate models interchangeably. The results indicate that our method is effective and robust across different CIR models and surrogate models. \n\nHowever, to address your concerns, we conducted additional experiments using different surrogate models (i.e., **g**) to evaluate the same CIR model. Due to time constraints, we focused on the clarifying question task and used ERNIE, ELECTRA, and DeBERTa as surrogate models. The results demonstrate that: 1) our HumCoE approach outperforms baselines, 2) HumCoE consistently outperforms different surrogate models, and 3) surrogate models with larger model parameters do not necessarily lead to higher performance in terms of evaluation consistency.\n\n| **CIR**         | **#** | **Active Top-K** | **HMCEval** | **MCM** | **Ours (g=ERNIE-base)** | **Ours (g=ERNIE-large)** | **Ours (g=DeBERTa-base)** | **Ours (g=DeBERTa-large)** | **Ours (g=ELECTRA-base)** | **Ours (g=ELECTRA-large)** |\n|-----------------|-------|------------------|-------------|---------|-------------------------|--------------------------|---------------------------|----------------------------|---------------------------|----------------------------|\n| BERT            |     5 | 90.88            | 97.52       | 97.52   | **99.76**               |                **98.44** |                 **98.56** |                  **98.60** |                 **98.29** | **98.43**                  |\n|                 | 10    | 56.80            | 97.52       | 97.52   | **99.69**               |                **99.20** |                 **99.13** |                  **99.46** |                 **98.81** | **99.21**                  |\n|                 | 15    | 53.02            | 97.50       | 97.50   | **99.93**               |                **99.32** |                 **99.38** |                  **99.64** |                 **99.12** | **99.44**                  |\n|                 | 20    | 51.12            | 97.40       | 97.40   | **99.91**               |                **99.56** |                 **99.56** |                  **99.82** |                 **99.11** | **99.47**                  |\n|                 | 25    | 59.07            | 97.32       | 97.32   | **99.75**               |                **99.79** |                 **99.80** |                  **99.77** |                 **99.45** | **99.82**                  |\n|                 | 30    | 60.58            | 97.32       | 97.30   | **99.73**               |                **99.74** |                 **99.63** |                  **99.34** |                 **99.97** | **99.61**                  |\n| avg.performance | -     |            61.91 |       97.43 |   97.43 |               **99.80** |                **99.34** |                 **99.34** |                  **99.44** |                 **99.12** | **99.33**                  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Detailed experiments on three CIR tasks with promising improvement over baselines\n- first approach at leveraging this type of active testing technique for CIR tasks", "reasons_to_reject": "- The methods section (section 4) was a bit confusing and this may have affected my ability to judge the soundness of the approach.  Some of the variables and equations need to be explained more fully.  In particular, I think there needs to be more explanation given about the usage of the surrogate model.  A few running examples would probably help clarify this.\n- I'm also less familiar with the evaluation metrics that are being used here (the stability and consistency ones) and I think that they should be explained a bit more.  Authors could provide references to other works that use these or they could just explain them in more depth.\n- How sensitive are the final humcoe scores to the choice of surrogate model g? I would like to see more discussion or experiments showing how using different models for g (with the same model for f) causes scores to vary.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides a new evaluation method for the CIR task that makes use of active testing techniques to reduce the amount of needed human annotation.  They first select a small set of data to be annotated (found using scoring with a surrogate model) and then weight the final set with a calibration technique from active learning literature.  They compare against a handful of similar active testing approaches and note how theirs compares favorably with human evaluation on this task using consistency and stability metrics.\n\n--------\nNote: I've adjusted my soundness scores during the discussion period.  See discussion with authors regarding main concerns below.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 349 – “very few cost” → very low cost"}, "extra_text": {}}, {"review_id": "8DGRtUOBJT", "reviewer": "Reviewer_sHgf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. However, we cannot agree with the main points of your criticism. Please see our detailed response to your \"Reasons To Reject\". Hope we can clarify your misunderstanding.  If any part of our description is unclear, please don't hesitate to ask us for clarification.\n\n# 1.Limited novelty and appears incremental compared to HMCEval (R1)\nAs noted by ourselves and other reviewers, accurately evaluating a CIR system at a low cost is an urgent problem that requires attention. In response to this problem, we made a successful attempt and implementation based on our proposed active testing method. In addition to the innovation inherent in our method, we hope that the reviewers will pay more attention to the significance of our paper in solving the problem of the CIR evaluation. \n\nHaving experimentally shown our superiority over HMCEval, we would also like to highlight that our method is largely different from HMCEval in terms of the methodology, optimization strategy, and targeted evaluation task, although we are both human-machine collaborative methods for evaluation. We will provide more explanations to delve deeper into their differences.\n\n**Methodology**.  1) **HMCEval** is based on the human-machine collaboration paradigm of **learning to defer**[1-2] (or called rejection learning [3]), where tasks are jointly fulfilled by the machine or deferred to humans in making decisions. HMCEval takes the first step to involve learning to defer to model evaluation task, it requires humans to evaluate a subset of data and machines to evaluate the remaining data. The primary research question in HMCEval, as well as the learning-to-defer studies, is to decide which data should be assigned to humans and which to machines to achieve accurate evaluation results for the whole dataset. The most common approach [4,5] is to assign data with high machine prediction uncertainty to humans, while assigning the rest, which are easier to predict, to machines. This is also the approach adopted by HMCEval. 2) **Our method** follows the human-machine collaboration paradigm of **active testing** [6], where humans evaluate a subset of data. Our research question, as well as the active testing studies, is how to select a subset of data for human evaluation that accurately estimates the evaluation results for the entire dataset. Unlike HMCEval, our method does not require machines to evaluate the remaining data. Instead, we use machines to assist with subset selection. Additionally, Unlike other active testing methods[6,7], our approach incorporates machine assistance to calibrate the evaluation results so that it improves evaluation performance.\n\n**Optimization strategy**. 1) HMCEval formulates a constrained optimization problem (cf. Lines 306-310); 2) We directly sample representative data based on the estimated probabilities, following the active testing studies (cf. Section 4).\n\n**Targeted Evaluation Tasks**. HMCEval is designed for malevolence identification evaluation, while our method aims at evaluating the conversional information retrieval system (cf. Lines 184-188, 306-310, 968-971).\n\n# 2.Treating each sample equally for human labor cost might be a strong assumption (R1)\nWe want to emphasize that the method of treating human labor equally on each data point is widely used and necessary in current research on Active testing [6,7,8], CIR [9,10,11], and human-machine interaction [12-15]. While it would be preferable to have a more detailed estimation of human labor, it is challenging as human labor is not only influenced by the difficulty of the data but also by the complexity of human behavior, as recognized in many IR/CIR studies [16]. Therefore, current studies typically treat human labor equally on each data point as a compromise. We also follow this common practice in our work.\n\n# 3.Recommend to validate the performance of the approach on few other conversational datasets (R2)\nBased on a recent CIR survey [11], conversational QA and clarifying questions are common use cases of CIR, both of which we evaluated in our paper. We validated the effectiveness of our method on different CIR tasks, models, and evaluation metrics, as acknowledged by Reviewer ReMS. However, given the vast scope of research in the CIR field, it would be challenging for us and other researchers to conduct experiments on all tasks. To provide a better understanding, we conducted additional experiments based on the suggestions of Reviewer Hngp. We present the results on TopiOCQA [5], where the CIR model is DPR [6] and the surrogate model is FiD [7] (which are used interchangeably). The results demonstrate that our HumCoE approach continues to achieve promising results and outperforms the baselines.\n\n| **CIR** | **#** | **HumCoE (ours)** | **Active Top-K** | **HMCEval** | **MCM** |\n|---------|-------|-------------------|------------------|-------------|---------|\n|   DPR   |     5 |        **82.87%** |           17.05% |    76.19%   |  20.90% |\n|         |    10 |        **77.73%** |           19.43% |  76.19%     |  23.80% |\n|         |    15 |        **82.68%** |           32.58% |      76.19% |  21.30% |\n|         |    20 |        **80.94%** |           24.43% |      76.00% |  23.70% |\n|         |    25 |        **79.81%** |           39.09% |      76.00% |  22.10% |\n|         |    30 |        **79.55%** |           32.58% |      76.00% |  23.60% |\n|   FID   |     5 |        **90.18%** |           30.33% |      72.70% |  15.40% |\n|         |    10 |        **91.98%** |           45.26% |      72.79% |   7.70% |\n|         |    15 |        **90.89%** |           50.28% |      72.78% |   3.80% |\n|         |    20 |        **89.41%** |           65.35% |      72.82% |  10.60% |\n|         |    25 |        **90.62%** |           72.29% |      72.85% |   5.40% |\n|         |    30 |        **90.61%** |           64.11% |      72.83% |   9.20% |\n\n**Reference**\n- [1] Improving Fairness and Accuracy by Learning to Defer, 2018\n- [2] consistent estimators for learning to defer to an expert, 2020\n- [3] on optimum recognition error and reject tradeoff, 1970\n- [4] on the calibration of multiclass classification with rejection, 2019\n- [5] support vector machine with a reject option, 2008\n- [6] Active testing: An efficient and robust framework for estimating accuracy, 2018\n- [7] Efficient test collection construction via active learning, 2020\n- [8] Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation, 2022\n- [9] Conversational Recommendation: Formulation, Methods, and Evaluation, 2020\n- [10] Conversational Information Seeking: Theory and Evaluation: CHIIR 2022 Half Day Tutorial, 2022\n- [11] Neural Approaches to Conversational Information Retrieval, 2022\n- [12] Interactive information extraction with constrained conditional random fields, 2004\n- [13] MATILDA - Multi-AnnoTator multi-language InteractiveLight-weight Dialogue Annotator, 2021, EACL\n- [14] From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains, 2020, ACL\n- [15] Semi-Automated Data Labeling, 2021, NeurIPS\n- [16] User behavior modeling for Web search evaluation, 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper studies an important and timely problem of evaluating model generations with a limited budget for human annotations. \n\nEvaluations are conducted on multiple tasks such as Conversational Question Answering and generation of Clarifying Questions. Experimental results demonstrate they improve over other baselines for selecting hard samples.", "reasons_to_reject": "The proposed method has limited novelty and appears incremental compared to HMCEval. Additionally, treating each sample equally for human labor cost might be a strong assumption.\n\nWould also recommend to validate the performance of the  approach on few other conversational datasets at least.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method for active testing of model generations named HomCOE. The method actively selects a few model generations to be annotated by humans , to limit the significant cost of human labor for manual evaluation. \n\nThe methodology selects representative data for human annotation by factoring in the hardness of examples. Hardness of examples are measured as the probability of the model making mistakes against pseudo labels. To moderate the effect of only selecting hard examples for evaluation they balance with sample weights.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T7iV1HfrUe", "reviewer": "Reviewer_ReMS", "summary": "", "questions": "For the clarifying question generation task, ChatGPT is used to automatically predict relevance/naturalness scores. But the agreement of ChatGPT with human scores was low based on Krippendorff's alpha. Could the authors comment more on the reliability of using ChatGPT for automatic scoring?\n\nHave you experimented with different surrogate model architectures? Does the choice of surrogate model significantly affect HumCoE performance?\n\nWhat are some scenarios where HumCoE does not approximate full human evaluation well? Is there a theoretical analysis of the lower bounds on human labeling needed?", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed feedback and suggestions. In the following, we clarify your concerns in order.\n\n# Regarding \"Reasons To Reject\"\nWe appreciate the reviewer's positive feedback and valuable suggestions. As mentioned, ChatGPT offers a promising alternative to address the challenge of requiring differentiable evaluation metrics. We will explore other non-differentiable evaluation metrics using ChatGPT to assess the effectiveness more broadly. Additionally, we plan to conduct further error analysis and include it in the journal extension once our work is accepted.\n\n# Could the authors comment more on the reliability of using ChatGPT for automatic scoring? (Q1)\nAs shown in our study (Line 537-565), the consistency between ChatGPT-generated scores and human scores for relevance and naturalness is not high, which aligns with recent findings that point-wise scoring by ChatGPT is unreliable[1]. This suggests that ChatGPT alone may not be a reliable tool for automatic CIR evaluation, at least for relevance and naturalness scoring. One possible way to improve the reliability of ChatGPT's scores is to use pair-wise and list-wise ranking, treating the ranking index as the score for the dataset. However, in our case, we require ChatGPT to provide specific scores (0-5) to align with the manual scores as much as possible. This implies that the point-wise approach is currently the best solution for us to achieve automated evaluation. Fortunately, our method does not rely solely on ChatGPT scores to calculate evaluation results; instead, we use these scores in the data sampling step. The experimental results demonstrate that our method achieves high consistency between human evaluation even with the imperfect scores generated by ChatGPT.\n[1] Uncovering ChatGPT's Capabilities in Recommender Systems, 2023\n\n# Have you experimented with different surrogate model architectures? Does the choice of surrogate model significantly affect HumCoE performance? (Q2)\n\nIn our experiments, We explored surrogate models that differ from the CIR model, including BART and GPT2, BERT, and ERNIE. We are excited to present further discussions and results on this topic.\n\n**More Discussions**. We believe that using surrogate models with different model architectures fundamentally promotes diversity [1]. As mentioned in Lines 246-250, it is crucial to encourage diversity between the CIR model f and the surrogate model g. To illustrate this, consider an extreme example: Our method relies on g to provide pseudo-labels, which allows us to estimate the probability of f making mistakes on each data point and select data with high probabilities of being errors. If f and g are the same, we would not be able to accurately select data since the predictions of g and f would be identical, and the probability of error for each data point would be zero. In such cases, our method would randomly select data without calibration. Therefore, we recommend using different surrogate models when evaluating CIR systems.\n\n**More results**. We conducted additional experiments using different surrogate models (i.e., **g**) to evaluate the same CIR model. We focused on the clarifying question task and used ERNIE, ELECTRA, and DeBERTa as surrogate models. The results demonstrate that: 1) our HumCoE approach outperforms baselines, 2) HumCoE consistently outperforms different surrogate models, and 3) surrogate models with larger model parameters do not necessarily lead to higher performance in terms of evaluation consistency.\n\n| **CIR**         | **#** | **Active Top-K** | **HMCEval** | **MCM** | **Ours (g=ERNIE-base)** | **Ours (g=ERNIE-large)** | **Ours (g=DeBERTa-base)** | **Ours (g=DeBERTa-large)** | **Ours (g=ELECTRA-base)** | **Ours (g=ELECTRA-large)** |\n|-----------------|-------|------------------|-------------|---------|-------------------------|--------------------------|---------------------------|----------------------------|---------------------------|----------------------------|\n| BERT            |     5 | 90.88            | 97.52       | 97.52   | **99.76**               |                **98.44** |                 **98.56** |                  **98.60** |                 **98.29** | **98.43**                  |\n|                 | 10    | 56.80            | 97.52       | 97.52   | **99.69**               |                **99.20** |                 **99.13** |                  **99.46** |                 **98.81** | **99.21**                  |\n|                 | 15    | 53.02            | 97.50       | 97.50   | **99.93**               |                **99.32** |                 **99.38** |                  **99.64** |                 **99.12** | **99.44**                  |\n|                 | 20    | 51.12            | 97.40       | 97.40   | **99.91**               |                **99.56** |                 **99.56** |                  **99.82** |                 **99.11** | **99.47**                  |\n|                 | 25    | 59.07            | 97.32       | 97.32   | **99.75**               |                **99.79** |                 **99.80** |                  **99.77** |                 **99.45** | **99.82**                  |\n|                 | 30    | 60.58            | 97.32       | 97.30   | **99.73**               |                **99.74** |                 **99.63** |                  **99.34** |                 **99.97** | **99.61**                  |\n| avg.performance | -     |            61.91 |       97.43 |   97.43 |               **99.80** |                **99.34** |                 **99.34** |                  **99.44** |                 **99.12** | **99.33**                  |\n\n[1] A Unified Theory of Diversity in Ensemble Learning\n\n\n# What are some scenarios where HumCoE does not approximate full human evaluation well? Is there a theoretical analysis of the lower bounds on human labeling needed? (Q3)\n\nTheoretically, we will try to determine the theoretical bound for our method in the future. We believe diversity analysis may be a good place to start. But, this still requires more effort. \n\nEmpirically, surrogate models with very low diversity (discussed above) or poor performance can undermine the advantages of our method. For instance, if the surrogate model provides pseudo-labels randomly, the calibration step may not function as intended. Nevertheless, it is not difficult to find a surrogate model with good performance and high diversity for evaluating CIR systems, given the abundance of publicly available LLMs. Therefore, we intentionally used an open-source language model in our experimental study, which is more aligned with real-world scenarios and eliminates the need to search for a suitable surrogate model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Addresses the important challenge of designing labor-efficient evaluation methods for conversational systems. Reducing human annotation effort would have significant practical impact.\n\n2. Proposes a novel approach by combining ideas from active learning and human-AI collaboration. Adaptation of active testing for CIR is innovative.\n\n3. Comprehensive empirical analysis on multiple datasets and CIR tasks demonstrates the effectiveness over baselines. The method consistently approximates full human evaluation with minimal labor.\n\nThe problem is timely and the solution is novel. The approach could inspire more work on human-machine collaboration for efficient system evaluation.", "reasons_to_reject": "The weaknesses are minor compared to the contributions. One limitation is the need for a differentiable evaluation metric to estimate sample hardness (relevance scores addressed via ChatGPT).\n\nThe authors could do more error analysis and discuss scenarios where the approach may not work as well. But overall the claims seem properly supported.\n\n", "questions_to_authors": "For the clarifying question generation task, ChatGPT is used to automatically predict relevance/naturalness scores. But the agreement of ChatGPT with human scores was low based on Krippendorff's alpha. Could the authors comment more on the reliability of using ChatGPT for automatic scoring?\n\nHave you experimented with different surrogate model architectures? Does the choice of surrogate model significantly affect HumCoE performance?\n\nWhat are some scenarios where HumCoE does not approximate full human evaluation well? Is there a theoretical analysis of the lower bounds on human labeling needed?", "paper_topic_and_main_contributions": "This paper proposes a human-machine collaborative approach called HumCoE to reduce the amount of human labor needed for evaluating conversational information retrieval (CIR) systems. They propose to utilize active testing for CIR evaluation by strategically selecting representative data for human annotation. This is the first work to apply active testing for CIR evaluation.\nThe proposed HumCoE method selects data based on the probability of inconsistency between the CIR model prediction and pseudo labels from a surrogate model.\nEmpirically results demonstrate that HumCoE produces evaluation results comparable to full human evaluation with minimal human labor (<1%) across different CIR tasks like conversational QA and clarifying questions. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2: \"CIR evaluation methods is still an unaddressed problem\" -> \"CIR evaluation methods are still an unaddressed problem\"\n\nSection 5.1.3: \"we also consider ClariQ in our experiments\" -> \"we also consider ClariQ dataset in our experiments\"\n\nFigure 1: Subcaptions are not readable. Please make them larger.\n\nSome minor grammar/style issues exist but overall the paper is well written."}}]

Author rebuttal:

Meta-review: The paper presents a novel approach to address the problem of automatic evaluation for conversational information retrieval (CIR) tasks. The authors propose a method called HumCoE that leverages active testing to reduce the amount of human labor required for evaluation. The selection of representative data for human annotation is done using a surrogate model, which generates pseudo labels that differ from the CIR model's prediction. The paper presents empirical results, demonstrating that HumCoE can produce evaluations comparable to full human evaluation with much less human labor. The authors apply this method across multiple CIR tasks, showing its versatility and effectiveness.

The paper addresses a significant challenge in CIR research, particularly the labor-intensive aspect of human evaluation. The proposed method brings together active learning and human-machine collaboration, making it an innovative solution in the field. The paper presents a comprehensive empirical analysis across multiple datasets and tasks, demonstrating the effectiveness of the approach compared to baseline methods. The application of active testing in CIR is a novel contribution, and the approach could potentially inspire future research in human-machine collaboration for system evaluation.

While the paper's contributions are notable, there are several areas where it could be improved. One of the main concerns raised by the reviewers relates to the clarity and depth of explanation in the methods section, which seemed to be hard to understand. The use of the surrogate model could be better explained, and it is suggested that a few running examples could help clarify this. There are also questions about the sensitivity of the final HumCoE scores to the choice of the surrogate model. It would be beneficial to include more discussion or experiments showing how using different models for the surrogate model affects the scores. It has also been highlighted the need for a clearer explanation of the evaluation metrics used (or cite relevant works), namely the stability and consistency ones, which will increase the readability. It would be helpful if the authors could provide references to other works that use these metrics, or explain them in more depth. The choice of datasets and models used in the study has also been questioned. The reviewers have asked for more clarity on why certain datasets were chosen, and why only Bert and Bart were used instead of other models. They have also suggested that the authors could provide more detail on scenarios where HumCoE may not work as well, perhaps through additional error analysis.

Despite these areas for improvement, the paper presents an innovative approach to a significant problem in CIR research. The authors are encouraged to address the concerns raised by the reviewers to further strengthen the paper. In particular, the authors should provide more clarification on their methodology, the choice of datasets and models, and the evaluation metrics used. Further experimentation with different surrogate models and a deeper analysis of scenarios where HumCoE may not work well could also enhance the paper. AC acknowledged that the authors tried to address most of these issues during rebuttal.
Decision: Accept-Main