Interview Evaluation: A Novel Approach for Automatic Evaluation of Conversational Question Answering Models
Authors: Xibo Li
Bowei Zou
Yifan Fan
Yanling Li
AiTi Aw
Yu Hong
Venue: EMNLP 2023 Main (2023)
Keywords: Conversational Question Answering
Evaluation metrics
Conversational history
Conversational question generation
Prompting

Abstract: Conversational Question Answering (CQA) aims to provide natural language answers to users in information-seeking dialogues. Existing CQA benchmarks often evaluate models using pre-collected human-human conversations. However, replacing the model-predicted dialogue history with ground truth compromises the naturalness and sustainability of CQA evaluation. While previous studies proposed using predicted history and rewriting techniques to address unresolved coreferences and incoherencies, this approach renders the question self-contained from the conversation. In this paper, we propose a novel automatic evaluation approach, interview evaluation. Specifically, ChatGPT acts as the interviewer (Q agent) with a set of carefully designed prompts, and the CQA model under test serves as the interviewee (A agent). During the interview evaluation, questions are dynamically generated by the Q agent to guide the A agent in predicting the correct answer through an interactive process. We evaluated four different models on QuAC and two models on CoQA in our experiments. The experiment results demonstrate that our interview evaluation has advantages over previous CQA evaluation approaches, particularly in terms of naturalness and coherence. The source code is made publicly available.

Interview Evaluation: A Novel Approach for Automatic Evaluation of Conversational Question Answering Models
Xibo Li', Bowei Zou', Yifan Fan', Yanling Li', Ai Aw?, Yu Hong
'School of Computer Science and Technology, Soochow University, Suzhou, China Institute for Infocomm Research, A SSSRR, Singapore
(cipolee200000210, yifanfannlp, 1i4861988, tianxianner)@gaill com {zou_bowel) aaiti]@i2 --sssrrrddu....
Abstract
has been made in modeling CQA systems in recent years, the evaluation of CQA tasks has received less attention and remains flawed.
Conversational Question Answering (CQA) aims to provide natural language answers to users in information--secking dialogues. Exist- ing CQA benchmarks often evaluate models us- ing pre-collected human-human conversations. However, replacing the model- -prediccd dia- logue history with ground truth compromises the naturalness and sustainability of CQA eval- uation. While previous studies proposed us- ing predicted history and rewriting techniques to address unresolved coreferences and inco- herencies, this approach renders the question self-contained from the conversation. In this paper, we propose a novel automatic evaluation approach, interview evaluation. Specifically, ChatGPT acts as the interviewer (Q agent) with a set of carefully designed prompts, and the the intervie-
The prior CQA evaluation asks model questions in turn based on the held-out human-human con- versations due to the high cost of involving humans during the evaluation process. However, this auto- matic evaluation method utilizes golden answers from the conversation history, disregarding the ac- tual predictions made by the model. It has been noted by Mandya et al. (2020) that it deviates from real-world scenarios. They suggest using the mod- els' own predictions for automatic evaluation. On the other hand, Li et al. (2022a) highlights that us- ing predicted history can introduce issues such as unresolved coreference and incoherence and detect invalid questions by employing a coreference res- olution model (Lee et al., 2018) and subsequently rewrite these questions by substituting incorrect mentions with correct ones.
CQA model under wee (A agent). During the interview evaluation. questions are dynamically generated by Q agent to guide the A agent in predicting the correct answer through an interactive process. We evaluated four different models on QuAC and two models on CoQA in our experiments. The experiment results demonstrate that our interview evaluation has advantages over previ- ous CQA evaluation approaches, particularly in terms of naturalness and coherence. The source code is made publicly available.'
Although the evaluation based on prediction combined with a rewriting mechanism mitigates the gap with real-world scenarios and has been proven to align better with human evaluation, the substitution in rewriting makes the conversational question self-contained, which destroys the coher- ence of the conversation. Furthermore, clarification is a significant feature in dialogues (Yatskar, 2019) 9), where questioners often reformulate their questions if they are not satisfied with the previous answers received. Building upon this observation, we pro- pose novel evaluation method called interview evaluation, which takes into account the dynamic nature of conversations.
1 Introduction
Conversational Question Answering (CQA) ex- pects machines answer questions in conversa- tion given the evidence passage. Existing CQA datasets, such as QuAC (Choi al., 2018), CoQA (Reddy et al., 2019), and DoQA (Campos et al., 2020), are derived from human-human conversa- tions, where the questioner poses questions to gain knowledge a specific topic, and the answerer responds based on the evidence passage and the conversation history. While significant progress
During interview, interviewers often rely on set of pre- pre-prepared questions to assess interviewees. However, they also have the flexibility generate new questions based on the interviewee's answers and provide prompts to guide them toward the cor- rect response. Similarly, our interview evaluation for CQA simulates this human interview process.
"Conresponding author. "hanss//github.com/cppllee interview interview_evaluation

---

Qi is typically generated by a human, taking into account both P and H;-1-
AgSSA
An ideal evaluation method for CQA systems would involve human participation, where humans can reflect on the conversation history and initiate new round of questioning and answering. However, such way accurate but also expensive. As a result, existing benchmarks rely on collections of human- human conversations for automatic evaluation, us- ing golden history as a reference. Nevertheless, Mandya et al. (2020) argue that such evaluations do not truly reflect the model's performance they predictions. However, when incorrect predictions occur in conversational history, it leads to unresolved coreferences and in- coherence the current questions. To tackle this issue, Li et al. (2022a) propose two approaches: "rewriting" or "replacing" invalid questions. In "rewriting". entity names in invalid questions are substituted with correct entities, while in "replac- ing". the invalid question is replaced with context- independent question sourced from CANARD (EI- gohary al., 2019). Figure shows a case of "rewriting" where the question When did they re- lease the album? is rewritten as When did they release In This Light? when the model incorrectly answers the question What was In This Light?.
the prope hiceiiiad
beg Cauda?
soid god
Viether aporece reccoolile)
QN cadi
Wacher Mickii
lnte iiniiaiio saredd Canda?
Whatee the redil
faxil axeed
vien be difl.
overlook the models'
Figure 1: Architecture of interview evaluation.
It involves a Q agent that acts as the interviewer and an A agent representing the CQA model as the interviewee. This setup allows us to dynamically generate questions and prompts, mimicking the interactive nature of real interviews. Specifically, the agent consists of selector and a generator. The selector sequentially chooses the next ques- tion from the question set, assuming the previous question was answered correctly. Otherwise, the generator iteratively generates new questions to prompt the CQA model until a question is marked successful or unsuccessful.
However, the process of rewriting and replacing questions leads to paraphrasing, resulting in the loss of conversational question characteristics. As a result, these approaches still fall short of accurately reflecting the models' performance in real-world scenarios. This raises crucial question: How can preserve the dialogical properties, such reference and omission, in evaluation schemes? This question emphasizes the need maintain the conversational nature of the questions during the evaluation process.
We summarize
contributions follows:
1) We propose novel approach, interview evalu- ation, assess CQA models by simulating realis- tic and interactive interview scenario. This evalua- tion framework incorporates a question-generation generation agent (Q agent) that dynamically generates ques- tions based on the CQA model's predictions.
2) We propose three metrics specifically tai- lored for interview evaluation, allowing for multi- faceted analysis of the models' performance from different perspectives.
3) We first introduce the ability large language models in conducting CQA evaluation without the need for costly data collection of new datasets.
2.2 CQA Models for Verification
To assess the effectiveness of interview evalua- tion and conduct further analysis, we adopt the approach proposed by (Li et al., 2022a) and se- lect four CQA models that exhibit variations in both model structures and training methods. More details of the CQA models are in Appendix B. For comparison,
2 Preliminary
2.1 Existing Evaluation for CQA
The current typical evaluation process for CQA be described as follows: In the i-th dialogue turn, CQA model M is expected answer given question Q based on the provided reference passage P and the conversation history Hi-1 = A,, Qi-1, Q Ar-1). It is worth noting that
initially train these models on QuAC and CoQA, following the original imple- mentations. However, the answer format of CoQA is free-form and may not necessarily be present in the given passage, which makes it impractical to

---

Topi: Editors (on Laglish rock band)
3 Interview Evaluation
What was This ightt
3.1 Interview Process Imitation
The lund und while studbing Music Techandory at Staffardebire lnlvenitty
Interviews have the nature of conversational ques- tion answering, where interviewers typically lect questions from pre-prepared set to assess the interviewee. In addition, interviewers often generate new questions based on the interviewee's responses and provide prompts to guide them to- ward the correct answers. The introduction of in- terview datasets in programming, such as APPS (Hendrycks et al., 2021), has contributed to ad- vancements in the field. Expanding this, Zhang et al. (2023) propose the utilization of key prompts, such compile or runtime error messages de- rived from test results, to enhance debugging and problem-solving in code generation models. To simulate realistic human behavior, Park et al. (2023) introduce agents that rely on generative models. These agents' capabilities, including mem- ory and planning, are evaluated through interviews covering various question categories.
Cuu you tell me the none of album relzaved by the hand?
new
Light www.eoa.........................
When they release albang
alstery match. The question is rewritten by gold bistary coreforeace
9
What like
the group's direction.
o
What about the new dirretion did thrs
CANNOTANSWER
What spectfic sspect of the alba "In This Light and This Eveaing" did they nut Hir?
wothetic
allum,
currect aniwer
corefereace
Inspired by Zhang et al. (2023); Park al. (2023), we propose an evaluation method that simu- lates an interview process for assessing CQA mod- cls. During an interview, the interviewer evalu- ates the interviewee's 's knowledge by assessing the correctness of their answers. In addition, the in- terviewer generates new questions based on the interviewee's responses and provides prompts guide them toward the correct answers when they respond incorrectly or answer with "unknown", In our proposed interview evaluation for CQA, we take the Q agent as the interviewer and the A agent as the CQA model being tested. The Q agent eval- uates the abilities of the A agent from two perspec- tives: (1) the number of prompts required for the A agent to answer question correctly, which reflects their comprehension of questions and utilization of knowledge (i.e., evidence passage), and (2) the ratio of the A agent providing correct answers un- der prompting, indicating their understanding of the conversational context. This ability is further demonstrated by the ratio of transitioning from re- sponding "unknown" to answering correctly. A higher-performing A agent is characterized by the ability to answer questions correctly with fewer prompts and a lower ratio of not being able to an- swer questions correctly under prompts.
subutitate
Figure 2: An example illustrating the generated-histon interview evaluation. The green-colored questions are from the CQA dataset, while bluc-colored ques- tions generated by the agent. Based the CQA model's predictions, the Q agent employs dif- ferent strategies to determine the subsequent question.
apply the posHAE component of HAM. In addi- tion, CoQA lacks rewriting dataset like QuAC, which hinders the implementation of the question rewriting model utilized in ExCorD, As result, our evaluation CoQA is limited to using BERT and GraphFlow.
We adopt different evaluation settings following Li et al. (2022a): Auto-golden for evaluation using golden history, Auto-Pred for evaluation based on model predictions, Auto-Rewrite for evaluation in- volving question rewriting, and Auto-Replace for evaluation utilizing replacing. Subsequently, we assess the performance of CQA models in Auto- golden and Auto-Pred. We also incorporate the human performance results reported by Li et al. (2022a), obtained through interactively question- ing and checking model predictions Amazon Mechanical Turk. Furthermore, we analyze the cor- relation between our proposed interview evaluation and human evaluation.
The framework for interview evaluation is illus- trated Figure 1. The Q agent consists of three modules: a question set, a selector, and a generator.

---

The question encompasses series of sequential questions derived from previously gathered human- human conversations. The selector functions as a state execution mechanism, determining the next question in the set based on specific triggers within the state. The generator generates new questions based on the model's predictions.
conversation cell up to Q The golden history is denoted as:
H (C CT,
(3)
Similar to Hj, the history generated by model is represented
For each conversation, the selector initially se- lects question from the beginning of the pre- collected human question set. If the model (A agent) answers correctly, the selector proceeds to the next question. However, the model answers incorrectly, the generator generates a new question. Notably, in cases where the model answers incor- rectly, the generator persists in generating questions prompts until two exit states are encountered: success and failure. The success state occurs when the model generates a correct answer with higher Fl score between the golden answer than a config- urable threshold. failure state further divided into two scenarios: 1) when the model refuses to answer (e.g., responding with CANNOTANSWER in QuAC or unknown in CoQA), and 2) when the num- ber of newly generated questions reaches the upper limit, yet the model still answers incorrectly.
H; (Co, C,,
(4)
For the original question Q if the model predicts an incorrect answer, new questions are generated to prompt the model. In the j-th turn of prompting, the generator generates
question O based the generated history H;-1, the conversation cell C for Q; and the golden answer A;:
Qf - - - .Ai)
(5)
Corresoondingly, the model predicts an answer A². given the generated history H;-1, the conversation cell Cy and the new question Q/ We name it as imnerated-history interview evaluation.
Af -M (P.
(6)
For comparison, we also explore another way that utilizes the golden history for model evaluation. without the questions generated by the generator and their corresponding model answers. We name it as golden-history interview evaluation. In contrast to the intricate and conflicting nature of the generated history, the golden history represents a simple and ideal scenario of the generated history:
3.2 Methodology and Metric
The centerpiece of interview evaluation is the Q agent, which generates questions to elicit model predictions and guide the model provide cor- rect answers. Rather than disregarding the model predictions and asking the question directly in the subsequent turn, the generator 9 within the Q agent generates multiple questions to steer the model to- wards providing a correct answer corresponding to the original human question. Therefore, unlike pre- vious evaluations, the conversation in the interview evaluation is dynamically generated based on the model predictions. Let Q AJ and C represent the human question, human answer, and golden conversation turn, respectively.
= (P
(7)
The interview evaluation compares models three aspects. Both the golden-history interview evaluation and the generated-hise history interview evaluation utilize three metrics to evaluate CQA models. The first metric is the average number of prompt questions required for each round of answering until the model answers correctly or reaches the maximum number of questions, re- ferred to as Questions Per Round (QPR). Denot- ing the number of questions marked in the success state as N₂ and the number of questions generated by G as Ng. QPR is computed as follows:
= - A7)
(1)
Let Qi represent the j-th new question generated by G for Q;, and let A² denote the model prediction for Q?. Specifically, the model prediction for Q is denoted as AQ. The generated conversation cell for Qi is represented as:
N N QPR
(8)
C. =
(2)
N
where n represents the maximum number of gen- erator prompts. Similarly, we use Ci to denote the
The second metric is the Persistent Failure Rate (PFR), which measures the failure rate of

---

CQA models even after receiving multiple prompts from the Q agent. Denoting the number of ques- tions marked in the failure state as NG, PFR is computed as follows:
is and the maximum history window of A agent (CQA models) set to 2. We implement interview evaluation using both golden history and generated history, with further details provided in Subsection 3.2.
N PFR N₂ N5
Prompt Engineering for Question Generation. We conduct prompt tuning and compare the ef- fect of prompts by manually checking the fluency, plausibility and relevance of generated questions. The final template consists of three components: (a) task definition, which introduces the interview task to ChatGPT and assigns the roles of the inter- viewer and interviewee; (b) background, which a fll-in-the-blank text requiring the inclusion of the conversation history, current question, prediction and golden answer; (c) instruction, which guides the context generated by the language models. The template of prompts for generating new questions as follows:
The third metric is the Answer Conversion Rate (ACR), which measures the rate which the QA model transitions from incorrectly refusing swer (e.g., incorrectly answering with unknown or CANNOTANSWER, indicating non-unanswerable) to correctly answering. Denoting the number of non-unanswerable questions that the model predicts incorrectly as Nus. and the number of times the model correctly answers initially non- unanswerable question under prompting as Nuc, ACR is computed as follows:
ACR = Nuc
(10)
Forget the instruction you have previ- ously received. The following is an inter- view assessment where you play the role of the interviewer and the other person plays the role of the interviewee. The interviewee needs to incorporate conver- sation history to answer the current ques- tion. For questions that the interviewee cannot answer correctly, the interviewer needs to ask new questions with hints to guide the model to the correct answer without revealing it. The conversation history is (history): J. The current question is (cur_question and the interviewee an- swers (prediction), J, however, the golden answer is (golden _anseerr____r.... Then you ask new question with hint:
Nui
Compared with solely using ChatGPT to score CQA model predictions, the evaluation with the three metrics propose for the predictions is more reliable and fair.
4 Experimentation
4.1 Settings
Dataset. We conduct experiments QuAC (Choi al., 2018) and CoQA (Reddy al., 2019).
QuAC comprises 98k question-answer pairs, ob- tained from 13k conversations. It contains 20.2%
unanswerable questions and the average length of answers is 14 tokens. CoQA comprises 127K question-answer pairs obtained from 8k conversa- tions in seven domains. It contains only 1.3% unan- swerable questions and contains more factoid ques- tions, which results in average answer length of 2.7 tokens. Due to the average dialogue length (number of dialogue turns) of CoQA being twice that of QuAC, we curate a test set consisting 400 QuAC dialogues and 200 CoQA dialogues, com- prising total of 2.8k and 2.5k question-answer pairs, respectively.
Then, we fill the necessary information into this template and utilize the ChatGPT API to gen- erate new questions, as shown in Figure 3.
4.2 Main Results
We conduct a comparison between the interview evaluation and the automatic evaluation, namely Auto-golder and Auto-Pred Furthermore, we ana- lyze the alignment between our proposed metrics and the human evaluation in QuAC. The human evaluation process entails conversation compris- ing 8-12 questions posed by an annotator who is unaware of the passage content.
Agent Settings. When the A agent provides the correct answer, the Q agent sequentially selects the next question from the pre-collected conversational question to ask the A agent, otherwise the Q agent will generate new questions to guide the agent to answer correctly, as illustrated in Appendix A. The maximum number of prompts in Q agent
Consistency of Evaluation Metrics. The per- formance of the evaluated models on QuAC and

---

Task Definitios
conversation contest "latersirwer: colbert report about "In interviewee: The Culbert Report was Daily Show...
interview
you
Backgroand
queslion- ""what influcuce did his have?"
Uerv are convenation histery -litory--jon econeesttionn bistory--hise hissory.... Squestien).-ppreee hpwppppreeiinn.............. -aaiiiinni---dddd ipalden anever)
prediction "Thsugh
"What viewership Repart!"
Instraction
interviewer new questiens with hints.. without revealing It..
gulder The arries opeord to strong averaging millian viewers
Template
Input
Ouptut
Figure 3: Process of prompt engineering for question generation using the ChatGPT API.
QuAC
CoQA
BERT
GraphFlow
HAM
ExCorD
BERT
GraphFlow
Auto-golden (FI %)
61.5
65.1
64.4
66.9
78.1
76.1
Auto-Pred (FI %)
53.0
50.7
56.2
60.3
70.1
62.8
Interview Evaluation (QPR)
1.36
1.41
1.43
1.28
1.19
1.18
(PFR %6)
19.7
21.6
19.3
16.0
14.0
21.8
(ACR %)
59.3
54.8
63.8
57.9
23.5
0.
Interview Evaluation (QPR)
1.31
1.37
1.28
1.24
1.16
1.17
(PFR 56)
18.3
20.6
15.4
15.6
12.8
21.8
(ACR %)
54.9
54.5
64.6
56.1
29.7
Human Evaluation (Accuracy %)
82.6
81.0
87.8
87.9
(Li ct al., 2022a)
Table Performance of evaluated CQA models on QUAC and CoQA. Note: indicates interview evaluation based generated history; & indicates interview evaluation based on golden history. The accuracy of Li et al. (2022a)'s evaluation on CoQA is missing as they only report human evaluation in QuAC. Note that model performance is negatively correlated with the values of QPR and PFR.
CoQA is listed in Table 1, and the key observations can be summarized as follows: 1) In the golden- history interview evaluation, when considering the metrics of QPR and PFR, the performance ranking QuAC is ExCorD z HAM BERT > Graph- Flow. However, in the snverrate-histooy interview evaluation, the performance ranking changes due to the significant performance degradation HAM while the rankings other models remain con- sistent. 2) In terms of the ACR metric, whether in the golden-history interview evaluation or the imnerated-history interview evaluation, the perfor- mance ranking on QuAC is HAM > ExCorD > BERT GraphFlow. The ranking of BERT and GraphF-low CoQA is also consistent.
swerable question predictions, which causes drop in overall performance; HAM and ExCorD en- hance the ability to understand the question in a conversational context and yield superior perfor- mance; ExCorD adopts question rewriting mod- ule that generates context-independenr questions based on conversational history; HAM emphasizes the mentions questions and conversation history by special embeddings (turn marker and PosHAE) and uses a history attention mechanism to select the most relevant part from the history. 2) We notice that in the interview evaluation, the ACR improve- ment of HAM is the most obvious, which benefits from the unique history attention mechanism and special embeddings that make it easy for the model to correct the answer from unanswerable to answer- able, and give most suitable passage span as the answer. 3) However, HAM does not perform well in the other two metrics on generated-histor in- terview evaluation. Our analysis is because the history generated by the Q agent and the A agent introduces lot of conffict-laden history content,
We verify that our proposed interview evaluation provide more comprchensive insight into the strengths and weaknesses of CQA models through an analysis of the performance on evaluation met- rics. 1) GraphFlow performs poorly in predict- ing unanswerable questions due to it utilizing a difficultly-calibratee separate network for unan-

---

G
H
Auto-golden (FI %)
60.3
63.7
64.2
65.1
Auto-Pred (F1 %)
50.1
49.1
53.9
$7.7
RAH
Interview Evaluation (QPR)
1.42
1.48
1.52
1.33
241
(PFR %)
16.1
15.6
14.7
13.2
12
3
13
28
Table 2: Performance for answerable questions on QuAC. B: BERT; G: GraphFlow; H: HAM: E: ExCorD. indicates the evaluation based on generated history.
36
36
(a) Turn 1
(b) Turn 2
Toplc: Don Shola (an American professional In Football League)
Figure Overlap degree of new questions for the four CQA models. The numbers within the Venn diagram indicate the count of identical questions generated for the corresponding model by the Q agent.
Did Shota any cuzchre?
Super Bowi lowses
Who were some cuacites that Don Shain hnd loning records against?
affected. Nevertheless, HAM remains competitive in terms of the PFR metric. Its focus on the prompt related to the question enables it to make correct predictions during subsequent rounds of prompts. Therefore, compared to existing CQA evaluation methods, our interview evaluation provides a better assessment of the CQA model's 's ability to contin- uously understand dialogue and answer questions, bringing it closer to real-world scenarios. However, it's s important to note that existing evaluation meth- ods still rely on single round of evaluation due the use of golden history.
Tom Flters 1-6 Raymond Berry (3- R),
Dos Shalls bad Raymond
lastng records against
Berry?
Michaels 55 Vince Lombordi (5-
knswers had againet
Rayaoud Barry (3-5) Michach (5-7-1, Lambardi (5-8).
Figure 4: A bad case for interview evaluation.
which limits both the historical attention mecha- nism and special embeddings and reduces the abil- ity to correct answers from a wrong passage span to correct passage span.
4.3 Case Study
Results of Answerable CQA. Considering the neg- ligible impact of the 1.3% unanswerable questions in QuAC, we only show the results for answerable questions in Table As the ACR metric remains the same for both answerable and overall questions, we refrain from redundant ACR reporting and focus on QPR and PFR. Based on these metrics, the CQA models are ranked as: for QPR. ExCorD > BERT GraphFlow > HAM; and for PFR, ExCorD HAM > GraphFlow > BERT.
We present two cases to illustrate the advantages and disadvantages of interview evaluation. In the first case depicted in Figure 2, the Q agent prompts the CQA model with more specific ques- tions such as Can you tell me the name ... or What specific aspect ... did they not like? when the model provides an incorrect answer or responds with CANNOR In comparison to the Auto- Rewrite approach, interview evaluation leverages the Q agent to guide the model toward providing correct answers, thereby avoiding the invalidation of questions. As a result, interview evaluation not only preserves the coherence of the dialogue but also evaluates the model's 's ability to reconsider the conversation history under the provided prompts. Figure 4 illustrates limitation of the interview evaluation, as it may not fully capture all correct predictions made by the model. In this case, the model accurately predicts Tom Flores (1-6) Ray- mond Berry (3-8), but it is incorrectly misjudged as an incorrect answer because its prediction has a low overlap with all candidate answers. Future studies should focus on optimizing the prediction
We observe a notable decrease in the perfor- mance of GraphFlow from Auto-golden to Auto- Pred. It is attributed to GraphFlow modeling incor- rect predictions in the conversation history, which introduces noise and hampers the understanding of the current question. On the other hand, the perfor- mance of HAM shows only slight changes. This is because HAM incorporates history attention mechanism that selects the most relevant part of the history, helping to mitigate the impact of incorrect predictions. However, in the interview evaluation, where the conversation history is significantly ex- panded, HAM's performance on the QPR metric is

---

Heemap Evalugtion
Intervice Evaltation (Gencetoo-lattrr)
Turn 0
Actraic (h))
PFR (7822%)
Turn
X2
Turn 2
BERT
HAM
GraphFllw ExCorD
GraphFlow
Turn
history)
20
400
800
1200
1600
2000
Generated-history Interview Evaluation
Accorsc (16)
84
PER (S)
82
Tern
Tarn 1
76
GrapAFlew
ExCorD
Figure 7: Accuracy of human evaluation vs PFR of interview evaluation.
BERT
HAM
GraphFlow
Turn
ExCorD
analysis across all four models. Our analysis re- sulted in dataset of 408 examples with conver- sation cell length of least 2, and 37 examples with conversation cell length of at least We considered questions be overlapping if they were an exact match in terms of their text. To visual- ize the degree of overlap, we utilized a Venn dia- gram, which is presented in Figure 5. The diagram demonstrates that questions generated by ChatGPT for different models exhibit low degree of over- lap, and this degree decreases the number of prompts increases. In conclusion, the generated questions used the evaluation process are diverse and effectively reflect the predictions the eval- uated models. This diversity and alignment with model predictions represent key advantage of the interview evaluation approach.
400
1200
1600
2008
Gold-history Interview Evaluation
Figure 6: Number prompts required to correctly an- swer questions QuAC. "Turn O" indicates the mod- els' ability to accurately predict human questions, while "Turn 1" indicates correctly predicting with the prompt of the i-th new question.
QPR
QPR
Spearman -0.32
0.54
-1.0
-0.80
Pearson -0.40
0.20
-0.91
-0.97
Table Pearson and Spearman correlation coefficients between human evaluation and interview evaluation. & and 0 indicate the metrics for interview evaluation based generated history and golden history, respectively.
assessing module and the question generator of the Q agent to account for cases where the model's answer is partially correct.
Golden History vs. Generated History. We con- ducted comparison between two modes of inter- view evaluation: based on golden history and based on generated history. The number of prompts re- quired to successfully answer question for four CQA models on QuAC is reported in Figure 6. The results indicate that all models perform bet- ter on human human-generatte generated questions compared to the new questions generated by ChatGPT. It is evident that the questions requiring ChatGPT-generatee prompts are more challenging for the models, they have been answered incorrectly before. No- tably, we observed that HAM performs worse than the other three models on Turn in the generated- history interview evaluation. However, HAM per-
5 Analysis and Discussion
Adaptability and Flexibility for Interview Evalu- ation. One notable characteristic of interview eval- uation is its ability to generate mediction-awarr questions and can adapt to different evaluated CQA models. To investigate this aspect, analyze the number of overlapping questions generated by ChatGPT across different models on QuAC. Specif- ically, we focused on prompts originating from the same human question and selectively collected con- versation cells that contained these prompts for

---

forms better when answering new questions gener- ated by ChatGPT, benefiting from its history atten- tion mechanism.
Conclusion
This paper presents the concept of interview evalu- ation as novel approach for assessing CQA mod- els. Our interview evaluation framework employs a question-generation agent (Q agent) to dynami- cally generate questions based on the model's pre- dictions, facilitating a more realistic and interactive evaluation scenario. We introduce three metrics specifically designed for interview evaluation sess models from different perspectives. Through our analysis, we compared two modes of interview evaluation: golden history and generated history. We also examined the performance of these modes with automatic evaluations and human evaluation. However, it is important to acknowledge that the interview evaluation has its limitations. In some cases, it overlooks correct predictions when the model's output partially aligns with the correct answer. Besides, it is difficult to evaluate the di- versity and creativity answers provided by the generative model. Future research should focus on optimizing the answer-matching module and the question generator of the Q agent to handle such cases more effectively.
Furthermore, we analyze the PFR performance and compare it with the results of human evaluation, depicted in Figure 7. The results demonstrate that the performance rankings of the golden-history interview evaluation and human evaluation align with each other. However, the performance rank- ings of the generated hissoory interview evaluation and human evaluation are not consistent. Com- pared to human evaluation, interview evaluation introduces Q agent that is aware of model predic- tions and gives prompts. Therefore, the generated- history interview evaluation expands the dialogue history to make history full of conflict. Conse- quently, the generated interview evaluation scenar- ios are more intelligent and complex to better ex- pose the limitations of the model such as HAM.
To gain better understanding of the two eval- uation modes, we analyze their agreement with human evaluation in Table 3. In terms of QPR, both the golden-history interview evaluation and human evaluation exhibit a strong agreement with a Spearman correlation coefficient of -1.0 and a Pearson correlation coefficient of -0.91. However, the correlation coefficients between the generated- history interview evaluation and human evaluation only range between -0.5 and O. It indicates that the relationship between human evaluation and golden- history interview evaluations strong, while the relationship with the sneerated-historg interview evaluation is weak. The same conclusion applics to the PFR metric.
Limitations
In interview evaluation, we determine predic- tion as correct if the F1 score between the golden answer and the model's prediction exceeds pre- defined threshold. However, relying solely on F1 scores can introduce bias towards longer answers and may not capture all correct answers, particu- larly shorter ones.
Additionally, our approach leverages LLMs to dynamically generate questions about model pre- dictions. The introduction of LLMs also im- proves generalization since LLMs can be effec- tively adapted to downstream tasks by building few-shot task-specific prompts. However, LLMs suffer from the issue of overfitting on the train- ing data. Therefore, the quality of the generated questions will also be affected, especially if the original questions contain the latest knowledge that is not used to train the LLMs. Moreover, the new questions generated by LLMs exhibit diversity for different incorrect predictions, and quantifying the difficulty of these questions becomes challenging.
It is important note that the high consistency between human evaluation and golden-history in- terview evaluations does not imply that the inter- view evaluation based on golden history is superior. The reason for this high consistency is that, both of them utilize the golden history, which means the CQA model does not have to account for its previous performance in the conversation. As we previously analyzed, the interview evaluation based on generated history provides better assessment of the model's ability to continuously understand the conversation and the interactive ability, which are not adequately captured by existing CQA eval- uation methods.
A Acknowledgements
The research is supported by National Key R&D Program China (2020YFB 1313601), Na-

---

tional Science Foundation of China (62376182, 62076174) and Institute of Infocomm Research of A*STAR (CR-2021-001).
conversational question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Voiume I: Long Papers), pages 6130-6141.
References
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Kenton Lee, Luheng He, and Luke Zeulemoyer. 2018. Higher-order coreference resolution with coarse-to- fine inference. Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume (Short Papers), pages 687-692 692.
Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Mi- lan Deriu, Mark Cieliehak, and Eneko Agirre. 2020. Doqa: accessing domain-specific faqs via conversa- tional qa. In 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), online, 5-10 July 2020. pages 7302-7314, Association for Computational Linguistics.
Huihan Li, Tianyu Gao, Manan Goenka, and Danqi Chen. 2022a. Ditch the gold standard: Re-evaluating conversational question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 8074 8085.
Yu Chen, Lingfei Wu, and Mohammed J Zaki. 2021. Graphflow: exploiting conversation flow with graph neural networks for conversational machine compre- hension. In Proceedings of the Twenty-Ninth Interna- tional Conference International Joint Conferences on Artificial Intelligence, pages 1230-1236.
Xiang Lisa Li and Percy Liang, 2021. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597.
Eunsol Choi, He He, Mohit Iyyer. Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174-2184.
Xibo Li, Bowei Zou, Mengxing Dong, Jianmin Yao, and Yu Hong. 2022b. Concession-fine learning and coarse-to-fine to-fine retrieval for open-domain conversa- tional question answering. In 2022 IEEE 34th Inter- national Conference on Tools with Artificial Intelli- gence (ICTAI), pages 317-325. IEEE.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Angrosh Mandya, James O' Neill, Danushka Bollegala, and Frans Coenen. 2020. Do not let the history haunt you: Mitigating compounding errors in conversa- tional question answering. Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 2017-2025.
Ahmed Elgohary, Denis Peskov, and Jordan Boyd- Graber. 2019. Can you unpack that? learning to rewrite inestions-in-cconen
In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5918-5924,
Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442..
Dan Hendrycks, Steven Basart, Saurav Kadavath, Man- tas Mazeika, Akul Arora, Fahan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv: 2005:::900
Jin Qian, Bowei Zou, Mengxing Dong, Xiao Li, Aiti Aw, and Yu Hong. 2022. Capturing conversational interaction for question answering via global history reasoning. In Findings of the Association for Com- putational Linguistics: NAACL 2022, pages 2071- 2078.
Jacob Devlin Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W Bruce Croft, and Mohit Tyyer. 2019. Attentive history selection for conversational ques- tion answering. In Proceedings of the 28th ACM
Gangwoo Kim, Hyunjac Kim, Jungsoo Park, and Jac- woo Kang. 2021. Learn to resolve conversational dependency: A consistency training framework for

---

International Conference on Information and Knowl- edge Management, pages 1400-
Algorithm 1 Interview Evaluation
Initialize Selector S, Generator G
Alec Radford, Jeffrey Wu, Rewon Child, David Luan,
Initialize a question set Q collected from the given human-human conversation
Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.
Set the maximum number of prompting to Initialize the index of question in Q
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
5: while i< Q do
6: S selects a question d from Q
7: if QA model answers correctly then
8: Mark qi success state
Siva Reddy, Danqi Chen, and Christopher Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266 266.
9: else
10: Initialize the number of prompting j 0
while no state marked in 4; do G generates new question 12:
11:
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with au- tomatically generated prompts. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235.
13:
if QA model answers correctly then Mark q. as a success state else if j > N or QA model predicts the question is unanswerable then Mark @: as a failure state
14:
15:
16:
Wissam Siblini, Baris Sayil, and Yacine Kessaci. 2021.
17:
end if j -j++
Towards a more robust evaluation for conversational question answering.
18:
Proceedings of the 59th An-
19:
end while
nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer-
20:
if q is marked as failure then
ence on Natural Language Processing (Volume 2:
21:
Show the model golden answer
Short Papers), pages 1028-1034.
22:
end if
23:
end if
Mark Yatskar. 2019. A qualitative comparison of coqa, squad 2.0 and quac. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2318 -2233..
24:
i = i +
25:
end while
GraphFlow: Chen al. (2021) combine the reasoning layer on top of the encoding layer capture rich semantic relations within the con- text. The reasoning layer adopts a recurrent graph neural network to process the output of the encoding layer.
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code genera- tion. arXiv preprini arXiv:2305.0440
A Algorithm for Interview Evaluation
Algorithm illustrates the process of interview evaluation.
Qu et al. (2019) apply a history atten- tion module to assign different weights to his- tory turns and softly select the most relevant previous turns. Besides, The PosHAE is em- ployed to map each token to history answer embedding in the passage, which enhances history answer embedding by incorporating position feature of history turns.
HAM:
B
CQA Models for Test
To verify the effectiveness of our interview evalu- ation, employ the following four CQA models and conduct them on QuAC and CoQA.
To establish it, we construct model by adding multiple linear layers on top of the encoder of BERT (Kenton and Toutanova, 2019). The input comprises the concatenation of two preceding rounds of historical question- answer pairs, along with the current question and the passage. The model is trained to pre- dict the answer span.
BERT:
ExCorD: Drawing upon CANARD (Elgohary al., 2019), Kim al. (2021) introduce a question rewriting model to enhance CQA models' comprehension of conversational con- text.

---

Related Work
C.1
Conversational Question Answering
After several conversational question answering datasets have been proposed, such QuAC (Choi al., 2018) and CoQA (Reddy al., 2019), the studies focusing on conversational question answer- ing (Qu al., 2019; Chen et al., 2021; Kim al., 2021; Qian et al., 2022; Li et al., 2022b) have emerged. Recently, some studies have noticed that the current method of evaluating conversational QA models is flawed. Mandya et al. (2020); Siblini et al. (2021) point out that utilizing golden answer in history deviates from real-world scenarios and proposes to predicted history. Li al. (2022a) points out rewrite these questions which suffer from unresolved coreference and incoherence. Dif- ferent from prior works, in this paper, we propose a novel evaluation, namely interview evaluation, uti- lizing ChatGPT to generate dynamic and coherent dialogue. In the evaluation of CQA systems, prior studies have commonly employed the Fl score as the primary metric, similar to reading comprehen- sion tasks such as SQuAD (Rajpurkar et al., 2016). The FI score indicates the extent of lexical over- lap between the model prediction and the golden answer.
C.2 Prompting
Kojima et al. (2022); Radford et al. (2019) propose to utilize the prompts to direct models toward the given task or topic, thereby reducing the task of generating irrelevant inaccurate answers. The prompts divided into "hard" prompts made from interpretable words and tokens and "soft" prompts consisting of continuous feature vectors such as Prefix-Tune Tunnng (Li and Liang, 2021). (Shin et al., 2020) designs a hard template that acts as a fill-n-the-bllane format where specific slots or placeholders are defined to represent missing infor- mation. This text-based prompt controls the gen- eration of large models. Recent, The development of LLMs such GPT3 (Brown al., 2020) and PaLM (Chowdhery et al., 2022) facilitates the zero- shot instruction. Zero-shot instruction highly effective due to the extensive data used to pre-train large-scale language models, enabling them to de- velop a comprehensive understanding of language and reasoning ability. Thus, we carefully craft hard prompt to control ChatGPT with the instruc- tion following capability to generate the text require.


Reviews:
[{"review_id": "pXab2rbqkX", "reviewer": "Reviewer_5zAa", "summary": "", "questions": "-\tHow can \"reference and omission\" be measured using an evaluation method not covered well in Li et al. (2022)? (lines 147 – 150)\n-\tDoes the generator in the Q agent implicitly know about the questions? For example, during an interview, an interviewer might know the answer to a question, leading them to generate more appropriate subsequent questions. Similarly, does ChatGPT know the answers to every question?\n-\tWhat is the configurable threshold for overlap between the model prediction and the golden answer? Does this threshold significantly affect the performance of the evaluation metrics? (lines 259 – 253)\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your feedback is highly valued and has undoubtedly played a crucial role in enhancing the quality of our work. We have carefully considered each of your suggestions and comments, and we provide our responses to address your concerns below:\n\n\n\n----------Response to Comments----------\n\n\n**Comment #1:** The main concern revolves around the measurement of the appropriateness of the proposed evaluation methods. It is challenging to determine whether QPR, PFR, and ACR align with human evaluation accuracy as shown in Table 1. Furthermore, it is unclear how the human evaluation results in Table 3 were conducted. The authors should provide more details on the human evaluation process and ensure that it reflects the perspective of the suggested evaluation methods. For instance, if evaluating fluency, human evaluators should be asked to assess grammar errors and fluency explicitly. So they can set up the fine-grain questions for showing the comprehensive insights.\n\n\n**Response to #1:** We divide your comment into two aspects: Alignment with Human Evaluation, Details on the Human Evaluation Process, and reply separately.\n\n\n* **Alignment with Human Evaluation**\n\n\nIn our paper (lines 553-566), we have conducted statistical analyses using Pearson and Spearman correlation coefficients to establish the alignment between the interview evaluation metrics and human evaluation. We deeply apologize for any inconvenience caused by the typos that have appeared in the original text, and we will address and correct all the issues in the expanded version. In a word, we observed a strong negative correlation (Spearman's $\\rho$ = -1.0,  Pearson's $r$ = -0.91) in the QPR metrics of the golden-history interview evaluation. The closer the correlation coefficients is to 1 or -1, the more aligned it is, while the closer it is to 0, the less aligned it is. These results demonstrate the alignment between our proposed metrics and human evaluations, supporting the reliability of our evaluation approach.\n\n\n* **Details on the Human Evaluation Process**\n\n\nWe apologize for the oversight in excluding the details of the human evaluation process in our initial submission. The human evaluation metrics for the four CQA models were referenced from Table 2 of the paper Li et al (ACL, 2022) [1]. The human evaluation process described in the referenced paper involved three main steps conducted on the Amazon Mechanical Turk platform. \n\n\nFirstly, the annotators were provided with questions without access to the passage. Secondly, after the conversation end, the annotators were shown with the passage and asked to verify the correctness of the model predictions. Lastly, two additional annotators were involved to validate the annotations.\n\n\nThe referenced paper has been widely used in CQA evaluation, including being referenced in the ACL 2023 tutorial [2] on \"Open Challenges for Conversational Agents' Awareness and Beyond: Evaluation for Conversational Agent's Awareness.\"\n\n\nWe believe that the key advantage of human evaluation lies in introducing the interaction between humans and machines during the evaluation process. This insight inspires us to incorporate this interactivity in our automatic evaluation as well.\n\n\nThank you for acknowledging the specific instances provided. Grammar errors and fluency are indeed crucial dimensions for evaluation. Due to the answers extracted from well-written articles, such as Wikipedia or news sources, we assume that the answers are grammatically correct and fluent. Therefore, the explicit evaluation of grammar errors and fluency is not within the scope of our evaluation.\n\n\n\n----------Response to Questions----------\n\n\n**Question #1:** How can \"reference and omission\" be measured using an evaluation method not covered well in Li et al. (2022)? (lines 147–150)\n\n\n**Response to #1:** We apologize for any confusion. Li et al. (2022) introduced three evaluation methods: human evaluation and two automated methods simulating human evaluation. While human evaluation naturally captures linguistic phenomena like reference and omission in human-machine dialogues, the two automated methods deviate from assessing CQA models based on \"reference and omission\" in conversational questions. In our interview evaluation, we manually analyzed 20 conversations, comprising 141 questions. Among them, 39 questions exhibited omission, and 93 questions involved reference phenomena. This observation confirms the inclusion of reference and omission properties in the our interview evaluatetion.\n\n\n**Question #2:** Does the generator in the Q agent implicitly know about the questions? For example, during an interview, an interviewer might know the answer to a question, leading them to generate more appropriate subsequent questions. Similarly, does ChatGPT know the answers to every question?\n\n\n**Response to #2:** We explicitly provided ChatGPT with the answers to every question.\n\n\n**Question #3:** What is the configurable threshold for overlap between the model prediction and the golden answer? Does this threshold significantly affect the performance of the evaluation metrics? (lines 259–253)\n\n\n**Response to #3:** The threshold represents the level of lexical overlap between predictions and gold answers, aligned with downstream tasks. If a downstream task demands high similarity between predicted and gold answers, the threshold can be increased. In our proposed interview evaluation, if the overlap between prediction and gold answer exceeds the configurable threshold (0.5), the current question is considered successfully answered, and the question generation process is terminated. Therefore, the configurable threshold impacts the QPR and FPR metrics. Configurable parameters are common in evaluations, such as binary classification F-score.\n\n\n\n----------Reference----------\n\n\n[1]Li, Huihan and Gao, Tianyu and Goenka, Manan and Chen, Danqi. Ditch the gold standard: Re-evaluating conversational question answering. In ACL, 2022.\n\n\n[2]Deng, Yang and Lei, Wenqiang and Huang, Minlie and Chua, Tat-Seng. Goal Awareness for Conversational AI: Proactivity, Non-collaborativity, and Beyond. In ACL, 2023 tutorials.1.\n\n\nWe sincerely appreciate the constructive critique provided by the reviewers, which has undeniably enriched the final version of the paper. Thank you for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "One of the significant contributions of this paper is the interesting and straightforward method for evaluating conversational question answering models. Evaluation methods for natural language generation (NLG) tasks are challenging yet crucial, and the suggested idea can be extended to other NLG tasks, such as dialogue systems, due to its simplicity.", "reasons_to_reject": "The main concern revolves around the measurement of the appropriateness of the proposed evaluation methods. It is challenging to determine whether QPR, PFR, and ACR align with human evaluation accuracy as shown in Table 1. Furthermore, it is unclear how the human evaluation results in Table 3 were conducted. The authors should provide more details on the human evaluation process and ensure that it reflects the perspective of the suggested evaluation methods. For instance, if evaluating fluency, human evaluators should be asked to assess grammar errors and fluency explicitly. So they can set up the fine-grain questions for showing the comprehensive insights.", "questions_to_authors": "-\tHow can \"reference and omission\" be measured using an evaluation method not covered well in Li et al. (2022)? (lines 147 – 150)\n-\tDoes the generator in the Q agent implicitly know about the questions? For example, during an interview, an interviewer might know the answer to a question, leading them to generate more appropriate subsequent questions. Similarly, does ChatGPT know the answers to every question?\n-\tWhat is the configurable threshold for overlap between the model prediction and the golden answer? Does this threshold significantly affect the performance of the evaluation metrics? (lines 259 – 253)\n", "paper_topic_and_main_contributions": "The paper proposes a novel evaluation method for conversational question answering models (CQA) by generating related subsequent questions for wrong answers from a CQA model. This approach aims to provide more opportunities for the model to answer correctly by asking detailed and generated questions, simulating an interview-like setting. The authors introduce three metrics to quantify this evaluation: questions per round (QPR, higher is better), persistent failure rate (PFR, lower is better), and answer conversion rate (ACR, higher is better). The proposed evaluation methods are tested on two CQA datasets – QuAC and CoQA – and their performance is compared to human evaluation accuracy. The results demonstrate that the suggested methods partly align with human evaluation results, and the paper also investigates various aspects of the evaluation approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vwrxDAcrAx", "reviewer": "Reviewer_uuh1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We apologize for any reading difficulties caused by the typos in the paper. We take your feedback seriously, and our response is as follows:\n\n\n\n----------Response to Comments----------\n\n\n**Comment #1:** The model's performance depends on the performance of the generator G, but the validity of the questions generated by G has yet to be examined.\n\n\n**Response to #1:** The issue you raised is indeed significant, as ensuring the validity of generated questions poses a challenge. \n\n\nFirst, we compare the metrics obtained after applying method G with human evaluations, demonstrating alignment between our proposed interview evaluation using real dialogue history and human evaluation. In lines 553-566, we discuss the alignment with human evaluation, validating our proposed evaluation metrics.\n\n\nSecond, in our paper, the generator G is ChatGPT. Fang et al (2023) [1] stated that \"we can conclude that ChatGPT generates grammatically corrected sentences that are highly fluent\". Therefore, we hold the belief that the generated question exhibits a high level of grammatical quality. Furthermore, the configuration of G used for generating questions was kept fixed across different models being evaluated. Therefore, any impact resulting from generating grammar errors or lack of fluency in questions can be considered as a systematic error that does not alter the ranking of the evaluated models.\n\n\nThird, our approach involves manual analysis of the questions generated by the generator G. We conducted approximately 20 iterations of prompt tuning, to determine the final prompt. We show several prompts in Table 1 (shown in the Appendix at the end of response), including those with improper instruction settings and lacking information. We check the generated questions in 50 dialogues of each prompt, and compare the effect of prompts by manually checking the fluency, plausibility and relevance of generated questions.\n\n\nFinally, allow me to further elaborate on the necessity of introducing G. The automatic evaluation of the dialogue ability of the CQA model has always faced the problem of static evaluation [2,3]. We introduced interactivity in the automatic evaluation of CQA models by introducing G, and designed three indicators based on this feature.\n\n\nIn future iterations or expanded versions of the paper, we intend to conduct a more extensive quality evaluation, incorporating methodologies to ensure generator quality.\n\n\n**Comment #2:** They have analyzed the model using the three metrics they proposed, but the validity of these metrics themselves has not been adequately examined.\n\n\n**Response #2:** We cite the evaluation accuracy (Acc) scores for the BERT, GraphFlow, HAM, and ExCorD models from \"Ditch the Gold Standard: Re-evaluating Conversational Question Answering\" and calculate Spearman and Pearson correlation scores between these metrics and our QPR and PFR metrics. This demonstrates the alignment between our metrics and human evaluations.\n\n\nOur three metrics are tailored to the interactive nature of interview evaluations, where the Q agent plays a role in guiding the model to provide correct answers. The three metrics are characterized as below:\n\nQPR can be used to measure the average number of interactions required for the model to correctly answer a question. A lower QPR indicates the model's ability to answer questions more quickly.\n\n\nPFR can be used to evaluate the model's capability to successfully respond to questions during interactions. A higher PFR indicates more frequent failures in answering questions, while a lower PFR indicates fewer failures.\n\n\nACR is a dynamic evaluation method. When a model refuses to answer a question, the Q agent generates new probing questions to guide the model towards providing the correct answer. We measure this ability using ACR, which reflects the model's capacity to refine its answers based on new feedback, similar to how humans refine their responses in dialogues (lines 330-340).\n\n\n----------Response to Typos----------\n\n\n**Typos #1:** What is stated in lines 554-558 of the text is inconsistent with Table 3.\n\n\n**Response to #1:** We apologize for any confusion caused by the error in the description of Table 3. The ♣ symbol corresponds to the metrics for interview evaluation based on generated history, while the ♢ symbol corresponds to the metrics for interview evaluation based on golden history. The description in lines 557-558 was incorrect. In terms of QPR, both the golden-history interview evaluation and human evaluation exhibit strong agreement, with a Spearman correlation coefficient of -1.0 and a Pearson correlation coefficient of -0.91. Apologies for any confusion caused by a typo in the formula, where the denominator should be $N_s$ in line 322.\n\n\n\n----------Reference----------\n\n\n[1] Fang, Tao and Yang, Shu and Lan, Kaixin and Wong, Derek F and Hu, Jinpeng and Chao, Lidia S and Zhang, Yue. Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation. In arxiv, 2023.\n\n[2] Li, Huihan and Gao, Tianyu and Goenka, Manan and Chen, Danqi. Ditch the gold standard: Re-evaluating conversational question answering. In ACL, 2022.\n\n[3] Deng, Yang and Lei, Wenqiang and Huang, Minlie and Chua, Tat-Seng. Goal Awareness for Conversational AI: Proactivity, Non-collaborativity, and Beyond. In ACL, 2023 tutorials.1.\n\n\n----------Appendix----------\n\n\n**Tabel 1 Comparison of Generated Questions for Each Prompt**\n\nWe provide three examples of question generation for each prompt, with the \"origin question\" column presenting three questions from the question set, and the \"generated question\" column showcasing three questions generated using the corresponding prompt.\n\n   | Prompt |Origin questions| Generated questions| Performance |\n   | ------ | -------|----|--------- |\n   | You play a interviewer and someone else plays a interviewee, you ask a question, and if you judge the interviewee to answer correctly,. Otherwise you need to prompt the interviewee for the wrong answer and give the interviewee some new hints based on the correct answer without revealing the answer.this is history conversatio{history}\\nThen when you ask {cur_question}, the interviewee answer {cur_answer},however the gold answer is {gold_answer}. | 1.What was In This Light? 2.Did Shula lose to any coaches?3.What happened next in their business? | 1.Interviewer: Welcome to the interview! ...Interviewee: ... . Music Technology at Staffordshire University. 2.Interviewer: Can you tell me about Don Shula's coaching career?Interviewee: ... .  3.Interviewer: Let's dive into our history conversation...Interviewee: ... | negative    |\n| [               {'role':'system','content':\"\"\"This is a turing test. You play a interviewer and someone else plays a interviewee, you ask a question, and if you judge the interviewee to answer correctly,. Otherwise you need to prompt the interviewee for the wrong answer and give the interviewee some new hints based on the correct answer without revealing the answer.\"\"\"}, {'role':'user','content':f\"\"\"this is history conversation{history}\"\"\" },{'role':'system','content':f\"\"\"Then when you ask {cur_question}, the interviewee answer {cur_answer},however the gold answer is {gold answer}.\"\"\"}] | 1.What was In This Light? 2.Did Shula lose to any coaches? 3.What happened next in their business? | 1.Thank you .... Can you tell me about the significance of the Magna Carta in English history? 2.That's close, but not quite the answer I was looking for Don Shula did have Super Bowl losses, but I was actually referring to his regular season record against other coaches... . 3.Thank you for that response. While it is true that Orson Welles worked extensively in radio as an actor, ...? | negative |\n| [{'role':'system','content':\"\"\"This is a turing test. You play a interviewer and someone else plays a interviewee, you ask a question, and if you judge the interviewee to answer correctly, you keep asking questions. Otherwise you need to prompt the interviewee for the wrong answer and give the interviewee some new hints based on the correct answer without revealing the answer. The question is a single-hop question and end the conversation, and should not be redundant.\"\"\"}, {'role':'user','content':f\"\"\"this is history conversation {history}\"\"\"},              {'role':'system','content':f\"\"\"Then when you ask '{cur_question}', the interviewee answer '{cur_answer}',. What question do you ask to prompt interviewee? The question is a single-hop and short!\"\"\"}] | 1.What was In This Light? 2.Did Shula lose to any coaches? 3.What happened next in their business? | 1.Can you provide more details about the band's formation and how they came together while studying Music Technology at Staffordshire University? 2.Can you provide more information about Don Shula's record against Vince Lombardi? 3.Can you provide an example of a radio production that Orson Welles worked on during this time? | negative    |\n| [{'role':'system','content':\"\"\"This is a turing test. You play a interviewer and someone else plays a interviewee, you ask a question, and if you judge the interviewee to answer correctly, you keep asking questions. Otherwise you need to prompt the interviewee for the wrong answer and give the interviewee some new hints based on the correct answer without revealing the answer. The question is a single-hop question and end the conversation, and should not be redundant.\"\"\"}, {'role':'user','content':f\"\"\"this is history conversation {history}\"\"\"},{'role':'system','content':f\"\"\"Then when you ask '{cur_question}', the interviewee answer '{cur_answer}', however the gold answer is '{gold answer}'. What question do you ask to prompt interviewee? The question is a single-hop and short!\"\"\"} ] | 1.What was In This Light? 2.Did Shula lose to any coaches? 3.What happened next in their business? | 1.Can you tell me the name of the latest album released by the band?  2.Can you name any other coaches that Don Shula had losing records against? 3.What famous radio adaptation brought Welles instant fame? | positive |\n\nThank you again for your insightful feedback, which undoubtedly contributes to enhancing the quality and enriching the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method considers the actual conversational situation that the answerer's answer may lead to asking supplementary questions.\n- They propose several evaluation metrics that allow the evaluation of various aspects of the model.", "reasons_to_reject": "- The model's performance depends on the performance of the generator G, but the validity of the questions generated by G has yet to be examined.\n- They have analyzed the model using the three metrics they proposed, but the validity of these metrics themselves has not been adequately examined.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors propose an evaluation architecture for conversational question answering. Existing evaluation methods use a history of manually generated correct answers, which does not match the history predicted by the model. In contrast, the proposed method uses a question generator that generates questions according to the model's predictions without losing naturalness. The proposed method mimics an interview, where the interviewer asks questions to the model. If the model fails to answer, the interviewer generates other questions based on the previous exchanges. The authors also propose a rating scale according to this method.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- What is stated in lines 554-558 of the text is inconsistent with Table 3."}, "extra_text": {}}, {"review_id": "DiovralF0r", "reviewer": "Reviewer_deJG", "summary": "", "questions": "Question A: Following reject reason 1. Could you discuss potential issues related to the proposed Interview Evaluation, such as bias, overfitting, or challenges in understanding and evaluating conversational question answering models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have thoroughly considered each of your suggestions and comments, and we now provide our detailed responses to address each point below:\n\n\n\n----------Response to Comments----------\n\n\n**Comment #1:** The authors propose using large language models to conduct CQA evaluations as an alternative to costly data collection. However, while LLMs are powerful, they can introduce biases present in their training data into their evaluations. Similarly, there could be issues of overfitting, where the model is excessively tailored to its training data and may not generalize well to new, unseen data. Furthermore, the ability of these models to understand and evaluate the aspects of conversational dynamics remains questionable. This lack of a discussion about these potential challenges undermines the paper's argument for the efficacy of using large language models for CQA evaluation.\n\n\n**Response to #1:** We divide your comment into three aspects: Bias, Overfitting and The ability of these models to understand and evaluate, and reply separately.\n\n\n* **Bias**\n\nThe issue you raised is highly relevant, as models like ChatGPT can exhibit certain biases. We believe that ChatGPT, trained on large-scale corpora with extensive knowledge and fine-tuned through RLHF, aligns with human evaluations. Hence, we employ a Q agent consisting of ChatGPT to replace human involvement in the evaluation process. Even in human evaluations, the generation of new questions may vary among individuals and different models, potentially introducing human biases. Our experiments demonstrate the alignment between our evaluation approach and human assessments (lines 552-566).\n\n\n* **Overfitting**\n\n\nWe think that ChatGPT is sensitive to prompts. In our approach, we include instructions such as \"give the interviewee some new hints\" in the prompts to mitigate overfitting and encourage the model to generate novel outputs.\n\n\n* **The ability of these models to understand and evaluate**\n\n\nWe propose two metrics, ACR and QPR, which effectively evaluate a model's understanding in dynamic dialogues. QPR measures the average number of interactions required for the model to correctly answer a question, with lower values indicating faster and more accurate responses. Additionally, in interview evaluations, the Q agent generates new prompting questions to guide the model towards providing correct answers. ACR captures this ability by assessing the model's capacity to refine its answers based on new feedback, mimicking the way humans refine their responses in dialogues (lines 330-340).\n\n\n**Comment #2:** In section 5, Adaptability and Flexibility for Interview Evaluation: The author rely solely on 'exact match' of text to evaluate the diversity, which overlooks other factors of diversity. While this can show some level of diversity, it does not accurately or comprehensively reflect the actual depth of diversity and adaptability that the ChatGPT might be offering. Aspects such as question types, semantic meanings, context relevancy, and many other factors should be considered when evaluating diversity. Also, more details on how diversity correlates with model prediction are needed.\n\n\n**Response to #2:** We fully agree with your perspective. In our analysis, diversity is considered as a subordinate feature, used to assess whether the G agent in interview evaluations is adaptable to different tested models, and doesn’t determine the quality of generated questions. We acknowledge that EM may impose overly strict limitations on diversity assessment. In future work, we plan to incorporate semantic evaluations such as BERTScore to address this limitation. We are actively working on this aspect.\n\n\n\n\n\n----------Response to Questions----------\n\n\n\n**Question #1:** Following reject reason 1. Could you discuss potential issues related to the proposed Interview Evaluation, such as bias, overfitting, or challenges in understanding and evaluating conversational question answering models?\n\n\n**Response to #1:** The potential issues related to the proposed Interview Evaluation is worth discussing. \n\n\nRegarding bias, we believe that, by RLHF, ChatGPT works with a higher similarity to the preference of human, including the behavior of evaluating test results. On the other side, unfortunately, ChatGPT may produce biases similar to those present in the evaluations of human. We argue that their influence might not be severe.\n\n\nRegarding overfitting, it might be a crucial consideration in our evaluation methodology. it can restrict the selection of the question set in the Q agent. If the selected question set includes knowledge specific to the years 2023, our Q agent may encounter challenges in generating appropriate questions. \n\nRegarding the last challenge, we propose ACR and QPR as metrics. These metrics effectively evaluate the model's ability to refine its answers based on new feedback (ACR), and measure the average number of interactions required for accurate responses (QPR).\n\n\n\n\n\n----------Response to Typos----------\n\nI'm glad that you found the suggestions helpful for improving readability. We will make every effort to rectify and improve the quality of future work. Thank you for bringing this to our attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is commendable for its clear and fluent writing style. The authors effectively communicate complex ideas, which significantly aids understanding and the dissemination of their proposed method and findings.\n\n2. The paper explores LLMs' abilities to conduct CQA evaluations, eliminating the need for expensive data collection of new datasets. The proposed idea is novel while practical and easy to implement on evaluating different CQA models. This represents a step forward in making CQA evaluations more accessible and cost-effective.", "reasons_to_reject": "1. The authors propose using large language models to conduct CQA evaluations as an alternative to costly data collection. However, while LLMs are powerful, they can introduce biases present in their training data into their evaluations. Similarly, there could be issues of overfitting, where the model is excessively tailored to its training data and may not generalize well to new, unseen data. Furthermore, the ability of these models to understand and evaluate the aspects of conversational dynamics remains questionable. This lack of a discussion about these potential challenges undermines the paper's argument for the efficacy of using large language models for CQA evaluation.\n\n2. In section 5, Adaptability and Flexibility for Interview Evaluation: The author rely solely on 'exact match' of  text to evaluate the diversity, which overlooks other factors of diversity. While this can show some level of diversity, it does not accurately or comprehensively reflect the actual depth of diversity and adaptability that the ChatGPT might be offering. Aspects such as question types, semantic meanings, context relevancy, and many other factors should be considered when evaluating diversity. Also, more details on how diversity correlates with model prediction are needed.", "questions_to_authors": "Question A: Following reject reason 1. Could you discuss potential issues related to the proposed Interview Evaluation, such as bias, overfitting, or challenges in understanding and evaluating conversational question answering models?", "paper_topic_and_main_contributions": "The paper identifies and addresses a crucial yet unsolved problem of Conversational Question Answering (CQA) task. Current CQA benchmarks use pre-collected human-human conversation to evaluate models performance, however, this method ignores the actual predictions made by model. Also, other method use substitution and rewriting mechanism to mitigate the gap with real-world scenarios destroys the coherence of the conversation. The authors not only identify these flaws in existing evaluation methods but also propose the interview evaluation method which considers the dynamic and interactive nature of conversations, which is often neglected in other evaluation methodologies. There are three new metrics (Questions Per Round, Persistent Failure Rate, Answer Conversation Rate) specifically designed for interview evaluation, providing a multi-dimensional analysis of model performance. The experiment results suggest that the proposed evaluation approach can provide more comprehensive insight into the strengths and weakness of CQA models. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 177: CQA mmodels -> models\n2. Line 384 and 392: It would be more readable if both ranking order are in ascending order or descending order. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper advocates a novel method for evaluating QA models, especially those dialog-oriented: the authors propose an architecture, relying on Chat-GPT, to conduct an "interview" with a QA system. During the interview, Chat-GPT augments a pre-recorded sequence of questions (interview script) with generated hints and paraphrased Qs in order to nudge the system to arrive at a correct answer. This has several advantages: (1) it can reduce the cost of expensive human-based evaluation and (2) it can considerably improve our understanding of QA performance according to different criteria. 

All the reviewers agree that the idea is novel and definitely interesting. Moreover, it is very impactful and this research would be immediately beneficial for the NLG community.

The paper is written very well and provides multiple insights on the proposed metrics, highlighting their meaning, interpreting the results on two datasets and discussing alignment with human judgement. 

Two major issues have been raised by the reviewers. First, all the reviewers have given middle-low reproducibility scores and raised specific technical issues. Generally speaking, the authors provide a lot of technical details in the appendix -- yet, some questions remain.  I believe it is absolutely crucial for evaluation papers to arrive at maximum reproducibility, otherwise effective adoption of proposed metrics by the community is impossible. The authors, however, have very thoroughly clarified these issues in the response. One particular issue: the algorithm depends on the specific prompt template. The template is not presented in the paper -- but was revealed by the authors in the discussion. The omission of the template from the final version would considerably undermine reproducibility. (Figure 7 in the Appendix provides a very good idea of the template, yet it might be insufficient to reproduce the results.)

Second, reviewers uuh1 and dejG argue that the approach relies crucially on the properties of the interviewer (Chat-GPT). This might introduce multiple effects, from non-valid questions to biases etc. This is a more general problem that requires an in-depth discussion -- and the authors have provided some insights in the rebuttal, to be incorporated into the paper. I believe, however, that this issue is not a weakness per se, but rather a starting point for important further research.
Decision: Accept-Main