Just Adjust One Prompt: Enhancing In-Context Dialogue Scoring via Constructing the Optimal Subgraph of Demonstrations and Prompts
Authors: Jiashu Pu
ling Cheng
Lu Fan
Tangjie Lv
Rongsheng Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: dialogue evaluation
in-context learning
large language models
prompt generation
TL;DR: Leverage the in-context learning capability of the LLM and diverse human labels to automatically score dialogues.

Abstract: The use of modern Large Language Models (LLMs) as chatbots still has some problems such as hallucinations and lack of empathy. Identifying these issues can help improve chatbot performance. The community has been continually iterating on reference-free dialogue evaluation methods based on large language models (LLMs) that can be readily applied. However, many of these LLM-based metrics require selecting specific datasets and developing specialized training tasks for different evaluation dimensions (e.g., coherence, informative). The developing step can be time-consuming and may need to be repeated for new evaluation dimensions. To enable efficient and flexible adaptation to diverse needs of dialogue evaluation, we propose a dimension-agnostic scoring method that leverages the in-context learning (ICL) capability of LLMs to learn from human scoring to the fullest extent. Our method has three key features. To begin with, rather than manual prompt crafting, we propose automatically generating prompts, allowing the LLM to observe human labels and summarize the most suitable prompt. Additionally, since the LLM has a token limit and ICL is sensitive to demonstration variations, we train a selector to finely customize demonstrations and prompts for each dialogue input. Finally, during inference, we propose to request the LLM multiple times with a subgraph of demonstrations and prompts that are diverse and suitable to maximize ICL from various human scoring. We validate the efficacy of our method on five datasets, even with a small amount of annotated data, our method outperforms all strong baselines. Code is available at https://github.com/iamlxb3/EMNLP2023-ADOROR.

Just Adjust One Prompt: Enhancing In-Context Dialogue Scoring via Constructing the Optimal Subgraph of Demonstrations and Prompts
Jiashu Pul, Ling Cheng', Lu Fan', Tangjie Lv and Rongsheng Zhang
Fuxi AI Lab, NetEase Inc., Hangzhou, China 2 Zhejjang University, Hangzhou, China
Singapore Management University, Singapore 4 The Hong Kong Polytechnic University, Kowloon, Hong Kong, China
@corp.neteuse.com
lingcheng. 20nchenng2200@phhhess htsp.sg......eeposssssu....uuu. wtw.:/s.....u..u.......... cslfan coaimmmoooppp.uduue.
Abstract
Training
Dirain
The use of modern Large Language Models (LLMs) chatbots still has some problems such as hallucinations and lack of empathy. Identifying these issues can help improve chat- bot performance. The community has been continually iterating on reference-free dialogue evaluation methods based large language models (LLMs) that can be readily applied. However, many of these LLM-based metries re- quire selecting specific datasets and developing specialized training tasks for different evalua- tion dimensions (e.g., coherence, informative).
1.
Manually write and tune one inversion prompt.
Request infer prompts dialogue evaluation from Human examples.
Prompts with the highest Spearman's carrelation humon training
Filled inversion Templete
Few shott dermas
Inserssion prompt
Prompt Candidates
Selected Ps Prompts
1LM
Human Score
LLM score
Filled Dislogue
Scoring Template
elr
The developing step can be time-consuming and may need to be repeated for new evaluation dimensions. To enable efficient and flexible adaptation to diverse needs of dialogue evalua- tion, we propose dimension-agnostic scoring method that leverages the in-context learning (ICL.) capability of L.LMs to learn from human scoring to fullest extent. Our method has three key features. To begin with, rather than manual prompt crafting, we propose automat- ically generating prompts, allowing the LLM to observe human labels and summarize the most suitable prompt. Additionally, since the LLM has token limit and ICI. sensitive demonstration variations, we train selector to finely customize demonstrations and prompts for each dialogue input. Finally, during infer- ence, we propose to request the LLM multiple times with a subgraph of demonstrations and prompts that are diverse and suitable maxi- mize ICL from various human scoring. We vali- date the efficacy of method five datasets, even with small amount of annotated data, our method outperforms all strong baselines. Code is available at EMNLP2023-ADORO
Few shot
Selected
Input
demos
Prompt
Dialogue
X
3. To enhance the in-context learning ability of the LUM, we train a selector I selector that can find the most suitable demonstrations prompts based on different inputs.
Inference
1. Construct the Optimal Sub-Graph of DemonstrationsPPoopp..
Dialogue
Fow Sample from Ps
demos
Optimal Sub-Graph
Sample from
2. Request LLM figtt..s.
Evaluation Score
12g'
LUM
outcomes
Figure I: Overview our ADOROR method.
2023; Yang et al.. 2023), even the most advanced LLMs (such ChatGPT) still exhibit notable flaws, including hallucination (Alkaissi and McFarlane, 2023; Ji et al., 2023) and lack of emotion percep- tion when used as a chatbot (Tlili et al., 2023). By precisely evaluating the dialogue, we can identify areas for improvement and make necessary adjust- ments to enhance the chatbot's performance.
1 Introduction
Although modern language models have made significant strides in language fluency, informa- tiveness, and user understanding (Taecharungroj.
The gold standard for dialogue evaluation is hu- man scoring (Adewumi et al., 2022), but human
Corresponding Author.

---

scoring is subjective (Zhang et al., 2020), time- consuming and expensive (Pang al., 2020; Ych et al., 2021; Mehri and Eskenazi, 2020b). In recent years, many automated reference- free evaluation schemes have emerged; these schemes infuse pre- trained LLMs with prior knowledge at training time by selecting data and designing specific tasks for various evaluation dimensions (Huang et al., 2020; Sinha al., 2020; Zhong al., 2022; Zhang et al., 2022). However, the dimension of interest may change with the evolution of LLMs, for modern LLMs, it is more valuable for us to focus on per- sona consistency, the presence of bias (Ray, 2023), etc. Whenever a new dimension is to be evaluated, previous approaches necessitate two-stage proce- dure of data selection and training task formulation, which is inefficient. Not only that, they only sup- port contexts and responses as input, and cannot imbue specific background information, which can pose challenge when dealing with diverse scoring criteria. As an illustration, if we utilize a chatbot as an academic aide (Tiili et al., 2023), is imper- ative that the chatbot customizes its replies to the various age groups of the students. Responses to younger students ought to be engaging and capti- vating, whereas responses older students should prioritize more formal and proficient tone. Apart from this, in specific use cases, such as those involv- ing medical chatbots, the standards for assessing responses must be custom- desttned (Chow et al., 2023). Our view is that it is unfeasible to prepare an off-the-shelf pre-train-based metric that can ad- equately accommodate all scoring criteria.
strations/prompts. Based on the idea of ensemble learning, we request LLM multiple times with dif- ferent inputs on the optimal subgraph and finally use the averaged score as the output of our dia- logue evaluation system. We validate our method on five datasets and it exceeds all supervised and self-supervised learning baselines.
2 Related Work
In- Co-tett Learning ICL the process by which models learn from context examples during for- ward propagation, without parameter updates. ICL currently performs very well on traditional NLP tasks (Kim et al., 2022; Min et al., 2022), but ICL is still sensitive to how the prompts are written (Zhao et al., 2021), the selection and the order of demon- strations (Lu et al., 2022; Liu et al., 2022). For auto- matic labeling, ICL has shown good results and low cost on tasks such as word sense disambiguation, text summarization, and question generation (He et al., 2023; Wang et al., 2021), I), but using ICL for dialogue evaluation is rare.
Auto Prompt Generation One branch of prompt generation relies paraphrasing of the seed prompt, including back-translation (Jiang al., 2020), synonym substitution (Yuan et al., 2021), etc. Other methods use training data and labels to guide the prompt generation process, but they rely on the gradient of the model (Shin et al., 2020) or the confidence level of the output (Ding et al., 2022; Gao et al., 2021). Unlike the above approaches, we utilize the ability of LLM to learn from human demonstrations and induce appropriate prompts. Our approach easy to use, intuitive, and well- adapted to black-box LLMs.
To enable efficient adaptation to novel evalua- tion dimensions or unique scenarios, we design a new dialogue evaluation method - Enhancing In-Context Dialogue Scoring through constructing the Optimal Sub-Graph Of Demonstrations and Prompts (ADOROR), with an overview in Figure 1, With in-context learning (ICL) its core, our ap- proach requires only a few labeled data and has three main features. First, instead of relying on experts to write prompts, we devise a method of generating prompts that is friendly to black- box LLMs. Second, due to the token limit of the L.M,, we train model to learn how to pair different in- puts with appropriate demonstrations and prompts alleviate the sensitivity problem of ICL. Third, during inference, we aim to maximize the L.LM's ICL capability by exposing it to human labels via constructing an optimal subgraph of diverse demon-
Dialogue Evaluation Most dialogue evaluation methods exploit the self-supervised learning ca- pabilities of LLMs. A small amount of work de- sign methods specifically around coherence (Huang et al., 2020; Ye et al., 2021), but most of the work centers around enabling a pre-trained LLM to perform multidimensional evaluation (Pang et al., 2020; Zhang et al., 2021b). Zhong et al. (2022) con- verts the fine-grained dimension evaluation into a unified Boolean QA problem, training T5 (Raffel et al., 2020) four self-supervised tasks. Sinha et al. (2020) utilizes Noise Contrastive Estimation to train the model and construct one-size-fits-all score. Whereas Mehri and Eskenazi (2020b) take the opposing view that the evaluation of dialogue is diverse and there is no one-size- mee-size-iits-al metric,

---

thus proposing multiple interpretable sub-metrics. Zhang et al. (2022) propose pairwise ranking loss to learn different sub-metrics, and integrate them by metric ensemble and multitask learning as the dialogue-level score. These works bring strong prior knowledge to the LLM via adjusting the training tasks and data, but none of them can be quickly adapted to new scoring criteria. Addition- ally, some works propose using the log probability of the LLM, such BartScore (Yuan et al., 2021) and GPTscore (Fu al., 2023), to evaluate the qual- ity of dialogues. However, for certain evaluation dimensions, these methods rely on customizing specific task descriptions and prompt templates.
Spearman Correlation
FED Dataset
0.7
H
0.5
Overall
Understanding dand
Flexible
Coherent
Informative
Likeable
Depth
Diverse
(a) The box plot of each evaluation dimension encom- passes the outcomes derived from of manually written prompts.
FED-Overall
Spearman Correlation
3 Problem Formulation
0.8
Our goal is to construct a dialogue evaluator that automates human-like scoring. We denote the input of the dialogue evaluator x, which may contain three parts: the conversation context (usually con- tains several rounds of conversation), a response r immediately following the context, and the con- versation background (e.g., knowledge contained, persona information, etc.). Dialogue evaluation is categorized into dialogue-level evaluation, where the evaluator scores the overall context from dif- ferent dimensions (e.g., topic depth, informative, etc.), and tum-level evaluation, where the evaluator scores the next round of response succeeding the context and the possible background Given an input v, the evaluator is expected to output a score ý for a particular evaluation dimension. We adopt the reference- feeerenceeeeeeeeeee setting for both turn-level and dialogue-Ieve evaluation due to the one-to-many characteristic of reference-basee based methods (Zhao et al., 2017). Exhausting all potential responses for reference unfeasible.
0.6
0.4
0.2
0.0
(b) Each column represents different random seed for sampling the demonstrations. Error bars correspond to the results five different test
Figure Results of requesting the LLM via ICL.
demonstrations Xdemo the prompt describing the current task, and the input v. The demonstra- tions Xdemo contains k pairs of labeled samples (( Videmo, 1.... (.ddmm...BBeeemno)).) and we denote Xdemo as demonstrations without human scores. We have a template function T fill all inputs, where the filled template denoted as T [Xdermo P; X). The exact format of template is as follows (see Table 9 for a complete filled template);
[Xiernol [Task Prompt p] [x] Score:.
The template for demonstrations [Xdemo] is:
We denote the labeled dataset as Dtrain ((x1, .i., (Inz vn)),, where y is the average score several human annotators. We define the set of inputs and labels for Dtrain as Di and DV train Given D Di ~ Prr and D train we aim to automate the scoring process of another dataset Diest from the same distribution Pa, ensuring as high consistency as possible between the model's scoring D lost and the humans' scoring D test
[ 1 Score:
[Idemo, Score: [ydemo..)
To obtain the dialogue evaluation score, we re- quest the LLM with T(Xdemo. P, x)) We extract the first floating point number from the LLM's out- put as the predicted score ÿ. We define the LLM scoring function as LCM TX P, + ÿ.
4.2 Inverting Prompts from Training Set In practice. the variance of manually written prompts can be quite large (Figure 2(a)). and it may take us constant trial and error to manually find a suitable prompt. The whole process ineffi- cient and trivial. To reduce the manual workload
4 Method
4.1
Dialogue Scoring via In-Context Learning We employ a black-box LLM to score dialogues via in-context learning. We provide the LLM with

---

of composing prompts, we let the LLM invert suit- able prompts by observing several pairs of labeled examples from Derain at time (limited by the con- text length of the LLM). The templates used for inversion are as follows:
ples (Zhao et al., 2021) (proved by Figure 2(b) well), it is not optimal randomly select demon- strations for each x and apply the same globally preferred prompt to all the samples, nor can we include the entire training set due to the token limit of the LLM. To resolve the uncertainty caused by random sampling, we opt to tailor the combination of demonstrations and the prompt for each input s.. To this end, we think of mining the pairing knowledge directly from the training set Dirain-
[Xdemo] [Inversion Prompt (Pinver)]
Below we provide filled template with simple inversion prompt Pinver:
[omitted few-shot demonstrations xeemol Please design a prompt that will be used to request the language model score the response. Examples of responses and scores are provided for reference. Prompt:
Inversion Prompt:
4.3.1 Dataset Construction
We construct a training set O for training a selector Sselector to choose the optimal prompt and few shot demonstrations for each input x. Specifically, we denote the training set as O (zf. vf",, ..., (2f, ...),...=%%,,, yh,)), ), where no is the size of the O and x! contains the input zy that needs be scored, k-shot demonstrations Xdemo,i and a task prompt p;. The label yf represents the confidence level at which we should select Xaemo, and p; for the given 24. When we feed the LLM with a higher-confidence combination (input as 7 (Xdemmaá: Pt, 22),, the selector faelentor sees that the LLM's scoring will be closer to the human scor- ing. For the exact calculation of yf. please refer to Equation 6. For detailed description of construct- ing O, please refer Algorithm 1.
For brevity, we omit the few shot demonstrations in the above example. For a complete filled tem- plate, please refer to Table 8. For more inversion prompts used in practice, please refer to Table 10. Writing Pinser still requires trial and error but instead of observing the entire training set, you sim- ply verify whether the generated prompt is reason- able for the current evaluation dimension. (An in- adequate inversion prompt may lead L.M to mimic the content in the demonstration, continue the inver- sion prompt, or produce content that is completely unrelated to the current evaluation dimension). In practice, we keep one optimal inversion prompt for each evaluation dimension.
In order to derive l optimal prompts for the LLM to perform ICL for a dialogue scoring task, the initial step involves generating m prompt candi- dates (m is set to be 20 times f). To this end, we request the LLM with m filled inversion tem- plates (the inversion template function is denoted Tinver (Xdemwo: Pinwer)), )), each with different set of few shot demonstrations Xdemo- The best & prompts are selected according the squared er- ror S. (lower is better) on the training set Drrain. which is defined as
Build the training set for the demo and prompt selector fvelector-
Algorithm
Input Training set Dtrain: LLM:
Set of selected prompts Ps: Template function T;
Number of repetitive sampling 7; Output: Data Oz, Oy for seleator O₂ + 00.00 + for 24, E DDrrii do for P; E Ps do
(Derain] Σ
S.
(CCM(T(Xdemo... P. x:)) - y (1)
for 1 to y do
demo + select k random
where the candidate prompt p is the output of re- questing the LLM with Tineer (Xdemor Pinver) and few shot demonstrations Xdemo are randomly se- lected for each p. We denote the set of selected prompts as Ps.
samples from Dtrain \ fai, y
O + -Or U (Xiemos Ps; x;)
i: + LLM(T (NMMeTo Pi, (;));;
O, + O, U (1/ell 1 RB:;
end
end
4.3
Search for the Best Combination of Demonstrations and Prompt
end return O,, O,,
It is believed that ICL is sensitive to the prompt, the selection, and the order of in-context exam-

---

ba
bio
K
output
Demonstration
Input Tower
Two-Layer Transformer Enceder Relative
Lager
Linnar
Lincar
Linvar
Sentence Trandwrrer
Dialogue-1 1
Distogue-2
...
Diatogue-k l
Prompt
Prompt
Embedding
E text
text
Ftest
F text
Poshion Embedding
pos (i-11
pos (l 2)
pos (0)
Closest to Imput
Score Embodding
score
Figure 3:
Main structure of Sselector, diagram of the input and output of the demonstration and input towers. The number in brackets after E pos the input to position embedding layer and represents distance between the current example dialogue and the input The three linear layers independent and do not share parameters.
4.3.2 Demonstration and Prompt Selector
Next we introduce the main function the demonstration tower, which is denoted as Jdemo (Edemo: Impned (p)) -> ha E R It con- sists of an affine transformation ft, for prompt em- bedding fmpnet(() (p) and two-layer transformer en- coder (Vaswani et al., 2017), with each transformer layer composed of multi-heate self-attentin layer followed by feed-forward network. is worth noting that we replace the vanilla self-attention layer with self-attention layer with relative posi- tion (Shaw et al., 2018) 8) to capture the changes in y caused by the different relative positions between the demonstrations. The output of the demonstra- tion tower fdemo is calculated as
We construct the dataset O train selector Seelector -> y². The model Soclector a two- tower model, where Xdemo and æ are fed into two different towers, while the prompt is shared by both. Please refer to Figure 3 for an overview struc- ture of selector.
Demonstration Tower The input of the demon- stration tower fdomo is Xdemo and p. The Xdcmo consists of k pairs of dialogues and scores, for which we do three levels of embedding extraction. Firstly, we use the Sentence Transformer Smppet to convert the score-free demonstra- tion Xdemo into an text embedding matrix: Exext = ( Iddemm..)))) E kk The exact form of input can be found in Table 7.
ha = -poott (ettro/oeeeemmo))) + fú, (mppeeme))())) (2)
where ftron is the transformer encoder, fpoot is the average pooling layer and fi, is a linear layer. We briefly discuss the reasons for introducing both the relative position self-attention layer and the absolute position encoding. Both of them can perceive the changes brought by different permu- tations of demonstrations, but the focus of the captured position information is different. We adopt the relative position layer to learn the effec- tive order information among demonstrations, e.g. demonstration Edemo.a needs to be before Tdemo.b- But the two-tower architecture makes it impossible for the relative position layer to perceive the posi- tional relationship between the demonstrations and the input 24. Thus we adopt the absolute position encoding allow the model to learn positional features when the distance relationship between demonstrations and the input o; matters. These two positional encodings are complementary.
The second level joins the absolute posi- tion information (Devlin et al., 2019). We denote the absolute position embedding as .... .ooss) C TR where fpos
Epos = fpos (ppos).... is the embedding layer. The number pos; refers to the distance between I and x (for the closest position to H, the number pos; equals 0).
The third level is to add the scoring informa- tion. To obtain a legitimate score embedding, we bucket the whole scores D Dirain with Freed- man Diaconis Estimator (Freedman and Diaconis, 1981). Each score y is converted into an inte- ger ÿ. We calculate the score embedding as fol- lows: Expore = fscure (iyaeemmy : Bltemo.k)) E Rixh where Sacore is an embedding layer. Fi- nally we obtain the input of the demo tower as Edemo = (Etext Epos + Escore) C B4xh
We use the all-mpnet-basc-v2 model from www.shert.net

---

Input Tower The structure of the input tower model Femput (fmpper(z); (fmmma(()) + han E Re is simple, which consists of two linear layers f and f1,3, with the output being calculated as hin - J1 (x))e + ft (m)pec()))))
Derain, and integrate multiple results the end. At inference, given an input I, we randomly sample Ç different combinations of Xdemo repeatedly for each prompt in Ps, forming a set of size Ç x [Psl. We re- gard this set as an undirected fully connected graph g. Specifically, we define G = (V, E, W., We), where each node V represents a pair of Xdemo and p. The function W, + yp E IR acquires the confidence value from Ssclector (Xdemor p, We denote We - We(e) IR as an edge map- ping function. The weighted edge (egeg) between nodes represents the discrepancy between node v; and node Uy, which is computed as follows,
(3)
Output and Loss Function
The output of
seelector is calculated as
J" =
= (had hin) 1 (harhin)
(4)
and the loss function is Mean Squared Error:
MSF 1 Σ O0 39)²,
Wo =1 [Ndemoos Xdemooa
(7)
k
where the ground truth label of is calculated as:
where Wo (eig) ranges between [0-1], 1), means that V/ and By do not have any identical demonstrations, and 0 means that v, and uy share exactly the same demonstrations.
Di U/ CCCM( -NN-CCCCMMTTNAee............ (N
We design our Seelector as a two-tower structure for fast inference. The output of fdemo depends only on the training set and can be calculated in advance so that the main inference process is the multiplication of two matrices. Even if the number demonstration combinations reaches 100, 000 or more, the calculation can be done within a second.
We aim to find optimal subgraph g" such that the confidence scores (y") of the nodes in the optimal subgraph are high while satisfying that the discrepancy between nodes is as large as possible. We create a formula for scoring the subgraph g' when a new node i3 is added:
"od' (Wa(es)) (2( b1 Eg'
4.4 Constructing the Optimal Subgraph of Demonstrations and Prompts
Q(
04)
Wo(cg) (W((o))+W((r))
(8)
In practice, we find that for the same input æ. multiple combinations of Xdomo and have close high scores d" at inference. Inspired by this phc- nomenon, we intend to request the LLM multi- ple times, each time with different examples from
We denote "ig as the size of subgraph g'. Combin- ing the above Equation 8 with the summation oper- ation in the parentheses of Equation 9, we can use it to evaluate the overall quality of a subgraph. In the ideal situation, based the Equation 9, where Lg represents the sampling space of G' of size neg-.
find the optimal subgraph
Algorithm 2: Greedy Algorithm
input :Graph g
Optimal subgraph G 0. ng!
ng' Q(Omm ()),)))) (9)
output: : Optimal subgraph g
g' arg arg max g' g'cn
0x arg max max((W(wx)))
meg g - g {u"}:
Due to the NP-hardness of the problem², deter- mining the globally optimal subgraph is not fea- sible within a finite time frame. Consequently, in practice, we resort to the Greedy Algorithm (See Algorithm 2). Once retrieved the optimal subgraph g iterate through all nodes of the optimal sub- graph, obtain no different sets of filled templates
g' + G'U
while 1 < ng and | > 0 do
0¹ arg maxQ v4); wEg
G g {
g' - g [ur}:
end
return gr
'Our problem is a variant of the Maximum Weighted Clique problem (Östergård, 2001), which is NarHard.

---

T[Nemoo p)),, request the LLM, and obtain ng' scores. We simply take the average score as the final score for a certain evaluation dimension.
yet powerful. We set the decoding temperature and the top-p value (Holtzman al., 2020) 0.0 and 1.0 respectively.
5 Experiments
5.2.1 Baselines
5.1 Datasets
D-score (Zhang et al., 2021a) D-score is one of the best self-supervised turn-level evaluation frameworks. It devises a range of evaluation tasks and a multi-task learning framework. In der maximize the performance of the D-score, we adopt the approach described in (Zhao et al., 2020a) and further fine-tune the D-score check- point on the target datasets.
We use five datasets for dialogue evaluation, in- cluding FED (Dialogue) (Mehri and Eskenazi, 2020a), Topical-USR, Persona-USR (Mchri and Eskenazi, 2020b), DailyDialog-Ziie and Persona- Zhao (Zhang al., 2020) (See š A.1 for more de- tails). The FED dataset is evaluated the dialogue level while the other four are evaluated at the turn level. We choose specific evaluation dimensions for each dataset, as shown Table 1 and Table 2. These evaluation dimensions are selected because they have high inter-annotator agreement (with an average Spearman's 's rank correlation coefficient of 0.79), which is guarantee of label reliability and consistency.
(Zhang et al., 2022) We select and it is the SOTA supersupe method for dialogue-level evaluation. It tailors specific pre-training tasks to different evalua- tion dimensions, highlighting a metric ensemble method
FineD-Evalna
Ruber (Tao et al., 2018) Ruber is one of the best supervised methods to perform dialogue evalua- tion, for faireess, changed Ruber's ' backbone from RNN to Roberta Large (Liu et al., 2019). Ruber encodes context and responses separately, so we only experiment with Ruber on the turn- level evaluation datasets.
5.2 Experimental Settings
Since all evaluation datasets contain very few sam- ples (< 1000), we adopt the 5-fold vals-vvaiiation method, in each split, D is partitioned into a train- ing set Drrain and a test set Diest. For each dimen- sion, we report the mean value on the test sets. As the LLM is extremely sensitive to the choice of demonstrations (Figure 2(b)), for those baselines that require random sampling of demonstrations, we repeat the experiments 4 more times (each time with a different random seed for sampling) and report the mean value of these experiments.
GTPScore (Fu et al., 2023) GTPScore is a novel evaluation framework that leverages the emer- gent capabilities of pre-trained generative mod- els to score generated texts in a training-free free manner. For turn-level assessment, we use GTP- Score to evaluate the overall quality of the re- sponses, while for dialogue-level overall assess- ment, we compute the GTPScore by averaging the scores obtained at the turn-level. We choose text-davinci-003 from OpenAl for the experi- ment.
We set the number of demonstrations filling the inveraion-emmplaee Tinver to be 14. After inverting prompts from Dirwin, we keep only four of the best prompts (L is set to 4, thus |Ps| is 4).
For training details, we set the learning rate, the batch size, and the epoch of frelector to le 1e-4, 32, and 50 respectively. The hidden size h for fdemo and fimput set to 768. We chose Adam (Kingma and Ba, 2014) as our optimizer. In the process of constructing O, the number of repetitive sampling y is set to 4; during inference, the number of repet- itive sampling for each selected prompt in Ps is set to 512, rendering the size of & being Ç x [Ps]. We set the size of the optimal subgraph "ggg to 5.
Fine-tuning of Pre-trained Language Models Before the emergence of in-context learning ca- pabilities for LLMs, most tasks in this area fol- low the paradigm of pre-training and fine-tuning. We select the Roberta-L as the fine-tuning model, trained on the classification (based on the bucketed ?)) and regression tasks. The loss func- tions for classification and regression are Cross- Entropy and MSE. The input for supervised base- lines tokenized contexts and responses.
The black-box LLM we request our experi- ments is -pr3.---urb--0001- which is economical
'htps://plafform.oo opwnaiioennniiiommmmm www.e/ppppannnoneeseeeeeeeeeeeee com/doss/moddels/gt---55
"Our results are different from those in the FineD-Eval paper due to the different experimental settings.
*Charge only 2$ per 1, (000, 000 token

---

DDZ
PZ
UTC
UPC
Appropriate.
Appropriate.
Ground.
Overall
Ground.
Overall
D-score
0.523
0.555
0.327
0.478
0.389
0.501
GPTScore (text-davinci-0033
0.427
0.321
Roberta-Large (els)
0.321
0.454
0.403
0.217
0.617
0.237
Roberta-Large (reg)
0.623
0.710
0.482
0.167
0.654
0.563
Ruber-Roberta-Larg
0.202
0,580
0.352
0.497
0.174
0.211
Human-P-Selectee (2.s)
0.662
0.678
0.507
0.474
0.579
0.459
Human-P-Selected (5.s)
0.684
0.679
0.482
0.527
0.690
0,494
Auto-P-Random (5.s)
0.585
0.589
0,460
0.525
0.645
0.571
Auto-P-Selected (5.s)
0.690
0.680
0.508
0.583
0.669
0.610
Auto-P-Selected (5.s with RM25)
0.576
0.643
0.523
0.616
0.658
0.653
Auto-P-Selected Enteembbdd- (5.s)
0.705
0.694
0.485
0.609
0.680
0.602
ADOROR (ours, 5.s)
0.733
0.711
0,529
0.623
0.703
0.637
who Optimal subgraph
0.714
0.709
0.510
0.618
0.687
0.641
Main results for turn-level evaluation. Statistically insignificant scores (p >> 0.05) are italicized. The best results are in bold and the second best are underlined. 'P" stands for prompt. 'cls" and 'reg" stand for classification and regression. 'z,s' means zero-shot. "5.s' means five-shot. 'Appropriate. and 'Ground," are Appropriateness and Groundness (the degree which the response "UPC" are abbreviations for DailyDiaoguue Zhao, Persona Zhao, Jopical-USK, and Persona- respectively, "wlo Optimal subgraph' indicates that we choose the Ndema and p with the highest yo inference.
Table 1:
knowledge from the background). 'DDZ', 'P7', 'UTC', and
Coh.
Div.
Dep.
Lik.
Und.
Inf.
Overall
FineD-Evalmu.
0.604
0.357
0.363
0.374
0.389
0.382
0.442
0.392
GPTScore (fext-davinci-003)
0.279
Roberta-Large (cls)
0.082
0.183
0.267
-0.032
0.082
0.185
0.159
0.047
Roberta-I aarge (reg)
0.299
0.125
0.280
0.165
0.261
0.326
0.285
0.258
(z.s)
0.622
0.260
0,235
0.545
0.439
0.529
0.409
0.590
Humun-P-Selected (5.s)
0.632
0.432
0.451
0.534
0.635
0.617
0.516
0.616
Auto-P-Random (5.s)
0,612
0.366
0.451
0.482
0.590
0.514
0.510
0.613
Auto-P-Scled P-Selected (5.s)
0.559
0,383
0.428
0.502
0.629
0.468
0.490
0.576
Auto-P-Sclectte (5.s with BM25)
0.599
0.352
0.430
0.475
0.533
0.451
0.513
0.591
Auto-P-Seleee P-Selected-EEseeeeee (5.s)
0.613
0.385
0,439
0,510
0.633
0.540
0.534
0.614
ADOROR (5.s, ours)
0.681
0.506
0.554
0,590
0.681
0.582
0.660
0.698
who Optimal Subgraph
0.566
0,512
0.455
0,523
0.640
0.515
0.619
0.649
Table 2: Main results for the FED dataset. 'Coh.', 'Div', 'Dep.. 'Lik',, 'Und', FFle'' and 'Inf.' are abbreviations for Coherent, Diverse, Topic Depth, Likeable, Understanding, Flexible, Informative, and Overall, respectively.
Request LLM with Human Expert Prompts We also test the results of requesting the LLM directly with ing the zero-shot and few-shot cases). For each evaluation dimension, we ask experts write 20 prompts. we keep the best one for each dimen- sion based on the prompt's performance on the training set (according to the Equation 1).
ensemble variant, we request the LLM & times for each sample and take the average score. expert-written prompt (includ-
Inspired by previous work (Rubin et al., 2022), we also test retrieving demonstrations with Okapi BM25 (Robertson al., 2009).
If not specified, all baselines starting with Auto-P and Human-P, their Xdemo is randomly sampled for each (same random seed for different methods).
Request LLM with Inverse-Generated Prompts This branch contains several variants: randomly sampling a generated prompt (Auto- P-Random, not evaluated on Drrain), choosing the prompt that gives the lowest square error on Dtrain (Auto-P-selected), using all prompts in Ps Peeleeed Enngteelee For the
5.3 Results and Analysis
We report the main results (Spearman's rank corre- lation coefficient with human labels) in Table and Table 2, from which we can see that our ADOROR method is significantly better than all other base- lines. It also noticeable that ADOROR with-

---

DDZ
PZ
UTC
UPC
FED
Avg
w/o Relative Position Attention
2.046%
0.000%
-0.562%
-1.908%
-8.164%
-2.536%
w/o Absolute Position Emehdding
-1.637%
-1.828%
-3.387%
1.376%
-6.171%
-2.329%
w/o Score Emebdding
-2.993%
-0.146%
-1.850%
-0.281%
-7.238%
-2.322%
w/o Optimal Subgraph Construction
-2.592%
-0.281%
-2.197%
-0.824%
-9.461%
-3.071%
Table The percentage of performance degradation without certain components for ADOROR.
Min
Max
Average
greater use of the LLM's in-context learning capa- bility. Other factors (including relative position attention layer, absolute position embedding, and score embedding) provide similar benefits. These results show is beneficial to incorporate all these structures into ADOROR.
DDZ-Appropriate
1,57%
4.05%
2.89%
PZ-Appropriate
1.40%
3.15%
2.24%
Gro-ndnus
0.68%
11.76%
5.62%
UTC-Overall
0.53%
6.63%
3.25%
UPC-Groundnes
-0.94%
3.82%
1.56%
UPC-Overall
2.77%
7.82%
5.62%
FED-Coherent
7.24%
23.62%
14.04%
FED-Diverse
-3.61%
31.43%
14.10%
5.5 Sensitivity Analysis for Hyperparameters In Table 4 we analyze the sensitivity of the hy- perparameters T and Ç. for which we perform a grid search, with 7 and Ç searches the ranges (2,4,8) and (256,5 12, 1024) respectively. We report the improvements of ADOROR relative to Auto-P- Selected-Ensene Ensemble. Min, Max, and Average corre- spond the minimum, maximum, and mean values of the percentage of exceeding -eeeet-- Ensemble under all combinations of y and Ç. For a more detailed comparison, please refer to Figure 7, Figure 8 and Figure 9. This result proves that train- ing eeeectorr and building the optimal subgraph at inference are relatively robust.
FHD-Depth
-2.55%
26.23%
9.96%
DiD-Likeable
8.27%
24.97%
15.66%
TED-Understandiin
0.87%
14.73%
7.08%
FED- DeD-Flexible
-6,67%
22.43%
8,27%
FED Infommarneeee
4.75%
23.43%
17.25%
FED-Overall
10.08%
14.12%
12.08%
Table 4: Sensitivity analysis for y and
out constructing an optimal subgraph shows a de- cent improvement over Auto-P-selected and Auto- Eeelected-Ensemblle on almost all evaluation di- mensions (except for Coherence, Flexible of the FED dataset), result illustrates that finding the optimal prompt not good enough, we need to more finely customize the demonstrations and the prompt for each input x, which emphasizes the need for training Sselector.
We also test the sensitivity of ADOROR to dif- ferent few-shot & and conclude that ADOROR sistently outperforms the --Seee-ede method regardless of the few-shot variation, see Figure 10 for detailed experimental results.
Comparing the expert hand-written prompts and the inverse-generaie prompts, find no signifi- cant differences, but the latter requires writing only one inversion prompt, which requires much less manual effort and relies less on expert experience. For the traditional fine-tune paradigm, we see that Roberta-Large (reg) performs best, especially on turn-level datasets, but underperforms on dia- logue datasets. In addition, MSE-bbsed regression tasks are significantly better than cross-entropy- based classification tasks. For different sampling methods, we find that BM25-based retrieval does not have an advantage over random sampling.
Conclusion
Our automated dialogue scoring method allows the LLM to learn from human labels using in-context learning (ICL) in three stages. In the first stage, we let the LLM observe human scoring examples, from which the LLM induces suitable prompts for dialogue scoring. Additionally, the order and se- lection of demonstrations are crucial for ICL, and therefore, we train a demonstration/promyy selec- tor using a new training set created by comparing the LLM and the human scoring. During infer- ence, we provide the LLM with diverse and suitable demonstrations and prompts as possible. Our ADOROR approach flexible and adaptable to new evaluation dimensions or criteria, requiring only a small amount of human-labeled data.
5.4 Ablation Study
We report the results of the ablation study in Ta- ble 3, where we see that the construction of optimal subgraph at inference leads the greatest improve- ment, and we believe that the fusion of the most suitable and diverse demonstratioon/prompps makes

---

Limitations
sion of the number of demonstrations, However, this approach reduces the representational power of the model. We may need to fine-tune the struc- ture that encodes each dialogue to learn how to select the most appropriate demonstrations for the LLM. One way is to view l demonstrations whole for tokenization and then feed l tok- enized Xdemo into LLM that supports modeling long sequences.
Experimental Dataset Regarding the datasets, the dialogues used for evaluation mostly origi- nated from models such Transformer (Vaswani et al., 2017), LSTM (Graves and Graves, 2012), GPT-2 (Radford al., 2019), and others, which are much weaker than the current state-of-the- art LLMs. our method can precisely imitate human scoring when annotating the dialogue his- tory of more powerful chatbots, it would further validate the effectiveness of our approach. To accomplish this, we collecting annotations from fresh set of dialogue data generated by the strongest language models via crowdsoureing. We intend to incorporate additional annotation dimensions, such as persona consistency, verbal authenticity, etc. in the near future, as well as expand our experiments.
Regarding the way in which the LLM is re- quested, we currently require the LLM to out- put a score directly. However, on other tasks, a number of works have proposed adding a Chain- Of-Thought (COT) process to the LLM to allow the model have an explicit thought process before output (Wei et al., 2022). We do not intro- duce COT into our experiments for two reasons: firstly, the evaluation datasets do not contain hu- man reasons for scoring. Second, we did some preliminary experiments and found that using COT instead reduced the correlation between model scoring and human scoring. Why COT is not effective for the task of dialogue evaluation remains to be investigated.
Choice of Large Language Model We have solely experimented with gpt-3.5-turbe for au- tomatic dialogue evaluation in great detail. For other LLMs, also test how GPT-4 (Ope- nAI, 2023) performs on the PersonaZhao dataset. ADOROR achieves a Spearman rank's correla- tion coefficient of 0.817, which is 4.2% higher than the Aute-P-Selected baseline. This experi- mental result preliminarily demonstrates that our method has good LLM adaptability. Due to high costs, the non-a avaiiabbbbyy of open APIs, and the limited capabilities of the language model itself, we have not tested more LLMs with comprehen- sive experiments, such llama (Touvron al., 2023), bard (Rahaman et al., 2023), and so on. As our approach is LLM-agnostie, we anticipate that future researchers can explore more LLMs.
Ethics Statement
Our method pertains to ethics in two ways. Firstly, concerning data confidentiality, we adopt a power- ful third-party LLM to analyze the dialogue data. It may be necessary to de-identify it to avoid pri- vacy violations. The second aspect concerns the LLM's inherent bias. The LLM may exhibit bi- ases towards gender, ethnicity, religion, or specific ideologies (Ray, 2023). The LLM's bias mainly stems from its training data, which is challenging for us as users to eliminate. Even after utilizing our automated scoring method, it is necessary to manually inspect the model's 's output to ensure that it is unbiased in particularly sensitive situations.
Method We did not conduct additional experi- ments to confirm the efficacy of our approach on white-box LLMs. Our method tailored for black-box LLMs, but we anticipate that training ADOROR's s demo selector and ensemble on an optimal subgraph could also enhance white-box LLMs in dialogue scoring (White-box denotes the model that allows access to gradients and confidence score distribution).
A Acknowledgement
Our work is supported by the Key Research and Development Program of Zhejiang Province (No. 2022C01011). We would like to express our special thanks to the four reviewers on OpenReview for their valuable comments. Currently, we are apply- ing our ADOROR method in practice for evaluat- ing Chinese dialogues, and we would like to extend our special thanks to the NetEase crowd-soureing platform? for their strong support.
In addition, to make approach lighter, chose extract an off-the-shelf, fixed-dimension embedding of the dialogue using the Sentence Transformer. This allows us to compress an ar- bitrary dialogue of fewer than 512 tokens into a vector of dimension 768, facilitating the expan-
hhups://ffuxi 163 cottc/oonnnetDttaaUUU

---

References
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
Tosin Adewumi, Foteini Liwicki, and Marcus Liwicki. 2022. State-of-the-art in open-domain conversational ai: A survey. Information, 13(6):298.
Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. Grade: Automatic graph- enhanced coherence metric for evaluating open- domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9230-9240.
Hussam Alkaissi and Samy I McFarlane. 2023. Artifi- cial ballucinations in chatgpt: implications in scien- tific writing. Cureus, 15(2).
James CL Chow, Leslic Sanders, and Kay Li. 2023. Impact of chatgpt on medical chatbots as a disruptive technology. Frontiers in Artificial Intelligence, 6.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023, Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: tre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Haman Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo. and Sang-goo Lee. 2022. Self Gelf-generated in-context learning: Leveraging auto- regressive language models as a demonstration gen- erator. arXiv preprint 280.02200..8882..
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Xi- aobin Wang. Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gos Gee Kim. 2022. Prompt- learning for fine-grained entity typing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6888 (188-6991,A0 Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141..999..
David Freedman and Persi Diaconis. 1981. On the his- togram as a density estimator: 1.2 theory. Zeitschrift für Wahrscheinichheettsheerii und verwandte Gebi- ete, 57(4):453 476.
Yanran Li, Hui Su. Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-tum dialogue dataset. In Proceedings of the Highth International Joint Conference Nat- ural Language Processing (Volume I: Long Papers), pages 986-995.
Jinlan Fu, See- Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint 202.02230..04166
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures. pages 100-114.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 381 3816-3830.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1rXiv:1907.1166
Alex Graves and Alex Graves. 2012. Long short-term memory. Supervised sequence labelling with recur- rent neural networks, pages 37-45.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational I Linguistics (Volume I: Long Papers), pages 8086-8098.
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 379-391.
Xingwei He, Zhenghao Lin, Yeyun Gong, A Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023, Annollm: Making large language models to be better crowdsourced annota- tors. arXiv preprint 230iv:2303. 16854.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsuper- vised evaluation of interactive dialog with dialogpt. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235.

---

Shikib Mehri and Maxine Eskenazi. 2020b. Usr: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707.
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 4684468.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with au- tomatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLIP), pages 4222-4235.
Sewon Min, Mike Lewis. Luke Zettlemoyer, and Han- nanch Hajishirzi. 2022. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 2791-2809.
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L Hamilton, and Joelle Pineau. 2020. Learning an unreferenced metric for online dialogue evaluation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguisticx, pages 2430-2441.
OpenAL. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Patric RJ Östergård. 2001. A new algorithm for the maximum- Beiight-weightt clique problem. Nordic Journal of Computing, 8(4):424-436.
Viriya Taccharungroj. 2023, "what can chatgpt do?" an- alyzing early reactions to the innovative ai chatbot on twitter. Big Data and Cognitive Computing. 7(1):35,
Bo Pang. Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yix- ian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue gener- ation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619-3629.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for au- tomatic evaluation of open-domain dialog systems. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
Ahmed Tlili, Boulus Shehata, Michael Agyemang Adarkwah, Aras Bozkurt, Daniel T Hickey, Ronghuai Huang, and Brighter Agyemang. 2023. What the devil is my guardian angel: Chatgpt as a case study of using chatbots in education. Smart Learning Envi- ronments, 10(1):15.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text- text-to-tex, to-text transformer. Journal of Machine Learning Research,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
21(140))1-67.
Md Saidur Rahaman, MM Ahsan, Nishath Anjum, Md Mizanur Rahman, and Md Nafizur Rahman. 2023. The ai race is on! google's 's bard and openai's chat- gpt head to head: an opinion article. Mizanur and Rahman, Md Nafizur The Al Race is on.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background, applications, key challenges. bias, ethics, limitations and future scope. Internet of Things and Cyber-Physicee Systems.
Shuohang Wang. Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce label- ing cost? gpt-3 can help. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 4 4255
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 3(4):333-389
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-2453
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, pages 2655-2671.
RF Woolson. 2005. Wilcoxon signed-rank test. Ency- clopedia of Biostatistics, 8.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North
Jingfeng Yang. Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and

---

Xia Hu. 2023. Harnessing the power of Iims in prac- tice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304. 13712.
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020a. Designing precise and robust dialogue re- sponse evaluators. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 26-33.
Zheng Ye, Liucun Lu, Lishan Huang, Liang Lin, and Xiaodan Liang. 2021. Towards quantifiable dialogue coherence evaluation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence Natural Language Processing (Volume I: Long Papers), pages 27 1729.
Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020b. Designing precise and robust dialogue re- sponse evaluators. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguisties, pages 26-33, Online. Association for Computational Linguistics.
Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. comprehensive assessment dialog evaluation metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15-33.
Zihao Zhao, Eric Wallace. Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text gener- ation. Advances in Neural Information Processing Systems, 34:27263-2747
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chen Zhang, Luis Fernando D' D'Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li. 2022. FineD- eval: Fine- grainerr automatic dialogue-level evalu- ation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3336-3355, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chen Zhang, Grandee Lee. Luis Fernando D' Haro, and Haizhou Li. 2021a. D-score: Holistic dialogue evalu- ation without reference. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:2502- 2516.
Chen Zhang, João Sedoe, Luis Fernando D' Haro,, Rafael Banchs, and Alexander Rudnicky. 2021b. Automatic evaluation and moderation of open-domain dialogue systems. arXiv preprint arXiv:2111.cm 11.02110..
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018 Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020. Dialogpt: Large- scale generative pre-training for conversational re- sponse generation.
Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-leve diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers), pages 654-664, 664, Vancouver, Canada. Association for Computational Linguistics.

---

A Appendix
Topical-USR and Persona-USR Mehri and Es- kenazi (2020b) created two human evaluation datasets called Topical-USR and Persona-USR. Both datasets use the same annotation scheme, where three dialogue researchers evaluate each context-response pair using six different cate- gories to assess dialogue quality: Understandabil- ity (scored 0-1), Naturalness (scored 1-3), Con- text Maintenance (scored 1-3), Interest (scored 1-3), Knowledge Usage (scored 0-1), and Over- all Quality (scored 1-5). The inter-annotator agreements for cach category were measured using Spearman's 's rank correlation coefficient. For Topical-USR, the scores are 0,5102, 0,4871, 0.5599, 1, 0.7090, and 0.7183, while for Persona-USR, the scores are 0.2984, 0.4842, 0.6125, 0.43 18, 0.8115, and 0.6577.
the appendix, we provide more details of the dataset in $A.1. In addition, we provide a spe- cific example of how to request the LLM to output an evaluation score in Table 9. We also provide the inversion prompts use in our experiments in Table 10 and some examples of automatically generated prompts in Table 11.
Regarding the experimental results, we provide the correlation scores of different prompts on the training and test sets in Figure 4, Figure 5 and Figure 6, exploring whether prompts with better correlation scores on the training set can generalize better on the test set. In Figure 7, Figure 8, and Figure we provide more detailed data for the sen- sitivity analysis of hyperparamettrs.. In Figure 10, we show that our ADOROR method is consistently better than the Auto-P-Selected method for all few shot k setting. For other baseline training details, please refer to $A.2. In addition, provide hy- pothesis testing for method in $A.3, which statistically demonstrate the effectiveness of our core innovation.
FED (Dialogue) FED dataset contains 124 con- versations, with 40 conversations being gener- ated by Meena, 44 by Mitsuku, and another 40 being human-human conversations (Mehri and Eskenazi, 2020a). These conversations were eval- uated using 11 different criteria, with a high level of agreement among the evaluators (Spearman correlations ranging from 0.5 to 0.84).
A.1 Details of the evaluation datasets
For more detailed statistics on the datasets, please refer to Table 5.
A.2 Details regarding other baselines
DailyDialog-Zhar Zhao et al. (2020b) created evaluation dataset called DailyDialog-Zhhar which was based on 100 dialogues randomly se- lected from the test set of the DailyDialog cor- pus (Li et al., 2017). DailyDialog-ZMn ZZhao assesses four criteria: appropriateness, language usage, relevance, and content. Each context-response pair is evaluated by four annotators using a 5- point Likert scale. After removing outliers, the inter- annotaooo agreement for appropriateness was measured using Krippendorff's alpha and found be above 0.8.
D-score has three models trained on different datasets: DSTC6, DSTC7, and PERSONA-CHAT. We pick the checkpoint of DSTC7 for further fine- tuning. For FineD-Eval Eval, we pick the model trained on convai2 for evaluation. When evaluating the Overall dimension using GPTScore, the task de- scription and the aspect definition are omitted. For supervised baselines using Roberta-Large, the la- bels for the classification task obtained by buck- cting the scores with the Freedman Diaconis Esti- mator (Freedman and Diaconis, 1981). For train- ing of these supervised baselines, we split 10% of D Dirain into the validation set Deat- The model with the lowest validation set is evaluated on the test set Dzest- We set epoch, batch size, and the gradient clipping value to 20, 16, and 1.0 respectively. We adjust the learning rate for each model to ensure that the model's loss on the training set converges consistently to below 0.5. We a learning rate of 5e 6 for RUBER-base models and 1e for the other models. The input of supervised baselines is tokenized contexts and responses.
Persona-Zhao Gupta et al. (2019) created an evaluation dataset called Persona-Zhao, which was constructed in a similar way to the DailyDialog-Zhan dataset. The context-response pairs used in Persona-Zhao were obtained from dialogues in the test of the PersonaChat cor- pus (Zhang et al., 2018). In Persona-Zhao, only the appropriateness quality was annotated, and the inter-annotator agrcement for this category is above 0.8, as measured by Krippendorff's ' alpha.

---

DailyDialog-Ze Zhao
times the number of evaluation dimensions of all concerned datasets (multiply by because we perform 5-fold cross-validation). Given two meth- ods, and B, their Wilcoxon Sign Rank test re- sult (p-value) can be represented as fw AA Rg). LSTM Seq2Seq. Random and GPT-2
#instances
900
Avt.#Utts.
4.7
Avg.#t WCg.HCyyHyppp Words
47.5 / 11.0
Type
Turn
#Annotations
14,400
We present the Significance test results in Table 6. From this table. can see that constructing the op- timal subgraph steadily improve the performance and the improvement becomes more pronounced when integrate both modules.
Models
Persona- Zhao
#instances
900
Avg.HUtts.
5.1
Avg/HCyx/lyyp Words
48.8
11.5 Turn
Type
#Annotations
3,600
LSTM Seq2Seq and GPT-2
Models
ToPical-USR
#Instances
360
Avg.#Utts.
11.2
Avg.#Ctx/Hyp Words
236.3
22.4
Type
Turn
#Annotations
6,480
Models
Transformers
Persona-USR
#Instances
300
AUt.
9.3
ACt//Ctx/Hyp Words
98.4
12.0
Type
Turn
#Annotations
5,400
Transformer Seq SSe. LSTM
Models
Memory Network
FED (Dialogue)
#Instances
125
Avg.#Utts.
12.7
ACt/HCtx/HHp Words
113.8/
Type
Dialogue 6,720
#Annotations
Models
Mcena, Mitsuku
Table 5: Details of five evaluation datasets, 'Ctx' and 'Hyp' indicate dialogue context and model hypothesis. 'Utts' indicates utterance.
A.3 Significance tests of ADOROR
Our core contribution is the proposal of two modules training demonstratioon/pommp selector and constructing the optimal subgraph. Here we provide additional results of significance tests to demonstrate the effectiveness of these two modules. We use the Wilcoxon Sign Rank test (Woolson, 2005) to determine if there are significant differ- enees between any two methods, with the calcula- tion function denoted ()) We represent the results of any method m as Rm, (ri....,Pius rn))- where r; represents the the value of Spearman's 's correlation coefficient between a method's ratings and human ratings on a test set, and R equals

---

FED
UTC
UPC
Average
reggie miller, nha hall of famer, ADOROR w'o Optimal subgraph
10.56% (0.009) 2.13% (0.074) 1.00% (0.139)
4.56% (0.0002)
Auto-P-Selected (5.s) 22.72% (0.0001) 5.6% (0.036) 4.7% (0.028) 11.01% (5.542c-07)
Table Average improvement of ADOROR over ADOROR w/o Optimal subgraph and Auto-P-Selected (5.s) with few shot k=5. In parentheses are the p-values of Wilcoxon Sign Rank test.
With Fact
Context: context \nFact: (Fact) InResponse: (response)
With Persona
Context: context| \nPersona: (Perssonan:aPoonnaaa / (resonse))
without Fact or Persona
Context: (coteex) Cecoontext//RRsponnnstttt \nResponse: (reppoon))
Userl: hi do you watch espn ?
User2: hi yes i do sometimes. they cover some nba games
Userl: yes i am huge bucks fan did you see...
(context) example
User2: no did n't. what did he say ?
Userl: that they should n' 't consider signing him agree.
User2: what made him such a bad player last year ?
Userl: he did n't do much for the team and was overpaid.
could n't escape his older sister cheryl 's shadow while in high school. on january 26 1982, he scored a career high 40 points. he tried bragging about it on the car ride home, only to find out cheryl scored 105 points and broken 8 national records
(Fact) example
your persona:
have a children and dogs. own house in florida.
(Persona) example
enjoy american sports.
am male.
(response) example
User2: "m not sure i 'm sure there are some violin out there.
Table 7: In the top column, we present the input form of Sentence Transformer, where we feed the entire multi-round conversation (context) along with the response into Sentence Transformer, outputs vector of fixed length (768), the dialogue embedding. For the USR dataset, the input context contains extra Fact or Persona.

---

Context:
Userl: congratulations are you excited or nervous
User2: i'm excited as my cousin is getting married. do you study or work
Userl: i do both. not tonight though.
User2: wow so good for you yes is weekend enjoy it.
Userl: i work in the morning, as a tour guide.
Response:
User2: yea rock skinny jeans and leggings you appropriateness Score: 1.5 [END]
# 1e
Context:
Userl: awesome am going to college to become a physical therapist.
User2: same when can get my parents of my back
Userl: we should meet sometime after college
User2: yes sounds like great plan
Userl: you still in high school ?
Response:
User2: am good, that is pretty cool. where you work
appropriateness Score: 2.5 [END]
Your objective is to construct a task prompt that can generate an appropriateness score based on the given context and response. The score will determine the quality of the response in terms of appropriateness, and will fall within the range of (1.0, 5.0). The prompt should include the score range. In order to accomplish this, please carefully review the provided few shot demonstrations, note in particular how the above example is scored based context and response. Please include as much detail as possible in the prompt. Please DO NOT include Context and Response in the task prompt.
The created task prompt will be utilized the following manner during inference:
(few shot demonstrations)
(prompt) task prompt:
Table 8: A concrete example of a filled inversion template Tineer requesting the LLM. Here the number of demonstrations set The evaluation dimension is "appropriateness" for the DailyDialogue-Zhao dataset.

---

Context:
Userl: what kind of job do you have?
User2: i walk dogs to pay for college and food
Userl: that is not such a bad job. it sounds fun.
User2: it is not stable enough and it does not pay enough
Userl: but least you get lots of fresh ai.
Response: User2: yes that's true and love fresh air
appropriateness Score: 4.25 [END]
Context:
Userl: i'm twitch streamer and famous at it
User2: that sounds like the casiest money every
Userl: very easy but allergic water
User2: that must be the worst Userl: i'm recluse. what about you" ?
Response:
User2: i'm chef. love to cook.
appropriateness Score: 3.25 [END]
Please rate the appropriateness of the response on a scale of 1.0 to 5.0 based on the given context. The uppropriateness score should reflect how well the response aligns with the context and the conversation as whole. Please consider factors such as relevance, coherence, and tone when assigning the score.
*** 
Context:
Userl: yes, our shop sells watches help with repairs.
User2: very nice maybe i'll stop in sometime. Userl: i would like to meet you?
do you enjoy shopping
User2: i do enjoy shopping i think many women do i must get going now.
Userl: was nice talking to you
Response: User2: same to you have anyone gone with your friend outside ?
appropriateness Score:
Table 9: concrete example of prompt requesting the LLM, which returns the appropriateness score. Here the few-shot is 2 and the dataset is Persona-Zhao. [END] is the end symbol for LLM. We can see that the demonstration is separated hy separator (*) and the prompt also separated by separator (-). practice, we find that adding separators improves the performance of the LLM on in in-context learning.

---

Topical-USR and Persona-USR:
Create a prompt that can generate score between (Score Range] based on the (Evaluation Dimen- sion) of given Response to a (Background Type) and Context. This prompt is solely for evaluating an existing dialogue system during inference, and not for chatbot development or conversation generation. Review the few shot demonstrations provided and note how the example is scored based on the (Background Type], Context, and Response. Do not include the (Background Type), Context, or Response in the prompt. Provide much detail possible. prompt:
MailyDialogue-hhae and Persona-Zhao:
Your objective to construct a task prompt that can generate an (Evaluation Dimension score based on the given context and response. The score will determine the quality of the response in terms of (Evaluation Dimension), ), and will fall within the range of [Score Range). The task prompt should include the score range. In order to accomplish this, please carefully review the provided few shot demonstrations, note in particular how the above example is scored based context and response. Please include much detail as possible in the task prompt. Please DO NOT include Context and Response in the task prompt. The created task prompt will be utilized in the following manner during inference:
(few shot demonstrations) (task prompt) task prompt:
FED:
The dataset to be evaluated includes sets of conversations between human and two open-domain dialog systems, Meena and Mitsuku2. In addition, the dataset also includes human-human dialogues collected in the same environment where one of the humans was selected to play the role of the system. You task now is to create a prompt that evaluates the (Evaluation Dimension) } of dialogue system's 's responses to a user's inputs, with a score within (Score Range). Use the provided few-shot examples to guide the prompt creation in a way that generalizes beyond them. The prompt is solely for evaluating existing dialogue system during inference, NOT for chatbot development or conversation generation. Avoid any user-system conversations in the prompt. DO NOT include system and user in the prompt! The created task prompt will be utilized in the following manner during inference:
(few shot demonstrations)
(task prompt)
prompt:
Table 10: Inversion prompts Pinser used in our experiments. (Score Range), (Evaluation Dimension), J, [Background Type] are placeholders for the score range of the current evaluation dimension, the name of the evaluation dimension (c.g. Coherent), and the type of background knowledge, respectively. There are two types of background knowledge in the USR dataset, Fact and Persona.

---

DailyDialogue-Zhaa Appropriateness:
Please rate the appropriateness of the response on scale of 1.0 to 5.0 based on the given context. The appropriateness score should reflect how well the response fits the context and the conversation flow. A score of 1.0 indicates highly inappropriate response, while a score of 5.0 indicates highly uppropriate response. Please consider the relevance, coherence, and tone of the response when assigning a score.
Persona-Zhan Appropriatenes::
Given a context and response, rate the appropriateness of the response on a scale of 1.0 to 5.0, with 1.0 being completely inappropriate and 5.0 being highly appropriate. Consider the relevance of the response to the context, the coherence of the response, and the overall tone and language used in the response.
Topical-USR Groundness:
Based on the given Persona and Context, how well does the Response demonstrate an understanding of the Persona's 's interests, background, or experiences? Rate the Response on a scale of 0.0 to 1.0, where 0.0 indicates no understanding and 1.0 indicates a strong understanding. Consider the Persona's stated hobbies, family background, job, and other relevant details.
Persona-USR Overall:
Please rate the quality of the response on scale of 1.0 5.0, with 1.0 being poor and 5.0 being excellent. Consider the coherence, relevance, and naturalness of the response in rela- tion the given context and persona. Use your best judgement evaluate the quality of the response.
FED Coherent:
Evaluate the coherence of the dialogue system's responses to user's 's input, with score within (0.0, 2.0). A coherent response should be relevant, informative, and logically connected to the previous input. A score of 2.0 indicates highly coherent response that effectively addresses the user's input, while score of 0.0 indicates a completely incoherent response that irrelevant or nonsensical. Use the provided few-shot examples to guide your evaluation.
FED Topic Depth:
Evaluate the depth of the dialogue system's 's responses to the user's input, with a score within (0.0, 2.0). The depth score measures the level of engagement, coherence, and relevance of the system's responses to the user's input. A score of 0.0 indicates lack of engagement, coherence, and relevance, while score of 2.0 indicates a high level of engagement, coherence, and relevance. The system's responses should demonstrate understanding of the user's input, provide relevant information, and maintain coherent and engaging conversation. Use the provided few-shot examples guide for evaluation.
We present some top-per performing prompts generated by requesting the LLM with the inversion prompt. These prompts are filtered based on proximity to human labels on Drrain-
Table 11:

---

TopicalUSSR-rounddres
Topical-JSt-Overall
0.6-
Spoarman Conee Conelntiinan Tex:
0.5
0.4
0.7
0.325
0.350
0.575
0.000
0-425
0.450
0,475
0.500
0.30
0.35
0.40
0.45
0.50
0.55
0.60
Spearthan Conrlltiionnn Training Sa4
Spaartmn Corralation Tratning 544.
Porsona USR. Groundness
Persona-URR.Ovee
Speammn Conwarten Teat Set
0.70
0.5
Speamman
0.55
0.56
0.58
0.50
0.62
0.64
0:56
0.68
0.48
0.50
0.52
0.54
0.56
0.58
0.50
0.62
Spearment Commlatioon Thaining Set
Speer
Correlation on Thaining Set
Figure 4: Each point corresponds to a prompt in Ps. With each prompt, traverse the complete training and test sets, each time requesting L.LM with the demonstration, prompt, and input that has been filled the template. Based the requested of evaluation scores, calculate the corresponding Spearman's 's rank correlation coefficient. The X-axis represents the correlation on the training set and the Y-axis represents the correlation on the test set. We find clear pattern the USR data set, where prompts with better correlation the training set also correlates better with Human in the test set.
Uailydialog-Zhao/e hha.ppggisseeeeppoooittteeess
Persona- Persona-Zhac-Apornee hrtepmmm/hhh-ppppppriiteeeett
0.7
0.6
0.5
0.5
0.6
53
5.6
35
2.6
27
02
0.1
2.4
0.5
0.6
87
Spcarman Concelaioo on Training Set
Spcorman Conclation on Training Set
(a)
Figure 5: The Dailydialog Zhao and Persona-Zhao datasets: association of Spearman's 's rank correlation coefficient on training and test sets. can see a linear relationship between Spearman's 's rank correlation coefficient the test and training sets, with slowing upward trend in the tail of the x-axis, but very stable overall trend.

---

FED FED-Coherent
FED Diverse
0.--
Sparmman Candlrtee se Test Set
0.7-
0.4
0.6-
0.2
0.4
0.0
0.2
0.525
0.500
0,575
0.600 025 0,600
0.65
0,700
0.725
0.20
6.25
0.50
0.35
0.40
0.45
Spearmnn Corrention on on Triining See
Speaman Correlation on Training Set
FED-Depth
FED-Ükeable
0.6
Sprrrtee Corvertion on Text 546
0.6
0.4
0.2
0.0
0.0
0.25
0.35
0.45
0.50
0.55
0.40
0.45
0.50
0.55
0.60
0.85
Spcarman Conclation on Training Set
Sprorman Conreation Training Set
FED- ULD-Understanddnng
FED-Flexible
0.8
Sjeur on Doreaatonn 44 Teat 544
0.7
0.7-
0.6-
0.5
0.5
03
0.6-
0.1-
0.35
0.40
0.45
0.50
0.50
0.65
0.70
0.55
Comreattio Training
Speerman Comrection Training Set
FED-Infarmative
FED-Overall
0.9
0.8
0.8
Spaiman Comeaaton on Text
0.8
0.7
0.6
0.4-
0.5
0.4
8.525
0.569
0.675 Comelation Set
0.750
Spearman Cometation Training
Figure 6: The FED dataset: association of Spearman Correlation the training and test sets. We that except for the evaluation dimension of Understanding, the trend of the other evaluation dimensions not obvious, ie. filtering based on the performance of the prompt on training set alone does not guarantee that we obtain the optimal prompt.

---

USpicallURR.Grounne Groundnoss
Topical-UR..Ov
0.6
3.10%
2. 7.%
3:475
0.10
7.50%
1.58%
7.47%
0.55
Gentrre Cil
0.44
11.70%
9.10%
6.92%
0.06
4.40%
2.32%
6.63%
0.3
0.02
0.68%
5.69%
2.%
4.00%
2.40%
0.52%
0.02
0.81
256
51.2 2ets (4)
1024
Rnnton--USR-Groundnedd
PelSora-USl-Overall
0 21.
1.00%
3.77%
5,45%
6.00%
6.19%
0.07
Ganure Iv
0.00
-0.5 0..9%%
3,47%
3.82%
5.39%
5.82%
6.00%
0.01
0.05
0.04
1.69%
5.13%
2.77%
7.82%
0.03
255
$12 Zeta (4)
3024
25B
512
1024
2eta (4)
Figure 7: Sensitivity analysis of and ( on the USR dataset.
baliydlllog-2hho--pppepiiiee
Persona-Zhoo-kgpporriittee
0.00
0.03
0.26
Cumsua 10
2.59%
1.47%
0.222
0.18
3.97%
2.15%
5.5%%
3.19%
2.21%
1.70%
1054
Figure 8: Sensitivity analysis for y and Ç on the Dailydialog Zhao and Persona-Zhao datasets.

---

Diverse
0.30
0.22
10.67%
19.32%
0.18
Carrra NO
10 10.22%
11.19%
7.24%
0.16
3.61%
31.49%
26.15%
0.15
0.12
17 92%
23.62%
15 29.23%
0.10
4.70%
24-10%
10.709
0.00
1024
256
1024
FED
Likeable
0.25
7.7%%
18.00 18.009%
0.20
Corrtroe trl
0.15
20 20-40%
26.23%
0.72%
24.97%
15 62%
20.22%
0.10
0.66
0.92%
23 N9%
2.6%%
23-25%
8.27%
11.52%
0.00
0.10
si2
1024
FEO-Understanding
FED-Flexible
.66%
11 49%
14.
0.12
-1.17%
6.6 6.67%
7.83%
0.10
Ganona IV
0.10
0.87%
7/40%
7.5774
15,77%
7.80%
6.93%
0.06
0.05
2. 2.58%
9.19%
8 38%
12 05%
9.50%
22.45%
0.00
512 FED.Overall
0.140
9.75%
1B 09%%
23. 04%
10 94%
13.20%
09.0%9
Gemrie to
Tarmo XX
0.22
71. 71.31%
73.43%
19. 19.015
70%
133.70%
11..74%
0.120
0.315
0.100
13.60%
19. 19.70%
13.24%
0.075
10.08%
10.85%
14.12%
0.10
0.050
256
512
1024
256
512 Zata (4)
1024
Figure 9: Sensitivity analysis for y and ( on the FED dataset.

---

Appropnateness
method
ADcROR
Auto Seltted
0.73
Syctman Corretation
0.72
0.70
0.69
Fow Shot K
Fawancc
(a) DailyDialog-Ziin Zhao
(b) Persona Zhao
Figure 10: Sensitivity analysis for few-shot & the DailyDialog-Zhonn and Persona-Zhao dataset. We conclude that the ADOROR method is consistently better than the Auto-P-selecte baseline, regardless of the number of demonstrations.


Reviews:
[{"review_id": "WFTf3VskNQ", "reviewer": "Reviewer_Hsj3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our special appreciation for your recognition of our model architecture design and experimental details. At the same time, your constructive criticism is also very valuable. It is true that our original paper does miss some information on significance tests and has few inappropriate wordings, as you pointed out in the three weaknesses. We will include the results of significance test in our future versions. Below, we have provided some additional statistics to address your concerns.\n\n**Response to Weakness1**:\n\nWe apologize for not explicitly stating the results of significance test in the original paper. Here we use the **Wilcoxon Sign Rank test** [1] to determine if there are significant differences between any two methods, with the calculation function denoted as $f_w(\\cdot, \\cdot)$. We represent the results of any method as $R=({r_1, …, r_{|R|}})$, where $r_i$ represents the the value of Spearman’s correlation coefficient [3] between a method's ratings of and human ratings on a test set, and $|R|$ equals 5 times the number of evaluation dimensions of all concerned datasets (multiplied by 5 because we perform 5-fold cross-validation). Given two methods, A and B, their Wilcoxon Sign Rank test result can be represented as $f_w(R_A, R_B)$.\n\nBefore listing the comparison results, we briefly review the meaning of each method abbreviation:\n\n \\- **Auto-P-Selected**: Generating and selecting the best prompt based on the inversion prompt, training set and the LLM. (**Module 1**, ours)\n\n\\- **ADOROR w\\o Optimal subgraph**:  **Module 1** (ours) + training demonstration/prompt selector (**Module 2**, ours)\n\n\\- **ADOROR**: **Module 1** (ours) + **Module 2** (ours) + constructing optimal subgraph (**Module 3**, ours)\n\nWe present in Table 1 the differences between **ADOROR w\\o Optimal subgraph** and **Auto-P-Selected (5.s)**, with p-values obtained from the Wilcoxon Sign Rank test (it should be noted that these p-values are different from those mentioned in Table 1 Caption of the original paper, which correspond to Spearman's correlation coefficients). For the DDZ and PZ datasets, which have a small number of results (5), we omit their p-values [2].\n\n| Average Improvement (p-value)                                | FED                     | DDZ  | PZ    | UTC                    | UPC                   | AVG                    |\n| ------------------------------------------------------------ | ----------------------- | ---- | ----- | ---------------------- | --------------------- | ---------------------- |\n| **ADOROR w\\o Optimal subgraph** vs. **Auto-P-Selected (5.s)** | 11.03% (p-value: 0.022) | 3.4% | 4.26% | 3.39% (p-value: 0.075) | 3.5% (p-value: 0.059) | 5.88% (p-value: 0.004) |\nTable.1 Performance on all datasets with few shot k = 5 (For AVG, we integrate the results of all datasets.).\n\nWe can see from Table 1 that the introduction of demonstrations & prompt selector (**Module 2**) leads to an improvement averaging around 6%, and the improvement is statistically significant (p<0.05 when considering all datasets).\n\n**Response to Weakness2**:\n\nSimilarly to the previous comparison, we perform Wilcoxon Sign Rank test to determine whether there was a significant difference between **ADOROR** and **Auto-P-Selected**. For each dataset and method, we integrate the results from few-shot-k=1 to few-shot-k=7 (k = 1, 3, 5, 7), resulting in 20 data points. As shown in Table 2, although the improvement of **ADOROR** over **Auto-P-Selected** (i.e., the introduction of **Module 2 & 3**) is not particularly large, it is stable and statistically significant. Based on the results, the wording (significantly better) used in Figure 10 caption of the original paper is not very inappropriate, and we will make corrections in future versions.\n\n| Average Improvement (p-value)             | DDZ                       | PZ                         |\n| ----------------------------------------- | ------------------------- | -------------------------- |\n| **ADOROR**  vs. **Auto-P-Selected** | 6.06% (p-value: 5.72e-06) | 5.74%  (p-value: 9.53e-06) |\nTable.2 Performance on DDZ and PZ datasets with few shot k = 1,3,5,7.\n\n**Response to Weakness3**:\n\nUsing the same method, we can see from Table 3 (first row) that constructing the optimal subgraph (**Module 3**) steadily improve the performance (though the improvement is minimal on the PZ dataset). The results are statistically significant when considering data points from all datasets.\n\nIn addition, our core contribution is the proposal of **Module 2** and **Module 3**. Although the improvement of each individual module is not particularly large, the improvement becomes more pronounced when we integrate **Module 2** and **Module 3** (see Table 3, second row).\n\n| Average Improvement (p-value)                   | FED                       | DDZ   | PZ     | UTC                    | UPC                    | AVG                         |\n| ----------------------------------------------- | ------------------------- | ----- | ------ | ---------------------- | ---------------------- | --------------------------- |\n| **ADOROR**  vs. **ADOROR w\\o Optimal subgraph** | 10.56% (p-value: 0.009)   | 2.66% | 0.282% | 2.13% (p-value: 0.074) | 1.00% (p-value: 0.139) | 6.34% (p-value: 0.0002)     |\n| **ADOROR**  vs.  **Auto-P-Selected (5.s)**      | 22.72%  (p-value: 0.0001) | 6.23% | 4.5%   | 5.6% (p-value: 0.036)  | 4.7% (p-value: 0.028)  | 14.32% (p-value: 5.542e-07) |\nTable.3 Performance  on all datasets with few shot k = 5.\n\n**References**\n\n[1] Woolson, R.F., 2007. Wilcoxon signed‐rank test. *Wiley encyclopedia of clinical trials*, pp.1-3.\n\n[2] Dwivedi, A.K., Mallawaarachchi, I. and Alvarado, L.A., 2017. Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method.\n\n[3] Sedgwick, P., 2014. Spearman’s rank correlation coefficient. BMJ: British Medical Journal, 349, p.g7327.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "# Strengths\n\n- Technically **well written** paper. The experiments & ablations are thoughtful and well documented.\n\n- Baselines like Ruber, Auto-P, Human-P (with BM 25) are comprehensive and widely used by community for these setups.\n\n- Method has been tested on wide range of datasets (FED (various dimensions), Topical-USR, Persona-USR, DailyDialog-Zhao).\n\n- Two Tower setup enables faster interference for large number of combinations of demonstrations.", "reasons_to_reject": "# Weakness\n\n- While it is interesting that the context demonstrations & prompt selection (two towers) are important, The results in terms of performance of these formulation is not as significantly better as expected. It questions many of assumptions in this formulation in the first place.\n\n- Sensitivity analysis for few-shot k by proposed method ADOROR vs  Auto-P-selected baseline in various datasets, the difference between spearman correlation scores is less than 0.05, which may not be statistically significant for proving one way or other.\n\n- Authors report that **ADOROR without suboptimal graph performs on par with ADOROR  in one dimension of FED, DDZ, PZ, UTC, UPC**, which goes to question can the difference attributed to randomness rather than this formulation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose method called ADOROR to auto evaluate data across various dimensions by leveraging power of in context learning of Large Language Models. They first do prompt generation by essentially asking it to LLM. Then evaluate the generated prompts & various sampled in-context examples in two-tower fashion to get value function for effective prompt & demonstration selection. Authors, then formulate this task of selecting demonstrations and prompt as optimal sub-graph problem. As the problem NP-Hard in nature, They do greedily select most value among the set of demonstrations at every iteration.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UgJwYXyhOg", "reviewer": "Reviewer_r8LW", "summary": "", "questions": "Line 428 mentions the general cost of GPT-3.5-turbo, is there any estimation on the total cost (and time spent) of evaluating on the datasets? Just tracing from the writing it is a bit hard to get a feeling of the cost of ADOROR.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your recognition of our work. We also believe that one of the main practical values of our work is to some extent alleviate the problem of expensive manual annotation in dialogue evaluation (the paradigm can be transformed from manual massive annotation to manual fine-grained annotation + AI automated annotation). Regarding the shortcomings and issues you mentioned, we have provided supplementary explanations and answers in the following text.\n\n**Response to Reject Reasons**:\n\nWe conducted additional experiments on the PersonaZhao dataset using GPT-4 and compared the results with GPT3.5-turbo as follows:\n\n|                                      | Spearman correlation coefficient (Average over 5-fold cross validation) |\n| ------------------------------------ | ------------------------------------------------------------ |\n| Auto-P-Selected (5.s, gpt-4)         | 0.784                                                        |\n| ADOROR (gpt-4)                       | 0.817                                                        |\n| Auto-P-Selected (5.s, gpt-3.5-turbo) | 0.680                                                        |\n| ADOROR (gpt-3.5-turbo)               | 0.711                                                        |\nTable1:  Results on Persona-Zhao dataset\n\nFrom Table 1, it can be observed that our proposed core modules, the prompt/demo selector and the construction of optimal subgraph, still improve the scoring performance on a new LLM—GPT4. We conduct 5-fold cross-validation and find that the Spearman coefficient of **ADOROR** (GPT-4) are consistently higher than the baseline (**Auto-P-Selected**) on all test sets. This experimental result preliminarily demonstrates that our method has good LLM adaptability.\n\nAdditionally, we also attempted to use other large language models, such as LLAMA-2 [1] and Claude-2-100k [2], but preliminary experiments showed that LLAMA-2 [1] could not effectively follow the correct score format as shown in the demonstrations, leading to difficulties in post-processing parsing. Claude-2 also had output format issues, and tended to output very low scores regardless of context variations, so we did not further experiment with these models.\n\n**Response to Questions**:\n\nWe can take the Persona-Zhao dataset as an example. The Persona-Zhao dataset contains 900 dialogues, with an average of 5 rounds per dialogue, and each round is relatively short. The cost of reproducing all the results (few shot k = 5, LLM = gpt-3.5-turbo) in the paper is approximately 160 USD, with 40% used for generating and selecting the most suitable prompts, 45% used for generating the dataset $\\mathcal{O}$, and the remaining 8% used for test set inference. The cost of the first step, tuning the inversion prompt, can be ignored. Since we use 5-fold cross-validation, **the cost divided by 5 is roughly the cost in the real scenario (32 USD)**, which we consider reasonable. For inference, suppose the number of tokens per dialogue is between 300 and 800, if we set the size of the optimal graph $n_{\\mathcal{G}^{'}}$ to 5, the cost of inferring once (**ADOROR** annotating one response with gpt3.5-turbo) is approximately between **0.003-0.008 USD**. Compared to manual annotation, the cost of labeling can be **reduced by a factor of 20** — for the current task, the cost of manual annotation using Amazon Mechanical Turk is around 0.06 USD per response [3].\n\n|                     | Training                                    | Inference                 |\n| ------------------- | ------------------------------------------- | ------------------------- |\n| Ruber-Roberta-Large (we adopt a fine-tuning setup similar to [4]) | 20 minutes / 900 samples (1-fold, epoch=10) | 10 seconds  / 100 samples |\n| ADOROR              | 22 minutes / 900 samples (1-fold)           | 40 seconds / 100 samples  |\nTable2. Comparison of training  and inference speed on Persona-Zhao dataset.\n\nAlso taking the Persona-Zhao dataset as an example, we can see the training and inference time of ADOROR in Table 2. (Our machine configuration is as follows: CPU: Intel(R) Xeon(R) Gold 6242R. RAM: 32G. GPU: NVIDIA RTX 3060TI. The concurrent number of requests to the OpenAI LLM API is set to 20.)\n\n**References**\n\n[1] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S. and Bikel, D., 2023. Llama 2: Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*.\n\n[2] Wu, S., Koo, M., Blum, L., Black, A., Kao, L., Scalzo, F. and Kurtz, I., 2023. A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology. *arXiv preprint arXiv:2308.04709*.\n\n[3] https://aws.amazon.com/sagemaker/data-labeling/pricing/\n\n[4] Zhao, T., Lala, D. and Kawahara, T., 2020, July. Designing Precise and Robust Dialogue Response Evaluators. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics* (pp. 26-33).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Study on prompting-based dialogue evaluation methods could benefit the community (e.g. this class of methods could be beneficial when it is expensive/hard to obtain a sufficient amount of human annotations for training evaluation models from scratch)\n- The proposed approach has good performance compared to various baselines, supported with coherent evaluations with clear experimental details.\n", "reasons_to_reject": "I agree with the authors that experimenting with large language models is relatively expensive, however since ADOROR is a *model-agnostic* method I still feel it's a bit jumpy to draw conclusions from evaluating solely on GPT-3.5-turbo (for example, how does RLHF/instruction tuning affect results? what about the quality of the backbone LLM? etc..). \n\nWhile running a model for all experiments are expensive, it could be fairly helpful if the author could consider reporting a smaller-scale sanity check (e.g. on 1 dataset, etc..) with another model.", "questions_to_authors": "Line 428 mentions the general cost of GPT-3.5-turbo, is there any estimation on the total cost (and time spent) of evaluating on the datasets? Just tracing from the writing it is a bit hard to get a feeling of the cost of ADOROR.", "paper_topic_and_main_contributions": "This paper proposes ADOROR, a prompting-based method for scoring dialogue systems w.r.t. flexible evaluation dimensions leveraging GPT-3.5-turbo. Experiments on five dataset demonstrate ADOROR outperforms various previous scoring methods.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f0wnieW2qt", "reviewer": "Reviewer_JKu9", "summary": "", "questions": "- it's hard to follow how many inversion prompts are being used in the overall training dataset. \n- It is not clear how the method will perform on out of domain conversations. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your thorough and insightful review comments and questions. In response to the shortcomings and related issues you have pointed out, we have provided detailed explanations and clarifications.\n\n**Response to Reject Reason1**:\n\nYes, we agree with this view, but we believe that there are several paths to improve the reliability of LLM and reduce bias:\n\n1. Improve the diversity [2] and quality [1] of human-scored samples to better align with human scoring when conducting in-context learning.\n2. Increase the number of human-scored samples, especially for sensitive data. With a sufficient amount of data, our framework can automatically find prompts that can reduce LLM bias.\n3. Perform RLHF or instruction tuning before using LLM (of course, this path is cost-intensive, and the first two paths are more practical).\n\n**Response to Reject Reason2**:\n\nYes, this is one of the limitations of our work because the currently available open-source dialogue datasets with good annotation consistency generally contain short dialogues. We are actively working on building a dialogue scoring dataset with better dialogue quality and longer dialogues.\n\n**Response to Reject Reason3**:\n\nThank you very much for your suggestion. We will simplify the content of the abstract and adjust its structure to highlight the key points of our work.\n\n**Response to Reject Reason4**:\n\nThank you for your suggestion. We will make adjustments to Figure 1 to better illustrate the important steps in our pipeline.\n\n**Response to Question1**:\n\nWe apologize for not explicitly mentioning the number of inversion prompts in our paper. We will update this information in future versions. For each evaluation dimension in the dataset, we tune and keep one best inversion prompt. In practice, we further simplify the process by designing two inversion prompt templates for turn evaluation and dialogue evaluation, respectively, in which different {evaluation dimension} can be filled in (e.g., …Your objective is to construct a task [prompt] that can generate an {evaluation dimension} score…). Therefore, it is very convenient to use, as we only need to adjust two templates manually.\n\n**Response to Question2**:\n\nWe believe that if the human demonstrations come entirely from others domains, it will significantly increase the difficulty of LLM in performing in-context learning. It is unknown whether our method will still be effective as our method highly relies on the in-context learning capability of the LLM. Some works have shown that the effectiveness of in-context learning is directly proportional to the similarity between the demonstration and the query [3]. In extreme cases, such as when two domains are very different (such as gaming and medical scenarios), it is questionable whether the LLM's ICL capability can still transfer well to different domains, while it is obviously much easier to find demonstrations similar to the query from the in-domain data. Therefore our initial work chose a relatively pure in-domain evaluation scheme. (In-domain evaluation has also been applied to other related work of dialogue evaluation, including supervised [4], semi-supervised [5], and self-supervised [6] ones.)\n\nNevertheless, we also believe that evaluating OOD performance is important. One of the core contributions of our work is to replace human annotation with AI, but one of the prerequisites is the need for a batch of seed data manually generated by humans. If there is a need to frequently annotate dialogues in new domains, then it would be very valuable if the model can generalize to new domains. When the method has good OOD generalization performance, we can reuse the previously annotated data (from other domains), which can further reduce labor costs.\n\n**References**\n\n[1] Chen, J., Chen, L. and Zhou, T., 2023. It Takes One to Tango but More Make Trouble? In-Context Training with Different Number of Demonstrations. *arXiv preprint arXiv:2303.08119*.\n\n[2] Levy, I., Bogin, B. and Berant, J., 2022. Diverse demonstrations improve in-context compositional generalization. *arXiv preprint arXiv:2212.06800*.\n\n[3] Liu, J., Shen, D., Zhang, Y., Dolan, W.B., Carin, L. and Chen, W., 2022, May. What Makes Good In-Context Examples for GPT-3?. In *Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures* (pp. 100-114).\n\n[4] Lowe, R., Noseworthy, M., Serban, I.V., Angelard-Gontier, N., Bengio, Y. and Pineau, J., 2017, July. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)* (pp. 1116-1126).\n\n[5] Zhao, T., Lala, D. and Kawahara, T., 2020, July. Designing Precise and Robust Dialogue Response Evaluators. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics* (pp. 26-33).\n\n[6] Zhang, C., Lee, G., D’Haro, L.F. and Li, H., 2021. D-score: Holistic dialogue evaluation without reference. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, *29*, pp.2502-2516.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a novel method for dialog evaluation using automatic prompt selection, demonstration selection and using the LLM multiple times with various demonstrations for dialog scoring. The method performs surprisingly better than many existing dialog evaluation benchmarks and looks very promising. \n- Overall the method seems to correlate well with human scoring using the gpt3.5 human aligned models to help with automatic scoring. ", "reasons_to_reject": "- There's over reliance on the LLM to trust the automated scoring with the knowledge that LLMs have their complex biases and sensitivity to prompts (and order).\n- It is not clear how the method will perform on long conversations (the dialog datasets used for prompt and demonstration selection seem to contain short conversations) \n- The paper can be simplified in writing - the abstract is too long and does not convey the findings well. \n- Fig. 1 can also be drawn better to show the processing pipeline (prompt generation and manual check, demonstration selection with ground truth scores, and automatic scoring alongwith showing where model training is being used to optimize the selection modules.   ", "questions_to_authors": "- it's hard to follow how many inversion prompts are being used in the overall training dataset. \n- It is not clear how the method will perform on out of domain conversations. ", "paper_topic_and_main_contributions": "Pre-trained dialog evaluation metrics do not scale well to different dimensions and possibilities available today with use of LLMs.  This paper proposes design and development of a dynamic dialog evaluation method to enable efficient adaptation to novel evaluation dimensions or unique scenarios. The paper utilizes an ICL-enhanced prompt generation method for training the black-box LLM and at inference, the LLM is queried multiple times alongwith a subgraph of diverse demonstrations to generate responses. Dialog evaluation method is compared against multiple existing metrics and shows promising results. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "page 6 second column last paragraph type \"... contains contain ...\""}, "extra_text": {}}, {"review_id": "12UxGqTzmR", "reviewer": "Reviewer_yAzT", "summary": "", "questions": "* L186-188: I haven’t got the idea of “aim to automate the scoring process of another dataset $D^x_test$ from the same distribution $P_x$”. Could you explain more about why the goal is to test a dataset from the same distribution? I guess what you mean for the distribution $P_x$ is just a concept, an underlying distribution of real data, but not the distribution of $D^x_train$, right? That is, you split a collected dataset into the training and testing sets and say that they are from the same distribution, right? By the way, should the notation $D^test_y$ be changed to $D^y_test$ to be consistent with other notations?\n* Experimental Question: Section 5.2 you mention “We set the decoding temperature and the top-p value to 0.0 and 1.0 respectively.” Just to confirm, does it mean you are using Argmax (since temperature = 0) and also not top-p sampling (since top-p value = 1)? Otherwise, I guess you want to mean you were using temperature=1.0 with just softmax sampling?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your appreciation of our paper. Your comments, suggestions and questions are also very helpful and insightful, and we have provided the following responses to address them.\n\n**Response to Reject Reason1**:\n\nAs an example, we compared the training and inference time between the Supervised baseline and the ADOROR method on the Persona-Zhao dataset in Table 1. Our machine configuration is as follows: CPU: Intel(R) Xeon(R) Gold 6242R. RAM: 32G. GPU: NVIDIA RTX 3060TI. The concurrent number of requests to the OpenAI LLM API is set to 20.\n\nFurthermore, we would like to emphasize that the time consumption of the ADOROR method mainly depends on the requests to the LLM API, which in turn depends on the speed of the API provided by OpenAI and the maximum concurrent number supported by your current environment. To speed up the training process, we have implemented caching for certain repetitive intermediate results, such as sentence embeddings and the results of requests with identical contexts.\n\n|                     | Training                                    | Inference                 |\n| ------------------- | ------------------------------------------- | ------------------------- |\n| Ruber-Roberta-Large  (we adopt a fine-tuning setup similar to [4]) | 20 minutes / 900 samples (1-fold, epoch=10) | 10 seconds  / 100 samples |\n| ADOROR              | 22 minutes / 900 samples (1-fold)           | 40 seconds / 100 samples  |\nTable1. Comparison of training  and inference speed on Persona-Zhao dataset.\n\n**Response to Reject Reason2**:\n\nThank you for your suggestion. We will definitely consider incorporating these methods (such as BARTscore) in future versions, as long as the space of layout permits.\n\n**Response to Question1**:\n\nYes, you are right. The phrase \"from the same distribution\" can be understood as referring to a batch of dialogue data generated in a similar manner (e.g., from self-chats of the same model, dialogues recorded by the same batch of annotators and machines). We chose to study the generalization of model scores under the same distribution rather than different distributions because our method relies on the in-context learning (ICL) ability of LLM, and the effectiveness of ICL is directly proportional to the similarity between the demonstration and query [1]. Compared to the Out-Of-Distribution (OOD) data, it is obviously easier to find demonstrations similar to the query within the same distribution. In extreme cases, such as when the two distributions are very different (such as gaming and medical scenarios), it is questionable whether ICL has transferability, so we first chose a pure same distribution evaluation scheme in our experiments. (Same-distribution evaluation has also been applied to other related work of dialogue evaluation, including supervised [3], semi-supervised [4], and self-supervised [5] ones.)\n\nThank you very much for correcting our notation. We will make the changes in the future versions.\n\n**Response to Question2**:\n\nYes, we set the decoding temperature to 0.0 and top-p to 1.0, expecting the model to perform greedy decoding. However, in reality, we found that temperature=0.0 is not always equivalent to greedy decoding. According to the official explanation from OpenAI, the model's output can be affected by the non determinism in GPU calculation [2], which can result in different outputs for the same context in rare cases. To ensure maximum reproducibility of our results, we cached all requests locally, meaning that the same context was only requested once.\n\n**References**\n\n[1] Liu, J., Shen, D., Zhang, Y., Dolan, W.B., Carin, L. and Chen, W., 2022, May. What Makes Good In-Context Examples for GPT-3?. In *Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures* (pp. 100-114).\n\n[2] https://community.openai.com/t/a-question-on-determinism/8185\n\n[3] Lowe, R., Noseworthy, M., Serban, I.V., Angelard-Gontier, N., Bengio, Y. and Pineau, J., 2017, July. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)* (pp. 1116-1126).\n\n[4] Zhao, T., Lala, D. and Kawahara, T., 2020, July. Designing Precise and Robust Dialogue Response Evaluators. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics* (pp. 26-33).\n\n[5] Zhang, C., Lee, G., D’Haro, L.F. and Li, H., 2021. D-score: Holistic dialogue evaluation without reference. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, *29*, pp.2502-2516.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is full of information and well constructed.\n* The in-context learning of LLM for dialogue evaluation is a timely method to explore.\n* The method development and experiments have been carefully designed and are sound.\n", "reasons_to_reject": "Minor:\n* The method still looks time-consuming. Could you provide a detailed inference time comparison with other methods? It can be interesting to see the tradeoff between score correlation improvement and the extra consumed resources.\n* The experiments can have a general NLG evaluation metric for reference for the overall score, such as BARTscore, SEScore, GPTscore etc.\n", "questions_to_authors": "* L186-188: I haven’t got the idea of “aim to automate the scoring process of another dataset $D^x_test$ from the same distribution $P_x$”. Could you explain more about why the goal is to test a dataset from the same distribution? I guess what you mean for the distribution $P_x$ is just a concept, an underlying distribution of real data, but not the distribution of $D^x_train$, right? That is, you split a collected dataset into the training and testing sets and say that they are from the same distribution, right? By the way, should the notation $D^test_y$ be changed to $D^y_test$ to be consistent with other notations?\n* Experimental Question: Section 5.2 you mention “We set the decoding temperature and the top-p value to 0.0 and 1.0 respectively.” Just to confirm, does it mean you are using Argmax (since temperature = 0) and also not top-p sampling (since top-p value = 1)? Otherwise, I guess you want to mean you were using temperature=1.0 with just softmax sampling?\n", "paper_topic_and_main_contributions": "This paper proposes a systematic method that leverages in-context learning of LLM for dialogue evaluation in both dialogue-level and turn-level on multiple dimensions. The method includes inverting prompts from the training set, training a prompt selector, and finding an optimal subgraph during the inference stage. The experiments are done on five datasets and compared with multiple baselines.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Very useful paper on the customization of very-long prompts to improve in-context learning. The paper is principled, well-written and impactful.
Decision: Accept-Main