LLM-RUBRIC: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts* (2024)

Abstract: This paper introduces a framework for the automated evaluation of natural language texts. A manually constructed rubric describes how to assess multiple dimensions of interest. To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. The LLM predictions often fail to agree well with human judges—indeed, the humans do not fully agree with one another. However, the multiple LLM distributions can be combined to predict each human judge’s annotations on all questions, including a summary question that assesses overall quality or relevance. LLMRUBRIC accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters. When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-RUBRIC with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges’ assessment of overall user satisfaction, on a scale of 1–4, with RMS error ă 0.5, a 2ˆ improvement over the uncalibrated baseline.

heading: 1 Introduction
text: Many fields that must assess large numbers of short documents have turned to NLP-assisted workflows. For example, lawyers conducting legal discovery must identify all relevant documents (Quartararo et al., 2019)—a task also faced by journalists and historians. Social scientists and market researchers must code survey responses (Mellon et al., 2024; enumerate.ai; ATLAS.ti). Teachers or examiners must evaluate student writing (Page, 1968; Ramesh and Sanampudi, 2022) and provide feedback (Meyer et al., 2024). Doctors, social workers, or public health agencies or researchers may assess an individual’s mental health or safety from their social media posts (Chancellor and De Choudhury, 2020; Xu et al., 2024; Al-Garadi et al., 2022) or
*Code and data available at https://github.com/ microsoft/llm-rubric. At the time of this work, the first author was a Ph.D. intern at Microsoft Semantic Machines.
from clinical interviews and assessments (GalatzerLevy et al., 2023).
The above settings evaluate human-authored texts. In addition, NLP developers must assess the quality of their machine-generated texts—texts that are consumed by end users, but also hidden intermediate steps in agentic workflows (such as chains of thought, tool calls, and revisions). With the recent commercialization of conversational AI, for example, it is crucial to evaluate dialogue systems during development and monitor them after deployment. Special care is needed in high-stakes settings like medical dialogue (Huang et al., 2024).
Manual evaluation has long been the gold standard for assessing text, including generated text (Saphra et al., 2023; van der Lee et al., 2021). Humans are often asked to consider multiple criteria and then provide a final assessment (Hosking et al., 2023). Humans may also be asked to produce reference answers to which other humans can compare the target text. Yet manual evaluation is expensive, time-consuming, and not without its own quality and reliability issues (Hosking et al., 2023; Liu et al., 2016; Smith et al., 2022). Because of these challenges, and the increasing abilities of large language models (LLMs) (Brown et al., 2020), experimenters have recently been eliciting judgments directly from an LLM (Chiang and Lee, 2023; Fu et al., 2023; Liu et al., 2023a; Thomas et al., 2024; ChainForge; and others). But can LLM evaluation be trusted? It solves the time, scaling, and possibly cost issues, but leaves open the problem of aligning these LLM judgments with human judgment.
We present a general approach to this alignment problem. We demonstrate its value for the evaluation and comparison of LLM-powered dialogue systems, in an information-seeking dialogue task (Zamani et al., 2023) similar to Lowe et al. (2015). Evaluation in this setting is complex owing to competing factors that might affect a human judge’s assessment of the dialogue. These may include correctness of responses, accuracy and helpfulness of citations, length and complexity of responses, and more (Smith et al., 2022).
13806
Our LLM-RUBRIC approach begins with a manually authored evaluation rubric. The rubric’s multiple-choice questions cover various evaluation dimensions, and it may also include a question that assesses overall quality or relevance. Evaluating a text, such as a dialogue, then consists of two main steps: (1) for each rubric question we elicit the LLM’s probability distribution over possible responses, by prompting it with the text and the rubric question, and (2) we aggregate and calibrate these distributions with a small feed-forward network that has been trained to match the individual preferences of human judges. A high-level overview of LLM-RUBRIC is shown in Figure 1.
For research in generative NLP, once the rubric and LLM are fixed, LLM-RUBRIC can be used like other metrics (BLEU, ROUGE, etc.) to drive system development, monitor quality, demonstrate the value of a new technique, and conduct competitions. In our dialogue evaluation experiments, each user–AI dialogue is evaluated by 3 trained annotators (randomly drawn from a larger pool) who each answered the same 9 rubric questions. Our method uses these data to train an automatic LLM-based evaluator, without treating the 24 human annotators as interchangeable. Overall, we find1 that
• Personalized calibration of an LLM evaluator of overall satisfaction on ă 750 synthetic dialogues significantly improves its prediction of human judgments and correlations with human judgments, but still works poorly. • Incorporating LLM evaluations of 8 additional criteria (LLM-RUBRIC) improves these metrics by over 2ˆ over the uncalibrated LLM.
1See Table 1, right side, rows 3, 4, and 6.
Accurate automated text assessment could replace human assessment in many other settings, such as those reviewed at the start of this paper. It could also be used in new settings where human assessment was never feasible. In AI-powered user interfaces, instantaneous scoring of user-written text can feed into downstream decisions such as providing writing feedback or deciding how to proceed with a dialogue. An AI reasoning engine may internally apply a rubric to assess the validity of a proposed natural-language reasoning step (Weir et al., 2024). When processing a large document collection, an LLM can be used to assess the compatibility of two text passages (Zhang et al., 2023; Viswanathan et al., 2023; Choi and Ferrara, 2024), potentially in a more nuanced way than vector similarity; this problem arises in workflows for matching, routing, clustering, and fact-checking (Charlin and Zemel, 2013; Harman, 1996; and the papers just mentioned). Finally, automated assessments could provide signals for training text generation (Keskar et al., 2019; Tambwekar et al., 2019; Bai et al., 2022).
To allow LLM-RUBRIC to support all of these use cases, we release general code along with the datasets we created for this paper (see URL on page 1). We discuss limitations at the end of the paper.

heading: 2 The LLM-RUBRIC Method
text: It is challenging to model human preferences in a combinatorial space such as text. Reasonable human judges may differ (Aroyo and Welty, 2015) on (1) what textual properties they happen to prefer (e.g., concise vs. detailed, formal vs. informal, novice vs. expert audience), (2) how they combine multiple preferences into an overall assess-
ment, and (3) how they convey that assessment through a numerical score. Figure 2 shows that in our dataset (§3), different human judges indeed have very different marginal distributions of overall score. Clearly these cannot all be matched by a judge-independent system (e.g., the LLM shown at the lower right of Figure 2).
To expose the different properties and preferences at play, we ask the human judges a series of finer-grained questions about different evaluation criteria. It is already common in practical settings (§1) to at least mention such criteria in instructions to human judges. We use the same questions to query an LLM,2 and train a calibration network to jointly adjust the LLM’s scores to match the scores of any given human judge. We refer to this methodology as LLM-RUBRIC. The gray curves in Figure 2 show that on held-out dialogues, the calibrated overall score is now distributed like that of the given judge. We will see later that these scores are also more accurate on the individual dialogues.
In this section, we present LLM-RUBRIC in a general way, but for concreteness, we also introduce details of our specific experimental setup.
Evaluation Rubric Construction. We wrote 8 dialogue evaluation questions (Q1, . . . , Q8) inspired by the NLG evaluation literature (Zhou et al., 2022; van der Lee et al., 2021). These questions are shown in Appendix C. They address various dimensions such as naturalness, relevance, attribution, citation quality, and conciseness. Our final
2It is convenient to use the same questions, as we have already crafted them. However, different or additional questions could in principle be used—or multiple variants of each question, or multiple LLMs. This could potentially provide more useful evidence to the calibration network below, at the cost of slowing down evaluation and at the risk of overfitting.
question (Q0) asked the judge to assess the overall quality of the dialogue (in this case, focusing only on whether the user would be satisfied), on a Likert scale of 1–4. Each question stated its allowed multiple-choice responses (usually scores 1–4, with a meaning provided for each score).
Multi-Dimensional Evaluation with LLMs. We use an LLM to evaluate a given text T (in our case, a dialogue transcript). For each question Qi (0 ď i ď 8 in our case), we instruct the LLM to generate a label yi P Yi, where Yi is the set of allowed responses to Qi (e.g., t“1”, “2”, “3”, “4”u). Specifically, we prompt it with a preamble, the text T , and the question Qi, where Qi also specifies the allowed responses Yi (see Appendix D). We chose to do this independently for each question Qi to avoid confounding the LLM’s responses. We thus obtain pLLMpyi | T,Qiq for all questions Q0, . . . , Q8 and each possible response yi P Yi.3
Aggregated Evaluation with Personalized Calibration. We then use a small feed-forward calibration network (Figure 1 and equations (3)–(5) below) to map this collection of LLM probabilities pLLMpyi | T,Qiq to a collection of adjusted probabilities p̂apyi | T,Qiq that predict the responses of a particular judge a. Note that each p̂apyi | T,Qiq is predicted from the LLM’s behavior on all questions about T , not just Qi. This design lets the calibration network inspect some additional proper-
3The LLM also allocates some probability to responses outside Yi, so Zi def“ řyiPYi pLLMpyi | T,Qiq ă 1. We do not normalize the probabilities by Zi before presenting them to the calibration network. This allows our calibration network, in principle, to notice when Zi ! 1 and to learn not to rely on the LLM’s answer to Qi in such cases. In practice, however, our prompts result in Zi being very close to 1.
ties of T that might influence a’s response to Qi.4 This design also extends to the case where the LLM was not asked the specific question Qi for which we are predicting a’s response (see footnote 2).
We train the calibration network by maximum likelihood (regularized by early stopping). That is, given a dataset D of annotations, we maximize5
ÿ
pT,i,a,yai qPD log p̂apyai | T,Qiq (1)
where pT, i, a, yai q P D means that judge a answered Qi on T with response yai .
Decoding. Given a new text T , the trained calibration network predicts any judge a’s possible responses to question Qi via the distribution p̂apyi | T,Qiq. If we wish to output a single predicted value ŷai for downstream use, then we also need a decoding principle that extracts ŷai from p̂a. In our experiments, actual responses yai are integers, predictions ŷai are real numbers, and we will be evaluating the predictions by L2 loss, pŷai ´ yai q2.6 Thus, our principle is to minimize the expected L2 loss (our “Bayes risk”). This is accomplished simply by predicting the mean of distribution p̂a,
ŷai “ ÿ
yiPYi p̂apyi | T,Qiq ¨ yi (2)
We remark that we could have constructed a network that directly predicted the ŷai values, and trained it to minimize L2 loss on training data—a regression problem. However, by modeling the entire distribution p̂a and not just its mean, we make fuller use of the training data for representation learning—our representations are trained to be able to predict the full distribution. Indeed, we found in pilot experiments that our method slightly outperforms the regression method. Furthermore, modeling p̂a lets us report our predictive uncertainty— e.g., the entropy or variance of p̂apyi | T,Qiq and not just its expectation ŷai . Finally, equation (2) guarantees that 1 ď ŷai ď 4 on any example.
4In the future, for this reason, the calibration network’s input could also include an embedding of the full text T .
5This formula models the yai for different i as conditionally independent given T . This assumption could be relaxed. For example, perhaps all of the yai should be made to also depend on a latent variable, e.g., judge a’s mood while annotating T .
6This setup treats the integers as falling on an interval scale, not just an ordinal scale. For example, outputting 1.4 when the true answer is 1 is considered exactly as bad as outputting 2.6 when the true answer is 3. This is not always appropriate.
Calibration Network Architecture. Our network’s input is a feature vector x “ rpLLMpyi | T,Qiq : i P t0, . . . , 8u, yi P Yis. These are already extremely high-level text features, extracted by the LLM. We next use a feed-forward neural net to transform x into a representation z2 P Rh2 :
z1 “ σ ` pW1 ` W a1 q r1;xs ˘ P Rh1 (3) z2 “ σ ` pW2 ` W a2 q r1; z1s ˘ P Rh2 (4)
Here W1,W a1 P Rh1ˆp1`9q and W2,W a2 P Rh2ˆp1`h1q. The parameters Wk are shared across all judges while W ak are judge-specific.
The learned representations z2 are shared across all questions. For each i P t0, . . . , 8u, we obtain tp̂apyi | T,Qiq : yi P Yiu as a probability vector
softmaxppVi ` V ai q r1; z2sq P R|Yi| (5)
The collection of matrices Vi P R|Yi|ˆp1`h2q can be implemented as a 3D tensor V (padding Vi with extra rows when |Yi| is small). Multi-Task Learning. Our calibration network performs multi-task learning: each rubric question is a different task. When the accurate prediction of ya0 is our main task, the other tasks serve only as regularizing auxiliary tasks, which help training to discover useful hidden features z2. The weighting of the auxiliary tasks could be dynamically adapted during training (using a validation set), for example with the AuxiNash training algorithm (Shamsian et al., 2023). However, we currently use a simpler, faster shortcut that divides training into two phases. In the pre-training phase, we optimize the full log-likelihood objective (1). This learns useful initial representations.7 In the fine-tuning phase, we continue training with a modified objective that sums over only the tuples in D with i “ 0. This adjusts the parameters to focus on the main task— predicting responses ya0 to Q0. In both phases, we use early stopping to avoid overfitting.8
7However, in contrast to AuxiNash, this shortcut does not try to identify and favor more useful auxiliary tasks. Equation (1) simply weights each question Qi in proportion to its number of annotated answers in the training dataset D. (In our experiments, all questions are equally represented in D.)
8We also tried a variant where pre-training was itself divided into two stages and we fixed W ak “ 0 and V ai “ 0 during the first stage. This was intended to prevent overfitting of these judge-specific parameters, but we observed no improvement compared to the simpler method.
Using the Predictions. Since LLM-RUBRIC can predict any judge’s scores on a new text T , how should it be used in practice? In Appendix A, we propose approaches to score aggregation, system quality monitoring, and other practical issues.
Future Extensions. The idea of a calibration network is quite general and can easily be extended to various types of human and LLM judgments. In Appendix B, we sketch some natural extensions that were not needed in this paper’s experiments.

heading: 3 Data
text: Conversational AI systems are now being widely deployed. To test our methods on dialogue evaluation, we invest in developing both synthetic and real datasets of human–AI conversations.
We focus on English information-seeking dialogues in the “IT help” (enterprise information technology) domain (Lowe et al., 2015; Carletta et al., 2005). As in many real world domains, dialogue data here is often proprietary to the system owner and/or private to the user. Acquiring experimental access to live systems for evaluation is even more difficult. Thus, we build and evaluate several LLM-powered dialogue systems, which differ in their ability to search a corpus of websites related to Microsoft Azure9 help topics.
For training data, we generate a corpus of synthetic dialogues with simulated users, and have human judges rate them. Collecting these artificial dialogues is efficient, since judges only have to annotate conversations and not interact with the systems first. For our final test data, we have our judges actually interact with the live systems as users and then annotate their own dialogues. All of our judges are professional annotators.
To mine the topics for both synthetic and live evaluation, we use real user queries and click data from a large commercial web search engine, which further increases the realism of our experiments.
Below, §3.1 explains how we compile a corpus of background documents and how we select topics to ensure that the generated and collected conversations are diverse and are indeed informationseeking, rather than navigational or transactional. §3.2 and §3.3 explain our approaches to synthetic dialogue generation and real dialogue collection.
9https://azure.microsoft.com/

heading: 3.1 Mining Topics for RAG
text: To simulate or collect diverse information-seeking dialogues, we need to know what information our users will seek. We picked an arbitrary IT help topic, Azure, for which many answers can be found on the subreddit r/azure. We hypothesize that search queries are enterprise information-seeking topics related to Azure if they lead to satisfactory clicks on the Azure subreddit.10 Using this heuristic to help filter query logs obtained from the Bing search engine, we construct a set S of 2275 common English queries about Azure. We will use these as topics to prompt the creation of realistic and diverse conversations.
Some of our dialogue systems will condition their responses on relevant documents, as in retrieval-augmented generation (RAG) (Lewis et al., 2020). To build a corpus of potentially relevant documents, we mined and crawled all 37,982 clicked URLs in the web search engine’s results to the queries in S. This includes but is not limited to the Azure subreddit URLs. We discard the ones that require login, are behind a paywall, or are no longer available (broken links). To ensure that the URLs are of high quality, we also make sure they exist in Clueweb 2022 Set B (Overwijk et al., 2022) top 200M most popular URLs. After filtering, we arrived at 23,243 unique webpages. We used BeautifulSoup to convert each webpage’s title and body into a plain text document, without any truncation. The mean document length is 1246 ˘ 1651 words (denoting mean ˘ standard deviation).

heading: 3.2 Synthetic Dialogue Generation
text: To generate synthetic dialogues in English of varying quality, we use 5 different LLM-based approaches (DS1–DS5), described in Appendix F. These approaches have different levels of access to the document corpus. Also, the true topic (which is always provided to the simulated user) is only revealed to the dialogue system in DS1–DS3.
We use gpt-3.5-turbo-16k with its default parameters (OpenAI, 2024) for all of our data generation (§3.2, §3.3) and rubric-based evaluation (§4).
We randomly selected 50 topics, and used each of the systems DS1–DS5 to generate a synthetic conversation on that topic, resulting in 250 unique dialogues of varying quality. Each dialogue was evaluated by 3 judges (randomly assigned from
10A satisfactory click in a search engine is defined as a click that leads to a dwell time longer than a given threshold (Jiang and Allan, 2016). Here we use a threshold of 30 seconds.
a pool of 24 judges), resulting in 741 personalized data points for dialogue evaluation after some guardrail quality checks (see Appendix G). The average judge annotated 31 ˘ 13 dialogues.

heading: 3.3 Real Dialogue Collection and Evaluation
text: To obtain more realistic data for evaluation, we collect conversations with DS1–DS3 where the user turns are not generated by the LLM but by a real human. The assistant in these three systems may be summarized as “no RAG” (DS1), “oracle RAG based on the topic” (DS2), and “BM25 RAG based on the topic” (DS3).
The human who plays the user role in the dialogue then also serves as the judge for that dialogue, making them particularly well qualified to judge overall user satisfaction Q0. Details about the web interface and instructions to the humans can be found in Appendix H.
We collected a total of 223 evaluated human conversations by having 13 of the original 24 judges converse with systems DS1–DS3 (some judges were no longer available). Each judge engaged in and annotated 17.2 ˘ 3.4 dialogues on average. The evaluations are summarized in Appendix I.

heading: 4 Experiments
text: We will evaluate how well LLM-RUBRIC can predict individual judges’ assessments ya0 of our Q0 (overall user satisfaction). We evaluate predictions ŷa0 both in absolute terms (whether they achieve low root-mean-squared error, or RMSE) and in relative terms (how well ŷa0 correlates with y a 0 , i.e., whether ŷa0 can be used to rank pT, aq pairs). We train our calibration networks on synthetic dialogues. We then evaluate them not only on heldout synthetic dialogues but also on real dialogues, to demonstrate that the LLM scoring and its calibration can generalize from synthetic to real data.
Hyperparameter Selection. To train a system on a given training set, we evaluate hyperparameter settings from a grid by 5-fold cross-validation on the training set, and then use the selected hyperparameters to train on the entire training set. We select the hyperparameters that maximize the main task objective, namely the log-likelihood of (held-out) annotations ya0 . The hidden layer sizes h1, h2 each range over t10, 25, 50, 100u, the batch size ranges over t32, 64, 128, 256u, the learning rate of the Adam optimizer ranges over t0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005,
0.01u, and the numbers of epochs for pretraining and fine-tuning each range over t5, 10, 20, 30, 40, 50u.11 Synthetic Data Evaluation. We test our calibration network on all 741 synthetic dialogues, using 5-fold cross-validation; the dataset is split at the dialogue level so that each dialogue appears in only one fold. Different folds may select different evaluation hyperparameters, resulting in different architectures for the calibration network.12
Real Data Evaluation. We test our calibration network on all 223 real dialogues, after training on all of the synthetic dialogues (again selecting hyperparameters by 5-fold cross-validation).
Baseline Methods. As Table 1 shows, we compare LLM-RUBRIC to these 5 baselines:
1. Random. For each dialogue independently, we produce 1, 2, 3, or 4 uniformly at random.
2. Argmax LLM Q0. We use the top LLM prediction for Q0: argmaxy0PY0 pLLMpy0 | T,Q0q. Note that this system always produces an integer.13
3. Expected LLM Q0. We use the expected value of the LLM’s prediction for Q0: ř y0PY0 y0 ¨pLLMpy0 | T,Q0q{Z0 (where
Z0 normalizes the probabilities over Y0—see footnote 3).
4. Calibrated LLM Q0. An ablated version of LLM-RUBRIC that uses only Q0, i.e., the feature vector x “ rp̃py0 | T,Q0q | y0 P Y0s is restricted to the Q0 answer probabilities. We train and evaluate the calibration network just as for LLM-RUBRIC, including crossvalidation and hyperparameter selection.
5. FActScore (Min et al., 2023). This is a recent retrieval-based automatic evaluator14 that predicts the percentage of factually correct sentences as the overall evaluation score. We use the Azure corpus described in §3.1 as the retrieval corpus in FActScore, which performs better than the default Wikipedia corpus.
11Instead of including the number of epochs in the hyperparameter grid search, an alternative would be to use a standard early stopping heuristic at each phase, by evaluating that phase’s training objective periodically on held-out data.
12When training on 4 folds to evaluate the 5th, we select the hyperparameters by an inner 5-fold cross-validation on this training set of about 593 examples, as explained above.
13In a pilot experiment, we found no significant improvement from few-shot prompting.
14https://github.com/shmsw25/FActScore
Oracle Methods. Table 1 also shows upper bounds on performance. The Oracle system is the same as LLM-RUBRIC, but the calibration network’s input x—at both training and test time— includes the judge’s actual response to each question Qi (except for Q0, which we aim to predict!) as a four-dimensional one-hot vector, in addition to the LLM response vector pLLMpy0 | T,Qiq.
We ablate different components of the Oracle model by withholding the LLM response vector from the model input and by depersonalizing the calibration network (Oracle w/o Personalized Calibration) by dropping W ak . To restore a judge a’s idiosyncratic distribution of Q0 scores (Figure 2), without restoring their idiosyncratic computation of Q0 from other dimensions, we try correcting the output of the depersonalized calibration network using an a-specific isotonic regression model.
Our Depersonalized Oracle is similar to the Oracle, but instead of using the responses of the actual target judge a, it uses the distribution of responses of all other judges (averaging their onehot vectors), holding out the target judge.15 It also drops the personalized weights W ak .
Thus, the Oracle provides a rough upper bound on LLM-RUBRIC. The Depersonalized Oracle provides a rough upper bound on a version of LLMRUBRIC that produces a-independent results.

heading: 5 Results
text: A trivial baseline of predicting a constant Q0 (the overall mean from training data) achieves an RMSE of 0.82 on both synthetic and real conversations. LLM-RUBRIC roughly halves this (row 6 of Ta-
15We cannot run this on the real conversation dataset, where each dialogue is annotated only by a single judge.
ble 1), so it explains « 34 of the variance in human judgments of Q0 across judges a and texts T . Its predictions of Q0 have tolerably low error and correlate reasonably well with those of human judges.
In sharp contrast, the LLM’s direct response to Q0 (row 2 or 3) does worse than the constant baseline. Even calibrating its response distribution for each judge (row 4) barely improves on the baseline, explaining only 5–10% of the variance in human judgments and achieving only « 0.2 correlation with them. This suggests that the LLM cannot help assess Q0 (user satisfaction) until we ask it about the finer-grained dimensions Q1–Q8.
The results obtained by FActScore (row 5) do not correlate any better with overall satisfaction, so percentage of factually correct sentences is also not a good indicator of overall user satisfaction. Moreover, Liu et al. (2016) showed that dialogue systems were poorly evaluated by simple metrics of lexical overlap with human responses.

heading: 6 Analysis
text: Calibration. Does our trained LLM-RUBRIC produce well-calibrated probability distributions for Q0 (as one would expect from maximumlikelihood training)? We checked on synthetic data. It obtained excellent smECE values of ă 0.05 for each y0 P Y0 “ t1, 2, 3, 4u, where smECE is the smoothed expected calibration error (Błasiok and Nakkiran, 2023). Informally, this means that for each y0 P Y0, when we examine the held-out examples pT, 0, a, ya0q with p̂apy0 | T,Q0q « p, the fraction where ya0 “ y0 was in fact « p. Appendix K shows calibration plots and discusses how to use calibrated probabilities for downstream decisions.
Ablation Studies. §5 showed that LLM responses on 8 additional questions were useful, but was our calibration network the best way to incorporate them into our prediction of Q0? To justify each design decision, we try omitting pre-training, fine-tuning, and personalized weighting from our calibration network. The results on the real conversation data in Table 2 show that predictions were improved by each step. In particular, it was indeed useful to do multi-task pre-training of the calibration network (which required human judgments on all questions) and to then fine-tune on the main task. Personalized weighting had the greatest impact.
Also, were all 8 questions useful? We measured
the impact of each question by omitting it from the evaluation rubric for the LLM-RUBRIC model (bottom half of Table 2). All rubric dimensions contributed significantly to the Q0 prediction, except for Q6, which focuses on redundancy in the dialogue. Using even more rubric dimensions might improve performance further (footnote 2 and Appendix B). That said, considering more rubric dimensions would mean more human annotations at pre-training time and/or more LLM computation.
Oracle study. Giving LLM-RUBRIC access to a judge’s true responses to Q1–Q8 lets us see how well the judge’s overall quality score Q0 is predictable from our particular rubric dimensions. This gets rather better results, including an excellent 0.72 Pearson’s ρ correlation between predicted and actual satisfaction scores on real dialogues (row ‘a’ in Table 1). Almost all of this performance can be obtained from only the judge’s responses, without access to the pLLM score distributions (row ‘b’).
This suggests a future strategy (discussed below) of improving the input to LLM-RUBRIC by getting the LLM to better predict the judge-specific human rubric responses that were available to the Oracle (row ‘a’), or at least judge-independent versions (rows ‘e’–‘f’). Once such responses are available, the ensembling is still best done by a calibration network that understands an individual judge’s preferences—though under oracle conditions and with our population of judges, dropping that personalization would not be dramatically worse (row ‘c’), and a fraction of the difference can be made up simply by adjusting the predicted scores ŷ0 with personalized isotonic regression (row ‘d’).

heading: On which dimensions do zero-shot LLMs need
text: improvement? Table 3 shows these results. Redundancy (Q6), Conciseness (Q7), and Efficiency (Q8) were especially difficult for the LLM to predict—it showed close to zero correlation with human annotators. LLM-RUBRIC much better predicted these scores, as well as overall Satisfaction Q0, by exploiting the full response vector x: e.g., it improved RMSE by ą 0.5 in all of these cases.
The LLM’s errors on a difficult question Qi could potentially be reduced through prompt engineering, few-shot prompting, fine-tuning the LLM, or calling a larger LLM. Is that worth it? To assess the potential improvement to Q0 prediction from better answering Qi, one could use cross-validation to evaluate the benefit to Q0 from replacing just Qi with oracle scores before training LLM-RUBRIC.
How much human judge data is needed to train calibration? See Appendix J for learning curves.

heading: 7 Related Work
text: LLM Evaluation Zero-shot or few-shot LLM evaluators have been shown to have higher agreement with human annotators than traditional lexical overlap or even earlier generations of transformer embedding models across a variety of NLG tasks (Fu et al., 2023; Lin et al., 2024). Furthermore, when compared to crowdworkers, LLMs can have higher agreement with expert annotators (Gilardi et al., 2023; Chiang and Lee, 2023). Additional techniques like chain-of-thought prompting and auto-prompt engineering can also further improve alignment with human ground truth (Liu et al., 2023a,b; Lin et al., 2024). It seems that LLMs are capable of measuring an increasing range of evaluation dimensions including factuality (Min et al., 2023; Gekhman et al., 2023; Yue et al., 2023), interpretability (Lu et al., 2023), and relevance (SaadFalcon et al., 2023). These works generally focus on average judge preferences on individual evaluation attributes, while we focus on using LLMs to capture the interplay of individual attributes to better predict all judgments (particularly of overall text quality) for a given judge.
Calibration of LLM evaluators. Zhao et al. (2023) develop a Pareto-optimal method for estimating the error rate of an LLM-based predictor by combining both LLM and heuristic predictions, which can in turn be used to correct the initial LLM prediction. While they similarly take advantage of an ensemble of predictors, they assume specific ground-truth answers, whereas LLM-RUBRIC produces distributions over reasonable answers.
Subjectivity in Evaluation. While LLMs can agree with expert judges, in cases where experts have low agreement, LLMs tend to have low agreement with the judges as well (Chiang and Lee, 2023). It is increasingly acknowledged that accounting for subjectivity (as opposed to collapsing or removing disagreements) in NLP evaluation is a key part of evaluation design (Pavlick and Kwiatkowski, 2019; Basile et al., 2021; Uma et al., 2021a,b; Plank, 2022; Plepi et al., 2022; Sandri et al., 2023). By training a single network to model all judges, we take the view that “disagreement is not noise but signal” (Aroyo and Welty, 2015). Baan et al. (2022) put it more starkly: without
modeling the judge distribution, metric calibration is itself nonsensical on subjective tasks. Making downstream use of these disagreeing judges—or rather LLM-RUBRIC’s simulation of them on new texts—is discussed by Appendix A, Gantt et al. (2020), and Uma et al. (2021b).
While our work is similar conceptually to Gantt et al. (2020) in that we include judge-specific parameters to predict each human judge’s responses, we show that this can be further improved by predicting responses to multiple questions (our auxiliary tasks Q1–Q8 along with our main task Q0).
Xiao et al. (2023) analyze common NLG evaluation dimensions and metrics using the concepts of reliability and validity from measurement theory. They find that while manual judges may rate generated texts by different dimensions like ‘coherence’ or ‘relevance,’ these dimensions can exhibit poor validity structure. In their case, this means that they find that an individual judge’s correlation with their own ratings across coherence and relevance can be as high or higher than correlation between other judges within each dimension, supporting the idea individual judges may have idiosyncratic or conflated mappings of different evaluation criteria. Xiao et al. (2023) suggest several ways to improve the dimensions account for this. We did not preform a similar analysis on our judges and rubric dimensions, although improvements here would be orthogonal to the benefits of LLM-RUBRIC, since judges may reasonably disagree even in the absence of validity structure issues.

heading: 8 Conclusions
text: This work proposed LLM-RUBRIC—a rubricbased framework for automatic evaluation of text. We trained and tested it on novel datasets of information-seeking dialogues. LLM-RUBRIC performs multidimensional evaluation using a blackbox LLM, then aggregates and calibrates these multidimensional responses for each human judge.
Although the LLM’s raw responses do not highly correlate with human judgments in such a complex task, we found that combining its response distributions on all questions can predict each human judge’s responses, including overall satisfaction. We obtained substantial improvements on RMSE and on both linear and rank-based correlation metrics, on held-out synthetic conversations (development data) and real ones (test data). Below, we discuss limitations, ethics, uses, and extensions.

heading: Acknowledgments
text: We thank Val Ramirez and the data specialists who contributed to the creation of this work.
Limitations
Robustness. In general, one might hope that the trained LLM-RUBRIC can successfully predict human scores even in new test domains—at least when it is given a broadly competent LLM, a broadly worded rubric, and training examples that exhibit a variety of score profiles on that rubric. However, we did not evaluate this, other than showing that our trained LLM-RUBRIC worked well when applied to a slightly different test distribution (real rather than synthetic dialogues) on the same topics. Robust score prediction is needed when the test distribution shifts over time, perhaps even adversarially (e.g., in settings like essay grading where test-takers have an incentive to submit texts that will fool the evaluation system). Robustness is particularly important when privacy rules prevent having human judges inspect the actual test examples, as in some deployed dialogue systems or when coding medically or legally sensitive data.
Efficiency. We used a large pretrained LLM to answer each rubric question. It would be cheaper to use smaller models where possible, perhaps finetuned on specific questions. One could also decide which questions are worth asking (and which models to ask) by using an adaptive rubric: e.g., choose the next evaluation question to maximize the expected information gain, and stop at the point of diminishing returns, so that it is not necessary to ask all questions. An adaptive rubric could in principle be quite large, with only a small portion of it used on any particular text T . This direction and other possible extensions are discussed in Appendix B, but we did not try them.
Downstream Evaluation. Although we measured overall correlation between predicted and human scores on each rubric question, we did not evaluate the usefulness of our predicted scores for difficult downstream tasks such as choosing among similar candidate answers or dialogue systems. More rubric questions might be needed for sufficiently accurate evaluation (see footnote 2 and Appendix B).
A particularly challenging but important downstream use is to improve natural language generation. We have not addressed this. However, a metric such as our predicted overall quality ŷ0
(averaged over a set of judges as in Appendix A) could be used as a reward signal, for example to improve an LLM by proximal policy optimization (Schulman et al., 2017). More ambitiously, one could train the LLM using multi-objective reinforcement learning (e.g., Yang et al., 2019; Abels et al., 2019; Ramé et al., 2023; Wu et al., 2023) to consider idiosyncratic preferences at runtime and generate text that achieves a high predicted user-specific reward. For example, one could use ŷa0 as the runtime reward function if one modified our calibration network to do regression (§2) as ŷa0 “ pv0 ` va0q ¨ r1; z2s where z2 is judge-independent (compare equation (5)). Then z2 serves as a multi-objective reward vector, and v0 ` va0 is the preference weighting that linearly scalarizes this reward at runtime, where va0 may be regarded as a preference embedding of the user a (possibly computed from features of a).
Fine-Grained Evaluation. We only considered evaluating entire texts. However, humans often perform finer-grained evaluation tasks—such as highlighting problematic spans in human- or machinewritten text (e.g., to provide feedback and opportunities for revision), or highlighting relevant spans (e.g., to call a human or machine’s attention to them). We have not presented methods for performing or calibrating fine-grained evaluation.
Ethics Statement
Beyond User Satisfaction. Evaluation metrics drive engineering and so have real-world consequences. Our experiments focused on predicting overall user satisfaction (our choice of Q0), but we do not in fact recommend this as the actual goal of dialogue system development. In practice, quality evaluation of a dialogue agent should also assess potential harms to the user (e.g., false information), to the dialogue system owner (e.g., reputational harm through violating policies on content or style), and to third parties (e.g., encouraging violence or outputting private or copyrighted information).
Fairness Auditing. Our aligned LLM’s ability to approximately match human judges does not answer the question of whether the unaligned LLM, the aligned LLM, the human judges, or the manually constructed rubrics are fair or unbiased. Even when our system does achieve low total error at matching fair judgments, it is not guaranteed that its errors or their downstream harms are evenly dis-
tributed. Thus, accuracy (Table 1), calibration (Appendix K), and rubric validity should be checked for various important subsets of the data. For example, in essay grading, does the calibrated LLM systematically underestimate the quality of the ideas of speakers of a particular dialect? In dialogue system evaluation, is a particular user population frustrated with a certain kind of error that they experience heavily, yet this type of error is underdiagnosed?
Human Data. LLM-RUBRIC requires collecting data from human judges that reveal their personal preferences, such as their affinity for specific textual passages. Such data should always be carefully safeguarded. In certain cases it may even be appropriate to train the calibration network using differential privacy, to make it impossible to guess information about particular judges from the network weights.
Harmful Uses. LLM-RUBRIC may enable generating or chooosing content that appeals to a specific human’s preferences. This could improve their satisfaction with the NLG output, but it could also be used to optimize for their engagement—even when this is harmful (for example, confirming biases, spreading misinformation, provoking outrage, swindling, or recommending antisocial actions or self-harm).
Environmental Costs. LLM-RUBRIC is compute-intensive, as it involves calling an LLM several times for each NLG output. On a small evaluation dataset, the compute cost may be modest, but LLM-RUBRIC will add to the environmental footprint of a system if it is applied to a substantial fraction of user traffic, or is called many times during a hyperparameter tuning loop or to compute the reward signal for reinforcement learning. Costs might be reduced through distillation or an adaptive rubric, as discussed in the Limitations section.

heading: A Aggregating Predicted Scores
text: Our use of judge-specific distributions p̂a can be regarded principally as a technique to improve training on human annotations. Judges are heterogeneous (Figure 2) and a training document will only be judged by some of them (§3), as discussed in Appendix B below. Knowing who the judges were can help us model the training data. For example, suppose T got mostly low LLM scores, yet all judges randomly assigned to T in training data gave it a high overall score. The model might “explain away” the high scores if it knows that those particular judges are generous or are focused on dimensions where T did well—and thus could still predict low overall scores from the remaining judges.
However, this means that our trained calibration network does not produce ground truth. It only models the idiosyncrasies of individual judges a (successfully, as shown in Figure 2 and Table 1). We do not even suggest that purely objective scores exist (see §7), except on extremely precise rubric questions. So which judge should we use in the end? That is, after training LLM-RUBRIC, how should we practically obtain a final assessment of a new text T ?
We might use the mean predicted overall quality, ŷ0, where ŷi “ meanaPA ŷai for a fixed set A of trusted judges.16 This assumes that Q0 calls for numerical responses on an interval scale (see footnote 6), so that the mean is defined and meaningful. An unweighted mean also assumes that we equally want to please all judges in A (see the start of §2). The benefit of LLM-RUBRIC is that we do not actually query these judges—we predict how each of them would respond by querying an LLM and calibrating its response distributions.
What makes a judge “trusted”? The judges in A might have had additional training, insight, information, or time. For example, Thomas et al. (2024) distinguish between trained assessors and third-party crowdworkers. If LLM-RUBRIC scores are used to nominate interesting documents for more careful manual review, for example in a legal document review workflow, then A might consist of the experienced lawyers or paralegals who perform the manual review (and who will continue to add to the training set by answering at least Q0 on newly nominated documents). Alternatively, a trusted judge, rather than being a single human, might correspond to the result of a discussion and reconciliation process among multiple untrusted human judges.
The various applications in §1 might call for other ways to aggregate the predicted judgments (or the resulting document rankings). E.g., to be safe, lawyers may want to replace mean with max to review any document that at least one judge in A would have deemed relevant. The predicted judgments can also be used without aggregation (Uma et al., 2021b; Plank, 2022) to train or evaluate other systems for generating or scoring text.
Dashboards. In our setting of dialogue evaluation (or NLG evaluation), the mean predicted score ŷ0 for a given text T can be used as a target metric for system development and monitoring.
To aid system developers, we can go beyond ŷ0 and compute ŷi on T for each Qi (using a version of the network that has been re-fine-tuned to predict ŷai as its main task). We can also quantify the importance of improving T to raise its mean human Qi rating: ∇xŷ0¨∇xŷi∇xŷi¨∇xŷi estimates the improvement in the prediction ŷ0 per unit of improvement in the prediction ŷi, if one could change T so as to change x in the direction of steepest ascent of ŷi.17
A dashboard for the system developers could show how all of the above quantities are distributed over a representative set of texts T , using kernel density estimation (or a histogram). The dashboard could also display these distributions for different subsets of T representing specific topics or groups of users, could compare them across different versions of the system, and could track their means or quantiles over time. Uncertainty bands around each density curve can be found by computing it many times, each time
16Any judges not in A still help regularize the training. They might be omitted during fine-tuning (just as Qi was for i ‰ 0). 17Of course, it will not usually be possible to change T in quite this way: the desired direction ∇xŷi may send x out of the feasible space of texts. Thus, a more sophisticated approach is to estimate the manifold of plausible x vectors from known training texts (including desirable texts), so that each x can be represented in terms of underlying manifold coordinates w and residuals. Now ∇x may be replaced with ∇w throughout. This constrains the steepest-ascent direction to point along the manifold. The manifold may be estimated with methods such as Isomap (Tenenbaum et al., 2000), LLE (Roweis and Saul, 2000), or VAE (Kingma and Welling, 2019). Less ambitiously, one could merely represent the pLLM distributions within x using softmax parameters w, to constrain their steepest-ascent directions to the probability simplex.
substituting bootstrap replicates of A and T and—in the case of the density of ŷi—replacing each ŷai for each text T with a sample from p̂apyi | T,Qiq.18 Thus, small A, small T , and high-variance distributions p̂a for a P A will all lead to wider uncertainty bands. This procedure also yields confidence intervals on the statistics (means, differences of means, etc.).
Each of the above distributions over T could optionally be disaggregated into a distribution over T ˆA. Suppose Yi is a 1–4 Likert scale of “strongly disagree, disagree, agree, strongly agree” and |A| “ 2. If one judge probably disagrees and the other probably strongly agrees with Qi for a given text (ŷai « 2.0, ŷa 1 i « 4.0), then these two opinions would be recorded separately in the disaggregated view, rather than being averaged into “agree” (ŷi « 3.0). Averaging Likert responses is often discouraged because it homogenizes diverse opinions and because it treats the Likert scale as an interval scale rather than an ordinal scale (Barry, 2017).19 We suspect, however, that both aggregated and disaggregated views are useful in practice. Clicking on the lower tail of an aggregated distribution will display problematic dialogues that are predicted to have a low average score on Qi. For a disaggregated distribution, the same click displays dialogues that are predicted to be problematic for specific judges, according to their idiosyncratic interpretations of Qi.

heading: B Handling Other Types of Datasets
text: Our experimental datasets used a certain kind of rubric and a simple data collection mechanism. However, the idea of predicting human judgments with a calibration network is quite general and can be extended to a variety of practical settings. We discuss some useful examples in this appendix.
Additional Features. Our calibration network’s input is only x, the vector of LLM responses on T . To give it access to other predictive features, x or z2 could also be augmented with a fixed-dimensional embedding of T (as already noted in footnote 4). The embedding function could be pretrained, or it could be fine-tuned jointly with the calibration network.
To avoid overfitting if the embeddings are high-dimensional, the embeddings can be replaced with 0 during initial training. When the embeddings are revealed in the next phase of training, they may reveal properties of the text that systematically cause the calibrated LLM to overestimate or underestimate the human judges’ answers to certain questions. The calibration network can then learn to use them to further correct its estimates.
This is an example of the general principle that regression can be more accurate with more regressors. For the same reason, it may be useful for x to include additional LLM questions (see footnote 2), which might cover additional criteria or use variant prompts. Ambitious questions might potentially ask the LLM to think step-by-step about the user’s goals and whether they are achieved (chain of thought), or to take on specific personas (Wang et al., 2024) that might reflect the values and needs of some human judges. If internal states of the (Transformer) LLM are available, x can be further enriched with information about how it computed each distribution pLLMpyi | T,Qiq, such as a high-layer encoding of the final token of the T,Qi prompt, which strongly influences this distribution.20 Similarly, x could include other features of T that are extracted by manual code or trained classifiers rather than by prompted LLMs. It could even include features of the judge a, which allows sharing parameters across similar judges—especially useful when end users are enlisted as judges (see Appendix B) or when data are . Finally, it may improve accuracy to include available metadata about T , such as its domain, date, and author—but such metadata
18As a caveat, this procedure assumes that the true yai values actually follow this joint distribution, i.e., that the calibration network is correct. To incorporate uncertainty about the network’s parameters as well, we would also have to retrain them each time on a bootstrap replicate of the training set. Then a small training set would also lead to wider uncertainty bands. We would also likely get wider uncertainty bands by modeling and sampling the judgments yai jointly (for each i). We currently model them as independent (see footnote 5), but this assumes that the errors ŷai ´ yai are uncorrelated. In fact they are likely to be positively correlated across judges a on the same text T and also across similar texts, since they are derived from the same or similar LLM response vectors x.
19Disaggregation therefore avoids averaging over judges. Even then, however, each ŷai is still itself a weighted average over possible responses by a. This inner average may be problematic as well (footnote 6). Still, it elides only uncertainty, not disagreement, so disaggregating it seems less useful.
20Thanks to Zhichu (Brian) Lu for this observation.
should be masked for predictions that will be used to compare performance on different domains, dates, or authors, so that the predicted scores are fair in the sense that they depend only on the text T .
Missing Features. The Limitations section suggested using an “adaptive rubric” to reduce the number of queries to the LLM at test time. An adaptive rubric would query the LLM dynamically to ask the most useful questions first and to ask only as many questions as are needed to predict target quantities such as ŷ0.
However, this requires being able to predict yai values even when some of x is missing. 21 If we train LLM-RUBRIC with dropout, then it will be able to handle this case. Furthermore, we can extend the LLM-RUBRIC output so that it predicts not only distributions over the human responses yai , but also distributions over the missing parts of x (Uria et al., 2016; Kachuee et al., 2019; Covert et al., 2023)—that is, over what the LLM might say if asked. This can be used for dynamically choosing the next LLM question, as follows.
Dynamic feature selection dates back at least to He et al. (2012). We envision an approach similar to that of Covert et al. (2023) and Zhong et al. (2023), which greedily selects the next LLM question Qi based on information gain—and thus must be able to compute how much the variance of the predicted ŷ0, for example, is expected to decrease after observing the LLM’s distributional answer to Qi, pLLMpyi | T,Qiq. This requires us to guess how the LLM is likely to respond to Qi, given its responses to previous questions (i.e., the part of x that has been observed so far).
Dealing with missing features is also necessary if the input feature set evolves over time. We may not wish to compute old features on new data, or new features on old data. Indeed, we may not be able to do so, if the feature has changed because the underlying LLM has been replaced with a new version.
Irregular Datasets. Our training objective (1) tries to predict yai for each pT, i, a, yai q tuple in the training dataset D. Any collection of tuples can be used. That is, it is not necessary to obtain answers to all human questions for every text, or to use the same judge for all questions on a text. This is often useful.
First, in practice, new texts T or questions Qi may periodically be added to the training dataset D to better cover the observed distribution of test data and to track newly identified issues. The set of judges may also change over time due to staff turnover. As new tuples are collected, LLM-RUBRIC can simply be retrained on the growing, heterogeneous dataset.
Second, perhaps not every question is applicable to every text, and not every human judge has the same expertise. Thus, each text T might select a different set of questions Qi, and might route different questions to different judges. A manually written policy can rule out inapplicable questions (for both human judges and LLM evaluators) by consulting text classifiers or the results of earlier questions. The applicable questions should be routed to judges with appropriate expertise,22 which—depending on the question—may include familiarity with T ’s topic, dialect, or type of user. Yoshikawa et al. (2021) review and propose methods for routing texts to judges—a problem that is closely related to dynamic feature selection above. Some questions may require special expertise independent of T , e.g., questions that assess the harmfulness or inappropriateness of a dialogue system’s response according to the policies of the system’s owner.
Third, even when it is reasonable to ask a particular question of a particular judge, doing so may not be the best use of a limited annotation budget. One may use an active learning workflow (Settles, 2012) that prioritizes annotations that are not already predictable by LLM-RUBRIC—that is, where p̂apyi | T,Qiq still has high variance after p̂a has been trained on previously collected data.
Fourth, in a dialogue system, we may be able to enlist our end users as additional judges, which is especially useful on private dialogues that only the users can see. For example, it is currently common to give users the opportunity to click or (which may in turn trigger followup questions). We regard this opportunity as just another rubric question, which is judged by the user (though not by the LLM). Note that
21We can represent a missing LLM response in x by having pLLMpyi | T,Qiq put all of its probability on a special value yi “ MASK R Yi.
22We remark that to complement the judges’ own expertise, one might equip them with information beyond T . That is, for some pa, iq pairs, the judge a could consistently be shown additional information, such as the output of a fact-checker or machine translation system, or the user’s reaction to the system response. The trusted judges A of overall quality (Q0) could be shown expert judges’ responses to other rubric questions, or their response distributions as predicted by LLM-RUBRIC.
this question is distinct from the question Q0 that asks for the overall quality of the text (which usually uses a Likert scale and which may ask about aspects of the dialogue beyond user satisfaction). The calibration network can be used to predict the user’s response to this question—that is, a click23—from various LLM questions. Some of these LLM questions may be designed to detect various kinds of verbal feedback from the user, such as praise or specific complaints, rather than assessing the system’s responses directly. In fact, Lin et al. (2024) present a method for automatically creating questions of this sort from private dialogues. Questions about verbal feedback may also be presented to additional human judges—though only on synthetic or other non-private dialogues—so that they contribute to multi-task regularization of LLM-RUBRIC and so that calibrated versions can be shown on a dashboard (Appendix A).
Heterogeneous Response Types. Equation (5) constructed a softmax distribution p̂a over a small finite response set Yi. But if some Qi demands real-valued responses (e.g., Yi “ R), then p̂apyi | T,Qiq for that i can simply be changed to a density model, where the calibration network predicts the parameters of some parametric density function from z2. Similarly, if some Qi demands textual responses (e.g., Yi “ Σ˚), then p̂apyi | T,Qiq can be changed to an autoregressive language model conditioned on z2.
Next, consider the case where Yi is finite but large. Here the matrices in equation (5) are large, so generalization might be improved by smoothing them. This can be done by parameterizing Vi “ YiUi and Vi “ YiUai , where the rows of matrix Yi serve as embeddings of the various responses yi P Yi. Similar responses should have similar embeddings. The unsmoothed case takes Yi to be the identity matrix (yielding one-hot embeddings), but using a learned matrix Yi with fewer columns can reduce the number of parameters. In some cases, Yi does not even need to be learned: pre-trained word embeddings can be used if Yi is a natural-language vocabulary, and systematic number embeddings (Gorishniy et al., 2022) can be used if (e.g.) Yi “ t1, . . . , 100u.
The preceding paragraph treats large response sets for human judges, which are predicted by the output of the calibration network. If the LLMs are also permitted to use large response sets, which appear in the input of the calibration network, a similar solution applies: premultiply the vector pLLMpyi | T,Qiq by Y Ji to reduce its dimensionality before including it in x. For infinite response sets as in the first paragraph, standard schemes can be used to embed numbers (Gorishniy et al., 2022) or text (Devlin et al., 2018).
Finally, returning to the setting of our own experiments, we observe that when Yi is an ordinal scale with n possible responses, such as a Likert scale, it is not strictly necessary to use a flexible softmax distribution as we did in equation (5). Instead, yai could be modeled with fewer parameters as a quantized version of an underlying real value rai whose distribution is predicted by the calibration network.
24 Furthermore, we could then (if desired) evaluate the text T using our best prediction of the underlying rai rather than of the observed yai (e.g., using the expected value as before, if we wish to minimize expected L2 loss). The intuition is that the reconstructed unquantized rai values contain more information than their quantized versions—and that they might also be more comparable across judges a, if their different judgments (e.g., in Figure 2) mainly arise from different quantization boundaries.25
23“No click” will usually have probability close to 1. To avoid a large number of low-information training examples, one can downsample the “no click” examples in training data from this domain, provided that x indicates whether the example comes from a downsampled domain (since this kind of downsampling will shift the priors on many questions yi toward more extreme responses, and thus should shift the hidden features z1, z2 guessed from an ambiguous example x). Also, to control the number of parameters in a system with many users, a reasonable simplification is to fix W ak “ 0 when a is a user. Then for each user a, we only have to learn a matrix V ai with two non-zero rows (for and ; the row for no response can be fixed at 0, WLOG). Note that in the common case where user a has never provided any explicit feedback, so that V ai “ 0, the backoff matrix Vi still ensures a reasonable prediction—particularly if a’s demographics and/or user behavior are represented in x when predicting the answer to this question, allowing the network to share statistical strength with similar users.
24That is, p̂apyi | T,Qiq “ p̂apri P binyi | T,Qiq where rai has a normal (or logistic) distribution whose 2 parameters are predicted from T by the calibration network for a, and where the n bins are a partition of R by n ´ 1 learned thresholds that are specific to a or to pa, iq. This gives a discrete distribution over yai , which can be used in the log-likelihood objective (1). This is a nonlinear, heteroskedastic version of ordered probit (or logit) regression.
25However, a trick is needed to ensure that rai values are interpretable and comparable across judges a. The issue is that the method in the previous footnote does not identify the position or scale of rai . (If we adjusted our model to double the predicted means, predicted standard deviations, and thresholds for judge a, we would get exactly the same distribution over observables yai and achieve the same log-likelihood. But rai would now have twice the range and so would count more in a mean over judges A.) To break this tie, we can augment the log-likelihood objective with a second term (perhaps with infinitesimal weight) that does care about position and scale. Assuming that our ordinal scale Yi is numeric, a natural choice for this second term is the
Comparative Judging. Our maximum-likelihood training principle (1) can be extended to other question types. In particular, Qi “ “Does T or T 1 score higher on criterion i?” can be interpreted as a comparison of underlying real values as in the preceding paragraph: the human judge is being asked whether ri ą r1i. The calibration network could predict the probability of a “yes” response either directly, or else by integrating over a latent distribution p̂apri, r1i | T, T 1, Qiq (perhaps modeling it as p̂apri | T,Qiq ¨ p̂apr1i | T 1, Qiq via an independence assumption).26

heading: C LLM-RUBRIC Questions
text: These were the questions in our evaluation rubric. The human and LLM prompts in which these questions appeared are given in Appendix D and Appendix E respectively. When presenting the questions to the LLM (Appendix D), boldface was omitted. When presenting them to human judges on real data (Appendix I), boldface was again omitted, and the response choices were not numbered; instead, radio buttons were used (Figure 3b).
Question instances where the correct answer was “NA” were not included in our training dataset D and were not used for evaluation.
Q1– In terms of naturalness and tone of the assistant utterances, to what degree are they likely to be produced by an intelligent human in a conversation? Disregard whether they are grounded in the search results.
1. Unlikely. 2. Somewhat unlikely. 3. Somewhat likely. 4. Likely.
Q2– If the references are provided, to what degree user’s questions can be answered or resolved using the references? The assistant’s responses should not impact your response to this question. If no references are provided in the conversation, please write “NA” for this question.
1. None of the questions that user has asked could be answered using the reference documents. 2. Less than half of documents that user has asked could be answered using the reference document. 3. Half or more than half of the questions that user has asked could be answered using the reference
documents. 4. All the questions the user has asked could be answered with the reference documents.
Q3– Independent of what sources are cited in the conversation, to what degree the claims made by the assistant are followed by a citation. If no references are provided in the conversation, please write NA.
1. None of the claims are followed by a citation. 2. Less than half of the claims are followed by a citation. 3. Half, or more than half of the claims are followed by a citation. 4. All claims are followed by a citation.
Q4– What percentage of citations accurately support the claims made in the conversation? If no references are provided in the conversation, please write NA.
1. None of the citations accurately support the provided claims. 2. Less than half of citations accurately support the provided claims. 3. Half, or more than half of citations accurately support the provided claims. 4. All citations accurately support the provided claims.
unquantized log-likelihood: that is, we ask the normal curve to assign a high log-density to the exact value yai and not just a high log-probability to its bin. This ties ri to the yi scale, making it interpretable.
26Unfortunately, any reporting of the predicted rai values (e.g., r̂ a i ) as quality metrics runs into the same non-identifiability problem as in the previous footnote. We cannot know the position or scale of a judge’s rai values if we only observe the results ofą comparisons. A simple fix is to apply an affine transform to each judge’s rai values so that on a given reference set of texts, the transformed values have mean 0 and variance 1. Then report these transformed values.
Q5– To what degree the cited sources are the best candidates among all the provided sources? If no references are provided in the conversation, please write NA.
1. For all citations, there is a better source to be cited. 2. For more than half of the citations, there is a better source to be cited. 3. For half or less than half of the citations, there is a better source to be cited. 4. The best sources are cited in all cases.
Q6– To what degree the content of the assistant utterances is free of redundant elements, such as repetition, overspecification, etc.
1. The conversation has a large number of redundant elements. 2. The conversation has some redundant elements. 3. The conversation has a few redundant elements. 4. The conversation is completely free of redundant elements.
Q7– To what degree the assistant responses are concise? 1. In all assistant utterances, the responses could have been shorter. 2. In more than half of the assistant utterances, the responses could have been shorter. 3. In half, or less than half of the assistant utterances, the responses could have been shorter. 4. In all assistant utterances, the responses are concise and the utterance length is appropriate.
Q8– Do you think the number of exchange turns or back and forth is appropriate given the complexity of the user information need?27
1. No, fewer interactions would be sufficient and would make this conversation more pleasant. 2. No, more interactions are needed for a better conversation experience. 3. Yes, the rate of exchanges between the user and the assistant is reasonable.
Q0– Imagine you are the user who had this conversation with the assistant. All in all, how you would rate your overall satisfaction while interacting with the assistant? The higher the rating, the better the experience.
1. 1 2. 2 3. 3 4. 4

heading: D Evaluation Prompt for LLM
text: In our LLM-RUBRIC experiments (§4), we use the following prompt template to ask the LLM an evaluation question Qi about a conversational text T .
The variable {conversation} is the complete dialogue between the user and the assistant, and the variable {question} is one of the questions from the evaluation rubric presented in Appendix C.
The citation-related questions Q2, Q3, Q4, and Q5 are not presented to the LLM if no references are provided in the conversation. In this case, we simply pretend that the LLM would have correctly answered “NA,” which means that the probability vector over the responses 1–4 is r0, 0, 0, 0s (see footnote 3).
You are given a conversation between a user and an intelligent assistant for an enterprise chat scenario. In some cases, some references and citations are provided to back up the claims made by the intelligent
27For Q8, the numeric responses unfortunately do not form an ordinal scale. Response “3” should reasonably be considered closer to “1” than it is to “2”. Thus, L2 is not an appropriate loss function here. However, for simplicity we did still use L2 when decoding ŷa8 (it motivates equation (2)) and when evaluating the quality of ŷa8 (it motivates RMSE). This affects only the Q8 line of Table 3, all of whose metrics would presumably be improved if we fixed the problem by swapping “2” and “3” in both the human data and the LLM data. All of our other results would be unaffected by this relabeling.
assistant. Your primary job is to evaluate the quality of the conversation based on a criterion. To do so, read the conversation and references, and answer the followed question, by selecting only one of the choices.
Conversation: {conversation} Question: {question} Only print ’1’, ’2’, ’3’, or ’4’.

heading: E Evaluation Prompt and Preliminary Data Quality Questions for Humans
text: Below are the instructions we gave to human judges with the main questions in Appendix C. The preliminary questions DQQ0–DQQ2 are used only to screen for problems with the generated synthetic dialogues of §3.2 (see Appendix G). They are not included when the human is judging the real dialogues of §3.3. Note that if the answer to DQQ is “No,” then the remaining questions are not answered, which is why our synthetic training dataset D had only 741 examples rather than 750.
You are given a conversation between a user and an intelligent assistant for an enterprise chat scenario. You are also given an information need that the user wants to fulfill through the course of the conversation (e.g., a problem the user faces and wants to resolve). In some cases some references and citations are provided to back up the claims made by the intelligent assistant. Each assistant utterance can only cite the references listed in the adjacent cell in the table. Your primary job is to evaluate the quality of the conversation through a series of criteria that we define later in the document. To evaluate the conversation, you need to answer a questionnaire. Each question captures one evaluation criteria that we care about. Read about the definition of labels criteria below:
Naturalness (both content and form): The degree to which the form and content of the conversation is realistic, and likely to happen in real-world. To measure naturalness you should answer below questions:
DQQ0- Is this a conversation between a user and an assistant? 1. Yes 2. No (if you select ‘No’, you can skip the rest of the questions)
DQQ1- To what degree the user tries to fulfill the information need during the course of conversation? 1. The conversation is not about the user information need at all. 2. The conversation does not exactly address the user information need, but it is somewhat related. 3. The conversation addresses the user information need but it also talks about other topics. 4. The conversation only addresses the user information need.
DQQ2- To what degree the form and content of the user utterances are likely to be produced by a human in a conversation?
1. Unlikely. 2. Somewhat unlikely. 3. Somewhat likely. 4. Likely.
{Q1}
Citation quality: To what degree the claims made by the assistant are backed by reliable sources. Note that not all the sentences in a conversation require citation; only facts and claims need to be cited. To measure citation quality answer the following questions:
{Q2} {Q3}
{Q4} {Q5}
Dialogue efficiency: To what degree the dialogue has been conducted in an cost effective manner. To measure the dialogue efficiency answer the following questions:
{Q6} {Q7} {Q8}
User Satisfaction: Answer the following question to rate the overall user experience with the assistant.
{Q0}

heading: F Synthetic Dialogue Generation
text: This section describes the 5 approaches that we used in §3.2 to generate a variety of synthetic dialogues.
DS1: LLM-Only Assistant with Simulated User. In our baseline, the dialogue system has no access to external documents and can only answer the user from its internal knowledge. In this setting, the assistant cannot provide citations for its claims.
DS2: Oracle RAG Assistant with Oracle Simulated User. In this variant, the prompt includes highly relevant documents: the 5 documents that were most frequently clicked when the given topic appeared as a query in the real logs of §3.1. Thus, the assistant is essentially a RAG system with unrealistically good retrieval. In addition, the simulated user is unrealistically knowledgeable, having full access to the same documents for the initial question and all followup questions.
DS3: RAG Assistant with Oracle Simulated User. This variant resembles DS2, except that it uses the 5 documents that are most similar to the topic string according to the BM25 metric. We use the ElasticSearch28 implementation of BM25.
DS4: RAG Assistant with Simulated User. This variant resembles DS3, but the topic is included in the prompt only when generating simulated user turns, and the 5 documents are included in the prompt only when generating assistant turns. In addition, the BM25 query is not the topic string but rather the dialogue history (all past utterances); thus, each assistant turn may be prompted using a different set of 5 documents.
DS5: Retrieval-Augmented Dialogue Generation + Query Generation with Simulated User. This variant resembles DS4, but the BM25 query is not the dialogue history. Instead, it is derived from the dialogue history by a separate prompt to the LLM (also shown in Table 6). This may be required as calling a query generation tool.
The prompts used for synthetic dialogue generation (DS1–DS5) are presented in Table 6.

heading: G Quality of the Generated Synthetic Dialogues
text: As mentioned in §3.2, each of the systems DS1–DS5 (Appendix F) was used to generate 50 synthetic dialogues, each of which was evaluated by 3 human judges, resulting in 5 ˆ 50 ˆ 3 “ 750 completed questionnaires. The first question we asked (DQQ0) was “Is this a conversation between a user and an assistant?” As expected based on the findings presented in (Li et al., 2023), the answers to this question were vastly positive: 98.8% of the dialogues received a positive answer.
Table 4 shows the mean and standard deviation of the human judgments for the questionnaires that passed the DQQ0 quality check. The results on DQQ1 and DQQ2 suggest that all systems often simulated
28https://www.elastic.co/
the user turns competently, and the results on Q1–Q8 and Q0 suggest that all systems often produced reasonably good assistant responses to these simulated users. In fact, the DS2 and DS3 systems obtained an average ě 3.0 over their dialogues for all questions (except for Q8, where the response scale is 1–3).
Of course, the point of our LLM-RUBRIC experiments is not to generate good dialogues but to determine which dialogues show more satisfactory behavior by the assistant. These generated dialogues simply provide synthetic training and development data for that task.
Questions on the naturalness of dialogues (DQQ1, DQQ2, Q1). Table 4 indicates that system DS1 produces the most natural conversations. This is a non-RAG system that simply asks the LLM to write a plausible dialogue on the given topic. The other four systems perform comparably in terms of generating natural dialogues. DS2 performs slightly better than the rest; this it may be due to the high quality of its references, which can be less noisy and confusing than the other variants.
Questions on citations (Q2, Q3, Q4, Q5). On citation quality and usage, DS2 achieves the highest average rating, thanks to its “oracle” RAG. Among the methods that perform RAG with various BM25 queries, DS3 and DS4 perform slightly better than DS5, which prompts an LLM to generate the BM25 query.
Questions on conciseness (Q6, Q7, Q8). All systems are similar at generating an appropriate number of turns (Q8). DS2 and DS3 seem to have less concise dialogues (Q6, Q7), perhaps because the simulated user has access to the retrieved documents.
Question on overall satisfaction (Q0). The results suggest that the quality of retrieved documents is the most important factor for our judges, with DS1 clearly doing worst and DS2 doing slightly better than the others.

heading: H The User Interface for Human-Agent Dialogue Collection and Evaluation
text: We designed a web interface (Figure 3) to enable humans to converse with a dialogue system as users and then evaluate their interactions as judges (§3.3). In each session, the website shows the human a random Azure-related topic from the set described in §3.1, and randomly selects one of the dialogue systems DS1–DS3 for the human to converse with. The human does not know which system was selected (although DS1 might be distinguishable as it does not give citations).
This is a standard guided data collection procedure that has been previously used in prior work (Zamani et al., 2023). If the user does not understand the topic, they may refresh the website to get a different topic. Once the user is finished with their conversation, they click on the ‘End of Conversation’ button and judge the conversation (see Appendix C).

heading: I Evaluating the Collected Human-Agent Dialogues
text: We asked 13 trained judges to use the website for dialogue collection and evaluation. The judge set for the synthetic dialogues presented above includes these 13 judges. We collected a total of 223 conversations, ranging from 14–27 conversations per judge. The judge scores for the three dialogue systems evaluated are summarized in Table 5.

heading: J How much human judge data is needed to train calibration?
text: We plot learning curves in Figure 4. To make these plots, we train the model on the synthetic data and test on the real conversation data, but reduce the training portion of the data to a random x% sample. To reduce the impact of random selection, we repeat this process 50 times and plot the average performance, ˘ 1 standard deviation. As expected, the average performance improves and drops in variance as we increase the amount of training data per judge.29
Performance is reasonably good with even 20% of our training set, and appears to have essentially converged by the time we reach 80–100% of our training set. (Here 100% represents 741 dialogues, where each judge has evaluated only „30 dialogues on average.) Further improvements would, therefore, require more dimensions or more accurate modeling of each dimension, or perhaps training data targeted at fixing the residual errors.

heading: K Calibration Plots (Reliability Diagrams)
text: Well-trained neural networks tend to produce well-calibrated probabilities (Niculescu-Mizil and Caruana, 2005), an outcome that is incentivized by the log-likelihood training and validation objectives. Figure 5 shows smoothed calibration plots when training and evaluating our system on synthetic dialogues, as explained in §6. The system is indeed well-calibrated, meaning that the red curves stay close to the diagonal, as measured by smoothed expected calibration error (smECE).
The plots are produced by the relplot package30 of Błasiok and Nakkiran (2023), using 5-fold crossvalidation. All graphs plot the same examples, namely the tuples pT, i, a, yai q P D, where T is a synthetic dialogue. However, each graph considers the calibration for a different possible score y0. Tick marks just above (or below) the horizontal axis are a sample of held-out examples for which the true judgment yai is (or is not) y0. Their position along the horizontal axis shows their predicted probabilities p̂apy0 | T,Q0q under cross-validation. Thus, the tick marks above the axis (true score is y0) tend to appear farther to the right (y0 is predicted with high probability).
For each predicted probability p, the height of the red curve estimates the actual probability that yai “ y0 among held-out examples where p̂apy0 | T,Q0q « p. One may think of this visually as the approximate fraction of tick marks that are plotted near p on the horizontal axis that are just above the axis rather than just below. The thickness of the red curve at p corresponds to the density of tick marks near p. The gray band around the red curve is a 95% bootstrap confidence interval.
The smoothed expected calibration error (smECE) is the average absolute distance between the height of the red curve and the diagonal, weighted by the thickness of the curve. In other words, it estimates the average difference between the predicted and actual probability for a random held-out example. The smECE number for each graph is printed on the graph with a 95% confidence interval.
Calibration of p̂a is important because it means that the predicted probabilities are meaningful. The system can use them to assess its own uncertainty and make decisions accordingly. Some applications in our case:
• Text evaluation. The expected score (2) will be approximately unbiased: that is, on average over held-out examples, it will match the judge’s actual score. Table 1 assesses this match directly. Beyond expected score, various other interesting quantities that we might derive from p̂a are also approximately unbiased: for example, the probabilities that the score is ď 1, ď 2, and ď 3.
• Text selection. At runtime, a dialogue system might generate several candidate responses, and then choose the one with maximum expected reward. If the reward has the form ř i,a fi,apyai q, for any set
of reward functions fi,a, then its expectation under p̂ will be unbiased.
• Dynamic feature selection. LLM-RUBRIC can be sped up at test time by asking fewer questions of the LLM. As briefly mentioned in §8 and Appendix B, p̂a can be used to greedily choose the
29The reduction in variance is partly because the larger training sets are more similar to the population and thus to each other, but also because they overlap more and thus are less independent.
30https://github.com/apple/ml-calibration
question with the greatest information gain—that is, whose answer is predicted to most reduce the variance of the evaluation ŷ0 or most reduce the entropy of a text selection decision.
• Distillation. LLM-RUBRIC can be used to stochastically label a large dataset of naturally occurring texts according to p̂a (“multiple imputation”). A faster scoring function can then be trained to have low loss on this dataset.
• Rubric improvement. p̂a can be used to identify difficult texts where LLM-RUBRIC is unsure what judge a would say, or controversial texts where LLM-RUBRIC predicts that two judges will disagree more than usual. This can be used to improve the LLM questions or the human questions, respectively.
As a caveat, the plots in Figure 5 measure calibration only for the dataset as a whole. One could create calibration plots for subsets of the data to verify that the model remains calibrated within each user category, judge, or dialogue topic that is sufficiently represented in training data—as one would expect with maximum-likelihood training—rather than overestimating probabilities in some categories and underestimating them in others. The correlation coefficients in Figure 5 do show that the predicted scores provide a reasonable ranking of examples.

heading: Future Work
text: This suggests a future strategy (discussed below) of improving the input to LLM-RUBRIC by getting the LLM to better predict the judge-specific human rubric responses that were available to the Oracle (row ‘a’), or at least judge-independent versions (rows ‘e’–‘f’). Once such responses are available, the ensembling is still best done by a calibration network that understands an individual judge’s preferences—though under oracle conditions and with our population of judges, dropping that personalization would not be dramatically worse (row ‘c’), and a fraction of the difference can be made up simply by adjusting the predicted scores ŷ0 with personalized isotonic regression (row ‘d’)