Can Large Language Models Capture Dissenting Human Voices?
Authors: Noah Lee
Na Min An
James Thorne
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Model
Natural Language Inference
Human Disagreement
TL;DR: Large language models (LLM) show limited inference and human alignment performance on the traditional evaluation task of natural language inference (NLI).

Abstract: Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely align with the human disagreement distribution has not been well-studied, especially within the scope of natural language inference (NLI). In this paper, we evaluate the performance and alignment of LLM distribution with humans using two different techniques to estimate the multinomial distribution: Monte Carlo Estimation (MCE) and Log Probability Estimation (LPE). As a result, we show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution. The inference and human alignment performances plunge even further on data samples with high human disagreement levels, raising concerns about their natural language understanding (NLU) ability and their representativeness to a larger human population.

Can Large Language Models Capture Dissenting Human Voices?
Noah Lee"
Na Min An"
James Thorne KAIST AI
KAIST AI
KAIST AI
noah. moah.lee@kiist..o ac.
naminan@kaist. ac kr
thorne@kaist.c wtt.ekaiistt............ ac.kr
Abstract
1) Generation
Large language models (LLMs) have shown im- pressive achievements solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely align with the human disagree- ment distribution has not been well-studied, es- pecially within scope natural language inference (NLI). In this paper, evaluate the performance and alignment of L.LM dis- tribution with humans using two different tech- niques to estimate the multinomial distribution: Monte Carlo Estimation (MCE) and Log Prob- ability Estimation (LPE). As result, show LLMs exhibit limited ability in solving NLI lasks and simultaneously fail to capture human disagreement distribution. The inference and human alignment performances plunge even further on data samples with high human dis- agreement levels, raising concerns about their natural language understanding (NLU) ability and their representativeness to larger human population.'
Read the following and determine if the hypothesis Inferred from the premise.
Premise:
She smiled back.
Hypothesis: was so happy she couldn't stop smiling,
OPTIONS: entailment, contradiction, neutral
Answer:
MCE
LPE
2) Estimation
ent
neutral
nour
contradictory
C
3) Comparison
VS.
LLM
Human
Figure 1: The Proposed LLM Distribution Estimation Techniques, MCE and I.PE. We estimate LLM disagree- with either MCE LPE utilizing generated LLM outputs and compare the estimated LLM distribution with human disagreement distribution.
1 Introduction
Natural language inference (NLI) has long served as fundamental testhed to evaluate the ability of model to recognize entailment and capture plausi- ble inference relations between pairs of sentences (Dagan et al., 2006; Bowman et al., 2015; Williams et al., 2018 8). When constructing datasets, conven- tional processes result in single label per instance even if multiple annotators contribute, which lim- its the full representation of diverse opinions that might arise in a larger human population. Thus, recent datasets have become more attentive to in- corporating multiple interpretations (Pavlick and Kwiatkowski, 2019; Nie et al., 2020b; Glockner et al., 2023) to capture dissenting human opinions.
Meanwhile, instruction fine-tuning large lan- guage models (LLMs) has elicited remarkable gen- cralizability to diverse unseen tasks (Zhao et al., 2023). Not only can they generate free-form texts, but they can also select one answer from multiple options given in the input prompt. However, while many works study user interaction and conversa- tional usage (Liang et al., 2022), limited works evaluate these instruction-folloviii LLMs foundational NLI task. Therefore, aim an- swer the following questions: Can LLMs capture dissenting voices that naturally arise in the dataset? Are LLMs representative the voices of the anno- tators in inference tasks?
"Equal contribution
'The source code for the experiments is available at https//jtt...ooooftat///////////

---

With this in mind, we jointly assess on a number of instruction-folloving LLMs, Flan-T5 (Chung et al., 2022), Flan-UL2 (Tay et al., 2022), OPT- IML-Max (Iyer et al., 2022), and GPT-3 (Ouyang et al., 2022), on their performance on human opin- ion distribution datasets ChaosNLI (Nie et al., 2020b) and PK2019 (Pavlick and Kwiatkowski, 2019). For the process of using the model output distribution estimate of human disagreement distribution, we offer novel estimation methods: Monte Carlo Estimation (MCE) and Log Probabil- ity Estimation (LPE) (Figure 1).
the majority vote from these additional annotations, 22% of the instances exhibited change in label compared the original dataset (Nie et al., 2020b).
To characterize and reproduce the extent of hu- man disagreement in NLI tasks, previous works directly fine-tuned language models (Nie et al., 2020b) and implemented distribution estimation methods (Zhou et al., 2022) using the labeled data. Other studies have constructed losses to better cal- ibrate the ambiguity (Meissner al., 2021) and proposed an ensemble models to detect disagree- ing samples (Zhang and de Marneffe, 2021).
We find that the state-of-the-an GPT-3 model does not outperform smaller models such as fine- tuned BERT (Devlin et al., 2019) and partially fine- tuned Flan-T5-XXL in solving inference problems. Furthermore, it yields higher tance (JSD) (Endres and Schindelin, 2003) and Distribution Calibration Error (DCE) (Baan al., 2022) than BERT for the Cha datasets. Each model is optimized using different estimation meth- ods and prompt types, where GPT/Flan-T5-XX0 attains the best performances in NLI capability and human alignment when using LPE/MCE. Our pa- per's contributions are as follows:
For measuring the distance between two distri- butions, Kullback-Leibler (KL) Divergence (Kull- back and Leibler, 1951) or its symmetric version, Jensen-Shannon Distance (JSD) (Lindres and Schin- delin, 2003) are widely used. Baan al., 2022 argued that Expected Calibration Error (ECE), the difference between the average accuracy and confi- dence (Naeini et al., 2015; Guo et al., 2017), cannot capture inherent human disagreement. Therefore. for models to better calibrate to human disagree- ment, accuracy-agnostic metrics such DCE have been introduced (Baan al., 2022).
Dis-
To the best of our knowledge, we are the first to test generative LLMs jointly the perfor- mance and human disagreement on NLI.
2.2 Alignment of Instruction-tuned LLMs
LLMs have demonstrated the ability follow ex- amples provided in-context (Brown et al., 2020) and have further been developed to follow natural language instructions (Mishra et al., 2022; Ouyang et al., 2022; Chung et al., 2022). Instruction- following L.MMs are fine-tuned with various tasks and are expected to generalize well to tasks the model was not trained on (Zhao et al., 2023). For example, GPT-3 is fine-tuned using reinforce- ment learning with human feedback to produce responses that align with human values (Ouyang et al., 2022). Despite such efforts, Santurkar et al., 2023 identified that LLMs capture only a single perspective, exhibiting left-leaning tendencies and excluded demographic groups. Here, we study whether LLMs appropriately reflect diversity of viewpoints in the NLI task setting.
We suggest two probability distribution esti- mation techniques for LLMs to represent dis- agreement and perform empirical evaluations to with respect to the human disagreement distribution.
We study the model sensitivity to estimation methods and prompt types to demonstrate how these contribute to the ability of models to represent human-level disagreement for NLI.
2 Related Works
2.1 Disagreement in NLI
Considering only a single label in NLI datasets is bound to fail in capturing the diverse range of user opinions and could lead to misrepresentations of language models. To measure inherent human disagreements in NLI, Nie et al., 2020b and Pavlick and Kwiatkowski, 2019 collected large number of human annotations (e.g., 100 and 50 annotations for ChaosNL.I and PK2019) per instance for common NLI datasets such as SNLI (Bowman et al., 2015) and MultiNL (Williams et al., 2018). When taking
3 Methods
We estimate and quantify dissenting human voices using the multinomial soft-label distribution
LLMs with two proposed methods:

---

3.1 Log Probability Estimation (LPE)
4 Experimental Design
We use a single instance returning log probabilities of top-k² token candidates to estimate the categor- ical distribution of the labels. This method sums over all valid options* (vz) to estimate the model probability for class j. a method often adopted in a multiple-choice style evaluation of generative lan- guage models (Hendrycks et al., 2021; Santurkar et al., 2023). Although the LPE method requires single generation for each instance, it cannot be applied to all types of models Additionally, the method is limited in cases where more than one to- ken is generated as it requires exhaustive mapping of the determining token probability. Furthermore, as models only return prohabilities for top-k tokens, there is an unknown non-constant probability mass. We estimate this as follows, where C' is the total number of classes of the task:
4.1 Data
First, we test the inference ability of LLMs in chal- lenging datasets, ANLI (Adversarial NLI) (Nie et al., 2020a) and QNLI (Wang et al., 2018). We opt for the round version of ANLI (n = 1,200), which contains more contexts from diverse do- mains such as Wikipedia, Common Crawl, Sto- ryCloze (Mostafazadeh et al., 2016), and CBT (Hill et al., 2016). QNLI (Wang et al., 2018) (n = 5,463) is converted from the Stanford Question Answer- ing Dataset (SQuAD) (Rajpurkar al., 2016) to NLI dataset, and the task is decide whether the sentence contains the answer to the question.
Second, we jointly evaluate LLMs on ChaosNLI (Nie et al., 2020b), and PK2019 (Pavlick and K Kwiatkowski, 2019) to examine both the accuracy and how the model distribution aligns with the hu- man disagreement distribution. These datasets con- sist of two task settings: ChaosNLI-a (n 1,532), where models must select one of the two hypothe- ses, and ChaosNLI-S (n = 1,514), M (n = 1,599), and a subset? of PK2019 (n 299) where mod- els must assign the relationship (e.g., entailment, contradiction, or neutral) for a pair of premise and hypothesis. We also pick out a challenging sub- set of the ChaosNLI datasets, which we denote HighChaosNLI consisting of the top 100 samples having the greatest human disagreement level.
p(y]))== 20 Et exp lp; 1ievs
2j-1 Ek exp explpr- 1ievy
3.2
Monte Carlo Estimation (MCE)
Decoding strategies such as beam search or greedy search do not exploit the full distribution of the pos- sible generation options. Furthermore, API-based language model services limit the number of re- turned token-level probabilities. Alternatively, to reconstruct the distribution outputs from genera- tive L.Ms, we introduce an intuitive way that sam- ples a large numbers of generated outputs consider- ing the valid options" (vi) for class j. This method is based on a Monte Carlo method (Metropolis and Ulam, 1949) to estimate the probability distribution. Even though the MCE method can be computation- ally expensive, it can be applied to any model and prompt type to capture the multinomial distribu- tion of a classification setting. MCE is defined as follows:
Lastly, to trace possible causes of the disagree- ment occurring in LLMs, we use the round ver- sion of the DisagreementNLL dataset (n = 318). where the samples from ChaosNLI are annotated with one of the 10 categories (e.g., probabilistic) of potential sources of disagreement (Jiang and Marn- effe, 2022). While the primary focus is slanted towards identifying why humans disagree, we uti- lize and link the disagreement taxonomy un- cover whether the disagreement in LLMs aligns with those of humans.
4.2 Models
x()) ~
1ievy
(2)
We categorize numerous LLMs with varying levels of supervision on the NLI task Full Exposure (FE), Partial Exposure (PE), Minimum/ Unnknowwnnnnknwnnnnnno Exposure (MUE), and No Exposure (NE). For FE models, we follow the baseline setup of Nie et al., 2020b by fine-tuning BERT (340M) (Devlin et al.,
"k is set to for all the models to match the maximum logprobs size of OpenAl Completion API.
"See Appendix C for examples.
3.PT-3.5-Tufe Turbbo does not support logprobs.
'aamlle size 100 is heuristically chosen to match the
'JOCI& & DNC datasets of PK2019 are discarded as the annotation setting greatly varies from ChaosNLI. *See Appendix B for more details.
size of human unnotation for ChaosNLI.
'See Appendix C for examples.

---

LPE (NS)
MCE (NS)
MCE (OS)
Model
Acct
JSDI
DCE1
Acct
JSD1
DCE1
Acct
JSD4
DCEL
Flan-T5-L. (780M)
59.3
0.293
0.326
62.3
0.289
0.321
59.7
0.290
0.322
Flan-T5-XL (3B)
65.7
0.253
0.282
72.0
0.236
0.254
70.3
0.238
0.256
Flan-T5-XXL (11B)
68.7
0.258
0.277
71.0
0.263
0.277
74.3
0.232
0.244
Flan-UL22 (20B)
67.7
0.260
0.281
72.3
0.247
0.259
76.0
0.241
0.246
OPT-IML-M-S (1.3B)
57.0
0.294
0.337
54.7
0.312
0.354
59.3
0.298
0.337
OPT-IML-M-L (30B)
72.0
0.273
0.286
62.0
0.280
0.303
72.7
0.233
0.252
GPT-3-D3 (175B)
66.7
0.330
0.345
67.0
0.334
0.349
58.0
0.344
0.376
GPT-3-D2 (175B)
64.0
0.282
0.317
62.7
0.279
0.313
49.3
0.315
0.364
Stable Vicuna (13B)
45.7
0.328
0.379
43.7
0.504
0.568
41.7
0.502
0.567
Table 1: Human Alignment Performances of LLMs on Subsets ChaosNLI Datasets with Different Estimation Methods LPE/MCE (Prompt Types Shuffled NS/OS). We present the average results of ChaosNLI-oc, S, and M; for each dataset, we randomly sample 100 instances. The model categorizations are the same as Table Bold texts indicate the best value for each model and metric.
OPTIONS: hypothesis hypothesis 2
4.3 Prompt Types
OPTIONS: entaiiment, contradiction, neutral
We adopt mostly the same prompt template" across different types of models within each sub- dataset. Within a dataset and model, we test two types of prompts: (1) Option Selection (OS), which the model has to predict the name of the class label for the entailment relation, and (2) Num- ber Selection (NS), in which the model has to select the number assigned the relationship class (Figure 2). Additionally, LLMs are known to be sensitive to minor input modifications (Liang al., 2022; Sun al., 2023), we test the effect prompt variations over a single prompt. NS requires the model to predict a single token of target number and can be used with both MCE and LPE. OS, the other hand, is not considered in the LPE formulation encourage scalable, comprehensive generation strategy to estimate hu- man disagreement distribution since we allow LPE-OS, the token -seecfi probability of a model output which may vary by instance/datasetttask has to be mapped per class. We implement random ordering of the options in the prompt, as also men- tioned in Santurkar et al., 2023, to mitigate the sensitivity due the order of the options, which call shufiled OS and NS throughout the paper.
OPTIONS: hypothesis 2: hypothesis 2
OPTIONS: 1: entailment, 2: contradiction, neutral
Figure Two Prompt Types OS and NS for Two Types of Tasks. The former does not have number in front of each option choice.
2019) and RoBERTa (355M) (Liu et al., 2019). Since instruction-folloiingg LLMs not have full supervision of NLI, we assign these LLMs to one of the PE, MUE, and NE models.
First, the PE models include Flan-T5 (780M,
3B, I1B) (Chung et al., 2022), Flan-UL.2 (20B)
(Tay et al., 2022), and OPT-IML-Max (1.3B
and 30B)a (Iyer et al., 2022). We label
GPT-3-D2 (text-davinci-o0) (text-davinci-003) (175B) (Ouyang al. 2022) MUE models because, although the models variants from Ouyang et al., 2022, it is unknown which extent the model exposed to NLI tasks. Finally, the sole NE model that we test is Stable Vicuna (13B) (Chiang et al., 2023)10 because has no exposure to NLI tasks. All the hyperparameters we use to generate the outputs of these models are listed in Appendix A.
and GPT-3-D3
4.4 Metries
We investigate the distribution differences between humans and LLMs the sample level with JSD, which is a symmetric version of KL divergence
"hhee models will referred as OPT-IML-M-S OPT: IML-M-L for convenience. 10 Results of poor-performing models such as Stanford Al- paca (7B) and Dolly-v2 (12B) are not reported.
Refer to Appendix for specific prompt examples. 12 multiple-choic format similar to the prompt suggested in the MMLU Benchmark (Hendrycks et al., 2021)

---

Model
ANLI-R3
QNLI
ChaosNLI-o
ChaosNLI-S
ChaOsNLI-M
PK2019
Chance
33.3
50.0
50.0
33.3
33.3
33.3
Full Exposure (FE)
BERT-L (340M)
43.5
92.7
68.2 (+0.2)
73.8 (+1.2)
56.9 (-4.3)
RoBERTA-L' (355M)
44.4
98.9
83.7 (+1.6)
78.7 (+3.8)
63.5 (-3.9)
Partial Exposure (PE)
Flan-T5-L (780M)
46.3
90.2
73.1 (+1.9)
54.8 (-4.6)
59.7 (+7.8)
76.6 (+6.5)
Flan-T5-XL (3B)
54.3
93.1
83.3(+1.2)
71.2(+11)
60.2 (+1.0)
76.9 (-7.4)
Flan-T5-XXL (11B)
58.2
93.7
84.9 (+1.6)
67.9 (+0.8)
72.6 (+8.5)
82.1 (-0.6)
Flan-U1.2 (20B)
56.6
94.9
86.5 (+1.8)
(+6.4)
71.7 (+4.8)
74.6 (-14.3)
OPT-IML---- (1.3B)
34.6
80.6
53.6 (-0.7)
66.1(+7.2)
50.3 (+1.2)
57.5 (-3.1)
OPT-LML-M-L (30B)
38.5
70.4
72.7 (-1.8)
77.1 (+7.3)
(5.4.-3.55
68.3 (-14.5)
Minimal/Unkkown Exposure (MUE)
GPT-3-D3 (175B)
47.8
79.0
76.5 (+2.3)
62.7 (+5.6)
63.3 (+9.1)
69.5 (-0.2)
GPT-3-D2 (175B)
44.8
77.1
72.6 (+1.5)
56.3 (+6.0)
49.9 (-0.7)
45.5 (-10.6)
No Exposure (NE)
Stable Vicuna (13B)
33.5
49.5
55.6 (+2.1)
34.2 (-5.6)
45,4 (+8.5)
61.2 (+14.5)
Table 2:
Inference Performances of LLMs on Various Datasets. We MCE (n 5 for ANLI-R3/ONLI and 500 for ChaosNLI/PK2015 with shuffled OS. For GPTs and Stable Vicuna, we use LPE with shuffled NS. The values inside the parentheses indicate the accuracy change from the old to new labels. We report accuracy results of the FE models (+) from Nie et al., 2020a for ANLI-R3, Devlin et al., 2019 and Liu et al., 2019 for BERT-L and RoBERTa in QNLI, and Nie et al., 2020b for ChaosNLI datasets. All the outputs are averaged over three runs, and bold and underlined texts indicate the first and the second best value for each column.
(Endres and Schindelin, 2003). addition, we evaluate human uncertainty with DCE (Baan et al., 2022) to examine how the tendencies of these two measures compare.
negative effect of using MCE methods exhibited, especially when using OS. Hence, we choose MCE (OS) for the PE models and LPE (NS) for the MUE and NE models.
JSD(pl)a) = KL m) + (L
The NLI capability of LLMs does not only increase due to model size. In Table 2, even though GPT-3-D3 has the largest parameters (175 billion) and surpasses GPT-3-D2 and Stable Vi- cuna, its accuracy is significantly outperformed by the PE models across ANLI-R3, QNLI, ChaosNLI, and PK2019 datasets. For ChaosNLI-S espe- cially, GPT-3-D3 shows comparably lower perfor- mances than any FE and PE models. The lead- ing PE models are Flan-UL.2 and F5aX-T5-XXL across most of the tested datasets (Table 2). The best PE model achieves to 15% higher accu- racy ANLI-R3/ChaoSNLI-M than the best FE model (i.e., RoBERTa-L). However, Flan-T5-UL2 is marginally higher than RoBERTa-L by to % in ChaosNLI-o/S, and Flan-T5-XX: XX-XXL even achieves 9.1 % higher than RoBERTa-L. for ChaosNLIIMM, Within the Flan-T3 -5 family, scaling the model leads enhanced inference performances. However, the largest model across all the tested models GPT-3- D3 does not always attain the best accuracy, sug- gesting that model size alone is not a critical factor for performance on NLI.
2
1
DCE(p.q) 2
l1pp -qlh
where KL(p]/q) P;log(
pt) ptq
5
Results
LLMs are sensitive to different estimation meth- ods and prompt types. To select the optimal es- timation methods and prompt types for each model, examine three cases (1) LPE (NS), (2) MCE (NS), and (3) MCE (OS) for 100 randomly selected examples in ChaosNLI subsets (Table 1). All the PE models perform the best using MCE (OS) or MCE (NS), Meanwhile, GPT-3-D3 performs better using LPE (NS) than either MCE method, hinting that larger models (> 100B) may not need costly methods to estimate the model distribution. Sim- ilarly, for GPT-3-D2 and Stable Vicuna. drastic
13 We exclude I.PF. (OS) due to the reason outlined in Sec-
tion 4.3.

---

RB
FL 2
FXL
FXXL
FU1.2
OS 2
OI
G3
G2
SV
Chance
0.5
Distance from Human Distrihution
0.4
0.3
0.2
0.1
JSD (a:
DCE (a)
JSD (S)
DCE (S) JSD (M) DCE (M) JSD (PK) DCE (PK)
Figure 3: Human Alignment Performances of LLMs on Cha and PK2019 Datasets. The model categorizations and estimation methods are the same Table 2. All the outputs are averaged over three runs. We additionally visualize pairwise model similarity using JSD in Appendix D.
Does multiple annotation help? For most of the models making inferences the re-annotated datasets of ChaosNLI, improvements in NLI ac- curacy are observed, with the exception of OPT- IML-M-S. (Refer to the values inside parentheses in Table 2). This supports the necessity of hav- ing increased multiple annotations for tasks that humans are expected to disagree with. Also, it is noticeable how all these models, even if they were exposed to sample of the train set with the orig- inal label, show better performances in the newly annotated ChaosNLI. However, we detect an accu- racy decrease between the old and new labels in the PK2019 dataset for most of the models except for Flan-T5-L and Stable Vicuna. We hypothesize this is due to the way in which the final label was selected in Pavlick and Kwiatkowski, 2019; anno- tators were asked to select an interval score which later manually discretized.
L in ChaosNLI-o/S. Within LLMs, there is leading model that performs well across all datasets. For example, while Flan-UL2 scores the lowest JSD/DCE value in the ChaosNL.I-o dataset, OPT- IML-M-L shows the lowest distance from human distribution in the ChaosNLI-M dataset. It is impor- tant to note that GPT-3-D3 shows worse JSD/DCE than RoBERTA-L. for all ChaosNLI datasets, and it performs worse than Stable Vicuna in ChaosNLI-M. Intriguingly, the Flan-T5 family benefits from scaling model size in ChaosNLI datasets, but Flan-T5-large does not show the high- est JSD/DCE in PK2019 datasets.
Effect of Human Entropy on LLM Disagree- ment We filter out challenging subset, High- ChaosNLI, which the top 100 selected samples with the highest human disagreement levels based on the entropy of each instance. We observe a plunge in accuracy as well as a rise in JSD/DCE for every model (Table 3) compared to the human alignment performances for full datasets in Table 2. Still, the leading model concerning inference abil- ity (i.e., -5xn-T5-XXL) is unchanged, obtaining the highest accuracy of 52% in HighChaosNI.L On the other hand, is notable how Stable Vicuna displays the lowest ISD/DCE compared to the other models (Table 3). Nevertheless, with the hint the worst accuracy out of all the models for full ChaosNLI datasets (Figure 3) and high entropy levels (Fig- ure 4), we conclude that it is mere coincidence that Stable Vicuna exhibits the best performance in terms of human alignment performances in High- ChaosNLI dataset (Table 3).
Alignment with human disagreement is not al- ways better for larger models. To examine how closely the estimated distribution of LLMs aligns with the human disagreement distribution, we com- pare sample-level measures of JSD and DCE be- tween humans and LLMs (Figure 3). Similar to the accuracy results (Table 2), GPT-3-D3 fails to align with the human label distribution compared to some well- perf-oerforming PE models, such as Flan-T5, XXL and Flan-T5-UL2. Also, each model displays a similar tendency between JSD and DCE, suggest- ing that either one of the metrics might be enough to measure human alignment.
As can be observed in Figure 3, none of the LLMs show less JSD/DCE values than RoBERTa-
We further attempt to investigate the possible

---

-haps
ChaosNLI-S
ChaosNLI-M
1200
Human
Flan-T5-XX
GPT-3.5-D3
Stable Vicuna
800
Count
600
400
200
Figure 4: Histogram of Human and LLM Entropy Levels for ChaosNLI Datasets. The distributions Flan-T5-XX and GPT-3-D3/Stable Vieuna estimated using MCE (OS) and LPPE (NS), respectively, same as Table 2.
ChahChasNNLI
ants from the Flan repository 14 and make sensi- ble variants for ChaosNLI-o as is not part of the Flan mixture. From Table 4, it is shown that the prompt variation generally benefits Flan mod- els in the ChaosNLI-S/M datasets they were exposed to the prompt templates. However, the pre-selected single prompt beneficial in perfor- mance for the ChaosNLI-a dataset for Flan models and all datasets in GPT-3-D3 and Stable Vicuna. The performance drop using prompt variation is even more severe for GPT-3-D3, suggesting the preferred usage of carefully crafted single prompt over using unseen input templates. However, this does not mean that the single prompt should always be preferred since variations of prompts may dis- play fairer performance trends of diverse models within the ground of robustness.
Model
Acct
JSDJ
DCEL
Flan-T5-L (780M)
44.0
0.256
0.318
Flan-T5-XL (3B)
48.0
0.268
0.336
Flan-T5-XXL (IIB)
52.0
0.300
0.362
Flan-UL2 (20B)
50.3
0.321
0.378
OPT-IML-M-S (1.3B)
51.0
0.254
0.293
OPT-IML-M-L (30B)
50.7
0.266
0.312
GPT-3-D3 (175B)
50.0
0.435
0.494
GPT-3-D2 (175B)
45.7
0.310
0.354
Stable Vicuna (13B)
42.7
0.189
0.240
Table 3: Inference and Human Alignment Performances LLMs on HighChaosNLI The model categorizations and estimation methods are the same Table 2. All the outputs are averaged over three runs.
causes of this phenomenon by spanning out the entropy distribution. On the consistent finding that GPT-3-D3 performs worse than Flan-T5-XX in solving NLI tasks (Table 2) and capturing human disagreement levels (Figure 3), even in the High- ChaosNLI dataset (Table 3), as can be observed in Figure GPT tends to be more overconfident, showing entropy of less than 0.1 in most samples. contrast, the human entropy is mostly evenly distributed in the range of 0.4 0.6 for ChaosNLI- a and 0.8 to 1.0 for ChaosNLI-S/M. On the other hand, -lan-T5-XXL exhibits lower confidence than GPT-3-D3 but higher confidence than humans, and Stable Vicuna is uncertain in most instances.
What causes LLMs to disagree? Sources of hu- man disagreement have been well studied, but there is lack of study of the disagreement sources for LLMs. We try to find the causes of LLM disagree- ments by drawing a relationship between LLM en- tropy level and human disagreement sources (dis- cussed in Jiang and Marneffe, 2022) for each sam- visible correlation
ple (Figure 5). However, of LLM entropy on human entropy is displayed across identified sources of human disagreement. This suggests that the cause LLM disagreements may be due to factors other than human entropy and disagreement sources. Thus, Under the naive assumption that LLMs will attend to similar cues to humans, we are not fully uncovering the lens of why LLMs truly disagree.
Effect of Varying Prompts To observe the effect of prompt sensitivity on varying prompt templates, we craft variations of the pre-selected prompt. For SNLI and MNLI, we sample out five prompt vari-

---

ChaosNLI-o
ChaosNLI-S
ChaosNLI-M
Model
Acct
JSD1
DCEJ
Acct
JSD4
DCE1
Acct
JSD4
DCEL
Flan-T5-I (780M)
72.9
0.228
0.255
54.6
0.341
0.382
59.7
0.299
0.325
w/ Prompt Variation
64.9
0.279
0.317
63.3
0.303
0.330
64.7
0.303
0.331
Flan-T5-XL (3B)
83.2
0.166
0.171
71.8
0.255
0.264
64.5
0.272
0.304
w/ Prompt Variation
81.6
0.184
0.193
73.8
0.231
0.243
68.3
0.271
0.297
Flan-T5-XXL (1IB)
84.9
0.159
0.154
67.9
0.270
0.289
72.6
0.271
0.293
w/ Prompl Variation
83.8
0.162
0.164
68.7
0.259
0.279
71.6
0.260
0.285
GPT-3-D3 (175B)
76.5
0.254
0.249
62.7
0.348
0.374
63.3
0.373
0.402
w/ Prompt Variation
72.3
0.285
0.29
50.1
0.402
0.453
51.5
0,403
0,452
Stable Vicuna (13B)
55.6
0.310
0.368
34.2
0.390
0.451
45.4
0.303
0.342
w/ Prompt Variation
51.4
0.324
0.386
29.9
0.431
0.503
42.4
0.337
0.382
Table 4: Inference and Human Alignment Performances on the ChaosNLI Datasets with and without Prompt Variations. The estimation methods for each model are the same as Figure 4, All the outputs are averaged over three runs, and bold texts indicate the best value for each model and column.
6 Discussion
definitions of ambiguity, uncertainty, and disagree- ment have the potential to conflated and disam- biguated. In our paper, we use the multinomial soft label estimate of a model as a representation of "disagreement". When estimating this distribution with MCE, our modeling assumption treats each query to the model is analogous to asking an in- dividual annotator provide label. In contrast, LPE is analogous asking an individual assign the scores to each option. Whereas most works exploit disagreement or uncertainty to improve var- ious NLP task performances (Zhang et al., 2021; Fornaciari et al., 2021b; Yu et al., 2022: Zhou et al.. 2023) our study focuses on evaluating the mod- els. We find that using both methods for estimating the multinomial label distribution by querying the language model not calibrated well with the human annotations.
LLMs do not perform well in NLI. Despite min- imal, unknown, absence of exposure to the NLI task, we anticipated that state-of-the-art LLMs such as GPT-3 and Stable Vicuna could reason with this relatively basic inference problem. The models are trained with billions of parameters and are known to be effective in helping real-world users solve diverse, complex tasks (Ouyang et al., 2022). How- ever, the unforeseen poor performance of these models casts doubt to whether they possess true general language understanding abilities.
The problem is exacerbated for distilled models (e.g. Stable Vicuna) that are fine-tuned using pro- prietary LLMs. a performance discrepancy issue similarly raised by Gudibande et al., 2023. Since smaller LLMs fully or partially trained with NLI tasks could perform much better than the MUE and NE models, this hints at task-specific latent factor in NLI tasks where supervised training benefi- cial and required for wider definition of natural fact, as these LLMs can simply be fine-tuned to perform better for NLI tasks, a stricter evaluation criterion is needed to as- sess the genuine understanding capability of LLMs.
Other domain tasks are transferable to NLI. Our work can be expanded to test LLMs other NLP applications (Plank, 2022) such Question Answering (De Marneffe et al., 2019), Fact Verifi- cation (Thorne et al., 2018), and Toxic Language Detection (Schmidt and Wiegand, 2017: Sandri et al., 2023). Further, our method can be applied for tasks that contain disagreements since they are easily transferable to NLI tasks (Dagan et al., 2006) like the QNLI dataset from Table 2. for example, instead of directly asking controversial questions (e.g., abortion) the model (Santurkar al., 2023), the question format can be modified into declara- tive statement in the premise and place a possible answer in the hypothesis with a binary True/False
language understanding.
Characterizing Disagreement with respect to Ambiguity and Uncertainty Previous studies re- late multiple annotations not only to disagreement (Uma al., 2021; :: Gordon al., 2021), but also to ambiguity (Min al., 2020; Tamkin et al., 2022; Liu and Liu, 2023), and mostly to uncertainty (Fox and Ülkümen, 2011; Xiao and Wang. 2021; Kuhn et al., 2022; Zhan et al., 2023; Hu et al., 2023). The

---

14
10
10-
10"
00
0.7
02-
6.6
inpartatioon
larppre
02
0.6
0.3
10
06
0.8
10
0.4
0.6
0.8
1.0
Human end opy
Human enoray
human entrpy
Figure 5: Relationship between Human and L.M Entropy Levels Divided with Different Human Disagreement Reasons. The estimation methods for each model are the same as Figure 4.
label (Dagan al., 2006). Thus, if these compli- cated tasks can be formulated in a way where the LLM can estimate a multinomial distribution over a set of classes, our methods are applicable.
fully represent a larger population, preferably in- cluding the voices of minorities. Thus, we suggest that future LLMs could be improved to reflect hu- man disagreements in diverse means, for example, by fine-tuning with ambiguous instances (Liu et al., 2023). As LLMs are shown to be aware of their ignorance (Kadavath al., 2022) and have the abil- ity express their level of confidence (Lin et al., 2022; Zhou al., 2023), expect future works address similar approaches in the aspect of align- ment towards the human disagreement distribution. In this way, the reconstructed model distribution with MCE and LPE may better capture different interpretations from human individuals, aiding ac- countability.
However, we should consider the target tasks when tracing "human disagrcement" only when it is significant signal that needs be captured. For example, since it important to include diverse opinions, we can easily apply our methods to detect disagreements in hate speech (Schmidt and Wie- gand, 2017). In contrast, spotting disagreement in the arithmetic reasoning task (Cobbe et al., 2021) might be less important since it often requires logical step-by-step reasoning procedure to obtain accurate answer.
How hetter align LLMs to represent dis- senting voices? We point out the current limi- tation of utilizing LLMs represent a larger hu- man population, especially when disagreements are present. The causes of this phenomenon are indiscernible due to the entanglement of miscali- bration of out-of-distributio (OOD) inference, ad- ditional noise due to disagreement and ambiguity, prompt sensitivity, and more aspects that are yet be identified. Even though simple remedies of temperature scaling (Ackley al., 1985; Wang al., 2022), incorporating logit bias, constrained decoding (Ziems et al.. 2023), or direct supervision to multiple annotations (Zhang et al., 2021; Forna- ciari et al., 2021 a) might mitigate the misalignment, these methods are unrealistic and not scalable due to the exhaustive hyperparameter tuning and ad- ditional data collection required represent the population of interest.
Conclusions
In this paper, we compare the performance of instruction-followwe following generative LLMs with other fully fine-tuned smaller models the fundamental NLI task. First, by experimenting on four differ- ent NLI datasets, we show LLMs are not perform- ing well in the NLI task. considering their touted language comprehension capabilities. Further, agreement with the need for multiple annotations for disagreeable NLP tasks, LLMs also fail to align with human disagreements in the ChaosNLI and PK2019 datasets. Additional development is needed capture representative human distribu- tions, as well as discover key factors disagree- ment sources that can influence the L.MM's answer distribution.
Limitations
However, as LLM applications are becoming more ubiquitous, it is important for them to faith-
This work shows the limited ability of billion-scale LLMs inference and disagreement tasks. Al-

---

though we test with the dataset annotated with nu- merous human subjects per sample, 100 people may not be enough to represent the human dis- agreement distribution well. After more releases of human label variation datasets, our study can be extended by covering wider range of model types and creating evaluation benchmarks to measure the degree of disagreement. If we have robust LLMs in inference and disagreement, we could then try to find the latent factors that might not be human- interpretable but lead to disagreement in LLMs and compare them with those of humans.
Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing, 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetun language models. arXiv preprint arXiv:2210.114::
Ethics Statement
As our work directly employs trained large lan- guage models without any extra process of fine- tuming, the risks and potential biases incurred by the model checkpoints (e.g., dataset selection, train- ing configurations) remain the same as the original works.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian. Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 10,1110.,1116 14168.
Ido Dagan, Oren Glickman, and Bernado Magnini. 2006. The pascal recognising textual entailment chal- lenge. Machine Learning Challenges. Evaluat- ing Predictive Uncertainly, Visual Object Classifi- cation, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 1 11-13, 2005, Revised Selected Papers, pages 177-190. Springer.
Acknowledgments
This work was supported by Institute of Informa- tion & communications Technology Planning & Evaluation (IITP) grant funded by the Korea gov- ernment (MSIT) (No. 2--200011000077 Artificial In- telligence Graduate School Program (KAIST)) and Artificial intelligence industrial convergence clus- ter development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City.
Marie-Catherine De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee,, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
References
David H Ackley, Geoffrey E Hinton, and Terrence Se- jnowski. 1985. A learning algorithm for boltzmann machines. Cognitive science, 91):1477169.
Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Lanuugge Processing, pages 1892-1915, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Dominik Maria Endres and Johannes E Schindelin. 2003. new metric for probability distribu- tions. IEEF Transactions on Information theory, 49(7):1858-186)
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational L. Linguistics.
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021a. Beyond black & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2591-2597, Online. Association for Computational Linguistics.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy. Massimo Poesio, et al. 2021b. Beyond black& & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. Associ- ation for Computational Linguistics.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. In The Eleventh International Conference on Learn- ing Representations.
Craig R Fox and Gülden Ülkümen. 2011. Distinguish- ing two dimensions of uncertainty. Fox, Craig R. and Gülden Ülkümen (2011)," 'Distinguishing Two Dimensions of Uncertainty," in Exsays in Judgment and Decision Making, Brun, W. Kirkebøen, G. and Montgomery, H., eds. Oslo: Universteesforlaget.
Solomon Kullback and Richard A Leibler. 1951, On information and sufficiency. The annals of mathe- matical statistics, 22
Max Glockner, leva Staliūnaité, James Thome, Gisela Vallejo, Andreas Vlachos, and Tryna Gurevych, 2023. Ambife: Fact-checking ambiguous claims with evi- dence. arXiv e-prints, pages arXiv-2106
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar. et al. 2022. Holistic evaluation of language models. arXiv preprint 1..910221..09110.
Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tat- sunori Hashimoto, and Michael Bernstein. 2021. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2027 CHII Conference on Human Factors in Computing Systems, pages 1-14.
Stephanic Lin, Jacob Hilton, and Owain Evans, 2022. Teaching models to express their uncertainty in words. arXiv preprint 223:22205.14334.
Arnay Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary 1lms. arXiv preprint arXiv:2305. 15717.
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2023. We'r re afraid language models aren't i modeling ambiguity. arXiv preprint arXiv:2304. 14399.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PML..
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.11692.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In International Conference on Learning Representations.
Zhu Liu and Ying Liu. 2023. Ambiguity meets uncer- tainty: Investigating uncertainty estimation for word sense disambiguation. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 3963-3977, Toronto, Canada. Association for Com- putational Linguistics.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks prineiple: Reading children's 's books with explicit memory representa- tions. In 4th International Conference on Learning Representations, ICIR 2016.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph. Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint XrX::22301. 13688.
Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. 2023. Uncertainty in natu- ral language processing: Sources, quantification, and applications. arXiv preprint arXiv:2306.04455 04459.
Johannes Mario Meissner, Napat Thumwanit, Saku Sug- awara, and Akiko Aizawa. 2021, Embracing ambi- guity: Shifting the training target of NLI models. In Proceedings of the 59th Annual Meeting the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 862-869, Online. Association for Computational Linguistics.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shus- ter, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Scaling language model instruc- tion meta learning through the lens of generalization. arXiv preprint arXiv:2212 12017.
Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natu- ral language inference. Transactions of the Associa- tion for Computational Linguistics, 13:777
Nicholas Metropolis and Stanislaw Ulam. 1949. The monte carlo method. Journal of the American statis- tical association, 44(247):335-344 3411.

---

Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elis- abetta Ježek. 2023. Why don't you do it right? analysing annotators' disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computa- tional Linguistics, pages
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties Volume
I: Long Papers). pages 3470-3487 Dublin, Ireland. Association for Computational Linguistics.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee. Percy Liang, and Tatsunori Hashimoto, 2023. Whose opinions do language models reflect? arXiv preprint arXiv:2303. 17548.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He. Devi Parikh, Dhruv Batra, Lucy Vanderwende. Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguis- tics.
Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language pro- cessing. In Proceedings of the fifth international workshop on natural language processing for social media, pages 1-10.
Jiuding Sun, Chantal Shaib, and Byron C Wallace. 2023. Evaluating the zero-shot robustness of instruction-tuned language models. arXiv preprint arXiv:2306.11270.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference artificial intelligence, volume 29.
Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. 2022. Task ambiguity in humans and language models. arXiv preprint arXiv:2212.1001
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020a. Adversarial nli: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 4901.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Rep- resentations.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020b. What we learn from collective human opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131-9143 Online. Association for Computational Linguistics.
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguisties.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedhack. Advances in Neural Information Processing Systems, 35:27730-27744,
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 12:1555-
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018 Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018. page 353.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements human textual inferences. Transac- lions of the Association for Computational Linguis- rics, 7:677-694.
Barbara Plank, 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671-10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yuxia Wang, Minghan Wang. Yimeng Chen, Shimin Tao, Jiaxin Guo. Chang Su, Min Zhang, and Hao Yang. 2022. Capture human disagreement distribu- tions by calibrated networks for natural language inference. In Findings of the Association for Compu- tational Linguistics: ACL 2022, pages 1535-1335.

---

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A croad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisticy: Humun Language Technologies, Volume 1 (Long Papers); pages 11 11121122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Model
Precision
Acc
JSD
FP32
75.4
0.233
Flan- XXLXX
BF16
75.1
0.233
Table 5: Effect of Precision on Inference and Human Alignment Performances for Flan-T5-XXI.. The distri- bution is estimated using MCE (OS), same Table' 2. The outputs are averaged over three ChaosNLI sub- datasets.
Yijun Xiao and William Yang Wang. 2021. On hal- lucination and predictive uncertainty conditional
language generation. Proceedings of the 16th Con- ference of the European Chapter of the Association for Compulational Linguistics: Main Volume, pages 2734-2744
A Hyperparameters
Generally, we try to set similar hyperparameters to all the models with some exceptions due to model performance and/or cost issues.
Yu Yu, Hassan Sajjad, and Jia Xu. 2022. Learning uncertainty for unknown domains with zero-target- assumption. In The Eleventh International Confer- ence on Learning Representations.
Temperature To scale the confidence of the gen- erated output in a post-hoe manner, we unify the temperature to be 1 (i.e., no scaling). There ex- ist other precedents that use a smaller temperature for more deterministic output (Santurkar et al., 2023) compare outputs of models with varying temperatures (Ouyang et al., 2022). However, we jointly assess LLMs on the accuracy of NLI and human disagreement alignment, we argue that having a fixed, un un-scaled temperature to generate model outputs better aligns with our research goal of estimating model outputs to capture human dis- agreement distribution.
Runzhe Zhan, Xuebo Liu, Derek Wong, Cuilian Zhang. Lidia S Chao, and Min Zhang. 2023, Test- time adaptation for machine translation evaluation by uncertainty minimization. In Proceedings of the 61 Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 807-820.
Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with different amounts of annotation: From zero to many labels. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 7620-7632.
Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021. Identifying inherent disagree- ment in natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4908-4915.
Generation Length Easily adjustable by all APIs, including OpenAPI and Huggingface, we have varying generation lengths per prompt design. As discussed in Section 4.3, NS is cost-efficient alternative method for OS, solely needing single output token of numbers. Thus in LPE, a method for single token probability output, we only use the OS prompt for effective token probability calcula- tion. We set a maximum token output length of 10 for MCE and for LPE.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 18223.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: Expressions of overconfidence and uncertainty in language models, arXiv preprint arXiv:2302.144 13439.
Floating Point We load models of size greater than 10 billion parameters (except for GPT-3) with half the precision (bfloat 16; BF16). We observe Flan- Flan-T5-XX XXL shows negligible increase in per- formance when using the original precision (single- precision seating-point. FP32) (Table 5).
Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022. Dis- tributed NLI: Learning to predict human opinion dis- tributions for language reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 972-987, Dublin, Ireland, Association for Computational Linguistics.
B Levels of NLI Exposure
We outline the level of exposure the NLI task for each model since it is an influential factor that affects the accuracy and numan-alignmen perfor- mances of the models.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence? arXiv preprint arXiv:2305.03514

---

B.1 Full Exposure (FE) Models
B.4 No Exposure (NE) Models
The models below are fine-tuned with the training set of an NLI task as outlined in Nie et al., 2020b.
The below model does not have any exposure to a specific NLI task.
Models: BERT and RoBERTa
Model: Stable Vicuma
B.2 Partial Exposure (PE) Models
C Postprocessing
These models are partially exposed to the NLI task in the fine-tuning stage. However, the extent of exposure is different by the adopted fine-tuning strategy, thus listed in decreasing order.
Unlike conventional approaches of fine-tuning models directly on the downstream NLI dataset, one of the challenges in assessing an NLI task is the variability of generated outputs. To transform and choose valid options from the generated out- puts, we conduct postprocessing through a manu- ally crafted dictionary for each option (See Valid option examples on the last page.).
Flan Collection
Models: Flan-T5 models and Flan-UL2
The Flan Collection (Longpre et al., 2023) is collection of datasets in the format of instruc- tions enable generalization diverse un- seen tasks. employs fine-tuning strategy of maximum of 1836 NLP tasks with some NLI tasks taken into account (e.g., ANLI, RTE, MNLI, QNLI, SNLI, etc.).
D Distribution Alignment Among LLMs
We illustrate the averaged sample-level JSD en- tropy for each model pair (Figure 6) to visualize the trend of alignment among LLMs. Throughout all four JSD distribution plots, the scale and range of the JSD values differ for each data. Still, the gen- eral trend is maintained, where ChaosNLI-a shows low JSD values overall, likely attributed to lower task difficulty witnessed by the performance gap among datasets in Table 2. The perfoorinn models, Flan-T5-XXI and Flan-UL2, present the lowest disagreement in entropy for all plots.
Instruction Meta-Learning (IML) Bench
Models: OPT-IML-M models
Instruction Meta Leaming (IML) Bench (Iyer et al., 2022) a more common benchmark that uses 1 1500+ NL.P tasks the fine-tuning stage. Flan is a major portion of this bench- mark, with other major portions in other large datasets. We expect some NLI exposure but not strong as the models fine-tuned by the Flan dataset.
Although the size and type of model are influ- encing factors, the most consistent factor is the type of instruction fine-tuning introduced for each model. Throughout all plots, the alignment is well shown for the group of models fine-tuned by the Flan dataset and the IML Bench. As we expect more research the scope of human alignment NL.,, the evaluation of the human alignment among the models with the same fine-tuning process can also be studied and reported.
B.3 Minimal/Unknown Exposure (MUE) Models
The models below are unknown to the extent of exposure to a specific NLI task.
However, strong distinction needs to be made in which overall lower number of JSD values in this plot does mean that model has always had good performance in human disagreement align- ment. This figure merely delineates the alignment trends among models.
Models: GPT-3-D2. GPT-3-D3
The InstructGPT paper (Ouyang et al., 2022) does elaborate that the models utilizes a reward model in the process of RLHF (Reinforcement Learning from Human Feedback), and it is fine- tuned by variety of NLP datasets, including MNLI. However, the serviced models are not di- rectly mapped the models of the paper, leaving the exposure to NLI largely unknown
E Effect of Few-Shot Examples
We observe no consistent benefit harm in ex- perimenting with few-shot settings that resemble the human annotation process more than zero-shot settings (Table 6). In fact, zero-shot evaluation gen- erally seems to show better performances across
15 httpggghhhh.ooogggglreeeeeeeeeee 16 "Reeerr to the OpenAl Documentation

---

Chaos NL
ChaosNLI-S
Flan-TS-L
0.18
0:24
0.25
0:21
0.17
0.33
0.25
0.24
Honn-T--L 73-L-
0
0:23
0.26
0.35
0.36
0.38
0.42
0.45
0.27
Flan-T5XLL-
0.17
0.21
Flan-TS-XL
0.36
Rlan-T5-XXL.
0.13
0.34
0.25
0.21
0.21
0.31
Flan-TS-XXL
0:22
0.37
0.3
0.33
0.42
Flan-UL2
0.93
0.26
0.21
0.22
0.34
Flan-UL2
0.35
0.21
0.32
0.39
0.47
LLMS
LLMSs
OPT-IMI-M-S
0.33
OPT-IMI-MA
0.33
0.24
0.22
OPT-IML-M-L
0.34
0.34
0.15
GPT-3.5-D3
0.24
0.37
GPT-3.5-D3
GPT-3.5-D2
GPT-3.5-D2
0.45
Stable Vicuna
Stabbl Vicuna
0
Flan-T5-1
Flan-T5-XX
Flan-T5-XXL
Han-JUL
OPT M-M------
OPT-MML M--
GPT GPT-3.5 3.5.D3
GPT:3 GPT-3.5-D2 5-D222
Stable Vicuna
Flan-T5-1
Fan-T5-X:
Fan-ULL
OPT-IML-MSS M.M-MM-..
OPT M-L
GPT-3 5-D3
GPT D2
FlanTT
LLMs
LLMs
ChaosNLI-M
PK2019
Flan-TS-L-
0.24
0.29
0.36
0.41
0.37
0.41
0.47
0.27
Flan-T5-L.
0
0.22
0.25
0.32
0.36
0.32
0.37
0.45
0.29
Flan-TS-XL-
0:23
0.4
Flan-TS-XL-
0.2
Rlan-T5XXXL-
0.28
Flan-TS-XXL
0.15
0.39
0.21
0.27
0.43
Flan-UL2
0.39
0.24
0.34
0.37
0.45
Flan-UL2
0.37
0.3.
0.26
0.39
0.48
LLMS
LLASs
0.35
0.43
OPT-IML-M-L-
0.36
0.344
0.42
OPT-IML-M-L
0.3
0.35
0.44
GPT-3.5-D3
0.39
0.47
GPT-3.5-D3
GPT-3.5-D2
GPT-3.5-D2
0.42
Stable Vicuna
Stable Vicuna
0
Flan-TS-L
Flan-TS-XL
Hlan-UL2
OPT MT-IML-M-S-.
OPT- M-L
GPT-3.5 3.5-3.5.03-
GPT-3.5-D2 3.5-2
Stable Vicuna
Flan-TS-L-
Flan-75-XL XL
Flan-UL2
OPT-IML-M-S.
OPT M-L-
GPT-3.5-03-
GPT 02
Stable Vicuna
Flan-T5-XXC
Flan-T5-XX
LLMs
LLMs
Figure 6: ISD Distribution between All Combinations of Pairs for LLMs. The darker plot indicates similar distribution between pair of models. The estimation methods for each model are the same as Table 2.
datasets and models compared to the few-shot eval- uations. the case of Stable Vicuna, the perfor- mance increases in the I-shot setting for a-NLI and the 3-shot setting for SNLL. However, we notice a plunge in 5-shot performance, especially for the MNLI dataset.
gested general prompt template pre-specified for specific model. For example, we implement hu- man and assistant-style prompt template for Stable Vicuna. Otherwise, we leave the template format the same for the rest of the models.
F Prompt Examples
We present examples of prompts we used during the generation process in Figure (See two prompt examples on the last page.). We incorporate a sug-

---

ChaosNLI-a
ChaosNLI-S
ChaosNLI-M
Model
Acct
JSDJ
DCE|
Acct
JSD
DCE1
Acct
JSD
DCE
Flan-T5-XXL (0 Shot)
85.0
0.160
0.155
67.3
0.271
0.291
72.6
0.269
0.290
+ Shot
84.9
0.159
0.154
68.0
0.278
0.296
74.8
0.261
0.278
+ 3 Shot
83.6
0.163
0.160
67.0
0.285
0.304
74.0
0.271
0.290
+ 5 Shot
84.9
0.158
0.154
65.7
0.288
0.309
73.2
0.275
0.295
GPT-3-D3 (0 Shot)
76.1
0.254
0.249
62.4
0.348
0.374
63.0
0.376
0.405
Shot
77.7
0.240
0.235
6.2
0.400
0.433
61.1
0.414
0.445
+ Shot
80.7
0.233
0.216
55.7
0.407
0.442
58.2
0.436
0,471
+ + Shot
81.7
0.233
0.213
57.9
0.396
0.426
62.2
0.425
0.454
Stable Vicuna (0 Shot)
55.7
0.310
0.368
33.5
0.391
0.454
46.2
0.304
0.342
+ Shot
64.3
0.290
0.336
38.6
0.377
0.436
41.7
0.311
0.355
+ 3 Shot
56.2
0.296
0.346
43.4
0.351
0.405
32.8
0.352
0.418
+ 5 Shot
56.1
0.298
0.350
35.8
0.379
0.441
28.8
0.370
0.441
Table 6: Inference and Human Alignment Performances on the ChaosNLI Datasets for Zero- shoo-shot and Few-shot Settings. The estimation methods for each model are the same as Table 2. Bold texts indicate the best value for each model and column.

---

Valid option examples for ChaosNLI-c/S/MM and two prompt types OS and NS
dict_alphanli_05 C''' C'1, Hyypthhssis .'...
'2' ['2", Hypothesis 2', 2',...]>
dict_alphanli_NS (''' '1' 22 '2')
...] dict_s8mnli_0n ('e' C'entail', 'infer', 'yes', '
'c' L'contradict' 'oppose', 'no', ...J
I'neutral', uunanseerable ....
dict_s&mnli.Ng e'e' '1'
'c' '2' 'n' '3'
Prompt example for ChaosNLI-a using OS
INPUT
Read the following and determine if the hypothesis can be inferred from the premise. Observation Start: My roommates put up their Christmas tree this year.
Observation End: This is what it's like living with cat. Hypothesis 1: The roommates soon had to take the tree down.
Hypothesis 2: The cat enjoyed the ornaments and garland and slept under the tree.
Options: Hypothesis 1, Hypothesis 2
OUTPUT
Answer: <Generated Output>
Prompt example for ChaosNLI-S/M using NS
INPUT
Read the following and determine if the hypothesis can inferred from the premise.
This town, which flourished between 6500 and 5500 b.c. appear on Anatolian kilims. Hypothesis: This town is over 8000 years old. Options: 1: entailment, 2: contradiction, 3: neutral
Premise:
OUTPUT
Answer: <Generated Output>


Reviews:
[{"review_id": "tzynkxYbUI", "reviewer": "Reviewer_qn54", "summary": "", "questions": "1. Both MCE and LPE rely on a set of valid options for each class. How many valid options do you have per class, and do you have an idea how sensitive the estimated distribution is to the choice of these valid options? \n2. How slow/fast are MCE and LPE? NLI is a relatively simple task compared to others with long outputs, so it'd be interested to know scalable they are to more complicated tasks.\n3. In section 5, the first conclusion is about how model size alone isn't a critical factor for high performance on NLI. But based on my reading of Table 1, it's really GPT-3.5 that's clearly the outlier. I feel like it says more about GPT-3.5's unique properties rather than its size -- within the same model family, you do see model size matter (see various sizes of Flan models, and OPT-IML models). \n4. I might have missed this, but what exactly do you mean by 'shuffled option selection'?\n5. I'm not sure I understand the argument in line 393-397 -- what's the characteristic of the dataset that led you to conclude it's mere coincidence?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer qn54,\n\nThank you for your thoughtful review. We are glad that you find our attempt to quantify and analyze the alignment between the LLM distribution and human disagreement distribution using various LLMs to be thoroughly conducted and valuable. \n\nBelow, we have tried our best to address your questions and concerns. Please take a look at our response and let us know if further clarification is needed.\n\n**Q1) Both MCE and LPE rely on a set of valid options for each class. How many valid options do you have per class, and do you have an idea how sensitive the estimated distribution is to the choice of these valid options?**\n\n| | | *ChaosNLI* | *$\\alpha$* | *ChaosNLI* | *S* | *ChaosNLI* | *M* |\n|------------------|:-----:|:----------:|:-------:|:----------:|:-----:|:----------:|:-----:|\n| | | *MCE* | *LPE* | *MCE* | *LPE* | *MCE* | *LPE* |\n| *FLAN-T5-XXL* | *OS* | 99.96% | - | 89.24% | - | 90.85% | - |\n| *FLAN-T5-XXL* | *NS* | 99.74% | 99.81% | 99.49% | 99.45% | 99.51% | 93.55% |\n| *GPT-3.5-D3* | *OS* | 100.00% | - | 99.91% | - | 99.99% | - |\n| *GPT-3.5-D3* | *NS* | 99.44% | 99.57% | 99.93% | 99.74% | 99.83% | 99.15% |\n| *Stable Vicuna* | *OS* | 97.84% | - | 85.20% | - | 86.04% | - |\n| *Stable Vicuna* | *NS* | 99.57% | 99.53% | 99.28% | 99.39% | 99.30% | 99.36% |\n\nTable A | The Percentage of Valid Options for Three Models - Flan-T5-XXL, GPT-3.5-D3, and Stable Vicuna on Subsets of ChaosNLI Datasets. We present the average results of ChaosNLI-α, S, and M; for each dataset, we randomly sample 100 instances. For MCE, we record the percentage of the number of captured valid options out of a total of 100 generated outputs. For LPE, we report the sum of the probability of valid options captured within the top-k (k=5) tokens.\n\nOur implementation of the parsing of answers relies on string matching the stems of the candidate space that we sample out in the initial stage of experiments rather than having all the variations of each class label counted. As a matter of fact, the distribution was not sensitive to these options as most models were capable of generating a combination of strings that was parsable into a single class label for most (> 99%) time (Table A). Furthermore, by prompting the model to generate a number (Number Selection; NS) rather than the actual option (Option Selection; OS), the models were able to perfectly parse out the generated option into a specific class. \n\n**Q2) How slow/fast are MCE and LPE? NLI is a relatively simple task compared to others with long outputs, so it'd be interested to know scalable they are to more complicated tasks.**\n\n| | | *ChaosNLI* | *$\\alpha$* | *ChaosNLI* | *S* | *ChaosNLI* | *M* |\n|------------------|:-----:|:----------:|:-------:|:----------:|:-----:|:----------:|:-----:|\n| | | *MCE* | *LPE* | *MCE* | *LPE* | *MCE* | *LPE* |\n| *FLAN-T5-XXL* | *OS* | 8.21 | - | 8.28 | - | 8.94 | - |\n| *FLAN-T5-XXL* | *NS* | 3.91 | 0.16 | 3.14 | 0.15 | 3.49 | 0.16 |\n| *GPT-3.5-D3* | *OS* | 4.93 | - | 3.92 | - | 4.11 | - |\n| *GPT-3.5-D3* | *NS* | 3.55 | 0.37 | 3.47 | 0.39 | 3.57 | 0.37 |\n| *Stable Vicuna* | *OS* | 13.97 | - | 10.58 | - | 11.61 | - |\n| *Stable Vicuna* | *NS* | 5.64 | 0.12 | 3.94 | 0.09 | 4.48 | 0.09 |\n\nTable B | Comparison of Inference Time Between MCE and LPE for Models - Flan-T5-XXL, GPT-3.5-D3, and Stable Vicuna on Subsets of ChaosNLI Datasets. We present the average results of ChaosNLI-$\\alpha$, S, and M; for each dataset, we randomly sample 100 instances. For both MCE and LPE, we record the generation time in seconds using a Python package called “microbench.” \n\nReferring to lines 156 (for MCE) and 177-178 (for LPE) of the paper, LPE is n times faster than MCE since LPE requires only one sample to be generated (O(1)), whereas MCE must produce n samples (O(n)) to estimate the final distribution capturing disagreement. We further provide an empirical comparison of the average inference time per instance between MCE and LPE (Table B). \n\n**Q3) In section 5, the first conclusion is about how model size alone isn't a critical factor for high performance on NLI. But based on my reading of Table 1, it's really GPT-3.5 that's clearly the outlier. Within the same model family, you do see model size matter.**\n\nYou are on point regarding that size is a very critical component for the NLI performance within the same model family. However, as mentioned in lines 282-283 and 307-310 of the paper, we want to emphasize that size alone is not the sole factor that can affect NLI performance. In fact, the NLI exposure seems to be a more important factor since a 355 million parameter RoBERTa-Large can significantly outperform a 175 billion model GPT-3.5-D2/D3. Still, we deeply understand your comment on our expression which can be better rephrased in a less misleading way. We will revise the first Results section based on your valuable suggestion.\n\n**Q4) Explain Shuffled Option Selection.**\n\nWe will clarify that as the details of the shuffling are laid out in Appendix A, but not in the main paper. The shuffled option selection basically indicates randomly permuting the class options (e.g., 1, 2, 3 to 3, 1, 2). This is needed to reduce the variability of performances caused by the high prompt sensitivity of generative LLMs (Hendrycks et al., 2021)  to increase the robustness of performances (Santurkar et al., 2023). In fact, the prompt sensitivity of LLMs also resembles the sensitivity of the position incurred in human respondents (Santurkar et al., 2023), which is why we thought it more natural to adopt this method.\n\n**Q5) What's the characteristic of the dataset that led you to conclude it's mere coincidence?**\n\nAs you can observe in Table 1 and Figure 3, Stable Vicuna is certainly not the best model in terms of accuracy (NLI Performance) and JSD/DCE (human alignment performance) for the full ChaosNLI datasets. However, when we test it on top-100 human disagreeing instances (which we entitle, Chaos^2NLI), Stable Vicuna “seemingly” becomes the best model in human alignment with the lowest JSD/DCE among all the other models (Table 3).\n\nThis phenomenon is not due to the fact that Stable Vicuna could estimate the most disagreeable instances well, but because *Stable Vicuna is uncertain in most instances since it displays higher entropy levels compared to the other models (Figure 4).* Hence, paired with the hint of the worst accuracy out of all the models for full ChaosNLI datasets but with the best human alignment performance score for top-100 human disagreeing instances, we conclude that it is a mere coincidence that Stable Vicuna exhibits the best performance in terms of human alignment performances in Chaos^NLI dataset. We will clarify this aspect more in detail in the Results section.\n\n**My main concern is regarding how applicable MCE and LPE methods are for other more complicated tasks**\n\nWe understand your concern about how MCE and LPE methods can be applied to further complicated tasks other than NLI. We briefly mention the possibility of transferring other NLP tasks to the NLI task (lines 488-505). In addition to this, if these complicated tasks can be formulated into multiple-choice style formats, with the assumption that our models can generate valid options for each class, our methods are easily scalable as Table B from Q2) suggests. For example, more complex tasks such as ambiguous fact verification (Glockner et al., 2023) or TruthfulQA (Lin et al., 2022) are in or are capable of directly being formulated into a multiple choice format; thus, our methods are easily applicable. However, with tasks such as Abstractive QA such as HotpotQA (Yang et al., 2018) that can’t be directly transferred, we can exploit the power of LLMs to perform semantic matching (Kamalloo et al., 2023) and even further.\n\nHowever, it should be noted how our methods are only useful in tasks where “human disagreement” is a significant signal that needs to be captured. For example, since it is important to include major and minor opinions, we can easily apply our methods to detect disagreements in hate speech (Schmidt et al., 2017). In contrast, spotting disagreement in the arithmetic reasoning task (Cobbe et al., 2021) might be not as important since it often requires a logical step-by-step reasoning procedure to obtain an accurate answer.\n\nWe also deeply appreciate you for the comments on missing references and typos/presentation improvements to bring a stronger presentation for the paper. We will respond to some of them:\n\n**Typos Grammar Style And Presentation Improvements:**\n\n\n•\t**[Table 1 Revision]** Thank you for this suggestion. We will add the model parameters in Table 1.\n\n\n•\t**[Figure 3 Y-axis Label Revision]** Thank you so much for pointing this out. We will revise the label as you suggested.\n\n**References:**\n\n•\tHendrycks, D. et al., (2021). Measuring Mathematical Problem Solving With the MATH Dataset. Sort, 2(4), pages 0-6.\n\n\n•\tSanturkar, S. et al. (2023). Whose Opinions Do Language Models Reflect? In International Conference on Machine Learning, pages 29971-30004.\n\n\n•\tGlockner, M. et al., (2021). AmbiFC: Fact-Checking Ambiguous Claims with Evidence. arXiv e-prints, arXiv-2104.\n\n\n•\tLin, S. et al., (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214-3252.\n\n•\tYang, Z. et al. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380.\n\n\n•\tKamalloo, E., Dziri, N., Clarke, C. L., & Rafiei, D. (2023). Evaluating Open-Domain Question Answering in the Era of Large Language Models. In Association for Computational Linguistics, pages 5591–5606.\n\n•\tCobbe, K. et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.\n\n•\tSchmidt, A., & Wiegand, M. (2017). A survey on hate speech detection using natural language processing. In Proceedings of the fifth international workshop on natural language processing for social media, pages 1-10.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Efficiently quantifying the degree to which LLM's distribution differs from human agreement/disagreement is a valuable effort, esp given the challenge of API-based models that are becoming more and more in use nowadays.\n2. Very clearly written, and very thorough set of experiments that make the main point of the paper that LLMs fail to adequately capture human disagreement.", "reasons_to_reject": "My main concern is regarding how applicable MCE and LPE methods are for other more complicated tasks. See the Questions section for more details.", "questions_to_authors": "1. Both MCE and LPE rely on a set of valid options for each class. How many valid options do you have per class, and do you have an idea how sensitive the estimated distribution is to the choice of these valid options? \n2. How slow/fast are MCE and LPE? NLI is a relatively simple task compared to others with long outputs, so it'd be interested to know scalable they are to more complicated tasks.\n3. In section 5, the first conclusion is about how model size alone isn't a critical factor for high performance on NLI. But based on my reading of Table 1, it's really GPT-3.5 that's clearly the outlier. I feel like it says more about GPT-3.5's unique properties rather than its size -- within the same model family, you do see model size matter (see various sizes of Flan models, and OPT-IML models). \n4. I might have missed this, but what exactly do you mean by 'shuffled option selection'?\n5. I'm not sure I understand the argument in line 393-397 -- what's the characteristic of the dataset that led you to conclude it's mere coincidence?", "paper_topic_and_main_contributions": "This paper focuses on the NLI task and studies the uncertainty in the learned distribution of LLMs and how it differs from human disagreement. The paper also proposes two simple techniques to estimate LLM's learned distribution to overcome the challenge of API-based LLMs that limits the number of queries you can make.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. I would have liked to see another column in Table 1 to show each model's parameter count.\n2. Y axis in Figure 3 would make sense with 'distance to human distribution' as the title instead of 'closeness', since lower means better alignment.\n"}}, {"review_id": "IGfVoGIfOT", "reviewer": "Reviewer_DgQC", "summary": "", "questions": "A) On Lines 154-156, you cite the decoding strategy as a limiting factor for exploring the whole space. How does this apply to your NLI tasks, where the model is selecting between a finite number of options rather than performing open-ended decoding? With lines 162-163, it seems like you are saying each class label will be inferred based on the open-ended generation, but this isn't totally clear.\n\nB) Line 182 - 185, suggest the probability mass is non-constant, but this is a choice by this calculation. Why don't you use the adjustment from Santurkar et al. 2023 in your citations to make the probability mass constant by estimating the mass of missing options as their upperbound?\n\nE) The FLAN Models use the full ANLI and QNLI training datasets as part of their instruction fine tuning. Why are they marked as Partial Exposure?\n\nD) Line 245-246, How can you confidently say that Davinci 2 has no exposure to these tasks? It received supervised finetuning on an unknown dataset, which based on my understanding means it could have partial or complete exposure to these tasks.\n\nE) Line 264-268, Why can't you compare the probabilities of the first token of each option for OS LPE? Entailment, Contradiction, and Neutral all have different leading tokens so the probability of each leading token seems a valid proxy for the whole word.\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DgQC,\n\nThank you for your appreciation of our work and insightful review. We are grateful that you find our problem, which is modeling human disagreement with LLMs, crucial. We are also encouraged that you see the potential of our proposed metrics to have impacts in LLM and human opinion distribution domains. \n\nBelow, we have tried best to address your questions and concerns. Please take a look at our response and let us know if further clarification is needed.\n\n**Q1) You cite the decoding strategy as a limiting factor for exploring the whole space. How does this apply to your NLI tasks, where the model is selecting between a finite number of options rather than performing open-ended decoding? With lines 162-163, it seems like you are saying each class label will be inferred based on the open-ended generation, but this isn't totally clear.’**\n\nAs mentioned in lines 154-156, using decoding strategies such as beam search might hinder models to explore the entire space. Within the application to our NLI task, the decoding strategy is not a major problem for LPE which requires a single output token. However, for MCE, which we do not limit the number of tokens, we are able to mitigate this source of limitation as the LLM is able to output a wider set of answer space than a single generation equipped with a decoding strategy can.\nAlso, you are correct on “*each class label will be inferred based on the open-ended generation.*” We will clarify this in more detail: we prompt instruction fine-tuning LLMs to select between a finite number of options (ex., entailment, contradiction, and neutral for ChaosNLI-M/S datasets) and to perform open-ended decoding, where the models have to predict the entire string approximating valid options. Please look into our Prompt examples in Appendix (p. 14) to see the exact prompt we used for some instances. \n\n**Q2) Line 182 - 185, suggest the probability mass is non-constant, but this is a choice by this calculation. Why don't you use the adjustment from Santurkar et al. 2023 in your citations to make the probability mass constant by estimating the mass of missing options as their upper bound?**\n\nThe unknown probability mass is displayed due to two reasons: a missing option or the model's characteristic to be unconfident. Santurkar et al. assigns the probability mass of missing options to the minimum value of the remaining mass or the minimum probability assigned to any of K token choices (p. 29 in Santurkar et al. 2023) and normalizes after, which will lead to an unsound overestimation of unspecified (i.e., missing) options. We make adjustments by not assigning probability mass for the missing options and directly normalize our output which might be overestimating the probabilities of existing options, but refraining from any problematic assignment to missing options.\n\n**Q3) The FLAN Models use the full ANLI and QNLI training datasets as part of their instruction fine tuning. Why are they marked as Partial Exposure?**\n\nThank you for pointing this out. Here we provide more details to clarify the concept of Partial Exposure (PE). Our intent in marking FLAN models as PE is because they are not directly fine-tuned with only NLI datasets. In fact, yes, it is true that they are fine-tuned with ANLI and QNLI datasets, however, in the FLAN paper (Chung et al., 2022), it is stated that “*We apply the maximum cap for each task because there are tasks that are much larger than others in the same mixture, which can dominate the sampling.*” This suggests that they also have used tasks other than NLI to fine-tune the FLAN models.\n\nIn addition, in Table 23 (p. 46 in Chung et al., 2022), the maximum cap and proportion rates for the Muffin mixture, which contains 80 tasks including the NLI tasks, are capped with at most 30k instances per task. This is a smaller proportion compared to the entire training dataset size of the full ANLI (r3: 100k,  r2: 45.5k, and r1: 16.9k, thus, theoretically possible to be fully sample r1, but highly unlikely) and QNLI (116k) datasets. To get a better sense of the utilized NLI proportion, in Table 22 (p. 46 in Chung et al., 2022), FLAN-T5-XXL is fine-tuned with a batch size of 64 for 14K steps, summing up to approximately 896K instances, and the FLAN models utilize a mixture weighting around 40-50% for the Muffin task. Hence, we can conclude that unlike the Full Exposure (FE) models which are directly fine-tuned to only NLI tasks, the FLAN models are assigned as PE models due to comparably under usage of NLI datasets for finetuning, according to our definition. We will clarify the definition of PE in Section 4.2 in the later version of this paper. \n\n**Q4) How can you confidently say that Davinci 2 has no exposure to these tasks?**\n \nWe were also aware of the uncertainty of whether Davinci 2 has exposure to NLI tasks or not. Ouyang et al. elaborate that “*the dataset consists primarily of text prompts submitted to the OpenAI API … Playground interface*,” thus; unless the authors added a small proportion of NLI data or the API users input NLI problems out of their own interests, the model would not have been fine-tuned to the NLI dataset. \n\nHowever, to be sure, we reached out to one of the authors of the paper and received confirmation that no NLI dataset and/or instances were used in their first response. Unfortunately, another author responded later that they, in fact, do have NLI instances but with a number less than 10. We will add a Minimal Exposure (ME) level to have Davinci models separated from Partial Exposure (PE) FLAN models with thousands of NLI supervision instances and with No Exposure (NE) model to encourage a fairer comparison.\n\n**Q5) Why can't you compare the probabilities of the first token of each option for OS LPE?** \n\nWe try to respond to this issue in the paper by lines 264-268, *“OS, on the other hand, is not considered in the LPE formulation for simplicity in both implementation and application, where the token-specific probability of a varying length and output format has to be mapped per class.”* \n\nTo further elaborate, our main purpose of the paper as you kindly phrase is to *develop a sound metric of mapping LLMs to distributions of opinions has potential impacts in each of those domains*. If we allow LPE-OS, we would have to allow multiple tokens to be generated which will lead to an exhaustive, rule-based processing: 1) saving log probabilities of every token, 2) selecting the determining token probabilities that may vary by instance/dataset/task, and 3) mapping the designated probability to the correct class label. For example, FLAN models’ frequent representation of the label “neutral” was by an answer “It is not possible to tell”. To incorporate the generation characteristics for every used model along with the varying deterministic token position, we thought it best to leave out LPE–OS to encourage a scalable, comprehensive generation/evaluation strategy to estimate human disagreement distribution.\n\n**The fundamental comparison seems to incompletely answer the question layed out in the title, confounded by the way the models are compared to human annotators. The probability distribution of labels assigned by different human annotators is the average of opinions from multiple agents. However, disagreement does not neccesarily mean that each individual annotator was uncertain in the label they selected. Given the high variance from prompting, it seems not sufficiently sound to compare \"agreement\" of LLMs by using the sampled variance over one prompt.**\n\nYes, we appreciate and understand your concerns that using sampled responses over one prompt may not perfectly align with human disagreement distribution. Our title, “Can LLMs Infer and Disagree Like Humans?” may lead to confusion that we use multiple LLMs to estimate the human disagreement distribution. However, our intention with the title is whether each of the state-of-the-art instruction fine-tuning LLMs, with their emergent ability, are capable of representing the dissenting voices of humans. Since LLM decodes different responses even when prompted with the same question, we explore whether utilizing varying responses over n instances (MCE) or top-k tokens over a single instance (LPE) might be enough to represent the human disagreements. \n\nWe believe your concern on “*disagreement does not necessarily mean that each individual annotator was uncertain in the label they selected*,” especially roots in our LPE method since it is calculated with a single instance. Although we regard MCE as a more intuitive, human-like way to represent disagreement (e.g., aggregating voices of n=100 people), in Table 2, we can observe how LPE outperforms MCE in several models like GPT-3.5-D3 and Stable Vicuna with higher efficiency (i.e., less cost for the number of instances).\n\nWe also want to point out that we do not use the sampled “variance” to calculate uncertainty. Rather, we aggregate the LLM uncertainty over one prompt using either the LPE or MCE method to estimate human disagreement distribution. While Zhou et al. injected expressions of uncertainty into prompts (e.g., “I think the answer is …”) to increase the accuracy in terms of predicting one ground-truth answer correctly by generating 10 tokens (p. 4 in Zhou et al.), our work not only aims to achieve high accuracy but also to capture different perspectives of humans with MCE/LPE.\n\nAs you mention, high prompt sensitivity can possibly debunk the soundness of our paper. We try to mitigate this problem by shuffling the order of the class options and averaging them over three trials. We are aware of the effect of the variability occurring in answers generated with different instruction prompts. However, Chiang et al. claim that variants in prompts do not affect the overall conclusion, suggesting that averaging different outputs generated from slightly different prompts with MCE/LPE may be enough to draw the general conclusion that each generative LLM cannot infer like humans and represent their disagreements. \n\n**The paper does not clearly communicate actionable insights from their work. This work has the potential to more systematically understand whether we can map model uncertainty to human uncertainty, but at present it misses that mark.**\n\nWe agree that the present work does not clearly address actionable insights, hence, we extend the discussion below and will rephrase the introduction and discussion sections to better communicate: Compared to previous works you mentioned, such as Santurkar et al. and Gillardi et al., our work is the first study to explore whether billion-scale generative LLMs could capture human disagreements in the most fundamental NLP task, NLI. Santurkar et al. found “misalignment between the views of current LLMs and US demographic groups,” by prompting personal sensitive questions. One of the reasons for this misalignment issue is stated as “the left-leaning tendencies in human feedback” in the paper, which can be easily inferred from the result of current LLM steering towards particular demographic groups.\n\nHowever, it is more difficult to find the explicit reason for misalignment between current LLMs and humans on NLI tasks since there might be diverse sets of reasons leading to human disagreement, such as annotator disagreement, subjectivity, and multiple plausible answers (Plank et al., 2023). We try to find a subtler insight into why the LLMs disagree in some instances by a simple regression model (Figure 5) on LLM vs. human disagreement level (i.e., entropy) per instance. However, we did not find a direct relationship between the reasons of humans and LLM entropy level (See “What causes LLM to disagree?” in Discussion). In conclusion, we allude to the need to reflect a larger population in generative LLMs into less-sensitive issues by our experimentation on NLI, a simple yet effective testbed to represent casual disagreements. \n\nWe also deeply appreciate you for the comments on missing references and typos/presentation improvements to bring a stronger presentation for the paper. We will respond to some of them:\n\n**Missing References:**\n\nWe will integrate the following important references in our Related Works section to clarify and strengthen our position:\n\n•\tIntercoder agreement results (Gilardi et al., 2023)\n\n•\tConstrained decoding & logit bias adjustments to LLMs to generate a more confident distribution output. (Ziems et al., 2023)\n\n•\tExplicit expression of uncertainty can be exploited to better performance (Zhou et al., 2023)\n\n•\tThe limitation of status-quo model training which leads to poor calibration towards human perspectives (Gordon et al., 2021)\n\n\n**Typos Grammar Style And Presentation Improvements:**\n\n\n•\t**[GPT Naming Convention]** Thank you for this suggestion. We will change GPT-3.5-D2/D3 to GPT-3-D2/D3.\n\n\n•\t**[Chaos^2NLI]** We will change Chaos^NLI to HighChaosNLI to avoid confusion on the “squared” part.\n\n\n•\t**[Figure 3 Label Shorthand]** Thank you for this suggestion. We will put the same shorthand of model names across all the figures and tables (e.g., Flan-T5-XXL to FlanXXL).\n\n\n•\t**[Figure 5 Better Communication]** Yes, it would be ideal to have a linear relationship between humans and LLM entropies. We will add the y=x line to show the ideal case. We were expecting to see whether there is a relationship between human disagreement reason categories and LLM entropy level. If there is, we would have observed the same colored dots clustered with similar y-axis values. However, this was not the case. We also want to point out that whereas Figure 4 presents the overall counts of entropy levels for humans and each model, Figure 5 shows the relationship between human and model entropy levels for each instance, and the instance is categorized by human disagreement sources. Thank you for your suggestion on improving the clarity of uncertain parts in the paper. We will surely extend this explanation in or after lines 422-427.\n\n**References:**\n\n\n•\tSanturkar, S. et al. (2023). Whose Opinions Do Language Models Reflect? In International Conference on Machine Learning, pages 29971-30004.\n\n•\tChung, H. W. et al. (2022). Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.\n\n•\tOuyang, L. et al. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Volume 35, pages 27730-27744.\n\n\n•\tZhou, K., Jurafsky, D., & Hashimoto, T. (2023). Navigating the Grey Area: Expressions of Overconfidence and Uncertainty in Language Models. arXiv e-prints, arXiv-2302.\n\n\n•\tChiang, C. H., & Lee, H. Y. (2023). Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pages 15607–15631.\n\n\n•\tGilardi, F., Alizadeh, M., & Kubli, M. (2023). Chatgpt outperforms crowd-workers for text-annotation tasks. In Proceedings of the National Academy of Sciences of the United States of America, Volume 120, No 30, pages 1-3.\n\n\n•\tPlank, B. et al., (2022). The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Understanding the mechanisms we can use to model human disagreement with LLMs is increasingly important as other works aim to use LLMs as annotators, models of human opinion, and even to score other ML systems. The focus of this work on developing a sound metric of mapping LLMs to distributions of opinions has potential impacts in each of those domains.\n\n- The selection of models is extremely thorough, a significant positive aspect of this work over similar prior works of this kind.", "reasons_to_reject": "- The fundamental comparison seems to incompletely answer the question layed out in the title, confounded by the way the models are compared to human annotators. The probability distribution of labels assigned by different human annotators is the average of opinions from multiple agents. However, disagreement does not neccesarily mean that each individual annotator was uncertain in the label they selected. Given the high variance from prompting, it seems not sufficiently sound to compare \"agreement\" of LLMs by using the sampled variance over one prompt. \n\n- The paper does not clearly communicate actionable insights from their work. Santurkar et al. looks at model agreement to understand how models reflect society to provide actionable insights about the biases they may capture. Gillardi et al. looks at model agreement to understand whether uncertainty can be a proxy for annotation quality. Both of these works have already captured varying aspects of the overconfident single perspectives LLMs represent. This work has the potential to more systematically understand whether we can map model uncertainty to human uncertainty, but at present it misses that mark.", "questions_to_authors": "A) On Lines 154-156, you cite the decoding strategy as a limiting factor for exploring the whole space. How does this apply to your NLI tasks, where the model is selecting between a finite number of options rather than performing open-ended decoding? With lines 162-163, it seems like you are saying each class label will be inferred based on the open-ended generation, but this isn't totally clear.\n\nB) Line 182 - 185, suggest the probability mass is non-constant, but this is a choice by this calculation. Why don't you use the adjustment from Santurkar et al. 2023 in your citations to make the probability mass constant by estimating the mass of missing options as their upperbound?\n\nE) The FLAN Models use the full ANLI and QNLI training datasets as part of their instruction fine tuning. Why are they marked as Partial Exposure?\n\nD) Line 245-246, How can you confidently say that Davinci 2 has no exposure to these tasks? It received supervised finetuning on an unknown dataset, which based on my understanding means it could have partial or complete exposure to these tasks.\n\nE) Line 264-268, Why can't you compare the probabilities of the first token of each option for OS LPE? Entailment, Contradiction, and Neutral all have different leading tokens so the probability of each leading token seems a valid proxy for the whole word.\n\n", "paper_topic_and_main_contributions": "This work looks at LLM performance on the classic task of NLI and evaluates metrics which estimate the uncertainty of LLM models for these fixed label tasks. It then uses these metrics to look at whether model size, exposure, and architecture has significant effects on both model performance and the agreement between model uncertainty and human uncertainty. From this they drive towards surrounding whether or not LLMs are accurate models of human disagreement and uncertainty using NLI as a clear testbed that has a format which is broadly applicable.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Includes results on Internal Agreement from LLMS:\nhttps://arxiv.org/abs/2303.15056\n\nThis work seems to solve the challenge of options not occurring in the top-k tokens using OpenAI's logit bias, see Table 1:\nhttps://arxiv.org/pdf/2305.03514.pdf\n\nAn example of how minor wording shifts could be used to more accurately model different perspectives and how these effect model uncertainty\nhttps://arxiv.org/pdf/2302.13439.pdf\n\nA strongly motivated work looking at how model training leads to poor calibration with human perspectives\nhttps://dl.acm.org/doi/abs/10.1145/3411764.3445423\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Lines 241-252: The naming conventions you use for davinci-002 and 003 are confusing if you are familiar with the api names. The first model officially referred to as GPT 3.5 in the official documentation is ChatGPT. Since here you aren't referring to ChatGPT, I'd remove the \".5\" from your shorthand.\n\nLine 382: Is Chaos^2NLI a widely used subset and shorthand? If so, cite it. If not, I'd suggest changing the shorthand, as the Chaos^2NLI ends up looking like ChaosNLI with a footnote, especially since the \"squared\" part doesn't have clear semantics.\n\nFigure 3: The label shorthand was not apparent on first pass and required scrolling back and forth between table 1 and figure 3 to figure out the mapping. I'd try to establish this shorthand earlier in the work so that readers expect it throughout rather than changing shorthand depending on the space limitations of different tables.\n\nFigure 5: It's not clear what this figure is communicating. What is the ideal trend we'd like to have in this graph? A linear relationship between entropies? If so, communicate that in either text or visualization. How do we expect the trends to change when delineated by categories? Are we hoping to see a more clear trend for some categories than for others? If so, communicate this in the text. Right now, these figures take up a good bit of space, but they just seem to communicate the same info as Figure 4 in a different format."}}, {"review_id": "e5fO0qVz20", "reviewer": "Reviewer_UZ9d", "summary": "", "questions": "Question A: in line 184 you mention the unknown non-constant probability mass in LPE. Is this a limitation? How does it affect the conclusions one can draw from this metric? Is there a way to mitigate it?\n\nQuestion B: are the encoder-only models (BERT, RoBERTa) also prompted in the same ways as the encoder-decoder (Flan-T5) and decoder-only (GPT, OPT, Vicuna) models? The usual methodology is to fine-tune them with a classification layer, but from the paper it seems they are prompted. Is there previous work comparing these approaches?\n\nQuestion C: which distribution estimation is used for the experiments reported in Figure 3?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UZ9d,\n\nThank you for the time to review our paper thoroughly. We deeply appreciate how you found our methods useful for estimating LLMs, which currently lack accuracy and alignment with human disagreement in NLI.\n\nBelow, we have tried our best to address your questions and concerns. Please take a look at our response and let us know if further clarification is needed.\n\n**Q1)  You mention the unknown non-constant probability mass in LPE. Is this a limitation? How does it affect the conclusions one can draw from this metric? Is there a way to mitigate it?** \n\nThe unknown probability mass is displayed due to two reasons: a missing option or the model's characteristic to be unconfident. It is important this is carefully dealt with as the amount of disagreement can be represented as a part of the unknown mass as well.\n\nFor the prior component, Santurkar et al. assign the probability mass of missing options to the minimum value of the remaining mass or the minimum probability assigned to any of K token choices and normalizes after, which will lead to an unsound overestimation of unspecified (i.e., missing) options. For Ziems et al., logit bias adjustments were also shown to be effective. In contrast, we make adjustments by not assigning probability mass for the missing options and directly normalizing our output which might be overestimating the probabilities of existing options, but refraining from any problematic assignment to missing options.\n\nFor the latter component, the underconfidence of the LLM outputs is a major problem. While, numerous methods to re-calibrate the model confidence, through temperature scaling or label smoothing (Wang et al., 2022); however, this is not an option we adopt as we plan to investigate the unscaled evaluation across the model without optimizing it for every single task/model.\n\n**Q2) Are the encoder-only models (BERT, RoBERTa) also prompted in the same ways as the encoder-decoder (Flan-T5) and decoder-only (GPT, OPT, Vicuna) models? The usual methodology is to fine-tune them with a classification layer, but from the paper it seems they are prompted. Is there previous work comparing these approaches?**\n\nNo, the encoder-only models are not prompted but evaluated by the usual method that you mentioned. Plus, this is why we separate Full Exposure (FE) models from Partial/No Exposure (PE/NE) models, but we should have been clearer. Additionally, as far as we are concerned, no previous approaches jointly compare encoder-only (FE models in this work), encoder-decoder, and decoder-based models in the scope of NLI and human disagreement. Thus, such comparisons can be considered as the main novelty introduced first in our paper in the era of generative LLMs which are too large for single-task fine-tuning.\n\n**Q3) Which distribution estimation method is used for the experiments reported in Figure 3?**\n\nWe are aware of this confusion, mainly because of the mismatch of label shorthand for model names across figures and tables. We will first revise the label shorthand (e.g., Flan-T5-XXL (Table 1) and FXXL (Figure 3) $\\rightarrow$ FlanXXL, GPT-3.5-D3 (Table 1) and G3 (Figure 3) $\\rightarrow$ DAV3). In Figure 3, we use LPE with a prompt type of Number Selection (NS) for G3, G2, and SV, and  MCE with a prompt type of Option Selection (OS) for the other models, same as Table 3. We will add these details in the caption for Figure 3 to avoid confusion.\n\n**Insufficient positioning with respect to previous work on label distribution and capturing human disagreements (see Missing References)**\n\nYes, we appreciate your comment on how our paper has insufficient positioning w.r.t. previous work. This is possibly due to our emphasis on mainly showing comprehensive, easily applicable evaluations for diverse tasks and models including generative LLMs on the alignment of human disagreement.  We will take on board your suggestion to discuss papers concerning uncertainty recalibration (Wang et al. 2022, Xiao et al. 2022), and disagreement (Uma et al., 2021, Fornaciari et al., 2021, Zhang et al. 2021). We will further position ourselves on these previous studies regarding label variation and human disagreements for the final version of the manuscript.\n\n\n**While Distributed NLI [1] is cited in passing, for example, a more thorough comparison is warranted between it and the current approach, as it also proposes similar output label distribution estimation methods and evaluates them on NLI.**\n\nYes, Distributed NLI is a highly related work that we were paying attention to. However, as we previously mentioned our goal is to inspect general methods that are applicable to different models of size (Encoder, Decoder, Encoder-Decoder) and openness (Huggingface, Closed API such as OpenAPI, etc.), We try to evaluate the simple methods that we suggest without calibrating the confidence on the model for every model to respective datasets.\n\n\n**While clearly a form of anthropomorphism, it is not clear what \"disagreement\" refers to when used to describe LLMs - is it about disagreement between different LLMs or within the same LLM? From the paper, it is evident that the latter is meant, but this metaphorical narrative obscures the real research question in a way that makes it difficult to assess its validity.**\nThank you for confirming that our methods only regard the agreement coming from within the same LM. Our intention for the initial titling comes from how LLMs are capable of representing dissenting voices through the lens of NLI. With such scaling parameters and datasets, which include voices of a significant population, we originally hypothesized LLMs to be able to represent disagreement, which we found not to be true in our experiments. \n\n\n**The characterization of models as having \"full exposure\", \"partial exposure\" or \"no exposure\" to NLI seems too coarse grained to me.**\n\nWe strongly agree with your comment on the coarse-grained categorization. In our initial stage of the research, the Partial Exposure (PE) level was divided into PE and Minimal Exposure (ME), where ME is assigned to GPT-3.5-D3 because its reward model was fine-tuned with the MNLI dataset. This way, GPT-3.5-D3 was differentiated from FLAN and OPT-IML models, which might explain our first result (lines 282-283) better. We will take your valuable comment to consider more fine-grained categorization of models.\n\n**A more rigorous approach could be to apply few-shot learning and then compare the effect of exposure size with a single baseline model. In particular, the conclusion that \"LLMs do not perform NLI like humans\" is simply misleading, since human annotators almost certainly do see examples before or during the annotation process.**\n\nWe appreciate your concern about the unfairness of the evaluation for LLMs without any given instances. However, we do not try to align LLMs with the same procedure with human annotators as we are focused more on the representation of human dissent, though maybe confounded with our title. Nevertheless, we are very interested in your concern, and it will be interesting to provide the “prescriptivist” approach evaluation with additional experimental results in the future study.\n\nWe also deeply appreciate you for the comments on missing references and typos/presentation improvements to bring a stronger presentation for the paper. We will respond to some of them:\n\n**Missing References:**\n\nWe will integrate the following important references in our Related Works section to clear our position:\n•\tComprehensive survey on the need to represent disagreement in various tasks in NLP & CV (Uma et al., 2021)\n\n\n\n•\tIncorporates *disagreement* as an informative signal in training models and uses the multinomial distribution to train the models  (Fornaciari et al., 2021) \n\n\n\n•\tIntroduces a novel annotating scheme to assign multiple labels for each instance, which can be used as high-quality datasets to better train models to achieve higher accuracy and lower JSD scores. (Zhang et al. 2021)\n\n\n\n•\tUses calibration methods such as label smoothing and temperaturing scaling to lower divergence scores between model probability and true human opinion distribution. (Wang et al. 2022)\n\n\n\n•\tIntroduces major features that better the pipeline of calibration error minimization expected calibration error (ECE) as a main metric to quantify uncertainty. (Xiao et al. 2022) \n\n**Typos Grammar Style And Presentation Improvements:**\n\n\n\n\n•\t**[Additional Explanation]** Thank you for pointing this out.  DCE is a total variation distance measure between LLM distribution estimated with MCE/LCE and human disagreement distribution (Baan et al., 2022). This metric is different from widely used Expected Calibration Error (ECE), which can often result in high ECE even if the model can perfectly estimate the human judgement distribution (Baan et al., 2022). We will add this explanation in the Related Works section (line 131).\n\n\n\n•\t**[Sentence Revision]** We will revise the sentence starting in line 141 to the following sentence: For example, GPT-3.5 is fine-tuned to follow instructions with human feedback using reinforcement learning method (Ouyang et al., 2022).\n\n\n\n•\t**[Clarification on Model Names]** GPT-3.5-D2/D3 refers to Text-Davinci-002/3, which is different from GPT-3.5-turbo. We exclude GPT-3.5-turbo in our experiment since at the time of experimentation it does not provide token-level log probability unlike Text-Davinci-002/3. We will make this clearer by changing the shorthand of GPT-3.5-D2/D3 to GPT-3-D2/D3.\n\n\n\n\n•\t**[Miscitation]** Thank you for your correction for the miscitation on StoryCloze. We will change the original citation to the citation that you suggested (Mostafazadeh et al., 2016).\n\n\n\n\n•\t**[Table 1 Clarification]** In Table 1 caption, clarify what \"old to new labels\" means. $\\rightarrow$ We will clarify that there exists two labels (old and new) for each instance in ChaosNLI datasets. The old labels are the majority class options selected by only five annotators, a strategy selected for MNLI, SNLI, and $\\alpha$-NLI datasets. The new labels are the partially revised ground-truth labels, by introducing 100 annotators for the ChaosNLI datasets.\n\n\n\n•\t**[Figure 2 Revision]** Figure 2 should be more explicit about how the full prompt looks and how the output from the model is parsed and mapped to labels. $\\rightarrow$ Yes, we should have been more explicit on the actual full prompts we used in Figure 2. Combining prompt examples in Appendix (p. 14) for different types of datasets and prompts, and parts of Figure 1 showing how the model outputs are sometimes discarded if they are not valid options or kept for vice versa might be needed to gain a better understanding on the process of generating LLM outputs.\n\n**References:**\n\n\n•\tSanturkar, S. et al. (2023). Whose Opinions Do Language Models Reflect? In International Conference on Machine Learning. pages 29971-30004.\n\n\n•\tZiems et al., (2023). Can Large Language Models Transform Computational Social Science?. arXiv e-prints, arXiv-2305.03514.\n\n\n•\tZhou, X., Nie, Y., & Bansal, M. (2022). Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 972-987.\n\n\n•\tWang, Y. et al. (2022). Capture human disagreement distributions by calibrated networks for natural language inference. In Findings of the Association for Computational Linguistics: ACL 2022, pages. 1524-1535.\n\n\n•\tNie, Y., Zhou, X., & Bansal, M. (2020). What Can We Learn from Collective Human Opinions on Natural Language Inference Data?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9131-9143.\n\n\n•\tBaan, J., Aziz, W., Plank, B., & Fernández, R. (2022). Stop Measuring Calibration When Humans Disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892-1915.\n\n\n•\tOuyang, L. et al. (2022). Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Volume 35, pages 27730-27744.\n\n\n•\tMostafazadeh, N. et al. (2016). A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Capturing human disagreements is an important open problem in LLMs, and this paper utilizes the extensive work done previously on human disagreements in NLI to evaluate LLMs on this problem.\n\nThe proposed output distribution estimation methods can be useful for future work estimating and calibrating LLMs.\n\nThe results highlight the limitations of current LLMs and identify factors that limit their accuracy and alignment with human label distributions in NLI.", "reasons_to_reject": "[Addressed in rebuttal] Insufficient positioning with respect to previous work on label distribution and capturing human disagreements (see Missing References). While Distributed NLI [1] is cited in passing, for example, a more thorough comparison is warranted between it and the current approach, as it also proposes similar output label distribution estimation methods and evaluates them on NLI.\n\n[Addressed in rebuttal] As acknowledged in line 480, the authors \"formally define and represent the multinomial soft label estimate of a model as \"disagreement,\" but there is no theoretical support to link these terminologies\". I could not say it better, and I believe this is a serious limitations that was a major source of confusion for me while reading the paper: while clearly a form of anthropomorphism, it is not clear what \"disagreement\" refers to when used to describe LLMs - is it about disagreement between different LLMs or within the same LLM? From the paper it is evident that the latter is meant, but this metaphorical narrative obscures the real research question in a way that makes it difficult to assess its validity.\n\nThe framing of the paper suggests that the NLI task as operationalized in datasets bears some kind of \"truth\" that LLMs (with no exposure to NLI data) fail to capture despite their language capabilities. In reality, as acknowledged by both Manning [2] and Pavlick and Kwiatkowski, humans have different assumptions and follow different processes in NLI annotation, even if following a \"natural\" rather than a \"prescriptivist\" ask formulation. Especially where the \"prescriptivist\" approach is used for annotation, though, it is not fair to evaluate the \"no-exposure\" models and expect them to approximate humans who have read the (potentially artificial) task guidelines.\n\n[Addressed in rebuttal] The characterization of models as having \"full exposure\", \"partial exposure\" or \"no exposure\" to NLI seems too coarse grained to me. While it may simply not be known to which extent GPT-3 was exposed to NLI data, there is a large difference between fine-tuning on one or thousands of examples. A more rigorous approach could be to apply few-shot learning and then compare the effect of exposure size with a single baseline model. In particular, the conclusion that \"LLMs do not perform NLI like humans\" is simply misleading, since human annotators almost certainly do see examples before or during the annotation process.\n\n[1] Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022. Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 972–987, Dublin, Ireland. Association for Computational Linguistics. https://aclanthology.org/2022.findings-acl.79/\n\n[2] Christopher D. Manning. 2006. Local textual inference: It’s hard to circumscribe.", "questions_to_authors": "Question A: in line 184 you mention the unknown non-constant probability mass in LPE. Is this a limitation? How does it affect the conclusions one can draw from this metric? Is there a way to mitigate it?\n\nQuestion B: are the encoder-only models (BERT, RoBERTa) also prompted in the same ways as the encoder-decoder (Flan-T5) and decoder-only (GPT, OPT, Vicuna) models? The usual methodology is to fine-tune them with a classification layer, but from the paper it seems they are prompted. Is there previous work comparing these approaches?\n\nQuestion C: which distribution estimation is used for the experiments reported in Figure 3?", "paper_topic_and_main_contributions": "Evaluation of several recent Large Language Models (LLMs) on several Natural Language Inference (NLI) benchmarks, in particular benchmarks designed to capture human disagreements.\n\n- Models are prompted on multiple-choice NLI in two settings, Option Selection (OS) and Number Selection (NS).\n- Two methods are proposed to estimate the model output distribution, Monte Carlo Estimation (MCE) and Log Probability Estimation (LPE), which can be used in different situations depending on whether the model exposes the estimated probabilities of its output tokens.\n- Evaluation metrics include accuracy as well as distribution differences between estimated LLM output and human labels, using three metrics, Jensen-Shannon Divergence (JSD), Kullback-Leibler Divergence (KL) and Distribution Calibration Error (DCE).\n\nResults show that larger models with no exposure to NLI data underperform and do not align with human label distributions.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Uma, Alexandra N., et al. \"Learning from disagreement: A survey.\" Journal of Artificial Intelligence Research 72 (2021): 1385-1470. https://jair.org/index.php/jair/article/view/12752\n\nTommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591–2597, Online. Association for Computational Linguistics. https://aclanthology.org/2021.naacl-main.204/\n\nShujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with Different Amounts of Annotation: From Zero to Many Labels. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7620–7632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. https://aclanthology.org/2021.emnlp-main.601/\n\nYuxia Wang, Minghan Wang, Yimeng Chen, Shimin Tao, Jiaxin Guo, Chang Su, Min Zhang, and Hao Yang. 2022. Capture Human Disagreement Distributions by Calibrated Networks for Natural Language Inference. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1524–1535, Dublin, Ireland. Association for Computational Linguistics. https://aclanthology.org/2022.findings-acl.120/\n\nYuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022. Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7273–7284, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. https://aclanthology.org/2022.findings-emnlp.538/", "Typos_Grammar_Style_and_Presentation_Improvements": "116: \"Meissner et al., 2021\" should be in parentheses.\n\n130: \"DCE\" should be explained somewhere - how is it different from the proposed methods?\n\n141: the sentence implies that RLHF and instruction tuning are the same thing, but they are not.\n\nFootnote 7 refers to GPT-3.5-turbo but elsewhere you refer to GPT-3.5-D2/3. Clarify the relationship between these.\n\n195: Citing Shakespeare for StoryCloze seems wildly inappropriate. Better cite Mostafazadeh et al. [1].\n\nIn Table 1 caption, clarify what \"old to new labels\" means.\n\nFigure 2 should be more explicit about how the full prompt looks and how the output from the model is parsed and mapped to labels.\n\n349: \"less\" should be \"lower\".\n\n[1] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California. Association for Computational Linguistics. https://aclanthology.org/N16-1098/"}}]

Author rebuttal:

Meta-review: This paper presents examines alignment of LLMs with humans on the task of natural language inference. Thorough experiments on many models indicate that LLMs fail to adequately capture human disagreement. Concerns about the paper include the lack of actionable insights and the framing of the paper, which were addressed in the lengthy rebuttals.
Decision: Accept-Main