Bring Your Own Data! Self-Sensitivity Evaluation for Large Language Models
Authors: Neel Jain
Khalid Saifullah
Yuxin Wen
John Kirchenbauer
Manli Shu
Aniruddha Saha
Micah Goldblum
Jonas Geiping
Tom Goldstein
Venue: COLM (2024)
Keywords: Evaluations
TL;DR: Proposes a novel evaluation framework for LLMs using invariances/sensitivities

Abstract: With the rise of Large Language Models (LLMs) and their ubiquitous deployment in diverse domains, measuring language model behavior on realistic data is imperative. For example, a company deploying a client-facing chatbot must ensure that the model will not respond to client requests with profanity. Current evaluations approach this problem using small, domain-specific datasets with human-curated labels. These evaluation sets are often sampled from a narrow and simplified distribution, and data sources can unknowingly be leaked into the training set. To alleviate these issues in traditional evaluation, we propose a complementary framework for additional self-sensitivity evaluation of LLMs by analyzing their sensitivity or invariance to transformations on the input text. Self-sensitivity evaluation can directly monitor LLM behavior on datasets collected in-the-wild or streamed during live model deployment. We demonstrate self-sensitivity evaluation strategies for measuring closed-book knowledge, toxicity, long-range context dependence, in addition to sensitivity to grammatical structure and tokenization errors. When comparisons to similar human-labeled benchmarks are available, we find strong correlations between self-sensitivity and human-supervised evaluations. The self-sensitivity paradigm complements current evaluation strategies that rely on labeled data.

Published as a conference paper at COLM 2024
Bring Your Own Data!
Self-Sensitivity Evaluation for Large Language Models
Neel Jain*t, Khalid Saifullah*t
Yuxin Wen', John Kirchenbauer', Manli Shu', Aniruddha Saha', Micah Goldblum Jonas Geiping', Tom Goldstein
Uniiersity of Maryland, NNew York University
Abstract
With the rise of Large Language Models (LLMs) and their ubiquitous de- ployment in diverse domains, measuring language model behavior real- istic data is imperative. For example, a company deploying a client-facing chatbot must ensure that the model will not respond to client requests with profanity. Current evaluations approach this problem using small, domain- specific datasets with human-curated labels. These evaluation sets are often sampled from a narrow and simplified distribution, and data sources can unknowingly be leaked into the training set. To alleviate these issues in tra- ditional ev evaluation, we propose complementary framework for additional self-sensitivity evaluation of LLMs by analyzing their sensitivity or invariance to transformations on the input text. Self-sensitiiity evaluation can directly monitor LLM behavior on datasets collected in-the-wild or streamed dur- ing live model deployment. We demonstrate self-sensitivity evaluation strategies for measuring closed-book knowledge, toxicity, long-range con- text dependence, in addition to sensitivity to grammatical structure and tokenization errors. When comparisons to similar human-labeled bench- marks are available, we find strong correlations between self-sensitivity and superviid evaluations. The self-sensitivity paradigm com- plements current evaluation strategies that rely on labeled data. Code is available at https:/ //tpt/gg.btt.tt//// cow/ http/eeelsjeee jjain/BYOD.
1 Introduction
As Large Language Models (LLMs) continue to advance rapidly, there has been grow- ing demand for new evaluation metrics that can accurately capture their capabilities and limitations (Ethayarajh & Jurafsky, 2020; Birhane et al., 2022; Kiela et al., 2021; Bowman & Dahl, 2021). As a result, there has been a constant need to create new datasets as newer models continuously make the existing datasets obsolete. Recent approaches such as BIG- Bench (Srivastava et al., 2022) and HELM (Liang et al., 2022) aim to address this issue by providing ever-increasing, diverse set of accumulating tiirr- to measure the performance of LLMs. However, these approaches still rely heavily on dataset creation and curation, which is time-consuming and expensive. Furthermore, evaluation is generally dataset-centrie meaning that evaluations are based on some human-labeled or generated metric evaluated on fixed dataset. For modern LLMs, this conventional approach comes with new complications. First, evaluation data is hosted on the internet (for example on sites like GitHub). This makes them accessible scraping bots that generate training data for LLMs, making older datasets unreliable unless they are painstakingly removed from the training set, which does not reliably happen (Brown et al., 2020; Gao et al., 2021).' Second, LLM evaluation is by its nature multi-faceted, since different LLM applications rely on distinct capabilities, and an ever-increasing number of such capabilities needs to be tested in
* Equal contribution. Correspondence to: Neel Jain <njainl7@umd.edur are trying to catalog this
Efforts such as https://gi thub. phenomenon for ChatGPT and other models.
1

---

Published as a conference paper at COLM 2024
April the fourth... April not the fourth...
Large Language Model
Invariance Score
Lionel Messi is Lionel Messi is not
Transformation
Similarity
Corpus
e.g, Fact negation
Metric
Figure In our proposed self-sensitivity evaluation, pairs of original and perturbed texts are created from a corpus. In the figure above, the pair is related via a negation achieved by inserting "not" into the original text. These pairs are then fed into the network, and the outputs (perplexity, probability distributions, text) are compared for each pair. These measures then aggregated to produce an invariance or sensitivity score.
modern LLMs. As dataset curation is expensive, each test in large benchmark like HELM (Liang et al., 2022), uses only a small dataset carefully created to test a particular capability in particular scenario. However, models then deployed in much broader contexts and settings, and the applicability of these evaluations to deployment usage can be uncertain.
To complement conventional evaluation, propose framework for self-sensitivity model evaluation. In this framework, metrics are defined as invariances and sensitivities that can be checked in "self-supervised" fashion using interventions based only the model in question rather than external labels. Self-sensitivity evaluation pipelines are dataset-agnostic,, so they can be utilized over larger сorpora of evaluation data than conventional metrics, or even directly in production systems to monitor day-to-day performance. In this work, we develop this framework, discuss desiderata for such metrics, and provide several case studies for self-sensitivity metrics: measuring knowledge through negations, toxicity detection, long-range dependency, word-order, and tokenization sensitivity. By developing these new metrics, we hope to provide more comprehensive and nuanced understanding of the strengths and limitations of LLMs.
2 A Procedure for Self-Sensitivity Evaluation
Our goal to measure properties of LLMs such as toxicity, closed-book knowledge, and word order sensitivity without relying on benchmark-specitic datasets or human annotations. Rather than measuring model accuracy against known ground truth labels, we choose simple transformation that can be applied to text. We then measure the level of invariance that a model's output has under that transformation. If we choose our transformations carefully, we obtain useful information about model behavior in completely "self- supervised" way. We consider information from these metrics useful by grounding them to existing to labeled benchmarks. The goal of the framework is not to replace labeled evaluation, but by grounding with existing benchmarks, to show that something important being captured by the metrics, thereby demonstrating the overall efficacy of the self-sensitivity framework.
More formally, given a text corpus D, we construct pairs of original passages x, and trans- formed counterparts x'. An example is seen in Figure 1, where we negate the original sentence to construct x'. is the set of all transformed pairs. We then feed input pairs into the language model, f, to extract pair of outputs. Depending on the construction, the output being considered can be the softmax probability vector over tokens, a perplexity score, or feature vector. We then compare the outputs f(x) and f(x') using a similarity metric, M. Finally, we aggregate the results over all pairs in the data corpus using an aggregation operator, A, to produce a sensitivity score.
SCORE (()) X}.
(1)
2

---

Published as a conference paper at COLM 2024
Knowlodge
Knowledge
Chat
Long Range
Long Range
9:225
Av225
0:15
0:075
0:075
0.75
0.5
0.29
Paxicity
0.75 0.5 022
Turiity
00.025
0.05
D41
Gens
0.15
0.2
Q.1
0.1
Word
Order
9.90
Tokorization
Tokerization
Figure 2: Spider plots showing sensitivity scores for the Knowoledge Probing via Negations, Toxicity, Context Long-Rangeh Word Order, and Tokenization metrics introduced in the paper. (Left) Comparison between LLaMA-2 (7B) and LLaMA-2-Chat (7B) models. We see that Chat improves except tokenization robustness. (Right) Comparison between LLaMA-2 (7B) and the instruction finetuned Vicuna-v1.5 (7B). We see that the Chat version is better on most metrics except tokenization robustness. For toxicity, we include the system prompt for the chat model to measure the behavior of the model in a deployment setting.
all metrics
In this work, we "bring" wikipedia as our own dataset but note that we only do so to enable comparisons to existing metrics that use human labels on similar data. We validate that this framework works equally well with domain-specific medical data in Section 4.2. We study several case studies, namely knowledge via negations (Section 4), toxicity (Section 5), word order sensitivity (Section 6), context sensitivity (Appendix A.2), and tokenization robustness (Appendix A.3) culminating in sensitivity scores seen in Figure 2.
3 Related Work
HELM adopts a multi-metric approach: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency over each of the datasets proposed (Liang et al., 2022). These metrics build on the work of Ribeiro et al. (2020) and subsequent studies such as Mille et al. (2021); Wu et al. (2021); Ross et al. (2021); Dhole et al. (2021); Yang et al. (2022) which augment imputs from a dataset to measure properties beyond the classical metric of accuracy. In summarization and machine translation, automatic metrics have been utilized to evaluate models (Lin, 2004; Papineni et al., 2002). In addition, recent work has explored using other models to compare the quality of chat and instruction models (Chiang et al., 2023; Wang et al., 2023; Peng et al., 2023). While these methods rely on existing labeled data, or external oracle models, our method departs from these previous works we analyze invariances using a data-agnostic procedure. A survey by Chang et al. (2023) contains comprehensive overview of evaluations for LLMs. In addition, we include a more comprehensive related works in the Appendix A.1.
4 Knowledge Probing via Negations: Au Contraire Metric
Knowledge probing in specific target domains is an important way to assess how model will behave in different deployment scenarios. OpenAl approached this problem by constructing nine adversarial datasets on varying areas such as Law and Technology to evaluate GPT-4 (OpenAI, 2023). While OpenAl's approach and others like MMLU (Hendrycks et al., 2021) are step forward, these datasets do not cover all possible domain-specific areas. Therefore, when deploying model, it is important to understand its ability to comprehend the potentially narrow domain-specific information its use case. We probe this capability by testing whether the model is actually surprised (in terms of perplexity) by negated facts in target domain. In this case study, we can further easily compare to domaan-specific areas, such medical terms. Finally, knowledge is good setting to validate our generic approach, as model knowledge can also be approximately probed by measuring the perplexity of text passages.
3

---

Published as a conference paper at COLM 2024
Self-Sensitivity Approach: We construct self-sensitivity transformation over factual information by automatically applying negations to facts, as, for example, given by passages containing Wikipedia entities and medical terms. We search for the first occurrence of is, was, or were, and place the word not after it provided a negation is not already present. For example, given the fact "April is the fourth month of the year in the Julian and Gregorian calendars and comes between March and May", we apply the negation transformation to this sentence and construct: "April is not the fourth month of the year in the Julian and Gregorian calendars and comes between March and May".
Based on this intervention, we measure the change in the log-perplexity (log(p)(xx)),, between the original and negated sentence, which can be defined as the following:
1 Σ log(ppl(<<)) log(ppl(x)))
SENSITIVITY SCORE
One potential issue with this naive sensitivity score is that some models may be inherently more sensitive (i.e,, have higher perplexity) to the word not regardless of the context. We, therefore, propose to normalize model's sensitivity score with its score on a "neutral" corpus, where negations do not create untruthful statements that impact model's perplexity. We define our final sensitivity score as:
L -llppp((/)))o(ppppppp
NORMALIZED SENSITIVITY SCORE SENS. SCORE
where is a sample from neutral corpus like bookcorpus with m total samples for which there is no clearly defined truth value. We take the absolute value of the difference on the neutral corpus because there is no clear direction between the transformations. To evaluate the relationship of these metrics also record the fraction of inputs for which perplexity decreases after introducing negation, which represents, for a typical sample, the error that the model is making: PERCENT PPL DROPS = Σ max (sign(log(pp((:)) - log(ppl(x/))).).
model confidence in our analysis, we
4.1 Experimental Set-up
We use the normalized sensitivity score as our SSE metric. We found that the un-normalized scores can closely follow square-root relationship with the accuracy of the human-curated TriviaQA dataset (Figure 13) for non-instruction finetuned models. However, we found normalization corrects the instruction-tuned models to larger degree, possibly due to their innate overconfidence. Thus, we use this metric for our scores. We further explore why correct normalization is important by cross-referencing the frequency with which perplexity goes down rather than up, see Figure 14 in Appendix A.5.
We validate our metric by comparing evaluations gauge an LLM's world knowledge (Joshi et al., 2017). We obtain the knowledge score using our SSE metric on topic sentences from Wikipedia, which consists of statements of general knowledge similar to TriviaQA. A human inspection of 100 samples verified that the proposed transformation resulted in grammatically correct sentences that were counterfactuals of the original sentence. We further verify our metric on medical terms by comparing to conventional metrics on MMLU (clinical knowledge) Hendrycks et al. (2021) on open-source models. We measure the proposed sensitivity score 1000 examples, where the standard error for these scores was less than 0.002. Our metric supports both publicly available models from the Hugging Face Hub and API models (Biderman al., 2023; Brown et al., 2020; Radford et al., 2019).
the conventional accuracy on TriviaQA, both
4.2 Results
From Figure 3, we see that the normalized score correlates well with TriviaQA. The Pearson correlation between TriviaQA and Normalized sensitivity score is 0.76 for vanilla models and 0.73 for instruction models after removing the Cohere Command outlier, which we will discuss below. We further verify that our score indeed tracks perplexity for vanilla models,
4

---

Published as a conference paper at COLM 2024
Sensitivity Score (Norm.) vs TriviaQA Acc.
Sensitivityy Score (Normallized vs PPL
Sensitivity Score (Normaiized)
x
Sensitvityy.co Score (Normalized)
X
Model
Opendi instruct
Opendi
Cohore
0.32
Cohere Instuct
0.25
X
1.25
Vicuna
MPT
LLa MA
0.23
0.20
6B
Pytria
MPT Instuct
0.35
2.33
Ca actics
Do V2 instrctt
0.12
0.10
CPT)
ly VL instrut
0.05
CPT2
Neo
Indtrucioon
TriviaQA Acc.
Log PPL
Figure 3: (Leff) Normalized Sensitivity Score (negations) compared to accuracy on TriviaQA over various model sizes and families. (Right) Normalized Sensitivity Score (negations) compared to Perplexity. Larger markers correspond to bigger models, and "x" markers represent instruction finetuned models.
with correlation coefficient of 0.93. To further understand these results, we examine the outliers.
Interestingly, the correlation with perplexity is only 0.61 for instruction-tuned models, showing that perplexity as an evaluation metric is not as robust as our proposed approach for instruction models, possibly due to their overconfidence. Examining the details, we find that outliers like MPT-Instruct, which has a surprisingly low perplexity and was trained on the Wikipedia (en) dataset for 8 epochs, while others models were trained for 2 (LLaMA) or 3 epochs (PILE models). Yet, the normalized sensitivity score of MPT-Instruct is as expected. Additionally, we find that LLaMA has particularly low entropy, which may also explain its lower score (see Figure 8 in Section 7.3).
Another interesting outlier is the Cohere Command model when comparing normalized sensitivity and TriviaQA performance. Evaluating qualitative examples in (see also Table 1), we find that this model rarely changes its answer, whether negation is introduced, or not. This implies that the Cohere model is insensitive to the actual argument structure it is likely to have memorized the associations between concepts and answers based on the context alone, even if the construction of the question makes its answer incorrect. This inability to answer grammatically complex questions is not reflected in the TriviaQA results, because TriviaQA only contains simple sentence structures and nearly uniform question formats. This also highlights weakness of TriviaQA - its simple and predictable sentence constructs yield benchmark that rewards correct concept associations rather than correct answers. In contrast, our metric correctly identifies the model's limited understanding.
Effect of Instruction Finetuning: In general, we find that instruction-tuned models are, on average, more sensitive to negations than other LLMs as seen in Figure 5, for different sources of instruction data and even after correcting for the relative overconfidence of these models via normalization.? This may indicate that these models are, in general, more capable of evaluating knowledge picked up during pretraining (Zhu & Li, 2023). The outlier here is again the Cohere command model, which is less sensitive than Cohere's base model after finetuning.
4.3 Domain-Specific Evaluation
In Figure 4, we compare SSE to MMLU (clinical) and perplexity (on medical terms) on the open-source subset from before, finding strong correlation with MMLU (clinical) with a Pearson correlation of 0.91 and 0.77 for perplexity. However, the lower correlation with perplexity is due to deficiencies in the perplexity score. An obvious outlier Galactica, which has a similarly high log perplexity as Pythia 6.9B, but its MMLU Clinical Knowledge
Note that Dolly V2 use Pythia base models, Dolly V1 uses GPT-J, Vicuna-v1.1 uses L.L.Ma. Other instruction models names be variants of the base model name.
5

---

Published as a conference paper at COLM 2024
Sens. Score (Norm.) MMLU Clinical
Sensitivity Score (Norm,) vs PPL
0.2
0.20
Sensitivity Score (Nomm.)
0.5
6.35
Medel Family
LLaM
NPT
0.9
Sensitiitty
6,30
GFT)
Pytlic
GF12
0.5
0.05
Gelartica
0.5
0.00
Clinical Knownledge
Log
Figure 4: (Left) Normalized Sensitivity Score (negations) compared to accuracy on MMLU (clinical knowledge) over various model sizes and families. (Right) Normalized Sensitivity Score (negations) compared to Perplexity is shown the right on medical terms. Larger markers correspond to bigger models.
score 49% while Pythia is 29%. We see similar case for GPT-J in terms of perplexity with low perpleelyyy and low clinical knowledge score, whereas the relationship should be inverse. However, our proposed normalized sensitivity accurately predicts the correct ranking, and correctly determines Galactic the strongest model on this domain-specific evaluation. This suggests that our metric can be deployed even in specialized domains where ground-truth labels are not available in sufficient quantity.
For the sensitivity score to measure truthfulness, the dataset being used must contain larger fraction of sentences whose truth value is true, rather than neutral false. This is likely to hold for many corpora, if only to varying degrees. As such, this metric might be less meaningful on a fan-fiction corpus, but more meaningful on collection of medical or legal textbooks. We extend this argument in Appendix A.4. Finally, we chose a simple rule-based construction for the negation transform and found it to be effective. While LLMs like ChatGPT could be utilized to construct counterfactual sentences for more complicated phrases, our construction has the additional benefit of reproducibility, as it does not depend on an external model that may change over time.
Limitations:
5 Toxicity: Buckingham Guard Metric
Before LLMs are deployed in commercial settings, for example, as customer service chatbot, it is important to audit their potential to produce profanity or other toxic language. Most methods for measuring toxicity involve feeding an LLM toxic prompts and then analyzing
Different Instruction Finetuning Metheds (Negations)
RLHF
Human Curated
ShamGGP
Instruct
Selt-Instruct
Senstbilt Score
0.2
0.1
7ythe (2.08)
PATW (6:8)
(FT.)155)
HIT
LLINA
Ada
Subsage
Curie
Dexirel
Cohara DUL
Base Model
Figure 5: Normalized Sensitivity Score (negation) comparing pretrained LLMs with their instruction finetuned counterparts. It can be seen that, average, instruction finetuning increases the Sensitivity Score.

---

Published as a conference paper at COLM 2024
Frac. of Gen. Toxicc (Ours) vs Frac. Gen. Texic (Perspective API)
Different Instrction Finetuning Methods (Tadrity)
Frec. of Generaton Tosi (Ours)
6:73
Famiy Cless
6.72
PytRis
Prac of Generrntinnn Toak: (Durc))
0.5
GP1-2
GFT:
MPT
Eas
Eat
641
3.00
930 6i5 0.19 0.75 7.25
Generation Texic (Perspective
0.50
0.5
Pybla 0.8 (UB0)
CIT1 1981
11476
Figure 6: (Left) The fraction toxic generations under our metric versus Perspective API. (Right) The fraction of generated outputs that are toxic (ie., the proposed metric) for instruction finetuned models and their pretrained counterparts. The models represented along the x-axis by their model family names. The rightmost bar is WizardLM, which trained uncensored ShareGPT data, containing toxic content.
the outputs using a black-box commercial tool (egg,, the Perspective API) or an additional trained model (usually encoder). However, using model to measure the generation may be problematic. For example, although work like Fortuna et al. (2020) has tried to understand how Perspective API classifies toxic text, the API continues to change, and as it changes, our understanding of how toxic generations are being classified starts to dissipate (Pozzobon et al., 2023).
One simple and reproducible approach is to analyze toxic generation through invariance. We construct a metric that quantifies how stoic the model is to profanity, i.e,, whether the model responds to profane and aggressive comments with its own profanity or aggression. Although we study profanity, this can be extended other forms of toxicity as well, or more broadly to model behaviors, such as the tone, that the model should not mimic from user queries.
Self-Sensitivity Approach:
To build self-sensitivity evaluation, we transform input prompts by "F-bombing:" we append trigger words (e.g,, "F**k F**K") at the end of the original text. We then feed the modified prompts into the model, examining the generated output.
We construct two sensitivity scores: one that analyzes open-ended generated text, and another that analyzes the softmax probability distribution of the next predicted token. For the text generation score, we examine the output to determine whether it contains a profane word from the LDNOOBW dataset". The softmax score metric computes the next-token predictions each position in block of text and sums the probabilities of LDNOOBW word occurring. We validate our metric by comparing it with Perspective API toxicity scores over the generations that contained the appended trigger phrase. This approach enables us to understand the model's stoicism towards profanity and extends to other forms of toxicity.
5.1 Experimental Set-up
We employ our self-sensitivity approach using the Wikipedia dataset, which we split into prompts on a per-sentence level. For the generated text, we report the fraction of times generation contains "bad" word, minus the fraction of times the prompt contains bad word. For the next word prediction metric, we compute the change in the average probability of profane word between x' and x.
We compared our metric with the Perspective API (Fortuna et al., 2020) toxic scores on the prompts. We consider a generation toxic when the toxicity score is greater than or equal to 0.5. We run all models greater than 6B parameters in their FP16 configuration. The full list of models can be found in the Appendix A.6.

---

Published as a conference paper at COLM 2024
5.2 Results
The results of our toxicity metric, evaluated in text space, are presented in Figure 6 (Left). We also consider measuring the changes in the logit space, which can be found in Appendix A.6 (Figure 16). The figures clearly demonstrate a close correlation between our metric (Pearson correlation of 0.97), which measures the fraction of generated toxic word counts, and the toxicity scores obtained from the Perspective API. We conducted tests using models of different types and scales (Figure 6). Furthermore, from Figure 16, there appears to be no relation between the sensitivity of models to profane words and model size.
Instruction finetuning models with their corresponding system prompts appear to have a noticeable effect on toxicity levels compared to their pretrained counterparts, based on the results shown in Figure 6 (right). The model with the lowest toxicity score is Vicuna-v1.1 (7B), making it the least toxic across the six models examined. On the other hand, the Dolly models (i.e., finetuned Pythia models and GPT-J), which were trained on self-instruct and human-curated datasets, have a higher toxicity score on average. This suggests that certain instruction tuning datasets may be more "aligned" with human values like toxicity than others.
Effect of Instruction Finetuning:
Limitations:
Our analysis focuses on explicit profanity and may not capture nuanced forms of toxicity beyond explicit language. We rely on predefined lists of profane words, which may not encompass all variations of toxicity. The effectiveness of our metric and the model's stoicism could vary with different datasets and prompt distributions.
Word Order: Word Salad Metric
Close adherence to word order is a requirement for accurate factual responses beyond simple completions based on associative recall. Large Language Models have an incredible ability understand association but have been shown to lack the necessary representations for certain types of reasoning. One of many potential reasons for this is their occasional inability to understand word order. Yuksekgonul models trained on image captions exhibit this behavior. People have also demonstrated that BERT can often behave like a bag-of-words classifier (Juneja et al., 2023).
al. (2023) showed that multimodal
Self-Sensitivity Approach: To evaluate a model's sensitivity to word order, we utilize sentences from a given corpus and apply a transformation where two random words are swapped in each sentence, creating modified versions denoted as x'. Next, we analyze the impact of word order changes on the model's predictions by examining the predicted token softmax probability distribution from the original sentence x and its modified counterpart x'. Specifically, we examine the JSD between the two distributions to quantify the divergence in attention focus resulting from the random word swaps in x'. We use JSD instead of ppl as the swap may affect tokenization differently for different tokenizers. Thus, we measure the next token prediction via JSD. Since there are no datasets that study word order, we compare our self-sensitivity approach to HellaSwag a commonsense reasoning task as the two might be related (Zellers et al., 2019).
WORD ORDER SCORE = mediam//SSD(((())))llff(())
V(x,x) E X),
where is the last token for the input sequence for and j' is the last token for x'.
6.1 Experimental Set-up
For this experiment, we take our corpus and break it down into sentences. Then, for every sentence, we swap two random words (not tokens) to construct our x' over 5000 examples. Due to the long-tailed distribution in scores that were observed over the 5000 examples, we report the median, as described. For reference, if we had computed the mean, we would observe standard error 2e-3. We report the median JSD for each model, again including Pythia, Neo, GPT-2, and others. We run all models greater than 6B parameters in their FP16 configuration.
8

---

Published as a conference paper at COLM 2024
6.2 Results
From Figure 7 (Left), we can see that there is a positive correlation between Word Order Score and HellaSwag with a Pearson correlation 0.88. We observe that GPT-J has the highest word score closely followed by MPT and LLaMA. However, we can see that there appears to be a plateau for Word Score. Nevertheless, we can just increase the number of swaps in the sentence (Appendix A.8 (Figure 19)).
Effect Instruction Finetuning: Figure 7 (Right) shows that most instruction finetuning approaches make the model more sensitive to word order over the five model families stud- ied. Particularly, we see that only finetuning on the human-curated databri www.iccksss----oeeeeeeeeieololee seems to make the model more sensitive, irrespective of the size.
Limitations: For this Word Order Score, we make the assumption that the next token prediction when swapping two words randomly is good proxy to measure model's sensitivity to word order.
7 Discussion
7.1 Limitations
Although our SSE metrics correlate well with existing datasets, models should be chosen through rigorous testing for real-world deployment settings. Our metrics are intended to supplement existing labeled data that can be used as another measure of domain-specific data. For additional details, we refer to the limitations of each of the metrics in their individual sections.
7.2 Memorization
Machine learning evaluation benchmarks for studying statistical generalization almost always assume idealized train and test set separation. However, in reality, some amount of overlap often exists in modern web-scale pretraining corpora. As result, there have been various efforts to measure and address the impact of these overlaps on the training and evaluation of large models (Brown et al., 2020; Gao al., 2021). Additionally, Kandpal et al. (2022) showed that language model's d ability to answer fact-based question relates to how many documents associated with that question were seen during pretraining. In the context of sensitivity scores, this collection of results in the literature suggests that it is hard to make strong statements about whether training-timg time exposure to certain documents or token sequences would confound the trends observed in proposed sensitivity metrics. We leave a detailed analysis of the interactions between memorization behaviors based on training data and our sensitivity metrics for future research. Nevertheless, potential advantage of self-sensitivity scores and other automatic evaluations is that we might be able to circumvent the potential effects of memorization by evaluating sensitivities novel text, i.., the latest news articles, as no labeling and additional curation of data sources is required.
Word Order Score vs HellaSwag
Different Instruction Finetuning Mothods (Word Order)
Seff-intruatt
SharaCoT
Curatod
0.050
0.08
0.003
Word
Word
0.06
Mode family we
0.04
0.02
0.05
0.03
0.50
0.30
0.40
0.45
0.50
0.32
Pydre
Pythu
CIT)) (18) Base Modal
LLAMA (78)
HellaSwag
Figure 7: (Left) Word Order Score vs LRS Score acToss various model sizes and families. (Right) Word Order Score of instruction finetuned models and their pretrained counterparts.

---

Published as a conference paper at COLM 2024
Next Taken Entropy vs Senkence Entropy
Model
Sth (100
2 decee
Next Token Entropy
May
OFT greasmn
4-3
GIT-Meo 221
3.4
OF (FT12-99
2F12 DL
3.8
Pytfo ( A11
Pyhto
3.0
HIT 120
2.75
3.00
3.25
350
3.75
4.03
4.5
45%
435
Sentence Entropy
Figure 8: Plot showing the next token prediction Shannon entropy (y-axis) and mean token Shannon entropy (x-axis) over sentences on Wikipedia. We find that LLaMA (7B) has the lowest entropy over the next token and mean token over sentence.
7.3 Entropy
The entropy of model's output distribution can impact many aspects of text generation. A lower entropy may require more aggressive sampling strategy for text generation to achieve a diverse set of generations from the model, or might indicate a miscalibration of the output distribution. Similarly, the model's entropy can affect sensitivity scores. If the entropy of the model low, then the sensitivity may naturally be lower as well. The exact impact of the model's entropy on these sensitivity scores and how to appropriately incorporate it into invariances/ / sensitivity scores should be explored in future work. Figure shows the Shannon Entropy of the Next Token Prediction and Sentence Entropy (the mean token entropy over a sentence of the model). We use the Wikipedia (our corpus) sentences E Ex(x) (x)).. From Figure
to calculate the Shannon Entropy, defined as H(x) we see that LLaMA has the lowest entropy on both the next token and mean token over sentence, with large models having lower entropy than smaller models on average. This
may partially explain why the sensitivity scores for LLaMA are lower.
8 Conclusion
In this paper, we introduce procedure for self-sensitivity evaluation by analyzing invari- ances for Large Language Models. The key advantage of self-sensitivity evaluation is that it has the potential to remove the need to laboriously label new data, leading to more efficient forms of evaluation in real deployment settings. We showcase several case studies, where empirically validate this approach to reliably track existing supervised metrics. Additionally, there are number of future questions to consider when measuring model's sensitivity that we have not fully explored yet- like entropy and memorization. Nevertheless, these self-sensitivity evaluation approaches have the potential measure properties beyond what is currently capable of the traditional dataset approach - like sensitivity to word order. We believe the future of LLM evaluation lies in automatic evaluation, as it can be conducted on large scale. We hope that this is only a starting point for self-sensitivity or more largely self-supervised metrics in the future that can lead deeper understanding of how LLMs behave and complement classical supervised benchmarks.
Acknowledgements
This work was made possible by the ONR MURI program, the Office of Naval Research (N000142112557), and the AFOSR MURI program. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IIS-2122822, and by the NSF TRAILS Institute (2229885).
*Vocabulary size does play an additional role in the entropy of model. For example, in completely uniform distribution, the Shannon Entropy of a model with smaller vocabulary size will be smaller than another model with larger vocabulary size.
10

---

Published as a conference paper at COLM 2024
References
Anthropic. Introducing 100k context windows, May 2023a. URL U.L.ttps://www.aaabrrpp.
Anthropic. Introducing claude, March 2023b. URL htt.:tpp/www.aaaarropicccoo/////
introducing-cl.n claudde.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. arXiv preprint ar ari:::304400137. 2023.
Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. The values encoded in machine learning research. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 173-184, 2022.
Samuel R Bowman and George E Dahl. What will it take to fix benchmarking natural language understanding? arv preprint ar Xio:2104.002145 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ar ar preprint ar *xir:2307.03109 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing, Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //msys..ssss... herg//log//2200 -0-----0000----i-- vicuna.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, pp. 862-872, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.344522 URL https://doi org/10. 10. 1145/3442188 3445924.
Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, al. Nl-augmenter: A framework for task-sensitive natural language augmentation. arv preprint Xiv:2112..07211, 2021.
Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. arXiv preprint ar Xrx:::230 12420, 2023.
Kawin Ethayarajh and Dan Jurafsky. Utility is in the eye of the user: A critique of NLP leaderboards, arxiv preprint ar Xiw::2009.13888, 2020.
Paula Fortuna, Juan Soler, and Leo Wanner.
Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets. In Proceedings of the Twelfth Language Resources and Evaluation Conference, Pp. 6796, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95544 10-15556--4-4. URL https://aclanthol1 org 2erg/2020.oom frrec-1833.
11

---

Published as a conference paper at COLM 2024
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https:
//doi.org//0. (rooo..or//0..55//zemddd 5371628.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- ToxicityPrompts: Evaluating Neural Toxic Degeneration Language Models, September 11462. arXiv:2009.11463 [cs].
2020. URL htttp://axxiv.rrrg
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 1601-1611, 2017.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. In The Eleventh International Conference on Learning Representations, 2023. URL https::/
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. arXiv preprint arXim:2211.084111 2022.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. ar preprint arXiv:2104.14337, 2021.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. ar Xrv preprint ar arXi:2221.009100 2022.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, pp. 74-81, 2004.
Microsoft. Guidance. Microsoft, June 2023. URL httpps//ggithaa..cmm///rrrressse guidance.
Simon Mille, Kaustubh D. Dhole, Saad Mahamood, Laura P'ree--ellrachini, Varun Gangal, Mihir Kale, Emiel van Miltenburg, and Sebastian Gehrmann. Automatic construction of evaluation suites for natural language generation datasets. ArXiv, abs abs/2116.09069, 2021.
Joe O'Connor and Jacob Andreas. What context features can transformer language models use? arviv preprint arXiv:2106.08367, 2021.
OpenAL. Gpt-4 technical report. ArXiv,
2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, 311-111118 Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115 011111//177383.1073133 URL wrgpp// 1040.
https:/
12

---

Published as a conference paper at COLM 2024
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint Xrxv:2304.002777, 2023.
Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. On the challenges of using black-box apis for toxicity evaluation in research. arXiv preprint Xfff:23344.13977 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Πla Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4902 4912, Online, July 2020. URL
Association for Computational Linguistics. doi: 10.18653 1111666//112222aacmm/// httts://accaanthol.g. org/2020 aac-maaaiii 442.
Alexis Ross, Tongshuang Sherry Wu, Hao Peng, Matthew E. Peters, and Matt Gardner. Tailor: Generating and perturbing text with semantic controls. In Annual Meeting of the Association for Computational Linguistics, 2021.
Jessica Rumbelow and Mwatkins, Solidgoldmagikarp (plus, prompt genera- tion), 2023. URL https://wwx lesswrong. pttpssssaaeeeaaooorrrlloooo
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Xrviv preprint ar Xix::2200 2K4.:2206.04615, 2022.
Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. On the safety of conversational models: Taxon- omy, dataset, and benchmark. In Findings of the Association for Computational Linguistics: ACL 2022, pp. -9233,,,,, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1. (118653/112022..mmil 22t.2222..ininns-al..... 2022.findinge. findingnnnncllln.. 308.
URL https: /aclaaatolll org/
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations, 2020.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality, 2022.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXiv preprint ar Xii:22006005087 2023.
Tongshuang Sherry Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S. Weld. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Annual Meeting of the Association for Computational Linguistics, 2021.
Guanqun Yang, Mirazul Haque, Qiaochu Song, Wei Yang, and Xueqing Liu. Testaug: A framework for augmenting capability-based nlp tests. In International Conference on Computational Linguistics, 2022.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreet wttpeiiwwwweeeeeeeeeeeeeeeeeeeee feet/for /or//oormm///// forum?idd-KLUuxieeee K-KLUUxxhBua...
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the 2019.
Association for Computational Linguistics, pp.
13

---

Published as a conference paper at COLM 2024
Zeyuan Allen Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. arxiv:2339..1316(css) September 2023, doi: 10.48550/ 10.48550/arXiv.2000 arXiv:2309 14316. URL http://arxiv..t/ wttpv//.rggabb///////ee/ awtp///s//.///// 14316.
14

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Extended Related Work
Knowledge Probing via Negation: The MMLU benchmark (Hendrycks et al., 2021) is widely used to assess the knowledge base of language models, evaluating their performance on task-specific micro datasets. In production, the GPT-4 technical report (OpenAI, 2023) advertises the model's capabilitiis across various knowledge categories, yet the evaluation suite used in the report is not publicly available. Diao et al. (2023) proposed using negative log-likelihood to probe a model's knowledge for instruction models. Furthermore, Wu et al. (2021) introduces a general-purposs counterfactual generator, Polyjuice, that allows for control over perturbation types and locations and is trained by finetuning GPT-2 on multiple labeled datasets of paired sentences. Our evaluation method allows us to assess the model's understanding and knowledge representation by examining its ability to handle negations without the need for in-domain labeled datasets or model finetuning.
Toxicity: DealoxxicityProme
is the most prominent benchmark for toxicity in LLMs (Gehman et al., 2020). This method relies on the Perspective API to score the model's gen- eration based on series of prompts. This API is also used as the toxicity metric for HELM. However, with the proprietary API constantly changing, comparing evaluations across time is difficult (Pozzobon et al., 2023). Another common benchmark is BOLD (Dhamala et al., 2021). BOLD trains another model to classify toxic generations. This approach of utilizing another model to measure toxicity is common (Sun et al., 2022). Our approach differs from these methods as we do not build a dataset nor rely on auxiliary models to classify the generations.
Word Order: While previous efforts have made significant contributions to testing the compositional and word order understanding of language models (O'Connor & Andreas, 2021; Thrush et al., 2022), these efforts predominantly rely on small sets of hand-crafted examples. Moreover, these tests often encompass wide range of knowledge types, making it challenging to isolate and evaluate the specific role of word order knowledge. Our work aims to investigate the word order sensitivity of LLMs from the lens of invariance in data-agnostic manner.
Long-Range Dependency: As conversational AI models become more prevalent (Ouyang et al., 2022; Anthropic, 2023b), the importance of accommodating large context lengths has become evident. Recent endeavors have focused on developing chat models with extensive context capabilities, such 32k and 100k (OpenAI, 2023; Anthropic, 2023a), utilizing techniques like memory-efficient attention (Dao et al., 2022). However, it is equally crucial to gauge how far back into the context the model truly operates and can refer to. LAMBADA (Paperno et al., 2016), addresses this by assessing language models' comprehension of broad contexts. In contrast, our self-sensitivity approach creates texts through closed-form transformations that evaluate language models' grasp of long-range sensitivity.
Tokenization Sensitivity: HELM approaches this problem by inducing spaces, misspellings, etc., over the datasets in question to determine if these slight changes can affect changes when evaluating over established datasets (Liang et al., 2022). Additionally, Rumbelow & Mwatkins (2023) found set of anomalous tokens that result in previously undocumented failure mode for GPT-2 and GPT-3 models. Inspired by these works, we designed a test to see how the same text tokenized differently affects model behavior without changing the underlying text.
A.2 Context (Long-Range) Sensitivity: Back to the Future Metric
As LLM context window sizes have increased in recent models, is important to understand how changes in the previous context can affect the representations and generation across long ranges. Datasets like Long-Range Arena (Tay et al., 2020) offer a very broad set of tasks, focusing on context lengths ranging from 1K to, 16k and aim to evaluate architectural choices. There are other datasets like LAMBADA that focus on the capability to successfully
hattpe///prppeciivapp.com
15

---

On average, we see that instruction-fiietunee models are more sensitive to changes in context than their pretrained counterparts, suggesting that they may be sensitive to long-range changes (beyond locally attending to a sentence). Moreover, we find this gain appears independent of base model size. Both the smaller and larger Pythia base models have a similar sensitivity, and finetuning on Dolly-V2 (" ("human nuuman-curated curated""""" in Published as a conference paper at COLM 2024
predict the conclusion to paragraph (Paperno et al., 2016). The dataset is designed such that the prediction of the word is clear given the full context, but it impossible to predict given just the last sentence. This measures an LLM's ability to comprehend text beyond locally attending to sentence.
We can utilize self-sensitivity evaluation to understand how the model's predictions change when a prior sentence or multiple sentences from a passage are altered. We conduct this test by taking three sentences from stream of data in order and replacing the first two sentences with two random sentences from the corpus. For example, the original passage had three sentences, {S,, S2, S where S3 is the first sentence of the input passage, then the altered passage would be {S Sy, S1,, where Sxx, Sly are random sentences from another passage in the corpus. more concrete example can be found in Appendix A.7 (Figure 18). We then look at the probability distribution at each position of S1 for both and x', and compare them using the Jensen-Shannnn Shannon divergence. This is to determine how the representations of the last sentence change different context is presented.
Self-Sensitivity Approach:
The Jensen-Shannon divergence (JSD) is symmetric variation of KL-divergence, defined as:
[SD(PIIQ) = _ (L(M) + 22 (LQIIMM,
(2)
2
1
where M = (P+
(3)
2
For our invariance /eeaancce//essiiiityyy score, we take the mean of JSD over the last sentence, averag- ing over all samples. Concretely,
1 " E Σ (SD(()))LA(()))))
LRS SCORE
n
where m represents the sentence length and xff is the ith sample in the set at token position j in the last sentence.
A.2.1 Experimental Set-up
For this sensitivity test, we compare our method to LAMBADA using Fleuther Language Model Evaluation Harness (Gao et al., 2021). It is worth noting that the tests here are different. The LAMBADA dataset measures long-range dependency on fiction and its ability to comprehend the previous passage. On the other hand, we analyze the invariance of the probability distributions over the last sentence when the passage has been altered. To calculate our metric, we use the same corpus examples with the standard error 2e-3 of the mean value record. We report the JSD for range of models including Pythia, Neo, GPT-2, and others. We run all models greater than 6B parameters in their FP16 configuration.
Languuag
the other tests and calculate over 1000
A.2.2 Results
From Figure 9 (Left), we see that as our LRS Score increases, the model performs better on LAMBADA. Furthermore, bigger models generally tend be more sensitive changes in the context. We see that Pythia and GPT-J are more sensitive to changes in the context compared to MPT and L.LMA. Whereas, smaller models like Pythia-70M and GPT-2 small produce lower LRS Score. We see Pearson correlation with Lambada (OpenAl) of 0.91.
Effect of Instruction Tuning;
Figure 9) leads to similar gain in sensitivity.
16

---

Published as a conference paper at COLM 2024
LRS Score Lambada (OpenAl)
Different Instruction Finetuning Methods (LRS)
0.40
0.23
Masic kunkk
Tydhe
None
Self-instruct
ShareGPT
0.35
Human Curated
Nix
0.2
LRS Score
DFT2
0.25
0.23
0.20
0.19
URS
0.15
0.10
0.05
0.00
07
PytMe (2.080
Pytrie (5.98)
[STS) (M1
MIT
LL+MA (77)
LAMBADA (OpenAl)
Base Model
Figure 9: Left LRS Score vs LAMBADA (OpenAl) across various model sizes and families. Right LRS Score of instruction finetuned models and their pretrained counterparts.
Limitations: Although we are analyzing long-range sensitivity in token probability space, for transformers in particular, analyzing attention probabilities may be more effective. However, to make the metric applicable to generic architectures, including RNNs, LSTMs, efficient attention variants, etc., we believe that the token probability space is more appro- priate.
A.3 Tokenization Sensitivity: Broken Token Metric
Text pre-processing is rarely perfect. Raw text often contains extra spaces, weird formatting, and other quirks that affect how the tokenization of the text occurs. HELM explored some of these phenomena (Liang et al., 2022). Others, such as Rumbelow & Mwatkins (2023), found anomalous tokens that represent failure modes in GPT-2 and GPT-3 models, showing that our understanding of how different tokenization impacts the model behavior is still limited.
To quantify this phenomenon, we randomly chop strings of raw input text at regular intervals x, and then we tokenize each of the chopped strings inde- pendently. This way, we mimic "broken" tokenization, that might occur the pretraining corpus due to document breaks and misspellings. A broken tokenization can also occur during model generation when incomplete user input is provided (Microsoft, 2023). After tokenizing each chopped string separately, we concatenate these tokenizations back together. Note that the original content is unchanged the alternative tokenization still decodes to the same raw input text. We then compare the concatenation of chopped tokenization to the original text over the next token prediction using JSD, similar our Word Order Metric.
Self-Sensitivity Approach:
TOKENIZATION SENSITIVITY SCORE
(4)
[ISD((())))(1
(5)
A.3.1 Experimental Set-up
For this experiment, we take our corpus and break it down into sentences. Then, for every sentence, we apply our procedure (described above) construct x' over 1000 examples. We report the mean JSD for each different model like Pythia, Neo, GPT-2, and others, where the standard error is about 3 for all models. We run all models greater than 6B parameters in their FP16 configuration. Here, we specifically explore a split stride of 5, splitting every 5th character.
A.3.2 Results
From Figure 21, we see that MPT and LLaMA are the least sensitive (lower is better) to changes in token inputs. More broadly, we observe negative trend with training FLOPS
17

---

Published as a conference paper at COLM 2024
Tokenization Sensitivity Score FLOPS
Different
Finetuning Methods
Sosel Fumbl
0.25
None
Self Instruct
ShareGPT
Tythle OPT
0.20
Tokenization Sensitiiit
Tolenization Score
0.25
der
0.15
0.30
2.25
0.20
0.15
0.95
0.00
19c
10M
Pybhia (2.88) Pychi
GPTJ 158) Base Model
LLaNA
Approx. FLOPS
Figure 10: (Left) Tokenization Sensitivity Score with split stride of five versus Approx. FLOPS lower is better. Note that the OPT models have seen the fewest tokens during training, c.f. Figure 21. (Right) Impact of different instruction-tuned methods.
(i.e increasing the FLOPs decreases the sensitivity to tokenization changes). We suspect that as the amount of training increases, alternative tokenizations are more likely to be observed, and invariance to these abnormal tokenizations increases. This is supported by measurements on the OPT models, which are strong outliers the trend observed above. Each of these models was trained only 180B tokens, less than fifth of the tokens seen by MPT and LLaMA (1 Trillion) and about half of what GPTT2, GITT-Neo, and Pythia have seen. We include Figure 21 for variant of Figure 10 in terms of tokens observed during training in Appendix A.9.
Effect of Instruction Finetuming: Figure 10 (Right) shows the impact of different instruction finetuning methods. In contrast to previously observed metrics, there seems to be no reliable trend in tokenization robustness after instruction finetuning. Furthermore, even when only model size differs (Dolly-V2s) the instruction finetuned dataset can have a different impact on this metric. It is worth noting that the Dolly-V2s were only trained on 15k instructions.
Limitations We test a limited type character splits of tokenization error, particularly the same text just being processed differently by the tokenizer. There are additional tokenization errors to consider well, based on minor edits of the raw input text (i.e explicit word splits, extra spaces, unusual punctuation, etc), that could also be considered. Additionally, examined the change in the next token probabilities, we believe it is a good proxy to measure this phenomenon.
A.4 Do All Transformations Need to Be Informative?
The following is why we believe that transformations need only be informative on average. We will use knowledge via negations as an example here.
Assume that you have some corpus, C, which contains set of facts X1 and set of non- facts, X2. Let a be the percentage of facts, F be the model, T' be the transformation, and be the measurement procedure.
Now, let M1 1 Ex, (F(x),, F(T(x)) be the measurement received from the a set of facts, where X1| is the cardinality of X1. This the informative measure.
Similarly, let M2 X21 Ex, (FF(x) FxT(x)) be the measurement received from the a set of non-facts, where X2 is the cardinality of X2. This is the non-informative measure.
the transformation is non-informative over some set (i.e. X2), then either two things will happen: (1) the transformation will have some bias which will affect both the measure- ments taken from X, and X2 or (2) it will have no effect on measurement (i.e it would be zero). To confirm that X, is informative, manually examination of the sentences and their
18

---

Published as a conference paper at COLM 2024
transformations is necessary. We did this for 100 examples, and all of them were correctly formed.
Case (1) may occur, which is why we use normalization term that applies "not" to sentences in a neutral corpus, In our case, we use fiction books as the neutral corpus, as few sentences in this corpus are statements of fact. Let this normalization term be M. Now, we subtract M from M1 and M2. Ideally, this would set the M2 - M measurement to zero, and M1 should be non-zero. Now, can look at both (1) and (2) together M2 both zero.
Now, we can look at how alpha can affect the measurement, which central to our claim that "these transformations need only be effective on average to generate informative signal."
Let the final measurement be M a/C. (M, M)+ CM M) Notice that the second term
C
goes to zero now. This leaves us with M = a (M M). Now, only when a (Mi M) is smaller than standard error is there a concern. Notice that here the relative order will not change given the same X1 no matter what the mixture of facts (informative) and non-facts (non-informative), measured by a, is here.
We acknowledge that the above argument assumes the bias in the neutral corpus is of the same magnitude as the bias in X2 (the non-fact sentences of the original corpus). While this is not gauranteed to hold, it appears to be a reasonable assumption for the cases we consider, as our metric correlates well with ground truth metrics like TriviaQA and MMLU (clinical knowledge).
A.5 Extended Knowledge Probing via Negations
Example: Figure 11 shows an example of the original x and the transformed x' for the Knowledge Probing via Negations experiments.
Original (x): April the fourth month of the year in the Julian and Gregorian calendars and comes between March and May.
Perturbed (x'): April is not the fourth month of the year in the Julian and Gregorian calendars and comes between March and May.
Figure 11: Knowledge probing via negations example over topic sentences in wikipedia. (Top) is the original, x, from wikipedia. (Bottom) is the transformed, x', where we add "not" according to the rules described in the main paper.
Adding Negations in TriviaQA To understand whether adding negations and measuring the change in log perplexity is reasonable assessment of probing the knowledge in an LLM, we added negations to questions following the same rule described in the main paper. We then recorded the change in perplexity for each of the models given the question-answer pair. This was to understand how different models may understand negations. Figure 12 (Left) shows that adding negation in the question and observing the change in perplexity can give us an indication of performance on TriviaQA.
Medical Terms
For the medical terms, we use open source dataset from huggingface: We suspect that a text book of definitions would also suffice.
TriviaQA Accuracy We calculate the accuracy for TriviaQA for the teeiiteredwwbbb split by simply counting a correct answer from the model one of the given answers was contained in the output string. Additionally, since found that the answer list sometimes had the answer entity in the question, we excluded these answers when calculating accuracy. We use the template "Question: [input question] Annwwerr."
19

---

Published as a conference paper at COLM 2024
Sensitivity Score (Vlkipedia) vs Answer in Output After Negation
Senditviity Score on TrVvSAA TIVISQA Acc.
Hodd Fambl
Output Ather Nagotton
6.99
Osenk intruct
OoanAl
X
Cohara Inttuct
Sensiliity Score on
2:3-
Uarr
Made Famm
Cllven Intrue
Dely Dely
0.29
Ansgwer
0.8
6:40
3/6
Sensitivity (Wikipedis)
Figure 12: (Left) The change in perplexity in the question-answer pair when a negation applied to the question versus TriviaQA Acc. There appears be square-root relationship between the Sensitivity Score on TriviaQA versus TriviaQA Acc. (Right) The percentage of times when the correct answer was contained in the solution even when applying the negation versus Sensitivity Score (Wikipedia) for few instruction models. We that text-ada-001 changes its answer often, whereas the Cohere model does not.
Sensitivity Score vs TriviaOA Acc.
Sensitivity Score MMLU Clinicl Knownledge
Model Famiy
Osend Inttuct
1.82
x
Cohere instrct
LUMMA
Miano
Senstttiit Score
0.3
Mode Fanily
Galactica
K0- Sensitiity
MPT
Pydhi
OPTicee
(PT1
Nov
BrviaQA
MMLU Clinical Knownledge
Figure 13: Left shows the SENSTIVITY SCORE with TriviaQA and Right shows the SENSTIVITY SCORE with MMLU (clincial knowledge).
Percent Drops va BlivaGAA Act.
Moad Rimily
0.22
Openki intruct
Openki
Conure
Draps
Concre Iusrout
6x
GFT2
0.00
Figure 14: Percentage of samples where perplexity drops versus accuracy on TriviaQA. We observe reliable negative correlation.
20

---

Published as a conference paper at COLM 2024
Models From Huggingface: gpt2, gpt2-large, gpt2-medium, gpt2-xl,
EleutherAl/gpi-noo-11 1B EloutherA//gpt-noo-2. EloutherAT/gpt-j-6B.
EleutherrA//yythaal1.... EleutherA//pytthia-2.1 EleutherAl/ppythaa--.bbb
mosaicml/mmtt-b,
nosaical//mpt---hhhstuutt databriccss/oolly-2277
databricks/bblll-wi-.bb, databrickes/bolly-22-9..
Other Models: LLaMA-base-7B. Vicuna-7B
OpenAl API Models: ada, babbage, curie, davinci, text-ada-001,
dext-babbage-001. text-curie-001, text-davinci
Cohere API Models: xlarge, command -laaeed--large-bea
Table 1: Example outputs of text-ada-001, text-davinci-OUs and Cohere command. These examples are selected where text-ada-001 would produce sensible answer to both the original question and the negated question. The Cohere model is sometimes entirely insensitive negations, compared the OpenAl models, although even text-davinci can fail this task. This trend was observed over several generations, from which we show two qualitative examples here.
Model
Original
Transformed
Question
A sterlet what type of creature?
A sterlet not what type of creature?
A sterlet is a creature that has spiny body and long, sharp tongue.
text-ada-001
A sterlet not creature.
text-davinci-001
A sterlet type of sturgeon.
A sterlet type of sturgeon.
Cohere command
Fish
Fish
What is the only natural food that never goes bad?
What is not the only natural food that never goes bad?
Question
There is no one natural food that never goes bad. There are, however,
The only natural food that never goes bad is sugar.
text-ada-001
some foods that more likely to do
These include: milk, yogurt, ice cream, and cake.
There is no single natural food that never goes bad.
text-davinci-0U3
Honey. Honey never goes bad.
Cohere command
Honey goes bad.
A.6 Extended Toxicity
Example:
Figure 15 shows an example of the original x and the transformed x' for the toxicity experiments.
Original (x): Lionel Andrés Messi is Argentine professional footballer who plays as a forward for Ligue club Paris Saint-Germain and captains the Argentina national team.
Transformed (x'): Lionel Andrés Messi is Argentine professional
footballer who plays as a forward for Ligue 1 club Paris Saint-Germain and captains the Argentina national team. pr+* paw*
Figure 15:
Toxicity test with one sentence. (Top) is the original, x, from wikipedia. (Bottom) transformed, x', where we add the trigger phrase to the original input.
Increasing the number of F-bombs: From Figure 17, we see, that on average, increasing the number of F-bombs increases the toxicity of the model. However, here two F-bombs induce more toxic behavior than three F-bombs, suggesting that on average, the two bombs might be trigger for toxic generation.
21

---

Published as a conference paper at COLM 2024
Bad Werds Logits (Ours) vs (Perspertive
0.80
in Bad Word Logits (Ours)
0.75
Family Class
0.70
Pythia
0.65
GPT-J
MPT
0.60
Neo
0.55
0.55 0.00
0.65
Frac. Generation Toxic (Perspective API)
Figure 16: The change in the next-token probability of profane word (token) versus the fraction of toxic generations according to Perspective API.
Texicity Ablation - Increasing F-Bombs
Model
GPT-2 Ismall)
Pythia
GPT-2 (medium)
[large)
Pythia (1.48)
GPT:) (68)
(XL)
LLaMA (7B)
GPT-Neo (2.22 (2.38)
Pythia (2.08) 28:
(2.78)
Number of F-Bambs Appended
Figure 17: As we increase the number of F-bombs, the toxicity of the generation increases except when two F-bombs are present, which is notable outlier. This suggests that to most models this is toxic trigger. We measure toxicity the generated text by observing whether term from LDNOOBW contained the generation. From this figure, we GPT--Ne (2.7B) the most toxic according to our metric.
Models From Huggingface: gpt2, gpt2-large, gpt2-medium, gpt2-x1,
EloutherAt/ppt-j-bbb, FloutherA//go-nno--..33 3B, BleutherAI/pp--neo
EleutherA//pythia-1.. EleutherAI/pythia-2.. 8b, EleutherA//yythia-..0 mosaicml/npt-7b,
mosaicmll/ppt----nnnnrrtt
databricks/dolly--11666 databbricks/8ll1yy22-- databrickss/bblly-v211
Other Models: LLaMA- Vicuna-7B, WizardLM-7B (uncensored)
A.7 Extended Context (Long-Range) Sensitivity
Figure 18 shows an example of the original and the transformed x' for the LRS experiments.
Example:
Increasing the Amount of Context: From Figure 19, we see that increasing the context (or the number of sentences swapped) increases the sensitivity. For the 7B parameter range, see that Pythia (6.9B) is the most sensitive.
Models From Huggingface: gpt2, gpt2-large, mpt2-medium, gpt2-xl, facebook/opt 1.3b, 3b,
facebook/opt- o. 7b,
eehartford
22

---

Published as a conference paper at COLM 2024
Original (x): Lyrically, the song begins with the absence of her man, but then, in the chorus, transitions into warning not to fall in love with material things. The second track, "Lágrimas Cálidas" ("Warm Tears"), is vallenato-stylizer pop ballad, expressing her suffering due to being abandoned by her lover." "Te Arrepentiras" ("You'll Regret"), is about woman who surrendered completely to man who did not appreciate her.
Transformed (x'): Ireland has won more medals in boxing than in any other Olympic sport. Boxing is governed by the Irish Amateur Boxing Association. "Te Arrepentiras" ("You'll w'lll Regret"), is about woman who surrendered completely to a man who did not appreciate her.
Figure 18: Long-Range Sensitivity test with four sentences. (Top) is the original, from wikipedia. (Bottom) is the transformed, x', where the first two sentences replaced with random two sentences from wikipedia.
Word Order Ablation Number ber of Word Swass
Sontences Swasped
Mn Lst
Pylic (1.55
Sager
CMC coeel Tylu (2B4)
3.
0.51
Mumbbrr of Words Guapped
Nambor Serteness
Figure 19: (Left) We plot JSD the next token prediction against the number of swaps for the token. (Right) Increasing the context length (the number swapped sentences) increases, the change the probability distribution over the last sentence.
ElettherAl/gppt-n--1.BB.. EloutherAA//gpt-ne---.77 7B, EleutherAT/gpt-j-Ob,
EleutheraAl/pythh---00,, EleutherrA//pydhha-11600, EleutherA//pryhia-4100
Eleuthersl//pyyiia-l,, EleatherAA/ppythi---.4, EleutherA(/pyycaaa-..8b.
EleutheraA/ppythia-...b. mosaicml/mpt-7b,
mossiccll/mpcc7--hhstruuu databrickks/bolly-v1-666
databrickss/ollyyv2--bb.
databriicks/bblly-vv27bb databrickss/bolly-22277
Other Models: LLaMA-base-7B, Vicuna-7B
A.8 Extended Word Order Sensitivity
Example: Figure 20 shows an example of the original x and the transformed x' for the word order experiments.
Original (x): Media. Vision would return the franchise with the development of Valkyria: Azure Revolution for the PlayStation 4.
Transformed (x'): Media. Vision would return PlayStation the franchise with the development of Valkyria Azure Revolution for the to 4.
Figure 20: Word Order Sensitivity test sentence. (Top) the original, x, from wikipedia. (Bottom) is the transformed, x', where two words are randomly flipped. This a 1-Swap.
Different Number of Swaps: Figure 19 shows the median JSD on the next token as we increase the swaps. Here, increasing the number of swaps increases the sensitivity.
23

---

Published as a conference paper at COLM 2024
increasing Split
Tokenization Sensitiityy Score vs Tökens Seen
0.6
Mode
Pokennzstoon. Sonetvve Score
0.45
0.5
Pythi (10:00)
Tyshe 13.30)
0.40
Pyth
Gmo
6.
607) (14)
GP02
Get2 (arge)
6.3
Crtbiia 12.0
Pythe (1440
0.25
Tylha 12:50
62-
0.20
6972 lovel
0.1
0.35
11.30
0.10
0.0
2.5
5.0
Spilt Stride Iby Character
7.5
10.0
12.5
15.0
17.5
20.0
Total Tokens Seen During Training
Figure 21: (Left) Increasing the split stride decreases sensitivity, see that the OPT family cannot handle this type of transformation. Additionally, we see LLaMA and MPT are good at handling these types of tokenization changes. Lower is better. (Right) Increasing the total number of tokens seen during training decreases the sensitivity score. We that the OPT family is the most sensitive to this LLaMA
type of transformation, they have seen the least number of tokens. Additionally, and MPT are good at handling these types of tokenization changes as they have seen more tokens. Lower better.
Table 2: Example sentence of the transformation with split stride of 10. (Left) shows the original unaltered sentence. (Right) shows the transformed sentence after splitting every 10th character. The underlined dashes are where the sentence split.
Original (x)
Transformed (x')
Media. Vision would return the franchise with the development of Valkyria: Azure Revolution for the PlayStation.
Media. Visi-on would r-eturn to the franchi-se with th-e developm-ent of Val-kyria: Azu-re Revolut-ion for th-e PlayStat-ion 4.
Models From Huggingface: gpt2, gpt2-large, gpt2-medium, gpt2-x1,
ElouthersA//gpt-nno---S5 EloutherA//gpp--ee--... SeottherA//gpt-ne-22.777 EleutherAllggpt-j-Bbb EleutherAl/pythia-L.oo
Eleuther///ptthia-2.00..
EleutherAA//gythia--..bb mosaicml/apt-7b,,
mosaicall/mptt---inntrreet databrickss/bolly-22111 dstabricks/bolly--11.66
databricks//bllyy--22
Other Models: LLaMA-base-7B, Vicuna-7B
A.9 Extended Tokenization Sensitivity
Example: Figure 20 shows an example of the original and the transformed x' for the tokenization experiments.
Increasing Split Stride: Figure 10 shows the median JSD on the next token as we increase the split stride. Here, we see that LLaMA and MPT are much less sensitive (better at handling tokenization changes) regarding the change in the probability distribution over the next token as we increase the split stride. Figure 10 shows the number of tokens seen versus the tokenization sensitivity score. Here, we see that there is a negative correlation.
Models From Huggingface: gpt2, gpt2-large, gpt2-medium, gpt2-x1, faceboook/ott-1.3.. facebook/opt-2.77 ELeutherA//gpt-ne--122 Eleutheratl/gtinno---33 EleutherAt/gpt-nne-22 7B, EleutherAl/gpt-j-66
EleutherrAI/yythha-166.. EleutherA///pythia--4
EleutherA[/pythha-bb.
EleutheraAA/pythia--.44b EleutherA//pyythiaa2.. 8b, Eleuttaaaaar/
mesticm//nnggrrr..nnaaaa///ppppp sraas//sscccttcacccccooooooaaaa- databrickss/oolyyv116bb0
databricks//bolly-2222 databricks/6bllly-2222 7
Other Models: LLaMA-base-7B, Vicuna-7B
24

---

Published as a conference paper at COLM 2024
A.10 Additional Experiment Details
For all these experiments, we use NVIDIA RTX A4000 GPUs, finding that evaluating most models is quite inexpensive over 1000 examples, with compute requirements of less than 30min per model for most tests. Additionally, for sentence and word parsing/ tokenization, we use the nltk package.
25


Reviews:
[{"review_id": "2rQuUnLRMa", "reviewer": "Reviewer_bvvQ", "summary": "This work proposed a new evaluation method based on comparison between a text its negations. In detail, authors proposed to make handcrafted negation perturbation on original text and compute various difference between this counterpart and its originals, like perplexity, token softmax probability distribution or text. Authors proposed this new test method can provide more accurate metrics for various domain datasets. \n\nAuthors motivation contains these points:\n1. New general automatic evaluation for LLM text generation ability is an interesting topic and authors do provide a lot of interesting results. \n2. Authors stressed the importance of data contamination and intended to propose a corresponding solution. \n3. They hope to have this new automatic evaluation metric to be highly aligned with human intention.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, bvvQ.\n\n> Convenience of SSE.\n\nWe agree self-sensitivity metric necessitates human design, as with all evaluations. Every evaluation process requires significant human effort to develop appropriate input and label pairs. However, our self-sensitivity framework offers an *efficient* approach compared to labeled evaluations as there is no need to laboriously collect data. While domain-specific adaptations are necessary, they are less time-consuming and complex than creating entirely new benchmarks for each specific problem. \n    \n> SSE and human evaluation \n\nHuman evaluation is the gold standard for many aspects of model performance. However, due to the reproducibility issues and laborious nature of human evaluation, labeled data is still used ubiquitously. Our primary aim was to show the advantages of our SSE relative to existing automatic evaluation metrics. This approach aims to overcome several limitations associated with labeled data, such as data contamination, laborious collection, and scalability issues.\n\n> data-centric\n\nOur evaluation is dataset-agnostic, which should be understood as its flexibility and applicability to various types of data, a \"one-size-fits-many\" approach rather than \"one-size-fits-all.\" SSE still necessitates tailoring to the specific characteristics of each dataset. For instance, in knowledge, the focus is on ensuring that factual content exists, irrespective of the proportion of non-factual statements, as long as the standard error remains below the sensitivity score (see Appendix A.5). A dataset with no factual sentences such as fictional books would not be an ideal candidate for the knowledge SSE metric.\n\n>  Data Contamination\n\nSSEs are meant to be data-agnostic in that a new dataset from live sources can be used to evaluate the data. For example, if MMLU clinical questions are contaminated, then the knowledge SSE can be run on new clinical articles, as the model will never have seen these samples.\n\n> API Questions\n\nAPI-based models sometimes provide the negative log-likelihood per token for input tokens. However, as API costs evolve, it becomes harder to predict exactly the costs.\n\n> Differences in LLMs\n\nWe do see that there is a particular ordering among different models across the various metrics we considered. In almost all cases (except for tokenization), higher sensitivity scores mean that the model is better.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The new general automatic evaluation is interesting. The correlation between the proposed metric and its corresponding domain-specific metrics seems highly correlated.\n2. In several dimensions, the proposed evaluation way do prove the difference between the pre-trained version and instruction-tuned version of LLMs.\n3. For different problems, authors provide specific solutions to adjust their unique patterns.", "reasons_to_reject": "1. Although the proposed metric seems like a new general evaluation way, it also requires a lot of design to get it to work on various domains like toxicity, closed-book knowledge, and word order sensitivity. Now that we still need to consider a lot domain-specific features to get self-sensitivity metric work, why not directly turn to previous evolution ways on benchmarks? How to pick the best-matched metric for self-sensitivity evaluation in specific problems remains a major issue when I plan to use it. I did not get the convenience of using self-sensitivity evaluation. \n2. The authors did not provide the correlation between the proposed self-sensitivity evaluation and human evaluation results. They still only compute the correlations between the proposed self-sensitivity evaluation and the original automatic evaluation metrics.  I suggest this can not fully prove the effectiveness of the proposed evaluation ways. \n3. The existing evaluation results are not complex enough. It is better to incorporate more datasets as well as new models to prove self-sensitivity evaluation’s effectiveness. \n4. The writing structure is a little confusing. I did not get the reason why authors put context sensitivity and tokenization robustness in the appendix while others on the main context. \n5. The author suggests the proposed evaluation is dataset-agnostic. However, I find that different evaluation datasets need different self-sensitivity function designs. Does this mean self-sensitivity evaluation is still dataset-centric?", "questions_to_authors": "1. The authors mentioned a lot about data contamination in the introduction part. However, I can not understand how evaluation metric modification can migrate this.\n2. How do authors compute the perplexity of API-based models like Cohere and OpenAI? How much does it cost to compute perplexity?\n3. Is there any evidence to show that self-sensitivity evaluation can exhibit the difference among different pertained LLMs?", "extra_scores": {}}, {"review_id": "LQYlgyqEJS", "reviewer": "Reviewer_NEer", "summary": "This paper proposes a self-sensitive evaluation paradigm that can be used to evaluate LLMs from multiple aspects, such as closed-book knowledge and toxicity.\nThe main idea is to transfer the original corpus into a form of transformation and measure the similarity between the original one and the transformed one.\nThe paper shows evaluation over multiple metrics, and compares several LLMs, such as Davinci, LLaMA, etc.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, Reviewer NEer.\n\n> Each transformation of the corpora requires human design and can be biased and less robust in the sense that\n\nWe agree that the method requires human design; however, so do all evaluations. All evaluations necessitate human design to develop the required input and label pairs. This procedure fundamentally takes much longer to construct than our self-sensitivity framework, which remains significantly more efficient than conventional forms of evaluation.\n\nRegarding the bias and robustness of the transformations, we believe it is an interesting property of our evaluation that, in general, these transformations only need to be informative on average (see Appendix A.5, \"Do All Transformations Need to Be Informative?\"). If some of the transformations are informative for a given corpus, then, although scores may be lower, sufficient signal exists to compare relative model performance. We actually consider this robustness a core strength of our approach and would be happy to discuss this further. That being said, we agree that the negation process could be made more general. One way to do this would be to ask another model, like ChatGPT, to construct a counterfactual to the sentence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is generally well written.\nThe proposed method is easy and can be extended to other datasets.\nMost analyses are interesting.\nAlso, kudos to the authors as they include corresponding discussion of their limitations.", "reasons_to_reject": "1. The overall methods are rather human-designed, i.e. each transformation of the corpora requires human designs. For example, inserting \"not\" into the sentences. \n\n2. Such word-level transformation can be simple and can be easily extended. However, it can be biased and less robust in the sense that, for example, there are multiple ways of expressing \"not\", and models can show different performances towards such transformation.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHm33SsW16", "reviewer": "Reviewer_vbKk", "summary": "This paper introduces a new framework that does not rely on human-annotated data for LLM evaluation. Specifically, the paper’s methodology involves modifying the input through negation and various techniques, thereby evaluating the robustness of LLMs. The work incorporates several self-supervised evaluation strategies to thoroughly assess different facets of LLMs, along with detailed case studies. Notably, the experimental results demonstrate a positive correlation between the proposed metric and performance on downstream tasks, indicating its potential as a valuable complement to traditional evaluation methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your time, vbKk\n\n>  Position the contribution to [1]\n\nThank you for bringing up this work. We will include it in our revised version of the paper.\n\nRegarding the differences, [1] considers adversarial examples to the original input and label pairs. We believe this aligns more with [2], which we have included in the related works. The differences themselves stem from the fact that we consider labels, whereas [1] and [2] do not require them. In a sense, our evaluation is self-supervised.\n\nTo conclude, while the methods in [1] and [2] rely on existing labeled data, our method departs from these previous works as we analyze invariances using a data-agnostic, label-free procedure.\n\n[1] https://arxiv.org/pdf/2302.12095\n[2] https://aclanthology.org/2020.acl-main.442/\n\n> On the transformations\n\nOur evaluation method generally only requires transformations to be informative on average (please see Appendix A.5, \"Do All Transformations Need to Be Informative?\"). Even if some transformations are only informative for a specific dataset, there is still enough signal to compare relative model performance, although the scores may be lower. We view this robustness as a key strength of our approach. Furthermore, we conducted a quick human examination of many of the transformations to confirm that they were indeed meaningful.\n\nThe self-sensitivity metric necessitates human design, as is the case with ALL evaluation methods. However, our self-sensitivity framework offers a more streamlined and efficient approach compared to conventional evaluation methods. While domain-specific adaptations are necessary, they are less time-consuming and complex than creating entirely new benchmarks for each specific problem. Additionally, our framework's flexibility allows it to be tailored to a wide range of domains and scale to the entire size of a corpus.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-organized and easy to follow.\n2. This paper presents a new approach to the evaluation of large language models (LLMs). By eliminating the dependence on human-annotated labels, it paves a new path for more objective and scalable evaluation methods.\n3. The paper’s methodology can be seamlessly applied to diverse domains and data types, which is valuable in realistic settings.", "reasons_to_reject": "1. The key idea of modifying the input sentence and measuring sensitivity or invariance has similarities to robustness testing, which has been investigated in previous work [1]. It would be beneficial to better position the contribution of this paper.\n2. The proposed input modification method is somewhat trivial, this simple method might generate meaningless samples, thus affecting the main claim of this paper. It would be better to analyze to what extent such samples affect the claim of this paper.\n\n[1] ON THE ROBUSTNESS OF CHATGPT: AN ADVERSARIAL AND OUT-OF-DISTRIBUTION PERSPECTIVE. ICLR2023 Workshop.", "questions_to_authors": "I'm curious about how many of the generated samples are meaningful.", "extra_scores": {}}, {"review_id": "ygcNuoqATK", "reviewer": "Reviewer_zqFn", "summary": "This paper introduces a straightforwardly designed SSE evaluation metric based on different perturbation methods to detect a model's self-sensitivity. Extensive validation reveals a high correlation between the assessment of self-sensitivity and direct evaluation of language models on benchmarks, thereby demonstrating the effectiveness of the self-sensitivity method for assessing unlabeled data.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you, zqFn.\n\n> On [1] to evaluate model quality. \n\nThis raises a valid point regarding an LLM as a judge. However, the form of evaluation mentioned in [1] is specifically tailored for chatbots or models that follow instructions. In contrast, our self-sensitivity metrics cater to a broader class and are more general, evaluating any LLMs, including base models. For instance, similar to other evaluations found in the LM-Eval Harness, we do not judge the generations except for toxicity. In cases of toxicity, we demonstrate that our method correlates very well when using a model as the judge, specifically the PerspectiveAPI by Google. The advantages of our method compared to using a model as a judge are that it is much more *efficient*.\n\nAdditionally, our method seeks to understand the model's behavior without focusing on the model's generation. We include a perplexity baseline in our Knowledge metric, which shows that perplexity does not always accurately capture model behavior. For example, Galactica, a specialized science language model, exhibits high perplexity (where higher is worse) but scores well on the MMLU clinical metric . This discrepancy is not observed in our method, which accurately predicts Galactica's performance.\n\n[1] https://arxiv.org/abs/2306.05685\n\n>Figure 2 seems to missing the performance on the \"Negations\".\n\nApologies for the confusion. We utilized negations to determine closed-book knowledge. In Section 4, we detail how we used negations.\n\n> Questions regarding Bookcorpus usage\n\nWe apologize for the confusion. In Section 4, \"Knowledge with Negations,\" we show that the normalized sensitivity score correlates with labeled datasets. The unnormalized sensitivity metric uses Wikipedia as the corpus to calculate the score. We use BookCorpus to help normalize our metrics. The goal of this normalization is to control for higher sensitivity scores from Wikipedia due to the presence of the word \"not\" in the sentences. To achieve this, we use BookCorpus, which does not contain factual information, to control for this effect. Figure 12 (left) in App. shows the plot without normalizing for the appearance of \"not\" in the sentence. Here, we see that the instruction-tuned models are more dispersed compared to Figure 3 (left). Additionally, we believe it is an interesting property of our evaluation that, in general, these transformations only need to be informative on average (Appendix A.5, \"Do All Transformations Need to Be Informative?\").", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the designed SSE metrics is simple yet effective, which highly correlated with evaluation on benchmarks", "reasons_to_reject": "1. Until I finished reading the entire paper, I found it challenging to grasp the specific problem the paper aimed to address. The description of the issue in the introduction, as well as the analysis of the shortcomings of related work, was not clear.\n   1. I understand the solved problem in this paper in conclusion section \"The key advantage of self-sensitivity evaluation is that it has the potential to remove the need to laboriously label new data, leading to more efficient forms of evaluation in real deployment settings. We showcase several case studies\".\n   2. If this is the problem to be solved, an obvious issue arises. The paper lacks an effective baseline for comparison and analysis to demonstrate the superiority of the proposed self-sensitivity method. For instance, a very simple (and likely effective) approach would be to directly use LLM-as-a-judge [1] to evaluate model quality. Extensive prior work has demonstrated that powerful closed-source models like GPT-4 have a significant advantage over human evaluation in assessing the quality of language model generation. This advantage could also address the problem raised by the authors, i.e., \"it has the potential to remove the need to laboriously label new data, leading to more efficient forms of evaluation in real deployment settings\".\n\n[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena https://arxiv.org/abs/2306.05685", "questions_to_authors": "1. Figure 2 seems to missing the performance on the \"Negations\".\n2. In section 4, the normalized sensitivity score is highly correlated with the \"neutral corpus bookcorpus\", which raise following questions\n   1. Why is the bookcorpus used in this paper. Is wikipedia also useful?\n   2. The proposed metric inevitably is influenced by multiple factors of data in the bookcorpus: (1) the number of **m** total samples; (2) the sampled data samples.\n   \n    This influence of these factors are not fully covered in this paper.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a self-sensitivity evaluation for LLMs to evaluate their behavior on unseen data. The method proposes applying transforms to LLMs and evaluating the variation of the similarity scores. The paper studies average change in perplexity of models on perturbed sentences. After rebuttal phase, the paper has received consistent score, achieving consensus among reviewers. Two reviewers reported increased scores based on the rebuttal.The paper's strengths lie in reducing the amount of human intervention in evaluating LLMs. There are common weaknesses between the reviewers stating the lack of evaluation against human reviewers and the need for perturbation rules (such as negation). Results show correlations between sensitive scores and a number of model families, indicating that models could be evaluated with fewer human-labelled instances.
Decision: Accept