On Robustness-Accuracy Characterization of Language Models using Synthetic Datasets
Authors: Ching-Yun Ko
Pin-Yu Chen
Payel Das
Yung-Sung Chuang
Luca Daniel
Venue: COLM (2024)
Keywords: synthetic data; evaluation
TL;DR: A real-data-free evaluation framework for language models

Abstract: In recent years, language models (LMs) that were pretrained at scale on diverse data have proven to be a successful approach for solving different downstream tasks. However, new concerns about proper performance evaluation have been raised, especially for test-data leakage caused by accidentally including them during pretraining, or by indirectly exposing them through API calls for evaluation. Motivated by these, in this paper, we propose a new evaluation workflow that generates steerable synthetic language datasets and proxy tasks for benchmarking the performance of pre-trained LMs on sentence classification tasks. This approach allows for better characterization of the joint analysis on the robustness and accuracy of LMs without risking sensitive information leakage. It also provides a more controlled and private way to evaluate LMs that avoids overfitting specific test sets. Verified on various pretrained LMs, the proposed approach demonstrates promising high correlation with real downstream performance.

Published as a conference paper at COLM 2024
On Robustness-Acuraccy Characterization of Language Models using Synthetic Datasets
Ching-Yun Ko
Pun-Yu Chen & Payel Das
MIT cyko@mit.edu
IBM
pin-yu.chen@bmmccm, daspa@us.imm.com
Yung-Sung Chuang & Luca Daniel
MIT
yungsung@mit.edu, luca@mit.edu
Abstract
In recent years, language models (LMs) that were pretrained at scale on diverse data have proven to be a successful approach for solving different downstream tasks. However, new concerns about proper performance evaluation have been raised, especially for test-data leakage caused by accidentally including them during pretraining, by indirectly exposing them through API calls for evaluation. Motivated by these, in this paper, we propose a new evaluation workflow that generates steerable synthetic language datasets and proxy tasks for benchmarking the performance of pre-trained LMs on sentence classification tasks. This approach allows for better characterization of the joint analysis on the robustness and accuracy LMs without risking sensitive information leakage. It also provides a more controlled and private way to evaluate LMs that avoids overfitting specific test sets. Verified on various pretrained LMs, the proposed ap- proach demonstrates promising high correlation with real downstream performance.
1 Introduction
In recent years, language models (LMs) have emerged, showcasing remarkable capabilities across a wide range of natural language processing (NLP) applications (Peters al., 2018; Devlin et al., 2019; Yang et al., 2019; Raffel et al., 2020; Rae et al., 2021; Thoppilan et al., 2022; Hoffmann et al., 2022). While new opportunities present themselves with foundation models, they also bring forth potential risks and challenges (Bommasani et al., 2021; Blodgett & Madaio, 2021; Thieme et al., 2023; Biderman et al., 2023). For example, despite the unprecedented publicity of LMs and beliefs in their emergent abilities (Wei et al., 2022), some also argued the emergent abilities of LMs are a mirage (Schaeffer et al., 2023) and change in metric choice can lead to different conclusion. Recently, researchers have also expressed concerns about the potential for LMs to be trained on test sets (Li, 2023; Zhou et al., 2023; Golchin & Surdeanu, 2024; Oren et al., 2024; Xu et al., 2024). Even worse, private or held-out unpublished test sets may well be vulnerable to data leakage through querying the LMs via APIs for evaluation purposes. Extraction attacks (Carlini et al., 2019; 2021), membership inference attacks (Hisamoto et al., 2020; Thomas et al., 2020; Mireshghallah et al., 2022), and generative embedding inversion attack (Li al., 2023), caused by unintended memorization (Carlini et al., 2019; Shi et al., 2024) further deepened our concerns about test set contamination.
To address the caveat of test set contamination, in this paper, aim to propose new testbed for evaluating LMs with synthetic data. We link the design of the synthetic test set to two fundamental skills infants must master during language acquisition: identifying words and understanding linguistic structures (Frost et al., 2020). One intuitive approach is to generate labeled synthetic sentences using existing generative LM and then evaluate

---

Published as a conference paper at COLM 2024
SynTextennh
modall
margin
Lexicon with word-level labels
Synthetic datasets
model 5
accuracy
Figure 1: Overview of Syn TextBench. SynTextBench generates set of synthetic datasets from any given lexicon with word-level labels. We test the given LM on sentence-level tasks with these datasets and obtain robustness-accuaacy characterization under a range of steerable task difficulties. For each LM, we can plot the robustness-accuralt trade-off curve and make model comparisons.
LMs with the constructed test sets. By this, the generated sentences would harness language structural heuristics learned by the LM, and a decent probing result also requires the ability to distinguish words and their associated meanings, such as semantics. However, this workflow does not permit the active manipulation of synthetic task difficulties due to the limited level of interpretability (Zhang et al., 2023) and intrinsic bias of specific LMs (Acerbi & Stubbersfield, 2023). Motivated by the limitation, we explore an alternative route by entirely eliminating the reliance on LMs for test set generation. Specifically, we leverage existing sentiment lexicons, such as SentiWordNet 3.0 (Baccianella et al., 2010), to generate working word lists based on the word (or synset) level labels. We build positive, negative, and neutral word lists from the lexicon, and construct sentences following the nesting parentheses (Papadimitriou & Jurafsky, 2020), which mimics the recursion structural hypothesis about the narrow language faculty in humans (Hauser et al., 2002) and the dependency tree structure in natural language (Chiang & Lee, 2022). By maneuvering the mixing percentage of binary words (positive /negative words) and neutral words, a configurable testbed for evaluating the performance of LMs on different levels of difficulty and complexity: Finally, benchmark and quantify the ability of each LM on sentence classification tasks by comparing their performance on a set of our synthetic datasets with varying difficulty levels.
languagge
create
We dub our evaluation framework using synthetic data by SynTextBench and present the workflow in Figure where we focus on benchmarking LM sentence embeddings in terms of their accuracy and robustness. By accuracy, we are interested in analyzing the linear separability of sentence representations rendered by different pretrained LMs. We note that in learning sentence embeddings, the go-to metrics are cosine distance or linear probing accuracy, both of which imply separability. By robustness, we refer to the decision margin on these sentence embeddings with respect to the optimal classification strategy. We derive both measures using only the constructed synthetic datasets, which allow for contamination- free benchmarking of LMs. SennexBBench complements existing benchmarks by providing a controlled environment to assess accuracy and robustness, which might be essential for reasoning and planning tasks. SynTextBench is designed the evaluation of language sentence representations that covers a range of controllable task difficulties.
extendable framework for
Our main contributions are:
We introduce SynTextBench, novel theoretically-gronnde framework to generate steerable synthetic datasets towards a holistic evaluation of LMs. The use of syn- thetic datasets alleviates the risk of test-data leakage and offers new tools for LM testing and auditing.
SynTextennc provides a configurable lightweight testbed and quantifiable metric for evaluating the robustness and accuracy of LMs on different levels of difficulty and complexity for sentence classification tasks, with no restrictions on the model architecture.
We conduct experiments with several state-of-the-art LMs testbed and report their performance and behavior. SynTextBench, as a real-data-free evaluation method, shows high correlation with robustness-accurace performance evaluated
2

---

Published as a conference paper at COLM 2024
real data. Further study demonstrates its capability of making quick attribution comparisons such as analyzing fine-tuning effects for LMs.
2 Methodology
2.1 Why using synthetic datasets for LM evaluation?
To reduce the reliance on real-world data, we propose to build synthetic NLP tasks by generating synthetic sentences model inputs at test time. This way, we no longer need to exchange sensitive private data or label-annotated data as test sets with LM APIs. In making steerable and transparent evaluation framework for LMs, we first detail the desiderata of proxy tasks and the evaluation metric.
Task substance: representations that preserve class separability when evaluated by a linear classifier.
Tasks should test a pretrained LM's ability to encode sentence
Task difficulty: Tasks' difficulty should be configurable to allow for comprehensive analysis, ie., one can generate tasks of various levels of difficulty.
Task feasibility: Tasks should be feasible to solve, i.e., the sentences should be distinguishable certain degree by an algorithm that works on the raw sentences input.
Task independence: Tasks should be independent of the LM to be evaluated, in order to avoid biased evaluation, i.e., neither sentences nor labels should be given by an LM.
Task equity: Tasks should be able to be generated by anyone and affordable for any- one without requiring any private data or favoring any party with more resources.
Metric informativeness: The designed framework should give quantifiable metric that has a clear implication (e.g., the larger the better) and correlates well with the real performance.
With these in mind, is straightforward to see why we should not opt for synthetic datasets generated by any LM: (1) task difficulty would not be configurable (see more evidence in Appendix A. (2) the evaluation might be biased and favor the LM that generates the synthetic sentences or labels due to the intrinsic bias of each LM, and (3) any auditor without access proprictary LMs or datasets cannot run independent evaluation.
In the following, we explain how we leverage sentiment lexicons, such as SentiWordNet 3.0, to create building blocks for our framework. Then, we put together building blocks and generate synthetic inputs to LMs by observing nesting structure. We adjust the mixing ratio of ingredients the recipe simulate tasks of different difficulties. We depict this procedure in Figure 2. Finally, we will introduce our evaluation workflow and how arrive at a quantifiable metric.
2.2 Constructing synthetic datasets and tasks
Word List. Building a synthetic task requires us to define the synthetic inputs to be used. Here, we utilize sentiment lexicons with word-level labeling. SentiWordNet labels the synsets of WORDNET (Miller, 1995) according to the notions of "positivity",, "negativity", and "neutrality". Each of the entries in Senti has PosScore and NegScore denoting the positivity and negativity score, and ObjScore is calculated by 1 (PosScoree NegScore), denoting the neutrality score. When categorizing these words, we remove the sense number associated with the words and group words into individual word list based on the following criteria: for a word 20,
if PosScore NegScore, we categorize into the positive word list;
if PosScore NegScore, we categorize into the negative word list;
if PosScore NegScore O, categorize into the neutral word list.
3

---

Published as a conference paper at COLM 2024
a. Generate word lists from SentiWordNet
c. Running example
Neutral words
Ungaired
P.Cl
SentiWordNet
Pos/Neg words
Positive words
rich accepnable, trus. clear repellent, abuned, unocrepratle, wrong emerging, atsorpo
Uippaited
words
Negative words
Unsomebeesna pizt)
words
b. Synthetic sentence generation with nesting parenthesis
Pa. 'seos>' +1 p.) +11 (1.) p.)
Poired words
16
DD
New
Unpaired words
words
Pashivv
12
LA
words
words
Unpaired
Figure 2: Overview of the sentence generation procedure. In block a, we generate word lists from SentiWord 3.0. In block we generate each sentence token following nesting parenthesis and mixing distribution D. In block c, we show a running example of sequen- tially generating f6, t7, fs.
We give running examples in the Appendix A.3 for better understanding. In practice, we perform the procedure on SentiWordNet 3.0 and gather a positive word list with 23147 words, a negative word list with 26440 words, and neutral word list with 154993 words. The same procedures can be applied to any sentiment lexicons with word-level labeling, which will result in different word lists. To this end, we created the word lists from Net 3.0 as depicted in Figure 2(a).
Sentence structure. recent literature (Papadimitriou Jurafsky, 2020) explored the power of music and Java code in training models that transfer to NLP tasks. It further stated that, not only music and Jave code, non-linguistic artificial parentheses languages can also train LMs that yield substantial gains compared to random data when testing natural language (Chiang & Lee, 2020; Ri & Tsuruoka, 2022; Papadimitriou & Jurafsky, 2023). Motivated by this, we follow one of the abstract structures, nesting parenthesis, when generating the synthetic sentences in our proxy tasks. The inclusion of the parenthesis is to guarantee we test for the linguistic structures, whose importance is repeatedly advocated in literature from both machine learning and cognitive science (Frost et al., 2020; Wilson et al., 2020; Manning al., 2020). Specifically, nesting parenthesis involves paired tokens and recursive structure. For example, by referring to Figure 2(b), one sees that I and f4 are paired words, while 12 and f3 are another paired words. In our example, the words are hierarchically nested, meaning the token to be paired with l2, which is fa in our case, should appear before the pairing token with f1. other words, observes a "last first out" data structure, and the arcs in Figure 2(b) do not cross.
Sentence generation and difficulty level. With the created word list from above, we will now explain how to do sentence generation following the structure introduced. Let us revisit the case in Figure 2(b). Assume we want to generate a positive sentence (label y = 1), and we already generated the first five tokens is in the sentence with colors denoting the picked word. Now, to decide the next token, we sample t6 from a mixing distribution D, where
D= <eos>' + Pn - pe) last unpaired word + (1 pw) - pe) Dnew-
(1)
To interpret distribution D, we realize that there are essentially possible outcomes for the incoming f token: (1) it can be the end of sentence indicator <<eos>', (2) it can be the popped token from the stack that stores the unpaired words, i.e., the last unpaired word, (3) it can be new word. If it is to pick distribution of new words Dnew, which directly depends on the label y of the sentence to be generated and the desired task difficulty. For positive sentence (y 1), Drewly- is
word, this word will be sampled from the
4

---

Published as a conference paper at COLM 2024
described by the probability density function (PDF) p. (NFU(X) + (1- p) (x)s(x),, where specifies the percentage of neutral words in a synthetic sentence, fNEU gives the PDF of neutral words, and fros gives the PDF of positive words. Similarly, if we are to generate negative sentence (y = - 1), we have D newlyy- described by p fNNU(x) + (1 - p) [NE where fNEG gives the PDF of negative words.
In Figure 2(c), we show a running example of the sentence generation process, where we flip a coin with 3 outcomes each time to decide on new token. When the realization is "new words" (like in is and ty), this word will also be pushed to the stack "Unpaired words" that stores unpaired words. When we are deciding is, we draw "unpaired words" and hence is is defermined by Unpaired woodds. In essence, with the generated sentence, its label is determined by construction, which guarantees the task independence since the label is not given by an LM. It also allows configurable task difficulty by adjusting the percentage p of neutral words in synthetic sentence. That is, it is easier to predict the sentiment of sentences consisting of 90% positive words and 10% neutral words than that of sentences constructed all by neutral words. On the whole, by fixing a mixing ratio p, together with the fixed Pe and Pn given in the above, one synthetic dataset will be constructed as well resulting proxy sentiment classification task. By varying the mixing ratio p, a set of tasks with diverse difficulties can be created. In the Appendix A.4, we prove the task feasibility by demonstrating the separability of generated synthetic datasets by SentiWordNet sentiment analysis algorithm (Denecke, 2008). With an increasing mixing ratio p, while the task becomes harder, we show there least exists an algorithm that can separate the data to a certain degree, showcasing lower bound on the optimal classification strategy. By our workflow of constructing synthetic datasets and tasks, we also guarantee task equity since the generation process requires access to any LM or private data, and can be readily replicated by anyone with limited resources. Furthermore, we note that the construction of synthetic datasets and tasks described herein is also extendable to other lexicons and tasks by swapping the lexicon used for extracting word lists.
Lastly, we note that during the construction of synthetic sentences, the probability Pe associated with the special token '<eos>' is determined by its frequency in the English Wikipedia corpus. For the remaining mass 1 Pe, pn portion is assigned to new words, with its value picked following Papadimitriou & Jurafsky (2020), which is pn = 0.5. Additionally, when there are no unpaired words in the stack (e.g., when drawing the starting token of the sentence, or when all the unpaired words are popped), we assign its probability pn (n Pc) to new words. We show the length profile of our synthetic data in Appendix A.5.
A workout example. Suppose we are to generate synthetic task with difficulty = 0.7. Now, to generate a positive sentence for the task, we consider sampling the first token l following equation 1. At the first token, the mixing distribution D = Dnew since last unpaired word is an empty stack and the first token can not be the special token "<eos>'. We draw I from 0.7 fx) + 0.3 (x) and obtain 4 'nice'. At the second token, 12 is sampled from the mixing distribution D = Pr. '<eos>' + p,, - pc) last unpaired word + (1 (1) - Pe) Dnew, where last unsairedd word= ['micee'..
Discussions. The inclusion of parenthesis in our sentence structure guarantees we test for the linguistic structures but at the same time makes -rammaamatiica test sets. While grammar might be crucial in some NLP tasks that requires more advanced reasoning. For sentiment analysis, we believe it should not have strong dependency on grammar (we exclude the scenario of negation which can be detected by a rule-based method). For example, the reviews "love love fantastic", "love fantastic love" and their word permutations should all be predicted as positive, regardless of their grammar. We support this intuition by additional experiment where we noticed that 86% of the labels given by Huggingface sentiment analysis pipeline on product reviews classification (Hu & Liu, 2004) remain the same after removing 284 stop words from the sentences and hence making them non- grammatical. We leave more details and sentence examples to Appendix A.6 and A.7.
5

---

Published as a conference paper at COLM 2024
Algorithm 1 Benchmarking LMs using synthetic datasets (SynTextBench))
Imput: Sentiment lexicons S, a range of difficulty levels P, an LM g, threshold accuracy ar. Output: SynTextBench score that quantifies the robustness-accuraccy performance. Construct
word lists from sentiment lexicon S.
2: for p in P do
3: Generate synthetic binary classification task and obtain training set (xtrain, y yytrain) and test set (xtest, ytest).
4:
Calculate transformation Ti and T-1 from 2/rain {g(x;) | (x, E (xtrain ytrain), 1}
and = {g(x) | (x, E (xxrain, b g/rrai)) -1}.
5:
Transform training set and test set zi train Ti (z(rain) z train T (2!"q(i)) and
2 lest = (z 2 test (z
6:
Derive the Bayes optimal classifier f according to sign (j (2-
M1 (m]11) (2)) based on
2 train and z train i.e. #1 = mean(zi train), ), 122 = 1 train)
7: Read out the accuracy a of on zi test and =11 test and calculate the average scale margin := aog(II4:112) according to || (2- 21 y' for correctly-clsssiied
1
sentence embeddings.
8: Denote the accuracy and average margin pair on the task by (ap,
9: end for
10: Define a goodness function s(a) |b) E(pePa,>a) 6p, for E [[00,1].
11:
SynTextBench score = S (a))aa
2.3
Robustness-accuraccy evaluation
Given an LM g, let y be the input sentence and its label, z be the sentence embeddings
z = g(x) E R", we are interested in evaluating the accuracy of the sentence embedding classifiers f, and the average distance A from sentence embeddings to the linear classifiers (i.e., decision margins). We let 2 be {z:2= = gg())) = 1} and z be {z: 12:2 = 8(x), = -1}.
Preparing sentence embeddings. Recall that Bert-flow (Li et al., 2020) and Bert- whitening (Su et al., 2021) transformed the sentence embeddings into an isotropic Gaussian distribution to remedy the anisotropic behavior in the sentence embedding vector space. We thereby also perform whitening on sentence representations before we draw the deci- sion rule on the embeddings. Transforming set of sentence embeddings of a class into an isotropic Gaussian involves two steps: (1) model the mean by and covariance Σy of original embeddings Zy, (2) apply transformation the embeddings g-122/2 where FSF" = Σy is the singular value decomposition of Σy- Nevertheless, since Σy can be ill- conditioned, directly applying S-1/2 on embeddings Zy might amplify noisy signals due to numerical instability. Thus, we propose to reduce the dimension according to energy- preservation (Leskovec et al., 2020) (also called variance-based methods by Falini (2022)).
0.99, where s; = diag is the i-th largest singular value of Till now, we see that the sentence embeddings are trans- 1/2 1-2 formed to an R vector space via F.T 1.1. S 1= Zy. We perform these operations for both classes (y= and y - 1) separately. Since we want the transformed embeddings to observe the original relative distance between two classes, we further scale the distance between two whitened Gaussians by d / /nttar-cclasss where the numerator dinter-class = ||b; b-1ll calculates the inter-class distance (the distance between two class centers b1 and b 1), and
We select to keep K dimensions according to arg min,
2i-1
m2 (E,") 1 ||| by + -"T ||2/ b-ill) calculates the intra- class distance (the average distance from class data to class mean) with m, and m2 being the number of positive sentences and negative sentences, respectively. We let Ty denote the overall transformation operations and obtain transformed embeddings E1 = (2,) and
the denominator dintra-class
z"1 T 1(2 1)-

---

Published as a conference paper at COLM 2024
Decision margins induced by robust Bayes optimal classifiers. Recall that robust Bayes optimal classifiers explicitly give the optimal classification strategy for cossiiondiii Gaussian distribution in the presence of data perturbations (Bhagoji et al., 2019; Dan et al., 2020). Here, we see that (2 are modeled as P P,,,...kk y = 1 N ( Ix), -1
(u Ik), and y E C = {+1,- -1.. While finding the robust Bayes optimal classifier generally involves solving the optimization problem arg min |=lla
(1r | 2 z) (cf. Appendix A.1), we can prove that, when the covariance is an identity matrix, the class priors Fyy- 1) = T, Ply = -1) = 1 T, the perturbation radius e, then the optimal simply f sign(w w = (1(1 - / /| (2), and fi = "1222. Furthermore, when the classes are balanced (i.e., I = 1/22), the robust Bayes optimal classifier overlaps with the Bayes optimal classifier. That is, the (robust) Bayes optimal classifier is plainly sign(f"( === )), which is independent of e. We then use this given classifier to calculate the accuracy on the synthetic datasets.
classifier is given
"1 9/2), where 9 log - -)/
In fact, we prove in Appendix A.8 that, as long ā lies completely within degenerate subspace of the eigenspace of the covariance matrix (i.e., with eigenpairs { (Λy v2), k E [n]] }
for Vij E(k: : Ax * 0, ji Ux # 0}, A; A A), the e-robust Bayes optimal classifiers overlap for all e. In the case of an identity covariance matrix, the degenerated subspace of the eigenspace expands the whole RK, hence ş lies in the space naturally.
Now that we have specified the optimal robust classification rule on the transformed sentence embeddings, we write out the decision margin induced by the classifiers using an informal but more intuitive statement: For any sample z, the Bayes optimal classifier f of class-balanced dlass-oodditinas Gaussian distribution P Pruu2lxx yields a decision margin
of |||a [22 "1 TT
and if we scale the margin by the distance between two Gaussian F2
centers || we obtain scaled margin of || = [[2 " We give the formal
Tplli
results for the generic class prior in Appendix A.8. To this end, we have prepared sentence embeddings and specified the way of calculating decision margins induced by a robust Bayes optimal classifier. In the following, we will state the complete algorithm for characterizing robustness-accuracy performance of LMs using synthetic datasets.
2.4 SynTextBench score and algorithm
With Section 2.2 and Section 2.3, we now can simulate synthetic tasks of configured level of difficulty and evaluate their accuracy and margin. In our benchmarking process, we essentially build on this foundation to generate sequence of tasks with different difficulty levels and inspect how the magnitude of decision margins changes with the classifier accuracy. In terms of robustness-accuracy characterization, it is desirable for LM to consistently yield high classification accuracy, while maintaining a big decision margin (that is, less sensitive to perturbations in the embedding space). The pseudocode of the proposed framework, Texteenheh,,, is given in Algorithm 1.
In practice, we let P {0, (0,0.05,... 0.9,, 0.95), and subsequently generate 20 synthetic datasets with p- = being the easiest and p 0.95 being the hardest (cf. Section 2.2). Then, we perform analysis on the sentence embeddings of various synthetic datasets, and threshold the accuracy at ar based on utility. The threshold serves as a penalty for poor sentence embeddings that lead to an undesirable accuracy under this threshold, matching our task substance of testing LM's ability to preserve linear separability. By referring to Figure 1, Line 1 in Algorithm determines the word lists from a given lexicon. From Line 2 to Line 9, the for-loop generates one synthetic dataset at one time, on which we compute an (accuracy, average margin) pair (αp, and draw one point on the margin- accuracy 2D plot in Figure 1. We apply Algorithm 1 on various models and obtain margin-accuracy curve for each model. Since we not only care about the curvature of the curve but also how the (accuracy, average margin) pairs span on the curve, we define goodness function s(a) = [P| E(pePap2n) op on IR [0, 1] in Line 10 account for the span. By our definition, saa) will be monotonically decreasing function (e-g., Appendix A.9) and
7

---

Published as a conference paper at COLM 2024
calculate the expected margin conditioned on the accuracy level. The final SynTextBench score is defined by the integration over the desirable range of threshold accuracy in Line 11, quantifiable score to inform
Lee SynTextBench score = Jer s da. We use Syn TextBench the accuracy-robustres aspect of a pretrained LM. In the later section, we will demonstrate the metric informativeness by measuring the correlation between SynTextBench scores and the average real-world sentence classification task performance.
Experiments
3.1 Setups
LMs. In the experiment, we will analyze the pretrained LMs predominantly considered by the sentence embedding literature Gao et al. (2021); Su et al. (2021); Chuang et al. (2022), and also larger models such as LLaMA and OPT (Touvron et al., 2023a;b; Zhang al., 2022).. Specifically, we consider encoder models such as BERT base. BERT large (Devlin et al., 2019), RoBERTabase (Liu et al., 2019), DifICSE-B, DifíCSE-R (Chuang et al., 2022); encoder-decoder models such as base, aarge (Raffel et al., 2020), ST5 (Ni et al., 2022); and decoder models such as DialogRPT (Gao et al., 2020)), LLaMA-7B, LLaMA-13B, LLaMA-30B (Touvron et al., 2023a), LLaMA-2-7B, LLaMA-2-13D (Touvron et al., 2023b), OPT-13B, OPT-30B (Zhang et al., 2022). For models that have an encoder component (encoder-only or encoder-decoder), we use the average output from the first and the last layer as sentence embeddings. For the decoder-only model, we use the embedding of the last token as sentence embeddings.
Baselines. We followed the open-s implementation of the literature (Whitney et al., 2020) and fed the pretrained LMs with synthetic texts generated according to Section 2.2 and reported the validation accuracy (Val loss), minimum description length (MDL), surplus description length (SDL), and e-sample complexity (eSC) as baselines (Blier & Ollivier, 2018; Voita & Titov, 2020; Whitney et al., 2020). Since these methods take one dataset as inputs, we choose relatively easy synthetic proxy task generated by 0.2 as the input dataset.
Objectives. Through the experiments, our main aim is to verify the feasibility of making per- formance assessments of possible downstream tasks by real-data cal-dtt-frre evaluation methods. To achieve this, we will compare the Pearson correlation coefficients of assessments given by different real-data-free evaluation methods with the performance on real-world tasks. Since SynTextBench is intended to inform the robustness-accuracy performance, we will report both the accuracy and robustness on real-world tasks for studying correlation. We use PWWS attack (Ren et al., 2019) through TextAttack, a Python framework for adversarial attacks in NLP, to generate attacks. Essentially, the attacker will perturb the inputs gradually by changing more and more words until the perturbation leads to a wrong classification result. Therefore, we report the average number of perturbed words in successful attack as an indicator of the level of model robustness. We will also demonstrate how SynTextBench can be used do attribute comparisons. Finally, as more attentions have been drawn to large LMs lately, we will also conduct an extended study on large LMs and include discussions on in-context learning performance on SynTextBench synthetic data. We defer experimental details to the appendix due to the page limit.
3.2 Performance evaluation and discussion
We evaluate encoder models listed in Section 3.1 by SynTextBench framework as well on real-world sentence embedding tasks. Specifically, we simulated 20 synthetic datasets as described in Section 2.4 and obtained one goodness function s(a) for each LM. We plot these functions together in Figure 7, from which the final Syn TynTeetBench score can be determined by definition. We refer readers to Appendix Table 7 for the exact numbers due to the page limit. To gauge the performance of these pretrained LMs on downstream real-world tasks, we evaluate the given models on SentEval (the Evaluation Toolkit for Universal Sentence Representations (Conneau & Kiela, 2018)) and show the detailed numbers in Appendix Table 6 and Figure 8. SentEval tasks include seven semantic textual similarity tasks (denoted by "STS tasks"), where results are given by the Spearman's correlation with output range
8

---

Published as a conference paper at COLM 2024
Table 1: Correlation between real-data-free evaluation metric and real-data accuracy at different synthetic dataset sizes.
Table Aggregated correlation with real-data-
free evaluation metric and the robustness-
accuracy performance, and its breakdown.
32768
Correlation. loss
Rob-Acc.
Rob.51S
Rob-Transfer
Vass
0.2910.30
0.65
0.6 10.01
0,2710.02
0.0610015
0.0810.13
-0.131021
MDL
0.52+-0.11
0.52+0.04
0.51+00.3
0.48++0.03
MDL SDL
0.6400.6
0.55+00.8
0.2++005
S-L
0.5710.11
0.5110.04
0.3:.0022
0.3110.01
0.6010022
0.1110.04
0.58.10.05
rSC,
0.04+0000
SS,
SynJexiBench
0.94
0.9720.01
0.96:0000
0.93:00.0
SynJextBench
0.7620.04
0.76:0003
0.69:0.05
[-1,1], and seven transfer learning tasks (denoted by "Transfer task"), where results are given by the standard accuracy with range [0, 1]. We scale the former to the same range as the latter, [o, 1], and take an average as the final accuracy indicator.
Correlation with real-world tasks. To demonstrate the informativeness of SynTextBench score, we list the Pearson correlation coefficients between real-data-free evaluation methods and the accuracy of SentEval tasks in Table 1. Five real-data-free metrics are considered that includes Val loss, MDL, SDL, eSC, and the proposed SynTextBench. Since the smaller the baseline metrics are, the better, we add a negative sign in front of them when calculating the Pearson correlation coefficient. As we have the flexibility of generating synthetic datasets with various sizes (number of sentences), we compare four configurations n = {4096, 8192, 16384, 32.683. From Table 1, we observe that Syn TextBennn consistently gives scores highly correlated with real-world task accuracy, with correlation coefficients that are above 0.9. For the four baselines, the highest correlation ever achieved is when n 8192 and evaluated by Val loss, 0.65. It is noteworthy that SynTextBench is also a stabler metric as substantiated by the smaller standard deviation.
Ablation on the nesting structure. To showcase the effect of the nesting structure, we see that no nesting structure is special case of our proposed framework when Px: = 0 (cf. Equation 1). In Table 1, we have Syn SynTextBench(pp = 0.5) = 0.97. In comparison, we run the analysis for pn 0 and obtain TentexcBench SynTextBench, with both parameters, outperform the baselines by large margins. Between the two, SynTextBench with the imposed structure further improves the correlation.
0) = 0.92. In conclusion,
Robustness implications. To understand how real-data-free evaluation methods correlate with real-world task robustness-accuracy performance, we further analyze the correlation with the robustness indicator, the average number of perturbed words, on Transfer tasks when n 8192. We focus on these tasks as they are classification tasks where adversarial attacks are well-defined. To combine robustness correlation with accuracy correlation, we add up two ranking vectors by robustness and accuracy measures, and calculate its Pearson correlation with the ranking by one of the real-data-free evaluation metrics (Val loss, MDL, SDL, eSC, TynTexteechh) This way, we effectively obtain the aggregated Spearman corre- lation coefficient between real-data-free evaluation metrics and joint robustness-accuracy performance. We refer readers to Appendix A.12 for more experimental details. We list the results in Table 2. From the "Rob.-Acc." column, we see Syn TextBench has overall higher correlation with robustness-accuracy performance compared to other baselines. To be more precise, SynThetBeenh shows coefficient of 0.76, whereas MDL and SDL are 0.64 and 0.60. Recall that accuracy results were aggregated from STS tasks and Transfer tasks. In Table 2, we also show how each component contributes to the correlation. In the "Rob.-STS" and accuracy measure when ranking the models, and the remaining steps follow. From the two columns, we see that SynText still shows a stronger correlation compared to baselines, while having slightly better correlation with Robustness-STD accuracy performance than Tobustness-Transfer
columns, we use only STS or Transfer task results as the
Transfer accuracy performance.
Case study on model comparisons. Besides having high correlation with real-world task performance, we show how SynTextBench can be used to make model comparisons. From Table 7, one sees that, the SynTextBench score of ST5 is significantly higher than that of T5 across all dataset sizes n1, e.g., ST5's s 0.223 vs. T5's 0.130 when 8192. This indicates contrastive fine-tuning is beneficial for improving sentence embeddings. This conclusion is in sync with the observations from real-world tasks, where we see ST5 yields both higher

---

Published as a conference paper at COLM 2024
Table 4: Pearson correlation comparison between the in-context learning accuracy on Syn- TextBench synthetic tasks and the average in-context learning accuracy on the real-world tasks of decoder models.
Name
MAMA-7
1LaMA-130
LLaMA-30B
LLaMA-2-7B
L1aMA-2-138
OPT-138
OPT-308
Pearson
Reallife acc.
73.58
80.07
68.37
70.49
8192
SynlextBench
50.82
53.43
59,09
$1.48
38.83
52.87
51.79
0.13
accuracy and robustness according to Table 7 and Table 11. Specifically, ST5 has an average accuracy of 90.17 and robustness 13.23, whereas T5 has average accuracy of 82.78 and robustness 12.21.
3.3 Extended study on large LMs
Since Syn TextBench focuses on the sentence embed- dings of LMs, of which larger decoder models gen- erally do not have better performance than smaller encoder models (Ethayarajh, 2019), we have given most of our analysis encoder models in Gao et al. (2021). Here, to demonstrate the generality of Syn- TextBench to various LM types, we analyze more large decoder LMs such as LLaMA and ÓPT (Tou- vron et al., 2023a;b; Zhang et al., 2022)) Similar Table 1, we calculated the Pearson correlation coef- ficients between real-data-fress evaluation methods
Table 3: Correlation between real- data-free evaluation metric and real- data accuracy.
Name Val loss MDL
Pearson correlation
0.80
-0.47
$
-0.55
Textexbeecch
0.87
and the accuracy of SentEval tasks in Table 3. According to the table, SynTextBench also
gives scores highly correlated with real-world task accuracy on decoder models, with correlation coefficient of 0.87. We refer readers to Appendix Table 9 for the complete results.
In-context learning Besides evaluating linear probing performance on our SynTextBench synthetic tasks, we also evaluate the few-shot in-context learning (ICL) performance SentEval transfer tasks and SynTextBench synthetic task. We do not include STS tasks since they are typically measured by cosine distance, whose ICL prompts are less obvious to us. We also excluded TREC as we have not found proper prompts that could lead to reasonable accuracy. The instructions we give include two demonstrations with one demonstration for each class. For example, in CR (customer review), we use the instruction: "Answer the sentiment of the following review, either Positive or Negative. |n)nnQ We tried it out christmas night and it worked great. AnA Positive nn: very bad quality AnA: Negative)n
In Table 4, we calculate the correlation between the ICL accuracy on Syn synthetic tasks and the average ICL accuracy on subset SentEval tasks. We can see that the ICL accuracy on TenTextBench synthetic tasks shows strong correlation (above 0.8) with ICL accuracy on SentEval tasks. Future research will also be dedicated to investigate whether the success of SynTextBench can be explained by its ability to check the compositional features (e. induutiioo head (Olsson et al., 2022)) of transformers.
4 Conclusion
In this paper, we have proposed SynTextBench, a novel framework for evaluating the accuracy and robustness of LM sentence embeddings. SynTextBench is a configurable real-date-free lightweight testbed that generates steerable synthetic language datasets and proxy tasks, avoiding the risk of test-data leakage. SynTextBench is the pioneering effort in developing synthetic benchmarking methodologies for NLP, with primary focus on sentence classification tasks and does not cover other NLP tasks such question answering, machine translation, or summarization. By concentrating on this specific aspect, we have provided solid foundation upon which future research can build.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
This work was partially supported by the MIT-IBM Watson AI Lab and by the National Science Foundation.
References
Alberto Acerbi and Joseph M Stubbersfield. Large language models show human-like content biases transmission chain experiments. In Proceedings of the National Academy of Science, volume 120, pp. e2313790120, 2023.
Pranjal Awasthi, Natalie Frank, and Mehryar Mohri.
On the existence of the adversarial bayes classifier. Advances in Neural Information Processing Systems, 34:2978-2999 2990, 2021.
Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. SentiWordNet 3.0: An en- hanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta, May 2010. European Language Resources Association (ELRA). URL http://www. 1ww.wwo.oecooooffo.mm wtr/ppoocedingg//eeee00///// /769 Paper.
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness from optimal transport. Advances in Neural Information Processing Systems, 32, 2019.
Kush Bhatia, Avanika Narayan, Christopher De Sa, and Christopher Ré. Tart: A plug-and- play transformer module for task-agnostic reasoning. arXiv preprint ar 077:53306.07536, 2023.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arviv preprint ar arXiv:2304.11158 2023.
Léonard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems, 31, 2018.
Su Lin Blodgett and Michael Madaio. Risks of ai foundation models in education. arXiv preprint Xiv::2 10. 10024, 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. ar preprint arXiv:2108.0725 2021.
Margaret M Bradley and Peter J Lang. Affective norms for english words (anew): Instruction manual and affective ratings. Technical report.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposuum, volume 267, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021.
Cheng-Han Chiang and Hung-yi Lee. Pre-training language model without human language, arXiv preprint urXfo:2012.11995 2020.
Cheng-Han Chiang and Hung-yi Lee. On the transferability of pre-trained language mod- els: A study from artificial datasets. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10518-10525, 2022.
11

---

Published as a conference paper at COLM 2024
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. Diffcse: Difference-based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4207-4218, 2022.
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC. 2018), 2018.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Super- vised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, PP. 370-680,22 2017.
A Dadhich and B Thankachan. Opinion classification of product reviews using naive bayes, logistic regression and sentiwordnet: Challenges and survey. In IOP Conference Series: Materials Science and Engineering, volume 1099, pp. 012071. IOP Publishing, 2021.
Chen Dan, Yuting Wei, and Pradeep Ravikumar. Sharp statistical guaratees for adversarially robust gaussian classification. In International Conference on Machine Learning, pp. 2345- 2355. PMLR, 2020.
Kerstin Denecke. Using sentiwordnet for multilingual sentiment analysis. In 2008 IEEE 24th international conference on data engineering workshop, pp. 507-512. IEEE, 2008
Jacob Devlin, Ming- Wing Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186 2019.
Andrea Esuli. Automatic generation of lexical resources for opinion mining: models, algorithms and applications. In Acm sigir forum, volume 42, pp. 105-106. ACM New York, NY, USA, 2008
Andrea Esuli and Fabrizio Sebastiani. Sentiwordnet: A publicly available lexical resource for opinion mining. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06), 2006.
Andrea Esuli and Fabrizio Sebastiani. Sentiwordnet: a high-coverage lexical resource for opinion mining. 2007.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 55-65, 2019.
Antonella Falini. A review on the selection criteria for the truncated svd in data science applications. Journal of Compulational Mathematics and Dala Science, pp. 100064, 2022.
Rebecca LA Frost, Andrew Jessop, Samantha Durrant, Michelle S Peter, Amy Bidgood, Julian M Pine, Caroline F Rowland, and Padraic Monaghan. Non-adjacent dependency learning in infancy, and its link to language development. Cognitive Psychology, 120: 101291, 2020.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, 2019.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-69100 2021.
12

---

Published as a conference paper at COLM 2024
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and William B Dolan. Dialogue response ranking training with large-scale human feedback data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 386-395, 2020.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. Declutr: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Pp.
2021.
Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /wwpeeeveeeeeeeeeeeeeeeeeeeeeeep wwt. Forum?id=2Rww@cc
Marc D Hauser, Noam Chomsky, and W Tecumseh Fitch. The faculty of language: what is it, who has it, and how did it evolve? science, 298(5598))1569--577) 2002.
Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks on sequence- to-sequence models: Is my data in your machine translation system? Transactions of the Association for Computational Linguistics, 8:49-63, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arv preprint ar X51:222003.15566 2022.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, PP. 168-177, 2004.
Mujtaba Husnain, Malik Muhammad Saad Missen, Nadeem Akhtar, Mickaël Coustaty, Shahzad Mumtaz, and VB Surya Prasath. A systematic study on the role of sentiwordnet in opinion mining. Frontiers of Computer Science, 1544):154614 2021.
Richard Arnold Johnson, Dean W Wichern, et al. Applied multivariate statistical analysis, volume 5. Prentice hall Upper Saddle River, NJ, 2002.
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. SentiLARE: Sentiment- aware language representation learning with linguistic knowledge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6975- 6988, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ 11/2220.emmll-main..55. URL https: / aclaatoooooo org/2020. emnlp-main.566 567.
Farhan Hassan Khan, Usman Qamar, and Saba Bashir. Sentimi: Introducing point-wise mutual information with sentiwordnet to improve sentiment polarity detection. Applied Soft Computing, 39:140-153, 2016.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systemss, 28, 2015.
Ching- Yun Ko, Pin-Yu Chen, Jeet Mohapatra, Payel Das, and Luca Daniel. Synbench: Task- agnostic benchmarking of pretrained representations using synthetic data. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022.
Kundan Krishna, Jeffrey P Bigham, and Zachary C Lipton. Does pretraining for summariza- tion require knowledge transfer? In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3178-3189, 2021.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empiriaal Methods in Natural Language Processing, pp. 785-794, 2017.
13

---

Published as a conference paper at COLM 2024
Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of massive data sets. Cambridge university press, 2020.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119-9130, 2020.
Haoran Li, Mingshi Xu, and Yangqiu Song. Sentence embedding leaks more information than you expect: Generative embedding inversion attack to recover the whole sentence. Xiv preprint arXto:2305.03010, 2023.
Yucheng Li. Estimating contamination via perplexity: Quantifying memorisation in lan- guage model evaluation. ar e-prints, pp. arXiv-2309, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Isipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. ar Xiv preprint ar Xii::2211009100 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. ar Xiv preprint ar wxXi::197.11192, 2019.
Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48))3004630000 2020.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatte Kirkpatrick, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. ar preprint ar arXto:2203.03923 2022.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACI. 2022, pp. 1864-1874, 2022.
Finn Arup Nielsen. A new anew: Evaluation of a word list for sentiment analysis in microblogs. In Workshop on Making Sense of Microposts: Big things come in small packages, pp. 93-98, 2011.
Bruno Ohana and Brendan Tierney. Sentiment classification of reviews using sentiwordnet. Proceedings of IT&T, 8, 2009.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 2022.
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination for black-box language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:/// https:://pppevveeeeeeee net net/forum?id KS8mIvetg2.
Isabel Papadimitriou and Dan Jurafsky. Learning music helps you read: Using transfer to study linguistic structure in language models. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 6829-6839, 2020.
Isabel Papadimitriou and Dan Jurafsky. Pretrain on just structure: Understanding linguistic inductive biases using transfer learning. arXiv preprint ar aix::2330 13060, 2023.
14

---

Published as a conference paper at COLM 2024
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zett Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 2217-22-7222222 2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/ v1/N18-1202. URL pettps://aclanthollse hww.lloggooogg/eeee org/N18- 12002.
Sergios Petridis and Stavros Peranttnnis On the relation between discriminant analysis and mutual information for supervised linear feature extraction. Pattern Recognition, 37 (5):857-874, 2004.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. Xiv preprint ar #rXiv:2112.11446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: +00,000+
questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pp. 2383,2392, 2016
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswer- able questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 784-789, 2018.
Vijjiii Anvesh Rao, Kaveri Anuranjana, and Radhika Mamidi. sentiwordnet strategy for curriculum learning in sentiment analysis. In Natural Language Processing and Information Systems: 25th International Conference on Applications of Natural Language to Information Systems, NLDB 2020, Saarbrüicke,, Germany, June 24 26, 2020, Proceedings 25, pp. 170-178. Springer, 2020.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adver- sarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1085-1097, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ /0066/////1191110/ URÍ. attpss//aclanntoll.gg htwp:/aatntoooooooo/o/eeeeee org/P19-1103.
Rezvaneh Rezapour, Saumil H Shah, and Jana Diesner. Enhancing the measurement of social effects by capturing morality. In Proceedings of the tenth workshop on computational approaches to subjectivity, sentiment and social media analysis, pp. 35-45, 2019.
Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with artificial language: Studying trans- ferable knowledge in language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7302-7315, 2022.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? arv preprint arXio:2304.1500 2023.
Samira Shaikh, Kit Cho, Tomek Strzalkowski, Laurie Feldman, John Lien, Ting Liu, and George Aaron Broadwell. Anew+: Automatic expansion and validation of affective norms of words lexicons in multiple languages. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pp. 1127-1132, 2016
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevie wttpeeww.wweooooniddeeeeeeeee 3MOuNs.
15

---

Published as a conference paper at COLM 2024
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Cupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semantics and faster retrieval. Xrviv preprint ar arXio:2103.15316, 2021.
Anja Thieme, Aditya Nori, Marzyeh Ghassemi, Rishi Bommasani, Tariq Osman Andersen, and Ewa Luger. Foundation models in healthcare: Opportunities, risks & strategies forward. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-4, 2023.
Aleena Thomas, David Tfeoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow. Investigating the impact of pre-trained word embeddings on memorization in neural networks. In Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8-11, 2020, Proceedings 23, pp- 271..81. Springer, 2020.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arv preprint Xiv::22011.022. 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint 7rXiv:2307.99288 2023b.
Elena Voita and Ivan Titov. Information- probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183-196, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020.
Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior research methods, 45:1191-1207, 2013.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
William F Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, and Kyunghyun Cho. Evaluating representations by the complexity of learning low-loss predictors. Xrv preprint ar XrXo:22009.07368 2020.
16

---

Published as a conference paper at COLM 2024
Benjamin Wilson, Michelle Spierings, Andrea Ravignani, Jutta L Mueller, Toben H Mintz, Frank Wijnen, Anne Van der Kant, Kenny Smith, and Arnaud Rey. Non-adjacent depen- dency learning in humans and other animals. Topics in cognitive science, 12(3))843-858, 2020.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arx preprint 2rXio2444418824, 2024.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 93-104, 2018.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song, A survey of controllable text generation using transformer-basee pre-trained language models. ACM Comput. Surv., 56(3), oct 2023. ISSN 0360-0300. doi: 10.1145/3617680. URL https: //doi.org/10. 1145/3617680.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arviv preprint axXiv:205.00166, 2022.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your Ilm an evaluation benchmark cheater. ar Xiv preprint XxXt:2311.00196 2023.
17

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Related Work and Background
19
A.2
Generating synthetic datasets with language model
20
A.3
SentiWordNet 3.0 synsets
A.4 Task feasibility
22
A.5 Histograms of synthetic datasets versus English Wikipedia corpus
22
A.6 List of stop words
22
A.7 Synthetic sentence examples and discussions
23
A.8 Robust Bayes optimal classifier and proofs
24
A.9 Goodness function
26
A.10 Complete results for Section 3.2
27
A.11 Complete results for Section 3.3
28
A.12 Experimental details
29
18

---

Published as a conference paper at COLM 2024
A.1 Related Work and Background
Sentence representations. To obtain performant LMs, learning universal sentence rep- resentations that capture rich information for various downstream NLP tasks without task-specific finetuning is an active research field and has also been studied extensively in the past years (Kiros et al., 2015; Conneau et al., 2017; Gao et al., 2019; Li et al., 2020; Su et al., 2021; Giorgi et al., 2021; Gao et al., 2021; Chuang et al., 2022). While learning to extract ideal sentence embeddings, (Gao al., 2019; Li et al., 2020; Ethayarajh, 2019) have pinpointed the anisotropic behavior in the sentence embedding vector space as a reason behind sentence embeddings' poor capture of semantic information. To remedy the situa- tion, Bert-flow (Li et al., 2020) and Bert- embedding distribution into an isotropic Gaussian distribution through normalizing flow and whitening post-processing. Through contrastive learning, SimCSE (Gao et al., 2021) and DiffCSE (Chuang et al., 2022) also achieved new state-of-the -f-t-----te-rtttt sentence embedding performance by promoting uniformity and alignment (Wang & Isola, 2020).
(Su et al., 2021) transformed the sentence
Evaluations of pretrained models. In evaluating the performance of LMs, the current de facto evaluation paradigm is to utilize widely-used NLP benchmarks such as the General Language Understanding Evaluation (GLUE (Wang et al., 2018) /Stp///u.perEE (Wang et al., 2019)) benchmark, the Stanford Question Answering Dataset (SQuAD v1.1 (Rajpurkar et al., 2016)/v2.0 (Rajpurkar et al., 2018)), the Situations With Adversarial Generations (SWAG (Zellers et al., 2018)) dataset, the ReAding Comprehension from Examinations (RACE (Lai et al., 2017)) dataset, the Evaluation Toolkit for Universal Sentence Represen- tations (SentEval (Conneau & Kiela, 2018)), BIG-Bench (Srivastava et al., 2022), etc. In many cases, these NLP benchmarks are supersets of datasets, e.g., GLUE is collection of 9 datasets for evaluating natural language understanding systems, and SentEval collection of 7 Semantic Textual Similarity (STS) tasks and 7 transfer datasets that have partial overlap with GLUE The heavy reliance on real-world tasks can be exemplified by broad literature. For example, Bert (Devlin et al., 2019) was evaluated on GLUE, SQuAD v1.1/2.0, SWAG; Roberta (Liu et al., 2019) was evaluated on GLUE, S1.12v..12200, RACE; and T5 (Raffel et al., 2020) was evaluated on GLUE/SuperGLuUS abstractive summarization and WMT translation. HELM (Liang et al., 2022) proposes a holistic evaluation framework for LMs that measures 7 metrics on 42 scenarios. However, when confronting the challenge of test-data leakage, to the best of our knowledge, there is no real-data-free evaluation method for NLP pretrained representations. In recent literature (Ko et al., 2022), authors reported the validation loss (Val loss), minimum descrip- tion length (MDL) (Blier& & Ollivier, 2018; Voita & Titov, 2020), surplus description length (SDL) and c-sample complexity (cSC) (Whitney et al., 2020) on class-conditions. Gaussian distribution data as an effort build task-agnostic evaluation baselines for pretrained representations in computer vision. Our proposed framework differs from this line of work in that we focus on the domain of natural language processing and we do not assume the data inputs are sampled from an idealized distribution. Instead, we create synthetic sentences and proxy tasks based on a lexical resource for LM evaluation.
v1.1/2.0,
SQuAD, CNN Mail
Sentiment lexicons. SerdiWordNet 3.0 (Baccianella et al., 2010) is a lexical resource that pro- vides sentiment information for each word in WordNet (Miller, 1995), a widely-used lexical database of English words and their relationships. SentiWordNet 3.0 is an improved version of Sent 1.0 (Esuli & Sebastiani, 2006), 1.1 (Esuli & Sebastiani, 2007), 2.0 (Esuli, 2008). Nett automatically assigns synsets of WordNet according to notions of " "positivity", "neeativity", and "neutrality" The sentiment scores of a synset are assigned on a scale from 0.0 to 1.0 and sum to 1, reflecting fine-grained opinion-related word-level labeling. SentiWord has been used in a variety of natural language processing tasks, such as sentiment analysis (Denecke, 2008; Ohana Tierney, 2009; Khan et al., 2016), opinion mining (Husnain et al., 2021; Dadhich & Thankachan, 2021), representation learning (Ke et al., 2020), and curriculum learning (Rao et al., 2020). Besides SendiWroddet, other sen- timent lexicons include Affective Norms for English Words (ANEW) (Bradley & Lang), Warriner lexicon (Warriner et al., 2013), a new ANEW (Nielsen, 2011), and ANEW+ (Shaikh et al., 2016). In this paper, we will demonstrate the use of sentiment lexicon with word-level labels in constructing synthetic datasets using SentiWordNet 3.0; however, the framework proposed in this paper can take any lexicon with word-level labels. We also envision our
19

---

Published as a conference paper at COLM 2024
framework to benefit from richer vocabulary and extend to other value lexicons like moral
lexicons (Rezapour et al., 2019).
Robust Bayes optimal classifier. Despite the difficulty of characterizing the optimal classi- fier with the minimum loss for generic data, for data drawn from class-conditional Gaussian distribution, the explicit optimal strategy is given by Fisher's 's linear discriminant rule (John- son et al., 2002; Petridis Perantonis, 2004). Likewise, the optimal classification strategy can also be given for such data in the presence of input perturbations (Bhagoji et al., 2019; Dan et al., 2020). Let N (H,, Σ) denote Gaussian distribution with mean and variance Σ. Generally, for binary classification problems with data pair (x, y) generated from a prob- (x,y)
ability distribution Pu,z: xly = 1 ~ .N (1,, I), xly = -1 ~ (--p. pt, Σ), the classifier that minimizes the adversarial loss (Awasthi et al., 2021) max Bayes optimal classifier (Bhagoji et al., 2019; Dan et al., 2020), is given by sign sign(uf x), where wp I (j (1)) and zy, is the solution of the convex problem
(f(x') # y), the robust
arg min(N - Z (1= z)
(2)
In the following sections, we will exploit robust Bayes optimal classifier in giving the explicit optimal classifier on whitened sentence embeddings and develop our theoretical groundings on top of it.
A.2 Generating synthetic datasets with a language model
To generate synthetic sentences with configurable difficulties with an LM, we reuse the word lists constructed in Section 2.2 and constrain the LM vocabulary to be within the word lists. Concretely, let V be the original tokenizer vocabulary, POS be the set of positive words, NEU be the set of neutral words, NEG be the set of negative words, and STOP be the set of stop words (see A.6), then we constrain the LM vocabulary to be V = STSPPP where composes of p x 100% NEU V elements and (1 p) x 100% POS n V elements for positive sentence generations ((1 - p) 100% NEG n V elements for negative sentence generations). Similar to the use of the mixing ratio in Section 2.2, we intend to create of tasks with diverse difficulties herein via varying p. We generate synthetic sentences by completing any of the starting tokens {"There", "I", "You", "She", "He", "It", "They", "The"). We print some generated sentence examples below:
POSITIVE
"She's a sweet and kind girl."
"The one thing that you have to do is look for other people."
"here are also a number new content that have been rolled out in recent times." "
"I had a lot of fun with this design."
"She was one several of several hundred people in the group to speak out against the police and their use of force."
"You are very close to the truth.... if you are one of the first to see what is being done, that is very much a sign of an error... you have to be very clear that it is good thing that you are doing what you have to do....."
NEGATIVE
They were the worst of the worst."
"She has no other option." "
"There's no question that the new and aggressive international community is headed for bad start with its future in mind."
"They do not want to see you there."
"There's some real bad blood out there."
20

---

Published as a conference paper at COLM 2024
"I just want to make sure that are talking about our state government."
Discussions. Using LM-generated synthetic test sets, the rest of robustness-accural evalu- ation follows Section 2.3 and 2.4. We calculate the SynTextBench scores from LM-generated synthetic sentences and find that the Pearson correlation coefficient between these scores and the actual downstream task performance is 0.603++0.011. This in contrast to the higher correlation coefficient of above 0.9 observed from the LM-free synthetic sentences discussed in Section 2.2, as shown in Table 1.
LM-frre
LM generated
Average percentage of positieeneggaive
0.0
0.2 0.4 0.6 0.8
Figure 3: The average percentage of vositive/negation words in the generated labeled posi- tive /egaative synthetic sentences. With an increasing mixing ratio p, we aim at configuring the task to be harder (data to be more mixed). While the percentage of positive/negative words does decrease in both LM-free synthetic sentences and LM-generated synthetic sentences, we have more control over LM-free generations in generating tasks at various difficulty levels (various y-axis values).
A.3 SentiWordNet 3.0 synsets
We drop columns POS, ID, GLOSS in the examples for easier illustration. By performing the procedure on synsets in Table 5, we obtain a positive word list (able, living, accurate, concrete, active), negative word list (unfaithful, unable), neutral word list facroscopic, straight).
Table 5: Examples of synsets in SentiWordNet 3.0.
SynsetTerms
PosScore
NegScore
Senset
PosScore
NegScore
able#1
0.125
unable#1
0.75
acroscopic#1
0
0
unquestioning#2
0.5
0.5
living#3
0.5
0.125
concrete#1
0.625
0.25
accurate#1
0.5
0
straight#5
0
0
unfaithful#4
0.5
active45
0.5
0.125
21

---

Published as a conference paper at COLM 2024
A.4 Task feasibility
accuracy
0.7
0.6
random guess
0.5
0.0
0.2
0.4
0.6
0.8
Figure 4: The reference accuracy given by SentiWordNet sentiment analysis. With an increasing mixing ratio p, the task becomes harder and the reference accuracy also shows decreasing trend.
A.5 Histograms of synthetic datasets versus English Wikipedia corpus
wiki-en
positive
negative
30.0%
20.0%
10.0%
0.0%
20
40
60
80
100
120
140
Length
Figure 5: The histograms of sentence lengths in the English Wikipedia corpus (stop words removed) and the constructed synthetic corpus poosiivve/negatiin sentences).
A.6 List of stop words
('must', 'meanwhile', 'among', 'same', 'you', 'formerly', 'already', 'take', 'he', 'thereupon', 'done', 'anyhow,, 'almost', 'ca', 'rgarddngg, 'will', 'mostly', ''say', 'again', 'forty', 'seemed', "still', 'they', "re', 'seem', 'latter', 'why', 'hers', 'thereby', 'themselves', 'your', 'nine', 'be- come', 'may', 'beyond', 'it', 'back', 'our', 'himself', "m', 'via', 'we', 'seems', 'throughout' 'yourself', 'bottom', 'only', 'whereby', 'move', 'else', 'front', 'within', 'after', 'every', 'quite' "hereby', 'now', 'since', 'became', 'herself' 'behind', 'any', 'those', 'used', 'indeed', "ve', 'first', 'moreover,, 'ourselves', 'she', 'should', 'her', 'various', 'few', huundred' 'whoever', "give', 'latterly', 'between', 'in', 'most', 'make', 'sixty', 'therefore', "s", 'hence', 'amount',
'saa'''again',
22

---

Published as a conference paper at COLM 2024
'otherwise', "m', "re', "s', 'are', 'could', 'along', 'ours', 'of', 'that', 'everywhere', 'during', "his', 'then', 'fifty', 'namely', 'when', 'around', 'all', 'keep', 'these', 'Ii', 'third', 'being', 'thus', 'more', "s", 'is', 'where', 'further', 'them', 'towards', 'next', 'and', 'a', 'does', 'here', 'ten', 'whom', 'except', 'myself', 'somehow', 'ever', 'enough', 'there', 'mine', 'other', 'so', 'hereupon', 'who', 'eight', 'one', 'hereafter', 'amongst', 'seeming', 'its', 'each', 'sometime', 'this', 'me', "II', 'until', 'him', 'because', 'many', 'anyway', 'part', 'from', 'have', 'over', 'to'' "rre", 'becomes', 'too', 'as', 'name', 'whence', 'whole', 'herein', 'everything', 'against', 'call', 'upon', 'both', 'i, 'whenever', 'across', 'anywhere', 'six', 'us', 'thereafter', 'also', 'former' 'whither', 'whose', 'such', 'really', 'was', "d', 'someone', "ve', 'eleven', 'wherein', 'yours', 'by,, 'their', 'beside', 'or', 're', 'has', 'off', 'which', 'put', 'whether', 'per', 'four', 'whereafter" 'often', 'doing,, 'had', 'out', 'some', 'fifteen', 'others', 'once', 'somewhere', 'either', 'besides', 'though', 'been', 'do', 'very', 'thru', "go', 'please', 'sometimes', ""I", 'perhaps', 'whereupon' 'whatever', 'about', 'for', 'itself, 'thence', 'at', 'how', 'made', 'three', 'might' 'another', 'did', 'alone', 'elsewhere', 'toward', 'were', 'would', 'due', 'what', 'an', 'wherever', 'be', 'can', "something', 'side', "'d", 'with', "'m", 'am', 'therein', 'into', 'through', "'ve", 'everyone', 'on', 'my', 'even', 'own', 'see', 'several' 'two', 'afterwards'' 'show', "d", 'beforehand', 'nowhere', 'becoming', 'last', 'onto', 'the', 'yourselves', 'five', 'anyone', 'together', 'before', 'always', "get', 'using')
A.7 Synthetic sentence examples and discussions
POSITIVE
perfectibility lotus-eater shine shine health. care health care pleasant-tasting"
"convincingly gruesomely gruesomely convincingly deserve feeder exhaust exhaust debonaire stuffily stuffily anne sexton wholeness wholeness rarefy conformable pretension pretension"
"smarmily smarmily fairnsss covetously infuse soothing subtly subtly soothing"
right.way absoluteness absoluteness" reeasonn pleasurable sleekness cryptographically cryptographically cor- rect delineate sink in authenticated"
preciios grace the right way the right way
"perfectibility lotus-eater shine shine health_care health_care pleasant-tasting"
NEGATIVE
"unpleasant unpleasant mortal sympathetic dead dead choker nubbly fallout"
'counterrevolutionary apprehensive thunderclap unskilled unskilled thunderclap apprehensive cheat shanny shanny cheat counterrevolutionn smooth smooth de- cayed decayed imagine imagine loser unpicturesque unnaturalized unnaturalized unrelieved unrelieved unhewn"
unpleesant unpleasant mortal sympathetic dead dead choker nubbly fallout"
jostling weka offend engorged fouled fouled engorged intermittence space im- paction impaction space intermittence dishonesty disgustingly"
"blindly blindly"
"second class criminal possession lousiness nonextensile linanthus dianthiflorus nonarbitrary regular foolishness stabbing"
Discussions. As we mentioned earlier the paper, the inclusion of the parenthesis to guarantee we test for the linguistic structures, whose importance is repeatedly advocated in literatures from both machine learning and cognitive science. Therefore, when building synthetic test for the linguistic structures, also follow the parenthesis and thus have non-grammatical test sets.
We would like to motivate their use based on the following example of sentiment analysis in food reviews. Upon seeing the review "love love fantastic!" food review, reasonable language model should recognize the entailed positive sentiment, even though the sentence
23

---

Published as a conference paper at COLM 2024
is non-grammatical. In our framework, to test the other basic skill for language acquisition in a systematic and scalable manner, we put words associated with binary labels (positive and negative) in the synthetic sentence and test sentence embeddings of LMs in identifying the words for sentence classification. Related to our setups herein, Krishna et al. (2021) also studies range of summarization tasks from nonsense documents, in which a task is also designed to classify whether there are keywords indicating positive or negative sentiments (Krishna et al. (2021), Figure 1). Additional evidence of the usage of non-grammatical sentences can be found in Bhatia et al. (2023), where authors also exploit non-grammartical synthetic sentence (Bhatia et al. (2023), Appendix A) for constructing Gaussian logistic regression problems in improving reasoning ability in LMs, which manifests the value of non-graan non-grammatical language in learning/ aarriig/testing basic skills. Our high correlation with real-world tasks further suggests that better understanding of the synthetic sentences indeed implies better performance on real tasks. By construction, our framework is not limited to sentiment analysis as one can readily change the base lexicon to test how LMs identify words describing other notions. For example, if we use the moral foundation lexicon, one can test how each LM identifies words that describe care, fairness, loyalty, authority, and sanctity.
A.8 Robust Bayes optimal classifier and proofs
To motivate our findings, we first plot the Bayes optimal robust classifiers together with the Bayes optimal classifier in three 2D cases in Figure 6. From the as the direction of μ is in parallel to one of the two eigenvectors, the robust Bayes optimal classifiers would overlap with the Bayes optimal classifier.
plot, we see that as long
Counsian
20 Couss
(a) No alignment
(b) ş || 1
(c) z 8/2
Figure 6: Three 2D examples of the Bayes optimal classifier and robust Bayes optimal classifiers with different magnitudes of expected perturbation Figure 6(a) no no alignment between the mean vector μ and the eigenvectors. Figure 6(b) and Figure 6(c) is parallel to the eigenvector corresponding to either of the two eigenvalues.
To generalize the result, we prove the following theorem that specifies a sufficient condition for all e-robust Bayes optimal classifiers to overlap with each other (including 0, i.e. Bayes optimal classifier). Intuitively, if the e-robust Bayes optimal classifiers overlap with the Bayes optimal classifiers, then there is no robustness-accuracy trade-off.
Result A.1. The e-robust Bayes optimal classifiers overlap for all e if the vector differ- ence ş between the centers of the two gaussians lies completely within a degenerate sub- space of the eigenspace of the covariance matrix, i.e. with eigenpairs {(Ax, Ux), k E [m]], for V i,j { : d + 0, n² Dk + 0}, A; = A== = A.
Proof. Let Vi,..., Un and An be the orthonormal eigenbasis and the correspond- ing eigenvalues of the covariance matrix Σ, then we have Σ 1 Σ x daoo Follow-
ing Dan et al. (2020), we that the e-robust classifier is given as sign w x, where wf =--- (μ 25(1)) and
(p) = arg min|| 1
|
24

---

Published as a conference paper at COLM 2024
Let | = - a and we re-parmmeteii Li b Then,
(a; b₁)
Σ bf where be = (b mig Σ
A
3 17 i=1
By using the Lagrange multiplier "e with first-order optimality condition, we see that V i
by -4a + tebf = a; by
by
(3)
A
A
MrT
and Σ (bg) I e². In order for all the robust classifiers overlap we need w / /||w>|| to the independent of e. That is,
2
of A
We
E 1 VEL (b9)
1
Eies
100
(b9) VE'es (b7)²
Ei=1
where the S in the last equation denotes the set of indices for which aj + For V with a; from equation 3, we clearly have bf =0 O.
The condition y lies completely within a degenerate subspace of the eigenspace of E is equivalent to saying A Aj À for v j,j E S. In this case, we see that for 4 i E S,
e² N
Σ (bg) i=1
[b9)
Σ a7
ATe 1
1
iES
iES
so
by
of dj.
So, we get by me aj where me
Lies
min 1,
Viee
we
Lies bfu;
Lies mcase;
Σ iCS
Vics(a;)
Eies (bg)²
me
Lies a?
which is independent of
Result A.2. Consider the robust Bayes optimal classifier!, fe, for Privadaa with class prior Pyy = 1) = T, P, Py -1) = - T, is in the following form
fe(x) sign { (x-
M1 122
p(1 e/11p112) 9/2
where f - #2 and q - Inf (1 - T}} For any sample x, fe gives the lower bound on
the decision margin 8
M1 + U2 T
x 5
á1 - e/(l[]122 - 9/2 =0
2
+ 8T (T e/ c/11a|12) = 9/2 (x 11 + 122 T
p1 e/11
2
[x - /1 TT - e/ (17l11 - 9/2l
|
[ (1 e/ e/11
which then yields the worst-case bound
||||2 min |01| (xx H1 11 TT e/ 11112 9/2|
| - c/ ( 12)12
Dobriban, F., Hassani, H., Hong, D. and Robey, A., 2020. Provable tradeoffs in adversarially robust classification. arXiv preprint arXiv:2006.0516i.
25

---

Published as a conference paper at COLM 2024
Since the bound | is subject to the positions of two Gaussians, scale the bound by the distance from Gaussian centers to the classifier. We note that, since the class are imbalanced, the distances from the two Gaussian centers to the classifier fe are different, L.. | and [ ((-A(1 -
respectively. We hereby take their average ((1l-//122))
2
as the scaling factor and obtain
||Al|2 = 1xx "1 y 711 e/l (112) 9/21
2 fT - e/ || //2) p|1 (2)2 + 9/2
[11 - < [ 2)112
(x - - fT - e/11 9/2)
f11 - | F' F(1
+
A.9 Goodness function
BERT pase
BERTage
12
RoBERTa
DiffCSE-B
DiffCSE
T5aas
T5urge
STS
GPT
0.6
0.0
65
70
75
80
85
90
95
100
Figure 7: The goodness function s(a) of nine pretrained LMs. The SynTextBench score is calculated by the area under the curve.
26

---

Published as a conference paper at COLM 2024
A.10 Complete results for Section 3.2
Table The detailed SentEval linear probing performance. For STS tasks, we report Spearman's correlation (%), and for Transfer task, we report the standard accuracy (%).
STS taoks
Transfer taoks
Midlel
STS12
51513
51514
51313
51516
S15-8
SK..KK
MR
CK
MIXJA
SST
TRFC
MMP
BURT
52.11
55105
07.91
33.35
02.05
52.56
85.56
55.15
89:92
B5.15
8935
7100
B3:50
DiACSE-B
76.21
75.51
67.20
84.46
88
DERT
58.06
58.18
63.78
64.14
71.83
68.94
60.17
58.77
80.54
88.34
93.04
89.73
81.27
85,8
67.36
82.78
15trge
56.34
62.99
63.50
71.36
67.88
59.67
58.02
70
86.86
93.55
90.43
80.72
82.8
68.75
82.36
RoBERTabee
57.28
55.21
59.76
61.64
58.55
61.63
81.08
86.91
95.63
89.52
88.25
91.6
74.49
83.83
DiICSF-R
70.54
95.2
87.92
89.4
88.19
48.16
78.01
74,32
82.83
81.50
86 14
85.95
86.04
79.76
88
91.81
94.
91.00
90.83
95.8
74.26
90.17
Table 7: Pearson correlation comparison between real-data-free evaluation methods and the average linear probing accuracy the real-world tasks included in Table 6. Since the smaller the Val loss, MDL, SDL and eSC, the better, we add a negative sign in front of them when calculating the Pearson correlation coefficient.
Tha
Pownsm
Realifi
MEX
5302
4755::22 29
5822-367
7818e119
67246222
539e-181
4773=256
56382366
4033:330
0,57 109
3000
2641.12
2011
48221120
346
2663
039010110
3666-3
360620
Dam Jawirsttinchh
0127 16200
0.0 1116-110
0116 (UM3)
BELS CHP
3019 1010
0100
1600
DKON LOILL
1040
SDL c-1
3352-99
3141-252
656
109442426
$2512489
5562:175
634
0,00
702.0
25221B
70720
SanTestBench
1112200000
015650001
034450002
0137-0001
319
LIN65000
065560003
T630s
Knw
23 15:2
9.5
72:01E1005
21 01 92
2.5
UXnEEn DOT
MEX
355402436
14099-775
20001500
2560642767
18635-1177
15445-34P
16744-440
0.0620 182
ISC cml
.4745:5
L
14745e)
1474220
1P4550
14745
LEN4200
1.16Hn0 300
064-000
0151=0001
01:1-1000
218050 100
00065000
0,67-0000
0227+0001
OMRReO DOC
MEN
2234242244
2001
2007 201
1666
SWACL 1550
22151155
20032
SDL c-l
35417=382
977
17377-253
S12804 1940
1710
14775-500
15014:489
9422-:95
0076-106
0ALL-0 DOS
Symiestteenhh
0.121111333
3101101000
01862110000
Comparison Models on SentEval Tasks
accuracy
robustness
80
14
12
accuracy
robustness
40
2
BERT base
iERT
RobaRTatase
DiffCSE-B
DifiCSE-R
T5psse
T5urge
ST5
GPT
Figure 8: The accuracy and robustness (average number of perturbed words) performance of pretrained models on SentEval tasks.
27

---

Published as a conference paper at COLM 2024
A.11 Complete results for Section 3.3
Table 8: The detailed SentEval linear probing performance on decoder models. For STS tasks, we report Spearman's correlation (%)) and for Transfer tasks, we report the standard accuracy (%).
S1S13
MFQA
avp
LLMMA-715
10.51
9.68
5.85
2.54
3.27
15.38
15.01
71.23
75387
87.39
81.54
77.59
62.80
64.12
64..55
LLaMA-136
12.08
7.05
2.85
-0.84
7.38
3.50
10.93
70.88
77
68.04
61.53
76.77
64.00
63.19
63.78
LLaMA-306
7.04
16.29
5.39
3.12
5.04
16.02
14.77
71.99
78.12
88.81
62.53
76.84
61.00
60.70
64.53
1LMMA-2-76
11.95
22.85
10.85
16.31
44.12
20.13
17.17
91,07
91.55
97.30
89.
91.78
96,80
67 67.55
76.13
LMMA-2-138
21.90
33.07
50.67
50.83
32
97.70
89.72
97.20
20.38
OFT-138
91
OFT-308
38,83
Table 9: Pearson correlation comparison between real-data-free evaluation methods and the average linear probing accuracy the real-world tasks of decoder models. Since the smaller the Val loss, MDL, SDL and eSC, the better, we add a negative sign in front of them when calculating the Pearson correlation coefficient.
Name
LLaMA-7B
LLaMA-138
LLaMA-30B
LLaMA-2-7B
LLaMA-2-13B
OPT-138
DPT-30B
Pearson 1,0
Reallide
61.55
63,75
64.53
76.13
75.31
74.72
75.41
0.149492
SDL
4135.77
3321.93
3090.58
6791.69
579191
4291.41
4005.95
-0.518
ISC, c -1
7372
7372
7372
7372
7372
7372
7372
SynTextBench
0.062
0.027
0.048
0.097
0,075
0.089
0.093
0.871
Table 10: The detailed subset SentEval in-context learning accuracy on decoder models.
Transfer tasks
Models
CR
MR
MPQA
SUB]
SS12
MRP
avg.
IL.MA-BB
85.35
90.49
74.34
48.47
88.47
53.86
73.53
LLaMA-13B
91.07
62.78
70.07
50.02
69.74
66.20
68.31
LLaMA-30B
91.97
92.60
83.77
50.01
95.83
66.26
80,07
LLaMA-2-7B
90.83
53.25
47.06
81.60
71.00
66.49
68.37
LLaMA-2-13B
91.84
91.92
80.26
52.73
95.55
66.49
79,80
OPT-13B
90.01
49.85
76.99
70.49
OPT-30B
90.78
82.04
63.56
50.00
87.10
66.61
73.35
28

---

Published as a conference paper at COLM 2024
A.12 Experimental details
When we calculate the correlation between real-data-free evaluation methods and real- world task robustness-accuracy performance, we need to aggregate two metrics accuracy and robustness. For this purpose, we can obtain a ranking of the models according to the accuracy measure, R1, and a ranking of the models according to the robustness measure, R2- We aggregate two rankings by the simple and commonly-used mean aggregation which yields the overall ranking of models based on securacyroobustees performance, R On the other hand, we can obtain another ranking of models based on one of the real-data- evaluation methods (e.g. Val loss, MDL, SDL, eSC, SynTextBench), R. Lastly, we calculate the Pearson correlation coefficient between and Rie--
Moreover, when we calculate the robustness measures, we only perform attacks Transfer tasks as they are classification tasks where adversarial attacks are well-defined. Since use the average number of perturbed words by PWWS attacks (Ren et al., 2019) the robustness indicator, we also excluded MPQA and TREC due to their short sentence lengths (MPQA and TREC average sentence lengths are 3.03 and 6.48, respectively). PWWS attacks focus on the text adversarial example generation that could guarantee little semantic shifting and therefore rarely cause ground truth label change (also lexical and grammatical correctness). To meet the semantic constraint, PWWS replaces words in the input texts with synonyms and replace named entities (NEs) with similar NEs to generate adversarial samples. Synonyms for each word can be found in WordNet, large lexical database for the English language. NE refers to an entity that has a specific meaning in the sample text, such a person's 's name, location, organization, or proper noun. Replacement of an NE with a similar NE imposes a slight change in semantics but invokes no lexical or grammatical changes.
We list the robustness results the following table:
Table 11: The robustness (average number of perturbed words) of pretrained representations on Transfer tasks.
Models
SUB]
SST
avg.
BERT base
14.48
13.99
20.2
15.07
5.45
13,838
DifCCSE-B
14.46
14.7
18.64
15.19
6.39
13.876
BERT
14.3
14.22
19.87
15.46
5.26
13.822
TEease
12.71
12.82
16.8
13.66
5.05
12.208
T5
13.67
14.28
16.93
13.82
12.774
RoBERTa
16.4
18.35
20.74
17.26
7.12
15:974
DifCSE-R
GPT
12.53
13.11
15.75
13.52
5.17
12.016
ST3
18.36
14.22
We also list the ranking of models from different metrics in the following table.
Table 12: Ranking of models from different metrics -8192
Name
BERT
DiffCSEBB
BERT
T5
T5agg
RoBERTapase
DifCSE-R
GPT
STS
Overall accuracy
9
SIS accuracy
Transfer accuracy
3
7
Robustness
Val loss
MDL
6
4
SDL, e=1
6
eSC, e=1
5
SynTextBench
2
9
For example, to calculate SynTextBench correlation with robustnes-and-acuuaay perfor- mance, calculate the Pearson correlation between (row "Overall accuracy" row "Ro-
Wald, R., Khoshgoftaar, T.M. and Dittman, 2012, December. Mean aggregation versus robust rank aggregation for ensemble gene selection. In 2012 11th international conference machine learning and applications (Vol. pp. 63-69). IEFF.
29

---

Published as a conference paper at COLM 2024
bustness") / and "SynTextBench". To calculate SynTextBench correlation with robustness- and-STS accuracy performance, we calculate the Pearson correlation between (row "SIS accuracy" row "Robustness") / and "SynTTextBench" To calculate SynTextBench cor- relation with Sruustness-an-Transiee accuracy performance, we calculate the Pearson correlation between (row "Transfer accuracy" row "Robustness") 2 and "SynTexttenchh We note that in all our results prior to Table 12, we always infer the correlation in individual runs before we take an average over all trials. Different from that, the rankings from Val loss, MDL, SDL, eSC, and SynTextBench in Table 12, are inferred from the average metric results over 3 trails for an easier illustration. Therefore, the ranking correlation suggested by the table might have some deviation from what is shown in Table 2.
30


Reviews:
[{"review_id": "oHZVfpbMXZ", "reviewer": "Reviewer_d3w1", "summary": "Summary:\nThis manuscript introduces \"SynTextBench,\" a new framework for generating synthetic datasets to evaluate the robustness and accuracy of sentence embeddings in large language models (LLMs). By leveraging sentiment lexicons to construct sentences with varying levels of difficulty, the method aims to provide a safer, more transparent, and controllable way of evaluating LLMs. The framework seeks to create various difficulty levels through the manipulation of word mixing ratios and assesses LLMs based on their performance on these synthetic datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and positive comments. We appreciate the recognition of our innovative approach and focus on data leakage. We will address all questions and welcome further discussion if needed.\n\nAnswer to Q1: We chose sentiment polarity as it tests the ability of identifying words with polarity. Capturing sentiment polarity ensures the LM understands nuanced semantic information. Although SynTextBench can use any labeled lexicon, SentiWordNet is practical since most words in real tasks are included, e.g. 81% of CR words are in SentiWordNet. SynTextBench also allows controlling synthetic task difficulties by tuning polarity levels.\n\nAnswer to Q2: SynTextBench focuses on studying the sentence embedding space of an LM, commonly evaluated by sentence classifications (e.g., SentEval). As effective sentence embeddings are crucial for many NLP tasks[1-2], we indirectly measure other tasks through sentence embedding quality. For specific applications, specialized synthetic tests are needed, and we hope this work serves as a foundation for such extensions.\n\nAnswer to Q3: SynTextBench supports any labeled lexicon, ensuring applicability across languages and cultural backgrounds, provided labeled lexicons exist (e.g., Spanish lexicon[3]). If not, synthetic sentences can be translated from popular languages like English. To assess complex linguistic phenomena like sarcasm and humor, SynTextBench can be extended by adding templates wrapping synthetic sentences, such as distinguishing between \"'AAA' is positive\" and \"No way, you believe 'AAA' is positive?\" This extension is future work.\n\nAnswer to Q4: The conventional approach for evaluating LMs involves various benchmarks in different languages, and SynTextBench eases reliance on popular languages with labeled lexicons, such as English, Spanish, and Mandarin. For low-resource languages, future strategies may involve creating synthetic datasets using transfer learning from high-resource languages and augmenting data through paraphrasing/translation.\n\nScalability clarification: SynTextBench only requires inferences on the synthetic samples and does not rely on real datasets. Our experiments have shown that 4k samples are sufficient.\n\n[1] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data\n\n[2] Diffcse: Difference-based contrastive learning for sentence embeddings\n\n[3] CSL: a combined Spanish Lexicon-resource for polarity classification and sentiment analysis", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1. Innovative Evaluation Approach: \"SynTextBench\" introduces a new method for generating synthetic datasets, offering a novel pathway for testing LLMs under controlled conditions.\n2. Focus on Data Leakage: The method provides a solution to the data leakage issues in the evaluation process, helping to protect data privacy.\n3. Dual Assessment Metrics: The paper demonstrates the potential of the framework to assess both the robustness and accuracy of LLMs.", "reasons_to_reject": "Weaknesses:\n1. Limited Application Scope: The paper primarily considers only evaluating sentence embeddings from LLMs, which while important, is a small part of the overall evaluation landscape of LLMs. However, the current evaluation focuses mainly on sentence classification tasks and has not yet addressed other NLP tasks like question answering or summarization, which may limit its broader application.\n2. Insufficient Coverage of Linguistic Phenomena: Although a complex sentence generation method is proposed, the framework still lacks effective handling of more complex linguistic phenomena such as sarcasm and humor.Whether this metric can be leveraged for other tasks (especially for a different class of tasks) needs to be demonstrated in my opinion.\n3. Dependency on Sentiment Lexicons: The reliance on sentiment lexicons for generating synthetic datasets raises concerns about the framework's adaptability to different languages or dialects, potentially limiting its effectiveness in multicultural or multilingual contexts.\n4. Scalability Issues: The paper does not clearly address how the framework scales with increasingly large models or datasets, which is vital for its application in real-world scenarios where data and model sizes are continuously growing.", "questions_to_authors": "Questions:\n1. What is the reasoning behind selecting sentiment polarity as the foundational task for creating the evaluation benchmark? Specifically, why is it believed that capturing sentiment polarity leads to more practical sentence embeddings, and why would failing to do so (especially in non-grammatical sentence structures) result in poorer performance in real-world downstream tasks?\n2. Have the authors considered applying this framework to other NLP tasks beyond sentence classification? If so, what are the potential directions for expansion?\n3. Given the framework's reliance on sentiment lexicons, how do the authors ensure its applicability across different languages and cultural backgrounds?\n4. Are there any specific considerations or adjustments for low-resource languages or less mainstream linguistic structures?\n\n\nThis paper proposes a promising new framework for assessing the accuracy and robustness of large language models. Although it primarily focuses on sentence classification tasks, its correlation with real task performance indicates practical applicability. However, to enhance its universality, it is recommended that the authors explore applying this assessment framework to a broader range of NLP tasks. Discussing and experimentally validating these potential applications within the paper could significantly enhance its impact and applicability.", "extra_scores": {}}, {"review_id": "2bpZjYxRKz", "reviewer": "Reviewer_t26e", "summary": "LLMs have shown remarkable capabilities on multiple NLP tasks, but there have been consistent concerns about the evaluation strategy used for assessing  their actual capabilities. Towards that front, a major issue pertains to test-set contamination. Test-set contamination can primarily be categorized into the following headings: \n\n(a) the evaluation dataset might actually have been a part of the pretraining dataset for LMs \n\n(b) data-leakage through querying LMS via API calls.  \n\nThis paper tries to address the issue of test-set contamination in the evaluation of LLMs. To that end, it additionally proposes a new testbed for LM evaluation via synthetic data. The creation of this testbed is inspired by infant learning skills during language understanding/acquisition. - identify words, understand linguistic structures. The resulting test set called SynTextBench is constructed without reliance on any LMs. The focus is on benchmarking LM sentence embeddings in terms of accuracy as well as robustness.  \nThe final measure of LMs abilities is done using a SynTextBenchScore, which is theoretically well grounded.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and positive comments. We are pleased to see the reviewer finds our test dataset easy to construct, our comparisons comprehensive, and our SynTextBench-score theoretically well-grounded. We are committed to addressing all questions and welcome further discussion if needed.\n\nAnswer to Q1: \nWe motivate ungrammatical sentences by sentiment analysis in food reviews. For instance, \"love love fantastic!\" in a food review should still be recognized as positive despite its non-grammatical nature. Related studies also utilize non-grammatical sentences, as seen in [1] for sentiment classification from nonsense documents (Figure 1) and in [2] for Gaussian logistic regression problems to improve LLM reasoning (Appendix A), which manifest the value of non-grammatical language in learning/testing basic skills. \n\nAnswer to Q2: \nWe thank the reviewer for making the suggestion and we will revise accordingly!\n\nAnswer to Q3:\nSynTextBench focuses on studying the sentence embedding space of an LM, which were commonly evaluated by sentence classifications (e.g., SentEval). As effective sentence embeddings are crucial for many NLP tasks[3-4], we indirectly measure other tasks through sentence embedding quality. For specific applications, specialized synthetic tests are needed, and we hope this work serves as a foundation for such extensions.\n\nAnswer to Q4:\nThanks for the suggestions. We will include a worked-out example in our paper for better understanding!\n\nAnswer to Q5:\nWe will correct the typos!\n\nAnswer to Q6:\nThe reviewer's intuition on the idealized LM is correct - ideally LMs should latch on to any positive/negative words, and have uniformly high accuracies regardless of the neutral word percentage. However, this is not always true in practice, highlighting the need for our metric. Difficulty levels challenge the LM's ability to distinguish sentiments amidst neutral token noise. Higher proportions of neutral tokens increase ambiguity, making it harder for the model to rely solely on sentiment-bearing words. In SynTextBench, difficulty levels are adjusted by varying the mixing ratio $p$ from 0 to 1.0.\n\n[1] Does Pretraining for Summarization Require Knowledge Transfer?\n\n[2] TART: A plug-and-play Transformer module for task-agnostic reasoning\n    \n[3] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data\n    \n[4] Diffcse: Difference-based contrastive learning for sentence embeddings", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides an easy-to-construct mechanism for test dataset for evaluating LM abilities.\n    \n2. Comprehensive comparison of multiple models including encoder only, decoder only and encoder-decoder frameworks.\n    \n3. Theoreticaly well grounded approach for robustness measure using optimal classifiers and token edits.", "reasons_to_reject": "1. Ungrammatical sentences used for evaluation hinders interpretability for humans. While theoretically it may improve upon the evaluation working with separability of the encoded sentences, the pragmatic use for NLU is limited, despite having high correlation coefficients.\n    \n2. Most of the comparison of different baselines is moved to the appendix. Verbosity of the text should be worked upon and some comparison tables with their associated conclusions should be moved to the main paper.\n    \n3. Sentence separability only measures one of the aspects of LMs, which might not reflect its main goal. It is not clear why a sentence separability measure would serve as a better evaluation proxy for language understanding.\n    \n4. It would help if the authors could provide a worked out example in section 2.2 to elucidate the algorithm for sentence (technically just a set of words) construction.", "questions_to_authors": "Abstract:  \n“pertrained” -> pre-trained\n\n“comparing their performance on a set of our synthetic datasets with varying difficulty levels”: \nHow is this difficulty decided ? Why should increasing neutral tokens increase difficulty of a task given that the sentence is not well-formed, and the model should ideally latch on to any of the positive/negative words.", "extra_scores": {}}, {"review_id": "TwXVls1dmM", "reviewer": "Reviewer_mBig", "summary": "This paper proposes a new framework (SynTextBench) to evaluate LMs’ performance. SynTextBench is a theoretically grounded framework which generates synthetic data for LM evaluation. It does not rely on LMs’ generated data, so it can alleviate the risk of data leakage problems. SynTextBench provides a configurable lightweight testbed that can evaluate LMs’ performance under various difficulties. The proposed quantifiable metric for evaluating the robustness and accuracy of LMs is very effective. Experiments demonstrate the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and positive comments. We appreciate the recognition of our idea's clarity, method's reasonableness, and framework's effectiveness. We will address all questions and welcome further discussion if needed.\n\n1. Ungrammatical sentences & separability: We motivate ungrammatical sentences by sentiment analysis in a food review - \"love love fantastic!\" should be recognized as positive despite its ungrammatical nature. Related studies also utilize ungrammatical sentences, e.g. [1] does sentiment classification from nonsense documents (Fig 1) and [2] uses Gaussian logistic regression to improve LLM reasoning (App. A), which manifest the value of ungrammatical language in learning/testing basic skills.  \nSynTextBench focuses on the sentence embeddings of LMs, which were commonly evaluated by sentence classifications (e.g. SentEval). As effective sentence embeddings are crucial for many NLP tasks[3-4], we indirectly measure other tasks through sentence embedding quality. For specific applications, specialized synthetic tests are needed, and we hope this work serves as a foundation for such extensions.\n\n2. Sec 2.3: We will elaborate on each equation to provide context and motivation, ensuring the concepts and significance of our mathematical models are clear.\n\n3. Reference: Thank you for pointing out! We will include them in our revised manuscript.\n\n4. Transforming sentence embeddings: Transforming sentence embeddings into an isotropic Gaussian distribution mitigates anisotropy and helps design an evaluation metric independent of the robustness parameter $\\epsilon$ - we proved in App. A.8 that, when $\\tilde{\\mu}$ lies within a degenerate subspace of the covariance matrix's eigenspace, $\\epsilon$-robust Bayes optimal classifiers overlap for all $\\epsilon$. \n\n5. Effectiveness of validation loss: We compared SynTextBench with validation loss on our synthetic sentences (Tab 1-3, 6, 8, row \"Val loss\"). While validation loss shows lower correlation (worse) than SynTextBench, it remains a strong baseline. Evaluating validation loss on pre-train data may be infeasible since LMs are often trained on different data.\n\n[1] Does Pretraining for Summarization Require Knowledge Transfer?\n\n[2] TART: A plug-and-play Transformer module for task-agnostic reasoning\n    \n[3] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data\n    \n[4] Diffcse: Difference-based contrastive learning for sentence embeddings", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed framework is very effective and can avoid potential bias introduced by LMs’ generated data.\n- The experiments show promising results of the proposed method.\n- The idea is clear, and the method is reasonable.", "reasons_to_reject": "- My primary concern with this paper is the **interpretability** of the proposed method.\n  - Although the authors address this topic in Section 2.2, the explanation provided does not convincingly demonstrate that non-grammatical test sets can effectively evaluate LMs’ performance. While sentiment analysis tasks may not heavily depend on grammatical integrity, other NLP tasks could be impacted differently. The readability of synthetic data is compromised, thereby reducing the interpretability of the generated test set.\n  - The paper does not clearly establish how the language model’s performance on this test set correlates with its overall language processing capabilities. It is questionable whether producing more discriminative sentence representations equates to improved LM performance. The relationship between these elements remains elusive and warrants further exploration.\n- Upon a detailed review of the experimental results, I observed a notable finding: for auto-regressive LMs (such as GPT, LLaMA, and OPT), the Pearson correlation coefficients between the real-data-free evaluation metric and actual data accuracy are significantly higher compared to other LMs, as indicated in Tables 3 and 6. Numerous studies have suggested that training loss is an effective indicator of LMs' performance. This paper could benefit from a more thorough comparison between the proposed method and the LMs’ loss metrics to bolster this claim.\n- The paper would improve if it provided more motivation and explanation, particularly in Section 2.3, where numerous equations are presented without adequate elucidation. This makes it challenging for readers to grasp the underlying concepts or the significance of these mathematical models in the context of the discussed methodology.\n\nMissing reference:\n- Don't Make Your LLM an Evaluation Benchmark Cheater, Zhou et al.\n- Benchmarking Benchmark Leakage in Large Language Models, Xu et al.", "questions_to_authors": "1. In Section 2.3, why we should transform the sentence embeddings into an isotropic Gaussian distribution? Is it necessary for the evaluation?\n2. Is there any discussion about the effectiveness of the validation loss?", "extra_scores": {}}, {"review_id": "R6RRhhcQyk", "reviewer": "Reviewer_mAMg", "summary": "This work proposes SynTextBench, a framework for evaluating the accuracy and robustness of LM sentence embeddings. It aims to generate steerable synthetic language datasets and proxy tasks, avoiding the risk of test-data leakage.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort in reviewing our paper. Our paper's main merits are laying a foundation for evaluating LMs with synthetic data and demonstrating how simple compositional synthetic tasks may inform real-data task performance. We are pleased the reviewer finds our dataset useful, pipeline reasonable, and experiments extensive. We are committed to addressing all questions and welcome further discussion if needed.\n    \nAnswer to Q1: SynTextBench focuses on studying the sentence embeddings of LMs. As effective sentence embeddings are crucial for many NLP tasks[1-2], we indirectly measure other tasks through sentence embedding quality. We further note that sentence classification is among the most important use cases for LLM[[ref1](https://ubiops.com/llm-use-cases/)]. As of now, our approach complements existing benchmarks by providing a controlled environment to assess accuracy and robustness, which might be essential for reasoning and planning tasks. We will expand the motivation section to clarify the relationship between the LM capabilities SynTextBench tests and other high-level capabilities, and how they contribute to the overall evaluation of LMs.\n\nAnswer to Q2: The misalignment between human and LLM-generated texts (e.g. the more aggressive emotions in human texts) indeed supports our setup as we do NOT use LLMs to generate synthetic datasets. We also reveal the problem with LLM-generated texts as it doesn't allow accurate controls over task difficulties in App A.2. Comparatively, we utlize labeled lexicons to construct the synthetic data utilizing some preliminary linquistic structure, ensuring straightforward emotion expression and a controlled testbed that isolate specific linguistic capabilities without real-world data confounding factors. Moreover, despite potential pattern differences, our high correlation with real-life tasks suggests better understanding of synthetic sentences correlates well with better performance on real tasks.\n\nAnswer to Q3: We appreciate the suggestion. We currently defer evaluation results of existing LLMs on human and synthetic texts to the appendix. We agree that including these results in the main paper will make the paper more complete and better reflect the testbed's value. We will update the manuscript accordingly.\n\n[1] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data\n\n[2] Diffcse: Difference-based contrastive learning for sentence embeddings", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.the proposed dataset is useful to evaluate the capability of LLMs, and can fill the blank in testing the text generation ability of LLMs.\n2.the whole evaluation pipeline is reasonable, and can test both the accuracy and robustness.\n3.extensive experiments have verified the effectiveness of the dataset for evaluation.", "reasons_to_reject": "1. The motivation of this paper is weak, and it is not clear whether the studied topic (e.g., identifying words and understanding linguistic structures) is an important skill for evaluating LLMs. Actually, existing work has shown that LLMs can well learn the linguistic features about texts. Thus, the most popular LLM evaluation benchmarks mostly focus on the reasoning and planning abilities, as they are more useful high-level capabilities for LLMs.\n2. It is not clear if the synthetic dataset is able to faithfully reflect the capability of LLMs, as existing work[1] also mentions the difference between human and synthetic texts. Actually, it might lead to unalignment issue between real-world human requirement and synthetic test data.\n3. The evaluation results of existing LLMs should be reported in this paper, as the comparison across different LLMs can also reflect the value of the proposed testbed.\n\n[1] Muñoz-Ortiz A, Gómez-Rodríguez C, Vilares D. Contrasting linguistic patterns in human and llm-generated text[J]. arXiv preprint arXiv:2308.09067, 2023.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Pros:

* The paper proposes a novel method to evaluate LLMs, especially to mitigate test set leakage in modern LLMs trained on web scale datasets.
* They provide clear explanation of the method and run experiments on various models to show efficacy of the approach. 
* The paper assesses both the accuracy and robustnesses of LLMs using dual metrics.

Cons:

* The proposed approach may have limited applicability due to the limited nature of evaluating sentence embedding 
* The evaluation of existing LLMs would significantly strengthen the paper.
Decision: Accept