Bridging Information-Theoretic and Geometric Compression in Language Models
Authors: Emily Cheng
Corentin Kervadec
Marco Baroni
Venue: EMNLP 2023 Main (2023)
Keywords: compression
information theory
language models
TL;DR: Intrinsic dimensionality of linguistic data representations tracks information-theoretic coding length under LMs and predicts ease-of-adaptation.

Abstract: For a language model (LM) to faithfully model human language, it must compress vast, potentially infinite information into relatively few dimensions. We propose analyzing compression in (pre-trained) LMs from two points of view: geometric and information-theoretic. We demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding length under the LM. We then show that, in turn, high compression of a linguistic dataset predicts rapid adaptation to that dataset, confirming that being able to compress linguistic information is an important part of successful LM performance. As a practical byproduct of our analysis, we evaluate a battery of intrinsic dimension estimators for the first time on linguistic data, showing that only some encapsulate the relationship between information-theoretic compression, geometric compression, and ease-of-adaptation.

Bridging Information-Theoretic and Geometric Compression in Language Models
Emily Cheng and Corentin Kervadec Universitat Pompeu Fabra / Barcelona {name (lastname)@uup edu
Marco Baroni ICREA Barcelona marco. baroni@upff edu
Abstract
grammatical utterances. That is, it must perform successful form of data compression.
For a language model (LM) to faithfully model human language, must compress vast, poten- tially infinite information into relatively few dimensions. We propose analyzing compres- sion in (pre-trained) LMs from two points of view: geometric and information thherretcc... We demonstrate that the two views are highly cor- related, such that the intrinsic geometric di- mension of linguistic data predicts their coding length under the LM. We then show that, in turn, high compression of linguistic dataset predicts rapid adaptation to that dataset, con- firming that being able to compress linguistic information is an important part of successful LM performance. As a practical byproduct of our analysis, we evaluate a battery of in- trinsic dimension estimators for the first time on linguistic data, showing that only some en- capsulate the relationship between information- theoretic compression, geometric compression, and addattt-daattttion..
Thus, following the line of research that aims to better understand LM behavior (Wei et al., 2022; Zhang et al., 2021; Rogers et al., 2020), in this work we provide initial insights on how they compress linguistic knowledge. We demonstrate empirical link between two types of compression: geometric and cooommationtthooettee In particular, we ask: how, and by how much, do LMs compress linguis- tic data? Furthermore, what are linguistic correlates to compressibility? Is compression a good predic- tor of rapid adaptation? We show that (1) intrinsic dimension (ID) of linguistic data representations under an LM tracks information-theone
coding length; (2) greater data compression predicts ease- of-adaptation in causal language modeling tasks; (3) interpretable linguistic properties such as vo- cabulary size and syntactic structure modulate ID; and (4) different model sizes recover similar ranges of ID. Finally, as practical contribution, (5) we explore different ways to estimate ID of linguistic data, and find only some to capture the relation between ID, coding length, and ease- add-tttaiot....
1 Introduction
To speak a language not to memorize all pos- sible utterances, but to instead extract the finite ruleset and lexicon that generates them (Chomsky, 1986). That is, language, though nominally high- dimensional, can be compressed to comparatively small intrinsic dimension.
Related Work
Causal Language Models State-of-the-art lan- guage models are based on the Transformer archi- tecture (Vaswani et al., 2017). which consists of alternttin fred-forward and self-attention modules (Bahdanau al., 2015). They are trained using self- supervised learning on sequences of tokens, where a token is defined as the atomic unit (e.g., word or a sub-word) fed into the language model. Due to their current ubiquity, we focus on autoregressive models trained causal language modeling ob- jective, that is, next token prediction given context of previous tokens (Brown al., 2020; Radford and Narasimhan, 2018: Zhang et al., 2022).
The recent success of (large) language models demonstrates that artificial neural networks, too, can acquire linguistic knowledge. Current language models (LMs) are Transformer-basee architectures at-scale (Vaswani et al., 2017) that are trained on the conditional distribution of natural language (OpenAI, 2023; Zhang et al., 2022; Touvron et al., 2023; Chowdhery al., 2022) and that are inching closer to human-like linguistic robustness (Brown al., 2020; Liang et al., 2022; Wang et al., 2019b). For an LM to faithfully model language, it must en- code linguistic training data into finitely many vari- ables that allow generalization to infinitely many
Language models are typically measured against human performance on linguistic benchmarks.

---

Evaluation may be done post-finetuning or in a low-shot regime, where the model completes a lin- guistic task given few or zero examples. A variety of benchmarks, such as GLUE (Wang et al., 2019b), SuperGI.UE (Wang et al., 2019a), and Big (Srivastava et al., 2022) have been proposed, evalu- ating, for instance, model performance on textual entailment, question-answeriing
2023). We similarly focus on information-lheoretic coding length of inputs under a pre-trained model, which a simple measure of compression that, to our knowledge, has not been shown to be analyt- ically equivalent to geometric compression. We develop this measure further in section 3.2.
Intrinsic Dimension (ID) Geometric compres- sion is commonly quantified using dimensional- ity reduction techniques. Often underlying these approaches is the manifold learning hypothesis (Goodfellow et al., 2016), or the notion that real- life, high-dimensional data often lie on a low- dimensional manifold. Intrinsic dimension (ID), or the number of degrees of freedom in the data, is the dimension of this data manifold.
semantic equiva- lence, or sentiment analysis. Indeed, human base- lines for GLUE and SuperGLUE have already been surpassed by LMs that contain billions of parame- ters (c.g., PaLM 540B (Chowdhery et al., 2022)).
2.1 Compression in LMs
There is a wide body of work analyzing compres- sion in deep neural architectures. In statistical learning theory, compression has been empirically and theoretically linked to gencralization (Shwartz- Ziv and Tishby, 2017; Arora et al., 2018). More- over, deep learning models are thought to mini- mize description length (Perez et al., 2021; Voita and Titov, 2020; Blier and Ollivier, 2018). In large LMs, implicit compression of neural network parameters has been linked to cuse-of-finetunigg and generalization (Aghajanyan et al., 2021). Our work complements this line of research by focus- ing on compression of data representations rather than network parameters. We consider compres- sion from two different perspectives: information- theoretic and geometric (intrinsic dimension).
Perhaps the most prototypical ID estimators are linear projective methods like random projec- tion (Li et al., 2018) or Principal Component Analy- sis (PCA) (Jolliffe, 1986). While these project data to a linear subspace, the underlying geometric ob- ject need not be linear; therefore, e.g., PCA poorly estimates ID for curved manifolds (Campadelli et al., 2015). Nonlinear ID estimators include Cor- relation Dimension (Grassberger and Procaccia, 1983); Fisher Separability (Albergante et al., 2019); and a host of Thearest-neighhoor (NN)-based meth- ods, which use the fact that manifolds look locally Euclidean to fit the ID based on local neighbor dis- tributions (Facco et al., 2017; Levina and Bickel, 2004; Haro et al., 2008; Amsaleg et al., 2018). Such methods outperform linear ones on ID estimation benchmarks (Campadelli et al., 2015). In section 4, will assess these methods in the context of lin- guistic data, and analyze how each of them relates to coding length and case-of-adaptaation.
Information-theoreeie compression Compres- sion in neural networks can be quantified from an informationntheorreie point of view (Shannon, 1948). For instance, the information plane (IP). a widely studied framework introduced by Shwartz- Ziv and Tishby (2017) and Tishby and Zaslavsky (2015), quantifies compression per-layer as the mu- tual information between representations and in- puts. However, there is little consensus in the rele- vant literature on the appropriate estimator to mea- sure this internal compression of inputs (Saxe et al.. 2018; Goldfeld et al., 2019; Noshad et al., 2019; Chelombiev et al., 2019; Geiger, 2022).
In deep learning, there has been recent interest in using ID to characterize learning complexity. Intrinsic dimension has been quantified for neu- ral network parameters (Li et al., 2018), well as for input data and their representations in vi- sual and protein-sequence domains (Cohen et al., 2020; Recanatesi et al., 2019; Ansuini et al., 2019; Valeriani et al., 2023: Pope et al., 2021). These studies show that deep neural architectures learn low-dimensional structures, encoding parameter weights and training data into orders-of magniuue lower ID than their ambient dimension.
Instead, as probabilistic models, LMs are nat- ural black-box compressors: the negative log- likelihood of the next token given context is, by def- inition, its Shannon coding length in bits (Shannon, 1948). Concurrent work explores the equivalence between self-supervised prediction and lossless compression, demonstrating that LMS can be pow- erful general-purge purpose compressors (Delétang et al.,
In the linguistic domain, low ID of LM parame- ters has been shown to underlie efficient task adap- tation (Aghajanyan et al., 2021), where optimiza- tion occurs in oow-dimessional task specifie sub-

---

spaces (Zhang al., 2023). Moreover, parameter redundancy in pre-trained LMs can be exploited to design parameter-effcient finetuning methods such as LoRA (Hu et al., 2022). We are interested in ID data representations as opposed to LM parame- ters, as (1) we want to study how different linguistic properties affect their coding; (2) ID estimation of model parameters can be expensive- large LMs have billions of parameters, while input repre- sentations are lower-dimensional, e-g.. D 4096 in OPT-6.7H 7b (Zhang et al., 2022). In related work on LM representation ID, contextual word embed- dings have been found to lie oow-dimensional linear subspaces (Mamou et al., 2020; Hernandez and Andreas, 2021). Most similar to our work, Cai et al., 2021 show that Transformer embeddings of the Wikitext and Penn TreeBank datasets constitute nonlinear manifolds of ID O(10).
Benchmark
Datasets
GLUE
cola. mnli, mipp. quli qqp. rtc, sst2, stsb
SuperGL.UE
boolq, multire, wic
IMDB, Penn Treebank, Bookcorpus, Wikitext fr, Tweets, Pile-10k, CNN Dailymail, Openwebtext-10k, CONCODE, OPTCorpus, OPTCorpus-permutee,
OPTCorpus-sswapped. OPTCorpus-random, Wiki- text, Wikitext-permuted Wikitext-swapped, Wikitext- random
Table I: List of datasets used in experiments. We use all datasets for ID and PPL estimation and the ones in the last block for finetuning (except for OPTCorpus, which already reflects the distribution of the model).
3 Methods
Our work attempts to bridge notions of geometric and information-theorrtiec compression of linguistic data under an LM, and subsequently relate these to ease-of-adaptation. We do so by quantifying the ID of data representations, information-thewt coding length of linguistic inputs under an LM, and case-of-fie offftnnunnnin in order to determine whether these three phenomena are correlated.
pus, of~ 24 million tokens by repeatedly randomly sampling from M until [EOS] is reached. The con- ditional next-token distribution of OPTCorpus ap- proximates that of M so serve as reference datapoint.
In order to determine the effects of syntax and lexical semantics compression, define three transformations which we apply dataset: (I) cautase--permuled for each sequence in dataset. randomly permute its tokens. This ablates syn- tax to retain bag-of-tokens lexical information. (2) dataset-swapped: excluding special tokens, cre- ate random permutation a over the vocabulary. For each sequence in dataset, deterministically map each token by This ablates lexical patterns, taining syntactic structure. (3) dalaset-random: randomly replace each token in dataset with an- other (excluding special tokens). This ablates both syntactic and lexical structure.
Notation Let linguistic dataset X ( consist of N sequences of tokens, where each sequence 2(() has length ()) Let M be a (pre-trained) causal language model described by PMM(-122)) ), the conditional probability distribution of the jth token given its past context of tokens.
)==
Models & Datasets Experiments are performed for the product of models M [OPT-350m, OPT- 1.3b, OPT-6.7b] and datasets X E table 1. We focus OPT suite of causal language models due their accessibility (Zhang et al., 2022). For the datasets, we start from a list of corpora including the GLUE and SuperGI.UE benchmarks, then pick those whose size is large enough for ID estimates to converge (N 2 10000). Then, for computational efficiency, we randomly subsample each dataset to size N' = max(N, 50000), where 50000 is cho- conservatively based on preliminary analyses of convergence of bootstrapped ID estimates (see appendix A.2).
Notably, several shallow linguistic descriptors are preserved with these transformations: dataset size, sequence length and vocabulary size (1-3), vocabulary entropy (1,2), and token frequency (1). We apply the transformations to OPTCorpus and wikitext, producing six additional datasets.
3.1 Intrinsic Dimension
Given model M and dataset we estimate the representational ID of X under M as follows (sec also fig. I):
In addition to external datasets, we create one baseline dataset per model, which we call OPTCor-

---

dj of Rj using ID estimator g : RN Z+: R2 + dj.
Layer
feedforward
Ieprosentation
E N
DOORCOTING Receserntteeeii
We test 12 different ID estimators g. grouping them into categories based on technique: nine NN- based (Facco et al., 2017; Farahmand et al., 2007; Carter et al., 2010; Amsaleg et al., 2019, 2018;
feedforward
Layer
attention
Layer
embedd
Fl. e N
buoth
Haro et al., 2008; Johnsson et al., 2015; Rozza al., 2012; Ceruti et al., 2014), one projective (PCA), one based fine-grained clustering (Fisher Sep- arability, Albergante et al., 2019), and one fractal- based (Correlation Dimension, Grassberger and Procaccia, 1983). Further details on estimators can be found in appendix A.I. We implement all estimators using the skdim Python package (Bac, 2020).
Data
Jim
Embeddings
Intrinsic Dimension
Figure I:
ID estimation. Data (bottom left) are fed into an LM with M blocks (left). Activations post- attention] blocks i 1... are extracted and aggregated across the sequence length to produce an N dimensional matrix represen- tations R4, Then, the ID of each R; is estimated to produce an ID "profile" (right plot).
mmbedding/(feedforwaar
3.2 Information-Theoree Compression
1. Preprocess data. Let the context window length of M be IM- We preprocess X by splitting all r(() with length 200) > [M into sequences with maximum length LM-
Information thecorttc compression is directly re- lated to the training objective of the model, which minimizes the average negative log-likelihood loss of next-token prediction over the training set.
2. Gather representations. Evaluate M(X). gathering intermediate representations after contextualized embedding and after each uttention-ffed-forwaal block. In particular, representations are extracted after the residual connection and LayerNorm.
Learning minimizes coding length The average negative log-likelihood (NLL) training objective of causal LMs is given by
((2(8)
Σ Σ - logpm( 0),0)
min Σ ((x())
3. Aggregate representations. Because input sequences z00) are variable-length, vector associated the last token of each layer to represent Due to rugo-regees self- attention, the last token the sequence the only one to incorporate information from all other tokens in the sequence. Moreover, in the context of causal language modeling, the last token representation is the one used for next- token prediction in the top layer of the model, so is the where all information relevant to the prediction task should be concentrated. We leave testing alternative aggregation strate- gies, such as average pooling (cf. Valeriani et al., 2023) to future work.
the
that is, to minimize the empirical negative log- likelihood of the next token given its context with respect to model parameters O. This is analytically equivalent to minimizing the average number of bits encode the j"h token under p.-
We are interested in quantifying information- theoretic compression at the sequence level. We do so by using perplexity, a common metric in NLP.
The perplexity (PPL) of a sequence ( is the exponentiated negative oglliiel
Perplexity
loss
After the aggregation step, have dataset representations
(z(() PPL() 23a E logpm(sf")
(2)
R = ( Rg NXD
We compute the average PPL for each dataset X by performing forward passes through M. As PPL is monotonic in coding length, we use PPL as our measure of interest to proxy information-theoreiic compression, later relating this quantity to the rep- resentational ID of X.
where D, the hidden dimension of the model, is the ambient (extrinsic) dimension of data representations, and M has M layers.
4. Estimate ID. Per layer j. compute the ID

---

3.3 Ease-of-Adaptation
is significantly lower than their ambient dimen- sion: in our case, by roughly orders of magni- tude. In particular, while the ambient dimension of representations in OPT-350m, OPT-1.3b, and OPT-6. 7b are D = 1024, 2048, and 4096, respec- tively (Zhang et al., 2022), dataset representational ID is d = (10), see fig.
Low ID of pre-trained LM parameters has been shown to predict aase-of-finetuinng (Aghajanyan et al., 2021). We complement this finding by cor- relating the ID of data representations to LM's ease-of-adaptas adaattttt to that dataset.
Ease-of-adaptation to downstream task de- pends not only on the inputs X but also on task- specific oulputs. For instance, binary classification can be less complex than causal language model- ing given the same inputs X. As it is not always clear what is the best way to encode the outputs in order measure the quantities of our interest, focus on adaptation under causal language mod- cling objective, which is the same as the model's pre-training objective. This entails little loss of gen- erality, as task adaptation is nowadays commonly framed as language model adaptation problem.
For simplicity, in the main article we present sults on one representative NN-based estimator, the Expected Simplex Skewness (ESS) method of Johnsson et al., 2015, as it is the only one to significantly correlate with all other estimators (a = 0.1) (see fig. 6). We present results on other estimators in appendix E, and we comment on other (non-NN-based) ID estimators in section 4.4. Also for practicality, we report in the main article results obtained with one model, OPT-6.7b, 7b, only com- menting on other models when relevant, and one aggregated ID mcasure across layers: the max ID value, seen as a conservative upper bound on ID across layers. This choice is supported by the ob- servation that, for most datasets, the ID profile over layers is quite flat (appendix D). Results with other aggregated measures are presented in appendix E. Our primary result is that, in the pre-trained LMs tested, information-theoreti compression (PPL) predicts geometric compression (ID), and low ID predicts ease-of- ease-of-adaptattoe adaptatioo.. We then take closer look at which linguistic attributes dataset pre- dict its ID, finding that not only several shallow lin- guistic descriptors but also grammatical and lexical structure enable geometric compression. Finally, we find that, qualitatively, ID tends to be stable across model sizes and types and comment the differences between ID estimators.
Adaptation procedure We perform finetuning for each of OPT-350m, 1.3b, and 6.7b on the datasets X in table that are suited to causal lan- guage modeling, i.e., omitting [Super]GLUE.
Due to resource constraints, and as we com- pare between datasets and not models, we perform full finetuning for OPT-350m and finetune using LoRA (Hu et al., 2022) for the larger sizes. We end finetuming at maximum of 15 epochs or when validation loss converges. Loss is considered to have converged soon it fails to decrease for 3 evaluation steps, each 500 iterations apart. De- tailed hyperparameter settings may be found in appendix C.
Adaptation metrics We quantify ease-of- adaptation with the following, where T defined as the number of iterations until convergence:
4.1
ID tracks informationn compression
1. PPLT: final evaluation perplexity.
Informattion-thcocticc thcoretic and geometric description length of data under OPT are eeearmmnn-corrllate As shown for 7b in fig. 2a, data PPL pre- dicts ID, = 0.51 (p 0.01). The significant positive correlation between PPL and ID, more- over, holds for all model sizes tested: p 0.66, p 0.01 for OPT-350m and p 0.49, p 0.01 for OPT-1.3b, the correlation being most salient for the smallest model (see figs. E.3a and E.3d). We hypothesize that model optimization in higher dimensions permits discovery of better representa-
2. Sample complexity $ - + ET PPL, where PPL is the evaluation PPL at eval- uation step t.
Finally, we compute Spearman correlations be- tween these metrics, zero-shot perplexity (PPLo). and ID to assess whether the two types of compres- sion and lise-of-adaptation are linked.
4 Results
We find that, similar to protein models and visual networks (Ansuini et al., 2019; Valeriani et al., 2023), the ID of linguistic data representations
Wee also experimented with ensembling ID metrics, but found the various ensemhling methods hard to justify as NN- hased estimators are very over-represented in our panel.

---

opt-6.7b
Sample Complestyy (log)
opt-6.75 7b
opt-6.7b
300
Finetune Eval. PPL, (leg)
1p-2.011
p- (p-0.00)
(p-0.00)
250
12
200
8
E55
6
50
6
B
10 12 14
50
100
150
200
250
300
50 100
150 200
250
300
Dataset Perplexity (log)
ESS Max
ESS Max ID
(a) Intrinsic dimension (ESS max ID) Sample complexity vs. Intrinsic dimen- Final log-PPL vs. Intrinsic dimension sion (ESS max ID)
vs. data perplexity Nog-PPI.).
(ESS max ID).
Figure 2: Intrinsic dimension significantly positively correlated to (a) dataset PPL (p 0.01): and also predicts case-of-adaptation metrics (b) sample complexity (p < 0.01) and (c) final evaluation PPL (p < 0.01). Both ID and dataset perplexity in (a) are measured before finetuning; in (b) and (c), ID measured before finetuning and the y-axes after finetuning.
tions in d-dimensional intrinsic space, as evidenced by the correlation between performance and model size in LM scaling laws. Then, on a given dataset, larger model sizes may encounter an ID floor effect, thus weakening the correlation between PPL and ID compared to smaller sizes.
V
Hy
Nto 0.15
Max ID
0.50
0.44
0.15
p=0.01
p=0.02
Table For OPT-6.7b, Spearman correlations p be- tween ESS max ID and dataset vocabulary size (V), cabulary entropy (Hv)), average sequence length L. and size in tokens Nook Significant correlations at 0.05 displayed with the corresponding p-value.
4.2 Compression is linked to Cass-of-Adaptation
As evidenced by the positive trends in figs. 2b and 2c, ID predicts sample complexity (p = 0.72,
Linguistie Structure Permits Compression Ab- lating linguistic structure increases ID and per- plexity of data representations. This may be seen in fig. 2a for OPT-6.7b, where both the ID and perplexity of the permuted, swapped, and random variants increase with respect the baseline for the wikitext dataset. Moreover, this relationship holds for all model sizes, see figs. E.3a and E.3d, and generally for other ID estimators, see appendix E. This indicates that learned grammatical and lexical structure permits compression in LMs; furthermore, the small increase in ID from wikitext to wikitext- permuted compared to other variants suggests that bag-of-words lexical semantics, rather than com- plex syntactic structure, accounts for much of geo- metric compression. Interestingly, this ties in with general estimates of the relative information load of syntax and lexical semantics in human language (Mollica and Piantadosi, 2019).
p < 0.01) as well as final PPL after convergence (p = 0.73, < 0.01). These trends, moreover, are robust to model size: ID predicts sample com- 0.01 for OPT-1.3b and P 0.81, p 0.01 for OPT-350m (figs. E.3b and E.3e); and ID predicts final PPL after fine- tuming at p 0.65, p 0.01 for OPT-1.3b and P = 0.81, p 0.01 for OPT-350m (figs. E.3c and E.3f).
plexity at p 0.61,
Results indicate that data which are pressed zero-shot under the LM are casier to adapt to. Moreover, they corroborate findings in Agha- janyan et al., 2021, which low parameter in pre-trained LMs predicts rapid adaptation. We hypothesize that this is because intrinsic data rank bottlenecks intrinsic parameter rank and vice-versa (see Rozza et al., 2012 for discussion).
4.3 Linguistic Correlates to Compression
Some Shallow Linguistic Descriptors Predict ID but Are Uncorrelated Perplexity Beyond lin- guistic structure, two related shallow linguistic de- scriptors also predict ID: vocabulary size V (num- ber of unique tokens in the dataset) and vocabulary entropy Hy (Shannon entropy of token frequency
In pre-trained OPT, geometric compression can be explained partially by data perplexity. But, taking a closer look, ID also correlates with interpretable linguistic descriptors such as syntactic structure or token entropy.

---

wikitext
sst2
tweets
90
90
90
Intrinsic Dimension
80
70
70
0014
80
60
40
40
40
30
20
20
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8
1.0 0.0
0.2
0.4
0.6
0.8
1.0
Relative Layer Depth
Relative Layer Depth
Relative Layer Depth
Figure 3: Example evolution of ID (ESS) over layers for three tasks (left to right): wikitext, sst2, and tweets. general, ID profiles be dissimilar for different datasets under the model, and ID profiles for OPT-350m, 1.3b, and 6.7b appear correlated, with larger extrinsic dimension lending itself to slightly (not proportionately) larger intrinsic dimension.
Spearman (opt-6.7b, a=0.1)
1.00
1.00
0.46
0.75
350m
Ipput. Length
0.50
-0.46
1.00
0.72
0.43
0.25
Model Size
Vocab Size
0.72
1.00
0.74
0.00
1.3b
0.25
0.43
0.74
1.00
Entropy
0.50
0.75
-1.00
6.7b
PPL
Input. Length
Vocab
Size
Entropy
Figure 4: Spearman correlations between data descrip- tors: information-theorate descriptor PPI. (top/left), and shallow linguistic descriptors (bottom/right); only correlations significant at a 0.1 shown. Shallow metrics highly correlate to each other but not to PPL.
100
200 250 300
ESS Max
Figure 5: Distributions of max ID (aggregated lay- ers of the Transformer) across tasks, computed using the ESS estimator. Despite the model extrinsic dimension doubling (ED 1024, 2048, 4096 from top to bottom), the ID remains fairly stable. Apart from a few outliers. the ID over layers less than 100, that all IDs computed over all layers are generally O(10).
in the dataset), Table 2. In contrast, average sequence length (in tokens) L and dataset size (in tokens) Nock do not predict ID.2
That descriptors such as vocabulary size and en- tropy predict ID is intuitive: with, e.g., a larger vocabulary, more information needs to be encoded, Furthermore, as expected, these descriptors are cor- related to one another. However, the descriptors positively correlated to PPL. (fig. 4), suggesting that the relationship between information-theoree and geometric compression not explained by shallow dataset properties.
4.4 Intrinsic Dimension of Representations
We have presented ID it relates information- theoretic compression and ease-of-adaptation; now we address the question of geometric compression itself. First, we show that the various models com- press data into similar ranges of ID regardless of extrinsic hidden dimension, lending weight to the manifold hypothesis for linguistic data. We fur- ther report on how ID estimation with different methods produces complicated picture, and we comment results evaluated the full battery of 12 estimators.
The relation between shallow linguistic descrip- tors and ID generally holds across model size and ID estimators; see appendix E for further discus- sion, extending other layer-aggregate measures of ID.
Different model size, similar ID We find that all tested models compress data to a similar range of ID, around ((10). Although the model hidden dimension doubles from OPT-350m to 1.3b to 6.7b. the range of data representational ID does not sig-
"It is crucial that the datasct be big enough to prevent spu- rious correlations hetween size and ID due to scaling effects (see appendix A.2).

---

Layer-wise Spearman Corr. of ID Methods (opt-6.7b, a 0.1)
1.00
PCA
1.00
0.94
-0.12
-0.09 0.20 -0.28 0.23 0.05
0.75
FisherS
0.94
1.00
-0.12
-0.18
0.19
-0.27
0.12
Corrint
1.00
0.50
0.14
0.56
0.18
0.09
0.50
TLE
-0.12
-0.12
0.50
1.00
0.69
0.72
0.59
0.49
0.42
0.25
MLE
-0.09
-0.18
0.14
0.69
1.00
0.33
0.38
0.83
0.86
0.00
ESS
0.20
0.19
0.56
0.72
0.33
1.00
0.45
0.27
0.12
0.25
TWoNN
-0.28
-0.27
0.18
0.59
0.38
0.45
1.00
-0.50
MOM
0.23
0.12
0.09
0.49
0.83
0.27
1.00
0.93
-0.75
MADA
0.05
0.42
0.86
0.12
0.93
1.00
-1.00
PCA
Fishers
Corrint
TLE
MLE
ESS
TWoNN
MOM
MADA
Figure 6: Spearman correlations between ID metrics: the bottom-right block of correlated metrics correspond to NN-based methods, while the top-left are PCA, Fisher Separability, and Correlation Dimension, respectively a linear projective, fine-grained clustering, and fractal method.
TWEETS Representational ID
et al., 2019) and evidence the manifold hypothe- sis for linguistic data. Together with past work, our results suggest that different-sized (but similar- performing) models which are trained on the same data and objective can independently recover the la- tent dimension of data and exhibit similar patterns of processing.
Relative Intrinsic Dimension
ESS
TwoNN
FisherS
Corrint
MLE
TLE
0.02
MADA
0.01
ID estimators aren't equal; some are useful Though ID estimators based on similar analytical methods are correlated (fig. 6), different ID estima- tors can produce different ID profiles for the same dataset (e.g., fig. 7). This can stem from number of factors pertaining assumptions and analytical method. For instance, we find the NN-based meth- ods be most predictive of data perplexity and case-of-adappation and PCA and Fisher Separabil- ity to be least predictive. This may be because (1) PCA assumes that the underlying data manifold is linear, which may lead to poor ID estimation (con- sider a 1D line embedded in 2D space; PCA will estimate an ID of 2 as there are two principal direc- tions of variance); (2) Fisher Separability system- atically underestimates ID for non-uniformly dis- tributed data (Albergante al., 2019), and Trans- former representations are highly anisotropic (Cai et al., 2021). ). Lastly, among the 12 estimators tested, we could not produce sensible results for three
0.00
0.2
0.4
0.8
1.0
Relative Layer Depth (opt-6.7b)
Figure 7: Different ID methods' relative estimates (= ID ED) over layers in OPT-6.7b 7b for the Tweets dataset. While some groups of methods are correlated, they produce different ID profiles for the same data.
nificantly change, fig. 5.
Moreover, the evolution of ID across the layers different model sizes follows similar trajecto- ries: qualitatively, for all tasks, the OPT models' ID profiles follow similar global patterns (c.g., sim- ilar number of peaks), with larger models having slightly larger intrinsic dimension, examples in fig. 3. For extended diseussion of ID evolution across layers, see appendix D.
Similar ID values and evolution across different models echo results in the visual domain (Ansuini

---

them (Rozza al., 2012; Ceruti al., 2014; Carter al., 2010), see appendix A.3 for discussion.
specific linguistic constructions (ef. Hernandez and Andreas, 2021 for analysis using PCA).
While we cannot claim that any single estimator produces the "true" ID, it appears that, for purposes of ID estimation of linguistic datasets encoded in LMs, NN-based methods are the most useful ones, being reliable predictors of information-theoreiic compression and laas-oofddappatioon (appendix E). More generally, the differing results obtained with various ID estimators reveal need to validate them against linguistic data, which may violate under- lying assumptions of common estimators, such as the global isotropy assumption in Fisher Separa- bility (Albergante et al., 2019). While there in- deed exist ID estimation benchmarks for synthetic manifolds and image data, and while NN-based estimators outperform linear ones in these bench- marks (Campadelli al., 2015), a benchmark has not yet been developed for linguistic data, to our knowledge.
While our work investigates the relationship be- tween ease-of- -aas-of-ddaptation and zero-shot compres- sion, a logical next step is to investigate how fine- tuning dynamically affects data compression under the model. We hypothesize that the result may de- pend whether the dataset used for finetuning is memorized by the model during training.
Finally, while LMs are trained reproduce the yet unclear
distribution of human language, whether analyzing the linguistic representations of the former allows us to make statements about the dimensionality of the latter. Then, an open question remains: what is the "true" dimensionality of natural language, and to what extent do LMs recover it?
5 Discussion
We have quantified geometric compression in neu- ral language models using the ID of representations, where ID tracks Shannon information -iemmatioo--hhoreticc coding length. This bridges two notions of descrip- tion length in pre-trained neural LMs by showing they are significantly positively correlated. Our re- sult has also practical implications, suggesting that ID and perplexity predict how easy it to finetune a model to task (similarly to what observed in the context of zero-shot prompting by Gonen et al., 2022). More speculatively, the relation between ID and task adaptation may inform future modeling work that actively encourages data compression training time, to indirectly inject fast-adaptation capabilitics into a model.
ID estimators are not equal: we focus on NN- based methods because they explain useful proper- ties of the data and ease-of-finetuning. Our work is first attempt to evaluate wide range of ID estimators on natural language representations, and reveals the need for a further principled study of ID estimation linguistic data.
That nonlinear ID estimators predict informationn-thooree compression and case- of-adaptation over linear ones highlights a need go beyond PCA in analyzing compression of specific linguistic phenomena. Our experiments on wikitext and variants also demonstrate a need for further experiments on, e.g., idiomaticity, or

---

Limitations
only, and the ERC is not responsible for any use that may be made of the information it contains.
While we confirmed that modern LMs do com- press language data, and that this compression is correlated with ease-of-learning, we only provided a limited characterization of the rela- tion between LM compression and linguistic properties of the input, such as lexical infor- mation and syntactic structure.
References
Armen Aghajanyan, Sonal Gupta, and Luke 7ettle- moyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Inter- national Joint Conference on Natural Language Pro- cessing (Volume I: Long Papers), pages 7319-7328, Online. Association for Computational Linguistics.
Due to both access restriction and computa- tional limitations, we cannot replicate our in- vestigations on huge language models such as ChatGPT.
Luca Albergante, Jonathan Bac, and Andrei Zinovyev. 2019. Estimating the effective dimension of large biological datasets using fisher separability analysis. In 2019 International Joint Conference on Neural Nenworks (IJCNN, page 1-8.
A related question is what extent the cor- relation we report between compression and ease of learning would hold (or even how could be meaningfully formulated) in the con- text of prompt-based zero-shot task adaptation as afforded by huge LMs.
Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stéphane Girard, Michael E. Houle, Ken-ichi Kawarabayashi, and Michael Nett. 2018. Extreme- theoe-theoretic estimation of local intrinsic dimen- sionality. Data Mining and Knowledge Discovery. 32(6):1768 16711188888
More generally, it remains to be explored how number of modeling choices, such as non- causal predictive objectives or instruction tun- ing, would affect our generalizations.
I Laurent Amsaleg, Oussama Chelly, Michael E. Houle, Ken-ichi Kawarabayashi, Miloš Radovanović, and Weeris Treeratanajaru. 2019. Intrinsic Dimensional- ity Estimation within Tight Localities. Proceedings. page 181-189. Society for Industrial and Applied Mathematics.
Ethics Statement
This paper does not introduce new models or datasets, and it presents an abstract analysis of language model data compression that, we think, should not raise ethical concerns. We believe the other hand that our focus on improving the understanding of how language models process information can be generally beneficial, in an AT landscape in which powerful language models are deployed with little understanding of the mechan- ics by which they work, and, consequently, little ability to control their behavior.
Alessio Ansuini, Alessandro Laio. Jakob H Macke, and Davide Zoccolan. 2019. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems. volume 32. Curran Associates, Inc.
Sanjeev Arora, Rong Ge. Behnam Neyshabur, and Yi Zhang. 2018. Stronger generalization bounds for deep nets via compression approach. In Pro- ceedings of the 35th International Conference Machine Learning, volume of Proceedings of Ma- chine Learning Research, pages 254-263. PMLR.
Acknowledgements
Jonathan Bac. 2020. Intrinsic dimension estimation in python scikit-dimension 0.3.2 documentation.
We thank Alessandro Laio for generous guidance on intrinsic dimensionality estimation. Jacob An- dreas provided very helpful feedback on the project. We also thank the members of the UPF COLT lab, especially Gemma Boleda, Roberto Dessi and Lucas Weber for early feedback, the members of the Barcelona Apple Machine Learning Research group and the participants in the EviL seminar for helpful feedback and suggestions. Our work was funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 101019291). This paper reflects the authors' view
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. TweetE- Vnl:Undfied Benchmark and Comparative Evaluation for Tweet Classification. In Proceedings of Findings of EMNLP.
Stas Bekman. 2022. Openwebtext- 10k dataset.

---

Christopher M. Bishop. 2007. Pattern Recognition and Machine Learning [Information Science and Statis- tics), I edition. Springer.
Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Eriia Mor- eira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling lan- guage modeling with pathways. (2rXiv:2204.0231),, ArXiv:2204.0331 11 [cs].
Léonard Blier and Yann Ollivier. 2018 The descrip- tion length of deep learning models. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Uri Cohen, Yeon Chung, Daniel D. Lee, and Haim Sompolinsky. 2020. Separability and geometry of object manifolds in deep neural networks. Nature Communications, 11(1):746.
Grégoire Delétang. Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hut- ter, and Joel Veness. 2023. Language modeling is compression.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church, 2021. Isotropy the contextual embedding space: Clusters and manifolds. In International Con- ference Learning Representutions.
Francesco Denti, Diego Doimo, Alessandro Laio, and Antonictta Mira. 2022. The generalized ratios in- trinsic dimension estimator. Scientific Reports, 12(1 (0:2000
P. Campadelli, E. Casiraghi, C. Ceruti, and A. Rozza. 2015, Intrinsic dimension estimation: Relevant tech- niques and a benchmark framework. Mathematical Problems in Engineering, 5:7755577
Vittorio Erba, Marco Gherardi, and Pietro Rotondo. 2019. Intrinsic dimension estimation for locally un- dersampled data. Scientific Reports, 9(1:111133
Kevin M. Carter, Raviv Raich, and Alfred O. Hero III. 2010. On local intrinsic dimension estimation and its applications. JEEE Transactions Signal Process- ing, 58(2)6600-663.
Elena Facco, Maria d' d'Errico, Alex Rodriguez, and Alessandro Laio. 2017. Estimating the intrin- sic dimension of datasets by minimal neighbor- hood information. Scientific Reports, 7(1):12140. ArXiv: 1803.0699 [cs, stat].
Claudio Ceruti, Simone Bassis, Alessandro Rozza, Gabriele Lombardi, Elena Casiraghi, and Paola Cam- padelli. 2014. Danco: An intrinsic dimensionality estimator exploiting angle and norm concentration. Pattern Recognition, 2566-2555.
Amir massoud Farahmand, Csaba Szepesvári, and Jean- Yves Audibert. 2007. Manifold-adaptive dimension estimation. In Proceedings of the 24th international conference on Machine learning, page 365-272, Cor- valis Oregon USA. ACM.
Tvan Chelombiev, Conor Houghton, and Cian O' O'Donnell. 2019. Adaptive estimators show infor- mation compression in deep neural networks. In International Conference on Learning Representa- tions.
K. Fukunaga and D.R. Olsen. 1971. An algorithm for finding intrinsic dimensionality of data, IEEE Trans- actions on Computers, C-20(2): 17--183.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, al. 2020. The Pile: An 800GB dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0003
Noam Chomsky, 1986. Knowledge of Language: Its Nature, Origin, and Use. Praeger, Wesport, CT.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia,
Bernhard C. Geiger. 2022. On information plane anal- yses of neural network classifiers review. IEEE Transactions on Neural Networks and Learning Sys- tems, 33((2):7039-7551 ArXiv:2003.0667 [cs, math, stat].
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Ste- fanie Tellex. 2019. Openwebtext corpus. http: //Skylion007 thub. wwt.ppeeeebeettorraaeeee

---

Ziv Goldfeld, Ewout Van Den Berg, Kristjan Gree- newald, Igor Meinyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. 2019. Estimating information flow in deep neural networks. In Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pages 2299-2308. PMLRR
Chunyuan Li, Heerad Parkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Man- ning. Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak. Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji. Omar Khattab, Peter Henderson, Qian Huang. Ryan Chi, Sang Michael Xic, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li. Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models.
Hila Gonen, Srini Iyer. Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. https: / //arxiv.org/aaaee baarrg/bb//220 2ttpp///22...0200
lan Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www. deeplearningbook.og org.
Peter Grassberger and Itamar Procaccia. 1983. Measur- ing the strangeness of strange attractors.
Gloria Haro, Gregory Randall, and Guillermo Sapiro. 2008. Translated poisson mixture model for stratifi- cation learning. International Journal of Computer Vision, 35833335.337.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015, Teaching machines to read and comprehend. In NIPS, pages 1693-1700
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Evan Hernandez and Jacob Andreas. 2021. The low- dimensional linear geometry of contextualized word representations. In Proceedings of the 25th Confer- ence on Computational Natural Language Learning, pages 82-93, Online. Association for Computational Linguisties.
Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang. Yoon Kim, and Sueyeon Chung. 2020. Emergence of separable manifolds in deep language representations. In Proceedings of the 37th International Conference on Machine Learning, page 6713-6723. PMLR.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor- pus of English: The Penn Treebank. Computational Linguistics, 19(2):33-333.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zeullmmyer. 2018. Mapping language to code programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, page 1643-1652, Brussels, Bel- gium. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- cls. In International Conference on Learning Repre- sentations.
Kerstin Johnsson, Charlotte Soneson, and Magnus Fontes. 2015. Low bias local intrinsic dimension estimation from expected simplex skewness. IEEE Iransactions pattern analysis and machine intelli- gence, (7))):196200
Francis Mollica and Steven Piantadosi. 2019. Humans store about 1.5 megabytes of information during language acquisition. Royal Society Open Science. 6(3):181393.
Ian Jolliffe. 1986. Principal Component Analysis. Springer.
Neel Nanda. 2022. Pile- 10k dataset.
Morteza Noshad, Yu Zeng, and Alfred O. Hero. 2019. Scalable mutual information estimation using depen- dence graphs. In ICASSP 2019 2019 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), page 2962-2966.
Elizaveta Levina and Peter Bickel. 2004. Maximum likelihood estimation of intrinsic dimension. In Ad- vances in Neural Information Processing Systems, volume 17. MIT Press.

---

OpenAI. 2023. Gpt-4 technical report. (arXIv:2303.08774) ArXiv:2303.08774 [cs].
Zou, Angela Jiang, Angelica Chen, Anh Vuong. Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Ar- gueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng. Chitta Baral, Chiyu Wu, Chris Dallison-Burchhe Chris Waites, Christian Voigt. Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raf- fel. Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette. Dan Hendrycks, Dan Kilman. Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Ju- rgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret. Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dim- itri Coelho Mollo, Diyi Yang. Dong-Ho Lee, Ekate- rina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, El- lie Pavlick, Emanuelc Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang. Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice En- gefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinezz-Plunee Francesca Happé, Francois Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski. Giambattista Parascandolo. Gior- gio Mariani, Gloria Wang. Gonzalo Jaimovitch- López, Gregor Betz, Guy Gur-Ari, Hana Galijase- vic, Hannah Kim. Hannah Rashkin, Hannaneh Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jae- hoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Ko- coñ, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Ja- son Yosinski, Jekaterina Novikova, Jelle Bosscher. Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse En- gel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jil- lian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Oralle Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gim- pel, Kevin Omondi, Kory Mathewson, Kristen Chi- afullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan. Lianhui Qin. Lidia Contreras- Ochando, Louis- Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. Rissanen data analysis: Examining dataset character- isties via description length. In Proceedings of the 38th International Conference on Machine Learning, page 8500 8513. PMLR.
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Gold- blum, and Tom Goldstein. 2021. The intrinsic di- mension of images and its impact on learning. In International Conference on Learning Representa- lions.
Alec Radford and Karthik Narasimhan. 2018, Im- proving language understanding by generative pre- training.
Stefano Recanatesi, Matthew Farrell, Madhu Advani, Timothy Moore, Guillaume Lajoie, and Eric Shea- Brown. 2019. Dimensionality compression and ex- pansion in deep neural networks. ArXiv:1906.04443 [cs, stat].
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866 -66.
A. Rozza, G. Lombardi, C. Ceruti, E. Casiraghi, and P. Campadelli. 2012. Novel high intrinsic dimension- ality estimators. Machine Learning. 89(1):37-66 65.
Andrew Michael Saxe, Yamini Bansal, Joel Dapello. Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. 2018. On the infor- mation botteneck theory of deep learning. In Inter- national Conference on Learning Representations.
CE Shannon. 1948. A mathematical theory of commu- nication.
Ravid Shwartz-Ziv and Naftali Tishby. 2017. Opening the black box of deep neural networks via informa- tion.
Antoine Simoulin and Benoit Crabbé. 2021. Un modèle Transformer Génératif Pré-entrainé pour le français. In Traitement Automatique des Langues Naturelles, pages
Lille, France. ATALA.
Ahmed Soliman. 2022. Codexllue-concode dataset.
Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Sauriga-Alonso Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain. Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüiler,, An- drew Dai, Andrew La, Andrew Lampinen, Andy

---

He, Luis Oliveros Colón, Luke Metz, Lütf Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Pouthast Matthew I., Leavitt, Matthias Hagen, Mátýás Schu- bert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Co- hen, Michael Gu. Michael Ivanitskiy, Michael Star- ritt, Michael Strube, Michat Swędrowski. Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng. Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers. Niklas Muennighoff, Ni- tish Shirish Keskar, Niveditha S. Iyer, Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy. Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang. Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao. Percy Liang, Peter Chang, Peter Eckensley, Phu Mon Htut, Pinyu Hwang, Piotr Mitkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu. Qinlang Chen, Rabin Banjade, Rachel Eita Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg. Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le- Bras. Rosanne Liu, Rowan Jacobs, Rui Zhang, Rus- lan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Sto- vall, Ryan Techan Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman. Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous. Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebas- tian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas De- haene, Stefan Divic, Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stuart M. Shieber, Summer Misherghi. Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang. Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz. Uri Sha- ham, Vedant Misra, Vera Demberg. Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu,
Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang. Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2022. Beyond the imita- tion game: Quantifying and extrapolating the ca- pabilitiies of language models. (arXiv:2206.0461) ArXiv:2206.0461 [cs, stat].
Naftali Tishby and Noga Zaslavsky. 2015, Deep learn- ing and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), page 1-5.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich. Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov. Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schel- ten, Ruan Silva, Eric Michael Smith, Ranjan Sub- ramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, An- gela Fan, Melanie Kambadur, Sharan Narang, Au- relien Rodriguez. Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open founda- tion and fine-tuned chat models. (arXiv:2307.0925) 009288).. ArXiv:2307.00 00.888888888 [cs].
Lucrezia Valeriani, Diego Doimo, Francesca Cu- turello, Alessandro Laio, Alessio Ansuini, and Al- berto Cazzaniga. 2023. The geometry of hid- den representations of large transformer models. (arXiv:2302.00224)) ArXiv:2302.00294 [cs, stat].
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Elena Voita and Ivan Titov, 2020. Information probing with minimum description length. Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNIP), page 183-196, Online. Association for Computational Lin- guistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpest language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32, Curran Associates. Inc.

---

Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel Bowman. 2019b. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of ICLR, New Orleans, LA. Published on- line: https: / /opppreeeeiir net net/group?id-Ic... cc/2019/conferenen
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi. Tatsunori Hashimoto. Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Transactions on Machine Learning Research.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language mod- els. (arXiv:2005.10068) ArXiv:2205.01 [cs].
Yu Zhang, Peter Tiño, Aleš Leonardis, and Ke Tang. 2021. A survey neural network interpretability. IEEE Transactions on Emerging Topics in Computa- tional Intelligence, 5(5):726-742.
Zhong Zhang, Bang Liu, and Junming Shao. 2023. Fine- tuning happens in tiny subspaces: Exploring intrinsic task-specific subspaces of pre-trained language mod- els. In Proceedings of the s1 Anmual Meeting of the Association for Computational Linguisties ( (Volume Long Pupers), pages 1701-1713, Toronto, Canada. Association for Computational Linguistics.
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urta- sun, A. Torralba, and S. Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference Computer Vision (ICCV), pages 19-27, Los Alamitos, CA, USA. IEEE Computer Society.

---

A ID Estimation
A.1.1 PCA Projective method that assumes a linear manifold. The ID d the number of principal values À > Auyus where hyperparameter & 20 (Fukunaga and Olsen, 1971).
Global Estimators
this appendix, we detail the tested ID estimation methods (appendix A. and validate their conver- gence on a sample of 50, 000 datapoints of the bookcorpus dataset in appendix A.2.
Fisher Separability (FisherS) Fine-grained clus- tering estimator that relies on "clumping" phenom- ena in high dimensions. To estimate d, (1) the data are transformed using PCA (hyperparameter k = 10) and projected onto the unit sphere; (2) the proportion of linearly separable datapoints is compared to that of theoretical equidistribution on d-sphere (Albergante et al., 2019).
A.1 ID Estimator Details
For each ID estimator tested, we provide brief description well the hyperparameters used (which we will denote kt) in relation to the skdim package. As ID estimation theory not our focus, we refer the reader to Campadelli et al., 2015 for a comprehensive review of methods.
Note that ID estimators can be classified as global local. Global ID estimation assumes the entire dataset lies on single geometric object, and estimates its dimensionality from the entire dataset at once. In contrast, local ID estimation computes wiint-wise ID of data neighborhoods, then aggregates the per-neighborhood estimates to arrive at global estimate. This aggregation of local estimates correct for negative biases of global ID estimators in high where this bias is symptomatic of concentration of measure in high dimensions (Carter et al., 2010). While Mamou et al., 2020 show that linguistic representations lic on separable manifolds, we assumed for simplicity that our data lie on one geometric object and use either global or local methods to estimate its ID) leaving per-manifold ID estimation to future work.
Correlation Dimension (CorrInt) The volume of d-dimensional radius-r hypersphere grows as s ~ yd For two values of r: rı and r2, estimate by counting the nearest neighbors in a radius-r sphere around each point. Then, fit (Grassberger and Procaccia, 1983).
We choose corresponding to hyperparameter h1 10 nearest neighbors and 72 corresponding to hyperparameter k2 20 nearest neighbors.
TwoNN A nearest-neighhors (NN)-based estima- tor that assumes local uniform data density (up to the second neighbor). Regresses
log(1- (o(1-/F1 (()))))()))))))))))))))))))))( d, log where F is the cumulative density function and P4 = 22 is the 2NNs distance ratio for each 34 E X. Following Facco et al., 2017, we dis- card hyperparameter & 0.1 fraction of largest 14; from the regression, heuristic that improves estimation accuracy.
Estimator
Method
Global/Local
PCA
Projective
Global
A.1.2 Local Estimators
FisherS
Fine-grained
Global
clustering
ESS For each datapoint, take hyperparameter 10 nearest neighbors. This neighborhood is assumed to be approximable by a uniform dis- tribution of datapoints on a tangent plane to the manifold. Expected Simplex Skewness, or ESS (called ESSa in Johnsson et al., 2015), constructs a simplex with one vertex at the centroid of the local dataset and other vertices the other datapoints. Then, the expected simplex skewness measure is defined as the volume of the simplex divided by the volume if all edges to the centroid were orthogonal.
CorrInt
Fractal
Global
TwoNN
NN-based
Global
KNN
NN-based
Global
MIND ML
NN-based
Global
DANCo
NN-based
Global
TLE
NN-based
Local
MLE
NN-based
Local
MOM
NN-based
Local
MADA
NN-based
Local
ESS
NN-based
Local
The ESS computed over all such local datasets then compared to the theoretical expected value in d-dimensions in order to estimate d.
Table A: Taxonomy of 12 ID estimators tested (3 of which unsuccessfully; KNN, MIND ML, and DANCo), by method and by whether global or local. If local, point- wise ID estimates are computed and mean-aggregated.
TLE Tight Local ID Estimation consists of com- puting a modified Correlation Dimension on small

---

neighborhoods of hyperparameter k 20 neigh-
Convergence of Methods (bookcorpus)
bors around query datapoints, then fitting using maxinun-likelihood estimation (Amsaleg et al., 2019).
50
Estimate
40
Conint
KGM
TLE MADA
MLE We use the Maximum Likelihood Estima- tor of Haro et al., 2008, which extends Levina and Bickel, 2004 to handle noise. For simplic- ity, we assume one underlying data manifold (the method extends to multiple underlying manifolds). Then, MLE models the number of points falling into a small neighborhood around query points by a (Translated) Poisson process. The maximum- likelihood estimator for the Poisson À parameter is fit given data, from which one can solve for the local ID at that neighborhood. Finally, local IDs are averaged to arrive the overall estimate.
20
10
1.0
4.0
N Samples (1e4)
Figure A.1: Convergence of ID estimates for bookcorpus on the last layer of OPT-6.76 7b, from N' 200 to N' 50000. Each datapoint is the mean ID estimate, shown with one standard deviation, boot- strapped sample of size N', computed over 3 random seeds.
MOM The Method-of- Momeets estimator from Amsaleg et al., 2018 takes as reference the dis- tribution of distances to query point over hyperparameter = 100 NNs of q. The procedure estimates empirical moments of X and compares them to the thcoretical value, which is function of d, in order solve for d.
Ceruti et al., 2014), see Libba et al., 2019 for discus- sion.
Similar to Ansuini et al., 2019, justify our choice of 50000 with a convergence analysis of ID estimators on large, complex dataset, that is, bookcorpus (N 7/00-1288,, using OPT-6.7b (D = 1096), and a one layer representation (the last). Starting at N = 50000, we systematically subsample the representations on a log-scalc and show that ID estimates averaged over three ran- dom seeds appear to converge for all estimators by N' 10000 (see fig. A.1). Convergence of these estimates permits to reliably include them in our analysis linking ID to PPL and -fa-p-ptatttttnonn.
MADA Manifold Adaptive Dimension Estima- tion (MADA) compares the empirical probability that datapoints fall into a ball around query dat- apoint to the theoretical probability, which a function of d. More precisely,
P(x; E B(x,)) r)) = n(x, r)r"
for some function n, where for small-enough r. n(xrr) is essentially constant. Then, d is fit over many query points x and z; in the dataset (Farah- mand al., 2007).
Finally, our approach is limited for small datasets, as N needs to grow exponentially in d in order to estimate (Bishop, 2007). As attested in the literature (Campadelli et al., 2015; Alber- gante et al., 2019), several ID estimators are empir- ically more robust to this curse of dimensionality, such non-NN-based estimators PCA and FisherS, which converge earlier than N' 10000. More- over, note that for the remaining estimators. for small N' E 10000, they are increasing, thus sys- tematically underestimating ID, until starting to converge at around N' ~ 10000. Therefore. while N' - 10000 is not an end-all be-all cut-off for ID convergence of linguistic data, we employ it as heuristic for dataset selection.
A.2 ID Estimation Convergence Analysis
In section 3.1, state that ID estimation is per- formed for datasets X of size N > 10000. For large datasets (N 50000), we compute ID on random subsample of (NN, 50000) data points for efficiency. Here, we verify that ID computed on sample of size 10000 < N < 50000 for max(D) ()) - 4096 reasonably converges, the reason being that NN-based ID estimators can be scale- dependent. Scale-dependeneer.
sensitivity to data resolution, of NN-based estimators is well- documented in the literature. Due concentration of measure high dimensions, these estimators are sensitive to data density and underestimate ID for 2 10 (Facco et al., 2017; Ansuini et al., 2019;
A.3 Other Estimators Tested
We tested total of 12 estimators. In addition the discussed in appendix A.1, we tested three NN-based global estimators, DANCo (Ceruti et al.,

---

2014), MIND ML (Rozza et al., 2012), and KNN (without bootstrapping) (Carter et al., 2010), but encountered difficulty tuning the hyperparameters to produce sensible ID estimates (MiND ML pro- duced d = 10 for almost all layers on all datasets; the other estimators produced d = D). For this rea- son, we omit them from discussion of the results. Finally, in future work, it is worth testing the recent FCI estimator of Erba al., 2019 and the Generalized Ratios ID estimator of Denti et al., 2022, which are more robust to scaling effects by design.
Corpus
Summary
IMDB (Maas al., 2011)
Movie reviews
Penn Treebank (Maruus et al., 1993)
Text-only version of Penn Treebank
(content from 1989
Wall Street Journal)
Bookcorpus (Zhu al., 2015)
Text from books
Wikitext (Mer- ity et al., 2017)
Cleaned Wikipedia
Wikitext fr (Simoulin and Crabbé, 2021)
Cleaned Wikipedia
French
B Dataset Details
In this appendix, we outline how [Super]GLUE data are preprocessed for PPL and ID computation, and describe the remaining corpora in table B.2. All datasets are freely available on HuggingFace at the time of writing.
Tweets (Barbieri al., 2020)
Twitter dataset's sentiment evalua- tion subset
Pile- File-10k (Nanda, 2022)
First 10k sequences
of The Pile (Gao et al., 2020) sequences
Preprocessing [Super]GLUE GLUE and Super- GLUE are two linguistic classification benchmarks designed to test language models on tasks such as textual entailment and semantic equivalence (Wang al., 2019b.a). For example. in Quora Question Pairs (qqp). two questions are given to the model as natural language, and the model tasked to predict whether they are equivalent. there are multiple inputs, we treat them as individual data points in the ID/ / PPL computations and feed them to the model separately as in Wang et al., 201 9a, leaving other strategies, such concatenation with delimiters (Brown et al., 2020), to future work.
Openwebtext- 10k (Bekman, 2022)
10k from Openwebtext
(Gokaslan al.,
2019)
CNN Dailymail (Hermann et al., 2015)
News articles from CNN
CONCODE (Soliman, 2022)
Set of Java classes from CONCODE (Iyer et al., 2018)
Table B.2: Description of external (LL- corpora
C Finetuning Details
Hyperparameter Batch size
Value
We implement full finetuning for OPT-350m and parameter-efficient finetuning using LoRA (Hu et al., 2022) for the other LMs on standard cross- entropy causal language modeling objective, op- timized using AdamW (Loshchilov and Hutter, 2019).
8 (opt-350m); 16 (opt-
6.7b); 32 (opt-1.30) 5e-5. 5e-6, 5e-7
max train epochs
15
LoRA rank 8
LoRA
8
To avoid an expensive hyperparameter search, we set constant batch size for each model (the largest fitting in memory), with constant learning rate schedule. Then, the only hyperparameter we vary is the learning rate from 5c-5, 5c-6 to 5c-7. For each learning rate, average final evaluation PPL after training over three random seeds, and we choose the best learning rate by the lowest final evaluation PPL. All values reported are averaged over random seeds for the best learning rate.
AdamW 31, 31,82
(0.9, 0.999)
AdamW e
1e-6
Table C.3: Hyperparameters used finetuning, where OPT-350m is fully finetuned and all other models are finetuned using LoRA.
Hyperparameters can be found in table C.3. Code is adapted from HuggingFace implementa- tions and can be found at https: //gi tit..b.... comb

---

chengemily1/idd.rriiiing
change (last-first) and range (max-min), which summarize the spread of ID, i.c., how data is pro- cessed in terms of expansion or reduction over lay- ers.
D ID over layers
In Section 4, we have presented results focusing on max ID as our aggregate measure across layers. However, the evolution of ID across layers is interesting problem in itself, and reveals how Trans- formers sequentially compress (or decompress) lin- guistic representations (cf. Valeriani et al., 2023 and Ansuini et al., 2019 for equivalent studies on protein sequence and vision models, respectively).
We find that when one aggregation of ID is cor- related to PPL, so are other aggregations. For in- stance, as the max ID influences the mean ID, they are often both correlated to PPL. For estimators CorrInt, TwoNN, ESS, TLE, and MLE, we notice virtually all aggregations of ID correlating to per- plexity in OPT-350m (right column of figs. E.6 and E.7).
In the large majority of cases, ID is relatively stable across layers, justifying the choice of focus- ing on an aggregate measure: see representative examples in fig. D.2a. We do, however, also ob- serve small clusters datasets where the ID profile across layers exhibits different patterns. For exam- ple, for a number of datasets (example fig. D.2b), we observe first increase in ID in the early layers followed by a later increase at the middle layers. which might suggest sionality expansions before new reductions, along the lines of what was observed by Ansuini al., 2019. Another interesting pattern pertains to the OPTCorpus, showing a late increase in dimension- ality. Intriguingly, this pattern is not disrupted by the ablations, as shown in fig. D.2c. Still, we want to emphasize that in most cases ID profiles are rel- atively flat: even when they aren't (as in figs. D.2b and D.2c), the fluctuations stay within relatively narrow ranges. We conjecture that residual con- nections have a stabilizing effect on ID across lay- ers. Notably, however, Valeriani et al., 2023 found that Transformers applied to visual and protein se- quence data displayed an early peak in ID that is missing in our linguistic datasets, whose profiles vary in shape. We leave fuller understanding of the differences to future work.
ID vs. Perplexity We observe the following about the Spearman correlation between aggrega- tions of ID and dataset perplexity:
Correlation strength tends to increase model size decreases. This is seen by look- ing the leftmost columns of the correlation plots: there are more significant correlations (colored squares) as model size decreases figs. E.6 and F.7.
dimen-
When significant, min, max, first, last, mean, and median ID are positively correlated to PPL for all model sizes and ID estimators.
When significant, change in ID is negatively correlated with PPL for all model sizes and ID estimators. That is, intuitively, if an input is harder for the model to process (larger PPL), it projects its features into higher-dimensional space for next-token prediction, which we in- terpret be a ceemel--ppansion-liie operation à la Ansuini al., 2019.
When significant, range of ID positively cor- relates to PPL. for all model sizes and ID estimators. This correlation can be partially explained by those between the min and max ID and PPL.
E Extended Results
ID vs. Ease-of-Adaptation We find that sev- eral ID aggregates are good predictors of ease-of- adaptation across model size and ID estimator. We observe the following:
Here, we review results on the relationship between geometric compression, information -thooreticc com- pression, and ease-of-ade ease-of-adaptation for all model sizes, ID estimators, and aggregations of ID.
For Correlation Dimension and NN-based ID estimators, when significant, point estimates of ID (e.g., max, mean) positively correlate to final evaluation PPL and sample com- plexity, as evidenced by the orange-colored squares in the bottom-right of plots in figs. F.6 and E.7. This trend breaks down for PCA and FisherS, but appears robust to model size.
Alternative Aggregations of ID In addition to the max ID over layers, which is discussed in the main article, we test the following other aggrega- tions: min, first (ID of positional embeddings), last (ID before next-word prediction), mean, and median, which are sointt-ggregateer of ID, and

---

Relatvve Intrind Dimscnaion (ESS)
0.08
B220
6153
Delative Intrinsic Dimension
0.08
Dimension
0.06
aptcrpuu nandor
sptceppus
3ptcrpuu por pertuused
0.08
2-2
pelative
0.02
0.00
0.0
0.00
0.6
1.0
0.8
0.
Reletive Layer Depih (opt-6.7b)
Relative Layer Depth 0.70)
Relabive Layer Depth (opt-6.76)
(a) Relatively stable ID
(b) Expansions then reductions in ID
(c) ID profiles for OPTCorpora
Figure D.2: Representative (relative) ID profiles, computed with ESS, of dataset representations against layer depth for OPT-6.7b, qualitatively grouped into three categories: (a) "flat" (the most common case), (b) "hilly", and (c) gradual increase then sudden decrease, corresponding to OPTCorpora. profiles are calculated on min 50000) sequences for each dataset shown, where N is the number of sequences in the dataset.
Results on ID spread (change and range) are inconclusive, giving sometimes negative and sometimes positive correlations depend- ing on the estimator and model size.
ID vs. Shallow Linguistic Descriptors The re- lationship between ID and shallow linguistic de- scriptors like vocab size, vocab entropy, average sequence length, and number of tokens appears to diverge by ID method category. We observe the following (see figs. E.6 and E.7):
Trends uppear to be opposite for PCA/FisherS and the other TD estimators.
For PCA/FisherS, ID generally correlates neg- atively to linguistic descriptors like average input length, vocab size, and vocab entropy.
When significant, vocab size and entropy gen- crally correlate positively to ID aggregates for estimators that aren't 't PCA/FisherS.
ID and Linguistic Structure Beyond ESS (sec- tion 4.3), the increasing trend in ID/PPL for vari- ants of wikitext, successively baseline < permuted swapped S random, also holds for virtually all ID estimators and model sizes (fig. E.5)- all but PCA/FisherS (not pictured in the figure), who esti- mate ID of all variants to be baseline = permuted swapped < random.

---

opt-1.3b
Finetune Sumppi Compleiity (log)
opt-1.3b
opt-1.3b
tokone
Finetune Eval. PPL, (lag)
25
140 p-d
1p-0.011
p-1 (p-0.00
F-0.55 (p-
20
20
ESS Mak
1000
80
-0
30
60
40
Dataset Perplexity (log)
50
100
ESS Max ID
150
200
250
50
100
ESS Max ID
150
200
250
(a) ESS max ID vs. log-PPL PPL
(b) ESS sample complexity vs. max ID
(c) ESS final PPL (log) vs. max ID
opt-350m
Fiosttun Somppe Complestt (egg
opt-350m
opt-350m
140
pmd 66 1p-00.00
12
pm (p-0.003)
Fineture Eval. PPL (log)
#-0.01 (p 0.00)
120
Mod, ootte
-0
30
Max
80
13
60 100
60 80
100
Dataset Perplexity (log)
ESS
(d) ESS max ID vs. log-PPL
(e) ESS sample complexity max ID
(1) ESS final PPL (log) vs. max ID
Figure E.3: For OPT-1.3b (top row) and OPT-350m (bottom row), max estimated with ESS is positively correlated to (a,d) dataset PPL (significant with S 0.01 for both models); predicts aase-of-adaptation metries (b,e) sample complexity (significant with p I 0.01 for both models); and (c,f) final evaluation PPL (significant with p S 0.01 for both models). Datapoints for wikitext and its perturbations subset of datasets tested that are suitable for a causal language modeling objective.
labeled for reference. The right two plots contain the
Spearman Corr. (opi-350m, am0.1)
Layer-wise Spearman Corr. ID Methods (opt-350m. e
0.1]
3,00
PEA
1.00
8.89
0:30
-0.22
1228
-0.10
Perplesity
9.75
Rohers
0.89
1.00
0:30
10:31
0.33
0.11
-.32
-016
0.26
0.75
Input Longth
1.00
0.68
0.43
0.25
TLE
0.22
0.30
0.52
1.00
0.95
0.89
0.73
0.82
0.02
Vocab Size
0.68
1.00
0.78
5,00
MLE
0.17
0.33
0.51
0.95
1.00
0.85
0.75
0.76
0.85
0.02
Vocab
0.78
1.00
3.25
0.50
TwchN
0.28
0.33
0.39
0.73
0.75
0.60
1.00
0.42
0.50
3.50
0.75
NON
0.24
.0.16
0.42
0.82
0.76
0.74
0.42
1.00
0.79
0.75
Perplestt
Impust Langeth
Vosut Enerupy
Stze
0.10
0.82
0.77
Hooaa
Rillss
TLE
MILE
TwoMN
NADA
(a) OPT-350m data correlations
(b) OPT-350m ID correlations
Spearman Corr. (opt-1.3b, a=0.1)
Layer-wist Spearman Com..
(opt-
30,
0.1)
PCA
1.00
0.94
0:37
00:30
0.28
0.22
Perp
:.00
3.75
Raherr
0.32
0.38
Length
3.50
Contro
0.17
0.25
1.00
0:44
0.27
0.60
0:22
0:25
0.13
2.00
0.25
Vocab
3:76
0.72
3.00
MLE
0.11
0.30
0.27
0.85
0.00
0.59
0.64
0.68
0.77
0.00
Vocab Entropy
3.50
0.
77
1.00
-025
ES5
0.28
0.32
0.60
0.79
0.59
1.00
0.28
0.60
0.45
1.00
-0.75
MOM
0.47
0.38
0.25
0.48
0.68
0.60
1:00
0.93
Perpieetty
Inpoll Length
Vacatt Rize
Werah Earray
Size
1.00
MADA
0.22
0.12
0.13
0.52
0.77
0.40
0.38
0.98
1.00
-3.75
1.000
Fighers
TUE
MIF
E3S
TwoWW
MMDX
(c) OPT-1.3b data correlations
(d) OPT-1.3b ID correlations
Figure E.4: Data metrics Spearman correlations for left column (a) OPT-350m and (c) OPT-1.3b; and ID metrics Spearman correlations for right column (b) OPT-350m and (d) OPT-1.3b. Note the inter-correlations between shallow linguistic descriptors imput length, vocab size, and vocab entropy for all model sizes, as well as the grouping of ID metrics into one block corresponding to NN-based methods (bottom right of each plot).

---

opt-6.7b
ppt-1.3b
opt-350m
30
Corfint Max
25
Conrint Max
Confin Max
18
20
15
15
15
10
10
Dataset Perplexity (log)
Dataset Perplexity (log)
Dataset Perplexity (log)
pp
opt-1.3b
apt-350m
100
p-0.15 (2-05
p.0.22 (p#0:271
p-3.50 12=0.000
TwoNN Max ID
40
TwomM Max ID
TwoMN Max tD
6
50
Wrsporiinaa rad, ahhrr
40
20
10
10
Dataset Perplexity (log)
8
12 14
Dataset Perplexity (log)
32 Dataset Perplexity (lag)
opt.6.7b
opt-1.3b
apt-350m
p-0.3 12-0.02)
2p=
p-3:30 (2-0.00)
TLE Mox ID
TLE Max ID
15
TLE Max
'8
40
Breee parr atoee
15
20
10
10 Dataset Perplexity (log)
6
12 14
Dataset Perplexity (log)
Dataset Perplexity (log)
10
12
opt-6.7b
opt-350m
opt-1.3b
140
p-0.1 1p-00..000 251
45
5-
1p=0.001
120
40
MLE Max ID
1000
MIEE Max ID
MLE Max
20
gini pereuted Jairss
23
60
10
10 Dataset Perplexity (log)
8
10 Dataset Perplexity (log)
5
8
10
Dataset Perplexity (log)
ppt-6.7b
opt-1.3b
apt-350m
140
p-02 (2-)
p- (p-rd)
p-0.42 (b46.64)
MOM Max ID
120
MOM Max ID
100
50
Med Good souped Claes.
MOM Max
40
20
Onders
L5
2 aarorr seoppee
10
12 Datasel Perplexity (log)
Dataset Perplekity (log)
Datasel Perplexity (log)
ppt-6.7b
opt-1.3b
apt-350m
175
p=0024
(2-0.24)
p-2.12
(o-0.56)
p-2.42 (2-0.04)
MADA Max: 1D
150
MADA Max ID
MADA Max:
125
60
pvaszed Jobers
100
8
75
30
Dataset Perplexity (log)
4
8. 10 12 14
Dataset Perplexity (log)
Dataset Perplexity (lag)
10 12
Figure E.5: For OPT-6.7b (left), O.T-1.3b (middle) and OPT-350m (right), and for all other ID estimators considered, we plot max ID vs. dataset PPL (log), labeling the points for wikitext. Across estimators and model sizes, we see a general trend that ablating linguistic structure increases ID well PPL., in the order of baseline permuted < swapped & random.

---

upt-6.72 ICA Speammn Sammary (om0
spi-1.36 ICA Sprarmm Car.. Summarr (pwd.A)
opt-350m PLA Spearman Cari. Summary lomb.1)
Lo
(a) OPT-6.7b, PCA
(b) OPT-1.3b, PCA
(c) OPT-350m, PCA
opt-8.7 Saiterr Spearman Corr. Summary (am0.3) up
Fisherr Spaurman Conr. Summary lamb 3)
opt-35 Fisherr Spearman Corr. Summary (am0.1). 2.13
(d) OPT-6.7b, FisherS
(e) 1.3,1.3,, FisherS
( 3PT-350m, FisherS
apt-d.7b Contre Spearman Summary 0.30
opt-1.1b Contrel Speaman Corr. Summary (amm
apt-755m Contert Spearmen Canr.. Suammary Jomo
Con
T-r Ca
(g) OPT-6.7b, Corrint
(h) 1.3, Corrint
ii OPT-350m, Corrint
opt/5/b TeehhN Spear man Summary (ow0
opt-1.bb Twokt Spearman Corr. Summary lawD
vpt-350m Twcht Spearman Carr. Summary law0 11
hD
i OPT-6.7b, TwoNN
(k) OPT-L.3b, TwoNN
() OPT-350m, TwoNN
Figure H.6: Global ID Estimators: full panel of Spearman correlations for models (left to right columns) OPT-6.7b, 1.3b, and 350m. Each figure is a summary of Spearman correlations given a model and global ID metric, significant at a 0, between aggregated ID and data metrics (top left), case-of-finetuming metrics and data metrics (bottom left), and ease-of- of-finttuning and ID (bottom right).

---

opt-6.75 ESS Speartan Corr. Summary 5.1
opt-3 Te ESS Spearman Carr. Summary (am
ope-750m ESS Speamann Conr. Summary Co
(a) OPT-6.7b, ESS
(b) OPT-1.3b, ESS
(c) OPT-350m, ESS
Summary (pm0.1)
Speattmen Summary (ym0.33
(d) OPT-6.7b, TLE
(c) OPT-L.3b, TLE
(1) OPT-350m, TLE
opt-6.b MLE Spearmnn Carr. Summary (.m0.11
opt-1 10 MLE Speemmnn Curr. Summary (pm0.31
mpt-350m MI.Et Speatman Corr. Summary (am0.
225
C.1
(g) Ob,-6.7b, MLE
(b) L.3. MLE
i OPT-350m, MLE
Summary
Speasman Summory
Summary
i OPT-6.7,, MOM
(k) OPT-1.3h, MOM
(I) OPT:350m, MOM
7p WAGA Spearmnn Corr. Summarry (gmn D.1)
opt-3.36 2. MALAA Spuarman Corr. Summary (gm 2:11
300-350m MA.DA Spearman Cort. Summary (pm
(m) OPT-6.7b, MADA
(n) OPT-L.3b, MADA
(0) OPT-350m, MADA
Figure E.7: Local ID Estimators: full panel of Spearman correlations for models (left right columns) OPT-6.7b, 1.3b, and 350m. Each figure is summary of Spearman correlations given a model and ID metric, significant at a = 0.1, between aggregated ID and data metrics (top left), casc-of-finee -ffeetuinng metrics and data metrics (bottom left), and ease-of case-of-finetuning finetuning and ID (bottom right).


Reviews:
[{"review_id": "yzcGdIJUDH", "reviewer": "Reviewer_78nc", "summary": "", "questions": "A. It is not clear to me that the datasources are diverse enough. Most datasets seem to contain factual information and are likely to contain simple sentences. How likely is it that the results would differ on datasets containing a higher proportion of complex sentences, like classic novels, etc.?\n\nB. To what extent are the effects of ablating syntax related to the choice of representation? In your opinion, would using a different representation, like an average or concatenation of all the token embeddings change any trends? I know this is out of scope of this paper but informed speculation would still be helpful.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback! We are glad you found our work to be interesting and thorough. To respond to your points,\n\n1. In light of limited computational resources, we chose to focus on experiments spanning a breadth of model sizes and datasets. We agree, though, that our results can be strengthened by testing different model families. Therefore, towards including a more expansive set of experiments in the camera-ready, we have thus far launched experiments for BLOOM-560m this week, so far finding similar ranges of perplexity and ID to OPT-6.7b. \nWe plan to replicate the full experiments for BLOOM-560m and for LLAMA 2 to include in the appendix of the camera-ready.\n\n2. __Clarity points__ (we will also clarify them in the revision):\n\n    a. We state in l338-341 that, like all other datasets that are not [Super]GLUE, the models are fine-tuned on the transformed datasets (e.g. wikitext-permuted) so as to compute their finetuning sample complexity and final PPL. \n\n    b. Fig 2a uses models before finetuning, or the \"zero-shot perplexity\" mentioned in your next comment. We will make this clear in the figure caption.\n\n    c. We agree that (l359) zero-shot perplexity may be unclear. We'll change it to \"zero-shot perplexity ($PPL_0$), or perplexity evaluated on the pre-trained model before finetuning\".\n\n3. __Questions__\n\n    A. We took care to include a variety of data sources, including one containing classic novels (bookcorpus, see Table 1), as well as those which are \"non-language\" (the syntax-ablated datasets), a Java code dataset, encyclopedic data, encyclopedic data in a different language (wikitext french), data scraped from the web (openwebtext-2), etc, which we believe represent a diverse set of language that covers in-distribution and out-of-distribution for the model. Moreover, the data are diverse in terms of perplexity and ID, which is the main focus of our study.\n\n    On a more general note, we think that having, e.g., a high proportion of complex sentence structure or rare vocabulary items is interesting insofar as it is a __distribution shift__ from the model's learned distribution of language. In future work, we definitely plan to experiment with these distribution shifts in a controlled way, seeing how types of shifts, e.g. increasing syntactic complexity or changing word frequency, affect perplexity and ID.\n\n    B. Thanks for the interesting question! We would like to try other representations that take all tokens into account. It is possible that the choice of representation will change the sensitivity of the analysis to syntax ablation. Note that averaging all of the token embeddings might actually make the representation less sensitive to syntactic structure, as it muddles information about word ordering. A concatenation-based representation is more promising, but we need to solve the technical issue of how to compare sentences/paragraphs with a different number of tokens in this setup.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Experiments are thorough and expansive, with variety of estimation methods, and datasets being tested, which lends credence to results.\n- Although many of the results presented in the paper may have been expected, it is valuable that they have been demonstrated and quantified through rigorous experimentation.\n- This paper might stimulate interest in analyzing linguistic characteristics via LLMs that might also enhance our understanding of human language.", "reasons_to_reject": "1. Lack of diversity in models. All the models are of the OPT family. Perhaps models with different training data like BLOOM, or different architectures and training methodolgies like Llama should have been included to marginalize out the impact of modelling choices made in OPT.\n1. Lack of clarity in some places:\n    1. It is unclear if the model is fine-tuned on the transformed dataset.\n    1. It is unclear which results are based on the fine-tuned models. Does Fig 2 (a) use fine-tuned models, or does it simply evaluate the pretrained model with these datasets?\n    1. Zero-shot perplexity is not defined.", "questions_to_authors": "A. It is not clear to me that the datasources are diverse enough. Most datasets seem to contain factual information and are likely to contain simple sentences. How likely is it that the results would differ on datasets containing a higher proportion of complex sentences, like classic novels, etc.?\n\nB. To what extent are the effects of ablating syntax related to the choice of representation? In your opinion, would using a different representation, like an average or concatenation of all the token embeddings change any trends? I know this is out of scope of this paper but informed speculation would still be helpful.\n", "paper_topic_and_main_contributions": "This paper analyzes the relationship between geometric compression (as indicated by the intrinsic dimensionality), information-theoretic complexity (measured as perplexity) and ease of adaptation (measured as perplexity after fine-tuning) for the representations learned by large language model. Their experiments show that geometric compression predicts information-theoretic complexity and ease of adaptation. They verified that this relationship holds even when linguistic structure is progressively destroyed -- ablating syntax and semantics increases both perplexity and intrinsic dimensionality. Furthermore, their results also indicate that despite having different extrinsic dimensionality (defined by the representation size), LLMs tend to have similar and rather small intrinsic dimensionality that is in O(10).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FuU0WOmBAF", "reviewer": "Reviewer_xjpS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reading our work! We appreciate that you found the ablation study interesting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Their ablation study of the relationship between compression on data representations and network reduction.", "reasons_to_reject": "None", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors investigate the relationship between compression on data representations and network reduction. \nThey find that correlation between adaption of casual language models and data compression. They also show an ablation study of different techniques and the extent to which they capture the relationship beween the intrisic dimension of the data, coding length and ease of adaptation for casual models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sg5T4uVxnh", "reviewer": "Reviewer_LFBq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reading our work and for your positive response! Following your feedback, we intend to include a wider range of LMs in the camera-ready including BLOOM-560m and LLAMA 2, and experiments computing ID and PPL for BLOOM-560m are currently running. So far, the ID and PPL numbers are looking similar to those of OPT-6.7b.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clearly written\n- The findings of the paper could be of interest to the community ", "reasons_to_reject": "- The authors could have evaluated on a wider range of LMs in order to make their arguments stronger. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyses the Language Models in terms of geometric and information-theoretic compression. In particular, it measures the correlation between geometric compression, information-theoretic compression, and ease of adaptation. For each of these properties, it proposes to use different methods, i.e., intrinsic dimension for geometric compression, perplexity for information-theoretic compression, and, sample efficiency and validation perplexity as a measure of ease of adaptation while fine-tuning. It shows that there is a positive correlation between intrinsic dimension and perplexity as well as ease-of-adaptation. Moreover, the intrinsic dimension among different LMs in size suggests that data resides in a much smaller manifold than the extrinsic dimension. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were unanimous that this is a solid paper, and I concur. To add to their notes, I appreciate the distinctive hypothesis that this tests relative to much of the analysis work in NLP.

As for positives, reviewers note the rigor and likelihood of stimulating discussion and understanding of LMs. For negatives, reviewers wish for more LMs in the evaluation; I’m not overly concerned about this. I think the paper is rigorous and relatively clear, but additional work could be put into presentation of the various compression metrics and their evaluation (some nice figures would help.)

Adding my own concern, I think the destruction of linguistic structure through vocabulary swapping etc. is rather severe, and so the increase in intrinsic dimensionality seems less interesting. I encourage the authors to consider some less severe transforms, like noisy paraphrase.

The limitations section notes the disconnect between the somewhat lofty discussion of infinite use of finite means in the introduction and the concrete experiments in the paper; I think that walking readers along the connections would help this!
Decision: Accept-Main