On Linear Representations and Pretraining Data Frequency in Language Models
Authors: Jack Merullo
Noah A. Smith
Sarah Wiegreffe
Yanai Elazar
Venue: ICLR 2025 Poster (2025)
Keywords: pretraining data
pretraining
linear
linear features
interpretability
linear representations
corpus frequency
Primary area: interpretability and explainable AI
TL;DR: Language Models (LMs) form linear representations of relations when the average subject-object frequency surpasses a certain threshold

Abstract: Pretraining data has a direct impact on the behaviors and quality of language models (LMs), but we only understand the most basic principles of this relationship. While most work focuses on pretraining data's effect on downstream task behavior, we investigate its relationship to LM representations. Previous work has discovered that, in language models, some concepts are encoded "linearly" in the representations, but what factors cause these representations to form (or not)? We study the connection between pretraining data frequency and models' linear representations of factual relations (e.g., mapping France to Paris in a capital prediction task). We find evidence that the formation of linear representations is strongly connected to pretraining term frequencies; specifically for subject-relation-object fact triplets, both subject-object co-occurrence frequency and in-context learning accuracy for the relation are highly correlated with linear representations. This is the case across all phases of pretraining, i.e., it is not affected by the model's underlying capability. In OLMo-7B and GPT-J (6B), we discover that a linear representation consistently (but not exclusively) forms when the subjects and objects within a relation co-occur at least 1k and 2k times, respectively, regardless of when these occurrences happen during pretraining (and around 4k times for OLMo-1B). Finally, we train a regression model on measurements of linear representation quality in fully-trained LMs that can predict how often a term was seen in pretraining. Our model achieves low error even on inputs from a different model with a different pretraining dataset, providing a new method for estimating properties of the otherwise-unknown training data of closed-data models. We conclude that the strength of linear representations in LMs contains signal about the models' pretraining corpora that may provide new avenues for controlling and improving model behavior: particularly, manipulating the models' training data to meet specific frequency thresholds. We release our code to support future work.

Published as a conference paper at ICLR 2025
ON LINEAR REPRESENTATIONS AND PRETRAINING DATA FREQUENCY IN LANGUAGE MODELS
Jack Merullo
Noah A. Smith +
Sarah Wiegrelfe
Yanai Elazar 9
Brown University, Allen Institute for Al (Ai2), University of Washington
* Co-senior authors.
ja ck.meruu lorlll@@breee wowroon. edu, {noah, sarahw, yanaie)@allemai.co
ABSTRACT
Pretraining data has a direct impact on the behaviors and quality of language mod- els (LMs), but only understand the most basic principles of this relationship. While most work focuses on pretraining data's effect on downstream task be- havior, investigate its relationship to LM representations. Previous work has discovered that, in language models, some concepts are encoded 'linearly' in the representations, but what factors cause these representations form (or not)? We study the connection between pretraining data frequency and models' linear rep- resentations of factual relations (e.g., mapping France Paris in capital predic- tion task). We find evidence that the formation of linear representations is strongly connected to pretraining term frequencies: specifically for subject-relation-biee fact triplets, both subject-object co-occurrence frequency and in in-tontext learning accuracy for the relation are highly correlated with linear representations. This is the case across all phases of pretraining, i.e., it is not affected by the model's underlying capability. In OLMo-7B and GPT-J (6B), we discover that linear representation consistently (but not exclusively) forms when the subjects and ob- jects within a relation co-occur at least 1K and 2k times, respectively, regardless of when these occurrences happen during pretraining (and around 4k times for OLMo- IB). Finally, we train regression model on measurements of linear rep- resentation quality in fully-trained LMs that can predict how often term was seen in pretraining. Our model achieves low error even on inputs from a different model with different pretraining dataset, providing a new method for estimat- ing properties of the otherwise-unknowd conclude that the strength of linear representations in LMs contains signal about the models' pretraining corpora that may provide new avenues for controlling and improving model behavior: particularly, manipulating the models' training data to meet specific frequency thresholds. We release our code to support future work
training data of closed-data models. We
INTRODUCTION
Understanding how the content of pretraining data affects language model (LM) behaviors and per-
formance is an active area of research (Ma et al. 2024: Xie et al.) 2023 Aryabumi ct al., 2025 Longpre al. 2024 Wang et al.] 2025 Seshadri et al. 2024 Razeghi al.) 2023 Wang et al. 2024]. For instance, it has been shown that for specific tasks, models perform better on instances containing higher frequency terms than lower frequency ones (Razeghi al., 2022; Mallen al.) 2023 McCoy et al. 2024). However, the ways in which frequency affects the internal representa- tions of LMs to cause this difference in performance remain unclear. We connect dataset statistics to recent work in interpretability, which focuses on the emergence of simple linear representations of factual relations in LMs Hermannezz al.] (2024); Chanin al. (2024). Our findings demonstrate strong correlation between these linear representations and the frequency of terms in the pretraining corpus.
Cooe is available http:::://ggthuubbmmmaaaaaaaaa/a
and for efficient batch search
https: : httpps://gitubb.com a /

---

Published as a conference paper at ICLR 2025
Linear representations in LMs have become central to interpretability research in recent years (aav- fogel et al.
2020
Elazar et al.
2021
El2gee et al.
2021,
Slobodkin et al. 2023 Olah et al. 2020
Park et al. 2024 Jiang al.
2024 Black et al.
2022 Chanin et al. (024). Linear representa-
tions are essentially ly linear approximations (linear transforms, directions in space) that are simple to understand, and strongly approximate the complex non-linear transformations that networks implementing. These representations are crucial because they allow us to localize much of the be- havior and capabilities of LMs to specific directions in activation space. This allows for simple interventions to control model behaviors, i.e., steering (Todd et al., 2024 Subramani et al. 2022
Hendel et al., 2023 Rimsky al., 2024).
Recent work by Hernandez et al. (2024) and Chanin et al. (2024) highlight how the linearity of dif- ferent types of relations varies greatly depending on specific relationships being depicted. For example, over 80% of entities in the "country-largest-city relation, but less than 30% of entities in the iitar-in-constellation" relation can be approximated this way (Hernandez et al. (024). Such findings complicate the understanding of the Linear Representation Hypothesis, which proposes that LMs will represent features linearly (Park et al. 2024) without providing when/why these form. While Jiang al. (2024) provide both theoretical and empirical evidence that the training objec- tives of LMs implicitly encourage linear representations, it remains unclear why some features are represented this way while others not. This open question is central focus of our investigation.
Whether linear representations for "common" concepts are more prevalent models or simply easier to identify (using current methods) than those for less common concepts remains unclear. We hypothesize that factual relations exhibiting linear representations are correlated with higher mention frequencies in the pretraining data (as has been shown with static embeddings, see Ethayarajh al.) 2019). which we confirm in Section 4 Our results also indicate that this can occur at any point in pretraining, as long as a certain average frequency is reached across subject-object pairs in relation. In order to count the appearance of terms in data corpora throughout training, we develop an efficient tool for counting tokens in tokenized batches of text, which we release to support future work in this area. We also explore whether the presence of linear representations can provide insights into relation term frequency. In Section 5 we a regression model to predict the frequency of individual terms (such "The Beatles") in the pretraining data, based on metrics measuring the presence of linear representation for some relation. For example, how well a linear transformation approximates the internal computation of the "lead-singer-of" relation mapping "John Lennon" to "The Beatles" can tell us about the frequency of those terms in the pretraining corpus.
Our findings indicate that the predictive signal, although approximate, is much stronger than that encoded in log probabilities and task accuracies alone, allowing us to estimate the frequencies of held-out relations and terms within approximate ranges. Importantly, this regression model gen- eralizes beyond the specific LM it was trained on without additional supervision. This provides a valuable foundation for analyzing the pretraining corpora of closed-data models with open weights.
To summarize, in this paper we show that:
1. The development of linear representations for factual recall relations in LMs is related to frequency as well as model size.
2. Linear representations form at predictable frequency thresholds during training, regardless of when this frequency threshold is met for the nouns in the relation. The formation of these representations also correlates strongly with recall accuracy.
3. Measuring the extent to which relation is represented linearly in model allows to predict the approximate frequencies of individual terms the pretraining corpus of that model, even when we do not have access to the model's training data.
4. We release tool for accurately and efficiently searching through tokenized text to support future research on training data.
2

---

Published as a conference paper at ICLR 2025
Pretraining Corpus
Measuring linear structure
Miles Davis
trumpet
Bill Evans
piano
token_ cooccurrence( corpus, "Miles Davis", "trumpet"
Cat Stevens
guitar
Frequency Efficacy of LRC. r=.82
Causallyy (LAEquality)
(2128, 0.8)
2128
0.6
Log Subj-Obi Co-Oocurence
Figure 1: Overview of this work. Given a dataset of subject-relation-ohjee factual relation triplets, count subject-object co-occurrences throughout pretraining batches. We then measure how well the corresponding relations are represented within an LM across pretraining steps, using the Linear Relational Embeddings (LRE) method from Hernandez al. (2024). We establish strong relation- ship between average co-occurrence frequency and model's 's tendency to form linear representations for relations. From this, we show that we can predict frequencies in the pretraining corpus
2 BACKGROUND
2.1 LINEAR REPRESENTATIONS
Vector space models have long history in language processing, where geometric properties of these spaces were used to encode semantic information (Salton et al. 1975 Paccanaro Hinton 2001]. When and why linear structure emerges without explicit bias has been of considerable interest since the era of static word embeddings. Work on skipgram models (Mikolov et al., 2013a) found that vector space models of language learn regularities which allow performing vector arithmetic be- tween word embeddings to calculate semantic relationships (e.g.. France - Paris + Spain = Madrid) MMikolov al., 2013b Pennington al., 2014). This property was subject to much debate, was not clear why word analogies would appear for some relations and not others (Köper et al.) 2015 Karpinska et al. 2018 Gladkova et al. 2016). Followup work showed that linguistic regular- ities form in static embeddings for relations under specific dataset frequency constraints for relevant terms (Ethayarajh et al. 2019), but does not clearly relate to how modern LMs learn. More recently, there has been renewed interest in the presence of similar linear structure in models with contextual embeddings like transformer language models (Park al. [024 Jiang al. 20244) Merullo al.) 2024). As result, there are many ways to find and test for linear representations modern LMs. though the relationship to pretraining data was not addressed (Huben et al. 2024 Gao et al.) 2025
Templeton et al., [2244] Rimsky et al. 2024: Todd et al.) 2024 Hendel et al. 2023 Hernandez al. 2024| et al. (2024). Many these share similarities in how they compute and test for linear particular class of linear representations called Linear Relational
representations. We focus
Embeddings (LREs) (Paccanaro & Hinton 2001).
Linear Relational Embeddings (LREs) Hernandez et al. (2024) use a particular class of lin- ear representation called a Linear Relational Embedding (Paccanaro & Hinton] (200T] to ap- proximate the computation performed by a model to predict the objects that complete common reeejeetteelaiionn
triplets as an affine transformation. This transform is calculated from hidden state the subject token representation some middle layer of the model, to the hidden state at the last token position and layer of the model (i.e., the final hidden state that decodes token in autoregressive transformer) within natural language description of the relation. For example, given the input sequence "Miles Davis (subject) plays the (relation)) the goal is to approximate the computation of the object "trumpet", assuming the model predicts the object
3

---

Published as a conference paper at ICLR 2025
rectly. It was found that this transformation holds for nearly every subject and object in the relation set (such as "Cat Stevens plays the guitar") for some relations. This is surprising because, despite the nonlinearities within the many layers and token positions separating and o, simple structure within the representation space well approximates the model's prediction process for a number of factual relations. In this work we study LREs under the same definition and experimental setup, because it allows us to predefine the concepts we want to search for (e.g., factual relations), as well as use handful of representations to relate thousands of terms in the dataset by learning linear representations on per-relation level.
Hemandez al. calculate LREs to approximate an LM's computation as a first-order Taylor Series approximation. Let Fss, c) o be the forward pass through model that produces object represen- tation o given subject representation and few-shot context c, this computation approximated as F(s, c) Rv Ws + b F(s,,c) c) + W (s s:) where we approximate the relation about a specific subject s, Hernandez et al. propose to compute W and using the average of examples from the relation (n 8 here) with Us
o
representing the Jacobian Matrix of F:
OF
and = Eas,er F(s, OF
W = Ex,.
(1)
Üs
Øs (8p)))
(8))0))
In practice, LREs are estimated using hidden states from LMs during the processing of the test example in few-shot setup. For relation like paapeeeeetpaaadd---------e see four examples (in the form "[X] plays the [Y]") and on the fifth example, when predicting e.g., "trumpet" from "Miles Davis plays the",, the subject representation and object representation o are extracted.
the model may
2.2 INFERRING TRAINING DATA FROM MODELS
There has been significant interest recently in understanding the extent to which is possible to infer the training data of a fully trained neural network, including LMs,, predominantly by performing membership inference attacks (Shokri al.
2017 Carlini et al. 2022). judging memorization of text (Carlini et al., 2023 Oren et al. 2024 Shi et al.] (2024), or inferring the distribution of data sources (Hayase al. [0244] Ateniese al.
2015 Suri & Evans 2022). Our work is related that we find hints of the pretraining data distribution in the model itself, but focus on how linear structure
in the representations relates to training data statistics.
METHODS
Our analysis is twofold: counts of terms in the pretraining corpus of LMs, and measurements of how well factual relations are approximated by affine transformations. We use the OLMo model v1.7 (0424 7B and 0724 1B) (Groeneveld et al.) 2024) and GPT-J (6B) (Wang & Komatsuzakij 2021 and their corresponding datasets: Dolma (Soldaini et al.) 2024) and the Pile (Gao et al, 2020), respectively. To understand how these features form over training time, we test eight model checkpoints throughout training in the OLMo family of models (Groeneveld et al. 2024).
3.1 LINEAR RELATIONAL EMBEDDINGS (LRES) IN LMS
We use a subset of the RELATIONS dataset Hernandez et al. (024), focusing on the 25 factual relations of the dataset, such as capital-city and person-mother (complete list in Appendix B Across these relations, there are 10,488 unique subjects and objects. Following Hernandez et al. (2024), we an LRE for each relation 8 examples from that relation, each with a 5-shot prompt. We use the approach from this work as described in Section 2.1
Foor the analysis, we drop "landmark-on sonndmarkkon--comiinent because 74% of the answers are Antarctica, making potentially confounding for extracting a representation for the underlying relation. Factual relations are much easier to get accurate counts for, so we leave non contuactual relations for future work (e.g.. although LMs associate the "pilot" occupation with men, this relation does not map the word "man" the way "France" maps to "Paris": see 43.2
4

---

Published as a conference paper at ICLR 2025
Fitting LREs Hernandez et al. (2024) find that Equation 1 underestimates the optimal slope of the linear transformation, so they scale each relation's i W by a scalar hyperparameter 3. Unlike the original work, which finds one 3 per model, we use one 3 per relation, this avoids disadvantaging specific relations. Another difference in our calculation of LREs is that we do not impose the constraint that the model has to predict the answer correctly be used as one of the 8 examples used to approximate the Jacobian Matrix. Interestingly, using examples that models predict incorrectly to fit Equation I works as well as using only correct examples. We opt to use this variant as it allows us to compare different checkpoints and models ($4) with linear transformations trained on the same 8 examples, despite the fact that the models make different predictions on these instances. We explore the effect of example choice in Appendix B and find that it does not make significant difference. We also explore the choice of layer in Appendix
Metries To evaluate the quality of LREs, Hernandez et al. (2024) introduce two metrics that mea- sure the quality of the learned transformations. Faithfulness measures whether the transformation learned by the LRE produces the same object token prediction as the original LM. Causality mea- sures the proportion of the time prediction of an object can be changed to the output of a different example from the relation (e.g., editing the Miles Davis subject representation so that the LM pre- dicts he plays the guitar. instead of the trumpet). For specifics on implementation, we refer the reader to Hernandez et al. (2024). We consider an LRE to be high 'quality' when it scores highly on these metrics, as this measures when an LRE works across subject-object pairs within the relation. In general, we prefer use causality in analysis, as faithfulness can be high when LMs predict the same token very often (like in early checkpoints).
3.2 COUNTING FREQUENCIES THROUGHOUT TRAINING
A key question we explore is how term frequencies affect the formation of linear representations. We hypothesize that more commonly occurring relations will lead to higher quality LREs for those relations. Following Elsahar et al. 2):18; Elazar et al.] (2022), we count an occurrence of a relation when subject and object co-occur together. While term co-occurrence is used proxy for the frequency of the entire triplet mentioned in text. Elsahar et al. (2018) show that this approximation quite accurate. We now discuss how to compute these co-occurrence counts.
What's in My Big Data? (WIMBD) Elazar et al. (2024) index many popular pretraining datasets, including Dolma (Soldaini et al. 2024 and the Pile (aoo et al. 2020), and provide search tools that allow for counting individual terms and co-occurrences within documents. However, this only gives us counts for the full dataset. Since we are interested in counting term frequencies throughout pretraining, we count these within training batches of OLMo instead. When per-batch counts are available, WIMBD offers a good approximation for final checkpoints, which is what we do in the case of GPT-J. We compare WIMBD co-occurrence counts to the Batch Search method (described below) for the final checkpoint of OLMo in Appendix[D] and find that the counts are extremely close: The slope of the best fit line for BatchCount against WIMBDCount is .94, because co-occurrence counts are overestimated when considering the whole document.
Batch Search Data counting tools cannot typically provide accurate counts for model checkpoints arbitrary training steps. Thus, we design tool efficiently count exact co-occurrences within sequences of tokenized batches. This also gives us the advantage of counting in a way that is highly accurate to how LMs are trained; since LMs are trained on batches of fixed lengths which often split documents into multiple sequences, miscounts may occur unless using tokenized sequences. Using this method, we note every time one of our 10k terms appears throughout a dataset used to pretrain an LM. We count a co-occurrence batch (a (batch-size, sequence-length) array). We search 10k terms in the approximately 2T tokens of Dolma (Soldaini et al. 2024) this way. Using our implementation, on 900 CPUs in about a day. To support future work, we release our code as Cython bindings that integrate out of the box with existing libraries.
any time two terms appear in the same sequence within
are able to complete this
5

---

Published as a conference paper at ICLR 2025
OLMo-7B 0424 Development LREs over Training Time
41B Tokens (10k steps)
Final Model
Causality
Co-Occurrence Threshold
Model
0.6
(Mean Causality >.9)
0.4
GPT-J (6B)
1,097
1,998
0.2
OLMo-7B
1,998
10
12
OLMo-1B 4,447
Log Obj Cooccurrence
OLMo-1B 0724 Development of LREs Training Time
GPT-J Development LREs Training Time
Causality
0.0
Causality
0.8
0.6
0.6
0.4
4, 447
1,097
0.2
Log Subj-Obj Cooccurrence
Log Subj-Obj Cooccurrence
We find that LREs have consistently high causality scores across relations after some average frequency threshold is reached (table, top right). In OLMo models, red dots show the model's LRE performance at 41B tokens, and blue dots show the final checkpoint performance ( (550k steps in 7B). Gray dots show intermediate checkpoints. We highlight Even at very early training steps, if the average subject-object cooc. count is high enough. the models are very likely to already have robust LREs formed in the representation space. Symbols represent different relations. Highlighted relations are shown darker lines f
Figure 2:
FREQUENCY OF SUBJECT-OBIECT CO-OCCURRENCES ALIGNS WITH EMERGENCE OF LINEAR REPRESENTATIONS
In this section, explore when LREs begin appear at training time and how these related to pretraining term frequencies. Our main findings are that (1) average co-occurrence frequency within relation strongly correlates with whether LRE will form; (2) the frequency effect is independent of the pretraining stage: if the average subject-object co-occurrence for a relation surpasses some threshold, it very likely have a high-quality LRE, even for early pretraining steps.
4.1 SETUP
Using the factual recall relations from the Hernandez et al. (2024) dataset, we use the Batch Search method (...p
count subject and object co-oceurrences within sequences in Dolma (Soldaini 2024) used to train the OLMo-1B (v. 0724) and 7B (v. 0424) models (Groeneveld et al.) The OLMo family of models provides tools for accurately recreating the batches from Dolma, which allow us reconstruct the data the way the model was trained. We also use GPT-J (Wang) Komatsuzaki 202T) and the Pile (Gao et al. 2020) as its training data, but since we do not have access to accurate batches used to train it,
et al.
2024
WIMBD (Elazar et al.) 2024) to count subject- object counts in the entire data. We fit LREs on each relation and model separately. Hyperparameter sweeps in Appendix C OLMo also releases intermediate checkpoints, which use track development over pretraining time. We use checkpoints that have seen [41B, 104B, 209B, 419B, 628B, 838B, IT, and 2T} tokens
We use the Pearson coefficient for measuring correlation.
4.2 RESULTS
Our results are summarized in Figure 2 We report training tokens because the step count differs between 7B and IB. Co-oceurrence frequencies highly correlate with causality (r 0.82). This
In Mo-B- 0424, this corresponds to 10k, 25k, 50k, 100k, 1.0kk 200k, 250k, 409k pretraining steps "These are: 'country largest city', 'country currency', 'company hq', "company CEO', and 'star constella- tion name' in order from best worst performing final checkpoints.

---

Published as a conference paper at ICLR 2025
is notably higher than the correlations with subject frequencies: 0.66, and object frequencies:
r = 0.59 for both OLMo-7B and OLMo- IB, respectively.
We consider causality score above 0.9 to be nearly perfectly linear. The table in Figure 2 shows the co-occurrence counts above which the average causality is above 0.9 and is shown by dashed black lines on the scatterplots. Regardless of pretraining step, models that surpass this threshold have very high causality scores. Although we cannot draw conclusions from only three models, it is possible that scale also affects this threshold: OLMo-7B and GPT-J (6B params) require far less exposure than OLMo-IB.
4.3 RELATIONSHIP TO ACCURACY
Increased frequency (or proxy for it) was shown to lead better factual recall in LMS (Chang et al. 2024] Mallen et al. 2023). However, it remains unknown whether high accuracy entails the existence of linear relationship. Such a finding would inform when we expect an LM achieve high accuracy on a task. We find that the correlation between causality and subject-object frequency higher than with 5-shot accuracy (0.82 v.s. 0.74 OLMo-7B), though both are clearly high. In ad- dition, there are few examples of high accuracy relations that do not form single consistent LREs. These relations are typically low frequency, such as star constellation name, which has 84% 5-shot accuracy but only 44% causality (OLMo-7B), with subjects and objects only co-occurring about 21 times on average across the full dataset. In general, few-shot accuracy closely tracks causality, con- sistent with arguments that in-context learning allows models to identify linear mappings between input-output pairs (Hendel et al.) 2023) Garg et al. (022). We find that causality increases first in some cases, like "jooc--from-countyy" having causality of 65% but a 5-shot accuracy of only 42%. This gap is consistently closed through training. In the final model, causality and 5-shot accuracy are within 11 % on average. We report the relationship between every relation, zero-shot, and few-shot accuracy for OLMo models across training in Appendix [F]
A fundamental question in the interpretability community is under what circumstances linear struc- tures form. While previous work has shown that the training objective encourages this type of representation (Jiang et al., 2024), our results suggest that the reason why some concepts form linear representation while others do not is strongly related to the pretraining frequency.
5 LINEAR REPRESENTATIONS HELP PREDICT PRETRAINING CORPUS FREQUENCIES
In this section, we aim to understand this relationship further by exploring what we can understand about pretraining term frequency from linearity of LM representations. We target the challenging problem of predicting how often term, or co-occurrence of terms, appears in an LM's training data from the representations alone. Such prediction model can be useful, if it generalizes, when applied to other models whose weights are open, but the data is closed. For instance, such predictive model could tell us whether a model was trained on specific domains (e.g.. Java code) by measuring the presence of relevant LREs. First, we show that LRE features encode information about frequency that is not present using probabilities alone. Then, we show how a regression fit on one model generalizes to the features extracted from another without any information about the new model's counts.
5.1 EXPERIMENTAL. SETUP
We fit a regression to the Relations dataset (Hernandez al. 2024) using OLMo-7B LRE features and log probabilities. We fit 24 models such that each relation is held out once per random seed across seeds. We train random forest regression model with 100 decision tree estimators to predict the frequency of terms (either the subject-object frequency, or the object frequency alone; e.g., predicting "John Lennon" and "The Beatles" or just "The Beatles") from one of two sets of features. Our baseline set of features is based on likelihood of recalling a fact. Given some few-shot context from the relations dataset ("John Lennon is a lead singer of"") we extract the log probability of the correct answer, as well as the average accuracy on this prompt across 5 trials. The intuition is that models will be more confident about highly frequent terms. The other set of features include the first, as well as faithfulness and causality measurement.
7

---

Published as a conference paper at ICLR 2025
Object Occurrences
Subject-Objece Co-occurrence
Random Baseline
0.7
Mean Baseline
0.7
Accuracy
0.6
Accuracy
0.5
0.5
0.4
0.3
0.3
0.2
0.2
0.1
LRES+LM
only
LRES+LM
LM only
Figure 3: Within-Magnitude accuracy (aka the proportion of predictions within order of mag- nitude of ground truth) for models predicting object and subject-object co-occurrences in heldout relations. Using LRE features outperforms LM only features by about 30%. We find that it is much easier to predict object frequencies; the subj-obj. prediction models with LRE features only marginally outperform baseline performance.
We use Faithfulness and Causality as defined in HnHranndeez et al. (002)) well as two other metries: Faith Prob., which the log probability of the correct answer as produced by an LRE, and Hard Causality, which is the same the "soft" variant, but only counts the proportion of times the causality edit produces the target answer as the number one prediction. We use every example from the relations for which there more than object occurrence subject-object co-occurrence. We do not provide an explicit signal for which relation an example comes from, but due to the bias of subjects/objects having similar frequencies within relation, train multiple models and evaluate on held out relations and average performance. all settings, the held out set objects and relations guaranteed not have been in the training set.
5.2 LRE METRICS ENCODE FINE-GRAINED FREQUENCY INFORMATION
Because the difficulty predicting the exact number occurrences, we report accuracy within one order of magnitude of the ground truth. This measures whether the predicted value is within reasonable range of the actual value. Results are shown in Figure We find that language modeling features do not provide any meaningful signal towards predicting object subject-object frequen- cies, and are only marginally above the baseline predicting the average or random frequencies from the training data. On object frequency predictions, find that LRE features encode strong signal allowing for accurate predictions about 70% the time. Mean absolute error of the pre- dictions (in natural log space) for LRE features (LM-only features) are 2.1, (4.2) and 1.9, (2.3) on object prediction and subject-object prediction tasks, respectively. We find that subject-object co- occurrence frequency likely too difficult to predict given the signals that we have here, our predictions are higher than, but within one standard deviation of the mean baseline.
Feature Importance: How important are LRE features for predicting the frequency of an item? We perform feature permutation tests contributes to the final answer. First, we check to see which features used to fit the regression are correlated, as if they are, then perturbing will leave the signal present in another. In Appendix we show that only faithfulness and faith probability are strongly correlated, so for this test only, we train models with single PCA component representing 89% of the variance of those two features. We find that hard causality is by far the most important feature for generalization performance, causing a difference of about 15% accuracy, followed by faithfulness measures with 5% accuracy, providing evidence that the LRE features are encoding an important signal.
how much each feature (LRE features and log probs)
5.3 GENERALIZATION TO NEW LM
Next, test the ability to generalize the regression fit of one LM to another, without requiring further supervision. If such a model could generalize, we can predict term counts to models for which do not have access to their pretraining data. We keep the objective the same and apply
8

---

Published as a conference paper at ICLR 2025
Table 1: Within-Magnitude accuracy for different settings of train and test models. Overall, we find that fitting a regression on one model's LREs and evaluating on the other provides a meaningful signal compared fitting using only log probability and task performance, or predicting the average training data frequency. The metric here is proportion of predictions within one order of 10x the ground truth. Here, Eval. on GPT-J means the regression fit on OLMo and evaluated on GPT-J.
Model
Predicting Object Oces.
Predicting Subject-Object Co-Oces.
Eval. on GPT-J
Eval. on OLMo
Eval. GPT-J
Eval. on OLMo 0.68+0.08 0.6040.07
LRE Features
0.65+0.12 0.42+0.10 0.31+0.15
0.49+0.12 0.41#0.09 0.41=0.17
0.76+0.12 0.66#0.09 0.57=0.15
LogProb Features
Mcan Freq. Baseline
0.67=0.16
the regression model, fit for example on OLMo ("Train OLMo" setting), to features extracted from GPT-J, using ground truth counts from The Pile (and vice versa, i... the "Train GPT-J" setting).
We again train a random forest regression model to predict the frequency of terms (either the subject- object frequency, or the object frequency alone; e.g., predicting "John Lemnon" and "The Beatles" or just "The Beatles") on features from one of two models: either OLMo-7B (final checkpoint) or GPT- J, treating the other as the 'closed' model. We test the hypothesis that LRE features (faithfulness, causality) are useful in predicting term frequencies across different models, with the hope that this could be applied to dataset inference methods in the future, where access to the ground truth pretraining data counts is limited or unavailable.
Results Our results are presented in Table First, we find that there a signal in the LRE features that does not exist in the log probability features: We are able to fit much better generalizable model when using LRE features as opposed to the LM probabilities alone. Second, evaluating on the LRE features of heldout model (scaled by the ratio of total tokens trained between the two models) maintains around the same accuracy when fit on exact counts from OLMo, allowing us to predict occurrences without access the GPT-J pretraining data. We find that predicting either the subject- object co-occurrences or object frequencies using LREs alone is barely better than the baseline. This task is much more difficult than predicting the frequency of the object alone, but our model may just also be unable to account for outliers in the data, which is tightly clustered around the mean (thus giving the high mean baseline performance between approx. 60-70%). Nevertheless, we show that linear structure for relations within LM representations encode rich signal representing dataset frequency.
5.4 ERROR ANALYSIS
In Tableeae we show example predictions from our regression model that we on OLMo and evaluate on heldout relations with LREs measured on GPT-J. We find that some relations transfer more easily than others, with the star constellation name transferring especially poorly. In general, the regression transfers well, without performance deteriorating much (about 5% accuracy: Figure Figure compared comparee to the evaluation of GPT-J in Table [T. suggesting LREs encode information in a consistent way across models. We also find that the regression makes use of the full prediction range, producing values in the millions (see Table ( as well as in the tens; The same regression shown in the table also predicts 59 occurrences for "Caroline Bright" (Will Smith's 's mother) where the ground truth is 48.
DISCUSSION
Connection to Factual Recall Work interpretability has focused largely around linear repre- sentations in recent years, and our work aims to address the open question of the conditions in which they form. We find that coherent linear representations form when the relevant terms (in this case subject-object co-occurrences) appear in pretraining at consistent enough rate. Analogously, Chang et al. (2024) show that repeated exposure encourages higher retention of facts. Future work could investigate the connection between factual recall accuracy and linear representations.
9

---

Published as a conference paper at ICLR 2025
Table 2: Examples of regression fit on OLMo LRE metrics and evaluated GPT-J on heldout relations, demonstrating common еттог patterns: 1. Predictions are better for relations that are closer those found in fitting the relation (country related relations), 2. Some relations, like star- constellation perform very poorly, possibly due to low frequency, 3. the regression model can be sensitive to the choice of subject (e.g., William vs. Harry), telling us the choice of data measure LREs for is important for predictions.
Predicting Object Frequency GPT-J, Regression fit on OLMo
Relation
Subject
Object
Prediction
Ground Truth
Error
landmark- country
Menangle Park
Australia
2,986,989
3,582,602
1.2x
country language
Brazil
Portuguese
845,406
561,005
1x
star-constellation name
Arcturus
Boötes
974,550
2,817
346x
person-mother person-mother
Prince William
Princess Diana
5,826
27,094
4.6x
Prince Harry
Princess Diana
131
27,094
207x
Linear Representations in LMs The difficulty of disentangling the formation of linear represen- tations from increases in relation accuracy, especially in the few-shot case, is interesting. Across 24 relations, only the cottteconttlllatinnn naaiiioonnnmm" and accuracies that far exceed their causality scores (and both are low frequency). Thus, it is still an open question how LMs are able to recall these tasks. The fact that few-shot accuracy and causality seem so closely linked is consistent with findings that ICL. involves locating the right task (Min et al., 2022) and applying a 'function' to map input examples to outputs (Hendel et al., [2023 Todd et al., 2024.. The finding that frequency controls this ability is perhaps unsurprising, as frequency also controls this linear structure emerging in static embeddings (Ethayarajh et al., 2019). Jiang et al. (2024) prove strong frequency- -based condition (based on matched log-odds between subjects and objects) and an implicit bias of gradient descent (when the frequency condition is not met) encour- age linearity LLMs; our work empirically shows conditions where linear representations tend to form in more realistic settings. If LMs are 'only" solving factual recall or performing ICL through linear structures, it is surprising how well this works at scale, but the simplicity also provides a promising way to understand LMs and ICL in general. An interesting avenue for future work would be to understand if and when LMs use a method that is not well approximated linearly to solve these types of tasks, recent work has shown non-linearity can be preferred for some tasks in recurrent networks (Csordás et al. 2024).
relations have few-shot
Future Work in Predicting Dataset Frequency The ability to predict the contents of pretraining data is an important area for investigating memorization, contamination, and privacy of information used to train models. In our approach, we show it is possible to extract pretraining data signal without direct supervision. Without interpretability work on the nature of representations in LMs, would not know of this implicit dataset signal, and we argue that interpretability can generate useful insights more broadly as well. Extensions on this work could include more information to tighten the prediction bounds on frequency, such as extracting additional features from the tokenizer (Hayase) al. 2024). We hope this work encourages future research in other ways properties of pretraining data affect LM representations for both improving and better understanding these models.
7 CONCLUSION
We find a connection between linear representations of subject-relation factual triplets in LMs and the pretraining frequencies of the subjects and objects in those relations. This finding can guide future interpretability work in deciphering whether linear representation for given concept will exist in a model, since we observe that frequencies below a certain threshold for a given model will not yield LREs (a particular class of linear representation). From there we show that we can use the presence of linear representations to predict with some accuracy the frequency of terms in the pretraining corpus of an open- weights, closed-data model without supervision. Future work could aim to improve on our bounds of predicted frequencies. Overall, our work presents a meaningful step towards understanding the interactions between pretraining data and internal LM representations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was performed while JM was intern Ai2. We thank the anonymous reviewers and members the Aristo and AllenNLP teams at Ai2 for valuable feedback.
REFERENCES
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadace, Ahmet Üstün, and Sara Hooker. To code or not to code? exploring impact of code in pre- training. The Thirteenth International Conference on Learning Representations, 2025. URL
Mttps://openreviit net. nett/forummi0==/ffeNlla.
Giuseppe Ateniese, Luigi V Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International Journal of Security and Networks, 10(3):137- 150, 2015. URL https:://///m... //cp:p:://d... acm. wwtp//i/g/...oo//o 1504/ IJSN TTt.://NNN........11
Sid Black, Lee Sharkey, Leo Grinsztajn. Eric Winsor. Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy, Interpreting neural networks through the polytope lens, 2022. URL
12312
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Mem- bership inference attacks from first principles. In 2022 IEEE Symposimm on Security and Pri- vacy (SP), pp. 1897-1914, 2022. URL httpss://eeexplore ieee. org/ document/ 9833649/
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. The Eleventh International Conference on Learning Representations, 2023. URLhttps: httptttpsss//ppeeeeeeeee. wpppreeeeeeiieeeeeeeeeee net forum?
Id=TatRHT 1ck]
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Scong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
net pest/forum?iddTXVe j1EBPP
David Chanin, Anthony Hunter, and Oana-Maris Camburu. Identifying Linear Relational Concepts in Large Language Models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 1524-1535. Association for Computational Linguistics, 2024. doi: 10. 1225:///112222mmaallllooooo
URL
wwt.oo////2.......a...oo naacl -ong. 85
Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Yonatan Be- linkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 248 262, Miami, Florida, US, November 2024. Association for Computational Lin- guistics. doi: 10. 00.18853/v1/2024 2024. bl a www.bckbooxrto.. www.bbllccbxxmll.....
URL https: patee//acllttttt loyy. wow.ooyoooog.o. orgg
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 03 2021. 1ttp:ppppp/oo..ooo.0....1222///0
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mos- bach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. Measuring causal effects of data statistics on language model's 'factual' predictions. arXiv preprint 1xXiv:2207.14251, 2022. URL
https://arxiis www.vvorgggaaa aww. 14251

---

Published as a conference paper at ICLR 2025
Yanai Elazar, Akshita Bhagia, lan Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannanch Ha- jishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Conference on Learning Representations, 2024, URLhttps: httpss///ppeeeieeeeeee net forum?
id=RvfPnOkPV4]
Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical frame- work for transformer circuits. Transformer Circuits Thread, 2021. URI. https:// transformer-c rcuits
Hady Elsahar. Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Fred- erique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry De- clerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceed- ings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL org/ 1544414444
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards Understanding Linear Word Analogies. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3253-3262. As- sociation for Computational Linguistics, 2019. doi: 10.18653/11/11--1355 URL https:
Taclaathhloogycrrg/P1....
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of di- verse text for language modeling. arXiv preprint 1rXiv:2101.00027, 2020. URL https:
7/arxiv. orq. http://aaa//1//1....00.. 00027
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Forum?id=tcss 9ZNKD.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps:://ppereeeeeese
n / forum? = fNZ
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based detection of morpholog- ical and semantic relations with word embeddings: what works and what doesn't. In Jacob An- dreas, Eunsol Choi, and Angeliki Lazaridou (eds.), Proceedings of the NAACL Student Research Workshop, pp. 8-15, San Diego, California, June 2016. Association for Computational Linguis- tics. doi: 18653/11/N16-- 2002. URL htt.:ttss:022 http://cclaattoolyo....oooo
20022
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Pa- pers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Lin- URL htwp://aatttoooloooo///2//444
guistics. doi: 10.18653/w1/2022 1/20244////0222aaalll ael-ooog.ol--o
acl- long. 841/
Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A. Smith. Data mixture inference: What do BPE tokenizers reveal about their training data? In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. URL httpttt//ppeeeeiiiiii nnt/
Eorum?id 2dd- Brrrumiid--EXyymmmaaaa EHXyeImuxx
Roee Hendel, Mor Geva and Amir Globerson. In-Context Learning Creates Task Vectors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational
12

---

Published as a conference paper at ICLR 2025
Linguistics: EMNLP 2023. pp- 9318 93333. Association for Computational Linguistics. 2023.
doi: 10. 10.18663//1/2023..1 1106:5/w/2022..mmmmgmmmmmmmmmmm findings-emnlp.624. URL
findings-emnlp... 624
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:
7/ //openeeview..o http://www..e//oooom///d///////
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfih International Conference Learning Representations, 2024. httphtttpss///pppeeeeee
net net/forum?
[dd=76bwRSS
Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In Forty-first International Conference on Machine Learning, 2024. URL
net forum?rum? ?id=
otuTw4Mghk
Marzena Karpinska, Bofang Li, Anna Rogers, and Aleksandr Drozd. Subcharacter information in When is it worth it? In Georgiana Dinu, Miguel Ballesteros, Avirup Sil, Sam Bowman, Wael Hamza, Anders Sogaard, Tahira Naseem, and Yoav Goldberg (eds.), Proceedings of the Workshop on the Relevance of Linguistic Structure in Neural Architectures for NLP. pp. 28-37, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 18.18653/v 1t////1822900 URL https: http://aaaaaaaaoaooco BRLttttss///ccaattit
Japanese embeddings:
org/wl3 /905/
Maximilian Köper, Christian Scheible, and Sabine Schulte im Walde. Multilingual reliability and "semantic" structure of continuous word spaces. In Matthew Purver, Mehrnoosh Sadrzadeh, and Matthew Stone (eds.), Proceedings of the 11th International Conference on Computational Se- mantics, pp. 40-45, London, UK, April 2015, Association for Computational Linguistics. URL http::ss//aclaathlllgggo org/ W1 01005/
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), pp. 5245-3276 Mexico City, Mexico, June URL
2024. Association for Computational Linguistics. doi: https://acl
105533/11202..maacll logy. wow.oooo.................. 002/2024 maacl-lons 1797
Yingwei Ma. Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL TRRLittpss//openzeeee http:///pppeevvieeeeee net/ forum?
id=KIPJKST4qg
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non parameerieiic memories. In Anna Rogers, Jordan
and Nacaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. (0.18653/11/2222.aaao
URL. https: / //acl h/aa/aalltthhhtcc antho 1 ogy. org/2023.
ael acl-long. 546
Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, (2144))e232222012,, 2024. URI.
https: httpp:://www.... wwws. org/ dci/ abs 10. 1073 w073/ppas..... 232222012222
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple Word2Vec- style vector arithmetic. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational
13

---

Published as a conference paper at ICLR 2025
Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5030-5047 Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.28453/1/2024.
naacl-long.288 281. URU/httss://cclatttolooo -tt.:/y....oooooaaaaaaaaaaaaaaa- naacl-long..222
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 2013a. URL /RRpss//:///a org/
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Dis- tributed representations of words and phrases and their compositionality. In C.J. Burges, L. Boutou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Ad- vances in Neural Information
P Processing Systems, volume 26. Curran Associates, Inc.,
2013b. URL
seceer_flesspppeee welllesppppee wttrr_flll//papr/211.. paf
File/ /tta://2222112200003363332222999
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (cds.), Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Processing, pp. (1048-11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022.
emnlp-main.7593 759. URL htt.ts////alaaattllllooo//////// org www.//2....2ooe.ooooo emn http://ooooo..rrro/2022eeeeeaeaa in. 7597
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024-001, 2020. URL https://
distill. pubt////pubb//00
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Prov- ing test set contamination in black-box language models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:://pperreview. net. forummii KS8mIvetg2
Alberto Paccanaro and Geoffrey E Hinton. Learning Hierarchical Structures with Linear Rela- tional Embedding. In Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001. URL /tt.://////.//cfcccccccccccccc/ 81 3
nips. cc/pape: /tlles//apeer220//hssht/
Abstract html
Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geom- etry of Large Language Models. In Forty-First International Conference on Machine Learning, 2024.
net/ httt:/ooruunn---GGCC......
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daclemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10. 3115/v1/D14-110
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by iterative nullspace projection. In Proceedings the the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 7237-7256, Online, July 2020. As- sociation for Computational Linguistics. URL wttph:tts/////t //wp::://wwww... aclweb, org/anthologyy macl-mainn... 647.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Yoay Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840 854, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Lin- guistics. doi: 10.18653/1/2022..imm www.iiingggeennl......... URL https:/ httpss///caaaahooog....o org/
2022. findi www.gmmmmmmm 597
Yasaman Razeghi, Hamish Ivison, Sameer Singh, and Yanai Elazar. Backtracking mathematical reasoning of language models to the pretraining data. In NeurlPS Workshop on Attributing Model Behavior Scale, 2023. URL httts::///ppnrreeee
net: wttpummmmmmdkkkx//kkkkk.
14

---

Published as a conference paper at ICLR 2025
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steer- ing llama via contrastive activation addition. pp. 15504-15522 August 2024, doi: 10. 10.18653/ v1/2024.acl-long.888 828. BtLhhtts:: httpa//////aaaaaoooooooooooooe
acl-aacll11 -ong--l-on-... 828/
G. Salton, A. Wong. and C. S. Yang. A vector space model for automatic indexnng. Commun.
ACM, (8(1)))613662) November 1975. ISSN 0001-0782. doi: 10.1145/362219.361220 URL
https://doi http://di..rgg./// .ogg/0. 111.1...111111111 1111444119.36112222221
Naomi Saphra and Sarah Wiegreffe. Mechanistic? In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NL.,, pp. 480-498, Miami, Florida, US, November 2024, Association for Computational Linguistics. doi: 10. 10.18653/11/2024 blackboxnlp- 1.30.
303
Preethi Seshadri, Sameer Singh, and Yanai Elazar. The bias amplification paradox in text-to-image generation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6367 6384, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.
URL https: /acl/acccaac//ccc aalllnnnhhloggo 200. naac loog..3.033 353/
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview. net wtttt/forumiiii (irrrmmii=z72qr3333000 3MQuNs
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18, 2017. doi: 10.1 10.1109/5P.217..11 URL bttps://Leexxoooee.eee... stamp/ stamp/stamg stp jsp?tp šarnumber 7958568
Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of ballucinatory (un)answerabilit:: Finding truths in the hidden states of over-confident large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3607-3622 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v I /tt02222222.mpppp. 220. URL /ttp:pps///ecc.aaaoooooyyyyoyye org/2233 22033
emn 1m ma mamin.. 220/
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15725-15788, Bangkok. Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18665////222.a/llooooggg 2ttp:///aaaaaaaooooooooo///2//.. www-0/22...llooonn.. 840/
URL
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting Latent Steering Vectors from Pretrained Language Models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 566-581. As- sociation for Computational Linguistics, 2022. doi: 10.1863///2022..mmnnn--aaaa www.yyy.rg//222.
URL
mondings-acl....
Anshuman Suri and David Evans. Formalizing and estimating distribution inference risks. Proceed- ings on Privacy Enhancing Technologies, 2022. URL
06024
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. 2024. URL /t/erraannformereriiiiitttteeeee cexxaatta.pao///ee//aaaeeeeeeeen index. html
15

---

Published as a conference paper at ICLR 2025
Eric Todd, Millicent Li, Arnb Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Tweifth International Conference on Learning Representations, 2024. URL
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:// com hommmiinggee http:/iinnoooollo z www.ssstttassooooooooo -jax May 2021.
Xinyi Wang, Alfonso Amayuelas, Kexun Zhang. Liangming Pan, Wenhu Chen, and William Yang Wang. Understanding reasoning ability of language models from the perspective of reasoning paths aggregation. In Forty-first International Conference on Machine Learning, 2024. URL https://openrevier net dostrfo
Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization v.s. memorization: Tracing language models' capabili- ties back to pretraining data. In The Thirteenth International Conference on Learning Represen- tations, 2025, URL http::tt////t /ttt:::://ppeerevieww net/ fooo//idd
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Thirth-seventh Conference Neural Information Processing Systems, 2023. URL Bttlhttts///ppeeeiiiie net/ http:arwweeeeeeeeeeooorroooooooo Seeea/oorummid=XuuyByauu
A LIMITATIONS
While our approach thoroughly tracks exposure to individual terms and formation of LRE features across pretraining, we can not draw causa o claims about how exposure affects individual represen- tations, due to the cost of counterfactual pretraining. We try to address this by showing the frequency of individual terms can be predicted with some accuracy from measurements of LRE presence. We motivate this approach we are not able to make any guarantees on its efficacy in settings not shown here, and would caution drawing strong conclusions without additional information. Furthermore, we find that our method is relatively worse at predicting subject-object co-occurrences than object occurrences, and our method fails to account for the harder task. Future work could expand on this tool by incorporating it with other data inference methods for greater confidence. We also do not discuss the role of the presen- tation of facts on the formation of LRE features, but following Elsahar et al. (2018) and the strength of the relationship we find, we speculate this has minimal impact. Note that the BatchSearch tool we release tracks the exact position index of the searched terms, thus facilitating future work on questions about templates and presentation of information.
possible way to detect the training data of closed-data LMs; however,
B EFFECT OF TRAINING ON INCORRECT EXAMPLES
In Hernandez et al. (024). examples are filtered to ones in which the LM gets correct, assuming that LRE will only exist once a model has attained the knowledge to answer the relation accuracy (e.g., knowing many country capitals). We find that the choice of examples for fitting LREs is not entirely dependent on the model "knowing" that relation perfectly (i.e., attains high accuracy). This is convenient for our study, where we test early checkpoint models, that do not necessarily have all of the information that they will have seen later in training. In Figure 5] we show faithfulness on relations where the LRE was fit with all, half, or zero correct examples. We omit data for which the model did not get enough incorrect examples. Averages across relations for which we have enough data shown in Figure 4 which shows that there is not considerable difference in the choice of LRE samples to train with.
C LRE HYPERPARAMETER TUNING
There are three hyperparameters for fitting LREs: layer at which to edit the subject, the beta term used to scale the LRE weight matrix, and the rank of the pseudoinverse matrix used make edits for
And thus mechanistic, in the narrow technical sense of the term Saahra & Wiegreffe 2024
16

---

Published as a conference paper at ICLR 2025
Average Falthfuinees for Settings Training Examples
Average Causality Different Settings of Training Examples
0.8
0.7
0.6
0.6
Felthfiness
0.5
0.5
0.4
0.3
0.3
0.2
0.2
0.1
Correct
Hair Correct
None Correct
Correct
Half Cornect
None Correct
Figure 4: Average Causality and Faithfulness results across relations depending on if the LRE was fit with correct incorrect samples. We find no notable difference in the choice examples.
Causality for all Relations for Different Settings of Training Examples
Correct
0.8
Half Correct
None Correct
Causality
0.6
Majority Label Baseline
Majority Frequency Baseline
0.4
0.2
0
city
tity
iacgge
Faithfulness for all Relations for Different Settings of Training Examples
All Correct
0.8
Half Correct
0.7
None Correct
Faithfulness
Majority Label Baseline
0.5
0.4
Majority Frequency Baseline
0.3
0.2
0
secrica tern
couttry
sarpee moics
Figure 5: Causality and Faithfulness results for each relation depending on if the LRE was fit with correct or incorrect samples. Note that relations with only one bar do not have zeros the other categories. It means that there not enough data that the model (OLMo-7B) got wrong to have enough examples to fit.
17

---

Published as a conference paper at ICLR 2025
Best Layer Beta vs. Falthfuiness
pokeman evolution
country
country
landmark
06
428
D-5
10ST
superhero person
person sport position
person mother
person university
plays sport
0.4
0.5
0.4
0.2
0.4
06
company ha
person lead singer of band
president birth year
person fother
person plays instrument
0.5
0.4
0.2
company CEO
landmark on continent
president election year
Food from country
product by company
0.4
0.0
country largest
capital city
superhere archnemesis
constellation
02
Figure 6: OLMo 0424 7B per layer faithfulness scores as function of the choice of layer which to fit the LRE. Note we do not use these results to choose the layer for the LRE, instead preferring the results from the causality sweep.
measuring causality. Beta is exclusive measuring faithfulness and rank is exclusive causality- We test the same ranges for each as in Hernandez et al. (2024): [0. 5] beta and [0, full. rank] for causality at varying intervals. Those intervals are every 2 from [0, 100), every 5 from [100,200], every 25 from (200, 500], every 50 from [500, 1000], every 250 from [1000,
sadden_size] We
perform the hyperparameter sweeps across faithfulness and causality, but choose the layer edit based on the causality score. In cases where this is not the same layer as what faithfulness would decide, we use the layer causality chooses, as would metric. We refer the reader to Hernandez et al. (2024) for more details on the interactions between hyperparameters and the choice of layer. The results of our sweeps on OLMo-7B across layers in Figures 6 and and across beta and rank choices in Figures 8 and
make sense train one LRE for each
D BATCH SEARCH COUNTS COMPARED TO WIMBD
In Figure 10 we find that What's in My Big Data Elazar et al. 2024) matches very well to batch search co-occurrences; however, WIMBD tends to over- over-predict co-occurrences (slope less than I), due to the sequence length being shorter than many documents, as discussed in the main paper.
E FEATURE CORRELATIONS AND IMPORTANCES
Our feature importance test is shown in Figure 12 This permutation test was done on the heldout data to show which features contribute the most generalization performance. We use PCA to
18

---

Published as a conference paper at ICLR 2025
Layer vs. Cousality
country currency
country language
country
country
05
0.5
0.5
10
20
20
Superhero person
person sport position
person mother
person university
plays pro sport
0.4
but
compenyy hg
person lead singer of band
president birth year
person feither
person plays instrument
0.4
0.2
company CEO
Jandmark continent
president election
Food from country
product company
0.6
20
20
50
20
30
10
20
10
20
30
20
country largest
country capital city
superhero
person accupetion
name
V
0.3
0.5
0.5
10
20
30
20
30
10
20
10
20
30
25
10
Figure 7: OLMo 0424 7B per layer causality scores function of the choice of layer at which to fit the LRE.
19

---

Published as a conference paper at ICLR 2025
Layer vs. Faithfulnness
country currency
countryy language
country
country
0.5
0.5
0.4
0.2
02
2057
superhero person
person sport position
person mother
person university
plays pro sport
0.8
0.3
0.8
0.2
0.2
0.2
company hq
person lead singer band
president birth year
person father
person plays instrument
0.5
0.6
6.4
02
company CEO
Jandmark
president year
from country
product company
0.5
0.5
0.8
D.4
country largest
country capital
superhero archnemesis
person occupation
0 constellation name
0.8
0.6
0.4
Figure 8: OLMo 0424 7B LRE Beta hyperparameter sweep at highest performing layer.
20

---

Published as a conference paper at ICLR 2025
Best Layer Rank Causality
pokemon evolution
country currency
country language
city country
landmark in country
0.5
05
0.5
05
3.05
superhero
sport position
person mother
person university
0.6
0.4
0.5
0.7
100
202
300
200
200
300
10C
200
200
100
200
302
100
200
300
company
person singer
year
person
gerson plays instrument
0.8
aa
088
0.4
0.2
b.2
100
203
300
100
200
102
203
200
100
200
302
100
300
company CEO
landmark continent
president election year
from country
product by company
0.6
0.5
0.2
1.00
202
300
102
200
100
300
country largest city
country capital city
superhero archnemesis
person occupetion
star constellation name
Figure 9: OLMo 0424 7B LRE Rank hyperparameter sweep at highest performing layer.
21

---

Published as a conference paper at ICLR 2025
WIMBD vs Batch Cooccurrence. slope=0.94, r=0.99
Batch Cooccurrence
15
5
10
15
WIMBD Cooccurrence
Figure 10: Comparison between WIMBD and Batch Search subject-object co-occurrences
Feature Correlations OLMo-7B
Feature Correlations GPT-)
2:3
E:E
3/2
8:2
312
hard causality
3-14
hand cRumai
Recuracy
Ir. brgloggano
bith prs-
2/
hittummb prob-
8.20
coft cit_owsolity
2.3
est pet_.oolityy
313
left couculiy
lr, peb
bccursce
left, caudulity
In Freb
ercorecy
hard cautallyy
penb
paussity
prob
Figure 11: Correlations between each feature in our regression analysis. Because of the high cor- relation between faithfulness metrics, we use single dimensional PCA to attain one feature that captures 89% of the variance of both for the purposes of doing feature importance tests. Note that we zero out the diagonal (which has values of 1) for readability.
reduce the faithfulness features to one feature for the purposes of this test. Correlations are shown in Figure 11
22

---

Published as a conference paper at ICLR 2025
Permutation Importances
Change in Accuracy
0.15
0.1
0.05
0
Soft Causality
Accuracy
Lm Log Prob
Faith Pca
Hard Causality
Figure 12: Hard causality is by far the most important feature for generalizing relations when predicting Object frequencies, causing change in about 15% accuracy.
F RELATIONSHIP BETWEEN CAUSALITY AND ACCURACY
In this section, we provide more detail on the relationship between the formation of linear represen- tations and accuracy on in-context learning tasks. Although the two very highly correlated, argue that accuracy and LRE formation are somewhat independent.
We show this relationship across training for OLMo-1B 1B in Figure and 7B in Figure 14
G EXTENDING TO COMMONSENSE RELATIONS
Following Elsahar et al. 2018). focus factual relations because subject-object co-occurrences are shown to be good proxy for mentions of the fact. For completeness, we consider 8 additional commonsense relations here. Results for OLMo-7B are shown Figure [5] We show that fre- quency is correlated with causality score (.42) in these cases well, but it is possible subject-object frequencies do not accurately track occurrences of the relation being mentioned. For example, in the "task person type" relation, the co-occurrence count of the subject "researching history" and the object "historian" does not convincingly describe all instances where the historian concept is defined during pretraining. Co-occurrences learns that the outside of a coconut is brown, however (the fruit outside color relation). Therefore, caution treating these under the same lens the factual relations. Nevertheless, we believe these results are an interesting perspective on how a different relation family compares to factual relations.
perhaps more convincingly related to how model
23

---

Published as a conference paper at ICLR 2025
Zero Shot, Shot, Causality: OLMo 18
Figure 13: Zero shot, 5-hhot accuracies against causality for each relation across training time in OLMo-1B
24

---

Published as a conference paper at ICLR 2025
Zero Shot, Shot, Causality: OLMo 7B
Figure 14: Zero shot, 5-shot accuracies against causality for each relation across training time in OLMo-7B
OLMo-7B 0424 Development of Commonsense LREs over Training Time
Causality
0.8
- autssten pi mader
0.7
- ord Sa
0.5
12
Log Subj-Obj Cooccurrence
Figure 15: Commonsense relations compared to pretraining time in OLMo-7B.
25


Reviews:
[{"review_id": "v0gJ7xq2T7", "reviewer": "Reviewer_eSj4", "summary": "The authors investigate the correlation between linear representations and pre-training data frequency in language models. The work is conducted on recent findings that the linearity of different types of relations varies significantly depending on the specific relationship. Existing work does show that language model exhibit such linear structures, but do not reveal the underlying reason why some relations exhibit such structure while other do not. The main contribution of this work is to empirically draw the correlation between such linear structure and data frequency. It shows that that linear representations for factual recall relations are related to mention frequency and the model size. In addition, more detailed results show that linear representations form at predictable frequency thresholds during training, which allows the prediction of term frequencies in the training data. Finally, the authors release a tool for searching through tokenized text for understanding training data characteristics.\n\nOverall, the findings are insightful for understanding linear representation structures in language models. This empirical study complements existing theoretical evidence on the same subject. It provides a perspective to the problem, which can be among many other factors in driving the formation of linear structures. On the utility side, the findings can be used for understanding training data, which are typically not published for current LLMs.", "questions": "Could there be some theoretical discussion on the training dynamics and the frequency thresholds?\n\nOne related work is \n\nGuangsheng Bao, Zhiyang Teng, and Yue Zhang. 2023. Token-Level Fitting Issues of Seq2seq Models. In Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP) at ACL 2023. Toronto, Canada from July 9th to July 14th, 2023.\n\nwhich also discusses the correlation between term frequencies and model accuracy.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "A perspective for understanding the reason that some features from LLMs demonstrate linear structures while others do not.\n\nA tool for search through tokenized text to support the understanding of training data.", "weaknesses": "It provides one perspective to the problem empirically, with a specific set of metrics. While giving useful information, the depth of understanding and the utility domain is constrained mostly to the correlation between term frequency, the model size and the linear structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBXOCMWrsy", "reviewer": "Reviewer_J8rP", "summary": "This paper explores the question of why linear structures form in LLMs by investigating the connection between training data frequency and the formation of linear representation, focusing specifically on factual recall relations. The study reveals that (1) the formation of linear representations is strongly correlated with subject-object co-occurrence frequency, and (2) the presence of linear representations can help predict relation frequency. Experiments are conducted using OLMo-1B, oLMo-7B, and GPT-J to validate these findings.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Exploring the origin of linear representation is an important question in LM interpretability. This work identifies a correlation between linear representations of factual recall relations and the subj-obj co-occurrence frequency in pretraining.\n- This paper investigates the relationship between few-shot accuracy and the existence of a linear representation.\n- Using the existence of linear representations to predict the frequency of terms in the pretraining corpus is interesting.", "weaknesses": "- The scope of the work is somewhat limited, as only 25 factual relations are investigated. It is unclear whether the identified correlation is also valid for other relation types. Expanding the analysis to include more factual relations and other types of relations could further enhance the robustness of the findings and potentially offer additional insights.\n- The linear representation seems to be affected by the context in LREs (e.g., four \"X plays the Y\" examples before the fifth one. Are the findings universally applicable to LLM generation without involving ICL formats?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KQk1j3RbNh", "reviewer": "Reviewer_zXUa", "summary": "This research paper explores how the frequency of certain words appearing together in the data used to train a language model (LM) affects the LM's ability to learn simple, linear rules for representing facts. The authors found that the more often two words related to a fact appear together in the training data, the more likely the LM is to learn a simple rule to represent that fact. This discovery helps to understand how LMs learn factual information and could be used to figure out what kind of data was used to train secret LMs. The authors also created a tool to help others count how often words appear together in large datasets", "questions": "Refer the previous section.\n\nIt would be good if authors can dedicate a section to discuss the potential impact of confounding factors, such as context, syntax, and semantics. Explain why controlling for these factors is challenging in the current study but emphasize the importance of future work to disentangle their effects from the influence of frequency.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The study finds a strong correlation  between the average co-occurrence frequency of subjects and objects within a relation and the quality of linear representations (LREs) formed for that relation. This correlation surpasses the individual correlations with subject frequencies or object frequencies, highlighting the significance of subject-object co-occurrence.\n\nThe study focuses uses Linear Relational Embeddings (LREs), which effectively approximate the computations performed by an LLM to predict objects in factual subject-relation-object triplets. This paper builds upon this research by examining how the frequency of subject-object co-occurrences in pretraining data directly impacts the emergence and quality of these LREs\n\nThe paper introduces a promising technique for analyzing the pretraining data of closed-source models by leveraging the connection between linearity and frequency.", "weaknesses": "The paper presents valuable findings, however they should provide some discussion along the following directions:\n\n(a) The paper primarily focuses on Linear Relational Embeddings (LREs) as a representative class of linear representations in LLMs. However, LLMs might employ various other forms of linear or non-linear structures to encode information. This focus on LREs could limit the generalizability of the findings to other types of representations. Is there any strong hypothesis to strict to LREs?\n\n(b) While the study demonstrates that LRE features can be used to predict the frequencies of individual terms with reasonable accuracy, predicting the frequency of subject-object co-occurrences is challenging. The regression models achieve only marginal improvements over baseline performance in this task.  Integrating additional features might be helpful here.\n\n(c) The study analyzes a set of 25 factual relations from the Relations dataset. However, LLMs are trained on vast and diverse data, encompassing a much wider range of relations and concepts. Expanding the scope of analysis to encompass a broader range of relations would provide a more comprehensive understanding of the role of frequency in shaping LLM representations.\n\n(d)  The paper focuses primarily on the frequency of terms in the pretraining data. However, other factors, such as the context in which terms appear, the syntactic structure of sentences, or the semantic relationships between words, could also influence the formation of linear representations. For example, LLMs are proven to not do well is facts  are stored in templates , as it tend to remember the template and not the facts. The proposed approach may not be applicable in those scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ifNDPVkPlg", "reviewer": "Reviewer_juu2", "summary": "This paper finds that the formation of linear representations for factual recall relations in LMs is highly correlated with the frequency of subject-object cooccurrence in the pretraining data. The formation of linear representation can happen at any stage of pretraining as long as the subj-obj cooccurrence exceeds some threshold, i.e., a linear representation can form consistently when the subjects and objects co-occur at least 1-2k times even at early stages of pretraining. The results also indicate that the frequency threshold is related to the model size, and larger models tend to require smaller thresholds. Using the metrics that evaluate the quality of linear representations, the authors can predict the approximate frequencies of individual terms as well as the co-occurrence of terms in the pretraining corpus better than using the LMs' uncertainty alone.", "questions": "1. The experiments follow previous work and only analyze 25 relations. What are the reasons for not including other relations?\n2. Section 4.3 is interesting to some degree but I am not sure about the implication of the results. Looks like it is just a description of what is observed. What is the research question you want to answer here?\n3. Typos:\n   1. Line 455 the regression model can \"be\" sensitive to ...\n   2. Line 416: the paragraph does not end properly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper draws an interesting connection between pretraining term frequency to the formation of linear representation of factual recall relations. The fact that the formation of linear representations could happen at any pretraining stage is particularly intriguing. The experiments and results are easy to understand and the discussion of related work is comprehensive.", "weaknesses": "- Being able to predict the frequencies of individual terms as well as the co-occurrence seems to be a direct implication of high correlation and therefore does not sound like a major standalone contribution. Also, \"Importantly, this regression model generalizes beyond the specific LM it was trained on without additional supervision.\": the prediction results seem to be very noisy and not much better than the mean frequency baseline.\n- Some claims are not properly justified:\n  - Line 75: \"This frequency threshold decreases with model size\": This is only tested two model sizes (OLMo-7B and OLMo-7B), and the fact that GPT-J (6B) has as smaller threshold than OLMo-7B is an counterexample for this. Would be good just to be consistent with the rest of discussion to claim there is a connection to scale.\n  - Line 93-94: \"Linear representations form at predictable frequency thresholds during training, regardless of when this frequency threshold is met for the nouns in the relation.\" The term \"predictable\" can be understood as there is a strong correlation between the linear representation quality and the co-occurrence frequency. However, the fact that this threshold is predictable regardless of when this frequency threshold is met is not well supported by results. It is necessary to show that the threshold (mean causality >.9) is consistent across different checkpoints.\n    - Line 319-320: \"Regardless of pretraining step, models that surpass this threshold have very high causality scores.\" It would be nice if you could arrange results into different scatter plots for each pretraining stage and compare them, say by fitting a linear model and comparing their slopes and biases.\n  - Line 100: The efficiency of the proposed searching tool is not well discussed.\n  - Line 455: \"Some relations, like star-constellation perform very poorly, possibly due to low frequency\" Why low frequency is the cause?\n  - Line 471: \"Second, evaluating on the LRE features of a heldout model (scaled by the ratio of total tokens trained between the two models) maintains around the same accuracy,\" How do the results support \"around the same accuracy\"? If it is comparing Train OLMo and Train GPT-J in Table 1, the drop of accuracy is larger than the performance gap between LRE features and mean baseline. I am not sure if this entails \"around the same accuracy\".\n  - Line 483: \"In general, the regression transfers well, without performance deteriorating much (about 5%), suggesting LREs are encoding information in a consistent way across models.\" What results support this? Table 2 only shows a few examples which is insufficient for supporting the claim. Are there aggregated numbers of all pairs? How many of them have errors less than 5%?\n- Some important details are missing from the experiments\n  - The results in Figure 3 and Table 1 do not match: 1) why is the mean freq. baseline performance different? 2) why do LRE features (Table 1: 0.76) seem to perform better than LRE + LM (Figure 3: ~0.67) for OLMo, if Figure 3 shows the results for OLMo. The author should explain how are the numbers related to each other.\n  - Table 2: 1) \"Predictions are better for relations that are closer to those found in fitting the relation (country-related relations)\" What does closer mean here? How did you measure this? 2) \"Some relations, like star-constellation perform very poorly, possibly due to low frequency\"\n- Some figures and tables need to be more carefully explained:\n  - The two left plots in Figure 2 need more explanations or presented in a better way. Specifically, why are some points darker than the others? What do the lines (light grey and dark grey lines) mean? Also, why do all dots for GPT-J have the same shape while the dots for the 2 left plots do not?\n  - Table 1: \"Train OLMo\" and \"Train GPT-J\" are hardly self-explanatory, the authors should consider better ways to explain the settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)