Interpretability of Language Models via Task Spaces (2024)

Abstract: The usual way to interpret language models (LMs) is to test their performance on different benchmarks and subsequently infer their internal processes. In this paper, we present an alternative approach, concentrating on the quality of LM processing, with a focus on their language abilities. To this end, we construct ‘linguistic task spaces’ – representations of an LM’s language conceptualisation – that shed light on the connections LMs draw between language phenomena. Task spaces are based on the interactions of the learning signals from different linguistic phenomena, which we assess via a method we call ‘similarity probing’. To disentangle the learning signals of linguistic phenomena, we further introduce a method called ‘fine-tuning via gradient differentials’ (FTGD). We apply our methods to language models of three different scales and find that larger models generalise better to overarching general concepts for linguistic tasks, making better use of their shared structure. Further, the distributedness of linguistic processing increases with pre-training through increased parameter sharing between related linguistic tasks. The overall generalisation patterns are mostly stable throughout training and not marked by incisive stages, potentially explaining the lack of successful curriculum strategies for LMs.

heading: 1 Introduction
text: Recently, language models (LMs) have reached a level of sophistication in language production where their output is often indistinguishable from human-generated language (Liang et al., 2022). However, the complexity inherent in language production means that effective models are also inherently complex, making them challenging to interpret.
*Now affiliated with Fraunhofer IIS; corresponding author
†Shared senior authorship
…
Commonly, linguistic interpretability involves assessing an LM’s ability through simple evaluation tasks like grammatical acceptability judgments of various language constructions (e.g. Linzen et al., 2016; Marvin and Linzen, 2018). While these methods inform us about a model’s performance, they do not provide insights into the quality of the model’s solutions. This is especially the case when error analysis is not possible due to high model performance. However, it is the quality of processing that is interesting from the viewpoint of the interpretability researcher, the cognitive scientist or linguist. Here, we introduce a method to interpret the language processing of LMs holistically. We show how linguistic knowledge in LMs interconnects. We build upon the framework of Weber et al. (2021) that proposes to consider linguistic
4522
phenomena as ‘tasks’ an LM has to optimise and allows us to analyse the interactions of those tasks, similar to ideas from multi-task learning (MTL). For example, consider the following sentences:
(1) John did not see anything.
(2) If John sees anything, he will be surprised.
In both sentences, a downward-entailing environment (negation vs. conditional) allows for the negative polarity item (NPI) anything to be used. Understanding whether it is acceptable to produce an NPI in either sentence can be considered a different task. LMs can use different rules to solve these tasks: an LM might learn the co-occurrence statistics of certain trigger words (e.g. ‘not’ vs. ‘if ’) with NPIs. On the other hand, it might generalise to a more abstract linguistical conceptualisation and understand that both – negation and conditionals – create downward-entailing environments permitting NPIs. With either rule, the model resolves acceptability judgements correctly, while the quality of both solutions is decisively different. Hence, assessing the generalisation of linguistic tasks reveals how LMs conceptualise language.
Similar to ‘task spaces’ in MTL (more details in § 2.1), we can represent an LM’s generalisation behaviour in a linguistic task space, a multidimensional space relating linguistic tasks according to their similarity. To construct linguistic task spaces, we introduce similarity probing, a method to estimate linguistic similarity. This method involves selectively fine-tuning LMs on specific linguistic tasks and assessing the impact of the finetuning on other tasks (see Figure 1), as well as the alignment of tasks in gradient space. We extricate single linguistic tasks from their entanglement in natural language via a method we call fine-tuning via gradient differentials (FTGD). FTGD selectively updates a small, relevant subspace of parameters with ‘gradient differentials’.
The contributions of this paper can be summarised as follows:
1. Propose linguistic task spaces as an interpretability method for deeper model understanding and as a tool for linguistic theory testing.
2. Introduce FTGD, a technique to disentangle linguistic tasks from their language context and selectively fine-tune them in LMs.
3. Introduce similarity probing, an efficient
method for generating large linguistic task spaces.
4. Analyze the development of language conceptualisation of LMs throughout pre-training by constructing language task spaces at various stages of LM pre-training.

heading: 2 Background and related work
text: In this section, we summarise related work and additional background on task spaces in MTL (§ 2.1), the use of LMs for linguistic theorizing (§ 2.2) and fine-tuning machine learning models in lowdimensional subspaces (§ 2.3).

heading: 2.1 Task-similarity spaces in MTL
text: In MTL, the transfer between different tasks is thought to be determined by their ‘similarity’ (BenDavid and Borbely, 2008). Constructing similarity spaces and task taxonomies to determine which tasks should be trained together has been a prominent goal in the literature. One of the earliest examples of constructing task-similarity spaces can be found in Thrun and O’Sullivan (1996). More recently, Zamir et al. (2019); Standley et al. (2020) constructed task taxonomies for computer-vision tasks based on the transferability of task-specific representations. Similarly, Achille et al. (2019) create ‘task embeddings’ for visual classification tasks by comparing their task structure through Fisher Information Matrices. From a theoretical perspective, Lee et al. (2021) investigate task similarity using synthetic tasks in a controlled setting, finding that their similarity measure can predict learning outcomes.

heading: 2.2 Linguistic spaces
text: Their ability to consistently construct acceptable language made LMs interesting, explicit linguistic theories (Baroni, 2022). However, similar to humans (Watson, 1913; Titchener, 1912; Nisbett and Wilson, 1977), LMs cannot introspectively report their internal processes. Consequently, there has been growing interest in developing methods to gain theoretical insights by analysing internal processes of LMs in what has been described as ‘synthetic linguistics’ (Chowdhury and Zamparelli, 2019). A collection of interpretability work assumes implicit linguistic similarity spaces in LMs revealed in their generalisation behaviour: Weber et al. (2021) demonstrate how language models generalize across linguistically related construc-
tions, suggesting an implicit task hierarchy within broad tasks like language modeling.
Chowdhury and Zamparelli (2019) observe that pre-trained models more easily learn grammatical than ungrammatical structures, showing how LMs generalise across consistent linguistic structure. Prasad et al. (2019) and Sinclair et al. (2022) use priming experiments to determine the relationship between different linguistic tasks and recover their hierarchical organization. Pérez-Mayos et al. (2021) fine-tune LMs on various downstream tasks and evaluate the effects on their syntactic understanding. Müller-Eberstein et al. (2023) probe for linguistic subspaces in language models using information theoretic probes. To our knowledge, we conduct the first attempt to explicitly construct a linguistic similarity space. Ultimately, our linguistic spaces are akin to knowledge representations in conceptual spaces (Gardenfors, 2004, 2014), popular in the cognitive sciences.

heading: 2.3 Fine-tuning in low-dimensional subspaces
text: Recently, the idea that tasks can be fine-tuned in low-dimensional subspaces of overparameterised models has gained popularity. A range of previous work shows how tasks can be effectively trained using projections into low-dimensional subspaces (Li et al., 2018; Aghajanyan et al., 2020; Gressmann et al., 2020; Hu et al., 2022; Li et al., 2022; Zhang et al., 2023). Other research demonstrates the reducibility of tasks to small subnetworks by learning discrete maskings for task-irrelevant parameters or activations (Frankle and Carbin, 2018; Mallya et al., 2018; Sanh et al., 2020; Zhang et al., 2021; Zhao et al., 2020; Csordás et al., 2020; Guo et al., 2020; Chintam et al., 2023), without explicitly projecting representations into lower-dimensional space.

heading: 3 Methods
text: In this paper, we connect work on task spaces in MTL (Zamir et al., 2019; Standley et al., 2020; Achille et al., 2019) with the idea of linguistic similarity spaces (Weber et al., 2021) in a method we call similarity probing. Similarity probing consists of three steps: First, we evaluate our untuned model on a wide range of linguistic phenomena (i.e. a ‘linguistic task’). Then, we selectively tune a separate LM on each linguistic task. Finally, we evaluate each model again on all linguistic tasks and assess the tuning’s impact in terms of performance transfers and compare different properties of
the gradient updates (more details follow in § 3.2). Fine-tuning a linguistic phenomenon is not straightforward. For that reason, we start by introducing a modified training procedure we will call ‘finetuning via gradient differentials’ (§ 3.1).

heading: 3.1 Fine-tuning via gradient differentials (FTGD)
text: The major problem when fine-tuning an LM on a linguistic phenomenon is what we here call ‘linguistic entanglement’.
Linguistic entanglement Within language data, linguistic tasks are necessarily interwoven (Weber et al., 2021). For example, certain tasks are present in every sentence (e.g. subject verb agreement [SVA]). This presents a challenge if we want to use natural language data to selectively fine-tune a separate task A since any potential data point to train task A necessarily also contains information on SVA. The learning signals of A and of SVA are overlapping and can not be unambiguously attributed to either task.
On the other hand, natural language data is also rich in spurious correlations between different task distributions. For example, two tasks A and B might occur in similar contexts or frequently share vocabulary in their realisations. The similarity in learning signal between A and B in such cases may solely be due to these spurious correlations instead of any conceptual similarity.
Disentangling linguistic tasks Our method builds on the assumption that gradients in LM training are a linear combination of an ensemble of ‘sub-
gradients’, representing different linguistic tasks. Following this assumption, we can isolate a specific linguistic task by subtracting two gradients that only differ in that task’s specific subgradient, effectively eliminating all other subgradients. To obtain minimally different gradients, we make use of ‘minimal pair’ sentences. A minimal pair consists of two almost identical sentences, only distinguished by a minimal difference that renders one of them ungrammatical with respect to a specific linguistic task. An example of a minimal pair for NPI licensing through negation is
(3) John did not see anything.
(4) * John did see anything.
where (4) differs minimally from (3) to render it unacceptable. Using minimal pairs as training data, we proceed in the following way: At every update, we calculate the gradients g+(θ) for the grammatical examples and g−(θ) for their ungrammatical counterparts, with θ being our model parameters. We then calculate the gradient differentials g∆ = g+ − g−.
Estimating similarity in the resulting highdimensional gradient space is challenging (see e.g. Beyer et al., 1999; Zimek et al., 2012). Therefore, we additionally reduce the gradients to a small parameter subspace by dropping parameters for which the differential g∆(θ) does not differ from 0 by a margin of ϵ = 10−3:
θ0 = {θ : |g+(θ)− g−(θ)| > ϵ}
In other words, we select those parameters θ0 where the gradients for positive and negative examples are sufficiently different. We then fine-tune the model by using only g∆(θ0). With this approach, we reduce the number of trainable parameters to an average of 5% of the full parameters while maintaining 81% of the gradient mass (see Figure 2).

heading: 3.2 Similarity Probing
text: We determine the similarity between tasks A and B by examining various aspects of how LMs learn them. Following the MTL literature, we concentrate on transfer learning (Zamir et al., 2019; Standley et al., 2020) and the gradient alignment (Yu et al., 2020) between A and B.
Transfer probing We determine the transfer between A and B by fine-tuning a language model on A and measuring the performance on a benchmark test of B before and after the fine-tuning (see Figure 1). Fine-tuning on task A may have three potential influences on task B. We interpret them as follows:
1. The performance of B increases: A and B are related and have high similarity.
2. The performance of B decreases: A and B are related and have low similarity.
3. The performance of B is unchanged: A and B are unrelated.
We normalise1 all transfers to mitigate any floor and ceiling effects. Assessing similarity in this way
1For negative transfers, we normalise by the maximally possible accuracy decrease on the benchmark (i.e. the pre-fine-
has the advantage of being technically easier to apply and does not require a machine with a large memory.
Gradient Probing We can also directly relate tasks in the parameter space: we compare the overlap of different task subspaces and the alignment of gradients g∆ within those subspaces. Following Yu et al. (2020), we predict:
1. A and B have great subspace overlap and g∆
is aligned: A and B are similar and will benefit each other.
2. A and B have a great subspace overlap, but g∆
is not aligned: A and B are dissimilar and will interfere.
3. A and B have a small or no subspace overlap: A and B are unrelated and will not interact.
We determine the overlap between subspaces of tasks A and B by calculating their Jaccard-similarity Jsim
Jsim(θ A 0 , θ B 0 ) = |θA0 ∩ θB0 | |θA0 ∪ θB0 |
and use cosine similarity (CS) to measure gradient alignment between tasks. The gradient-based method enables us to get a more detailed insight into the interaction between tasks.

heading: 4 Experiments
text: In the empirical analysis of our method, we pretrain three different generative LMs up to various stages. Then, we test the FTGD on a trained-out checkpoint to ensure that it works as intended. Subsequently, we apply it to all intermediate checkpoints to interpret the change in the LMs’ language conceptualisation throughout the training process.

heading: 4.1 Training details
text: Data We pre-train our LMs on the standard split of a common English Wikipedia corpus (; Merity et al., 2017). For probing their linguistic ability, we use the BLiMP corpus (Warstadt et al., 2020). BLiMP consists of minimal pairs for 13 higherlevel linguistic phenomena in the English language, which can be subdivided into 67 distinct realisations called paradigms. Each paradigm contains 1000 individual minimal pairs, sizing the whole corpus at 67,000 data points. For our experiments,
tuning accuracy), and for positive transfers, we normalise by the maximally possible accuracy increase on the benchmark (i.e., 1 - the pre-fine-tuning accuracy).
we consider every paradigm as a separate linguistic task. We use 85% of each paradigm’s data for probe training and retain 15% for evaluation.
Models and pre-training We employ decoderbased generative transformer language models based on the fairseq library (Ott et al., 2019). We consider three models of different sizes with ∼27M, ∼70M and ∼203M trainable parameters2 respectively, with all other hyperparameters kept constant across models3. After 20 epochs of pre-training, we reach average final perplexities of 65.21, 38.32 and 27.61 on the wiki103 validation set of training across 5 runs.
FTGD We adapt the setup for fine-tuning in the probing phase to avoid potential confounds: we switch to plain stochastic gradient descent (SGD) to preempt interference of Adam’s momentum terms with the probing process. Additionally, we change the batch size to 850 to contain the entire training data for the probed paradigm to minimize the influence of individual data point variations on the learning signal. We fine-tune on the probed paradigm until the model’s performance on the validation set converges4 or we reach a maximum of 20 updates.

heading: 4.2 Results
text: We first show how FTGD compares to full-gradient fine-tuning (§ 4.2.1), continue with an analysis of the similarity probing method (§ 4.2.2) and end with the analysis of the development of similarity space throughout the LM pertaining process (§ 4.2.3).

heading: 4.2.1 FTGD
text: The desideratum of our method is that it improves a specific linguistic task in isolation. This requires effectiveness and selectivity: it improves the model’s performance on a specific linguistic task while not interfering with unrelated capacities. We assess effectiveness by comparing FTGD with regular finetuning using the full gradients g+ and observe that the difference method achieves equivalent or higher fine-tuning performance (see Figure 3a). To assess selectivity, we compare how much both fine-tuning methods interfere with the LM’s general ability to
2Hyperparameters for (LM27, LM70 and LM203): layers = (3, 6, 12), hidden- and embedding-size = (256, 512, 1024), attention-heads = (4, 8, 16), ffn size = (1024, 2048, 4096)
3Hyperparameters: batch size = 16, dropout = 0.1, learning rate = 0.0001; Optimiser: Adam (Kingma and Ba, 2015)
4The stopping criterion is defined as current performance being lower or equal to the average of the last five steps.
generate language. We measure this ability via the LM’s perplexity on the wiki103 validation set and find FTGD to be much less disruptive than full gradient fine-tuning (see Figure 3b). Both results show how FTGD is indeed effective and selective.

heading: 4.2.2 Linguistic task spaces
text: After applying our similarity probing method, we obtain linguistic task spaces containing similarity values between all possible pairings of BLiMP paradigms. The heatmap in Figure 1 visualises a transfer space for a LM203 model. From here, we can take two perspectives on the analysis of linguistic task spaces: we can use them to test linguistic hypotheses or to interpret the linguistic conceptualisation of the LM. While we will concentrate on the latter use for the remainder of the paper, we will touch upon the possibility of doing linguistic hypothesis testing in the discussion.
Comparing similarity measures As laid out in Section 3.2, we can construct task spaces based on different similarity measures (via performance transfers i.e. ‘transfer probing’ or properties of their gradients i.e. ‘gradient probing’). Additionally, we consider multiple options to construct gradient task spaces: the overlap between gradient subspaces Jsim (i.e. the degree to which tasks are learned with the same parameters), the alignment of the gradients in those subspaces (using cosine similarity [CS]) or the gradient alignment weighted by the degree of subspace overlap (Jsim × CS). In the upper part of Table 1, we compare how predictable the resulting gradient task spaces (GTS) are for the respective TTS. Subspace overlap alone (GTSJsim) yields comparatively low correlations, as sharing parameters is necessary but not sufficient for transfer to occur. Accordingly, we find the similarity of gradients within the overlapping subspaces (GTSCS) to be much more predictive of transfers between tasks. If we now weigh the alignment by the degree of overlap (GTSJsim×CS), we expect the GTS to become even more predictive of transfers (i.e. high alignment with larger parameter sharing should lead to higher transfer than high alignment with little parameter sharing). Surprisingly, GTSJsim×CS does not lead to improved prediction of transfers leaving gradient alignment as the best predictor of transfers.
5For WD, we report the absolute value of the correlation, as we relate a distance with a similarity measure.
Global transfer patterns We can get a global idea of the type of features the LMs generalise across, by comparing the task space with a ‘hypothesis space’, a synthetic space representing a hypothesis that we expect a model to generalise across. To see whether the task spaces capture meaningful higher-level features within linguistic phenomena, or rather are due to low-level, spurious features, we generate three different types of hypothesis spaces. First, we test the clustering of paradigms into their higher-level BLiMP phenomena as it is shown in Figure 1. A high correlation of task spaces with this hypothesis space means that LMs indeed find the higher-level structural features that all paradigms from the same phenomenon have in common. We compare this against low-level, spurious controls in the form of vocabulary-level similarities between tasks (normalised vocabulary overlap [V. overlap] and the Wasserstein distance [WD] between vocabulary distributions; additional information in Appendix A.1.1). We observe that the phenomenon structure is much more predictive of the generalisation patterns than the low-level controls (see the bottom portion of Table 1), con-
firming that the models generalise related tasks beyond their shared vocabulary and that we can capture this generalisation in our task spaces. The generalisation within phenomena further increases with increasing model size as shown by higher correlations for larger models.
Individual phenomena In the previous paragraph, we found that LMs tend to generalise according to the higher-level linguistic structure globally. We can get a more differentiated picture by looking at the within-phenomena transfers for individual phenomena. As laid out in § 3.1, there are three main transfer patterns that we can expect within a phenomenon: First, the paradigms within a phenomenon have high similarity values, as we can see in the phenomenon determiner noun agreement [DN-AGR]) in Figure 4a). In this case, the model has discovered the overarching phenomena. Beyond DN-AGR, we observe a similar pattern for ELLIP, QUANT and SV-AGR, showing in the high average within-phenomenon similarity values with relatively low standard deviations in Figure 5 (see Appendix A.3 for a full table of abbreviations). Second, the model discovers the similarity between paradigms, but cannot reconcile them, as we see it in filler-gap dependencies [FG-DEP] (see Figure 4b). Those phenomena have
low similarity values but very high standard deviations or negative similarity. Models find subclusters of paradigms to transfer across, but cannot reconcile the different subclusters with each other. As a consequence, the subclusters are highly interfering with each other. In irregular forms [IRR-F], the ‘subclusters’ consist of single paradigms that test different usages of lexical items with irregular morphology (e.g. as a verb vs. as an adjective). Our models do not resolve this ambiguity, leading to high interferences between the tasks. Third, for some phenomena, we do not observe any interactions between their paradigms (see, e.g. binding [BIND] in Figure 4c). The LM finds idiosyncratic solutions to all the paradigms and does not discover the more general phenomenon. With increasing size, models tend more towards the first pattern, solving paradigms more by generalising to the higher-level phenomenon.

heading: 4.2.3 Analysis of the training process
text: We have established that linguistic task spaces inform us about LMs language conceptualisation. But how do they change throughout training? Performance on the BLiMP benchmark increases starkly in early training (for learning curves, see Appendix A.2.1). What can we learn from task spaces beyond that observation?
Development of task spaces We construct similarity spaces for nine model checkpoints throughout pre-training (see Appendix A.5 for visualisations of all similarity spaces). Overall, we find that similarity spaces are remarkably stable: similarity patterns are present from very early in training (within the first epochs; for details, see Appendix A.4). At the same time, the generalisation within phenomena continuously increases, showing a continuous reinforcement of the existing generalisation pattern without major structure shifts in the pattern (see Figure 6a).
The continuity of generalisation patterns is surprising and contrasts with human learning, which is marked by learning stages (Piaget et al., 1952; Gopnik et al., 1999, 2004): as humans deepen their knowledge, new patterns emerge. Language learning in LMs is not marked by such incisive shifts.
Development of subspaces Another interesting angle of analysis is the change of the subspaces θ0 in which the model is learning specific paradigms. The average size of |θ0| continuously grows during pre-training (see Figure 6b), showing that LMs learn linguistic tasks initially more localised but become more distributed throughout training. While the relative subspace overlap Jsim increases generally with training, the within-phenomenon overlap is overall higher (see Figure 6c top). Additionally, the alignment of gradients increases selectively for
paradigms from the same phenomenon (see Figure 6c bottom). This shows how the processing of linguistic tasks starts idiosyncratic (separated and in non-aligned subspaces) and with training, the sharing of structure increases (shared and in aligned subspaces, where appropriate).

heading: 4.3 Linguistic hypothesis testing
text: In addition to testing linguistic task spaces against hypothesis spaces with known structural similarities (such as ‘clustering by phenomena’ or our vocabulary controls), we can also use them to test assumed similarities in linguistic structure. We can construct a hypothesis space that represents contested ideas in linguistic theory and test whether our LM generalises according to the hypothesis by calculating a simple correlation. Our methodology is a step towards model-based theorising or ‘synthetic linguistics’ (Chowdhury and Zamparelli, 2019). Doing linguistic hypothesis testing, however, is beyond the scope of the current paper.

heading: 5 Discussion
text: In this paper, we construct linguistic task spaces, representing an LM’s language conceptualization, which can be used for linguistic hypothesis testing and as a holistic interpretability tool. We introduce FTGD to selectively fine-tune latent, entangled concepts such as linguistic tasks, and ‘similarity prob-
ing’ to estimate similarities between linguistic tasks through their transfer learning and gradient analysis. We analyse the resulting similarity spaces of LMs throughout pre-training to interpret their learning process.
We find that learning of linguistic tasks begins localized and becomes more distributed with training, with increased parameter sharing among linguistic tasks and gradient alignment especially between linguistically related tasks. Learning theory suggests that a trained model requires fewer dimensions due to increasingly efficient compression rules (e.g. Cheng et al., 2023). Opposing the assumed reduction in intrinsic dimension, training in our experiments actually increases the number of extrinsic dimensions on which a task is learned. Intrinsic and extrinsic dimensions might be inversely related in language models, as previously observed by Aghajanyan et al. (2021). A more distributed processing of tasks allows for more overarching structure sharing and generalisation across different subconcepts, which potentially helps to achieve lower intrinsic dimensionality. Furthermore, we find that generalization patterns remain surprisingly stable throughout pre-training, without stark shifts to new patterns—a behaviour more typical of human-like learning. This potentially reflects the weakness of classical neural network models to generalise systematically (Hupkes et al., 2020; Lake and Baroni, 2023, see also Lake and Baroni (2018); Ettinger et al. (2018); Bahdanau et al. (2019); Keysers et al. (2019); Yu and Ettinger (2020); Kim and Linzen (2020); Press et al. (2022)). Future generations of LMs employing more human-like learning paradigms (see e.g. Lake and Baroni, 2023) may exhibit stronger shifts in generalization patterns. The observed continuity might explain the lack of successful curriculum learning strategies for language modelling in the past (see e.g. Surkov et al., 2022; Campos, 2021; Weber et al., 2023): in a learning process without notable shifts in generalisation patterns, changes in the data distribution during training are not beneficial.
Future reseach Beyond language, our approach to interpreting LM conceptualisation can be applied to other domains to better understand the current weaknesses of LMs, such as numerical reasoning and cross-lingual concept learning. Furthermore, the potential for explicit linguistic hypothesis testing, though underexplored in this paper, can help bridge the gap between formal linguistic and com-
putational linguistic research. Large, state-of-theart LLMs may uncover subtle structural similarities that are informative to linguists.

heading: 6 Limitations
text: There are several limitations to the presented methods. First, our fine-tuning and evaluation data are i.i.d. and come from a very narrow distribution: the data are not natural but synthetic, and all data are generated using the same templates. We use this very narrow i.i.d. data to asses the fine-tuning success during probing. However, we cannot be entirely sure whether we succeeded in fine-tuning a specific linguistic task rather than some idiosyncracies of the narrow data distribution. While our FTGD approach might elevate this issue slightly, it does not dispel our doubts completely. The optimal way to guarantee our results would be the evaluation on a set from a separate distribution.
Second, while our approach applies to all types of knowledge domains, it requires minimal pairs of tasks within that domain to fine-tune them selectively. Minimal pairs are primarily used in linguistics and are uncommon in other knowledge domains.
Third, as discussed in the previous section, a major weakness of our probing approach lies in the necessary top-down definition of ‘anchors’ that we use to span the space. We utilise human-defined tasks and relate them to each other. However, a more accurate linguistic space can probably be described by ‘anchors’ that are defined through the model itself and span the conceptual space with maximal expressivity.

heading: Acknowledgements
text: We thank the COLT group at UPF for the discussions and their useful feedback; Further, LW thanks the Department of Translation and Language Sciences at the University Pompeu Fabra for funding. Further, this research was possible through computational resources acquired through a grant of the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 101019291). This paper reflects the authors’ view only, and the ERC is not responsible for any use that may be made of the information it contains. Ultimately, we thank the anonymous reviewers for their time and useful feedback!

heading: A Appendix
text: The supplementary material to this paper contains additional information about the control hypothesis spaces that we employ to verify the meaningfulness of our linguistic spaces (Appendix A.1). Further, we document the development of the LMs’ performance on BLiMP in different scenarios (Appendix A.2). Ultimately, we show all heatmaps for all transfer and gradient spaces for all models throughout the whole training process (Appendix A.5).
A.1 Controls
We include control conditions and baselines for our experiments. This appendix section provides additional details.
A.1.1 Vocabulary baselines We calculate two baselines to estimate the amount of transfer that is due to mere vocabulary overlap between different paradigms:
1. Normalised vocabulary overlap (NVO) between the vocabularies VA and VB of paradigms A and B – calculated simply as the size of their intersection normalised by the maximum vocabulary overlap between any paradigms X and Y :
NV O = |VA ∩ VB|
max(|VX ∩ VY |)
2. Wasserstein distance (WD; Kantorovich, 1960) between the vocabularies distributions. These vocabulary controls can be correlated with any task space or hypothesis space. For example, the correlation between these controls and the transfer task spaces § 4.2.2 indicates how much of the transfer between different paradigms can be attributed to the vocabulary overlap between tasks alone.
A.2 BLiMP performance
Throughout our experiments, we pre-train and fine-tune our LMs. We here document the performance of the models in different scenarios: first, we show how the models perform on the whole benchmark throughout the pre-training process. Second, we show how different pre-training checkpoints adapt.
A.2.1 BLiMP learning curves During the pre-training process, we evaluate each saved checkpoint on all paradigms of the BLiMP benchmark and average the results. The following plot shows the respective learning curves for the different models. While none of the models achieve very good performance, the largest model achieves their final performance much faster than the smaller ones.
A.2.2 BLiMP probe tuning The final performance after fine-tuning a specific task changes with the amount of pre-training. The final performance of that model for that specific task is shown in Figure 9. With more pre-training, models adapt better during the fine-tuning. Larger models generally adapt better than smaller models. FTGD works better for models that are pre-trained for longer. This makes sense, as the method requires the difference between minimal pairs to be meaningful (i.e. it requires previous knowledge already contained in the model parameters). The subspace selection will be more accurate as a consequence.
A.3 BLiMP abbreviation map
A.4 Similarity space stability We here show the correlation of similarity spaces of different epochs with the final similarity space (epoch 20). Generally, similarity spaces are remarkably stable and do not show larger shifts in generalisation patterns.
A.5 Details similarity spaces This section contains additional information about similarity spaces that we constructed throughout the paper.
A.5.1 Similarity spaces through training We construct all similarity spaces throughout the training process. Figure 11 on the following page illustrates the transfer and gradient matrices for each saved model checkpoint. The respective indices for the rows and columns of the heatmaps can be found in Table 3 on the subsequent page.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Paradigm 2
LM 27
| Ep
oc h
0 LM
27 |
Ep oc
h 1
LM 27
| Ep
oc h
2 LM
27 |
Ep oc
h 3
LM 27
| Ep
oc h
4 LM
27 |
Ep oc
h 5
LM 27
| Ep
oc h
10 LM
27 |
Ep oc
h 15
LM 27
| Ep
oc h
20
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Paradigm 2
LM 70
| Ep
oc h
0 LM
70 |
Ep oc
h 1
LM 70
| Ep
oc h
2 LM
70 |
Ep oc
h 3
LM 70
| Ep
oc h
4 LM
70 |
Ep oc
h 5
LM 70
| Ep
oc h
10 LM
70 |
Ep oc
h 15
LM 70
| Ep
oc h
20
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Paradigm 2
LM 20
3 | E
po ch
0
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch
1 1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch 2
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch 3
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch 4
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch 5
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch 1 0
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch 1 5
1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 Pa ra di gm 1 LM 20 3 | E po ch 2 0
(a )T
T S
fo ra
ll m
od el
s th
ro ug
ho ut
tr ai
ni ng
.
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 Paradigm 2
LM 27
| Ep
oc h
0 LM
27 |
Ep oc
h 1
LM 27
| Ep
oc h
2 LM
27 |
Ep oc
h 3
LM 27
| Ep
oc h
4 LM
27 |
Ep oc
h 5
LM 27
| Ep
oc h
10 LM
27 |
Ep oc
h 15
LM 27
| Ep
oc h
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 Paradigm 2
LM 70
| Ep
oc h
0 LM
70 |
Ep oc
h 1
LM 70
| Ep
oc h
2 LM
70 |
Ep oc
h 3
LM 70
| Ep
oc h
4 LM
70 |
Ep oc
h 5
LM 70
| Ep
oc h
10 LM
70 |
Ep oc
h 15
LM 70
| Ep
oc h
20
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 Paradigm 2
LM 20
3 | E
po ch
0
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
1
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
2
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
3
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
4
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
5
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
1 0
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
1 5
1 3
5 7
9 11
1 3
15 1
7 19
2 1
23 2
5 27
2 9
31 3
3 35
3 7
39 4
1 43
4 5
47 4
9 51
5 3
55 5
7 59
6 1
63 6
5 67
Pa ra
di gm
1
LM 20
3 | E
po ch
2 0
(b )G
T S
fo ra
ll m
od el
s th
ro ug
ho ut
tr ai
ni ng
.
Fi gu
re 11
A.5.2 Paradigm/index map A complete list of BLiMP phenomena, paradigms and the respective indices for the heatmaps throughout the paper.

heading: Future Work
text: Future generations of LMs employing more human-like learning paradigms (see e. g. Lake and Baroni, 2023) may exhibit stronger shifts in generalization patterns. The observed continuity might explain the lack of successful curriculum learning strategies for language modelling in the past (see e. g. Surkov et al. , 2022; Campos, 2021; Weber et al. , 2023): in a learning process without notable shifts in generalisation patterns, changes in the data distribution during training are not beneficial. Future reseach Beyond language, our approach to interpreting LM conceptualisation can be applied to other domains to better understand the current weaknesses of LMs, such as numerical reasoning and cross-lingual concept learning. Furthermore, the potential for explicit linguistic hypothesis testing, though underexplored in this paper, can help bridge the gap between formal linguistic and com-
putational linguistic research. Large, state-of-theart LLMs may uncover subtle structural similarities that are informative to linguists