Venue: NAACL 2025

# Evaluating Contextualized Representations of (Spanish) Ambiguous Words:

A New Lexical Resource and Empirical Analysis

Pamela D. Riviere\({}^{1}\), Anne L. Beatty-Martinez\({}^{1}\), Sean Trott\({}^{1,2}\)

\({}^{1}\)Department of Cognitive Science, UC San Diego

\({}^{2}\)Computational Social Science, UC San Diego

{pdrivier, abeattymartinez, sttrott}@ucsd.edu

###### Abstract

Lexical ambiguity--where a single wordform takes on distinct, context-dependent meanings-serves as a useful tool to compare across different language models' (LMs') ability to form distinct, contextualized representations of the same stimulus. Few studies have systematically compared LMs' contextualized word embeddings for languages beyond English. Here, we evaluate semantic representations of Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models. We develop a novel dataset of minimal-pair sentences evoking the same or different sense for a target ambiguous noun. In a pre-registered study, we collect contextualized human relatedness judgments for each sentence pair. We find that various BERT-based LMs' contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark. In exploratory work, we find that performance scales with model size. We also identify _stereotyped trajectories_ of target noun disambiguation as a proportion of traversal through a given LM family's architecture, which we partially replicate in English. We contribute (1) a dataset of controlled, Spanish sentence stimuli with human relatedness norms, and (2) to our evolving understanding of the impact that LM specification (architectures, training protocols) exerts on contextualized embeddings.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8322–8338

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

State-of-the-art language models (LMs) display a remarkable level of formal linguistic competence Mahowald et al. (2024). To date, however, we currently lack a precise accounting of the mechanisms underlying LMs' fundamental linguistic capabilities. The opacity of model internals has motivated work probing the transformations that inputs undergo as they are processed through various model components Tenney et al. (2019); Hu et al. (2020); Wang et al. (2022); Zou et al. (2023). This work has focused on LMs trained with English-language corpora, with a smaller subset of studies investigating cross-linguistic representations in multilingual models Chang et al. (2022); Wendler et al. (2024); Michaelov et al. (2023) or comparing representations across multiple monolingual models Edmiston (2020). As others have noted Blasi et al. (2022b); Bender (2009), an over-reliance on English as a "model language" limits the generalizability of findings, as well as potential applications Blasi et al. (2022a). Here, we extend interpretability work to Spanish, a language spoken by almost 600M people (with almost 500M native speakers)1. Specifically, we: (1) evaluate the trajectory of ambiguous Spanish words' semantic representations within mono- and multilingual LMs and (2) identify layers along a model's architecture whose semantic representations best capture human judgments of semantic relatedness.

Footnote 1: [https://www.exteriores.gob.es/en/](https://www.exteriores.gob.es/en/) PoliticExterior/Paginas/ElEspanolEtEnDMudo.aspx

Excision for Computational Linguistics

contend with in LMs and a useful _tool_ to understand LM representations.

We first provide a survey of related work on ambiguity and interpretability in LMs (Section 2), then present a novel dataset of human relatedness judgments about Spanish ambiguous words--in context (SAW-C) (Section 3). Section 4 documents our use of this dataset to empirically probe the representation of ambiguous words in pre-trained spanish LMs, focusing first on pre-registered analyses2 of the monolingual BETO [10]. We next systematically compare multiple monolingual Spanish and multilingual BERT-based models (Section 5). We conclude with a discussion of the implications of the current results (Section 6), as well as limitations and directions for future work (Section 7).

Footnote 2: Code and data to run all analyses are available at: [https://github.com/seantrott/spanish_norms](https://github.com/seantrott/spanish_norms). Preregistration is available at: [https://osf.io/n5htp](https://osf.io/n5htp).

## 2 Related work

To date, lexical ambiguity has been largely explored within English monolingual models [9, 8, 10]. Less work is available with models trained on other languages, such as Spanish [11, 12], and crucial distinctions emerge across studies within this literature: (1) the operationalization of same and different sense conditions, (2) the degree to which sentential context is controlled around the target word, and (3) the extent to which human semantic judgments are collected to set usage-based expectations for the _context-dependence_ and _graded quality_ of ambiguous word meanings [13, 14].

In Spanish, mono- and multilingual BERT-based models can capture information about semantic relationships between homonyms and their synonyms [10] and can approximate words' degree of polysemy [12]. However, available studies tend to leverage naturalistic sentence stimuli from sense-annotated corpora. While valuable, the variability in token sequence length and target word position within naturalistic sentential contexts may make it challenging to isolate the precise effect of the context's semantic _content_ from the uncontrolled effects of sentence frame [9]. We thus follow as closely as possible the experimental structure leveraged (for English) in Trott and Bergen [15], creating Spanish-language sentence pairs that vary along a single context cue evoking either the same or different sense of the target ambiguous noun (see Section 3.1). For our dataset, we document the extent to which context cues presented true minimal pairs (e.g. zero-token-differences across sentence pairs) for the BERT-based models we tested (see Appendix A.1; Table 3).

Using more controlled sentence stimulus design, coupled with empirically collected human benchmarks, prior studies in English have shown that BERT-based models' contextualized embeddings capture some--though not all--variance in human similarity [9] and relatedness [14, 15] judgments for ambiguous English words. Some of this work has also argued that the _continuous_ nature of LM contextualized representations makes them well-suited as models of human word meanings, which are likely graded to some extent [16, 17, 18, 19, 15, 16, 17, 18, 19, 20, 21, 22]--though importantly, may also exhibit marked _categoriality_ compared to LM representational spaces [14].

Finally, another important line of work has used techniques like classifier probes [10] and activation patching [15, 16, 17] to decode the putative functional role of different model components (e.g., layers, attention heads, etc.) in producing observed behavior.

To our knowledge, there is little to no work combining these strands of research: i.e., making use of graded human judgments about ambiguous Spanish words to trace the dynamics of contextualized representations in pre-trained Spanish language models. This is the gap we aim to address.

## 3 Human Annotation Study

We created a dataset containing graded human judgments about Spanish ambiguous words--in context (SAW-C). This process involved first producing and curating materials (i.e., target words and sentences), collecting judgments from native Spanish speakers, and validating those judgments for quality and reliability. This study was pre-registered on the Open Science Framework (OSF) platform.

 



### Materials

All sentences were developed by two native (Puerto Rican) Spanish speakers, both authors in this study. \(102\) target words were drawn from noun lists collected in earlier studies of Spanish lexical ambiguity [86, 37], as well as spontaneously generated and then verified using the online Real Academia Espanola3 dictionary. We excluded wordform meanings that corresponded to distinct parts of speech, but we accepted a small fraction (\(8/102\)) of nouns whose grammatical gender changed across meanings. Each target ambiguous noun was embedded within sentence pairs that differed by a single modifier4, termed context cue5. The average number of words in sentences was \(4.72\). The context cues across the sentence pair could evoke either the Same or Different sense of the word across the contexts.

Footnote 3: [https://www.rae.es/](https://www.rae.es/)

Footnote 4: For \(187/812\) sentence pairs, the modifiers varied along more than just a single word, as was the case when the modifiers required different prepositions, or the grammatical gender of the modifier differed across sentences and required distinct determiners and contractions. Of these, \(173\) pairs differed by \(1\) word; \(14\) differed by \(2\) words.

Footnote 5: Context cues were adjectival modifiers for \(100/102\) target nouns; verbs for \(2/102\).

**1a.** Compro el **aceite** de oliva (_[S/he] bought the olive oil_)
**1b.** Compro el **aceite** de cocina (_[S/he] bought the cooking oil_)
**2a.** Compro el **aceite** de motor (_[S/he] bought the motor oil_)
**2b.** Compro el **aceite** de carro (_[S/he] bought the car oil_)

The minimum number of _sentence pairs_ per target noun was \(6\), with a maximum of \(28\) (M = \(7.96\)). A total of \(812\) sentence pairs and \(102\) target nouns were included in the dataset.

Finally, for the purpose of human norming, the \(812\) sentence pairs were assigned to \(10\) experimental lists using a Latin Square design, where each list had approximately \(81\) or \(82\) sentence pairs.

### Participants

Our goal was to collect a minimum of \(10\) judgments per sentence pair (i.e., a minimum of \(100\) participants). Because we anticipated a non-zero exclusion rate, our pre-registration specified: 1) an initial goal of \(120\) participants; and 2) a plan to sample more participants as needed, if any sentence pairs had fewer than \(10\) observations after applying the pre-registered exclusion criteria.

Using the two-step process described above, we recruited an initial pool of \(139\) participants through Prolific. Participants received $\(2.40\) for participating and the median completion time was 12 minutes and 23 seconds, for an average rate of $\(11.64\) per hour. On Prolific, we screened for participants who reported that their primary language was Spanish; we specifically recruited participants from the United States, as well as countries in which the dominant language was Spanish (including Chile, Mexico, and Spain).

We excluded participants (1) who failed "catch" trials (where the sentences in the pair were identical), (2) whose task completion times exceeded the sample mean by 3 standard deviations, (3) whose inter-annotator agreement for the items they rated was very low (Pearson's \(r^{2}<0.1\)), and (4) who self-identified as non-native Spanish speakers. After all exclusions, we considered data from a total of \(131\) participants.

Participants' self-identified nationalities corresponded heavily to Mexico (69), Spain (39), and Chile (20); 1 participant was from the United States and 2 participants were from Venezuela.6 54 participants self-identified as female (74 male, 3 nonbinary). The average age was \(30.97\) (median = \(28\)), and ranged from \(20\) to \(59\).

Footnote 6: As described in Section 3.4, we found relatively high correlations between mean relatedness judgments produced by participants across Chile, México, and Spain.

### Procedure

After providing consent, participants were given instructions explaining that some words can have different meanings in different contexts (using an example that was _not_ included in the experimental materials), and that the goal of the experiment was to collect ratings about the relatedness of the meanings expressed by a given word across two sentence contexts.

Each sentence pair was presented on a separate page. Participants were instructed to rate each word on a scale from 1 (_totalmente sin relacion_, "totally unrelated") to 5 (_mismo significado_, "same meaning")7. The target word (e.g., _acetic_, "oil") was centered in larger font, and the target sentences were presented side-by-side (the side was randomized across trials). Participants indicated their re

sponse via button-press; see **Figure 1** for an example. We also included one "catch" trial in the experiment, which simply contained the same sentence, repeated (i.e., the correct answer was \(5\)).

The entire experiment (including consent form, instructions, and debrief page) was conducted in Spanish. Participants were randomly assigned to lists, and the order of items within each list was randomized. After completing the primary experiments, participants read a debrief form explaining once more that the goal of the experiment was to collect judgments about ambiguous Spanish words, and that their data would be anonymized before analysis and publication.

### Validation of Final Dataset

We validated the final dataset using several approaches. First, we applied multiple exclusion criteria (Section 3.2) and collected a minimum of \(10\) ratings per sentence pair. The average number of ratings per pair was \(13.1\); the maximum was \(17\).

After applying exclusion criteria, we recalculated inter-annotator agreement to estimate the reliability of the ratings in the final dataset. Following past work Hill et al. (2015); Trott and Bergen (2021) we calculated inter-annotator agreement using a _leave-one-annotator-out_ scheme. For each of the final \(131\) participants, we calculated Spearman's \(\rho\) between the judgments produced by that participant and the mean judgment for those same sentence pairs, leaving out that participant's data. The resulting distribution of correlation coefficients ranged from \(0.39\) to \(0.88\), with an average correlation of \(0.77\). This number is comparable to past work using similar methods Hill et al. (2015); Trott and Bergen (2021).

Finally, we compared average relatedness judgments across the three main demographic groups reported by participants Chile, Mexico, and Spain). Judgments across each group were all strongly correlated (\(r>0.82\) in all cases).

### Relatedness of Same vs. Different Sense Contexts

We then asked whether and how human relatedness judgments varied as a function of whether two uses of a word (e.g., "aceite") corresponded to the Same Sense or Different Sense. These will hereafter be considered the levels of the binary variable we call Sense Relationship.

Using the entire dataset of trial-level judgments (\(10639\) observations), we fit a linear mixed effects model in R using the _lme4_ package De Boeck et al. (2011), which had Relatedness as a dependent variable and a fixed effect of Sense Relationship. The model also contained by-participant and bylist random slopes for the effect of Same Sense, and random intercepts for participants, lists, and words. (The specification of random effects was determined by beginning with the maximal model, then reducing as needed for model convergence Barr et al. (2013).) The full model explained significantly more variance than a reduced model omitting only the effect of Sense Relationship \([\chi^{2}(1)=39.59,p<.001]\). As expected, Same Sense contexts were rated as more related on average (\(M=4.35,SD=1.14\)) than Different Sense contexts (\(M=2.11,SD=1.41\); **Figure 2**).

## 4 Analysis of BETO, a Pre-Trained Spanish LLM

Using SAW-C as a probe, we conducted multiple pre-registered analyses on how BETO--a pretrained monolingual Spanish BERT-based model Canete et al. (2020)--represents ambiguous words in context, whether and to what extent these representations are predictive of human semantic representations, and _which layers_ of the model contained the most information Tenney et al. (2019). **Table 1** summarizes the research questions and their results.

Figure 1: Sample item from task. Participants emitted graded relatedness judgments from a scale of 1 (totally unrelated) to 5 (same sense) for a given target word (here: **aceite** – _oil_), using information provided by the context cue across the sentence pair (here: **de oliva**/ **de motor** – _olive l motor_).

 



### Model Details

Our pre-registered analyses used the cased version of a Spanish monolingual BERT-based model: BETO, comprised of 12 layers, each made up of 12 self-attention heads, and hidden size of 768, trained on a corpus of approximately 3B words Canete et al. (2020)8. Exploratory analyses leveraged the models summarized in the **Appendix A.1** (Table 2). Each sentence in the dataset (bracketed by special tokens [CLS] and [SEP]) was tokenized separately according to each model's tokenizer. Sentences contained periods, unlike the sentences viewed by human participants9. We report sentence pair tokenization differences across models in **Table 3** (**Appendix A.1**). When the target noun was represented by multiple subword tokens, we took the average embedding across tokens. We extracted embeddings from each model layer.

Footnote 8: Accessed via [https://huggingface.co/dccuchile/bert-base-spanish-wmm-cased](https://huggingface.co/dccuchile/bert-base-spanish-wmm-cased)

Footnote 9: In a previous version of this study, we presented models with sentences that did not contain periods, to match the stimuli exactly as human participants saw them. However, we found that LMs’ outputs explained more variance in human judgments when the periods were included at the end of the sentences.

### Which layer of BETO best captures sense boundaries?

We first assessed which layers of BETO produced representations that best distinguished between Same Sense and Different Sense uses of word. To address this question, we calculated the Cosine Distance between these contextualized representations of the target word from each sentence pair for each layer. Concretely, this yielded \(812\) Cosine Distance values for each layer of BETO.

Then, we asked how Cosine Distance evolved through the network's layers with respect to the Same/Different Sense distinction. We built a series of logistic regression models in R with Sense Relationship as a dependent variable, and Cosine Distance from a given layer \(\ell_{i}\) of BETO as an independent variable. We then measured the Akaike Information Criterion (or \(AIC\)) Akaike (2011); Burnham and Anderson (2004) of the resulting model as a measure of model fit. The best-fitting model used Cosine Distance from _layer 6_. **Figure 3** highlights the change in Cosine Distance across layers of BETO as a function of Sense Relationship, suggesting that the _difference_ between conditions was largest at this layer.

### Which layer of BETO best predicts relatedness?

Our second question was whether certain layers of BETO produced representations that better predicted _human relatedness judgments_ than others. For each layer, we calculated the correlation coefficient (both Pearson's \(r\) and Spearman's \(\rho\)) between Cosine Distance values obtained from BETO and the distribution of Mean Relatedness judgments obtained for each sentence pair. We also calculated \(R^{2}\) as an estimate of the amount of _variance explained_ in human relatedness judgments as a function of Cosine Distance from that layer alone.

As depicted in **Figure 7**, the layer of BETO with the highest \(R^{2}\) was _layer 12_ (\(R^{2}=0.36,r=-.60,\rho=-.61\)). However, performance did not

Figure 3: Average Cosine Distance between the contextualized representations of the target ambiguous word across each layer of BETO, depicted as a function of whether the contexts cued the Same Sense or Different Sense.

Figure 2: Density plot representing the distribution of mean relatedness judgments for sentence pairs. As expected, word meanings were rated as more related when used in Same Sense than Different Sense contexts.



meaningfully improve beyond _layer 7_ (\(R^{2}=0.35\)). This suggests that the operations performed by later layers were less useful in terms of producing contextualized representations that captured relevant variance in relatedness judgments (see also **Appendix A.2**).

### BETO under-performs inter-annotator agreement

We then compared the best-performing layer (\(\ell=12\)) to human inter-annotator agreement (calculated in Section 3.4). The correlation magnitude of the best-performing layer (\(\rho=0.61\)) was considerably lower than the average inter-annotator agreement (\(\bar{X}_{\rho}=0.77\)), with BETO's performance lying approximately in the bottom \(5\%\) of the distribution of agreement values (**Figure 4**).

### BETO is Less Sensitive to Sense Boundaries

Past work conducted in English Trott and Bergen (2021, 2023) suggests that LMs are less sensitive to sense boundaries--the distinction between Same and Different Sense--than humans. However, it is unclear whether this effect generalizes to Spanish speakers and Spanish LMs.

We constructed a linear mixed effects model in R with Relatedness as a dependent variable, fixed effects of Cosine Distance and Sense Relationship, by-participant random slopes for both fixed effects, and random intercepts for participants, words, and lists. The full model explained significantly more variance than a model omitting only the effect of Sense Relationship [\(\chi^{2}(1)=325.21,p<.001\)]. The full model also explained more variance than a model omitting only Cosine Distance [\(\chi^{2}(1)=236.73,p<.001\)]. Further, the \(R^{2}\) of a linear regression model predicting Mean Relatedness using Sense Relationship alone (\(R^{2}=.61\)) explained more variance than Cosine Distance alone (\(R^{2}=0.36\)); adding both predictors resulted in a modest improvement over the Sense Relationship model (\(R^{2}=0.66\)).

Finally, we extracted the _residuals_ of a linear regression model predicting Mean Relatedness from Cosine Distance alone. We then plotted the distribution of these residuals according to Sense Relationship. As illustrated in **Figure 5**, BETO (as well as all the Spanish language models tested in Section 5) consistently _underestimated_ the relatedness of Same sense pairs, and consistently _overestimated_ the relatedness of Different sense pairs.

## 5 Comparing Pre-Trained Spanish Language Models

Section 4 tested several pre-registered hypotheses (**Table 1**) with respect to a single pre-trained Spanish LM. Here, we extend this work in exploratory analyses of additional pre-trained Spanish BERT-based models. Testing multiple models is an important step towards establishing the _external validity_ of a finding; additionally, it is useful for testing hypotheses about model scale Kaplan et al. (2020) or other model specifications (e.g., architecture, multilingual status).

Figure 4: Distribution of human inter-annotator agreement scores, calculated using a _leave-one-annotator out_ scheme. The vertical dashed line represents the correlation between human judgments and Cosine Distance values extracted from BETO.

Figure 5: Residuals of linear regression models fit for each LM, predicting relatedness from the interaction between cosine distance and layer position; residual distributions are separable as a function of Sense Relationship.

 



### Models

We considered \(10\) monolingual Spanish language models (including BETO) and \(2\) multilingual models. Models varied in their training procedures (e.g., BERT vs. RoBERTa; Liu et al. (2019)), tokenization scheme, number of layers, training corpus size, total number of parameters, and whether or not they were multilingual (**Table 2**).

### Impact of model scale

Past work Kaplan et al. (2020) suggests that increases in a model's number of parameters may correlate with metrics of model performance (e.g., perplexity). At the same time, there is some evidence that increasing scale does not always produce more human-like representations, i.e., large models with lower perplexity do not always better predict human reading times Kuribayashi et al. (2021).

To assess this question, we compared the maximum \(R^{2}\) achieved by each of the \(12\) language models10 and asked whether a model's best \(R^{2}\) was related to its size. We found that model size was in fact correlated with its ability to predict human relatedness judgments in Spanish 11 (**Figure 6**). RoBERTa-large (with the most parameters) outperformed all other models. Multilingual models, however, performed just as poorly as monolingual models with many fewer parameters. Notably, models within the Spanish-language ALBERT family--carefully controlled for various training and architectural properties Canete et al. (2022)--failed to show clear, consistent evidence of scaling beyond the jump from -tiny to -base architectures. None of the models matched the variance explained in average judgments by individual human annotators (i.e., inter-annotator \(R^{2}\)).

Footnote 10: I.e., the \(R^{2}\) from the best-performing layer of each model.

### Performance across layers

In Section 4, we found that cosine distance measures extracted from the _middle and last layers_ of BETO were most useful for predicting whether two contexts belonged to the same sense, and for predicting human relatedness judgments. Do other models or model families show the same trajectory of performance across layers?

Models varied in their number of layers. Thus, we first compared the trajectory of \(R^{2}\) across layers for the subset of models with the same number of layers, i.e., \(12\) layers (**Figure 7**) or \(24\) layers (**Figure 9**). In each case, we identified two qualitatively distinct "classes" of trajectory: a _rise and plateau_ trajectory, in which performance improves up until a point (e.g., layer 6) and then stays relatively stable; and a _rise and fall_ trajectory, in which

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Research Question** & **Result** & **Section** \\ \hline Do humans judge Same & Yes & 3.5 \\ Sense uses to be more & & \\ related than Different & & \\ Sense uses? & & \\ Which layer of BETO & & \\ is most sensitive to the Same/Different & & \\ Sense distinction? & & \\ Which layer of BETO & & \\ is most correlated with & & \\ human relatedness judgments? & No & 4.4 \\ Does BETO match & & \\ human inter-annotator & & \\ agreement? & No & 4.5 \\  & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of pre-registered research questions and their results.



performance improves and then decays substantively in the final layers (**Figure 8**).

In order to compare all models on the same axis, we calculated the _layer depth ratio_, which divides each layer position in a given network by the total number of layers in that network. We then visualized the average \(R^{2}\) by _layer depth ratio_ across the three model families tested: ALBERT, BERT, and RoBERTa. As **Figure 8** suggests, the two putative trajectories appear to covary with model family: the ALBERT family of models shows a _rise and fall_ trajectory, while the BERT and RoBERTa family of models shows a _rise and plateau_ trajectory.

### Sensitivity to sense categoriality across models

Lastly, we asked whether the LMs tested _explained away_ the sense boundary effect in humans. Each model showed the same pattern as BETO: Cosine Distance _underestimated_ the relatedness of Same Sense meanings and _overestimated_ the relatedness of Different Sense meanings (**Figure 5**).

## 6 Discussion

We introduced a novel dataset (SAW-C) containing human relatedness judgments about ambiguous Spanish words in controlled, minimal pair contexts. Using this dataset, we probed pre-trained BERT-based models' representations of ambiguous words, finding that: 1) LM representations correlate with human judgments but do not match inter-annotator agreement, and exhibit _systematic errors_; 2) performance varies across layers, with model families showing distinct _trajectories_ of performance; and 3) performance scaled with model _size_.

The systematic _underestimation_ and _overestimation_ errors observed with respect to same vs. different sense contexts (**Figure 5**) is consistent with past work conducted in English Trott and Bergen (2021, 2023). One explanation for this is that the initial (static) embedding for an ambiguous wordform might entangle all of its multiple meanings Grindrod (2024), which must then be "teased apart" in context--but which might nevertheless persist as "attractors" in subsequent layers. Disambiguation could be made even more difficult by the presence of minimal pair contexts Garcia (2021).

Psycholinguistic research suggests that humans also activate uncued, dominant meanings in certain tasks Duffy et al. (1988); Martin et al. (1999); here, however, humans appeared to distinguish target meanings with relative ease. It is possible that humans represent distinct homonymous (though not necessarily polysemous) meanings along clearly differentiable regions of meaning space Rodd et al. (2004); Trott and Bergen (2023, 2021); Haber and Poesio (2020), which would be consistent with the fact that categorical sense boundaries Same vs. Different sense conditions) explained an overwhelming share of human relatedness judgments (Section 4.5, Trott and Bergen (2023)).

Lastly, this work contributes to expanding the linguistic diversity of both human-annotated benchmarks and interpretability research. Although Spanish ranks among the most widely spoken lan

Figure 7: Depiction of seven pre-trained Spanish LMs’ ability to predict human relatedness judgments across layers (measured as \(R^{2}\)). For ease of illustration, this plot shows only LMs with \(12\) layers.

 

guages, it suffers from a surprising dearth of resources (with notable exceptions, Baldiissin et al. (2022)), pre-trained models, and interpretability research--particularly when compared to English. In one study, a sample of 550 corpora (spanning 22 languages) contained >\(50\%\) English-language corpora, while <\(10\%\) represented Spanish-language corpora Anand et al. (2020). A wider research perspective--considering varied languages--is critical for ensuring the generalizability of findings.

## 7 Limitations

### Limitations of the Dataset

SAW-C is limited in _scope_, containing only \(812\) sentence pairs. This is considerably smaller than English benchmarks such as BLiMP Warstadt et al. (2020), which contain tens of thousands of examples. However, it is larger than or comparable in size to other, more targeted datasets involving crowd-sourced human annotations Erk et al. (2013); Haber and Poesio (2020); Trott and Bergen (2021). Relative to specifically Spanish-language datasets, ours is the only one we are aware of that collects human judgments for target ambiguous words embedded _within_ minimal pair stimuli12. Importantly, SAW-C includes not only the sentence pairs but also over \(10000\) validated _human judgments_ from \(131\) participants about those sentence pairs.

Footnote 12: For examples of other Spanish-language datasets that collect human judgments under different experimental conditions, see Estevez Monzo (1991); Gómez-Veiga et al. (2010); Dominguez et al. (2001); Haro et al. (2017); Fraga et al. (2017); Zamora-Reina et al. (2022); Baldissin et al. (2022).

Another limitation concerns stimulus generation. The ambiguous nouns included in our dataset were spontaneously produced by native (Caribbean) Spanish speakers, or selected from previously published lists Estevez Monzo (1991); Fraga et al. (2017), rather than via automated searches. Spontaneous production of ambiguous words may (1) overrepresent homonymous words Estevez Monzo (1991), and (2) underrepresent words whose multiple meanings have large dominance asymmetries Duffy et al. (1988). In future work, we intend to collect dominance norms for these items. Differences in Spanish varieties may have also led some items to appear awkward to annotators from different nationalities Baldiissin et al. (2022); Lipski (2014). We address this concern by showing strong correlations in relatedness judgments from the three heavily represented groups Chile, Spain, and Mexico, Section 3.4), but other groups' relatedness judgments may not be represented in this sample.

Finally, our sentence pairs are not _naturalistic_. Our controlled minimal pair design was intentional, enabling us to identify key differences between LM and human representations, e.g., LMs display less sensitivity than humans to the manipulation of word meaning across minimal pair contexts (**Figure 5**). At the same time, it is important to know whether and to what extent the current results replicate with naturalistic stimuli. Thus, we aim to augment SAW-C with naturalistic examples of the target ambiguous words.

### Limitations of the Analysis

All the analyses presented here are essentially _correlational_. As others have noted Hewitt and Liang (2019); Niu et al. (2022); Zhou and Srikumar (2021), supervised methods for probing LM representations are more informative about the ability of the _probe_ to learn specific features than the question of whether the LM "naturally" encodes that feature and deploys it for token prediction. Future work would benefit from the selective application of "knock-out" methods or "activation patching" Meng et al. (2022), both of which have proven more successful in characterizing the causal, mechanistic role of model components. We view the current work as a useful starting point, which can motivate future work isolating the mechanistic role of specific model circuits within each layer.

The finding that distinct model families exhibit distinct _trajectories_ in performance (**Figure 8**) is intriguing, but due to its exploratory nature, it is unclear to what extent this finding is reliable and robust to different datasets or probing methods. In a supplementary analysis (see **Appendix 11**), we found qualitatively similar clusters of trajectories in pre-trained English models, though these differences appeared considerably weaker than in the Spanish models. Future work could build on the question of whether--and more importantly, _why_--distinct model _architectures_ and training schemes lead to different _processing mechanisms_.

We also note that analysis of performance as a function of model size was not an ideal test of the scaling hypothesis, given that many features of these models (e.g., training data, model architecture) were not controlled, except within the Spanish ALBERT family.

Lastly, the structure of Spanish modifiers (which _follow_ the target noun meant that we could only 

test encoder (e.g. bidirectional) models in this study. To offset this limitation, we tested a large suite of Spanish-trained encoder models, but future studies may leverage stimuli modified to suit decoder-only architectures.

## 8 Ethical Considerations

This research was conducted with IRB approval. All data from human participants has been fully anonymized before analysis and publication.

## Acknowledgments

Pamela D. Riviere is supported by UCSD's Chancellor's Postdoctoral Fellowship Program. We thank Benjamin Bergen and Cameron Jones for valuable discussions.

## References

* Akaike [2011] Hirotugu Akaike. 2011. Akaike's information criterion. _International Encyclopedia of Statistical Science_, page 25.
* Anand et al. [2020] Pranav Anand, Sandra Chung, and Matthew Wagers. 2020. Widening the net: Challenges for gathering linguistic data in the digital age. _Response to NSF SBE_.
* Baldissin et al. [2022] Gioia Baldiissin, Dominik Schlechtweg, and Sabine Schulte im Walde. 2022. DiaWUG: A dataset for diatopic lexical semantic variation in Spanish. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 2601-2609, Marseille, France. European Language Resources Association.
* Barr et al. [2013] Dale J Barr, Roger Levy, Christoph Scheepers, and Harry J Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. _Journal of Memory and Language_, 68(3):255-278.
* Bender [2009] Emily M. Bender. 2009. Linguistically naive l= language independent: Why NLP needs linguistic typology. In _Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?_, pages 26-32, Athens, Greece. Association for Computational Linguistics.
* Blasi et al. [2022a] Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022a. Systematic inequalities in language technology performance across the world's languages. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5486-5505, Dublin, Ireland. Association for Computational Linguistics.
* Elasi et al. [2022b] Damian E Blasi, Joseph Henrich, Evangelia Adamou, David Kemmerer, and Asifa Majid. 2022b. Overreliance on English Hinders Cognitive Science. _Trends in cognitive sciences_, 26(12):1153-1170.
* Burnham and Anderson [2004] Kenneth P Burnham and David R Anderson. 2004. Multimodel inference: understanding AIC and BIC in model selection. _Sociological Methods & Research_, 33(2):261-304.
* Canete et al. [2022] Jose Canete, Sebastian Donoso, Felipe Bravo-Marquez, Andres Carvalho, and Vladimir Araujo. 2022. ALBETO and DistilBERTO: Lightweight Spanish language models. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 4291-4298, Marseille, France. European Language Resources Association.
* Cafiete et al. [2020] Jose Canete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Perez. 2020. Spanish pre-trained BERT model and evaluation data. In _PMLADC at ICLR 2020_.
* Chang et al. [2022] Tyler Chang, Zhuowen Tu, and Benjamin Bergen. 2022. The geometry of multilingual language model representations. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 119-136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Conneau et al. [2020] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* De Boeck et al. [2011] Paul De Boeck, Marijan Bakker, Robert Zwitser, Michel Nivard, Abe Hofman, Francis Tuerlinckx, and Ivailo Partchev. 2011. The estimation of item response models with the lmre function from the lme4 package in R. _Journal of Statistical Software_, 39:1-28.
* DeLong et al. [2023] Katherine A DeLong, Sean Trott, and Marta Kutas. 2023. Offline dominance and zeugmatic similarity normings of variably ambiguous words assessed against a neural language model BERT. _Behavior Research Methods_, 55(4):1537-1557.
* Dillion et al. [2023] Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can AI language models replace human participants? _Trends in Cognitive Sciences_, 27(7):597-600.
* Dominguez et al. [2001] Alberto Dominguez, Fernando Cuetos, and Manuel de Vega. 2001. 100 palabras polisiccas con sus acepciones. _Revista Electronica de Metodologia Aplicada_, 6(2):63-84.
* Duffy et al. [1988] Susan A Duffy, Robin K Morris, and Keith Rayner. 1988. Lexical ambiguity and fixation times in reading. _Journal of memory and language_, 27(4):429-446.
* Edmiston [2020] Daniel Edmiston. 2020. A systematic analysis of morphological content in BERT models for multiple languages. _arXiv preprint arXiv:2004.03032_ 

Jeffrey L Elman. 2009. On the meaning of words and dinosaur bones: Lexical knowledge without a lexicon. _Cognitive science_, 33(4):547-582.
* Erk et al. (2013) Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2013. Measuring word meaning in context. _Computational Linguistics_, 39(3):511-554.
* Monzo (1991) Adelina Estevez Monzo. 1991. Estudio normativo sobre ambigiuedad en castellano. _Cognitiva_, 3(2):237-246.
* Fraga et al. (2017) Isabel Fraga, Isabel Padron, Manuel Perea, and Montserrat Comesana. 2017. I saw this somewhere else: The Spanish ambiguous words (SAW) database. _Lingua_, 185:1-10.
* Garcia (2021) Marcos Garcia. 2021. Exploring the representation of word meanings in context: A case study on homonymy and synonymy. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3625-3640, Online. Association for Computational Linguistics.
* Soler and Apidianaki (2021) Aina Gari Soler and Marianna Apidianaki. 2021. Let's play mono-poly: BERT can reveal words' polysemy level and partitionability into senses. _Transactions of the Association for Computational Linguistics_, 9:825-844.
* Gomez-Veiga et al. (2010)Isabel Gomez-Veiga, Nuria Carriedo Lopez, M Rucian Gallego, and Jose O Vila Chaves. 2010. Estudio normative do ambigiuedad lexica en Castellano, en ninos y en adultos. _Piscologica_, 31(1):25-47.
* Grindrod (2024) Jumbly Grindrod. 2024. Transformers, contextualism, and polysemy. _Preprint_, arXiv:2404.09577.
* Haber and Poesio (2020) Janosh Haber and Massimo Poesio. 2020. Word sense distance in human similarity judgements and contextualised word embeddings. In _Proceedings of the Probability and Meaning Conference (PuM 2020)_, pages 128-145, Gothenburg. Association for Computational Linguistics.
* Haro et al. (2017) Juan Haro, Pilar Ferre, Roger Boada, and Josep Demestre. 2017. Semantic ambiguity norms for 530 Spanish words. _Applied Psycholinguistics_, 38(2):457-475.
* Hewitt and Liang (2019) John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
* Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. _Computational Linguistics_, 41(4):665-695.
* Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1725-1744.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _Preprint_, arXiv:2001.08361.
* Kuribayashi et al. (2021) Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryyo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_, pages 5203-5217. Association for Computational Linguistics.
* Li and Armstrong (2024) Jiangtian Li and Blair C Armstrong. 2024. Probing the representational structure of regular polysemy via sense analogy questions: Insights from contextual word vectors. _Cognitive Science_, 48(3):e13416.
* Li and Joanisse (2021) Jiangtian Li and Marc F Joanisse. 2021. Word senses as clusters of meaning modulations: A computational model of polysemy. _Cognitive Science_, 45(4):e12955.
* Lipski (2014) John M Lipski. 2014. The many facets of spanish dialect diversification in latin america. _Iberian imperialism and language evolution in Latin America_, pages 38-75.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. RoBERTa: A robustly optimized BERT pretraining approach.
* Mahowald et al. (2024) Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelia Fedorenko. 2024. Dissociating language and thought in large language models. _Trends in Cognitive Sciences_, pages S1364-6613.
* Martin et al. (1999) Charles Martin, Hoang Vu, George Kellas, and Kimberly Metcalf. 1999. Strength of discourse context as a determinant of the subordinate bias effect. _The Quarterly Journal of Experimental Psychology Section A_, 52(4):813-839.
* Martinez et al. (2024) Gonzalo Martinez, Juan Diego Molero, Sandra Gonzalez, Javier Conde, Marc Brysbaert, and Pedro Reviriego. 2024. Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. _arXiv preprint arXiv:2408.16012_ 

* Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In _Advances in Neural Information Processing Systems_, volume 35, pages 17359-17372. Curran Associates, Inc.
* Merullo et al. (2024) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024. Circuit component reuse across tasks in transformer language models. In _The Twelfth International Conference on Learning Representations_.
* Michaelov et al. (2023) James Michaelov, Catherine Arnett, Tyler Chang, and Ben Bergen. 2023. Structural priming demonstrates abstract grammatical representations in multilingual language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 3703-3720, Singapore. Association for Computational Linguistics.
* Nair et al. (2020) Sathvik Nair, Mahesh Srinivasan, and Stephan Meylan. 2020. Contextualized word embeddings encode aspects of human-like word sense knowledge. In _Proceedings of the Workshop on the Cognitive Aspects of the Lexicon_, pages 129-141, Online. Association for Computational Linguistics.
* Niu et al. (2022) Jingcheng Niu, Wenjie Lu, and Gerald Penn. 2022. Does bert rediscover a classical nlp pipeline? In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 3143-3153.
* Piantadosi et al. (2012) Steven T Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. _Cognition_, 122(3):280-291.
* Rodd et al. (2004) Jennifer M Rodd, M Gareth Gaskell, and William D Marslen-Wilson. 2004. Modelling the effects of semantic ambiguity in word recognition. _Cognitive science_, 28(1):89-104.
* Sahn et al. (2019) Victor Sahn, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In _Proceedings of Thirty-third Conference on Neural Information Processing Systems (NeurIPS2019)_.
* Schlechtweg et al. (2025) Dominik Schlechtweg, Tejaswi Choppa, Wei Zhao, and Michael Roth. 2025. CoMeDi shared task: Median judgment classification & mean disagreement ranking with ordinal word-in-context judgments. In _Proceedings of Context and Meaning: Navigating Discagreements in NLP Annotation_, pages 33-47.
* Schlechtweg et al. (2018) Dominik Schlechtweg, Sabine Schulte im Walde, and Stefanic Eckmann. 2018. Diachronic usage relatedness (durel): A framework for the annotation of lexical semantic change. In _Proceedings of NAACL-HLT_, pages 169-174.
* Schlechtweg et al. (2024) Dominik Schlechtweg, Shafat Mumtaz Virk, Pauline Sander, Emma Skoldberg, Lukas Treuer Linke, Tuo Zhang, Nina Tahmasebi, Jonas Kuhn, and Sabine Schulte Im Walde. 2024. The DURel annotation tool: Human and computational measurement of semantic proximity, sense clusters and semantic change. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_, pages 137-149, St. Julians, Malta. Association for Computational Linguistics.
* Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
* Trott (2024) Sean Trott. 2024. Can large language models help augment english psycholinguistic datasets? _Behavior Research Methods_, pages 1-19.
* Trott and Bergen (2021) Sean Trott and Benjamin Bergen. 2021. RAW-C: Relatedness of ambiguous words in context (a new lexical resource for English). In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 7077-7087, Online. Association for Computational Linguistics.
* Trott and Bergen (2023) Sean Trott and Benjamin Bergen. 2023. Word meaning is both categorical and continuous. _Psychological Review_, 130(5):1239-1261.
* Wang et al. (2022) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In _NeurlPS ML Safety Workshop_.
* Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. _Transactions of the Association for Computational Linguistics_, 8:377-392.
* Wendler et al. (2024) Chris Wendler, Venimim Veselovsky, Giovanni Monea, and Robert West. 2024. Do Llamas Work in English? on the Latent Language of Multilingual Transformers. _Preprint_, arXiv:2402.10588.
* Zamora-Reina et al. (2022) Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022. LSCDiscovery: A shared task on semantic change discovery and detection in Spanish. In _Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change_, pages 149-164, Dublin, Ireland. Association for Computational Linguistics.
* Zhou and Srikumar (2021) Yichu Zhou and Vivek Srikumar. 2021. DirectProbe: Studying representations without classifiers. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5070-5083.
* Zipf (1945) George Kingsley Zipf. 1945. The meaning-frequency relationship of words. _The Journal of General Psychology_, 33(2):251-256 

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to AI transparency. _arXiv preprint arXiv:2310.01405_.

## Appendix A Appendix

### LLM Specifications and Dataset Tokenization

See **Table 2** for a summary of the LMs considered in Section 5, including architecture, multilingual status, corpus size, tokenization scheme, training objective, number of layers, and number of parameters. Because models used different tokenizers, we also calculated summary statistics about the number of tokens in each sentence in each sentence pair, as well as the average number of _token differences_ (i.e., 5 vs. 4 tokens across the members of a given pair) for each tokenizer (see **Table 3**).

### Analysis of Expected Layer

In the primary manuscript, we identified which layers of BETO provided the _best_ fit, i.e., which were most effective for predicting Sense Relationship (layer 6) and which were most effective at predicting Mean Relatedness judgments (layer 12). However, in some cases, the improvements across layers are fairly marginal. Thus, in this supplementary analysis, we implemented a version of the _Expected Layer_ analysis described by Tenney et al. (2019). This analysis considers the size of the improvement across layers and estimates the layer at which particular kinds of information is expected to resolve in the network.

#### a.2.1 Methods

The _Expected Layer_ statistic considers the _improvement in performance_ (measured here as \(AIC\) or \(R^{2}\), depending on the analysis in question) across progressively more complex regression models fit with cosine distance information from each layer. This improvement in performance measure was defined as:

\[\Delta^{\ell}=Score(P_{T}^{\ell})-Score(P_{T}^{\ell-1}) \tag{1}\]

Where \(Score(P_{T}^{\ell})\) is defined as the performance (\(AIC\) or \(R^{2}\)) of a regression model equipped with cosine distance information from a given layer \(\ell\) and each previous layer, i.e., such that the number of parameters in the regression model was equal to \(\ell\). (Note that this was distinct from the approach taken in the primary manuscript, in which distinct univariate regression models were fit for each layer.) The Expected Layer statistic itself was defined as follows:

\[\bar{E}_{\Delta}[\ell]=\frac{\sum_{\ell=1}^{L}\ell\cdot\Delta_{T}^{(\ell)}}{ \sum_{\ell=1}^{L}\Delta_{T}^{(\ell)}} \tag{2}\]

#### a.2.2 Results

Using this approach, we obtained Expected Layer statistics for both predicting Sense Relationship (3.6) and for predicting Mean Relatedness (3.4). Note that in both cases, the Expected Layer was smaller than the layer at which optimal performance was achieved; this is consistent with the observation that past a certain point, additional LM layers resulted in only marginal gains in prediction.

### Additional analyses of pre-trained English BERT-based models

In the primary manuscript, we reported the results of work using ambiguity as a probe for understanding and interpreting how pre-trained Spanish LMs process word meanings. We found that larger models exhibited better performance; and secondly, that different model _families_ exhibited different trajectories of performance across layers. We asked whether these results replicated in pretrained English models, using an openly available dataset of relatedness judgments about ambiguous English words, in context (RAW-C) (Trott and

Figure 9: Depiction of three pre-trained Spanish LMs’ ability to predict human relatedness judgments across layers (measured as \(R^{2}\)). For ease of illustration, this plot shows only LMs with \(24\) layers.

 



\begin{table}
\begin{tabular}{l c l l l r l} \hline
**Model** & **\# Lang** & **Corpus** & **Tok.** & **Trn Obj** & **\# Layers** & **\# Params** \\ \hline BETO & 1 & \(\sim\) 3B & SP & DM, WWM & 12 & \(\sim\) 110M \\ BETO-uncased & 1 & \(\sim\) 3B & SP & DM, WWM & 12 & \(\sim\) 110M \\ mBERT & 104 & ? & WP & MLM, WWM\({}^{?}\), NSP & 12 & \(\sim\) 178M \\ \hline DistilBETO & 1 & \(\sim\) 3B & SP & DistillLoss, MLM & 6 & \(\sim\) 66M \\ \hline ALBETO-tiny & 1 & \(\sim\) 3B & SP & MLM & 4 & \(\sim\) 5M \\ ALBETO-base & 1 & \(\sim\) 3B & SP & MLM & 12 & \(\sim\) 12M \\ ALBETO-large & 1 & \(\sim\) 3B & SP & MLM & 24 & \(\sim\) 18M \\ ALBETO-xlarge & 1 & \(\sim\) 3B & SP & MLM & 24 & \(\sim\) 59M \\ ALBETO-xxlarge & 1 & \(\sim\) 3B & SP & MLM & 12 & \(\sim\) 223M \\ \hline RoBERTa-BNE-base & 1 & \(\sim\) 135B & byte-BPE & MLM (DM\({}^{?}\)) & 12 & \(\sim\) 125M \\ RoBERTa-BNE-large & 1 & \(\sim\) 135B & byte-BPE & MLM (DM\({}^{?}\)) & 24 & \(\sim\) 355M \\ XLM-RoBERTa & 100 & ? & SP & MLM & 12 & \(\sim\) 278M \\ \hline \end{tabular}
\end{table}
Table 2: Spanish language model properties and training procedures. Models are used (distinguish between upper and lowercase characters) unless otherwise specified. All monolingual models are trained on Spanish-language corpora; multilingual models include Spanish-language corpora. _Model Notes:_ For mBERT, (a) the corpus size per language varied and we are unsure of the total corpus size[13], (b) it was unclear to us whether the current version of the model on HuggingFace is updated with the whole-word masking (WWM) technique[14] during pre-training. For RoBERTa-BNE models, it was unclear to us whether authors used dynamic masked (DM) modeling, as in the English RoBERTa. For XLM-RoBERTa, the corpus size per language may have varied, but we were uncertain to what extent pretraining text was sampled proportionally to its representation in the corpus (Conneau et al., 2020). _Acronyms:_**BNE:** Biblioteca Nacional de Espana (National Library of Spain); **byte-BPE:** byte-level Byte-Pair Encoding; **DistillLoss:** Distillation loss (Sahn et al., 2019; Cánete et al., 2022); **DM:** Dynamic Masking (Liu et al., 2020); **MLM:** Masked Language Modeling; **NSP:** Next Sentence Prediction; **SP:** SentencePiece; **WWM:** Whole-Word Masking; **WP:** WordPiece. This summary represents our best attempt at gathering and reconstructing some model specifications—they are necessarily incomplete, and may contain inaccuracies borne from either a lack of knowledge regarding more recent updates or an imperfect understanding of the training protocols as described in the relevant primary literature and repositories.



Bergen, 2021).

#### a.3.1 A correlation between model scale and performance

We tested English versions of the Spanish language models tested in the primary manuscript (with the exception of _ALBERT-tiny_, and with the addition of two different versions of _ALBERT-base_). We then asked whether there was a relationship between the number of parameters in each model and the maximum \(R^{2}\) achieved in predicting human relatedness judgments about English words, in context. Unlike Spanish, we found a clear, positive relationship between the logarithm of the number of parameters and the maximum \(R^{2}\) (see Figure 10): a linear model estimating maximum \(R^{2}\) from number of parameters and a model's multilingual status estimated that for every order of magnitude increase in a model's number of parameters, \(R^{2}\) increased by approximately \(0.2~{}[\beta=0.198,SE=0.03,p<.001]\). On average, multilingual models also performed worse (adjusting for number of parameters), though the small number of multilingual models tested makes it difficult to determine whether this is a reliable finding.

#### a.3.2 Layer-wise trajectories by model family

We also asked whether different model _families_ displayed different performance trajectories across _layers_. In the primary manuscript, we found that the ALBERT family models displayed a _rise and fall_ trajectory, while both BERT and RoBERTa displayed _rise and plataeau_ trajectories (see Figure 8). Surprisingly, we found qualitatively similar (albeit weaker) classes of trajectories using the pretrained English models on the RAW-C dataset (see Figure 11).

### Analysis of GPT-4 Turbo

In the primary manuscript, we found that the Large Language Models tested produced representations that were _correlated_ with human judgments, but nonetheless systematically underestimated how related people judged Same Sense meanings to

Figure 10: Pre-trained monolingual English language models show evidence of _scaling_, i.e., models with more parameters achieve a higher \(R^{2}\) in predicting human relatedness judgments about ambiguous English words.

\begin{table}
\begin{tabular}{l c l} \hline \hline
**Model** & **Avg ; Modal ; Max** & **Target Noun** \\  & **Token Differences** & **\# Tokens** \\ \hline BETO & \(\sim\) 0.682 ; 1 ; 3 & \(\sim\) 1.07 \\ BETO-uncased & \(\sim\) 0.670 ; 1 ; 3 & \(\sim\) 1.03 \\ mBERT & \(\sim\) 0.997 ; 1 ; 4 & \(\sim\) 1.27 \\ \hline DistilIBETO & \(\sim\) 0.670 ; 1 ; 3 & \(\sim\) 1.03 \\ \hline ALBETO-tiny & \(\sim\) 0.619 ; 1 ; 3 & \(\sim\) 1.04 \\ ALBETO-base & \(\sim\) 0.619 ; 1 ; 3 & \(\sim\) 1.04 \\ ALBETO-large & \(\sim\) 0.619 ; 1 ; 3 & \(\sim\) 1.04 \\ ALBETO-xlarge & \(\sim\) 0.619 ; 1 ; 3 & \(\sim\) 1.04 \\ ALBETO-xxlarge & \(\sim\) 0.619 ; 1 ; 3 & \(\sim\) 1.04 \\ \hline RoBERTa-BNE-base & \(\sim\) 0.643 ; 0 ; 3 & \(\sim\) 1.05 \\ RoBERTa-BNE-large & \(\sim\) 0.643 ; 0 ; 3 & \(\sim\) 1.05 \\ XLM-RoBERTa & \(\sim\) 0.929 ; 1 ; 4 & \(\sim\) 1.28 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Average, modal, and maximum token differences across sentence pairs per LM. Tokenization schemes for all Spanish monolingual LMs were heavily represented by either non-zero or single-token differences across sentence pair stimuli, whereas the multilingual models tested here tended to more frequently generate non-zero token differences across the sentence pairs.

 

be--and overestimated how related people judged Different Sense meanings to be (see Figure 5). However, recent work Trott et al. (2024); Dillion et al. (2023) suggests that state-of-the-art LLMs like GPT-4 are capable of producing "norms" that accurately predict human judgments across various domains, including the relatedness of English words.

Because these models are "closed source", this work typically relies on _prompting_ the models with instructions and directly eliciting a judgment (e.g., a relatedness rating). Thus, a key limitation is that even if these judgments are highly correlated with human judgments, it is very difficult (in some cases impossible) to know _why_, i.e., which representations or mechanisms give rise to the behavior in question--making them less well-suited to questions about model interpretability.

Nonetheless, the question of the empirical fit between these LM judgments and human judgments is still an interesting one--particularly because past work has primarily focused on judgments in English, and it is unclear whether these LMs would excel at other languages. Thus, in this supplementary analysis, we asked whether GPT-4 Turbo, a state-of-the-art LM, produced judgments that were more predictive of human judgments than the models tested in the primary manuscript.

#### a.4.1 Methods

Following past work Martinez et al. (2024); Trott (2024); Dillion et al. (2023), we prompted GPT-4 Turbo (_gpt-4-1106-preview_) using the OpenAI Python API. GPT-4 Turbo was presented with a system prompt containing the same instructions (in Spanish) that were presented to human participants, explaining the purpose of the task. Then, for each sentence pair, Turbo was presented with the same instructions given to human participants (again in Spanish) asking them to rate the relatedness of the target word across the two contexts. The two sentences were presented on separate lines, as was the target word (e.g., "Word: acetic"). Finally, we included an additional instruction requesting a single number in response. Turbo was prompted using a temperature of 0 and its responses were limited to a maximum of 3 tokens.

#### a.4.2 Results

The ratings produced by Turbo were highly correlated with human judgments, approaching or even exceeding average human inter-annotator agreement (\(\rho=0.79\)).

We then asked whether Turbo's ratings explained away the sense boundary effect observed in the primary manuscript. First, we fit a linear model with Mean Relatedness as a dependent variable and two predictors: Rating (from GPT-4 Turbo) and Sense Relationship. The coefficients assigned to each predictor were significant, suggesting that they explained some amount of independent variance: both Rating [\(\beta=0.45,SE=0.02,p<.001]\) and Same Sense [\(\beta=1.12,SE=0.05,p<.001]\) exhibited a positive relationship.

As in Figure 5, we also visualized the residuals of a linear model containing only Rating as a predictor (Figure 12). Notably, even though Rating was highly correlated with Mean Relatedness, the residuals suggest that Turbo's ratings follow a similar pattern with respect to sense boundaries as was observed with BETO and the other models tested: GPT-4 consistently _underestimates_ the relatedness of Same Sense pairs, and consistently _overestimates_ the relatedness of Different Sense pairs.

#### a.4.3 Discussion

In this supplemental analysis, we found that a larger model trained with Reinforcement Learning from Human Feedback (RLHF) produced behavior that was much more correlated with human relatedness judgments than the other Spanish language models tested.

On the one hand, this analysis has considerable limitations: as noted above, a major challenge with relying on large language models such as GPT-4 is that, despite their impressive performance, much is still unclear about how exactly they were trained--either in terms of the original training data

Figure 11: Depiction of English LMs’ ability to predict mean relatedness judgments (measured as \(R^{2}\)), broken down by _Model Family_ and _Layer Depth Ratio_, i.e., with each layer divided by the the total number of layers in a given model.

 

or the procedure for implementing RLHF. Thus, as a tool for testing scientific hypotheses, they may be less useful than open-source models, i.e., the ones tested in the primary manuscript, especially if the scientific question under investigation concerns the _representations_ or _mechanisms_ used by the LLM. These are, of course, the kinds of questions that research on interpretability is generally interested in.

On the other hand, the fact that an LLM produces behavior that rivals inter-annotator agreement suggests that the representations required to produce this behavior can be learned provided sufficient training data and fine-tuning; in this case, direct data contamination is an unlikely concern given that the materials were entirely novel and the ratings had never been published before. This analysis is also notable in that it reveals that even a very large-scale model trained with RLHF appears to be less sensitive to sense boundaries than human judgments, as depicted in Figure 12.

However, because Turbo is a closed-source model, it is difficult to draw firm conclusions from these results precisely because we cannot investigate where or how this behavior emerges. For example, we cannot investigate which _layer_ of GPT-4 Turbo appears to be most helpful for producing high-quality relatedness judgments. Future work would thus benefit from investigating which architectural features, training objectives, or fine-tuning procedures are likely candidates for producing this improvement in performance, ideally in open-source Spanish language models.

### Additional sentence stimuli examples

All sentence stimuli will be made available following publication. Here, we list additional examples from the dataset. Since target ambiguous nouns in Spanish are not always ambiguous in English, we mark the target noun's English translation in bolded italics within each parenthetical.

**1a.**: Era un **banco** financiero. (_It was a financial bank_.)
**1b.**: Era un **banco** exitoso. (_It was a successful bank_.)
**2a.**: Era un **banco** comodo. (_It was a comfortable bench_.)
**2b.**: Era un **banco** de plastico. (_It was a plastic bench_.)
**1a.**: Disfruto el **ba\({}^{\text{no}}\)** caliente. (_[S/he] enjoyed the warm bath_.)
**1b.**: Disfruto el **ba\({}^{\text{no}}\)** frio. (_[S/he] enjoyed the cold bath_.)
**2a.**: Disfruto el **ba\({}^{\text{no}}\)** remodelado. (_[S/he] enjoyed the remodeled bathroom_.)
**2b.**: Disfruto el **ba\({}^{\text{no}}\)** privado. (_[S/he] enjoyed the private bathroom_.)
**1a.**: Tena una **mueneca** fracturada. (_[S/he] had a fractured wrist_.)
**1b.**: Tena una **mueneca** lesionada. (_[S/he] had an injured wrist_ .)
**2a.**: Tenia un **mueneca** preciosa. (_[S/he] had a lovely doll_.)
**2b.**: Tenia una **mueneca** plastica. (_[S/he] had a plastic doll_.)

Figure 12: Residuals from a linear regression predicting Mean Relatedness from ratings elicited from GPT-4 Turbo. Although Turbo’s ratings are highly correlated with human judgments, they still systematically underestimate the relatedness of Same Sense pairs and overestimate the relatedness of Different Sense pairs.