Venue: NAACL 2025

# Characterizing the Role of Similarity in the

Property Inferences of Language Models

 Juan Diego Rodriguez\({}^{\tau}\) Aaron Mueller\({}^{\tau,t}\) Kanishka Misra\({}^{\star,x}\)

The University of Texas at Austin \(~{}^{\nu}\)Northeastern University

Technion - Israel Institute of Technology \(~{}^{\chi}\)Toyota Technological Institute at Chicago

juand-r@utexas.edu aa.mueller@northeastern.eduk kanishka@titc.edu

###### Abstract

Property inheritance--a phenomenon where novel properties are projected from higher level categories (e.g., birds) to lower level ones (e.g., sparrows)--provides a unique window into how humans organize and deploy conceptual knowledge. It is debated whether this ability arises due to explicitly stored taxonomic knowledge vs. simple computations of similarity between mental representations. How are these mechanistic hypotheses manifested in contemporary language models? In this work, we investigate how LMs perform property inheritance with behavioral and causal representational analysis experiments. We find that taxonomy and categorical similarities are not mutually exclusive in LMs' property inheritance behavior. That is, LMs are more likely to project novel properties from one category to the other when they are taxonomically related and at the same time, highly similar. Our findings provide insight into the conceptual structure of language models and may suggest new psycholinguistic experiments for human subjects.1

Footnote 1: Data and code available at [https://github.com/aaromueller/lm-property-inheritance](https://github.com/aaromueller/lm-property-inheritance).

+
Footnote †: Work partly done at UT-Austin before joining TTIC.

+
Footnote †: Work partially done at UT-Austin before joining TTIC.

## 1 Introduction

Categories are fundamental to human semantic cognition. Our knowledge of categories allows us to draw everyday inferences; a prominent example of which is _property inheritance_, where properties are projected from a category to its members. For example, if we learn that _dogs_ have the T9 hormone, we can reasonably assume that _corgs_ also have the T9 hormone. In cognitive psychology, an obvious and previously popular explanation for property inheritance is that it is the natural consequence of our minds organizing categories hierarchically into taxonomies Collins and Quillian (1969); Glass and Holyoak (1974); Murphy (2002).

At the same time, empirical evidence has called this assumption into question. Sloman (1998) showed that humans were highly sensitive to the similarity of the categories when performing property inheritance; for example, they were more likely to project novel properties from _birds_ to _robins_ (a more typical bird) than from _birds_ to _penguins_ (a less typical bird), despite them agreeing that both were members of the bird category. Sloman concluded that instead of explicitly using stored taxonomies, humans might simply be computing categorical similarities to demonstrate inheritance-compatible behavior. In such a case, a property is more likely to be shared between categories infosfar as the similarity between them is high enough.

Debates about the mechanisms that underlie human property inheritance offer an exciting avenue to analyze conceptual organization and use in language models (LMs). First, due to the close connec

Figure 1: **Property inheritance involves projecting properties from a category to its members. For example, if dogs are _dxable_, are corgis _dxable_? Here, _dxable_ is a nonce word used to study property inheritance without any confounding effects from parametric knowledge. Language models may rely on taxonomic (left) and/or similarity (right) relations to perform property inheritance. We investigate the interplay between these two effects in LMs’ property inheritance judgments using both behavioral and mechanistic analyses.**

tion of a word to its conceptual meaning Murphy2002, Lupyan and Lewis2019, Lake and Murphy2021, property inheritance provides an opportunity to diagnose meaning-sensitivities in LMs Piantadosi and Hill2022, Bender and Koller2020. Next, similarity effects like those found by Sloman1998 are related to the notion of _content effects_ shown by humans and LMs on logical reasoning tasks Wason1968, Evans1989, Lampinen et al.2024; the presence of similarity effects goes against the idea that humans rely on abstract taxonomic principles (where every category member would be equally likely to inherit the property). Finally, the principles of taxonomy vs. similarity offer two clear hypotheses about what mechanisms could govern property inheritance behavior in a system. This allows us to go beyond prior work investigating property inheritance in LMs Misra2023, which has largely focused on their _behavior_, leaving open questions about the underlying mechanisms.

In this study, we empirically investigate the roles of taxonomic relations and categorical similarities on LMs' property inheritance behavior, using both behavioral and causal interpretability methods on four LMs. First, we behaviorally analyze LMs' sensitivity to taxonomic relations when performing property inheritance (SS3.4). We demonstrate that noun similarity, known to correlate with human property inheritance behavior Sloman1998, also correlates strongly with property inheritance judgments in language models, and explains many of the false positives and false negatives. Then, we causally localize property inheritance to specific activation subspaces (SS4) using distributed alignment search DASGeiger2024b. By training and evaluating DAS on controlled subsets of the data, we show that the subspaces responsible for property inheritance reflect both taxonomic and similarity features. Together, these results indicate that LMs are susceptible to non-trivial content effects rooted in noun similarities, and moreover that taxonomy and similarity are _fundamentally entangled_ in model representations.

## 2 Conceptual Organization in LMs

To what extent is the similarity vs. taxonomy debate relevant to LMs? By definition, LMs learn representations from distributional statistics of tokens in context, so any reasoning they demonstrate is a result of distributional similarity. While technically true, this viewpoint may be an oversimplification: two words can be distributionally similar if they participate in anatomy (_hot_ vs. _cold_), metonymy (_car_, _wheel_), hypernymy (_robin_, _bird_), or even if they share thematic relations (_dog_, _bone_). For property inheritance, however, it is only the **hypernymy** relation that plays a critical role Murphy2002. Is it possibile that something structural like a taxonomy might arise through distributional statistics? It has been postulated that symbolic behavior can emerge as a result of sub-symbolic processes internal to a neural network Smolensky1986,1988, McClelland et al.2010. Empirically, this has been shown in a number of recent works, e.g., Nanda2023 on LMs and modular arithmetic, and Feng2023 on LMs learning to simulate dynamic programming--both of which require operating over symbolic representations.

The above discussion raises multiple interrelated questions: when performing property inheritance, do LMs show sensitivity to taxonomic relations? If so, is their behavior _graded_, where properties are less likely to be projected to concepts that are taxonomically related to the higher-level category but low in similarity? Or, is it more _binary_, where there is no significant difference between how likely a property is projected from the higher-level category to its high- and low-similarity members? If their behavior is graded, then does it differ between non-category members that might share similarities with the higher level category (e.g., bird and giraffe) vs. those that do not (e.g., bird and chair)?

Related workBy focusing on the mechanisms that underlie LMs' property inheritance behavior, our work contributes to a rich body of work that investigates conceptual representations in language models Bhatia and Richie2021, misra2021, patel2022, abdou2021, grand2022, wu2023a, park2024, _i.a._). Complementing these, our focus in this work is to understand how LMs _use_ their conceptual structure to project properties from one category to another--an important task in the broader space of inductive problems solved by humans Kemp and Jern2014.

In this vein, recent works have investigated LMs' property inheritance behavior--either by purely behavioral analyses Misra2023,wu2023a, or by manipulating LMs' internal representations to edit their taxonomic knowledge Powell2024, cohen2024, and measuring 

the implications of the edit. Importantly, these works assume that changes to taxonomic membership must be directly reflected in the LMs' behavior, without necessarily considering the impact of categorical similarity. Grounding our investigation in empirical observations about humans (Sloman, 1998), we abstract away from this assumption, and explicitly consider the interplay between similarity and taxonomic relations by characterizing it using both behavioral and causal analysis methods.

## 3 Experimental Materials

### Data

We used the THINGS dataset (Hebart et al., 2019, 2023) as our primary repository of noun categories. THINGS is a repository of human responses on odd-one-out tasks for 1,854 unique object categories--e.g., _dogs_ and _chairs_. Each object concept is also annotated with its superordinate category (_dog-animal_). We performed a few modifications to this dataset. First, many objects belonged to more than one category--e.g., _cat_ is an _animal_ but it is also a _mammal_; we make these into separate entries. Second, we also paired each object with its WordNet sense, since one of the similarity methods we use involves ground-truth knowledge of word senses (we expand on this in SS3.2). We discarded instances for which we could not find a valid sense in WordNet. These manipulations result in 2,016 pairs of taxonomically related object-category pairs. Our final dataset has 44 superordinate and 1,281 subordinate categories, respectively. Next we describe our method to sample pairs that are not taxonomically related.

Negative SamplingBecause we are explicitly analyzing the effect of _similarity_ in LMs' property inheritance behavior, we include similarity measures as a central component for deriving pairs of related categories that are _not_ taxonomically related. We perform this sampling as follows: for each superordinate category \(C\), we construct the negative sample space \({\cal N}\) as the set of items that are _outside_ that category--e.g., for _bird_, this could mean non-birds such as _zebra_, _sofa_, etc. We then compute the similarity of each item in \({\cal N}\) to the superordinate category using a given similarity measure. Finally, if \(k\) is the number of category members of \(C\), then we sample the top \(\nicefrac{{k}}{{2}}\) and bottom \(\nicefrac{{k}}{{2}}\) concepts from \({\cal N}\), based on the similarity values. This way, for each superordinate category, we have \(k\) category members, and \(k\) non-category members, with an even split between high and low-similarity non-taxonomic items.

Stimuli designWe follow precedent from research in the psychology of concepts (Osherson et al., 1990; Sloman, 1998), and create stimuli in a premise-conclusion format, commonly used to analyze category-based inference in humans. In each instance, the premise expresses a statement where a category is said to have some property (_birds have the T9 hormone_), and the conclusion asks if a different category also has that property (_robins have the T9 hormone_). Following the same precedent, we use properties that are expressed by nonce words (e.g., _are daxable_). Since LMs are likely to have less--if any--prior knowledge about these nonce properties, this design decision allows us to isolate LMs' inference behavior to the relations between categories (i.e., taxonomy and similarity), devoid from any interference from knowledge of real properties (e.g., _can fly_) which may already be embedded in their learned weights. We experimented with multiple different surface form realizations of the stimuli, and found the following template to work best on average:2

Footnote 2: Some models demonstrated better performance when we omitted “The answer is:” from this prompt. The entire set of templates is made available in Appendix C.

Answer the question. Given that Ais/are daxable, is it true that Bis/are daxable? Answer with Yes/No. The answers is:

where A and B are the premise and conclusion categories, respectively. Most nouns were pluralized, except for those which were more naturally expressed in singular form (mass nouns, e.g., _honey_). A model that is sensitive to taxonomic relations when performing property inheritance should be more likely to generate _Yes_ than _No_ for a taxonomic pair such as (_toy_, _ doll_), while the reverse should be true for non-taxonomic pairs such as (_fruit_, _puppy_).

### Similarity scores

It is non-trivial to predict the nature of the 'similarity' that might have an effect on a system's category-based inference behavior. This is true even for humans--most experiments used human derived similarity ratings to construct their stimuli (Osherson et al., 1990; Sloman, 1998), but what these ratings actually captured was largely unknown. To this end, we used two different types of similarity metrics, one sensitive to global distributional contexts of word-senses, and the other sensitive to the visual and conceptual properties of 

objects (_animacy_, _color_, etc.). Both metrics were derived from vector-space embeddings, and similarity was calculated using cosines between vectors of the input concepts.3

Footnote 2: We discuss other possible metrics we could have used, and our reasons for not using them in the limitations.

Word-sense SimilarityWe used the LMMS-ALBERT-xxl model Loureiro et al. (2022) which was constructed by aggregating contextualized embeddings from the ALBERT-xxl LM Lan et al. (2020) for lexical items with tagged WordNet senses. This model can be considered as a distributional semantic model like Glove Pennington et al. (2014) that isolates the distributional information of the particular sense of a word--i.e., the embedding for _bat_ when used as a mammal is different from that used as sporting equipment.

SPoSE SimilarityWe used SPoSE Zheng et al. (2019), a sparse, non-negative vector space model fitted to human behavioral judgments on odd-one-out tasks, where participants were tasked to choose a concept from a triple that was least similar to the other two (e.g., \(\{\textit{crow},\textit{sparrow},\textbf{ship}\}\)). We used the 49-dimensional embedding released by Hebart et al. (2023), which achieved a correlation of up to 0.9 with human judgments of perceived similarity Kaniuth et al. (2024). Since the embeddings in SPoSE are for subordinate categories, we augmented this dataset with superordinate category vectors by taking the mean of the vector representations of the category members for each category--e.g., for _bird_, we took the average of the embeddings of \(\{\textit{crow, sparrow,...}\}\).

Table 1 shows examples of taxonomically and non-taxonomically related neighbors of a few categories according to both types of similarity.4 For each similarity type, we bin the category pairs in our stimuli into 'high' or 'low' similarity by calculating the median similarity for premise category and categorizing instances where the similarity between the premise and the conclusion is above the median to be 'high', and those below to be 'low'. This way, every premise category in our stimuli is associated with equal numbers of high and low similarity conclusion categories. In total, we have 4032 category pairs, for each type of similarity.

Footnote 4: Additional examples are given in Appendix E.

### Models

We analyze four instruction-tuned LMs: Mistral 7B Insruct v0.2 Jiang et al. (2023), the 2B and the 9B parameter variants of the Lemma 2 family Riviere et al. (2024), and the 8B parameter variant of Llama 3 Instruct Dubey et al. (2024).5 We found varying degrees of success in using the chattemplates for these LMs, and therefore used it only when it resulted in a non-trivial improvement over the standard method of passing inputs to the LMs. All models were accessed using the transformers library Wolf et al. (2020).

Footnote 5: Initial experiments showed these LMs’ non instrumented counterparts to perform worse at property inheritance, which corroborates recent evidence about the advantage of instruction-tuning for property inheritance Misra et al. (2024).

### Behavioral Characterization of Property Inheritance

We begin by characterizing the role of taxonomic relations and similarity in LMs' property inheritance behavior. To this end, we collect LMs' responses to our stimuli, and analyze the conditions under which they are more likely to extend the property from the premise category to the conclusion category. We do this by comparing their relative probabilities for 'Yes' vs. 'No' on our stimuli, which we compute as follows:6

Footnote 6: The LMs that we use have two different tokens to represent ’Yes’—on with and one without space, and similarly for ‘No’. To address this potential issue, we take the maximum of the probabilities over the two options for both cases. That is, \(p_{\theta}(\texttt{Yes})=\max(p_{\theta}(\texttt{Yes}),p_{\theta}(\_\texttt{ yes}))\), and similarly for ‘No’. We note that these tokens appeared as the top predicted tokens for all models evaluated except Germma-2-2B-IT.

\[P_{\texttt{rel}}(x)=\frac{p_{\theta}(x~{}|~{}\text{prefix})}{\sum\limits_{ x\in\{\text{Yes, No}\}}p_{\theta}(x~{}|~{}\text{prefix})},\]

where 'prefix' is the stimulus, formatted as discussed in SS3, and \(p_{\theta}(.)\) denotes the LM's next

\begin{table}
\begin{tabular}{l c l l l} \hline \multicolumn{1}{l}{**Similarity Type**} & \multicolumn{1}{l}{**Category**} & \multicolumn{1}{l}{**Taxonomic Neighbors**} & \multicolumn{1}{l}{**Non-taxonomic Neighbors**} \\ \hline \multirow{2}{*}{Word-Sense} & \multirow{2}{*}{vehicle} & _car (0.85), sled (0.70), hot-air balloon (0.59)_ & _headlight (0.82), spinach (0.11), calsome (0.09)_ \\ \cline{1-1} \cline{3-5}  & & bird & _angle (0.95), pelican (0.90), penguin (0.82)_ & _bee (0.92), gacelle (0.87), cabinet (0.08)_ \\ \hline \end{tabular}
\end{table}
Table 1: Examples of taxonomically related and non-taxonomically related neighbors of the _vehicle_ and _bird_ categories determined by the two different types of similarities. Values in parentheses indicate similarities.

 

token distribution which we computed using the minicons library Misra (2022). Using these relative probabilities, we compute the following metrics to analyze LMs' property inheritance behavior:

Taxonomic Sensitivity (TS)We measure the extent to which LMs are sensitive to the taxonomic relations between the premise and conclusion category. We do so by computing the proportion of examples where \(P_{\mathsf{rel}}(\text{Yes})>0.5\) when the premise and conclusion are taxonomically related, and \(P_{\mathsf{rel}}(\text{Yes})<0.5\) when they are not.

Property Sensitivity (PS)The TS metric is computed for stimuli where the only nonce property is _is/are daxable_. To what extent does the specific surface form of the property play a role in LMs' property inheritance behavior? To this end, we generate a different set of stimuli where we randomly replace both instances of _is/are daxable_ with _has/have feps_ for half of the examples. We maintain the even balance between taxonomically and non-taxonomically related premise-conclusion pairs. Given this data, we then compute the taxonomic sensitivity in this setting as before. This metric penalizes bias for a particular property.

Mismatch Sensitivity (MS)We measure the extent to which the LMs are sensitive to whether the properties mentioned in the premise and conclusion match. If they are mismatched, then regardless of the taxonomic relationship between the concepts, the LMs should always prefer 'No' over 'Yes'. To test this, we generate stimuli where we uniformly substitute either the premise or the conclusion with a different property (e.g., _birds are daxable_ vs. _robins have feps_), and compute the proportion of time \(P_{\mathsf{rel}}(\text{Yes})<0.5\).

Spearman correlation with similarity (\(\rho\))Finally, we measure the Spearman correlation between \(P_{\mathsf{rel}}(\text{Yes})\) and the similarity between the premise and conclusion categories, to quantify the role of similarity on LMs' inference behavior.

We compute these metrics separately for both types of similarity. For the three sensitivity metrics (TS, PS, MS), chance sensitivity is 0.5, as they all involve pairwise comparisons. We also measure the average \(P_{\mathsf{rel}}(\text{Yes})\) across four slices of our data based on when the premise and conclusion categories were (i) non-taxonomically related, low similarity (-Max, -Sim), (ii) non-taxonomically related, high similarity (-Max, +Sim), (iii) taxonomically related, low similarity (+Max, -Sim), and (iv) taxonomically related, high similarity (+Max, +Sim).

### Results and Analysis

Table 2 shows the aforementioned metrics for all four LMs, across both similarity types, while Figure 2 shows their average \(P_{\mathsf{rel}}(\text{Yes})\) values across the four different slices of our data. For both similarity types, all four LMs demonstrate substantially high TS and PS values--i.e., they are more likely to extend the novel property when the premise category is a superordinate of the conclusion category. This suggests a non-trivial role of taxonomic category membership in the models' inference behavior. At the same time, the LMs also show positive correlation with categorical similarity, suggesting that their tendency to extend properties from the premise to the conclusion is also sensitive to the similarity between the two. More specifically, they show greater correspondence with similarities derived from SPoSE (\(\rho\in[0.59,0.68]\)) than from the sense embeddings (\(\rho\in[0.26,0.42]\)).

While LMs demonstrate sensitivity to similarity, their tendency to produce 'Yes' over 'No' seems to correspond better with the presence of taxonomic relations. In Figure 2, LMs on average prefer to extend the property from premise to conclusion for high-similarity pairs than for low-similarity ones. However, we see the average \(P_{\mathsf{rel}}(\text{Yes})\) to be largely greater than 0.5 for the slices of our data where the premise is a taxonomic superordinate of the conclusion, regardless of the similar

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{3}{*}{**Model**} & \multicolumn{4}{c}{**Word-Sense**} & \multicolumn{4}{c}{**SPoSE**} \\ \cline{2-7}  & **TS** & **PS** & **MS** & \(\rho\) & **TS** & **PS** & **MS** & \(\rho\) \\ \hline Gemma-2-2B-FT & 0.84 & 0.85 & 0.89 & 0.26 & 0.79 & 0.79 & 0.88 & 0.59 \\ Mistral-7B-Instruct-v0.2 & 0.84 & 0.85 & 0.84 & **0.42** & 0.80 & 0.81 & 0.83 & 0.62 \\ Llama-3-8B-Instruct & 0.86 & 0.86 & **0.98** & 0.37 & 0.79 & 0.79 & **0.98** & **0.68** \\ Gemma-2-9B-TT & **0.90** & **0.89** & 0.85 & 0.34 & **0.85** & **0.84** & 0.84 & 0.65 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Behavioral sensitivities of LMs across both similarity types (Word-Sense and SPoSE). TS: Taxonomic Sensitivity; PS: Property Sensitivity; MS: Mismatch Sensitivity; \(\rho\): Spearman correlation of model scores (\(P_{\mathsf{rel}}(\text{Yes})\)) with similarity (\(p<.001\) throughout).

 

ity bin (high vs. low), for both types of similarity. For non-taxonomically related premise conclusion pairs, while the LMs show greater \(P_{\mathsf{re1}}(\text{Yes})\) value when the premise and conclusion are highly similar, it is seldom above 0.5 (with the exception of Llama-3-8B-Instruct, for the SPoSE similarity). Finally, all LMs show robustness to cases where the property being projected between the premise and conclusions was different (largely high MS values). That is, whatever taxonomic/similarity sensitivities LMs demonstrate are largely for the well-formed instances of the task (i.e., when the properties between premise and conclusion match). These results suggest that **LMs' property inheritance behavior can be explained jointly by taxonomic relations as well as categorical similarity**.

## 4 Causal Investigation of Taxonomy and Similarity

Given that LMs perform property inheritance with high taxonomic sensitivity, we now use causal interpretability tools Mueller et al. (2024); Geiger et al. (2024) to localize this behavior. We then investigate (i) whether the subspaces that are causally responsible for inheritance perform inductive generalization in a manner that is better explained by the similarity of the noun concepts, rather than their taxonomic relationship; and (ii) whether taxonomic information and noun similarity are fundamentally entangled in the model representations. We investigate both questions using **distributed alignment search**DASGeiger et al. (2024), a causal interpretability method which has been used to localize and mechanistically explain syntax-sensitive behaviors Arora et al. (2024) and arithmetic abilities Wu et al. (2023) in LMs. To avoid needing to search over all possible subspaces of an activation vector manually, we employ the Boundless DAS variant Wu et al. (2023) implemented in pyvene Wu et al. (2024) to learn the appropriate subspace.

We use DAS rather than other causal interpretability techniques7 because it allows us to investigate to what extent a specific hypothesized _causal model_ is being implemented by the neural network. Because DAS isolates a specific subspace corresponding to a given hypothesized causal variable, the subspace can be evaluated in novel contexts post hoc to better characterize its functional role. We leverage this to evaluate whether subspaces that are sensitive to _taxonomic_ relationships are also sensitive to _similarity-based but non-taxonomic_ relationships. This type of experimental design involving ambiguous training sets with separate test and generalization sets has been used to characterize the inductive biases of LMs McCoy et al. (2019); Si et al. (2023); Mueller et al. (2024), though this has typically been evaluated behaviourally rather than at the level of internal mechanisms. There is a limited but growing literature validating how well discovered mechanisms (or explanations thereof) generalize to novel types of contexts Geiger et al. (2020); Huang et al. (2023, 2024), _i.a._).

Footnote 7: For example, those based on sparse autoencoders Bricken et al. (2023); Marks et al. (2024); Huben et al. (2024) or neurons Vig et al. (2020); Finlayson et al. (2021).

Given an LM \(M\), a hypothesized causal graph \(L\), and counterfactual input (source, base) pairs \((s,b)\) that target a specific intermediate variable \(V\) in the causal graph, DAS learns a rotation \(R\) for the activations of a given submodule, and then performs counterfactual interventions in this rotated space. This is done as follows: \(L_{V}(s)\), the value of variable \(V\) in \(L\) when applied to source \(s\), is patched into the corresponding variable of the base \(b\),

\[L_{V}(b)\gets L_{V}(s),\]

Figure 2: LMs’ Average Relative Probability of ‘Yes’ for different conclusion categories and different types of similarities (Word-Sense vs SPoSE). LMs show a clear sensitivity to taxonomic relations, but also show an effect of similarity, where they are more likely to extend the property to a conclusion category when the premise and the conclusion categories are highly similar. Chance behavior is 0.50, as indicated by the dashed line.

 

and then the final output of \(L\) is computed. Translating this to operations on the low-level computation graph, a rotation \(R\) is learned for activations in the network \(M\) from a position \(i\) to position \(j\). We patch \(R(M(s)_{i})\) into \(M(b)_{j}\):

\[M(b)_{j}\gets R\big{(}M(s)_{i}\big{)},\]

and then compute the output of \(M\).

After learning the rotation, we compute the interchange intervention accuracy (IIA) on a held-out test set. This scores how frequently the learned subspace performs as expected given the hypothesized causal graph. Intuitively, IIA is defined as the proportion of \((s,b)\) pairs where patching in the rotated source activation into the base model \(M\) causes \(M\)'s output to match the causal model \(L\)'s output _after_ the intervention. It thus measures the degree of alignment between the neural network and the causal graph. The intuition for learning a rotation is that the features of interest may not necessarily be aligned to the bases of the original activation space Hinton et al. (1986); Smolensky (1986); Elhage et al. (2022); Mueller et al. (2024), rendering neuron-based approaches insufficient for capturing the target variable. We refer readers to Geiger et al. (2024) for details.

In our experiments, we hypothesize that a given subspace will be sensitive to taxonomic relationships--if this is true, then passing an activation into this subspace from taxonomically unrelated inputs should yield a variable output of False. However, if we intervene on this subspace passing in activations from an alternate input where the nouns _are_ taxonomically related (while holding all other activations in the model constant), then the variable's output should flip to True. We illustrate the intuition and the causal graph that we hypothesize LMs implement in Figure 3. By only changing the premise and conclusion nouns across inputs (e.g., _birds_, _robins_, _orchids_), and keeping the rest of input constant, we effectively target the node "B is a kind of A" in Figure 3.8

Footnote 8: We target this node since our focus is on taxonomic relations rather than property binding—where the goal is to check if LMs have appropriately bound the property with the noun concept—which has been investigated in other work Feng and Steinhardt (2024).

We train DAS on a subset of 3,000 stimuli and evaluate on the rest. Implementation and training details are given in Appendix D. Next, we describe variations of the training and test sets used in our experiments.

BalancedFor each _base_ stimulus in the set of 3,000 stimuli, we pick a _source_ counterfactual stimulus by sampling from this set of stimuli without replacement. This ensures a balanced coverage of counterfactual stimuli in terms of similarity and whether or not the premise and conclusion are taxonomically related. This experimental setting is used to localize property inheritance in the network.

ControlDAS requires training, raising concerns that its expressivity may lead to discovered causal effects where none exist Arora et al. (2024), similar to concerns raised in the probing literature Hewitt and Liang (2019). To mitigate this, we follow Arora et al. (2024) and map the labels Yes and No to the semantically irrelevant words chart and view, respectively. This setting evaluates the extent to which DAS memorizes an arbitrary token mapping versus learning task-specific structure.

Figure 3: We hypothesize that language models rely on this causal graph to perform the categorical inference task. We are interested specifically in the node responsible for taxonomic judgments, so we use DAS (§4) to isolate the subspace encoding this causal variable. We causally verify its sensitivity to taxonomy by setting its value to what it would have been on counterfactual _source_ inputs, and observing whether model behavior changes appropriately. Here, activations at the isolated subspace for the _base_ are replaced with activations from the _source_. IIA measures to what extent this intervention results in the expected behavior given the hypothesized causal graph across inputs.

 

AmbiguousIn this setting, we filter the training set such that learned rotations can be consistent with _either_ similarity or taxonomic features. By only training on taxonomic examples with high similarity and negative examples with low similarity, we learn an intervention which is ambiguous with respect to what features it is using. To disambiguate these, we evaluate on two different test sets: an in-domain (**Ambiguous-Test**) test set composed in the same way as the train set (taxonomic examples with high similarity and negative examples with low similarity), and an out-of-domain generalization test set (**Ambiguous-Gen**), composed of taxonomic examples with low similarity and non-taxonomic examples with high similarity. A drop in IIA values between the Test and Generalization settings would show that learned interventions rely more on similarity than taxonomic features.

UnambiguonsFinally, we test whether the subspace sensitive to taxonomic relationships is also sensitive to similarity-based relationships--i.e., whether taxonomic information and noun similarity are entangled in the model. To do this, we take the intervention trained under the **Balanced** setting and evaluate it on (**Ambiguous-Gen**), defined above. Here the training signal for DAS is unambigious-it is only compatible with capturing taxonomic information. A drop in scores would then indicate that taxonomic and similarity features are entangled.

We use \(P_{\text{rel}}\)(Yes) rather than the top predicted token as done in Wu et al. (2023b) when computing IIA across experiments, in order to have a reasonable comparison against the control setting.9

Footnote 9: The interventions are not strong enough to force the model to output either of the control tokens chart, or view as their top-predicted token, resulting in 0% accuracy; on the other hand the difference between using the top predicted token and \(P_{\text{rel}}\)(Yes) was negligible in most other settings (<0.02 across experiments, except for Gamma-2-2B-IT).

### Results and Analysis

Figure 4 shows the IIA results for Gemma-2-9B-IT and Mistral-7B-Instruct-v0.2 for the first three settings, and Table 3 shows results on the **Unambiguous** setting. Full results for the other models are given in Appendix B. Higher IIA values indicate a better causal alignment between the intervention at a particular location in the network and the causal graph. The activations at the premise token positions have poor alignment with the causal graph, while activations around the last conclusion token at layer 10 (for Mistral) and layers 10-20 (for Gemma) have a much bigger causal effect on property inheritance.

IIA values for the control experiment are always lower than for the balanced setting; thus, despite its expressivity, DAS can yield task-dependent results (i.e., comparatively lower IIA values in the control setting establishes that DAS is telling us something about the LM's property inheritance abilities). High values for IIA occur in roughly the same positions for Ambiguous-Test and Ambiguous-Gen. There is a substantial drop from Ambigious-Test to Ambigious-Gen for all models and for every in

Figure 4: Interchange Intervention Accuracies (IIA) for the causal graph in Figure 3 for **Gamma-2-9B-IT** (Top) and **Mistral-7B-Instruct-0.2** (Bottom), when intervening at various layers and token positions, with negative samples derived using SPoSE similarities. Since the premise and conclusion nouns are often multiple tokens, we show IIA when intervening at the first and last positions of each. **Note:** Both models have different numbers of layers.

 

tervention position, showing that, **when given ambiguous data, the learned rotations tend to rely on similarity rather than taxonomic relations.** Similar observations hold for the Unambigious experiment, with a drop in scores showing that, even when trained on data that is unambigiously consistent with taxonomic relations rather than similarity, the causal model is sensitive to similarity effects. In other words, **similarity and taxonomic features appear to be fundamentally entangled**.

## 5 Discussion and Conclusions

Our goal in this paper was to characterize category-based inferences in LMs. We derived inspiration from psychological experiments that have sought to understand how humans transfer properties from a higher level category like _mammals_ to lower level ones such _dogs_ and _cats_(Collins and Quilian, 1969; Glass and Holyoak, 1974; Murphy, 2002)--i.e., show _property inheritance_. Findings from these experiments show that instead of relying purely on abstract category-membership principles (where all members of a category are equally likely to inherit properties), humans show graded behavior proportional to the similarity between the categories (Sloman, 1998).

Through behavioral and representational analyses methods across four LMs, we found that **language models' property inheritance behavior is sensitive to both taxonomic relations _and_ similarity**. Behaviorally, LMs were more likely to extend the property from the premise to the conclusion when they were taxonomically related, and at the same time, showed positive correlations with the similarity between them. This finding was further reinforced in our representational investigations, where **we found taxonomy and similarity to be largely entangled in the LM subspaces causally responsible for property inheritance.** Our results cast doubt on previous works' assumption of pure taxonomic principles with respect to property inheritance in LMs, without considering similarity (Misra et al., 2023; Powell et al., 2024; Wang et al., 2024). In the broader context of LMs' reasoning behaviors, our results showcase yet another instance where LMs were susceptible to human-like content effects (Lampinen et al., 2024).

Our findings also suggest interesting hypotheses for future human analyses. Importantly, Sloman (1998)'s experiments never tested humans on cases where the categories in a property inheritance setting did _not_ share taxonomic relations (e.g., birds and bats). This makes it unclear if the principles of taxonomy and similarity are _also_ not mutually exclusive for humans (Murphy et al., 2012)--e.g., it could be possible that both mechanisms are at play, where taxonomic relations demarcate the inference (i.e., whether the property will be inherited) and similarity modulates its strength. Future experiments could test this hypothesis and shed further light on the extent to which LMs and humans show similar sensitivities and biases during reasoning. Overall, our work shows how tools from modern interpretability research, primarily used to isolate sub-structures in neural networks, may also be used to perform hypothesis testing of the abstractions encoded within them.

### Limitations

Alternative similaritiesThere are other similarities that we could have used in our experiments. Examples include similarities computed over WordNet (e.g., Wu-Palmer similarity; Wu and Palmer, 1994), co-occurrence based similarities, or similarities computed from LM token embeddings or activations (Chronis and Erk, 2020). While it is tempting to use these, there are various shortcomings to each of them. WordNet similarities do not distinguish between co-hyponyms: concepts that are at the same level will have the same similarity with a given high level concept. Next, the sense based similarity used in this work _is_ a form of distributional similarity as it is formed by aggregating distributional semantic embeddings of words occurring in free-text corpora, and has the added advantage of being informed by the sense of the word. Finally, using LM-specific similarities makes it difficult to use the same stimuli across models, since similarities from different models would have resulted in different negative samples. However,

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**Word-Sense**} & \multicolumn{2}{c}{**SPoSE**} \\ \cline{2-5} Model & **Bal** & **Gen** & **Bal** & **Gen** \\ \hline Gemma-2-2B-IT & 0.86 & 0.81 & 0.80 & 0.63 \\ Mistral-7B-Instruct-v0.2 & 0.88 & 0.85 & 0.83 & 0.65 \\ Llama-3-8B-Instruct & 0.87 & 0.82 & 0.80 & 0.63 \\ Gemma-2-9B-IT & 0.90 & 0.85 & 0.85 & 0.67 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Maximum IIA values for the Unambiguous experiment. Interventions are trained under the Balanced train set, and evaluated on both the Balanced test set (Bal) as well as the Ambiguous-Gen set (Gen). The drop on the generalization test set suggests that the learned interventions rely non-trivially on similarity.

 

future work could explore possible correlations between LM-derived similarities and their property inheritance behavior.

Abstract and Ad-hoc ConceptsWe have exclusively focused on concrete, object noun concepts in this work. However, these do not account for the entire range of concepts that humans acquire in their life times--i.e., it is unclear how property inheritance applies to abstract concepts like _love_ or ad-hoc concepts Barsalou (1983) like _things you would pack for a camping trip_, which may not have taxonomic structure. We leave these investigations to future work.

Knowledge editingPrevious work has studied whether language models are able to carry out inferences related to entities Cohen et al. (2024) or concepts Wang et al. (2024); Powell et al. (2024) which have been modified through knowledge editing. While we do not engage directly with knowledge edits, it would be interesting to measure the impact of knowledge editing on the LMs' rotated subspaces during property inheritance.

Broader space of inductive problemsWhile property inheritance is an important consequence of category structure and organization, it is only one type of inductive generalization that humans exhibit Kemp and Jern (2014). For instance, humans may also learn about novel concepts and ascribe known features to them Smith and Estes (1978); Murphy (2002); Kemp (2011), or they might learn a new higher level category and judge category memberships of known concepts Xu and Tenenbaum (2007), or they might generalize in a completely non-deductive manner (e.g., from robins to birds, or to its co-hyponyms, see Rips (1975); Osherson et al. (1990).

Contextual similaritiesWhile we experimented with different types of similarities, human inductive inferences might not be explained by the same similarity every time--i.e., similarity in inductive problems can be context-sensitive Heit and Rubinstein (1994); Rogers and McClelland (2004); Kemp and Tenenbaum (2009). For instance, generalization of biological-sounding properties (_has an ulnar artery_) are often sensitive to taxonomic similarities. Bears and whales are likely to share these properties since they are mammals. On the other hand, behavioral properties (_studies its food before attacking_) might be influenced by similarities derived from other, non-taxonomic conceptual structures. The aforementioned property could be shared by predatory animals such as tigers and eagles. It is, however, important to note that these caveats apply only when the properties are not _nonce_, like in our work--e.g., it is not clear if _is daxable_ or _has feps_ are biological or otherwise. Future studies could explore these deeper nuances of similarity and category-based inferences in LMs using the methods we employ in this work.

## Acknowledgments

A.M. is supported by a postdoctoral fellowship under the Zuckerman STEM Leadership program. K.M. acknowledges postdoctoral funding supported by NSF Grant 2139005 awarded to Kyle Mahowald. We also thank Martin Hebart for help on the THINGS dataset and the SPoSE embeddings, Atticus Geiger and Zhengxuan Wu for assistance with pyvene, and Katrin Erk and the anonymous reviewers for helpful discussion and feedback.

## References

* Abdou et al. (2021) Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Sogaard. 2021. Can language models encode perceptual structure without grounding? A case study in color. In _Proceedings of the 25th Conference on Computational Natural Language Learning_, pages 109-132, Online. Association for Computational Linguistics.
* Arora et al. (2024) Eryaman Arora, Dan Jurafsky, and Christopher Potts. 2024. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14638-14663, Bangkok, Thailand. Association for Computational Linguistics.
* Barsalou (1983) Lawrence W. Barsalou. 1983. Ad hoc categories. _Memory & cognition_, 11(3):211-227.
* Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5185-5198, Online. Association for Computational Linguistics.
* Bhatia (2023) Sudeep Bhatia. 2023. Inductive reasoning in minds and machines. _Psychological Review_.
* Bhatia and Richie (2021) Sudeep Bhatia and Russell Richie. 2021. Transformer networks of human concept knowledge. _Psychological Review_.
* Brucken et al. (2015) Trenton Brucken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askel, 

Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Braydem McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. _Transformer Circuits Thread_.
* Chronis and Erk (2020) Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? when it's like a rabbit! multi-prototype BERT embeddings for estimating semantic relationships. In _Proceedings of the 24th Conference on Computational Natural Language Learning_, pages 227-244, Online. Association for Computational Linguistics.
* Cohen et al. (2024) Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evaluating the ripple effects of knowledge editing in language models. _Transactions of the Association for Computational Linguistics_, 12:283-298.
* Collins and Quillian (1969) Allan M Collins and M Ross Quillian. 1969. Retrieval time from semantic memory. _Journal of verbal learning and verbal behavior_, 8(2):240-247.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Scheletin, Amy Yang, Angela Fan, et al. 2024. The llama 3 Herd of Models. _arXiv preprint arXiv:2407.27178_.
* Elhage et al. (2022) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. _Preprint_, arXiv:2209.10652.
* Evans (1989) Jonathan St BT Evans. 1989. _Bias in human reasoning: Causes and consequences_. Lawrence Erlbaum Associates, Inc.
* Feng et al. (2023) Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. 2023. Towards revealing the mystery behind chain of thought: A theoretical perspective. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Feng and Steinhardt (2024) Jiahai Feng and Jacob Steinhardt. 2024. How do language models bind entities in context? In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Finlayson et al. (2021) Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1828-1843, Online. Association for Computational Linguistics.
* Geiger et al. (2024) Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Izard. 2024a. Causal abstraction: A theoretical foundation for mechanistic interpretability. _Preprint_, arXiv:2301.04709.
* Geiger et al. (2020) Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 163-173, Online. Association for Computational Linguistics.
* Geiger et al. (2024b) Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Izard, and Noah D. Goodman. 2024b. Finding alignments between interpretable causal variables and distributed neural representations. In _Causal Learning and Reasoning, 1-3 April 2024, Los Angeles, California, USA_, volume 236 of _Proceedings of Machine Learning Research_, pages 160-187. PMLR.
* Glass and Holyoak (1974) Arnold L Glass and Keith J Holyoak. 1974. Alternative conceptions of semantic theory. _Cognition_, 3(4):313-339.
* Grand et al. (2022) Gabriel Grand, Idan Asher Blank, Francisco Pereira, and Evelina Fedorenko. 2022. Semantic projection recovers rich human knowledge of multiple object features from word embeddings. _Nature Human Behaviour_, pages 1-13.
* Han et al. (2022) Simon Jerome Han, Keith Ransom, Andrew Perfors, and Charles Kemp. 2022. Human-like property induction is a challenge for large language models. In _Proceedings of the 44th Annual Conference of the Cognitive Science Society_.
* Han et al. (2024) Simon Jerome Han, Keith J Ransom, Andrew Perfors, and Charles Kemp. 2024. Inductive reasoning in humans and large language models. _Cognitive Systems Research_, 83:101155.
* Hayes and Heit (2018) Brett K Hayes and Evan Heit. 2018. Inductive reasoning 2.0. _Wiley Interdisciplinary Reviews: Cognitive Science_, 9(3):e1459.
* Hebart et al. (2023) Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. 2023. Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. _eLife_, 12:e82580.
* Hebart et al. (2019) Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris I Baker. 2019. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. _PloS one_, 14(10):e0223792.
* Heit and Rubinstein (1994) Evan Heit and Joshua Rubinstein. 1994. Similarity and property effects in inductive reasoning. _Journal of Experimental Psychology: Learning, Memory, and Cognition_, 20(2):411 

John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
* Hinton et al. (1986) G. E. Hinton, J. L. McClelland, and D. E. Rumelhart. 1986. _Distributed representations_, page 77-109. MIT Press, Cambridge, MA, USA.
* Huang et al. (2023) Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. 2023. Rigorously assessing natural language explanations of neurons. In _Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP_, pages 317-331, Singapore. Association for Computational Linguistics.
* Huang et al. (2024) Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024. RAVEL: Evaluating interpretability methods on disentangling language model representations. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8669-8687, Bangkok, Thailand. Association for Computational Linguistics.
* Huben et al. (2024) Robert Huben, Hoagy Cunningham, Logan Riggs, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. _arXiv:2310.06825_.
* Kaniuth et al. (2024) Philipp Kaniuth, Florian P Mahner, Jonas Perkuhn, and Martin N Heburt. 2024. A high-throughput approach for the efficient prediction of perceived similarity of natural objects. _bioRxiv_, pages 2024-06.
* Kemp (2011) Charles Kemp. 2011. Inductive reasoning about chimeric creatures. _Advances in Neural Information Processing Systems_, 24:316-324.
* Kemp and Jern (2014) Charles Kemp and Alan Jern. 2014. A Taxonomy of Inductive Problems. _Psychonomic Bulletin & Review_, 21(1):23-46.
* Kemp and Tenenbaum (2009) Charles Kemp and Joshua B Tenenbaum. 2009. Structured Statistical Models of Inductive Reasoning. _Psychological Review_, 116(1):20.
* Lake and Murphy (2021) Brenden M Lake and Gregory L Murphy. 2021. Word meaning in minds and machines. _Psychological Review_.
* Lampinen et al. (2024) Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2024. Language models, like humans, show content effects on reasoning tasks. _PNAS neus_, 3(7).
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In _International Conference on Learning Representations_.
* Loureiro et al. (2022) Daniel Loureiro, Alipio Mario Jorge, and Jose Camacho-Collados. 2022. Imms reloaded: Transformer-based sense embeddings for disambiguation and beyond. _Artificial Intelligence_, 305:103661.
* Lupyan and Lewis (2019) Gary Lupyan and Molly Lewis. 2019. From words-as-mappings to words-as-cues: The role of language in semantic knowledge. _Language, Cognition and Neuroscience_, 34(10):1319-1337.
* Marks et al. (2024) Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. _CoRR_, abs/2403.19647.
* McClelland et al. (2010) James L McClelland, Matthew M Botvinick, David C Noelle, David C Plaut, Timothy T Rogers, Mark S Seidenberg, and Linda B Smith. 2010. Letting structure emerge: connectionist and dynamical systems approaches to cognition. _Trends in Cognitive Sciences_, 14(8):348-356.
* McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
* Misra (2022) Kanishka Misra. 2022. minicons: Enabling flexible behavioral and representational analyses of transformer language models. _arXiv:2203.13112_.
* Misra et al. (2024) Kanishka Misra, Allyson Ettinger, and Kyle Mahowald. 2024. Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently. _EMNLP 2024_.
* Misra et al. (2021) Kanishka Misra, Allyson Ettinger, and Julia Rayz. 2021. Do language models learn typicality judgments from text? In _Proceedings of the 43rd Annual Conference of the Cognitive Science Society_.
* Misra et al. (2023) Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. COMPPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2928-2949, Dubrovnik, Croatia. Association for Computational Linguistics.
* Misra et al. (2022) Kanishka Misra, Julia Taylor Rayz, and Allysson Ettinger. 2022. A property induction framework for neural language models. In _Proceedings of the 44th Annual Conference of the Cognitive Science Society_.

 

Aaron Mueller, Jannik Brinkmann, Millicent L. Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarasayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. 2024a. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. _CoRR_, abs/2408.01416.
* Mueller et al. (2024b) Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. 2024b. In-context learning generalizes, but not always robustly: The case of syntax. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4761-4779, Mexico City, Mexico. Association for Computational Linguistics.
* Murphy (2002) Gregory L Murphy. 2002. _The Big Book of Concepts_. MIT press.
* Murphy et al. (2012) Gregory L Murphy, James A Hampton, and Goran S Milovanovic. 2012. Semantic memory redux: An experimental test of hierarchical category representation. _Journal of Memory and Language_, 67(4):521-539.
* Nanda et al. (2023) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. In _The Eleventh International Conference on Learning Representations_.
* Osherson et al. (1990) Daniel N Osherson, Edward E Smith, Ormond Wilkie, Alejandro Lopez, and Eldar Shafir. 1990. Category-based Induction. _Psychological Review_, 97(2):185.
* Park et al. (2024) Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2024. The geometry of categorical and hierarchical concepts in large language models. _CoRR_, abs/2406.01506.
* Patel and Pavlick (2022) Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In _International Conference on Learning Representations_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Piantadosi and Hill (2022) Steven T Piantadosi and Felix Hill. 2022. Meaning without reference in large language models. _arXiv:2208.02957_.
* Powell et al. (2024) Derek Powell, Walter Gerych, and Thomas Hartvigsen. 2024. TAXI: Evaluating categorical knowledge editing for language models. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 15343-15352, Bangskok, Thailand and virtual meeting. Association for Computational Linguistics.
* Rips (1975) Lance J Rips. 1975. Inductive judgments about natural categories. _Journal of Verbal Learning and Verbal Behavior_, 14(6):665-681.
* Riviere et al. (2012) Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesward, Bobak Shahriari, Alexandre Ramc, Johan Ferret, Piet Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliakesei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coonen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danilia Sinopalkov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Eric Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klinczak-Plucinska, Harlean Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Iping Zhou, Joan Carrasqueira,Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersoort, Josh Gordon, Josh Lipschultz, Josh Newman, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kirambir Sodhia, Kish Greene, Lars Lowe Sjosund, Lauren Usui, Laurent Sifre, Lena Heureman, Leticia Lago, and Lilly McCasulas. 2024. Germna 2: Improving open language models at a practical size. _CoRR_, abs/2408.00118.
* Rogers and McClelland (2004) Timothy T Rogers and James L McClelland. 2004. _Semantic Cognition: A Parallel Distributed Processing Approach_. MIT press.
* Si et al. (2023) Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Daniqi Chen, and He He. 2023. Measuring inductive biases of in-context learning with underspecified demonstrations. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11289-11310, Toronto, Canada. Association for Computational Linguistics.
* Sloman (1993) Steven A Sloman. 1993. Feature-based induction. _Cognitive psychology_, 25(2):231-280.
* Sloman (1998) Steven A Sloman. 1998. Categorical inference is not a tree: The myth of inheritance hierarchies. _Cognitive Psychology_, 35(1):1-33.
* Smith and Estes (1978) Edward E Smith and William K Estes. 1978. Theories of semantic memory. _Handbook of learning and cognitive processes_, 6:1-56.
* Smolensky (1986) P. Smolensky. 1986. _Neural and conceptual interpretation of PDP models_, page 390-431. MIT Press, Cambridge, MA, USA.
* Smolensky (1988) Paul Smolensky. 1988. On the proper treatment of connectionism. _Behavioral and brain sciences_, 11(1):1-23.

 

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Wang et al. (2024) Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. 2024. Editing conceptual knowledge for large language models. _CoRR_, abs/2403.06259.
* Wason (1968) Peter C Wason. 1968. Reasoning about a rule. _Quarterly journal of experimental psychology_, 20(3):273-281.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, Joe Davison, Sam Sheifer, Patrick von Platen, Clara Ma, Yicine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu et al. (2023a) Weiqi Wu, Chengyue Jiang, Yong Jiang, Pengjun Xie, and Kewei Tu. 2023a. Do PLMs know and understand ontological knowledge? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3080-3101, Toronto, Canada. Association for Computational Linguistics.
* Wu et al. (2024) Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, and Christopher Potts. 2024. pyvene: A library for understanding and improving PyTorch models via interventions. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations)_, pages 158-165, Mexico City, Mexico. Association for Computational Linguistics.
* 16, 2023_.
* Wu and Palmer (1994) Zhibiao Wu and Martha Palmer. 1994. Verb Semantics and Lexical Selection. In _32nd Annual Meeting of the Association for Computational Linguistics_, pages 133-138, Las Cruces, New Mexico, USA. Association for Computational Linguistics.
* Xu and Tenenbaum (2007) Fei Xu and Joshua B Tenenbaum. 2007. Word learning as Bayesian inference. _Psychological Review_, 114(2):245.
* Zheng et al. (2019) Charles Y. Zheng, Francisco Pereira, Chris I. Baker, and Martin N. Heburt. 2019. Revealing interpretable object representations from human behavior. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.

## Appendix A Sensitivity to directionality

Our previous analyses suggest that LMs demonstrate non-trivial taxonomic sensitivity in their property inheritance behavior. Are they also sensitive to the fact that the taxonomic relation is, in its strictest sense, asymmetrical Murphy (2002)? For instance, while _robins_ are _birds_, one cannot say that _birds_ are _robins_. Is this fact captured in LMs' property projections? That is, if _robins_ are daxable, are _birds_ daxable? A finding of insensitivity might not necessarily be a negative result in the context of LMs' conceptual representations. After all, generalizing from subordinates (_robins_) to superordinates (_birds_) is directly connected to contemporary work targeting property induction in LMs Misra et al. (2022); Han et al. (2022); Bhatia (2023); Han et al. (2024), where similarity has more of an uncontroversial role Osherson et al. (1990); Sloman (1993), and where humans show a myriad of interesting phenomena Hayes and Heit (2018).10 Instead, it might be informative to further characterize the LMs' subspaces responsible for property inheritance--our previous findings suggest that they are sensitive to similarity, are they also sensitive to directionality? Or are they simply encoding the presence of taxonomic (and similarity) relations?

Footnote 10: We refer to this analysis as the _directionality_ of property projection given that _induction_ is often used more generally to refer to many kinds of reasoning which are _ampliative_, and not determined by deductive logic Kemp and Jern (2014).

We behaviorally evaluate the sensitivity of LMs to the direction of the property projection as follows. We measure the **Directional Sensitivity** (DS) as the fraction of times an LM flips its relative preference from Yes to No when the order of the premise and conclusion nouns in our stimuli is swapped, _only_ over all taxonomically related items. Additionally, we also measure the spearman correlation of DS with that of TS--i.e., the taxonomic sensitivity metric computed for the non-reversed property inheritance stimuli, computed for the tax 

onomically related premise and conclusion. We refer to this as \(\rho_{\text{DS}}\)

Next we consider whether the learned DAS interventions from SS4 generalize when reversing the direction of the premises and conclusions. We consider interventions trained under the **Balanced** setting with high IIA values (i.e., which localized property projection behavior in the regular property inheritance direction and showed taxonomic sensitivity), and evaluate them on source and base pairs with flipped premise and conclusion nouns. In order to measure the effect of direction, we only evaluate on examples where the intervention successfully flipped the label from No to Yes in the non-reversed direction, as illustrated in Figure 5. The evaluation set is constructed from source and base pairs where the premise and conclusions are taxonomic and non-taxonomic, respectively. We then filtered these so that (i) the LM predicts the right labels in the non-reversed case (i.e., shows taxonomic sensitivity) and (ii) the DAS intervention correctly flips the label from No to Yes. The order of the nouns are then reversed to establish whether the intervention causes the same behavior when the direction of inheritance is flipped. This is illustrated in Figure 5.

We refer to the IIA values for this evaluation as **Subspace Directional Nsensitivity** (SDI), since higher scores mean the subspace is _less_ sensitive to direction.

ResultsThe behavioral DS results across models are given in Table 4. Gemma-2-9B-IT is the most directionally sensitive model: the order of the premise and conclusion concepts has an impact on property projection roughly half of the time. The other three models are far less sensitive to direction. This is also reflected in higher Spearman correlations for those models.

Subspace Directional Nensitivity results are shown in Table 5 for interventions at different network locations. Since IIA values were high at both the final token position (Last) and the last token of the conclusion noun (Con-\(\ell\)), we evaluated with interventions at both, in each case selecting the layer with highest IIA across each token position.11

Footnote 11: For Gemma-2-9B-IT, this was layer 15 for Con-\(\ell\), and layer 35 for Last; for the other models this was layer 10 for Cont-\(\ell\) and layer 15 for Last.

SDI is nearly always lower for Con-\(\ell\) than for Last positions: it is harder to use the taxonomically sensitive property inheritance intervention to make the LM behave in the same way with reversed examples when intervening earlier in the network. Similar to the behavioral results, the subspace at conclusion-last token position and layer 15 for Gemma-2-9B-IT is the most sensitive to the directionality of the inference, while Gemma-2-2B-IT is the least sensitive. We note, however, that this only holds for the prompts used in our experiments--for example, using _Prompt 2_ with chat template (Appendix C) makes Gemma-2-2B-IT more directionally sensitive than it is with _Prompt 1_ without chat template.12

Figure 5: DAS interventions on pairs where the direction of the inference is flipped. Low IIA values for this experiment reveal whether the learned subspaces for property inheritance are sensitive to direction.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **DS** & \(\rho_{\text{DS}}\) \\ \hline Gemma-2-2B-IT & 0.24 & **0.54** \\ Mistral-7B-Instruct-v0.2 & 0.37 & 0.37 \\ Llama-3-8B-Instruct & 0.38 & 0.40 \\ Gemma-2-9B-IT & **0.49** & 0.31 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Directional sensitivities of LMs. DS: Directional Sensitivity; \(\rho\): Spearman correlation of \(P_{\text{rel}}\)(Yes) between the original and reverse directions. DS and \(\rho\) are computed only over the taxonomic items.

 

Additional DAS Results

Additional results for the DAS experiments for Gemma-2-2B-IT and Llama-3-8B-Instruct are given in Figure 6. This figure also includes the full set of results from using the sense-based similarity measure to derive our negative samples. The full set of results from the **Unambiguous** DAS experiments are shown in Figure 7.

## Appendix C Prompts

The formats for the prompts used in our experiments are given below.

```
prompt1:Answerthequestion.GiventhatAis/aredaxable,isttruethatBsis/aredaxable?AnswerwithYes/No.\nfrom{}2:Answerthequestion.GiventhatAis/aredaxable,isttruethatBsis/aredaxable?AnswerwithYes/No.Theansweris:Prompt3:Answerthequestion.GiventhatAis/aredaxable,isttruethatBsis/aredaxable,isttruethatBsis/aredaxable?AnswerwithYes/No.\textless{}\)
```

Table 6 indicates which the prompts used for each language model and whether the chat template14 was used. These selections were made on the basis of the models' behavioral taxonomic sensitivities.

Footnote 14: [https://huggingface.co/docs/transformers/main/en/chat_templating](https://huggingface.co/docs/transformers/main/en/chat_templating)

## Appendix D DAS Implementation Details

Boundless DAS interventions were trained for 2 epochs, with a batch size of 16 on NVIDIA RTX A6000 and NVIDIA A40 GPUs. We used the Adam optimizer with a learning rate of 1e-3, including gradient accumulation and a temperature schedule.15 All DAS experiments were performed using the pyvene library, version 0.1.1. We loaded all models using the default huggingface configuration, except for Gemma-2-9B-IT, which we loaded with torch_dtype torch.bfloat16 due to memory constraints. For the **Balanced**, **Unaphigous** and **Control** experiments, we trained on 3,000 stimuli and evaluated on 1,018 stimuli. For the **Unambigious** setting we trained on 1,527 stimuli for the Word-Sense dataset and 2,017 stimuli for the SPoSE dataset. The **Ambiguous-Test** set contained 536 stimuli for Word-Sense and 671 for SPoSE, while the **Ambiguous-Gen** test set containd 482 stimuli for Word-Sense and 347 for SPoSE.

Footnote 15: As used in [https://github.com/stanfordnlp/pyvene/blob/main/tutorials/advanced_tutorials/Boundless_DAS.ipynb](https://github.com/stanfordnlp/pyvene/blob/main/tutorials/advanced_tutorials/Boundless_DAS.ipynb)

## Appendix E Premise and Conclusion Concepts

The list of all premise categories used in our experiments is shown in Table 9. Tables 7 and 8 show examples from different slices of our data using both SPoSE and sense-based embeddings.

 

Figure 6: Interchange Intervention Accuracy (IIA) for the causal graph in Figure 3, for all LMs.

 



## References

* [1] A. B. Krizhevsky, S. M. Gershtein, and R. L. Krause, "Theoretical evaluation of the performance of a new machine learning approach to machine learning," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 209-213, 2014.
* [2] D. J. Bishop, C. V. Oliva, and P. I. Jordan, "A practical guide for machine learning," _IEEE Transactions on Neural Networks_**58**, no. 6, pp. 77-82, 1994.
* [3] T. H. Cover, N. J. Freeman, and E. W. Tucker, "On the number of hidden layers," _Advances in neural information processing systems_**10**, no. 1, pp. 1-10, 1970.
* [4] Y. Bengio, F. Chollet, J. Erhan, and X. Zisserman, "Batch normalization: Accelerating deep networks with applications to image recognition," _arXiv preprint arXiv:1512.01581_**, 2015.
* [5] Y. Bengio, J. Salomon, and O. Vitells, "Deep learning by reducing the dimensionality of data from tiny images," _Journal of Machine Learning Research_**10**, no. 1, pp. 1-10, 2015.

[MISSING_PAGE_POST]





\begin{table}
\begin{tabular}{l l} \hline
**premise category** & \(N\) \\ \hline food & 291 \\ animal & 177 \\ tool & 142 \\ clothing & 107 \\ container & 105 \\ mammal & 88 \\ electronic device & 74 \\ vehicle & 70 \\ weapon & 48 \\ plant & 47 \\ home decor & 45 \\ vegetable & 42 \\ accessory & 37 \\ dessert & 36 \\ furniture & 36 \\ breakfast & 35 \\ fruit & 34 \\ musical instrument & 33 \\ toy & 33 \\ fastener & 31 \\ toiletry & 31 \\ auto part & 30 \\ sea animal & 30 \\ bird & 28 \\ kitchen tool & 27 \\ medical equipment & 26 \\ school supply & 26 \\ office supply & 24 \\ seafood & 24 \\ kitchen equipment & 20 \\ drink & 19 \\ game & 19 \\ headwear & 19 \\ water vehicle & 19 \\ women’s clothing & 19 \\ livestock & 18 \\ garden tool & 17 \\ insect & 17 \\ outerwear & 16 \\ protective clothing & 16 \\ candy & 15 \\ commitment & 15 \\ footwear & 15 \\ jewelry & 15 \\ \hline \end{tabular}
\end{table}
Table 9: List of premise categories and their sizes (in terms of number of members), sorted by size. Many of these categories overlap—e.g., mammals are also animals, etc.