Competence-Based Analysis of Language Models

## Abstract

Abstract Despite the recent successes of large, pretrained neural language models (LLMs), comparatively little is known about the representations of linguistic structure they learn during pretraining, which can lead to unexpected behaviors in response to prompt variation or distribution shift.
To better understand these models and behaviors,
we introduce a general model analysis framework to
study LLMs with respect to their representation and use of human-interpretable linguistic properties.
Our framework, CALM (Competence-based Analysis of Language Models), is designed to investigate LLM competence in the context of specific tasks
by intervening on models’ internal representations of different linguistic properties
using causal probing, and measuring models’ alignment under these interventions with a given ground-truth causal model of the task.
We also develop a new approach for performing causal probing interventions using gradient-based adversarial attacks, which can target a broader range of properties and representations than prior techniques.
Finally, we carry out a case study of CALM using these interventions to analyze and compare LLM competence across a variety of lexical inference tasks,
showing that CALM can be used to explain and predict behaviors across these tasks.

## 1 Introduction

The rise of large, pretrained neural language models (LLMs) has led to rapid progress in a wide variety of natural language processing tasks
Brown et al. (); Chowdhery et al. (); Dubey et al. ().
However, these models can also be quite sensitive to minor changes in input prompts Elazar et al. (); Moradi and Samwald (); Mizrahi et al. ()
and fail to generalize outside their training or fine-tuning distribution Wang et al. (); Yang et al. ().
It is usually unclear where these limitations come from, as LLM task performance is generally studied using only “black box” behavioral analysis,
in which case one can only detect limitations that are adequately represented by the benchmark, which cannot cover every possible limitation using a finite dataset Raji et al. (); Siska et al. ().
Understanding the means by which these models can perform as well as they do while exhibiting such limitations is a key question in the science of LLM interpretation and analysis Bereska and Gavves (), and is likely necessary in enabling robust, trustworthy, and socially-responsible LLM-enabled applications
Shin (); Liao and Vaughan (); Zou et al. (); Bereska and Gavves ().

We approach this question in terms of *competence*, drawing on the traditional competence-performance distinction in linguistic theory (see Section 2.1) to motivate the study of LLMs in terms of their underlying representation of language.
We define LLM competence in the context a given linguistic task as the alignment between the ground-truth causal structure of the task and the LLM’s latent representation of the task’s structure, measured by intervening on the LLM’s representation of task-causal or non-causal properties and observing how its behavior changes in response.
While such representations are not directly observable,
we take inspiration from *causal probing*, which damages LLMs’ latent representations of linguistic properties using causal interventions to study how these representations contribute to their behavior Elazar et al. (); Lasri et al. ().
We introduce a general framework, CALM (for Competence-based Analysis of Language Models), to study the competence of LLMs using causal probing and define the first quantitative measure of LLM competence.

While CALM can be instantiated using a variety of existing causal probing interventions (e.g., Ravfogel et al., , , ; Shao et al., ; Belrose et al., ), we develop a new intervention methodology for damaging LLM representations using gradient-based adversarial attacks against structural probes, extending causal probing to arbitrarily-encoded representations of relational properties
and thereby enabling the investigation of new questions in language model interpretation.
We carry out a case study of CALM using two well-studied LLMs
by implementing interventions as GBIs in order to measure and compare these LLMs’ competence across 14 lexical inference tasks,
showing that CALM can indeed
explain and predict important patterns in behavior across these tasks by distinguishing between models’ use of causal and spurious properties.

Our primary contributions are as follows:

- 1.
We introduce CALM, a general interpretability framework for studying LLM competence using causal probing.
- 2.
We provide a causal formulation of linguistic competence in the context of LLMs, using CALM to define the first quantitative measure of LLM competence.
- 3.
We establish a gradient-based intervention strategy for causal probing, which directly addresses multiple limitations of prior methodologies.
- 4.
We implement a case study of CALM using gradient-based interventions, demonstrating its utility in explaining and predicting LLM behaviors across several lexical inference tasks.

## 2 Competence-based Analysis of Language Models

### 2.1 Linguistic Competence

Linguistic competence is generally understood as the ability to utilize one’s knowledge of a language in producing and understanding utterances in that language, and is typically defined in contrast with linguistic performance, which is speakers’ actual use of their language in practice, considered independently of the underlying knowledge that supports it Marconi ().(^1^11 There has been significant debate in linguistics and the philosophy of language regarding the precise definition and nature of competence Lyons (); Newmeyer (); Sag and Wasow (); Marconi (). However, the formalization of competence provided in this work is sufficiently general to incorporate most notions of competence, which may be flexibly specified by instantiating CALM in different ways.)
Given a linguistic task, we may understand competence in terms of the underlying linguistic knowledge that one draws upon to perform the task.
If fluent human speakers rely on (implicit or explicit) knowledge of the same set of linguistic properties to perform a given task, then we may understand their performance of this task as being causally determined by these properties, and invariant to other properties.
For example, if we consider the two utterances “the chicken crosses the road” and “the chickens cross the road”, the grammatical number of the subject (i.e., singular and plural, respectively) determines whether the verb “(to) cross” should be conjugated as “crosses” or “cross”.
As English (root) verb conjugation always depends on the grammatical number of the subject,
grammatical number may be regarded as having a causal role in the task of English verb conjugation, so we may understand fluent English speakers’ (usually implicit) mental representation of verb tense as having a causal role in their behavior.
In this work, we focus on *lexicosemantic competence*, the ability to utilize knowledge of word meaning relationships in performing tasks such as lexical inference Marconi (, ).

While the study of human competence has a rich history in linguistics, there is currently no generally accepted framework for studying LLM competence Mahowald et al. (); Pavlick ().
Our primary goal in this work is to define and test a general empirical analysis framework for interpreting and measuring LLM competence, as outlined in the following section.

### 2.2 CALM Framework

In order to make the study of competence tractable in the context of LLMs,
we introduce the CALM (Competence-based Analysis of Language Models) framework,
which describes an LLM’s competence with respect to a given linguistic task in terms of its latent representation of the causal structure of the task.

Figure: Figure 1: Structural causal model (SCM) of task ${\mathcal{T}}$’s data-generating process and how it may be performed by model $M$. Shaded and white nodes denote observed and unobserved variables, respectively. In CALM, the goal is to determine which representations $Z_{j}=z_{j}$ are causally implicated in $M$’s predictions $\hat{\mathbf{y}}$.
Refer to caption: extracted/5805793/img/scm.png

#### Task Structure

Formally, given supervised task ${\mathcal{T}}\sim P(\mathcal{X},{\mathcal{Y}})$ where the goal is to correctly predict ${\mathbf{y}}\in{\mathcal{Y}}$ given $\mathbf{x}\in\mathcal{X}$, and a collection of latent properties ${\mathbf{Z}}=\{Z_{j}\}_{j=1}^{m}$ that are (potentially) involved in generating $\mathbf{x}$, we formulate the causal structure of ${\mathcal{T}}$ in terms of the data-generating process

$$ $\mathbf{x}\sim\Pr(\mathbf{x}|{\mathbf{Z}}_{c},{\mathbf{Z}}_{e}),\quad{\mathbf{ y}}\sim P({\mathbf{y}}|{\mathbf{Z}}_{c})$ (1) $$

where
${\mathbf{Z}}$ may be decomposed into ${\mathbf{Z}}={\mathbf{Z}}_{c}\cup{\mathbf{Z}}_{e},{\mathbf{Z}}_{c}\cap{\mathbf
{Z}}_{e}=\varnothing$, where ${\mathbf{Z}}_{c}$ contains all properties that causally determine $\mathbf{y}$, and ${\mathbf{Z}}_{e}$ are the remaining properties that may be involved in generating $\mathbf{x}$ (cf. Ilse et al., ).
However, there may be an unobserved confounder $S$ that produces spurious correlations between ${\mathbf{y}}$ and ${\mathbf{Z}}_{e}$, which, if leveraged by language model $M$ in the course of predicting $\hat{\mathbf{y}}$, can lead to unexpected failures on ${\mathcal{T}}$ when the spurious association is broken Pearl ().
The structural causal model (SCM)(^2^22 An SCM is a directed acyclic graph where each node represents a variable and directed edges indicate causal dependencies (see Bongers et al. for an introduction to SCMs).) of this data-generating process is visualized on the left side of Figure 2.

For example, suppose a speaker wants to communicate that orangutans are a genus of primate. She might say “orangutans are primates” or “orangutans, a genus of apes, are primates”.
In both cases, the conjugation of the root verb would be “are” because it is independent of whether the subject is complemented by an appositive phrase like “a genus of apes”, and this phrase does not change the grammatical number of the subject “orangutans”;
so if we define ${\mathcal{T}}_{\text{VC}}$ as English verb conjugation, $Z_{\text{NS}}$ as the grammatical number of the subject, and $Z_{\text{AP}}$ as the presence of an appositive phrase modifying the subject, then it is clear that $Z_{\text{NS}}\in{\mathbf{Z}}_{c}$ and $z_{\text{AP}}\in{\mathbf{Z}}_{e}$.
However, if we instead consider the task ${\mathcal{T}}_{\text{H}}$ of predicting hypernyms – for example, predicting $y$ in “orangutans are $\mathbf{y}$s”, where $\mathbf{y}=$ “primate” and $\mathbf{y}=$ “ape” would both be correct answers –
the causal property $Z_{\text{H}}\in{\mathbf{Z}}_{c}$ will be the hypernymy relation, and $Z_{\text{NS}}\in{\mathbf{Z}}_{e}$
(e.g., the same answers will be correct if the question is instead posed as “an orangutan is a $\mathbf{y}$”).
Thus, we expect competent English speakers to be invariant to grammatical number when performing hypernym prediction (see Figure 2).

#### Internal Representation

Our main concern is measuring how attributable an LLM $M$’s behavior in a given task ${\mathcal{T}}$ is to its representation of various properties ${\mathbf{Z}}=\{Z_{1},...,Z_{m}\}$, and how these properties correspond to the causal structure of the task.
If $M$ respects the data-generating process of ${\mathcal{T}}$, then its behavior should be attributable only to causal properties $Z\in{\mathbf{Z}}_{c}$ (and not to environmental properties $Z\in{\mathbf{Z}}_{e}$), in which case we say that $M$ is *competent* with respect to ${\mathcal{T}}$ (see Figure 2).
We study model $M$’s use of each property $Z_{j}\in{\mathbf{Z}}$ by performing causal interventions $\operatorname*{do}(Z_{j})$ on its representation of $Z_{j}$ in the course of performing task ${\mathcal{T}}$, and measure the impact that these interventions have on its predictions.

### 2.3 Measuring Competence

We evaluate the competence of $M$ with respect to task ${\mathcal{T}}\sim P({\mathcal{X}},{\mathcal{Y}})$ by measuring its causal alignment with a *competence graph* ${\mathcal{G}_{\mathcal{T}}}$,
which we define as a structural causal model (SCM) of ${\mathcal{T}}$ with nodes corresponding to each latent variables $Z_{j}\in{\mathbf{Z}}$ and an additional node for outputs $\mathbf{y}\in{\mathcal{Y}}$
and directed edges denoting causal dependencies between these variables.
That is, the set of causal properties ${\mathbf{Z}}_{c}$ defined by ${\mathcal{G}_{\mathcal{T}}}$ is the set of all properties $Z_{j}\in{\mathbf{Z}}$ such that there is an edge or path from $Z_{j}$ to $\mathbf{y}$.

To determine the extent to which $M$’s behavior is correctly explained by the causal dependencies (and lack thereof) in ${\mathcal{G}_{\mathcal{T}}}$, we measure their consistency under interventions $\operatorname*{do}({\mathbf{z}})$, where
setting ${\mathbf{z}}=\{z_{j}\}_{j=1}^{m}\sim\operatorname*{val}({\mathbf{Z}})$ is a combination of values $Z_{j}=z_{j}\in\operatorname*{val}(Z_{j})$ taken by each corresponding latent variable $Z_{j}\in{\mathbf{Z}}$.
For instance, under the hypernym prediction task ${\mathcal{T}}_{H}$, for input ${\mathbf{x}}_{i}=$“orangutans are $\mathbf{y}$s” and ground-truth output $\mathbf{y}=$“primate”,
the values taken by ${\mathbf{z}}_{i}$ would be $Z_{\text{H}}=1,Z_{\text{NS}}=1$ (where $1$ indicates the presence of hypernymy and a plural noun subject, respectively),
and we might define an alternative ${\mathbf{z}}^{\prime}$ where $Z_{\text{H}}=0,Z_{\text{NS}}=1$, under which a competent model’s prediction would be expected to change with the causal variable $Z_{\text{H}}$ (i.e., $M(\mathbf{x}|\operatorname*{do}({\mathbf{z}}^{\prime}))\neq M(\mathbf{x})$).

The alignment of $M$ with ${\mathcal{G}_{\mathcal{T}}}$ is measured in terms of the similarity $S$ of their predictions under interventions $\operatorname*{do}({\mathbf{z}})$ given input ${\mathbf{x}}\sim P({\mathcal{X}})$,
and can be computed using a given similarity metric $S:{\mathcal{Y}},{\mathcal{Y}}\to[0,1]$ (e.g., equality, n-gram overlap, cosine similarity,
etc.) depending on the SCM ${\mathcal{G}_{\mathcal{T}}}$ and output space ${\mathcal{Y}}$.
That is, we define ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ as $M$’s competence with respect to task ${\mathcal{T}}$ as a function of its alignment with corresponding task SCM ${\mathcal{G}_{\mathcal{T}}}$ under interventions $\operatorname*{do}({\mathbf{z}})$ measured by similarity metric $S$, as follows:

$$ ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})=\\ \mathbb{E}_{{\mathbf{x}},{\mathbf{z}}\sim P({\mathcal{X}},\operatorname*{val}( {\mathbf{Z}}))}S\big{(}M({\mathbf{x}}|\operatorname*{do}({\mathbf{z}})),{ \mathcal{G}_{\mathcal{T}}}({\mathbf{x}}|\operatorname*{do}({\mathbf{z}}))\big{)}$ (2) $$

This ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ metric (bounded by $[0,1]$) is an adaptation of the Interchange Intervention Accuracy (IIA) metric (Geiger et al., , ) to the context of causal probing, where instance-level interventions are replaced with concept-level interventions enabled by the gradient-based intervention methodology we introduce in Section 3. (See Section B.1 for a detailed comparison of our competence metric with IIA.)

### 2.4 Causal Probing

A key technical challenge in implementing CALM (and causal probing more generally) is designing an algorithm to perform causal interventions $\operatorname*{do}(Z)$ that maximally damage the representation of a property $Z$ while otherwise minimally damaging representations of other properties $Z^{\prime}$ Ravfogel et al. ().
For example, *amnesic probing* Elazar et al. () uses the INLP algorithm Ravfogel et al. () to produce interventions $g_{Z}$ that remove all information that is linearly predictive of property $Z$ from a pre-computed set of embedding representations ${\mathbf{H}}$,
showing that BERT makes variable use of parts-of-speech, syntactic dependencies, and named-entity types in performing masked language modeling.
However, Elazar et al. () also found that, when INLP is used to remove BERT’s representation of these properties in early layers, it is often able to “recover” this representation in later layers, which is likely due to BERT encoding these properties nonlinearly; and later work has
found that the same “recoverability” problem persists even when linear information removal methods like INLP are kernelized Ravfogel et al. ().
Thus, it is necessary to develop interventions that do not require restrictive assumptions about the structure of LLMs’ representations (e.g., linearity; see Vargas and Cotterell ), a problem which we aim to solve in the following section.

## 3 Gradient-Based Interventions

Our goal in developing gradient-based interventions (GBIs) as a causal probing technique is to enable interventions over arbitrarily-encoded LLM representations.
GBIs allow users to flexibly specify the class of representations they wish to target, expanding the scope of causal probing to arbitrarily-encoded properties.
We take inspiration from Kos et al. (), who developed a technique to perturb latent representations using gradient-based adversarial attacks.(^3^33 Notably, Tucker et al. () developed a similar methodology without explicit use of such attacks (see Section 6).)
They begin by training probe $g_{Z}$ $:\mathbf{h}\mapsto z$ to predict image class $z\in Z$ from latent representations $\mathbf{h}=f_{\text{enc}}(\mathbf{x})$ of images $\mathbf{x}$, where $f_{\text{enc}}$ is the encoder of a VAE-GAN Larsen et al. () trained on an unsupervised image reconstruction task (i.e., $f_{\text{dec}}(f_{\text{enc}}(\mathbf{x}))=\hat{\mathbf{x}}\approx\mathbf{x}$, for decoder $f_{\text{dec}}$ and reconstructed image $\hat{\mathbf{x}}$ approximating $\mathbf{x}$).
Next, gradient-based attacks like FGSM Goodfellow et al. () and PGD Madry et al. () are performed against $g_{Z}$ in order to minimally manipulate $\mathbf{h}$ such that it resembles encoded representations of target image class $Z=z^{\prime}$ (where $z^{\prime}\neq z$, the original image class), yielding perturbed representation $\mathbf{h}^{\prime}$.
Finally, $\mathbf{h}$ and $\mathbf{h}^{\prime}$ are each fed into the VAE decoder to reconstruct corresponding output images $\hat{\mathbf{x}}$ and $\hat{\mathbf{x}}^{\prime}$ (respectively),
where $\hat{\mathbf{x}}$ resembles input image class $Z=z$ and $\hat{\mathbf{x}}^{\prime}$ resembles target class $Z=z^{\prime}$.

Figure: Figure 3: Gradient-Based Interventions. Input tokens $\mathbf{x}=(x_{1},...,x_{|\mathbf{x}|})$ are passed through layers $L=1,...,l$, where embedding $\mathbf{h}_{i}^{l}$ (encoding the value $Z=z$) is extracted from layer $l$ and given to $g_{Z}$ as input. Next, the embedding is modified by gradient-based attacks on $g_{Z}$ to encode the counterfactual value $Z=z^{\prime}$, then fed back into subsequent layers $L=l+1,...,|L|$ and language modeling head $f_{\text{LM}}$ to obtain the intervened predictions $M(\mathbf{x}|\operatorname*{do}(Z=z^{\prime}))$.
Refer to caption: extracted/5805793/img/gbi.png

We reformulate this approach in the context of causal probing as visualized in Figure 3, treating layers $L=1,...,l$ as the encoder and layers $L=l+1,...,|L|$ (composed with language modeling head $f_{\text{LM}}$) as the decoder, allowing us to target representations of property $Z$ across embeddings $\mathbf{h}_{i}^{l}$ of token $x_{i}\in\mathbf{x}$ in layer $l$.
We train $g_{Z}$ to predict $Z$ from a set of such $\mathbf{h}_{i}^{l}$,
then attack $g_{Z}$ using FGSM and PGD to intervene on $\mathbf{h}_{i}^{l}$ (representing the original value $Z=z$), producing $\mathbf{h}_{i}^{l^{\prime}}$ (representing the counterfactual value $Z=z^{\prime}$).
Finally, we replace $\mathbf{h}_{i}^{l}$ with $\mathbf{h}_{i}^{l^{\prime}}$ in the LLMs’ forward pass from layers $L=l+1,...,|L|$, simulating the intervention $\operatorname*{do}(Z=z^{\prime})$, and observe the impact on its word predictions $M(\mathbf{x}|\operatorname*{do}(Z=z^{\prime}))$.

#### Benefits and Drawbacks

The key advantage of gradient-based interventions (GBIs) as a causal probing methodology is that they may be applied to any differentiable probe. For example, if we are investigating the hypothesis that $M$’s representation of $Z$ is captured by a linear subspace of representations in a given layer (see Vargas and Cotterell, ), then we may train a linear probe and various nonlinear probes on representations and observe whether GBIs against the linear probe have a comparable impact to those against the nonlinear probes.
Alternatively, if we believe that a probe’s architecture should mirror the architecture of the model it is probing (as argued by Pimentel et al., ), we may implement probes as such.
Finally, where previous intervention methodologies for causal probing have focused on *nullifying* interventions that remove the representation of the target property $Z$ Ravfogel et al. (, , ); Shao et al. (); Belrose et al. (), GBIs allow one to perform targeted interventions that set LLMs’ representations to counterfactual values $do(Z=z^{\prime})$, effectively simulating the model’s behavior under counterfactual inputs, which may be useful for predicting behaviors under various distribution shifts (see Section B.1).
However, the benefits associated with GBIs do come with some important limitations, as we discuss in Section 8.

## 4 Experiments

In this work, we begin by examining BERT Devlin et al. () and RoBERTa Liu et al. (),(^4^44 Specifically, [BERT-base-uncased](https://huggingface.co/bert-base-uncased) and [RoBERTa-base](https://huggingface.co/roberta-base) Wolf et al. ().) two language models which have been extensively studied in the context of probing
Rogers et al. (); Ravfogel et al. (); Liu et al. (); Elazar et al. (); Lasri et al. ().
Our primary goal in the following experiments is to develop and test an experimental implementation of CALM using GBIs in the context of comparatively small, well-studied models and tasks in order to validate whether CALM can explain behavioral findings of earlier work in this simplified environment.
(We motivate this choice in greater detail in Section A.1.)

### 4.1 Tasks

Masked language models like BERT and RoBERTa are trained to predict $\Pr(x_{\texttt{[MASK]}}=w|\mathbf{x})$ for text input (token sequence) $\mathbf{x}=(x_{1},x_{2},...,x_{|\mathbf{x}|})$, mask token $x_{\texttt{[MASK]}}\in\mathbf{x}$, and token vocabulary $V=\{w_{1},w_{2},...,w_{|V|}\}$.
As such, it is common to study them by providing them with “fill-in-the-blank” style masked prompts (e.g., “a cat is a type of [MASK]”) and evaluating their accuracy in predicting the correct answer (e.g., “animal”, “pet”, etc.), a task known cloze prompting (Liu et al., ).

We use the collection of 14 lexical inference tasks included in the ConceptNet Speer et al. () subset of LAMA Petroni et al. (),
each of which are formulated as a collection cloze prompts.
For example, the LAMA “IsA” task contains $\sim$2K hypernym prompts corresponding to the “IsA” ConceptNet relation (including, e.g., “A laser is a [MASK] which creates coherent light.”, where the task is to predict that the [MASK] token should be replaced with “device”, a hypernym of “laser”), with the remaining 13 LAMA ConceptNet tasks corresponding to other lexical relations such as “PartOf”, “HasProperty”, and “CapableOf”. (See Section A.2 for additional details.)

Using these task datasets allow us to
test how the representation of each relation is used across all other tasks.
In the context of a single task ${\mathcal{T}}_{j}$, intervening on a model’s representation of the task-causal relation $Z_{j}$ allows us to measure the extent to which its predictions are attributable to its representation of the causal property ${\mathbf{Z}}_{c}=\{Z_{j}\}$ (where a large impact indicates competence).
On the other hand, intervening on the representations of the other 13 lexical relations $Z_{k}\in{\mathbf{Z}}_{e}$ allows us (in the aggregate) to measure how much the model is performing task ${\mathcal{T}}_{j}$ by leveraging representations of general, non-causal lexical information (where a large impact indicates incompetence).(^5^55 Note that the strictest interpretation of this formulation of competence makes the simplifying assumption that each non-causal property is equally (un)related to the target property, which is not generally true; see Section 8.)

### 4.2 Experimentally Measuring Competence

Given LLM $M$ and task ${\mathcal{T}}$, measuring the competence ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ of $M$ given ${\mathcal{G}_{\mathcal{T}}}$ requires us to specify an experimental model $E=({\mathbf{Z}},{\mathcal{G}_{\mathcal{T}}},S)$, where ${\mathbf{Z}}$ is a set of properties, ${\mathcal{G}_{\mathcal{T}}}$ is a competence graph for task ${\mathcal{T}}$, and $S$ is a scoring function that compares the predictions of $M$ and ${\mathcal{G}_{\mathcal{T}}}$.
Given that each task ${\mathcal{T}}_{i}$ is defined by a single causal lexical relation $Z_{i}$ (i.e., ${\mathbf{Z}}_{c_{i}}=\{Z_{i}\}$), we model settings ${\mathbf{z}}$ as a collection of values $Z_{j}=z_{j}$ taken by each property $Z_{j}$ in the context of a specific task instance $({\mathbf{x}},{\mathbf{y}})\sim{\mathcal{T}}_{i}$,
where $Z_{j}=1$ if $i=j$ (i.e., where the property $Z_{j}$ is the causal property for the task ${\mathcal{T}}_{i}$) or $Z_{j}=0$ otherwise.
That is, for each instance $({\mathbf{x}},{\mathbf{y}})\sim{\mathcal{T}}_{i}$, the corresponding setting ${\mathbf{z}}$ is a one-hot vector whose $i$-th element ${\mathbf{z}}_{i}=1$.
We may specify ${\mathcal{G}}_{{\mathcal{T}}_{i}}$ in a similar manner: for task ${\mathcal{T}}_{i}\sim P({\mathcal{X}},{\mathcal{Y}})$, outputs ${\mathbf{y}}\in{\mathcal{Y}}$ are causally dependent on the property $Z_{i}$,
and invariant to other concepts $Z_{j},j\neq i$.,
meaning that the only direct parent node of $\mathbf{y}$ in ${\mathcal{G}}_{{\mathcal{T}}_{i}}$ is $Z_{i}$.
Finally, as we are dealing with masked language models whose output space ${\mathcal{Y}}$ for each task consists only of single tokens in $M$’s vocabulary $V_{M}$, our experimental model can define the scoring function $S$ as the overlap $\operatorname*{overlap}({\mathbf{y}}_{i},{\mathbf{y}}_{j})$ for top-$k$ token predictions ${\mathbf{y}}_{i}=\{y_{1},...,y_{k}\}\subset V_{M}$, where $\operatorname*{overlap}(\cdot,\cdot)$ is the size of the intersection of each set of predictions divided by the total number of predictions $\operatorname*{overlap}({\mathbf{y}}_{i},{\mathbf{y}}_{j})=\frac{|{\mathbf{y}}
_{i}\cap{\mathbf{y}}_{j}|}{k}$.
(See Section B.2 for additional details on how we compute competence in each experiment.)

### 4.3 Probes

We implement probes $g_{Z}$ as a 2-layer MLP over each language model’s final hidden layer, and train the probe on the task of classifying whether there is a particular relation $Z$ between a final-layer [MASK] token in the context of a cloze prompt
and the final-layer object token from the “unmasked” version of the same prompt. All reported figures are the average of 10 runs of our experiment, using different randomly-initialized $g_{Z}$ each time. (See Section A.3 for further details.)

### 4.4 Interventions

We implement GBIs against $g_{Z}$ using two gradient attack strategies, FGSM Goodfellow et al. () and PGD Madry et al. ().
We bound the magnitude of each intervention as follows: where $h$ is the input to $g_{Z}$ and $h^{\prime}$ is the intervened representation following a GBI, $||h-h^{\prime}||_{\infty}\leq\epsilon$.
For all experiments reported in our main paper, we use FGSM with $\epsilon=0.1$. (See Section A.4 for more details and PGD results.)

## 5 Results

Figure: Figure 4: Performance (top) and competence (bottom) of BERT (left bars) and RoBERTa (right bars) for all tasks, using FGSM with $\epsilon=0.1$. In the competence plot, y-values are the average competence score and error bars are the maximum and minimum competence score, as measured over 10 experimental iterations (each with a different randomly-initialized probe $g_{Z}$).
Refer to caption: extracted/5805793/img/perfplot.png

In Figure 4, we visualize the performance and competence of BERT and RoBERTa across the test set of each LAMA ConceptNet task.
Performance is measured using $(0,1)$-accuracy, competence is measured using the experimental competence metric in Equation 3, and both metrics are averaged across the top-$k$ predictions of each model for $k\in[1,10]$.
Specifically, for accuracy, we compute

$$ $\frac{1}{n}\sum_{k=1}^{n}\mathbbm{1}[y\in\operatorname*{top-\mathit{k}}_{\hat{ y}}\Pr(\hat{y}|\mathbf{x})]$ $$

for ground truth $(\mathbf{x},y)$ and $n=10$; and for competence, we compute

$$ $\frac{1}{n}\sum_{k=1}^{n}{\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{ T}}})$ $$

To account for stochasticity in initializing and training probes $g_{Z}$, scores are also averaged over 10 randomized experiments for each target task where the probe is randomly re-initialized each time (resulting in different GBIs).

### 5.1 Analysis

#### Performance

While their accuracies on individual tasks vary, BERT and RoBERTa have quite similar aggregate performance: BERT outperforms RoBERTa on just over half (8/14) of the tasks, achieving essentially equivalent performance when averaged across all tasks (0.3099 versus 0.3094).

#### Competence

Given our experimental model $E$ with $m=14$ tasks, consider a random baseline language model $R$ whose predictions always change in response to each intervention, making equal use of all properties in each task. $R$ would yield a competence score of ${\mathcal{C}}(R|{\mathcal{G}_{\mathcal{T}}})=\frac{1}{m}\approx 0.0714$ for each task.
Both BERT and RoBERTa score above this threshold for all tasks, meaning that their competence is consistently greater than that of a model ($R$) that does not distinguish between causal and environmental properties.
However, RoBERTa is consistently less competent than BERT (on 12/14 tasks), and also has lower competence scores averaged across all tasks (0.381 vs. 0.334).

We also observe that, for the two tasks (HasSubevent and MotivatedByGoal) where RoBERTa is more competent than BERT, it also achieves substantially higher performance.
Specifically, relative performance and competence are correlated: the Spearman’s Rank correlation coefficient between the average difference in accuracy and average difference in performance is a moderately strong positive correlation $\rho=0.508$ with significance $p=0.064$.

### 5.2 Discussion

*A priori*, we might expect an LLM with nontrivial performance
to also exhibit greater competence than a random baseline like $R$; but this is not necessarily the case, given that it is common for deep learning models to achieve remarkable performance by exploiting spurious correlations inherent in a given task dataset McCoy et al. (); Geirhos et al. (); Feder et al. ().
Thus, the finding that BERT and RoBERTa’s performance on each task is supported by an intermediate level of competence on the part of both models is meaningful: for each task, their behavior is generally more attributable to their representations of causally-invariant properties than to spurious lexical associations, and this competence varies substantially between tasks.

#### Explananda

Prior work has shown that BERT and RoBERTa have widely varying performance in response to lexical inference tasks, depending on the specific manner in which they are prompted (Hanna and Mareček, ; Ravichander et al., ; Ettinger, ; Elazar et al., ; see Section 6).
One possible explanation for this phenomenon is that these models may not consistently utilize a representation of the task-causal lexical relations (i.e., they are not highly competent for these tasks), instead relying (at least in part) on spurious lexical associations learned from its training data.
Previously, it has not been possible to empirically assess this hypothesis;
but using CALM, it is possible to provide direct evidence in its favor, as we find that both models possesses (only) an intermediate degree of competence for lexical inference prompt tasks.

#### Future Work

While the simplified experimental context considered in this work is a necessary first step in empirically validating our theoretical CALM framework, competence metric, and GBI methodology, we anticipate a much broader range of future research directions and potential applications for CALM.
We provide a brief summary of several possible directions below, with more detailed discussions of each direction available in Appendix C.

First, the CALM framework can be easily extended to study how various model training and fine-tuning choices impact learned representations (see Section C.1).
CALM can also be used to characterize tasks based on mutual dependency structures, which may be useful for predicting model behaviors in the context of related tasks or selecting related tasks for multitask learning or intermediate fine-tuning (see Section C.2).
Alternatively, CALM also allows one to discover conflicting task dependencies and potentially protect against negative task interactions (see Section C.3).
Finally,
it is also possible to discover a causal model describing an LLM’s implicit task representation, rather than comparing against a pre-specified ground truth task structure ${\mathcal{G}_{\mathcal{T}}}$, by synthesizing CALM and traditional causal graph discovery algorithms (see Section C.4).

## 6 Related Work

#### Hypernym Prompting

The performance of BERT-like models on lexical inference tasks such as hypernym prediction is known to be highly variable under small changes to prompts Hanna and Mareček (); Ravichander et al. (); Ettinger (); Elazar et al. ().
Our findings offer one possible explanation for such brittle performance: BERT and RoBERTa’s partial competence in hypernym prediction indicates that it should be possible to prompt these models in a way that will yield high performance, but that its reliance on spurious lexical associations may lead it to fail when these correlations are broken – e.g., by substituting singular terms for plurals Ravichander et al. () or paraphrasing a prompt Elazar et al. ().

#### Causal Probing

Most related to our work is amnesic probing Elazar et al. (), which we discuss at length in Section 2.4.
Lasri et al. () applied amnesic probing to study the use of grammatical number representations in performing an English verb conjugation prompt task.
As this experiment involves intervening on the representation of a property which is causal with respect to the prompt task, it may be understood as an informal instantiation of CALM (albeit without considering environmental properties or measuring competence).

#### Gradient-based Interventions

Tucker et al. () developed a similar approach to our GBI causal probing methodology (as outlined in Section 2.4) without explicit use of gradient-based adversarial attacks. Their methodology is equivalent to performing a targeted, unconstrained attack using standard gradient descent.(^6^66 I.e., they continue running gradient updates until the targeted probe loss saturates, irrespective of resulting perturbation magnitude.) In such attacks, it is standard practice to constrain the magnitude of resulting perturbations Goodfellow et al. (); Madry et al. (); Kos et al. (), which we do here in order to minimize the effect of “collateral damage” done by such attacks (see Sections 4.4 and A.4);
so failing to impose such constraints may result in indiscriminate damage to representations.

#### Unsupervised Probing

Instead of training supervised probes to predict a pre-determined property of interest (as we do here), an alternative approach is to train *unsupervised* probes such as Sparse Auto-Encoders (SAEs; Subramanian et al., ; Yun et al., ; Cunningham et al., ) to automatically learn an overcomplete basis of features that are useful for sparsely representing embeddings,
which can also be used to control models’ use of these learned features Bricken et al. (); Templeton et al. ().
However, as SAEs are unsupervised probes, they yield feature vectors that are not inherently interpretable and must be retroactively interpreted,
meaning that the task of creating a supervised probe training dataset (as required for conventional causal probing) is substituted for the task of interpreting learned features Davies and Khakzar ().
However, given features that can be reliably interpreted as representing task-causal or -environmental features, it is also possible to implement CALM using unsupervised probes like SAEs.

## 7 Conclusion

In this work, we introduced CALM, a general analysis framework that enables the study of LLMs’ linguistic competence using causal probing, including the first quantitative measure of linguistic competence.
We developed the gradient-based intervention (GBI) methodology, a novel approach to causal probing that can target a far greater range of representations than previous techniques, expanding the scope of causal probing to new questions in LLM interpretability and analysis.
Finally, we carried out a case study of CALM using GBIs, analyzing BERT and RoBERTa’s competence across a collection of lexical inference tasks,
finding that even a simple experimental model is sufficient to
explain and predict their behavior across a variety of lexical inference tasks.

## 8 Limitations

#### Gradient-Based Interventions

While GBIs are applicable to a more general range of model representations than other interventions (see Section 3), this generality comes with a lack of constraints on probes ($g_{Z}$); and as a result, GBIs cannot provide the strong theoretical constraints on collateral damage as can methods like, e.g., INLP Ravfogel et al. (), which provably preserves distances between embeddings as well as possible while completely removing the linear representation of the target property.
To minimize collateral damage to representations, the magnitude of perturbations should be modulated via constraints on gradient attacks against $g_{Z}$ (see Section 4.4) and experimentally validated to control the damage done to representations (see Section A.4).
Thus, in cases where the structure of representations is believed to satisfy strong assumptions (e.g., being restricted to a linear subspace; Vargas and Cotterell, ) or strong upper bounds on collateral damage are required, CALM interventions can be implemented with methods like INLP rather than GBIs.(^7^77 It may also be possible to control for collateral damage by developing GBI strategies that offer more principled protection against damage to non-targeted properties, such as adding a loss term to penalize damage to non-targeted probes or leveraging interval bound propagation Gowal et al. () to place intervened embeddings inside the adversarial polytope for non-targeted properties. We leave such possibilities to future work.)

#### Tasks

In our experiments, we modeled the 14 LAMA ConceptNet tasks as representing fully independent properties, which is not necessarily true – e.g., knowing that a tree is made of bark or contains leaves tells us something about whether it is a type of plant.
However, in the aggregate (with impacts summed across 14 widely-varying lexical relation types in computing the final competence score for each task; see Section B.2), it may nonetheless be appropriate to treat the relations which are not causal with respect to a given task as collectively capturing spurious lexical associations.

## Appendix A Experimental Details

### A.1 Simplified Environment

As noted in Section 4, our primary goal in our experiments is to validate CALM by testing it in a simplified experimental setting consisting of comparatively small, well-studied models and tasks.
As such, we need models that are *just complex enough* for CALM to be applicable (i.e., neural language models that are capable of performing the tasks we consider at a nontrivial level of performance), making BERT and RoBERTa ideal candidates;
and in future work plan to scale CALM to more complex contexts covering larger, more powerful models as they perform more difficult tasks (see Appendix C).
This is a common setting in the context of substantial recent interpretability work: first, a theoretical framework is developed for interpreting an internal representation or mechanism and initially tested in the context of “toy” models or tasks Elhage et al. (); Olsson et al. (); Zhong et al. (); Geiger et al. (), and subsequent work scales these frameworks to the context of larger models “in the wild” Wang et al. (); Conmy et al. (); Wu et al. ().
We anticipate that all of our major contributions (the CALM framework, competence metric, and GBI causal probing method) will in principle be scalable to much larger, more recent LLMs (e.g., Zhang et al. ; BigScience et al. ; Touvron et al. , ; Groeneveld et al. , etc.), and predict that the main challenge will be in finding an appropriate probing architecture (see Pimentel et al. ).

### A.2 Tasks

The full set of LAMA ConceptNet tasks is as follows: IsA, HasA, PartOf, HasSubEvent, MadeOf, HasPrerequisite, MotivatedByGoal, AtLocation, CausesDesire, NotDesires, CapableOf, UsedFor, ReceivesAction, and HasProperty.
We split each task dataset into train, validation, and test sets with a random $80\%/10\%/10\%$ split. Train and validation instances are fed to each model to produce embeddings used to train $g_{Z}$ and select hyperparameters, respectively; and test instances are used to measure LLMs’ competence with respect to each task by observing how predictions change under various interventions.
In all experiments, we restrict each model $M$’s output space for each task ${\mathcal{T}}$ to the subset of vocabulary $V_{M}$ that occurs as a ground-truth answer $y^{*}$ for at least one instance $({\mathbf{x}},y^{*})\sim{\mathcal{T}}$ in the respective task dataset. This lowers the probability of false negatives in evaluation (e.g., penalizing the model for predicting $\hat{y}=$ “mammal” for “a dog is a type of $y$” instead of $y^{*}=$ “animal”).

### A.3 Probes

We use BERT’s final layer $L$ to encode $h_{i}^{l}$ embeddings for each such example, where $i$ is the index of the [MASK] token or target word in the input prompt $x_{i}$.
To encode the [MASK] token, we issue BERT masked prompts (as discussed above) to extract $h_{\texttt{[MASK]}}$, then repeat with the [MASK] token filled-in with the target word to encode it as $h_{+}$ (e.g., “device” in “A laser is a device which creates coherent light.”),
and concatenate matching embeddings $h=(h_{\texttt{[MASK]}};h_{+})$ to produce positive ($y=1$) training instances. We also construct one negative ($y=0$) instance, $h=(h_{\texttt{[MASK]}};h_{-})$, for each $h_{\texttt{[MASK]}}$ by sampling an incorrect target word $x_{i}$ corresponding to an answer to a random prompt from the same task, feeding it into the cloze prompt in the place of the correct answer,
and obtaining BERT’s contextualized final-layer embedding of this token ($h_{-}$).
Finally, we train $g_{Z}$ on the set of all such $(h,y)$.

We implement $g_{Z}$ as a multi-layer perceptron with 2 hidden layers, each with a width of 768 (which is one half the concatenated input dimension of 1536), using ReLU activations and dropout with $p=0.1$, training it for 32 epochs using Binary Cross Entropy with Logits Loss(^8^88[https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html))
and the Adam optimizer, saving the model from the epoch with the highest validation-set accuracy for use in all experiments.

For all competence results reported in Section 5, we run the same experiment 10 times – each with a different random initialization of $g_{Z}$ and shuffled training data – and report each figure as the average among all 10 runs.

Figure: Figure 5: Competence of BERT (left bars) and RoBERTa (right bars) for all tasks, using PGD with $\epsilon=0.1$. Y-values are the average competence score and error bars are the maximum and minimum competence score, as measured over 10 experimental iterations (each with a different randomly-initialized probe $g_{Z}$).
Refer to caption: extracted/5805793/img/compplot2.png

### A.4 Interventions

For instance $(h,y)$, classifier $g_{Z}$, loss function $\mathcal{L}$, and $L_{\infty}$-bound $\epsilon\in\{0.01,0.03,0.1,0.3\}$(^9^99 All reported results use $\epsilon=0.1$, as greater $\epsilon$ resulted in unacceptably high “collateral damage” across target tasks (e.g., even random perturbations of magnitude $\epsilon=0.3$ do considerable damage), and lesser values meant that predictions changed on target tasks consisted of only a few test instances.),
each intervention (gradient attack) $g_{z}$ may be used to produce perturbed representations $h^{\prime}=g_{z}(h,y,f_{\text{cls}},\mathcal{L},\epsilon)$ where $||h-h^{\prime}||_{\infty}\leq\epsilon$.
In particular, given $h=(h_{\texttt{[MASK]}};h_{\pm})\in\mathbb{R}^{2d}$, let $h_{\texttt{[MASK]}}^{\prime}$ be the first $d$ dimensions of $h^{\prime}$ (which also satisfies the $L_{\infty}$-bound with respect to $h_{\texttt{[MASK]}}$, $||h_{\texttt{[MASK]}}-h_{\texttt{[MASK]}}^{\prime}||_{\infty}\leq\epsilon$).
To measure BERT’s use of internal representations of $Z$ on each prompt task, we evaluate its performance when perturbed $h_{\texttt{[MASK]}}^{\prime}$ is used to compute masked-word predictions, compared to unperturbed $h_{\texttt{[MASK]}}$.

Our intent in intervening only on the final-layer mask embedding $h_{\texttt{[MASK]}}$ in our experiments is that, in the final layer of a masked language model such as BERT or RoBERTa, the only embedding which is used to compute masked-word probabilities is that of the [MASK] token. Thus, any representation of the property that is *used* by the model in its final layer must be a part of its representation of the [MASK] token, preventing “recoverability” phenomena such as those observed by Elazar et al. ().

#### FGSM

We implement Fast Gradient Sign Method (FGSM; Goodfellow et al., ) interventions as

$$ $h^{\prime}=h+\epsilon\cdot\text{sgn}(\nabla_{h}\mathcal{L}(f_{\text{cls}},x,y))$ $$

#### PGD

We implement Projected Gradient Descent (PGD; Bubeck et al., ; Madry et al., ) interventions as $h^{\prime}=h^{T}$ where

$$ $h^{t+1}=\Pi_{N(h)}\big{(}h^{t}+\alpha\cdot\text{sgn}(\nabla_{h}\mathcal{L}(f_{ \text{cls}},x,y))\big{)}$ $$

for iterations $t=0,1,...,T$, projection operator $\Pi$, and $L_{\infty}$-neighborhood $N(h)=\{h^{\prime}:||h-h^{\prime}||\leq\epsilon\}$.
This method also introduces two hyperparameters, the number of PGD iterations $T$ and step size $\alpha$.
We use hyperparameter grid search over $\alpha\in\{0.001,0.003,0.01,0.03\}$
and $T\in\{20,40,60,80,100\}$, finding that setting $\alpha=\frac{\epsilon}{10}$ and $T=40$ produces the most consistent impact on $g_{Z}$ accuracy across all tasks; so we use these values for the results visualized in Figure 5.

### A.5 Compute Budget

BERT-base-uncased has 110 million parameters, and RoBERTa-base has 125M parameters.
As our goal is to study the internal representation and use of linguistic properties in existing pre-trained models, and we are not directly concerned with training or fine-tuning such models, we use these models only for inference (including encoding text inputs, using embeddings to train probes, and feeding intervened embeddings back into the language models).
The only models we trained were probes $g_{Z}$, which each had 1.77M parameters.

Each experimental iteration (including encoding text inputs, training probes on all 14 tasks, and performing all GBIs) for either BERT or RoBERTa took less than one hour on a single NVIDIA GeForce GTX 1080 GPU, meaning that running all 10 iterations across both language models took less than 20 hours on a single GPU. Each iteration, probe, and GBI can easily be parallelized across GPUs: in our case, running all iterations across both models took less than 3 hours total across 8 GTX 1080 GPUs.

## Appendix B Competence Metric

### B.1 Comparison With IIA

As noted in Section 2.3, the ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ metric defined in Equation 2 is an adaptation of the Interchange Intervention Accuracy (IIA) metric (Geiger et al., , ),
which evaluates the faithfulness of a causal abstraction like ${\mathcal{G}_{\mathcal{T}}}$ as a (potential) explanation of the behavior of a “black box” system like $M$.
In our case, this is equivalent to evaluating the competence of $M$ on task ${\mathcal{T}}$, provided that ${\mathcal{G}_{\mathcal{T}}}$ is the appropriate SCM for ${\mathcal{T}}$, as an LLM is competent only to the extent that its behavior is determined by a causally invariant representation of the task.(^10^1010 For many tasks, there is more than one valid ${\mathcal{G}_{\mathcal{T}}}$ (see, e.g., the “price tagging game” constructed by Wu et al. ()). In such cases, ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ should be computed with respect to each valid ${\mathcal{G}_{\mathcal{T}}}$ and the highest result should be selected, as conforming to any such ${\mathcal{G}_{\mathcal{T}}}$ carries the same implications.)
IIA requires performing *interchange interventions* $M({\mathbf{x}}_{i}|\operatorname*{do}({\mathbf{z}}_{i}))$, where the part of $M$’s intermediate representation of input ${\mathbf{x}}_{i}$ hypothesized to encode latent variables ${\mathbf{Z}}$ (taking the values ${\mathbf{z}}_{i}$ when provided input ${\mathbf{x}}_{i}$) is replaced with
that of ${\mathbf{x}}_{j}$ (which, in the ideal case, causes $M$’s representation to encode the values ${\mathbf{z}}_{j}$ instead of ${\mathbf{z}}_{i}$),
and compute the accuracy of ${\mathcal{G}_{\mathcal{T}}}({\mathbf{x}}_{i}|\operatorname*{do}({\mathbf{z}}_{
j}))$ in predicting $M$’s behavior under these interventions.
Thus, given access to high-quality interchange interventions over $M$, IIA measures the extent to which ${\mathcal{G}_{\mathcal{T}}}$ correctly models $M$’s behavior under counterfactuals, and thus its faithfulness as a causal abstraction of $M$.

To adapt IIA to the context of causal probing and define ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$, we replace instance-level interchange interventions with concept-level interventions: instead of swapping $M$’s representation of variables ${\mathbf{Z}}$ given input ${\mathbf{x}}_{i}$ with that of ${\mathbf{x}}_{j}$, we intervene on representations at the level of arbitrary concept settings ${\mathbf{z}}$ that need not correspond to previously sampled ${\mathbf{x}}$, allowing us to simulate the behavior of $M$ under previously-unseen distribution shifts (i.e., settings ${\mathbf{z}}$ representing previously-unseen combinations of property values) and therefore make broader predictions about $M$’s consistency with a given causal model ${\mathcal{G}_{\mathcal{T}}}$ under such conditions.
As one of the key desiderata in studying LLM competence is to predict behavior under distribution shifts where spurious correlations are broken, ${\mathcal{C}_{\mathcal{T}}}$ is more appropriate than IIA in this setting.
However, it also introduces an additional challenge: where interchange interventions only require localizing candidate representations – as counterfactual representations are obtained merely by “plugging in” values from a different input – computing ${\mathcal{C}_{\mathcal{T}}}$ instead requires one to both localize representations and directly intervene on them to change the encoded value.
Previous causal probing intervention strategies (e.g., Ravfogel et al., , ) have generally performed interventions by *neutralizing* concept representations, not modifying them to encode specific counterfactual values; so in order to carry out our study, it is also necessary to develop a novel approach to perform such interventions.
We develop a solution to this problem, gradient-based interventions (GBIs), in Section 3.

### B.2 Experimental Competence Metric

$$ $\displaystyle{\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})\approx \frac{1}{n\cdot m}\sum_{i=1}^{n}\sum_{j=1}^{m}\operatorname*{overlap}\Big{(}M \big{(}{\mathbf{x}}_{i}|\operatorname*{do}(Z_{j}=0)\big{)},{\mathcal{G}_{ \mathcal{T}}}\big{(}{\mathbf{x}}_{i}|\operatorname*{do}(Z_{j}=0)\big{)}\Big{)}$ (3) $$

To compute the expectation in Equation 2 for test set $\{{\mathbf{x}}_{i},{\mathbf{y}}_{i},{\mathbf{z}}_{i}\}_{i=1}^{n}\sim{\mathcal{
T}}\times{\mathbf{Z}}$, we sum the competence score over all samples ${\mathbf{x}}_{i}$ and perform one intervention $\operatorname*{do}(Z_{j}=0)$ corresponding to each concept $Z_{j}\in{\mathbf{Z}}$.(^11^1111 Note that this intervention changes the prediction ${\mathcal{G}_{\mathcal{T}}}({\mathbf{x}}_{i})\neq{\mathcal{G}_{\mathcal{T}}}({ \mathbf{x}}_{i}|\operatorname*{do}(Z_{j}=0))$ if and only if $({\mathbf{x}}_{i},{\mathbf{y}}_{i})\in{\mathcal{T}}_{j}$ – i.e., where the corresponding $({\mathbf{z}}_{i})_{j}=1$ – otherwise, $({\mathbf{z}}_{i})_{j}$ is already 0 0, so the intervention has no effect. Thus, as ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ measures $M$’s consistency with ${\mathcal{G}_{\mathcal{T}}}$, then to the extent that $M$ is competent, its prediction should change under all and only the same interventions as ${\mathcal{G}_{\mathcal{T}}}$.)
As our goal is to measure the extent to which $M$’s behavior is attributable to an underlying representation of the causal property $Z_{c}$ or environmental property $Z\in{\mathbf{Z}}_{e}$, our experimental model defines ${\mathcal{G}_{\mathcal{T}}}$’s predictions with reference to $M$’s original predictions $M({\mathbf{x}}_{i})=\hat{{\mathbf{y}}_{i}}$, according to the following principle: if $M$ is competent, then its prediction $M({\mathbf{x}}_{i})=\hat{{\mathbf{y}}_{i}}$ is wholly attributable to its representation of causal property $Z_{c}$, so its predictions $M({\mathbf{x}}_{i}|\operatorname*{do}(Z_{c}))=\hat{{\mathbf{y}}_{i}}^{\prime}$ will not overlap with its original predictions $\hat{{\mathbf{y}}_{i}}$ (i.e., $\operatorname*{overlap}(\hat{{\mathbf{y}}_{i}},\hat{{\mathbf{y}}_{i}}^{\prime}
)=0$); and conversely, a competent $M$ will make the *same* predictions $M({\mathbf{x}}_{i}|\operatorname*{do}(Z_{j}))=\hat{{\mathbf{y}}_{i}}^{\prime\prime}$ for any $Z_{j}\in{\mathbf{Z}}_{e}$, because its prediction is not caused by its representation of these environmental properties (i.e., $\operatorname*{overlap}(\hat{{\mathbf{y}}_{i}},\hat{{\mathbf{y}}_{i}}^{\prime
\prime})=1$).
Motivated by this reasoning, our experimental model defines ${\mathcal{G}_{\mathcal{T}}}({\mathbf{x}}_{i}|\operatorname*{do}(Z_{j}=0))=M({
\mathbf{x}}_{i})$ for environmental $Z_{j}\in{\mathbf{Z}}_{e}$; and for causal property $Z_{c}$, defines ${\mathcal{G}_{\mathcal{T}}}({\mathbf{x}}_{i}|\operatorname*{do}(Z_{c}=0))=\{y^
{\prime}\in V_{M}:y^{\prime}\notin M({\mathbf{x}}_{i})\}$ (i.e., the set of all tokens $y^{\prime}$ in $M$’s vocabulary that were not in its original prediction $M({\mathbf{x}}_{i})$).
Thus, under experimental model $E$, we approximate ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ by computing Equation 3.

Notably, our experimental model $E$ only accounts for the relationship between $M$’s intervened and non-intervened predictions, independently of ground truth labels – instead, what is being measured is $M$’s consistency under meaning-preserving interventions $\operatorname*{do}(Z_{j^{\prime}})$ and its mutability under meaning-altering interventions $\operatorname*{do}(Z_{j})$.
However, as we find in Section 5.1, the resulting competence metric ${\mathcal{C}_{\mathcal{T}}}(M|{\mathcal{G}_{\mathcal{T}}})$ is nonetheless useful for predicting $M$’s accuracy.

## Appendix C Future Work

### C.1 Representation Learning

The CALM framework, competence measure, and GBI methodology developed in Sections 2 and 3 are sufficiently general to be directly applied to analyze arbitrary LLMs on any language modeling task whose causal structure is already well understood (or, for tasks where this is not the case, we may apply the causal graph discovery approach described in Section C.4), allowing us to study the impact of various model architectures, pre-training regimes, and fine-tuning strategies on the representations LLMs learn and use for arbitrary tasks of interest.

### C.2 Multitask Learning

Are high competence scores on task ${\mathcal{T}}$ correlated with an LLMs’ robustness to meaning-preserving transformations (see, e.g., Elazar et al., ) on tasks ${\mathcal{T}}^{\prime}$ that share several causal properties ${\mathbf{Z}}_{c}$ with task ${\mathcal{T}}$.
Through the lens of causally invariant prediction (Peters et al., ; Arjovsky et al., ; Bühlmann, ), this hypothesis is likely true (however, see Rosenfeld et al. for appropriate caveats) – if so, this would make it possible to use clusters of related tasks to predict LLMs’ robustness (and other behavioral patterns, such as brittleness in the face of distribution shifts introduced by spurious dependencies) between related tasks using CALM, given an appropriate experimental model.
Furthermore, the ability to characterize tasks based on mutual (learned) dependency structures could be valuable in transfer learning applications such as guiding the selection of auxiliary tasks in multi-task learning (Ruder, ) or predicting the impact of intermediate task fine-tuning on downstream target tasks (Choshen et al., ).

### C.3 Task Dependencies

Another possible application of CALM concerns causal invariance under multi-task applications.
Existing approaches in invariant representation learning generally require task-specific training Zhao et al. (), as the notion of invariance is inherently task-centric (i.e., the properties which are invariant predictors of output values vary by task, and different tasks may have opposite notions of which properties are causal versus environmental; see Section 2.2), so applying such approaches
to train models to be causally invariant with respect to a specific downstream task ${\mathcal{T}}$ is expected to come at the cost of performance on other downstream tasks ${\mathcal{T}}^{\prime}$.
Therefore, considering the recent rise of open-ended, task-general LLMs Zhang et al. (); BigScience et al. (); Touvron et al. (, ); Groeneveld et al. (),
it is important to find alternative approaches for studying models’ causal dependencies in a task-general setting to account for applications involving tasks with different (and perhaps contradictory) causal structures, such as CALM.

### C.4 Causal Competence Graph Discovery

One of the key benefits of CALM is that, instead of simply measuring consistency with respect to a known, static task description ${\mathcal{G}_{\mathcal{T}}}$, the competence metric in Equation 2 can also be used to discover a competence graph ${\mathcal{G}}$ which most faithfully explains a model $M$’s behavior in a given task or context (see Section 2.3) by computing ${\mathcal{C}}(M|{\mathcal{G}})$ “in-the-loop” of existing causal graph discovery algorithms like IGSP (Yang et al., ).
Such algorithms can be used both to suggest likely competence graphs based on interventional data collected by running CALM experiments, to recommend the experiments that would yield the most useful interventional data for the graph discovery algorithm, and to evaluate candidate graphs ${\mathcal{G}}$ using our competence metric, terminating the graph discovery algorithm once a competence graph ${\mathcal{G}}$ that offers sufficiently faithful explanations of $M$’s behavior has been found.
In this case, it is still necessary to define the set of properties ${\mathbf{Z}}$ being probed and the scoring function $S$ used to compare the predictions of $M$ and ${\mathcal{G}}$; but no knowledge of the causal dependencies (or structural functions $F:{\mathbf{pa}}(Z_{j})\mapsto Z_{j}$ mapping from causal parents ${\mathbf{pa}}(Z_{j})$ to causal dependents $Z_{j}$; see Bongers et al. ) is required.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why should I use CALM over alternatives?
2. What surprising finding have you derived from applying CALM that substantially contributes to the body of scientific knowledge?
soundness: 3
strengths: 1. Nice approach and easy to follow exposition.
2. It makes a lot of sense to use GBIs for this purpose.
3. Promising initial results. Thorough appendix.
summary: This paper proposes analyzing LLM capabilities via competence-based testing, specifically via intervening on the model’s representations via gradient-based adversarial attacks for causal probing. It is important to examine whether the linguistic competence of LLMs corresponds to that of humans, and to decouple competence from performance, since the latter might be spuriously obtained. This is especially true with modern LLMs where humans increasingly rely on them behaving in predictable and consistent ways. The CALM (competence-based analysis of LMs) framework conducts causal probing via gradient-based adversarial attacks and reports on preliminary experiments using the approach.
weaknesses: 1. Lack of substance. The paper reports on some preliminary experiments, but without any clear findings. I guess it is a little surprising that BERT is more competent than RoBERTa? The preliminary analysis is conducted on language models (I wouldn't call them "large" language models) that are 6 years old, which is an eternity in our field. To really showcase the value of CALM, I would have wanted to see much more convincing experimental results. These could either be about deriving interesting and important novel insights from LLMs (ideally recent ones); or, they could be about showing why the proposed CALM framework is better than alternatives (see below).

2. Lack of comparison. Given the lack of substantial findings by utilizing the proposed approach, the paper lacks a direct empirical comparison to alternative methods to show that it is better.

Overall, I like this paper. With more work and more substance, it could be a very good submission -- but this is overall too lightweight to merit acceptance, as it stands.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. BERT/RoBERTa are unusual choices for models to focus on in 2024. Is there a particular reason they were chosen?

2. I am not sure I understand the argument in the paragraph beginning on line 141. The final conclusion seems to have counterexamples in English, though. Consider "letters"/"mail" and "chairs"/"furniture". I actually am not sure what the significance of this is to the paper, though. Do my examples suggest that Figure 2 is in error? If not, why not? If so, does it matter for the paper?

3. The results in Figure 4 seem to show that the models cannot do the task, which settles the question "Are they competent?" (No.) Why not study situations in which models are successful, and try to understand what causal structures they use to be successful? Those causal structures could be argued to represent their competence.
soundness: 2
strengths: It is great to see creative work that is seeking to push beyond simple behavioral assessments to more deeply understand how models represent linguistic data. This seems critical to the project of interpretability, which itself seems like an important building block for foundational work on robustness and out-of-domain generalization.

I also appreciate that the authors have defined some creative causal probing techniques.
summary: This paper seeks to develop a framework for assessing the relative competence of LLMs at specific linguistic phenomena. In essence, this means assessing whether LLMs conform to simple causal models of how linguistic items depend on other linguistic items. The experiments focus on mostly lexical relations, and the models assessed are BERT and RoBERTa. Overall, these models are only moderately successful at the lexical tasks, and, by the paper's methods, only moderately competent as well.
weaknesses: 1. Really a lot of relevant literature is not cited. The paper makes very few connections to causal interpretability research and very few connections to work on targeted analysis of specific linguistic phenomena.

    * For an overview of interpretability methods in this space, I recommend [Mueller et al. 2024](https://arxiv.org/abs/2408.01416) -- this article will help fill in gaps in the current paper's connections to prior wrok.

    * For linguistic analyses, the major benchmarks are SyntaxGym and BLiMP, each of which now have variants in other languages. The most closely related paper to the current one is [Arora et al. 2024](https://aclanthology.org/2024.acl-long.785/) (CausalGym). That paper extends SyntaxGym to support supervised interpretability, and it evaluates lots of different methods on the SyntaxGym/CausalGym.

4. The linguistic investigations in this paper are a good deal more basic than those covered by the above work. SyntaxGym/CausalGym phenomena range from simple agreement patterns all the way up to long-distance dependencies.

5. Possibly as a results of the above gaps in literature coverage, the paper does not include any comparisons with top methods in this field. I think the CausalGym paper includes a pretty good overview of salient baselines and more cutting-edge methods. The current paper also includes only one very, very weak baseline (Section 6.1), which makes the results very difficult to assess. The models are not even very good at the tasks themselves (Figure 4), and so the interpretability results seem almost guaranteed to be hard to interpret.

6. I am not convinced of the value of importing the competence/performance distinction into LLM evaluation research. It's not clear that it has been a productive dichotomy within linguistics and cognitive science, and we are going to have to confront the same fraught issues here in AI. All we have is behavioral evidence from finite samples of real-world performances. We could consider trying to use this evidence to characterize capabilities that will hold for all inputs -- a challenging analytic task -- but calling this "competence" seems unlikely to clarify the nature of such results.

7. I would add that the current paper seems to use competence in an usual way relative to the linguistics literature. In linguistics, competence is one's abstract knowledge of one's language, separate from considerations that might lead to mistakes and so forth (fatigue, cognitive load, etc.). The current paper seems to want to classify models as competent or not based on prior theoretical characterizations (Section 2.3). This is very different. If we are going to adopt the  competence/performance framing, we should try to get at the underlying "knowledge" of the model rather than insisting that it conform to our prior expectations.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: I would recommend making more clear the concept of *environmental properties* and the difference between casual properties. As the paper is now you only mention environmental properties 8 times without further explanation or examples.

 In the abstract, you mention investigating competence by "intervening on models’ internal representations of different linguistic properties." I suggest replacing the broad references to "linguistic properties" and " linguistic competence" with "lexicosemantic properties" and "lexicosemantic competence." Since your work specifically addresses a formal aspect of linguistic competence and not functional linguistic competence, the term "linguistic competence" may come across as overly general and potentially overstates the study's scope.

If this is a competence framework I would suggest validating the framework’s effectiveness across real-world applications to strengthen its practical relevance beyond controlled experiments. How well does CALM generalize to tasks outside of lexical inference?

Figure 4's different y-axis makes it hard to compare accuracy vs competence scores (minor but I would remove the lama_ in the x labels. It is mentioned throughout the paper that the categories are a set of LAMA ConceptNet tasks, you could mention it in the Figure caption).

Spelling: 
355-356: each of which are formulated as a collection cloze prompts -> each of which **is** formulated as a collection **of** cloze prompts
361-362: Using these task datasets allow us to test how the representation of each relation is used across all other tasks. -> Using these task datasets allows us to test 
485: as we find that both models **possess** (only) an intermediate degree of competence for lexical inference prompt tasks.
There are some “?” (missing citations) throughout the paper.
soundness: 3
strengths: The strength of the paper is that it uses an evaluation approach that combines causal probing with Gradient-Based Interventions (a concept similar to adversarial perturbations). The introduction of GBIs allows for a broader and more precise range of probing interventions, addressing some possible previous limitations in probing techniques.
summary: The paper introduces an evaluation framework, which seeks to understand language models (LMs) by examining their internal representations of linguistic properties. It uses causal probing to measure the alignment of an LM’s representations with human-understandable causal structures of specific tasks.
weaknesses: Overall the novelty of the paper is limited as it uses an existing dataset (ConceptNet), and the established method of causal probing. Could you clearly articulate what aspects of your approach are novel beyond the gradient-based intervention strategy for causal probing? It seems to me that your 5 contributions are summarized by this strategy and its application.

Framework: (1) The title seems somewhat misleading; this work is less a "Competence-Based Analysis of Language Models" and more an evaluation approach focused on an engineered relational database (ConceptNet). Moreover, it’s unclear whether this can be accurately described as a measure of "linguistic competence," given that it examines only specific semantic relations. Could the authors clarify what constitutes the "analysis" aspect of the framework? Is this not more accurately described as an evaluation framework? (2) From the title and abstract the reader expects some interpretability or causal analysis to better understand models and their behaviors. There is a need to clarify what are the interpretable aspects of this framework and how it leads to interpretable insights about model behavior. (3) Additionally, I would like to see some motivation regarding the usefulness of this framework. Maybe comparing the CALM framework with other existing causal probing frameworks could indicate its importance. (4) The study’s focus on lexical inference tasks may limit the generalizability of CALM to other types of language tasks. Could you provide some motivation as to why you focused on lexicosemantic tasks?

Experiments: (1) There is no evidence that the framework is scalable to more complex, bigger, auto-regressive models. Could the authors motivate why no auto-regressive model architectures were used? (2) The framework is tested in a simplified experimental setting consisting of comparatively small, well-studied models and tasks. (3) How is this framework useful or applicable in the recent research landscape of LLMs? (4) The authors note that model "competence varies substantially between tasks," a pattern likely to persist as this framework is expanded to additional tasks and models. Previous studies on lexical inference tasks (Hanna and Marecek, 2021; Ravichander et al., 2020; Ettinger, 2020; Elazar et al., 2021a) have also highlighted such inconsistencies in model performance. Given these known variations, what novel evaluative insights does the CALM framework provide?

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: **Q1**
> "Once we instantiate the general CALM framework with a specific probing technique such as the GBI introduced in the previous section, CALM would be “operational” and can be used in many novel ways to both facilitate understanding of representations learned in LLMs and predict behaviors of LLMs in many application contexts, which would otherwise be impossible without such a framework."

This seems like a very strong claim. Could you give an example of a behaviour that could be predicted with CALM that would otherwise be impossible to predict without such a framework? Ideally a specific hypothetical scenario.

**Q2**
> "For example, just using the proposed competence measure as an additional dimension of evaluation as we have done in our experiments should already enable obtaining additional insights about the behaviors of the models. As the competence measure can be expected to have better correlation with the behavior of a model than a regular task performance measure, using the competence measure or using it in addition to regular performance measures can lead to better decisions in optimizing all kinds of decisions such as model architecture and hyperparameters."

Can you expand on what this means, and perhaps give a concrete example?

**Q3**
> "On the other hand, intervening on the representations of the other 13 lexical relations Zk ∈ Ze allows us (in the aggregate) to measure how much the model is performing task Tj by leveraging representations of general, non-causal lexical information (where a large impact indicates incompetence)."

Could you expand on what you mean by leveraging representations of non-causal lexical information? How do you distinguish this outcome from your method simply removing/replacing more information than intended? Is there a reason that the gradient-based approach you give should only perturb the intended property?

**Q4**

Line 412 - why do you use intersection in top-k rather than some kind of probability measure?

**Q5**

Eq 1: Is there a reason you use both Pr and P?

---



My review is based on the impression that this framework, applied in its current form, is unlikely to yield novel or significant insights about LM behaviour. I would be open to revising my score if the authors can address this concern. I'm particularly interested in precise situations where CALM could explain otherwise unexpected behaviours.
soundness: 2
strengths: - The introduction of a formal notion of competence in terms of causal models is interesting, and the application of insights from linguistics seem to be novel and original
- In general, the mathematical framework seems sound and well-presented
- The goal of the paper seems important and valuable: I am excited by the prospect of work that helps us to understand and predict the behaviour of models
summary: The paper is intended to shed light on the behaviour of LMs. The authors propose 'competence-based analysis of language models' (CALM):
- Firstly giving a causal formulation of the linguistic notion of competence in the context of LMs (roughly, invariance of the output to perturbation of irrelevant lexical features)
- Secondly giving a framework for testing this notion of competence using causal probing and preexisting gradient-based methods
- Finally giving some basic demonstrations of the framework being applied to BERT and RoBERTA on a few different tasks
weaknesses: The paper's main contribution is a theoretical framework, and I think they do not provide a particularly compelling case that the framework actually shows meaningful results. For instance, in the abstract they say "CALM can be used to explain and predict behaviors across these tasks", but as far as I can tell, what they show empirically is only the competence metric they propose is reasonably correlated with accuracy across a handful of task/model combinations. They suggest many potential contexts where the framework could be applied, but it is not clear to me what the framework would actually show.

The paper would be much stronger if the authors spelled out far more clearly what kinds of behaviours CALM might predict or explain, and then demonstrated that it does in fact predict or explain them, especially in contexts where this would be useful and significant - for example, if the paper gave concrete examples of how CALM could predict a model's performance on out-of-distribution data, or explained unexpected failures on certain linguistic phenomena.

**Nitpicks (not affecting the score):**
- Use \citep for parenthetical citations
- Define GBI when it's first used
- Define FGSM and PGD, and give some indication of how they work / how they are different
- The paper is missing a lot of articles, eg/ 158 "study *a* model M's", 159 "performing *a* task T", 254 "training *a* probe", and there are a few other missing prepositions: I recommend checking over the text carefully for these