Bilinear MLPs enable weight-based mechanistic interpretability

## Abstract

Abstract A mechanistic understanding of how MLPs do computation in deep neural networks remains elusive. Current interpretability work can extract features from hidden activations over an input dataset but generally cannot explain how MLP weights construct features. One challenge is that element-wise nonlinearities introduce higher-order interactions and make it difficult to trace computations through the MLP layer. In this paper, we analyze bilinear MLPs, a type of Gated Linear Unit (GLU) without any element-wise nonlinearity that nevertheless achieves competitive performance. Bilinear MLPs can be fully expressed in terms of linear operations using a third-order tensor, allowing flexible analysis of the weights.
Analyzing the spectra of bilinear MLP weights using eigendecomposition reveals interpretable low-rank structure across toy tasks, image classification, and language modeling. We use this understanding to craft adversarial examples, uncover overfitting, and identify small language model circuits directly from the weights alone. Our results demonstrate that bilinear layers serve as an interpretable drop-in replacement for current activation functions and that weight-based interpretability is viable for understanding deep-learning models.

## 1 Introduction

Multi-layer perceptrons (MLPs) are an important component of many deep learning models, including transformers (Vaswani et al., ). Unfortunately, element-wise nonlinearities obscure the relationship between weights, inputs, and outputs, making it difficult to trace a neural network’s decision-making process.
Consequently, MLPs have previously been treated as undecomposable components in interpretability research (Elhage et al., ).
(^0^0footnotetext: Code at: [https://github.com/tdooms/bilinear-decomposition](https://github.com/tdooms/bilinear-decomposition))

While early mechanistic interpretability literature explored neural network weights (Olah et al., ; ; Voss et al., ; Elhage et al., ), activation-based approaches dominate contemporary research (Petsiuk et al., ; Ribeiro et al., ; Simonyan et al., ; Montavon et al., ). In particular, most recent studies on transformers use sparse dictionary learning (SDL) to decompose latent representations into an overcomplete basis of seemingly interpretable atoms
(Cunningham et al., ; Bricken et al., ; Marks et al., ; Dunefsky et al., ).
However, SDL-based approaches only describe which features are present, not how they are formed or what their downstream effect is.
Previous work has approximated interactions between latent dictionaries using linear and gradient-based attribution (Marks et al., ; Ge et al., ), but these approaches offer weak guarantees of generalization. To ensure faithfulness, ideally, we would be able to capture nonlinear feature interactions in circuits that are grounded in the model weights.

One path to better circuit discovery is to use more inherently interpretable architectures. Previous work constrains the model into using human-understandable components or concepts (Koh et al., ; Chen et al., ), but this typically requires labeled training data for the predefined concepts and involves a trade-off in accuracy compared to learning the best concepts for performance (Henighan et al., ). Ideally, we could more easily extract and interpret the concepts that models naturally learn rather than force the model to use particular concepts. To this end, Sharkey () suggested that bilinear layers of the form $g({\bm{x}})=({\bm{W}}{\bm{x}})\odot({\bm{V}}{\bm{x}})$ are intrinsically interpretable because their computations can be expressed in terms of linear operations with a third order tensor. This enables the use of tensor or matrix decompositions to understand the weights. Moreover, bilinear layers outperform ReLU-based transformers in language modeling (Shazeer, ) and have performance only slightly below SwiGLU, which is prevalent in competitive models today (Touvron et al., ).

We show that bilinear MLPs can be decomposed into functionally relevant, interpretable components by directly decomposing the weights, without using inputs. These decompositions reveal a low-rank structure in bilinear MLPs trained across various tasks. In summary, this paper demonstrates that bilinear MLPs are an interpretable drop-in replacement for ordinary MLPs in a wide range of settings. Our contributions are as follows:

- 1.
In section 3, we introduce several methods to analyze bilinear MLPs. One method decomposes the weights into a set of eigenvectors that explain the outputs along a given set of directions in a way that is fully equivalent to the layer’s original computations.
- 2.
In section 4, we showcase the eigenvector decomposition across multiple image classification tasks, revealing an interpretable low-rank structure. Smaller eigenvalue terms can be truncated while preserving performance. Using the eigenvectors, we see how regularization reduces signs of overfitting in the extracted features and construct adversarial examples.
- 3.
Finally, in section 5, we analyze how bilinear MLPs compute output features from input features, both derived from sparse dictionary learning (SDL). We highlight a small circuit that flips the sentiment of the next token if the current token is a negation (“not”). We also find that many output features are well-correlated with low-rank approximations. This gives evidence that weight-based interpretability can be viable in large language models.

## 2 Background

Throughout, we use conventional notation as in Goodfellow et al. (). Scalars are denoted by $s$, vectors by ${\bm{v}}$, matrices by ${\bm{M}}$, and third-order tensors by ${\bm{\mathsfit{T}}}$. The entry in row $i$ and column $j$ of a matrix ${\bm{M}}$ is a scalar and therefore denoted as $m_{ij}$. We denote taking row $i$ or column $j$ of a matrix by ${\bm{m}}_{i:}$ and ${\bm{m}}_{:j}$ respectively. We use $\odot$ to denote an element-wise product and $\cdot_{\text{axis}}$ to denote a product of tensors along the specified axis.

Defining bilinear MLPs.
Modern Transformers (Touvron et al., ) feature Gated Linear Units (GLUs), which offer a performance gain over standard MLPs for the same number of parameters (Shazeer, ). GLU activations consist of the component-wise product of two linear up-projections, ${\bm{W}}$ and ${\bm{V}}$, one of which is passed through a nonlinear activation function $\sigma$(Equation 1). We omit biases for brevity.

$$ $\displaystyle g({\bm{x}})$ $\displaystyle=({\bm{W}}{\bm{x}})\odot\sigma({\bm{V}}{\bm{x}})$ (1) $\displaystyle\text{GLU}({\bm{x}})$ $\displaystyle={\bm{P}}(g({\bm{x}}))$ $$

A bilinear layer is a GLU variant that omits the nonlinear activation function $\sigma$. Bilinear layers beat ordinary ReLU MLPs, and perform almost as well as SwiGLU on language modeling tasks (Shazeer, ). We corroborate these findings in Appendix G, and show that bilinear layers achieve equal loss when keeping training time constant and marginally worse loss when keeping data constant.

Interaction matrices and the bilinear tensor. A bilinear MLP parameterizes the pairwise interactions between inputs. One way to see this is by looking at how a single output $g(\textbf{x})_{a}$ is computed.

$$ $\displaystyle g({\bm{x}})$ $\displaystyle=({\bm{W}}{\bm{x}})\odot({\bm{V}}{\bm{x}})$ $\displaystyle g({\bm{x}})_{a}$ $\displaystyle=({\bm{w}}_{a:}^{T}{\bm{x}})\ ({\bm{v}}^{T}_{a:}{\bm{x}})$ $\displaystyle={\bm{x}}^{T}({\bm{w}}_{a:}{\bm{v}}_{a:}^{T}){\bm{x}}$ $$

We call the matrix ${\bm{w}}_{a:}{\bm{v}}_{a:}^{T}={\bm{B}}_{a::}$ an *interaction matrix* since it defines how each pair of inputs interact for a given output dimension $a$.

The collection of interaction matrices across the output axis can be organized into the third-order *bilinear tensor*, ${\bm{\mathsfit{B}}}$, with elements $b_{aij}=w_{ai}v_{aj}$, illustrated in Figure 1A. The bilinear tensor allows us to easily find the interaction matrix for a specific output direction ${\bm{u}}$ of interest by taking a product along the output axis, ${\bm{u}}\ \cdot_{\text{out}}{\bm{\mathsfit{B}}}$, equal to a weighted sum over the neuron-basis interaction matrices, $\sum_{a}u_{a}{\bm{w}}_{a:}{\bm{v}}_{a:}^{T}$.

Simplifications due to symmetry. Because an interaction matrix is always evaluated with two copies of the input ${\bm{x}}$, it contains redundant information that does not contribute to the activation of a layer. Specifically, any square matrix can be expressed uniquely as the sum of a symmetric and an anti-symmetric matrix.

$$ $\displaystyle{\bm{B}}_{a::}=\dfrac{1}{2}(\underbrace{{\bm{B}}_{a::}+{\bm{B}}_{a::}^{T}}_{{\bm{B}}_{a::}^{sym}})+\dfrac{1}{2}(\underbrace{{\bm{B}}_{a::}-{\bm{B}}_{a::}^{T}}_{{\bm{B}}_{a::}^{anti}})$ $$

However, evaluating an anti-symmetric matrix ${\bm{A}}$ with identical inputs yields 0 and can be omitted:

$$ $\displaystyle{\bm{x}}^{T}{\bm{A}}{\bm{x}}={\bm{x}}^{T}(-{\bm{A}}^{T}){\bm{x}}=-({\bm{x}}^{T}{\bm{A}}{\bm{x}})^{T}=0.$ $$

Therefore, only the symmetric part ${\bm{B}}_{a::}^{sym}$ contributes. From here on, we drop the $\cdot^{sym}$ superscript and assume the symmetric form for any interaction matrix or bilinear tensor ($b_{aij}=\frac{1}{2}(w_{ai}v_{aj}+w_{aj}v_{ai})$). Symmetric matrices have simpler eigendecompositions since the eigenvalues are all real-valued and the eigenvectors are orthogonal, by the spectral theorem.

Incorporating biases. If a bilinear layer has biases, we can augment the weight matrices to adapt our approach.
Given activations of the form $g({\bm{x}})=(W{\bm{x}}+{\bm{b}}_{1})\odot({\bm{V}}{\bm{x}}+{\bm{b}}_{2})$, define ${\bm{W}}^{\prime}=[{\bm{W}};{\bm{b}}_{2}]$, ${\bm{V}}^{\prime}=[{\bm{V}};{\bm{b}}_{2}]$, and ${\bm{x}}^{\prime}=[{\bm{x}},1]$. Then, $g({\bm{x}})=({\bm{W}}^{\prime}{\bm{x}}^{\prime})\odot({\bm{V}}^{\prime}{\bm{x}}^{\prime})$ in a bilinear layer with biases.
In subsection 4.3, we study a toy classification task using a model trained with biases, illustrating how biases can be interpreted using the same framework.
For the rest of our experiments, we used models without biases for simplicity, as it did not harm performance. See Appendix J for details.

## 3 Analysis methods

Since bilinear MLPs can be expressed in terms of a third-order tensor, ${\bm{\mathsfit{B}}}$, they can be flexibly analyzed using techniques from linear algebra, such as decompositions and transformations. The choice of analysis approach depends on what additional information, in terms of previously obtained input or output features, is provided.

### 3.1 Input / output features → \rightarrow direct interactions

If we have already obtained meaningful sets of features for the bilinear MLP’s inputs and outputs, for example, by training sparse autoencoders (SAEs), then we can directly study the interactions between these features. Using the feature dictionaries $D_{\text{in}}$ and $D_{\text{out}}$, we can transform the bilinear tensor into the feature basis via $\tilde{b}_{abc}=\sum_{ijk}d^{\text{\ out}}_{ai}\ b_{ijk}\ d^{\text{\ in}}_{bj}\ d^{\text{\ in}}_{ck}$. For a given set of sparse input and output activations, only a small subset of the interactions (with $a,b,c$ all active) will contribute, and the statistics of these active interactions can be studied.

If, instead, we want to determine the output feature activations from the inputs, we would use the output SAE’s encoder directions: $\tilde{b}_{abc}=\sum_{ijk}e^{\text{\ out}}_{ai}\ b_{ijk}\ d^{\text{\ in}}_{bj}\ d^{\text{\ in}}_{ck}$. In section 5, we use this approach to identify the top relevant interactions for features in a language model.

### 3.2 Output features → \rightarrow eigendecomposition

Figure: Figure 1: A) Two ways to represent a bilinear layer, via an elementwise product or the bilinear tensor. B) Diagram of the eigendecomposition technique. Multiplying the bilinear tensor by a desired output direction ${\bm{u}}$ produces an interaction matrix ${\bm{Q}}$ that can be decomposed into a set of eigenvectors ${\bm{v}}$ and associated eigenvalues $\lambda_{i}.$
Refer to caption: x1.png

Given a set of meaningful features for the MLP outputs, we can identify the most important input directions that determine the output feature activations. The output features could come from a sparse autoencoder, from the unembedding layer (demonstrated in section 4), or from a previous decompilation of the later layers.

The interaction matrix, ${\bm{Q}}={\bm{u}}\cdot_{\text{out}}{\bm{\mathsfit{B}}}$ for a given output feature ${\bm{u}}$ can be decomposed into a set of eigenvectors (Figure 1). Since ${\bm{Q}}$ can be considered symmetric without loss of generality (see section 2), the spectral theorem gives

$$ $\displaystyle{\bm{Q}}=\sum_{i}^{d}\lambda_{i}\ {\bm{v}}_{i}{\bm{v}}_{i}^{T}$ (2) $$

with a set of $d$ (rank of ${\bm{W}}$, ${\bm{V}}$) orthonormal eigenvectors ${\bm{v}}_{i}$ and real-valued eigenvalues $\lambda_{i}$. In the eigenvector basis, the output in the ${\bm{u}}$-direction is

$$ $\displaystyle{\bm{x}}^{T}{\bm{Q}}{\bm{x}}=\sum_{i}^{d}\underbrace{\lambda_{i}\ ({\bm{v}}_{i}^{T}{\bm{x}})^{2}}_{\text{activation for }{\bm{v}}_{i}}$ (3) $$

where each term can be considered the activation for the eigenvector ${\bm{v}}_{i}$. That is, the bilinear layer’s outputs are quadratic in the eigenvector basis.

The eigenvector basis makes it easy to identify any low-rank structure relevant to ${\bm{u}}$. The top eigenvectors by eigenvalue magnitude give the best low-rank approximation to the interaction matrix ${\bm{Q}}$ for a given rank. And since the eigenvectors diagonalize ${\bm{Q}}$, there are no cross-interactions between eigenvectors that would complicate the interpretation of their contributions to ${\bm{u}}$.

### 3.3 No features → \rightarrow higher-order SVD

If we have no prior features available, it is still possible to determine the most important input and output directions of ${\bm{\mathsfit{B}}}$ through a higher-order singular value decomposition (HOSVD). The simplest approach that takes advantage of the symmetry in ${\bm{\mathsfit{B}}}$ is to reshape the tensor by flattening the two input dimensions to produce a $d_{\text{out}}\times d_{\text{in}}^{2}$ shaped matrix and then do a standard singular value decomposition (SVD).
Schematically, this gives

$$ $\displaystyle{\bm{B}}_{\text{out},\text{in}\times\text{in}}=\sum_{i}\sigma_{i}\ {\bm{u}}^{(i)}_{\text{out}}\otimes{\bm{q}}^{(i)}_{\text{in}\times\text{in}}$ $$

where ${\bm{q}}$ can still be treated as an interaction matrix and further decomposed into eigenvectors as described above.

## 4 Image classification: interpreting visual features

We consider models trained on the MNIST dataset of handwritten digits and the Fashion-MNIST dataset of clothing images. This is a semi-controlled environment that allows us to evaluate the interpretability of eigenvectors on a nontrivial task. This section analyses a shallow feedforward network (FFN) consisting of an embedding projection, a bilinear layer, and a classification head; see Appendix F for details.

First, we qualitatively survey the eigenvectors and highlight the importance of regularization in feature quality. Second, we consider the consistency of eigenvectors across training runs and sizes. Third, we turn toward an algorithmic task on MNIST, where we compare the ground truth with the extracted eigenvectors. Lastly, we use these eigenvectors to construct adversarial examples, demonstrating their causal importance.

### 4.1 Qualitative assessment: top eigenvectors appear interpretable

The eigenvectors are derived using the unembedding directions for the digits to obtain interaction matrices that are then decomposed following subsection 3.2. We can visualize them by projecting them into the input space using the embedding weights. Because the activation of an eigenvector ${\bm{v}}$ with eigenvalue $\lambda_{i}$ is quadratic in the input, $\lambda({\bm{v}}^{T}{\bm{x}})^{2}$, the sign of the eigenvector ${\bm{v}}$ is arbitrary. The quadratic leads to XOR-like behavior where high overlap with an eigenvector’s positive regions (blue) *or* the negative regions (red)—but not both—leads to large activation magnitude, while the overall sign is determined by the eigenvalue (Figure 2A).

For MNIST, the top positive eigenvector for each output class emphasizes a curve segment specific to its digit or otherwise resembles a prototypical class image (Figure 2B). Top eigenvectors for FMNIST function as edge detectors, focusing on important edges for each clothing article, such as the leg gap for trousers. Edge detecting relies on the XOR-like behavior of the eigenvector’s quadratic activation.

Figure: (a)
Refer to caption: x2.png

Figure: Figure 3: The top four positive (top) and negative (bottom) eigenvectors for the digit 5, ordered from left to right by importance. Their eigenvalues are highlighted on the left. Only 20 positive and 20 negative eigenvalues (out of 512) are shown on the left images. Eigenvectors tend to represent semantically and spatially coherent structures.
Refer to caption: x4.png

Only a small fraction of eigenvalues have non-negligible magnitude (Figure 3). Different top eigenvectors capture semantically different aspects of the class. For example, in the spectrum for digit 5, the first two positive eigenvectors detect the 5’s horizontal top stroke but at different positions, similar to Gabor filters. The next two positive eigenvectors detect the bottom segment. The negative eigenvectors are somewhat less intuitive but generally correspond to features that indicate the digit is not a five, such as an upward curve in the top right quadrant instead of a horizontal stroke. In Appendix C, we study this technique towards explaining an input prediction. Details of the training setup are outlined in Appendix F while similar plots for other digits can be found in Appendix A.

Because we can extract features directly from model weights, we can identify overfitting in image models by visualizing the top eigenvectors and searching for spatial artifacts.
For instance, the eigenvectors of unregularized models focus on certain outlying pixels (Figure 4).
We found adding dense Gaussian noise to the inputs (Bricken et al., ) to be an effective model regularizer, producing bilinear layers with more intuitively interpretable features.
Increasing the scale of the added noise results produces more digit-like eigenvectors and results in a lower-rank eigenvalue spectrum (Appendix D).
These results indicate that our technique can qualitatively help uncover overfitting or other unwanted behavior in models.
Furthermore, it can be used to evaluate the effect of certain regularizers and augmentation techniques, as explored in Appendix B.

Figure: Figure 4: Top eigenvector for models trained with varying Gaussian input noise. For reference, the norm of an average digit is about 0.3; adding noise with a norm of 1 results in a heavily distorted but discernible digit. Finally, the test accuracy for each model is shown at the top.
Refer to caption: x5.png

### 4.2 Quantitative assessment: eigenvectors learn consistent patterns

One important question in machine learning is whether models learn the same structure across training runs (Li et al., ) and across model sizes (Frankle & Carbin, ). In this section, we study both and find that eigenvectors are similar across runs and behave similarly across model sizes. Furthermore, we characterize the impact of eigenvector truncation on classification accuracy.

Figure: (a)
Refer to caption: x6.png

Both the ordering and contents of top eigenvectors are very consistent across runs. The cosine similarities of the top eigenvector are between 0.8 and 0.9 depending on size (Figure 5). Generally, increasing model sizes results in more similar top eigenvectors. Further, truncating all but the top few eigenvectors across model sizes yields very similar classification accuracy. This implies that, beyond being consistently similar, these eigenvectors have a comparable impact on classification. In Appendix E, we further study the similarity of eigenvectors between sizes and show that retaining only a handful of eigenvectors results in minimal accuracy drops (0.01%).

### 4.3 Comparing with ground truth: eigenvectors find computation

To perform a ground-truth assessment of eigenvectors, we consider a task from a mechanistic interpretability challenge, where the goal was to determine the labeling function (training objective) from a model Casper (). Specifically, the challenge required reverse-engineering a binary image classifier trained on MNIST, where the label is based on the similarity to a specific target image. The model predicted ‘True’ if the input has high cosine similarity to this target or high similarity to the complement of that target (^1^11By complement, we mean one minus the grayscale value of the target.) and ‘False’ otherwise. This target is chosen as an instance of a ‘1’.

Previous work (Stefan Heimersheim, ) reverse-engineered this through a combination of methods, all requiring careful consideration and consisting of non-trivial insights. Furthermore, the methods required knowledge of the original dataset and a hint of what to look for. While our method does not work on the original architecture, we show that we do not require such knowledge and can extract the original algorithm from the weights alone.

We perform our decomposition on the output difference ($\text{True}-\text{False}$) since this is the only meaningful direction before the softmax. This consistently reveals one high positive eigenvalue; the rest are (close to) zero (Figure 6). The most positive eigenvector is sufficient for completing the task; it computes the exact similarity we want. If the input is close to the target, the blue region will match; if it is close to the complement, the red will match; if both are active simultaneously, they will somewhat cancel out. The remaining two eigenvectors are separated as they seem to overfit the data slightly; the negative eigenvector seems to penalize diagonal structures.

Contrary to other models, this task greatly benefited from including biases. This arises from the fact that the model must not only compute similarity but also make its binary decision based on a learned threshold. If no bias is provided, the model attempts to find quadratic invariances in the data, which don’t generalize well, especially given the important but sensitive role of this threshold in classification. Here, the bias (shown in the bottom corner of Figure 6) represents a negative contribution. The role of biases in bilinear layers is further discussed in Appendix J.

Figure: Figure 6: Eigenvalues and eigenvectors of a model trained to classify based on similarity to a target. The most important eigenvector (top-left) is a generalizing solution; the other features sharpen the decision boundary based on the training dataset. The latter features disappear with increased regularization. On the right, the target digit is shown along with the learned bias from the model.
Refer to caption: x8.png

### 4.4 Adversarial masks: general attacks from weights

To demonstrate the utility of weight-based decomposition, we construct adversarial masks for the MNIST model without training or any forward passes. These masks are added to the input, leading to misclassification as the adversarial digit. The effect is similar to steering, but the intervention is at the input instead of the model internals.

We construct the adversarial masks from the eigenvectors for specific digits. One complication is that the eigenvectors can have nontrivial cosine similarity with each other, so an input along a single eigenvector direction could potentially activate multiple eigenvectors across different digits. To help avoid this, we construct the mask ${\bm{m}}_{i}$ for a given eigenvector ${\bm{v}}_{i:}$ as the corresponding row of the pseudoinverse $({\bm{V}}^{+})_{i:}$ for a set of eigenvectors ${\bm{V}}$ (specifically the top 10 positive). In an analogy to key-value pairs, the pseudoinverses effectively act like keys that activate with more specificity than the eigenvectors themselves, since ${\bm{v}}_{j:}\cdot({\bm{V}}^{+})_{i:}=\delta_{ij}$.

Figure: Figure 7: Examples of an adversarial mask constructed from the given eigenvector along for models trained A) with Gaussian noise regularization (std 0.15) and B) without regularization. The average accuracy and the rate of misclassification as the adversarial digit show stronger effects for adversarial masks than random baselines. In B), the mask is only applied to the outer edge of pixels that are active on less than 1% of samples.
Refer to caption: x9.png

We construct an adversarial mask from an eigenvector for the digit 3 (Figure 7A). Even though the original eigenvector resembles the digit, the pseudoinverse-based mask does not (see Appendix K for more examples). The accuracy, averaged over masks from the top three eigenvectors, drops significantly more than the baseline of randomly permuting the mask, despite regularizing the model during training using dense Gaussian noise with a standard deviation of 0.15. The corresponding rise in misclassification indicates effective steering towards the adversarial digit.

Ilyas et al. () observe that adversarial examples can arise from predictive but non-robust features of the data, perhaps explaining why they often transfer to other models. Our construction can be seen as a toy realization of this phenomenon because the masks correspond to directions that are predictive of robust features but are not robust. We construct masks that only exploit the patterns of over-fitting found on the outer edge of the image for a model trained without regularization (Figure 7B). Since we can find this over-fitting pattern from the eigenvectors, in a general way, we can construct the mask by hand instead of optimizing it.

## 5 Language: finding interactions between SAE features

Each output of a bilinear layer is described by weighted pairwise interactions between their input features. Previous sections show that this can be successfully leveraged to trace between a bilinear layer’s inputs and outputs. Here, we turn towards tracing between latent feature dictionaries obtained by training sparse autoencoders (SAEs) on the MLP inputs or outputs for a 6-layer bilinear transformer trained on TinyStories (Eldan & Li, ) (see training details in Appendix F).

### 5.1 Sentiment negation circuit

We focus on using the eigendecomposition to identify low-rank, single-layer circuits in a bilinear transformer. We cherry-pick and discuss one such circuit that takes input sentiment features and semantically negates them. Unlike previous work on sparse feature circuits (Marks et al., ) that relies on gradient-based linear approximations, we identify nonlinear interactions grounded in the layer’s weights that contribute to the circuit’s computation.

The sentiment negation circuit computes the activation of two opposing output features in layer 4 (index 1882 and 1179) that form a fully linear subspace. The cosine similarity of their decoder vectors is -0.975. Based on their top activations, the output features activate on negation tokens (“not”, “never”, “wasn’t”) and boosts either positive sentiment tokens (“good”, “safe”, “nice”) or negative sentiment tokens (“bad”, “hurt”, “sad”), so we denote the two features as the *not-good* and the *not-bad* features respectively. See Appendix M for the top activations of all features mentioned.

Figure: Figure 8: The sentiment negation circuit for computing the not-good and not-bad output features. A) The interaction submatrix containing the top 15 interactions. B) The top interacting features projected onto the top positive and negative eigenvectors using cosine similarity, with the symbols for different clusters matching the labels in A. Clusters coincide with the projection of meaningful directions such as the difference in “bad” vs “good” token unembeddings and the MLP’s input activations for input tokens “[BOS] not”. C) The not-good feature activation compared to its approximation by the top two eigenvectors.
Refer to caption: x10.png

Focusing on the not-good feature, the top interactions for computing its activations resemble an AND-gate (Figure 8A). Input features that boost negative sentiment tokens (blue squares) have strong positive interactions with negation token features (green triangles), but both have negligible self-interactions. So both types of input features are needed to activate the not-good feature and flip the boosting from negative to positive sentiment. The one positive sentiment feature (orange downward triangle) has interactions with the opposite sign. The interactions shown are significantly larger than the typical cross-interactions with a standard deviation of 0.004 (Figure 24.

The eigenvalue spectrum has one large positive (0.62) and one large negative value (-0.66) as outliers (Figure 24). We can see the underlying geometry of the circuit computation by projecting the input features onto these eigenvectors (Figure 8). By itself, a positive sentiment feature (blue squares) would equally activate both eigenvectors and cancel out, but if a negation feature is also present, the positive eigenvector is strongly activated. The activation based on only these two eigenvectors, following Equation 3, has a good correlation (0.66) with the activation of the not-good feature, particularly at large activation values (0.76), conditioned on the not-good feature being active.

Figure: Figure 9: Activation correlations with low-rank approximations. A) Average correlation over output features computed over all inputs before the SAE’s non-linearity (TopK) is applied or only on inputs where the feature is active. B) The distribution of active-only correlations for approximations using the top two eigenvectors. C) Scatter plots for a random set of nine output features. Approximations use the top two eigenvectors.
Refer to caption: x11.png

### 5.2 Low-rank approximations of output feature activations

The top eigenvectors can be used to approximate the activations of the SAE output features using a truncated form of Equation 3. To focus on the more meaningful tail of large activations, we compute the approximation’s correlation conditioned on the output feature being active. For comparison, we also compute the approximation’s correlation with the output feature activations before the SAE’s nonlinearity (TopK) is applied. These correlations are generally lower because they are dominated by inputs with low activations.

We find that features are surprisingly low-rank, with the average active-only correlation starting at 0.67 for approximations by a single eigenvector and rising steadily with additional eigenvectors (Figure 9A). Most features (69%) have a high correlation ($>0.75$) even when approximated by just two eigenvectors (Figure 9B). Scatter plots for a random sample of features show that the low-rank approximation often captures the tail dependence well (Figure 9C). Overall these results suggest that the interactions that produce large output activations are low-rank, making their interpretability potentially easier.

## 6 Discussion

Summary.
This paper introduces a novel approach to weight-based interpretability that leverages the close-to-linear structure of bilinear layers. A key result is that, using an eigenvector decomposition, we can identify the most important input directions that explain the layer’s output along a given direction. The top eigenvectors are often interpretable, for example for MNIST they function as edge-detectors for strokes specific to each digit. The lack of element-wise nonlinearity in bilinear MLPs allows us to transform their weights into interaction matrices that connect input to output features and then extract the low-rank structure. In language models, we find that many SAE output features are well-approximated by low-rank interaction matrices, particularly at large activations. We highlighted one example of an extracted low-rank circuit that flips the sentiment of the next token if the current token is a negation (“not”). The behavior of this circuit can be easily understood in terms of the top eigenvectors, whereas finding a similar circuit in conventional MLPs would be more difficult. Overall, our results demonstrate that bilinear MLPs offer intrinsic interpretability that can aid in feature and circuit extraction.

Implications.
The main implication of our work is that weight-based interpretability is viable, even for large language models. Bilinear MLPs can replace conventional MLPs in transformers with minimal cost while offering intrinsic interpretability due to their lack of element-wise nonlinearities and close-to-linear structure.
Current circuit analysis techniques rely on gradient-based approximations (Syed et al., ; Marks et al., ) or use transcoders (Dunefsky et al., ) to approximate MLPs. Both approaches depend on an input dataset, potentially leading to poor performance out-of-distribution, and they may not fully capture the nonlinear computations in MLPs. In contrast, bilinear MLPs can be transformed into explicit feature interaction matrices and decomposed in a way fully equivalent to the original computations. Extracting interactions more directly from the weights should lead to better, more robust circuits. Weight-based interpretability may also offer better safety guarantees since we could plausibly prove bounds on a layer’s outputs based by quantifying the residual weights not captured in a circuit’s interactions.

Limitations. Application of our methods typically relies on having a set of meaningful output directions available. In shallow models, the unembedding directions can be used, but in deeper models, we rely on features derived from sparse autoencoders that are dependent on an input dataset. Another limitation is that, although the eigenvalue spectra are often low-rank and the top eigenvectors appear interpretable, there are no guarantees the eigenvectors will be monosemantic. We expect that for high-rank spectra, the orthogonality between eigenvectors may limit their interpretability. Applying sparse dictionary learning approaches to decompose the bilinear tensor may be a promising way to relax the orthogonality constraint and find interpretable features from model weights.

## Acknowledgements

We are grateful to Narmeen Oozeer, Nora Belrose, Philippe Chlenski, and Kola Ayonrinde for useful feedback on the draft. We are grateful to the [AI Safety Camp](https://aisafety.camp/) program where this work first started and to the [ML Alignment & Theory Scholars](https://https://www.matsprogram.org/) (MATS) program that supported Michael and Alice while working on this project. We thank CoreWeave for providing compute for the finetuning experiments. This research received funding from the Flemish Government under the ”Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

## Ethics statement

This paper proposes no advancements to the state-of-the-art in model capabilities. Rather, it provides new methods to analyze the internals of models to increase our understanding. The only potential misuse the authors envision is using this technique to leak details about the dataset that the model has learned more efficiently. However, this can be avoided by using this technique during safety evaluation.

## Reproducibility statement

We aspired to make this work as reproducible as possible. First, Appendix F (among others) aims to provide detailed and sufficient descriptions to independently recreate our training setups. Second, our code (currently public but not referenced for anonymity) contains separate files that can be used to generate the figures in this paper independently. We ensured setting seeds across training runs such that recreated figures would be equivalent. Third, all models that are compute-intensive to train, such as the SAEs and the LMs, will be shared publicly. Lastly, we will publish an interactive demo, which will allow independent analysis of the figures in Appendix A, Appendix B, and Appendix M in a way this document cannot.

## Appendix A Eigenspectra: showing eigenvectors across digits

The following are plots showing multiple positive and negative eigenvectors for certain digits. Positive features either tend to look for specific patterns in the target class (first eigenvector of 2, matching the bottom part) or tend to match an archetypal pattern (second eigenvector of 6, matching the whole digit). Negative eigenvectors tend to look for a specific part that would change the class of the digit. For instance, if the pattern highlighted by the first negative eigenvector of 4 were on, it would most likely be a 9.

Figure: Figure 10: eigenvectors for digit 2.
Refer to caption: x12.png

Figure: Figure 11: eigenvectors for digit 4.
Refer to caption: x13.png

Figure: Figure 12: eigenvectors for digit 6.
Refer to caption: x14.png

## Appendix B Regularization & augmentation: ablations & observations

Following the observation that regularization improves feature interpretability for image classifiers, we study several popular regularization and augmentation techniques. In summary, input noise sparsifies the features, while geometric transformations blur the features. Some popular techniques, such as dropout, have little impact on features.

### B.1 Regularization

Input noise has the largest impact on features from any of the explored techniques. Specifically, we found dense Gaussian noise (already depicted in Figure 4) to provide the best trade-off between feature interpretability and accuracy. We also considered sparse salt-and-pepper noise (blacking or whiting out pixels), which resulted in both lower accuracy and interpretability. Lastly, we explored Perlin noise, which is spatially correlated and produces smooth patches of perturbation. However, this performed worst of all, not fixing the overfitting.

Model noise adds random Gaussian noise to the activations. This had no measurable impact on any of our experiments. However, this may simply be because our models are quite shallow.

Weight decay generally acts as a sparsifier for eigenvalues but does not significantly impact the eigenvectors. This is extremely useful as it can zero out the long tail of unimportant eigenvalues, strongly reducing the labor required to analyze a model fully (more details in Appendix D).

Dropout did not seem to impact our models. Overfitting was still an issue, even for very high values ($>0.5$). We suspect this may change in larger or capacity-constrained models.

### B.2 Augmentation

Translation stretches features in all directions, making them smoother. The maximal shift (right) is about 7 pixels in each direction, which is generally the maximal amount without losing important information. Interestingly, translation does not avoid overfitting but rather results in smoother overfitting patches. High translation results in split features, detecting the same pattern in different locations (Figure 13). This also results in a higher rank.

Rotation affects the features in the expected manner. Since rotating the digit zero does not significantly impact features, we consider the digit 5, which has a mix of rotation invariance and variance. Again, it does not stop the model from learning overfitting patches near the edges without noise. These features become broader with increased rotation.

Blur does not significantly affect features beyond making them somewhat smoother. Again, it still overfits certain edge pixels in a blurred manner without noise.

Figure: Figure 13: Important eigenvectors for a model trained with high translation regularization (7 pixels on either side). Similar patterns manifest as multiple eigenvectors at different locations.
Refer to caption: x15.png

All these augmentations are shown separately in Figure 14. Combining augmentations has the expected effect. For instance, blurring and rotation augmentation yield smooth and curvy features.

Figure: Figure 14: Important eigenvectors for models with different hyperparameters.
Refer to caption: x16.png

## Appendix C Explainability: a small case study with eigenvectors

While this paper focuses on bilinear layers for interpretability, the proposed techniques can also be used for post-hoc explainability to understand what has gone wrong. Our explanations are generally not as human-friendly as other methods but are fully grounded in the model’s weights. This section explores explaining two test-set examples, one correctly classified and one incorrectly.

The figures are divided into three parts. The left line plots indicate the sorted eigenvector activation strengths for the digits with the highest logits. The middle parts visualize the top positive and negative eigenvectors for each digit. The right displays the input under study and the related logits.

The first example, a somewhat badly drawn five, results in about equal positive activations for the output classes 5, 6, and 8 (which all somewhat match this digit). The negative eigenvectors are most important in this classification, where class 5 is by far the least suppressed. This is an interesting example of the model correctly classifying through suppression.

The second example, a seven-y looking two, is actually classified as a 7. From looking at the top eigenvectors of the digit 2 (shown in Figure 10), we see that the more horizontal top stroke and more vertical slanted stroke activates the top eigenvector for the digit 7 more strongly than the 2-eigenvectors that look for more curved and slanted strokes. The negative eigenvectors are not very important in this incorrect classification.

Figure: Figure 15: Study of a correctly classified 5. The output is strongly influenced by negative eigenvectors, resulting in strong suppression for the other digits.
Refer to caption: x19.png

Figure: Figure 16: Study of a misclassified 2. The model mostly classifies twos based on the bottom line and top curve, which are both only partially present.
Refer to caption: x20.png

## Appendix D Sparsity: weight decay versus input noise

Throughout, we make the claims that input noise helps create cleaner eigenvectors and that weight decay results in lower rank; this appendix aims to quantify these claims. To quantify near-sparsity, we use $\left(L_{1}/L_{2}\right)^{2}$, which can be seen as a continuous version of the $L_{0}$ norm, accounting for near-zero values. We analyze both the eigenvalues, indicating the effective rank, and the top eigenvectors, indicating the effective pixel count.

As visually observed in Figure 4, this analysis (left of Figure 17) shows that input noise plays a large role in determining the eigenvector sparsity; weight decay does not. On the other hand, input noise increases the number of important eigenvectors while weight decay decreases it. Intuitively, input noise results in specialized but more eigenvectors, while weight decay lowers the rank.

Figure: (a)
Refer to caption: x21.png

## Appendix E Truncation & similarity: a comparison across sizes

This appendix contains a brief extension of the results presented in Figure 5.

Figure: (a)
Refer to caption: x23.png

## Appendix F Experimental setups: a detailed description

This section contains details about our architectures used and hyperparameters to help reproduce results. More information can be found in our code [currently not referenced for anonymity].

### F.1 Image classification setup

The image classification models (section 4) in this paper consist of three parts: an embedding, the bilinear layer, and the head/unembedding, as shown in Figure 19. The training hyperparameters are found in Table 1. However, since the model is small, these parameters (except input noise; Appendix B) do not affect results much.

Figure: Figure 19: The architecture of the MNIST model.

**Table 1: Training setup for the MNIST models, unless otherwise stated in the text.**
| MNIST Training Parameters |  |
| --- | --- |
| input noise norm | 0.5 |
| weight decay | 1.0 |
| learning rate | 0.001 |
| batch size | 2048 |
| optimizer | AdamW |
| schedule | cosine annealing |
| epochs | 20-100 |

### F.2 Language model setup

The language model used in this paper is a 6-layer modern transformer model (Touvron et al., ) where the SwiGLU is replaced with a bilinear MLP (Figure 20). The model has about 33 million parameters. The training setup is detailed in Table 2. As the training dataset, we use a simplified and cleaned version of TinyStories (Eldan & Li, ) that remedies the following issues.

- Contamination: About 20% of stories are exact duplicates (in both train and test).
- Corruption: Some stories contain strange symbol sequences akin to data corruption.

Furthermore, we use a custom interpretability-first BPE tokenizer. The tokenizer is lower-case only, splits on whitespace, and has a vocabulary of only 4096 tokens.

Figure: Figure 20: Simplified depiction of a bilinear transformer model. A model dimension of 512 is used, and an expansion factor of 4 (resulting in 2048 hidden dimensions in the MLP).
Refer to caption: x27.png

**Table 2: Model architecture and training setup. Omitted parameters are the HuggingFace defaults.**
| Transformer Training Parameters |  |
| --- | --- |
| weight decay | 0.1 |
| batch size | 512 |
| learning rate | 0.001 |
| optimizer | AdamW |
| schedule | linear decay |
| epochs | 5 |
| tokens | $\pm$ 2B |
| initialisation | gpt2 |

### F.3 Sparse autoencoder setup

All discussed SAEs use a TopK activation function, as described in Gao et al. (). We found $k=30$ to strike a good balance between sparseness and reconstruction loss. This paper studies quite narrow dictionaries (4x expansion) for simplicity. The exact hyperparameters are shown in Table 3, and the attained loss added (Equation 4) across layers is shown in Figure 21.

$$ $L_{added}=\dfrac{L_{patch}-L_{clean}}{L_{clean}}$ (4) $$

Figure: Figure 21: Loss added for the mlp_out and resid_mid SAEs across layers.
Refer to caption: x28.png

**Table 3: SAE training hyperparameters.**
| SAE Training Parameters |  |
| --- | --- |
| expansion | 4x |
| k | 30 |
| batch size | 4096 |
| learning rate | 1e-4 |
| optimizer | AdamW |
| schedule | cosine annealing |
| tokens | $\pm$ 150M |
| buffer size | $\pm$ 2M |
| normalize decoder | True |
| tied encoder init | True |
| encoder bias | False |

## Appendix G Bilinear transformers: a loss comparison

While experiments on large models show bilinear layers to only marginally lag behind SwiGLU (Shazeer, ), this section quantifies this accuracy trade-off through compute efficiency. We performed our experiments using a 6-layer transformer model trained on TinyStories. For these experiments, we use $\text{d_model}=512$ and $\text{d_hidden}=2048$, resulting in roughly 30 million parameters. However, we have found these results to hold across all sizes we tried.

**Table 4: The loss of language models with varying MLP activation functions. Bilinear layers are 6% less data efficient but equally compute efficient.**
|  | Bilinear | ReGLU | SwiGLU |
| --- | --- | --- | --- |
| constant epochs | 1.337 | 1.332 | 1.321 |
| constant time | 1.337 | 1.337 | 1.336 |

Considering the data efficiency (constant epochs), both SwiGLU and ReGLU marginally beat the bilinear variant. Concretely, SwiGLU attains the same final loss of the bilinear variant in 6% less epochs. On the other hand, when considering compute efficiency (constant time), we see that these differences vanish (^2^22An improvement in implementation efficiency, such as fusing kernels, may change this fact.). Consequently, if data is abundant, there is little disadvantage to using bilinear layers over other variants.

## Appendix H Finetuning: your transformer is secretly bilinear

Many state-of-the-art open-source models use a gated MLP called SwiGLU (Touvron et al., ). This uses the following activation function $\text{Swish}_{\beta}(x)=x\cdot\text{sigmoid}(\beta x)$. We can vary the $\beta$ parameter to represent common activation functions. If $\beta=1$, that corresponds to SiLU activation, used by many current state-of-the-art models. $\beta=1.7$ approximates a GELU and $\beta=0$ is simply linear, corresponding to our setup. Consequently, we can fine-tune away the gate by interpolating $\beta$ from its original value to zero. This gradually converts an ordinary MLP into its bilinear variant.

To demonstrate how this approach performs, we fine-tuned TinyLlama-1.1B, a 1.1 billion-parameter transformer model pretrained on 3 trillion tokens of data, using a single A40 GPU. For simplicity, we trained on a slice of FineWeb data. Due to computing constraints, we only tried a single schedule that linearly interpolates towards $\beta=0$ during the first 30% (120M tokens) and then fine-tunes for the remaining 70% (280M tokens). We compare this to a baseline that does not vary $\beta$ during fine-tuning, corresponding to continued training. We use this baseline to compensate for the difference in the pretraining distribution of TinyLlama (consisting of a mixture of RedPajama and StarCoder data).
This shows that this fine-tuning process increases the loss by about (0.05) but seems to benefit from continued training (Figure 22). We plan to extend this result with a more thorough search for an improved schedule, which will probably result in a lower final loss. We also expect longer training runs to close to gap even further.

Figure: Figure 22: Comparison of fine-tuning versus a baseline over the course of 400M tokens. The loss difference is noticeable but decreases quickly with continued training.
Refer to caption: x29.png

## Appendix I Tensor decompositions: extracting shared features

### I.1 Re-expressing the bilinear tensor

Given a complete or over-complete set of $m$ ${\bm{u}}$-vectors, we can re-express ${\bm{\mathsfit{B}}}$ in terms of the eigenvectors, which amounts to a change of basis. To avoid multiple contributions from similar ${\bm{u}}$-vectors, we have to use the pseudo-inverse, which generalizes the inverse for non-square matrices.
Taking the ${\bm{u}}$-vectors as the columns of ${\bm{U}}$, the pseudo-inverse ${\bm{U}}^{+}$ satisfies ${\bm{U}}{\bm{U}}^{+}=I$, as long as ${\bm{U}}$ is full rank (equal to $d$). Then

$$ $\displaystyle{\bm{\mathsfit{B}}}$ $\displaystyle=\sum^{m}_{k}{\bm{u}}^{+}_{:k}\otimes{\bm{Q}}_{k}$ (5) $\displaystyle=\sum^{m}_{k}\sum_{i}^{d}\lambda_{\{k,i\}}\ {\bm{u}}^{+}_{:k}\otimes{\bm{v}}_{\{k,i\}}\otimes{\bm{v}}_{\{k,i\}}$ (6) $$

where ${\bm{u}}^{+}_{:k}$ are the rows of $U^{+}$ and $Q_{k}=\sum_{i}\lambda_{\{k,i\}}{\bm{v}}_{\{k,i\}}\otimes{\bm{v}}_{\{k,i\}}$ is the eigendecomposition of the interaction matrix corresponding for ${\bm{u}}{k:}$. We can then recover the interaction matrices from ${\bm{Q}}_{k}={\bm{u}}_{k:}\cdot_{\text{out}}{\bm{\mathsfit{B}}}$ using the fact that
${\bm{u}}_{k:}\cdot{\bm{u}}^{+}_{:k^{\prime}}=\delta_{kk^{\prime}}$ (Kronecker delta). Note that the eigenvectors within a single output direction $k$ are orthogonal but will overlap when comparing across different output directions.

## Appendix J Bilinear layers: a practical guide

Bilinear layers are inherently quadratic; they can only model the importance of pairs of features, not single features. Interestingly, we haven’t found this to be an issue in real-world tasks. However, linear structure is important for some toy tasks and, therefore, merits some reflection. Without modification, bilinear layers will model this linear relation as a quadratic function. To resolve this, we can add biases to the layer as follows: $\text{BL}({\bm{x}})=({\bm{W}}{\bm{x}}+{\bm{b}})\odot({\bm{V}}{\bm{x}}+{\bm{c}})$. In contrast to ordinary layers, where the bias acts as a constant value, here it acts as both a linear and a constant value. This becomes apparent when expanded:

$$ $\text{BL}({\bm{x}})={\bm{W}}{\bm{x}}\odot{\bm{V}}{\bm{x}}+({\bm{c}}{\bm{W}}{\bm{x}}+{\bm{b}}{\bm{V}}{\bm{x}})+{\bm{c}}{\bm{b}}$ $$

We disambiguate by calling the terms ‘interaction’, ‘linear’, and ‘constant’. Theoretically, this is very expressive; all binary gates and most mathematical operations can be approximated quite well with it. In practice, the training process often fails to leverage this flexibility and degenerates to using quadratic invariances instead of learned constants.

## Appendix K Adversarial masks: additional figure

Figure: Figure 23: More examples of adversarial masks constructed from specific eigenvectors for models with A) no regularization, B) Gaussian noise regularization with std 0.15, and C) Gaussian noise regularization with std 0.3.
Refer to caption: x30.png

## Appendix L Language: further details for feature circuits

Figure: Figure 24: A) Histogram of self-interactions and cross-interactions. B) The eigenvalue spectrum for the sentiment negation feature discussed in section 5. The dashed red line gives spectrum for a random symmetric matrix with Gaussian distributed entries with the same standard deviation.
Refer to caption: x31.png

## Appendix M Input features of the negation feature

**Table 5: Output SAE features in the negation feature discussed in section 5.**
| Output Feature (1882): not good |
| --- |
| lily wanted to climb the ladder , but her mommy told her to stay away from it because it was not safe . lily listened to her |
| . they looked everywhere . the train was lost . the train stayed in the hole , and it was not happy . the end . [EOS] |
| the man was angry . he grabbed sue , who screamed and cried . sue was very troubled . she never got to watch the play and |
| , and lily still couldn ’ t find her beloved jacket . she felt so sad that she stopped playing with her friends and stopped |
| big dog , and max said , ” i didn ’ t like the big dog . he wasn ’ t honest like me . ” lily |
| sick . they called the doctor , who said he was very i ll . sadly , the small boy never recove red and he passed |
| Output Feature (1179): not bad |
| me to the park , car ! ” the car didn ’ t respond , but lily didn ’ t mind . she was just happy |
| was a bee ! susie was afraid at first , but mommy explained that the bee was friendly and would not hurt her . susie was curious |
| was proud of himself . he was scared to collect the lemons , but he didn ’ t give up . he collected all |
| became friends . they went on many adventures together . the bear was still clumsy , but he didn ’ t mind . he was happy to |
| made many new friends and had lots of fun . the little frog was always very busy , but he never forgot the cow ’ s advice |
| ” tom says . he gives him a hug . the snowman is cold and wet , but tom does not mind . he likes the snowman |

**Table 6: SAE features that contribute to the negation feature discussed in section 5.**
| Input Feature (326): crashing and breaking |
| --- |
| but , as the lady was carrying the birdie across the road , a car didn ’ t see them and hit them both . the birdie |
| takes an egg and taps it on the edge of the bowl . but he tap s too hard and the egg breaks in his hand |
| and faster . he feels dizzy from the speed . he does not see the rock in his way . he hit s the rock and his |
| Input Feature (1376): dangerous actions |
| ” no , sweetie , you can ’ t touch the fish . they are not for touch ing . they might bite you or get scared |
| with them ! ” tom was worried . he said ” lila , we should not touch the balloons . they are not our s . maybe |
| was worried and said , ” no , sam . you mustn ’ t go in ! it would be too dangerous . ” but sam |
| Input Feature (1636): nervous / worried |
| big adventure . he decided to take a trip to the tip . joe was so excited , but he was a little worried too . he |
| two people doing yoga . joe was fascinated and he clapped with excitement . but the people weren ’ t happy . one |
| forest and soon he spotted a lovely tree with lots of delicious fruit . he was excited , but he was also a bit nervous . he |
| Input Feature (123): negative attribute |
| [BOS] once upon a time there was a poor rug . it was old and fa de d , but it |
| [BOS] once upon a time , there was a beach . the beach was very filthy . it had a lot of trash on it |
| to say hello . she ran to the door and knocked on it , but it was filthy and covered in dirt . the new kid opened |

**Table 7: SAE features that contribute to the negation feature discussed in section 5 (continued).**
| Input Feature (990): a bad turn |
| --- |
| . she went up to him and asked him why he was crying . the little boy said , ” i lost my li p bal m |
| a little boy crying . she went to him and asked why he was sad . the little boy said he lost his toy and didn ’ |
| h . spot asked , ” why are you sad , little bird ? ” the bird replied , ” i lost my favorite toy in the |
| Input Feature (1929): inability to do something / failure |
| ’ t listen to him . max tried again and again , but the water wouldn ’ t answer . he said please and asked nicely , |
| how . he tried moving it , pushing it , pull ing it . nothing seemed to work . he was getting frustrated . he asked his |
| couldn ’ t reach it on the counter . she tried and tried , but it was too high . lily ’ s mom came into the |
| Input Feature (491): body parts |
| a bucket and a shovel and start to fill the bucket with soil . they press the soil hard with their hands . they turn the bucket |
| day , he hopped up to a magazine that was lying on the ground . he poked it with his furry foot and it felt hard |
| spider on the ceiling . she wanted to catch it , so she reached up to pinch it with her fingers . but the spider was |
| Input Feature (947): bad ending (seriously) |
| the doctor , who said he was very i ll . sadly , the small boy never recove red and he passed away . [EOS] [EOS] |
| ’ t make it out of the water . her family was very sad and missed her very much . they remembered how jolly and happy she |
| was very sad . his friends could not help him . the hippo stayed stuck with the disgusting wrap forever . [EOS] |
| Input Feature (882): being positive |
| , there was a musician who wandered through the countryside . he was quite weak , but he was determined to make it to the |
| finger ! she yelped and quickly pulled away . the little girl was a bit scared but she was also brave . she did not |
| was tall and had long , bony fingers . the little boy was a bit scared , but he was also very curious . he went |
| Input Feature (240): negation of attribute |
| ran to it and asked the others what it was . he was so excited to know that it wasn t just a strange big box it |
| explore a big tunnel and see what he might find inside . joe s mum told him that it wasn t a safe tunnel and he should |
| go swimming in the ocean . he asked his mom to take him . his mom explained that it was not safe to swim alone and she |
| Input Feature (766): inability to perform physical actions |
| but one day , he got sick and started to suffer . his body hurt and he couldn ’ t play with his friends anymore . |
| [BOS] one day , a pretty bird hurt its wing . it could not fly . a kind girl named |
| to play with him too . one day , the puppy fell down and hurt his leg . he could not play and run like before . |
| Input Feature (1604): avoiding bad things |
| it in a more careful way . so he started playing more carefully , making sure that he wouldn ’ t get too dirty . he kept |
| shelf . she smiled and said yes . jimmy was careful to pull it off the shelf so it would not break . he proudly showed it |
| jane noticed the ice was very icy . mary suggested they wear gloves , so that the cold would not hurt their hands . jane agreed |
| Input Feature (1395): positive ending |
| . the old man watched the pony with a smile on his face . the pony was happy and no longer scared . she had found a |
| and jane seeing a deer in the woods and choosing to stay and appreciate it , instead of trying to catch it . [EOS] |
| . sam was happy , and so was the ball . now , sam and the ball could play together without mud . they rolled and bounced |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Check weaknesses.
soundness: 4
strengths: ORIGINALITY:
This work distinguishes itself from existing approaches to interpretable neural networks by using a unique characteristic of bilinear MLPs. While previous works have utilized bilinear MLP networks, this paper uniquely demonstrates how to exploit their inherent differences from other MLPs to achieve interpretable neural networks.

SIGNIFICANCE:
Interpretable AI is crucial for understanding how AI models encode information. 
This work offers a new framework for interpretable neural networks, with the potential to significantly advance the field.

QUALITY:
The quality of this work is good. It provides relevant information to support its claims. The experimental results sufficiently support the main claims in this paper.

CLARITY:
This paper is well-written. It provides enough information required to put this work into proper perspective. It is also well organized and provides enough background information needed to understand the main components of this work.
summary: This work provides an in-depth analysis of Bilinear Multilayer Perceptrons (MLPs). 
Bilinear MLPs use bilinear transformations to achieve nonlinearity for function approximation.
The analysis focuses on the spectral decomposition of the bilinear MLP weights. 
This reveals the low-rank structure of the weights and provides insight into crafting adversarial samples, overfitting, and so on. 
The low-rank structure is interpretable and shows that components corresponding to small eigenvalues can be removed while preserving performance.
The work provides an explainable framework, which would be difficult to achieve without bilinear weights because of the use of nonlinear activation functions.
The paper provides experimental results to support its claims.
weaknesses: This is a minor suggestion:

This paper introduces the use of sparse autoencoders in Section 3.1. 
While the stated limitations offer justification for this approach, a more detailed explanation of the role of sparse autoencoders in Section 3.1 would enhance the clarity and comprehensiveness of the work.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: None.
soundness: 3
strengths: Originality:  While the ideas of using bilinear layers and use of eigenanalysis on the corresponding reduced tensor are borrowed from past work (in particular, “A technical note on bilinear layers for interpretability), the precise demonstrations are new, to my knowledge. 

Clarity: The paper is well-written, making it easy to follow (even for a reader like me, less familiar with this area of work). The Discussion section well-summarizes the contributions, contextualizes the implications, and identifies key limitations.

Quality: The work seems technically sound, with a reasonable span of experiments to make the point that’s being made. 

Significance: The results are certainly intriguing, and help further the case for adoption of bilinear layers in modern models that are heavily MLP-based.
summary: The submission picks up on preliminary ideas developed in “A technical note on bilinear layers for interpretability” (Sharkey, arXiv, 2023) where the idea of using bilinear layers in place of traditional MLPs was promoted to enable forms of interpretability. This is enables by the fact that a bilinear transform of an input vector can be expressed as linear operations with a third-order tensor. This submission follows through by demonstrating several ways in which the tensor can be analyzed with tools from eigen-analysis because input vectors representing features act upon the tensor to yield a matrix (and one can further only consider the symmetric component of this matrix, which has a real eigen-spectrum). 

Examples showcased are: 
* Interpretability for image classification with a shallow feed-forward network with one hidden bilinear layer (MNIST and FMNIST)
    * Eigenvectors of the last layer when projected back into input space visually resemble class prototypes or key features
    * Top eigenvectors are consistent across training runs and model sizes
    * Success on the mechanistic interpretability challenge (Casper, 2023)
    * Successful adversarial masks created from leading eigenvectors for a chosen adversarial class
* Identifying interactions between sparse autoencoder (SAE) based features in a small language transformer model
    * An example of identifying a sentiment negation circuit by projecting input features onto leading (both positive and negative eigenvalues) eigenvectors
    * Pointing out that SAE features tend to be low-rank in terms of the interaction matrix, suggesting promise for future work in circuit discovery in such models
weaknesses: The illustrations in the submissions are some fairly reasonable demonstrations of the interpretability-advantages of networks with bilinear layers. However, these ideas may not be mature enough, hence the lack of robust demonstrations at a broader range of tasks, models, and circuits. Being able to showcase such range would significantly strengthen the influence of the work in the nearer future.

Perhaps some of the existing problems can be solved with advances in discovering semantic “output directions” as part of the method. Even if the top eigenvectors may correspond to distributed semantics, further work to enable interpretation of such semantics would also be significant.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: Can you justify the use of the HOSVD as used? Two concerns: 1) The tensor represents a non-linear function. 2) If removed all but but the largest singular values of B, would the resulting tensor functionally approximate B? Also, what were the results of the SVD analysis, what section is that in?


Minor Comment:
Giving some dimensions on W,V could help with the presentation.
soundness: 3
strengths: Good exhibition of content in all sections, both setup and presentation of results
Effective choice of experiments with convincing examples and helpful visualization. 
Work is significant: Presents analysis using well understood methods to produce precise, understandable and actionable interpretations of models with competitive performance.
summary: The paper gives several methods to interpret bilinear multilayer perceptron networks (BMLP). BMLP's use a bilinear activation function and produce comparable results to other activation functions. By posing the bilinear leayrs as tensor operations, the author is able to apply direct interaction, eigendecomposition, and SVD analysis. Using these, the author provides an analysis of internal features, construct low-rank approximations of a model, and constructs adversarial examples. Overall, the paper shows the viability of an interpretable BMLPs.
weaknesses: I did not notice much analysis using SVD, perhaps I missed it.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: - Could the authors introduce the dimensionality of the variables when defining them? I and perhaps also other readers would find that quite useful for understanding the equations.

- In line 121, is the $W$ in $g(x)=(Wx+b_1)\odot (Vx+b_2)$ supposed to be bold?

- I'm confused by the $d_{bj}^{in}$ in line 138 and the $d_{in}$ in line 189. Are they the different variables?

- In line 252, I can see that adding dense Gaussian noise to the inputs produces bilinear layers with more intuitively interpretable features. But why is it an effective regularizer?

- Is the bilinear MLP related to the gated linear network literature [1]?

  [1] Veness et al (2021). Gated linear networks. https://arxiv.org/abs/1910.01526
soundness: 3
strengths: - This paper is carefully composed. Though there are places that can be further clarified (as mentioned below), the overall structure and logic is good.
- A variety of experiments are carried out to demonstrate the usefulness of the proposed approach. The bilinear MLP does seem to provide interesting findings and have potential in the realm of mechanistic interpretability.
summary: This paper proposes to use bilinear MLP as a novel approach for weight-based interpretability. The bilinear MLP and its analysis method are introduced first. Then a handful of experiments are presented to demonstrate the usefulness of this approach, including visual feature interpretation in image classification and feature interactions in language tasks.
weaknesses: - I couldn't understand what Figure 2 is demonstrating. Firstly, the caption refers to subfigure A, B while the figures do not have A, B labels. Secondly, do you mean that the input images with "$|$" or "$-$" patterns of exactly the same size and position will have high activation? I imagine that when flattening images to vectors, an image with many small "$-$" edges isn't well aligned with the eigenvector that is one big "$-$" in the middle of the image. If that's the case, why are the "$|$" or "$-$" eigenvectors edge detectors?
- I feel that Section 4 and 5 can be clearer if the authors always specify "eigenvectors of *what matrix*" when saying eigenvectors. Section 3 introduced three different methods. So I find the terms like "decomposition" and "eigenvectors" ambiguous, even after I tried to trace back to the concepts introduced in Section 3. The term "eigenvector of a model" is also ambiguous, since models don't really have eigenvectors.

Meta-review: The paper explores Bilinear MLPs for their interpretability in deep learning. Bilinear layers are proposed as interpretable alternatives to current functions, offering a viable framework for understanding deep-learning models. The reviewers find the contribution interesting, however I do believe a much more detailed related work should be included in the camera-ready. Currently, there is no discussion about bilinear layers, despite their rich history in both ML and vision, or in fact the tensors and tensor decompositions. I hope the reviewers can rectify the issue - there are multiple papers on the topic and the authors can find even multiple review papers including summarized developments, e.g. the recent [1]. 

[1] Tensor methods in computer vision and deep learning.

Decision: Accept (Spotlight)