Bilinear MLPs enable weight-based mechanistic interpretability
Authors: Michael T Pearce
Thomas Dooms
Alice Rigg
Jose Oramas
Lee Sharkey
Venue: ICLR 2025 Spotlight (2025)
Keywords: interpretability
mechanistic interpretability
bilinear
feature extraction
weight-based
eigenvector
eigendecomposition
tensor network
Primary area: interpretability and explainable AI
TL;DR: The close-to-linear structure of bilinear MLPs enables weight-based analysis that reveals interpretable low rank structure across multiple modalities.

Abstract: A mechanistic understanding of how MLPs do computation in deep neural net-
works remains elusive. Current interpretability work can extract features from
hidden activations over an input dataset but generally cannot explain how MLP
weights construct features. One challenge is that element-wise nonlinearities
introduce higher-order interactions and make it difficult to trace computations
through the MLP layer. In this paper, we analyze bilinear MLPs, a type of
Gated Linear Unit (GLU) without any element-wise nonlinearity that neverthe-
less achieves competitive performance. Bilinear MLPs can be fully expressed in
terms of linear operations using a third-order tensor, allowing flexible analysis of
the weights. Analyzing the spectra of bilinear MLP weights using eigendecom-
position reveals interpretable low-rank structure across toy tasks, image classifi-
cation, and language modeling. We use this understanding to craft adversarial
examples, uncover overfitting, and identify small language model circuits directly
from the weights alone. Our results demonstrate that bilinear layers serve as an
interpretable drop-in replacement for current activation functions and that weight-
based interpretability is viable for understanding deep-learning models.

Published as a conference paper at ICLR 2025
BILINEAR MLPs ENABLE WEIGHT-BASED MECHANISTIC INTERPRETABILITY
Michael Pearce"
Thomas Dooms'
Alice Rigg Independent ri rigg.al al ce00 gmail com
Independent pea @
University of Antwerp
thomas. thomas.dooms@ uantwerpen.bo
& alumni. st stanford.edt
Jose Oramas
Lee Sharkey
University of Antwerp, sqIRL/DLabb jose. www.asuuant werpen. be
Apollo Research
lee@apolloreseerrhh.dd
ABSTRACT
A mechanistic understanding how MLPs do computation in deep neural net- works remains elusive. Current interpretability work can extract features from hidden activations over an input dataset but generally cannot explain how MLP weights construct features. One challenge is that element-wise nonlincarities introduce higher-order interactions and make difficult to trace computations through the MLP layer. In this paper, analyze bilinear MLPs, type of Gated Linear Unit (GLU) without any element-wise nonlinearity that neverthe- less achieves competitive performance. Bilinear MLPs can be fully expressed in terms of linear operations using third-order tensor, allowing flexible analysis of the weights. Analyzing the spectra of bilinear MLP weights using eigendecom- position reveals interpretable low-rank structure across toy tasks, image classifi- cation, and language modeling. We use this understanding to craft adversarial examples, uncover overfitting, and identify small language model circuits directly from the weights alone. Our results demonstrate that bilinear layers serve as an interpretable drop-in replacement for current activation functions and that weight- based interpretability is viable for understanding deep-learning models.
INTRODUCTION
Multi-layer perceptrons (MLPs) important component of many deep learning models, in- cluding transformers (Vaswani et al. 2017). Unfortunately, element-wise nonlinearities obscure the relationship between weights, inputs, and outputs, making it difficult to trace neural network's decision-making process, Consequently, MLPs have previously been treated as undecomposable components in interpretability research (Elhage al. 202T).
While early mechanistic interpretability literature explored neural network weights (Olah et al.,
2017 2020 Voss et al., 2021, Elhage al., (02I), activation-baser approaches dominate contem- porary research (Petsiuk et al. 2018 Ribeiro et al. 2016 Simonyya et al. 2014 Montavon et al.) 2018 In particular, most recent studies on transformers use sparse dictionary learning (SDL) to decompose latent representations into an overcomplete basis of seemingly interpretable atoms (Cun-) ningham et al. 2024 Bricken et al. 2203bb Marks et al. 2024: Dunefsky et al.) 2024). However, SDL-based approaches only describe which features present, not how they are formed or what their downstream effect is. Previous work has approximated interactions between latent dictionaries using linear and gradient-basee attribution (Marks et al. 2024 Ge et al. 2024), but these approaches offer weak guarantees of generalization. To ensure faithrulness, ideally, we would be able to capture nonlinear feature interactions in circuits that are grounded in the model weights.
"qqual contribution
Code at: http://gthhh..oomodommm//iieeeet

---

Published as a conference paper at ICLR 2025
One path better circuit discovery is to more inherently interpretable architectures. Previous work constrains the model into using human-underssanddbble components or concepts (Koh et al. 2020 Chen et al. 2019), but this typically requires labeled training data for the predefined con- cepts and involves a trade-off in accuracy compared to learning the best concepts for performance (Henighan et al. 2023). Ideally, we could more easily extract and interpret the concepts that mod- els naturally learn rather than force the model to use particular concepts. To this end. Sharkey (2023) suggested that bilinear layers Lin al. (2015); Li et al. (2017); Chrysos al. (2021) of the form ((x) (Wa x) @ (Vz) are intrinsically interpretable because their computations can be expressed terms of linear operations with third order tensor. This enables the use of tensor or matrix decompositions to directly understand the weights. Moreover, bilinear layers outperform ReL-U-based transformers language modeling (Shazeer 2020) and have performance only slightly below SwiGLU, which is prevalent in competitive models today (Touvron et al. 2023).
Tensor decompositions have long been studied in machine learning (Cichocki et al. 2015 Panagakis et al. 2021 Sidiropoulos et al., 2017) where most applications are based on an input dataset. Using decompositions to extract features directly from the weights of (ensor-based models is less explored. Here, we show that bilinear MLPs can be decomposed into functionally relevant, interpretable com- ponents by directly decomposing the weights, without using inputs. These decompositions reveal low-rank structure in bilinear MLPs trained across various tasks. In summary, this paper demon- strates that bilinear MIPs are an interpretable drop-in replacement for ordinary MLPs in a wide range of settings. Our contributions are as follows:
1. In section 3] we introduce several methods to analyze bilinear MLPs. One method decom- poses the weights into a set of eigenvectors that explain the outputs along a given set of directions in way that is fully equivalent to the layer's s original computations.
2. In In|section 4]
showcase the cigenvector decomposition across multiple image classifica- tion tasks, revealing an interpretable low-rank structure. Smaller eigenvalue terms can be truncated while preserving performance. Using the eigenvectors, see how regularization reduces signs of overfitting in the extracted features and construct adversarial examples.
3. Finally, in section we analyze how bilinear MLPs compute output features from input features, both derived from sparse dictionary learning (SDL). We highlight a small circuit that flips the sentiment of the next token if the current token is negation ("not"). We also find that many output features are well-correlated with low-rank approximations. This gives evidence that weight-based interpretability be viable in large language models.
2 BACKGROUND
Throughout, we use conventional notation as in Goodfellow et al. (2016). Scalars are denoted by s, vectors by v, matrices by M, and third-order tensors by T. The entry in row i and column j of a matrix M is scalar and therefore denoted as mij- We denote taking row i column of matrix by m. and my respectively. We use @ to denote an element-wise product and "axis to denote product of tensors along the specified axis.
Defining bilinear MLPs. Modern Transformers (Touvron et al. 2023) feature Gated Linear Units (GLUs), which offer a performance gain over standard MLPs for the same number of parameters Shazeerr 2020 Dauphin et al.) 2017). GLU activations consist of the component wise product of two linear up up-orojections of size [Chaddon: dimput).. W and V. one of which is passed through nonlinear activation function o (EEquatin 1p. The hidden activations g(x)) then pass through down- projection P of size (domtput; dhidsee). We omit biases for brevity.
g(w) = (Wa)Co GLU(x) = P(g(z))
bilinear layer is a GLU variant that omits the nonlinear activation function σ.. Bilinear layers beat ordinary ReLU MLPs and perform almost as well as SwiGLU on language modeling tasks (Shazeer 2020). We corroborate these findings in in/Appendix I and show that billnear layers achieve equal loss when keeping training time constant and marginally worse loss when keeping data constant.
2

---

Published as a conference paper at ICLR 2025
Interaction matrices and the bilinear tensor. A bilinear MLP parameterizes the pairwise interac- tions between inputs. One way to see this is by looking at how single output g(X)a is computed.
g(x) (Wæ) O (Vx)
g g(x) l (w",)) (v
x (wa: v
We call the (dimput: dinpue) matrix 0 = Bay an interaction matrix since it defines how each pair of inputs interact for given output dimension
The collection interaction matrices across the output axis can be organized into the third-order bilinear tensor, B, with elements baig WaiVaj. illustrated in Figure 1/A. The bilinear tensor allows us to easily find the interaction matrix for specific output direction u of interest by taking a product along the output axis, "out B, equal E. Un w. As written, has size (chidden: dinputs dinpar) ) but we will typically multiply the down- projection P into B resulting in a (drmpput d'nput dlippu)) size tensor.
weighted sum
the neuron-base -baion interaction matrices,
Simplifications due to symmetry. Because an interaction matrix is always evaluated with two copies of the input æ, it contains redundant information that does not contribute to the activation. Any square matrix can be expressed uniquely as the sum of a symmetric and anti-symmetric matrix.
B :::
B +B
B
B D!!
Bat
However, evaluating an anti-symmetric matrix A with identical inputs yields 0 and can be omitted:
TT a=Ac===(AAA"
(xx" Ax)
Therefore, only the symmetric part Beym contributes. From here on, we drop the .yym superscript and assume the symmetric form for any interaction matrix or bilinear tensor (baij = May + WayVas)). Symmetric matrices have simpler cigendecompositions since the eigenvalues all real- valued, and the eigenvectors are orthogonal by the spectral theorem.
Incorporating biases. If bilinear layer has biases, can augment the weight matrices to adapt our approach. Given activations of the form (xx) = (W x +b, too))oooo+-e> (Vx+bz), define W = [W; bbl, V = by]. and x' [zr, 1]. Then, g(x) (W' x') @ (V'x') in a bilinear layer with biases. In subsection 4.3] we study a toy classification task using model trained with biases, illustrating how biases be interpreted using the same framework. For the rest of experiments, used models without biases for simplicity, as it did not harm performance. See Appendix I. for details.
3 ANALYSIS METHODS
Since bilinear MLPs can be expressed in terms of a third-order tensor, B, they can be ficxibly analyzed using techniques from linear algebra, such decompositions and transformations. The choice of analysis approach depends on what additional information, in terms of previously obtained input or output features, is provided.
3.1 INPUT/ / OUTPUT FEATURES - DIRECT INTERACTIONS
If we have already obtained meaningful sets of features for the bilinear MLP's inputs and outputs, for example from set of latent feature dictionaries Fin and Four, then we can directly study the in- teractions between these features and understand how the output features
constructed from input ones. We can transform the bilinear tensor into the feature basis via bate = Eizk Gijk out bigk fajn J For given set of sparse input and output activations, only small subset of the interactions (with a, b, c all active) will contribute, and the statistics of these active interactions can be studied.
For dictionaries obtained from sparse autoencoders (SAEs) we can instead use the output SAE's encoder directions in the transformation: babbe = Eigk j o bisk Joj f Then the output activations are .ott = ReLU(E.. bute: .in .in) terms of the input directions in_ In section 5) we use this approach identify the top relevant interactions for features in language model.
3

---

Published as a conference paper at ICLR 2025
A)
Bilinear layer
Bilinear tensor
B)
Multiply by ourput vector
Interaction Matrix
Figendecomposttonn
W
A
U;
d
B
B
in
Σ
V
( = Wx O Væ g(x)
Σ
bass 2:3;
ạ B Q
Σ
A Vala
Figure A) Two ways to represent a bilinear layer, via an elementwise product or the bilinear tensor. B) Diagram of the eigendecomposition technique. Multiplying the bilinear tensor by desired output direction u produces an interaction matrix Q that can be decomposed into set of eigenvectors and associated eigenvalues X
3.2 OUTPUT FEATURES - EIGENDECOMPOSITION
Given a set of meaningful features for the MLP outputs, we can identify the most important input directions that determine the output feature activations. The output features could come from dictionary, from the unembedding (shown in ssecttiion 4), or from the decompilation of later layers.
The interaction matrix, Q u "our B for given output feature u can be decomposed into a set of eigenvectors (Figure 1 Since Q can be considered symmetric without loss of generality (see section 2), the spectral theorem gives
Q =
A
(2)
with set of d (the rank of W, V) orthonormal eigenvectors and real-valued eigenvalues ); In the eigenvector basis, the output in the u-direction
x² Qx = Σ A (o,
(3)
activation
where each term can be considered the activation for the eigenvector v. of size (dinpur). That is, the bilinear layer's outputs are quadratic the eigenvector basis.
The eigenvector basis makes it easy to identify any low-rank structure relevant to u. The top eigen- vectors by eigenvalue magnitude give the best low-rank approximation the interaction matrix Q for given rank. And since the eigenvectors diagonalize Q. there are no cross-interactions between eigenvectors that would complicate the interpretation of their contributions to u.
3.3 No FEATURES - HIGHER-ORDEE SVD
If we have no prior features available, it is still possible to determine the most important input and output directions of through higher-order singular value decomposition (HOSVD). The simplest approach that takes advantage of the symmetry in B is to reshape the tensor by flattening the two input dimensions to produce a (doutpat: (ingue) shaped matrix and then do standard singular value decomposition (SVD). Schematically, this gives
Bourinxxin Σ
where q can still be treated as an interaction matrix and further decomposed into eigenvectors as described above. We demonstrate this approach for an MNIST model in Appendix D
4 IMAGE CLASSIFICATION: INTERPRETING VISUAL FEATURES
We consider models trained on the MNIST dataset of handwritten digits and the Fashion-MNIST dataset of clothing images. This is semi-controlled environment that allows evaluate the

---

Published as a conference paper at ICLR 2025
A)
B)
Elgenwector Y
Inpat
Activation Ad6r
Low
High
trease
High
Low
Negative
Figure 2: A) Eigenvector activations quadratic in the input and have large magnitude if an input aligns with the positive (blue) regions or the negative (red) regions, but not both. B) Top eigenvectors for single-layer MNIST and Fashion-MNISST models, revealing the most significant patterns learned for each class. In MNIST, eigenvectors represent components of the target class, while Fashion-MNIST eigenvectors function as localized edge detectors. Best viewed in color.
interpretability of cigenvectors computed using the methods in musubscctionn 3.2 This section analyses shallow feedforward network (FFN) consisting of an embedding projection, bilinear layer, and classification head; see Appendix G for details.
First, qualitatively survey the eigenvectors and highlight the importance of regularization in feature quality. Second, we consider the consistency eigenvectors across training runs and sizes. Third, turn toward algorithmic task MNIST, where compare the ground truth with the extracted eigenvectors. Lastly, we use these eigenvectors to construct adversarial examples, demonstrating their causal importance.
4.1 QUALITATIVE ASSESSMENT: TOP FIGENVECTORS APPEAR INTERPRETABLE
The eigenvectors are derived using the unembedding directions for the digits as the output directions obtain interaction matrices Q "our B that are then decomposed following (subsection 3.2) So each unembedding direction (digit) has corresponding set of eigenvectors, although we may refer the full collection the eigenvectors of the layer or model.
We can visualize them by projecting them into the input space using the embedding weights Be- cause the activation of an eigenvector e with eigenvalue X; is quadratic the input, \(v ) the sign of the eigenvector is arbitrary. The quadratic leads to XOR-like behavior where high overlap with an eigenvector's s positive regions (blue) or the negative regions (red)-but not woth-eaads to large activation magnitude, while the overall sign is determined by the eigenvalue (Figure A
For MNIST, the top positive eigenvector each output class emphasizes curve segment specific to its digit or otherwise resembles a prototypical class image (Figure 2B). Top eigenvectors for FMNIST function as localized edge detectors, focusing on important edges for each clothing article, such the leg gap for trousers. The localized edge detection relies the XOR-like behavior of the eigenvector's quadratic activation.
0.34
8.1
0.00
<0.10
.0.13
8:1
Figure 3: The top four positive (top) and negative (bottom) eigenvectors for the digit 5, ordered from left to right by importance. Their eigenvalues highlighted the left. Only 20 positive and 20 negative eigenvalues (out of 512) are shown on the left images. Eigenvectors tend to represent semantically and spatially coherent structures.
5

---

Published as a conference paper at ICLR 2025
norm=0
norm-
Figure 4: Top eigenvector for models trained with varying Gaussian input noise. For reference, the norm of an average digit is about 0.3; adding noise with norm of results in a heavily distorted but discernible digit. Finally, the test accuracy for each model is shown at the top.
Only small fraction of eigenvalues have non-negligible magnitude (Figure 3). Different top eigen- vectors capture semantically different aspects of the class. For example, the spectrum for digit 5, the first two positive eigenvectors detect the 5's horizontal top stroke but at different positions, similar to Gabor filters. The next two positive cigenvectors detect the bottom segment. The negative eigenvectors
somewhat less intuitive but generally correspond features that indicate the digit is not a five, such as an upward curve in the top right quadrant instead of horizontal stroke. In Appendix C we study this technique towards explaining an input prediction. Details of the training setup are outlined in Appendix G while similar plots for other digits can be found in Appenddix A
Because we can extract features directly from model weights, can identify overfitting in image models by visualizing the top eigenvectors and searching for spatial artifacts. For instance, the eigenvectors of unregularized models focus on certain outlying pixels (Figure 4). We found adding dense Gaussian noise to the inputs (Bricken et al.) 2023a) to be an effective model regularizer, producing bilinear layers with more intuitively interpretable features. Increasing the scale of the added noise results produces more digit-like eigenvectors and results in lower-rank eigenvalue spectrum (Appendix E). These results indicate that our technique can qualitatively help uncover overfitting or other unwanted behavior in models. Furthermore, it can be used to evaluate the effect of certain regularizers and augmentation techniques, as explored in Appendix B
4.2 QUANTITATIVE ASSESSMENT: EIGENVECTORS LEARN CONSISTENT PATTERNS
One important question machine learning whether models learn the same structure across train- ing runs (Li al. 2 and across model sizes (Frankle & Carbin 2019). In this section, we study both and find that eigenvectors are similar across runs and behave similarly across model sizes. Furthermore, we characterize the impact of eigenvector truncation classification accuracy.
Both the ordering and contents top eigenvectors are very consistent across runs. The cosine similarities of the top eigenvector between 0.8 and 0.9 depending size (Figure 5). Generally,
A)
B)
Similartty Eigenvectors
Truncation
680
1102
Digeveeter mak
Dipeneectrr mik (per dig?)
Figure 5: A) The similarity between ordered eigenvectors of the same model size averaged over all digits. This shows that equally sized models learn similar features. B) Resulting accuracy after only retaining the n most important cigenvalues (per digit). Both plots are averaged over 5 runs with the 90% confidence interval shown.

---

Published as a conference paper at ICLR 2025
increasing model sizes results in more similar top eigenvectors. Further, truncating all but the top few eigenvectors across model sizes yields very similar classification accuracy. This implies that, beyond being consistently similar, these eigenvectors have comparable impact classification. In Appendix H we further study the similarity of eigenvectors between sizes and show that retaining only handrul of eigenvectors results in minimal accuracy drops (0.01%) %6).
4.3 COMPARING WITH GROUND TRUTH: EIGENVECTORS FIND COMPUTATION
To perform a ground-truth assessment of eigenvectors, we consider task from mechanistic in- terpretability challenge, where the goal was determine the labeling function (training objective) from model Casper (2023). Specifically, the challenge required reverse-enginent eegeee-nngini a binary im- age classifier trained on MNIST, where the label is based on the similarity to a specific target image. The model predicted "True' if the input has high cosine similarity to this target or high similarity to the complement (one minus the grayscale) of that target and 'False" otherwise. This target chosen instance of '1'.
Previous work (Stefan Heimersheim 2023) reverse-engineered this through a combination of meth- ods, all requiring careful consideration and consisting of non-trivial insights. Furthermore, the meth- ods required knowledge of the original dataset and a hint of what to look for. While our method does not work on the original architecture, show that we do not require such knowledge and can extract the original algorithm from the weights alone.
We perform our decomposition the output difference (True False) since this is the only mean- ingful direction before the softmax. This consistently reveals one high positive eigenvalue; the rest (close to) zero (Figure 6). The most positive eigenvector is sufficient for completing the task; it computes the exact similarity we want. If the input is close to the target, the blue region will match; if it is close to the complement, the red will match; if both active simultaneously. they will somewhat cancel out. The remaining two eigenvectors are separated as they seem to overfit the data slightly; the negative eigenvector seems to penalize diagonal structures.
Contrary to other models, this task greatly benefited from including biases. This arises from the fact that the model must not only compute similarity but also make its binary decision based on learned threshold. If no bias is provided, the model attempts to find quadratic invariances in the data, which don't generalize well, especially given the important but sensitive role of this threshold in classification. Here, the bias (shown in the bottom corner of Figure 6) represents negative contribution. The role of biases in bilinear layers is further discussed in Appendix
0.19
8.08
Target
0.00
Bias
Figure 6: Eigenvalues and eigenvectors of a model trained to classify based on similarity to a target. The most important eigenvector (top-left) is generalizing solution; the other features sharpen the decision boundary based on the training dataset. The latter features disappear with increased regu- larization. On the right, the target digit is shown along with the learned bias from the model.
4.4 ADVERSARIAL MASKS: GENERAL ATTACKS FROM WEIGHTTS
To demonstrate the utility of weight- based decomposition, we construct adversarial masks for the MNIST model without training any forward passes. These masks are added to the input, leading to misclassification as the adversarial digit. The effect is similar to steering, but the intervention is the input instead of the model internals.
7

---

Published as a conference paper at ICLR 2025
0.3
0.2
Original
0.6
Eigenwector
Advensarial
Mask
Misclassfied Example
Random Mask
Nask Std
Maaic
200000
Original
0.1
022
0.3
Neak SLd Der
Mlalh 5411 Dare
Figure 7: Examples of an adversarial mask constructed from the given eigenvector along for models trained A) with Gaussian noise regularization (std 0.15) and B) without regularization. The average accuracy and the rate of misclassification the adversarial digit show stronger effects for adversarial masks than random baselines. In B), the mask is only applied to the outer edge of pixels that are active on less than 1% of samples.
We construct the adversarial masks from the eigenvectors for specific digits. One complication is that the eigenvectors have nontrivial cosine similarity with each other, an input along single eigenvector direction could potentially activate multiple eigenvectors across different digits. To help avoid this, we construct the mask m, for given eigenvector Us: as the corresponding row of the pseudoinverse (V+)s: for a set of cigenvectors V (specifically the top 10 positive). In an analogy to key-value pairs, the pseudoinverses effectively act like keys that activate with more specificity than the eigenvectors themselves, since Vy:- (V+)= ); 8ij-
We construct an adversarial mask from an eigenvector for the digit (Figure 7A). Even though the original eigenvector resembles the digit, the pseudoinverse based mask does not (see [Appeddix M for more examples). The accuracy, averaged over masks from the top three eigenvectors, drops significantly more than the baseline of randomly permuting the mask despite regularizing the model during training using dense Gaussian noise with standard deviation 0.15, The corresponding rise in misclassification indicates effective steering towards the adversarial digit.
Iyyas et al. (2019) observe that adversarial examples can arise from predictive but non-robust features of the data, perhaps explaining why they often transfer to other models. Our construction can be seen as a toy realization of this phenomenon because the masks correspond to directions that are predictive of robust features but of over-fitting found on the outer edge of the image for a model trained without regularization FFiuure 7). Since we can find this over-fitting pattern from the eigenvectors, in general way, we can construct the mask by hand instead of optimizing it.
robust. We construct masks that only exploit the patterns
LANGUAGE: FINDING INTERACTIONS BETWEEN SAE FEATURES
Each output bilinear layer described by weighted pairwise interactions between their input features. Previous sections show that this can be successfully leveraged to trace between a bilinear layer's inputs and outputs. Here, we turn towards tracing between latent feature dictionaries ob- tained by training sparse autoencoders (SAEs) the MLP inputs outputs for 6-layer bilinear transformer trained on Stories (Eldan & Li 2023
(see training details in in/Apendix Gi..
5.1 SENTIMENT NEGATION CIRCUIT
We focus using the eigendecomposition to identify low-rank, single-layer circuits in bilinear transformer. We cherry-pick and discuss one such circuit that takes input sentiment features and semantically negates them. Unlike previous work sparse feature circuits (Marks et al. 2024 that relies on gradient- linear approximations, we identify nonlinear interactions grounded in the layer's weights that contribute the circuit's computation.
8

---

Published as a conference paper at ICLR 2025
A)
B)
40
Top negative elgenvector
84
Troe
42
Elgenvecton
12
144
ean-gedd boctun
positive eigenvector
Outouz feature activation
15
Figure 8: The sentiment negation circuit that computes the not-good and not-bad output features. A) The interaction submatrix containing the top 15 interactions. B) The projection of top interacting features onto the top cigenvectors using cosine similarity. The symbols for different clusters match the labels A. Clusters coincide with the projection of meaningful directions such the difference in "bad" "good" token unembeddings and the MLP's imput activations for the input "[BOS] not". C) The not-good feature activation compared its approximation by the top two eigenvectors.
The sentiment negation circuit computes the activation of two opposing output features in layer (index 1882 and 1179) that form a fully linear subspace. The cosine similarity of their decoder vectors is -0.975. Based their top activations, the output features activate on negation tokens ('not", "never", "wasn't "aass"t")) and boosts either positive sentiment tokens ("good", "safe", "nice") or negative sentiment tokens ("bad", "hurt", "sad"), we denote the two features as the not-good and the not-bad features respectively. See Appendixx O for the top activations of all features mentioned.
Focusing on the not-good feature, the top interactions for computing its activations resemble an AND-gate (Figure 8A). Input features that boost negative sentiment tokens (blue squares) have strong positive interactions with negation token features (green triangles), but both have negligi- ble self-interactions. So, both types of input features are needed to activate the not-good feature and flip the boost from negative to positive sentiment. The one positive sentiment feature (orange downward triangle) interacts with the opposite sign. The interactions shown significantly larger than the typical cross-interaction. with a standard deviation of 0.004 (Figure 27
The eigenvalue spectrum has one large positive (0.62) and one large negative value (-0.66) outliers (Figure 27). We can see the underlying geometry of the circuit computation by projecting the input features onto these eigenvectors (Figure 8). By itself, positive sentiment feature (blue squares) would equally activate both eigenvectors and cancel out, but a negation feature is also present, the positive eigenvector is strongly activated. The activation based on only these two eigenvectors, following [qqation 3) has a good correlation (0.66) with the activation of the not-good feature, particularly at large activation values (0.76), conditioned the not-good feature being active.
5.2 LOW-RANK APPROXIMATIONS OF OUTPUT FEATURE ACTIVATIONS
The top eigenvectors can be used to approximate the activations of the SAE output features using a truncated form Equation 3 To focus on the more meaningful tail of large activations, we compute the approximation's 's correlation conditioned on the output feature being active. The correlations of inactive features are generally lower because they are dominated by 'noise". We evaluate this on three bilinear transformers at approximately 2/3 depth: 6-layer TinyStories ("ts-tiny') and two Web models with 12 and 16 layers ('fw-small' and 'fw-medium',
We find that features are surprisingly low-rank, with the average correlation starting around 0.65 for approximations by a single eigenvector and rising steadily with additional eigenvectors (Figure 9)) Most features have a high correlation (> 0.75) even when approximated by just two eigenvectors Figure 9B). Scatter plots for random sample of features show that the low-rank approximation often captures the tail dependence well (Figure 9C). Interestingly,
find the approximation to drastically improve with longer SAE training times while other metrics change only slightly. This indicates "hidden' transition near convergence and is further discussed in Appendix H Overall, these results suggest that the interactions that produce large output activations are low-rank, making their interpretability potentially easier.

---

Published as a conference paper at ICLR 2025
A)
activatio approvimation
B)
Approx. by digonvectors
5.96
0.02
0.72
6.72
Tip digereectors
Corrs on Exctov oely
AcDivction
Figure 9: Activation correlations with low-rank approximations for differently-sized transformers. A) Average correlation over output features computed over every input where the feature is active. B) The distribution of active-only correlations for approximations using the top two eigenvectors. C) Scatter plots for a random set of nine output features on 'fw-medium'. Approximations use the top two eigenvectors. Low correlation scores generally only occur on low-activation features.
6 DISCUSSION
Summary. This paper introduces novel approach to weight-based interpretability that leverages the close-to-liness structure of bilinear layers. A key result is that we can identify the most im- portant input directions that explain the layer's output along a given direction using an eigenvector decomposition. The top eigenvectors are often interpretable, for example for MNIST they function as edge-detectors for strokes specific each digit. The lack of element-wise nonlinearity bilinear MLPs allows us to transform their weights into interaction matrices that connect input to output features and then extract the low-rank structure. In language models, we find that many SAE output features are -ppl-approximmtte by low-rank interaction matrices, particularly at large activations. We highlighted one example of an extracted low-rank circuit that flips the sentiment the next to- ken if the current token is negation ("not"). The behavior of this circuit can be easily understood in terms of the top eigenvectors, whereas finding similar circuit conventional MLPs would be more difficult. Overall, our results demonstrate that bilinear MLPs offer intrinsic interpretability that can aid in feature and circuit extraction.
Implications. The main implication of our work is that weight-basce -baed interpretability is viable, even for large language models. Bilinear MLPs can replace conventional MLPs in transformers with min- imal cost while offering intrinsic interpretability due their lack of element-win nonlinearities and close-to-linear structure. Current circuit analysis techniques rely on gradient-based approximations (Syed et al., 2023 Marks al. 2024) use transcoders (Dunefsky et al. 2024) to approximate MLPs. Both approaches depend on an input dataset, potentially leading to poor performance out- of-distribution, and they may not fully capture the nonlinear computations in MLPs. In contrast, bilinear MLPs can be transformed into explicit feature interaction matrices and decomposed in way fully equivalent to the original computations. Extracting interactions more directly from the weights should lead to better, more robust circuits. Weight-based interpretability may also offer better safety guarantees since we could plausibly prove bounds on a layer's outputs by quantifying the residual weights not captured in a circuit's 's interactions.
Limitations. Application our methods typically relies on having a set of meaningful output direc- tions available. In shallow models, the unembedding directions be used, but in deeper models, we rely on features derived from sparse autoencoders that are dependent on an input dataset. An- other limitation is that, although the eigenvalue spectra often low-rank and the top eigenvectors appear interpretable, there are no guarantees the eigenvectors will be monosemantic. We expect that for high-rank spectra, the orthogonality between eigenvectors may limit their interpretability. Ap- plying sparse dictionary learning approaches to decompose the bilinear tensor may be promising way to relax the orthogonality constraint and find interpretable features from model weights.
ACKNOW
We are grateful to Narmeen Oozeer, Nora Belrose, Philippe Chlenski, and Kola Ayonrinde for help- ful feedback the draft. We grateful to the Al Safety Camp program where this work first
10

---

Published as a conference paper at ICLR 2025
started and to the ML Alignment & Theory Scholars (MATS) program that supported Michael and Alice while working on this project. We thank CoreWeave for providing compute for the finetuning experiments. This research received funding from the Flemish Government under the "Onderzoek- sprogramma Artificiéle Intelligentie (AI) Vlaanderen" programme.
CONTRIBUTIONS
Michael performed the bulk of the work on the MNIST analysis and provided valuable insights across all presented topics. Thomas worked on the Language Models section and was responsible for code infrastructure. The paper was written in tandem, each focusing on their respective section.
ETHICS STATEMENT
This paper proposes no advancements to the state-of-the-ane in model capabilities. Rather, it provides new methods analyze the internals of models to increase our understanding. The only misuse the authors envision is using this technique to leak details about the dataset that the model has learned more efficiently. However, this can be avoided by using this technique during safety evaluation.
REPRODUCIBILITY STATEMENT
We aspired to make this work reproducible possible. First, Appendix G (among others) aims to provide detailed and sufficient descriptions to independently recreate our training setups. Second, our code (currently public but not referenced for anonymity) contains separate files that can be used to generate the figures in this paper independently. We used seeds across training runs so that recreated figures would be equivalent. Third, all models that are compute-intensive train, such as the SAEs and the LMs, will be shared publicly. Lastly, we will publish an interactive demo, which will allow independent analysis of the figures Appendix A Appendix and Appendix O way this document cannot.
REFERENCES
Trenton Bricken, Rylan Schaeffer, Bruno Olshausen, and Gabriel Kreiman. Emergence of sparse representations from noise. In International Conference Machine Learning, pp. 3148-3191
PMLR, 2023a.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden Mcl.ean, Josiah F. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Trans- former Circuits Thread, October 2023b.
2023 /ttp///ooooncccccccccc
html
Stephen Casper. Eis vii: A
URL https: AI Alignment Forum, Part of the Engineer's Interpretability Sequence, posted on February 18, 2023.
challenge for mechanists, 2023. www. alignment for um org/
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: Deep learning for interpretable image recognition, 2019. URL https:
//arxiv. orq/ httgg//bs11800 10574
Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos Zafeiriou. Deep polynomial neural networks. IEEE transactions pattern analysis and machine intelligence, 44(8):4021 40344, 2021.
Andrzej Cichocki, Danilo Mandic, Lieven De Lathauwer, Guoxu Zhou, Qibin Zhao, Cesar Caiafa, and HUY ANH PHAN. Tensor decompositions for signal processing applications: From two- way to multiway component analysis. IEEE Signal Processing Magazine, 145-11, March

---

Published as a conference paper at ICLR 2025
2015. ISSN 1053-5888. doi: 10.1 00.11099mpp22132297749.. URL /ttt:://xx.. doi org/10. 1109/MBB 2013.2297439
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR, January 2024. doi: (0-48550/rrXxi..300.06660 URL MrXiv:2309.08600 [es].
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, 2017.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders enable fine-
grained interpretable circuit analysis for language models, 2024. URL
https::/ hwwp gww.ntooruumnmmmmmmcmmmmmmm http:://wwww.alaggnnoooooooommmm
http:///////pppsst///////eteeeee Stanscoders-enabllee fine wtt.gaaeeedd-eeeeeerrraeeeeeeeee
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Haffield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https//transformee-crruuut wttp
Jonathan Frankle and Michael Carbin. The lottery hypothesis Finding sparse, trainable neural networks, 2019. URL https: http://aaivvvi/.ii owg. wttp://g/.bb.aooee 1803. 03635
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https:
//arxiv. 240b.///40..o 04093
Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Auto- matically identifying local and global circuits with linear computation graphs. arXiv preprint 13868,, 2024.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023. URL hittps::/ttrassformerr recitst.t...... pub/
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances neural information processing systems, 32, 2019.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models, 2020. URL httpp::/arrxv..orr/bb/////
04612
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Factorized bilinear models for image recognition. In Proceedings of the IEEE international conference on computer vision, pp. 2079- 2017. 2017.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations?, 2016. URL httpt:tt//a/aii....aaaa org/
fbs/1511. 07543
Tsung- Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear models for fine- ane-grained visual recognition. In Proceedings of the IEEE international conference on computer vision, pp.
1449-1457, 2015.
12

---

Published as a conference paper at ICLR 2025
Samuel Marks, Can Rager, Erie J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647 2024.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, February 2018. ISSN 1051-2004. doi: 10. (0.10166j.dpp2217110000 dsp.2017.10.00
URL http: doi http:://ax..o..og//11..oo 10
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.23915/disttll.00000
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Cireuits. Distill, March 2020. URI. htt.ttttt///sittt. ppbb
in
Yannis Panagakis, Jean Kossaifi, Grigorios G. Chrysos, James Oldfield, Mihalis A. Nicolaou, Anima
Anandkumar, and Stefanos Zafeiriou. Tensor methods in computer vision and deep learning- Proceedings of the IEEE, (69 860-890,,068880881 May 2021. ISSN 1558-2256. doi: 10.1 10.1109/jrro. 2021 1307429, URL //x doi wor..orr//..... 10. 1109/ 2021 3074329
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL http:/tsss//axiii.orrrreeee hbtp://b//2// hbtp://...rr/a///200.....2 17557
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized imput sampling for explanation of black-box models, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier, 2016. URL/attps:/arxxvvorr wwwpgg//aassssaa /602. 04938
Lee Sharkey. A technical note on bilinear layers for interpretability. 2023.
Noam Shazeer. Glu variants improve transformer, 2020.
Nicholas D. Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E. Papalexakis, and Christos Faloutsos. Tensor decomposition for signal processing and machine learning. IEEE
Transactions Signal Processing, 65(13):3551-3355 2017. doi: 10.1 109119992211..2995554
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014. URL https://aaxivv.
orq/ erq/abs/ 01112..4 6034
Marius Hobbhahn Stefan Heimersheim. 2023. URL http:///www...aaaaattttco///roe/
Ac-
cessed: 2024-09-02.
Aaquib Syed, Can Rager, and Arthur Commy, Attribution patching outperforms o automated circuit discovery, 2023. URL URL/Lppp::///aa....rr/////////1
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunoy, and Thomas Scialom. Llama 2: Open foundation and finettuned chat models, 2023.
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. NeurIPS, 30, 2017. URL
https://arxii wwt.oorgg///oooooooo 03762
Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Lim, and Chris Olah. Visualizing weights. Distill, 2021. doi: 10.2391 1/distill.00024. 007. http:///dtiillllu222222///rrrrrd
14

---

Published as a conference paper at ICLR 2025
A EIGENSPECTRA: SHOWING EIGENVECTORS ACROSS DIGITS
The following are plots showing multiple positive and negative cigenvectors for certain digits. Posi- tive features either tend look for specific patterns the target class (first eigenvector of match- ing the bottom part) or tend to match an archetypal pattern (second eigenvector of 6, matching the whole digit). Negative eigenvectors tend look for specific part that would change the class of the digit. For instance, if the pattern highlighted by the first negative eigenvector of 4 were on, it would most likely be a
0.14
0.09
0.00
-0.13
Figure 10: eigenvectors for digit 2.
0.15
0.12
0.00
-0.12
-0.14
Figure 11: eigenvectors for digit 4.
0.15
0.13
0.10
0.00
00.12
-0.14
Figure 12: eigenvectors for digit 6.
15

---

Published as a conference paper at ICLR 2025
B REGULARIZATION & AUGMENTATION: ABLATIONS & OBSERVATIONS
Following the observation that regularization improves feature interpretability for image classifiers, study several popular regularization and augmentation techniques. In summary, input noise sparsifies the features, while geometric transformations blur the features. Some popular techniques, such dropout, have little impact on features.
B. REGULARIZATION
Input noise has the largest impact features from any of the explored techniques. Specifically, we found dense Gaussian noise (already depicted in Figure 4) to provide the best trade-off between feature interpretability and accuracy. We also considered sparse salt-and-pepper noise (blacking or whiting out pixels), which resulted both lower accuracy and interpretability. Lastly, we explored Perlin noise, which spatially correlated and produces smooth patches of perturbation. However, this performed worst of all, not fixing the overfitting.
Model noise adds random Gaussian noise the activations. This had measurable impact any of our experiments, However, this may simply be because our models are quite shallow.
Weight decay generally acts as sparsifier for eigenvalues but does significantly impact the eigenvectors. This is extremely useful as it can zero out the long tail of unimportant eigenvalues, strongly reducing the labor required to analyze a model fully (more details (mppenddix E
Dropout did not seem to impact our models. Overfitting was still an issue, even for very high values (> 0.5). We suspect this may change in larger or capacity-constraine models.
B.2 AUGMENTATION
Translation stretches features in all directions, making them smoother. The maximal shift (right) about pixels in each direction, which is generally the maximal amount without losing impor- tant information. Interestingly, translation does not avoid overfitting but rather results in smoother overfitting patches. High translation results in split features, detecting the same pattern in different locations (Figure 13). This also results in a higher rank.
Rotation affects the features in the expected manner. Since rotating the digit zero does not signifi- cantly impact features, consider the digit 5, which has mix of rotation invariance and variance. Again, it does not stop the model from learning overfitting patches near the edges without noise. These features become broader with increased rotation.
Blur does not significantly affect features beyond making them somewhat smoother. Again, it still overfits certain edge pixels in blurred manner without noise.
0.12
0.09
20
0.00
-0.09
-0.11
-0.13
Figure 13:
Important eigenvectors for model trained with high translation regularization pixels on either side). Similar patterns manifest as multiple eigenvectors at different locations.
Combining augmentations has the ex- pected effect. For instance, blurring and rotation augmentation yield smooth and curvy features.
All these augmentations are shown separately in Figure 14
16

---

Published as a conference paper at ICLR 2025
37.6%
08.0%
97.6%
97.3%
26.8%
0.4 nUnnn
Noiso
98.2%
97.2%
normm
Translation
plxes
pixels
97.6%
97.9%
98.0%
97 57.5%
97.5%
0.4 (M1ET)
Nossa
08.1%
95.0%
95.0%
27.8%
97.4%
Rotation
0.4 nomn
90.1%%
95.1%
95.0%
97.0%
97.0%
Noise
95.1%
Blur
sigmaa
Lisiga
Figure 14: Important eigenvectors for models with different hyperparameters.
17

---

Published as a conference paper at ICLR 2025
C EXPLAINABILITY: A SMALL CASE STUDY WITH EIGENVECTORS
While this paper focuses on bilinear layers for interpretability, the proposed techniques can also be used for post-hoc explainability to understand what has gone wrong, Our explanations are generally not as human -ninndf as other methods but are fully grounded in the model's weights. This section explores explaining two test-set examples, one correctly classified and one incorrectly.
The figures are divided into three parts. The left line plots indicate the sorted eigenvector activation strengths for the digits with the highest logits. The middle parts visualize the top positive and negative eigenvectors for each digit. The right displays the input under study and the related logits.
The first example, a somewhat badly drawn five, results about equal positive activations output classes 5, 6, and 8 (which all somewhat match this digit). The negative eigenvectors are most important in this classification, where class by far the least suppressed. This is an interesting example of the model correctly classifying through suppression.
The second example, a seven-y looking two, is actually classified as From looking at the top eigenvectors of the digit (shown in Figure 1O), more vertical slanted stroke activates the top eigenvector for the digit 7 more strongly than the 2- eigenvectors that look for more curved and slanted strokes. The negative eigenvectors not very important in this incorrect classification.
that the more horizontal top stroke and
8
Input
3.88
0.00
10
log
22.51
Figure 15: Study of correctly classified 5. The output is strongly influenced by negative eigenvec- tors, resulting in strong suppression for the other digits.
input
4.67
2
1.81
1.25
2
log its
0.00
-0.43
00.87
10
Figure 16: Study of a misclassified 2. The model mostly classifies twos based on the bottom line and top curve, which are both only partially present.
18

---

Published as a conference paper at ICLR 2025
D HOSVD: FINDING THE MOST IMPORTANT SHARED FEATURES
In the case that no output features are available or we wish to find the dominant output directions, HOSVD the B tensor (described in subsection 3.3). Intuitively, this reveals the most important shared features. We demonstrate this approach on the same MNIST model used in section 4
Instead of contributing to single output dimension, each interaction matrix can contribute to an arbitrary direction, shown at the bottom right ("contributions") Further, the importance of the con- tributions is determined by their singular value, which is shown at the top right. The remainder of the visualization shows the top eigenvalues and the corresponding cigenvectors.
The most important output direction separates digits with prominent vertical line (I, 4, and 7) from digits with a prominent horizontal line (5 specifically). Similarly, the second most important direction splits horizontal from vertical lines but is more localized to the top half. Specifically, it splits by the orientation of the top stroke (whether starts/ends left or right).
eigenvalues
elgenvectors
singular value
0.63
cigenvalues
eigenvectors
contributions
Figure 17: The most important output direction of MNIST model roughly splits digits by its horizontality or verticality.
eigenvalues
eigenivectors
singular value
0.39
0.56
20
eigenvalues
eigenvectors
contributions
3
-0.33
Figure 18: The second most important output direction of an MNIST model splits digits according to the orientation of its top stroke.
We observe that the output directions uncovered through HOSVD somewhat correspond to mean- ingful concepts, albeit sometimes dominated by specific digit (such as 5 and 6). Less significant directions often highlight specific portions of digits that seem meaningful but are more challenging to describe. In summary, while in the case of MNIST, the results not particularly more inter- pretable than decomposing according to digits, we believe this technique increases in utility (but also computational cost) as the number of output classes increases.
19

---

Published as a conference paper at ICLR 2025
E SPARSITY: WEIGHT DECAY VERSUS INPUT NOISE
Throughout, we make the claims that input noise helps create cleaner eigenvectors and that weight decay results in lower rank; this appendix aims quantify these claims. To quantity near-sparsity, we use (L1/Lg) which can seen as continuous version of the Lo norm, accounting for near- zero values. We analyze both the eigenvalues, indicating the effective rank, and the top eigenvectors, indicating the effective pixel count.
As visually observed in Figure 4 this analysis (left of Figure 19) shows that input noise plays large role determining the eigenvector sparsity; weight decay does not. On the other hand, input noise increases the number of important eigenvectors while weight decay decreases it. Intuitively, input noise results specialized but more eigenvectors, while weight decay lowers the rank.
Eigenvector Sparsity
Eigenvalue Sparsity
402
390
We0 git bock
0.0
943
62
303
6.
6.
Irpur Nalee
Trout Notioe
Figure 19; Measuring the approximate Lo norm of eigenvalues (left) and the top eigenvectors (right) with varying Gaussian input noise and weight decay.
F TRUNCATION & SIMILARITY: A COMPARISON ACROSS SIZES
This appendix contains brief extension of the results presented in Figure 5
Truncation seress Sizes
Simiarity Across Bgonvectors
Hodd
Egenvddor
Figure 20: A) The accuracy drop when truncating the model to a limited number of eigenvectors. The first few cigenvectors result in similar drop across sizes. Narrow models tend naturally remain more low-rank. In general, few cigenvectors are necessary to recover the full accuracy. B) Inter-model size similarity of cigenvectors (using 300 comparison point). The top features for similarly-sized models are mostly similar. C) similarity comparison between all model sizes for the top eigenvector.
20

---

Published as a conference paper at ICLR 2025
G EXPERIMENTAL SETUPS: A DETAILED DESCRIPTION
This section contains details about our architectures used and hyperparameters to help reproduce results, More information can be found in our code [currently not referenced for anonymity).
G.1 IMAGE CLASSIFICATION SETUP
The image classification models (section 4) in this paper consist of three parts: an embedding, the bilinear layer, and the head/unembeddingg shown in Figure 21] The training hyperparameters are found in Table However, since the model is small, these parameters (except imput noise; Appendix B not affect results much.
Embed
Head
Figure 21: The architecture of the MNIST model.
MNIST Training Parameters
input noise norm
0.5
weight decay learning rate batch size optimizer schedule
1.0
0.001
2048
AdamW cosine annealing
epochs
20-100
Table Training setup for the MNIST models, unless otherwise stated in the text.
G.2 LANGUAGE MODEL SETUP
The language model used in section is 6-layer modern transformer model (Touvron et al.) 2023) where the SwiGLU is replaced with a bilinear MLP (Figure 22). The model has about million parameters. The training setup is detailed in Table 3 As the training dataset, we use simplified and cleaned version of TinyStories (Eldan Li 2023 that remedies the following issues.
Contamination: About 20% of stories are exact duplicates (in both train and test).
Corruption: Some stories contain strange symbol sequences akin to data corruption.
Furthermore, we use custom interpretability- first BPE tokenizer. The tokenizer lower-case only, splits whitespace, and has vocabulary of only 4096 tokens.
512
512
Nor
attn
Norm
2048
Figure 22: Simplified depiction of a bilinear transformer model. A model dimension of 512 is used, and expansion factor of 4 (resulting in 2048 hidden dimensions in the MLP).
21

---

Published as a conference paper at ICLR 2025
TinyStories Training Parameters
weight decay
0.1
batch size
512
context length
256
learning rate
0.001
optimizer schedule
Adam W
linear decay
epochs
tokens
+ 2B
initialisation
gpt2
Table 2: Tinystories training setup. Omitted parameters are the HuggingFace defaults.
The models used in the experiments shown in Figure 9 are trained of the FineWeb dataset (Penedo al. 2024). These follow the architecture of GPT2-small (12 layers) and GPT2-medium (16 layers) but have bilinear MLPs. Their parameter count is 162M and 335M, respectively. Both use the Mixtral tokenizer.
Fine Web Training Parameters
weight decay batch size context length
0.1
512
512
learning rate
6e-4
optimizer schedule
Adam W
linear decay
tokens initialisation
+ 32B
gpt2
Table Web training setup. Omitted parameters are the HuggingFace defaults.
G.3 SPARSE AUTOENCODER SETUP
All discussed SAEs TopK activation function, as described in Gao et al. (2024). We found
A = 30 to strike a good balance between sparseness and reconstruction loss. |seciion stsddies quite narrow dictionaries (4x expansion) for simplicity. The exact hyperparameters shown in Table 4 and the attained loss added [Equation 4) across layers is shown in Figure 23
Lepatch
Ledean
Ladded
(4)
Lelean
LO55 R/00
02
2.15
Figure 23: Loss added for the mlp out and resid.mid SAEs across layers.
22

---

Published as a conference paper at ICLR 2025
SAE Training Parameters
expansion
4x
k
30
batch size
4096
learning rate
le-4
optimizer
Adam W
schedule
cosine annealing
tokens
+ 150M
buffer size
- 2M
normalize decoder
True
tied encoder init
True
encoder bias
False
Table 4: SAE training hyperparameters.
H CORRELATION: ANALYZING THE IMPACT OF TRAINING TIME
Figure shows how few eigenvectors capture the essence of SAE features. This section discusses the impact of SAE quality, measured through training steps, on the resulting correlation. In short, we find features of SAEs that are trained longer are better approximated with few eigenvectors.
We train 5 SAES with an expansion factor of 16 on the output of the MLP at layer 12 of the 'fw- medium' model. Each is trained twice as long the last. The feature approximation correlations are computed over 100K activations; features with less than 10 activations (of which there are less than 1000) are considered dead and shown. The reconstruction error and loss recovered between SAES differ only by 10% while the correlation mean changes drastically (Table 5). The correlation distribution is strongly bimodal for the 'under-trained" SAEs (shown in Figure 24 with darker col- ors). Given more training time, this distribution shifts towards higher correlations. The activation frequencies features are mostly uncorrelated with their approximations.
Training steps (relative)
2
4
8
16
Normalized MSE
0.17
0.16
0.16
0.15
0.15
Loss recovered
0.60
0.61
0.65
0.65
0.66
Correlation mean
0.17
0.28
0.42
0.52
0.59
Table 5: The SAE metrics along with the mean of the correlations shown in Figure 24| tion improves strongly with longer training, while other metrics only change marginally.
The correla-
700
Count
400
200
100
-0.5
Corvelation (active only)
Figure 24: Feature approximation correlations using two eigenvectors across SAEs with different training times. Darker is shorter, and bright is longer. This shows clear bimodal distribution for 'under-trained" SAES, which vanishes upon longer training, indicating some form of convergence.
23

---

Published as a conference paper at ICLR 2025
1 BILINEAR TRANSFORMERS: A LOSS COMPARISON
While experiments on large models show bilinear layers to only marginally lag behind SwiGLU (Shazeer 2020), this section quantifies this accuracy trade-off through compute efficiency. We per- formed our experiments using a 6-layer transformer model trained on TinyStories. For these exper- iments, we use d mooeel = 512 and hidden = 2048, resulting in roughly 30 million parameters. However, we have found these results to hold across all sizes we tried.
Bilinear
ReGLU
SwiGLU
constant epochs
1.337
1.332
1.321
constant time
1.337
1.337
1.336
Table 6: The loss of language models with varying MLP activation functions. Bilinear layers 6% less data efficient but equally compute efficient.
Considering the data efficiency (constant epochs), both SwiGLU and ReGLU marginally beat the bilinear variant. Concretely, SwiGLU attains the same final loss of the bilinear variant in 6% less epochs. On the other hand, when considering compute efficiency (constant time), we see that these differences vanish] ] Consequently. if data is abundant, there is little disadvantage using bilinear layers over other variants.
J FINETUNING: YOUR TRANSFORMER IS SECRETLY BILINEAR
Many ttat-oof-the-an open-source models use gated MLP called SwiGLU Touvron al. 2023 This uses the following activation function Swish (æ)
( We can vary the parameter represent common activation functions. If 1, that corresponds to SiLU activation, used by many current state-of-th of-the-art- models. 5 = 1.7 approximates a GELU and / = 0 is simply linear, corresponding to our setup. Consequently, can fine-tune away the gate by interpolating from its original value to zero. This gradually converts an ordinary MLP into its bilinear variant.
To demonstrate how this approach performs, we fine-tuned TinyLlama- I., a 1.1 cillion-paramenn transformer model pretrained on 3 trillion tokens of data, using single A40 GPU. For simplicity, we trained on a slice of FineWeb data. Due to computing constraints, we only tried single schedule that linearly interpolates towards 3 = 0 during the first 30% (120M tokens) and then fine-tunes for remaining 70% (280M tokens). We compare this to baseline that does not vary 3 during fine- tuming, corresponding to continued training. We use this bascline to compensate for the difference in the pretraining distribution of TinyLlama (consisting of a mixture of RedPajama and StarCoder data). This shows that this fine-tuning process increases the loss by about (0.05) but seems benefit from continued training (Figure 25). We plan to extend this result with a more thorough search for improved schedule, which will probably result in lower final loss. We also expect longer training runs to close to gap even further.
Pretue
tardin
2.30
2.2
6.c
Figure 25: Comparison of fine-tuning versus baseline over the course of 400M tokens. The loss difference is noticeable but decreases quickly with continued training.
An improvement in implementation efficiency, such fusing kernels, may change this fact.
24

---

Published as a conference paper at ICLR 2025
K TENSOR DECOMPOSITIONS: EXTRACTING SHARED FEATURES
Given a complete or over-complete set of m u-vectors, we can re-express in terms of the eigen- vectors, which amounts change of basis. To avoid multiple contributions from similar u-vectors, we have to use the pseudo-inverse, which generalizes the inverse for non-square matrices. Taking the u- u-vectors as the columns of U, the pseudo-inverse U satisfies UU+ + I, as long as U is full rank (equal to d). Then
m
B
u & Qs
(5)
d
A(ko)) u @ (u..)) @ ((.s)
(6)
k
v(..)) the eigendecomposition of the interaction matrix corresponding for uk We can then recover the interaction matrices from
where u are the rows of U and Qs Ei A
Qx = Wk: 'out using the fact that uy: u des (Kronecker delta). Note that eigenvectors within single output direction k are orthogonal but will overlap when comparing across different output directions.
L BILINEAR LAYERS: A PRACTICAL GUIDE
Bilinear layers are inherently quadratic; they can only model the importance of pairs of features, not single features. Interestingly, haven't it found this to be an issue in real-world tasks. However, linear structure important for some toy tasks and, therefore, merits some reflection. Without modification, bilinear layers will model this linear relation we can add biases to the layer as follows: BL(x) = (Wx + b) @ (Væ + c). In contrast to ordinary layers, where the bias as constant value, here as both linear and a constant value. This becomes apparent when expanded:
quadratic function. To resolve this,
BL(x)= = W @ Væ + (cW bV + cb
We disambiguate by calling the terms 'interaction', 'linear', and 'constant'. Theoretically, this is very expressive; all binary gates and most mathematical operations can be approximated quite well with In practice, the training process often fails to leverage this flexibility and degenerates to using quadratic invariances instead of learned constants.
M ADVERSARIAL MASKS: ADDITIONAL FIGURE
B
Figure 26: More examples of adversarial masks constructed from specific cigenvectors for models with A) no regularization, B) Gaussian noise regularization with std 0.15, and C) Gaussian noise regularization with std 0.3.
N LANGUAGE: FURTHER DETAILS FOR FEATURE CIRCUITS
25

---

Published as a conference paper at ICLR 2025
A)
B)
10"
06
Interaction
Cnessiiteractions
Random Symmetncc Metri
107
041
10
Eigenvalue
Counts
103
010
207
2.4
-3.6
0.0 Interaction
100
500
Eigenvector index
Figure 27: A) Histogram of self-interactions and cross-interactions. B) The eigenvalue spectrum for the sentiment negation feature discussed in /eeciion 5 The dashed red line gives spectrum for random symmetric matrix with Gaussian distributed entries with the same standard deviation.
O INPUT FEATURES OF THE NEGATION FEATURE
Output Feature (1882): not good
lily wanted to climb ladder. but her momumy
told
her to
stay
away from
bocause was
not
safe
Bily listened ber
they
everywhere
trsin stayed
heppyy
[FOS)
mon angry lily
prohtied
who sereamed cried
was very
troubled!
she
nover
pox
play
and
beloved jacket
stopped.
playing
stopped
dog
big dog
bonese
"Blly
they called
very
small boyy
Output Feature (1179): not bad
me to the park, car the car
didn
resp end, but
lily didn
mind
she was just happy
was
was at mommy
explained
was friendlly
wonld
susic was curieus
was proud
lemons,
up
they many adventures togother. bear
clumsy
happy
many friends lad lots
frog always very buny
but
forgot
adviic
says. be
cold
does
mindd
likes
the
Table 7: Output SAE features in the negation feature discussed in section
26

---

Published as a conference paper at ICLR 2025
Input Feature (326): crashing and breaking
canying hirdie
road
hindii
and of bowl
he
hard
breaks
hand
and
fuster
he
fecls
from
speed
does
rock
his
anc
biss
Input Feature (1376): dangerous actions
sweetie
you
dhey
kng
they might
yonn
aet
should
they
noc our
muybe
worried and said. "no,
wouldd dangonous
Input Feature (1636): nervous worried
big adventure decided to take trip to the tip joe was so excited,
he
was little wurried boo be
two people doing yogs was fascinatad clappod with
the people heppey
spotted lovely noe
excited wax nervous
Input Feature (123): negative attribute
[BOS] unce upon time there was nug
was
old
and
de but
[805] once upon time, there was beach. the bosch was very dithyy ii
had
Jon
tmash
on
to say hello. she ran the dour and knocked on it, but was
fittyy and covered
dirt
the sow kid opened
Table 8: SAE features that contribute to the negation feature discussed in section 5
27

---

Published as a conference paper at ICLR 2025
Input Feature (990): a bad turn
she went to him and zoiked
hrim
why
hhe
was
crying
the little
boy
sond
Tost
my
hal
m
a little buy crying
she went to him and nskedd why hee
was
sead
the little
bvry
Inc
Tost
hris
Lory
and
didn
h
spot asked.
why are
you
sad.
little
bird
the
bird replied
Jost
Favocite
bay
in
the
Input Feature (1929): inability to do something / failure
Bisttn to him
miss
tried
agoin
and
again
bun
the
waster
workkin
Lanewerr
said plesse and asked nicely
how
Itc creed
mnoving
pushing
is
pull
motthing
seermed
wook
be
was
getting
Erustrated
bc
asked
his
couldn rreach
the counter
sice
cied
and
triee
but
wazs
boo
high
lily
came
into
the
Input Feature (491): body parts
buckes
shovel
dill the bucket with
they press
soul hard
with
their
hands
the bucket
day. hopped up
that
ground. poked with
Foot
and felt hard
spider the coiling she wanted catch it, she reached pinchs in
with
heer
fiegers
but the spider was
Input Feature (947): bad ending (seriously)
very
sodiyy
passed
awaay
farmiiy
verr
missed
very
they
remennheree
happy
very
Friendls
courl
hrippeo
steged
disgesating
weap
Input Feature (882): being positive
there was musician wandered through countryside quite wesk
was
determined
mokke the
finger! she yelped and quickly pulled oway. girl was
wax
site
not
was long. bony fingers little boy was
was
very
Input Feature (240): negation of attribute
ran to it and usleed the others what it wass he was so excited to know that it waeen
just
smange big box it
explore a big tunnel and see what might inside joe mum told him it waza safe tunnel and he should
go swimming in the ocean. he asked mom take him. his mom explained it was not
safee to swim alone and she
Input Feature (766): inability to perform physical actions
but one day be sick and started suffier
his
body
burt
and he couldin
play
with
his friends
anymure
[BOS] une day pretty bird burt its
wing
it could
not fly kind girl mamed
to play with him too. one day. the puppy fell dowwn and hurrt
his
leg he could
not
play and nun like before.
Input Feature (1604): avoiding bad things
it careful
started playing carefullly making that wouldn
dirty.
shatf. she smilled and soid jimny was careful to pull off the shelf would
noc break. he proudly showed
jane noticed the ice was very mary suggeste they wear gloves so that the cold would
noc hurt their hands. jane agreed
Input Feature (1395): positive ending
watched paay
pony happy and
longer seared
found
and juse sceing the woods chaosing appresiate instead
trying
[BOS]
huppy and the ball
the ball could play together
without
mudi
they nolleed and bounced
Table 9: SAE features that contribute to the negation feature discussed in section (continued).
28


Reviews:
[{"review_id": "eWJg541xQs", "reviewer": "Reviewer_QYom", "summary": "This work provides an in-depth analysis of Bilinear Multilayer Perceptrons (MLPs). \nBilinear MLPs use bilinear transformations to achieve nonlinearity for function approximation.\nThe analysis focuses on the spectral decomposition of the bilinear MLP weights. \nThis reveals the low-rank structure of the weights and provides insight into crafting adversarial samples, overfitting, and so on. \nThe low-rank structure is interpretable and shows that components corresponding to small eigenvalues can be removed while preserving performance.\nThe work provides an explainable framework, which would be difficult to achieve without bilinear weights because of the use of nonlinear activation functions.\nThe paper provides experimental results to support its claims.", "questions": "Check weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "ORIGINALITY:\nThis work distinguishes itself from existing approaches to interpretable neural networks by using a unique characteristic of bilinear MLPs. While previous works have utilized bilinear MLP networks, this paper uniquely demonstrates how to exploit their inherent differences from other MLPs to achieve interpretable neural networks.\n\nSIGNIFICANCE:\nInterpretable AI is crucial for understanding how AI models encode information. \nThis work offers a new framework for interpretable neural networks, with the potential to significantly advance the field.\n\nQUALITY:\nThe quality of this work is good. It provides relevant information to support its claims. The experimental results sufficiently support the main claims in this paper.\n\nCLARITY:\nThis paper is well-written. It provides enough information required to put this work into proper perspective. It is also well organized and provides enough background information needed to understand the main components of this work.", "weaknesses": "This is a minor suggestion:\n\nThis paper introduces the use of sparse autoencoders in Section 3.1. \nWhile the stated limitations offer justification for this approach, a more detailed explanation of the role of sparse autoencoders in Section 3.1 would enhance the clarity and comprehensiveness of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QBkWvHJYXa", "reviewer": "Reviewer_9dUb", "summary": "The submission picks up on preliminary ideas developed in “A technical note on bilinear layers for interpretability” (Sharkey, arXiv, 2023) where the idea of using bilinear layers in place of traditional MLPs was promoted to enable forms of interpretability. This is enables by the fact that a bilinear transform of an input vector can be expressed as linear operations with a third-order tensor. This submission follows through by demonstrating several ways in which the tensor can be analyzed with tools from eigen-analysis because input vectors representing features act upon the tensor to yield a matrix (and one can further only consider the symmetric component of this matrix, which has a real eigen-spectrum). \n\nExamples showcased are: \n* Interpretability for image classification with a shallow feed-forward network with one hidden bilinear layer (MNIST and FMNIST)\n    * Eigenvectors of the last layer when projected back into input space visually resemble class prototypes or key features\n    * Top eigenvectors are consistent across training runs and model sizes\n    * Success on the mechanistic interpretability challenge (Casper, 2023)\n    * Successful adversarial masks created from leading eigenvectors for a chosen adversarial class\n* Identifying interactions between sparse autoencoder (SAE) based features in a small language transformer model\n    * An example of identifying a sentiment negation circuit by projecting input features onto leading (both positive and negative eigenvalues) eigenvectors\n    * Pointing out that SAE features tend to be low-rank in terms of the interaction matrix, suggesting promise for future work in circuit discovery in such models", "questions": "None.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality:  While the ideas of using bilinear layers and use of eigenanalysis on the corresponding reduced tensor are borrowed from past work (in particular, “A technical note on bilinear layers for interpretability), the precise demonstrations are new, to my knowledge. \n\nClarity: The paper is well-written, making it easy to follow (even for a reader like me, less familiar with this area of work). The Discussion section well-summarizes the contributions, contextualizes the implications, and identifies key limitations.\n\nQuality: The work seems technically sound, with a reasonable span of experiments to make the point that’s being made. \n\nSignificance: The results are certainly intriguing, and help further the case for adoption of bilinear layers in modern models that are heavily MLP-based.", "weaknesses": "The illustrations in the submissions are some fairly reasonable demonstrations of the interpretability-advantages of networks with bilinear layers. However, these ideas may not be mature enough, hence the lack of robust demonstrations at a broader range of tasks, models, and circuits. Being able to showcase such range would significantly strengthen the influence of the work in the nearer future.\n\nPerhaps some of the existing problems can be solved with advances in discovering semantic “output directions” as part of the method. Even if the top eigenvectors may correspond to distributed semantics, further work to enable interpretation of such semantics would also be significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qp74Fm94IF", "reviewer": "Reviewer_w2jE", "summary": "The paper gives several methods to interpret bilinear multilayer perceptron networks (BMLP). BMLP's use a bilinear activation function and produce comparable results to other activation functions. By posing the bilinear leayrs as tensor operations, the author is able to apply direct interaction, eigendecomposition, and SVD analysis. Using these, the author provides an analysis of internal features, construct low-rank approximations of a model, and constructs adversarial examples. Overall, the paper shows the viability of an interpretable BMLPs.", "questions": "Can you justify the use of the HOSVD as used? Two concerns: 1) The tensor represents a non-linear function. 2) If removed all but but the largest singular values of B, would the resulting tensor functionally approximate B? Also, what were the results of the SVD analysis, what section is that in?\n\n\nMinor Comment:\nGiving some dimensions on W,V could help with the presentation.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Good exhibition of content in all sections, both setup and presentation of results\nEffective choice of experiments with convincing examples and helpful visualization. \nWork is significant: Presents analysis using well understood methods to produce precise, understandable and actionable interpretations of models with competitive performance.", "weaknesses": "I did not notice much analysis using SVD, perhaps I missed it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7igkrGQOXV", "reviewer": "Reviewer_zjLd", "summary": "This paper proposes to use bilinear MLP as a novel approach for weight-based interpretability. The bilinear MLP and its analysis method are introduced first. Then a handful of experiments are presented to demonstrate the usefulness of this approach, including visual feature interpretation in image classification and feature interactions in language tasks.", "questions": "- Could the authors introduce the dimensionality of the variables when defining them? I and perhaps also other readers would find that quite useful for understanding the equations.\n\n- In line 121, is the $W$ in $g(x)=(Wx+b_1)\\odot (Vx+b_2)$ supposed to be bold?\n\n- I'm confused by the $d_{bj}^{in}$ in line 138 and the $d_{in}$ in line 189. Are they the different variables?\n\n- In line 252, I can see that adding dense Gaussian noise to the inputs produces bilinear layers with more intuitively interpretable features. But why is it an effective regularizer?\n\n- Is the bilinear MLP related to the gated linear network literature [1]?\n\n  [1] Veness et al (2021). Gated linear networks. https://arxiv.org/abs/1910.01526", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper is carefully composed. Though there are places that can be further clarified (as mentioned below), the overall structure and logic is good.\n- A variety of experiments are carried out to demonstrate the usefulness of the proposed approach. The bilinear MLP does seem to provide interesting findings and have potential in the realm of mechanistic interpretability.", "weaknesses": "- I couldn't understand what Figure 2 is demonstrating. Firstly, the caption refers to subfigure A, B while the figures do not have A, B labels. Secondly, do you mean that the input images with \"$|$\" or \"$-$\" patterns of exactly the same size and position will have high activation? I imagine that when flattening images to vectors, an image with many small \"$-$\" edges isn't well aligned with the eigenvector that is one big \"$-$\" in the middle of the image. If that's the case, why are the \"$|$\" or \"$-$\" eigenvectors edge detectors?\n- I feel that Section 4 and 5 can be clearer if the authors always specify \"eigenvectors of *what matrix*\" when saying eigenvectors. Section 3 introduced three different methods. So I find the terms like \"decomposition\" and \"eigenvectors\" ambiguous, even after I tried to trace back to the concepts introduced in Section 3. The term \"eigenvector of a model\" is also ambiguous, since models don't really have eigenvectors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)