Attention as a Hypernetwork

## Abstract

Abstract Transformers can under some circumstances generalize to novel problem instances whose constituent parts might have been encountered during training but whose compositions have not.
What mechanisms underlie this ability for compositional generalization?
By reformulating multi-head attention as a hypernetwork, we reveal that a composable, low-dimensional latent code specifies key-query specific operations.
We find empirically that this latent code is predictive of the subtasks the network performs on unseen task compositions revealing that latent codes acquired during training are reused to solve unseen problem instances.
To further examine the hypothesis that the intrinsic hypernetwork of multi-head attention supports compositional generalization, we ablate whether making the hypernetwork generated linear value network nonlinear strengthens compositionality.
We find that this modification improves compositional generalization on abstract reasoning tasks.
In particular, we introduce a symbolic version of the Raven Progressive Matrices human intelligence test which gives us precise control over the problem compositions encountered during training and evaluation.
We demonstrate on this task how scaling model size and data enables compositional generalization in transformers and gives rise to a functionally structured latent space. 1 1 1 Code available at https://github.com/smonsays/hypernetwork-attention

## 1 Introduction

Abstract reasoning is a cornerstone of human intelligence.
Many intelligence tests in psychology measure intelligence using abstract reasoning tasks [e.g. , , ].
Arising from the idea that combinatorial structures are central to human cognitive abilities, the ability of connectionist models to reason has been debated for decades [e.g. , , ].
However, with the success of neural networks at scale, the capacity for reasoning has seemingly come into reach [].
Arguably key to this success is the ability for *in-context learning* paired with the capacity to flexibly recombine learned skills and knowledge to unseen settings.
While these capabilities emerge at least partially from training large models on huge datasets [], it is not clearly understood how the resulting networks implement them - and why they still often fail.

A number of recent studies that aim to illuminate in-context learning abilities have found that transformers can learn to perform gradient-based optimization within sequence [, , ].
They build on the insight that linear attention can be understood as a fast-weight programmer [],
that learns to construct an implicit model in-context, with weights given by a sum of input-dependent outer products [].
Complementing this perspective, [, ] have found that in-context learning creates task vectors in the hidden activations of deeper layers of the network which summarize in-context information and modulate the forward computation.
What is striking is that learning in-context can under some circumstances lead to compositional generalization, that is generalization to unseen combinations of previously observed constituents [, , ], a capacity that neural networks notoriously struggle with [, , ].
The mechanism behind this ability however is not well understood.
Here we aim to shed light on this question.

Central to our results is a novel decomposition of multi-head attention as a hypernetwork, a neural network that reconfigures the computation of another neural network in a data-dependent manner.
Our decomposition differs from that of [], revealing that multiple heads form a linear hypernetwork with key-query specific inputs, illustrated in Figure 1.
As a result, the attention scores along the head index represent a compact code that specifies the operation applied to each key-query pair via a linear value network.
We hypothesize that the shared hypernetwork in each attention layer is what supports the reuse and recombination of learned operations.
In order to investigate whether this constitutes a faithful characterization of what transformers learn in practice, we study in-context learning tasks that require recomposing knowledge obtained during training.
As one such task we develop a challenging abstract reasoning task based on the Raven Progressive Matrices human intelligence test [] and show how multi-head attention develops a structured latent code that captures information about the operations implemented.

Our main contributions can be summarized as follows:

- •
We reformulate standard multi-head attention from a hypernetwork perspective revealing that network computations are specified by a low dimensional latent code.
- •
We show that scaling up model size and data enables transformers to compositionally generalize on abstract reasoning tasks and leads to a structured latent space that is predictive of network function.
- •
To test the hypothesis that the hypernetwork mechanism supports compositionality, we introduce a simple modification to multi-head linear attention that makes the value network nonlinear without introducing additional parameters finding that it improves compositional generalization.
- •
We introduce sraven, a challenging Raven-based symbolic, abstract reasoning task whose difficulty can be controlled parametrically and which offers precise control over the problem compositions encountered during training in order to test for compositional generalization.

## 2 Attention as a hypernetwork

Figure: Figure 1: Hypernetwork attention. A A linear hypernetwork maps a latent code to a set of parameters that configure a value network to process the input. B The attention scores along the head index form the latent code of the hypernetwork. C Multi-head attention can be equivalently expressed as a linear hypernetwork that configures key-query specific computations of a linear value network.
Refer to caption: x1.png

In this section, we will first briefly recapitulate hypernetworks [] and standard multi-head dot-product attention [] before showing how attention can equivalently be expressed as a hypernetwork.
We will then introduce Hypernetwork Linear Attention (HYLA) as a simple modification of linear attention that renders the hypernetwork mechanism inside attention more expressive.

For notation we will use bold lower-case letters for vectors (e.g. $\bm{z}$), bold upper-case letters for matrices (e.g. $\bm{\mathrm{A}}$) and bold, italic upper-case letters for learnable parameter matrices (e.g. $\bm{W}_{V}$).

### 2.1 Multi-head attention

A hypernetwork is a neural network $h(\bm{z};\bm{\theta})$ parameterized by $\bm{\theta}$ that takes as input a latent code $\bm{z}$ and outputs parameters $\bm{\mathrm{W}}$ that parameterize a value network $f(\bm{x};\bm{\mathrm{W}})$ which then processes the inputs (see Figure 1A).
The latent code, $\bm{z}$, is typically low-dimensional and can be interpreted as a specification of the computation performed by the value network.
As we will see in the following, multi-head attention can be viewed as a linear hypernetwork producing the weights for a linear value network.
Note that this is different from the observation that linear attention can be understood as a fast weight programmer [] where the fast weights of a value network are constructed as a sum of outer products over the key indices instead of the head indices.

Self-attention maps a sequence of inputs $\bm{\mathrm{X}}\in\mathbb{R}^{T\times D}$ to a sequence of outputs $\bm{\mathrm{Y}}\in\mathbb{R}^{T\times D}$.
For each attention head $h\in\{1,\ldots,H\}$ the inputs are projected into keys $\bm{\mathrm{K}}_{h}=\bm{\mathrm{X}}\bm{W}^{\text{key}}_{h}$ and queries $\bm{\mathrm{Q}}_{h}=\bm{\mathrm{X}}\bm{W}^{\text{query}}_{h}$ using head-specific projection matrices $\bm{W}^{\text{key}}_{h},\bm{W}^{\text{query}}_{h},\bm{W}^{\text{value}}_{h}\in
\mathbb{R}^{D\times D^{\text{key}}}$.
The attention matrix can then be obtained as

$$ $\displaystyle\bm{\mathcal{A}}=\sigma\bigl{(}\;\left[\bm{\mathrm{\tilde{A}}}_{1 }\;\bm{\mathrm{\tilde{A}}}_{2}\;\ldots\;\bm{\mathrm{\tilde{A}}}_{H}\right]\; \bigr{)}\text{ with }\bm{\mathrm{\tilde{A}}}_{h}=\frac{\bm{\mathrm{Q}}_{h}\bm{ \mathrm{K}}_{h}^{\top}}{\sqrt{D^{\text{key}}}}$ (1) $$

where $\bm{\mathcal{A}}=(a_{h,q,k})_{h,q,k}\in\mathbb{R}^{H\times T\times T}$ stacks the head-specific unnormalized attention matrices $(\bm{\mathrm{\tilde{A}}}_{h})_{h}$ into a three-dimensional array and applies a normalization operation $\sigma(\cdot)$.
For linear attention, $\sigma(\cdot)=\mathrm{Id}(\cdot)$ is simply the identity whereas for softmax attention $\sigma(\cdot)=\mathrm{Softmax}(\cdot)$ applies the softmax operation for each head and query index independently across the key indices.
Considering a single query index $q\in\{1,\ldots,T\}$ and $\mathbf{x}_{k}$ the $k$-th row of $\bm{\mathrm{X}}$, multi-head attention (MHA) can be equivalently expressed as a hypernetwork with a key-query specific latent code:

$$ $\displaystyle\text{MHA}_{q}(\bm{\mathrm{X}}):=$ $\displaystyle\bm{W}^{\text{out}}\bigoplus_{h=1}^{H}\sum_{k=1}^{T}a_{h,q,k}\; \bm{W}^{\text{value}}_{h}\mathbf{x}_{k}$ (2) $\displaystyle=$ $\displaystyle\sum_{h=1}^{H}\bm{W}^{\text{out}}_{h}\sum_{k=1}^{T}a_{h,q,k}\;\bm {W}^{\text{value}}_{h}\mathbf{x}_{k}$ (3) $\displaystyle=$ $\displaystyle\sum_{k=1}^{T}\left(\sum_{h=1}^{H}\underbrace{a_{h,q,k}}_{\text{ latent code}}\;\underbrace{\bm{W}^{\text{out}}_{h}\bm{W}^{\text{value}}_{h}}_{ \text{hypernetwork}}\right)\mathbf{x}_{k}$ (4) $\displaystyle=$ $\displaystyle\sum_{k=1}^{T}\underbrace{\bm{\mathrm{W}}_{q,k}}_{\text{value network}}\mathbf{x}_{k}$ (5) $$

where $\oplus$ denotes concatenation along the leading axis and the $\bm{W}^{\text{out}}_{h}\in\mathbb{R}^{D^{\text{value}}\times D}$ are defined as the slices of $\bm{W}^{\text{out}}$ such that $\bm{W}^{\text{out}}=\bigoplus_{h=1}^{H}\bm{W}^{\text{out}}_{h}$.
See Figure 1C for a diagram of the computation.

As a result, the key-query specific attention scores over the multiple heads form a latent code where the number of heads $H$ corresponds to the latent dimension of a hypernetwork.
Accordingly, the dot products between any key-query pair over all heads can be interpreted as an amortized inference process to configure the computation implemented in the linear value network.

The key significance of our finding is that it suggests that multi-head attention can implement specialized, reusable operations through the latent code of the hypernetwork.
Notably, it has been shown that hypernetworks support compositional generalization [], providing a possible explanation for similar capabilities observed in transformers [, , ].
Note that inside of multi-head attention the same hypernetwork is used for every key-query index.
Specifically, the hypernetwork linearly combines the *same* matrices $(\bm{W}^{\text{out}}_{h},\bm{W}^{\text{value}}_{h})_{h}$ according to the weights given by $a_{h,q,k}$.
This incentivizes reuse of latent codes and their associated operations implemented through the value network.

### 2.2 Hypernetwork Linear Attention (HYLA)

If the perspective of multi-head attention as a hypernetwork indeed reflects how it operates in practice, it should be possible to modify multi-head attention in a way that further reinforces the hypernetwork mechanism.
For instance both the hypernetwork and the value network in multi-head attention are simple linear networks and could be replaced by nonlinear networks.
Furthermore the normalization operation $\sigma(\cdot)$ could be used to encode inductive biases such as competition or sparsity in the latent code by imposing structural constraints.

Here, we focus on a simple modification to linear attention where we make the value network nonlinear and normalize the latent code along the head index (instead of along the key index as in standard softmax attention).
Specifically, we define Hypernetwork Linear Attention (HYLA) to use a single hidden layer value network with nonlinearity.
This should increase the expressivity of the subfunctions that are learned and recomposed by the layer.
Noting that the output and value projections, $\bm{W}^{\text{out}}_{h},\bm{W}^{\text{value}}_{h}$, already form a deep linear network in standard multi-head attention (c.f. Eq. 4) this can be achieved without adding additional parameters as follows:

$$ $\displaystyle\text{HYLA}_{q}(\bm{\mathrm{X}})=$ $\displaystyle\sum_{k=1}^{T}\left(\sum_{h=1}^{H}a_{h,q,k}\;\bm{W}^{\text{out}}_ {h}\right)\phi\left(\sum_{h=1}^{H}a_{h,q,k}\;\bm{W}^{\text{value}}_{h}\mathbf{ x}_{k}\right)$ (6) $\displaystyle=$ $\displaystyle\sum_{k=1}^{T}\bm{\mathrm{W}}^{\prime}_{q,k}\phi(\bm{\mathrm{W}}_ {q,k}\mathbf{x}_{k}),$ (7) $$

where $\phi(x)$ is an element-wise nonlinearity; here we set it to be a ReLU, $\phi(x)=\max(0,x)$.

Additionally, we use $\sigma(\cdot)=\text{RMSHead}(\cdot)$, to normalize the attention scores for each key and query index independently across the head indices using the $\text{RMSNorm}(\bm{x})=\frac{\bm{x}}{\sqrt{\frac{1}{n}\sum_{i=1}^{n}x_{i}^{2}}}$.
This ensures that the parameters of the value network generated by the hypernetwork are appropriately initialized to prevent exploding/vanishing
gradients when backpropagating through it.
The resulting HYLA-layer is a simple drop-in replacement to standard attention.
Notably, since the normalization operation $\sigma(\cdot)$ is local to the query and key index, it requires no communication across keys as opposed to softmax attention.
By making the value network more expressive, we allow the network to implement more complex operations for a given latent code which we hypothesize strengthens the ability for compositional generalization.

## 3 Compositional generalization on fuzzy logic functions

Figure: Figure 2: Compositional generalization on fuzzy logic functions. A We split fuzzy logic functions according to their constituent terms into train and out-of-distribution (OOD) set to measure compositional generalization in a sequence model that learns these functions in-context. B The latent code of the query token is predictive of the constituent terms underlying each task. Shown is the F1 score on unseen tasks of logistic regression classifiers for each layer and term, trained to predict the terms underlying each task based on the attention scores across the head index for the query attending to itself. C Task performance on unseen tasks reported as OOD $R^{2}$ for varying number of in-context examples and fraction of tasks held out during training. D tSNE visualization of the latent codes (attention scores across the head index for the query token attending to itself) colored according to the target label (top) and colored according to the first term of the fuzzy logic function of each task (bottom).
Refer to caption: x2.png

We start by developing a fuzzy logic task with compositional structure, inspired by a similar task previously considered by Rahaman et al. [].
Learning such tasks in an in-context learning setting will allow us to study the ability of multi-head attention-based models to compositionally generalize and analyze the structure of their latent code when solving each task.

Given a set of $L$ scalars $\{x_{1},...,x_{L}\}$ with $x_{i}\in[0,1]$ we define fuzzy logic operators on the elements of this set as the Zadeh operators []:

$$ $\displaystyle x_{i}\land x_{j}:=\min(x_{i},x_{j}),\quad x_{i}\lor x_{j}:=\max( x_{i},x_{j}),\quad\bar{x}_{i}:=1-x_{i}$ (8) $$

These operators have the property that for boolean inputs $x_{i}\in\{0,1\}$ the outputs coincide with the outputs of the corresponding Boolean operation.
Each task is then defined as a function in disjunctive normal form consisting of $K$ terms such as for example for $L=4$ and $K=2$

$$ $\displaystyle f(x_{1},x_{2},x_{3},x_{4})=\underbrace{(x_{1}\land x_{2}\land x_ {3}\land x_{4})}_{\text{Term 1}}\lor\underbrace{(\bar{x}_{1}\land x_{2}\land \bar{x}_{3}\land x_{4})}_{\text{Term 2}}$ (9) $$

where each term is a conjunction of all variables with negations applied to some of them.
In order to test for compositional generalization, we index each of the $2^{L}$ possible terms, generate all possible combinations of $K$ terms and hold out a fraction of the combinations for out-of-distribution (OOD) testing, ensuring that their constituent terms have been encountered in some tasks during training.

For each task, we present $N$ examples in-context (see Figure 2A) where each token has dimension $L+1$ consisting of the concatenated inputs sampled uniformly from $[0,1]$ and the corresponding target of the function for the current task.
For the final token we mask out the target and use the output logits of the final token to predict the target, measuring the loss using the mean-squared error.

### 3.1 Compositional generalization

We first investigate how well multi-head attention based transformer models are able to learn our fuzzy logic tasks in-context and compositionally generalize to held out tasks.
We vary the number of examples presented in context as well as the fraction of combinations of fuzzy logic terms held out from the training set.
Figure 2C compares standard multi-head softmax attention and linear attention as well as our HYLA model in terms of their coefficient of determination ($R^{2}$) on the queries of held out tasks.
We find that all models are able to solve the fuzzy logic task given sufficient examples are presented in context.
As the fraction of held out tasks increases, compositional generalization performance starts decreasing for all models.
Interestingly, HYLA is less affected by this decline than softmax and linear attention.
We hypothesize that the more expressive value network allows to more efficiently represent the nonlinear terms being composed and thereby strengthens the inductive bias towards compositionality.
Consistent with this interpretation, in-distribution the models show comparably small differences in performance (see A2).
In Appendix A.1 we conduct additional model ablations that separate the contributions of the RMSHead normalization and the nonlinearity added to the value network, showing that both components in combination are required to obtain the observed performance improvements.

Our compositional split only contains novel combinations of known terms.
A priori, it might be possible that the system learns to compose the basic fuzzy logic operations in a way that allows it to generalize to any fuzzy logic function, including those that contain novel combinations of unknown terms.
However, testing on functions obtained as novel combinations of $K$ unknown terms, we find that none of the models considered here is able to solve such a task (see Figure A4 in the appendix).

### 3.2 Latent code structure

Next, we analyze the structure of the latent code for the different models solving the task.
As suggested by Eq. 4, the attention scores for a given key-query pair across the head dimension configure the computation performed in the value network and accordingly we would expect it to reflect the fuzzy logic function that needs to be implemented to correctly predict the output of the query token.
To test this hypothesis, we collect the attention scores for the final query token attending to itself, obtaining a vector of dimension $H$ for each layer and task.
For this purpose we use the tasks held out from training, varying the query token while keeping inputs shown in-context fixed.
In order to visualize the $H$-dimensional points we thereby obtain for each layer, we reduce the dimensionality using tSNE [].
The results of this analysis are shown in Figure 2D (for an extended figure showing all layers see Figure A3).
In line with our hypothesis, the latent codes for each model form clusters.
The structure of these clusters is only partially explained by the target value of the function for a given query value.
Strikingly, coloring data points according to the fuzzy logic terms present underlying each function reveals that clusters in the latent codes closely correspond to the functions.
Indeed, it is possible to decode the terms underlying each function from the latent code using a logistic regression classifier which is trained to predict the function terms given the latents and is evaluated on held out tasks as shown in Figure 2B.

## 4 sraven : Symbolic raven

Figure: Figure 3: sraven. A Illustration of sraven task generation and the construction of the compositional generalization split. B Example problem instance illustrating a key challenge of the original Raven Progressive Matrices of finding correspondences (adapted from []). When attempting to solve this instance, different hypotheses over which figural elements are governed by a consistent rule across rows are possible. This is akin to different orderings of the symbolic features.
Refer to caption: x3.png

Next we will introduce an abstract reasoning task based on Raven Progressive Matrices [] we refer to as sraven.
We develop this task to provide a challenging benchmark that specifically probes symbolic reasoning capabilities while giving us fine-grained control over the constituent parts of each task in order to assess compositional abilities and analyze latent code structure.

### 4.1 Abstract reasoning based on symbolic Raven Progressive Matrices

The original Raven Progressive Matrices test is a classic human intelligence test that probes the ability to induce abstract relations [].
It requires test subjects to generate and verify a potentially large number of hypotheses over rules that can parsimoniously explain a given problem instance [].
While there have been previous machine learning datasets inspired by Raven Progressive Matrices [, , , ], we seek to study a setting that is both symbolic, compositional and models an aspect which to the best of our knowledge has not been explicitly considered in prior benchmarks:
we seek to model the key difficulty of Raven tasks referred to as finding correspondences by Carpenter et al. [] which requires to search through a large number of possible hypotheses.
We discuss existing Raven-based benchmarks in Section 6.

### 4.2 Task description

Figure 3A illustrates the generative process of sraven.
For each task, a matrix of eight context panels is presented and the goal is to predict the contents of the final query panel.
In the original Raven Progressive Matrices Test, each panel typically displays a combination of geometrical shapes as for instance shown in Figure 3B.
For our symbolic version, each panel is defined as a vector of integers of length $M$ (in our experiments we set $M=4$ unless stated otherwise).
For each dimension $m\in\{1,2,\ldots,M\}$ one of $R=8$ possible rules is sampled as well as three corresponding sequences of three integers generated by the respective rule (see Figure 3A and Appendix B.3 for more details on the rules).
We restrict rules to apply within each row while generally rules could also be applied both across columns and across rows.
As a result we obtain nine panels where each row can be explained by the same set of underlying rules.
In order to maintain a finite set of possible symbols both for the context panels and the query panel, we limit all tasks to contain only integers $\{0,1,2,\ldots,K-1\}$ (in our experiments we set $K=8$).
Arithmetic rules such as progression, sum or difference are therefore defined using modular arithmetic modulo $K$.

We use the first eight panels as context panels and present them sequentially to a sequence model which needs to predict each dimension of the final query panel independently.
A prediction is considered correct if and only if all subpredictions were correct.
In the original Raven task, a number of possible answer panels are presented alongside the context panels and test subjects are asked to pick the one which contains the correct answer [].
Since in our case the correct answer will consist of a combination of a finite set of known symbols, we can ask the agent solving the task to directly predict the missing symbols.
Conveniently this helps avoid potential shortcuts for solving a task that might arise from a biased generative process to create multiple-choice answers as a popular prior Raven-based benchmark has been found to be affected by [, ].

One of the key features of sraven is its compositional structure.
Each task consists of a combination of a finite set of rule combinations.
To systematically test whether a system trained on a subset of rule combinations can generalize to unseen combinations, we split all rule combinations into a train set and a test set.
Unless noted otherwise, we hold out $25\%$ of all possible rule combinations for evaluation in our experiments.
Note that permutations of rule combinations (e.g. AB and BA) are considered to be the same and will only appear in one of the two sets.
This ensures that at test time the agent is confronted with instances of tasks whose underlying combination of rules it has never encountered during training.

### 4.3 Finding correspondences

Figure: Figure 4: Scaling data and model size on sraven. A Compositional generalization measured by OOD accuracy as a function of the number of training problem instances for different widths (scaling embedding and key-query-value dimensions) . B Same as A but increasing depth instead of width.
Refer to caption: x4.png

Making the task symbolic instead of presenting images of geometric shapes comes with the advantage of a smaller input space allowing us to scale to larger model and data sizes in our experiments.
However, it removes the visual representation learning step of discovering the right symbolic abstractions.
We argue that while this is a difficult problem worth studying by itself, it is typically not the main source of difficulty for the original Raven tasks which mostly consist of simple visual features such as numerosity, shape, or position.
As Carpenter et al. [] argue in their theoretical account of the original Raven test, what makes many tasks difficult is the problem of finding correspondences of rules to feature dimensions.

This is best illustrated when considering the example in Figure 3B showing an adapted version of Figure 5 of [].
When first attempting to solve this task, a test subject might hypothesize that there is a rule guiding the numerosity and orientations per shape.
For example one might to try find a rule that governs all curved shapes, one rule that governs all lines and one rule that governs all rectangles.
Translated into a symbolic encoding, this is akin to ordering symbols by the shape and trying to find rules along each dimension in this ordering.
However, this approach turns out to not provide a parsimonious answer.
The correct answer can instead be obtained by grouping the objects by their orientations, i.e. by finding a rule that applies to all horizontal objects and a rule that applies to all vertical objects.
In the symbolic encoding this new correspondence of features amounts to a new ordering of the same symbolic encoding where symbols are now ordered by orientation and one needs to find rules guiding the numerosity and shape per orientation.

To model the property of finding correspondences in sraven we similarly allow for permutations of the features.
Specifically, we sample and apply a column-specific permutation of the features when generating each task such that the entries of the vectors encoding each panel are permuted with a consistent permutation for a given column across all rows.
This makes finding the correct solution difficult since the number of possible hypotheses grows significantly (see Appendix B.1 for more details).
Note that in principle a problem instance might have multiple valid but divergent answers making the task ill-posed.
In the settings considered here this happens for less than half a percent of the problem instances.
See Appendix B.2 for more details on handling such ambiguities.

### 4.4 Results

Figure: Figure 5: Latent code structure of sraven. A tSNE visualizations of the final layer latent codes for the final query token colored by the magnitude of the predicted target value. B Same as A but colored by the ground-truth rule the model needs to apply to the query token to generated the correct prediction. C OOD accuracy for varying numbers of attention heads. For a single head the hypernetwork mechanism is absent which hampers OOD generalization. D The difficulty of sraven can be parametrically controlled by varying the number of modules $M$. E Heatmap showing the pairwise cosine similarity between the average latent code of each rule for HYLA revealing how semantically related rules form clusters. For instance rule F (sum) and G (difference) are implemented with a very similar latent code indicating that the same code might be reused by flipping the sign of the operands. F Decoding performance of a logistic regression classifier trained to predict the ground-truth sraven rule based on the latent code at the final query token of training tasks and evaluated on unseen OOD tasks revealing that the latent code is predictive of the implemented rule.
Refer to caption: x5.png

#### Scaling model size and data enables compositional generalization.

We first evaluate to what extent transformers can compositionally generalize on sraven as we increase the number of problem instances trained on and scale up the models both in terms of width and depth.
Figure 4 shows the results of these scaling experiments.
Given sufficient data and model size, all model classes are able to solve 80% of problem instances created from tasks held out during training.
The inductive bias of making the value network more expressive by introducing a nonlinearity in HYLA seems to be beneficial for this task.
Especially when presented with few problem instances and for small model sizes it outperforms both linear and softmax attention.
In contrast, disrupting the hypernetwork mechanism by decreasing the number of heads to $H=1$ noticeably decreases performance as shown in Figure 5C.

#### The latent code is structured according to the rules of the task.

We next analyze the latent code for the different models solving the task.
To this end we collect the attention scores of the final query token attending to itself.
Following the perspective of attention as a hypernetwork, we expect latent codes corresponding to the same rule to form clusters.
Indeed, Figure 5B reveals a strikingly structured code for the final layer across models with clusters closely matching the rule to be inferred to correctly predict the query output for each given task.
In comparison, Figure 5A colors the points in latent space by the magnitude of the inputs the query needs to attend to make the correct prediction.
This tends to explain clusters more prominently in early layers as shown in the appendix in Figure A5 and for a 16 layer softmax transformer in Figure A6.

To obtain a better understanding of the semantics of the latent code, we explore the pairwise cosine similarity between average latent codes for each rule across tasks for HYLA.
Figure 5C shows that semantically related rules like sum/difference can form clusters, suggesting that latent codes might even be reused to solve related task operations.
Finally, we train a logistic regression classifier to predict the ground-truth sraven
rule based on the latent code at the final query token of in-distribution sraven tasks and evaluate it
on unseen OOD tasks.
This reveals that especially for later layers the latent code is strongly predictive of the subfunctions performed by the network.

Figure: Figure 6: Language modeling. Performance of autoregressively trained decoder-only transformer models with 50M parameters over the course of training on the C4 dataset with 130B tokens [].
Refer to caption: x6.png

## 5 Language modeling

Multi-head attention has proven especially effective for language modeling.
With language itself being highly compositional, it stands to question whether the hypernetwork mechanism inside of multi-head attention might play a part in explaining its success.
To shed light on this question, we conducted language modeling experiments and evaluate the performance of HYLA in comparison to linear and softmax attention.
We train decoder-only transformer models with 50M parameters autoregressively for 130 Billion tokens on the C4 dataset [].
We find that strengthening the hypernetwork mechanism via HYLA improves performance over linear attention and performs closely to softmax attention despite being a linear attention variant itself.
This is noteworthy considering the importance of softmax in language modeling hypothesized to be due to its role in binding and associative recall problems on which linear attention methods typically struggle [, , ].
The fact that reinforcing the hypernetwork mechanism inside of multi-head attention as done by HYLA helps close the gap of linear attention to softmax attention suggests that the hypernetwork mechanism might be of practical relevance for understanding large-scale transformer models.

## 6 Related work

#### The role of multiple heads in attention.

Prior work that has studied the question why the attention mechanism benefits from multiple heads has counterintuitively found that it is possible to prune almost all but one head in some layers after training, sacrificing only relatively small amounts of performance [, ].
It has therefore been speculated that equipping the attention mechanism with multiple heads primarily aids stability during training [].
The hypernetwork perspective of attention offers an alternative account, suggesting that multiple heads create a way of configuring compositional computations in the value network.
Seen from this perspective, the observation of singular heads dominating could point to a connection to the phenomenon of module collapse often observed when training modular systems in practice [, , , ].

#### Compositional generalization.

Consistent with our observation on scaling model size on the sraven task, [] find that as pretrained large language models are scaled, their ability to compositionally generalize on in-context semantic parsing tasks improves.
Similarly, outside the in-context learning regime, [] have demonstrated that pretrained large language models outperform many architectures specialized towards compositional generalization.
Still, to what extent the ability for compositional generalization extends beyond the training distribution remains openly debated [, , ].

#### Raven-based tasks.

A number of Raven-inspired tasks have been introduced to assess abstract reasoning capabilities of neural networks [, , , ].
Common to all these variants including our version is an underlying generative task model based on a finite number of possible rules that are applied on one or more of the feature dimensions for each panel.
Different from our version, these variants render the problem instances as images using simple geometrical shapes resulting in larger datasets and computational demand.
Still we argue that the so generated tasks are not necessarily harder than the tasks of sraven, given that sraven models an additional difficulty of finding correspondences as detailed in Section 4.3 and its difficulty can be controlled parametrically; for instance by increasing the number of features.

## 7 Discussion

We have proposed a novel decomposition of multi-head attention revealing that it can be interpreted as a hypernetwork that composes value networks specific to each key-query pair.
Consistent with this perspective, we find empirically that the attention scores over the heads for each key-query pair form a functionally structured space, identifying reusable subfunctions in two abstract reasoning tasks.
Furthermore, modifying multi-head attention to strengthen the hypernetwork mechanism improves compositional generalization on these tasks.

Adopting the hypernetwork perspective more broadly, multi-head attention can be interpreted as a particular choice for the level of granularity at which the hypernetwork parameterizes the value network: The value networks are key-query specific and
are followed by a pooling step that sums the key-query specific outputs over the key index.
In principle, other levels of granularity are possible.
For instance the
hypernetwork could parameterize a query specific value network which subsumes the aggregation
over the key index.
In the extreme case, it could directly parameterize the full sequence operation potentially facilitating the re-use of sequence-level operations.
This also offers an interesting connection to attention-based graph neural networks [, ].
Classically, a non-causal single layer transformer is seen as a fully connected graph where each token corresponds to a node and aggregation is done via attention [].
Our derivation suggests an alternative interpretation where the message function is a hypernetwork that subsumes the attention weights while the aggregation becomes the sum operator.
Given the crucial role that aggregation plays in graph neural networks [, ], a natural question is whether different pooling operators should be considered more generally for multi-head attention.

#### Limitations.

We have focused our analysis on models trained from scratch in order to give us fine-grained control over what data is encountered during training.
Many interesting behaviors emerge in large-scale models pretrained on diverse tasks.
Investigating whether the resulting models form a similarly structured latent code is an interesting avenue for future work.

#### Ethics statement.

This paper conducts foundational research aiming to illuminate the mechanisms of the widely used multi-head attention layer.
While we foresee no immediate negative societal impact, we hope that it may improve our understanding of this widely deployed technology.

#### Reproducibility statement.

To ensure the reproducibility of our work, we are providing the code for all our experiments as part of the supplementary material.
We further expand upon the experimental details explained in the main text in Appendix C.

#### Acknowledgements.

We would like thank Angelika Steger as well as Alexander Meulemans, Johannes von Oswald, Maciej Wołczyk and Owen He for fruitful discussions throughout the development of the project.
This research was supported by an Ambizione grant (PZ00P3_186027) from the Swiss National Science Foundation and an ETH Research Grant (ETH-23 21-1).

## Appendix A Additional results

### A.1 Model ablations

Compared to linear attention, HYLA introduces three modifications: (1) normalization of the attention scores across the head indices using RMSNorm, (2) applying the attention-weighted sum operation also to the output projection and (3) inserting a nonlinearity in the value network (c.f. the pseudocode for HYLA shown in Figure 1).
To delineate the contributions of these modifications, we run an ablation study on the fuzzy logic task and sraven shown in Table A1.
We find that while simply applying RMSHead to linear attention (Linear Attention + RMSHead) and also the attention-weighted sum operation without nonlinearity (HYLA $-$ nonlinearity) both by themselves can slightly boost performance, the full HYLA model generally performs best.

#### Increasing depth of the value network.

To further test our hypothesis that it is the hypernetwork mechanism inside multi-head attention which improves compositional generalization, we explore making the value network configured by the hypernetwork deeper, adding an additional layer as follows,

$$ $\displaystyle\text{HYLA}^{+}_{q}(\bm{\mathrm{X}})=$ $\displaystyle\sum_{k=1}^{T}\left(\sum_{h=1}^{H}a_{h,q,k}\;\bm{W}^{\text{out}}_ {h}\right)\phi\left(\sum_{h=1}^{H}a_{h,q,k}\;\bm{W}^{\text{value}^{\prime}}_{h }\phi\left(\sum_{h=1}^{H}a_{h,q,k}\;\bm{W}^{\text{value}}_{h}\mathbf{x}_{k} \right)\right),$ (10) $$

where $\bm{W}^{\text{value}^{\prime}}_{h}\in\mathbb{R}^{D^{\text{key}}\times D^{\text
{key}}}$ is a second layer value matrix which increases the number of parameters of this model.
Consistent with our hypothesis, this model also demonstrates strong OOD performance as reported in Table A1.

#### Replacing RMSHead normalization with the softmax.

We perform an additional ablation where we replace the RMSHead normalization of HYLA with the standard softmax.
As shown in Table A1 this model performs poorly.
This might be partially explained by the fact that with the softmax the linear combination of weights produced by the hypernetwork is no longer guaranteed to maintain the scale of a standard neural network initialization corresponding to the value network.

**Table A1: Model ablations. Left column reports the OOD R2 score on the fuzzy logic task studied in Figure 2 for a sequence length of 32 and 70% of tasks held out from training. Right column reports the OOD accuracy on the sraven task studied in Figure 4 for a 4-layer transformer trained on 20M problem instances. We report the standard error over 3 seeds.**
|  | Fuzzy logic: | sraven: |
| --- | --- | --- |
|  | OOD R2 | OOD accuracy |
| Softmax attention | $63.28\pm 2.31$ | $56.56\pm 1.05$ |
| Linear attention | $59.89\pm 5.22$ | $56.30\pm 1.11$ |
| HYLA | $81.13\pm 7.77$ | $69.13\pm 1.90$ |
| Linear attention $+$ RMSHead | $68.93\pm 5.10$ | $59.01\pm 2.50$ |
| Linear attention $+$ RMSHead $+$ nonlinear value | $56.91\pm 4.25$ | $55.38\pm 0.32$ |
| HYLA $-$ RMSHead | $81.27\pm 7.88$ | $62.53\pm 5.73$ |
| HYLA $-$ nonlinearity | $84.54\pm 5.32$ | $56.54\pm 0.21$ |
| HYLA $-$ nonlinearity $-$ RMSHead | $79.42\pm 7.43$ | $56.33\pm 1.41$ |
| HYLA $-$ RMSHead $+$ softmax | $70.98\pm 7.08$ | $38.21\pm 2.98$ |
| HYLA $+$ deep value network | $87.65\pm 4.54$ | $66.59\pm 0.08$ |

Figure: Figure A2: Training coefficient of determination ($R^{2}$) for varying sequence length and fraction of held out tasks when in-context learning of fuzzy logic functions.
Refer to caption: x7.png

Figure: Figure A3: tSNE visualization of the attention scores across the head index for the query token on the fuzzy logic task for a sequence length of 32 and 70% of tasks held out from training colored according to (A) target label (B) fuzzy logic function, (C) term 1 and (D) term 2.
Refer to caption: x8.png

Figure: Figure A4: Performance on fuzzy logic functions containing terms not encountered during training. As expected all methods fail in this setting. Same setup as in Figure 2B.
Refer to caption: x9.png

Figure: Figure A5: Latent code on sraven. A tSNE visualization of the latent code predicting the final query token colored by the magnitude of the predicted target value for 4-layer transformers trained on 40M problem instances. B Same as A but colored by rule.
Refer to caption: x10.png

Figure: Figure A6: Latent code visualization in sraven for a softmax transformer of depth 16. A tSNE visualization of the latent code predicting the final query token colored by the rule to be inferred for a softmax transformer of depth 16 trained on 40M problem instances. B Same plot as in A but colored by the magnitude of the predicted target value.
Refer to caption: x11.png

## Appendix B sraven

### B.1 Number of possible tasks and instances

Figure: Figure A7: A Number of possible sraven tasks as a function the number of features ($M$). B Number of possible sraven problem instances as a function of the number of features ($M$) and the number of feature values ($K$).
Refer to caption: x12.png

In the general case, the number of possible tasks in sraven is a product of the number of possible rule combinations $\binom{R+M-1}{M}$ and the number of possible unique permutations $((M!)^{G-1}-M)$ where $G$ is the number of columns of the grid (in Raven-based tasks typically $G=3$).
Figure A7A shows the number of possible sraven tasks and Figure A7B shows the number of possible sraven problem instances.
Note specifically that the latter is orders of magnitudes larger than the number of training examples in our experiments for $M=4$, $K=8$.

### B.2 Ambiguity

**Table A2: Monte Carlo estimation of fraction of ambiguous examples over 4096 problem instances $\pm$ standard error. Bold used to denote the setting used for all experiments in this paper.**
| Number of features ($M$) | Number of feature values ($K$) | Estimated fraction of ambiguous examples |
| --- | --- | --- |
| 4 | 4 | $0.0642\pm 0.0038$ |
| 4 | 8 | $\bm{0.0032\pm 0.0009}$ |
| 4 | 16 | $0.0005\pm 0.0003$ |

In principle, a problem instance generated given a set of rules and permutations might have an ambiguous answer.
That is there is at least one other set of rules and permutations that fit all context panels but prescribe a different final query panel.
One possibility is to check for such examples when generating task instances and reject ambiguous ones.
This process is prohibitively expensive as it requires a search over the exponentially many possible permutations.
We therefore resort to a Monte-Carlo estimation of the fraction of ambiguous instances.
Table A2 shows the results for various parameterizations of the sraven task.
In the setting we consider here, ($M=4,K=8$), the probabilities are negligible and we do not take measures to filter out ambiguous examples given the computational overhead it produces in the data generation process.

### B.3 Rules

sraven contains the following rules:

- 1.
constant: Each row consists of a random but fixed integer in the range $1$ to $K$.
- 2.
progression (+1): The first element of each row is sampled uniformly at random and incremented by 1 modulo $K$ for each successive column.
- 3.
progression (+2): The first element of each row is sampled uniformly at random and incremented by 2 modulo $K$ for each successive column.
- 4.
progression (-1): The first element of each row is sampled uniformly at random and decremented by 1 modulo $K$ for each successive column.
- 5.
progression (-2): The first element of each row is sampled uniformly at random and decremented by 2 modulo $K$ for each successive column.
- 6.
addition: Two elements are sampled uniformly at random for each row and added modulo $K$ to obtain the last column.
- 7.
subtraction: Two elements are sampled uniformly at random for each row and subtracted modulo $K$ to obtain the last column.
- 8.
distribute three: Three elements are sampled uniformly at random and presented in three independently sampled random permutations for each row.

## Appendix C Experimental details

#### Architecture.

For all models we use a standard decoder-only transformer architecture where each block is structured as

$$ $\displaystyle\bm{\mathrm{Z}}$ $\displaystyle=\texttt{MultiHeadAttention}(\texttt{LayerNorm}(\bm{\mathrm{X}})) +\bm{\mathrm{X}}$ $\displaystyle\bm{\mathrm{Y}}$ $\displaystyle=\texttt{FeedForward}(\texttt{LayerNorm}(\bm{\mathrm{Z}}))+\bm{ \mathrm{Z}}.$ $$

MultiHeadAttention is either linear attention, softmax attention or hypernetwork linear attention (HYLA) and uses T5-style relative positional embeddings [].
For the FeedForward layer, we employ a standard single-hidden layer MLP with GeLU nonlinearity applied to each position in the sequence independently.
For our language modeling experiments we use rotary positional encodings [].

#### Tokenization.

For the fuzzy logic task we directly use the concatenated input examples as tokens.
Similarly for sraven we use one-hot encodings of the integer inputs as the input tokens.
In both cases the input tokens are mapped into the embedding space of the transformer using a fully connected dense layer and another fully connect dense layer is used to produce the output logits.
For the language modeling experiment we use the default sentencepiece tokenizer [] of NanoDo [] with a vocabulary size of $32000$ and use the transposed embedding matrix to produce the logits.

#### Optimization.

We use the AdamW optimizer [] with linear warmup starting from a learning rate of 0 0 followed by a cosine decay learning rate scheduler [] that decays the learning rate to $0.1$ times the base learning rate.
We exempt biases and LayerNorm parameters from weight decay.
In the fuzzy logic task we use the mean-squared error on the logits of the query token whereas in sraven we use the softmax cross-entropy loss on the logits of all query tokens.
The language modeling task uses an autoregressive softmax cross-entropy loss with causal masking applied to the attention mechanism.

#### Hyperparameters.

For all tasks and models we perform a grid search over the learning rate, weight decay and warmup steps.
We report the search grid as well as all other hyperparameters in Table A3.

**Table A3: Hyperparameters for all tasks and methods. Lists of values indicates that a grid search over all points contained in the list have been conducted and for each method the optimal combination has been picked. The width factor multiplies the embedding dimension, key-query-value dimension and MLP dimension.**
| Parameter | Fuzzy logic | sraven | Language modeling |
| --- | --- | --- | --- |
| batch_size | $128$ | $128$ | $256$ |
| num_layers | $2$ | $4$ / $8$ / $16$ | $6$ |
| width_factor | $1$ | $1$ / $2$ / $4$ | $1$ |
| emb_dim | $128$ | $128$ | $512$ |
| kqv_dim | $16$ | $64$ | $64$ |
| mlp_dim | $256$ | $256$ | $2048$ |
| num_heads | $8$ | $16$ | $8$ |
| learning_rate | $[0.001,0.003]$ | $[0.0003,0.001]$ | $[0.001,0.003]$ |
| weight_decay | $[0.03,0.1]$ | $[0.1,0.3]$ | $[0.1,0.03]$ |
| warmup_steps | $100$ | $[1000,3000]$ | $1000$ |

## Appendix D Additional details

### D.1 Compute resources

We used a Linux workstation with two Nvidia RTX 3090 GPUs with 24GB of memory each for development and conducted hyperparameter searches and experiments using 1 Linux server with 4 Nvidia RTX 3090 GPUs as well as a Slurm cluster equipped with Nvidia RTX 4090 GPUs.
With a single GPU of these two GPU models, a single run of the fuzzy logic task takes around $2-3$ minutes, a single run on sraven between $20-200$ minutes.
For the language modeling experiments, we used 16 Cloud TPU v5e with a complete run taking 72-100 hours.
In total, it takes around 24 GPU hours to reproduce all fuzzy logic results on our hardware, around 10 GPU days to reproduce all our sraven results and approximately 163 TPU days to reproduce our results on C4.

### D.2 Software and libraries

For the results obtained in this paper we built on free and open-source software.
We implemented our experiments in Python using JAX [, Apache License 2.0], Flax [, Apache License 2.0], NanoDo [, Apache License 2.0] and the Deepmind Jax Ecosystem [, Apache License 2.0].
We utilized WandB [, MIT license] to monitor the progress and results of experiments, and Plotly [, MIT license] for generating the plots.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - In HYLA, you not only add the nonlinearity but also change the normalisation. Did you perform ablations w.r.t. the normalization, meaning how well does HYLA perform with the standard normalization (or vice versa, if you add RMSHead in linear attention)?

- Please report the training performance of the models to shed light on the generalisation vs capacity question.

- Please provide more details on the OOD splitting of the datasets.
soundness: 3
strengths: The paper is well-written and studies an important problem, namely compositional generalisation, from a new angle and with promising empirical results. The writing, structure and related work is outlined clearly, and (almost) all relevant details with regards to the theory and the empirical experiments are available in either the main text or the appendix. The perspective to see multi-head attention as a hypernetwork is novel and might open interesting future avenues for study.
summary: This work reformulates linear multi-head attention as a hypernetwork to study compositional generalisation and introduces a simple non-linear variant that substantially increases generalisation both on abstract symbolic tasks as well as language modeling.
weaknesses: One problem in the empirical evaluation is a certain lack of control over whether the suggested modification (HYLA) is actually increasing compositional generalization or whether it's just a way of increasing the capacity of the model. To this end, it would be interesting to look at the training loss of the models: does HYLA reach the same minimum as linear / softmax attention, or is it already performing much better on the training set (suggesting it is more the capacity increase of the model rather than a genuine way of increasing compositional generalisation abilities).

Similarly, it didn't get clear to me how the OOD sets were sampled? E.g., in the fuzzy logic dataset, when you say "hold out a fraction of the combinations for OOD" (line 206), does this refer to combinations of K terms, or are all K terms sampled and you just subsample each (meaning, in the OOD set all combinations of K are present)?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: HYLA with a single head:
- In the caption for Figure 5C, the authors state that the hypernet mechanism is absent for a single head. Why is this the case?

Scaling compositional difficulty:
- In Section 4.2, the authors state that 25% of possible rule combinations are held out for evaluation. How does performance change when this fraction is changed? Specifically, as the compositional difficulty (% of held-out rules) varies from 0% to 100% on the X-axis, how do the performances of softmax attention, linear attention, and HYLA on the Y-axis vary? This important experiment will help us understand how these methods compare in a fully in-distribution setting to much harder OOD settings.

Drop in performance of HYLA with softmax:
- For HYLA, in Eq (6), both the value network weights $\\mathbf{W}'\_{q,k}$ and $\\mathbf{W}\_{q,k}$ depend on $a\_{h,q,k}$, which makes sense from the hypernet perspective where $a$ is the latent code. In standard MHA, as shown in Eq (2), $\\mathbf{W}^\\text{out}$ operates after attention scaling has already been applied. However, from this perspective, Eq (6) indicates a re-application of attention scaling. Could this reapplication of attention scaling be responsible for the drop in performance when using softmax in Table A1, perhaps due to the reapplication of softmax being more prone to vanishing gradients?
- Do the language modeling results improve if a potential issue with using softmax with HYLA is identified and fixed?
soundness: 4
strengths: - The paper presents a novel perspective of multi-head attention as a hypernet. More specifically, the perspective that queries and keys dynamically create a function to apply to the values is a straightforward interpretation of the attention mechanism, but the proposed perspective further reveals where the learned compositional structure is potentially exhibited.
- The core technical contributions of the "multi-head self-attention as a hypernetwork" perspective and HYLA are presented very clearly.
- The proposed method HYLA is a novel drop-in replacement for standard multi-head self-attention.
- The authors clearly show emergent compositionality in the "latent codes" for the intrinsic hypernets of multi-head self-attention and HYLA.
summary: Transforms have been shown to compositionally generalize in some situations and not in others. This paper presents a novel perspective of multi-head self-attention, a core component of transformers, which better exposes the module's capacity for representing and reusing compositional structure. 

Under this perspective:
- The multi-head self-attention mechanism comprises of $\\text{num\\_queries}\\times\\text{num\\_keys}$ value networks that are generated by a hypernetwork (hypernet) parameterized by the value and output projection weights
- The inputs of the hypernet are $\\text{num\\_heads}$-dimensional "latent codes," each corresponding to a query-key pair. Relating to standard multi-head self-attention, these latent codes are obtained by indexing the attention matrix stacked for all heads at a specific key-value index.

With this perspective:
- The authors show that the latent codes exhibit structure that reflects the learned compositionality for the task, which can explain why transformers compositionally generalize when they do. For instance, the latent codes are empirically predictive of the subtasks the network performs on unseen compositions.
- The paper introduces the hypernetwork linear attention (HYLA), a drop-in replacement of standard multi-head self-attention, that modifies the hypernet-generated value network from linear to non-linear to make it more expressive. This modification also replaces the expensive global softmax normalizations per head with local key-query-specific normalizations across heads.

The authors introduce SRAVEN, a symbolic version of Raven Progressive Matrices human intelligence test, which gives them more precise control over the compositions during training and evaluation. Evaluation on SRAVEN shows clearer latent code structure and improved OOD accuracy of HYLA over softmax and linear attention.

Finally, the authors evaluate HYLA on a language modeling task, but it fails to outperform softmax attention here.
weaknesses: The paper seems to consider the special case of self-attention:
- The proposed perspective of attention as hypernet is presented with the assumption of self-attention, which is a special case of attention (e.g., cross-attention is more general). While the assumption of self-attention can be seen in figures and equations, it is only explicitly mentioned in the text at line 114. Without clarifying that this perspective and the proposed modification (HYLA) consider the special case of self-attention, the paper can be seen as over-claiming its contribution. The paper's title, abstract, and introduction should clarify this constraint. Alternatively, the paper should clarify how its formulations work in more general cases of attention and if their experiments are limited to the particular case of self-attention.

Clarity and connection of multi-head to compositionality:
- The necessity and significance of the "multi-head" aspect for compositional generalization is underplayed. A relevant discussion is found in the first paragraph of Section 6, which could be discussed earlier in the paper to set up a clear connection to compositionality.
- The introduction of the paper is quite dense. After a clear flow of the first two paragraphs, it abruptly gets into too many technical specifics and jargon in the third paragraph that is difficult to parse without the context of the rest of the paper. Perhaps a less technical introduction of the paper's perspective and modification, with a motivation of how these relate to compositional generalization, can set up clearer expectations for what is to come and why it matters (maybe the Summary of this review can be helpful if accurate). Only after this, the details like how this paper's method differs from Schlag et al., 2021, or how the presented idea is evaluated become easier to place into context.

Correlation between compositionality and generalization:
- The paper claims that the compositional structure seen in the intrinsic hypernet of MHA can explain the compositional generalization of transformers. However, there is a missing connection here. An important experiment is to evaluate with confidence estimates how the compositionality of attention correlates with the OOD generalization of the model.

Weak performance on language modeling:
- Despite the positive results on synthetic tasks, the paper's proposed method, HYLA, fails to outperform softmax attention for language modeling with a clear performance gap (Figure 6). The authors hypothesize that softmax is essential here, but find that HYLA with softmax results in a drop in performance (Table A1). Please look at the last question in the Questions section for a possible actionable avenue.

RMSNorm:
- It looks like RMSNorm at lines 179-180 is the operation from [1], but the paper is not cited.
- The authors state that it ensures the parameters of the value network are appropriately initialized to prevent exploding/vanishing gradients. There needs to be an explanation of why this should be the case.

References:  
[1] Zhang, Biao and Sennrich, Rico. Root Mean Square Layer Normalization. NeurIPS 2019.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weaknesses section.
soundness: 3
strengths: - The idea of viewing multi-head attention as a hypernetwork is interesting.
- The experiments on a controlled domain such as symbolic RAVEN test are clear and thorough.
- The research problem is of great interest to the ML community.
- The paper is well-written and well-organized.
summary: The paper aims at understanding the ability for compositional generalization for the transformer networks. The idea is to view the multihead attention module as a hypernetwork, where a composable, low-dimensional latent code specifies key-query specific operations. Empirically, the paper finds that using non-linear value networks for the hypernetwork (i.e., the multihead attention module) leads to improved generalization ability. The paper further introduces a symbolic RAVEN test for fine-grained evaluations of compositional generalization ability.
weaknesses: - I may be missing something. But the modification of the linear attention layer seems a little bit restricted. Will other non-linearity functions or $\sigma(\cdot)$ functions other than `RMSHead` also lead to improved compositional generalization ability? It would be great if the authors can comment on this and explain why this modification would work.
- I am admittedly not an expert of this domain, and would like to hear my colleague reviewers' thoughts on this.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Could you provide a more formal explanation or theoretical basis for why framing multi-head attention as a hypernetwork enhances compositional generalization? Specifically, what is the role of the latent code in enabling the network to handle unseen task compositions?

2. Given the computational demands of Transformers in large-scale applications, would HYLA introduce significant memory or efficiency constraints, and if so, how might these be addressed?

3. Have you considered conducting deeper analyses, such as probing or interpretability methods, better to understand the latent codes’ alignment with specific subtasks?
soundness: 3
strengths: 1. The paper provides an innovative perspective by reformulating multi-head attention as a hypernetwork, a novel conceptual framework that opens up new avenues for understanding attention mechanisms in neural networks.

2. The paper is well-structured, with clear explanations of its objectives, methods, and findings.
summary: The paper investigates how Transformers generalize to new, unseen task compositions by reformulating multi-head attention as a hypernetwork—a model that reconfigures another model's computation based on data. This study reveals that a compact, low-dimensional latent code within the attention mechanism specifies key-query operations, which can generalize compositional tasks even if those specific combinations weren’t encountered during training.
weaknesses: 1. The paper makes a strong case for viewing multi-head attention as a hypernetwork, but it lacks a formal theoretical analysis showing why this approach specifically enhances compositional generalization over traditional attention mechanisms. Although the empirical results support the hypernetwork framing, a deeper theoretical exploration would add rigor and clarity.

2. The paper primarily contrasts the proposed approach with standard softmax and linear attention but lacks comparisons with other recent methods that enhance compositional generalization in Transformers.

3. While SRAVEN offers a modular benchmark for evaluating compositional generalization, it may not fully reflect the demands of compositional reasoning in language or perception tasks, which involve a broader range of dependencies and symbolic variations. 

4. Although the proposed Hypernetwork Linear Attention (HYLA) and hypernetwork framing show promising results, there is limited discussion on scalability and practical implementation challenges, especially as the model size increases. For instance, it remains unclear whether the approach would scale effectively in large language models (LLMs) or in high-dimensional data contexts.

5. The authors posit that the structured latent code in multi-head attention serves as a reusable resource for different compositional tasks. While the paper provides clustering analyses, there is limited interpretability regarding how the latent codes correlate with specific task elements or subtasks.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - How does this interpretation of MHA as a hypernetwork relate to the "residual stream" view, and related concepts on attentional circuits in previous mechanistic interpretability work? Does it have any implications on simple circuits like, say, induction heads?
- How do MoE models compare in these compositional generalization experiments?
- Typically, the value network in MHA is thought of as merely projecting out a certain component of the context token to be retrieved and incorporated into the query token. The value network is not intended to perform any complex processing, that is the job of the MLP that immediately follows. However, the interpretation of MHA as a hypernetwork seems to hinge on the hypothesis that the value network ought to be doing some more complex processing. Can you comment on this distinction? Why would we want the value network to perform complex processing rather than having that be performed by the MLP?
- Under the hypernetwork view, the "value network" has a pretty specific parameterization as a query/key-dependent linear combination of linear maps. Can you comment on why this type of parameterization would make sense? For example, is there some simple construction of a network where there is an interpretable view of the class of functions this could capture?
- In lines 370-374, the authors discuss the difference between standard RPM problems with visual inputs and their simplified version with symbolic input, arguing that the main source of difficulty in RPM tasks is the problem of finding correspondences of rules to feature dimensions. I found this unconvincing, however, and I hope the authors can expand. In particular, it seems to me that this argument relies on the assumption that the visual representation learning step of discovering the right symbolic abstractions is easy, citing work in cognitive science. But what might be easy for humans (who have in-built abilities through millions of years of evolution) may not be easy for machines. Learning the full visual RPM problem end-to-end is surely significantly more difficult for a machine learning system, no? Overall, I think it's unnecessary to try to make this argument. Yes, the symbolic RPM problem is easier, but it is still interesting; we don't need to argue that it captures all the interesting aspects of the full visual RPM problem.
soundness: 3
strengths: - The question of how compositional generalization arises in neural networks, and Transformers specifically, is of broad interest to the ML research community. I believe that the interpretation proposed in this paper is a meaningful contribution to this question.
- The paper is well-written, well-structured, and clear.
- To my knowledge, the interpretation of combinations of heads as a hypernetwork is novel.
- The proposal of the HYLA attention mechanism is well-motivated and makes sense given the presented hypernetwork interpretation of MHA.
- The experimental results provide some empirical support for the general claims made in the paper, with HYLA performing favorably in terms of compositional generalization on the tasks considered.
- I found the figures showing the correlation between the latent codes and the target label (i.e., 2B and 5A) compelling, making the argument that some notion of compositionality is made possible by having multiple heads and
summary: This paper presents an interpretation of multi-head attention as a "hypernetwork", in the sense of Ha et al. (2017)---a network that generates weights that parameterize another network. The main insight comes from writing MHA as: $\sum_{h=1}^{H} W_h^{out}\sum_{k=1}^{K} a_{h,q,k} W_h^{val} x_k = \sum_{k=1}^{T} \left(\sum_{h=1}^{H} a_{h,q,k} W_h^{out} W_h^{val}\right) x_k$, and interpreting the inner summation in the RHS as the parameterization of the "value network". This comes from swapping the summation over heads $h$ with the summation over keys in the typical formulation of attention. The authors argue that this can give rise to some notion of "compositionality" and OOD generalization. While the value network here is parameterized as a linear combination of linear maps, the authors use this interpretation to propose a variant of linear attention where the value network is instead a two-layer MLP, with each linear map in the MLP being parameterized by the hypernetwork. The paper then explores two experimental settings: the first is based on ICL on "fuzzy logic functions" and the second is based on a symbolic version of Raven's progressive matrices. The authors show that networks with linear attention and softmax attention achieve some level of OOD generalization on these tasks, and that their proposed HYLA model achieves further improved performance.
weaknesses: - The interpretation of MHA as a hypernetwork through re-ordering the summation over heads and keys makes sense in the case of linear attention, but seems less so in the case of softmax attention. In the case of softmax attention, the value network is a linear combination over heads, but with the normalization being over keys rather than heads. This doesn't fit into the claim that the value network is merely query-key dependent; rather it is a function of the entire sequence. Can the authors comment on this? You focus more on linear attention in the paper, and perhaps this is the reason. But, can you share more of your thinking about this?
- The idea of a value network being input-dependent seems closely related to Mixtures-of-Experts. However this is not directly addressed in the paper. Can you comment on this? In particular, is there a type of function that HYLA can represent that standard MHA followed by a MoE-MLP could not? Experimentally, how would the latter perform compared to HYLA? Would it be better or worse at compositional generalization?
- Although the experiments are clear, the scenarios where compositionality and the advantage of HYLA are demonstrated are somewhat restricted in scope. On complex real-world tasks such as language modeling, HYLA performs worse compared to Softmax attention (this is again related to HYLA being a variant of linear attention).

See also the questions below.

Overall, I think this is a nice research paper, with meaningful contributions, and I look forward to discussion with the authors.

Meta-review: This paper demonstrates a mathematical equivalence between multihead attention and linear hypernetworks, which is then used to provide an explanation for why Transformers are able to compositionally generalize to some extent. The attention scores per head can be interpreted as a low-dimensional latent code that predicts the resulting linear value network that processes the input. Then, this insight is used to construct a more expressive approach to compositional processing by modifying MHA to make the value network nonlinear, and also introduce a normalization scheme that leads to linear value networks that are more consistent with desirable network initialization properties. Through experiments on two toy reasoning datasets (a fuzzy boolean logic task and a symbolic Raven matrices task), the paper shows evidence of compositional generalization that is enhanced for the nonlinear version, lending support for the hypernetwork view of attention. Qualitative visualizations show that the heads are consistent with hypothesized compositional functions. Finally, experiments on language modeling show that the nonlinear hypernetwork approach improver over standard linear attention baselines, adding further support for the hypernetwork view of attention.

Strengths:
Well-written paper posing an interesting hypothesis with provocative results on compositional generalization. May open new design space choices for improving attention schemes in general and towards better generalization in reasoning tasks. The nonlinear modification and normalization scheme of HYLA may be valuable in its own right as a drop-in replacement for attention. The paper may also spur new ways of approaching interpretability in attention models. Experiments are nice in that they allow controlling degree of compositionality.

Weaknesses:
Not really a core weakness, but it would be nice to see controlled experiments with cross-attention, since it was claimed that the same treatment should hold. There are also some unexplained aspects of standard softmax self-attention that are not captured by the current treatment of linear MHA as a hypernetwork (and the HYLA variant), and one reviewer is not fully convinced that the hypernetwork equivalence is a useful construct to explain compositional generalization in softmax attention. 

Decision reasoning:
Reviewers unanimously felt the paper is a strong contribution, with 4/5 reviewers giving scores of 8. I agree with the reviewers’ assessments and find the paper quite interesting, particularly towards better understand why Transformers are able to reason compositionally in at least some cases. The authors also put a lot of effort into addressing reviewer concerns, which made the final paper significantly stronger. Papers like this are useful for understanding methodology, but may also lead to new developments — while the authors caution that the HYLA variant of MHA is more of an ‘experimental device’ rather than a legitimate enhancement, it may inspire new advances that enhance the expressiveness of attention.

Decision: Accept (Oral)