Disentangling and Integrating Relational and Sensory Information in Transformer Architectures
Authors: Awni Altabaa
John Lafferty
Venue: ICML 2025 poster (2025)
Keywords: relational
reasoning
attention
transformers
inductive biases
sensory
relational
architecture
attention
Primary area: deep_learning->attention_mechanisms
TL;DR: We introduce an extension of the Transformer architecture with explicit relational computational mechanisms, integrating sensory and relational processing.

Abstract: Relational reasoning is a central component of generally intelligent systems, enabling robust and data-efficient inductive generalization. Recent empirical evidence shows that many existing neural architectures, including Transformers, struggle with tasks requiring relational reasoning. In this work, we distinguish between two types of information: *sensory* information about the properties of individual objects, and *relational* information about the relationships between objects. While neural attention provides a powerful mechanism for controlling the flow of sensory information between objects, the Transformer lacks an explicit computational mechanism for routing and processing relational information. To address this limitation, we propose an architectural extension of the Transformer framework that we call the *Dual Attention Transformer (DAT)*, featuring two distinct attention mechanisms: sensory attention for directing the flow of sensory information, and a novel relational attention mechanism for directing the flow of relational information. We empirically evaluate *DAT* on a diverse set of tasks ranging from synthetic relational benchmarks to complex real-world tasks such as language modeling and visual processing. Our results demonstrate that integrating explicit relational computational mechanisms into the Transformer architecture leads to significant performance gains in terms of data efficiency and parameter efficiency.

Disentangling and Integrating Relational and Sensory Information in Transformer Architectures
Awni Altabaa
John Lafferty?
Abstract
1. Introduction
A central goal of machine learning research is to develop universal architecture capable of learning and reasoning across wide range of tasks and data modalities. Theo- retical frameworks for understanding human and animal intelligence seek to explain intelligent behavior through a small set of fundamental principles [1]. However, in ma- chine intelligence, there is tension between the objective of developing general architecture and the need incorpo- rate inductive biases that are beneficial for specific tasks [2, 3]. When faced with finite training data and numerous solutions to empirical risk minimization, inductive biases steer the learning algorithm towards solutions with desirable properties, enhancing data efficiency and generalization. core scientific challenge machine learning is to identify a complete and broadly applicable set of inductive biases that promote robust, flexible, and data-efficient learning across a diverse set of problems.
Relational reasoning is central component of generally intelligent systems, enabling robust and data-efficient inductive generalization. Recent empirical evidence shows that many existing neu- ral architectures, including Transformers, struggle with tasks requiring relational reasoning. In this work, we distinguish between two types of infor- mation: sensory information about the properties of individual objects, and relational information about the relationships between objects. While neural attention provides a powerful mechanism for controlling the flow of sensory information be- tween objects, the Transformer lacks an explicit computational mechanism for routing and pro- cessing relational information. To address this limitation, we propose an architectural extension of the Transformer framework that we call the Dual Attention Transformer (DAT), featuring two distinct attention mechanisms: sensory attention for directing the flow of sensory information, and novel relational attention mechanism for direct- ing the flow of relational information. We em- pirically evaluate DAT' diverse set of tasks ranging from synthetic relational benchmarks to complex real-world tasks such as language mod- eling and visual processing. Our results demon- strate that integrating explicit relational compu- tational mechanisms into the Transformer archi- tecture leads significant performance gains in terms of data efficiency and parameter efficiency.
Relational reasoning is central component of generally intelligent systems and is believed to underlic human abili- ties for abstraction and systematic generalization [4-7]. The power of relational reasoning lies in its capacity to generate inferences and generalizations in systematic and novel ways, even across instances which are superficially very different but have similarities grants humans a capacity for out-of-distributie generaliza- tion, data efficiency. and continual learning, which modern machine learning is yet to match [8-10]. Replicating this ability in machine intelligence can ultimately lead to univer- sal inductive generalization from a finite set of observations to an infinite set of novel instances [11].
abstract structural level. This
The ability of artificial intelligence systems to perform rela- tional reasoning has long been area of active study across various approaches to AI. In symbolic modeling frame- works, the relationships between symbols are explicitly de- fined in the language of logic and mathematics [12-14]. However, such approaches often require fand-crafted repre- sentations and suffer from the symbol grounding problem, limiting their ability generalize to variations in the task input outside certain domain. By contrast, deep learning approaches build data-dependent representations that are, in principle, capable of generalizing across diverse conditions.
'eepartment of Statistics and Data Science, Yale University 'Department of Statistics and Data Science, Wu Tsai Institute, Yale University. Correspondence to: Awni Altabaa wwwniaaltabaa@yale...o laffett edu..
John Lafferty
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Relational & Sensory Processing in Transformers
However, recent work exploring the ability of deep learning models to learn relational tasks finds that seemingly simple relational inferences can be remarkably difficult for pow- erful neural network architectures [15-23]. An emerging hypothesis, which we explore further here, explains this through the lens of inductive biases, arguing that neural networks struggle with relational reasoning because they overemphasize individual object representations- sensory information-wwite lacking explicit mechanisms for encod- ing and processing relational information [24].
We empirically evaluate this architecture on a diverse set of tasks, ranging from synthetic benchmarks on relational rea- soning to complex real-world tasks such language mod- eling and visual processing. Our results demonstrate that integrating explicit relational mechanisms into the Trans- former architecture leads to significant performance gains in terms of data efficiency and parameter efficiency.
2. Disentangling Attention over Sensory and Relational Information
In this work, we explore this idea in the context of the Transformer architecture [25], which offers a promising starting point for building a versatile, general-purpose neu- ral architecture. Although Transformers, like other neu- ral architectures, struggle learn abstract relational rep- resentations [19-23], there is encouraging evidence that Transformer-based foundation models acquire some rela- tional reasoning ability [26-32] through large-scale training on large amounts of data [33-36]. This presents an op- portunity to explore Transformer- baasseeeee-bbeeed architectures with built-in neural mechanisms and inductive biases for explicit and enhanced relational reasoning capabilities, seeking to imbue the Transformer architecture with a greater capacity for efficient induction of abstractions.
2.1. Standard Attention: Attention over Sensory Information
The attention mechanism of standard Transformers can be understood as differentiable computational mechanism for dynamically routing sensory information between different elements in the input. An object emits a query that is com- pared against the keys of each object in its context via an inner product. A "match" occurs when the inner product is large, causing an encoding of the features of the attended object to be retrieved and added to the residual stream of the receiver. Formally, attention between an object a Rd and a context y (y1..... Vn) E R takes the form
Attention(2, (.,))..;mn))
To introduce our proposal, we highlight distinction be- tween two types of information which are encoded in the internal representation of Transformer models: sensory in- formation, which represents features of individual objects, and relational information, representing comparisons and relationships between objects. In the standard attention mechanism of Transformers, relational information is entan- gled with sensory information, limiting the model's 's ability to learn to explicitly represent and reason about the relation- ships between objects. In particular, in standard attention, relational information directs the flow of information (i.e., attention scores encoding a selection criterion), while the values routed are representations of the sensory attributes of individual objects. In this work, we explore a new type of attention mechanism we call relational attention, in which the values being routed are themselves representations of relationships between the source and the target, computed explicitly as a series of comparisons under learned feature maps. This equips the model with a mechanism for explic- itly routing and processing relational information, decou- pling it from sensory features.
(z ())o where, 1=1
(1)
al2, y) Softmax [< (gattm (z),
of 1ith (w)))
where 0 @attm Ok dgtt are learnable query and key maps control- ling the selection criterion, and Φ is a learnable value map controlling what information about yi is sent. The attention scores axx, y) are used to retrieve a convex combination of the values, where (z y) denotes the i-th component.
Here, the retrieved information is sensory, comprising the features and attributes of individual objects in the context. For this reason, we refer to standard neural attention as "sensory attention"..
2.2. Relational Attention: Attention over Relational Information
Standard neural attention does not explicitly capture infor- mation about the relationship between the source/sender and the target/receiver, making relational learning in standard Transformers inefficient [15, 16, 18-24, 37]. We propose re- lational attention, an attention mechanism for dynamically routing relational information between objects.
We integrate relational attention with the standard attention mechanism of Transformers, yielding a variant of multi- head attention that processes both sensory and relational information parallel. The resulting Dual Attention Trans- former (DAT) architecture disentangles the two types of information during the information retrieval stage and inte- grates them during the local processing stage of each layer.
Mirroring standard attention, this operation begins with each object emitting a query and a key, which are compared via an inner product to compute attention scores determining which objects to attend to. Next, instead of retrieving the
2

---

Relational & Sensory Processing in Transformers
1
V1
r r(x, V1
+ $1
2
2
1/2
a22
r r(x, Y/2
+ 2
Y/2
X
X
n
an
:
Vn
Vn
r r(x, yn
+
Sn
V/2
(a) Attention(2, (Br....,...))
(b) RelationalAttennionnss (vi...., vr.)
Figure 1. Standard self-attention retrieves sensory information n about the attributes individual objects while relational attention retrieves relational information r(z, ys) about the relationship between the objects in the context and the target. Each relation is tagged with a symbol &, which acts as an abstract variable identifying the source. In both cases, information is aggregated according to the attention scores og, which are computed by softmax over inner products of queries and keys.
sensory features of the selected object, relational attention retrieves the relation between the two objects- series of comparisons between the objects under dif- ferent feature subspaces. In addition, symbolic identifier is sent to indicate the identity of the sender to the receiver. Formally, this operation is defined as follows.
structure to the relation function, transforming it into a posi- tive semi-definite kernel that defines a pseudometric on the object space, along with corresponding geometry.
2.3. Symbol Assignment Mechanisms
To process relational information effectively, the receiver must have two pieces of information: 1) its relationship to the objects in its context, and 2) the identity of the object associated with each relation. In relational attention, the former is captured by r(æ: yi) and the latter by 8i. The symbols s; are used to tag each relation with the identity information of the sender.
Relational At Attention (2r, (y) -.ll.-....p vn))
Σ a.( w, s,W.), where,
a(x, y) Softmax ((0gtt o2tt ((:""((":)))) (vi)))
r(z, 91) (f (z), 0F. (B.)) eejd,1
The symbol 8; identifies points to the object yi, but, im- portantly, is designed to not fully encode the features of the object. Instead, the symbols function as abstract refer- ences to objects, perhaps viewed as connectionist analog of pointers traditional symbolic systems. In particular, by drawing symbol vectors from a finite library Sub. relational attention maintains relation-centric representation. This separation between sensory and relational information is key to decoupling relational attention disentangled from sensory features, enabling generalization across relations.
($1, ........., sn) = SymbolRetriovor((; Smb)
Thus, relational attention between the object æ and the con- text y (......, .n.) retrieves convex combination of the relation vectors (r(x, w))) representing x's relationship with each object in the context. Relational attention also retrieves a symbol vector 8; that encodes the identity infor- mation of the attended object. The role and implementation of the symbols will be discussed in the next subsection. As with standard attention, d ob are learned feature maps that govern the attention selection criterion. A separate set of query and key feature maps, drel ofel é E [dz], are learned to represent the relation between the sender and the receiver. For each é E [d the feature maps 9. ote ex- tract specific attributes from the object pair, which are com- pared by an inner product. This produces a diddimensi relation vector representing a fine-grained series of com- parisons ((ote) otel (1x))) across different feature subspaces.
The notion of the "identity" of an object can vary depending on context. this work, we consider modeling three types of identifiers: 1) position, 2) relative position, or 3) an equiv- alence class over features. For each type of identifier, we model corresponding symbol assignment mechanism [22]. We find that different symbol assignment mechanisms more effective in different domains.
Positional Symbols. In some applications, it is suffi- cient to identify objects through their position in the in- put sequence. We maintain a library of symbols Sib = ($1..... Smax lea) E maaax_lexxe and assign s; to the i-th ob- ject in the sequence. These essentially learned positional embeddings.
In certain tasks [21-23], a useful inductive bias on the relations function r(..) ) is symmetry: i.e., r(x, y) = r(y, 2), Vx, y. This corresponds to using the same feature filter for the query and key maps, die = ote. This adds
3

---

Relational & Sensory Processing in Transformers
Position-Relane Symbols. Often, the relative position with respect the receiver is more useful identifier than absolute position. This can be implemented with position- relative embeddings. We learn a symbol library Sib = (+- A.....,& 8-1, 80; 81,-..,888 ,8A) E IR(22+1)xd, where is the maximum relative position, and relational attention be- comes E, 0ky (x) W, + 85=5 W.).
Algorithm 1: Dual Attention
Input: = (Z1....,Tn)) Rxxd
Compute self-attention heads
a t Soft ((x qihh (ae k. )T h [m
(b) Σ
(h) Biy By W iE [n]. h E [m
Symbolic Attention. In certain domains, some information about the objects' features is necessary to identify them for the purposes of relational processing. Yet, to maintain rela- tional inductive bias, we would like to avoid a full encoding of object-level features. In symbolic attention, we learn a set of symbol vectors, Sm = ($1,...,$r., E R" xd and a matching set of feature templates Fib = (fi,..., Sn.)-
e + concal (e, (1)
, e (1 Wan [
Assign symbols:
(s1....., 8n) t SymbolRetriover(a; Sib)
Compute relational attention heads
We retrieve a symbol for each object by an attention oper- ation that matches the input vectors 2; against the feature templates Sn and retrieves symbols 8..
a (b) + Softmax ((#: Wath") )ath h (*)
rig + ((0, q.e. Wi.)) eejjall i [n]
(b) t Σ (rij W + +s W!).
ie [m], h h [n
0 + concat (a (1) ....,, of W [n]
SymbolicAtttnntion((a)) = Softmax ((x Wg) Fiih)Sib Sib.
(3)
Output (opput:(connn) (concat(ee, a:))
Here, Süb: Fli: Wg are learned parameters. This can be thought of as implementing learned differentiable "equiv- alence class map" over feature embeddings. Crucially, the
3. Integrating Attention over Sensory and Relational Information
number of symbols (i.e., feature equivalence classes) is fi- nite, which enables relational attention to still produce a relation-centrie representation while tagging the relations with the necessary identifier.
3.1. Dual Attention
One of the keys to the success of the Transformer archi- tecture is the use of so-called multi-head attention. This involves computing multiple attention operations in parallel at each layer and concatenating the output, enabling the model to learn multiple useful criteria for routing informa- tion between objects. However, in standard Transformers, these attention heads focus solely on routing sensory in- formation, lacking explicit support for routing relational information between objects.
2.4. What Class of Functions can Relational Attention Compute?
To give some intuition about the type of computation that relational attention can perform, we present the following expressivity result, The following theorem states that rela- tional attention can approximate any function on X' x ) that 1) selects an element in (V1...., Vn), then 2) computes a relation with it. Both the selection criterion and the rela- tion function are arbitrary, and the selection criterion can be query-dependent. The formal statement and proof are given in Appendix A.
We posit that both sensory and relational information crucial for robust and flexible learning over sequences or col- lections of objects. To this end, we propose extension of multi-head attention comprising two distinct types of atten- tion heads: sensory attention (i.e., standard self-attention), and relational attention. This yields a powerful mechanism for dynamically routing both sensory and relational infor- mation in parallel. Our hypothesis is that by having access to both computational mechanism, the model can lear to select between them based on the current task or context, as well as compose them to create highly-expressive and flexible computational circuits.
Theorem 1 (Informal). Let Select X x y - y be an arbitrary preference selection function, which selects an element among (y1....,yy, based on a query- dependeen preorder relation {<=)aex- Let Rel: X x y -> Rdd be an arbitrary continuous relation function on X V. There exists relational atlention module that approximates the function Rel(z, Select(x, y)) to arbitrary precision.
Algorithm 1 describes the proposed module, referred to as dual attention. The number of sensory attention heads n
4

---

Relational & Sensory Processing in Transformers
Algorithm 2: Dual Attention Encoder Block
Algorithm 3: Dual Attention Decoder Block
Input:æ : E _
Input: :z y E 1a
x f [orm( + DualAttn(2)) a: - (orm(x + ML)(()))
x Norm( DualAttm(z)) a Norm(( CrossAtin (1)
x Norm( ML)(x))
Output:
Output:
and number of relational attention heads naa" are hyperpa- rameters. The sensory attention heads attend to sensory information while the relational attention heads attend to re- lational information. The combined ny = =n,"" + m2 heads are then concatenated to produce the output. The result is a representation of contextual information with integrated sensory and relational components. Appendix B provides further discussion on the details of the architecture and its implementation.
gorithms and in Appendix B define encoder and decoder blocks with dual attention. Composing these blocks yields the Dual Attention Transformer architecture.
The Dual Attention Transformer framework supports all architectural variants of the standard Transformer, making it applicable to a wide range of task paradigms. An encoder- decoder architecture with causal dual-head attention the decoder can be applied to pequence-to-sequenee tasks, as in the original Transformer paper [25]. An encoder-only archi- tecture can be used for a BERT-style language embedding model [40] or ViT-style vision model [41]. A decoder-onle architecture with causal dual-head attention can be used for autoregressive language modeling.
Attention Masks & Causality, Any type of attention mask (e.g., causal mask for autoregressive language modeling) can be implemented in relational attention in the same way as for standard self-ateention (i.e., mask is added to a pre-softmax).
Positional Encoding. There exists different methods in the literature for encoding positional information in the Trans- former architecture. For example, [25] propose adding posi- tional embeddings to the input, [38] propose adding relative- positional embeddings at each attention operation, and [39] propose rotary positional embeddings (RoPE) which apply a cesttoon-eeendemm map to the queries and keys pre-softmax. These methods are compatible with dual attention and are configurable options in our public implementation.
4. Empirical evaluation
We empirically evaluate the Dual Attention Transformer (abbreviated, DA7) architecture on range of tasks span- ning different domains and modalities. Our goal is to assess the impact of integrating relational inductive biases into the Transformer architecture. We begin with a synthetic relational learning benchmark to evaluate DAT's relational computational mechanisms in a more controlled setting. We then proceed to evaluate the proposed architecture on more complex real-world tasks, including mathematical wroblem-solvinge image recognition, and language model- ing. These experiments cover multiple task paradigms and architectural variants, including: discriminative (encoder- only architecture), sequence- gequeece-to-sequenae oe-sequence (encoder-decoder), autoregressive language modeling (decoder-only) and vi- sion (Vi7'-style architecture) tasks. For each experiment, compare DAT model that incorporates both sensory and relational heads against standard Transformer where all heads are ordinary sensory attention heads. The difference in performance highlights the impact of integrating both types of attention heads, enabling richer representation of sensory and relational information. We summarize the experimental results below and defer certain experimental details to Appendix C.
Computational complexity. The computational complex- ity of relational attention scales similarly to standard self- attention with a (nn²) dependence on sequence length. Like standard attention, relational attention can be computed in parallel via efficient matrix multiplication operations.
Symmetric relations. A symmetry constraint be in- jected into the relations Tú by imposing that Wrel Wrrel, which is useful inductive bias when the task-relevant rela- tions are inherently symmetric.
3.2. The Dual Attention Transformer Architecture
The standard Transformer architecture is composed of re- peated blocks of attention (information retrieval) followed by an MIP (local processing). Our proposed Dual Atten- tion Transformer follows this same structure, but replaces multi-head self-attention with dual attention (Algorithm l). At each layer, dual attention dynamically retrieves both sen- sory and relational information from the previous level of computation, which is then processed locally by MLP. Al-
4.1. Sample-Efficient Relational Reasoning: Relational Games
We begin our empirical evaluation with the "Relational Games benchmark for visual relational reasoning proposed
5

---

Relational & Sensory Processing in Transformers
8 Bane
occure
xoccura
Generalization Accuracy
01.9
0.9
0.8
0.8
0.8
0.7
0.7
0.6
0.6
0.6
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Training Set Size
Training Set Size
Training Set Size
between
match pattern
Model
Generoizationn Accuracy
DAT (rg" 02" 2) [421K]
0.9
0.9
DAT (ef' 1,n0" 1) [404K]
0.8
T (ug 8,0%" 0) [481K]
T (ng" 2. 0) [481K]
0.7
Y ("g" 8,0f" [386K]
(4g" 2, [366K]
1000
1500
2000
2500
10000
15000
20000
25000
Training Set Size
Training Set Size
Figure Learning curves on the relational games benchmark. Each subplot corresponds to a different task. Numbers in square brackets legend labels indicate parameter counts. Solid lines indicate the mean over 5 trials with different random seeds and the shaded regions indicate bootstrap 95% confidence intervals. DAT is more data-efficient at relational learning compared to Transformer.
by Shanahan et al. [20]. The dataset consists of family of binary classification tasks, each testing a model's 's ability to identify particular visual relationship among series of objects (see Figure 5 for examples). The input is an RGB image depicting a grid of objects, and the target is a binary classification indicating whether the particular relationship holds for this input. This forms a controlled synthetic setting for evaluating DAT's effectiveness in relational learning.
however, the difference is only marginal, suggesting that the model is able to learn to select the computational mecha- nisms that are most relevant to the given task. We provide further discussion in Appendix C.I, including results com- paring against previously -ropoosed relational architectures with stricter inductive biases.
4.2. Relational Inductive Biases for Symbolic Reasoning in Sequeencetto-Sqqeence tasks: Mathematical Problem Solving
Our goal in this section is to explore how the relational computational mechanisms of DAT affect data-efficiency in relational learning-thal is, how much data is necessary to learn a given task. We evaluate learning curves by varying the size of the training set, training each model until con- vergence, and evaluating on hold-out validation set. We test configurations of DAT: with only relational attention heads, and one with combination both sensory and relational heads. We include several Transformer base- lines, varying the number of attention heads and the model dimension, controlling for parameter count. The results are depicted in Figure 2.
Next, we evaluate DAT on a of mathematical problem- solving tasks based on the benchmark contributed by Saxton et al. [42]. Mathematical problem-solving is interesting test for neural models because it requires more than statisti- cal pattern recognition-in requires inferring laws, axioms, and symbol manipulation rules. The benchmark consists of a suite of mathematical groblem-solving tasks, with each task's dataset consisting of a set of question-ant -answer pairs. The tasks range across several topics including solving equa- tions, adding polynomials, expanding polynomials, differen- tiating functions, predicting the next term in a sequence, etc. An example of a question in the "polynom: als expand" task is "Expand (5x - (2x + 1)" with the target answer "10z" x - 3". This is modeled as a tooeneeet--seuuece task with character-level encoding.
We find that DAT is significantly more sample-efficient, particularly more difficult tasks. Both configurations of DAT are consistently more sample-efficient compared to the standard Transformer. The effect is particularly dramatic on the 'match pattern' task which is the most difficult and requires identifying second-order relation (i.e., a relation between relations). We note that these tasks are purely re- lational in the sense that pairwise same/different relations between objects are sufficient statistic for predicting the target. This suggests that relational attention is sufficient for solving the task. Indeed, the DAT variant with only rela- tional heads performs slightly better than the variant with combination of both sensory and relational heads. Notably,
We compare DAT against Transformers using an encoder- decoder architecture. The encoder processes the question, and the decoder autoregressively generates the answer while cross- aooss-at to the encoder. We explore how perfor- mance scales with model size by varying the number of layers. In the Transformer, all attention heads are standard self-attention with n 8, while in DAT we have combi- nation of both types of attention heads with n n 4.
6

---

Relational & Sensory Processing in Transformers
algebra_. 1igebra___inea__1
segebra...equeeee seatcc___et___ttteeee
colculus alcullss_ddifeeratiin
1.000
0.8
Accuracy
0.7
0.95
0.989
0.6
0.90
0.908
1.25M
1.75M
1.5M 1.75M
Model Size (Parameter Count)
Model (Parameter Count)
Model (Parameter Count)
polynomials__.cppp expand
polynomian als___ado add
0.9
0.58
Accuracy
Model
00.8
Transformer
DAT
0.84
750K
1M
1.25M
1.5M 1.75M
750K
1M
1.25M
1.5M 1.75M
Model Size (Parameter Count)
Size (Parameter Count)
Figure 3. Average character-level accuracy on different mathematical problem-solving tasks measured at different model sizes. Error bars indicate bootstrap 95% confidence intervals over 5 trials. DAT outperforms standard Transformer across model sizes, suggesting that relational computational mechanisms confer benefits on teuuuucc-o--ccquenc tasks that involve symbolic computation.
Figure 3 depicts the character-level accuracy for DAT and Transformers across varying model sizes. We find that the DAT model outperforms the standard Transformer at all model scales and across all tested tasks. This suggests that the relational computational mechanisms of DAT are beneficial for the type of symbolic processing involved in solving mathematical problems.
which are similar across some attributes (e.g., texture) but different across others (e.g., color). In particular, the rela- tions in relational attention can be interpreted as taking the source patch as a filter and comparing it against each patch in the image under different transformations.
We evaluate a ViT-style DAT architecture (ViDAT), and compare it against standard VIT on the CIFAR image recognition benchmarks [44]. We train directly on CIFAR- 10 and CIFAR-100, respectively, without pretraining on larger datasets. During training, we use random cropping, MixUp [45], and CutMix [46] as data augmentation tech- niques. We evaluate 8-layer models with dmodel = da 384. The VIT model has n 12 standard self-attention heads, while the DAT model uses both sensory and rela- tional heads, with even split of nr n 6. We use symmetric relations rij based on the intuition that visual processing involves symmetric attribute-smillariy relations. In Appendix C.3, we present ablations, additional results, visualizations of learn relations, and further discussion.
4.3. Visual Processing with Relational Inductive Biases
As a general sequence model, the Transformer architec- ture can be applied to visual inputs by dividing image into patches that are then flattened, linearly embedded into vectors, and passed in as sequence. Through series of attention and MLP operations, the visual input is processed for the downstream visual task. This architecture is referred to as a Vision Transformer (Vi7) [41]. Although Transform- ers lack the explicit spatial inductive biases found in models like convolutional networks, recent work has demonstrated its effectiveness at scale [43], demonstrating the versatility of attention as computational mechanism across several data modalities.
Table reports the classification accuracy of each model. ViDAT' outperforms ViT' on both datasets, suggesting that relational computational mechanisms enhance visual pro- cessing. These experiments show that relational inductive biases can be useful for image recognition. We hypothe- size that relational processing is even more important in visual tasks requiring complex scene parsing, where rea- soning about the relationships between constituent objects is essential. Recent work scene understanding in large vision-language models supports this view [47-49]. We leave the exploration of such tasks for future work.
In this section, we explore how the relational computa- tional mechanisms introduced in nam-namely, relational attention impact visual processing tasks. We hypothesize that visual processing benefits from attending both sen- sory and relational information. That is, when processing local region of visual input (e.g., patch, object, or object part), it is useful consider not only the sensory features of other regions but also the relationships between these re- gions. For example, this captures information about similar objects occurring in multiple places in the scene, or objects

---

Relational & Sensory Processing in Transformers
4.4. Relational Inductive Biases in Language Modeling
Perplexity
Language understanding involves processing and organizing relational information, such as syntactic structures, semantic roles, and contextual dependencies, to extract meaning from words and their connections within sentences. Transformers have been remarkably effective at language modeling, with neural scaling laws demonstrating that increasing model size and dataset size result in predictable improvements in performance across a range of language tasks [34, 35]. While the standard attention mechanism of Transformers is able to capture simple positional and syntactic relations in its attention scores, this is only used to control the flow of information between tokens rather than explicitly encoding relational information in the latent embeddings themselves. The relational attention mechanism of DAT enables explic- itly learning relational contextual information that is directly encoded in each token's latent embedding.
56
18.0
21.0
Takess
24,0
27.0
26
30.0
DAT
16
Trandfrmer
350M
730M
1.3
Param Count
Figure 4. Performance scaling of DAT compared to standard Transfomer on language modeling: DAT is more data-cfficient and more parameter-efficien.
Beyond performance improvements, we also find evidence that relational attention encodes human-interpretable seman- tic relations. Figure 8 depicts a visualization of the relations Tay learned by a DAT language model. We observe that the relations learned by relational attention tend to encode semantic relations, rather than syntactic relations. That is, re- lational activations raj E Rdd are large between tokens with related meanings. We believe that further exploration of this phenomenon from a mechanistic interpretability perspective could offer an exciting avenue for future research. Such an exploration would be complementary to interpretability efforts seeking to understand the attention scores of standard Transformers, which have been found to attend e.g., based position, syntax, and punctuation [51-53].
In this section, evaluate DAT causal language mod- eling, exploring the impact of its relational computational mechanisms in the domain of language. We use a decoder- only architecture, where the model receives a sequence of tokens as input and is trained to causally predict the next token at each position. We train on 10 billion tokens of the wie-Eeeb dataset [50], which is curated dataset of high-quality educational text data from CommonCrawl. We train models at multiple parameter scales, up to 1.3 billion parameters, to study the scaling properties of DAT on lan- guage modeling with respect to both model size and data size. Details of training and architectural hyperparameters are given in Appendix C.4, together with further discussion of the results.
5. Conclusion
Summary. The standard attention mechanism of Transform- ers provides a versatile mechanism for retrieval of sensory information from given context, but does not explicitly support retrieval of relational information. In this work, we presented an extension of the Transformer architecture that disentangles and integrates sensory and relational in- formation through a variant of multi-head attention with two distinct types of attention heads: standard self-attention for sensory information and novel relational attention mechanism for relational information. We empirically eval- uate this architecture and find that it yields performance improvements across range of tasks and modalities.
Figure 4 depicts the scaling properties of DAT's language modeling performance with respect to model size and data size, compared to a standard Transformer. We observe that DAT demonstrates greater data and parameter efficiency, achieving improved performance across model and data scales. This suggests that DA7's relational computational mechanisms confers benefits in language processing.
Dataset
Model
Params
Accuracy
VIT
7.1M
86.4 : 0.1%
Limitations & Future Work. The proposed architecture introduces several hyperparameters and possible configu- rations. Although we carried out ablations on the major configuration choices (e.g., composition of head types, sym- metry, symbol assignment mechanisms), an expanded em- pirical investigation would help develop improved under- standing of the behavior of this architecture under different configurations. We also note that our implementation of the Dual-Attention Transformer currently lacks the hardware- aware optimizations available for standard Transformers (e.g., Fllas-AAtennin [54]), which makes it slower to train
CIFAR-10
VIDAT
6.0M
89.7 + 0.1%
VIT
7.2M
68.8 + 0.2%
CIFAR-100
ViDAT
6.1M
70.5 1.5500114 0.1%
Table 1. Classification accuracy image recognition with the CIFAR-10 and CIFAR-100 datasets. Each training configuration is repeated 10 times with different random seeds; we report the mean accuracy + the standard error of mean. DAT outperforms stan- dard Vision Transformer, suggesting that relational computational mechanisms are useful for visual processing tasks.
8

---

Relational & Sensory Processing in Transformers
mathermll dioo
computatiion
fin te
state
machiie
F8
M
ailon
SA
pural
authmm
Iin
simply
state
machine
model
rinle
state
modd line
F5
M
Enila
sutom
ation
SA
plural
aithom
ata
Enita
zutto
aton
or
samply
stale machine
mathomath col model
computettion
mathemmatic
machine
machiin
finiss
stato
r3
M
of
finiee
atoon
SA
plural
autom
ats
finiie
autom
simply
statio
model
finile
stblo
machine
rS
M
finite
autom
aton
SA
plural
autom
sta
finte
autom
alon
simply
stale
machine
mathematicos
model
computatlion
Figure 5. Relational attention in DAT language models encodes human-interpretable semantic relations. A visualization of the relations Pij learned by 24-layer 343M parameeee DAT language model. Top. Visualization one relation dimension the first layer, focusing the token 'modcl' which has high activation with the tokens 'state', 'machine' and ical', Bottom. Visualization one relation dimension in the twelfth layer, focusing on the token "state', which has high activation with the tokens 'mathematical' 'model', and 'computation'
overall (though expect similar optimizations to be possi- ble). An important direction for future work is the mecha- nistic interpretability [53, 55, 56] of DAT models, focusing on identifying specific circuits that perform key computa- tions, to better understand the performance improvements observed in complex domains like language modeling.
A Acknowledgment
This research was supported by the funds provided by the National Science Foundation and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence).
Code and Reproducibility
Impact Statement
This work develops variant of the Transformer architec- ture, called the Dual Attention Transformer, that integrates explicit relational computational mechanisms enhance data-efficiency and generalization in learning relational pro- cessing. This advancement has the potential to improve performance in applications requiring structured reasoning. Like other developments in Transformer architectures, this research shares the potential societal consequences associ- ated with the broader field.
Our implementation of the Dual Attention Transformer ar- chitecture soun-sourced at https: /tte:://qithiee com/ Awni 00 10o//ual-oooo attent on and published Python package. Pre-trained model weights, including the 1.3B- parameter DAT language model, are made publicly available and can be loaded directly using the package. Additionally, we provide code for running the experiments described in this paper, along with instructions for reproducing our re- sults and access to the experimental logs.

---

Relational & Sensory Processing in Transformers
References
[16] David Barrett et al. "Measuring abstract reasoning in neural networks". In: International conference on machine learning. PMLR. 2018 (cited on pages 1, 2). [17] Felix Hill et al. "Learning to Make Analogies by Contrasting Abstract Relational Structure". In: Inter- national Conference on Learning Representations. 2019 (cited page 1).
[1] Gary F Marcus. "The algebraic mind: Integrating connectionism and cognitive science". MIT press, 2003 (cited on page 1).
[2] David H Wolpert, William G Macready, et al. "No free lunch theorems for search". Tech. rep. Citeseer, 1995 (cited on page l).
[18] Taylor W. Webb, Ishan Sinha, and Jonathan D. Co- hen. "Emergent Symbols through Binding in External Memory". Mar. 2021. arXiv: 2012. 1 14101 [cs] (cited on pages 1, 2).
[3] Jonathan Baxter. "A model of inductive bias learn- ing". In: Journal of artificial intelligence research (2000) (cited on page 1).
[4] Dedre Gentner. "Saapeuure-mapping: A theoretical framework for analogy". In: Cognitive science (1983)
[19] Adam Santoro et al. "A Simple Neural Network Mod- ule for Relational Reasoning". June 2017. arXiv: [cs] (cited on pages 1, 2).
(cited on page 1).
1706. 01427
[5] Richard E Snow, Patrick C Kyllonen, Brachia Mar- shalek, et al. "The topography of ability and learning correlations". In: Advances in the psychology of hu- man intelligence (1984) (cited on page 1).
[20]
Murray Shanahan et al. "An Explicitly Relational Neural Network Architecture". In: Proceedings of the 37th International Conference on Machine Learning. 2020 (cited on pages 1, 2, 5, 17).
[6] Charles Kemp and Joshua B Tenenbaum. "The dis- covery of structural form". In: Proceedings of the Na- tional Academy of Sciences (2008) (cited on page 1). [7] Keith J Holyoak. "Analogy and relational reasoning".
[21] Giancarlo Kerg et al. "On Neural Architecture Induc- tive Biases for Relational Tasks". June 2022. arXiv: [cs] (cited on pages 1-3, 17).
2206. 05056
[22]
Awni Altabaa et al. "Abstractors and relational cross- attention: An inductive bias for explicit relational reasoning in Transformers". In: The Twelfth Inter- national Conference on Learning Representations. 2024 (cited on pages 1-3, 17, 22-25).
In: The Oxford handbook of thinking and reasoning
(2012) (cited on page 1).
[8] Brenden M Lake et al. "Building machines that learn and think like people". In: Behavioral and brain sci- ences (2017) (cited on page 1).
[23] Awni Altabaa and John Lafferty. "Learning Hierar- chical Relational Representations through Relational Convolutions". Feb. 2024. arXiv: 2310 03240
[9] Brian Cantwell Smith. "The promise of artificial in- telligence: reckoning and judgment". Mit Press, 2019 (cited on page 1).
[cs] ] (cited on pages 1-3).
[10] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. "Investigating the Limitations of Transformers with
[24]
Taylor W. Webb al. "The Relational Bottleneck as an Inductive Bias for Efficient Abstraction". In: Trends in Cognitive Sciences (May 2024) (cited on page 2).
Simple Arithmetic Tasks". 2021. arXiv: 2102
13019
(cited page 1).
[11] Anirudh Goyal and Yoshua Bengio. "Inductive bi- ases for deep learning of bigher-level cognition". In: Proceedings of the Royal Society A (2022) (cited on
[25] Ashish Vaswani al. "Attention Is All You Need". In: Advances in neural information processing systems (2017) (cited pages 2, 16).
page 1).
[26]
Jason Wei et al. "Chain-of-thought prompting elicits reasoning in large language models". In: Advances in neural information processing systems (2022) (cited on page 2).
[12] Allen Newell. "Physical symbol systems". In: Cogni- tive science (1980) (cited on page l).
[13] Stevan Harnad. "The symbol grounding problem". In: Physica D: Nonlinear Phenomena (1990) (cited on page 1).
[27] Takeshi Kojima et al. "Large language models are zero-shot reasoners". In: Advances in neural informa- tion processing systems (2022) (cited on page 2).
[14] Jerry& A Fodor and Zenon W Pylyshyn. "Connection- ism and cognitive architecture: A critical analysis". In: Cognition (1988) (cited on page 1).
[28] Karthik Valmeekam et al. "Large Language Models Still Can't It Plan (A Benchmark for LLMs on Planning and Reasoning about Change)". In: NeurIPS 2022 Foundation Models for Decision Making Workshop. 2022 (cited on page 2).
[15] Brenden Lake and Marco Baroni. "Generalization without systematicity; On the compositional skills of sentennce-t-sequeeeee recurrent networks". In: In- ternational conference machine learning. PMLR 2018 (cited on pages 1, 2).
10

---

Relational & Sensory Processing in Transformers
[29] Jie Huang and Kevin Chen-Chuan Chang. "Towards
Scale". In: International Conference on Learning Representations. 2021 (cited pages 5, 6, 20).
Reasoning in Large Language Models: A Survey". [cs CL] (cited on [42] David Saxton et al. "Analysing Mathematical Rea- soning Abilities of Neural Models". In: International Conference on Learning Representations. 2019 (cited on pages 6, 17).
2023. arXiv: 2212. 10403
page 2).
[30] Taylor Webb, Keith Holyoak, and Hongjing Lu. "Emergent analogical reasoning in large language models". In: Nature Human Behaviour (2023) (cited on page 2).
[43] Xiaohua Zhai et al. "Scaling vision transformers". In: Proceedings of the IEEE/CVE conference on com- puter vision and pattern recognition. 2022 (cited on page 6).
[31] Jie Huang et al. "I.arge Language Models Cannot Self-Correct Reasoning Yet". 2024. arXiv: 2310.
01798 [Cs.CL] (cited on page 2).
[44] Alex Krizhevsky. "Learning multiple layers of fea- tures from tiny images". Tech. rep. 2009 (cited on pages 7, 18).
[32] Iman Mirzadeh et al. "GSM-Symbolicc: Understand- ing the Limitations of Mathematical Reasoning in Large Language Models". 2024, arXiv: 2410
[45] Hongyi Zhang et al. "mixup: Beyond Empirical Risk Minimization". In: International Conference on Learning Representations. 2018 (cited on pages 7, 20).
05229 [cs. LG] (cited on page 2).
[33] Tom B. Brown et al. "Language Models are Few-Shot Learners". 2020. arXiv: 2005. 14165 65 CL)
(cited on page 2).
[46]
Sangdoo Yun al. "Cutmix: Regularization strategy to train strong classifiers with localizable features". In: Proceedings of the IEEE/CVF international con- ference on computer vision. 2019 (cited on pages 7, 20).
[34] Jared Kaplan et al. "Scaling Laws for Neural Language Models". 2020. arXiv: 2001 08361
[cs.LG, (cited on pages 2, 2,2).
[35] Jordan Hoffmann al. "Training Compute-Optimal Large Language Models". 2022. arXiv: 2203 15556 [cs.CL] CL] (cited on pages 2, 7).
[47] Justin Johnson et al. "CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning". In: Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR). 2017 (cited on page 7).
[36] Aaron Grattafiori al. "The Ilama 3 Herd of Mod- els". 2024. arXiv: 2407 21783 [cs .AI] (cited
on page 2).
[37] Adam Santoro et al. "Relational Recurrent Neural Networks". In: Advances in Neural Information Pro- cessing Systems. Curran Associates, Inc., 2018 (cited on page 2).
[48]
Aimen Zerroug al. "A benchmark for composi- tional visual reasoning". In: Advances in neural infor- mation processing systems (2022) (cited on page 7).
[49] Bingchen Zhao et al. "Benchmarking Multi-Image Understanding in Vision and Language Models: Per- ception, Knowledge, Reasoning, and Multi-Hop Rea- soning". In: arXiv preprint arXiv:2406. 220 /2742 (2024) (cited on page 7).
[38] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. -Self-Attention with Relative Position Representa- tions". In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers). Ed. by Marilyn Walker, Heng Ji, and Amanda Stent. New Orleans, Louisiana: Association for Computational Linguis- tics, June 2018 (cited on page 5).
[50]
Anton Lozhkov et al. "FineWeb-Edu", May 2024 (cited on pages 8, 21).
[51]
Kevin Clark et al. "What Does BERT Look At? An Analysis of BERT's Attention".. 2019. arXiv: 1906. 04341
[cs. . CL (cited on page 8).
[39] Jianlin Su et al. "RoFormer: Enhanced Transformer with Rotary Position Embedding". Nov. 2023. arXiv: [cs] (cited on pages 5, 16).
[52] Phu Mon Htut et al. "Do Attention Heads in BERT Track Syntactic Dependencies?" 2019. arXiv: 1911.
2104. 098 64
12246
[cs. [C) (cited on page 8).
[40] Jacob Devlin et al. "BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding". In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguisties. Association for Computational Linguistics, June 2019 (cited on page 5).
[53] Nelson Elhage et al. "A Mathematical Frame- work for Transformer Circuits". In: Transformer Circuits Thread (2021). https//transformer- (cited on
page 8).
[54]
Tri Dao et al. "Flashattention: Fast and memory- efficient exact attention with I0-awareness". In: Ad- vances in Neural Information Processing Systems (2022) (cited on page 8).
[41] Alexey Dosovitskiy et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition

---

Relational & Sensory Processing in Transformers
[55] Catherine Olsson al. "In-context Learning and Induction Heads". In: Transformer Circuits Thread
[69] Guilherme Penedo et al. "The FineWeb Datasets: De- canting the Web for the Finest Text Data Scale". [cs. CL] (cited on
(2022). https://transfoomrr-ciruuutt pitp:rrcicrcuiisppp/po/fiiiii pub/2022/in-- context-learning- annddnduutioonheeeeee html (cited on page 8).
2024. arXiv: 2406 17557
page 21).
[70] Alec Radford et al. "Language models are unsuper- vised multitask learners". In: OpenAl blog (2019) (cited on page 21).
[56] Kevin Ro Wang al. "Interpretability in the Wild:
Circuit for Indirect Object Identification in GPT-2 Small". In: The Eleventh International Conference on Learning Representations. 2023 (cited on page 8). [57] Gerard Debreu et al. "Representation of a preference ordering by numerical function". In: Decision pro- cesses (1954) (cited on page 12).
[71] Ronen Eldan and Yuanzhi Li. "TinyStories: How Small Can Language Models and Still Speak Co- herent English?" May 2023, arXiv: 2305 07759 [cs] (cited page 24).
[58] Awni Altabaa and John Lafferty. "Approximation of Relation Functions and Attention Mechanisms". Feb. 2024. arXiv: 2402 08856 [cs, stat] (cited page 13).
[59] Noam Shazeer. "GLU Variants Improve Trans- former". Feb. 2020. arXiv: 2002 05202 [cs,
stat] (cited on page 16).
[60] Yann N Dauphin et al. "Language modeling with gated convolutional networks". In: International con- ference on machine learning. PMLR. 2017 (cited on page 16).
[61] Dan Hendrycks and Kevin Gimpel. "Gaussian Error Linear Units (GELUs)". 2016. arXiv: 1606. 08415
[CS.LG] (cited on page 16).
[62] Hugo Touvron al. "Llama 2: Open Foundation and Fine-Tuned Chat Models". July 2023. arXiv: 2307.
09288 [cs] (cited on page 16).
[63]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization" 2016. arXiv: 1607. [stat. ML] (cited page 16).
O 04550
[64]
Biao Zhang and Rico Sennrich. "Root mean square layer normalization". In: Advances in Neural Infor- mation Processing Systems (2019) (cited on page 16). Ruibin Xiong et al. "On layer normalization in the transformer architecture". In: International Confer- ence on Machine Learning. PMLR. 2020 (cited on page 16).
[65]
[66]
Francesco Locatello et al. "Object-Centrie Learn- ing with Slot Attention". Oct. 2020. arXiv: 2006. 15055 [cs, stat] (cited on page 16).
[67] Joshua Ainslie et "GQA: Training Generalized Multi-Query Transformer Models from Head Checkpoints". In: Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Pro- cessing. Singapore: Association for Computational Linguistics, 2023 (cited on page 20).
[68] Ekin D. Cubuk et al. "AutoAugment: Learning Aug- mentation Policies from Data". 2019. arXiv: 1805. 09501 [cs.CV] (cited on page 20).
12

---

Relational & Sensory Processing in Transformers
A. Function Class of Relational Attention: a universal approximation result
To gain better understanding of the types of functions that can be computed by relational attention, we presented simple approximation result (Theorem 1) in Section 2.4. Here, we will provide a formal statement of the result and prove it.
Recall that relational attention is a mapping on Rd x Raxd + R where d is the dimension of the input objects and dout is the output dimension. For convenience, we denote the "query space" by X' and the "key space" by J though both are Rd in this setting. Relational attention takes as input query x E X and a collection of objects y (vn.-... ..n) E ) and computes the following
RA(x, y) = Σ
a, y) W, s W.),
(4)
1=1
a(;; v) = Sofumax ([(ogtiu 0% (ztnm())) 1 1 E A
(5)
r(z y:) ((oT)) (E (E(d,]
(E(4)) Rdd,
(6)
($1 ................................ 8n) SymbolRetriever (y:; Sib) pnxdout,
where og 9k gatta dree tively. For this section, these are multi-layer perceptrons. Note that in Algorithm 1 these are linear maps, but they preceded by multi-layer perceptron in Algorithms 2 and 3, which makes the overall function class the same. Moreover, for this analysis we will take W, = dout = d, and We- = O. We will later discuss how the role of symbols fits within the message of the result.
Pi, Rd dede are the feature maps defining the attention mechanism and the relation, respec-
The following result states that relational attention can approximate any function of the form: 1) select an object in (y1.....,/% by an arbitrary query-dependent selection criterion, and 2) compute an arbitrary relation X x y Rd. with the selected object. This is formalized below.
To formalize (I), we adopt an abstract and very general formulation of "selection criterion" in terms of family of preference preorders, }< for each possible query the preorder <x defines a preference over objects in y to be selected. Intuitively, "yi 2o y22 means that 1/2 is more relevant to the query x than 3/.
More precisely, for each query x E X, 5x a complete (for each 1/1 1/2 E Y, either 1/1 < 1/2 or 1/2 %x 11). reflexive (y *x y for all y E )), and transitive (у/1 <x 1/2 and y/2 *5 ys implies y1 5. y3) relation. For each x E X, Z5. induces a preordered space (),, <.)) This implicitly defines two additional relations: <a and ~2 We will write yı -i y2 if "y Rx 2/2
and not y22 <x y1" and y1 y/2 if 1/1 <x 1/2 and y/2 <x V1".
For collection of objects y = (vn...., Vn) E 3mhh and query x E X, the preorder S. defines a selection function
Select (y1,...,,,h)) = max ((vi,..., ;yn), key =5a).
(8)
That is. Select(x, y) returns the most relevant element with respect to the query In particular, it returns ys when V: > yj, Vj # (and may return an arbitrary element if no unique maximal element exists in (y......y..)
We will assume some regularity conditions on the family of preorders (<x) which essentially stipulate that: 1) nearby elements in y have a similar preference with respect to each æ, and 2) nearby queries in X' induce similar preference preorders.
Assumption 1 (Selection criterion is query continoons and key-conti.uous). The family of preorder relations {<x satisfies the following:
1. Key-continuity. For each x E X, 2x is continuous. That is, for any sequence (y:)s such that yi Xx 2 and yi + you. we have You Sxx z. Equivalently, for any y E V, {2 y 4x y) and {z E y y Lx 2} are closed sets Y
2. Query-continuity. Under key-continuity, Debreu et al. [57] shows that each E X, there exists a continuous in utility function u : y - R for <x such that 31 to V2 - un( S u2 (V2). For query- query-contimuity, we make the further assumption that there exists a family of utility functions {uz : y - R]aex such that u(x, y) == U2 (y) is also continuous in its first argument.
13

---

Relational & Sensory Processing in Transformers
For technical reasons, for Equation (8) make sense, must assume that there exists unique element be selected. We formulate this in terms of assumption on the data distribution of the space A' y This is technical assumption, and different forms of such an assumption would be possible (e.g. instead condition on this event).
Assumption 2 (Selection is unique almost always). Let (æ. y) Fz.y- For each E > 0, there exists ne 0 such that ming (v.) - "n (1,) > 1e with probability at least 1 - E.
Theorem (Function class of relational attention). Let X,Y be compact Euclidean spaces. Let {5x)oex be an ar- bitrary family of relevance preorders y which are query-continuons and key-continuous (Assumption 1). Let selectionfuunctinn Select(z, (Ji...., ..)) max((th..... Wn), key =xx) be the selection function associated with {zziz. Let R : X X'y + Rdd be an arbitrary continuous relation function. Suppose v. y ~ Pa.y and that Assumption 2 holds (i.e, the data distri- bution is such that there exists a unique most-relevant element w.h.p). For any & > O, there exists multi-layer perceptrons gattn Th &g ok and a choice of symbols such that,
[[A( (V1....,, .n)) - R(x, Select (V1...... ((.....,,,))))...... yn))) <E
Proof. Condition on the event & == {(x, y) E X x y": minj (um(y)) - Ug un( > n Let 1' arg maax(())...-,,b key =x,)= = arg max (u, (y,):...,,,(((n))
By [58, Theorem 5.1], for any ei > O, there ex- ists MLPs dight 0f such that ag- (z, 3) > E1 for any (z, y) E E. That is, the attention score is nearly 1 for the <y--selected element uniformly over inputs in E
Similarly, by [58, Theorem 3.1], for any 6 0, there exists MLPs (d (ibb% (e(a) such that r(x, = ((op)) ord of (e)))))) approximates the target relation uniformly within an error of E2,
[|R(x, v) - r(x, ul < 62, Lebesgue almost every (x, v) x X
Thus, we have
[RA (yi,.... (b))) - R(x, Select(z, (z, ..l.....,n))))...
Σ (=(2;)) ((x, R(x, yr
||,(,;; r( R(x, ve- )l
s ag (z. y) [|, R(e, y1 )1
[a((x:)) v)) R(z, yr
jfH
$ (1 - 01)82 + max ||( v) - R(x, v/ oloo
2.y.0"
Note that max ||((,,)) v) - R(x, yn Io is finite since x, y are compact and r, A are continuous. Letting S1,82 be small enough completes the proof.
To summarize the analysis in this section, we showed that relational attention can approximate any computation composed of first selecting an object from a collection then computing a relation with that object. We can approximate any well-behaved selection criterion by formulating it in terms of an abstract preference preorder, and approximating the corresponding utility function (given by Debreu representation theorem) by inner products of query and key feature maps. We can then approximate the target relation function similarly by inner products of a different set of query and key feature maps.
In the analysis above, we set aside the role of the symbols. Note that the function class this approximation result proves involves retrieving a relation from a selected object, but does not explicitly encode the identity of the selected object. Informally, the receiver knows that has a particular relation with one of the objects its context, and knows that this relation is with an object that was selected according a particular selection criterion, but does not know the identity of the object beyond that. This is the purpose of adding symbols to relational attention the retrieved relation is tagged with symbol identifying the source.
14

---

Relational & Sensory Processing in Transformers
B. Architecture & Implementation Details
In this section, we briefly discuss some details of implementation that may be of interest to some readers. Our code is publicly available through the project git repository and includes detailed instructions for reproducing our experimental results. We also provide links experimental logs. Our code uses the PyTorch framework.
B.1. Relational Attention and Dual-Head Attention
The relational attention operation is defined as part of dual-head attention in Algorithm 1. We briefly mention some details of the implementation.
Learnable parameters. Let ni := ni, + ni be the total number of sensory and relational heads. The learnable parameters are
Sensory attention heads. For each head h E [mg]]:
Attention query/key projections: Watt Wattn E Dddndde x daey
q.h Value projections: t Redmodal Xdi,,
o Output projection: Waa E ggdmadeiXddoodd
Relational attention heads. For each head h E [m]]]] and each relation e & [dr]:
Attention query/key projections: Wattt, Wattn E [Rdmodde Xdhey,
9. h.
Relation query/key projections: WTel Wiel E R.dmede
g.E.
Symbol projection: Whth E Bddmodeddd
o Relation projection: Wh E Rddxxd,
o Output projection: W E R Rddmade X Xdmode)
We let they, dh = d n to maintain the same dimension for the input and output objects. Similarly, we let dproj = di- ng/// /d, so that the number of parameters is fixed d, varies. That is, we scale dproj down as dy increases; dproj has the interpretation of being the dimensionality of the subspace on which we computing comparisons. So, having larger number of relations corresponds to a more fine-grained comparison between the two objects.
To model symmetric relations, we let W = Wxy// Recall that this has the interpretation of computing comparison between the same attributes in the pair of objects.
Note that the same d,-diimnsional relation is used for all n attention heads, with a different learned linear map WAA for each head extracting the relevant aspects of the relation for that attention head and controlling the placement in the residual stream. This allows for useful computations to be shared across all heads. Note also that the head dimension dn dmodee// R is defined in terms of the total number of attention heads and is the same for both sensory attention and relational attention. The output of each head is a dj dimeennnsional vector. This means that after concatenating all the heads, the proportion in the final dmodel dimensoadi output that corresponds to each attention head type is proportional to the number of heads of that type. For example, if n 6, nt 2. then 75% of the the output of sensory attention heads and 25% is composed of the output of relational attention heads. This enables tuning the relative importance of each head type for the task.
output is composed of
Code. We briefly diseuss the code implementing relational attention. We use i nsum operations heavily in our implemen- tation due to the flexibility they offer for implementing general tensor contractions. From Algorithm 1, recall that relational attention takes the form:
0g3 < Σ
(b) oig (rig + s,W").
(h) where on are the sofumax attention scores for head h E [m/9), Tey E Rdd are relation vectors, 8j E Rddmedal is the symbol associated with the j-th input, and W Wh map rij and 8j, respectively, to dimeinnnsoo vectors. We assume those are already computed and focus on a particular portion of the computation of relational attention. We break up the computation as follows:
(1) (r.g + +,,Wh) Σ (a (1) sw! + Σ Σ a
(h) oij rsy W
(10)
15

---

Relational & Sensory Processing in Transformers
Note that we factor out the W linear map and apply it after computing E, oij Vag- This intentional, as will be explained below.
(h)
This can be computed in PyTorch via e einsum operations as follows.
av: (b, ny n_h, d_h) (b, n, n) relations: (b, n, d.)) self.wr: in d, z))
attended_symbols = torch. (th))ummmbb,,,,jhha2da attn_scores, sv) shape: (b, nhh, dh)
attended_relationn relations torch. (tb. attn_scores, relations) shape: (b, h d_))
attended relations torch. (' hdr->bihd', deteenedd_reltttin self.wr) shape: (b, n, h dhh)
attended_relations
output detended_symbol attended_relattions shape: (b, r_h, d_h)
Here, we assume sv, aattn_scoree and relatt atlati ons are already computed, and focus on particular part of the compu- tation. sv(:, ;,,,,,,, h, :1 = s W corresponds to the symbols of each object in the context, attn_scores [: h, :, :1 a are the softmax attention scores, and relations [:, i, ], :] = Tij are the relations, which can all be computed with simple matrix multiplication operations, very similar to the standard implementations of multi-head attention.
The first line corresponds to computing E, af The second line corresponds to computing E, adjray. The third line corresponds to applying the linear map W to the retrieved relations at each head. The reason we apply the map W after attending to the relations for memory efficiency reasons. If we were apply W first, we would need to manifest a tensor of dimension b it after computing attention, we only need to manifest tensor of dimension 6 d, < dmodel- This tensor is contracted to dimension b x d, first, then mapped up to X n x nh x d, This makes the memory footprint of relational attention of the same order as standard (sensory) attention when d, x f4.
nh dh, which is of order O(b n²² -moode)) Instead, by factoring out W and applying
dy, which is much smaller since
When using position-relative symbols, the implementation adjusted since we need to compute
(b) (rig w?"
(11)
instead, where the symbol 8j-; sent now depends on both the source and the target Thus, we now compute a symbols tensor which is indexed by both the source j and target i: sv [i, 1, j, h, :] sy-:W Then, the implementation is adjusted by replacing the first line in the code above with
attended_aymbols torch einsum (? ('bhij, ijhd->bihd''
attn_scores, sv)
The full implementation made available through the project's github repository.
Composing relational attention learn hierarchical relations. We remark that composing relational attention modules can be interpreted as representing hierarchical or higher-order relations. That is, relations between relations. An example of this is the relation tested in the matcc pattern task in the relational games benchmark. After one iteration of relational attention, an object's 's representation is updated with the relations it has with its context. A second iteration of relational attention now computes a representation of the relation between an object's relations and the relations of the objects in its context.
B.2. Encoder and Decoder Blocks
We briefly mention a few configurations in our implementation that appear in our experiments. We aimed to make our implementation configurable to allow for various tweaks and optimizations that have been found in the literature for training Transformer models.
16

---

Relational & Sensory Processing in Transformers
Symbol assignment. shared symbol assignment module is used for all layers in the model. We explore three types of symbol assignment mechanisms: positional symbols, position-relative symbols, and symbolic attention. Different symbol assignment mechanisms are more well-suited to different tasks. We discuss ablation experiments we carried out the effect of the symbol assignment mechanism in Appendix C.
MLP block. The MLP block uses a 2-layer feedforward network with a configurable activation function. The intermediate layer size is dø 4 dmodel by default. We also use the SwiGLU "activation function" [59] some of our experiments. SwiGLU is not merely an activation function, but is rather neural network layer defined as the component-wiss product of two linear transformations of the input. is a type of gated linear unit [60] with the sigmoid activation replaced with Swish activation [61]. SwiGLU(<) Swish(xW b) @ (=V + c). This is used in the Llama series of models and was found to be useful modification [62].
Normalization. Either LayerNorm [63] or RMSNorm [64] can be used. Normalization can be performed post-attention, like in the original Transformer paper [25], or pre-attention as in [65].
Positional encoding. Our experiments use either learned positional embeddings or RoPE [39].
17

---

Relational & Sensory Processing in Transformers
same
occurs
xoccurs
between
match patt
+
+
PEP
H
4
EEE
F
rH
7L4
ThI
Figure 6. Examples of different tasks in the Relational Games benchmark. Each column corresponds different task in the benchmark. The top row is an example of positive instance and the bottom row is an example of a negative instance.
C. Experimental Details & Further Discussion
C.1. Relational Games (Section 4.1)
EXPERIMENTAL DETAILS
Dataset details. The Relational Games benchmark datasets consists of 36 x 36 x 3 RGB images depicting 3 3 grid of objects which satisfy a particular visual relationship. The task is to identify whether a given relationship holds or not. The of objects consists of simple geometric shapes. Examples of each task are presented in Figure 5. For example, in the occurs task, in the top row occurs (i.e., is among) the objects in the bottom row. The most difficult task in the benchmark is the mat ch patte aitter rn task, where the grid contains triplet of objects in the top row and another triplet of objects in the bottom row. Each triplet satisfies some relationship (e.g., ABC, ABA, ABB, or AAB), and the task is to determine whether the relation in the first triplet is the same as the relation in the second triplet. The difficulty in solving this task is that it requires parsing a recond-order relation (a relation between relations). We remark that composing relational attention modules naturally captures this kind of hierarchical relations: the first relational attention operation produces objects representing relational information and the second would compute relations between those relations (i.e., second-order relations).
object is present in the top row and three in the bottom row, and the task is to determine whether the object
Model architectures. We use Vision- dikion-Trnsffomer--yye architecture where the input image is split up into patches, flattened, and passed through the sequence model with added learned positional embeddings. We use average pooling at the end and pass through an MLP to produce the final prediction. We use patch size of 12 x 12 which separates objects according to the grid structure. We note that in more general visual relational reasoning tasks where there isn't 't't this type of grid structure, would be appropriate to combine our approach with an object-discovery module such as Slot Attention [66].
We 2-layer models. The DAT' models dmodel 128, dn 256. One set of Transformer baselines uses the same, while another is larger with dmadel 144, de = 288. All models use SwiGLU "activation". dropout rate 0.1, and me-LayerrNormalizttoo For the DAT models, we use positional symbols as the symbol assignment mechanism. The composition of sensory and relational attention heads are depicted in the figure. In Figure 2, we use symmetric relations (i.e., imposing that W W Below, we also explore the effect of this inductive bias, evaluating variants without the symmetry constraint.
Training details. For each task and model, we evaluated learning curves by varying the training size and training the model until convergence, then evaluating on hold-out test set. For four out of five of the tasks, we evaluate learning curves within the range of 250 to 2, 500 samples, in increments of 250. For the more difficult match. pattern, the range is from 5, 000 to 25, 000 in increments of 5, 000. The ranges were chosen based on the difficulty of the different tasks in order to identify the right "resolution". When evaluating learning curves, each training set is sampled randomly from the full dataset. For each task, model, and training set size, we repeat the experiment times with different random seeds to compute approximate confidence intervals (accounting for randomness in sampling the dataset and random initialization). We use an Adam optimizer with learning rate of 0.001, 31 = 0.9, 82 = 0.99, and batch size of 512. We train for 50 epochs.
18

---

Relational & Sensory Processing in Transformers
FURTHER DISCUSSION, EXPLORATION, & ABLATIONS
Comparison previous relational architectures. Previous research has explored relational learning in synthetic settings, proposing various architectures with relational inductive biases. Here, compare DAT to three such architectures: PrediNet [20]. CoReINet [21], and Abstractor [22]. Unlike DAT, these architectures use subtractive rather than additive relational inductive biases, imposing constraints on the types of learnable representations to improve relational learning efficiency. As result, they are not general-purpose architectures and cannot be applied to broader domains such as language modeling. Nonetheless, is useful to compare DAT against those architectures to explore the trade-offs of strong inductive biases and evaluate DAT in comparison to alternative approaches to relational learning. Figure shows learning curves comparing DAT against those baselines. DAT performs competitively with previous relational architectures, generally outperforming PrediNet and Abstractor, while performing marginally worse than CoReINet. It is relevant note that CoReINet incorporates strong task-specific inductive biases, and was partially designed with this benchmark in mind.
same
Occure
xOccurs
Goneralization Accuracy
0.8
0.8
0.7
0.7
0.0
0.6
0.6
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Training Set Size
Training Set Size
Training Set Size
betueen
match pattern
Generalization Accuracy
Model
0.9
0.9
DAT (mg 0, mg" 2] [421K]
(mg 8, 0) [481K]
0.8
0.8
T (nf)" = 2,n 0) [481
0.7
0.7
(Hg]" 2. 0) [386K]
PrediNet [376K]
0.6
CoReiNet [215K]
[469K]
500
1000
1500
2000
2500
5000
10000
15000
20000
25000
Training Size
Training Set Size
Figure 7. Learning curves on the Relational Games benchmark, comparing DAT against wrreviously-propooo relational architectures. DAT performs competitively with previous relational architectures.
Ablation over symmetry. We performed an ablation over the symmetry inductive bias in the relations computed in relational attention. Our implementation exposes argument which controls whether the relation r(æ, y) ((Wrel q.e W )ee(a,)
Rd. modeled in relational attention is constrained to be symmetric by setting 9.6 = Wrrel k,g Indeed, we find symmetry to be useful inductive bias in this task. Figure depicts learning curves for the two configurations of DAT comparing symmetric RA against asymmetric RA. We find that symmetry results in faster learning curves for both configurations.
C.2. Mathematical Problem-Solving (Section 4.2)
EXPERIMENTAL DETAILS
Dataset details. Saxton et al. [42] propose a benchmark to assess neural models' ability to perform mathematical reasoning. The dataset consists of suite of tasks free-form textual input/output format. The tasks cover several topics in mathematics, including arithmetic, algebra, and calculus. For each task, the authors programmatically generate 2 10 training examples and a validation examples. Questions have maximum length of 160 characters and answers have maximum length of 30 characters.
Model architectures. We use an encoder- decoder architecture for this experiment, treating as a sequence-to-sequeee -eueence---eeeuence task. We use character-level encoding with a common alphabet size 85 containing small and upper case letters, digits 0-9, and symbols (e.g..*. *, /, +, -). We vary the number of layers to explore how performance scales with model size
19

---

Relational & Sensory Processing in Transformers
came
occurs
xoccurs
Gencralization Accuracy
0.9
0.8
0.8
0.8
0.7
0.6
0.6
0.6
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Training Set Size
Training Set Size
Training Set Size
between
match pattern
Generalization Accuracy
1.0
Model
0.8
0.8
Dual-Attn Transformer (Hi). 1, 1,
Dual-Attn Transformer (xq] = 0, N.
Symmetric RA
0.6
0.6
Asymmetric RA
500
1000
1500
2000
2500
5000
10000
15000
20000
25000
Training Set Size
Training Set Size
Figure ablation the effect of symmetry relational attention in relational games experiments.
in DAT compared to standard Transformers. Each encode/decoder block uses ReLU activation, dropout rate 0.1, and nost-normalization. We use dmooell 128, dæ 256 for the DAT models and dmoodel 144, de 288 in the Transformer models to control for parameter count and give the Transformer advantage in the evaluation. Sinusoidal positional embeddings are used as the positional encoding method. For all models, the total number of attention heads (across self-attention and relational attention) is 8. For the Transformer model, there are only self-attention heads: n = 8 for both the encoder and decoder. For DAT, we evaluated two configurations for the composition of head types, one with n = n, = in the encoder and sa = 8, nhh = in the decoder (i.e., standard Transformer Decoder), and one with n 4 = n 4 in the encoder and ni," 4 ra
the decoder. The number of cross-atis coss-attention heads in the decoder is 8 in all cases. No symmetry constraint is made relational attention. Position-relative symbols used the symbol assignment mechanism, and the symbol library is shared across all layers in both the encoder and decoder.
Training Details. Each model is trained on each task for 50 epochs. We use the Adam optimizer with 31 0.9, 32 = 0.995, a learning rate of x 10-4, and a batch size of 128. We evaluate and track the per-character accuracy over the course of training. We repeat this process 5 times for each combination of model and task with different random seeds to compute approximate confidence intervals.
FURTHER DISCUSSION, EXPLORATION, & ABLATIONS
Table 2 reports the full set of results obtained for this experiment, including certain configurations omitted from the figure in the main text.
C.3. Visual Processing (Section 4.3)
EXPERIMENTAL DETAILS
Dataset details. In this set of experiments, labeled small images. The CIFAR-I0 dataset consists of 60, 000 32 x 32 RGB images, evenly split across 10 classes. The CIFAR-100 dataset consists of 60, 000 RGB images of the same size, evenly split across 100 classes.
the CIFAR-10 and CIFAR-100 datasets [44] which datasets of
Model architectures. We use ViT-style architecture [41]. RGB images are divided into 4 x 4 patches, flattened, linearly embedded into a vector, and fed through an Encoder. We use average pooling followed by an MLP to produce the final prediction. We evaluate l-yerr models with dmodel dg 384, GeLU activation, Pre-LaayrNoormaizzation, and no
20

---

Relational & Sensory Processing in Transformers
Task
Model
Parumeter Count
# Layers
duold
Encoder
Encoder of
Decoder ng
Decoder ro
Aecuracy
Trunsfommer
692K
2
128
62.5 1.1%
DAT
783K
2
128
66.5 1.0%
Transformer
871K
2
144
64.0 1.5%
algebro Hinear_
6.5%
Transformer
1.3M
3
144
8
57.0 2.3%
DAT
1.43M
128
73.1 1.1%
Transtomer
692K
128
91.1 0.2%
DAT
783K
128
91.6 0.6%
Transformer
871K
144
91.4 0.2%
Sueebra___qqueecee temm
DAT
1,9M
128
97.0 0.59%
Transformer
96.1 0.5%
DAT
143M
128
Transfomeer
1.7M
144
93.1 2.0%
Transfomerr
692K
12H
10.9 0.07%
DAT
783K
2
12H
100.0 + 0.0%
Transformer
calculus differentiate
DAT
1.09M
128
Transfommer
1.3M
144
99.9 0.0%
DAT
0.0%
Transformer
1.M
4
144
99.9 1 0.0%%
Transformer
128
83.3 0.19
DAT
783K
12H
85.6 0.0%
Transformer
871K
144
84.5 0.3%
polynomiels
0.1%
Transformer
1.3M
144
86.4 0.3%
DAT
1.43M
128
88.7 0.0%
Transformer
1.7M
144
87.6 0.2%
Transforeer
692K
2
128
74.0 = 0.7%
0.1%%
Transformee
S71K
144
74.1 0.6%
polynomisls
expppp expand
DAT
1.0MM
128
Transformer
1.3M
144
1.2%
DAT
1443M
4
128
91.1 1 0.9%
Transformer
80.2 0.6%
Table Full results of mathematical problem-solve experiments. For each task, this table shows the mean character-level accuracy + the standard error of mean for each model configuration.
21

---

Relational & Sensory Processing in Transformers
Dataset
Model
Parameter Count
# Layers
dmodee
Symmetric rij
Accuracy
VIT
7.1M
8
384
12
0
NA
86.4 0.1%
CIFAR-10
6.0M
8
384
6
Yes
89.7 = 0.1%
ViDAT
6.6M
8
384
6
No
89.5 = 0.1%
VIT'
7.2M
384
12
0
NA
68.8 = 0.2%
CIFAR-100
6.1M
384
6
Yes
70.5 = 0.1%
ViDAT
6.7M
384
6
No
70.5 = 0.1%
Table Ablation over symmetry of riy relational attention image recognition experiments.
Dataset
Model
Parameter Count
# Layers
dmodel
7hh
1
Accuracy
VIT
7.1M
8
384
12
89.5 0.1%
CIFAR-10
ViDAT
6.0M
8
384
6
91.7 + 0.1%
VIT
7.2M
8
384
12
68.2 + 0.1% 70.9 + 0.1%
CIFAR-100
VIDAT
6.1M
8
384
6
Table 4. Classification accuracy on CIFAR- 10 and CIFAR- 110 with AutoAugment data augmentation during training. Each training configuration is repeated 10 times with different random seeds; report the mean accuracy + the standard error of mean. DAT continues to outperform the standard Vision Transformer.
dropout. The VIT' model has n = 12 standard self-attention heads, while the DAT model uses both sensory and relational heads, with processing involves symmetric attribute-simllarity relations. We also carried out experiments with asymmetric relations and discuss the results below. In DAT, we use position-relative symbols as the symbol assignment mechanism. Further, we use Grouped Query Attention [67] in DAT to reduce the parameter count to account for the added parameters in relational attention.
split ni n/."" = 6. In the main text, symmetric relations Tay with the intuition that visual
Training Details. We train for 100 epochs. We use the Adam optimizer with a learning rate schedule consisting of a gradual warmup 10-3 in the first 5 epochs, followed by cosine rate decay down 10-5 We the hyperparameters 31 = 0.9, 32 = 0.999, and weight decay of 5 10-5, We normalize the images channel such that pixels have mean zero and unit standard deviation. In the results reported in Table in the main text, we use random cropping, Mix [45], and CutMix [46] as data augmentation techniques during training. We also report results using AutoAugment [68] below.
FURTHER DISCUSSION, EXPLORATION, ABLATIONS
Effect symmetry in Tij. In the main text, Table reports DAT results with symmetric relations Thy by imposing 9
Wrell Wrrel Here, we explore the effect of this choice. Table compares DAT models with and without the symmetry constraint. We find no significant difference in performance. Though, note the smaller parameter count in the symmetric variant.
Interpretability Visualization. ?? depicts visualization of the relations learned by a ViDAT model trained on the CIFAR dataset. The relations Tsy in relational attention can be interpreted as applying the source patch i as aiileer, comparing it against each patch j. The different components of rıy can be seen as analogous to the channels in a convolution operation. We see that some relations in the ViDAT' model appear to capture a human-interpretable notion visual similarity across patches.
Alternative data augmentation. In the main text, use random cropping, MixUp, and CutMix data augmentation during training. Here, we report results on an alternative data augmentation technique: AutoAugment [68]. Automugment is an optimized set of data augmentation policies, found through a data-dependent automatic search procedure. At each batch a random sub-policy is chosen which consists of image processing operations such as translation, rotation, or shearing. Table 4 reports results using this data augmentation procedure. We continue to find that ViDAT outperforms the standard VIT' model.
22

---

Relational & Sensory Processing in Transformers
(a) Original Image
(b) Relation in the First Layer
(c) A Relation in the Fifth Layer
Figure 9. A visualization of the relations r:y[4] learned by 8-layer OM-parameter ViDAT model trained on CIFAR. The left panel depicts the original image input. middle and right panels depict the relation in relation vector Pky between the source patch i (red outline) and every other patch j. The relation activation is normalized with the tanh function for visualization. The normalized value the relation represented by tint each patch j-a green hue large positive activations and red for negative activations.
C.4. Language Modeling (Section 4.4)
EXPERIMENTAL DETAILS
Dataset details. The FineWeb-Ead [50] dataset is curated dataset of text data. is generated by filtering the large- scale FineWeb dataset for LLM pre-training [69] using educational quality classifier trained annotations generated by Bontaa--7B-insttruee Feb-Wed has been shown to outperform WineWeb several benchmarks, demonstrating the importance of data quality. We train our language models on random subset of 10 billion tokens of FineWeb- Web-Edu.
Model Architectures. We use Decoder-only architecture, with causal attention for autoregressive language modeling. We vary model size to explore the scaling properties of DAT with respect to both model size and data size, comparing to the scaling properties standard Transformers. Our architectural hyperparameters follow common choices at different model scales, based on scaling analyses ses performed for Transformers [69]. We explore 3 model scales: 350M (dmode) 1024, ni 16, L 24), 750M (dmodel 1536, Пą 24, L = 24), and 1.3B (dmodel 2048, n4 32, L 24) parameters. We use dg 4 d'model, GeLU activation, RoPE positional encoding, no bias, dropout, and rre-LayerNormalizatinn We use the GPT2 tokenizer [70]. We use symbolic attention as the symbol assignment mechanism, with the number of symbols in the symbol library scaling with model size: 1024 symbols and 8 heads for the 350M and 750M scale models, and 2048 symbols with 16 heads for the 1.3B scale model. We also increase the relation dimension with model size. We don't 't impose a symmetry constraint, with the intuition that linguistic relations can asymmetric. We use Grouped Query Attention in the DAT models to reduce parameter count to account for the added parameters in relational attention, making them smaller overall compared to the Transformer baselines each parameter scale.
Training Details. We train for 10B Tokens, with each batch containing 524, 288 tokens, split into context windows of 1, 024 tokens. We use gradient accumulation to fit micro-batches into memory. We use the Adam W optimizer with maximum learning rate of 6 x 10-4 and minimum learning rate of 6 x 10-5 first linearly warming over the first 715 steps, then decaying back down with a cosine schedule. We use 31 = 0.9, B3 = 0.95 and a weight decay of 0.1. We also use gradient clipping to unit norm.
FURTHER DISCUSSION, EXPLORATION, & ABLATIONS
Figure in the main text depicts the scaling properties of a DAT language model with respect to model size and data size compared to standard Transformer. Here, we provide few additional representations of the results. Table 5 reports the end-of-training validation perplexity of the different models.
Figure depicts training curves for the different model scales. We observe a power law scaling of the validation loss with respect to number of training tokens. This matches the neural scaling laws [34], which suggest that validation loss ought to scale roughly as d where is the amount of training data and the exponent a constant that depends on model
23

---

Relational & Sensory Processing in Transformers
architecture, training details, etc.
Table 5. End-of-training validation perplexity in language modeling FincWeb-Ead Edu dataset.
Model
Param count
Tokens
dmodel
Mlayers
h
d,
Rke
Perplexity
Transformer
353M
10B
1024
24
16.94
DAT
343M
10B
1024
24
8
8
64
16.09
Transformer
757M
10B
1536
24
24
14.65
DAT
734M
10B
1536
24
12
12
64
6
14.31
Transformer
1.31
10B
2048
24
32
13.63
DAT
1.27B
10B
2048
24
16
16
128
8
13.43
350M Scale
750M Scale
1.3B Scale
3.5
3.3
3.3
Transformer 353M
Transformer 757M
Transformer 1.31B
3.4
DAT 343M
3.2
DAT - 734M
3.2
DA 1.27B
3.1
3.2
3.0
Validation
3.1
2.9
2.6
2.7
2.5
18
10B
1B
10B
1B
10B
Tokens
Tokens
Tokens
Figure Validation loss on logarithmic scale examine data scaling laws. Dual Attention Transformer language models obey similar scaling laws standard Transformers with respect to the amount of training data, while consistently achieving smaller loss at multiple model scales.
24

---

Relational & Sensory Processing in Transformers
D. Comparison to Altabaa et al. [22]: Abstractors and Relational Cross-Attenticn
closely related work Altabaa et al. [22], which proposes Transformer-based module called the "Abstractor" with relational inductive biases. The core operation in the Abstractor is variant of attention dubbed "relational cross atttsntottttoon" (RCA). In this section, we will discuss the relation between the Dual Attention Transformer and the Abstractor.
D.1. Comparison between RA (this work) and RCA [22]
Altabaa et [22] propose a variant of attention called relational cross-attention which shares some characteristics with our proposal of what we're calling "relational attention" in this work. In this discussion, we will use the acronyms RCA and RA, respectively to distinguish between the two.
RCA processes sequence of objects æ (I), vn) and produces sequence of objects x' = -x.,,...,, x²)) via the following operation
x' c rell (0g(z)) SymbolRetricver(z)
where dg. 0% are query and key transformations, and the symbols s take the same role as in this work. Trel is referred to as a "relation activation". It may be either softmax or an element-win activation (e.g., tanh, sigmoid, or linear). For the purposes of this discussion, let us consider rel = Softmax, which was used the majority the experiments in [22].
To facilitate the discussion, let us write RA and RCA side-by-side using common notation.
RA (this work)
RCA [22]
(x(, ................. wnw - RA Sib),
(o/ onn)) t (a:(;; Smb)
x/ Σ an (5) W. + 83 W.), j=1
x Oij 8jz
j=1
= Softmax (x) (x)
a Softmax (x)))((()
r(x y) (x), (E()))'
(8) (81,..., 8p) = SymbolRRetriever(at Smb)
(s1,..., 8n) SymbolRettriever(::; Sii))
RCA can understood as self-attention, but the values are replaced with symbols (i.c., Attention(Q + x, K s)). By viewing the attention scores aij as relations, this has the effect of producing relation-centte -entricc representation. The rationale is that in standard self-attention, the attention scores form a type of relation, but these relations are only used as an intermediate processing step in an information-retrievw operation. The relations encoded in the attention scores entangled with the object-level features, which have much greater variability. This thinking also motivates the design of RA in the present work.
RCA can be understood as computing a pairwise relation (dgttm (r:)), oe dett" (xx,)) between d; and each 2. in the context, and retrieving the symbol sj associated with the object æj with which the relation is strongest. That is, RCA treats the relations and the attention scores as the same thing. By contrast, the attention operation and computation of relations are separate in RA. The attention component is modeled by one set of query/key maps of o and the relation component is modeled by another set of query/key maps (dre)) drel
gatta
q,8: (
The intuitive reason for this choice is that, for many tasks, the optimal "selection criterion" will be different from the task-relevant relation. For example, in a language modeling task, you may want to attend objects the basis of proximity and/or syntax while being interested in relation based on semantics. Similarly, in vision task, you may want to attend to objects on the basis of proximity, while computing a relation across certain visual attribute. Thus, the relational attention mechanism proposed in this work offers greater flexibility and expressivity compared to RCA.
In RA, the symbols maintain the role identifying the source. But they are now explicitly attached to a separately parameterized relation vector.
25

---

Relational & Sensory Processing in Transformers
D.2. Comparison between DAT and the Abstractor
We now briefly discuss the differences in the corresponding model architectures. Altabaa et al. [22] propose an encoder- like module called the Abstractor which consists of essentially replacing self-attention in Encoder with relational cross-attention. That is, it consists of iteratively performing RCA followed by an MLP. The paper proposes sev- eral ways to incorporate this into the broader Transformer architecture. For example, some of the experiments use Encoder + Abstractor + Decoder architecture to perform a sequenne-to-ssqqueee task. Here, the output of a standard Transformer Encoder is fed into an Abstractor, and the Decoder cross-attends to the output of the Abstractor, In another sequence-to-seqqeee
experiment, Altabaa et al. [22] use an architecture where the Decoder cross-attends to both the Encoder and the Abstractor, making use of both sensory and relational information. In particular, the standard encoder and decoder blocks are the same (focusing sensory information), but additional module is inserted in between with relational inductive bias.
By contrast, our approach in this paper is to propose novel encoder and decoder architectures imbued with two distinct types of attention heads, one with an inductive bias for sensory information and the other with an inductive bias for relational information. This has several potential advantages. The first is versatility and generality. The Abstractor architectures that were explored in [22] only explicitly support sequencce-t-sequeeee
or discriminative tasks. For example, they do not support autoregressive models like modern decoder-only language models (e-g., of the form we experiment with in Section 4.4), tasks, Abstractor architectures only support relational processing sequence, but they do not support relational processing over the target sequence (since the decoder does not have RCA), Another potential advantage of DAT is simplicity. The Abstractor paper proposes several architectures and configurations for the Saaadd///bbsractr//eoooooo
Moreover, even in sequence-to-sequeee
the input
modules, introducing several hyperparameters that are not trivial to choose. Moreover, is unclear how to interpret this kind of architecture as the number of layers increases, and the original paper does not experiment with scaling up the number of layers. The final potential advantage increased expressivity. In DAT, the two types of attention heads exist side by side in each layer. This allows relational attention heads attend the output of the self-attention heads at the previous layer, and vice-versa. This yields broader representational capacity, and potentially more interesting behavior we scale the number of layers.
D.3. How would RCA perform in an DAT-style dual head-type architecture?
One question one might ask is: how would DAT'sttyle dual head-type architecture perform if used Altabaa al. [22]'s RCA instead of the RA head-type proposed in this work? We carried out a few ablation experiments to answer this question.
Figure 10 compares learning curves on the relational games benchmark between standard DAT (with RA-heads) and a version of DAT with Altabaa et al. [22]'s RCA heads. We find that the two models perform similarly, with most differences small enough to be within the margin of error. This figure depicts the configuration with asymmetric RA and positional symbols.
Figure 11 depicts the validation loss curves on small-scale language modeling experiment based on the Tiny Stories dataset [71], comparing standard DAT against version with RCA heads. Here, we find that our relational attention heads yield setter-performing models, with the RCA-head variant of DAT performing no better than standard Transformer with a matching total number of heads.
26

---

Relational & Sensory Processing in Transformers
same
occurs
xoccurs
Generalization Accuracy
0.8
0.8
0.8
0.6
0.6
0.6
500
1000
1500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Training Set Size
Training Set Size
Training Set Size
between
match pattern
Generalization Accuracy
Model
0.8
Dual-Attn Transformer (rd) 1, 1,7g
(x4) 0, 0,74
Relational Attention
0.6
Abstractor's RCA
500
1000
1500
2000
2500
5000
10000
13000
20000
25000
Training Set Size
Training Set Size
Figure 11. Learning curves for DAT with RA compared with DA7 with RCA on the relational games benchmark. The performance is similar, most differences within margin of error.
2.2
Transformer (n 6, 2)
Dual-Atin Transformer (16 m,
RA Type
Disentangled Relational Attention
Validation Loss
Abstractor's RCA
2.0
1.9
1.8
Tokens
X10
Figure 12. Ablation of relational attention type. The solid line depicts the form of relational attention proposed in this work. The dotted line depicts RCA as proposed by Altabaa al. [22]. We find that our relational attention mechanism performs better, whereas RCA performs better than Transformer.
27


Reviews:
[{"review_id": "bvzuawYstB", "reviewer": "Reviewer_oPuj", "summary": "The authors introduce a modification/extension of the classical attention mechanism they term ‘dual attention’, which not only routes sensory information (as in classic SA) but adds a dedicated pathway to exchange relational information between tokens using its unique attention matrix – allowing for information flow which differs from the sensory one.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive review. We appreciate your positive feedback on the originality, significance, and clarity of our work. We are especially grateful for the time and effort you took to thoroughly engage with our work, reading the appendix and making note of typos. We'd also like to thank you for your specific and constructive feedback, which we believe has helped us improve the paper further. Below, we outline the key concerns you raised and attempt to address them.\n\n---\n\n**A1: Discussion & Experimental Comparison to Related Work on Relational Architectures (in main body of paper)**\n\n> Experimental analyses are often performed well but not necessarily contrasted to related methods – this has been deferred to the appendix, but might be better placed in the main paper for visibility\n\n> The similarity to [22] is discussed in detail in the appendix, but should be indicated much earlier\n\nWe agree that a more detailed discussion of related work in the main text would enhance clarity. Accordingly, we will use the additional page allowance to:\n- Integrate the experimental comparison to previous relational architectures [20,21,22], currently in Appendix C, into the main text.\n- Expand the discussion of Altabaa et al. [22], currently in Appendix D, to Sections 2.2 and 2.3.\n\n---\n\n**A2: Suitability of CIFAR for Evaluating Relational Processing in Vision**\n\n> CIFAR is very object-centric and a single-object dataset\n\n> I’d like to know why the authors think that Image classification on CIFAR benefits from the relational modelling. It clearly does, as we can see in the results\n\nAlthough CIFAR datasets contain only one object per image, we believe relational modeling is valuable at multiple levels of abstraction, from local patches to object parts and entire objects. The explicit relational mechanisms in our architecture enable the model to process and reason about visual relationships between object parts and image patches. For instance, this allows the model to detect symmetries or represent visual similarities between object parts across different regions of the image.\n\n> It can be enough to look at one or very few tokens of a CIFAR image and directly tell what the class would be.\n\nIn our models, images are divided into small 4x4-pixel patches (64 tokens per image), making it unlikely that a single token would suffice for classification. Since individual tokens represent very small regions at early layers, the models must consider information from several tokens/patches, including the relationships between tokens. At those early layers, the relations visually compare different patches, which can perhaps be thought of as analogous to applying one patch as a \"kernel\" or \"filter\" to another patch in the image. At later layers, tokens may come to represent more global higher-level features, and the relations can represent higher-level relations between object parts. Indeed, the improved results we observe for the *DAT* architecture demonstrate the utility of the enhanced relational processing capabilities of our architecture, even for the simple CIFAR benchmark.\n\nThat said, we agree that more complex tasks (e.g., multi-object detection, tracking, semantic segmentation) would better showcase our architecture’s capabilities. We will add a discussion on this limitation and potential future directions.\n\n> Have the authors visualised what relations are modelled? If not, is this possible and could be included? Do they represent any ‘expected’/intuitive relations (e.g. parts of the object)?\n\nFollowing your suggestion, we visualized the relations learned by the *ViDAT* model. We find that some relations do appear to represent intuitive \"visual similarity\" relations between object parts.\n\nTo illustrate this, we provide an example of a visualization of the learned relations on an image of a truck at [Layer 0](https://postimg.cc/PNC0fdLX) and [Layer 4](https://postimg.cc/gXQSL6wY). The patch labeled \"source\" represents the reference token, and the value annotations indicate sigmoid-normalized relation activations $r_{ij}[\\ell]$. The relation activations appear to be high for object parts that are visually similar, especially at earlier layers.\n\nWe will add a discussion of these findings in the revised paper.\n\n---\n\n**A3: Questions**\n\n> Are these subspaces particularly chosen? And if yes, how?\n\nThe 'feature subspaces' are not predefined; they are learned via $W_{q,\\ell}^{rel}, W_{k, \\ell}^{rel}$ during training. These are separate from the $W_{q,h}^{attn}, W_{k,h}^{attn}$ weights, which specify the selection criterion of the attention operation.\n\n> How many comparisons are performed between two tokens?\n\nThis is a hyperparameter of the model, denoted $d_r$. For example, in the 1.3B-parameter language model, $d_r = 128$.\n\nWe will revise the main text to make these points clearer.\n\n---\n\nThank you again for your thoughtful review and your helpful feedback.", "claims_and_evidence": "All major claims are in my opinion well justified.    \nThe authors do a good job in articulating and introducing their angle on the problem step by step, starting from the well-known Transformer self-attention mechanism. The relational information is explicitly computed (i.e. inductive bias) and added to a symbolic identity, hence justifying the claim that relational information is exchanged. (Although, actual insights into what exactly is exchanged would enhance the paper – currently placed in limited form in the appendix.)", "methods_and_evaluation_criteria": "The authors evaluate their method on four different tasks with corresponding evaluation criteria (acc for vision, perplexity for language, etc.);   \nThe selection of the vision task could in my opinion be significantly improved – Image classification on CIFAR seems rather ill-suited to show the power of relational processing:   \n- CIFAR is very object-centric and a single-object dataset, hence computation of relations between tokens might be rather straight-forward (around center of image) -- BUT more importantly:  \n- Image classification: It can be enough to look at one or very few tokens of a CIFAR image and directly tell what the class would be; Hence, this eval seems ill-suited to me.\n\n$\\textrightarrow$ There is a variety of other vision tasks where the benefit of relations between parts might be much more intuitive, e.g. multi-object detection, tracking, semantic segmentation, just to name a few.", "theoretical_claims": "I have checked the formulas and algorithms, and briefly read through the supporting evidence for Theorem 1 (appendix) – but couldn’t spot any obvious issues;", "experimental_designs_or_analyses": "As previously mentioned, I think the visual experiments could be significantly improved by choosing a task that requires relational modelling in a more obvious way that would be much more intuitive to the reader (e.g. multi-object detection, semantic segmentation, etc.); \nIt is unclear to me if an instance-based task like classification would benefit from this, as one token might already be enough to determine the correct class label. \n\nAlso: Experimental analyses are often performed well but not necessarily contrasted to related methods – this has been deferred to the appendix, but might be better placed in the main paper for visibility and to provide the reader with appropriate context.", "relation_to_broader_scientific_literature": "Relation to relevant literature can and should be improved – A very important relationship to the work of Altabaa et al. [22] is discussed in the appendix in detail, but the main body of the paper severely lacks in terms of discussing and attributing the similarity; \nWhile indeed different, both method are (in terms of underlying idea, choice of Transformer architecture and modified attention mechanism) very closely related – which should be appropriately indicated already in Sections 2.2 and 2.3.", "essential_references_not_discussed": "None that come to mind – but not extremely up-to-date in this particular area.", "other_strengths_and_weaknesses": "**Strengths:**  \n*Originality & Significance:*  \n- Clear motivation and step-by-step intro of the method based on a known shortcoming of a missing dedicated modelling mechanism of token-relationships within a Transformer architecture, addressing a known but important gap   \n- Proposed method applicable to range of modalities due to the choice of a Transformer architecture and the preservation of its generality (in terms of attention) \n- Authors demonstrate the applicability via a range of experiments across different tasks/modalities\n\n*Clarity:*  \n- Explanations well-supported through a good mix of figures, algorithms and formulas  \n- The paper is well written and easy to read and follow; several details moved to the appendix, but the paper provides a good level of depth to easily follow   \n\n---  \n\n**Weaknesses:**  \n- Discussion of & comparison to related works is lacking in parts of the main text: The similarity to [22] is discussed in detail in the appendix, but should be indicated much earlier and in a clearer manner in Sections 2.2 and 2.3; Similarity, almost all experiments exclusively compare default Transformers with DAT – although for some, there are related works available that might even outperform (see appendix Figure 6); Could be discussed in the respective section to be ‘up-front’ with the reader (I don’t expect the method to outperform task-specialist-methods, but a comparison to these works (even if treated in a class of their own) would in my opinion help the reader to better place the proposed method’s strengths \n- Experiments on the visual task, i.e. classification on CIFAR, seems rather ill-chosen to support a claim of modelling relations; see previous comments & questions-section\n- Details on the ‘subspace’ comparison for relationship computation, i.e. l \\in Rd could be extended -- see questions.", "other_comments_or_suggestions": "Typos:    \n- L 177 right: to both computational mechanismS (plural)\n- L 354 left: it is useful TO consider.. (to missing)", "questions_for_authors": "1. I’d like the authors to provide some more details and insights into the explicit relation computation between the feature maps: The authors state this operation is performed with “$l \\in [d_r]$”, and “for each $l \\in [d_r]$, the feature maps ..” (l151 f) – producing a relation vector “across different features subspaces”.   \n$\\textrightarrow$ Are these subspaces particularly chosen? And if yes, how?   \n$\\textrightarrow$ How many comparisons are performed between two tokens?  \nAs this is one of the main components of this approach (and differences to [22]), this aspect could be made a lot clearer to the reader.\n\n2. Comments on the visual task: I’d like to know why the authors think that Image classification on CIFAR benefits from the relational modelling. It clearly does, as we can see in the results, but as I’ve mentioned previously:  One token might already be enough to solve the entire task of classifying an image – so this choice doesn’t particularly feel well suited.    \n$\\textrightarrow$  Have the authors visualised what relations are modelled? If not, is this possible and could be included?  Do they represent any ‘expected’/intuitive relations (e.g. parts of the object)?  \n$\\textrightarrow$  The visualisation provided for the language task in Figure 8 (appendix) is quite interesting; so sth similar for the image task would be a good addition;\n\n3. Follow-up:  \n$\\textrightarrow$  Have the authors thought about evaluating their model on an alternative vision task that more intuitively require modelling of relations, e.g. multi-object detection or semantic segmentation?  Why/why now?\n\n*TLDR;* I think the paper presents an interesting approach, although some aspects could be improved (as detailled before); Depending on the response, I’m happy to consider further increasing my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u8Rf2IKnsT", "reviewer": "Reviewer_h4pW", "summary": "The authors propose a parameter-efficient variant of the self-attention mechanism in transformers called the Dual-Attention Transformer (DAT), which explicitly routes both sensory information (about individual tokens) and relational information (about relationships between pairs of tokens). The key differences between relational attention and standard attention are that 1) instead of routing a value projection from the query, a vector of relational similarities is constructed (by concatenating the dot products of multiple learnable relation projections between query and key), and 2) A symbol vector (from a learned codebook), retrieved for each key, is added to the corresponding relation vector, and the result is then routed with the usual attention weight.\n\nThe authors then incorporate both standard and relational attention heads in a multi-headed attention framework to construct DAT models and demonstrate their effectiveness across various domains - from explicitly relational tasks in the RelationsGame to language modeling, mathematical problem-solving, and image recognition tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response\n\nThank you for your detailed and thoughtful review. We appreciate your positive assessment of our work and are encouraged by your recognition of its methodological soundness, strong empirical results, and relevance to the literature on relational inductive biases and Transformer-based architectures. Below, we will aim to respond to the key comments and concerns you raised.\n\n---\n\n**B1: Computational efficiency**\n\n> A potential downside of DAT even within the experiments carried out is the possibility that the additional computations required for RelationalAttention outweigh the gains from parameter efficiency. Though differences are likely marginal, was this something you investigated?\n\nThank you for raising this important point. Our experiments were designed to carefully control for model size (i.e., parameter count) when comparing the *DAT* architecture to baselines. In particular, the parameter count is slightly *smaller* for the *DAT* model compared to the baselines. While we did not explicitly measure computational cost in terms of FLOPS, we expect the differences to be marginal.\n\nFrom a practical standpoint, we believe that the more important factor in computational efficiency is the availability of optimized GPU kernels, such as FlashAttention. This gap could perhaps be bridged given interest from the MLSys community to develop optimized kernels for relational attention, but for now, this would be an obstacle to scaling *DAT*-style architectures in a cost-effective way.\n\n---\n\n**B2: Scaling to Larger Models**\n\n> One potential limitation not fully addressed is whether the performance gains from RelationalAttention would still be significant at very large scales, or if they might be incompatible with optimization tricks for self-attention. However, the consistent improvements across the scales explored are promising and warrant further exploration.\n\n> While large-scale demonstrations aren't necessary for this paper, do you have intuitions about how the advantages of DAT might scale?\n\nOur intuition, guided by our scaling results up to ~1B-parameter scales, is that the explicit relational processing capabilities of *DAT* will continue to provide significant benefits at even larger scales. Our hypothesis is that relational processing is a key computational capability that is useful in many domains and at several levels of abstraction---explicit support for this in the model architecture can enable more efficient learning and greater generalization capabilities.\n\nOf course, confirming this hypothesis requires empirical validation, which we currently do not have the resources to do at our academic institution. As you rightly mentioned, and as discussed above, scaling further introduces new challenges, and the availability of various optimization tricks becomes an important consideration. We hope future work will explore this further.\n\n---\n\nThank you again for your thoughtful review and your engagement with our work.", "claims_and_evidence": "The paper makes two primary claims that are well-supported by the evidence:\n1. DAT outperforms standard transformers with multi-headed attention - this is demonstrated convincingly across multiple domains and tasks\n2. The relational attention mechanism better routes relational information than standard self-attention - this is most clearly shown on explicitly relational tasks from the RelationsGame", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for the demonstrations. The authors evaluate on a diverse set of tasks spanning different domains (visual reasoning, symbolic math problems, language modeling, image classification) and compare against comparable transformer baselines (or relational baselines) while controlling for parameter count.", "theoretical_claims": "A theoretical claim is made about the class of functions that can be computed by relational attention. A proof is provided in the appendices and was not checked. The claim is not particularly strong and the informal proof in the main text seems sufficient.", "experimental_designs_or_analyses": "The experiments and ablations are thorough, well-reported, and compelling. The learning curves on across most tasks show clear data efficiency advantages of DAT over standard transformers. Furthermore, the additional results in the supplementary material are comprenehsive.", "relation_to_broader_scientific_literature": "The paper's findings are moderately significant and relevant to current research on improving transformer architectures. The authors position their work appropriately within the literature on relational inductive biases and transformer models.\n\nOne potential limitation not fully addressed is whether the performance gains from RelationalAttention would still be significant at very large scales, or if they might be incompatible with optimization tricks for self-attention. However, the consistent improvements across the scales explored are promising and warrant further exploration.", "essential_references_not_discussed": "None of which I am aware", "other_strengths_and_weaknesses": "None of note", "other_comments_or_suggestions": "* The ICML Style guide requires citations within the text to include authors' last names and year, which this paper doesn't follow\n* Line 161 contains an unnecessary sentence fragment \"this adds structures...\"", "questions_for_authors": "1. A potential downside of DAT even within the experiments carried out is the possibility that the additional computations required for RelationalAttention outweigh the gains from parameter efficiency. Though differences are likely marginal, was this something you investigated?\n2. For very large transformers trained on extensive data, standard self-attention may eventually capture relational information in complex tasks while retaining greater efficacy for other common computations - potentially narrowing the performance gap at increasing scale. While large-scale demonstrations aren't necessary for this paper, do you have intuitions about how the advantages of DAT might scale?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RMczIkpO4y", "reviewer": "Reviewer_yBiU", "summary": "This paper presents the Dual Attention Transformer (DAT), an extension of the Transformer architecture that introduces a relational attention mechanism alongside the standard self-attention mechanism. The key idea is to explicitly represent and process relational information by replacing the standard value aggregation in self-attention with a weighted combination of a relation vector computed for each pair of objects and a symbol vector.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and positive feedback on the overall contribution of our work, especially for highlighting the empirical support for our claims, the contribution to relational reasoning in Transformer architectures, and the importance of structured reasoning in deep learning models. Below, we hope to address the key comments and concerns you raised.\n\n---\n\n**C1: Comparison with Baselines**\n\nWe would like to highlight Appendix C (specifically Section C.1) and Appendix D, where we compare our proposed method to several prior works on relational learning. In particular, we compare our model to PrediNet (Shanahan et al. 2020), CoRelNet (Kerg et al. 2022), and the Abstractor (Altabaa et al. 2024), positioning our work within this ongoing line of research on relational reasoning and extending these prior efforts.\n\nOne way to describe the architectures in these prior works is that they incorporate *\"subtractive\"* inductive biases that constrain the types of representations the model can compute (see [Ref 24] on the \"Relational Bottleneck\" for more on this). These strict biases enable strong performance on the benchmarks they were designed for (e.g.,  the Relational Games benchmark introduced by Shanahan et al. (2020)), but also constrain the models to a narrow domain of applicability. By contrast, our approach in developing the *Dual Attention Transformer* architecture is *\"additive\"*, in the sense that it incorporates new explicit relational processing capabilities without constraining existing components of the Transformer architecture, allowing the model to learn to select between the different computational mechanisms available to it based on the task or context, as well as compose them to create flexible and expressive computational circuits.\n\nDespite these differences, we find it valuable to compare *DAT* against those architectures on controlled synthetic benchmarks to explore the trade-offs of strong inductive biases and evaluate *DAT* against alternative approaches to relational learning. This was carried out and discussed in Appendix C.1.\n\nInitially, due to space constraints, we deferred this discussion to the appendix. However, with the additional page allowance, we plan to integrate this more detailed comparison and discussion of related works into the main body of the paper. Please also see our response numbered **A1** to reviewer oPuj, where we discuss a similar question.\n\n---\n\n**C2: Conceptual similarities to message-passing networks**\n\nWe view the aforementioned line of work on relational learning [Ref. 19, 20, 21, 22, 23] to be the most closely related literature to our work. However, we agree that there are some conceptual similarities between our *DAT* architecture and message-passing networks, which is a characteristic shared by Transformer models in general. In particular, like standard Transformers, the *DAT* architecture can be described in the language of the message-passing framework. In message-passing terminology, the messages exchanged in standard Transformer attention encode first-order sensory features of the sender, while in *relational attention*, the messages encode relational features between the sender and the receiver.\n\nWe will additionally incorporate a discussion on the conceptual connections between the *DAT* architecture and message-passing networks into the paper.\n\n---\n\nThank you again for your review. We hope we were able to address your main concerns.", "claims_and_evidence": "The paper argues that standard Transformers primarily process sensory information and struggle with relational reasoning due to the entanglement of sensory and relational information. The proposed DAT disentangles these components, leading to improved performance. The empirical results largely support these claims.", "methods_and_evaluation_criteria": "The authors employ standard benchmarks to assess the effectiveness of DAT, comparing it against standard Transformer models across various tasks.", "theoretical_claims": "This paper primarily focuses on the empirical aspects.", "experimental_designs_or_analyses": "It might be more convincing if more baseline comparisons with graph-based models and message-passing networks are included, given the conceptual similarities.", "relation_to_broader_scientific_literature": "The paper presents contribution to the development of relational reasoning in Transformer architectures, with potential implications across multiple domains, including language processing and vision. By introducing an explicit mechanism for processing relational information, the proposed approach highlights the importance of integrating structured reasoning capabilities into deep learning models.", "essential_references_not_discussed": "It might be helpful if more key works in graph neural networks that have incorporated relational attention mechanisms were discussed and compared.", "other_strengths_and_weaknesses": "The strengths and weaknesses have been addressed in earlier sections, and no additional ones require emphasis here.", "other_comments_or_suggestions": "The comments and suggestions have been addressed in earlier sections, and no additional ones require emphasis here.", "questions_for_authors": "How does DAT perform on more complex benchmarks beyond the current experimental setup?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ubiRtnh9a", "reviewer": "Reviewer_pBHH", "summary": "The authors describe a neural architecture (DAT) in which relational information is a first-class object, and via a series of experiments show that this architecture offers genuine empirical benefits.\n\n## update after rebuttal: I have kept my \"accept\" score for this strong paper. There is no change since my rebuttal responses below, but I am told I need to add this note here as well.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and thoughtful review. We appreciate your positive feedback about the significance, methodological soundness, and strength of the empirical evaluation. Below, we hope to address the main concerns you raised in turn.\n\n**D1: Interpretability of Learned Relations**\n\n> Figure 8 in the appendix is intriguing, and interpretability work would probably be an entire new paper. I do want to flag one thing about the claim that \"Relational attention in DAT language models encodes human-interpretable semantic relations\" from the caption: in many cases, classic transformers, too, encode human-interpretable semantic relations. ... I wonder if there's an example of an attention head which finds some kind of relation that is not normally seen in a classic transformer?\n\nThank you for your thoughtful comments regarding the interpretability of learned relations. This is indeed an interesting and important question that we’ve begun to explore further. We are glad you found the preliminary exploration in Figure 8 intriguing. As you rightly point out, the relations observed in Figure 8 appear to encode relatively simple semantic similarity between words/tokens, which has also been observed in the attention scores of standard Transformer attention. However, it is important to note that these relations serve a different functional role here: while the attention scores $\\alpha_{ij}$ in standard Transformers encode a *selection criterion* controlling *where* information is routed, in *DAT*, the relation vectors $\\boldsymbol{r}_{ij}$ serve as the *values* being transmitted between tokens, controlling *what* information is routed (with an independent set of attention scores controlling the selection criterion).\n\nWe agree with you that finding a single interpretable relational attention head may not be very revealing on its own, and we would be excited to investigate whether relational heads in *DAT* might encode novel relationships that are not typically seen in classic Transformers. In particular, it would be interesting to understand how these relations form *computational circuits* that are unique to the DAT architecture---that is, characterize specific computational circuits that use relational heads in unique ways to carry out a specific computation, understanding their functional role on a deeper level beyond just the fact that the relations themselves appear human-interpretable.\n\nFor now, this interpretability work is outside the scope of this paper, but we intend to explore it further in future work. One initial step we’ve taken is developing an interactive tool (accessible online) that allows users to load pre-trained DAT models and visualize relational representations at various layers on their own inputs. We plan to include a link to this tool in the final, de-anonymized version of the paper.\n\n---\n\n**D2: Question on proof of Representational Capacity Theorem.**\n\n> The theorem in Appendix A seems correct, although the Debreu representation theorem was new to me ... I'd recommend that the authors provide a paragraph outlining the result at a high level\n\nThank you for the question and the suggestion to provide further discussion on the Debreu representation theorem. We will add a high-level overview of the Debreu representation theorem and the related literature to improve the clarity and accessibility of the representation result presented in Appendix A\n\nYes, your interpretation of the result and its proof are correct. The Debreu representation theorem, due to the economics literature, identifies preference relations on a topological (e.g., metric) space with a continuous \"utility\" function, assuming certain continuity properties of the preference relations with respect to the underlying topology. For us, the key is to extend this idea to a *family* of query-dependent preference relations in order to specify the attention mechanism. That is, each query is associated with an ordered space, and we require continuity of the family of preference relations with respect to both queries and keys, which we formulate as query-continuity and key-continuity, respectively. From there, the result follows by the approximation properties of inner products of MLPs.\n\n**D3: Further discussion of related work**\n\n> If the authors want to expand a bit, there are two areas where I don't see citations; however, they may not really be essential. One is recent work on how transfomers do seem to represent relations, e.g. \"How do Language Models Bind Entities in Context?\" (Feng & Steinhardt) and papers that cite it. Another is theoretical ways that have been proposed for representing relations (vector symbolic architecture, etc.)\n\nThank you for these suggestions. We will incorporate them into our discussion of related work in the final version of the paper.\n\n---\n\nThank you again for your engagement with our work!", "claims_and_evidence": "The claims seem solid. The question with this type of paper is often less about the solidity of the claims, however, but more about their significance. To be concrete, why do we need yet another transformer-like architecture? In this case, I think the new architecture meets the bar of significance for publication. First, it can be seen as helping make explicit the type of computation that \"classic\" transformers are learning implicitly. Second, it appears that for certain tasks DAT actually outperforms transformers. Third, it opens up interesting possibilities for interpretability work, potentially making neural systems more transparent.", "methods_and_evaluation_criteria": "The paper has a good balance of methods, analyzing the architecture theoretically as well as performing experiments.", "theoretical_claims": "The theorem in Appendix A seems correct, although the Debreu representation theorem was new to me. Once I knew what that was, the result seemed correct. I'd recommend that the authors provide a paragraph outlining the result at a high level, which I read as: selection can be modeled with a preference ordering (with some mild conditions); the Debreu representation theorem allows us to think of computing such a preference ordering as computing a certain continuous function; we then just use standard results saying neural nets can approximate continuous functions. However, I'd note to the authors: if this outline is NOT what the proof is actually saying, then please clarify!", "experimental_designs_or_analyses": "The designs seem solid, and I appreciate that there are both language and vision tasks, which strengthens the overall claim.", "relation_to_broader_scientific_literature": "The question of whether and how transformers learn relationships is definitely central, so this seems squarely in the mainstream. See next section for more detail.", "essential_references_not_discussed": "I think the related work cited is fine. If the authors want to expand a bit, there are two areas where I don't see citations; however, they may not really be essential. One is recent work on how transfomers do seem to represent relations, e.g. \"How do Language Models Bind Entities in Context?\" (Feng & Steinhardt) and papers that cite it. Another is theoretical ways that have been proposed for representing relations (vector symbolic architecture, etc.)", "other_strengths_and_weaknesses": "Figure 8 in the appendix is intriguing, and interpretability work would probably be an entire new paper. I do want to flag one thing about the claim that \"Relational attention in DAT language models encodes human-interpretable semantic relations\" from the caption: in many cases, classic transformers, too, encode human-interpretable semantic relations. (And here the relation isn't particularly striking: it just seems like it's picking out generally related words?) The fact that one can find a single example of an attention head that is interpretable isn't particularly interesting or useful in itself. I wonder if there's an example of an attention head which finds some kind of relation that is not normally seen in a classic transformer?", "other_comments_or_suggestions": "I'm not fond of the term \"sensory\" here, because it seems actively misleading. There's already a common metaphor for deep networks where people talk about the early layers as doing sensory processing, and the final layers as being analogous to motor neurons.\n\nI'd recommend instead calling this something like \"first-order\" vs. \"relational\" information.", "questions_for_authors": "none, except for the note about confirming my read of the proof, mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Dual Attention Transformers, which modifies the standard attention mechanism to not only route sensory information (as in standard SA) but includes a dedicated pathway to exchange relational information between tokens. 

This paper received mostly positive reviews, and reviewers agree that the results/claims are well supported. Reviewers laud the quality of the work and in particular the explanation of the proposed mechanism, the diversity of the experiments (both on vision and language), and the comparison to relevant baselines. A few remaining concerns (though not all of them) were well addressed during the rebuttal.  One remaining concern of note is that image classification might not sufficiently test the model's ability to perform relational reasoning token space, even though improved performance is observed. However, given the other experiments and comparisons, this is not a major concern. All in all the reviewers agree this work presents enough of a contribution to be published at ICML and the AC concurs.
Decision: Accept (poster)