Attention-based Iterative Decomposition for Tensor Product Representation
Authors: Taewon Park
Inchul Choi
Minho Lee
Venue: ICLR 2024 poster (2024)
Keywords: tensor product representation
systematic generalization
compositional generalization
binding problem
structured representation learning
competitive attention
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Slot-based competitive mechanism that effectively binds sequential features to the structured representations (roles and fillers) of TPR

Abstract: In recent research, Tensor Product Representation (TPR) is applied for the systematic generalization task of deep neural networks by learning the compositional structure of data. However, such prior works show limited performance in discovering and representing the symbolic structure from unseen test data because their decomposition to the structural representations was incomplete. In this work, we propose an Attention-based Iterative Decomposition (AID) module designed to enhance the decomposition operations for the structured representations encoded from the sequential input data with TPR. Our AID can be easily adapted to any TPR-based model and provides enhanced systematic decomposition through a competitive attention mechanism between input features and structured representations. In our experiments, AID shows effectiveness by significantly improving the performance of TPR-based prior works on the series of systematic generalization tasks. Moreover, in the quantitative and qualitative evaluations, AID produces more compositional and well-bound structural representations than other works.

Published as a conference paper at ICLR 2024
ArTENTION-BASED ITERATIVE DECOMPOSITION FOR TENSOR PRODUCT REPRESENTATION
Taewon Park', Inchul Choi?. Minho L
l "Kyungoook National University, South Korea 2 ALL Co., Ltd., South Korea
(ptw7998, sharpic77, mholle)@ggail..cm
ABSTRACT
In recent research, Tensor Product Representation (TPR) is applied for the sys- tematic generalization task of deep neural networks by learning the compositional structure of data. However, such prior works show limited performance covering and representing the symbolic structure from unseen test data because their decomposition the structural representations was incomplete. In this work, we propose Attention-based Iterative Decomposition (AID) module designed to enhance the decomposition operations for the structured representations en- coded from the sequential input data with TPR. Our AID can be casily adapted to any TPR-based model and provides enhanced systematic decomposition through a competitive attention mechanism between input features and structured represen- tations. In our experiments, AID shows effectiveness by significantly improving the performance of TPR--baeed prior works the series of systematic generaliza- tion tasks. Moreover, in the quantitative and qualitative evaluations, AID produces more compositional and well-bound structural representations than other works.!
dis-
INTRODUCTION
Humans can understand the compositional properties of the surrounding world and, based on their understanding, systematically generalize over unfamiliar things. This systematic generalization abil- ity is one of the main characteristics of human intelligence and also the central issue of deep neural network research. However, the systematic generalization performance of deep neural networks is still far from human-level generalization (Fodor & Pylyshyn, 1988; Lake & Baroni, 2018; Hupkes al., 2020; O O'Reilly al., 2022; Smolensky al., 2022). Therefore, to improve the generalization performance, researchers have integrated symbolic system methodologies, such as Tensor Product Representation (TPR) (Smolensky, 1990), into neural networks.
TPR is general method that explicitly encodes the symbolic structure of data with distributed rep- resentations. It is constituted by the tensor product of roles vectors and fillers vectors, where each encodes structural information and content of data. For decoding, it obtains symbol information from the embedded representation by applying TPR decoding components, unbinding operators. In basdd neuro-symbolic approaches, deep neural networks learn to extract TPR components (roles, fillers, and unbinding operators) from the input data while training for TPR functions. Re- cently, TPR-based neuro-symbolic approaches have showed significant improvements in the gener- alization and capacity of neural networks (Schlag & Schmidhuher, 2018; Schlag et al., 2020; 2021; Shi et al., 2022). However, find that these approaches still encounter challenges in achieving compositional generalization. This is likely attributable to their reliance on simple MLP for de- composition, which may not be structured to learn the compositional nature of data. Therefore, when the model fails to sufficiently leam the systematic decomposition of TPR, it inevitably de- grades whole TPR operations (Smolensky, 1990; Bae al., 2022).
In another approach to systematic generalization. the attention mechanism is used with neural net- works to capture the compositional properties and thus generate meaningful objects representations
"corresponding authors
'The code of AID is publicly available http://gitthb.co//awwoaaaa////

---

Published as a conference paper at ICLR 2024
Downstream Tasks
TPR Encoding
Role;
Competttive
Faed Plomvard
Roley
Role,
Feed Forward (Routing)
Roles
+
Rolez
Roiey
Roley @ Kole:
filler
filler
filler
TPR Companents
Initia
K
V
TPR Components
Intormediote
TPR Components
Reftned
Token-level
Irput
Residuel Connection
Routing
mery
got
the
mik
2eput Features
Figure 1: Overview of AID assisted TPR decomposition. We illustrate the overall operations of the AlD-assisted part of the TPR-baser model (with Ninpurs = and Noom = 3). In the natural language task, each input feature is word vector for the sentence-level TPR models, while input features are Nippus sub-vectors partitioned from the word vector for the word-level TPR models. From those input features, the initial TPR components obtained with linear projection, and routed to structural representation slots of AID which iteratively decomposes TPR components with competitive attention mechanism.
by selectively attending to relevant information (Goyal et al., 2019; Locatello et al., 2020). Such attention-basse mechanisms have improved the sample efficiency and systematic generalization of neural networks in various fields, such as computer vision (Locatello et al., 2020; Singh et al., 2022) and reinforcement learning (Goyal et al., 2019; Yoon al., 2023).
In this paper, to address the incomplete decomposition problem in TPR-based neural networks, we propose novel Attention- based Iterative Decomposition (AID) module that can effectively decompose sequential data into structured representations of TPR. The AID uses slot-based com- petitive attention (Goyal al., 2019; Locatello et al., 2020; Singh al., 2022) bind sequential features to abstract structured representations (roles and fillers) of TPR, as shown in Fig. 1. While AID adopts an iterative competitive attention method similar to Slot Attention (Locatello et al., 2020), its distinctive contribution lies in its routing mechanism for the correct decomposition in the TPR framework. The TPR framework relies on pre-defined structural components, roles and fillers, to represent the underlying symbolic structure of data. To seamlessly integrate slot-based attention with the TPR framework, AID systematically routes individual structural components of TPR to specific slot and refines the structured representation through competitive attention. Fur- thermore, AID conditionally initializes each slot component based on the context-dependent input features, instead of randomly initialized values in other slot-attention approaches. This learnable routing mechanism of AID provides a differentiable connection between input features and TPR components while learning to bind structured representations from input features in an end-to-end manner. AID can easily enhance the decomposition mechanism in any TPR- model without introducing much computational overhead because of its parallel operations. Therefore, our method provides a simple and efficient way to enhance systematic generalization in any TaR-based models.
In experiments, apply AID to several recent TpR-based TPR equivalent approaches show its effectiveness and flexibility. We adopt three types of Tel-related models for our study: (a) TPR- RNN (Schlag & Schmidhuber, 2018), (b) Fast Weight Memory (FWM) (Schlag al., 2020), and (c) Linear Transformer (Katharopoulos et al., 2020). For comparison, we evaluate the AID-assisted version of all models with the baseline on the synthetic systematic generalization tasks, text/visual question-answering tasks, and even large-scale language modeling tasks. In all experimental results, AID shows effective generalization performance improvement for all TPR models.
Our main contributions are follows: (i) we propose a novel AID module that can easily enhance the systematic generalization performance of any TaR-based models. (ii) we show that an itera- tive attention with routing mechanism effectively enhances structural representations of TPR (roles, fillers, and unbinindg operators) for unseen compositions in a systematic way. (iii) by improving the decomposition performance, we enable the application of TPR for large-scale real-world data.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Binding problem The binding problem. discussed by Greff et al. (2020), represents one of the neural network's abilities flexibly and dynamically bind information from data to compositional structures. This problem comprises decomposing data into representations of meaningful entities and preserving their separation at the representational level facilitate compositional reasoning (Bengio et al., 2013; Lake & Baroni, 2018; Goyal et al., 2019; Webb et al., 2020; Madan et al., 2021; Goyal & Bengio, 2022). TPR one of the approaches for representational separation by explicitly binding roles and fillers representations using a tensor product. With TPR, this repre- sentational separation enhances the ability of neural networks to process the symbolic structure of data (Schlag & Schmidhuber, 2018; Schlag et al., 2019; 2020; Jiang et al., 2021; Shi et al., 2022). However, despite the binding capability of TPR, such TPR-based neuro-symbolie models do ex- plicitly learn the decomposition operations of TPR. Therefore, if they fail to decompose appropriate structural representations from unseen input data, it is likely to degrade all of the TPR functions of the network. To the best of our knowledge, our work represents the first attempt address the decomposition problem in TPR-based neural networks.
Compositional Generalization The compositional generalization, also known as systematic gen- eralization, has been explored in neural networks to enable them to generalize beyond what they learned (Fodor& & Pylyshyn, 1988; Lake al., 2017: Lake & Baroni, 2018; Liška al., 2018; Hupkes et al., 2020; Webb et al., 2020). Recent research on compositional generalization has incorporated inductive biases, including attention mechanisms, into neural networks capture self-contained. reusable, and disentangled representations (Goyal et al., 2019; Locatello et al., 2020; Mittal et al., 2021: Csordás al., 2021; Madan al., 2021; Singh al., 2022). Among these approaches, Lo- catello et al. (2020) introduces an iterative attention-based mechanism called Slot Attention, de- signed to discover object representations within visual scenes. In contrast to Slot Attention, which specific
assumes the permutation-invarianl slots, our AID systematically routes each component structural component of TPR, such as role and filler. This routing strategy enables the generated rep- resentations to be utilized in TPR functions. In another line of research, Schlag et al. (2019); Jiang et al. (2021) explore the integration between attention and TPR, akin to our approach. However, they are designed for specific tasks such math problems abstractive summarization. In addi- tion, Jiang et al. (2021) relies on a pre-defined role embedding dictionary. In contrast, the AID is a task-independent drop-in module that can be adapted to any TPR-based model. Also, it is designed to address more fundamental problem, the decomposition of data into the appropriate role and filler simultancously.
Fast Weight Programmer Our work is closely related to the field of Fast Weight Programmers (FWPs) (Schmidhuber, 1992; 1993). The concept of FWPs involves the construction of context- independent slow weights, which control context-dependent fast weights (Von Der Malsburg, 1994). During the training process, the slow weights are trained to program the context of input data ef- fectively into the fast weights through back-propagation. This foundational concept has been exten- sively researched in prior work (Ba et al., 2016a; Schlag Schmidhuber, 2017: 2018; Munkhdalai et al., 2019; Schlag et al., 2020; 2021; Irie et al., 2021). In FWPs, TPR is considered a high-order form of fast weight. In this context, our work is an extension of FWPs. In contrast to prior work, we focus extracting fast weight representations from unseen data rather than designing updating rules for fast weight.
3 METHOD
In this section, we illustrate how the AID module decomposes input data into TPR components with iterative attention. Also, we detail the overall integration of the AID with pre-existing TPR-based models and show how it assists them.
3.1 TENSOR PRODUCT REPRESENTATION
We first briefly review TPR (Smolensky, 1990), which is baseline method for our approach. TPR is a general method that explicitly encodes the symbolic structure of the objects in vector spaces. It is formed by outer product between role and filler representations derived from the object.
3

---

Published as a conference paper at ICLR 2024
Then, the connectionist representations for each object are combined via summation to represent multiple objects. During the decoding process, the filler is unbound from TPR by matrix multipli- cation with the unbinding operator. For the accurate encoding and decoding of symbolic structures in data, TPR requires three key conditions: (1) roles must be linearly independent of each other to prevent the overlap between fillers, (2) unbinding operators must exhibit high correlation with the corresponding roles to access associated fillers, and (3) fillers encompass object information utilized for downstream tasks. These requirements inherently highlight the significance of decom- posing roleffiller representations from the input data to perform TPR functions (the conventional TPR relies on prior knowledge for role/filler decomposition). The outer product form of roles and fillers provides representational separation, which is necessary for addressing the binding problem (Greff al., 2020). Based on this property, recent TPR- -baee models can enhance the generaliza- tion and interpretation of neural networks (Schlag & Schmidhuber, 2018; Schlag et al., 2019; 2020; Jiang et al., 2021; Shi et al., 2022). However, even in the framework of TPR, the representational separation is built upon the sufficiently learned role/filler decomposition in the neural network. Fur- thermore, our study finds that most TPR-based network models have difficulty when decomposing roleffiller representations from unseen data. To address this problem, introduce a slot attention- based iterative process (Locatello et al., 2020) for decomposition, which allows better generalization when dealing with unseen compositions of objects.
3.2 ATTENTION-BASEE ITERATIVE DECOMPOSITION MODULE
AID is an iterative attention-based decomposition mechanism designed for TPR enhancement. At every time step, it takes input features and generates structured representations for TPR with com- petitive attention among intermediate TPR components, as illustrated in Algorithm 1. During the decomposition process, the AID can refine the structured representations with attention as often as Nier iterations. This attention -baed competition mechanism enables AID to capture the composi- tional structure inherent in the data (Goyal et al., 2019; Locatello et al., 2020) and provides better generalization for unseen data.
For details, let us consider the AID module with single iteration at time step r. At each time step, the AID takes Ninpus input features and Noom TPR components, denoted as inputs E R. RR-aap x Duaus and
E R -.-ammDonn and maps them to common dimension of Doom using learnable parameters k, q, and v for competitive attention. Subsequently, the AID computes an attention score denoted as attn E IR Nupes x Nom through dot-product attention (Luong et al., 2015) between key and query. Based on attn, the AID applies a weighted mean to value to aggregate input information into componentss selectively, follows.
Ans
where A: key q T attny == EjeA,
E R Nupu Nem
(1)
attny
updates:: == W value e R Non x Den where Wi, == Mjttni
(2)
During the attention process, each component competes with others to discover input features that better explain the symbolic structure for TPR. The AID adds the aggregated features to components s for the update. According to Locatello al. (2020), we apply layer normalization (LayerNorm) (Ba et al., 2016b) and multi-layer perception (MLPupdate) of two layers with ReLU activation for the aggregated features. In the iterative process, the AID repeatedly refines those up- dated component s using one layer MLP final and the final components are used to perform TPR functions for downstream tasks.
Notably, in its pure form, the iterative attention mechanism produces connuutationinvvriinn components, posing challenge in directly linking components to elements in TPR func- tions (e.g., identifying the appropriate component for filler). We introduce a trainable routing mechanism to our iterative attention method to integrate slot-based attention correctly with the TPR framework. In this method, each initialleee srriialllooonnnttt is systematically linked to specific sym- bols. As illustrated in Fig. 1, for instance, the first component is mapped to rolel and the third to filler. Specifically, these initial.component work to the concatenated input features and are optimized to facilitate a systematic decomposition during training. Also, these context-dependent initial component.s significantly enhance the
are obtained by applying feed-forward net-
4

---

Published as a conference paper at ICLR 2024
training stability of the AID module. Furthermore, apply additional training techniques to im- prove performance, including incorporating a residual connection (He et al., 2016), concatenation with DropOut (Srivastava al., 2014), and non-linear activations for dot-product attention. The details of ablation studies for these techniques and comprehensive evaluation of their impact on the overall model's performance are illustrated in Appendix D.
3.3 TPR NETWORKS WITH AID
In this section, describe how the AID module is adapted other TPR-hased network models: (a) TPR-RNN (Schlag & Schmidhuber, 2018), (b) Fast Weight Memory (FWM) (Schlag et al., 2020), and (c) Linear Transformer (Katharopoulos al., 2020). In the integration process, apply the AID for both the encoding and decoding component generations with parameter sharing. Further- more, we maintain the original TPR functions of the baseline models; instead, we replace their decomposition mechanisms with our AID module to improve the effectiveness of the decomposi- tion.
3.3.1 TPR-RNN
TPR-RNN TPR-based memory network designed for sentence-level processing. This model aims to solve basic question-answering tasks (Weston et al., 2015). TPR-RNN takes a set of word vectors Ia {2] Iz Napu } at each time step and generates a sentence vector $ -
Nam x @
p', where p (p) .....,,' Napus) denotes learnalle position vectors and @ denotes element-wise product. Subsequently, TPR-RNN generates components based on the sentence vector st. In alignment with prior work, derive the
based the sentence vector st- The AID performs decomposition for 2; and initial s to generate components.
3.3.2 FWM
Fast Weight Memory (FWM) is TPR-based memory network for word-level processing. This algorithm is specifically designed to enhance the capacity for long sequential data. FWM takes an input vector and generates hidden state h, using internal LSTM (Hochreiter & Schmidhuber, 1997) at each time step. Subsequently, FWM generates components based on the hidden state ht- To incorporate the AID, form multiple input features using modular approaches akin to Henaff et al. (2016); Li et al. (2018). We transform I into i using a forwwfd network and partition into Ninptt sub-vectors, (..,,... Najas Follwing this transformation, the LSTT processes each af independently with shared parameters and generates multiple hidden states he = {h] .... h Munas }. In alignment with prior work, derive the initial components based on the concatenated hidden state. The AID performs decomposition for h, and initial to generate components.
3.3.3 LINEAR TRANSFORMER
Recent research by Schlag et al. (2021) has demonstrated the equivalence between TPR and linear attention mechanisms. Building upon this crucial insight, we integrate the AID into the Linear Transformer architecture to enhance the linear attention mechanism from TPR perspective. Our work uses the AID to extract key and value vectors for multi-hes attention, which can be considered TPR encoding components. The Linear Transformer takes an input vector x; at each position and subsequently generates components based on z., alignment with prior work, we also derive initial initial.component for competitive attention. To claborate further, We transform F, into á, using a foew-forwwrd network and partition it into Nipuut sub-vectors, (i,,..., to generate components.
based on the x, while simultaneously constructing muitiple input features
1 Nup"). The AID performs decompasition for h
and
4 EXPERIMENT
In this section, we evaluate the effectiveness and flexibility of our proposed AID module for both systematic generalization tasks and lorge-vocabulary language modeling task. These experiments
5

---

Published as a conference paper at ICLR 2024
100
80
Model
DNC
40
FUN
20
AID
10000 20000
30000
30000
20000
30000
0
20000
20000
30000
lleration
lteration
lcoraiio
Test accuracy curve [%] on the SAR task for 10 seeds with varying values of p. As increases, the models can learn more combinatorial information of data during training.
Figure 2:
show that the AID enhances the decomposition performance of TPR-based models and consequently provides overall systematic generalization performance improvement. For evaluation, we integrated the AID into established TPR-based approaches, as stated in Section 3.3. Also, we adopt the ex- perimental configurations used in prior works for fair comparison. All experimental details and parameter settings are illustrated in Appendix B. We first perform a synthetically designed system- atic generalization task (systematic associative recall) show and verify the effectiveness of AID operations. Then, we perform the experiments on more real-world tasks such as systematic bAbl task, Sort-of-CLEVE task, and WikiText-106 task.
4.1 SYSTEMATIC ASSOCIATIVE RECALL TASK
To show the effectiveness of the AID module, designed simplified Systematic Associative Recall (SAR) task that can measure the systematic generalization performance for unseen combi- natorial sequence data. Because the conventional associative recall tasks do not adequately capture the systematic generalization of memory networks (Graves et al., 2014; Le et al., 2020; Park et al., 2021) complex analyze (Bae et al., 2022), our SAR task mainly focused on assessing the model performance change according to varying compositional complexity of combinatorial input adopted the FWM, word-level TPR-based memory
data during training. In this experiment, network, as our baseline model and compared it to other representative memory networks, including Differentiable Neural Computer (DNC) (Graves al., 2016) and FWM.
Task Description The SAR task consists of two phases: a discovery phase involving memorizing input items and an inference phase recalling the memorized items. To introduce systematicity the items, we design each item combination of æ E X and E Y, where X = Xi U X2 U U33 and Y =Y= U 22 represent independent word sets. The task provides combinatorial sequence data to model during the discovery phase, and when the model is presented with an which is sampled from the sequence. it is required to recall the associated y during the inference phase (we illustrate an example of the SAR task in Appendix B.... We set different combination settings for each subset X; to evaluate the systematic generalization. During training, three types of combinations are provided to the models: (1) X1 and Y1, (2) X2 and Y2, and (3) X3 and Y. In contrast, during evaluation, the models are supposed to memorize and recall unseen combinatorial data, specifically X and Yg- We also introduce a seeper-parameter p. which adjusts the proportion of types (2) and (3). Here, p defined as |x]+| where |X,] denotes the cardinality of set X,, A larger value of p indicates that the models can learn more combinatorial information during training. By varying p. we can evaluate how the models" performance changes according the diversity of the combinatorial items given during training time.
Results AID shows better generalization performance for all values of p. as shown in Fig. 2. As expected, the performance of the baseline models is degraded as the value of p decreases, whereas the AID consistently shows high accuracy by successfully improving the baseline FWM.
4.1.1 ANALYSIS
We performed compositional analysis over the generated TPR component representations. For the quantitative aspect, disentanglement analysis is performed, and for the qualitative aspect, the orthog- onality of representation is analyzed.

---

Published as a conference paper at ICLR 2024
(b)
(a)
2.0
03
05
Infeenos
-05
-1.
-10
Figure 3: The heatmap for the cosine similarity between roles the SAR task.
Figure 4: The heatmap for the cosine similarity between roles (x-axis) and unbinding operators (y-axis) on the SAR task.
(b)
(c)
100
Figure 5: Test accuracy over different settings on the SAR task for seeds. (a) Varying Ninuts with Niters 2. (b) Varying training iterations with Ninputs = 3. (c) Varying test iterations for the models trained with Ninputs = 3 and Niter= 5.
Disentanglement Analysis We quantitatively evaluate the quality of disentanglement in the gen- erated representations using the DCI framework (Eastwood & Williams, 2018). The DCI assesses the disentanglement (D), completeness (C), and informativeness (I) of the generated representations through the mapping from representations to generative factors (e.g., and y). Specifically, evalu- ate multiple role and filler representations for each item, we employ a block-level DCI metric (Singh al., 2022) by concatenating the TPR encoding components. Each metric indicates the degree to which the block representation disentangles the factors (D), how specialized each factor is to spe- cific block representation (C), and the accuracy of predicting ground-truth factor values based on the representations (I).
Table shows the block-level DCI results for the baseline model and the AID across different p values. Notably, the AID module enhances the quality of representational dis- entanglement and completeness compared to the baseline model across all values.
Table 1: DCI results on the SAR task.
Model
p =
p =
p
C
C
FWM
0.91
0.63
0.99
0.92
0.64
0.59
0192
0.63
0.99
AID
0.95
0.66
0.99
0.97
0.67
0.99
0.97
0.67
0.99
These results demonstrate the AID mod- ule's s efficacy in capturing underlying factors during TPR component generation, which may explain why the AID improves task performance.
Orthogonality Analysis Our qualitative analysis focuses on the generated representations in terms of orthogonality. As posited by Smolensky (1990), to ensure undistorted encoding and decoding pro- cesses, the roles should exhibit lower correlations among themselves, and the unbinding operators should exhibit high correlations with roles for the same æ. To ascertain this, we evaluate the cosine similarity between TPR components for combinatorial data, with varying x and fixed y, in the case of 0.5.
Fig. shows the similarity between roles from the discovery phase, reflecting the quality of the en- coding process. While the FWM yields disentangled representations, as indicated in Table 1, its role representations lack orthogonality for distinct x. In contrast, the AID generates highly orthogonal role representations. Fig. 4 shows the similarity between roles from the discovery phase and unbind- ing operators from the inference phase, reflecting the decoding process's 's quality. While the FWM generates less correlated roles and unbinding operators, even for the same x, and fails to solve the SAR problem, the AID generates highly correlated representations for the same x and less correlated ones for distinct Furthermore, Figs. 3(b) and 4(b) show consistent patterns for roles and unbind-

---

Published as a conference paper at ICLR 2024
Table The mean word error rate [%]] on the sys-bAbl task for 10 seeds.
Model
wio sya did
w/ w/sys diff
Gap
IXL
3.71 + 0.47
8.72 3.27
5.01
DAM
0.48 + 0.20
5:25 1.64
4.77
STM
0.16
4.19
TPR-RNN
8.74
7.95
= 0.08 (0.10 4)
5.61 (3.13 4)
(3.03 4)
+AI
0.45 #: 0.16 (0.34 1)
1.21
0.66 (1.64 J)
0.76 (1.3 L)
ing operators, indicating that the AID module correctly decodes the filler information from the -PR-based memory. From all these results, is clear that the AID module learns to decompose data into meaningful component representations that better conform to TPR conditions than the baseline model.
The effect of Ninputs and Niter We analyze the effect of hyper- parameters, Niiutts and Niterss in the case of p 0.0. In Fig. 5(a), observe that the AID achieves its best performance when Ninurrs is set to 3. However, it shows slightly decreased performance at the other values of Ninuuts- These re- sults show an optimal number of input features for adequate competitive attention decomposition. Fig. 5(b) shows that more iterations of decomposition enhance accuracy until a certain threshold, which aligns with the result from slot-attention((ocatte loccttelo et al., 2020). When we increase Nitrs at test time, accuracy is increased until it reaches the training time iteration, as shown in Fig. 5(c).
4.2 SYSTEMATIC BABI TASK
Next, we evaluate the effect of the AID module basic question-answering task, the bAbl task (Weston et al., 2015), which tests text understanding and reasoning abilities. To evaluate systematic generalization, we slightly modify the contents of the test data. This modified task, named sys-bAbl, evaluates the model's 's performance under two aspects: (i) in an-distribution (w/o sys diff) and (ii) with the systematic difference (w/ sys diff). In this experiment, we use TPR-RNN and FWM as baseline models and compare our method to state-of-the-art memory networks, including Transformer-Xi (TXL.) (Dai al., 2019), Distributed Associative Memory (DAM) (Park al., 2021), Self-Attentive Associative Memory (STM) (Le et al., 2020), TPR-RNN, and FWM.
Task Description The bAbl task consists of 20 distinct question- answering tasks, such time reasoning and yes/no questions. Each task comprises stories, relevant questions, and corresponding answers. The bAbl task requires the models to remember the stories and recall information related to the questions to predict the correct answers. To evaluate systematic generalization, we modify the data of the bAbl task by replacing words across sub-tasks. Specifically, names, such as Mary, John, and Fred, which appear in various tasks (alternative word types, such as location names, can also be replaced). Through this replacement, the test data of each task will contain words that are not visible during training. The models are expected to learn task-independent text understanding to solve the sys-hAbl task. We train the models on all tasks and evaluate them on subsets of the bAbl tasks that are reasonable for implementing the replacement.
replace human
Results Table 2 shows the text understanding performance in w/o sys diff and w/ sys diff settings on the sys-bAbl task. Existing memory networks record high-performance gap between both settings despite their solid achievement the original test data, w/o sys diff. The AID improves the baseline models' systematic generalization and performs better than other memory networks in both settings. This result indicates that TPR-based approaches benefit systematic generalization but require a more elaborate design for decomposing structured representations.
4.3 SORV-OF-CLEVR TASK
We evaluate the competence of the AID module in visual relational reasoning tasks. We use the Sfrt-of-CLEVR task (Santoro et al., 2017). which evaluates compositional generalization for visual scenes. For our experiments, use Linear Transformer the baseline model for comparison.
8

---

Published as a conference paper at ICLR 2024
Table Test accuracy [%] on the Sort-of-CLEVR task for 10 seeds.
Model
Unary Accuracy
Binary Accuracy
Ternary Accusecy
Linear Transformer
69.3 148
75.5 13
56.4
98.9
(29.6
78.6 033
63.7 12.33
Table 4: Perplexity on the WikiText-103 task. The experimental results are obtained through open sources provided by (Schlag et al., 2021).
Model
Valid
Teat
Lincar Transformer"
36,473
37.533
36.189 (0.314 L)
37.151
Delta Network'
35,640
36.659
AID
35.361 (0.279 L)
36.253 (0.406 L)
Task Description The sort-of-CLEVR task is a visual question-answert task that evaluates the visual understanding of objects. It includes scene images, relevant questions, and corresponding swers. Introduced by Mittal et al. 1), this task consists of three question types: (i) the properties of single objects (Unary), such color and shape, (ii) the relationship between two objects (Binary), and (iii) the relationship among three objects (Ternary). To successfully solve this problem, models must capture the properties of the objects or their relationships to predict the correct answers.
Results Table shows the generalization performance on the sort-of-CLEVR task for different question types. Remarkably, the Linear Transformer struggles solve these fundamental visual question-answering tasks. However, with the integration the AID module, the baseline model's generalization performance improves significantly, demonstrating the effectiveness of the AID in enhancing the generation of the Linear Transformer's key and value vectors.
4.4 WIKITEXT-IO9 TASK
In addition to systematic generalization tasks, we extend our evaluation to assess the effectiveness of the AID on more practical large-scale task that may explicitly demand systematic general- ization capability but is an important problem. WikiText- task (Merity et al., 2016) comprises lengthy corpora from Wikipedia articles and evaluates the model's understanding of the probability distribution over sequence of words through perplexity. We use Linear Transformer and Delta Network (Schlag al., 2021) as our baseline models for this experiment and compare our method against them.
Results In the experiment, work as our baseline models, in accordance with Schlag et al. (2021). To mitigate the additional computational overhead by the AID, we strategically integrate our AID module into the baseline models at intervals of 4 out of the 16 layers. Table 4 shows the perplexity results of the WikiText- 103 task. Notably, our AID improves the perplexity of baseline models both baseline models, demonstrating its potential for enhancing performance even on large-scale tasks.
16-layered Linear Transformer and 16-layered Delta Net-
CONCLUSION
In this work, we propose a novel AID module that effectively decomposes sequential data into structured representations of TPR. AID iteratively applies slot-based competitive attention to bind sequential features to abstract structured representations of TPR and also learns to route specific symbolic meanings to each slot for the context-dependent initialization of roles and fillers. In ex- periments, we demonstrate the effectiveness of the AID module on synthetic systematic general- ization tasks, text/visual question-answeringg tasks, and even large-scale language modeling tasks. Our analysis shows that the AID provides bell-bound representations for based networks and successfully enhances overall systematic generalization performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea(NRFF grant funded by the Korea govemieen(MSSTTXNo. 2022RIA547026673),
REFERENCES
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advances in neural information processing systems, 29, 2016a.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: NXi:166070065000 2016b.
Jun-Hyun Bae, Taewon Park, and Minho Lee. Learning associative reasoning towards systematicity using modular networks. In International Conference on Neural Information Processing. pp- 113-122. Springer, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1788)117,- 2013.
Ajnrk-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07288) 2015.
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. The neural data router: Adaptive control flow in transformers improves systematic generalization. arXiv preprint arXiv:2110.07732, 2021.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02866 2019.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation disen- tangled representations. In International conference on learning representations, 2018.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analy-
sis. Cognition, 28(1-2):3-71, 1988.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 478(2266)20010008 2022.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909 190e:9999.,,9, 2019.
Alex Graves, Greg Wayne, and Ivo Danihelka. arXiv: 1410:544005501 2014.
Neural turing machines.
arXiv preprint
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):771-47.. 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem artificial neural networks. arXiv preprint arXiv:2012.05200 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. arXiv preprint arXiv: 16/v.336/. 2016.
10

---

Published as a conference paper at ICLR 2024
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Dieuwke Hupkes, Verna Dankers, Mathis Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Kazuki Iric, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear trans- formers with recurrent fast weight programmers. Advances in Neural Information Processing Systems, 34:7703-7717, 2021.
Yichen Jiang, Asli Celikyilmaz. Paul Smolensky, Paul Soulos, Sudha Rao, Hamid Palangi, Roland Fernande7, Caitlin Smith, Mohit Bansal, and Jianfeng Gao. Enriching transformers with structured tensor-product representations for abstractive summarization. arXiv preprint arXiv:2/06.01317, 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165. PMLR, 2020.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequeence-to-sequencc recurrent networks. In International conference on machine learning, pp. 2873-2882. PMLR, 2018.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In International Conference on Machine Learning, pp. 5682-5691. PMLR, 2020.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmnn): Building a longer and deeper rnn. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5457-5466. 2018.
Adam Liška, Germán Kruszewski, and Marco Baroni. Memorize generalize? searching for compositional rnn in a haystack. arXiv preprint
2018.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33: 1525-11538, 2020.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. arXiv preprint arXiv:1508.0400 2015.
Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Fast and slow learning of recurrent independent mechanisms. arXiv preprint arXiv:2 arXiv:2105.0877 105.08700, 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. models. arXiv preprint (tti::66000078433 2016.
Pointer sentinel mixture
Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish, Yoshua Bengio, and Guillaume Lajoie. Com- positional attention: Disentangling search and retrieval. arXiv preprint 1rXiv:2110.094 2021.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metaleamed neural memory, Advances Neural Information Processing Systems, 32, 2019.
Randall C O'Reilly, Charan Ranganath, and Jacob L Russin. The structure of systematicity in the brain. Current directions in psychological science, (142) 2022.
Taewon Park, Inchul Choi, and Minho Lee, Distributed associative memory network with memory refreshing loss. Neural Networks, 144:33-48, 2021.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Ad- vances in neural information processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2024
Imanol Schlag and Jürgen Schmidhuber. Gated fast weights for on-the-fly neural program genera- tion. In NIPS Metalearning Workshop, 2017.
Imanol Schlag and Jürgen Schmidhuber. Learning to reason with third order tensor products. Ad- vances in neural information processing systems, 31, 2018.
Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojie, Jürgen Schmidhuber, and Jian- feng Gao. Enhancing the transformer with explicit relational encoding for math problem solving. arXiv preprint arXiv: 1910.06611 2019.
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast weight memory. arXiv preprint 0rX.v:2011.0783/1 2020.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers secretly fast weight programmers. In International Conference on Machine Learning, pp. 9355-9366 PMLR, 2021.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-13, 1992.
Jürgen Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In ICANN'93: Proceedings of the International Conference Artificial Neural Networks Amsterdam, The Netherlands 13-16 September 1993 3, pp. 460-463. Springer, 1993.
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A benchmark for robust multi- hop spatial reasoning in texts. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 11321-1132, 2022.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh Inter- national Conference on Learning Representations, 2022.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159-216 1990.
Paul Smolensky, R Thomas McCoy, Roland Fernandez, Matthew Goldrick, and Jianfeng Gao. Neu- rocompositional computing in human and machine intelligence: A tutorial. 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15())11999-1988 2014.
Christoph Von Der Malsburg. The correlation theory of brain function. Models of neural net- works: Temporal aspects of coding and information processing in biological systems, pp. 95-119. Springer, 1994.
Taylor W Webb, Ishan Sinha, and Jonathan D Cohen. Emergent symbols through binding in external memory, arXiv preprint arXiv:2012.1 14601, 2020.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint 150i 05698, 2015.
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object- centric representations for reinforcement learning. arXiv preprint 0444443002.04419, 2023.
12

---

Published as a conference paper at ICLR 2024
A PSEUDO-CODE
Algorithm Attention-based Iterative Decomposition module.
1: Input: inputss E X initii i _component.s E R.Nam Dam Dom
2: Layer params: q, linear projections for attention; MLL updaris LayerNorm: MLP rinal
3: components = initial.compoeente
4: key, value k(inputs), Vinputts)
5: key = ELU(key)+
6: for Oer do
7:
query = > witti::tttitaaaaoooooooooo
8:
query = 7Dam 7
9:
query = HILU(query)+1
attn Softmax(key waeekeeyyyuurrruee wot.pryy"..iiiscoooooeeeeeete
11: updates =
Veighte/Nccatwiighrsttttt-- values=value)
components
Dan MLP upuu (lyeeouucelyy))eeuppdddtee))
end for
components Mi.Pinaccooppoeent DropOu((nnitaal (mppour(iiciaal.ccoooeeeett)))
return components
B EXPERIMENTAL SETTINGS
B.1 SYSTEMATIC ASSOCIATIVE RECALL TASK
Task Description The SAR task consists of two phases: a discovery phase involving memorizing input items and inference phase recalling the memorized items. To introduce systematicity the items, we design each item as combination of æ E X and and Y Yi U V2 represent independent word sets. The task provides combinatorial sequence data to model during the discovery phase, and when the model is presented with an which sampled from the sequence, it is required to recall the associated y during the inference phase. We set different combination settings for each subset X; to evaluate the systematic generalization. During training, three types of combinations are provided to the models: (1) X₃ and Y (2) X2 and Y2, and (3) X3 and Y. In contrast, during evaluation, the models supposed memorize and recall unseen combinatorial data, specifically X, and Yz-
Y, where X = X1 U X2 U X3
Fig. 6 illustrates example of the SAR task. At each training iteration, generative factors (o and y) are sam- pled from word sets X and Y construct the input se- quence. The sampled æ and values are randomly paired, creating combinations of one æ and one y each. These pairs are then embedded into vector space and concate- nated with flags, which are scalar values signaling the start of the discovery and inference phases. In the discov- ery phase, models sequentially receive these concatenated representations. During the inference phase, the model is presented only with z values (considered as role) and is tasked with predicting the corresponding y values (con- sidered as fillers).
X
Ya
Model
X2
X2
Xy
X4
@
Xa
76
Y2
y3
Time Step
Diccovryy Phase
Inferance Phose
Figure 6: Example of the SAR task for sampled pairs {(11,11), {I2 84), (23.82) {a4, ya]) from word sets X Experiment Details We use a set of arbitrary 1,000 words to construct each word set, as outlined in Table 15.
and Y.
We map each word in 50-dimensional space using the word embedding method. Therefore, each item is formed by concatenating the embedding vectors for & and y. The experiments on the SAR task repeated 10 times with distinct seeds. During training, 3, of 0.9, and B2 of 0.98 for training iterations of 30K
utilize the Adam optimizer with
batch size 64 and a learning rate (for training DNC, we use the R.Mppopp optimizer with learning rate of le and of 0.99).
13

---

Published as a conference paper at ICLR 2024
B.2 SYSTEMATIC BABI TASK
Task Description The bAbl task (Weston et al., 2015) consists of 20 distinct question-answering tasks, such time reasoning and yes/no questions. Each task comprises stories, relevant questions, and corresponding answers. The bAbl task requires the models to remember the stories and recall information related the questions to predict the correct answers. To evaluate systematic general- ization, we modify the test data of the bAbl task by replacing words across sub-tasks. As shown in Table we replace [Daniel, John, Sandra) to {Bill, Fred, Julie), and vice versa, during the replacement process. Since this modification is applied to the subset of the bAbl tasks, we evaluate the models on the modified tasks while training them on all tasks.
Experiment Details We use the experiment settings used TPR-RNN (Schlag & Schmidhuber, 2018) and FWM (Schlag et al., 2020). The experiments on the sys-bAbl task are repeated 10 times with distinct seeds. We also use the word embedding method to embed words in vector spaces.
For TPR-RNN, we set the embedding size to 179. For training, we utilize the Adam optimizer with a batch size of 64 and a learning rate of le -3, 11 of 0.9, and 32 of 0.99 for 100 training epochs.
For FWM, we set the embedding size to 256. For training. we utilize the Adam optimizer with batch size of 64 and learning rate of 1 31 of 0.9, and [32 of 0.98 for training iterations of 60K.
Additionally, we incorporate reconstruction loss, following the approach employed in Park et al. (2021), enhance representation learning on the sys-bAbl task. our reconstruction loss, the models are required to predict words based on their word embedding vectors. This loss is scaled to 1/100th and added to the main task loss.
Table 5: Word replacement across sub-tasks on sys-bAbl task.
Sub-task number
1, 7, 8,
10, 14
Word
(Daniel, John, Sandra, Mary)
(Bill, Fred, Julie, Mary]
B.3 SORT-OF-CLEVE TASK
Task Description The sort-of-CLEVR task is a visual question-answering task that evaluates the visual understanding of objects. It includes scene images, relevant questions, and corresponding answers. Each scene image contains objects, each characterized by a distinct shape and color selected from possible shapes (square or circular) and 6 possible colors (red, blue, green, orange, yellow, or gray). Introduced by Mittal et al. (2021), this task consists of three question types: (i) the properties of single objects (Unary), such as color and shape, (ii) the relationship between two objects (Binary), and (iii) the relationship among three objects (Ternary). An example of question type (i) is: "What is the shape of the red object?". An example of question type (ii) or (iii) is: "What is the shape of the object that is farthest from the blue object?".
We follow the experiment settings used in Mittal et al. (2021). To extract image features, we use single CNN layer with kernel size of 15 and stride size of 15. As our bascline model, we use 4-layered Linear Transformer with shared parameters. The experiments on the sort-of-CLEY task repeated 10 times with distinct seeds. For training, we utilize the Adam optimizer with batch size of 64 and a learning rate of le for 100 training epochs.
B.4 WIKITEXT- 103 TASK
WikiText- 103 task (Merity et al., 2016) comprises lengthy corpora from Wikipedia articles and evaluates the model's understanding of the probability distribution sequence of words through perplexity. The training set consists of 28,475 articles, while the validation and test sets contain 60 articles each.
We follow the experimental settings described in Schlag et al. (2021) for the WikiText-103 task. The training data are partitioned into segments of L words (back-propagation span). During the
14

---

Published as a conference paper at ICLR 2024
evaluation, with batch size of one, compute perplexity using sliding window with length of L. The perplexity computation considers only the last position segments, except for the first segment, where all positions evaluated. For training, of 96, an initial learning rate of 2, and a learning rate warmup step of 2,000 for 120 epochs.
Adam optimizer with batch size
16-layered Linear Transformer and 16-layered Delta Network baseline models, following Schlag et al. (2021). To mitigate the additional computational overhead by the AID, strategically integrate our AID module into the baseline models intervals of out of the 16 layers. Fig. 10 shows the experimental results for two cases: (1) where AID applied to the first layer and (2) where AID applied to the fourth layer.
In the experiment,
C
SETTINGS OF AID MODULE
Tables 6. and show our module's hyper-parameter settings for each task. Each parameter is denoted as follows: Noom as the number of components, Niees as the number of iterations, N NNipus as the number of input features, Dinputs as the dimension of input features, Deam as the dimension of components, DMLP,
the dimension of MLP update 's two layers, DML. as the dimension of MLP Páropout as the probability of Dropout. particular, we set different values as Neom in en- coding and decoding, in line with the TPR functions of the baseline model (Schlag & Schmidhuber, 2018; Schlag et al., 2020).
Also, the gray background represents the baseline models' hyper-parameters Each parameter is denoted as follows: Deniiy as the dimension of the entity vector, Dielation as the dimension of the relation vector, Need as the number of read heads, DLSTM the dimension of LSTM's hidden state, D Deemory as the dimension FWM's memory, Nhead, as the number of attention heads, Dreads as the dimension of attention heads.
Table Hyper- settings the AID on TPR-RNN.
Systematic bAbl task
Deniny
90
Dretution
20
Nene
5
Neke
Nusrs
2
Dingues
Deom
64
(128, 64)
DaaLPuo
64
Pdropout
0.5
15

---

Published as a conference paper at ICLR 2024
Table Hyper-parameter settings of the AID FWM.
SAR task
Systematic bAbl task
Nrad
DISTM
256
2566
D Demoory
32
32
Newn
3
3
Noke
++ Nian
Neead
Ness
2
3
Nasuus
3
6
D mpuas
32
32
Dom
32
32
DaA
(64, 32)
(64, 32)
DMALau
32
32
Paropou
0.5
0.5
Table 8: Hyper-parametee settings of the AID on Linear Transformer.
Sort-of of-CLVVR task
WikiText- 103 task
Nacuds
8
Drends
64
Neom
Noends
Nheak
Naas
N Niguus
Nisuds
Nheads
D Dinpas
Ducads
Doeuds
Deom
Dueads
Drends
DrLLP
(2 Dheats Dreais))
(2 Dueatss Dresks)
DMLPEus
Dueads
Dreuds
Paropont
0.5
0.5
D ABLATION STUDY
Module In Fig. 7, show the effect of incorporating residual connection (He et al., 2016) to query and a concatenation after the iterative process. Each of these techniques demonstrates the enhancement in performance when applied the sys-bAbl task. However, the concatenation tech- nique exhibits unstable and lower performance on the SAR task. Remarkably, when these techniques are jointly applied to the AID, clear improvement is evident across both tasks.
Activation function In Fig. 8, we show the effect of activation functions at dot-product atten- tion. ELU activation function (Clevert al., 2015) contributes to improved stability and enhanced performance across both tasks.
16

---

Published as a conference paper at ICLR 2024
90
w/o sys-diff
w/ sys-diff
Accuracy
80
Error Rate
2
70
1
60
Residual Concat
Residual
Concat
Both
Module
Module
(a) Systematic Associative Recall task
(b) Systematic bAbl task
Figure 7: Ablation Study for module.
90
w/o sys-diff
sys-diff
Accuracy
80
Error Rate
70
2
60
ReLU Activation
ReLU Activation
ELU
aa) Systematic Associative Recall task
(b) Systematic bAbl task
Figure 8: Ablation Study for activation functions.
E ADDITIONAL EXPERIMENT RESULTS
E.I SYSTEMATIC BABI TASK
Table 9; Additional experiment results on sys-bAbl task for 3 seeds.
Naas
Nagas
sys
s/s diff
Gap
paramns
2
5
0.69 1 0.11
3.69 1: 3.11
3.00
1.13
0.69 + 0.34
3.28 + 2.27
2.59
1.26 M
3
0,67 1: 0.19
2,80 0.88
2.13
0.87 M
0.67
3.42
2.75
1.00
0.55 + 0.08
2.73 4: 1.50
2.18
1.13
0.54 +: 0.38
3.32 + 0.80
2.78
1.26
17

---

Published as a conference paper at ICLR 2024
E.2 WIKITEXT-103 103 TASK
Table 10: Additional experiment results on WikiText-103 task.
Model
Volid
Test
paramns
Transformer'
36.473
37.533
44.02
AID (1" layer)
36.159 (0.314 L)
37.151 (0.382 L)
M
AID (4th layer))
36.301 (0.172 4)
37.425 (0.108 L)
44.16 M
Delta Network
35,640
36.659
44.03 M
ATD layer))
35,361 (0.279
36.253 (0.406
44.18 M
AID layer))
35,355 (0.285
36.291 (0.388
44.18 M
F COMPARISON TO BASELINE WITH MORE PARAMETERS
We conduct experiments with baseline models with more parameters. In the WikiText- 103 task, increase the size of the ceed-forward network in the attention part of the baseline model. The size increase is applied to the exact same positions of the model where AID adopted, for a fair comparison with our AID-assisted network architecture. In other tasks, apply different method to increase the model parameters, increasing either the hidden or head size of the baseline models. Fig. Tables 11, 12, and 13 show that improvements obtained with AID do merely come from the number of increased parameters in the models.
F.1 SYSTEMATIC ASSOCIATIVE RECALL TASK
0=0
p= 1.0
100
Accuracy
Model
FWW2200,
FWMM5122)
2.03M
Eeration
Figure 9: Comparison to baseline with more parameters the SAR task for 10 seeds.
F.2 SYSTEMATIC BABI TASK
Table 11: Comparison to baseline with more parameters the syy-bAbl task for 10 seeds.
Model
DiSTM
wlo sys diff
wys diff'
Gap
# params
FWM
256
0.79 1 0.14
2.85
2.35
0.73 M
256
0.45 0.16 (0.34
1.21
4)
0.76
4)
1.23 M
+
18

---

Published as a conference paper at ICLR 2024
F.3 SORT-OF-CLEVR TASK
Table 12: Comparison to bascline with more parameters on the Sort-of-CLEVR task for 10 seeds.
Model
Nusate
Duna
Unary Accuracy
Binory Acewacy
Ternory Aceuracy
params
Linear Transforner
32
82.5 # 18.3
78.3 = 2.7
60.0 = 5.0
0.68
AID
98.9 =b 0.2 (16.4 1)
78.0 = 0.6 (0.3 ()
61.0 2 1.7
(1.0
0.83
1 Linear Transformer
64
69.3
14.8
75.5 = 1.3
56.4 4.3
0.68 M
AID
98.9
(29.8
78.6 0.3 0.33
63.7 1.2 (7.3
0.83
Linear Transformer
64
57.5
59.7 11.1
13
2,55
Transformer
128
57.9 =
59.9
52.2
F.4 WIRITEXT-103 TASK
Table 13: Comparison to baseline with more parameters on the WikiTexxt-033 task. indicates an increasing feed-forward size of the attention layer.
Model
Volid
Text
params
Linear Transformer
36.473
37,533
44.02 M
(0.314
37.151
(0.382 4)
44.16
Linear Transformer'
37.306
Delta Network
640
36.659
44.03
AID
35.361 (0.279
36.253
(0.406 4)
44.18 M
Delta Network
35.468
36.639
44.23 M
G ADDITIONAL QUALITATIVE ANALYSIS
We conduct qualitative analysis for generated representations on the bAbl task. For this analysis, we consider the following two sentences where the desired answer is "kitchen".
(wfo sys-diff) sandra moved the office. afterward she journeyed to the kitchen. daniel went to the hallway. then he journeyed to the kitchen. where is sandra?
(w/ sys-diff) julie moved to the office. afterward she journeyed to the kitchen. bill went to the hallway. then he journeyed to the kitchen. where is julie?
Figs. 10 and 11 show the similarity between roles across the input sequence. FWM and AID exhibit a high correlation when the sentence subjects are identical, suggesting that word-level TPR-based Notably, FWM shows
models might learn represent symbolic structures sentence-by-sentene high intra-sentence word correlation, while AID shows a high correlation at sentence terminations (indicated by "!"). As highlighted in the yellow box comparison, FWM, when confronted with unfamiliar subjects (w/ sys-diff case), shows decreased correlation between relevant sentences and an increased correlation among irrelevant ones. Conversely, AID maintains consistent results irrespective of systematic differences.
Furthermore, we explore similarity patterns between roles and unbinding operators, as done in Schlag et al. (2020). We utilize the roles generated at each time step of the input sequence and the unbinding operators generated at "?" for each of the read heads (N, 3). Figs. 12 and 13 reveal that both models exhibit high correlation at the end of each sentence ("""). As seen from yellow box, the FWM struggles to link FWM struggles to link the "" of the question-related sentences in the sys-diff case, which may explain the prediction of incorrect answer ("office")). In contrast, the AID shows consistent patterns and accurately predicts the correct answer. These findings elucidate why FWM fails and AID succeeds in tackling the sys-bAbl task.
19

---

Published as a conference paper at ICLR 2024
3000
03
025
-023
-19
(a) w/o sys diff
mwst
1.00
050
015
-02
(b) w/ sys diff
Figure 10: The heatmap of FWM for the cosine similarity between roles on the sys-bAbl task. We use the roles generated at each time step of the input sequence.
20

---

Published as a conference paper at ICLR 2024
03
025
-023
-19
(a) w/o sys diff
mwsl
1.0
050
015
-02
(b) w/ sys diff
Figure 11: The heatmap of AID for the cosine similarity between roles on the sys-bAbl task. We use the rolex generated at each time step of the input sequence.
21

---

Published as a conference paper at ICLR 2024
(a) w/o sys diff (predicting kitchen)
(b) w/ sys diff (predicting office)
Figure 12: The heatmap of FWM for the cosine similarity between roles (X-axis) and unbinding operators (y-axis) on the sys-hAbl task. We use the roles generated at each time step of the input sequence and the unbinding operators generated at "?" for each the read heads (N, = 3).
(a) w/o sys diff (predicting kitchen)
(b) w/ sys diff (predicting kitchen)
Figure 13: The heatmap of AID for the cosine similarity between roles (x-axis) and unbinding operators (y-axis) on the sys-bAbl task. We use the roles generated at each time step of the input sequence and the unbinding operators generated at "D" for each the read heads (N, = 3).
22

---

Published as a conference paper at ICLR 2024
FWM++II
w/
0,09
1.70
3.34
0.06
5.33
1.47
0.11
0.44
0.06
0.67
0.41
0.79
w/o
0.00
0.94
0.07
1.13
0.22
0.02
0.41
0.00
0.00
0.43
2.
w
2.18
2.81
9.88
0.09
5.57
1.74
0.10
0.44
8.13
3.29
1.42
2.47
FWM
w/o
1.08
5.88
0.05
1.34
0.36
0.08
0.28
0.00
0.01
0.86
Table 14: The mean word error rate [%)] in detail on the sys-bAb/ task for 10 seeds.
w/
2.73
5.85
0.09
15.82
3.08
0.58
0.45
25.05
4.45
4.46
4.66
TPR FRR-NNN-A
w/o
0.00
0.19
1.11
0.03
1.23
0.31
0.12
0.29
0.43
1.07
2.34
0.36
TPR-RNN
w/
2.85
5.91
10.05
0.30
20.46
3.90
0.67
1.19
28.65
13.21
6.51
11.19
w/o
0.36
2.33
0.05
1.32
0.64
0.32
0.66
1.14
1.98
0.52
w/
0.12
1.79
7.66
0.15
12.85
3.77
0.34
0.59
17.90
2.55
1.05
1.48
STM
w/o
0.56
3.43
0.05
0.23
0.07
0.07
0.06
0.06
0.03
0.20
w/
3.78
7.27
0.70
6.09
2.31
0.10
0.27
22.45
8.76
8.83
1.76
DAM
w/o
0.00
0.82
2.88
0.08
1.34
0.31
0.04
0.06
0.00
0.01
0.04
0.13
w/
2.31
24.42
35.68
0.03
5.71
3.64
0.28
0.57
18.10
4.14
1.79
7.95
TXL
w/o
11.44
0.01
1.80
0.84
0.43
0.00
0.00
6.46
23.
3: three supporting facts
I: one supporting fact
2: two supporting facts
10: indefinite knowl.
basic coreference
13: compound coref.
6: yes/no questions
9: simple negation
14: time reasoning
12: conjunction
counting
8: lists/sets
Task
23

---

Published as a conference paper at ICLR 2024
Table 15:
Word sets in the SAR task.
We refer to the words in link
Set
Word
X1
a, ability, able, about, above, accept, according, account, across, act, action, activity, actually, add, address, ad- ministration, admit, aduli, affect, after, aguin, against, age, agency, agent, ago, agree, agreement, ahead, air, all, allow, almost, alone, along already, also, although, always, american, among, amount, analysis, and, animal, another, answer, any, anyone, anything, appear, apply, approach, area, argue, arm, around, arrive, art, article, anist, as, ask, assume, at, sttack, attention, attorney audience, author, authority, available, avoid, away, baby, back, bad, bag, ball, bank, bur, base. be, beal. beautiful, because, becume, bed, before, begin, behavior, behind, helieve, benefit, best, better, between, heyond, big, bill, billion, hit, black, blood, blue, hoard, body, book, born, both, box. buy, break, bring. brother, budget, build, building. business, but. buy, by, call, camera, campaign, can, cancer, candidate, capital, car, card, care, career, carry. case, catch, cause, cell, center, central, century, certain, certainly, chair, challenge, chance, change, character, charge, check, child, choice, choose, church, citizen, city, civil, claim, class, clear, clearly, close, coach, cold, collection, college, color, come, commercial, common, com- munity, company. compare, computer, concern, condition, conference, congress, consider. consumer. contain. continue, control, cost, could, customer, cut, dark, data, daughter, day, dead, deal, death, debate, decade, decide, decision, doep, defonse, de- gree, democrat, democratic, describe, design, despite, detail, deternine, develop. development, die, difference, different, difficult,, dinner, direction, director, discover, discuss, discussion, disease, do, doctor, dog, door, down, draw, dream, drive, drop, drig, during, each, carly, cast, casy, cat, cconomic, economy
couple,
cover, create, crime, cultural, culture, cup. current,
X2 U Uss
edge, education, ellect, effori, eight, either, election. else, employee, end, energy, enjoy, enough, enter, entire, environment, environmental, especially, establish, even, evening, event, ever, every, everyhody, everyone, every- thing, evidence, exactly, example, executive, exist, expect, experience, expert, explain, eye, face, fact, factor, fall, family, far, fast, father. fear, federal, feel, focling, fow, field, fight, figure, fil, film, final, finally, financial, find, fine, finger, finish, fire, firm, first, fish, five. floor, fly, facus, follow, food, foot, for, force, foreign, forget, form, former, forward, four, free. fricad, from, front, full, fund, future, game, garden, gas, general, generation. get, girl, give, glass, go, goel. good, government, great, green, ground, group. grow, growth, guess, gun, guy. half, hand, hang. happen. happy. hard, have, he, head, bealth. hear, heari, heat, heavy, help, her, here, herself, high, him, himself, his, histoey, hit, hold, home, hope, hospital, hot, hotel, hour, house, how; however, huge, human, bundred, hushand, i, idea, identify. image, imagine, impact, important, improve, in, include, including. increase. indeed, indieate, individual, industry. information. inside, instead, institution, interest, inter- esting, interattonal, interview, into, investment, involve, issue, it, item, its, itself, job, join, just, keep, key, kid, kill, kind, kitchen, know, knowledge, land, language. large, last, late. later, laugh, law, lawyer, lay, lead, leader, learn, least, leave, left, leg, legal, less, let, letter, level, lie, life, light, like, likely, line, list, listen, little, live. local, long. look, lose, loss, lot, love, low, machine, maguzine, main, muintain, major, majority, make, man, manage, management, manager, many, market, marringe, material, matter
Yi
may, maybe, mo, mean, measure, media, medical, meet. meeting, member, memory, mention, message, method. middle, might, militury, million, mind, minute, miss, mission. model, modemn, momentt money, month more, morning, most, mother. mouth, move, movement, movic,
much music, must, my, myself. name, nation. national, natural, nature, near, nearly, necessary, need, network, never, new, news, newspaper, next, nice, night, no, none, nor, nont, not, note, nothing, notice, now, number, occur, of, off, offer, office, officer, official, often, oh, oil, ak, old, on, once, one, only, onto, open, operation, opportunity, option, or, order, organization, other, others, our, outside, over, own, owner, page, pain, paioting, paper, parent, part, participant. particular, particularly. partner, purty, pass, past, patient, pattem, pay. peace, people, per, perform, performance, perhups, period, person. personal, phone, physical, pick. picture, piece, place, plan, plant, play, player, pm, point, po- lice. policy. political, polities. poor, popular, population. position. positive. possible, power, practice. prepare. present, president, pressure, pressurre pretly, prevent, price, privale, probubly, problem, process, produce, product, pro- duction, professional, professor, program project, propery, protect, prive, provide, public, pull, purpose, push, put, quality, question, quickly, quite, race, radio, raise, range, rate, rather, reach, read, ready, real, reality, realize, really, reason, receive, recent, recently. recogniae, record, red, reduce, reflect. region. relate, relationship. rell- gious, remain, remember. remove, report, represent, republican. require, research, resource. respond, response. responsibility, rest, result, return, reveal, rich, right, rise, risk, road, mel, role, rom, rule, run, safe, same, save, say, scene, school, science, scientist,
season, seat. second, section, security. see
Y2
seek, seem, sell, send, senior, sense, series, serious, serve, service, set, seven, several, sex, sexual, shuke, share, she, shoot, short, shot, should, shoulder, show, side, sign, significant, similar, simple, simply, since, sing, single, sister, sit, site, situation, six, size, skill, skin, small, smile, so. social, society, soldier, some, somebody, somcone. something. sometimes, son, song, soon, sort, sound, source, south, southern, space, speak, special, specific, start,
speech, spend, sport, spring, staff, stage, stand, standard, stock, stop, store, story, strategy, street, strong, structure, student, study, stuff, style, subject, success, successful, such, sudienly, suffer, suggest, summeer, support, sure, surface, system, table, talk, task, tax, teach, teacher, team, technology, television, tell, ten, tend, term, test, than, thank, that, the their, them, themselves, then, theory. there, these, they, thing, think, third, this, those, though, thought, thousand, threat, three, through, throughout, throw, thus, time, today, logether. tonight. too, top, total, tough, toward, town, trade, traditional, training. travel, treat, treatment, tree, trial, trip, trouble, true, truth, try, turn, bv, two, lype, under, understand, unit, until, up, upon, us, use, usmally, value, various, very, viatim, view, violence, visit, voice, vote, wail, walk, wall, want, war, watch, water. way, we, weapon, wear, week, weight, well, west, western, what, whatever, when, where, whether, which, while, white, who, whole, whom, whose, why, wide, wife, will, win, wind, window, wish, with. within, without, woman, wonder, word, work, worker, world, worry, would, write, writer, wrong, yard, yeah, year, yes, yet, you, young, your, yourself
statement, station, stay, step, still,
24


Reviews:
[{"review_id": "13CbIJhL4a", "reviewer": "Reviewer_8A4n", "summary": "Background information: TPRs are an approach for representing compositional structure in vector space; they work by encoding a compositional structure via pairs of fillers - the components of the structure - and roles - the positions of the fillers in the structure. For instance, in the sentence “cats chase dogs”, the fillers could be the words “cats”, “chase”, and “dogs”, and the roles could be “subject”, “verb”, and “object”, respectively. Each filler and each role is represented with a vector, and these vectors are then combined via tensor products and matrix addition to produce a representation for the whole compositional structure.\n\nWhat this paper does: The authors introduce an approach called Attention-based Iterative Decomposition (AID) designed to generate role and filler representations for models based on Tensor Product Representations (TPRs). TPRs require the input to be broken down into fillers and roles (both represented as vectors), and this is what AID is designed to do; it can be plugged into any TPR-based system as a way to produce the fillers and roles, which can then be processed in the way they normally are for TPRs. AID starts with an initial proposal for the values of the role and filler vectors. These values are then iteratively updated; at each iteration, each TPR component (i.e., role or filler) attends to the input elements, and the TPR components compete with each other for which component attends to which input element. The result of the attention process at each iteration is a new proposal for the role and filler vectors, which is then the input to the next iteration, until the iteration finishes and the final role and filler vectors are produced. The authors then run experiments where they test 3 TPR-based systems from prior work on 4 tasks that involve systematic generalization. They find that adding the AID module improves compositional generalization across tasks.", "questions": "- Q1: What are the input features? I was unable to figure this out. Specifically, is each input feature one object (such as one word)? Or is it one element within a vector representation? From most of the paper I was assuming that it was one object. However, I normally associate the word “feature” with an element of a vector representation. Also, if the input features are objects, that makes me confused about why N_inputs is considered a hyperparameter in Figure 5, since that’s really a property of the task rather than a hyperparameter. One thing that could help to clarify this is to show an actual example from an actual task in Figure 1, so that we can see what wach input feature is in the context of that task.\n- Q2: Near the top of page 4, it says that producing initial_components from the concatenated input features assigns symbolic meanings to each component, such as roles and fillers. I don’t understand how it achieves this. I can understand why this would be useful (because it would provide a better/optimized starting point), but I don’t see how it pushes each component to have a particular symbolic meaning such as “roles” or “fillers”", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- S1: The paper addresses an important problem - namely, how to get neural networks to produce effective compositional representations.\n- S2: The proposed AID module is intuitive and can act as a drop-in module in any TPR-based system, meaning that it will be straightforward for other authors to adopt.\n- S3: AID shows very strong performance in the experiments, often substantially increasing accuracy over previous approaches.\n- S4: The experiments are extensive, providing compelling evidence for the strength of the approach.\n- S5: In addition to the experiments based on accuracy, there are also analyses of the structure of the learned representations, which deepen the analyses and lend insight into the ways in which the AID module is enhancing the representations.", "weaknesses": "- W1: I believe there is a potential confound of number of parameters. That is, if I understand correctly, AID adds more parameters to the model. Therefore, it’s possible that the improvements created by AID are due to having more parameters rather than due to the effectiveness of the strategy. For most of the experiments, the difference in performance is so large that it’s probably not solely due to number of parameters, but for the Wikitext experiment, the improvement that AID brings is pretty small, so it does seem like a more important concern there. The most convincing way to address this concern would be have the same number of parameters in the model version that has AID and the model version that doesn’t; this could be achieved by, for example, making the feedforward size a bit smaller in the AID version than the non-AID version.\n- W2: I believe that the paper mischaracterizes the binding problem. The binding problem is the question of how different attributes of a structure can be appropriately bound together; for example, given an image with a red square and a blue triangle, how can a system appropriately associate (that is, bind) colors and shapes in order to represent the fact that you have a red square and a blue triangle, rather than a blue triangle and a red square? There is a separate problem that I’ll call the “decomposition problem” (I don’t think this is a standard term, but it will be useful for this review), which is how to decide what the attributes of a structure are. The paper seems to use the term “binding” or “binding problem” when in fact what it talks about is the decomposition problem. Specific places where this occurs are in the abstract (“because of the incomplete bindings”, “can effectively improve the binding”), the second paragraph of the intro (“these works still have a binding problem”), and the first section of related work (“Binding problem”). The reason why I think that this work is not really about binding is that the part of the TPR formalism that does binding is the step where tensor products are used to combine fillers and roles; the AID module does not alter that portion of the formalism, which is why, properly speaking, I believe it is really about decomposition rather than binding. I would recommend updating the wording to clarify this point. \n- W3: I think the paper is not as careful as it should be at distinguishing facts (things that have been empirically demonstrated), goals (things that the authors want to achieve), and plausible guesses (things that we think are likely to be true but can’t be certain of). I would recommend rewording the paper to be more careful about these points; as it stands, some points are presented as facts when I believe they are in fact goals or plausible guesses, and this could potentially mislead readers about how clearly these points have been demonstrated. Here are the specific points that stood out to me:\n    - The intro says “these works still have a binding problem … because the decomposition mechanism they employed relies on a simple MLP, which is known to be not effective in capturing the compositional nature of data.” I think that this is plausible but not something that can definitively be stated as a fact; a way to more clearly state what is known vs. not known would be “we find that these approaches still show some difficulties on compositional generalization, likely because the decomposition mechanism they employed relies on a simple MLP, which may not be sufficiently structured to learn the compositional nature of the data.” Specific motivations for these edits: adding “likely” to signal that this explanation is plausible but can’t be definitively said to be the cause; add “may” for a similar reason; changing “capture” to “learn”, because an MLP can capture anything (it’s a universal function approximator), so the actual difficulty would be when it needs to learn something.\n    - At the start of section 2, I think the word “effectively” should be removed from “we illustrate how the AID module effectively decomposes”. This section doesn’t show that the AID is effective - that is not demonstrated until later, when there are empirical results. Similarly, near the top of page 4, I would remove the word “effectively” again; I don’t think this work demonstrates that competitive attention on its own is effective at decomposing (as opposed to competitive attention being effective when used in combination with the rest of AID, such as the appropriate initial_components).\n    - At the start of section 3, and at the end of “Disentanglement analysis” under 3.1.1, I would recommend removing the word “consequently”. That word asserts a causal connection that has not been demonstrated (we know that the model gets better disentanglement and better task performance, but we can’t be certain that one causes the other); a more valid way to phrase this would be “These results demonstrate the AID module’s efficacy in capturing underlying factors during TPR component generation, which may explain why the AID improves task performance.”\n    - Near the top of page 7, it says that AID generates “more accurate representations.” I don’t think that “accurate” is the right word here; a better phrasing might be “representations that better conform to the formal requirements that Smolensky established for ideal TPRs”\n- W4: Some aspects of the experimental setup were not clear to me. First is what the input features are; see Q1 below. Second is that I found it somewhat difficult to understand exactly how the tasks worked; this concern could be addressed by providing some examples of the tasks (ideally in the main paper, or in an appendix if there isn’t room). For example, it’s not clear to me what the inputs are in the SAR task - is it just one x and one y? Or a sequence of x’s and y’s, and if so how are they arranged - x y x y x y, or x x x y y y? And how is the model presented with an x?\n\nOverall, I really enjoyed this paper, but I do think that these concerns currently decrease the paper’s understandability as well as the confidence that we can place in its results. If these concerns are addressed, I would be open to raising my score.\n\nUPDATE: The author response has sufficiently addressed my concerns. Therefore, I have raised my score from 5 to 8; previously I said 5 because my concerns prevented me from recommending acceptance, but with these concerns addressed I believe that the paper's helpful contributions are now effectively highlighted, enabling me to happily recommend acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SI1YfgSzBF", "reviewer": "Reviewer_rEt9", "summary": "The work proposes using iterative attention for learning Tensor Product Representations (TPR), meant to improve their systematic generalization capability, as measured through experiments over textual and visual reasoning tasks.\n\n**Update following rebuttal:**\n\nDear authors, thank you for addressing the comments in my review.\n\nI appreciate the additional experimental results over the bAbI task, both the analysis per question type on table 14 and the new attention maps -- these look great. Expanding the discussion in the related work section about how the approach compares to slot attention is useful too. Finally, I appreciate the detailed response to my review and to those of other reviewers.\n\nWhile I agree with reviewer bmNb that novelty-wise, the technical difference between slot attention and the paper is not large, nevertheless, for the usefulness of applying it in the context of TPRs and, most importantly, for the thoroughness of the rebuttal and paper updates, including performing additional experiments following the reviewers' comment, I'm happy to raise my score.", "questions": "- **Qualitative Evaluation**: It would be particularly useful for this work to have more qualitative evaluation for both bAbI and sort-of-CLEVR. What do the different TPR components actually attend to? Does their behavior make sense over specific instances? What mistakes do they tend to make? What type of mistakes are made by the baselines and eliminated by the new approach? How do they behave over examples with unseen names (systematic generalization cases)? This type of analysis can significantly help in demonstrating the actual impact of integrating attention into TPRs, beyond the overall accuracy metrics.  For bAbI, a more detailed breakdown of the performance by question type or story length will also be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Idea**: TPRs and attention fit well together: identifying and extracting the role and filler components seems like a natural application of attention and so the integration between them makes a lot of sense to me.  \n- **Evaluation**: Experiments are conducted on multiple datasets including both textual and visual modalities as well as both synthetic and realistic data (bAbI, Sort-of-CLEVR, WikiText and the Systematic Associative Recall (SAR) task). The experiments investigate using the attention module to extend several related models (TPR-RNNs, Fast Weight Memory, and Linear Transformers). Both quantitative (through e.g. DCI, downstream performance) and a bit of qualitative analysis (visualization of similarity between the representations of the TPR components). Overall these support the approach’s flexibility.\n- **Clarity**: The presentation is good and the paper is clearly written and well-organized. The introduction and model sections do a good job motivating the idea and presenting the necessary background and preliminaries. The overview figure is very helpful. Detailed description is provided for each of the 3 inspected models and the 4 tasks. The supplementary is also good, providing implementation details and ablation experiments.", "weaknesses": "- **Novelty**: The iterative attention decomposition works very similarly to slot attention, reducing the technical contribution of the paper. The paper introduces the idea as a novel attention-based module, not making it clear enough that effectively this strongly relies on slot attention. The comparison to slot attention appears only at the very end of the paper. Since the approach integrates together existing ideas, it will make sense in this case that the related work section will appear earlier on, before the model section.\n- **Empirical Results**: The improvements for WikiText (perplexity) and disentanglement (DCI) are relatively low. On the other hand, we see larger improvements on bAbI and Sort-of-CLEVR.   \n- **Related Works**:  A more detailed comparison to the prior related works, in particular to “Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization” that also integrates attention and TPRs. It is cited by the paper but more discussions on similarities and differences would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vSHk766QK5", "reviewer": "Reviewer_bmNb", "summary": "This paper aims to improve Tensor Product Representation (TPR) for systematic generalization tasks. The authors propose an Attention-based Iterative Decomposition (AID) module, which is plug-and-play and can be easily integrated into existing TPR models. AID is conceptually similar to Slot Attention, but with special designs tailored towards the task. Experimental results show that AID consistently improves existing TPR methods across a broad range of tasks.", "questions": "The Orthogonality Analysis in Sec. 3.1.1 shows that AID also helps extract more orthogonal *roles*. I am curious why this is the case. In my own experience with Slot Attention, the object-centric features (slots) are usually entangled, as there is no loss to force them to be orthogonal. Any insights here?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The considered challenge, roles/fillers decomposition, is indeed very similar to the object binding problem in object-centric learning (OCL). Therefore, it is intuitive to apply the SOTA OCL module Slot Attention here.\n- The experimental evaluations are thorough. AID shows consistent and non-marginal improvement in all the tasks.\n- The ablation and adapted designs from the original Slot Attention are insightful.", "weaknesses": "My background is in OCL so I am unfamiliar with these tasks and baselines. One concern I have is all the tasks (except the WikiText-103 one) are very simple. I understand that areas in the early stage experiment on simple data. However, for example for the CLEVR VQA task, people can train a Slot Attention model to extract object-centric features, and then attach a small Transformer head to predict the question's answer. According to my own experience, such a naive baseline can already achieve nearly perfect accuracy (on the original CLEVR dataset, not Sort-of-CLEVR). Therefore, it is hard for me to assess the importance of this paper.\n\nAlso, what is the difference in model size and computation cost of baselines with and without AID? For example, on the WikiText-103 task, the authors mention that they do not insert AID in every layer due to computation concerns. I wonder how will the baselines perform if they have more parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6PgNqbNZg", "reviewer": "Reviewer_bYLE", "summary": "This paper proposes an Attention-based Iterative Decomposition (AID) module that uses a competitive attention mechanism to decompose sequential input features into structured representations (roles, fillers, and unbinding operators) to improve systematic generalization for Tensor Product Representation (TPR) based models.\nThe AID module is flexible enough to integrate with existing TPR-based models such as TPR-RNN, Fast Weight Memory, and Linear Transformer.\nThe experiments support the improvements, show AID produces more compositional and well-bound structural representations, and exemplify applications with large-scale real-world data.", "questions": "(2) Do the intermediate TPR components always keep TPR conditions (the three key conditions required by TPR)?\nFor example, in integrating with TPR-RNN, the input features to the AID module $x_t$ are a set of word vectors, which may be in any form.\nDoes the AID module convert the input features to TPR?\n\n(3) TPR has its properties, such as the separation of roles and fillers.\nDoes the AID module use TPR properties in the module design, e.g., use role for attention key?\n\n(4) Though the AID module is designed to enhance TPR-based models, is it also informative to compare it with non-TPR-based models in experiments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- It is important to decompose sequential input to structured representations for systematic generalization, and the AID module enhances the performances for TPR-based models.\n\n- The module design is simple and clean, so it may be expected to keep the advantage in general cases.\n\n- It integrates with a wide range of TPR-based models in flexible ways.", "weaknesses": "(1) The WikiText-103 task shows the AID module performs well in a large-vocabulary language modeling task, but it seems not to be a systematic generalization task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXuaeHlkeR", "reviewer": "Reviewer_Qfs2", "summary": "The authors try to apply the Attention mechanism in the tensor product representation models. They also showed that the proposed AID block can be easily incorporated into many existing networks. Experiments show the advantages of introducing the AID block in previous network architectures.", "questions": "How is the scalability and complexity of the proposed AID module?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors proposed a new Attention based module for TPR. The proposed module can be combined with existing structures such as TPR-RNN, FWM and Linear Transformers.\n2. The authors conducted extensive experiments including ablation studies to show the advantages of the AID module and influences of hyperparameters.\n3. Code for all experiments is provided.", "weaknesses": "The authors mentioned that one advantage of TPR is to represent symbolic structures. I am wondering if this was demonstrated in experiments. I am not familiar with these tasks, but I did not find descriptions about this issue in experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)