A Framework for Inference Inspired by Human Memory Mechanisms
Authors: Xiangyu Zeng
Jie Lin
Piao Hu
Ruizheng Huang
Zhicheng Zhang
Venue: ICLR 2024 poster (2024)
Keywords: memory
transformers
architecture
inference
cognitive model
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose a PMI framework that incorporates perception, memory (dual) and inference, which can be embedded into mainstream models like Transformers for various QA and inference tasks.

Abstract: How humans and machines make sense of current inputs for relation reasoning and question-answering while putting the perceived information into context of our past memories, has been a challenging conundrum in cognitive science and artificial intelligence. Inspired by human brain's memory system and cognitive architectures, we propose a PMI framework that consists of perception, memory and inference components. Notably, the memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience. Through a differentiable competitive write access, current perceptions update working memory, which is later merged with long-term memory via outer product associations, reducing information conflicts and averting memory overflow. In the inference module, relevant information is retrieved from two separate memory origins and associatively integrated to attain a more comprehensive and precise interpretation of current perceptions. We exploratively apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets, as well as detecting equilateral triangles, language modeling and image classification tasks, and in each case, our PMI enhancements consistently outshine their original counterparts significantly. Visualization analyses reveal that relational memory consolidation, along with the interaction and integration of information from diverse memory sources, substantially contributes to the model effectiveness on inference tasks.

Published as a conference paper at ICLR 2024
A FRAMEWORK FOR INFERENCE INSPIRED BY HUMAN MEMORY MECHANISMS
Xiangyu Zeng, Jie L Hu, Ruizheng Huang, Zhicheng Zhang Laboratory of Intelligent Collaborative Computing, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China {zengxy, hupiao, huangrz, zhangzo)@std. westc. en, linjieluest cetpestt.eeuucco
ABSTRACT
How humans and machines make sense of current inputs for relation reasoning and question-answering while putting the perceived information into context of our past memories, has been challenging conundrum in cognitive science and artificial intelligence. Inspired by human brain's memory system and cognitive architectures, we propose a PMI framework that consists of perception, mem- ory and inference components. Notably, the memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience. Through a differentiable competitive write access, current perceptions update working mem- ory, which later merged with long-term memory via outer product associations. reducing information conflicts and averting memory overflow. In the inference module, relevant information is retrieved from two separate memory origins and associatively integrated to attain a more comprehensive and precise interpretation current perceptions. We exploratively apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbl-20k and Cort-of-CLEVE datasets, as well as detecting equilateral triangles, language mod- eling and image classification tasks, and in each case, PMI enhancements consistently outshine their original counterparts significantly. Visualization anal- yses reveal that relational memory consolidation, along with the interaction and integration of information from diverse memory sources, substantially contributes the model effectiveness on inference tasks.
INTRODUCTION
Cognitive science, neuroscience and artificial intelligence (AI) collectively advance grasp of in- telligence, defined as the general mental abilities of perception, memory and reasoning, each with a unique role in human cognition. To construct more human-like intelligent systems, often referred to as the standard model of the mind (Laird et al.) 2017), it is imperative to delve into the interactions among perception, memory and reasoning in unified system. Recently, scholars have uncovered significant flaw in previous deep learning architectures: the absence of dedicated memory module that is critical for long-term information retention and relational reasoning. This drawback becomes evident when considering the constraints of many intelligent systems, which either exclusively con- centrate on perception and reasoning or intricately interweave computation with implicit memory. Therefore, many memory-based studies have emerged, mainly focusing on designing item-based memory models with recurrent neural networks (RNNs) (Hopfield 1982. Hochreiter & Schmid- huber 1997 Dai et al. 2019; Ramsauer et al.) 2020 Schlag et al. 202 and seemoor-augmenter neural networks (MANNs) (Graves et al., 2014 2016a Le et al. 2018 Liang et al. (003)).
Nonetheless, existing approaches expose four limitations: (i) Implicit memory (hidden state) may gradually lose previous information as the model constantly updates its weights to accommodate new inputs, which prevents reusing the precomputed relations in sequential tasks (Vaswani et al.) 2017 Santoro et al. 2017 Devlin et al.; 2018). (ii) The memory system is configured in one of two forms: either as singular memory unit without hierarchical construction multiple separate

---

Published as a conference paper at ICLR 2024
memory components with identical data structures, both of which struggle align with human memory traits and achieve robust generalization (Goyal et al. 2022 Dai et al. 2019; Jaegle et al.
2021 Wu al.] 2022 Kang et al. 2023 Liang et al.) 2023) (ii)) The memory- memory relation
is either crude, expressed as weighted summation via neural networks or dot product attention,
or it undergoes intricate memory transformation algorithms. (Vaswani al. 2017: Santoro al., 2018). (iv) Memory exploitation is confined to rudimentary retrieval, whether it's content-based addressing (uu al. 2020. Goyal et al., 2022; Kang et al.] (023) or explicit address (Graves al., 2016b, Liang et al. (023). Arguably, modern MANNs have yet to develop general architectural frameworks for learning both diverse memory components and how they should interact internally
and externally.
Multiple Memory Systems Theory (MMS) asserts that working memory (WM) and long-term mem- ory (LTM) stand as pivotal subassemblies of human cognitive processes (Atkinson & Shiffrin 1968
Baddeley & Hitch 1974] Eichenbaum & Cohen, 2004). where the former serves to temporarily buffer and process data for current tasks, while the latter is responsible for the retention of relational knowledge and experiences. Additionally, the Global Workspace Theory (GWT) (Baars) 1993 De- haene et al. 2021 suggests communication and coordination scheme, in which disparate cognitive units write information into a shared workspace that is broadcast to all modules, along with the no- tion that write access is restricted.
Inspired by the MMS, GWT and cognitive theories, we assume that optimizing the structure of memory module and its internal and external correspondence mechanisms holds great promise in surmounting the extant restrictions. Accordingly, we hypothesize cognitive framework called PMI that consists of perception, memory and inference modules, wherein memory posited as a dual- layer memory block featuring distinct inner and outer communion principles[] More concretely, structurally, WM exists separately from LTM (especially the relational/declaratiie memory in LTM), with the latter possessing a higher- orgder-order order structure to preserve intricate patterns and relations (Ryan jet al.) 2000 Blumenfeld & Ranganath] 2006). Regarding interactions, there are two exterior proce- dures: perception-bbsed competitive writing and inference-oriented information retrieval, alongside one inner
to establish heterogeneous associations among the two memory units to facilitate efficient information filtering, storage and relational knowledge consolidation. We ap- ply modern different neural network models like Transformers attention based (Vaswani et al. [017]] Brown et al. 2020) and convolutional networks (He et al., 2016), which all equipped with our dual- memory module, to multifarious tasks that may require both WM and LTM: text and visual question- answering, detecting equilateral triangles, language modeling and image classification. Across all these tasks, models integrated with our memory block consistently outperform their original coun- terparts.
METHOD
2.1
OVERVIEW
An overview of our PMI framework is illustrated in Fig. which contains three pivotal compo- nents: perception, memory and inference (both potentially learned). Given an input X (e.g.. text, an image, an audio signal), it is processed through a series of computational stages indexed by to derive the cognitive understanding U the current perception, as outlined below:
1. P component: (Perception) - Convert the incoming input X to an internal feature repre- sentation H PXX).
2. M component: (Memory) Update old memories given the input representation Ht-1; M = M Mt-1).
3. component:
(Inference) Reason (interpret) the current content given the updated mem-
ories: U = I(H*-1, .M).
In this framework, trainable parameters are learned through backpropagation, while memory blocks are updated solely through the feedforward, which constitutes the process of memory precipitation through multiple iterations. A detailed methodological description follows.
"oode available http://gghhii...commmeegyyy//..
2

---

Published as a conference paper at ICLR 2024
in ference
h h'-
x²
P
M
M₁
My
MNK
MNC
M
MRC
>h²
M
I
memories update
out
(a) PMT framework
(b) Memories update and inference
Figure 1: Framework overview and the process of grasping the current input calculation step (a) The memory module consists of WM Mi and LTM M,, each characterized by distinct data structures. (b) WM is updated by current perception via differentiable and constrained write access, which is then integrated into LTM through outer product association. The inference component retrieves pertinent data from both WM and LTM using content-based addressing MHC and MHSC, respectively. Subsequently, through integration steps, it consolidates info from these sources to generate fresh insight into the input, which is used for next rounds of inference or to directly support the decision- decision-making process.
2.2 PERCEPTION
The perceptual operation maps the original input data to intermal entity representations. Focusing on the prevailing models and taking Transformers an example, the text input undergoes embedding and positional encoding to yield initial feature representation 19 e RT'D where T is the sequence length of D dimension. In Vision Transformer (ViT) (Dosovitskiy et al., 2020). a 2D image TR. H Wx xC split into patches Hp E TR.xx(PP.C) each of which linearly embedded. Then positional embeddings are added obtain the final embedding vector by Ieluss : x E: z E; .... 5 E]+ Epos E E ((P2.C)x Epos E ((N++)xD which contains fundamental feature informa- tion of the image, such color and texture, where (H, W) is the resolution of the original image, C' the number of channels and (P, ) is the resolution of each image patch. This process resembles the human perceptual system that receives external information and converts it into understandable internal representations, laying a foundation for subsequent memory and reasoning.
2.3 GLOBAL SHARED DUAL-LEVEL MEMORY
This section provides a detailed exposition of the proposed dual-layer memory module and the in- ternal and external communication mechanisms utilized for its update, as illustrated in Fig. [b We posit that the memory module should be globally shared, including working memory Mus which temporarily stores and processes information required for the current task, and long-term memory Mi for the encoding of higher-order relational knowledge and experiences primarily in its declar- ative/relational memory. We opt for M,, E NxD as a form of WM indexed based on m, slot. Here, N is the number of slots, each with dimension of Dm. While the LTM is represented 3D structure Mj e IR Cxx Dm, with C' denoting memory fragments, which is biologically plausi- ble (Marr& & Thach 1991).
2.3.1 EXTERNAL CHANNEL: -.riteitee
External communication serves to update the contents of WM via two pivotal steps: competitive writing and forgetting, which informed by fundamental aspect of the human memory system our inclination not to retain permanent record of every perception but rather to discerningly preserve essential elements within memory. Collectively, these processes guarantee the storage of the most critical information pertinent to the ongoing task, an indispensable facet in tasks involving reasoning.
Write with competition This process aims to selectively inscribe perceived inputs into M,, with finite capacity, also inspired by Miller's Law, which states that the number of information units that human WM can handle simultancously is limited, often around 7 2. We use multi-heades sparse cross-attention mechanism (MHSC) for this execution, as expressed Eq. [ Cognate to the MH mechanism used in Transformers, but MHSC exhibits two distinctive aspects: (i) it necessitates
3

---

Published as a conference paper at ICLR 2024
separate origins for Q and K and (if) it introduces a sparsity-inducinng operation the attention weight matrix. Specifically, the result of the - 1 step h E RT*h is projected into keys and values, along with M RN* that is projected into queries. The current inputs compete to write through our MHSC, in conjunction with some other operations to yield the intermediate state M The whole formulas are as follows:
8k = softmax
(1)
Vak
M W
(2)
M = LN,,M, + M,
(3)
M W.S ReLU (M,, -1)), iE (E......,
(4)
M 2 M
(5)
It's noteworthy that the input needs be linearly projected to the same dimension D,, as M (following the traditional practice of D = Dm) W WK and WV are weight matrices. 8% E Nx+++)) is the attention weight scores of M and ht-1 Unlike the standard soft competition, top-k softmax (Ke et al., 2018) select fixed number of entities for updating the M st denotes the post-softmax value, please consult Algorithm [ for details. LN, and LN, signify different LayerNorms, employed uphold memory stability over prolonged time steps. ReLU is the ReLU function, MLP; the 2t multilayer perceptron and M w.k through k multilayer perceptrons.
is the intermediate output
Forgetting Memory forgetting entails the climination or reduction of previously stored data to make space for new info, optimizing memory performance. is reasonable adopt the gating mechanism since it emulates the biological memory process and effectively alleviates information conflicts. This implemented in Eq. [6] where I, and F; indicate the input and forget gates respectively. as proposed in RMC (Santoro et al. 2018). Further details can be found in Appendix D.I
M Fe(Ma=l, _h'")) O M + IcM 1 M
(6)
2.3.2 INTERNAL CHANNNL: Mi-Write
The internal channel is utilized to update LTM that boasts a larger capacity to accommodate more relational info. As illustrated in Eq. 7 we conduct an outer product calculation between the updated M and the previous-ste LTM Mr; E CC to merge novel vital info into the current LTM M.. In contrast to scalar product computation that only yields numerical value, the outer product operation (Smolensky 1990 Halford et al., 1998) is used to capture relations and interactions be- tween vectors, which not only enhances higher-order representational capacity but also contributes to information precipitation and memory reinforcement.
M LN3 ((M", @ M + M;
Here, LN3 denotes LayerNorm, and @ signifies the outer product operation.
2.4 INFERENCE
Inference component, guided by the updated memories, provides insights of current perceptions. Our interpretation of the inference is that stems from an assumption on the form of the joint dis- tribution between perceptual inputs and current memories. To mimic human-like ability to focus on crucial details of the ongoing task while leveraging extensive knowledge and experience to navi- gate complex situations, we use content-based addressing MHC that is equivalent to MHSC without sparsity and MHSC to retrieve relevant memories from M and Mf based on current input 1'-1, getting h" and hf respectively, shown in Eq.
8 10
Ut = MHC W
W M W
(8)
C
M C
Σ M
where Mi E NxDm
(9)
U MHSC W N,M, M.W)
(10)
4

---

Published as a conference paper at ICLR 2024
Subsequently, the understanding Uj from LTM serves to further revise and supplement the understanding Uh from WM via the MHC mechanism, where Uj creates queries that match with keys and values from U to generate richer representation U Then a linear combination of U and U is conducted with a hyper-parameter a to yield the final cognition U", as shown in Eq. 12 This process of multiple correlation and fusion of various information sources contributes extract- ing richer and more valuable insights that support higher-level decision-making and reasoning.
U MHC (U/WQ, ,UT (
(11)
U oU, + u)
(12)
RELATED WORK
Cognitive Science In cognitive neuroscience, memory studies endeavor to unravel the intricacies of information storage, organization and retrieval in brains, and their profound impact thinking, cog- nition and behavior, building on the pioneering work of Ebbinghaus (1885) and Bartlett & Bartlett 1932). A Afterwards, Atkinson & Shiffrin (1688) proposed multi-store model including sensory, short-term and L.TM, which contributes to our insights of different memory types and stages. The successor Baddeley & Hitch (9744) further refined and delineated this model by substituting short- term memory with WM- a transient storage that can interact with LTM. Sigma (Rosenbloom et al.) 2016) and Soar (Laird 2019) are canonical cognitive frameworks of recent advancements, both of which employ a similar memory system comprising WM and LTM that play crucial roles in complex reasoning and moblem-ssolvngg tasks. Moreover, the Global Workspace Theory (Baars) 1993) put forward a coordination and collaboration mechanism with restricted write access, which sheds light on the interaction of diverse cognitive components.
Memory networks Semi-parametric MANNs, as form of using implicit knowledge to perform complex reasoning tasks, are a persistent theme in neural network research. Today MANNs typ- ically rely explicit memory and attention mechanisms, with pioneering models like Memory Networks (Weston et al., 2014) and Neural Turing Machines (NTMs) (Graves et al. 2014), both of which are equipped with storage for vector representations accessible via attention. Memory Networks use addressable memory to execute tasks through a series of read operations. In con- trast, NTMs also utilize addressable content storage, but unlike Memory Networks, which pre-load memories using all the inputs, NTMs write and read the memory one input at a time. Following this are Differentiable Neural Computers (DNC) (Graves et al. (016a) and Sparse DNC (Rac al., 2016). which are realized as recurrent neural networks (RNNs) capable of read and write operations on memory over time and are trained via BPTT (Werbos) 1990). A parallel research path involves enhancing RNNs like LSTM by incorporating data structures such as lists, stacks or queues (Joulin & Mikolov 2015) Grefenstette et al. 2015).
Transformers with memory extensions Memory is topic of active exploration in diverse Trans- former studies. Transformer-XI. (Dai et al. (0199) and its successors, Compressive Transformer (Rae) et al. 2019), RMT (Bulatov et al. (022) and Scaling Transformer (Bulatov et al. (023) re-introduce the notion of memory and recurrence by caching self-attention hidden states from each layer into fixed-size queue and reusing them in subsequent attention computations, with the difference that Compressive Transformer utilizes a compression network to further compress its memories into fewer vectors. In addition, various forms of global representations are introduced
model mem- ory that learns to gather information from input sequence tokens. Notable examples of these ap- proaches include Set Transformers (Lee et al. 2019), ETC (Ainslie et al., 2020), Longformer (Belt- agy et al. 2020) and TR+HSW (Goyal et al. 2022), all of which redesign the self-attention mech- anism reduce computational complexity. Memory modules, with their read-write global mem- ory operations, have recently attracted attention for their potential to remember prior information, driving movement towards more structured models. For instance, Memformer (Wu et al., 2020) proposes dedicated external dynamic memory module after the primitive self-attention layer and interacts with it through memory reader and writer components to store previous hidden states in concise representations for efficient sequence modeling. More recently, DT-Mem (Kang al. 2023) introduces a WM that contains N memory slots between the Transformer module and the MLP to store and retrieve information through an attention- approach, where the Transformer module is similar to the GPT-2 (Radford et al. 2019) module without the feedforward layer. Most pertinent to our work, Goyal et al. (0022) taking cues from the GWT theory, replace Transformers' pairwise
5

---

Published as a conference paper at ICLR 2024
interactions with shared workspace featuring constrained write access concept equivalent to our WM that can read and write. While these endeavors are closely related to explicit memories, their memory structures monolithic, which leads to boundaries in representing certain higher- order information or relations. Hence, one takeaway from our work is that it may be prospective revisit previous memory enhancement methods in light of insights from cognitive science into memory structures.
4 EXPERIMENTS
To assess the efficacy of the PMI module in discovering and learning inferring entities and their relations, we conduct a preliminary exploration by incorporating as replacement for the pairwise self-attention layers in Transformers and ViT (Dosovitskiy et al. 2020), where memory components are shared globally. This modified architecture, called PMI-TR, are then applied to a diverse range of tasks, including visual QA, text-based QA, detecting equilateral triangles and language model- ing. Readers can refer to Appendices and for the model hyperparameter settings and detailed descriptions of each task, respectively.
4.1 RELATIONAL. REASONING: SORT-OF-CLEVE
Sort-of-CLEVR (Santoro et al. 2017) is dataset similar CLEVR, designed specifically for search on relational reasoning. Each 2D image in Sort-ofCLEER is of size 75 x 75 and comes with 6 randomly placed geometric shapes of possible colors and 2 possible shapes. There 10 non-relational and 20 relational questions that are equally divided into binary and ternary types per image, along with corresponding answers (details Appendix (.44). Given the bounded answer space, this task is treated as a classification task. Each image is partitioned into a sequence of uni- form patches and then encoded as in ViT. Subsequently, we concatenate the image embedding with its corresponding question embedding as input into our PMI- TR, in line with Goyal et al. (2022).
For this task we evaluated our PMI-TR with the following five baselines: Standard Transformers with shared parameters across layers [TR] (Vaswani et al., 2017), Set transformer [ISAB]: Trans- formers where self-attention is replaced by ISAB module (Lee et al. 2019). Transformers with Shared Workspace with top-k competition [TR+HSW] (Goyal et al. 2022 and High Capacity Trans-
formers [TR+HC]: Standard Transformers with different parameters across layers.
50
46.9
Acaraya
Anorayy 25
IK ICA
19.00 IX:
100
150
200
100
1.00
200
100
150
220
Eaodo
Epocka
Epoco
(a) Unary Accuracy
(b) Binary Accuracy
(c) Ternary Accuracy
Figure 2: Test accuracy vs training iterations for the Sort-of- off-LEVR task.
The accuracy curves over 200 training epochs illustrated in Fig. We observe that Trans- formers equipped with our global shared memory module converges faster compared to all other baselines, demonstrating superior performance on both relational and non-relational tasks. In con- trast, the TR+HSW model excels in addressing non-relational questions but struggles with rela- tional problems. We conjecture this might be because non-relational problems frequently demand the model to tackle only small amounts of information about individual objects. Interestingly. the single-level memory slots the global workspace (similar to our WM) possess the capability to store and process scant present information, allowing them to handle these issues with ease. How- ever, relational questions regularly necessitate multi-step reasoning to obtain an answer, such as extracting object attributes followed by relational analysis. The introduction of LTM enables the model to deposit crucial information during the learning process. Consequently, can retrieve per- tinent knowledge from this memory module in upcoming reasoning steps, going beyond its reliance

---

Published as a conference paper at ICLR 2024
solely the current input. This contributes to more comprehensive understanding and handling of relational questions.
4.2 TEXT-BASED QA BABI
BAbI a pure text-based QA dataset (Weston et al. 2015) that is widely used to assess the ability of MANNs, attention mechanisms and other types of models to remember and son on textual information. This dataset con- tains 20 challenging tasks, each corresponding to a particular type of reasoning, such as logi- cal deduction, counting, pathfinding and induc- tion, all of which possibly require both WM and LTM. Each question is associated with set of supporting facts. For example, the facts "John journeyed to the office" and "John left the milk" support the question "Where is the milk?" (an- swer: "office") (more in Appendix [.2). Fol- lowing Le et al. (202bb), each story prepro- cessed into a sentence-level sequence. which fed into our PMI-TR model the input se-
Table 1: Test error rates: mean + std. (in %) on the 20 bAbl tasks for models trained using 10k examples and best error over 10 runs. is reported from Dehghani et al. (2018)
Error
Model
LSTM (Hochreiter & Schmidhuber
1997
27.3+0.8
25.2
TR'
Vaswani al 2017
22.1
NIA
DNC [Graves et al.
20160
12.814.7
3.8
H-Mem (Linkacher Legensien
2020
10.8
N/A
NUTN LLe et 202031
1.621.9
3.3
MemNet (Do & Principe 2023
NIA
TR+HSW Goyal al. 2022
3.640.46
3.25
PMI-TK (ours)
2.55+0.11
2.32
quence. A model succeeds on a task if its performance surpasses 95%. We compare our model with recent memory networks and report the results in Table I (more in Appendix G.2..
4.3 DETECTING EQUILATERAL TRIANGLES
In this binary classification task, our goal is to de- termine whether 64 x 64 sized image contains an equilateral triangle composed
three randomly placed point clusters (Ahmad & Omohundro 2009).
100
89.8
96.7
97.9
97.1
Test Test Accuracy
60.9
For equilateral triangles, the midpoints of these clus- ters are equidistant from each other. To feed an im- age into our PMI-TR, we adopt the same methodol- ogy employed in VIT (Dosovitskiy et al. 2020). Specifically, each image is divided into equally sized then utilized distinct
50
0
STR
EAB
18 TR-1SSW
PMt TR
PMt PMI-TR.S TR.S
4 4 patches, which input positions for the PMI-TR. In order to make precise judgments, this task requires the model adeptly comprehend and memorize the spatial rela- tions between disparate point clusters, embodying the relative positions and distances among them. By incorporating our PMI module, with shared WM and LTM across all layers, the model can preserve deci- sive info concerning each point cluster for subsequent inference procedures. Moreover, the con- strained capacity of WM compels the model to selectively inscribe crucial information into the memory module, which coincides favorably with the inherent sparsity intrinsic to the task.
Variants Transformers Models
Detecting Equilateral Trian- gles. This figure compares the performance of Transformers with our PMI [PMI-TR] against other Transformer baselines.
Figure 3:
The results in Fig. reveal that PMI-TR outperforms the standard TR model in terms of both convergence speed and accuracy, converging faster (curves available in Appendix G.I and achieving an impressive accuracy of 97.9%, 8.1% improvement. Additionally, baselines, further confirming the efficacy of the PMI module. Here, STR denotes Transformers with sparse factorizations of the attention matrix (Child et al., 2019), [PMI-TR+S] is variant of the PMI-TR without top-k sparsity, and other baselines are detailed in experiment 4.1
approach surpasses other
4.4 LANGUAGE MODELING
To further validate the effectiveness of the proposed approach in seng-neqeence language model- ing, apply PMI-TR-4 decoder-only transformer emhedded with PMI module, to variety
7

---

Published as a conference paper at ICLR 2024
of datasets on both character-level and word-level language modeling to have comparison with state-of-the-aue models, including Enwik8 (Matt 2011), WikiText- 103 (Merity et al. 2016) and PG- 19 (Rac al. 2019).
The experiment on Enwik8 employs a 12-layer PMI-TR (8 heads, 512 hidden size, 2048 in- termediate FF), WikiTeex- 103 uses a 16-layer PMI-TR (10 heads, 410 hidden size, 2100 inter- mediate FF), while PG19 uses a 12-layer PMI- TR (8 heads, 1024 embedding size, 4096 in- termediate FF). The complete hyperparameter settings are available Appendix E The test bits-per-character (BPC) on the Enwiki8 and the perplexity (PPL. on WikiText-1033 and PG- 19 are reported in the Table 3 Table 2 and Ta- ble respectively. Notably, we improve the re- sults of bpc/perplexity to 0.96 on Enwiki8, 16.5
Table 2: Comparison of valid and test perplexity on WikiText- 103 against other models.
Models
Layers Parama Valid
LSTM [Grave et al. 2016
48.7
RMC SSanrr 2015
30.8
31.6
Transfornst
151M
RMT Hulltvv è ml
16
24.85
Compressive Transt. Ral c
2019
18
16
17.1
Transforner XL. Large
18
257M
18.3
TIMS-HSW (Groyal et
2022
8
112M
35.9
16.7
PMI-TX
116M
24.9
23.8
PMI-TR (ours)
16
233M
15.3
16.5
on WikiText- 103 and 31.04 on PG19, which demonstrates the superiority of the PMI architecture. Additional qualitative analysis available in the Appendix
Table 3: The test bhts-per- on Enwik8.
Table 4: The valid and test perplexity on PG-19,
Models
Layers Params
PG19
Models
Layers
Transformer-LL Dai et 2019
12
41M
1,06
Valid PPL Test PPL
Transformer-X. DDa et al 2019
24
277M
0.99
Transfomer-XL et et al 2019
36
45.5
36.3
Compressive Trans.. [Rae
2019
24
0.97
Compressive Transt.
36
43.4
33.6
Sparse Transf. (Child st.
2019
30
95M
0.99
oo-former Martins et alr 2021
12
32.48
Adaplive Transf. Sukhhuular
2019
12
39M
1,02
Routing Transl. (Roy
12
33.2
RMT (Bulatov et al.
12
1,222
TR+HSW (Goyal et
12
39.46
32.46
TIMS+IIW (Goyal al (our))(ours)
2022
12
43M
1.36
PMF-TK (oW5)]
12
37.12
31.04
45M
4.5
MORE EXPLORATIONS OF MEMORY MODULE
4.5.1 MEMORY ATTRIBUTES AND COMMUNICATION MODES
This section delves into qualitative analyses of memory properties and communication modes on bAbl and Sort-of-CLEY tasks. Studies on memory properties seek to investigate how factors like capacity and persistence (global sharing) affect model performance. Experiments on commu- nication modes aim to evaluate the efficacy of competitive writing. as well as the correction and supplementation of LTM-derived data to relevant info from WM. To tackle these questions, we set up three models of distinct sizes PMI-TR, (1 4, h = 4), PMI-TR,,
= 8, 8) and PMI- TR, (1 = 12, h 16), and run them various combinations of N, M and k. where and h are the number of layers and heads in PMI-TR, respectively, considering their critical roles in model performance.
The results are reported in 3 where PMI-TR, w/s denotes PMI-TR without memory sharing among its layers, PMI-TR,, // indicates that info retrieved from LTM is directly aggregated with data from WM via a without correction step, and PMI-TR,,,// represents PMI-TR without WM involvement during inference. seft is a standard soft competition mode, not a top-k strategy. We can derive following key findings. For memory properties, firstly, greater memory capacity doesn't ' necessarily equate better performance. The optimal results are achieved N and M 5, aligning with discoveries in cognitive neuroscience. Secondly, memory persistence markedly im- proves the performance and speed of convergence in relational inference tasks, especially in binary and ternary relations, respectively, by 7.32%, 6.88% over non-globally shared cases (more in Ap- pendix I.II) Notably, independent memory modules result in an eightfold increase trainable parameters. Regarding the communication mode, constrained writing exhibits heightened sensi- tivity in binary and ternary inference tasks, albeit with contrasting effects. We speculate that this divergence may be attributed to the larger volume of info storage required for ternary problems, thus necessitating a slightly larger value. Moreover, the impact of lacking WM is notably less than lacking L.M. Without the guidance of LTM, as erudite scholar, there is minor uptick in
8

---

Published as a conference paper at ICLR 2024
rate for bAbl task under the same setup, and the three types of Sort-of- task exhibit respective decreases of 0.31% (unary), 7.34% (binary) and 4.54% (ternary) in accuracy, underscor- ing the constructive effect of previously accumulated relations and knowledge via outer product relational reasoning.
Table 5: Results of ablation studies on memory properties and communication modes.
bAbl
Sort-of-CLEVR
Model
N
M
Top-k
Params
Err%
Params
Unary%
Binary%
Ternary%
6
2.00M
2.81
2.03M
99.14
77.12
61.29
8
2.27M
2.72
2.29M
99.45
86.06
62.85
PMI-TR,
8
5
7
2.27M
2.73
2.29M
99.50
82.84
64.35
10
2.53M
2.78
2.55M
99.19
80.24
59.48
2.07M
2.61
2.09M
99.40
80.13
65.93
8
5
2.33M
2.55
2.36M
99.34
87.61
62.45
PMI-TR,
8
2.27M
2.57
2.36M
99.19
81.93
60.89
10
2.59M
2.62
2.62M
99.40
80.18
65.83
6
2.20M
2.73
2.22M
99.40
81.92
64.21
8
5
2.46M
2.58
2.49M
99.14
84.73
65.52
PMI-TR;
8
2.46M
2.59
2.49M
99.29
80.68
66.94
10
2.73M
2.71
2.75M
99.09
81.47
65.01
PMI-TR,, vn/0
8
16.48M
2.84
16.5M
99.14
80.29
60.06
PMI-TR,, u1o02
8
2.33M
2.91
2.36M
99.19
79.96
62.40
PMI-TR, wlo3
8
5
5
2.33M
2.64
2.36M
99.26
84.72
61.86
PMI-TR
8
5
soft
2.33M
2.75
2.36M
99.15
79.64
61.87
4.5.2 VISUALIZATIONS OF ATTENTION PATTERNS
To explore whether knowledge accumulates in LTM, we use visualizations of attention patterns between current perceptions and LTM on the bAbl task, shown in Fig. Here, current inputs act as queries, and the LTM matrix serves keys and values for cross- aotss-attntiion computation. As the depth increases, clear trend emerges the heatmaps: more colored regions appear that gradually stabilize and resemble, implying growing correlation between inputs and LTM that evolvingly converges (more explanations Appendix B.2). This may indicate that richer knowledge accumulated in LTM. leading to more consistent grasp of different elements within the input data across these layers.
5 CONCLUSION
Inspired by multiple memory systems and global workspace theory in cognitive neuroscience, we propose the PMI module containing perception, dual-layer memory and inference components, to generate a more comprehensive and accurate understanding current inputs while depositing vital contents into memory cope with more complex situations. We have explored PMT's dual utility: as an alternative to self-attention layers in Transformers and as a complement to convolutional net- works. Extensive experiments on images and texts provide compelling evidence for the effectiveness and adaptability of PMI, meaning could be core component in diverse architectures. We look forward to a broader application of our method, including its integration into various frameworks and its extension to a wide range of tasks across varying modalities.

---

Published as a conference paper at ICLR 2024
Attention patterns between inputs WM
(b) Attention patterns between inputs LTM
Figure 4: Attention patterns between perceptions and memories across different layers of the PMI- TR.
ACKNOWLEDGEMENTS
This work was supported by Sichuan Province Science and Technology Support Program, No.20222HCCG0008.
REFERENCES
Subutai Ahmad and Stephen M. Omohundro. Equilateral triangles: A challenge for connectionist vision. 2009. URL /ttppttss:/app..setea http::///.pp.emmmm..cccccoooo..o wtt.aaawww..ca.a.....oooooopoppa
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs transformers. arXiv preprint arXiv:2004. 08483, 2020.
Richard C Atkinson and Richard M Shiffrin. Human memory: A proposed system and its control processes. In Psychology of learning and motivation, volume 2, pp. 89-195. Elsevier, 1968.
Bernard Baars. A cognitive theory of consciousness. Cambridge University Press, 1993.
Alan D Baddeley and Graham Hitch. Working memory. In Psychology of learning and motivation, volume 8, pp. 47-89. Elsevier, 1974.
Frederic Charles Bartlett and Frederic C Bartlett. Remembering: A study in experimental and social psychology. Cambridge university press, 1932.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 15555 2020.
Robert S Blumenfeld and Charan Ranganath. Dorsolateral prefrontal cortex promotes long-term memory formation through its role working memory organization. Journal of Neuroscience, 26(3):916-92) 2006.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1990 2020.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:1 1099- 2022.
Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. Scaling transformer Im tokens and beyond with rmt. arXiv preprint arXiv:2304. 1062, 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /90/::9944///000 10509, 2019.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 022800, 2019.
10

---

Published as a conference paper at ICLR 2024
Stanislas Dehaene, Hakwan Lau, and Sid Kouider. What is consciousness, and could machines have it? Robotics, AI, and Humanity: Science, Ethics, and Policy, pp. 43- 43-56, 2021.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins arXiv: 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /929, 2020.
Ran Dou and Jose Principe. Universal recurrent event memories for streaming data. In 2023 Inter- national Joint Conference on Neural Networks (IJCNN), pp. 1-10. IEFE, 2023.
Hermann Ebbinghaus. Über das gedächinis: untersuchungen zur experimentellen psychologie. Duncker & Humblot, 1885.
Howard Eichenbaum and Neal J Cohen. From conditioning to conscious recollection:
Memory
systems of the brain. Number 35. Oxford Psychology, 2004.
Anirudh Goyal, Aniket Rajiv Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Curtis Mozer, and Yoshua Bengio. Co- ordination among neural modules through shared global workspace. In International Confer- ence on Learning Representations, 2022. URL https:://ppeneee// XzTtHjgPDST.
net net/forum?id
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with continuous cache. arXiv preprint arXiv:16/2.04426, 2016.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv: 1410.5401, 2014.
Alex Graves, Greg Wayne. Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agmieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471-476 2016a.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tom Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471-477 2016b.
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. Advances neural information processing systems, 28, 2015.
Graeme S Halford, William Wilson, and Steven Phillips. Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and brain sciences, 21(6):803-88) -310, 1998.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8)2555-225 2558.. 1982.
11

---

Published as a conference paper at ICLR 2024
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Interaational conference on machine learning, pp. 4651 1664.. PMLR, 2021.
Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in neural information processing systems, 28, 2015.
Jikun Kang, Romain Laroche, Xindi Yuan, Adam Trischler, Xue Liu, and Jie Fu. Think before you Decision transformers with internal working memory. arXiv preprint arXiv:2305.16336
2023.
Nan Rosemary Ke, Anirudh Goyal ALIAS PARTH GOYAL, Olexa Bilaniuk, Jonathan Binas, Michael C Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through reminding. Advances in neural information processing systems, 31, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4/2.6980, 2014.
John E Laird. The Soar cognitive architecture. MIT press, 2019.
John Laird, Christian Lebiere, and Paul S Rosenbloom. A standard model of the mind: Toward a common computational framework across artificial intelligence, cognitive science, neuroscience, and robotics. Ai Magazine, 38(4):13-26, 2017.
Hung Le, Truyen Tran, and Svetha Venkatesh. Dual memory neural computer for asynchronous two-view sequential learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1637-1645, 2018.
Hung Le, Truyen Tran, and Svetha Venkatesh. Neural stored proogapp memory. In International Conference on Learning Representations, 2020a.
Hung Le, Truyen Tran, and Svetha Venkatesh. Self-attentive associative memory. In International Conference on Machine Learning, pp. 5682-5691. PMLR, 2020b.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In Interna- tional conference on machine learning, pp. 3744-3753. PMLR, 2019,
Yao Liang, Hongjian Fang, Yi Zeng, and Feifei Zhao. A brain-inspired memory transformation based differentiable neural computer for reasoning-based question answering. arXiv preprint arXiv:2301.02809 2023.
Thomas Limbacher and Robert Legenstein. H-mem: Harnessing synaptic plasticity with hebbian memory networks. Advances in Neural Information Processing Systems, 33:21627-21637, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
arXiv preprint
1rXiv:1711.05101, 2017.
David Marr and W Thomas Thach. A theory of cerebellar cortex. From the Retina to the Neocortex: Selected Papers of David Marr, pp. 11-50, 1991.
Pedro Henrique Martins, Zita Marinho, and André F. Martins. co-forme:: Infinite memory trans- former. CoRR, abs/2109.00301. 2021. URL
00301
Mahoney Matt. Large text compression benchmark, 2011.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint
2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Jack Rae, Jonathan Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. Scaling memory- augmented neural networks with sparse reads and writes. Advances in Neural Information Processing Systems, 29, 2016.
12

---

Published as a conference paper at ICLR 2024
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.055000 2019.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprini arXiv:2008.00 02217, 2020.
Paul S Rosenbloom, Abram Demski, and Volkan Ustun. The sigma cognitive architecture and system: Towards functionally elegant grand unification. Journal of Artificial General Intelligence, 7(1):1, 2016.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguis- fics, 9:53-68, 2021.
Jennifer D Ryan. Robert R Althoff, Stephen Whitlow, and Neal J Cohen. Amnesia is a deficit in relational memory, Psychological science, 1166):454461, 2000.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Ad- vances in neural information processing systems, 30, 2017.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. Advances in neural information processing systems, 31, 2018
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative infer- ence using fast weight memory. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL. https:
//open wwwtt/ort// dtteefforrmm2.ddTuueaaaaadd
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46 46(1-2):159 2-16, 1990.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv: eetiii005999999 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Paul Werbos. Backpropagation through time: what does and how to do it. Proceedings of the IEEE, 78 18(10):550-1156) 1990.
Jason Weston, Sumit Chopra, and Antoine Bordes. 1rXIv:1410.3916, 2014.
Memory networks.
arXiv preprint
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint /50.i::500..0000 (502.55699 2015.
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A
transformer for sequence modeling. arXiv preprint arXiv:2010.0689,,
2020.
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing trans- formers. In The Tenth International Conference on Learning Representations, ICLR 2022, Vir- tual Event, April 25-29, 2022. OpenReview.net, 2022. URL httpps://ppenreeeeee forum?
d= f Ron
13

---

Published as a conference paper at ICLR 2024
Appendix
A
PSEUDO CODES
Algorithm 1:
PMI-TR Algorithm
Notation:
Given an original imput sequence of length T' with embedding dimension of for the proposed model. Let h' denotes the output of the 1th layer. The working memory is represented as a matrix M E IR D with distinet compartmentalized memories for each row, where M is the state of slot (the total number of slots is
N). The long-term memory is represented as 3D matrix M1 E IR R N Dm Dm,, where C is the number of memory segments. Initialization: Convert the raw input X x
hgg = Px. e, clstakens where ho E TR.TT P> signifies the positional encoding, er corresponds to the embedding of X and clstoken the classification head. Initialize the working memory matrix Mir and the long-term memory matrix M,, respectively, guaranteeing their universal sharing across all layers encompassed within our framework. Input to the layer I: h'-l having shape IR TxD
Step 1: The current perceptions compete with each other be selected to update the M
Q -!-1wa
5% so ftmax QQ(h' (Q("""**))) where 9ą E RNXT
Vdk
Construct a containing the indices of the k selected specialists with the top-k largest
values of 8g-
[(,,))
E Ft, L & Fr
for all N.....,,N,t. (I....,
0,
Step 2: External communication: Mw-Writt
residual = M
M' LN residual)
My. = ReLU (ML.,(M ie .......,,,, k signifies the number of layers
the MLP.
M LN2(M w. residual)
M -F(MM- " M
о tanh
refers the forget gate and I represents the input gate of tth layer.
Step 3: Internal communication: M:-Write with the updated M.r
M LN ((MM) & (
Step 4: Make sense of the current perceptions based the updated M., and Mi. Read and M,-Read
N²' (N-'W"(M! TP*
h = softmax
M W"
Vath
M 1 DE Mf where M Nx
= softmax hb' WW
Vak
h = = where T M.W
Similar to s% et represents the outcome of the top-k largest sparse attention scores associated with Ck-
h MHC (H/W9, f6 W.K Ny V
ht' ahi, + (1 - a)hh,
14

---

Published as a conference paper at ICLR 2024
B EXPERIMENTS ON MEMORY CONSOLIDATION
B.1 MORE ABLATION RESULTS
In this section, we present the results of ablation experiments concerning the global sharing of mem- ory components. The plot in Fig. 5 iillustrates the change in testing accuracy over training iterations. Here, the PMI-TR model denotes the usage of globally shared working memory module, while PMI-TR../ signifies the absence of global sharing. TR represents the vanilla Transformers model. The results demonstrate several key findings. Firstly, across all three tasks, both our PMI-TR and PMI-TRw//o models exhibit faster convergence compared to the traditional TR model. Secondly, when compared to the baselines TR and PMI-TRw/o- PMI-TR achieves the highest accuracy and convergence rate, particularly in complex binary and ternary relation tasks, showcasing the effec- tiveness of the adopted global sharing strategy in complex relational reasoning tasks. However, for non-relational tasks, their performance remains relatively on par, potentially due to the fact that non- relational tasks often emphasize extracting direct patterns and information from input data, which coincidentally aligns with the effective capture of these correlations by the self-attention mechanism in Transformers.
Untry Accuracy
Tenury Accuney
M
80-
Nucurac 5t
Axxrtay
Pr
52-
30
100
150
200
100
150
200
100 Ppochh
1.50
200
Pyoce
Ppocks
(a) Unary Accuracy
(b) Binary Accuracy
(c) Ternary Accuracy
Figure 5: Test accuracy vs. training iterations for the Cort-of-CLEVE task. Results of the ablation experiments on the memory module's global sharing or persistence.
B.2 VISUAALIZATION
We present four Q&& instances from the bAbl dataset, illustrating the attention matrix of perception towards both working memory (Fig. (6a) and long-term memory (Fig. (6b during the reasoning process within eight layers of the PMI-TR model. Each row corresponds to distinct Q&A examples, with columns representing the layers of our PMI-TR. Moreover, in each heatmap, the vertical axis is representation of perceptual information, functioning as queries, while the horizontal axis refers to long-term memory or working memory, serving as keys in the cross-attention mechanism.
As illustration, let's consider a specific problem with its set of stories and corresponding answer. In the first line of the working memory attention patterns, each Si represents a distinct sentence in the narrative. For instance, S1 corresponds to "the hallway is east of the bathroom', and S2 corresponds to 'the bedroom is west of the bathroom", and so forth until Smaz, where the maximum Smaz for each task may vary (in task 1, Smaz is S10). In cases where there are fewer than Smaz story sentences, placeholder sentences (all zeros) are used as filler input. The letter 'Q' represents the question: 'What is the bathroom east of?', and 'CLS' the classification header.
Firstly, it is well-known fact that the magnitude of attention weights is reflected through color intensity, where the deeper the color, the higher the weight. Figure 6a illustrates attention patterns between perception and working memory across various layers of PMI-TR for four instances from the bAbl dataset. During the inference process, with an increase in layers within the PMI-TR model, clear pattern emerges in the heatmaps: the presence of colored blocks markedly diminishes, yet the intensity of color within the remaining blocks steadily deepens. This implies a growing fo- cus of working memory essential segments of current inputs, especially those indispensable for This result consistent with the restrained write access elucidated Section 1 which encourages working memory concentrate the utmost critical elements of the task.
15

---

Published as a conference paper at ICLR 2024
Figure 6b depicts attention patterns between perception and long-term memory across different lay- ers. For each problem (per row), as layers increase (from left to right), the heat maps show clear trend opposite to the working memory pattern, with more and more colored areas but less inten- sity, and the last several heat maps tend to be stable and approximate. This suggests that long-term memory gradually accumulates more knowledge moderate correlation with a broader input area (unlike working memory with limited capacity, which focuses only on the most important info). Similar attention patterns in the final layers imply conver- gence of the firre-dimensional long- term memory matrix within finite calculation steps, resulting in a more consistent understanding of various input aspects.
computation step increases, and thus exhibits
(a) Attention patterns between perception and working memory across different layers of the PMI-TR for examples from the bAbl dataset.
(b) Attention patterns between perception and long-term memory across different layers of the PMI-TR for examples from the bAbl dataset.
Figure 6: Attention patterns between perception and memory.
16

---

Published as a conference paper at ICLR 2024
B.3 IMAGE CLASSIHICATION CIFAR-10
In order to establish the generality of the PMI framework, we extend its application beyond PMI- TR mentioned above to include convolutional series model and evaluate its performance on the CIFAR-10 dataset. CIFAR- 10 is benchmark image dataset commonly used in the field of computer vision, which consists of 50k training and 10k test images of resolution classes. Specifically, the original images, after four convolutional layers, serve as perceptions into our PMI module to obtain understandings, which then undergo linear and softmax transformations to yield final classification results.
x 32 with total of 10
The performance of the best models on test sets is reported in Table 6, where CNN_ w/o refers to CNN without guidance from LTM. It's obvious that both PMI-TR and CNN_PM, models exhibit superior performance, achieving accuracies of 79.12% and 78.69% in CIFAR-10, respec- tively, with an improvement of 2.94% (compared to TR) and 0.08% (compared to CNN MLP). These results further underscore the universality of our PMI module.
Table 6: Results of different models on CIFAR-10.
Trans.
Conv.
Models
VIT
ISAB
TR+HSW
PMI-TR (ours)
CNN_MLP
CNN PMI (ours)
CNN PMI (ours)
Acc (%)
76.18
76.39
76.28
79.12
78.61
78.69
78.63
Params (M)
0.75
2.21
2.01
2.0
0.11
1.75
1.68
C QUALITATIVE RESULTS
For qualitative analysis working memory and long-term memory in long-term language mod- eling, we illustrate an instance of the attention histogram between different memory types and the current input sequence during the inference stage on the WikiText-103 dataset when predicting the next token. The attention histogram for long-term memory in Figure highhlihht its ability focus on more long-range and relevant information, significantly exceeding the span of working memory depicted in Figure
Attention Weight
50
100
150
200
750
Input Sequence Position
Figure 7: Attention histogram of long-term memory to the current input sequence during inference in PMI-TR, when predicting the ground truth word "Japanese". The words in the LTM which receive higher attention (>0.5) are shaded.
Ground Truth: Although they had lost contact during the night the Americans did find the Japanese
Shortages of aircraft and serviceability problems greatly retarded pilot train- ing and the ships only had total of 17 D4Ys and 18 <unk>on hand on October
of these only 6 and 16 were operational The Japanese plan for the defence of the Philippines envisioned that the surviving carriers would be used to lure the American carrier forces away from the invasion area to position where the carriers could be attacked by land @-@ based aircraft and the transports by the rest of the IJN
respectively
The other carrier air groups
not in much better
shape and
17

---

Published as a conference paper at ICLR 2024
Attention Weight
50
100
150
200
250
Input Sequence Position
Figure 8: Attention histogram of working memory the current input sequence during inference in PMI-TR, when predicting the ground truth word "Japanese". The words in the WM which receive higher attention (>0.5) are shaded.
The assigned to the Northern Force under the command of Vice Admiral Jisaburó Ozawa and the sisters sailed from Yashima on 20 October On the morning of 24 October the bulk of the few aircraft aboard were launched to attack the American carriers as a distraction They inflicted no damage and caused the Americans to search in the direction from which they had attacked The Americans finally spotted the Japanese carriers 16: 40, some 200 miles (20 km east of Cape Engaño the northeastern
the Japanese decided retain the aireraft ashore for against the American carriers
Fourth Carrier Division
tip of Luzon The American carriers were spread out and was very late the day launch an airstrike so Admiral William Halsey, commander of the Third Fleet decided to mass his carriers position to attack the following momiig.... Ozawa reversed course during the night, correctly believing that the Americans would follow him north.
D IMPLEMENTATION DETAILS
D.1 GATING MECHANISM
To update the working memory, employ the gating mechanism introduced by Santoro et al. 2018 which consists of input gate and a forget gate. Let X [X- [X--1xX
X1 X
g fxxx represent the perceptual input, and M MM,,, Bxxx P.,. represent the previous and
updated working memory, respectively. M is the intermediate result described by Eq. 5 in Sec-
tion 2.3.I The gating mechanism can be formulated as follows.
1 X T Σ relu X, x W
K = X tanh (M x WFF
I sigmoid + +b:)
F, sigmoid + by)
M tanh + M
Here, 4e and F. represent the input and forget gates of the current calculation step with corre- sponding weight matrices W and F.F The biases for the input and forget gates are denoted as b, and by. In practice, we b 0, by
Dm
E HYPERPARAMETERS SETTING
The hyperparameter settings of the PMI-TR model on all tasks are shown in Table 7 and Table g where Adam and A were proposed by (Kingma & Ba 2014) and (Loshchilov & Hutter
18

---

Published as a conference paper at ICLR 2024
2011 7
respectively. The other baseline models remain the same configuration for the corresponding
tasks.
Table 7: The hyperparameter setting of PMI-TR model on four tasks.
Tasks
Parameters
bAbl
Sort-of- CLEVR
Triangle
Cifar-10
Top-k
5
5
Number of layers
8
4
2
4
Number of attention heads
4
Embedding dimensions
256
256
128
256
Optimizer
Adam
Adam
Adam
Adam
Weight decay
N/A
N/A
N/A
0.09
Learning rate
0.0002
0.0001
0.0001
0.0002
Batch size Inp Dropout
64
64
100
64
0.1
0.1
0.1
0.1
Seed
1
Number of working memory slots (N)
8
8
8
Number of long-term memory segments (M)
5
5
5
Size of each working memory slot (Dm)
256
256
256
Size of each long-term memory segment
5 x 256
x 256
5
256
5 x 256
Number of MLP layers in attention
5
4
Memory attention heads
8
1
1
Gate style
'unit'
'unit'
'unit'
'unit'
Initial value
0.7
0.75
0.7
0.55
Table 8: The hyperparameter setting of PMI-TR model on three language modeling tasks.
Tasks
Parameters
Enwik8
WikiText-103
PG-19
Top-k
5
5
Number of layers
12
16
12
Number of attention heads
8
10
8
Embedding dimensions
512
410
1024
Optimizer
Adam
Adam
Adam
Weight decay
0.5
0.5
0.5
Learning rate
0.00025
0.00025
0.00025
Batch size Inp Dropout
64
64
64
0.1
0.1
0.1
Seed
Number of working memory slots (N)
8
8
8
Number of long-term memory segments (M)
5
5
Size of each working memory slot (Dm)
512
410
1024
Size of each long-term memory segment
5 512
x 410
5 x 1024
Number of MLP layers in attention
4
4
5
Memory attention heads
8
Gate style
'unit'
'unit'
'unit'
Initial value
0.7
0.7
0.7
19

---

Published as a conference paper at ICLR 2024
F DESCRIPTION OF THE TASKS/DATASETS
F.1 ENWIKB&&WKKITXX--0044440
Enwiki8 Matt 201 is utilized for character-level language modeling and comprises 100M bytes of unprocessed Wikipedia text, where the first 90MB for training. 5MB for validation and the latter SMB for testing. Both WikiText-103 (Merity et al. 2016) and PG-19 (Rae al. 2019) bench- marks for word-level language modeling with long-term dependency. with the former containing 103M tokens from 28K English Wikipedia articles and the latter from English books published before 1919.
F.2 BABI
The bAbl-20k dataset consists of 20 distinct text-based QA tasks, each presenting unique rea- soning challenge, including counting, deduction and induction. Each task is divided into training, validation and test datasets, with 9k, 1k and 1k questions respectively. They presented in the form of short stories or text passages, comprising narratives, questions, answers and supporting facts. Narratives introduces entities, actions and contextual info relevant the question, and the answer is substantiated by facts from the narratives. Four detailed examples are shown below.
Task 1: Single Supporting Fact
Task 8: Lists/Sets
1. John travelled to the hallway.
1. Mary got the milk there.
2. Mary journeyed to the bathroom.
2. John moved to the bedroom.
3. Daniel went back to the bathroom.
3. John picked up the football there.
4. John moved to the bedroom.
4. John journeyed to the bathroom.
Where is Mary? A: bathroom S:
What is John carrying? A: football S:
Task 16: Basic Induction
Task 12: Conjunction
1. Julius is lion.
2. Lily is rhino.
1. Mary and Daniel travelled to the bathroom.
3. Bernhard is a swan.
4. Lily is white.
2. John and Daniel travelled to the office.
5. Bernhard is green.
6. Greg is rhino.
3. Sandra and Daniel moved to the kitchen.
7. Greg is gray.
8. Julius is white.
4. Sandra and John moved to the garden.
9. Brian is lion.
Where is Sandra? A: garden S:4
What color is Brian? A: white
S: 9 1
F.3 DETECTING EQUILATERAL TRIANGLES
(a)
(b)
(c)
(d)
Figure 9: An illustration of the equilateral triangle detection task. Here, (a) and (b) refer to equilateral triangle, (c) and (d) refer to equilateral triangle.
20

---

Published as a conference paper at ICLR 2024
F.4 SORT-OF-CLEVR
Ternary questions
Q: How many objects are in the rectangle formed by the centers of the and yellow objects?
Answer: 1
Q: Are there any objects on the line formed by the centers of the red and grey objects?
Answer: yes
Q: How many objects form an obtuse triangle with a blue object and a yellow object?
Answer: 3
Binary questions:
Q: What is the shape of the object that is furthest from the green object?
Answer: orange
Q: What is the color of the object that is closest to the red object?
Answer: green
Q: How many objects have the shape of the yellow object?
Answer: 3
Unary questions:
Q: What is the shape of the red object?
Answer: circle
Q: Is the green object on the left or right of the image?
Answer: left
Figure 4: An instance from the sort-of-clevr dataset.
Q: Is the grey object on the top or bottom of the image?
Answer: top
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 TEST CURVES FOR DETECTING EQUIVALENT TRIANGLES
As shown in Fig. 10 without top-k sparsity, show significantly faster convergence compared other models. is worth noting that the PMI-TR model achieves slightly higher accuracy compared to the PMI-TR+S model, highlighting the usefulness of our competitive writing mechanism.
we observe that our proposed models, PMI-TR and its variant, PMI-TR+S
100
97.9
Accuracy %
70
60
50
PATI-TRUS
100
150
200
Epochs
Figure 10: Comparison of convergence speeds on equilateral triangle detection task.
G.2 BABI DETAILED RESULTS Detailed results are shown in the Table 9
21

---

Published as a conference paper at ICLR 2024
Task
run1
run2
run3
run4
run5
run6
run7
run8
run9
run
Meanastd
1:Single Supporting Fact
0
0
0
0
0
0
T:wwo Supporting Facts
0.21
0.08
0
0.10
0.52
0.56
0.73
0.35
0.62
0.32+0.28
3:Three Supporting Facts
1.35
0.35
0.03
0.15
0.56
1.88
0.92
0.04
0.15
1.19
0.66+0.64
4:Two Arg. Relations
0
0
0
0
0.21
0
0
0.31
0
0.05+0.11
5:Three Arg. Relations
0.52
0.46
0.52
0.73
0.52
0.52
0.73
0.73
0.83
0.42
0.60+0.14
6: Yes/No Questions
0
0
0
0
0
0
0
7:Counting
0.31
0.25
0.04
0.73
0.35
0.35
0.35
0.46
0.15
0.04
0.30+0.21
8:List//Sets
0
0
0.21
0.21
0.31
0.10
0.52
0.10
0.10
0.21
0.18+0.16
9:Simple Negation
0
0
0
0
0
0
0
10:Indefinite Knowledge
0
0.31
0.10
0
0
0.10
0
0.05+0.10
11:Basic Coreference
0
0
0
0
0
0
0
0
0
12:Conjunction
0
0
0
0
0
0
13:Compound Coref.
0
0
0
0
0
0
0
14:Time Reasoning
0
0
0
0.10
0.01+0.03
15:Basic Deduction
0
0
0.21
0
0
0
0
0
0
0.02+0.07
16:Basic Induction
48.21
47.12
44.96
47.92
48.04
48.63
47.35
50.81
47.27
48.56
47.89+1.47
17:Positional Reasoning
0.94
0.03
0.03
0.31
0.58
1.67
0.83
0.60
0.77
0.38
0.61#0.49
18:Size Reasoning
0.42
0.01
0.00
0.52
0.21
0.21
0.21
0.31
0.52
0.21
0.2640.18
19:Path Finding
0.00
0
0
0.21
0
0.10
0
0.10
0
0.04+0.07
20:Agent's Motivations
0
0
0
0
0
Average
2.60
2.42
2.32
2.55
2.53
2.70
2.58
2.69
2.53
2.58
2.55+0.11
Failed t=5k())%)
Table 9: Results from 10 test runs of the PMI-TR model 20 bAbl tasks, each consisting of 10k samples, following 200 epochs of joint training. Bold denotes the best run.
22


Reviews:
[{"review_id": "jsZQpSJU2X", "reviewer": "Reviewer_SKET", "summary": "Inspired by human brain’s memory system and cognitive architectures,this paper propose a PMI framework that consists of perception, memory and inference components. Notably, the memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain more accumulated knowledge and experiences. \n\n\nIn my opinion, the motivation of this paper is meaningful because it comes from the human brain's memory. \nAnd the proposed memory module looks like powerful because it consists of working memory and long-term memory.\nHowever, the experiments may not be enough due to it not compare with other memory augment models, such as the memory augment language model. and this paper not take experiments on language generative task.\nBesides, the", "questions": "No", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is sometimes novel and comes from human's brain memory. \n\n2. The proposed model is meaningful with its novel motivation\n\n3. The paper is well written, and the image is easy to understand.", "weaknesses": "1. the experiments may not be enough to compare it with other memory-assisted language model\n\n2. The experiments is hard to understand, and i think it is not necessary to conduct experiments on image classification. And there is little work on the memory augment image model due to the image's too long context. \n\n3. I don't see any connection between your work and the title, the author maybe need change a title due to this model hard to help us underanding AI .\n\nI hope the author takes more experiments on the language model to solve the longer context challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1mg7nWFHQz", "reviewer": "Reviewer_AVcH", "summary": "The paper proposes a cognitive framework called PMI that consists of perception, memory, and reasoning modules. It is inspired by human memory mechanisms and aims to improve the understanding and handling of relational questions in AI systems. The memory module includes working memory (WM) and long-term memory (LTM), with LTM having a higher-order structure to retain accumulated knowledge. Current perceptions update WM through competitive write access and are merged with LTM via outer product associations. The inference module retrieves relevant information from both WM and LTM to generate comprehensive insights. The PMI enhancements consistently outperform their original counterparts in tasks such as question-answering, relation calculation, and image classification.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Integration of cognitive science and AI: The paper draws inspiration from multiple memory systems theory and global workspace theory in cognitive neuroscience, and applies these insights to develop the PMI framework for AI systems.\n\n- Novel memory module: The PMI framework introduces a dual-layer memory block with distinct communion principles, featuring working memory (WM) and long-term memory (LTM). This structure allows for efficient information filtering, storage, and knowledge consolidation.\n\n- Enhanced performance: The PMI enhancements consistently outperform their original counterparts in various tasks such as question-answering, and image classification. This demonstrates the effectiveness of the proposed framework in improving AI systems' understanding and reasoning abilities.\n\n- Clear experimental results: The paper provides detailed experimental results, including accuracy rates and convergence rates, to support the effectiveness of the PMI module. Visualizations of attention patterns further illustrate the model's ability to consolidate and integrate information from different memory sources.\n\n- Reproducibility: The authors plan to share their code once the review process is completed, ensuring the reproducibility of their experiments and allowing for further research and development in this area.", "weaknesses": "- The text appears to be excessively embellished. I would like to encourage the author to employ conventional terminology, as exemplified by the authors referencing \"relation calculation\" in the abstract.\n\n- The paper includes visualizations of attention patterns between perceptions and memories, but it could benefit from providing more detailed explanations and interpretations of these visualizations. \n\n- Examining the qualitative impact of your modules on various types of tasks would provide valuable insights, rather than solely relying on quantitative results. This approach would enhance the paper's overall credibility. You can achieve this by employing various visualization techniques and similar methods.\n\n\n\n**Additional Feedback and Future Experiments:**\n\n- Enhance the clarity of the text to facilitate a deeper comprehension of the paper. Despite grammatical accuracy, the writing occasionally comes across as artificial\n\n- Incorporate additional visualizations to facilitate a clearer and more easily comprehensible text.\n\n- Consider expanding the scope of this research to include more memory-based and cognition-inspired tasks. You may find the paper titled \"Decoding the Enigma: Benchmarking Humans and AIs on the Many Facets of Working Memory\" by Sikarwar et al. to be a relevant reference in this context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6oVKNZTdX", "reviewer": "Reviewer_4HGz", "summary": "This work presents a novel architecture inspired by memory systems in cognitive science. The method improves performance across multiple reasoning tasks in both transformer and CNN architectures.", "questions": "- In what sense does the 'longterm' memory module involve long term memory more than the 'working' memory module? They both seem to operate over the same timescale, the only difference being the presence of the tensor product to capture relational interactions (which is not related to longterm vs. working memory).\n- Is it possible to ablate the 'working' memory module?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed architecture improves performance across a diverse set of reasoning tasks.\n- A reasonable set of baseline comparisons are included.\n- An ablation study is performed to assess the impact of specific components.", "weaknesses": "- The primary limitation concerns the framing of the architecture as instantiating both working memory and longterm memory. It is not clear to me that the architecture actually involves longterm memory in any meaningful sense. I think the approach would be better described as a form of relational working memory (utilizing a tensor product to capture relational information). This of course doesn't concern the method itself, which seems to perform well across multiple tasks. But I think the contribution would be much more clearly framed as a kind of working memory that exploits *relational* information. The role of relations in working memory is very well-studied in cognitive science (see references below), and I think this would make an interesting topic for discussion.\n- Is it possible to study an ablation model that includes the 'longterm' memory component but not the 'working' memory? It seems likely that the tensor product in the longterm memory component is primarily driving the gain in performance, and it would be nice if this could be isolated.\n- It would be good to cite work from cognitive science on the role of tensor product representations in working memory [1,2] as this is highly related to the outer product mechanism in the 'longterm' memory module.\n\n[1] Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2), 159-216.\n\n[2] Halford, G. S., Wilson, W. H., & Phillips, S. (1998). Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and brain sciences, 21(6), 803-831.\n\nMinor comments:\n- It sounds like what is referred to as the transformer baseline in this work is actually a 'universal transformer' [3] in which parameters are shared across layers, and what is referred to as a 'high capacity transformer' is just a standard transformer (in which each layer has different parameters).\n\n[3] Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018). Universal transformers. arXiv preprint arXiv:1807.03819.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)