Coordination Among Neural Modules Through a Shared Global Workspace
Authors: Anirudh Goyal
Aniket Rajiv Didolkar
Alex Lamb
Kartikeya Badola
Nan Rosemary Ke
Nasim Rahaman
Jonathan Binas
Charles Blundell
Michael Curtis Mozer
Yoshua Bengio
Venue: ICLR 2022 Oral (2022)
Keywords: slot based recurrent architectures
attention
transformers
latent bottleneck.

Abstract:  Deep learning has seen a movement away from representing examples with a monolithic hidden state towards a richly structured state. For example, Transformers segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions and object-centric architectures make use of graph neural networks to model interactions among entities.  We consider how to improve on pairwise interactions in terms of global coordination and a coherent, integrated representation that can be used for downstream tasks. In cognitive science, a global workspace architecture has been proposed in which functionally  specialized  components share information through a common, bandwidth-limited communication channel. We explore the use of such a communication channel in the context of deep learning for modeling the structure of complex environments. The proposed method includes a shared workspace through which communication among different specialist modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have  a rational basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise  independent specialists.


Published as a conference paper at ICLR 2022.
COORDINATION AMONG THROUGH A SHARED GLOBAL WORKSPACE
NEURAL
MODULES
Anirudh Goyal Aniket Didolkar', Alex Lamb S, Kartikkya Badola 6, Nan Rosemary Ke Nasim Rahaman 1.3 Jonnthhnn Binas Charles Blundell ?, Michael Mozer 4, Yoshua Bengio
ABSTRACT
Deep learning has movement away from representing examples with monolithic hidden state towards richly structured state. For example, Trans- formers segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions and object-centric architectures make use of graph neural networks to model interactions among entities. We consider how to improve on pairwise interactions in terms of global coordina- tion and a coherent, integrated representation that can be used for downstream tasks. In cognitive science, global workspace architecture has been proposed in which functionally specialized components share information through a common, handwidth-limite communication channel. We explore the of such communi- cation channel in the context of deep learning for modeling the structure of complex environments. The proposed method includes shared workspace through which communication among different specialist modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have a rational basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise independent specialists.
INTRODUCTION
Deep Learning has seen a movement towards more structured models with cleaner separation between different pieces of information often handled by different components. The induced structure, and separation of knowledge has im- proved generalization, model-size scaling, and long-range dependencies (Berner et al., 2019; Vinyals al. 2019 Brown al. 20200. This opens up questions about how to achieve coher- ence and coordination between different com- ponents in such architectures. Looking back to the 1980s, the focus in AI was much less on learning and more constructing articulated, multi-component architectures and examining how intelligence might emerge from interactions among this collection of simple, functionally specialized components (Fodor) 1983] Braiten- berg] 1986. Minsky 1988) [1ook)]19991 Each
A
Figure 1: Step 1: ensemble of specialist mod- ules doing their own default processing; at a par- ticular computational stage, depending upon the input, a subset of the specialists becomes active. Step the active specialists get to write informa- tion in a shared global workspace. Step 3: the contents of the workspace are broadcast all spe- cialists.
of these specialist modules is on the scale of typical component of a computer program, like subroutine that implements a narrow, prespecified function from certain input contents to certain
Mila, University of Montreal, 2 Google Deepmind, 3 Max Planck Institute Germany, Google Research, Brain Team, Microsoft Research, New York, NY, Indian Institute of Technology, Delhi, Corresponding authors: ani ruchgoyal 91 i com

---

Published as a conference paper at ICLR 2022.
output contents] Through appropriate communication and coordination, set of specialists can achieve complex, dynamic, and flexible behavior patterns.
As concrete illustration, consider the task of driving car in terms of specialists. One specialist might monitor the position of the car with respect to lines on the road, and another specialist might adjust the steering direction based the perceptual data. In addition, there might be specialists which provide alerts when certain events occur, such as loud sounds, reaching a critical intersection on route, or coming into close proximity the car in front. To execute the task of driving the car properly, all these specialists need to interact coherently and broadcast their individual information to each other.
Arguably, modern ML and AI has yet to develop broad architectural frameworks for learning both the specialist modules and how they should interact, while the classical view lacks an articulate story about how learning could take place successfully in such frameworks. In this article, we revisit this classical view with modern machine learning tools based on end-to-end learning and differentiable memory and attention mechanisms. Inspired by the Global Workspace Theory (Baars)
1993 Dehaene et al. 1998, [Shanahan and Baars 2005| [hhanhhan 2006, 2010 2012 Dehaene et al.
2017) from cognitive neuroscience, we argue that more flexibility and generalization emerge through
an architecture of specialists if their training encourages them to communicate effectively with one another via the bottleneck of shared workspace (Figure. D
Distributed specialist modules. From a computational perspective, articulated multi-comppnent
architectures composed of sparsely interacting specialist modules show desirable scaling properties (e.g., more specialists can seamlessly be added), increased robustness (the system can tolerate the removal of or changes in individual specialists), and efficiency (information is processed predomi- nantly locally, reducing the cost of communication between specialists). However, modularization also requires mechanisms to establish sharing of compatible representations across specialists, form of shared internal language. While portions of task might be solved by independent specialists, synchronization is critical particularly when there are statistical, functional, or causal dependencies among the specialists.
Coherence through shared workspace. In cognitive neuroscience, the Global Workspace Theory (GWT) (Barrs| 1993] Dehaene et al. (017) suggests an architecture allowing specialist modules to interact. The key claim of GWT is the existence of a shared blackboard, sometimes workspace-thhe can be modified by any specialist and that is broadcast to all specialists, along with the notion that write access is limited to maintain coherence. Our interpretation of this restriction write access is that it stems from assumption on the form of the joint distribution between high-level concepts. In this paper, we explore a communication and coordination scheme similar to the proposed by GWT for modern neural network architectures like Transformers Vaswaani et al. 2017; Dehghani et al. 2018 Parmar et al., 20188] Radford et al..
called a
2019 Brown al. 2020) and attention-based modular architectures (Goyal et al.) 2019 Rahaman
et al. 2020 Mittal et al. 2020a Goyal et al. 2020 Madan et al. 2021).
In terms of our driving example, the workspace could be used override default behaviors by giving
high priority to specialist modules which provide alerts of various sorts (loud sounds, presence of child on the street), allowing specialists which respond to such alerts to take control of behavior over default driving routines. This scenario implies that prioritization of signals in a shared workspace is critical.
A shared communication channel necessitates common representations. For a multitude of specialist modules to cooperate, a common language is necessary (Baars) 1997). For example, in the driving scenario, alerts may come from auditory or visual processing specialists, but regardless of the source, signal for danger must be placed in the workspace to override default behavior, whether that behavior is controlled by radio-tuning specialist or steering specialist. Although specialist modules can be pre-wired to have compatible communication interfaces, we will model an architecture in which ensemble of specialist modules is trained in coordination, which should lead to a shared language (Colagrosso and Mozer 2005). Internally. individual specialists can use whatever form of representations that serves them, but their inputs and outputs require alignment with other specialists in order to synchronize. For example, an unusual event such as a rough thud under the wheels might not have been previously experienced, but the mere signalling of novelty
In the literature, specialists sometimes referred to as processes or agents.
2

---

Published as a conference paper at ICLR 2022.
und hopuoe
of RIMs
6) Transformer
<l TIMS
4) Univrrsal Transformer
specialte
TMM TMureederrio
Compusition
Broadcant dep
RIMM SW
Transformer 3W
TIMS SW
Universal Tranffommer SW
Figure 2: Using a Shared Workspace for creating global coherence in RIMs, Transformers, TIMs and Universal Transformers (UT), (Top Half) All four of these architectures use pairwise communication (using key-value attention) to establish coherence between individual specialist modules. In the case RIMs (Goyal al. 2019) and TIMs (Lamb et al. 202T). these specialists are independent modules that compete with each other in order to take control over the state update based on a given input. the case of Transformers (Vaswani et al. 2017) and Universal Transformers (Dehghani et al.) 2018), each specialist is associated with a different position. Activated specialists are denoted by a blue shade and the intensity depends on the degree of activation. In the case of Universal Transformers, the state update dynamics for each position is shared across all layers and all positions (denoted by a yellow triangle). (Bottom Half) We replace pairwise communication with a shared workspace to create global coherence between different specialists. Communication using the shared workspace is two-step process (as denoted by and 2 in the figures). In the first step (1), specialists compete for write access the shared workspace, resulting activated (in blue), and only the activated specialists perform the write operation on the workspace. In the second step (2), the contents of the shared workspace are broadcast to all the specialists.
subset of them being
could override default specialists. Without global communication channel, specialists would have to learn to communicate through pairwise interactions, which might limit coordination of behavior in novel situations: global communication ensures exchangeability of knowledge to achieve systematic generalization.
2 SYNCHRONIZING NEURAL MODULES THROUGH A SHARED WORKSPACE
We investigate neural architecture reminiscent of the GW model, where number of sparsely communicating specialist modules interact via shared working memory. In particular, we extend the Transformer (Vaswani et al., 2017), attention and slot-based modular architectures (Goyal et al. 2019) by adding a shared workspace and allowing modules (each representing an entity) to compete for write access in each computational stage.
Key-value attention. Key-value attention defines the backbone of updates to the hidden states in the proposed model. This form attention widely used in self-attention models and performs well on wide array of tasks (Bahdanau et al.) 2014] Vaswani et al., 2017 [Santoro et al., 2018). Key-value attention selects an input value based on the match of query vector to a key vector associated with each value. To allow differentiability and thus easier learnability, selection is soft and computes a convex combination both the source of imput and how the shared workspace is updated. It also makes the outputs of the specialists and the elements of the memory permutation invariant: they should be considered as an unordered set of elements to be selected by attention mechanism from the contents of specialists. More precisely, soft attention uses the product of a query (represented as a matrix Q of dimensionality N, d, with N, queries, and d the dimension of each query) with of No objects each associated with key as a row in matrix K" (N, x d). After normalization with a softmax the resulting convex weights are used combine the values V; (row of matrix V): where the softmax is applied each row of its argument matrix, yielding a set of convex weights. For our experiments, we use multihead dot product attention.
all the values. Such a mechanism makes possible to change on-the-fly
Neural modules with pairwise interactions. Our approach to synchronizing neural modules is highly general and mostly agnostic to the task, domain, or specific choice of architecture, with the
3

---

Published as a conference paper at ICLR 2022.
only requirement being that the model consists of multiple specialist modules which either operate independently or have sparse interactions requiring to only match pairs of modules at a time. Our goal is to explore how introducing a shared workspace
help these modules to become better synchronized and coordinated. We show the utility of the shared workspace for synchronization in (a) Transformers (Vaswani et al.) (017). in which all interactions between positions are performed via attention, and (b) slot-based architectures like Recurrent Independent Mechanisms or RIMs (Goyal al. 2019) in which all pairwise interactions between modules are performed via attention. In the context of slot-based architectures, each slot's content is associated with a specialist module, whereas in Transformers different entities each associated with a different position acts as a specialist module (Figure 2)
Both Transformers and RIMs utilize a self-attention mechanism for sharing information between modules, typically implemented in pairwise manner, i.e., each specialist attends to every other specialist. Instead, we facilitate information sharing among specialist modules through a limited capacity shared workspace. In this framework at each computational stage, different specialists compete for write access to the common workspace. The contents of the workspace, in turn, are broadcast to all specialist modules simultaneously.
Notation.
The input is processed through a sequence of computational stages indexed by t. and at each stage, n1 entities are operated on (i.e., n. different modules in slot-based architectures like RIMs or n, different positions in the case of Transformers). Each of these ng specialist modules has distinct internal п -iimennioonal state h,, for 1 E (1,..., ng). }. The specialist modules communicate with each other via a shared workspace divided into nm memory slots, each consisting of a vector of ny elements, denoted M- [mi;...m, different computational stages i.e., different time-steps in recurrent architecture and different layers in the case of Transformers. At each computational stage t, different specialists compete for writing in the shared workspace, but all specialists can read from the current state of the workspace. In the case of an autoregressive task, we can restrict the information sharing to previous positions and keep a separate version of the workspace for each position.
!- The shared workspace is updated across
2.1 SPECIFICS OF THE SHARED WORKSPACE.
Step 1: Process Input obtain an entity representation, for each specialist. The first step is external to the proposed method, and involves processing the input to form the initial representation vector for each of the different specialists. Different common deep learning architectures can be used to form the representation of different specialists. For example, Transformers start with a matrix n, X n& whose rows are initialized as the ni, dimeensioon embeddings of the input at each position of the sequence. Slot-Based Recurrent architectures like RIMs consist of a single-layer recurrent structure where the hidden state h, at computational stage 1 is decomposed into the substates of the n, specialists, h for == I,
In the proposed scheme, within each computational stage, the updates of the hidden state of different specialists follow a two-step process. First, specialists compete and write to a shared workspace. Second, information from the workspace gets broadcast to all the specialists, as detailed next.
Step 2: Writing Information in the shared workspace. The specialists compete to write into the shared workspace, whose contents need to be updated in the context of new information received from different specialists. This step ensures that only the critically important signals make it to the shared workspace, therefore preventing the workspace from being cluttered. Let matrix R represent the combined state of all the specialists (i.e. A Vk E [1,...., .?,) as the rows of R). In order to implement the competition between specialists write into the workspace, we use key-query-value attention mechanism. In this case, the query is function of the state of the current workspace memory content, represented by matrix M (with one row per slot of the memory), i.e Q = MW Keys and values are a function of the information from the specialists i.c., a function of R. We apply dot
product attention to get the updated memory matrix: M + softmax Q(RW-yT RW" The use dde
of regular softmax to write into M leads to standard soft competition among different specialists to write in the shared workspace. One can also use a top-k softmax (Ke et al., 2018) to select a fixed
number of specialists allowed to write in the shared workspace: based the pre- pre-softmax values, fixed number of & specialists which have the highest values are selected, and get access to write in the shared workspace. Selection with top-k softmax is hybrid between hard and soft selection. We
4

---

Published as a conference paper at ICLR 2022.
denote the set of thus selected specialists as Ft. We note that can apply the attention mechanism multiple times to distill information from different specialists into the shared workspace. Here, the contents of the shared workspace are updated in the gated way proposed in RMC (Santoro et al.) 2018). We ask the reader to refer to appendix section C for more details.
Step 3: Broadcast of information from the shared workspace. Each specialist then updates its state using the information broadcast from the shared workspace. We again utilize an attention mechanism to perform this consolidation. All the specialists create queries Q h W which are matched with the keys R, - (m, Weyz Vk E (I,..., ng3. {l,.... N from the updated memory slots, forming attention weights šk.j = softmax guRi vdd The memory slot values generated by each slot of the shared workspace and the attention weights are then used to update the state of all the specialists: h h + +E, $ where ,, my W. Vk E {1...., Rg). After receiving the broadcast information from the workspace, each specialist update their state by applying some dynamics function i.c., one step update of LSTM or GRU units in the case of recurrent architectures, and a feedforward layer in the case of Transformers. This yields the new value h for the k-th specialist, from which we start the next stage (t + 1).
Replacing pairwise interactions among neural modules with interaction facilitated by the shared workspace allows for the following:
1. Higher-order (HO) interaction among neural modules. The two-step write-read process first allows each memory slot to store a 'filtered summary' of the current input where the 'filter' is determined by the previous state of that slot ('Query' for the write step). Neural modules then summarize the information contained in these slots and update their state. Hence unlike pairwise interaction, messages passed among neural modules in the shared workspace setting also include HO interaction terms; those consisting of more than 2 modules at time. Naturally, HO interaction require that messages passed among neural modules lie in the same representation space, which is precisely what we aim to achieve by allowing message passing only via a singular global channel.
Dynamic filtering due to persistence of memory. With a shared workspace (SW), contents of the memory slot play key role in filtering and summarizing the information contained in the input at given time step. Persistence of memory throughout an episode 1) would allow the memory layer to summarize and filter information based on what it has seen thus far 2) should ideally lead to better generalization as the model is able to dynamically modify its filtering machinery for particular input. In contrast, "inducing points" in Set Transformers (Lee et al. 2019) are fixed after training and hence the bottleneck cannot adjust itself on the fly for any new input. We present comparisons on several tasks in section 4 They show the importance of these two properties by comparing performance of SW with a) Selff-Atttentio (to simulate HO interaction without global communication) b) a version without memory persistence, in Appendix D
Computational Complexity of using shared workspace for synchronizing different specialists. To encourage a coherent global coordination, Transformers and slot-based recurrent architectures rely on pairwise interactions captured via an attention mechanism. Unfortunately, such attention mechanisms scale quadratically with the number of specialists. Here, we propose method which uses shared workspace to create global coherence between different specialists and in the process, replaces the pairwise interactions of conventional dot dot-product attention. The computational complexity of the proposed method is thus linear in the number of specialists. In our experimentation, the number of memory slots is practically constant, which suggests a very favourable scaling behavior, and certainly much less than quadratic. As point of reference, what would correspond to the number of slots in human working memory
[Barr 1993) is indeed very small (less than 10 slots).
3 RELATED WORK
This work taps into a line of reasoning put forward by historical works, such as Minsky (1888;; BBaiteeberg 1986);; Fodor (1983), wherein is argued that in order to be able to deal with wide
spectrum of conditions and tasks, an intelligent system should be comprised of many interacting specialized modules or programs, rather than a single architectures have been the subject of number of research directions, (Jacobs al. 1991 Bottou
mimesize-fitt-all entity. While modular
and Gallinari 1991; Ronco et al. 1997 RRed and De Freitas 2013] Andreas et al. [0166] Rosenbaum
et al. 2017 Fernando et al. 2017 Shazeer et al. 2017
Rosenbaum al. 2019 Goyal and Bengio
5

---

Published as a conference paper at ICLR 2022.
2020), we focus here on a mechanism for achieving coherence and synchronization between specialist modules via a global workspace shared between all specialists.
Prior works have explored incorporating slot-based memory in the context of recurrent neural networks (Graves ct al., 2014 2016; Santoro et al. 2018). In the context of transformers, [Burtsev and Saaunov (2020) introduce memory tokens that are processed in addition to sequence tokens, whereas Dai et al. (2019) (Transformer-XL) propose to partition a long sequence to smaller segments and use the activations of the previous segment memory while processing the current segment. Building on the latter, Rac et (2019) propose to store activations from prior segments in a compressed memory, However, these methods do not restrict memory writes to be sparse and competitive. Recent advances in this direction include the global neuronal workspace (GNW) model (Dehaene and Changeux 2011. which identifies the global workspace with a large network of excitatory pyramidal neurons with long-range axonal processes connecting prefrontal and parietal cortices. Further, deploying shared workspace to establish coherence between different specialists as opposed to using all-pair communication has an added benefit, in that it allows us to tackle the O(n²) complexity of self- attention. This makes our work related to previous work on reducing the computational complexity of dot product attention in Transformers, Lee et al. (2019) introduce the ISAB module, which maps between sets and comprises two dot- product attention layers. In the first layer, a set of trainable parameters are used as queries and the elements of the input set as keys; in the second layer, the output of the first layer is used as keys and the input set as queries. However, unlike in this work, the intermediate states (corresponding
the output of the first layer) are not maintained across layers. Concurrent to our work, (Jaegle et al., (222)) also introduced the idea of using a latent bottleneck for addressing quadratic complexity by learning a bottleneck but there are important differences. For example. in Perceiver the latent bottleneck iteratively queries the information about different positions, and does not maintain the representation of the different specialists. More precisely, in our proposed method different specialists write information in the workspace and then information gets read from the shared workspace. In Perceiver, the latent bottleneck iteratively reads information from the set of positions. We also show the applicability of the proposed idea both for slot based models and Transformers.
The proposed model can also be seen integrating out different ideas popular in modular architectures (Andreas et al. 2016) Goyal et al. 2001)), memory networks (Graves et al. 2014 Santoro et al. 2018) and mixture of experts (Jacobs al. 199T), and hence combining some of their benefits in a unified architecture. The proposed model is factored as a set of specialists (incorporating modularity). The proposed model achieves coordination among different specialists via the use of a shared workspace (in the Neural Turing machines, there is only a single specialist i.e., without any modularity). Multiple experts can be active at the same time (generally not the case with a mixture of experts).
4 EXPERIMENTS
Here we briefly outline the tasks on which we applied the idea of the shared workspace and direct the reader to the appendix for some more experiments (Appendix G). full details on each task and details on hyperparameter settings for the model. The experiments have the following goals: (a) Demonstrate that the use of the shared workspace can improve results on wide array of challenging benchmark tasks, with the goal of demonstrating the practical utility and breadth of the technique. (b) Show that the shared workspace addresses coherence between different specialists by achieving improved performance without requiring all pairwise interactions. Finally, to show wide applicability of our model, we integrate SW in TIMs (Lamb et al. 2021), SCOFF (Goyal et al., (020)) and BRIMs MMittal et al. 2020b) and show improvements over the default communication method used in each.
Making sense of the visual input. Using a shared workspace introduces a bottleneck in sharing of information between specialists. Since the size of the workspace is limited and generally much lower than the number of specialists, there is limit to the amount of information that can be ex- changed among specialists. We hypothesize that mediating communication through a limited capacity workspace should encourage the model to look at relevant information that is important for the downstream objective. We test this hypothesis on set of visually challenging benchmarks. For our experiments, we use either Transformers or RIMs as a backbone. We consider variants of Trans- formers based on different subsets of important properties. Transformers [TR]: Self-attention based multi-layer architecture (Vaswani et al., (017)) with shared parameters across layers. Set transformer [ISAB]: Transformers where self attention replaced by ISAB module (Lee et al. 2019). Sparse

---

Published as a conference paper at ICLR 2022.
100
Model
Top-1
Top-5 %
Accuracy
ISAB
65.3
83.6
STR
70.6-n.0
87.33+000
10
TR
70.8 83
87.8
TR + HC
70.17
88. 88.33
20
TR HSW (OURS)
71.07
88.6
0 STR
TR
TR SSW (OURS)
71.33 -034
88.3
Figure 3: Detecting Equilateral Triangles. Here, we compare the performance of the Trans- formers with shared workspace to other Trans- former baselines. Here, we plot the test accuracy for each model.
Table 1:
Comparison on CATER Object Tracking. Here, we compare the Top-1 and Top-5 accuracy of Transformers with shared workspace and Transformers with self-attention We can see that Transformers with a shared workspace outperform those with pairwise self- attention.
Transformers [STR]: Transformers with sparse factorizations of the attention matrix (Child et al. 2019). High Capacity Transformers [TR+HC]: Same as TR but with different parameters across layers. Transformers with Shared Workspace with soft-competition [TR+SSW]: Transformers with different positions competing with each other to write in shared workspace using soft-competition. Transformers with Shared Workspace with top-k competition [TR+HSW]: Transformers with differ- ent positions competing with each other to write in shared workspace using top-k competition. For more detailed description of all the tasks described below, we ask the reader to appendix section
Detecting Equilateral Triangles. We first use a simple toy task to test our hypothesis where the model should detect equilateral triangles in images (Ahmad and Omohundro) 2009). Each image is of size 64 X 64 and contains 3 ran- domly placed clusters of points. For equilat- eral triangles, the midpoints of these clusters equidistant from each other. This is a binary classification task where the model has to pre- diet whether the three given clusters form an equilateral triangle or not. To feed an image into Transformer, we follow the same methodol- ogy as used in vision Transformers (Dosovitskiy et al. 2020). We first divide an image into equal sized& 4 x 4 patches and treat each patch as a different input position of the Transformer.
HSW
6AB
Non- Roon-Relaioonal Questions
100
Accuracy
20
Epoch
Relational Questions
Accuracy
80
60
20
40
80
100
Epoch
Figure 4: Comparison on Sort-of-CLEVR rela- tional reasoning. Speed of convergence for rela- tional and non-relational questions in the sort-of- clevr dataset. We can see that the proposed model converges much faster than the baselines in both cases.
To solve this task correctly, the model only needs to attend to relevant information i.c., patches that contain the cluster of points. Therefore, us- ing a limited capacity shared workspace should be useful here. Our results (presented in Figure
3 confirm this hypothesis. We can see that Transformers with shared workspace attention converge much faster and reach higher accuracy as compared to the baseline Transformer. Our method also outperforms Set Transformer by significant margin.
Multi MNIST Generation. In this task, we train an Image Transformer (Parmar et al. 2018) (pixel- by-pixel, raster-order generative model) for next-pixel prediction on the "MuliMNIST dataset" where each image consists of 4 independently sampled MNIST digits stacked horizontally to form one image (see Figure TO for demonstration). The main aim of this task is to observe the inductive biases that allow for specialization of mechanisms in TIMs (Lamb et al., (021). Each image in the MultiMNIST dataset can be broken down into different sets of independent spatial components. Since the digits which make up the image are independently selected, the joint distribution of pixel intensities in any one of the four sections of the image is statistically independent of the pixel intensities in any other section of the image. Moreover each section of the image can be further broken down into independent spatial components: one that pertains to the background and one that pertains the foreground. One expect that architectures that made up of sparsely interacting
7

---

Published as a conference paper at ICLR 2022.
different mechanisms to naturally capture this statistical independence by dividing labour among different mechanisms. While, for monolithic architectures, a major portion of their training time will be spent in learning these statistical independencies from scratch. We find that replacing the pairwise communication in TIMs with a shared workspace (TIMs + SW) leads to better and more interpretable division of labor among specialists shown in Figure 5 From the figure, It is clear that the TIMs model is unable to divide labour among specialists with mechanism 2 being activation for all the pixels in the image. On the other hand, we can see that TIMs + SW is able divide labor among specialists with each mechanism focusing on different aspect of the image. We can see that mechanism gets activated for the digits which are present towards the centre of each of the columns while mechanisms 3 and 4 cover the background of the digits, with mechanism 3 covering the area between adjacent digits and mechanism covering the area above and below the digits. Thus, we can see that using shared workspace aids the division of labor among different specialists. We also find that TIMs + SW results in the least cross-entropy loss in the test set when compared to TIMs and Image Transformers (Parmar et al. 2018). Results shown in appendix Table 3
CATER: Object Tracking- Cater is a spatio- temporal reasoning video dataset introduced in Girchar and Ramanan (2019). Each video con- tains 3D objects organized in 6 x 6 grid. Each object affords certain actions that can be per- formed on them. These actions result in move- ment of the concerned objects and change in their positions. Some of these actions include: rotate, pick-place, slide, contain. Throughout the duration of the video, number of these ac- tions are performed to get the final state of the grid. Note that only single object undergoes an action, at any instant. The task that we fo- cus on here is called localization. In this task, the goal is predict the location of the target object in the final frame. In this case the target well as
(a) Mechanism Activa-(b) Mechanism tion Maps for TIMs tivation
Ac-
Maps
for
TIMs+SW
Figure 5: This figure shows the mechanism acti- vation map for all 4 mechanims used in the multi- mnist generation task for both TIMs and TIMs + SW. Both the images in the figure correspond to the activation maps from 4 different examples. Each activation map contains mechanisms shown from left to right in a single row. Each mechanism is shown using a 32 x 32 image, a particular pixel in mechanism activation map is shown in white if that mechanism was used during the generation of that pixel while generating the image.
object is called a snitch. The snitch the other objects move across the 6 x 6 grid. In some scenarios, the snitch may be covered by other objects hence hiding it from the view. In such cases, tracking the movement of the snitch across frames becomes essential. Therefore, cap- turing long-range temporal dependencies is es- sential to solve this task.
The information exchange limit enforced by the limited capacity of the shared workspace should
be useful here as well. For CATER, in some frames the snitch is not visible as is covered by other objects. Therefore, ideally the model only needs to attend to frames in which the snitch is visible. Additionally. if the snitch is visible throughout the video in all frames, then to accurately predict the final position of the snitch, the model only needs to attend the final frame of the video and can completely ignore the initial frames. The results for this task are presented in Table [ We also experimented with both soft competition TR+SSW and hard competition TR+HSW with only k = 5 specialists writing into the shared workspace. We can see that models with shared workspace outperform those with pairwise multihead attention thus confirming our hypothesis about the benefits of shared workspace for this task. As shown in Table [proposed method convincingly outperforms the Set Transformer.
Relational Reasoning: Sort-of-CLEVE In relational reasoning, the model is tasked with answer- ing questions about certain properties of various objects and their relations with other objects. The model is presented with an image and a question for that image. This task has clear sparse structure as in order to answer the questions correctly, needs to only reason about a specific subset of objects that the question mentions. For this task, we use the Sort-of-CLEVR dataset (Santoro et al. 2017).
Each image in Cort-of-CLEVE of size 75 75 and contains randomly placed geometrical shapes of 6 possible colors and 2 possible shapes. Each image comes with 10 relational questions and 10 non-
8

---

Published as a conference paper at ICLR 2022.
relational questions. Non-relational questions only consider properties of individual objects. On the other hand, relational questions consider relations among multiple objects. For more details about the question see appendix Figure g The input the model consists of the image and the corresponding question. We first obtain a sequence of equal-sized patches for the image as in vision Transformers (Dosovitskiy et al.) 2020). We concatenate the resulting patch sequence with the representation of the question and pass the combined sequence through the Transformer. Sort-of-CLEVE has a finite number of possible answers, hence this task is setup as classification task.
We present the results for this task in Figure We observe that the Transformers with the shared workspace converge faster and outperform the baselines for relational as well as non-relation non-relational questions. The superior performance with shared memory can be attributed to the inherent sparsity of this task. For instance, model only needs to attend to single object referenced in the question to answer it correctly, while relational ques- tions only consider small subset of objects in the image, thus sparsity is helpful for both these types of questions. Therefore, the limited capacity of the shared workspace forces the model to attend to only relevant information.
Model SCOFF
Num Slots
ARI
MSE +
0.276+0000
0.083
SCOFF + SW
2
0. 0..00 0.154+
SCOFF + SW
4
0.487 1.0es
0.059
SCOFF + SW
0.915_.m0
0.035 _0.
non-relational questions, the
SCOFF SW
0.891
0:001
0.039 _009
SCOFF + SW
10
0.351 2:0000
0.08,g00
Table 2: Here we show the performance of SCOFF augmented with shared workspace attention on the bouncing balls task. We also analyse the ef- fect of varying number of slots in the shared workspace. This also shows that by increasing the number of slots performance decreases hence validat- ing claims regarding bandwidth lim- ited communication channel via shared workspace.
Shared Workspace for Physical Reasoning. In this task, we consider set of bouncing balls and the model is tasked with predicting the trajectory of the balls at each step. In order to solve this task, a coherent picture of where and which objects will collide needs to be estab- lished by the learner. We use the bouncing-bale dataset from Van Steenkiste et al. (2018). We train the model for next-step prediction. We compare the proposed approach against SCOFF (Goyal et al., 2020 The results of our comparison are shown in Table 2 We use the ARI and MSE metric for comparison. ARI measures how well the different balls are segregated into different slots, higher ARI means better segregation. We can see that using shared workspace results in higher ARI as compared to pairwise communication in SCOFF. Thus, using a shared workspace results in better division of labor among specialists. We also compare the proposed method against other baselines in appendix section F.1
Shared Workspace for Atari Video Games. We start by training RIMs, RIMs shared workspace (SW) on three "source" games (Pong. River Raid, and Seaquest) and test if the learned features transfer to a different subset of randomly selected "target" games (Alien, Asterix, Boxing, Centipede, Gopher, Hero, James Bond, Krull, Robotank, Road Runner, Star Gunner, and Wizard of Wor). We take sufficient number of specialists in RIMs (10). We train on source games for 10M steps, and then fine-tune transfer games for 10M more steps. We choose these games as they were also used in the original RIMs paper (Goyal et al. 2019). Using a suite of 36 game pairs, we find that RIMs + SW outperforms RIMs on both game A (a median performance ratio of 1.13; mean of 1.16) and game B (a median performance ratio of 1.11; I; mean of 1.15). The improved performance with RIMs + SW is due better forward transfer (knowledge acquired for game A facilitates the learning of game B) and reduced backward interference (knowledge acquired for game B does not disrupt knowledge acquired for game A), presumably thanks to more appropriate modularization of knowledge.
5 CONCLUSION
Inspired by cognitive neuroscience global workspace theories, we have proposed a shared workspace model for establishing coherence among modular neural specialists while exchanging information in a systematic way. We show that using a limited capacity shared workspace as a bottleneck for mediating communication among specialists results in better performance across wide range of visual reasoning benchmarks as compared to the pairwise interactions typically used in self-attention schemes. The proposed approach combines several key properties: knowledge and expertise is divided among specialists, they compete to post new contents to the workspace, and after being updated, the shared workspace is accessible to all specialists for their own updates.

---

Published as a conference paper at ICLR 2022.
ETHICS STATEMENT
The authors do not foresee any negative social impacts of this work, but of course the accumulation of improvements in ML could be misused may give more power nefarious agents.
REPRODUCIBILITY STATEMENT
We Algorithms | and for our experiments, will be releasing the code after the review process.
We also provide our code in the supplementary material.
REFERENCES
S. Ahmad and S. Omohundro. Equilateral triangles: A challenge for connectionist vision. 2009.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48, 2016.
Bernard J Baars. A cognitive theory of consciousness. Cambridge University Press, 1993.
Bernard J Baars. In the theatre of consciousness. global workspace theory, a rigorous scientific theory of consciousness. Journal of Consciousness Studies, 4(4):292-309 1997.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409. 409.0449..44 2014.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06660 2019.
Léon Bottou and Patrick Gallinari. A framework for the cooperation of learning algorithms. In Advances in neural information processing systems, pages 781-788, 1991.
Valentino Braitenberg. Vehicles: Experiments synthetic psychology. MIT press, 1986.
Rodney A Brooks. Intelligence without representation. Artificial intelligence, 47(1-3):139-153 1991.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Mikhail S Burtsev and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527 2020.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /90:. 10509, 2019.
Michael D. Colagrosso and Michael C Mozer. Theories of access consciousness. In L. K. Saul, Weiss, and L. Bottou, editors, Advances Neural Information Processing Sys- tems 17, pages 289-296. MIT Press, 2005. URL httt:://pappeeeee
.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint 1:X1::9901.02860 2019.
S. Dehaene, H. Lau, and S. Kouider. What is consciousness, and could machines have it? Science, 358(6362):486-492. 2017.
Stanislas Dehaene and Jean-Pierre Changeux. Experimental and theoretical approaches to conscious processing. Neuron, 70(2):200-221, 2011.
10

---

Published as a conference paper at ICLR 2022.
Stanislas Dehaene, Michel Kerszberg, and Jean- Changeux. A neuronal model of global workspace in effortful cognitive tasks. Proceedings of the national Academy of Sciences, 95(24): 14529-14534, 1998.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Eukasz Kaiser. Universal transformers. arXiv preprini urXiv:/807.038/1 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 19929, 2020.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: (0lo:/7011874,, 2017.
Jerry A Fodor. The modularity of mind. MIT press, 1983.
Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for compositional actions and temporal reasoning. CoRR, abs/1910.04744, 2019. URL http:://aa....gg/////a...
04744
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. arXiv preprint arXiv:2011. 2020.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint (rXiv:1909.10890. 2019.
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv:2006. 16225, 2020.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/ 1410.4410, 2014. URL http::://axiii.rrrt.t org/ wtwpg//////11.540.0000
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626): 471-476, 2016.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv preprint 2rX:v:21030000002 2021.
Andrej Karpathy. karpathy/mingpt, Aug 2020. MinGPT
URL http::://ggthuub.omm/kaaggg
Nan Rosemary Ke, Anirudh Goyal ALIAS PARTH GOYAL, Olexa Bilaniuk, Jonathan Binas, Michael C Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through reminding. In Advances in neural information processing systems, pages 7640-7651, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Lamb, Di He, Anirudh Goyal, Guolin Ke, Chien-Feng Liao, Mirco Ravanelli, and Yoshua Bengio. Transformers with competitive ensembles of independent mechanisms, 2021. URL
htttps://cpenre
net
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention- based permutation commutationinvvrrian neural networks. In International Conference Machine Learning, pages 3744-3753, 2019.
11

---

Published as a conference paper at ICLR 2022.
Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua Bengio. Meta attention networks: Meta-learning attention to modulate information between recurrent independent mechanisms. In International Conference Learning Representations, 2021. URL https:
//openreviews eww. net firee//foumiiii (oforrmmm111..c228A44444
Marvin Minsky. Society of mind. Simon and Schuster, 1988.
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. In International Conference on Machine Learning, pages 6972-6986 6986. PMLR, 2020a.
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. In International Conference Machine Learning, pages 6972-6986. PMLR, 2020b.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of Demonstrations, 2019.
NAACI-HIIT 2019:
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint 1911. 1.05507, 2019.
Nasim Rahaman, Anirudh Goyal, Muhammad Waleed Gondal, Manuel Wuthrich, Stefan Bauer, Yash Sharma, Yoshua Bengio, and Bernhard Schölkopf. S2rms: Spatially structured recurrent modules. arXiv preprint arXiv:2007.06533 2020.
Scott Reed and Nando De Freitas. Neural programmerrimterppetees arXiv preprint arXiv:1511.06279. 2015.
Eric Ronco, Henrik Gollee, and Peter Gawthrop. Modular neural networks and self-decomposition. Technical Report CSC-96012, 1997.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv: /2.74444..7777 2019.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4967-4976, 2017.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski. Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pages 7299-7310, 2018.
Murray Shanahan. A cognitive architecture that combines internal simulation with global workspace. Consciousness and cognition, 15(2):433-44) 2006.
Murray Shanahan. Embodiment and the inner life: Cognition and Consciousness in the Space of Possible Minds. Oxford University Press, USA, 2010.
Murray Shanahan. The brain's 's connective core and its role animal cognition. Philosophical Transactions of the Royal Society B: Biological Sciences, (171160))2704-2774 2012.
12

---

Published as a conference paper at ICLR 2022.
Murray Shanahan and Bernard Baars. Applying global workspace theory to the frame problem. Cognition, 98(2):157-176, 2005.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint urXiv:/701.06538 2017.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery objects and their interactions. arXiv preprint 17Xi::/802..0333,, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki. Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350 2019.
13

---

Published as a conference paper at ICLR 2022.
Part I Appendix
A
PSEUDO CODES
Alg. I shows the integration of shared workspace with RIMs Goyal et al. 2019). We replace the direct module module interaction via attention in RIMs, with shared workspace. Specialists compete to write in the shared workspace, and the contents of the workspace are broadcasted to all the specialists.
Alg. 2 shows the integration of the shared workspace with TIMs (Lamb et al. 2021). Again we replace the direct module module communication in TIMs, with shared workspace.
Algorithm 1: Shared Workspace integration with RIMs
Input: Current sequence element, 22 and previous state of the specialist, (h--1*)), for & E (1....,,,,) and structure of memory as matrix M with row wise compartmentalized memories, where m, refers to the state of slot (total numher of slots is Bm..)
Step 1: Process image by position p with fully convolutional net
Cp [CNN(x;))
Cp Cpp
(concatenate encoding of position CNN output)
Step Specialists compete to be selected to update the workspace based on current input
qx 1, W
S softmax( vd.): where (z, (EeWe)T
Construct set Fi which contains the indices of the nset specialists that have the largest S
hu,6 9s (s&2; W", h,-1k))
Fr.
14-1, a. S t Fi (Scaled Dot Product Attention)
Φ Fe.
Step 3: Activated specialists write in a shared workspace
Q MW4
R [M;A] A] where A is the matrix whose rows are the au E Fe
M + softmax Q(RWWyT RWW"
Step Broadcast of information from the shared workspace
T F W
Vk [1,..., 1gg
= softmax
where R = (m, WeyT Vk E ns, j E (l....,nm.)
Vdc
he. h + E3 8k.5 By where 0; = m. W
Ak E (1....,,n.}
B HYPERPARAMETERS
Table 3
lists the different typer-parametent
Parameters in RIMs+SW:
RIMs with shared workspace has three set of parameters:
Parameters corresponding to Input attention Parameters for the attention for the k-th specialist 0₃ = (W2, W W*) corresponding to query, keys, and values respectively. Each specialist has different query parameters but share the same keys and values (which are function the input). In the table it corresponds to the inp keys, inp values, inp heads respectively.
Writing in a shared workspace: Parameters corresponding to the writing in the memory. Here, we follow the similar mechanisms as in RMC Santoro [MCoSantoro al. 2018), where shared
14

---

Published as a conference paper at ICLR 2022.
Algorithm 2: Shared Workspace integration with TIMs
Notation: Consider h2 as output the 1d transformer layer. Let sequence length original input he and embedding dimension of transformer be D. Let the transformer he composed of ny mechanisms and memory be denoted as matrix M with row wise compartmentallize memories, where ma refers to the state of slot (total number of slots rem). Consider h = fu (k 1)D/mm kD/rwa to be the hidden state of mechanism indexed at layer l.
Initialization: Convert the raw input X E RTTocobbb.... size size to
ho positional_ _ecoldinnccggeeeee Embedding(A where ho E TxD, Initialize memory matrix M which remains for all layers in the transformer.
Input to the layer I: h;-1 having shape R
Step 1: Mechanisms compete be selected to update the workspace based on the input they receive from previous layer
W E D
= WE Yk (2,
c- coftmaz(concat(L))
c R R
For each time step t in the original sequence of length T', the soft score select the top nad mechanisms which would self attend and write to the memory. Hence generating set F, which stores indices of That mechanisms for position & {1,2, T}. Also construct E R T'x D/Ha where
[[[][[]] k Ee
[t..:]
k Fr,
Step 2: Selected mechanisms self-attend and update their hidden state residual; h
h = Attentiooion residuale Vk (1....,,oo)
Step 3: Selected mechanisms write on the shared workspace
Memory matrix M was last modified by mechanisms of layer |
Let @ h and concat (a..., (nn)), Absorb the first dimension (corresponding position in
sequence) the batch dimension by reshaping a. Perform same steps as algorithm MMW
R [M; A] where A aW"
Q(RWoyT
M + softmax
RW"
Va.
Step 4: Broadcast of information from the shared workspace
Reshape new memory to bring back sequence dimension. Perform same steps as in algorithm
T h W
YK E
C.R, {1,..., Tib
= soltmax
where R = (m, WeyT Vk E (L....,,m.),
m.) j (l...., mm
Vd.
h = hh + E3 Sk.
where T, m W Wk E (.,)..,mb)
15

---

Published as a conference paper at ICLR 2022.
Equilateral Triangles
Non Equilateral Triangles
Figure 6: A demonstration of the detecting equilateral triangles task.
Parameter
Value
Number of specialists (n,) Size of each specialist
6
85
Number of memory slots (nm)
Optimizer
Adam Adam<Kingma and Ba
2014
learning rate batch size
64
Inp keys
64
Inp Values
85
Inp Heads
Inp Dropout
0.1
Number of memory slots
4
Number of memory heads
Size of attention head
32
Key size
32
Number of MLP layers in Attention
Gate Style
'unit"
Memory Attention Heads
4
Memory Attention keys
32
Memory Attention Values
32
Table 3: Generic Hyperparameters for the proposed model (for RIMs)
workspace is seen as a Matrix with row wise compartmentaliize memories (i.e slots) i. W W W", In the table corresponds to number of memory slots, number of memory heads, size of attention head, key size and number of mlp layers attention. These are the same pyper-paramter as in RMC (Santoro et al.] 2018). We tried two different set of hyper-parameters (a) where we only have a single slot and (b) where we have slots.
Broadcast of Information from the shared workspace: this process, the information in the workspace gets broadcasted to all the specialists such that each specialist produces a query, and the keys and values are a function of the memory state. Each specialist gets information from the memory according to its query, and this information is used to update the state of each specialist in a residual fashion. This corresponds the parameters of W W We in the table i. memory attention heads, memory attention keys, and memory attention values. We did not do any hyper-parameter search for these hyper-parameters.
16

---

Published as a conference paper at ICLR 2022.
Resources Used:
For vision tasks like Sort-of-clever, Equilateral triangle, CIFAR classification, it takes about 6 hours to run 200 epochs on V100 (32G) GPU.
It takes about 2 days to train the proposed model on bouncing ball task for 100 epochs on V 100 (32G) GPU, We did not do any hyper-paramelte search specific to a particular dataset (i.e 4Balls or 678Balls Curtain Task). We ran the proposed model for different number of memory slots (i.e 2/4/8) for all the different datasets.
For Starcraft task, it takes about 5 days to train V100 (16G) GPU with batch size of 4.
C IMPLEMENTATION DETAILS
Writing Information the shared workspace. While writing information to the shared workspace, update the workspace using gating mechanism proposed in Santoro al. (2018). The gating mechanism consists of input and forget gates. Let M' and M be the previous and updated memory matrix respectively. Let M be the result of the attention mechanism as described in step of section [..1 Let X1...,, be the input to n, specialists. The gating mechanism can be formulated as follows.
X relu(X, X; X W
K X + tanh (M
I sigmoid (WW)
F sigmoid (KW*))
M Manh(M) F M
Here, I and F indicate the input and forget gates respectively. Note that W is shared across all specialists.
D PROPERTIES OF SHARED WORKSPACE
In section 2 we claim that higher-order interaction terms and effects due to persistence of memory are key contributors to Shared Workspace performance. We support those claims here:
Shared Workspace repeated self attention Higher-order interaction can be simulated by repeat- ing the self-attention step multiple times at the same layer/time-step. However, due to the absence of global communication channel, there is constraint that the messages passed among the neural modules should in the same representation space. We modify a standard transformer where we repeat the self-atte oeff-attention step two times in every layer. We expect that 2 Self Attention will perform worse than SW. We also run a model where both self-attention as well as shared workspace is used by the transformer to update its state.
Persistence of Memory To check whether persistence is crucial for our model to perform well, model where re-initialize the shared workspace every layer. Again expect that removing memory persistence should result in drop in performance and speed of convergence.
We these models ofrt-of-clevr dataset and present the results in figure 7
We note that removing persistence of memory results in significantly slower convergence. Replacing SW with 2xSA results in significant drop in performance.
17

---

Published as a conference paper at ICLR 2022.
Non Relational Questions
Relational Questions
100
Accuracy
Accuracy
80
8
60
20
40
60
100
20
60
100
Epoch
Epoch
memory parsitence)
Figure 7: Comparison on Sort-of-CLEVR relational reasoning. Speed of convergence for rela- tional and non-relational questions in the sort-of-clevr dataset. We can see that the Shared Workspace model converges faster and generalizes better as compared to all the other models. Here SW refers to shared workspace, 2xSA refers to applying self-attention twice in the same layer, SW+SA refers using both Shared Workspace and Self Attention in each transformer layer.
Relational questions:
What the shape the object closest to the red object? = square
What the shape of the object furthest to the orange object? = circle
How many objects have same shape with the blue object?
Non-relational questions:
What the shape the red object? Circle
Is green object placed on the left side of the image? ab yes
Is orange object placed the upside of the image? no
Figure 8: A sample from the sort-of-clevr dataset.
E TRANSFORMER TASKS
E. DETECTING EQUILATERAL TRIANGLES
demonstration this task can be found figure 6 We use images of size 64 x 64 for this task. Our training dataset consists of 50000 examples and we evaluate 10000 examples. We follow the same setup as vision transformers Dosovitskiy et al. (2020) for this task. We divide the image into patches of size 4 X 4. this sequence of patches is fed input 4-layered transformer along with the CLS token which is used for classification. We set hidden dim to 256 and ffn dim to 512. For the proposed model (TR+SSW, TR+HSW), We
query and key size of 32, and value size of 64. We use heads during reading from and writing into the shared workspace which consist memory slots. For the baseline models (TR. TR + HC, STR), we use query, key and value size of 64 and 4 heads. For training, we use batch size of 64. We train the model for 200 epochs using Adam optimizer with a learning rate of 0.0001. We anneal the learning rate using cosine annealing.
E.2 SORT-OF-CLEVR
Figure 8 shows sample from this dataset. The images in this dataset are of size 75 x 75. Each question encoded into bits. The first bits indicate color, the next bits indicate question type (relational or non-relational), and the remaining 3 bits indicate question subtype (according to figure 8) We use 4-layered transformer for this task with hidden dim set to 256 and ffn dim set to 512. For the proposed model (TR+SSW, TR+HSW), We use query and key size of 32, and value size of 64. We use 4 heads during reading from and writing into the shared workspace which consists of 8 memory slots. For the baseline models (TR, TR HC, STR), key and value size of 64 and 4 heads. We encode the 11 bit question into a 256 dimensional vector representation and concatenate it with the sequence of 15 x 15 sized patched obtained from the image.
query,
18

---

Published as a conference paper at ICLR 2022.
We use the representation corresponding to the CLS token for classification. We train the model using cross-entropy loss. We use a batch size of 64 and train the model for 100 epochs. We use Adam optimizer with learning rate of 0.0001 for training.
E.3 CATER: OBJECT TRACKING
Each CATER video consists of about 300 frames of size 224 X 224. We first sample frames at sampling rate of 6 which results 50 frames. From these 50 frames, we stack 5 consecutive frames together and pass each stack through a 18 layered resnet. The corresponding sequence of 10 frames is passed as input to the transformer. This task setup as a classification task where we have to predict which cell in the 6 with hidden dim set to 512 and ffn dim set to 2048. For the proposed model (TR+SSW, TR+HSW), We query and key size of 32, and value size of 64. We use heads during reading from and writing into the shared workspace which consists of 8 memory slots. For the baseline models (TR, TR + HC, STR), use query, key and value size of 64 and 8 heads.
grid contains the snitch in the final frame. We
6-layered transformer
F RIMS TASKS
F.1 BOUNCING BALL
The dataset consists of 50,000 training examples and 10,000 test examples showing ~50 frames of either 4 solid balls bouncing in a confined square geometry (4Balls), 6-8 balls bouncing fined geometry (678Balls), 3 balls bouncing in confined geometry with an occluded region (Cur- tain), or balls of different colors (Colored 4Balls) and (Colored 678Balls). We train baselines as well the proposed shared workspace extension (e.g., RIMs + SW). As shown in Fig. 9 we study the performance of the proposed model compared with LSTM, RIMs and RMC. The first 10 frames of ground truth are fed in and then the system is rolled out for the next 35 time steps. During the rollout phase, the proposed method performs better than the baselines in accurately predicting the dynamics of the balls as reflected by cross entropy (CE).
(a) Balls
00 Cursain
con-
Birary Cross Entrery
RMC
RING
R1M6 5M7
4%
45
32
Stops
Stops
Figure 9: Bouncing ball motion: Prediction er- ror comparison of the proposed method, LSTM, RIMs and RMC baseline. Given frames of ground truth, the model predicts the rollout over the next 35 steps. Here, we present the BCE for the 30th frame and 45th frame. The pro- posed SW extension performs better than other baselines in accurately predicting the dynamics, with an increasing advantage as the number of unrolled steps (30 vs 45) and balls ((a) (b)) increases. Results are an average over 5 random seeds.
We trained baselines well as proposed model for about 100 epochs. We use the same architecture for encoder as well decoder in (Van Steenkiste itt al. 2018). Hlyer-parametent specific to the pro- posed architecture are listed in Tab.
G INTEGRATING
SW WITH MORE ARCHITECTURES
G.1
TIMS
TIMs proposed by Lamb al. (2021). A transformer network is divided into 'independent mech- anisms' which update their state via sharing information between positions and sharing information between mechanisms. The information sharing step between mechanisms can be replaced by SW to create TIMs+SW.
G.1.1 MULTIMNIST GENERATION
In this task, we train an Image Transformer/ PaatnnerrPraaaaa Parmar et al. (2018) pixel-by-pixxe raster-order generative model) for next pixel prediction task the "MultiMNIST dataset"
19

---

Published as a conference paper at ICLR 2022.
Parameter
Value
Common Parameters
Optimizer
Adam(Kingma and Ba
2014
Learning rate
10
Batch size
12
Number of attention heads
8
TR
Size of transformer layer
256
TIMs
Number of mechanisms
4
Size of mechanism
48
TIMs
Number of mechanisms
Size of mechanism
40
Number of memory slots
2
Size of memory slots
160
Memory Attention Heads
8
Gate Style
unit'
Number of MLP layers in Attention
2
Table 4: Hyperparameters for MultiNNIST Task
Each 32 x 32 image in this dataset is made up of four randomly selected (and augmented) MNIST digits (resized to 32 x 8) placed side-by-side as shown figure T0 The digits themselves are selected independently of one-another.
The main aim of creating such a task is to observe the working of independent mechanisms in architectures such TIMs (Lamb et al. (021). Each image in the MulúMNIST dataset can be broken down into different sets of independent spatial components. Since the digits which make up the image independently selected, the joint distribution of pixel intensities in any one of the four sections of the image is statistically independent of the pixel intensities in any other section of the image. Moreover each section of the image be further broken down into independent spatial components: one that pertains to the background and one that pertains to the foreground.
is expected that a monolithic architecture (having single computational unit) would have devote a significant portion of its training to learn the statistical independence between the different con- stituents of the image. On the other hand, architectures made up of sparsely interacting independent mechanisms have natural way of capturing such statistical independence. A division of labour where each mechanism focused on the generation of distinct independent constituent of the image should allow for better generalization on the test set. Once the generation of constituent is completed, the task can handed over to some other mechanism based on current position in the image.
For this experiment we train a standard transformer with shared parameters across layers (denoted by TR), TIMs (Lamb et al.] (022)) with 4 mechanisms, and a modified version of TIMs with 4 mecha- nisms where the pair-wise communication between the mechanisms replaced by communication via shared workspace (denoted by TIMs+SW).
Training. We follow the minGPT Image Transformer setup Karpathy (022)) for our experiments. All three of the configurations have 8 layers, 8 heads for multi-headed attention and the exact same parameter initialization and base architecture. We train all three of the models for 20 epochs.
In the TR model, all of the 8 monolithic layers share the set of parameters. In TIMs and TIMs+SW, the first two layers are the standard monolithic layers having shared parameters. The middle four layers in both of these architectures are modular layers with four mechanisms. These four
20

---

Published as a conference paper at ICLR 2022.
Model
Loss
TR
0.000058
TIMs (4 mechanisms)
0.000050
TIMs+SW (4 mechanisms) 0.000042
Table 5: MultiMNIST Generation Task: We report cross-entropy loss between the generated pixel values and the true pixel values the test set of MultiMNIST Generation Task (smaller numbers are better)
41
4
38
20
3
S
5
700
0/9
3
00
7
0S
5
8
7765
16
1595
00
65 u20 12 4
4696
9
Figure 10: A randomly selected batch of 16 images from the MultiMNIST generation dataset (4 rows and 4 columns)
layers share the same set of parameters. the case SWWs the four mechanisms these layers communicate via shared workspace (having 2 memory slots). This shared workspace is common for all four middles layers and is absent in TIMs where the mechanisms communicate via pair- competition as proposed in the original paper. TIMs and TIMs+SW architectures are concluded by two more monolithic layers which again share the same parameters.
For all three models to have comparable number of parameters, chose the transformer embedding dimension to be 256 for TR model, 192 for TIMs model and 160 for TIMs+SW model. In TIMs and TIMs the embedding dimension is divided equally among the four specialists. Each memory slot the shared workspace of the TIMs+SW model has 160 dimensional embedding and the model uses four heads to perform read and write operations on the shared workspace. Total number of parameters for all three architectures lie between 1M and 1.8M.
Results. We observe the best cross-entropy loss in 20 epochs the dataset for the next pixel prediction task in the table [ We further plot the sixth layer "mechanism activation score" of TIMs and TIMs+SW while generating the first four images of the test in the best epoch (shown in figure 5h.
of the MultiNNIST
G.1.2 USING WORKSPACE FOR LANGUAGE MODELLING
We train our models on the WikiText-103 dataset by posing a language modeling problem. The dataset divided into train, test and validation sets which are composed out of 28.475, 60 and 60 articles respectively. The total number of tokens in the train set is more than 103 million, hence the name of the dataset. This dataset retains numbers, punctuation and case.
Training. We train our models for 15 epochs for the next word prediction task the WikiText- 103 dataset and report the perplexity on the validation set. We show the results using TIMs (Lamb) al. 2021) with 4 mechanisms and TIMs+SW with 4 mechanisms (where we replace the pairwise communication in TIMs with communication via a shared workspace like in the MultiNNIST experiment). We modify the FAIRSEQ Ott et al. (2019) transformer language model class for all of our experiments.
For TIMs+SW. we train and test two different variants: TIMs+SSW uses soft attention to generate the activation scores of competing independent mechanisms whereas TIMs+HSW uses top-k attention with k=2.
21

---

Published as a conference paper at ICLR 2022.
Since in this test, our aim is to compare the performance of the two models for the language modeling task, the architectures are only made up of transformer decoder. In both of the models, there are transformer decoder layers divided into 3 sets. The first layers are standard monolithic decoder layers which share the same parameters. The next 4 layers are modular layers (TIMs layers or TIMs+SW layers depending the model choice). These layers also share the same parameters among themselves. The last 2 layers are again standard monolothic decoder layers, both sharing the same parameters.
The inputs to the network are 1024 dimensional word embeddings, input to a transformer layer of dimension 1024 and feed forward dimension of 2048.
Both of the networks have 8 attention heads with head dimension of 128. The total transformer layer size of 8 x 128 1024 is equally divided among the four mechanisms. In the case of TIMs, these mechanisms (in layers 3,4.5) interact via pair-wise communication, whereas in TIMs+SSW and TIMs+HSW, these mechanisms interact via shared workspace. The shared workspace has memory slots, each 1024 dimensional, having 4 attention heads for reading and writing.
Parameter
Value
Common Parameters
Optimizer
(Kingma and Ba
2014
Learning rate
5
10
Adam betas
0.99,
0.98
Weight decay Ir scheduler
0.01
'inverse square root'
Max tokens per gpu
3078
Batch size multiple
8
Number of attention heads
8
Transformer layer size Number of Mechanisms
1024
4
Update frequency
4
Number of warmup updates
4000
Starting Warmup Ir
1-10 10-
TIMs+SSW
Number of memory slots
2 1024
Size of memory slots
Memory Attention Heads
Gate Style
'unit'
Number of MLP layers in Attention
top-k competition
False
TIMs+HSW
Number of memory slots
2
Size of memory slots
1024
Memory Attention Heads
Gate Style
'unit"
Number of MLP layers in Attention
top-k competition
True, k=2
Table 6: Hyperparameters for WikiTexxt-003 Language Modeling Task
Results. We plot the perplexity (per epoch) on the validation set. All models have comparable number of parameters (within a 10% difference). We note that TIMS performs poorly on this dataset but adding shared workspace improves the performance consistently. We also note that sparsity indeed helps as TIMs+HSW performed the best.
22

---

Published as a conference paper at ICLR 2022.
TIRS-5ON -55
Figure 11: Per epoch validation perplexity for TIMs, TIMs+SSW, TIMs+HSW for wikitext- -03 language modeling task
23


Reviews:
[{"review_id": "kuCxacSeCW1", "reviewer": "Reviewer_EPxc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper! I think the strengths fall into three main categories: good problem setup that took the reader from concept to formulation well, an interesting idea with good scoping, and an impressive experimentation suite. More details:\n\n**Good problem setup:** I found the discussion of advantages of a shared workspace compelling. The proposed structure is intuitive and simple, but nonetheless convincing about its claimed advantages of higher-order communications, dynamic filtering, and complexity advantages. The qualitative parts of the writing are strong, effectively building a case for reading this paper. \n\n**Interesting idea with good scoping:** I appreciate that this is one architectural change that considers all inputs and outputs, and shows its advantages over the right baselines. This work avoids the trap of making minor changes and then being unable to really ablate them. The paper does seem to be a unification of existing ideas, but I distinguish that from (and prefer it over) merely concatenating existing ideas. \n\n**Impressive experimentation suite:** There are of course many experiments with thematic consistency, which is great - always better than papers with two testbeds and virtually the same experiment over and over. The first four testbeds make a nuanced point about sparse inputs, and the results on Atari are particularly exciting because they make a general claim and show significant improvement. \n\nMy critiques are as follows, in order of significance: \n\n**Content**\n1. Main critique: the goals of this paper seem to be to show that accuracy and performance improve **overall**, with a mechanism of higher order communications, dynamic filtering, and linear complexity that we will be convinced are attributes of the SW. This paper doesn't quite get there, though it gets part of the way there. \n\nFrom my understanding, Triangles, CATER, and Sort-of-CLEVR all tell us that the higher-order communication and single channel will help identify relevant information earlier in the pairwise communication, leading to faster convergence - *if* information is sparse in the input. The MNIST generation experiment tells us something similar about independent regions in inputs. Figs 4 and 5 are convincing to this end. However, the main claim - higher-order communication and a sparser connection of graphs - help for inputs with sparse information in various locations - is less surprising than a claim of general improvement. Even if that's still interesting (it is), the intro and conclusion seem to make a more general claim. Physical reasoning and Atari show more general improvement (and the results on Atari are impressive), but they're also the weakest, most qualitative parts of Sec 4. The ideal solution would be more robust testing of general improvement in these two testbeds, but if not then I would at least appreciate a claim that acknowledges the common and specific nature of the first four testbeds. \n\nThis is the crux of the motivation behind my score: I worry that this paper overstates its claim and the real claim is elegant but not that surprising. It's made even murkier by a couple promising results that receive little attention. \n\n2. Relatedly, the Atari section makes a claim about considerable improvement due specifically to more appropriate modularization. That would be cool and a real testament to this approach, but I would need more specific experimentation to demonstrate that the effect really was more appropriate modularization. \n\n**Presentation**\n1. The mathematical formulation is quite clear, but it's not presented clearly. I had to puzzle over the paragraph for a while, not because it was overly complicated but because it took a lot of treasure hunting to find all the parts and put them together. The issue is that the *Notation* and *Step 1/2/3* sections write the math into large blocks of text without pause. It would be more helpful to present some end-to-end equations, then break those down in text. It can be unclear without doing it myself whether something is being element-wise updated, transformed, dotted, etc. \n\n2. I really appreciate the slew of testbeds. However, that's how they come across: a laundry list of experiments, one after another, even though they support similar claims. That robustness can be shown by having a claims-driven structure for the results section rather than a testbed-driven structure, which to the reader is arbitrary and doesn't deliver the important information as well as it could. E.g. section titles like \"Performance advantage on sparse data\", then talk about the different results that show it. If one experiment shows multiple advantages, multiple sections can still point to the same figures. \n\n3. The figures are confusing - they are cramped, small, and have lots of acronyms I have to hunt down in various parts of the paper. They're all helpful content-wise, but this could be shown better with something as simple as better aesthetics and tagging.\n\n4. Nit: the text seems to switch between \"specialist\" and \"neural module\". Are these the same thing? Is \"specialist\" an abstraction/metaphor for \"neural module\"? Is \"neural module\" an example of \"specialist\"? All of those are great, just signpost and explain the switch. \n\n \n", "summary_of_the_paper": "This paper proposes a communication framework to have multiple modules communicate and switch precedence efficiently, taking inspiration from Global Workspace Theory in cognitive science. The primary contribution is a scheme to replace complete pairwise communications in modularized architectures with a single, limited-capacity workspace that persists and changes over stages of computation. This workspace is implemented with a read/write scheme that iterates through stages of computation (layers in a Transformer or steps in a recurrent architecture): in the write phase, the shared memory is updated according to the current states of the modules that are most informative to the shared memory's current state as determined by a key-value attention scheme, with the modules competing via softmax. In the read phase, the internal states of all modules are updated via another key-value attention scheme. The advantages are claimed to be 1) higher order interaction among modules because every module learns from every other one (at least, more than pairwise), dynamic filtering because the memory persists and updates stage to stage; and linear computational complexity because the number of memory slots doesn't change much (and it's typically small, 1-10).  \n\nThe related work motivates this paper by the classical AI principle that intelligent systems should have multiple specialized modules rather than one general entity. It distinguishes from prior slot-based memory work in that memory writes here are sparse and competitive, and prior work on reducing computational complexity of Transformer dot product attention through its persistent memory. It aptly assesses itself as a unification of existing ideas. \n\nThe experimentation section tests different parts of the proposed scheme. The triangles experiment tests the comparative speed to convergence (and accuracy) of the HO communications here compared to pairwise in baseline Transformer. The MNIST generation experiment with TIMs shows that the shared workspace gives an advantage on domains where input dimensions are mostly independent. The CATER experiment shows a similar result to the triangles experiment (quickly picking out only relevant information) but in a time series, and the Sort-of-CLEVR experiment again reinforces the power of the shared workspace on sparse tasks. The physical reasoning shows general improvement, and the Atari performance shows considerable improvement due to modularization. \n", "summary_of_the_review": "**Strengths:**\n1. Good problem setup\n2. Interesting idea with good scoping \n3. Impressive experimentation suite \n\n**Critiques:**\n*Content:*\n1. The experimentation section mainly convinces me that this architecture is good for accuracy and convergence speed on sparse inputs, not that it's generally advantageous - but the latter seems to be the claim of the paper, and the former is less surprising given the nature of the architecture (though still interesting - it's compelling to see that the architecture works). \n2. The experimentation on general improvement is a bit weak. \n\n*Presentation*\n1. Formulation is clear, but it's presented piecemeal in long paragraphs that make it hard to parse and put together. Since the formulation is simple (a good, elegant thing), it's frustrating to spend a lot of time hunting down all the pieces.\n2. Excellent thoroughness in testbeds, but the results don't have to be presented in such a process-driven way. They can and should be claims/driven. \n3. The figures need to be more readable \n4. \"Specialist\"/\"neural module\" distinction is unclear (see main review)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FSBELVrj7I-", "reviewer": "Reviewer_2GPN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper is written very well. The introduction provides a good motivation and challenges and illustrates the use cases clearly using examples. The related work compares the proposed approach with other memory-based neural models. The description of methods is concise, but sufficiently detailed.\n\n- The proposed approach is original to the best of my knowledge. The key idea is very intuitive and motivated by insights from cognitive science literature.\n\n- The authors perform experiments on a wide variety of tasks including a toy task for detecting equilateral triangles, multi MNIST Generation, object tracking, relational reasoning, physical reasoning and Atari video games. I also like that the authors use different types of backbones (Transformers and RIMs) in different experiments which indicates the proposed Shared Workspace method is not specific to certain kind of backbones.\n\nWeaknesses:\n\n- Although the authors perform experiments with wide variety of tasks, multi-agent tasks are missing where I believe coordination is more important. In all the tasks used in the paper, different neural modules process different parts of the input to make a common prediction. Multi-agent tasks are more challenging as each agent would get a different input but also predict a different action.\n\n- It is unclear how certain hyper parameters are chosen. For example why is the patch size 4x4 in equilateral triangles, 6x6 in object tracking? The number of memory slots, size of memory slots, etc. different across different experiments. How are these chosen?\n\n- The authors propose two versions of Shared Workspaces, soft and hard (SSW and HSW). Some experiments one contain a single version. Why are both SSW and HSW not evaluated in all the experiments?", "summary_of_the_paper": "This paper presents a method of using external memory called \"shared workspace\" for communication among different neural modules or \"specialists\". The key idea is that there are limits on the communication bandwidth and the specialist modules must compete for access. The communication limit encourages specialization and compositionally and facilitate better synchronization. Experiments over a variety of tasks indicate the proposed Shared Workspace model improves performance over baselines.", "summary_of_the_review": "The paper proposes a novel method for coordination between neural modules which is well motivated. The experiments are comprehensive although some details are missing. Addition of multi-agent tasks would make the paper stronger.\n\nEdit after author response:\nI read the author response and I would like to maintain my positive rating. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Bn2Xqnb3LKT", "reviewer": "Reviewer_Cdd9", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. Novel and significant contribution by proposing a new mechanism for inter-subnets information exchange/learning.\n2. Of high relevance to the ICLR community. The generic approach cuts across different AI discipline as the experiments are performed on vision, language and reasoning tasks.\n3. Experiments are comprehensive and well-explained.\n\nWeaknesses\nMinor: Some technical details are not elaborated for readers who are less familiar with the respective topics. For example, the section on the key-query-value attention mechanism is quite brief. It's the key contribution on the success of the proposed approach.", "summary_of_the_paper": "Paper proposes a novel mechanism for information exchange between different neural subnets. It replaces the pairwise interactions with a share memory space. The memory is updated by a competitive scheme which the top-k updates are selected via the key-query-value\nattention mechanism. The shared memory/workspace then broadcast the updated state to all other upstream specialist subnets. Each specialist subnet then update its representation/weights accordingly.\n\nPaper claims 3 advantages of this approach: \n1. Higher-order (HO) interaction among neural modules\n2. Dynamic filtering due to persistence of memory. \n3. Lower computational complexity of using shared workspace for synchronizing different specialists\n\nExperimental results are very promising across a wide variety of tasks, namely DETECTING EQUILATERAL TRIANGLES; SORT-OF-CLEVR; CATER: OBJECT TRACKING;  for transformer architectures. Also, experiments on RIM for the BOUNCING BALL task and TIMS for the MULTIMNIST GENERATION and Language Modeling Task were also done. Finally, the approach was also experimented on model  free RL on ATARI game task.", "summary_of_the_review": "Paper proposes a novel approach which produces commendable experimental results across a wide range of tasks in vision, language and reasoning. The impact of this work is very significant and is of high relevance to the community. While I am not entirely familiar with the cited prior work on key-query-value attention mechanism which forms the backbone of the proposed approach, the entire paper is very well-written and comprehensible. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-LpTL33EiGz", "reviewer": "Reviewer_pjX4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. Interestingness of the approach\n\nThe proposed method is appealing since it applies a theory from cognitive science (the global workspace theory) to information processing in networks. The implementation of the idea is quite simple - individual ‘specialist’ units interact with a shared memory layer instead of with each other directly, and the shared memory then broadcasts information back to each of the specialists. The exact means by which this interaction is carried out is by using key-value attention. This approach when applied to transformers leads to much higher computational efficiency (linear instead of quadratic in the sequence length).\nFurther research in probing similar ideas for much more efficient information processing, backed by ideas from cognitive science, would be very beneficial for the field in my view and this paper would help the community in that regard.  \n\n2. Experimental Evaluation \n\nThe paper includes exhaustive experimental evaluation over 5-6 environments (including object tracking, and relational reasoning) where the solution requires considering a small portion of the input data, the authors demonstrate that adding the shared workspace model to attention based architectures like Transformers and RIMs leads to superior asymptotic performance, and faster learning. These environments include object tracking (CATER), relational reasoning (sort of CLEVR) and Atari games. The authors also include details about the experiments and algorithm implementation in the appendix, which will aid in reproducibility. \n\nWeaknesses \n\n1. Unclear if the approach will scale to larger, more unstructured datasets (where current attention based architectures have already been shown to thrive)\n\nMost of the experiments considered involve problems that only require a small portion of the input to be solved (eg: just the patch with the points for the equilateral triangles, or just the target object for Cater object tracking). I am concerned whether the framework proposed here will also be effective in settings where this is not necessarily the case, such as general modeling of language and images. For example, adding shared workspace to transformers imposes a communication bottleneck between representations at different positions of the sequence. It is possible that for problems where the solution does not depend on only a small portion of the input, considering the pairwise relationships of representations at every point in the sequence is critical for good performance. \nTo study this, the shared workspace model would have to be evaluated on larger, more unstructured datasets, where transformer based architectures have already been demonstrated to do well (for example the data on which GPT-2 was trained). Adoption of this approach would be much more widespread if the authors can demonstrate that on these larger datasets, training transformers with the shared workspace doesn’t lead to worse performance than training the regular transformer based models that are currently used. ", "summary_of_the_paper": "The paper proposes a modification for attention-based network architectures drawing inspiration from the global workspace theory in cognitive science. Essentially connections are made sparser, with different ‘specialist’ units communicating with each other through a limited bandwidth channel.\n", "summary_of_the_review": "The paper proposes an implementation of an interesting theory from cognitive science for more efficient information processing in networks, by making connections between entities sparser. While there is extensive evaluation on environments, most of these involve processing a small portion of the input. The paper will be made a lot stronger if the approach can be shown to scale to larger more unstructured datasets where transformers are known to work well. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper takes inspiration from Global Workspace Theory to propose a modification for attention-based network architectures. This is exemplified both in transformer models and in recurrent models (RIMs). The key idea is to replace the quadratic, pairwise communication between "specialist" units (which in transformers corresponding to the positions) by a higher-order communication model which consists in a competitive, sparse writing step into a shared workspace, followed by a reading step where information is broadcasted from the global workspace to all specialists. The competitive writing step establishes a limited bandwidth channel for this communication which encourages specialization.

The reviewers agree that this is an interesting and very well-written paper which unifies several existing ideas. The main contribution of this paper is in establishing a connection to GWT which may inspire future research to keep developing these ideas. The experiments on relatively small tasks (but challenging ones) provide a good proof of concept. Some concerns pointed out by some of the reviewers include a certain overstatement of the capabilities of the proposed model, as well as lack of experiments that scale up the model to larger and unstructured datasets. The authors replied with additional experiments included in the appendix, which in my opinion address these concerns convincingly. 

Overall, this is a strong paper and I recommend acceptance. I encourage the authors to take into account the reviewer's suggestions in the final version. I also think that the connection to related work could be improved, as there is several related works [1, 2, 3] which asks/investigates similar questions to this paper and should probably be acknowledged:
- The "shared global workspace" of this paper (Transformer + SW) is reminiscent of the Star-Transformer [1], as well as other more recent works which use special units (e.g. CLS tokens) to encode "global" representations. While that work does not include the competitive component (the "bottleneck"), I think it should be acknowledged.
- Variants of transformers with competition among specialists via sparsity have also been proposed, e.g. adaptively sparse transformers [2]. That framework is an alternative to top-k softmax used in this paper.
- Empirical studies which analyze the redundancy among specialists (in this case attention heads) and propose strategies to prune them have also been made by [3]. 

[1] https://arxiv.org/abs/1902.09113
[2] https://arxiv.org/abs/1909.00015
[3] https://arxiv.org/abs/1905.09418 

Minor point: "Hence unlike pairwise interaction, messages passed among neural modules in the shared workspace setting also include HO interaction terms" -- I believe higher-order interaction happens too every two layers with pairwise interaction. Perhaps this should be clarified.
Decision: Accept (Oral)