Discrete-Valued Neural Communication
Authors: Dianbo Liu
Alex Lamb
Kenji Kawaguchi
Anirudh Goyal
Chen Sun
Michael Curtis Mozer
Yoshua Bengio
Venue: NeurIPS 2021 Poster (2021)
Keywords: structured architecture
discretization
communication
specialist components
system 2

Abstract: Deep learning has advanced from fully connected architectures to structured models organized into components, e.g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes. The nature of structured models is that communication among the components has a bottleneck, typically achieved by restricted connectivity and attention. In this work, we further tighten the bottleneck via discreteness of the representations transmitted between components. We hypothesize that this constraint serves as a useful form of inductive bias. Our hypothesis is motivated by past empirical work showing the benefits of discretization in non-structured architectures as well as our own theoretical results showing that discretization increases noise robustness and reduces the underlying dimensionality of the model. Building on an existing technique for discretization from the VQ-VAE, we consider multi-headed discretization with shared codebooks as the output of each architectural component. One motivating intuition is human language in which communication occurs through multiple discrete symbols. This form of communication is hypothesized to facilitate transmission of information between functional components of the brain by providing a common interlingua, just as it does for human-to-human communication. Our experiments show that discrete-valued neural communication (DVNC) substantially improves systematic generalization in a variety of architectures—transformers, modular architectures, and graph neural networks. We also show that the DVNC is robust to the choice of hyperparameters, making the method useful in practice.

Discrete- Valued Neural Communication in Structured Architectures Enhances Generalization
Dianbo Liu"
Alex Lamb' Mila
Kenji Kawaguchi Harvard University
Mila
Anirudh Goyal Mila
Chen Sun
Michael C. Mozer Google Research, Brain Team
Yoshua Bengio Mila
Mila
Abstract
Deep learning has advanced from fully connected architectures structured models organized into components, c.g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes. The nature of structured models is that communication among the components has a bottleneck, typically achieved by restricted connectivity and attention. this work, further tighten the bottleneck via discreteness of the representations transmitted between components. We hypothesize that this constraint serves as a useful form of inductive bias. Our hypothesis is motivated by past empirical work showing the benefits of discretization in non-structured architectures as well as our own theoretical results showing that discretization increases noise robustness and reduces the underlying dimensionality of the model. Building on an existing tech- consider multi hellidheaded discretization
nique for discretization from the VQ-VAE. with shared codebooks as the output of each architectural component. One motivat- ing intuition human language which communication occurs through multiple discrete symbols. This form of communication is hypothesized to facilitate trans- mission of information between functional components of the brain by providing a common interlingua, just it does for human-to-He to-hummnn communication. Our ex- periments show that discrete-valued neural communication (DVNC) substantially improves systematic generalization in a variety of architectures ttanssormrrsss modular architectures, and graph neural networks. We also show that the DVNC is robust to the choice of hyperparameters, making the method useful in practice.
Introduction
In AI, there has long been tension between subsymbolic and symbolic architectures. Subsymbolic architectures. like neural networks, utilize continuous representations and statistical computation. Symbolic architectures, like production systems (Laird et al. 1986) and traditional expert systems, use discrete, structured representations and logical computation. Each architecture has its strengths: subsymbolic computation is useful for perception and control, symbolic computation for higher level, abstract reasoning. A challenge in integrating these approaches is developing unified learning procedures.
As a step toward bridging the gap, recent work deep learning has focused on constructing structured architectures with multiple components that interact with one another. For instance, graph neural networks are composed of distinct nodes (Kipf al. 2019 Scarselli al. 2008 Kipf al. 2018
"co- first author. Address correspondence to 1uudiambodgmail.com alex6200@gma1..oom and kkavaguchi@fas harvard.edu edu.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2017 Raposo et al. 2017; Bronstein et al. 2017 Gilmer et al. 2017 Tacchetti et al. Santoro et al.,
2018
Van Steenkiste et al. 2018). transtormers are composed of positional elements [Bahdanau et al.
2014
Vaswani et al. 2017 7) and modular models are divided into slots or modules with bandwidth
limited communication (Jacobs et al., 1991; Bottou and Gallinari] 1991
and Bengio, 2020 1997/Reet and De Freitas 2015] Lamb et al. 2021; Andreas et al. 2016 Rosenbaum
Goyal |
Ronco et al.
ct al. 2017
Fernando et al., 2017 Shazeer et al. 2017 Rosenbaum et al. 2019 Kucinski et al.
2021
Although these structured models exploit the discreteness in their architectural components, the present work extends these models to leverage discreteness of representations, which is an essential property of symbols. We propose learn common codebook that is shared by all components for inter-component communication. The codebook permits only discrete set of communicable values. We hypothesize that communication based on the use and re-use of discrete symbols will provide two benefits:
Discrete symbols limit the bandwidth of representations whose interpretation needs to be learned and synchronized across modules. It may therefore serve common language for interaction and facilitate learning.
The use of shared discrete symbols will promote systematic generalization by allowing for the re-use of previously encountered symbols in new situations. This makes it easier to hot-swap one component for another when new out-of-distribttion (OOD) settings arise that require combining existing components in novel ways.
Our work is inspired by cognitive science, neuroscience, and mathematical considerations. From the consider different components of structured neural architectures
cognitive science perspective, to be analogous to autonomous agents in a distributed system whose ability to communicate stems from sharing the same language. If each agent speaks different language, learning to communicate would be slow and past experience would be of little use when the need arises to communicate with new agent. If all agents learn the same language, each benefits from this arrangement. To encourage a common language, we limit the expressivity of the vocabulary to discrete symbols that can be combined combinatorially [oomasello] (009). From the neuroscience perspective, we note that various areas in the brain, including the hippocampus (Sun et al., 2020 Quiroga et al. 2005 Wills al. 2003), the prefrontal cortex (Fujii and Graybiel 2003), and sensory cortical areas (ssa et al. 2006) are tuned to discrete variables (concepts, actions, and objects), suggesting the evolutionary advantage of such encoding, and its contribution to the capacity for generalization the brain. From theoretical perspective, we present analysis suggesting that multi-head discretization of inter-component communication increases model sensitivity and reduces underlying dimensions (Section D. These sources of inspiration lead us to the proposed method of Discrete-Vare Valued Neural Communication (DVNC).
Architectures like graph neural networks (GNNs), transformers, and slot-based or modular neural networks consist of articulated specialist components, for instance, nodes in GNNs, positions in transformers, and slots/modules for modular models. We evaluate the efficacy of DVNC in GNNs, transformers, and in a modular recurrent architecture called RIMs. For each of these structured architectures, we keep the original architecture and all of its specialist components the same. The only change is that we impose discretization the communication between components (Figure [
Our work is organized as follows. First, we introduce DVNC and present theoretical analysis showing that DVNC improves sensitivity and reduces metric entropy of models (the logarithm of the covering number). Then we explain how DVNC can be incorporated into different model architectures. And finally we report experimental results showing improved OOD generalization with DVNC.
2 Discrete- Value Neural Communication and Theoretical Analysis
In this section, we begin with the introduction of Discrete- Value Neural Communication (DVNC) and proceed by conducting theoretical analysis of DVNC affects the sensitivity and metric entropy of models. We then explain how DVNC can be used within several different architectures.
Discrete- Eiscrete-Value Neural Communication (DVNC) The process of converting data with continuous attributes into data with discrete attributes is called discretization (Chmielewski and Grzymala- Busse

---

Node
Discretitation-
Step
Step
layer
Result attention
Sum edges
Node 1
Attertion
disceciaation
Discretization
Attention
Resuh of atternion
Node
Inpuc
Inout?
Inputt
<OS>, Position1, Jamny N, <EOS>
(a) DVNC in Graph Neural Network
(c) DVNC in Modular Recurrent Mccha- nisms (RIMs)
(b) DVNC in Transformer
Figure 1: Communication among different components in neural net models is discretized via a shared codebook. In modular recurrent neural networks and transformers, values results of attention are discretized. In graph neural networks, communication from edges is discretized.
1996 In this study, we use discrete latent variables to quantize information communicated among different modules similar manner as in Vector Quantized Variational AutoEncoder (VQ-VAE) (Ordd et al. 2017). Similar to VQ- VA-, we introduce a discrete latent space vector e E (mx size of the discrete latent space (i.e., an I--way categorical variable), and m the dimension of each latent embedding vector ej. Here, L and m are both hyperparameters. In addition, by dividing each target vector into G' segments or discretization heads, we separately quantize each head and concatenate the results (Figure 2). More concretely, the discretization process for each vector E 71 C R is described as follows. First, we divide a vector h into G segments $1, $2,.... ,,, $g sG); where each segment 8, Rm/ with E N+ Second, we discretize each segment separately:
where
with h = CONCATENATE(s1) (11, 82...., 82;
= DISCRETIZE(s:), where = argmin - ||sl- je(1....4)
Finally, we concatenate the discretized results obtain the final discretized vector
Z CONCATENATEEDIISCRTTER DISCETIZEss) ..., DISCRETIZE(SC)).
The multiple steps described above can be summarized by Z q(,, L, G). where q(-) is the whole discretization process with the codebook, L is the codebook size, and G is number of segments per vector. It is worth emphasizing that the codebook is shared across all communication vectors and heads, and is trained together with other parts of the model.
The overall loss for model training is:
G
G
L = Crask +
1 G
E1ss((s))
co, 112 +
sg((o,)))
where Lrask is the loss for specific task, e.g., cross entropy loss for classification or mean square error loss for regression, sg refers to a stop-gratd gradient operation that blocks gradients from flowing into its argument, and B is a hyperparameter which controls the reluctance to change the code. The second term Σ sg({s) 1 is the codebook loss, which only applies to the discrete latent vector and brings the selected cg, close to the output segment s;. The third term Σ ||s; sg(eo,) || is the commitment loss, which only applies the target segment s; and trains the module that outputs s; to make 8, stay close to the chosen discrete latent vector ep,. We picked / = 0.25 as in the original VQ-VAE paper (Oord et al., 2017/. We initialized e using k-means clustering
vectors h with = L and trained the codebook together with other parts the model by gradient descent. When there were multiple h vectors to discretize in model, the mean of the codebook and commitment loss across all h vectors was used. Unpacking this equation, it can be seen that we adapted the vanilla VQ-VAE loss directly suit our discrete communication method (Oord et al.) 2017). In particular, the VQ-VAE loss was adapted to handle multi-headed discretization by summing over all the separate discretization heads.
In the next subsection, we use the following additional notation. The function & is arbitrary and thus can refer to the composition of evaluation criterion and the rest of the network following discretization. Given any function Φ 1" + E and any family of sets = {SI...., Sk. ) with Si...., Sx C H. we define the corresponding function of by (h) k [K], where [K] {1, 2..... all the possible values after the discretization process: i.c., q(h, L, G) Useue {Qx} for all h H.
1fh Sa) for all
., K). Let E R f.x be fixed and we denote by (Q
3

---

Figure 2: In structured architectures, communica- tion is typically vectorized. In DVNC, this commu- nication vector is first divided into discretization heads. Each head is discretized separately to the nearest neighbor of a collection of latent codebook vectors which is shared across all the heads. The discretization heads are then concatenated back into the same shape as the original vector.
ht
Zt
h2
Shored codebook
7.2
h/3
2
Nearest neighbor
Table I: Communication with discretized values achieves noise-sensitivity bound that is independent of the number of dimensions m and network lipschitz constant & and only depends on the number of discretization heads G' and codebook size L.
Communication Type
Example
Sensitivity Bounds (Thm
Communication with continuous signals expressive but can take huge range novel values, leading to poor systematic generalization
Se RyL
2n
10
"John owns a car"
Communication with multi-head discrete-values is both expressive and sample efficient
G 15
o
L 30
This subsection shows that adding the discretization process has two potential advantages: (1) improves noise sensitivity or robustness and (2) reduces metric entropy. These are proved in Theorems 22 illustrated by examples (Table I. and explored in analytical experiments using Gaussian-distribute vectors (Figure 3).
Theoretical Analysis
To understand the advantage on noise sensitivity or robustness, we note that there is an additional error incurred by noise without discretization, i.c., the second term 5u. Rx / yn 2 0 in the bound of Theorem
2 (St and Rz are defined in Theorem 2 This term is defined to be the noise sensitivity (or robustness) term. This term due to noise disappears with discretization in the bound of Theorem Tas the discretization process reduces the sensitivity to noise. This is because the discretization process lets the communication become invariant to noise within the same category; e.g., the communication is invariant to different notions of "cats".
To understand the advantage of discretization on dimensionality, can that it reduces the metric entropy of m In (144)))) without discretization (in Theorem 2 to that of G Im(L) with discretization (in Theorem D. As a result, the size of the codebook L affects the underlying dimension in a weak (logarithmic) fashion, while the number of dimensions m and the number of discretization heads G scale the underlying dimension the metric entropy for any n 2 1 as long as G G'In(L) < m In(4/m). This is nearly always the case as the number of discretization heads Intuitively, a discrete language has combinatorial expressiveness, making it able to model complex phenomena, but still lives much smaller space than the world of unbounded continuous-valued signals (as G' can be much smaller than m.)
linear way. Thus, the discretization process successfully lowers
almost always much smaller than the number of units m.
Theorem 1. (with discretization) Let S (Qx) for all l E [L Then, for any 8 O, with probability at least 1 - 8 over an iid draw of n examples (ha)- the following holds for any
O:R' + R and all l [] if l (h)) S for all h E H. then
Σ (a(h,, L,))) = =0 (o(a)
G Gln(L) (In(L)+In220))
En (g(h, L, G])
(2)
2n
where no constant is hidden in O.
Theorem 2. (without discretization) Assume that |1Al|2 < Rx for all h H C RT. Fix C
argming { [C]: : C RT H C UlecBll]] where Bic = {o E Rm || - cll2 S RH/(2/T) Let Se for all k [[C[] where c E C and Uj (c(c C. Then, any ô 0, with probability at
least 1 - 6 over an iid draw of n examples (h,)²,, the following holds for any 6 R + R and all
k E [[C]] ]: if (0g (h)) S a for all h E H and (b) 25 S sı h 12' 1/112/2 for all h k E Sr. then
1
= =0 -o(o) m 1nln(4 + ln(2/8)
Sx Rx
E, 1 (h)
2e (ba))
2m
Vn
)
(3)

---

Figure 3: We perform empirical analysis Gaussian vectors to build intuition for theoretical analysis. Expressiveness scales much faster as we increase discretization heads than as we increase the size of the codebook. This be seen when measure the variance of collection of Gaussian vectors following discretization (left), and can also be seen when we plot a vector field of the effect of discretization (center). Discretizing the values from attention layer trained to select fixed Gaussian vector makes it more robust to novel Gaussian distractors (right). For more details, see Appendix B
where constant is hidden in O and & Sk Eh E B(cxl)).
The two proof' of these theorems use the same steps and are equally tight as shown in Appendix D Equation (3) is also as tight as that of related work as discussed in Appendix C.2 The set S is chosen cover the original continuous space H in Theorem 2 (via the c-covering C of H), and its discretized space in Theorem I Equations (2) 3 hold for all functions á : Rm + R. including the maps that depend on the samples (h,) via any learning processes. For example, we can set to be evaluation criterion of the latent space h or the composition of an evaluation criterion and any neural network layers that are learned with the samples (h,)² In Appendix A
we present additional theorems, Theorems 4 where we analyze the effects of learning the map F h and the
codebook e via input-target pair samples ((z,, y )=))==-
Intuitively, the proof shows that we achieve the improvement in sample efficiency when G and L small, with the dependency on G being significantly stronger (details Appendix). Moreover, the dependency of the bound on the Lipschitz constant && is eliminated by using discretization. Our theorems [ 2 are applicable all of our models for recurrent neural networks, transformers, and graph neural networks (since the function ø is arbitrary) in the following subsections.
Communication along edges of Graph Neural Network One model where relational information naturally used is in graph neural network for modelling object interactions and predicting the next time frame. We denote node representations by Ç, edge representations by eig. and actions applied each node by af. Without DVNC, the changes of each nodes after each time step is computed by </+ = Ç + AC, where ACς = fnode (C1, ai: Ejtr e) and e = fedge (C 25)-
In this present work, we discretize the of all edges connected each node with DVNC, So: AC fnode (C3, af {(Ee edd (2, L),
Communication Between Positions in Transformers In transformer model without DVNC, at each layer, the scaled dot product multi-hea soft attention is applied to allow the model to jointly attend information from different representation subspaces at different positions Vaswani et al. 2017) as:
Output = residual + MUTTIHEADAATEENTONNBBB.K.
where MULTIHEADATIENTION B, K, V) CONCATENATE head, = SOFTATTENTION BW KW. VWY). Here, W W" W and W are projection matrices, and B, K, and V are queries, keys, and values respectively.
....hedd,. W and
In this present work, we applied the DVNC process to the results of the attention in the last two layers of transformer model,
Output residual + -MMLLMMAAAAAAAATATTTNNOON o B, K, V), I,, G).
Communication with Modular Recurrent Neural Networks There have been many efforts to introduce modularity into RNN, Recurrent independent mechanisms (RIMs) activated different
5

---

modules at different time step based on imputs (Goyal et al. 2019). In RIMs, outputs from different modules are communicated each other via soft attention mechanism. In the original RIMs method, we have 2/+1 RNN (2j, x²) for active modules, and 2 a of for inactive modules, where t is the time step, i is index of the module, and T₁ is the input at time step t. Then, the dot product query-key soft attention is used to communication output from all modules E , M such that 11 1441= SOFTATTENTION 22 =f+1 .....)77).
In this present work, we applied the DVNC process to the output of the soft attention, like so: 2/+1 + q( I,, G). Appendix AmpendixEEpreseeeee presents the pseudocode for RIMs with discretization.
3 Related Works
The Society of Specialists Our work is related to the theoretical nature of intelligence proposed by Minsky (1988) and others (Braitenberg] 1986] Fodor 1983), in which the authors suggest that an intelligent mind can be built from many little specialist parts, each mindless by itself. Dividing model architecture into different specialists been the subject of number of research directions, including neural module networks (Andreas et al., 2016)), multi-agent reinforcement learning (Zhang et al. 2019) and many others (Jacobs et al., 1991 Reed and De Freitas 2015 Rosenbaum al. 2019). Specialists for different computation processes have been introduced in many models such as RNNs and transformers (Goyal et al., 2019 Lamb et al. 2021 Goyal al. 2021b). Specialists for entities or objects in fields including computer vision (Kipf et al. 2019). Methods have also been proposed to taking both entity and computational process into consideration (Goyal et 2020). In a more recent work, in addition to entities and computations, rules were considered when designing specialists (Goyal et al. 202 Our Discrete- Valued Neural Communication method can be seen as introducing specialists of representation into machine learning model.
Communication among Specialists Efficient communication among different specialist compo- nents in a model requires compatible representations and synchronized messages. In recent years, attention mechanisms are widely used for selectively communication of information among specialist components in machine learning modes (Goyal et al. 2019 (2221b | a) and transformers (Vaswani et al. 2017: Lamb al., 2021) 2.201.ccoleeeee..e... collecttive memory and shared RNN parameters have also been used for
multi-agent communication (Garland and Alterman 1996 Pesce and Montana (220)). Graph-basce models haven been widely used in the context of relational reasoning, dynamical system simulation multi-agent systems and many other fields. In graph neural networks, communication among different nodes were through edge attributes that are learned from the nodes the edge is connecting together with other information (Kipf et al., 2019; [Scarselli et al. 2008 Bronstein et al.) 2017 7
Watters et al. 20177 Van Steenkiste et al. 201 8 Kipf et al. 2018: Battaglaa et al. 201 8 Tacchetti et al. 2018 Veerapaneni et al.] 2019) Graph based method represent entities, relations, rules and other elements as node, edge and their attributes in graph (Koller and Friedman 2009: Battaglia
ct al. 2018). In graph architectures, the inductive bias is assuming the system to be learnt can be represented as a graph. In this study, our DVNC introduces inductive bias that forces inter-component communication to be discrete and share the same codebook. The combinatorial properties come from different combinations of latent vectors in each head and different combination of heads in each representation vector. While most of inter-specialist communication mechanisms operates in pairwise symmetric manner, Goyal et al.) (2021)) introduced a bandwidth limited communication channel to allow information from limited number of modules to be broadcast globally to all modules, inspired by Global workspace theory (Baars) 2019 1993).our proposed method selectively choose what information can communicated from each module. We argue these two methods are complimentary to each other and can be used together, which we like to investigate in future studies.
4 Experiments
In this study we have two hypothesis: 1) The use of discrete symbols limits the bandwidth of commu- nication. 2) The use of shared discrete symbols will promote systematic generalization. immeraliiation.Teeoretiaa Theoretical results obtained in section [ agree with hypothesis 1. In order to verify hypothesis 2, in this sec- tion, we design and conduct empirical experiments to show that discretization of inter-component communication improves OOD generalization and model performance.

---

Table Performance of Transformer Models with Discretized Communication on the Sort-of- of-Clevr
Visual Reasoning Task.
Method
Ternary Accuracy
Binary Accuracy
Unary Accuracy
Transformer baseline
57.25 + 1.30
76.00 1,41
97.75 + 0.83
Discretized transformer (G=16)
61.33 + 2.62
84.00 + 2.94
98.00 + 0.89
Discretized transformer (G=8)
62.67 1.70
88.00 0.82
98.75 = 0.43
Discretized transformer (G=1)
58.50 + 4.72
80.50 + 7.53
98.50 + 0.50
(a) 2D Shapes
(b) 3D Shapes
(c) Three-Body Body
(d) Adding
(e) Seq. MNIST
Figure 4: GNN models ba,b and e) and RIMs with DVNC (d and e) show improved OOD generaliza-
tion. Test data in OOD to 3 are increasingly different from training distribution.
4.1 Communication along Edges of Graph Neural Networks
We first turn our attention to visual reasoning task using GNNs and show that GNNs with DVNC have improved OOD generalization. In the tasks, the sequences of image frames in test set have different distribution from data used for training. We set up the model in a identical manner as in Kipf al. 2019 (Kipf et al., 2019) except the introduction of DVNC. A CNN is used to extract different objects from images. Objects are represented as nodes and relations between pairs of objects represented edges in the graph. The changes of nodes and edges can be learned by message passing in GNN. The information passed by all edges connected to a node is discretized by DVNC (see details in Section (
The following principles are followed when choosing and designing GNN visual reasoning tasks well as all other OOD generalization tasks in subsequent sections: 1) Communication among different components are important for the tasks. 2) meaning of information communicated among of information
components be described in discrete manner. 3) In OOD setting.distributtee communicated among specialist components in test data also deviate from that in training data.
Object Movement Prediction in Grid World We begin with the tasks to predict object movement in grid world environments. Objects in the environments are interacting with each other and their movements are manipulated by actions applied to each object that give a push to the object in a randomly selected direction. Machine learning models are tasked to predict next positions of all objects in the next time step + 1 given their positions and actions in time step t. We can think of this task detecting whether object be pushed toward direction. If object is blocked by the environment or another object in certain direction, then it can not be moved in that direction. Positions of objects are captures by nodes in GNN and the relative positions among different objects can be communicated a discrete manner via message passing through edges.
We adapted and modified the original 2D shapes and 3D shapes movement tasks from Kipf et al. (2019) by introducing different number of objects training or testing environment. In both 2D and 3D shapes tasks .ive objects are available in training data, three objects are available in OOD-1 and only two objects are available in OOD-2. Our experimental results suggest that DVNC in GNN improved OOD generalization in statistically significant manner (Figure In addition, the improvement is robust across different hyperparameters G and L (Figure B). Details of the visual reasoning tasks set up and model hyperparameters can be found in Appendix.
Three-body Physics Object Movement Prediction Next, we turn our attention to a three-body- physics environment which three balls interacting with each other and move according to physical laws in classic mechanics in a 2D environment. There are no external actions applied on the objects. We adapted and modified the ftree-body-phyeee environment from (Kipf et al. 2019). In OOD experiment, the sizes of the balls are different from the original training data. Details of the
7

---

Table Graph Neural Networks benefit from discretized communication on OOD generalization in
predicting movement in Atari games.
Game
GNN (aaselin))
GNN (Diereiize)
F-Vale (ss, haseline)
Game
GNN (Raseline)
GNN (Disrrdize))
P-Valuc (vs. Inseline)
Alien
0,1991 0.0756
0.2876 0.0752
0.00019
DoubleDunk
0,8600 0.0281
0,8793
0.0243
0.94444
BaniHeis
0.8234 + 0.0323
0.8459 + 0.0400
0.00002
MoPacnorn
0,2005 0.0362
0,2525 0.0648
0,5230
0,6077
0,6233
0,06228
0.0845
0.111
Boxing
0,9228 a: 0,0806
0,9502 à 0.0514
0.69408
Spacelavaders
0,0400 0.0224
0,0820 0U0239
00009000
experimental set up can be found in Appendix. Our experimental results show that GNN with DVNC
improved OOD generalization (Figure 4
Movement Prediction in Atari Games Similarly, we design OOD movement prediction tasks for 8 Atari games. Changes of each image frame depends previous image frame and actions applied upon different objects. A different starting frame number is used to make the testing set OOD, GNN with DVNC showed statistically significant improvement out of 8 games and marginally significant improvement in the other games (Table
4.2 Communication Between Positions in Transformers
In transformer models, attention mechanism used communicate information among different position. We design and conduct two visual reasoning tasks to understand if discretizing results of attention in transformer models will help improve the performance (Section (2}. In the tasks, transformers take sequence of pixel as input and detect relations among different objects which are communicated through attention mechanisms.
We experimented with the Sort-of-CLEVR visual relational reasoning task, where the model is tasked with answering questions about certain properties of various objects and their relations with other objects (Santoro et al., 2017). Each image in Sort-of-CLEVR contains randomly placed geometrical shapes of different colors and shapes. Each image comes with 10 relational questions and 10 non- relational questions. Nonrelational questions only consider properties of individual objects. On the other hand, relational questions consider relations among multiple objects. The input to the model consists of the image and the corresponding question. Each image and question come with a finite number of possible answers and hence this task classify and pick the correct the answerGGoyal et al. b02 Transformer models with DVNC show significant improvement (Table
4.3
Communication with Modular Recurrent Neural Networks
Recurrent Independent (RMM are RNN with modular structures. In RIMs, units communicate with each other using attention mechanisms at each time step. We discretize results of inter-unit attention in RIMs (Section 2 We conducted a numeric reasoning task and a visual reasoning task to understand if DVNC applied to RIMs improves OOD generalization.
We considered synthetic adding task in which the model is trained to compute the sum of sequence of numbers followed by certain number of dummy gap tokens(Goyal et al.) 2019). In OOD settings of the task, the number of gap tokens after the target sequence is different in test set from training data. Our results show that DVNC makes significnat improvement in the OOD tasgure D.
For further evidence that RIMs with DVNC can better achieve OOD generlization, we consider the task of classifying MNIST digits as sequences of pixels (Krueger et al. (016)) and assay generalization images of resolutions different from those seen during training. Our results suggest that RIMs with DVNC have moderately better OOD generalization than the baseline especially when the test images are very different from original training data (Figure
4.4 Analysis and Ablations
Discretizing Communication Results is Better than Discretizing other Parts of the Model:
The intuition is that discretizing the results of communication with shared codebook encourages more reliable and independent processing by different specialists. We experimentally tested this key hypothesis on the source of the DVNC's success by experimenting with discretizing other parts of the model. For example, in RIMs, we tried discretizing the updates to the recurrent state and
8

---

(a) 2D Shapes (G)
(b) 2D Shapes (L)
(c) 3D Shapes (G)
(d) 3D Shapes (L)
Iespyf
Teiple
Mumber of deciataric Medaad
Codabcak aoM31
Number Sacuciaiio NuadiS
Dodeicc MOWLL
(e) 2D Shapes(GNN)
(f) 3D Shanes(GNN)
(g) Adding (RMM
003-
Hodden sale
Dicoeeadd
Figure 5: Upper Row: Models with DVNC have improved OOD generalization in wide range of hyperparameter settings. Red dots are performances of models with DVNC with different codebook size (L. and number of heads (G). Black dashed lines are performance of baseline methods without discretization. Lower row: (e) and (f) compare OOD generalization in HIT's (higher is better) between GNNs with results of communication discretized vs. edge states discretized. (g) Compares RIMs model with results of communication discretized vs. other vectors discretized.
tried discretizing the inputs. On the adding task, this led to improved results over the baseline, but performed much worse than discretizing communication. For GNNs we tried discretizing the input to the communication (the edge hidden states) instead of the result of communication, and found that led to significantly worse results and had very high variance between different trials. These results are in Figure 5
VQ-VAE Discretization Outperforms Gumbel-Softmax: The main goal of our work was to demonstrate the benefits of communication with discrete values, and this discretization could po- tentially be done through a number of different mechanisms. Experimentally. we found that the nearest-neighbor and straight-through estimator based discretization tecmnire method used in VQ-VAE, outperformed the use of a Gumbel-Softmax to select the discrete tokens (Figure 6 in Appendix). These positive results led to focus on the VQ-VAE discretization technique, but in principle other mechanisms could also be recruited to accomplish our discrete-valued neural communication framework. We envision that DVCN should work complementarily with future advances in learning discrete representations.
5 Discussion
With the evolution of deep architectures from the monolithic MLP complex architectures with specialized components, we are faced with the issue of how to ensure that subsystems can communi- cate and coordinate. Communication via continuous, high-dimensional signals is natural choice given the history of deep learning, but our work argues that discretized communication results in more robust, generalizable learning. Discrete- Valued Neural Communication (DVNC) achieves much lower sensitstntt bound while allowing high expressiveness through the use multiple discretization heads. This technique is simple and casy-to-use in practice and improves out-of- distribution generalization. DVNC is applicable to all structured architectures that we have examined where compoeent communication important and the information to be communicated can be discretized by its nature.
Limitations The proposed method has two major limitations. First, DVNC can only improve performance if communication among specialists is important for the task. the different components do not have good specialization, then DVNC's motivation is less applicable. Another limitation is that the discretization process can reduce the expressivity of the function class, although this can be mitigated by using large value for G' and I, and can be partially monitored by the quantity of training data (e.g., training loss) similarly the principle of structural minimization. Hence future work could examine how to combine discrete communication and continuous communication.

---

Social Impact Research conducted in this study is purely technical. The authors expect no direct negative nor positive social impact.
Funding in direct support of this work Funding from Samsung Electronics Ltd
10

---

References
Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. (2016). Neural module networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 39 48.
Baars, B. J. (1993). cognitive theory of consciousness. Cambridge University Press.
Baars, B. J. (2019). On Consciousness: Science & Subjectivity. Nautilus Press.
Bahdanau, D., Cho, K., and Bengio, Y. (2014), Neural machine translation by jointly learning to align and translate. arXiv preprint wrXiv:149.00475.
Battaglia, P. W. Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M.,
Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al. (2018). Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806::/8000011111
Bottou, L. and Gallinari, P. (1991). A framework for the cooperation of learning algorithms. In Advances in neural information processing systems, pages 781-788.
Braitenberg, V. (1986). Vehicles: Experiments in synthetic psychology. MIT press.
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. (2017). Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18
Chmielewski, M. R. and Gizymala-Bussee J. W. (1996). Global discretization of continuous attributes as preprocessing for machine learning. International journa of approximate reasoning, 15(4):319- 331.
Fernando, C., Banarse, Blundell, C.,, Zwols, Y., Ha, D., Rusu, A. A., Pritzel, A., and Wierstra, D. (2017). Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: WrXiv:1701.0873:
Fodor, J. A. (1983). The modularity of mind. MIT press.
Fujii, N. and Graybiel, A. M. (2003). Representation of action sequence boundaries by macaque prefrontal cortical neurons. Science, 1446669)
Garland, A. and Alterman, R. (1996). Multiagent learning through collective memory. In Adaptation, Coevolution and Learning in Multiagent Systems: Papers from the 1996 AAAI Spring Symposium, pages 33-38.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017). Neural message passing for quantum chemistry. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1263-1272. JMLR. org.
Goyal, A. and Bengio, (2020). Inductive biases for deep learning of higher-level cognition. arXiv preprint arXiv:2011.1509)
Goyal, A., Didolkar, A., Ke, N. R., Blundell, C., Beaudoin, P., Heess, N., Mozer, M., and Bengio, Y. (2021a). Neural production systems. arXiv preprint arXiv:2103.01937.
Goyal, A.. Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., and Bengio, (2021b). Coordination among neural modules through a shared global workspace. arXiv preprint arXiv:2103.01197
Goyal, A., Lamb, A., Gampa, P., Beaudoin, Levine, S., Blundell, C., Bengio, Y.. and Mozer, M. (2020). Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv:2006.1 16225.
Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., and Schölkopf, B. (2019). Recurrent independent mechanisms. arXiv preprint arXiv: /909. 10093.
Jacobs, A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E, (1991). Adaptive mixtures of local experts. Neural computation, 3(1):79-87.

---

Kipf, T., Fetaya, E., Wang, K.-C., Welling, M., and Zemel, R. (2018). Neural relational inference for interacting systems. arXiv preprint 1rXiv:1802.04000 040870
Kipf, T., van der Pol, E., and Welling, M. (2019). Contrastive learning of structured world models.
arXiv preprint 1rXiv:1911.1222
Koller, D. and Friedman, N. (2009). Probabilistic Graphical Models: Principles and Techniques. MIT press.
Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N. R., Goyal, A., Bengio, Y., Courville, A., and Pal, C. (2016). Zoneout: Regularizing mns by randomly preserving hidden activations. arXiv preprint arXiv: 1606001335
Kucinski, L.. Korbak, T., Kotodziej. P., and Mitos, P. (2021). Catalytic role of noise and necessity of inductive biases in the emergence of compositional communication. Advances in Neural Information Processing Systems, 34.
Laird, J. E., Rosenbloom, P. S., and Newell, A. (1986). Chunking in soar: The anatomy of general learning mechanism. Machine learning, 1(1):11-46,
Lamb, A., He, D., Goyal, A., Ke, G., Liao, C.-F., Ravanelli, M., and Bengio, Y. (2021). Transformers with competitive ensembles of independent mechanisms. arXiv preprint arXiv:2103.00336
Minsky, M. (1988). Society of mind. Simon and Schuster.
Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K. (2017). Neural discrete representation learning. arXiv preprint /71 1.0009377.
Pesce, E. and Montana, G. (2020). Improving coordination in small-scale multi-agent deep reinforce- ment learning through memory- driven communication. Machine Learning, pages 1-21.
Quiroga, R., Reddy, L., Kreiman, G., Koch, C., and Fried, I. (2005). Invariant visual representation by single neurons in the human brain. Nature, 435(7045): 102-117777
Raposo, D., Santoro, A., Barrett, D., Pascanu, R., Lillicrap, T., and Battaglia, P. (2017). Discovering objects and their relations from entangled scene representations. arXiv preprint arXiv:1702.0520
Reed, and De Freitas, N. (2015). Neural programmer-interpretess. arXiv:1511.06279
arXiv preprint
Ronco, E., Gollee, H., and Gawthrop, J. (1997). Modular neural networks and self-decomposition. Technical Report CSC-96012.
Rosenbaum, C., Cases, I., Riemer, M., and Klinger, T. (2019). Routing networks and the challenges of modular and compositional computation. arXiv preprint 19044 12774.
Rosenbaum, C., Klinger, T., and Riemer, M. (2017). Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv: 1711001239
Santoro, A., Raposo, D., Barrett, D. G., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap,
T. (2017). A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4967-4976.
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. (2008). The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
Sokolic, J., Giryes, R., Sapiro, G., and Rodrigues, M. (2017a). Generalization error of invariant classifiers. In Artificial Intelligence and Statistics, pages 1094-1103.
Sokolic, J., Giryes, R., Sapiro, G., and Rodrigues, M. R. (2017b). Robust large margin deep neural networks. IEEE Transactions Signal Processing.
12

---

Sun, C., Yang, W., Martin, J., and Tonegawa, S. (2020). Hippocampal neurons represent events as transferable units of experience. Nature Neuroscience, 23(5):651-663.
Tacchetti, A., Song, H. F., Mediano, P. A., Zambaldi, V., Rabinowitz, N. C., Graepel, T., Botvinick, M., and Battaglia, P. W. (2018). Relational forward models for multi-agent learning, arXiv preprint arXiv: 1044.
Tomasello, M. (2009). The cultural origins of human cognition. Harvard university press.
Tsao, D. Y., Freiwald, W. A., Tootell, R. B. H., and Livingstone, M. S. (2006). A cortical region consisting entirely of face-sclective cells. Science, 311(5761):600-674,
van der Vaart, A. W. and Wellner, J. (1996). Weak Convergence and Empirical Processes. Springer New York.
Van Steenkiste, S., Chang, M. Greff, K., and Schmidhuber, J. (2018). Relational neural expec- tation maximization: Unsupervised discovery objects and their interactions. arXiv preprint arXiv:1802.10333.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, E., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, pages 5998-6008.
Veerapaneni, R., Co-Reyes, J. D., Chang, M., Janner, M., Finn, C., Wu, Tenenbaum, B., and Levine, S. (2019). Entity abstraction in visual model- reinforcement learning. CoRR, abs/1910.12227..
Watters, N., Zoran, D., Weber, T., Battaglia, P., Pascanu, R., and Tacchetti, (2017). Visual interaction networks: Learning physics simulator from video. In Advances in neural information processing systems, pages 4539-4547.
Wills, T., Lever, C., Cacucci, F., Burgess, N., and O' O'Keefe, J. (2005). Attractor dynamics in the hippocampal representation of the local environment. Science, 308(5723) 877-876.
Xu, H. and Mannor, S. (2012). Robustness and generalization. Machine learning, 86(3):391-423,
Zhang, K., Yang, Z., and Başar, T. (2019). Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint 191.
13


Reviews:
[{"review_id": "tsBia7fB-bX", "reviewer": "Reviewer_BL11", "summary": "The paper proposes a method of discretizing continuous nodes in a neural network, turning them into several discrete tokens. The method, DVNC, has theoretical and practical generalization improvements.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Originality_: The ideas proposed in this paper are novel and are centred on a generalization of the discretization node of VQ-VAE. The inspiration of VQ-VAE is clear in the paper, but oftentimes the differences are not made very clear in this work (some suggestions on _Clarity_ section).\n\n_Quality_: The method proposed is technically sound. The authors provide a thorough theoretical analysis of the improved noise robustness of DVNC when compared to a continuous representation. The experimental procedure supports these theoretical claims. Experiments are performed in three different architectures/tasks and the results improve significantly when compared to the baseline.\n\n_Clarity_: The paper is well-written. As briefly mentioned above, I think the authors should have used some space to better distinguish their method and VQ-VAE at the beginning of §2. A background section on that architecture would benefit the clarity of the paper.\n\n_Significance_: The analysis and method proposed in this paper are relevant to the community and this is a line of research that warrants more future work.\n\n__Comments__\n\n- I think the use of $m$ as the embedding size is not consistent. It is defined as the embedding size of the discrete latent space, but then the segments $s_i$ are of size $\\frac{m}{G}$ but it seems they should be of size $m$, given Eq. 1.\n\n- In Figure 3 (right) it seems like 3 lines are missing\n\n- Typos:\n\n    - Line 64, 74: analyses --> analysis\n    - Line 94: \"selected\" is repeated\n    - Line 98: decent --> descent\n    - Lines 194, 197, 201, 207, 210, 215, 274, 304: no space between period and next sentence, or comma has extra space/needs a space\n    - Line 198: that a learned --> that are learned\n    - Line 263: position --> positions", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "SSAd4XMO_S0", "reviewer": "Reviewer_X9GZ", "summary": "This paper proposes a method of discretizing communication between different modules in a network of specialists in order to improve generalization. They propose using a common codebook of discrete symbols shared by all network components that can be used to share information between modules. They propose a method of discretization based on the VQVAE architecture called Discrete-Valued Neural Communication (DVNC), in which a continuous vector is partitioned into \"discretization heads\", each of which is then mapped to its nearest neighbour among a set of learned latent vectors that are shared between modules. Gradients are propagated through the discretization using a straight-through estimator.\nThe authors provide theoretical analysis to argue that DVNC improves sample efficiency and robustness to noise, while decreasing intrinsic dimensionality. They also provide experiments with a number of different tasks and architectures to show that their method of discretization improves performance on out-of-distribution samples at inference time.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper’s analysis of the benefits of a discrete latent space for model generalization is novel and valuable, and highlights an interesting avenue for future research. The mathematics and theoretical results presented are thorough and rigorous, and provide a very interesting look at the benefits of discretization for generalization in general terms. The authors provide evidence for their claims through a variety of experimental settings, backed up by theoretical analysis and examples using Gaussian-distributed vectors. The work is clearly novel and provides a new and relevant framework for analysis that will hopefully be built upon by future research.\n\nMy main criticism of the paper would be that I think it needs some additional work in editing and improving the clarity of some sections, particularly the theoretical analysis in Section 2. From reading the paper it is not entirely clear to me how the results stated in Theorems 1 & 2  generalize to the claims about noise robustness and intrinsic dimensionality made in the text. First of all, in looking at the theorems, my impression is that they are discussing only functions restricted to act on a small specific subdomain of the input space, and I don't see from the content of the text or the theorems any justification for the more general claims you make. From your response I gather this does generalize, but I would make the generalization more clear. Second, I am unclear on what the paper means by 'noise robustness'. As far as I can tell, there is no explicit noise model mentioned in the paper or the appendices. The theorems rather appear to be analyzing the error of approximating the output of functions by Monte Carlo sampling. Finally, the paper’s discussion of ‘intrinsic dimensionality’ could benefit from some additional detail and discussion, as the definition or significance of this was never made clear in the paper (is this a reference to the Minkowski dimension, or maybe the packing dimension proposed by Kegl?). It would be helpful to have a more concrete discussion of what the intended meaning of these terms is in the context of the two theorems.\n\nI also think that the experiments may be overly limited. While the paper shows experiments on a number of tasks and architectures, many seem to be toy tasks rather than more realistic applications. In addition, aside from Figure 4, the reported results (e.g. Table 2, Table 3, Figure 5) show only the effect of the method on the out of distribution performance, and do not show the effect of the method on in-distribution performance, which I think would be beneficial to include for completeness’s sake.\n\nThere are also some clear editing mistakes in the paper that need to be fixed (e.g. ‘selected’ repeated on page 3, appendix B missing, etc…). Many of the diagrams are also far too small, making them very difficult to read – particularly figure 4. \n\nOverall I think that this paper is a worthwhile and relevant work, as long as the issues with clarity and editing are fixed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As this is a purely technical and theoretical work, the direct negative social impacts are not a major concern. The authors do address the limitations of their work, though I think it would be beneficial to more thoroughly address the concerns that discretization could harm expressivity or reduce in-distribution performance (perhaps with more robust experiments to show that this is empirically not the case).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "26zyWLf-6o", "reviewer": "Reviewer_ypgC", "summary": "The submission discusses discrete communication between neural network components and proposes 'discrete-valued neural communication (DVNC)', i.e. discretization of continuous representations in neural networks. The key contribution is a segmented discretization of vectors (see Eq. 1), in contrast to a joint discretization in a prior work (Oord et al., 2017).", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submission addresses an important research questions, namely how different neural submodules can communicate effectively. This is relevant since future neural networks may have several specialized components that need to interact in an efficient and effective way. However, the submission does not discuss clearly related prior work. Furthermore, is remains unclear what the paper actually claims to contribute. Hence, I believe that the submission cannot be accepted to this conference in its current form.\n\nMore details:\n\nMissing/vague contribution:\nOne of the main reasons for my reject recommendation is the vagueness of the submission. The title 'Discrete-Valued Neural Communication' and the abstract/introduction suggest that this paper proposes discrete-valued neural communication for the first time. However, this is clearly not the case. There are several works that have been working on this area (see details below). Hence, after reading the first parts of the submission, it is unclear what its actual contribution is. The actual contribution becomes more clear in section 2, and specifically in Equation 1, which is very similar to Equation 3 in the VQ-VAE paper (Oord et al., 2017). The key difference is the segmentation and the separate discretization of h. However, this does not become clear at all in the introduction of the submission. I would like to recommend to clearly state early on what the actual contribution of the paper is instead of vaguely talking about discrete communication.\n\nMissing related work:\nDiscrete-valued neural communication has already been addressed by several lines of work. Unfortunately, non of these areas is discussed. Examples are discrete communication between different agents in reinforcement learning (e.g. [1]), networks with discrete-continuous computation graphs (e.g. [2]), and memory networks with discrete states (e.g. [3]). Hence, I would like to recommend to discuss these works and how the current submission differs.\n\n[1] Jakob N. Foerster et al. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. In: CoRR abs/1605.06676 (2016). arXiv: 1605.06676. url: http://arxiv.org/abs/1605.06676.\n[2] Schulman, J., Heess, N., Weber, T., & Abbeel, P. (2015). Gradient Estimation Using Stochastic Computation Graphs. Advances in Neural Information Processing Systems, 3528–3536.\n[3] Wang, C., & Niepert, M. (2019). State-Regularized Recurrent Neural Networks. Proceedings of the 36th International Conference on Machine Learning, 6596–6606. http://proceedings.mlr.press/v97/wang19j.html\n\nMinor important comments/suggestions:\n- The submission states that 'graph neural networks are composed of distinct nodes'. I think this description is rather confusing. A GNN is not composed of distinct nodes. It operates over nodes as underlying structure. Similarly, an RNN for NLP is not composed of tokens, but operates on a sequence of tokens. Similarly, transformers are not composed of positional elements.\n\n- The submission argues that discrete encoding have an evolutionary advantage since different areas of the human brain are tuned to discrete variables. However, making strong connections between human brains and neural networks is far fetched. It remains an hypothesis if computers may benefit from similarities with the human brain.\n\n- There are several typos and missing spaces in the 'Communication among Specialists' section.\n\n- The text in Figure 4 should be larger. It is impossible to read it without zooming in.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses neural network architectures that exchange discrete messages, trained using a strategy similar to the successful VQ-VAE. The paper includes both theoretical results about such architectures, as well as encouraging empirical results.

The largest concerns have to do with the discussion of the scope, as the setup discussed here has large overlap with existing but different literature. (Reviewers point out examples from reinforcement learning or on RNN hidden state discretization.) Overall, some comparison to those strategies seems possible and would improve the work, but, at the same time, the work here seems like a good step. At the very least, the connections to this others applications of discrete variables should be discussed as clearly as possible and in a positive way. (Connections here are likely to be productive.)

The paper could be strengthened with results on more realistic applications, possibly where the discrete messages can be nicely interpretable. 

The authors have been very active in the discussion and have provided many clarification as well as promises for improvements of the presentation, including agreeing to a change of title that seems like it would improve clarity.
Decision: Accept (Poster)