Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs
Authors: Hung Le
Svetha Venkatesh
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Memory-augmented neuronal network

Abstract: Artificial Neural Networks are functionally equivalent to special-purpose computers. Their inter-neuronal connection weights represent the learnt Neural Program that instructs the networks on how to compute the data. However, without storing Neural Programs, they are restricted to only one, overwriting learnt programs when trained on new data. Here we design Neurocoder, a new class of general-purpose neural networks in which the neural network “codes” itself in a data-responsive way by composing relevant programs from a set of shareable, modular programs stored in external memory. For the first time, a Neural Program is efficiently treated as a datum in memory. Integrating Neurocoder into current neural architectures, we demonstrate new capacity to learn modular programs, reuse simple programs to build complex ones, handle pattern shifts and remember old programs as new ones are learnt, and show substantial performance improvement in solving object recognition, playing video games and continual learning tasks.

Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs
Anonymous Author(s)
Affiliation
Address
email
Abstract
Artificial Neural Networks are functionally equivalent to special- comput- ers. Their inter-neuronal connection weights represent the learnt Neural Program that instructs the networks on how to compute the data. However, without stor- ing Neural Programs, they are restricted only one, overwriting learnt programs when trained on new data. Here we design Neurocoder, new class of general- purpose neural networks in which the neural network "codes" itself in data- responsive way by composing relevant programs from a set of shareable, modular programs stored in external memory. For the first time, Neural Program is ef- ficiently treated datum in memory. Integrating Neurocoder into current neu- architectures, we demonstrate new capacity to learn modular programs, reuse simple programs to build complex ones, handle pattern shifts and remember old programs as new ones are learnt, and show substantial performance improvement in solving object recognition, playing video games and continual learning tasks.
10
11
12
Introduction
14
From its inception in 1943 until recently, the fundamental architectures of Artificial Neural Net- works remained largely unchanged a program is executed by passing data through a network of artificial neurons whose inter-neuronal connection weights are learnt through training with data.
16
18 These neteranluronal connection weights, or Neural Programs, correspond to program in modern computers [32]. Memory Augmented Neural Networks (MANN) are innovative solution allow- ing networks to access external memory for manipulating data [I] [2]. But they were still unable store Neural Programs in such external memory, and this severely limits machine learning. Stor- ing inter-neuronal connection weights only in their network does not permit modular separation of computer with fixed program. Recent works introduce
20
21
22
Neural programs and is analogous conditional computation via adjusting or activating parts of network in an input-dependent manner but networks remain monolithic. Current networks forget when retrained, old neter-ncuronal connection weights are merged with new ones or erased.
24
25
39
33
13 28
26
The brain is modular, not monolithic system 8 6). Neuroscience research indicates that the brain is divided into functional modules [19 7
28
7 9. If the neural program for each module is kept in separate networks, networks proliferate. Modular neural networks, another form of conditional computation, combine the output of multiple expert networks, but as the experts grow, the networks grow drasti- cally 20 14 35 29]. This requires huge computational storage and introduces redundancy these experts do not share common basic programs.
30
31
32
A pathway out of this bind is to keep such basic programs in memory and combine them required. This brings neural networks towards modern general-purpose computers that use the stored- program principle 37 40 to efficiently access reusable programs in external memory. Here show how conditional compu-
34
36
Neurocoder, a new neural framework, introduces a new class of
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

(a) MAIN NETWORK
(b) PROGRAM CONTROLLER
(c) PROGRAM MEMORY
Output
Layer
Werking Progr am
Active Programa
My
My
M2
(1)
Program Integration
Low wwwk approsimation
Querics
Gidea
Pragrama
(4)
Staras
(7)
Program am
Afiention
Welpht
pilate
(3)
(6)
(2)
Usage
Kega
Integration Networks
Compostin Network
Priggram
Studus
Input
Figure I: Neurocoder (a) The Main Network uses a working progrum to compute the output for input. Here only the final layer of the Main Network is adaptively loaded with the working program (2). Other layers use traditional Neural Programs as connection weights The Program Controller's composition network controls access to the Program Memory, emitting queries and interpolating gate control signals in response to the input (2). then performs recurrent multi-head program attention to the Program Status (3), triggering attention weights to the Singular Programs (4). The attended Singular Programs form an active program using low-rank approxi- mation (5). Residual program produced by the Program Controller's s integration network (6) plus the active program derives the working program. (c) The Program Memory stores the represen- tations (singular programs) required to reconstruct the active program be used by the Program Controller. Access controlled through the Program Status including keys (k),, and slot usage (m) that are updated during the training and computation (7).
seeed--fter-rranning (b)
37 tation machines in which neural network can be "coded" in an input-dependent manner. Efficient
38 decomposition of Neural Programs creates shareable modular components that can reconstruct the
39 whole program space. These components change their "shapes" based on training and are stored
40 in external Program Memory. Then, in data-responsive way, a Program Controller retrieves relevant components to build the Neural Program. This analogous to shape-shifting Lego bricks that can be reused to build unlimited shapes and structures (See Appendix Fig. 4
41
Using adaptive modular components vastly increases the learning capacity of the neural network 44 by allowing re-utilisation of parameters, effectively curbing network growth as programs increase.
45 More importantly, unlike pre-defined sub-networks or modules (20] I that combine activation
48 level, the construction of our modular components is dynamic and performed on the weight space.
47 The Neural Program construction is learnt through training via traditional backpropagation 30 as
48 the architecture is end-to-end differentiable.
2
Methods
50
2.1
System overview
51 A Neurocoder is a neural network (Main Network) coupled to external Program Memory through
52 a Program Controller. The working program of the Main Network processes the input data to pro- duce the output. This working program is "coded" by the Program Controller by creating an input-
54 dependent active program from the Program Memory (Fig. [. The following gives high-level
55 description of the Neurocoder framework and then the details.

---

58
Neurocoder stores Singular Value Decomposition of Neural Programs in Program Memory
57
The Neural Program needs to be stored efficiently in Program Memory. This is challenging as there may be millions of inter-neuronal connection weights, thus storing them directly ((22) is grossly inefficient. Instead, the Neurocoder forms the basis of subspace spanned by Neural Programs and stores the singular values and vectors of this subspace in memory slots of the Program Memory (hereafter referred to singular programs), Based on the input, relevant singular programs are retrieved, a new program is reconstructed and then loaded the Main Network to process the input. This representational choice significantly reduces the number of stored elements and allows each singular program to effectively represent a unitary function of the active program.
58
59
60
61
62
64
65
The active program matrix P can be composed by standard low-rank approximation as
I'm P= U-UVV Σ Un
66
where U and V are matrices of the left and right singular vectors, and S the matrix of singular values. I' is the total number of components we want to retrieve. {an I= is the attended singular values, {u, V' and {vp y the attended singular vectors of S, U, and V. respectively. The Program Memory is crafted as three singular program memories (My,My,Me My,Msh of their memory slot stores singular component singular program. The process "codes" the active program using singular programs from the program memories. The coding is conditioned on input 2t, yet drop index i for notation simplification and leave the details on the computation of any Uny Un in Sec.
68
69
71
73
74
The Program Memory also maintains the status for each singular program in terms of access and usage. To access a singular program, program keys (k) are used. These keys are low-dimensional vectors that represent the singular program function and computed by a neural network that ef- fectively compresses the singular program. The program usage (m) measures memory utilisation, recording how much memory slot is used in constructing program. The components of the Program Memory are summarised in Fig. (c).
75
76
77
78
79
80
Recurrent multi-head program attention mechanisms for program storage and retrieval Neural networks use the concept of differentiable attention to access memory [ 2] This de- fines a weighting distribution over the memory slots essentially weighting the degree to which each memory slot participates in a read or write operation. This is unlike conventional computers that use unique address to access single memory slot.
81
83
Here we use two kinds of attention. First is content-based attention [2 to ensure that the singu- 85
lar program is selected based on its functionality and the data input. This is achieved by producing 86
87 a query vector based on the input and comparing to the program keys (k) using cosine similarity. Higher cosine similarity scores indicate higher attention weights to the singular programs associated
88
89 with those program keys. Second, to encourage better memory utilisation, higher attention weights are assigned to slots with lower program usage (m) through usage-based attention 12 31. The
91 attention weights from the two schemas are then combined using interpolating gates to compose the final attention weights to the Program Memory.
93 We adapt multi-heal attention [ 38 that applies multiple attentions in parallel retrieve H singu- lar components. Besides, we introduce a recurrent attention mechanism, in which multi-head access 95 performed recurrently in steps. The j-th set of H retrieved components is conditioned on the 96 previous ones. This recurrent, multi-head attention allows the composition network to incrementally search for optimal components for building relevant active programs.
94
97
98
Neurocoder learns to "code" a relevant working program via training
99
The structure of the Program Memory and the role of the Program Controller facilitates the au- tomatic construction of working programs via training. The Program Controller controls memory access through its composition network that creates the attention weight defining how to weight the singular programs in the memories. A weighted summation of the singular programs results in the attended singular program. Applying the recurrent multi-head attention described earlier, multiple
100
101
102
103
104
attended singular programs are retrieved to construct an active program (Eq. D. Then the Program
105
Controller generates a residual program using its integration network, adding to the active program
3

---

106
to produce the working program of the Main Network. This addition enables creation of higher-rank working programs, which compensates for the low-rank coding process. The structure of the Program Controller is illustrated in Fig. (b).
flexible
107
108
The singular programs are trained represent unitary functions necessary for any computation whilst the composition and integration networks are trained to compose the relevant programs for the considering task. As such, beside minimising the task loss, we enforce orthogonality of stored singular vectors by minimising L.- = My - M -L.
110
111
112
I. The parameters of the networks, and the stored singular programs are adjusted using gradient training via minimising the total loss
113
L Ltusk + aao
(2)
where Crask represents the supervised task loss and Lo represents the orthogonal loss weighted by a haper-parameter a to enforce orthogonality of the singular vectors.
115
116
2.2
Attention mechanisms for Program Memory
117
Here we describe program attention mechanisms used in this paper. Given why wh w (jointly denoted win wx --the attention weight the i-th slot of the singular program memories My, My and Mg, we retrieve the n-th singular vector as follows,
118
119
Un
w i My (i)
(3)
i My (i)
(4)
1=1
For the singular values, need to enforce 01 > >0 ... Trm > 0, thus retrieve using
softplus (IE w Ms (i))
I'm
On 1nt1 softplus EEen wo Ms (i)
(5)
T'
Here, Pu, P. and Pa are the number of memory slots of My, My and Ms, respectively. In this paper, we set P Pa = P. = Ps as the number of memory slots of the Program Memory. We note specified for some data input x; and the index later maps to attention a.v.ar To simplify notations, we head h, and an attention step j. hence the full notation should be wiihh
122
123
that these notations
124
125
will drop u, U, a from now and describe the computation of a representative Wtigh for any of the three program memories in the following parts.
126
127 Recurrent Access to the Program Memory via the composition network
128
To perform program attention, the Program Controller employs composition network (denoted as fa), which takes the current input x, and produce program composition control signals (E?). fo performs all attentions concurrently via multi-head attention as [I] 38), may lead to program collapse [22). To have a better control of the component formation and alleviate program collapse, we propose to recurrently attend to the program memory. To this end, we implement so as recurrent neural network (LSTM [16) and let it access the program memory J times, resulting in At access step j. the recurrent network updates its hidden states and generates 4? using recurrent dynamics as
129
130
132
E = |
134
135
(6) where bo is initialized as zeros and <i is the program composition control signal at step that depends on both on the input data I, and the the previous state h Partieularly, the control signal contains the queries and the interpolation gates for each head to compute the program attention weight: St {ygh. Itijh )
Ei = fo (Kz,h;zi)
136
137
138
139
h- Here, at each attention step, we perform multi-head attention with H as the number of attention heads and thus, each E. consists of H pairs of queries and gates. Hence, the total number of retrieved components I'm = J x H and the index n j x H + h.

---

142
Attending to Programs by "Name"
143
Inspired by the content-based attention mechanism for data memory [1, we use the query to look for the singular programs. computer programming. to find the appropriate program for some computation, we often refer to the program description at least the name of the program. Here, we create the "name" for our neural programs by compressing the program content to low-dimensional key vector. As such, we employ a neural network (Sp) to compute the program memory keys as
144
146
k; = fp (M (i))
(7)
where F; R.K and is the row index of the program memory. Here, fg learns to compress each memory slot into a K-ddimnnsional vector. As the singular programs evolve, their keys get updated. In this paper, we update the program keys after each learning iteration during training.
149
151 Finally the content-based program memory attention Crijh is computed using cosine distance be- tween the program keys k; and the queries (tjh as
( Otjk h; [1atshll -|
Crihh softmax
(8)
Making Every Program Count
Similarly to [12 31,, in addition the content-based attention, employ least-used reading strategy to encourage the Program Controller to assign different singular programs to different com- ponents. In particular, calculate the memory usage for each program slot across attentions as
155
157
Majh = max Wiijh تSj
158
Since we want to consider only 4 amongst P memory slots that have smallest usages, let m denote the value of the 1y-th smallest usage, then the least-used attention computed as
159
Ligh max (meigh) Meigh Miigh S figgh
(10)
miijh > miih
180
The final program memory attention is computed as
Weijh = sigmoid (grigh) ) Crijh + (1 - sigmoid (gtigh)) Lijh
(11)
161
Since the usage record are computed along the memory accesses, the multi-step Neurocoder utilises this attention mechanism better than the single-step Neurocoder, creating different attention styles
163
(see Sec. 3.2
The composition the active program P, is illustrated Appendix's Fig. 5
184
2.3
Program Integration via the integration network
Since the working program P, only contains top I' principal components. it is low-rank and may be not flexible enough for sophisticated computation. We propose to enhance P, with a residual program R- traditional connection weight trained the integration network's parameters, which is constant after training w.r.t t. The residual program represents the sum of the remaining less important components. To this end, suppress R with multiplier that is smaller than otr... the smallest singular value of the main components resulting in the integration formula
186
168
170
W. = + w/
(12)
171
where =. sigmoid (fo (xx)) an adaptive gating value that controls the contribution of the residual program. fø is the integration network in the Program Controller and hence, in our imple- mentation, the integration control signal sent by the Program Controller X {w[ otr... } We note that in our experiments, the program integration can be disabled (W; is directly set to P,) to prove the contribution of P, or reduce the number of parameters. The working program W, is then used by the Main Network to exccute the input data Ie (see (Fig. (a))). For example, with linear
172
175
175
5

---

Liscar clasatteer
Nearotede: (I+1]
280
Nomber efrecercent atteatinn
a
S
(d)
12A K.L
3.58
digh
84.79
0.9
(c)
(b)
Figure 2: (a) MNIST test set classification error vs the number of steps (J) in Neurocoder (bluc), compared with linear classifier (red). (b) 1st column: Digit images; Middle column: Single-step attention weights for 30 slots in My (vertical axis) for first singular vectors (horizontal axis) for each digit; Last column: Multi-step attention weights for 10 slots in My (vertical axis) for first 3 singular vectors (horizontal axis). Multi-step attention is able to produce far more diverse patterns with fewer slots 10 slots compared to single-step 30 slots.
(c) Two attention patterns of single- step Neurocoder. The binary decision tree derived from single-step Neurocoder's 's attention patterns. The two patterns across components represent the decisions going up and down across the binary tree. Visualisation for (d) multi-step (J = 5, 20 memory slots) and (e) single-step (J = 1, 10 memory slots) cases showing while processing a sequence of the polynomial auto-regression task. The Neurocoder's attentions to My that form the first component of the active program are shown over sequence timesteps (upper) with Neurocoder's Ye prediction (orange) and ground truth (blue) (lower). The vertical dash green lines separate polynomial chunks. Each chuck represents local pattern, and thus ideally requires specific active program to compute the input æt. Although both predict well, only the multi-step Neurocoder discovers the chunk boundaries, assigning program attention the first component accordance with sequence changes.
177 classifier Main Network, the execution is y = x, We. Appendix's Table Table summarises the notations used for important parameters of Neurocoder.
3
Results
180
To demonstrate the flexibility of Neurocoder framework, we consider different learning paradigms: instance-based, sequential, multi-task and continual learning. We do not focus on breaking perfor- mance records by augmenting state-of-the-ar models with Neurocoder. Rather our inquiry is on re-coding feed-forward layers with the Neurocoder's programs and testing varied data types to demonstrate its intrinsic properties. For some experiments, we include ablation studies.
182
184
185
We compare the performance of diverse Main Networks (MN) with and without Neurocoder. We also augment the Main Networks with other recent conditional computing methods, either modular Stored-program form stronger baselines across our experiments. our experiments, we always apply Neurocoder
196
187
(sparse Mixture of Experts. Neural Stored- program Memory) monolithic (HyperNets, FiLM) to
188
to all layers of multi-layer perceptrons (MLP) just the final feed-forward layer of deep CNN networks (LeNet, DenseNet, ResNet), RNNs (GRU, LSTM), MANN (NTM). Other competitors
190
such MOE, NSM, HyperNet and FiLM are applied the Main Networks in the same manner.

---

Hyperrett
Neurocoder
143
Seaquest
103
MsPacman
103
Frostbite
155
Montozuma's Revenge
1.5
Average score
1.0
0.5
0
0.0
0.5
1.0
0.0
0.5
1.0
Env. Step (X16)
Eav. Step (x16)
kav. Step (x32)
147
Buv. Step (x32)
10
Figure 3: Learning curves (mean and std. over 5 runs) on representative Atari 2600 games. All baselines are applied to the actor/critic networks in the A3C agent.
192
3.1
Instance-based learning Object Recognition
We tested Neurocoder instance-based learning through classical image classification tasks using MNIST [24] and CIFAR [21] datasets. The first experiment interpreted Neurocoder's behaviour in classifying digits into 10 classes (0 - 9) using linear classifier Main Network. With equivalent model size, Neurocoder using the novel recurrent attention surpasses the performance of the linear classifier 24 by up to 5% (Fig. 2 (a)).
194
196
198
To differentiate the input, Neurocoder attends to different components of the active program to guide the decision-making process. Fig. (b) shows single-step and multi-step attention to the first singular vectors for each digit across memory slots. Multi-step attention produces richer patterns compared to single-step Neurocoder that manages only 2 attention weight patterns.
199
200
Fig. 2 (c) illustrates how Neurocoder performs modular learning by showing the attention assign- 202
203
ment for top 3 singular vectors as a binary decision tree. Digits under the same parental node share similar attention paths, and thereby similar active programs. Some digits look unique (e.g. 7) result-
204
205
ing in active programs composed of unique attention paths, discriminating themselves early in the decision tree. Some digits (e.g. and 9) share the same attention pattern for the first 3 components and are thus unclassifiable. They can only be distinguished by considering more singular vectors.
206
207
208
We integrated Neurocoder with deep networks 5-layer LeNet and 100-layer DenseNer- and tested on CIFAR datasets. Neurocoder significantly outperformed the original Main Networks with perfor- mance gain 5%. Compared with recent conditional computing models such as sparse Mixture of Experts (MOE (35) and Neural Stored-proo program Memory (NSM (22)), Neurocoder required tenth
209
210
211
of the number of parameters and performed better by up to 8 10% (see Appendix's 's Table 3).
212
213
3.2 Sequential learning Adaption to sequence changes and game playing using reinforcement learning
215 Recurrent neural networks (RNN) can learn from sequential data by updating the hidden states of the networks. However, this does not suffice when local patterns shift, as is often the case. We now demonstrate that Neurocoder helps RNNs overcome this limitation by composing diverse programs to handle sequence changes.
217
Synthetic polynomial ruto-regression We created simple auto-regression task in which data points are sampled from polynomial function chunks that change over time. The Main Network is strong RNN-Gated Recurrent Unit (GRU (5). We found that GRU integrated with single-step or multi-step Neurocoder converged much faster than all other baselines. The other conditional computing counterparts (HyperNet [13], FiLM (28) adapt by re-scaling weights or activation of the GRU, which were shown inferior to our modular approach (Appendix's 's Fig. 6).
220
222
224
225
Visualising the first singular vector attention weights in Mr, we find that the multi-step attention Neurocoder changes its attention following polynomial changes attends to the same singular pro-
226
gram when processing data from the same polynomial and alters attention for data from a different polynomial (Fig. d)). In contrast, the single-step Neurocoder only changes its attention when there is remarkable change in y-coordinate values (Fig. [e)) Although single-step Neurocoder
228
7

---

Method
MN (MLP (17))
MN (MLP ours)
NSM
Neurocoder
Adam
55.16=1.38
53.55
54.85
58.4610.46
Adagrad
18.08=11.06
57.83:1.2.74
58.42.1..87
62.28:14.03
L2
66.00+13.73
64.37-4-2.40
62.83+7.221
69.89
SI
64.76++3.09
64.41++3.36
64.36++2.99
67.96.
EWC
58.85+22.59
58.41+-2.37
58.1243..24
65.66+1.225
O-EWC
57.33+
1..7
58.55+3.40
73.97+1.50
Table I: Incremental domain continual learning with Split MNIST. Final test accuracy (mean and std.) over 10 runs.
converges well, it did not discover the underlying structure of the data, and thus underperformed the multi-step Neurocoder. We hypothesise that when recurrence is employed, usage-based atten- tion takes effect, stipulating better memory utilisation and diverse attentions over timesteps. We ran multi-step Neurocoder without usage-based attention. The results were worse than the full multi- step Neurocoder, which confirms our hypothesis (Appendix's Fig. 6).
231
233
234
235
Atari game reinforcement learning We used reinforcement learning further testbed to show the ability to adapt to environmental changes. We performed experiments on several Atari 2600 games wherein the agent was implemented as the Asynchronous Advantage Actor Critic (A3C
236
237
26)) In the Atari platform, agents are allowed to observe the screen snapshot of the games and act to earn the highest score. We augmented the A3C by employing Neurocoder's working programs for feed-forward layers of the actor and critic networks, aiming to decompose the policy and value function into singular programs that were selected depending on the game state.
239
241
242
Frostbite and Montezuma's 's Revenge. These games are known to be challenging for A3C and other algorithms [26]. We trained A3C and HyperNet-based A3C for over 300 million steps, yet these models did not show any sign of learning. performing equivalently to random agents. For such com- plicated environments with sparse rewards, both the monolithic neural networks and the HyperNet's unstored weights fail to learn (almost zero scores). In contrast, Neurocoder enabled A3C to achieve from 1, 500 to 3, 000 scores on these environments (Fig. 33, confirming the importance of decomposing complex solution to smaller, simple stored programs.
243
244
245
246
247
249
3.3
Multi-task learning Solving mutliple algorithms simultenously
Here explore the modular learning capability of Neurocoder in multi-task setting. Inspired by al- gorithmic sequencing tasks [22). we created a challenging sequential multi-task benchmark wherein the input sequence a series of sub-sequences from algorithms: Copy, Repeat Copy, Associative Recall and Priority Sort [T]] Each sub-sequence, following a task identification vector, represents the input for each task. In each input sequence, n tasks were sampled from the set of 4 algorithms randomly with replacement and the output sequences were created correspondingly.
251
252
253
254
255
256
We trained a MANN-Neural Taring Machine (NTM [TI) Main Network with FiLM, HyperNet and our Neurocoder augmentation on sequences of n = 4 tasks, and tested with sequences of n = 4 and tasks. Appendix's Fig. demonstrates that Neurocoder was performant in both test settings,
257
259
not only achieving lowest error on n = 4, but also being the only one generalised well to n scenario, which was unseen during training.
260
281
3.4
Continual learning Learning tasks sequentially without catastrophic forgetting
282
In continual learning, standard neural networks often suffer from "catastrophic forgetting" in which they cannot retain knowledge acquired from old tasks upon learning new ones [TO]. Our Neurocoder offers natural mitigation of such catastrophic forgetting in neural networks by attending to different singular programs whilst learning different tasks.
264
286 In this case, in addition to the Main Network, we examine several continual learning algorithms with and without Neurocoder. These algorithms, including Elastic Weight Consolidation (EWC 268 41) and Synaptic Intelligence (SI 41),, work by regularising the loss function and thus can be easily combined with Neurocoder by modifying the loss Lrask- We demonstrate that Neurocoder
8

---

270
can improve these continual learning algorithms without requiring additional assumptions as in other
271
approaches [2 36 34 that either utilise task embedding or replay memory.
272
Split MNIST We first considered the split MNIST dataset-a a standard continual learning bench-
273
mark wherein the original MNIST was split into 5 2-way classification tasks, consecutively pre- sented to Mulli-layer Perceptron Main Network (MLP). We followed the benchmarking as in [17] in which various optimisers and state-of-the-an art continual learning methods were examined under in- cremental task and domain scenarios. We measured the performance of the MI.P versus Neurocoder and NSM under each continual learning method. In both scenarios, Neurocoder was compatible with all continual leaning methods, demonstrating superior performance over MLP and NSM with performance gain between 1 to 16% (see Appendix's 's Table 5 Fand] I.
274
275
276
278
Split CIFAR We verified the scalability of Neurocoder to more challenging datasets. We split CIFAR datasets as in the split MNIST, resulting in 5-task 2-way split CIFAR 10 and a 20-task 5-way split CIFARIO0. We used Main Network ResNer [15 very deep CNN architecture.
281
283
When we stressed the orthogonal loss (a- = 10) and used bigger program memory (100 slots), Neu- rocoder improved ResNet classification by 15% and 10% on CIFARIO and CIFAR100, respectively. When we integrated Neurocoder with Synaptic Intelligence (SI 4., the performance was further improved, maintaining stable performance above 80% accuracy for CIFAR 10 and outperforming using SI alone by 10% for CIFAR 100 (see Appendix's 's Fig. R)
284
205
287
4
Discussion
289
Our experiments demonstrate that Neurocoder is capable of re-coding Neural Programs in distinctive neural networks, amplifying their capabilities in diverse learning scenarios: instance-based based, sequen- tial, multi-task and continual learning. This consistently results in significant performance increase, and further creates novel robustness to pattern shift and catastrophic forgetting. This ability for each architecture to re-code itself is made possible without changing the way it is trained, or majorly increasing the number of parameters it needs to learn (sce Appendix Table 77.
290
292
294
295
The MNIST problem illustrates the reasoning process of Neurocoder when classifying digit images wherein its singular program assignment resembles binary tree decision making process it shows how some singular programs are shared, others are not. The polynomial auto uugo-reegrssion problem highlights the importance of efficient memory utilisation in re-constructing the working program enabling discovery of hidden structures in sequential data. Training our framework with reinforce-
296
297
298
299
300
ment learning, we enable neural agents to solve complex games wherein traditional methods fail or learn slowly. Neurocoder also works well with multi-task setting, as shown in the challenging multi-algorithm benchmark. Finally, continual learning problems show that Neurocoder mitigates catastrophic forgetting efficiently under different learning settings/algorithms.
302
304
Our solution offers single framework that is scalable and adaptable various problems and learn- ing paradigms. Unlike previous attempts to employ a bank of separate big programs (20] 35 22), Neurocoder maintains only shareable, smaller components that can reconstruct the whole program space, thereby heavily utilising the parameters and preventing the model from proliferating. We note that Neurocoder is orthogonal to approaches employing tensor decomposition to reduce the number of parameters or hasten the computation (27 23). Neurocoder composes rather than decom- pose the neural weights. Our aim not only to enable efficient parameter usage, but also achieve general-purpose computing power, outperforming other methods in numerous learning problems.
305
306
307
308
310
312
One limitation of this work is the number of additional hyperparameters, which prevents us from fully tuning Neurocoder. Our research aims to add new capabilities to current neural networks to improve their performance and make them robust in different learning scenario. Hence, we do not see any intermediate negative societal impact. In future work, we will extend Neurocoder's application beyond feed-forward layers. It would be interesting to efficiently replace all neural layers including CNN or Transformer by Neurocoder's 's programs. We can also further extend Neurocoder's s
313
314
315
316
317
318
ability by allowing a growing Program Memory, in which the model decides to add or erase memory
slots as the number of data patterns grows or shrinks beyond the current program space's capacity.
320
Such a system represents a more flexible general-purpor computer that can dynamically allocate
computing resources by itself without human pre-specification.

---

References
322
323
[1] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39-48, 2016.
324
325
326
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Represen- tations, 2015.
327
328
[3] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
330
332
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradi- ents through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 2013.
334
335
[5] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734. Association for Computational Linguis- tics, October 2014.
336
337
338
339
340
[6] JC Eccles. The modular operation of the cerebral neocortex considered as the material basis of mental events. Neuroscience, 6(10):1839-1855) 1981.
341
342
[7] Gerald M Edelman. Neural darwinism: selection and reentrant signaling in higher brain func- tion. Neuron, 15-2): 1993.
343
[8] Gerald M Edelman and Vernon B Mountcastle. The mindful brain: cortical organization and the group-selective theory higher brain function. Massachusetts Inst of Technology Pr, 1978.
345
[9]
Richard SJ Frackowiak. Human brain function. Elsevier, 2004.
347
[10] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
348
349
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 14Xiv:1410.54000 2014.
350
351
[12]
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626):471447777 2016.
352
354
[13] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference Learning Representations, 2017.
356
[14] Bart LM Happel and Jacob MJ Murre. Design and evolution of modular neural network archi- tectures. Neural networks, 76677)9955-1044 1994.
358
359
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recogni- tion. pages 770-778, 2016.
360
381
382
[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):1735- 1997.
363
[17] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. In NeurIPS Continual learning Workshop, 2018
365
10

---

387
[18] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. Proceedings of the IEEE conference on computer vision and paltern recognition, pages 4700-4708, 2017.
368
370
[19] D.H. Hubel. Eye, Brain, and Vision. Scientific American Library series. Scientific American Library, 1988.
372
[20] Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey E Hinton, Adaptive mix- tures of local experts. Neural computation, 3(1):79-87, 1991.
374
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. TR-2009, 2009.
376
[22] Hung Le. Truyen Tran, and Svetha Venkatesh. Neural stored- stored-program memory. In Interna- tional Conference on Learning Representations, 2020.
378
[23] V Lebedev, Y Ganin, M Rakhuba, Oseledets, and V Lempitsky. Speeding-up convolutional neural networks using fine-tuned ep-decompasition. In International Conference on Learning Representations, 2015.
379
380
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(2122:7782222 1998.
382
383
[25]
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learn- ing. In Advances in neural information processing systems, pages 6467-6476, 2017.
384
385
[26]
Volodymyr Mnih, Adria Puigdomenech Badia, Mchdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. International conference on machine learning. pages 9928-1937, 2016.
386
387
[27] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances neural information processing systems, pages 442-450, 2015.
389
390
[28]
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI Conference on Artificial Intel- ligence, New Orleans, United States, February 2018
391
393
[29] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selec- tion of non-linear functions for multi-task learning. In International Conference on Learning Representations, 2018
395
[30] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagatpign errors. Nature, (23()888) 1986.
397
[31] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-leaming with memeeeeeugmmented neural networks. In International conference on ma- chine learning, pages 1842-1850, 2016.
399
400
401
[32] Jürgen Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environm TR FKI-126-90 1990.
402
403
nts.
404
[33] Jürgen Schmidhuber. Learning to control west- memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
405
406
[34] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548 4557-455,,25 2018
407
408
[35] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of- experts layer. In International Conference on Learning Representations, 2017.
410
11

---

412
[36] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pages 2990-2999, 2017.
413
[37] A.M Turing. On computable numbers, with application to the entscheidungssroobem.. In Proceedings of the London Mathematical Society, 1936.
416
417
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural informa- tion processing systems, pages 5998-6008, 2017.
419
420
[39] Christoph von der Malsburg. The correlation theory of brain function, 1981.
[40] John Von Neumann. First draft of report on the edvac. IEEE Annals of the History of Computing, 15(4):27-75, 1993.
422
423
[41] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intel- ligence. Proceedings of machine learning research, 70:3987, 2017.
425
12

---

428 Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Discussion and Appendix's
"Training procedure and hyper-parameter selections."
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] Data is public, provided with link. Code will be avaialble after published. All training details available and can be used to implement and reproduce the results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix's "Training procedure and hyper- parameter selections." (c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Experimental Results.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix's 's "Training procedure and syper-parameter selections.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] All assets are public. We will mention the license detail after the paper is published.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]


Reviews:
[{"review_id": "wgmZhxwWru", "reviewer": "Reviewer_DMv6", "summary": "This paper approaches the problem of multi-purpose neural networks that can learn several tasks or several methods for computing a task with a unique approach. This approach involves representing the last portion of a neural network as a task-configurable set of weights. The way this is accomplished is by storing several neural networks in memory in a compressed format, utilizing SVD, and then using the input to produce attention weights on each of the S V and U components of the SVD, thus reconstructing the weights, adding in a non-task-specific residue representing the remainder of the weights, and then using those weights for the remainder of the network.\n\nThe paper then uses several standard benchmarks both in standard learning tasks, RL tasks, and continual learning tasks to demonstrate that their network is able to outperform similarly powerful networks that do not contain the specific properties that they have built into their network.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the idea of attending directly over a compressed representation of a neural networks’ weights appears to me to be novel, as is the presented architecture in general.\n\nQuality: the paper presents a very interesting concept and has the experiments to demonstrate it in detail. I especially like the way that the compressed format is used to save on the storage efficiency of the program, and I wonder if it might also have some regularization advantages.\n\nThe one major critique I might give is that the idea seems to have a lot of components (the weights on the residual program, the use of the usage counters to promote less used programs, the idea that the singular values need be ordered, etc.) whose importance is only explained using theoretical justifications and some ablation analysis might go some way as to justifying the need for so many parts.\n\nClarity: While the high level of this paper seems really interesting, the clarity of the paper is lacking in a number of areas, to the point that I am having a hard time understanding many of the details being described, even after reading through the paper a number of times. I think this submission would be improved dramatically by addressing some of these concerns.\n\n- The section “neurocoder stores singular value decomposition of neural programs in program memory” I think could use much more direct information about the sizes of the various tensors involved. For example, even after reading through the paper twice I am unclear on what exactly P is: is it a matrix where rows are programs and columns are the parameters in each program? Are the matrices ones that represent the parameters in one program (somehow arranged into 2d) and there are several such matrices making P a tensor? Is it something else entirely? I lean towards the second explanation because that’s what M_U, M_V, M_S implies to me, but I am not sure. Additionally, I think equations that are fully qualified with all the relevant symbols would also be helpful in understanding exactly what is going on, even if it makes the equations somewhat less easy to read at first glance.\n- When you say “the singular programs are trained to represent unitary functions necessary for any computation whilst the composition and integration networks are trained to compose the relevant programs for the considering task” I believe that you are describing the result of an end-to-end training process rather than these networks being trained individually, but I am not 100% sure. The next sentence beginning “as such” seems disconnected from this point as well as it describes an auxiliary loss that seems to enforce the structure of the singular programs rather than anything to do with their function.\n- This might be an artifact of a misunderstanding created by the previous point, but I do not understand what’s going on in equations 3 and 4. Are those just linear combinations of attention weights and the memory modules for each of U and V? If so, why are the outputs denoted as vectors? Or since we’re retrieving the nth vector, why is n not one of the indices in M_U or M_V (both of which seem to be conditioned only on i).\n- The section “recurrent access to the program memory via the composition network” confused me at first because I thought you were accessing the program memory recurrently, that is feeding back information about the program memory into the recurrent module in order to inform future key generations. Based on a closer reading however, it seems that what you are doing is generating several queries that are dependent on the previous *queries* but not dependent on the state of the program memory in any way. If this is accurate, I think the wording should be updated to make it more clear. Perhaps “rolling out query heads rather than generating them in parallel allows us to ensure that each of the query heads is distinct from the rest and thus allows us to avoid program collapse”\n- Equation 12: I feel  like I’m missing something about how you are suppressing the magnitude of R, surely since R is a purely trainable parameter it can grow to an arbitrary size to counteract any suppression that might occur?\n- Figure 1: the diagram suggests that the residual program is generated out of the input separately from the composition network. I believe that one of the weights from the composition network is used here so an arrow should probably be added from the composition network to the residual program.\n\nSignificance: the paper seems to be fairly significant and I can see this approach being directly used with pretty much any architecture that seeks to solve multiple tasks or a single but varied task. I also see extensions of this work being applied in a number of different domains.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors seem aware of the limitations of their work and present them clearly. I agree with the authors that there are probably not many direct implications for this work, though I would say that there are possibly some positive implications if this technique in the long run allows for more efficient training of neural networks.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "r3Ot8VoPlEd", "reviewer": "Reviewer_D8Dy", "summary": "This paper proposes a general-purpose neural architecture with an explicit goal of modularity, called Neurocoder. Neurocoder dynamically produces the weights for a layer in a neural network, based on the input, by combining \"programs\" from memory. These programs consist of left and right singular vectors ($U$, $V$) and singular values ($S$). For a given input, a recurrent multi-head process is used to retrieve the program from memory. In each step of the recurrent process, we obtain a query $q$ and gate $g$ for each of the $H$ heads and for each of the $U$, $V$ and $S$. The query $q$ is used for content-based attention inside $U$, $V$, and $S$ respectively. The gate $g$ is used to combine the content-attention result with a least-recently-used attention. By putting all of this together we obtain a $u_n$, $v_n$, and $\\sigma_n$ for each head in each step of the recurrent process. We then sum $\\sum_n \\sigma_n u_n v_n^T$ to obtain the weights for the layer.\n\nThe authors evaluate this architecture on MNIST, CIFAR, RL on Atari games, and polynomial regression.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nOverall the paper seems similar to the previous work Neural Stored-Program Memory, but with differences such as the program vectors being stored in three parts as the singular vectors $U$, $V$ and singular values $S$; incorporating least-recently-used attention; using recurrence to build the output weight matrix over several time steps; and integrating a \"residual program\" that is fixed for every input and added to the dynamically-computed weights.\n\n## Quality\nThe paper aims to address important challenges about modularity and conditional computation in neural networks, and shows improvements upon baselines (although not SoTA results) on any task. Here are some general comments:\n- It's not clear if the proposed architecture addresses catastrophic forgetting in the manner mentioned, \"attending to different singular programs whilst learning different tasks\". It would be interesting to see whether this is actually the case, or if different \n- I felt that it is a bit strange to store and retrieve singular values from the memory, since they are scalars. Some statements in the paper don't make sense considering they are scalars, such as line 148-19 \"$f_\\varphi$ learns to compress each memory slot into a $K$-dimensional vector\". I am not sure attention over scalars makes a lot of sense either, especially considering that the retrieved values are processed further to ensure monotonicity in equation 5.\n- There is no mention of the computational complexity of the approach. It seems like it would be rather high considering the complexity of the approach, and lack of parallelization opportunities (for example, due to the recurrence).\n- It would be useful to see what happens if the Neurocoder layer is used in different parts of the neural network, rather than just the output layer.\n\n## Clarity\nThe overall model is quite complicated, and I would focus on making sure that the paper describes it clearly as possible. Some examples:\n- The notation can clarify the type of each variable. For example, in equation 1, my understanding was that $\\sigma_n$ is a scalar while $u_n$ and $v_n$ are (column) vectors, but there is no specific notation to clarify this.\n- In line 112, $\\mathcal{L}_{o}$ is a matrix, but it is added to a scalar in equation 2. It should be clarified how that matrix is turned into a scalar loss.\n- I felt that it was confusing to write $w_{tijh}$ as there are too many subscripts. I would drop $t$ since it is not relevant, flatten $j$ and $h$ into $n$ and make it clear in the recurrent section how the heads and steps map to $n$, and distinguish $i$ since it is used for softmax.\n\nAlthough it could take some space due to the complexity of the work, it would be very helpful as well to have the full set of equations for the computations performed written down together in one place.\n\nI couldn't understand the binary decision tree in figure 2(c) and lines 202-207 and how that is supposed to relate to the attention patterns.\n\n## Significance\nThe empirical results seem to improve upon the prior work and the work contains interesting ideas that others may build upon. However, the complexity of the approach will likely limit further adoption.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the biggest limitation of the work is the lack of ablation experiments, especially considering the complexity of the approach. The paper states on line 184 that \"For some experiments, we include ablation studies\" but I was unable to find any in the paper or appendix, except for varying the number of recurrent attention steps. Some example things that you can remove or adjust:\n- Storing the programs in full-rank form rather than as $U$ and $V$\n- Least-recently-used attention\n- Heads\n- Residual program\n\nIt would be particularly insightful if the paper can describe this work and the related prior work (HyperNetworks, FiLM, mixture of experts, Neural Stored-Program Memory) within some general framework where different axes of variation within the framework lead to the prior works and this one. \n\nWhile not technically an ablation, I suggest the authors to also report what happens when the Neurocoder layer is inserted in different places inside a neural architecture.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Y8ASeNT_1y", "reviewer": "Reviewer_4P5s", "summary": "This paper presents an architecture where the weights $P$ of a neural network’s linear layer can be determined by the input. The weight matrix $P$ of the linear layer is constructed by using its singular value decomposition $P=SEV$, with each of the columns of $S$ and $V$ determined by attending over a set of ``singular programs’’. Authors also use recurrent multi-headed attention to select multiple singular programs, using both content-based attention but also usage-based attention to improve the rank of $P$. \n\nThe authors run a range of experiments to test the architecture in a variety of settings. Starting with instance-based learning with MNIST/CIFAR, the author justifies the use of recurrent attention, and visualizes the attention pattern. The authors further justifies the use of recurrent attention with a piecewise polynomial regression sequential learning experiment, and demonstrates the attention shift at the polynomial boundaries. In the reinforcement setting, authors shows that their approach enables A3C to learn in an environment with sparse rewards. Finally, the authors demonstrate that the approach generalizes well in a multi-task learning setting, and that the approach mitigates catastrophic forgetting issues in a continual learning setting.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the central idea in the paper elegant and compelling: construct the weights of a MLP layer using its singular-value decomposition, where the singular vectors are selected (using attention) from a library of these vectors (aka ``singular programs’’). The use of recurrent attention is well justified with the experiments. The potentially low rank of the active program is fixed using a residual program.\n\nOne part of the architecture that I did not find compelling is the selection of the singular values using attention. It seemed odd to store and select these *scalar* values using attention, rather than computing the singular values directly using the attention weights (i.e. using $w_{in}^\\sigma$ in equation 5, without using $M_s(i)$ at all). I’m also not sure why the enforcement of the order of singular values is necessary: i.e. why not directly generate the singular values along with the attention weights using the recurrent attention mechanism?\n\nThe selection of experiments are impressive, although it is unfortunate that most of the details are in the supplementary material and are challenging to find. The details I had to hunt for include: (1) the authors mentioned in the MNIST experiment that the number of parameters of the baseline vs neural coder are comparable, and I did not find the comparison, (2) the author mentions that ablation studies are done, and it would have been helpful to see this information consolidated somewhere. \n\nThe writing is understandable, although it wasn’t until I saw the MNIST experiment hyperparameters that I understood which parts of the model were the trainable weights (which might just be me).\n\nHere are some more detailed feedback:\n\n\nAbstract line 8-9: I don’t know if the claim that Neural Program is treated as datum in memory is a fair statement, given the wealth of prior work that the authors cite.\n\nAppendix Figure 4: I recommend against using this analogy and omitting this figure. Experienced readers would not need this figure, and that inexperienced readers will draw misconceptions from this figure. The figure makes it look like the model is trying to reconstruct the input. It also made me think that the program memory was fixed, rather than a trainable weight.\n\nFigure 1: Minor suggestion to make $M_u$ a $6x4$ matrix and $M_v$ a $6x5$ matrix to show that you’re storing some number of singular vector (and that this number is a hyperparameter independent of the number of columns in $P$). Additionally, it is strange that $M_s$ has fewer columns than $M_u$ and $M_v$\n\nIf you need to cut content, the last two sentences in lines 82-84 can be cut. Additionally, the introduction can be tightened up significantly. \n\nFigure 2(b) Are you showing the first *3 singular vectors* or the first *5*? Looks like the x-axis in the plot ranges over 5 values. I’m also curious as to why, in the last column, the last two singular vectors look virtually identical for all the digits. \n\nFigure 2(d) Maybe helpful to make the x-axis range consistent. It is strange that even though the polynomial does not change in some transitions (e.g. x=200, 250, 300 in the top figure), the attention weights sometimes do (e.g. x=250, but not x=200) Conversely, there is a change in the polynomial in the transition x=350, but the attention weight stays the same.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I can't think of positive/negative societal impact beyond using the typical neural network architectures. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "KTNMsKGqGXm", "reviewer": "Reviewer_w4yd", "summary": "This paper proposes Neurocoder, a new general purpose neural network that “codes” itself in a data responsive way by composing relevant programs from a set of shareable, modular programs stored in external memory. Unlike previous work, Neurocoder treats modular programs as sharable datum in memory and composes relevant programs from stored “programs”. Experimental results show performance improvement in solving tasks such as object recognition, playing video games, and continual learning. \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed framework Neurocoder is interesting. The idea of learning and storing programs as external memory is interesting. \nTo demonstrate the flexibility of Neurocoder framework, the authors consider different learning paradigms: instance-based, sequential, multi-task and continual learning. The experiments were well conducted and demonstrated the effectiveness of the proposed framework. The proposed framework also required a much smaller number of parameters.\n\nThe general idea of Neurocoder is to retrieve relevant programs from memory using the attention mechanism. Similar ideas have been proposed in previous work such as the end-to-end memory networks (Sukhbaatar et al.2015), the neural turing machine (Graves et al. 2014), and the neural module networks (Jacob et al. 2016). It would be better if the authors explicitly describe the similarities and differences between Neurocoder and the previous work, and their novel contributions.\n\nNeurocoder induces modular programs and store them in external memory, so scalability may become an issue, which needs to be evaluated as well. In other words, how many programs can Neurocoder store in its external memory? \n\nThe proposed approach treats programs in the form of matrices/neural networks. There are some related work that synthesizes actual programs through input/output. For example: \n\nShu et al., Neural Programming by Example, Proc. the 31st AAAI Conference on Artificial Intelligence (AAAI 2017), February 2017, San Francisco, California, USA.\n\nMinor:\n\n. In Figure1, why only the last layer of the main network is adaptively loaded?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "To me, this work does not have negative societal impacts (including potential malicious or unintended uses, environmental impact, fairness considerations, privacy considerations, and security considerations).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate that this paper tackles an important problem of conditional computation, and proposed some interesting ideas.

All the reviews are slightly on the positive side, but some have low confidence.  Reading through the reviews, the AC recognizes that all reviewers were affected by the clarity issues of the paper, and there were quite a few misunderstandings about the model details, leading to perhaps inaccurate judgement of the paper.  This is in part caused by the fact that the model proposed in the paper is quite complicated with a lot of moving parts, which makes it challenging to present the model clearly.  On the other hand, only very limited ablations were done to understand the impact of each design decision.

The clarity issue and the absence of sufficient ablations are the two major factors holding back the AC’s decision to accept this paper.  After calibrating with SAC across a range of papers (we have both went through this paper ourselves) we decided to reject this paper but encourage the authors to improve it and send it to the next venue.
Decision: Reject