Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs
Year: 2021

Abstract:
Artificial Neural Networks are functionally equivalent to special-purpose comput1 ers. Their inter-neuronal connection weights represent the learnt Neural Program 2 that instructs the networks on how to compute the data. However, without stor3 ing Neural Programs, they are restricted to only one, overwriting learnt programs 4 when trained on new data. Here we design Neurocoder, a new class of general5 purpose neural networks in which the neural network “codes” itself in a data6 responsive way by composing relevant programs from a set of shareable, modular 7 programs stored in external memory. For the first time, a Neural Program is ef8 ficiently treated as a datum in memory. Integrating Neurocoder into current neu9 ral architectures, we demonstrate new capacity to learn modular programs, reuse 10 simple programs to build complex ones, handle pattern shifts and remember old 11 programs as new ones are learnt, and show substantial performance improvement 12 in solving object recognition, playing video games and continual learning tasks. 13

text: Artificial Neural Networks are functionally equivalent to special-purpose comput-1 ers. Their inter-neuronal connection weights represent the learnt Neural Program2 that instructs the networks on how to compute the data. However, without stor-3 ing Neural Programs, they are restricted to only one, overwriting learnt programs4 when trained on new data. Here we design Neurocoder, a new class of general-5 purpose neural networks in which the neural network “codes” itself in a data-6 responsive way by composing relevant programs from a set of shareable, modular7 programs stored in external memory. For the first time, a Neural Program is ef-8 ficiently treated as a datum in memory. Integrating Neurocoder into current neu-9 ral architectures, we demonstrate new capacity to learn modular programs, reuse10 simple programs to build complex ones, handle pattern shifts and remember old11 programs as new ones are learnt, and show substantial performance improvement12 in solving object recognition, playing video games and continual learning tasks.13
1 Introduction14
From its inception in 1943 until recently, the fundamental architectures of Artificial Neural Net-15 works remained largely unchanged - a program is executed by passing data through a network of16 artificial neurons whose inter-neuronal connection weights are learnt through training with data.17 These inter-neuronal connection weights, or Neural Programs, correspond to a program in modern18 computers [32]. Memory Augmented Neural Networks (MANN) are an innovative solution allow-19 ing networks to access external memory for manipulating data [11, 12]. But they were still unable20 to store Neural Programs in such external memory, and this severely limits machine learning. Stor-21 ing inter-neuronal connection weights only in their network does not permit modular separation of22 Neural programs and is analogous to a computer with one fixed program. Recent works introduce23 conditional computation via adjusting or activating parts of a network in an input-dependent manner24 [39, 33, 4, 13, 28], but networks remain monolithic. Current networks forget when retrained, old25 inter-neuronal connection weights are merged with new ones or erased.26
The brain is modular, not a monolithic system [8, 6]. Neuroscience research indicates that the brain is27 divided into functional modules [19, 7, 9]. If the neural program for each module is kept in separate28 networks, networks proliferate. Modular neural networks, another form of conditional computation,29 combine the output of multiple expert networks, but as the experts grow, the networks grow drasti-30 cally [20, 14, 35, 29]. This requires huge computational storage and introduces redundancy as these31 experts do not share common basic programs.32
A pathway out of this bind is to keep such basic programs in memory and combine them as required.33 This brings neural networks towards modern general-purpose computers that use the stored-program34 principle [37, 40] to efficiently access reusable programs in external memory. Here we show how35 Neurocoder, a new neural framework, introduces a new class of general-purpose conditional compu-36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
tation machines in which a neural network can be “coded” in an input-dependent manner. Efficient37 decomposition of Neural Programs creates shareable modular components that can reconstruct the38 whole program space. These components change their “shapes” based on training and are stored39 in an external Program Memory. Then, in a data-responsive way, a Program Controller retrieves40 relevant components to build the Neural Program. This is analogous to shape-shifting Lego bricks41 that can be reused to build unlimited shapes and structures (See Appendix Fig. 4).42
Using adaptive modular components vastly increases the learning capacity of the neural network43 by allowing re-utilisation of parameters, effectively curbing network growth as programs increase.44 More importantly, unlike pre-defined sub-networks or modules [20, 1] that combine at activation45 level, the construction of our modular components is dynamic and performed on the weight space.46 The Neural Program construction is learnt through training via traditional backpropagation [30] as47 the architecture is end-to-end differentiable.48
2 Methods49
2.1 System overview50
A Neurocoder is a neural network (Main Network) coupled to an external Program Memory through51 a Program Controller. The working program of the Main Network processes the input data to pro-52 duce the output. This working program is “coded” by the Program Controller by creating an input-53 dependent active program from the Program Memory (Fig. 1). The following gives a high-level54 description of the Neurocoder framework and then the details.55
Neurocoder stores Singular Value Decomposition of Neural Programs in Program Memory56
The Neural Program needs to be stored efficiently in Program Memory. This is challenging as there57 may be millions of inter-neuronal connection weights, thus storing them directly ([22]) is grossly58 inefficient. Instead, the Neurocoder forms the basis of a subspace spanned by Neural Programs and59 stores the singular values and vectors of this subspace in memory slots of the Program Memory60 (hereafter referred to as singular programs). Based on the input, relevant singular programs are61 retrieved, a new program is reconstructed and then loaded in the Main Network to process the input.62 This representational choice significantly reduces the number of stored elements and allows each63 singular program to effectively represent a unitary function of the active program.64
The active program matrix P can be composed by standard low-rank approximation as65
P = USVT = rm∑ n σnunv > n (1)
where U and V are matrices of the left and right singular vectors, and S the matrix of singular values.66 rm is the total number of components we want to retrieve. {σn}rmn=1 is the attended singular values,67 {un}rmn=1 and {vn} rm n=1 the attended singular vectors of S, U, and V, respectively. The Program68 Memory is crafted as three singular program memories {MU ,MV ,MS}–each of their memory69 slot stores a singular component or singular program. The process “codes” the active program70 using singular programs from the program memories. The coding is conditioned on input xt, yet71 we drop index t for notation simplification and leave the details on the computation of σn, un, vn in72 Sec. 2.2.73
The Program Memory also maintains the status for each singular program in terms of access and74 usage. To access a singular program, program keys (k) are used. These keys are low-dimensional75 vectors that represent the singular program function and computed by a neural network that ef-76 fectively compresses the singular program. The program usage (m) measures memory utilisation,77 recording how much a memory slot is used in constructing a program. The components of the78 Program Memory are summarised in Fig. 1 (c).79
Recurrent multi-head program attention mechanisms for program storage and retrieval80 Neural networks use the concept of differentiable attention to access memory [11, 2]. This de-81 fines a weighting distribution over the memory slots essentially weighting the degree to which each82 memory slot participates in a read or write operation. This is unlike conventional computers that use83 a unique address to access a single memory slot.84
Here we use two kinds of attention. First is content-based attention [11, 12] to ensure that the singu-85 lar program is selected based on its functionality and the data input. This is achieved by producing86 a query vector based on the input and comparing it to the program keys (k) using cosine similarity.87 Higher cosine similarity scores indicate higher attention weights to the singular programs associated88 with those program keys. Second, to encourage better memory utilisation, higher attention weights89 are assigned to slots with lower program usage (m) through usage-based attention [12, 31]. The90 attention weights from the two schemas are then combined using interpolating gates to compose the91 final attention weights to the Program Memory.92
We adapt multi-head attention [11, 38] that applies multiple attentions in parallel to retrieveH singu-93 lar components. Besides, we introduce a recurrent attention mechanism, in which multi-head access94 is performed recurrently in J steps. The j-th set of H retrieved components is conditioned on the95 previous ones. This recurrent, multi-head attention allows the composition network to incrementally96 search for optimal components for building relevant active programs.97
Neurocoder learns to “code” a relevant working program via training98
The structure of the Program Memory and the role of the Program Controller facilitates the au-99 tomatic construction of working programs via training. The Program Controller controls memory100 access through its composition network that creates the attention weight defining how to weight the101 singular programs in the memories. A weighted summation of the singular programs results in the102 attended singular program. Applying the recurrent multi-head attention described earlier, multiple103 attended singular programs are retrieved to construct an active program (Eq. 1). Then the Program104 Controller generates a residual program using its integration network, adding to the active program105
to produce the working program of the Main Network. This addition enables creation of flexible106 higher-rank working programs, which compensates for the low-rank coding process. The structure107 of the Program Controller is illustrated in Fig. 1 (b).108
The singular programs are trained to represent unitary functions necessary for any computation109 whilst the composition and integration networks are trained to compose the relevant programs for110 the considering task. As such, beside minimising the task loss, we enforce orthogonality of stored111 singular vectors by minimising Lo = MUM>U − I+MVM>V − I. The parameters of the networks,112 and the stored singular programs are adjusted using gradient training via minimising the total loss113
L = Ltask + aLo (2) where Ltask represents the supervised task loss and Lo represents the orthogonal loss weighted by114 a hyper-parameter a to enforce orthogonality of the singular vectors.115
2.2 Attention mechanisms for Program Memory116
Here we describe program attention mechanisms used in this paper. Given wuin, w v in, w σ in (jointly117 denoted as wu,v,σin )–the attention weight to the i-th slot of the singular program memories MU , MV118 and MS , we retrieve the n-th singular vector as follows,119
un = Pu∑ i=1 wuinMU (i) (3)
vn = Pv∑ i=1 wvinMV (i) (4)
For the singular values, we need to enforce σ1 > σ2 > ... > σrm > 0, thus we retrieve using120
σn = softplus (∑Ps i=1 w σ inMS (i) ) n = rm σn+1 + softplus (∑Ps i=1 w σ inMS (i) ) n < rm
(5)
Here, Pu, Pv and Ps are the number of memory slots of MU , MV and MS , respectively. In this121 paper, we set P = Pu = Pv = Ps as the number of memory slots of the Program Memory. We note122 that these notations are specified for some data input xt and the index n later maps to an attention123 head h, and an attention step j, hence the full notation should be wu,v,σtijh . To simplify notations, we124 will drop u, v, σ from now and describe the computation of a representative wtijh for any of the125 three program memories in the following parts.126
Recurrent Access to the Program Memory via the composition network127
To perform program attention, the Program Controller employs a composition network (denoted128 as fθ), which takes the current input xt and produce program composition control signals (ξ p t ).129 If fθ performs all attentions concurrently via multi-head attention as in [11, 38], it may lead to130 program collapse [22]. To have a better control of the component formation and alleviate program131 collapse, we propose to recurrently attend to the program memory. To this end, we implement fθ as132 a recurrent neural network (LSTM [16]) and let it access the program memory J times, resulting in133 ξpt = { ξptj }J j=1
. At access step j, the recurrent network updates its hidden states and generates ξptj134 using recurrent dynamics as135
ξptj , hj = fθ (xt, hj−1) (6) where h0 is initialized as zeros and ξ p tj is the program composition control signal at step j that136 depends on both on the input data xt and the the previous state hj−1. Particularly, the control signal137 contains the queries and the interpolation gates for each head to compute the program attention138 weight: ξptj = {qtjh, gtijh} H h=1. Here, at each attention step, we perform multi-head attention with139 H as the number of attention heads and thus, each ξptj consists of H pairs of queries and gates.140 Hence, the total number of retrieved components rm = J ×H and the index n = j ×H + h.141

heading: Attending to Programs by “Name”142
text: Inspired by the content-based attention mechanism for data memory [11], we use the query to look143 for the singular programs. In computer programming, to find the appropriate program for some144 computation, we often refer to the program description or at least the name of the program. Here, we145 create the “name” for our neural programs by compressing the program content to a low-dimensional146 key vector. As such, we employ a neural network (fϕ) to compute the program memory keys as147
ki = fϕ (M (i)) (7)
where ki ∈ RK and i is the row index of the program memory. Here, fϕ learns to compress each148 memory slot into a K-dimensional vector. As the singular programs evolve, their keys get updated.149 In this paper, we update the program keys after each learning iteration during training.150
Finally the content-based program memory attention ctijh is computed using cosine distance be-151 tween the program keys ki and the queries qtjh as152
ctijh = softmax (i)
( qtjh · ki
||qtjh|| · ||ki||
) (8)
Making Every Program Count153
Similarly to [12, 31], in addition to the content-based attention, we employ a least-used reading154 strategy to encourage the Program Controller to assign different singular programs to different com-155 ponents. In particular, we calculate the memory usage for each program slot across attentions as156
157
mtijh = max j̃≤j
( wtij̃h ) (9)
Since we want to consider only lI amongst P memory slots that have smallest usages, let m̂lItjh158 denote the value of the lI -th smallest usage, then the least-used attention is computed as159
ltijh =
{ max i
(mtijh)−mtijh ;mtijh ≤ m̂lItjh 0 ;mtijh > m̂ lI tjh
(10)
The final program memory attention is computed as160
wtijh = sigmoid (gtijh) ctijh + (1− sigmoid (gtijh)) ltijh (11) Since the usage record are computed along the memory accesses, the multi-step Neurocoder utilises161 this attention mechanism better than the single-step Neurocoder, creating different attention styles162 (see Sec. 3.2). The composition the active program Pt is illustrated in Appendix’s Fig. 5.163
2.3 Program Integration via the integration network164
Since the working program Pt only contains top rm principal components, it is low-rank and may165 be not flexible enough for sophisticated computation. We propose to enhance Pt with a residual166 program R– a traditional connection weight trained as the integration network’s parameters, which167 is constant after training w.r.t t. The residual program represents the sum of the remaining less168 important components. To this end, we suppress R with a multiplier that is smaller than σtrm– the169 smallest singular value of the main components - resulting in the integration formula170
Wt = Pt + w r tσtrmR (12)
where wrt = sigmoid (fφ (xt)) is an adaptive gating value that controls the contribution of the171 residual program. fφ is the integration network in the Program Controller and hence, in our imple-172 mentation, the integration control signal sent by the Program Controller is λpt = {wrt , σtrm} . We173 note that in our experiments, the program integration can be disabled (Wt is directly set to Pt) to174 prove the contribution of Pt or reduce the number of parameters. The working program Wt is then175 used by the Main Network to execute the input data xt (see (Fig. 1 (a))). For example, with linear176
classifier Main Network, the execution is yt = xtWt. Appendix’s Table 2 summarises the notations177 used for important parameters of Neurocoder.178
3 Results179
To demonstrate the flexibility of Neurocoder framework, we consider different learning paradigms:180 instance-based, sequential, multi-task and continual learning. We do not focus on breaking perfor-181 mance records by augmenting state-of-the-art models with Neurocoder. Rather our inquiry is on182 re-coding feed-forward layers with the Neurocoder’s programs and testing on varied data types to183 demonstrate its intrinsic properties. For some experiments, we include ablation studies.184
We compare the performance of diverse Main Networks (MN) with and without Neurocoder. We185 also augment the Main Networks with other recent conditional computing methods, either modular186 (sparse Mixture of Experts, Neural Stored-program Memory) or monolithic (HyperNets, FiLM) to187 form stronger baselines across our experiments. In our experiments, we always apply Neurocoder188 to all layers of multi-layer perceptrons (MLP) or just the final feed-forward layer of deep CNN189 networks (LeNet, DenseNet, ResNet), RNNs (GRU, LSTM), MANN (NTM). Other competitors190 such as MOE, NSM, HyperNet and FiLM are applied to the Main Networks in the same manner.191
3.1 Instance-based learning - Object Recognition192
We tested Neurocoder on instance-based learning through classical image classification tasks using193 MNIST [24] and CIFAR [21] datasets. The first experiment interpreted Neurocoder’s behaviour194 in classifying digits into 10 classes (0 − 9) using linear classifier Main Network. With equivalent195 model size, Neurocoder using the novel recurrent attention surpasses the performance of the linear196 classifier [24] by up to 5% (Fig. 2 (a)).197
To differentiate the input, Neurocoder attends to different components of the active program to198 guide the decision-making process. Fig. 2 (b) shows single-step and multi-step attention to the first199 3 singular vectors for each digit across memory slots. Multi-step attention produces richer patterns200 compared to single-step Neurocoder that manages only 2 attention weight patterns.201
Fig. 2 (c) illustrates how Neurocoder performs modular learning by showing the attention assign-202 ment for top 3 singular vectors as a binary decision tree. Digits under the same parental node share203 similar attention paths, and thereby similar active programs. Some digits look unique (e.g. 7) result-204 ing in active programs composed of unique attention paths, discriminating themselves early in the205 decision tree. Some digits (e.g. 0 and 9) share the same attention pattern for the first 3 components206 and are thus unclassifiable. They can only be distinguished by considering more singular vectors.207
We integrated Neurocoder with deep networks - 5-layer LeNet and 100-layer DenseNet - and tested208 on CIFAR datasets. Neurocoder significantly outperformed the original Main Networks with perfor-209 mance gain 1−5%. Compared with recent conditional computing models such as sparse Mixture of210 Experts (MOE [35]) and Neural Stored-program Memory (NSM [22]), Neurocoder required a tenth211 of the number of parameters and performed better by up to 8− 10% (see Appendix’s Table 3).212
3.2 Sequential learning - Adaption to sequence changes and game playing using213 reinforcement learning214
Recurrent neural networks (RNN) can learn from sequential data by updating the hidden states of215 the networks. However, this does not suffice when local patterns shift, as is often the case. We now216 demonstrate that Neurocoder helps RNNs overcome this limitation by composing diverse programs217 to handle sequence changes.218
Synthetic polynomial auto-regression We created a simple auto-regression task in which data219 points are sampled from polynomial function chunks that change over time. The Main Network is220 a strong RNN–Gated Recurrent Unit (GRU [5]). We found that GRU integrated with a single-step221 or multi-step Neurocoder converged much faster than all other baselines. The other conditional222 computing counterparts (HyperNet [13], FiLM [28]) adapt by re-scaling weights or activation of the223 GRU, which were shown inferior to our modular approach (Appendix’s Fig. 6).224
Visualising the first singular vector attention weights in MU , we find that the multi-step attention225 Neurocoder changes its attention following polynomial changes - it attends to the same singular pro-226 gram when processing data from the same polynomial and alters attention for data from a different227 polynomial (Fig. 2(d)). In contrast, the single-step Neurocoder only changes its attention when228 there is a remarkable change in y-coordinate values (Fig. 2(e)). Although single-step Neurocoder229
converges well, it did not discover the underlying structure of the data, and thus underperformed230 the multi-step Neurocoder. We hypothesise that when recurrence is employed, usage-based atten-231 tion takes effect, stipulating better memory utilisation and diverse attentions over timesteps. We ran232 multi-step Neurocoder without usage-based attention. The results were worse than the full multi-233 step Neurocoder, which confirms our hypothesis (Appendix’s Fig. 6).234
Atari game reinforcement learning We used reinforcement learning as a further testbed to show235 the ability to adapt to environmental changes. We performed experiments on several Atari 2600236 games [3] wherein the agent was implemented as the Asynchronous Advantage Actor-Critic (A3C237 [26]). In the Atari platform, agents are allowed to observe the screen snapshot of the games and act238 to earn the highest score. We augmented the A3C by employing Neurocoder’s working programs239 for feed-forward layers of the actor and critic networks, aiming to decompose the policy and value240 function into singular programs that were selected depending on the game state.241
Frostbite and Montezuma’s Revenge. These games are known to be challenging for A3C and other242 algorithms [26]. We trained A3C and HyperNet-based A3C for over 300 million steps, yet these243 models did not show any sign of learning, performing equivalently to random agents. For such com-244 plicated environments with sparse rewards, both the monolithic neural networks and the HyperNet’s245 unstored fast-weights fail to learn (almost zero scores). In contrast, Neurocoder enabled A3C to246 achieve from 1, 500 to 3, 000 scores on these environments (Fig. 3), confirming the importance of247 decomposing a complex solution to smaller, simple stored programs.248
3.3 Multi-task learning - Solving mutliple algorithms simultenously249
Here we explore the modular learning capability of Neurocoder in multi-task setting. Inspired by al-250 gorithmic sequencing tasks [22], we created a challenging sequential multi-task benchmark wherein251 the input sequence is a series of sub-sequences from 4 algorithms: Copy, Repeat Copy, Associative252 Recall and Priority Sort [11]. Each sub-sequence, following a task identification vector, represents253 the input for each task. In each input sequence, n tasks were sampled from the set of 4 algorithms254 randomly with replacement and the output sequences were created correspondingly.255
We trained a MANN–Neural Turing Machine (NTM [11]) Main Network with FiLM, HyperNet and256 our Neurocoder augmentation on sequences of n = 4 tasks, and tested with sequences of n = 4 and257 n = 8 tasks. Appendix’s Fig. 7 demonstrates that Neurocoder was performant in both test settings,258 not only achieving lowest error on n = 4, but also being the only one generalised well to n = 8259 scenario, which was unseen during training.260
3.4 Continual learning - Learning tasks sequentially without catastrophic forgetting261
In continual learning, standard neural networks often suffer from “catastrophic forgetting” in which262 they cannot retain knowledge acquired from old tasks upon learning new ones [10]. Our Neurocoder263 offers natural mitigation of such catastrophic forgetting in neural networks by attending to different264 singular programs whilst learning different tasks.265
In this case, in addition to the Main Network, we examine several continual learning algorithms266 with and without Neurocoder. These algorithms, including Elastic Weight Consolidation (EWC267 [41]) and Synaptic Intelligence (SI [41]), work by regularising the loss function and thus can be268 easily combined with Neurocoder by modifying the loss Ltask. We demonstrate that Neurocoder269
can improve these continual learning algorithms without requiring additional assumptions as in other270 approaches [25, 36, 34] that either utilise task embedding or replay memory.271
Split MNIST We first considered the split MNIST dataset–a standard continual learning bench-272 mark wherein the original MNIST was split into a 5 2-way classification tasks, consecutively pre-273 sented to a Multi-layer Perceptron Main Network (MLP). We followed the benchmarking as in [17]274 in which various optimisers and state-of-the-art continual learning methods were examined under in-275 cremental task and domain scenarios. We measured the performance of the MLP versus Neurocoder276 and NSM under each continual learning method. In both scenarios, Neurocoder was compatible277 with all continual leaning methods, demonstrating superior performance over MLP and NSM with278 performance gain between 1 to 16% (see Appendix’s Table 5 and 1).279
Split CIFAR We verified the scalability of Neurocoder to more challenging datasets. We split280 CIFAR datasets as in the split MNIST, resulting in 5-task 2-way split CIFAR10 and a 20-task 5-way281 split CIFAR100. We used Main Network ResNet [15]–a very deep CNN architecture.282
When we stressed the orthogonal loss (a = 10) and used bigger program memory (100 slots), Neu-283 rocoder improved ResNet classification by 15% and 10% on CIFAR10 and CIFAR100, respectively.284 When we integrated Neurocoder with Synaptic Intelligence (SI [41]), the performance was further285 improved, maintaining a stable performance above 80% accuracy for CIFAR10 and outperforming286 using SI alone by 10% for CIFAR100 (see Appendix’s Fig. 8).287
4 Discussion288
Our experiments demonstrate that Neurocoder is capable of re-coding Neural Programs in distinctive289 neural networks, amplifying their capabilities in diverse learning scenarios: instance-based, sequen-290 tial, multi-task and continual learning. This consistently results in significant performance increase,291 and further creates novel robustness to pattern shift and catastrophic forgetting. This ability for each292 architecture to re-code itself is made possible without changing the way it is trained, or majorly293 increasing the number of parameters it needs to learn (see Appendix Table 7).294
The MNIST problem illustrates the reasoning process of Neurocoder when classifying digit images295 wherein its singular program assignment resembles a binary tree decision-making process - it shows296 how some singular programs are shared, others are not. The polynomial auto-regression problem297 highlights the importance of efficient memory utilisation in re-constructing the working program298 enabling discovery of hidden structures in sequential data. Training our framework with reinforce-299 ment learning, we enable neural agents to solve complex games wherein traditional methods fail300 or learn slowly. Neurocoder also works well with multi-task setting, as shown in the challenging301 multi-algorithm benchmark. Finally, continual learning problems show that Neurocoder mitigates302 catastrophic forgetting efficiently under different learning settings/algorithms.303
Our solution offers a single framework that is scalable and adaptable to various problems and learn-304 ing paradigms. Unlike previous attempts to employ a bank of separate big programs [20, 35, 22],305 Neurocoder maintains only shareable, smaller components that can reconstruct the whole program306 space, thereby heavily utilising the parameters and preventing the model from proliferating. We307 note that Neurocoder is orthogonal to approaches employing tensor decomposition to reduce the308 number of parameters or hasten the computation [27, 23]. Neurocoder composes rather than decom-309 pose the neural weights. Our aim is not only to enable efficient parameter usage, but also achieve310 general-purpose computing power, outperforming other methods in numerous learning problems.311
One limitation of this work is the number of additional hyperparameters, which prevents us from312 fully tuning Neurocoder. Our research aims to add new capabilities to current neural networks to313 improve their performance and make them robust in different learning scenario. Hence, we do314 not see any intermediate negative societal impact. In future work, we will extend Neurocoder’s315 application beyond feed-forward layers. It would be interesting to efficiently replace all neural layers316 including CNN or Transformer by Neurocoder’s programs. We can also further extend Neurocoder’s317 ability by allowing a growing Program Memory, in which the model decides to add or erase memory318 slots as the number of data patterns grows or shrinks beyond the current program space’s capacity.319 Such a system represents a more flexible general-purpose computer that can dynamically allocate320 computing resources by itself without human pre-specification.321
References322 [1] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In323 Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39–48,324 2016.325
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by326 jointly learning to align and translate. In International Conference on Learning Represen-327 tations, 2015.328
[3] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning329 environment: An evaluation platform for general agents. Journal of Artificial Intelligence330 Research, 47:253–279, 2013.331
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradi-332 ents through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432,333 2013.334
[5] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares,335 Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–336 decoder for statistical machine translation. In Conference on Empirical Methods in Natural337 Language Processing (EMNLP), pages 1724–1734. Association for Computational Linguis-338 tics, October 2014.339
[6] JC Eccles. The modular operation of the cerebral neocortex considered as the material basis of340 mental events. Neuroscience, 6(10):1839–1855, 1981.341
[7] Gerald M Edelman. Neural darwinism: selection and reentrant signaling in higher brain func-342 tion. Neuron, 10(2):115–125, 1993.343
[8] Gerald M Edelman and Vernon B Mountcastle. The mindful brain: cortical organization and344 the group-selective theory of higher brain function. Massachusetts Inst of Technology Pr, 1978.345
[9] Richard SJ Frackowiak. Human brain function. Elsevier, 2004.346
[10] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive347 sciences, 3(4):128–135, 1999.348
[11] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint349 arXiv:1410.5401, 2014.350
[12] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka351 Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John352 Agapiou, et al. Hybrid computing using a neural network with dynamic external memory.353 Nature, 538(7626):471–476, 2016.354
[13] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on355 Learning Representations, 2017.356
[14] Bart LM Happel and Jacob MJ Murre. Design and evolution of modular neural network archi-357 tectures. Neural networks, 7(6-7):985–1004, 1994.358
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image359 recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni-360 tion, pages 770–778, 2016.361
[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation,362 9(8):1735–1780, 1997.363
[17] Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual364 learning scenarios: A categorization and case for strong baselines. In NeurIPS Continual365 learning Workshop, 2018.366
[18] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely con-367 nected convolutional networks. In Proceedings of the IEEE conference on computer vision and368 pattern recognition, pages 4700–4708, 2017.369
[19] D.H. Hubel. Eye, Brain, and Vision. Scientific American Library series. Scientific American370 Library, 1988.371
[20] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mix-372 tures of local experts. Neural computation, 3(1):79–87, 1991.373
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.374 TR-2009, 2009.375
[22] Hung Le, Truyen Tran, and Svetha Venkatesh. Neural stored-program memory. In Interna-376 tional Conference on Learning Representations, 2020.377
[23] V Lebedev, Y Ganin, M Rakhuba, I Oseledets, and V Lempitsky. Speeding-up convolutional378 neural networks using fine-tuned cp-decomposition. In International Conference on Learning379 Representations, 2015.380
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning381 applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.382
[25] David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learn-383 ing. In Advances in neural information processing systems, pages 6467–6476, 2017.384
[26] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap,385 Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce-386 ment learning. In International conference on machine learning, pages 1928–1937, 2016.387
[27] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing388 neural networks. In Advances in neural information processing systems, pages 442–450, 2015.389
[28] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM:390 Visual Reasoning with a General Conditioning Layer. In AAAI Conference on Artificial Intel-391 ligence, New Orleans, United States, February 2018.392
[29] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selec-393 tion of non-linear functions for multi-task learning. In International Conference on Learning394 Representations, 2018.395
[30] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by396 back-propagating errors. Nature, 323(6088):533–536, 1986.397
[31] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.398 Meta-learning with memory-augmented neural networks. In International conference on ma-399 chine learning, pages 1842–1850, 2016.400
[32] Jürgen Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent401 neural networks for dynamic reinforcement learning and planning in non-stationary environm402 nts. TR FKI-126-90, 1990.403
[33] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic404 recurrent networks. Neural Computation, 4(1):131–139, 1992.405
[34] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic406 forgetting with hard attention to the task. In International Conference on Machine Learning,407 pages 4548–4557, 2018.408
[35] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.409 Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-410 experts layer. In International Conference on Learning Representations, 2017.411
[36] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep412 generative replay. In Advances in Neural Information Processing Systems, pages 2990–2999,413 2017.414
[37] A.M Turing. On computable numbers, with an application to the entscheidungsproblem. In415 Proceedings of the London Mathematical Society, 1936.416
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,417 Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural informa-418 tion processing systems, pages 5998–6008, 2017.419
[39] Christoph von der Malsburg. The correlation theory of brain function, 1981.420
[40] John Von Neumann. First draft of a report on the edvac. IEEE Annals of the History of421 Computing, 15(4):27–75, 1993.422
[41] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intel-423 ligence. Proceedings of machine learning research, 70:3987, 2017.424
425
Checklist
1. For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] See Discussion and Appendix’s
"Training procedure and hyper-parameter selections." (c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Discussion. (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes] 2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)? [No] Data is public, provided with link. Code will be avaialble after published. All training details are available and can be used to implement and reproduce the results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix’s "Training procedure and hyper-parameter selections."
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Experimental Results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix’s "Training procedure and hyper-parameter selections.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets? [No] All assets are public. We will mention
the license detail after the paper is published. (c) Did you include any new assets either in the supplemental material or as a URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No] 5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
426

Review summaries:
This paper approaches the problem of multi-purpose neural networks that can learn several tasks or several methods for computing a task with a unique approach. This approach involves representing the last portion of a neural network as a task-configurable set of weights. The way this is accomplished is by storing several neural networks in memory in a compressed format, utilizing SVD, and then using the input to produce attention weights on each of the S V and U components of the SVD, thus reconstructing the weights, adding in a non-task-specific residue representing the remainder of the weights, and then using those weights for the remainder of the network.  The paper then uses several standard benchmarks both in standard learning tasks, RL tasks, and continual learning tasks to demonstrate that their network is able to outperform similarly powerful networks that do not contain the specific properties that they have built into their network.

This paper proposes a general-purpose neural architecture with an explicit goal of modularity, called Neurocoder. Neurocoder dynamically produces the weights for a layer in a neural network, based on the input, by combining "programs" from memory. These programs consist of left and right singular vectors ($U$, $V$) and singular values ($S$). For a given input, a recurrent multi-head process is used to retrieve the program from memory. In each step of the recurrent process, we obtain a query $q$ and gate $g$ for each of the $H$ heads and for each of the $U$, $V$ and $S$. The query $q$ is used for content-based attention inside $U$, $V$, and $S$ respectively. The gate $g$ is used to combine the content-attention result with a least-recently-used attention. By putting all of this together we obtain a $u_n$, $v_n$, and $\sigma_n$ for each head in each step of the recurrent process. We then sum $\sum_n \sigma_n u_n v_n^T$ to obtain the weights for the layer.  The authors evaluate this architecture on MNIST, CIFAR, RL on Atari games, and polynomial regression.

This paper presents an architecture where the weights $P$ of a neural network’s linear layer can be determined by the input. The weight matrix $P$ of the linear layer is constructed by using its singular value decomposition $P=SEV$, with each of the columns of $S$ and $V$ determined by attending over a set of ``singular programs’’. Authors also use recurrent multi-headed attention to select multiple singular programs, using both content-based attention but also usage-based attention to improve the rank of $P$.   The authors run a range of experiments to test the architecture in a variety of settings. Starting with instance-based learning with MNIST/CIFAR, the author justifies the use of recurrent attention, and visualizes the attention pattern. The authors further justifies the use of recurrent attention with a piecewise polynomial regression sequential learning experiment, and demonstrates the attention shift at the polynomial boundaries. In the reinforcement setting, authors shows that their approach enables A3C to learn in an environment with sparse rewards. Finally, the authors demonstrate that the approach generalizes well in a multi-task learning setting, and that the approach mitigates catastrophic forgetting issues in a continual learning setting. 

This paper proposes Neurocoder, a new general purpose neural network that “codes” itself in a data responsive way by composing relevant programs from a set of shareable, modular programs stored in external memory. Unlike previous work, Neurocoder treats modular programs as sharable datum in memory and composes relevant programs from stored “programs”. Experimental results show performance improvement in solving tasks such as object recognition, playing video games, and continual learning.