Continual learning using hash-routed convolutional neural networks
Authors: Ahmad Berjaoui
Venue:  (2021)
Keywords: Lifelong learning
continual learning
feature hashing

Abstract: Continual learning could shift the machine learning paradigm from data centric to model centric. A continual learning model needs to scale efficiently to handle semantically different datasets, while avoiding unnecessary growth. We introduce hash-routed convolutional neural networks: a group of convolutional units where data flows dynamically. Feature maps are compared using feature hashing and similar data is routed to the same units. A hash-routed network provides excellent plasticity thanks to its routed nature, while generating stable features through the use of orthogonal feature hashing. Each unit evolves separately and new units can be added (to be used only when necessary). Hash-routed networks achieve excellent performance across a variety of typical continual learning benchmarks without storing raw data and train using only gradient descent. Besides providing a continual learning framework for supervised tasks with encouraging results, our model can be used for unsupervised or reinforcement learning.

Under review as a conference paper at ICLR 2021
CONTINUAL LEARNING USING HASH-ROUTED CONVO- LUTIONAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning could shift the machine learning paradigm from data centric to model centric. A continual learning model needs to scale efficiently to handle semantically different datasets, while avoiding unnecessary growth. We introduce hash-routed convolutional neural networks: a group of convolutional units where data flows dynamically. Feature maps are compared using feature hashing and similar data is routed to the same units. A hash-routed network provides excellent plasticity thanks to its routed nature, while generating stable features through the of orthogonal feature hashing. Each unit evolves separately and new units can be added (to be used only when necessary). Hash-routed networks achieve excellent performance across variety of typical continual learning benchmarks without storing raw data and train using only gradient descent. Besides providing continual learning framework for supervised tasks with encouraging results, our model can be used for unsupervised or reinforcement learning.
INTRODUCTION
When faced with new modeling challenge, data scientist will typically train model from class of models based on her/his expert knowledge and retain the best performing one. The trained model is often useless when faced with different data. Retraining mance when trying to reuse the model on the original data. This is what is known as catastrophic forgetting [McCloskey& Cohen 1989). Although transfer learning avoids retraining networks from scratch, keeping the acquired knowledge in a trained model and using it to learn new tasks is not straightforward. The real knowledge remains with the human expert. Model training usually datu centric task. Continual learning (Thrun 1995) makes model training model centric task by maintaining acquired knowledge previous learning tasks.
new data will result in poor perfor-
Recent work in continual (or lifelong) learning has focused on supervised classification tasks and most of the developed algorithms do not generate stable features that could be used for unsupervised learning tasks, would more generic algorithm such the one we present. Models should also be able to adapt and scale reasonably to accommodate different learning tasks without using an ex- ponential amount of resources, and preferably with little data scientist intervention.
To tackle this challenge, we introduce hash-routed networks (HRN), A HRN is composed of mul- tiple independent processing units. Unlike typical convolutional neural networks (CNN), the data flow between these units is determined dynamically by measuring similarity between hashed feature maps. The generated feature maps are stable. Scalability is insured through unit evolution and by increasing the number of available units, while avoiding exponential memory use.
This new type of network maintains stable performance across a variety of tasks (including seman- tically different tasks). We describe expansion, update and regularization algorithms for continual learning. We validate our approach using multiple publicly available datasets, by comparing super- vised classification performance. Benchmarks include Pairwise-MNIST MNIST/Fashion-MNN (Xiao et al. 2017) and SVHN/merementall-ee -iffrrr Netzer et al. 2011 Krizhevsky et al. 2009). Relevant background is introduced in section Section details the hash-routing algorithm and discusses its key attributes. Section compares our work with other continual learning and dynamic network studies. A large set of experiments is carried out in section

---

Under review as a conference paper at ICLR 2021
2 FEATURE HASHING BACKGROUND
Feature hashing, also known as the hashing trick (Weinberger et al. 2009) is dimension reduction transformation with key properties for work: inner product conservation and quasi-orthogonality. feature hashing function IR + R,, can be built using two uniform hash functions h N f1,2..., s) and & f-1, 1), , such:
dx(x) Σ E()x,
]E[L.N] J:h())=
where o; denotes the 1t component of Φ. Inner product is preserved as E(ø(a) ((b) -aab provides an unbiased estimator of the inner product. It can also be shown that if ||a ||b|2 = 1, then n. = (g)).
Two different hash functions & and ø' (e.g. h + h { E') are orthogonal. In other words, v(v, w) E (#) x (m(("), E[v" w] Rs Furthermore, Weinberger al. (2009) details the inner product bounds, given Im(o) and x RN::
2/2
Pr([ (x()) e) S 2exp
(I)
x + l |x| c/3
Eq shows that approximate orthogonality is better when ø' handles bounded vectors. Data inde- pendent bounds can be obtained by setting ||x| = and replacing by livila"
which leads to
||x|| 2 S N and livll I 1, hence:
2/2
(-2212
2/2 (- N/8 + e/3
Pr([v" d'(x) <) < 2exp
S 2xpp
(2)
x
+ V|v
c/3
Better approximate orthogonality significantly reduces correlation when summing feature vectors generated by different hashing functions, as is done in hash-routed networks.
3
HASH-ROUTED NETWORKS
3.1 STRUCTURE
A hash-routed network maps input data feature vector of size that is stable across successive learning tasks. An HRN exploits inner product preservation to insure that similarity between gen- erated feature vectors reflect the similarity between input samples. Quasi-orthogonality of different feature hashing functions is used to reduce correlation between the output's components, as it is the sum of individual hashed feature vectors. An HRN H is composed of M units {U1 ..., ..., Each unit Ug is composed of:
A series of convolution operations fa. It is characterized by number of input channels and number of output channels, resulting in vector of trainable parameters Wą. Note that f& can also include pooling operations.
An orthonormal projection basis B It contains maximum of m non-zeros orthogonal vectors of size s. Each basis is filled with zero vectors first. These will be replaced by non-zero vectors during training.
A feature hashing function de that maps feature vector of any size a vector of size s.
The network also has an independent feature hashing function do. All the feature hashing functions are different but generate feature vectors of size s.
3.2 OPERATION
3.2.1 HASH-ROUTING ALGORITHM
H. maps an input sample x to a feature vector H(x) of size s. In a vanilla CNN, x would go through series of deterministic convolutional layers generate feature maps of growing size. In HRN,
2

---

Under review as a conference paper at ICLR 2021
U1
U1
U1
U2
U2
D2
90
d3
Ps
U3
Ds
Po
flattened feature map
U4
U4
U4
P2
hash vector
T3
H(x)
Figure 1: A hash-routed network with units and a depth In this example, Ug is selected first as the hashed flattened image has the highest projection (Pu) magnitude onto its basis. The structured image passes through the unit's convolution filters, generating the feature map in the middle. This process is repeated twice whilst disregarding used units at each level. The final output is the sum of all projection residues. Best viewed in color.
the convolutional layers that will be involved will vary depending intermediate results. Feature hashing is used to route operations. Since feature hashing preserves the inner product in the hashed features space, similar samples will be processed similarly. Intermediate features are hashed and projected upon the units' projection bases. The umit where the projection's 's magnitude is the highest selected for the next operation. Operations continue until maximum depth d reached (i.e. there is a limit of d 1 chained operations), or when the projection residue is below given threshold тa- H(x)) is the sum of all residues.
Let (U,,,,,,, Mu.: L..., be the ordered set of units involved in processing (assuming the final pro- jection residue's magnitude is greater than Ta). Operation simply involves hashing the (flattened) input sample using Φo. Let Zi, = fig fix-1 operation The normalized hashed features vector after operation & is computed as such:
fi, (x) be the intermediate features obtained at
Φix
Fig
h
(3)
Dig
Die
||arey
For operation 0. h, is computed using and Φo- Pi. Bigzi hü, and ri, h,, Pi, are the projection vector and residue vector over basis B resp. As explained earlier, this means that:
is+1 arg max
|Byba
(4)
JEZ fi
where is the subset of initialized units (i.e. units with bases containing least one non-zero vector). Finally,
H(x)
Σ r
(5)
jE (i
The full inference algorithm is summarized in Algorithm aad an example is given in Figure] [
3.2.2 ANALYSIS
The output a typical CNN feature map with a dimension that depends on the number of output channels used in each convolutional layer. In HRN, this would lead to variable dimension output as the final feature map depends on the routing. variable dimension feature maps would be impractical. Feature hashing circumvents these problems by generating feature vectors of fixed dimension. Similar feature maps get be processed by the same units, consequence of using feature hashing
continual learning setup, dealing with
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Hash-routed inference
Input: x
Output: H = H(x)
hg (r)x); J
H f- 0; h ho:
for j = -..... do
= arg max |
// select the best unit
r h h+,hB,,h;
// compute new residue
H - H +r;
// accunulate residue for output
JU
// update set of used units
if | 12 Ta then
break:
// stop processing when residue too low
else
V
fú (
// compute feature map
h
+
// new hash vector using flattened feature map
end
end
for routing. In this context, similarity is measured by the inner product flattened feature maps, projected onto different orthogonal subspaces (each unit basis span). Another consequence is that unit weights become specialized in processing certain type of features, rather than having to adapt to task specific features. This provides the kind of stability needed for continual learning.
For given unit U4, rank S m << 8. Hence, is reasonable to consider that the orthogonal subspace's contribution to total variance is much more important than that of Bg. This is why H(xr)
only contains projection residues. Note that in Eq h;g E Im( and | = 1. The operand under di, has infinite norm of 1, which under Eq leads to inner product bounds independent of input data when considering orthogonality.
Moreover, due to the approximate orthogonality of different feature hashing functions, summing the residues will not lead to much information loss as each residue vector r,, is in (m(-),,,) ) but this also explains why each unit can only be selected once. The residues' é -norms are added to the loss function to induce sparsity. Denoting Lr the specific loss for task T (e.g. KL-divergence for supervised classification), the final loss L is:
L Cr
Σ ||51112
JE (60
3.3
ONLINE BASIS EXPANSION AND UPDATE
The following paragraphs explain a unit's evolution during training. The described algorithms run requiring no external action.
each time unit is selected in Algorithm |
3.3.1 INIIIALIZATION AND EXPANSION
Units projection bases are at the heart of the hash-routing algorithm. As explained in section 3.2. bases are initially empty and undergo expansion during training. A hash vector (Eq is used to select unit according Eq 4 When all units are still empty, unit picked randomly and its basis is initialized using the hash vector. Let I denote the subset of initialized units. When I & but some units are still empty. units are still selected according to Eq under the condition that the projection's 's magnitude is above minimal threshold Tempty- When Tempty is not surpassed, random unit from the remaining empty units selected instead.
Assuming unit has been selected as the best for given hash vector, its basis can expand when the projection's 's magnitude is below the expansion threshold Tezpand- The normalized projection's 's residue is used as the next basis element. This follows Gram-Schmide orthonormalising process to maintain orthonormal basis for each unit. Each basis has a maximum size of m beyond which cannot expand. The unit selection and expansion algorithms summarized in Appendix

---

Under review as a conference paper at ICLR 2021
3.3.2 UPDATE
Once unit basis is full (i.e. it does not contain any zero vector), it still needs evolve to accommo- date routing needs. As the network trains, hashed features will also change and routing might need adjustment. If nothing is done to update full basis, the network might get "stuck" in a bad config- uration. Network weights would then need to change in order to compensate for improper routing, resulting in a decrease in performance. Nevertheless, bases should not be updated too frequently as this would lead to instability and units would then need to learn to deal with too many routing configurations.
An aging mechanism can be used to stabilize basis update training progresses. Each time unit is selected, a counter is incremented and when it reaches its maximum age, is updated. The maxi- mum age then be increased by means of geometric progression.
Using the aging mechanism, it becomes possible to apply the update process to bases that are not yet full, thus adding more flexibility. Hence, some bases can expand to include new vectors and update existing ones.
Bases can be updated by replacing vectors that lead to routing instability, Each non-zero basis vector Vy has a low projection counter сę. During training, when unit has been selected, the basis vec- tor with the lowest projection magnitude sees its low projection counter incremented. The update algorithm is summarized in Algorithm
Algorithm 2: Unit update
Input:
Current basis (excluding zero-vectors): B (Vi, Vm).
Current low projection counters: (c1,-.., Cm),
Current age: a, Current maximum age: a, Aging rate: p>> 1,
Latest hash vector h
Output: Updated basis if a a then
i = arg max (axx(c,11 };
1/ find basis vector to replace
V; f B B_h
// remove projection on the reduced basis B., (without v.
Vi f- V:
< pa;
// update maximum age // reset age counter
a +- 0:
0;
// reset low projection counter
else
+
// increment current age
i =
aarggmnn | hly,
// find low projection counter to increment
4 +
3.4 TRAINING AND SCALABILITY
HRNs generate feature vectors that can be used for a variety of learning tasks. Given a learning task, optimal network weights can be computed via gradient descent. Feature vectors can be used as input to fully connected network, to match given label distribution in the case of supervised learning. As explained in [ each input sample is processed differently and can lead to different computation graph. Batching still possible and weight updates only apply to units involved in processing batch data. Weight updates is regularized using the residue vector's norm at each unit level. Low magnitude residue vectors have little contribution to the network's output thus their impact training of downstream units should be limited. Denoting L learning task loss function, r the hash vector projection residue over a unit's basis, w the vector of the unit's trainable weights and y a learning rate, regularized weight update of becomes:
(7) An HRN can scale simply by adding extra units. Note that adding units between each learning task is not always necessary to insure optimal performance. In our experiments, units were manually added after some learning tasks but this expansion process could be made automatic. Indeed, or more extra unit(s) could be automatically added whenever all bases have been completely filled. Its architecture could be a copy of existing unit (chosen randomly).
W + w - ? (1, [[rll))V[((
5

---

Under review as a conference paper at ICLR 2021
4 RELATED WORK
Dynamic networks Using handcrafted rigid models has obvious limits in terms of scalability. Tanno et al. (2018) builds binary tree CNN with a routed dataflow. Routing heuristics requires intermediate evaluation on training data. It uses fully connected layers to select branch. [Spring & Shrivastava (0117) builds LSH (Gionis et al. 1999) hash tables of fully connected layer weights to select relevant activations but this does not apply to CNN. Rosenbaum et al. (2017)'s 's algorithm is closer to our setup. Each sample is processed by different blocks until a maximum processing depth is reached. It uses reinforcement learning to train router that selects the best processing block at each level, based on a supervised classification scheme. However, their network is task-aware and blocks at each level cannot be used other levels (unlike HRN units).
Continual learning Parisi et al. (019)) offers thorough review of of-te-of-thh-rrr continual learn- ing techniques and algorithms, insisting on a key trade-off: stability vs plasticity. [omonacc & Maltoni (217) groups continual learning algorithms into categories: regularization, architectural and rehearsal. Kirkpatrick et al. (2017) introduces a regularization technique using the Fisher in- formation matrix to avoid updating important network weights. Zenke et al. (2017) achieves the same goal by measuring weight importance through its contribution to overall loss evolution across given number of updates. |Rannen al. (2017) closer our setup. The authors continuously train an encoder with different decoders for each task while keeping a stable feature map. Knowl- edge distillation (Hinton et al., 2015) is used to avoid significant changes to the generated features between each task. A key limitation of this technique is, as mentioned in Rannen et al. (2017), that the encoder will never evolve beyond its inherent capacity as its architecture is frozen. Serra et al.) 2018) learns attention nearly-binary masks to avoid updating parts of the network when training for a new task. Similarly, Beaulleu et al. (2020) uses a primary model to modulate the update and response of secondary model. In both cases, scalability is again limited by the chosen architecture.
Li & Hoiem (017)) also uses knowledge distillation in a supervised learning setup but systemati- cally enlarges the last layers to handle new classes. Yoon et al. (2017) limits network expansion by enforcing sparsity when training with extra neurons. Useless neurons are then removed. [Xu & Zhu (2018) uses reinforcement learning optimize network expansion but does not fully take advantage of the inherent network capacity as network weights are frozen before each new task.
Lopez-Paz & Ranzato (017);; Rebuffi et al. (2017); Hayes et al. (2019) store data from previous tasks in various ways to be reused during the current task (rehearsal). Shin et al. (2017); van de Ven & Tolias (2018) Rios Itti (2018) make use of generative networks to regenerate data from Kemnker & Kanan (2017) use neuroscience inspired concepts such as phort-term/long-terrs memories and a fear mechanism to selectively store data during learning tasks, whereas we store limited number of hashed feature maps in each unit basis, updated using an aging mechanism.
previous tasks. Kamra et al. 2017 y Parisi et al. [Kener
5 EXPERIMENTS
5.1 SETUP
We test our approach in scenarios of increasing complexity and using semantically different datasets. Supervised classification scenarios involve single HRN that is used across all tasks to generate feature vector that is fed to different classifiers (one classifier per task). Each classifier is trained only during the task at hand, along with the common HRN. Once the HRN has finished training for a given task, test data from previous tasks is re-encoded using the latest version of the HRN. The new feature vectors are fed into the trained (and frozen) classifiers and accuracy for previous tasks is measured once more.
We compare our approach against other algorithms: a vanilla convolutional network (VC) for fea- ture generation with different classifier per task; Elastic Weight Consolidation (EWC) (Kirkpatrick) ct al. 2017). a typical benchmark for continual learning. Elastic weight consolidation is applied only to feature generator that feeds into different classifier per task; Encoder Based Lifelong learning (ELL) (Rannen et al. 2017). involving a common feature generator with a different classifier per task. For fair comparison, we used the same number of epochs per task and the same architecture for classifiers and convolutional layers. For VC, EWC and ELL, the convolutional encoder is equiv- alent to the unit combination in HRN leading to the largest feature map. Feature codes used in EL..

---

Under review as a conference paper at ICLR 2021
autoencoders (see Rannen et al. (2017) for more detail) have the same size as the hashed-featee vectors in HRN. For all experiments, we show the evolution of accuracy for the first task (TO) after each task training. This is clear measure of catastrophic forgetting. We also show the overall accuracy after each task training.
The following scenarios Pairwise-MNIST Each task a binary classification of handwritten digits: 0/1, 2/3, ...etc, for a total of 5 tasks (5 epochs each). In this case, tasks are semantically comparable. A units HRN with depth of 3 was used.
considered (implementation details be found in Appendix C
There are two 10-classes supervised classification tasks, first the Fashion- MNIST dataset, then the MNIST dataset. This 2 tasks scenario with semantically different datasets. units HRN (depth of 3) was used for the first task and units were added for the second task. SVHNiincremental-CT Cini-Cialrl0 00 This is an 11 tasks scenario, where each task is 10-classes super- vised classification. Task (8 epochs) involves the SVHN dataset. Tasks 10 (15 epochs each), involve 10 classes out of the 100 classes available in the Cifar100 dataset (new classes are intro- duced incrementally by groups of 10). All datasets are semantically different, especially task 0 and the others. A HRN of 6 units (depth of 3) was used for the SVHN task and 2 extra units were added before the 10 Cifar 100 tasks series. For this experiment, we provide the accuracy drop for each task between its first training and the final task. This is clear measure of catastrophic forgetting. We also provide the top accuracy score for each task. This measures the network's ability to learn new tasks.
MNIST/Fashion-MMN
VC
EWC
ELL
HRN
ewc
.. H ell
T hm
100%
90%
80%
TO accuracy % Globaa accuracy
70%
60%
50%
40%
30%
20%
10%
TO
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
EWC
HRN
EWC
HRN
hm
Fm
100%
100%
G.oboll accurrcy
90%
Glolbal accuracy %
19.4
TO accurory %
30%
70%
70%
...llbal
30%
TO
60%
50%
50%
40%
T2
T1
T2
T3
10
Figure 2: Task 0 accuracy evaluation after each task (continous lines) and global accuracy score (dotted lines). Top: THHHN/nncreeena--Cifarlllll Bottom-left: Pairwise-MNIST. Bottom-right:
MNIST/Fashion-MNISST. Best viewed in color.
5.2 COMPARATIVE ANALYSIS
Figure 2 shows that HRN maintains a stable performance for the initial task, in comparison to other techniques, even in the most complex scenario. Global accuracy plots show that HRN performance degradation for all tasks is very low (even slightly positive in some cases). Moreover, figure 3 shows that catastrophic forgetting is higher with other algorithms, and almost non-existent with HRN. However, alsso shows that maximum accuracy for each task is often slightly lower in comparison to other algorithms. Indeed, units in HRN are not systematically updated each training step and
7

---

Under review as a conference paper at ICLR 2021
Accuracy drop %
T0
T4
T5
T6
T9
10%
20%
30%
40%
50%
-00%
-70%
Top accuracy %
VC EWC ELL HRNN
100%
90%
80%
70%
60%
40%
30%
20%
10%
Figure 3: SVHNVincremental-Cffarara Top: Final accuracy drop for each task. Bottom: Top
accuracy score for each task. Best viewed in color.
would hence, require a few more epochs to reach top task accuracy as with VC. Furthermore, the trade-off between plasticity and stability can be controlled through the À parameter (see Eq the ablation study has shown (see for more detail). Plasticity can also be increased by adding extra units to an HRN.
5.3 ROUTING AND NETWORK ANALYSIS
After a few epochs, we observe that some units are used more frequently than others. However, we observe significant changes in usage ratios especially when changing tasks and datasets (see Figure This clearly shows the network's 's adaptability when dealing with new data. Moreover, some units almost never used (e.g. Us in Figure 4. This shows that the HRN only uses what it needs and that adding extra units does not necessarily lead to better performance.
4).
90.00
80.00
70.00
U7
60.00 96
50.00 96
40.00
30,00
T0
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
Figure 4: HRN units relative usage ratios for the SVHNViincrementallCiaaa scenario. Units 6 and 7 were added after TO (SVHN). Best viewed in color.
8

---

Under review as a conference paper at ICLR 2021
5.4 HYPERPARAMETERS AND ABLATION
Appendix B details the impact of key hyperparameters. Most importantly, the é 4i-norm constraint over the residue vectors (see Eq 6) plays crucial role in keeping long-term performance but slightly reduces short-term performance. This can be compensated by increasing the number of epochs per task. We have also considered keeping the projection vectors in the output of HRN (the output would be the sum of projection vectors concatenated with the sum of residue vectors) but we saw no significant impact on performance.
CONCLUSION AND FUTURE WORK
We have introduced the use of feature hashing to generate dynamic configurations in modular convo- lutional neural networks. Hash-routed convolutional networks generate stable features that exhibit excellent stability and plasticity across a variety of semantically different datasets. Results show excellent feature generation stability, surpassing typical and comparable continual learning bench- marks. Continual supervised learning using HRN still involves the use of different classifiers, even though compression techniques (such (Chen et al. 2015)) limitation is also a design choice, as it does not limit the use HRN to supervised classification. Future work will explore the use of HRN in unsupervised and reinforcement learning setups.
reduce required memory. This
REFERENCES
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.0955/, 2020.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pp- 2015.
Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search in high dimensions via hashing. In Vidb, volume 99, pp. 518-529, 1999.
Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA), pp- 9769-9776. IEEE, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1003.05503.02333 2015.
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint urXiv:1710. 10368, 2017.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. arXiv preprint 1111:17711..066: 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barkkksk-Baiikkaa,aaaaa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-294 2017.
Vincenzo Lomonaco and Davide Maltoni. Core50: new dataset and benchmark for continuous object recognition. arXiv preprint arXiv: 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.

---

Under review as a conference paper at ICLR 2021
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
German I Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter. Lifelong learning of spatiotempo- ral representations with dual-memory recurrent self-organization. Frontiers in neurorobotics, 12: 78, 2018.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Amal Rannen, Rahaf Aljundi, Matthew Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1320- 1328, 2017.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Laapper. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Amanda Rios and Laurent Itti. Closed-loop gan for continual learning. CoRR, 2018.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint 17 2017.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task, arXiv preprint 1OX.00//00,011233, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Ryan Spring and Anshumali Shrivastava. Scalable and sustainable deep learning via randomized hashing. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 445 454, 2017.
Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, Antonio Criminisi, and Aditya V. Nori. Adaptive neural trees. CoRR, Abs/1807.00699 2018. URL http::/arxiv. wtt.//ii.rrr.eooo abs/
1807 06699
Sebastian Thrun. A lifelong learning perspective for mobile robot control. In Intelligent Robots and Systems, pp. 201-214. Elsevier, 1995.
Gido M van de Ven and Andreas S Tolias. Generative replay with feedback connections general strategy for continual learning. arXiv preprint 1809 2018
Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th annual international con- ference on machine learning, pp. 1113-11 11200, 2009.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark-
ing machine learning algorithms, 2017.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Pro- cessing Systems, pp. 899-908, 2018
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 10...708.0.... 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3987- 3995. JMLR. org, 2017.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A SELECTION AND EXPANSION ALGORITIIMS
Algorithm 3: Unit selection and initialization
Input: Hash vector h.
Initialized units subset I (Io the subset of empty units), Used units subset J
Output: Selected unit 24;
if I = i then
Select random unit 24;
B, (h)
1/ initialize its basis using h
else
i = @ or maxjery, |B,h|| 2 Tempty then
Select unit according to Eq
else
Select random unit U, from the remaining empty units
B, + (h)
// initializo its basis using h
Algorithm 4: Unit expansion
Input: Hash vector h, initialized unit U; B,h
r h p
||p||2 < Tempand and nonzero(B.) m then
B; + (Bi, [
F
roplace a zero vector with the normalized residue

---

Under review as a conference paper at ICLR 2021
HYPERPARAMETERS AND ABLATION EXPERIMENTS
Experiment
Hyperparameters gradU pdate True
Metrics
= 3
TO accuracy = 96.41 %
$ 100
Max task accuracy = 98.87%
Baseline
Terpand 0.01
Global accuracy 95.24%
P
Training time = 59.3 min
À 1.0
TO accuracy 97.3%
Max task accuracy 98.87%
Gradient update
gradUpdate False
Global accuracy 92.55%
Training time 55 min
TV accuracy = 96.41%
Low residue constraint
Max task accuracy = -99.62%
=
Global accuracy 95.24%
Training time = 59.77 min
TO accuracy 99.24%
High residue constraint
Max task accuracy = 74.94%
=
Global accuracy 92.58%
Training time = 59.8 min
TO accuracy 99.05%
Max task accuracy 997.73%
Low aging rate
P 1.2
Global accuracy 93.47%
Training time = 58 min
TO accuracy = 97.45%
Max task accuracy 99.20%
High aging rate
p 10
Global accuracy 89.77%
Training time = 58.8 min
TO accuracy 98.25%
Max task accuracy = 92.81 %
Low basis size
m = 2
Global accuracy 94. 11%
Training time = 56.8 min
TO accuracy 99.10%
Max task accuracy = -3303%
High basis size
m = 5
Global accuracy 92.44%
Training time = 60 min
TV accuracy = 96.83%
Low embedding size
Max task accuracy 86.62%
50
Global accuracy 87.26%
Training time 58 min
TO accuracy = 99.67%
High embedding
Max task accuracy = 99.67%
$ 500
size
Global accuracy 98.50%
Training time = 58. min
TO accuracy 98.06%
Low expansion threshold
Max task accuracy = 98.72%
Texpand = 0.001
Global accuracy 96.33%
Training time = 58.6 min
TO accuracy = 97.45%
High expansion threshold
Max task accuracy 98.63%
Tempand 0.1
Global accuracy 93.15%
Training time 58.6 min
Table 1: Hyperparameters' ranges and observed impact (averaged over 10 runs) over accuracy and training/inference time, based on the Pairwise-MNY scenario.
12

---

Under review as a conference paper at ICLR 2021
C
IMPLEMENTATION DETAILS
This section details the network architectures and hyperparameters that were used for each experi- ment.
Layer
Type
Parameters
Filters: 16, Kernel: 5 X 5
1
Conv 2D
Stride: 3 x 3, Padding: 2 x
BatchNorm22, Activation: LeakyReLU
Hyperparameter
Value
Filters: 32, Kemel: 2 x 2
2
Conv 2D
Stride: 3 3, Padding: 0
batch size
16
BatchNorm22, Activation: LeakyReLU
epochs
5,5,5,5,5
3
Dense
Neurons: 60, Activation: LeakyReLU
Optimizer
Adam
4
DropOut
dropProb: 0.2
learning rate
0.001
5
Dense
Neurons: 10, Activation: LeakyReLU
6
DropOut
dropProb: 0.2
7
Dense
Neurons: 10, Activation: None
Table 2: VC: Pairwise-MNIST network architecture and hyperparameters
Laver
Type
Parameters Filters: 16, Kernel: 5 X 5 Stride: 3 x 3, Padding: 2 x
1
Conv 2D
Hyperparameter
Value
BatchNorm22, Activation: LeakyReLU
batch size
16
Filters: 32, Kernel: 2 x 2
epochs Optimizer
5,5,5,5,5
2
Conv 2D
Stride: 3 Padding: 0
Adam
BatchNorm2D, Activation: LeakyReLU
learning rate
0.001
3
Dense
Neurons: 60, Activation: LeakyReLU
Fisher matrix samples
4
DropOut
dropProb: 0.2
200
5
Dense
Neurons: 10, Activation: LeakyReLU
À
1024
6
DropOut
dropProb: 0.2
7
Dense
Neurons: 10, Activation: None
Table EWC: PaiRwise-MNIST network architecture and hyperparameters
Layer
Type
Parameters Filters: 16, Kerne:: 5 x 5 Stride: 3 x 3, Padding: x
Hyperparameter
Value
batch size
16
Conv 2D
epochs
5,5,5,5,5
BatchNorm221, Activation:
LeakyReL.U
Optimizer
Adam
Filters: 32, Kerne:: x 2
learning rate
0.001
2
Conv 2D
Stride: 3 3, Padding: 0
BatchNorm2D, Activation: LeakyReLU
Codes length
100
3
Dense
Neurons: 60, Activation: LeakyReL.U
Embedding Size
288
4
DropOut
dropProb: 0.2
Temperature
3.0
Stabilization
5
Dense
Neurons: 10, Activation: LeakyRel.U
2
epochs
6
DropOut
dropProb: 0.2
Stabilization
7
Dense
Neurons: 10, Activation: None
learning rate
0.001
Table 4: ELL: Pairwise-MNIST network architecture and hyperparameters
13

---

Under review as a conference paper at ICLR 2021
Units Quantity
Unit Layers
Parameters
Filters: 16. Kernel: 5 x
Hyperparameter
Value
2
Conv 2D
Stride: x 3. Padding: x 2
batch size
16
BatchNorm221
epochs
5,5,5,5,5
Activation: LeakyReLU
Filters: 16, Kernel: 2
extra units
0,0,0.0,0
Stride: 3 x 3, Padding: 0
Optimizer
Adam
2
Conv 2D
BatchNorm2D
learning rate
0.001
Activation: LeakyReLU
Classifier Layer
Type
Parameters
m
300
Neurons: 60
1
Dense
Activation: LeakyReI.U
Ta
0.2
2
DropOut
dropProb: 0.2
Texpond
0.01
Neurons: 10
3
Dense
Activation: LeakyReI.U
1.2
4
DropOut
dropProb: 0.2
1.0
Neurons: 10
5
Dense
Activation: None
Table 5: HRN: Pairwise-MNIST units/classifiers architecture and hyperparameters
Layer
Type
Parameters Filters: 32, Kernel: 5 x 5 Stride: 3 3, Padding: 2
Conv 2D
BatchNorm22, Activation: LeakyReLU
Hyperparameter
Value
Filters: 64, Kernel: x
2
Conv 2D
Stride: x 3. Padding: 0
batch size
16
BatchNorm21), Activation: LeakyRel.U
epochs
20,10
3
Dense
Neurons: 60, Activation: LeakyReLU
Optimizer
Adam
4
DropOut
dropProb: 0.2 Neurons: 10, Activation: LeakyReLU
learning rate
0.001
Dense
6
DropOut
dropProb: 0.2
7
Dense
Neurons: 10, Activation: None
Table 6: VC: Fashion-MNISTTMNIIST network architecture and hyperparameters
Layer
Type
Parameters
Filters: 32, Kernel: x
Conv 2D
Stride: 3 3, Padding: 2 x 2
Hyperparameter
Value
BatchNorm22, Activation: LeakyReLU
batch size
16
Filters: 64, Kernel: 2 x 2
epochs
20,10
2
Conv 2D
Stride: x 3, Padding: 0
Optimizer
Adam
BatchNorm21,, Activation: LeakyReL.
learning rate
0.001
Dense
Neurons: 60, Activation: LeakyReLU
Fisher matrix
4
DropOut
dropProb: 0.2 Neurons: 10, Activation: LeakyReLU
200
samples
5
Dense
400
DropOut
dropProb: 0.2
7
Dense
Neurons: 10, Activation: None
Table EWC: Fachion-MNIIST/MNIST network architecture and hyperparameters
14

---

Under review as a conference paper at ICLR 2021
Layer
Type
Parameters
Hyperparameter
Value
Filters: 32, Kernel: 5 x 5
batch size
16
Conv
Stride: 3 Padding: 2
epochs
20,10
BatchNorm21D, Activation: LeakyRel.U
Optimizer
Adam
Filters: 64, Kernel: x
learning rate
0.001
Conv 2D
Stride: x 3, Padding:
BatchNorm2D, Activation: LeakyReLU
Codes length
300
3
Dense
Neurons: 60, Activation: LeakyRel.U
Embedding Size
576
4
DropOut
dropProb: 0.2
Temperature
3.0
Stabilization
Dense
Neurons: 10, Activation: LeakyReLU
3
epochs
6
DropOut
dropProb: 0.2
Stabilization
Dense
Neurons: 10, Activation: None
learning rate
0.001
Table 8: ELL: F network architecture and hyperparameters
Units
Unit
Parameters
Quantity
Layers
Hyperparameter
Value
Filters: 6, Kernel: 5 x 5
batch size
16
Conv 2D
Stride: 3 x 3, Padding: 2 x
epochs
20,10
Activation: LeakyReLU
Filters: 8, Kernl: 2
extra units
0,2
Conv 2D
Stride: 3 3, Padding: 0
Optimizer
Adam
Activation: LeakyReL.U
learning rate
0.001
Classifier Layer
d
Type
Parameters
m
Neurons: 60 Activation: LeakyReI.U
Dense
300
DropOut
dropProb: 0.2
Ta
0.2
Neurons: 10
Tezpand
0.01
Dense
Activation: LeakyReL.U
4
DropOut
dropProb: 0.2
P
10
Neurons: 10
A
1.0
Dense
Activation: None
Table 9: HRN:
network architecture and hyperparameters
15

---

Under review as a conference paper at ICLR 2021
Layer
Type
Parameters
Filters: 36, Kernel: 3 X 3
Stride: x 2, Padding: x 2
Conv 2D
BatchNorm2D
Activation: LeakyReI.U
Filters: 99, Kernel: 2
Stride: x 1, Padding: x 1
Hyperparameter
Value
2
Conv 2D
BatchNorm221
batch size
32
Activation: LeakyReLU
8,15,15,15
DropOut 2D
dropProb: 0.5
epochs
15,15,15,15
Neurons: 200, BatchNorm
15,15,15
4
Dense
Activation: ReLU
Optimizer
Adam
5
DropOut
dropProb: 0.4
learning rate
0.001
Neurons: 100,, BatchNorm
Dense
Activation: ReLU
DropOut
dropProb: 0.4
Neurons: 100, BatchNorm
8
Dense
Activation: None
9
DropOut
dropProb: 0.2
Table 10: VC: SVHNNfincemennallCCaarr network architecture and hyperparameters
Layer
Type
Parameters Filters: 36, Kernel: 3 x 3
Stride: 2, Padding: 2
Conv 2D
BatchNorm211 Activation: LeakyRel.U
2
DropOut 2D
dropProb: 0.3
Hyperparameter
Value
Filters: 99, Kernel:
batch size
32
Stride: x 1, Padding: x
8,15,15,15
Conv 2D
BatchNorm2D
epochs
15,15
Activation: LeakyReLU
15,15,15
DropOut 2D
dropProb: 0.3
Optimizer
Adam
5
Dense
Neurons: 200, BatchNorm Activation: ReLU
learning rate
0.001
Fisher matrix
6
DropOut
dropProb: 0.5
samples
200
Dense
Neurons: 100, BatchNorm
250
Activation: ReLU
DropOut
dropProb: 0.4
Neurons: 100, BatchNorm
9
Dense
Activation: None
10
DropOut
dropProb: 0.2
Table 11: EWC: SVHNiincremental-Ciiiee network architecture and hyperparameters
16

---

Under review as a conference paper at ICLR 2021
Layer
Type
Parameters
Filters: 36, Kernel: 3 x 3
Conv 2D
Stride: x 2, Padding: x 2
Hyperparameter
Value
BatchNorm22
batch size
16
Activation: LeakyReLU
8,15,15,15
Filters: 99, Kernel: 2 x 2
epochs
15,15,15,15
Stride: x 1, Padding: x 1
Conv 2D
15,15,15
BatchNorm22
Activation: LeakyReLU
Optimizer
Adam
3
DropOut 2D
dropProb: 0.5
learning rate
0.001
Neurons: 200, BatchNorm
Codes length
2800
Dense
Activation: ReLU
Embedding Size
32076
DropOut
dropProb: 0.4
Temperature
3.0
Neurons: 100, BatchNorm
Stabilization
6
Dense
Activation: ReLU
epochs
7
DropOut
dropProb: 0.4
Stabilization
0.001
Neurons: 100, BatchNorm
learning rate
Dense
Activation: None
DropOut
dropProb: 0.2
Table 12: ELL: SVHVVicreemennal-CCfffrr0 network architecture and hyperparameters
Units Quantity
Unit
Parameters
Layers
Filters: 36, Kernel: 3 x 3
Conv 2D
Stride: Padding: 2 Activation: LeakyReLU
DropOut 2D
dropProb: 0.5
Hyperparameter
Value
Filters: 36, Kernel: 3 x 3
batch size
16
Conv 2D
Stride: 1, Padding: 2 2
8,15,15,15
Activation: LeakyReLU
epochs
15,15,15.15
DropOut 2D
dropProb: 0.5
15,15,15
Filters: 12, Kernel: 2 x 2
0,2,0,0
Conv 2D
Stride: Padding: 1
extra units
0,0,0,0
Activation: ReLU
0,0,0
DropOut 2D
dropProb: 0.5
Optimizer
Adam
Filters: 24, Kernel: x 4
learning rate
0.001
Conv 2D
Stride: 2 2, Padding: 1 x
d
3
Activation: LeakyReLU
3
DropOut 2D
dropProb: 0.5
2800
Classifier Layer
Type
Parameters
1.10-5
Neurons: 200
Texpand
0.01
Dense
Activation: ReL.U
5
DropOut
dropProb: 0.4
10
Neurons: 100
3
Dense
À
1.0
Activation: ReLU
DropOut
dropProb: 0.4
Neurons: 100
Dense
Activation: None
DropOut
dropProb: 0.2
Table 13: HRN: SVHNVinceeeenalCCiffatt0 network architecture and hyperparameters
17

---

Under review as a conference paper at ICLR 2021
D RUNTIME PERFORMANCE
This section compares the training and inference run times of each algorithm, considering the SVHNVincremennallCifarr/r experiment. All runs performed on 32 CPUs machine with an Nvidia P100 GPU and 16 GB of RAM. Performance figures have been averaged over 10 runs. Batch size is 16 for all runs and algorithms. Note that one training (resp. test) epoch for SVHN corresponds to 73257 (resp. 26032) samples; training (resp. test) epoch for incremental-Cifarr corresponds 5000 (resp. 1000) samples. HRN typically require more time for training and in-
SVNH epoch
Mncremental-CifarIle
epoch
Training
29s
3s
VC
Inference
6s
1s
Training
29s
3s
EWC
Inference
6s
1s
Training+
39s
34s
(4 epochs
ELL
(9min)
(36s)
Stabilization)
Inference
6s
Is
Training
1mmin44s
46s
HRN
Inference
2min24s
6s
Table 14: Training and inference run times (10 runs average) for each algorithm.
ference compared to classical convolutional neural network. This is due to dynamic routing, as hardly compatible with batch processing and therefore exhibits poor performance GPU. Each batch split into individual samples and each sample is processed differently in the dynamic net- work. A more adequate processing backend would be Many-coore/NerwootOOnnn-- processor or an FPGA.
18


Reviews:
[{"review_id": "a1yc6r29AJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nFor a learning model to learn continuously, it needs to handle new datasets without catastrophic forgetting or requiring the model to grow larger. This paper proposes a hash-routed convolution neural network where a different set of convolution filters are used depending on the data. New convolution filters can be added to the network without increasing the computational cost of the network.\n\nQuestions:\n1) What is the running time performance for hash-routed networks (HRN) during training and inference?\n\n2) How are new classes added to the Softmax layer during training?\n\n3) Couldn't you train a single model on the old and new datasets such that it performs well on both? If so, does the continual learning model train faster than building a data-centric model?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PoNifWam6K", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n### Summary\n \nOk, but not good enough. The authors present Hash-Routed Convolutional Neural Networks (HRNs), intended to enable learning of stable representations for continual learning, i.e. representations that change little for previously-learned tasks as more tasks are learned. The authors benchmark HRNs against several baselines in a number of tasks. Overall, the proposed method does not appear as conceptually and empirically convincing as those of other papers at ICLR and similar conferences. \n\n\n### Reasons for score\n\n- The presented algorithm does not appear as conceptually compelling as existing work on the subject (see below for papers that the authors could have cited and evaluated against, but did not). \n- Overall, the architecture, and in particular the use of hash routing as it is used here, does not seem natural to me. (See explanation below)\n- The results from the evaluation section are mixed, reinforcing my intuition that the proposed architecture is not as suitable as existing solutions. \n- The paper omits information about hyper-parameters, hyper-parameter tuning and choices, making the paper’s results impossible to reproduce. Overall, there is so little information that it is impossible to tell if the experimental evaluation was fair to existing algorithms. \n\n\n### Pros\n\n- The paper addresses an important question: how can we train models in a continual-learning set-up, build representations that are meaningfully shared across successive tasks, and avoid performance degradation on previously learned tasks as new tasks are learned? The authors attempt to solve this problem using feature hashing, which had not previously been used in this particular way before. The proposed architecture is novel. \n\n### Cons\n\n- Conceptually, I find the HRN algorithm presented here somewhat unnatural. The authors write “Similar feature maps get to be processed by the same units, as a consequence of using feature hashing for routing. ” That’s technically true, but since the similarity is measured right in the raw feature space, the similarity will often not be meaningful. For example, a black dog in front of a green background would have little measured pixel-level similarity with a beige dog in front of a blue background, even though conceptually both images contain dogs. I would expect that first computing higher-level representations and then using those to route to an appropriate CNN would be more useful than measuring similarity on raw inputs. I do understand that the HRN inference proceeds through multiple iterations, and that the output of the first iteration becomes an input to the second iteration. Still, I imagine that hashing raw inputs right at the start to decide where to route is  harmful. Note: while Weinberger et al (https://alex.smola.org/papers/2009/Weinbergeretal09.pdf) did hash their input features, but they were using bag of words features, not pixel features. \n- Figure 2 (top) is misleading and cherry-picks data that make the author’s proposed algorithm look better. The graph suggests (green line) that the authors’ HRN algorithm’s performance on Task 0 stays nearly constant at a high level as more tasks are added. However, a closer look at Table 1 (top) indicates that HRN performs significantly worse than baselines on several tasks (see e.g. task T1, T2 and T3), both in max and min accuracy.  \n\n- The paper does not provide (enough/any) information on the precise architectures and hyperparameters that were used for the “Experiments” section. This makes the paper very challenging to reproduce. Even for the HRN algorithm presented in this paper, I could not find what value was used for the maximum basis size parameter “m” was used. I could not find much information on whether the authors selected or tuned the hyper parameters for the algorithms that they benchmarked against in the “Experiments” section. Based on the information provided, I cannot tell if the experiments made were fair to the baselines. \n\n- Some relevant existing work has not been cited and compared to: the authors did not cite the ANML algorithm (https://arxiv.org/pdf/2002.09571.pdf), which has achieved state-of-the-art performance on tasks similar to the one that the papers of the paper reviewed here evaluate on. I would suggest that the authors cite this paper, and benchmark againnnst ANML unless there is a good reason not to do so. \n\n- The authors should also have cited “Routing Networks: Adaptive Selection of Non-Linear Functions for Multi-Task Learning” (ICLR 2018, https://openreview.net/forum?id=ry8dvM-R-) . \n\n- The authors write ' *The output of a typical CNN is a feature map with a dimension that depends on the number of output channels used in each convolutional layer. In a HRN, this would lead to a variable dimension output as the final feature map depends on the routing.* ' Why would there necessarily be a variable-length output dimension? One could fix the output width of multiple CNNs, and sum the outputs element-wise to get a fixed-length representation. \n\n- the authors write “As the network trains, hashed features will also change and routing might need adjustment. If nothing is done to update full basis, the network might get ”stuck” in a bad configuration. ” Would these basis updates not lead to changing output representations, thus breaking one of the key properties that the authors were looking for in this architecture? In fact, in Task T1 Table 1 (top), it appears that the model’s performance degraded quite significantly as tasks were learned. \n\n### Questions during rebuttal period:\n\n- I would be grateful if the authors could address as many of the cons listed above as possible. \n- In Figure 4, on the right, there are arrows going into U_4, but no arrows coming out of U_4. Is this an error? \n \n### Some suggestions:\n\n- There are possessive apostrophes missing here and there, e.g. “units” vs “unit’s”. I suggest that the authors review their use of possessive apostrophes one more time. \n\n- “0 This is a 11 tasks” should be “0 This is an 11 tasks”\n\n- “The following scenarios were considered.” might better be followed by a colon than a period.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EwFr3lgDjKn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "############## Summary ##############\n\nThis submission addresses the question of how to avoid catastrophic forgetting while being scalable and adaptable to multiple tasks. The paper introduces the hash-routed network (HRN), a new routing mechanism for neural networks whereby feature hashing (Weinberger et al.) is used to determine the similarity between data points; similar data points are then routed to the same units. HRNs  work by keeping various convolutional units, each coupled with an orthonormal projection basis. The parameters of the convolutional layers are trained via vanilla back-propagation and the orthonormal basis is created following the Gram Schmidt procedure. An aging mechanism enables updating basis vectors that have become obsolete over time. The proposed method is evaluated in three standard continual learning settings, and is demonstrated to avoid catastrophic forgetting.\n\n############## Strengths ##############\n\n1. The idea to use hashing to determine routing paths is novel and could be interesting to the general deep learning community.\n2. This is to my knowledge the first mechanism that enables learning routing networks for lifelong learning, which is certainly relevant.\n3. Code is included for reproducibility.\n\n############## Weaknesses ##############\n\n1. The obtained results show that, while catastrophic forgetting is avoided, this comes at the cost of a model that is too inflexible to handle new tasks.\n2. The proposed algorithm hinges too heavily on heuristics, including counters, thresholds, and manually deciding when new units should be added. While this is not necessarily wrong, it would require substantial experimentation and validation that each component is necessary to achieving the desired behavior.\n3. The paper is hard to follow and so it makes it tough to assess exactly how the different parts of the proposed method are implemented.\n\n############## Recommendation ##############\n\nUnfortunately, at this time I have to recommend this paper for rejection. I think a future iteration of this work could be interesting to the community if either much of the heuristics were replaced by more well-founded techniques, or each of the heuristics was appropriately analyzed in terms of its empirical effect. Moreover, the paper should provide other views of the obtained results, including final overall accuracy, which is omitted.\n\n############## Arguments ##############\n\nThe key idea studied in this work is how to leverage hashing to decide different routes in a routing net in a continual learning setting. I believe this to be a powerful and interesting idea, but I am of the opinion that the proposed approach uses too many heuristic tricks, and therefore makes it hard to assess the benefits of the key contribution. In particular, the method uses a threshold (1) on the projection residual magnitude to decide when to stop chaining, a threshold (2) on the projection magnitude to decide when to pick a random, un-initialized unit, another threshold (3) on the projection magnitude to decide when to add a new basis vector to the chosen unit, and a final threshold (4) on the age of a basis to choose when to update it. While it is possible that such a heuristic method is the right way to go, it warrants a much more comprehensive evaluation that enables us to understand what the contribution of each of the aspects is to overall performance.\n\nOn the other hand, it seems that the obtained results are not very strong, and so I didn't find enough evidence in the paper to convince me that the proposed method is appropriate for continual learning. In particular, it seems that the cost of avoiding forgetting is to almost entirely prevent the model from adapting to new tasks (e.g., Table 1-top). It would be much easier to evaluate these results if the authors provided other views into their results. For example, the authors could show accuracy evaluation of all tasks throughout the training process, average final accuracy of all tasks, average forgetting across all tasks. \n\nThe other major point I have is the lack of clarity in the paper. Section 2 is supposed to provide background for feature hashing. Ideally, the reader should finish reading this section and come out with an understanding of what the point of hashing is in general, how it's computed, and how it will be useful for the provided approach. Instead, this section gives a few mathematical properties of feature hashing, with no intuition or high-level description of it or how it will be used. It seems like the rest of the paper similarly doesn't really give an intuition for why hashing might be useful, other than the fact that it gives a constant-size dimensionality to the hashed features.  The manuscript places emphasis on the fact that inner product is maintained under hashing, so I suppose the point is that distance metrics can be computed in the hashed space, which is used to compute similarity. I believe this fact is never explicitly stated, even though it is a key point of the proposed approach.\n\nI'm also surprised that the paper did not provide a comparison to stronger lifelong baselines, like those based on experience replay, which have been shown to vastly outperform regularization-based methods in recent work (e.g., Lopez-Paz & Ranzato). \n\n############## Additional feedback ##############\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\nAbstract\n- The abstract mentions that the method is potentially useful for unsupervised or reinforcement learning unlike prior approaches, but this is never tested. Why should the reader believe this to be the case?\n\nIntro\n- Some paragraphs are spaced and some aren't. The rest of the paper has un-spaced paragraphs only. Please be sure to use consistent formatting.\n- What are \"stable features\", and how are they useful for unsupervised learning? The manuscript states that the proposed approach would find such stable features, but I didn't see any evidence of this in the evaluation.\n- The paper initially says that the method will be trained only with gradient descent, but then it mentions regularization techniques. Would it be fair to say that EWC is also trained only with gradient descent, since it only adds a regularization term and then uses the penalized objective for gradient descent?\n- If the three benchmarks are MNIST, Fashion/MNIST, and SVHN/incremental-Cifar100, then the text shouldn't say that benchmarks \"include\" those three, but rather that those three \"are\" the benchmarks used.\n\nSec 2\n- I recommend using `` '' for quotation marks\n- This section focuses on providing some theoretical properties of feature hashing, but no good intuition about what it's doing, how it's computed, or how it will be useful for the approach.\n    - What's N in the summation? In Weinberger et al., there's no N. It seems to be j=1^N where h(j)=i.\n    - Need to define \\sigma.\n\nSec 3.1\n- Elements of the proposed architecture are introduced, but we're still given no intuition of what they'll be useful for. This section should guide the reader by explaining from the beginning what the purpose of each piece of the architecture will be.\n\nSec 3.2.1\n- Why are feature maps of vanilla CNNs growing in size?\n- What is the i index? It seems to be the selected unit. Oddly, in Sec 3.1 k was the selected unit, but here it seems that k is the operation number in the chain of operations.\n- Does the set subtraction in Eq. 4 mean that the units used so far cannot be used for the subsequent operations? This should be stated in text.\n- Why is there no mention of how the bases are initialized yet?\n\nSec 3.2.2\n- Where is there an inner product between hashes computed for similarity? Is this the inner product with the basis? We can't know that at this point, since we have no information about how the bases are created.\n- Overall, this section could clarify a lot of things like why is orthogonality important in this context, why the residuals are chosen. It attempts to do so, but leaves me somewhat confused still.\n    - This statement: \"the orthogonal subspace’s contribution to total variance is much more important than that of B_k\" could be explained in more detail. Does this mean that, since hashes are similar, there is not much variance across projections and so the residual contains more information? \n- Why are residuals encouraged to be sparse?\n\nSec 3.3.1\n- basis --> bases (plurals)\n- Using two separate thresholds on the projection magnitude seems to introduce too many hyperparameters. It seems odd to use projection magnitude simultaneously for 1) choosing the current unit, 2) choosing whether to initialize a new unit, and 3) choosing when to expand a unit's basis projection. Why do this?\n\nSec 3.4\n- Scalability depends on manually adding units. Would it be possible to come up with an automatic way to do this? This section states that it should be possible, but I think a significant contribution of this work could come from designing a technique for it.\n- This manual addition seems to go against the stated goal of keeping data scientist involvement to a minimum.\n\nSec 5\n- Is EWC also allowed a task-specific classifier? A fair evaluation would give EWC a task-specific classifier like other methods get.\n- The description of the experimental setup is fairly complete, including details about when units were added, and which data sets were \"semantically different\"\n- Table 1 uses \",\" instead of \".\" for decimals. The \"%\" sign is missing.\n- The fact that catastrophic forgetting is avoided at the cost of lower overall performance suggests that the stability-plasticity trade-off was not well chosen: there is too much stability at the cost of very little flexibility.\n- We don't get to see average performance to accurately assess performance in Table 1.\n    - Manually computing it for SVHN/CIFAR-100 gives 55.46 (HRN) - 58.93 (ELL).\n- How were parameters of baselines chosen? It's odd that EWC neither retains performance on earlier tasks nor adapts to new tasks. This suggests a very poor choice of \\lambda.\n- It's nice to see unit usage plots, but these don't really show substantial differences in usage across tasks: all bars except the first look about the same.\n- The hyperparameters and ablation evaluation does not seem to add much value. Here, I would like to see a much more comprehensive ablation study that goes in depth into the effect of the different parts of the proposed algorithm.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "D_HrFrGCAQX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper studies the problem of continual learning and proposes a new learning framework named hash-routed convolutional neural networks (HRN). HRN has a set of convolutional units and hashes similar data to the same unit. With this design, the paper claims three key contributions. (1) HRN provides excellent plasticity and more stable features. (2) HRN achieves excellent performance on a variety of benchmarks. (3) HRN can be used for unsupervised or reinforcement learning. \n\n\n##########################################################################\n\nReasons for score: \n\nOverall, I like the scope of this paper that studies continual learning. The experiments in Figure 2 verify that the proposed HRN outperforms a number of baselines on incremental-Cifar100 dataset. However, I am not sure whether the dataset makes sense for benchmarking continual learning. It seems the authors split the 100 classes into 10 groups, with each group having 10 classes. The 10 groups and corresponding labels serve as 10 distinct tasks for evaluating continual learning. Such formulation is odd that leads to decreasing accuracy scores as shown in Figure 2. I cannot find a real-world application that can benefit from such formulation due to decreased accuracy scores. It will be good if the paper can clarity the above points to strengthen its motivation. \n\n##########################################################################\n\nPros:\n\n1. The paper proposes a novel neural network HRN for continual learning. HRN leverages multiple units of CNN and hashes similar data to the same unit for training.  \n\n2. The proposed HRN achieves impressive performance when compared with a selected set of baselines. HRN consistently shows more robust accuracy scores on three datasets, as shown in Figure 2 and Table 1. \n\n\n##########################################################################\n\nCons: \n\nThe most important point relates to the motivation of problem formulation. The paper splits a 100-classes dataset into 10 10-classes datasets. Any two datasets have no overlapping in the classes. Under such formulation, a model suffers from decreased accuracy score as shown in Figure 2. It is necessary to explain why decreased scores are appealing in practice given they are the base of the concerned continual learning. \n \nAccording to Figure 2 and Table 1, all models suffer from decreased accuracy scores. Besides, the accuracy scores decrease monotonously regarding the task ids. Task id should not be a factor of performance. It will be good if the paper can provide some explanations. All existing machine learning researchers seek better accuracy scores so the decreased accuracy scores seem unusual. \n \nIt may not be fair to compare baselines with HRN that exhibits a larger model size. HRN leverages multiple (e.g. 6 in figure 2) units of CNN. To be fair, the paper should use the same number of units (or equivalently ensembles) for a baseline method. It will be good if the paper can run some experiments or show a comparison on model sizes. \n\n\n##########################################################################\n\nQuestions during rebuttal: \n\nIt will be nice if the authors can add experiments or discussions related to decreased accuracy scores. It will be perfect if the authors can show real applications of the proposed continual learning with decreased accuracy scores. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers raised several concerns about the paper guided by unfounded heuristics as well as the artificiality of the tasks involved.  Rebuttal only answered a few of them and did not convince the reviewers which has been clearly stated in the response. We hope that the authors will improve the paper for future submission based on the reviews. 
Decision: Reject