Neural Routing by Memory
Authors: Kaipeng Zhang
Zhenqiang Li
Zhifeng Li
Wei Liu
Yoichi Sato
Venue: NeurIPS 2021 Poster (2021)
Keywords: convolutional neural networks
routing
memory

Abstract: Recent Convolutional Neural Networks (CNNs) have achieved significant success by stacking multiple convolutional blocks, named procedures in this paper, to extract semantic features. However, they use the same procedure sequence for all inputs, regardless of the intermediate features.
This paper proffers a simple yet effective idea of constructing parallel procedures and assigning similar intermediate features to the same specialized procedures in a divide-and-conquer fashion. It relieves each procedure's learning difficulty and thus leads to superior performance. 
Specifically, we propose a routing-by-memory mechanism for existing CNN architectures. In each stage of the network, we introduce parallel Procedural Units (PUs). A PU consists of a memory head and a procedure. The memory head maintains a summary of a type of features. For an intermediate feature, we search its closest memory and forward it to the corresponding procedure in both training and testing. In this way, different procedures are tailored to different features and therefore tackle them better.
Networks with the proposed mechanism can be trained efficiently using a four-step training strategy. Experimental results show that our method improves VGGNet, ResNet, and EfficientNet's accuracies on Tiny ImageNet, ImageNet, and CIFAR-100 benchmarks with a negligible extra computational cost.

Neural Routing by Memory
Kaipeng Zhang!? 1.2
Zhenqiang Li
Zhifeng Li
Wei Liu
Yoichi Sato
'Intiitute of Industrial Science, The University of Tokyo Tennent Data Platform
(kpzhang, liq] taa.suu-tton u-tokyo.ac. ac. michaelefli@tencent com v122238columbia, edu
yeato@iis.u-tokee ac.
Abstract
Recent Convolutional Neural Networks (CNNs) have achieved significant success by stacking multiple convolutional blocks, named procedures in this paper, to extract semantic features. However, they use the same procedure sequence for all inputs, regardless of the intermediate features. This paper proffers simple yet effective idea constructing parallel procedures and assigning similar intermedi- ate features to the same specialized procedures in divide-and-conque It relieves each procedure's learning difficulty and thus leads to superior perfor- mance. Specifically, we propose routing- routing-by-memon by-memory mechanism for existing CNN architectures. In each stage the network, we introduce parallel Procedural Units (PUs). A PU consists of memory head and procedure. The memory head maintains a summary of a type of features. For an intermediate feature, we search its closest memory and forward it to the corresponding procedure in both training and testing. In this way, different procedures are tailored to different features and therefore tackle them better. Networks with the proposed mechanism can be trained efficiently using four-step training strategy. Experimental results show that our method improves VGGNet, ResNet, and EfficientNet's accuracies on Tiny ImageNet, ImageNet, and CIFAR-100 benchmarks with negligible extra computational cost.
fashion.
Introduction
Human memory is often understood informational processing system. It plays essential role in humman intelligence and comprises short-term memory and long-term memory, inspiring many well-known machine learning models, such Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) [14, and Neural Turing Machine (NTM) 11.. Episodic memories, a type of long- term memory, are the collection of past personal experiences. They be retrieved and exploited by the brain when tackling problems that have been encountered before. Different memories activate different neurons the brain, directing us to perform specific procedures that have done before. Inspired by this observation, we introduce the routing-by-memon mechanism to the neural network. uses memory (a summary of seen features) to guide networks process different features with different procedures in a divide--ndd-ongeeee better performance. This paper applies the mechanism to the feature extraction in CNNs and refers to networks employing it as Routing-by -y-mmoorry Networks (RMNs). In the following, we will introduce conventional CNNs and our RMN.
manner, casing the difficulty of learning and achieving
Recent Convolutional Neural Networks (CNNs) achieve state-of-the-an results on computer vision tasks by stacking convolutional blocks (e.g., residual block (13), inception block (33], and Dense block (17), named procedures in this paper, to extract semantic features. However, they use the same
Co-corresponding authors
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Feature ft
Procedure
Feature
fi+1
(a) Typical feedforward neural network
Procedural Unit
Memory 7,,0
Procedure
3,00g
Feature f
R(fa. m))
Memory 1,1
Procedure 11.
Feature fi+1
Memory 1,
Procedure 12.
(b) Routing by memory network
Figure 1: (a) Conventional feed-forwaid network. It processes the features by stacked procedures. (b) Our proposed Routing by Memory Network (RMN). It employs parallel procedures (i.e., multi-branch network) and processes features in a divide-and-conquel manner. Different procedures are tailored to different types of features. is routing function, and we use the nearest neighbor algorithm for it. m, P, and & denote the memory, procedure architecture, and the parameters of the procedure, respectively. Memory indicates which type of features the corresponding procedure can handle. Given feature that a previous stage yields, it searches its most similar memory and forwards the feature to the corresponding procedure. For example, T0 1,0, 11.1 and m. represent scenery, food, and animal, respectively. Given lion's features, R will forward them to the third procedure.
stacked blocks to process all intermediate features, despite large variances of those features. Since similar intermediate features can be processed in the same way, propose simple yet effective idea of introducing parallel procedures (i.e., multi-branch CNN) and assigning similar intermediate features the same specialized procedures (i.e., the same branch). is a divide-and-conque structure, sceing Figure [ for an illustration. In this way, we can improve the model capacity and performance while not increasing the computational cost since forward an intermediate feature to only one procedure.
In RMN, introduce simple yet effective mechanism, routing by memory. Briefly, introduce the Procedural Unit (PU) to process features (see Figure and Figure for illustrations). It consists of memory summary/tepresentaive feature) with a procedure (some convolutional blocks). We use the memory to identify which type of features the corresponding procedure is expected to handle. Specifically, we split the network into different stages by downsampling layers. There are multiple PUs in each stage. All procedures within a stage use the same architecture but different parameters. In stage, given an intermediate feature produced from the previous stage, we will search its nearest memory and forward it to the corresponding procedure. In this way, different PUs are specialized to handle different types of features. Besides, the procedures, we introduce a routing -ependeent Squeeze-and -aeee-nnn---teeeee (SE)-like feature attention module, dubbed Conditional Attention (CA), to improve the performance further.
How to initialize and update memory and procedures is the main challenge when training our RMN, We propose an nos-lto-mpleement training strategy that includes four training steps: stem network training, procedure cloning, memory initialization, and routing-based training. In the stem network training step, we train a conventional CNN (c.g.. ResNet and EfficentNet) as a stem network. Then in the procedure cloning step, we generate multiple procedures (i.c., multi--banch CNN) in each stage of the network by cloning all learned procedures preset number of times. In the memory initialization step, we first extract intermediate features for all stages of the network. Then we use representative features as initialized memory. This paper clusters features each stage of the network and uses cluster centers as representative features. Finally, we continue the network training in the routing-basee training step, which similar to the first step, but the data flow inside the network is decided by routing results. In this step. the memory is updated in moving average fashion while other components are updated by the original optimization method used in the first stage. The overall training strategy is plug-and and--lad -lay to existing CNNs' training strategics.

---

This paper takes VGGNet. ResNet, and EfficientNet as backbones to train our proposed RMN. According to the experimental results, RMN significantly improves original models' results on some benchmarks while not increasing the computational cost. We summarize our contributions as follows:
We propose a novel divide-and-conquee mechanism called routing by memory, which constructs multi-branch CNN and assigns similar features to the same specialized branch. It can lead to better performance while not increasing the computational cost.
The proposed mechanism is plug-and-play to existing CNN architectures by virtue of proposed effective four-step training strategy.
We apply our mechanism to VGGNet, ResNet, and EfficientNet and achieve significant improvements in the accuracies on Tiny ImageNet, Imagenet, and CIFAR-100 benchmarks.
2 Related Work
2.1 Convolutional Neural Network Architecture
In 2012, AlexNet 23 outperforms handcrafted features engineering by a large margin in the ImageNet image classification competition. It combines feature extraction and classification in a deep CNN model to extract discriminative semantic features. After that, CNNs gradually dominate most computer vision tasks, such as image classification, object detection, and semantic segmentation. Classic CNN architecture (e.g., AlexNet 23 ZFNet [41], and VGGNet (32)) extract features by stacking convolutional layers. In 2016, ResNet 13], the most successful CNN architecture recent years, proposed to train very deep CNNs using skip connections and stacked residual blocks. After that, more and more efficient and effective stacked block-based CNNs (c.g., MobileNet[15], SENet [16), and EfficientNet (34) were proposed. However, they use the same block sequence for all inputs in feature extraction, regardless of the intermediate features. This paper introduces the plug-and-play routing-by-i seetinng-y-mennee memoor mechanism existing CNN architectures. It uses memory to guide different blocks to tackle different features in a divide-and -nnc-uue fashion, which boosts the accuracy while not increasing the computational cost.
2.2 Conditional Computation
Conditional computation 33 refers to activating only some of the modules in a network in an input- dependent fashion. Recent researches have introduced it to CNNs in order to accelerate network inference. AIG (36,, BlockDrop (39], and SkipNet (37] propose to learn the subset of blocks needed
to process given input. Since easy examples may not require deep layers' features to make the classification, SACT [9], Inside Cascaded CNN, 44 and Dynamic Routing (26) propose to do input-dependent early stopping at the stage of network inference. Routing Convolutional Network (RCN) (20) introduces a routing network that can perform anytime recognition. These methods aim to reduce the models' computational costs while maintaining comparable accuracies.
2.3 Mixture of Experts
Mixture of Experts (MoE) (19) 21 was proposed three decades ago. It is related to conditional
computation and introduces multiple experts (leamnes) divide the problem space into homogeneous regions. Given an input, MoE often uses a routing module to select the corresponding experts in a
hard soft selection fashion. In the deep learning era, sparsely- gatsedd mixture-of-expprts layer [i first introduces MoE to LSTM for language modeling. Lately, Switch Transformer (8] introduces
MoE to Transformer [35] to train a trillion parameter model and achieves great success in natural language processing, As for computer vision, there are two categories of MOE, dynamic parameter and dynamic architecture, which are compatible. Dynamic parameter methods 440
[ 6 learn
input-dependent convolutional kernels' parameters using parameters generation networks. Dynamic architecture methods employ multiple sub-networks and build dynamic computational graph by decision networks. NOEF [1] and HydraNets [27] divide the class label space and assign different sets of class labels to different sub-networks for handling. DeepMoE (38] proposes to increase the kernels and select each layer's features' channels by a decision network. Runtime Routing (28) proposes to learn an RNN-based decision network by reinforcement learning for multi-path networks. Akin to the Runtime Routing, DRNet 4 uses small CNN the decision network. Dynamic Routing (25).
3

---

ResNet-18
Lonv' & Pool
Block 2
Block
Block x
Block
Catt
Pr.0,
55,
2..9
ReoNet- 18- based RMN
Conv & Pool
mys
Block
mic
Block
Wa Block
CA
Pa. Ago
CA
CA
Py.
CA
Maa
Block
m2
f
Sy.
Stage
Stage
Stage
Stage
Feature extraction
Feature processing
Classification
memory stage
blocks (ie, (1l)
Conditional (CA) module
Block m
Procedure
Procedure Unit (PU)
Figure 2: An illustration of Re and ResNet- 18-based 2-way RMN. We divide the network into three phases: (preliminary) feature extraction, feature processing, and classification. In the feature processing phase, split the network into four stages by downsampling layers. RMN introduces the routing-by-mente mechanism to each stage. It uses memory to guide forwarding different features their corresponding specialized procedures. Different colors denote different stages. Different shades of color represent different kinds of features, memories, and procedures within stage.
Routing Networks [29]. ExperGate [2], and HardMoE [12 extend MoE to semantic segmentation, multi-task learning,
learning, and lifelong learning. respectively. These decision networks increase the training difficulty and bring extra computations. Besides, some of them require modifying relevant gradient descent algorithms employing reinforcement learning to train the decision networks.
Our RMN solves routing in perspective, routing by memory, rather than designing complicated decision networks to decide the routing path. is more straightforward and elegant. Besides, since the routing is based non-parametric nearest memory search, RMN is lightweight, easy train, and requires no modification in the back-propagation rule.
Routing by Memory Network
This section elaborates the architecture and training strategy of our proposed RMN.
3.1 Network Architecture
Our RMN is built by integrating the routing-by by-memory mechanism into an existing CNN architecture. In this paper, describe methods with the task of image classification. According to the functions, we treat the processing a common CNN as three separated phases: (preliminary) feature extraction, feature processing, and final classification. Our RMN enhances the intermediate features the feature processing phase by incorporating the suuting-by-meoooe mechanism. In the following sections, take ResNet- 18 [13 routing-by boagg--b-mmmrry mechanism into a conventional CNN(see Figure] 2).
example backbone to present how integrate the
3.1.1 ResNet
In this section, we elaborate on the ResNet-18 18. Given input image X, in the first (preliminary) feature extraction phase, its feature maps fo are generated by a series of operations Po consisting of convolution, pooling, and batch normalization, which can be formulated follows:
fo Po 0n),
where Gg denotes the learnable parameters in P. Low-level features such as edges and corners are represented in these preliminary feature maps.
In the feature processing phase, multiple residual blocks are utilized to process feature map fo for extracting features in higher semantic levels. We group the blocks into different stages according to their output feature maps' dimensions and resolutions (i.e., divide the stages by down-sampling operations). For example, shown in Figure 2 divide blocks into four stages by grouping

---

two adjacent blocks as their feature maps' shapes are the same. We represent the operations (i.e., convolution blocks) in the idh stage as P. with its parameters 6, The feature maps yielded from the 7t stage are ft. Specifically, we formulate the processing of the tth stage as follows:
J = P:(111,,,
== 1,23,44
(2)
The ResNet ends with a global average pooling layer and a 1000-way (the number of classes, 1000 in ImageNet) fully-connected layer with argmax (softmax in the training phase) function. They are used to classify one given image in the classification phase. We formulate this phase as follows:
y
arg max (GAP(fa)W + b),
DE cE(0..1....c-1))
where GAP refers to the global average pooling in the classification phase. W and b are learnable weights and biases of the conneccd layer (i.e., classifier), respectively. e is the class index, and is the total number of classes. y denotes the output class label.
3.1.2 Res RMN
In this section, we take the ResNet-18 as a backbone network to introduce our RMN. In the Resnet- 18-based RMN, we propose Procedural Units (PU) in the feature processing phase. There are multiple PUs in each stage, and we use the term N-way RMN to represent how many PUs per stage in our RMN. All PUs within stage share the same architecture but learn different parameters due the routing mechanism. Each PU consists of two modules, including memory and procedure. The memory is a representative feature that is learned from features by a moving average fashion in the training phase (see Section 3.2 for details). We use global average pooling for memory to reduce the storage cost. In the 2th stage, given the feature fi-1 and memories my = ma,0: Mi,1, NNN we first do the routing R by nearest neighbor searching as follows: arg min (D(ms.j. GAP(Ji-))), R(m;, JE(O.I....NN-11)
(4)
where m denotes the j*h PU's memory in the jth stage. To reduce the storage cost, we apply global average pooling GAP to f before routing. D refers to a distance measurement metric, and we use Euclidean distance in this paper. Since D is non- non-parametric and fast to compute, the computational cost of routing is negligible and can be omitted.
After the routing, we forward the features to the corresponding procedures. A procedure consists of residual blocks and optional Conditional Attention (CA) module. The CA module is optional module introduced to improve the accuracy further by a little additional computational cost. Without the CA module, RMN re-formulates the jth stage of the feature processing phase as follows:
f. 04
i i= 1, 4,
(5)
where 02.0 O,,,,.., - refers to N sets of parameters for P.:
We then introduce the CA module. The same channels of features produced from different procedural units may represent different seman- tic meanings. For example, the first convolution kernel of procedure may focus on animals' fur. However, the first convolution kernel of an- other procedure may focus on furniture's ' texture. The inconsistent semantic meaning of different features increases the learning difficultly. In- spired by the position-coding in ViT 77 we in- troduce the CA module to do routing-dependent channel-wise attention to the features, relieving the inconsistency adaptively. The CA module consists of conditional input, a routing result, and a Squeeze-and ----ann-tttxttinn (SE) module [16]
PiCfi (LRmms
GAP
RRmm, fi-))
ReLU
Sigmoid
Scale
Figure 3: An illustration of Conditional Atten- tion (CA) module. The CA is depended bobeeze-and-Exciiatiin (SE)-like module. (see Figure 3 for an illustration of the CA module). Specifically, it first computes the scalars
8 = [$1, 82, sx] that has the same number of channels as fi = [fr, Sea, Jix] This operation can be formulated as:
= Craeiit(ConcattSSqueee R(m,, f(-1))),
(6)
5

---

where Squeeze denotes using a global average pooling followed with a fully-connected layer to squeeze the feature dimension. Ercite denotes using fully-connected layer and a Sigmoid function to expand the feature dimension and compute attention values. Concal denotes channel-wise concatenation operation. can also use different coding methods for the routing results, such as one-hot coding, which sometimes makes the training more stable. Please note that all CA modules share parameters within a stage.
Then we scale the features fi according to the scalar vector s by channel-wise multiplication (we use to represent it). The CA module can be formulated as follows:
CA(A)) ft-
In summary, when using the CA module, RMN re-formulates the thh stage of the feature processing phase as follows:
St = P: 1))s i= 1,2,3,4.
3.2 Training Strategy
In this section, we take the BeaNet-18--base RMN example to introduce the four-step training strategy. Unlike the conventional training pipeline, after a few training epochs, we insert two steps to expand procedures and initialize memories. In the last step, we update memories in moving average fashion. The training strategy is alug-and-play to existing CNNs' training approach since can follow standard gradient descent algorithms in optimization.
3.2.1 Stem Network Training
In the first step, we aim to train a conventional CNN (c.g., VGG, ResNet, and EfficientNet), named stem network in this paper, extract reasonable features for memory initialization. Specifically, in this step, we train the standard ResNet-18 18 for few epochs and will resume the training in the fourth step. If using CA modules, denotes the discrete uniform distribution.
random routing results ~ U(1, N) for each CA module, where
3.2.2 Procedure Cloning
We clone the procedures in the second step. Specifically, clone the parameters of all procedures by N times (i.e., assign 0; to 1:. In this way, we can build branch procedural units in each stage of the feature processing phase. Please note that we also clone the optimizer state of the gradient in this step to maintain the training's 's consistency in the fourth phase.
3.2.3 Memory Initialization
We initialize the memories using representative features. In this paper, compute the representative features by cluster analysis. Specifically, in the jth stage of the feature processing phase, we extract each training sample's ft. Then we apply the Fuclidean distance-based K-means clustering algorithm to obtain N clusters. Finally, we initialize each memory in m,+1 by each cluster' center. In this way, different memories can dominate different kinds of features.
3.2.4 Routing-Bart -Base Training
In this step. we resume the model training. Inspired by the update of mean value in Batch Normal- ization (BN) [8]. we update the memory in moving-average fashion. Specifically, in training iteration and in the jth feature processing stage, given / samples that are routed to the jth PU, we have their features as fi, .... 1. We update the memory follows:
ΣH GAP(S!)
maj am,, + (1 (1
(9)
H
where a is Syper-parameter and denotes the momentum.
In this step, the computational graph (i.e., data flow) is dynamic due to the routing mechanism. But, the
mechanism uses the non-paramen nonaparaittric nearest neighbor algorithm to select the procedures, and the memory is updated in moving-average fashion. Thus, use the original

---

stochastic gradient descent algorithm with softmax cross-entropy loss function to train our RMN as the original ResNet-18. We also let each routing operation have a 0.05 probability of doing random routing, which can train procedures more stable.
4 Experiments
In this section, will first introduce our experimental setup. Then we present ablation experiments on our proposed main components and syper-parameters, Finally, we show our results on some image classification benchmarks.
4.1 Experimental Setup
In this section, we first introduce the datasets used. Then we introduce the network architectures. Finally, we present the training details of our RMN.
4.1.1 Datasets
In this paper, we take three image classification benchmarks, Tiny ImageNet (4), ImageNet 2012 [30], and CIFAR-100 [22] to evaluate method. ImageNet 2012 consists of 1.2M training images and 50,000 validation images for 1000 classes. Tiny ImageNet is a subset of ImageNet. It consists of 200 classes, and each class has 500 training images and 50 validation images. CIFAR-100 consists of 100 classes, and each class has 500 training images and 100 validation images. We use Tiny ImageNet for ablation experiments in Section 4.2 In Section 4.3 benchmarks.
show results on all three
4.1.2 Network Architectures
Our RMN can be applied to most existing CNNs architectures, and we take some widely-used architectures, i..., VGG, ResNet, and EfficientNet, in experiments. In the ablation experiments, we take ResNet-18 as the backbone network to train our RMN. In the section of evaluations on all benchmarks, present the results of using all architectures. For VGG-16, pooling layers to different stages and build the PUs. We also use batch normalization for VGG- 16. Since the image resolutions of Tiny ImageNet and CIFAR- -000 are 64 64, and 32 32, respectively, we drop the first two down-sampling operations for ResNet and EfficientNet when training on these two benchmarks. For VGGNet, we drop the first down-sampling operations Tiny ImageNet but keep them on CIFAR- 100. For the ImageNet, we use the resolution of 224 x 224.
split the network by
4.1.3 Training Details
Learning rate, We first follow the warmup strategy [00 to increase the learning rate from le-5 to 0.48 in the first five epochs. Then use the cosine learning rate strategy for the rest of the epochs, and we decrease the learning rate to le-5 at the final epoch.
The number of training epochs. For CIFAR-100 110 and Tiny ImageNet, the total numbers of training epochs for VGG-16, ResNet-18, ResNet-50, and EfficientNet-Bo are 120, 120, 200, and 300, respectively. For ImageNet, we use 160, 160, 220, and 400 epochs for them, respectively.
The number PUs. In this paper, N denotes the number of PUs per feature processing stage. Increasing N can improve the model capacity but brings in more parameters. Seeing Section 4.2 for the ablation experiments on N. We set N 8 for the accuracy and cost trade-off in other experiments. Please also note that, compared with memory consumption consume little memory (less than 1% overall consumption when doing inference on ImageNet with batch size 100). Hence, the extra GPU memory consumption by method is negligible. Besides, extra parameters bring an extra storage cost. But storage is abundant in real-world applications since disks are cheap.
feature extraction, the parameters
Batch Size. We use batch size 256 for all baselines. For our RMN, each feature processing stage, there are multiple PUs, and different features from the same batch are fed to different PUs. So, the batch size of each PU is much smaller than the total batch size, and accordingly, we have to enlarge the batch size to train our RMN. Simply, we multiply the original batch size (256) by N/2 though the

---

67
67
Accuracy (%)
66
66
66
65
Accuracy
65
Accuracy
65
64
64
64
63
63
63
62
62
62
61
10 12 14
16
512
1024 1536
2048
61
0.0
0.2
0.4 0.6 0.8
The number of PUs (a)
Batch Size (b)
Momentum in Memory Updating (c)
Figure 4: Experimental results Tiny ImageNet. The blue dashed line denotes the baseline ResNet- 18 with CA modules (without conditional routing inputs). (a) More PUs bring higher accuracy, and the accuracy got converged on 8 PUs, so set it to 8 in other experiments. (b) Since RMN assigns different samples to different PUs, it requires a large batch size to maintain the training stability. We set it 1024 in other experiments. (c) Higher momentum makes the memory updating smoother and more stable and leads to better results.
numbers of data instances among different PUs are imbalanced, which means that we use N 8 and batch size 1024 in this paper. Please see Section 4.2 for the ablation experiment of batch size.
Momentum in Memory Updating. The momentum is hyper-parameter used to make the memory updating smooth and stable. However, too high momentum will make the memory out of date, while too small momentum leads the memory updating unstable. We it to 0.9 in our paper. Please see Section 4.2 for its ablation experiment.
stochastic gradient descent with momentum 0.9 and weight decay le-5 for ResNet and VGGNet. We use the RMSProp optimizer for EfficientNet. We use Synchronized Batch Normalization (SyncBN) supported by the Nvidia APEX library. For CA modules, we use the reduction ratio 16. For data augmentation, we use random augmentation introduced in the ResNet [ paper. We 8x V100 (32GB memory version) with PyTorch.
Others. Regarding other training details,
4.2
Ablation and Hyper-Parameterr Experiments
4.2.1 Number of PUs
The key component in our RMN is the PU. So N, the number of PUs, is a critical hyper-parameter: Increasing the number of PUs will improve the model's capacity and lead higher accuracy. From Figure (1), (1), we can see that the accuracy increased along with larger N and got converged when N 8. However, larger N increases the number of parameters and requires larger batch size (batch = 256 x N/2). We make an accuracy and cost trade-off and set N to 8 in other experiments.
4.2.2 Batch Size
Our RMN introduces multiple branches with different PUs in the network, and the examples are assigned to the different branches. So, RMN requires a larger batch size than conventional CNNs. According the results in (b) too small batch sizes (128 or 256) lead to even lower accuracy than the baseline, which is because the memory updating and routing are unstable. Thus, network training becomes more difficult. Moreover, with the 128 batch size, we found about 40% PUs died after 50 training epochs, which means that these PUs' memories are far from the distribution of the current features, and thus no data can be assigned to these PUs. We consider that the small number of data instances assigned to the PUs will make the memory updating unstable and draw the corresponding memory features to the outliers. Too large batch size (2048) also leads to not good results. Simply, multiply the original batch size (256) by N/2 though the numbers of data instances among different PUs are imbalanced, which means that we set the batch size to 1024 in other experiments.
We also increased the batch size to 1024 baselines and reported the results for fair comparisons. We find that there is clear overfitting using batch size 1024 for the baselines, especially for ResNet-50 and EfficientNet-BH These results are lower than the results of our method. Due the limited pages, please the results our supplemental material.

---

4.2.3 Momentum in Memory Updating
We use the moving average to update the memories. Thus the momentum is key hyper- syper-parameter. Too small a will lead the memories to change quickly. The higher is, the more stable and smoother the memory updating becomes, while too large causes a very slow updating. According to Figure (c), we set a to 0.9 in other experiments.
4.2.4 Conditional Attention
The Conditional Attention (CA) module a SE- like modules to further improve accuracy by in- troducing a little extra cost. According to the results in Table it improves the ResNet-18's accuracy by 2.2% on Tiny ImageNet.
|
#Farmms
RMN RMN (ResNes- 18)
66.5%
89.5M
2.208
Table Ablations of the CA module on Tiny ImageNet.
4.3 Evaluation on Benchmarks
We evaluate methods three image clas- sification benchmarks: Tiny ImageNet, Ima- geNet, and CIFAR-100. According the eval- uation results ( on Tiny ImageNet, our RMN significantly improves the accuracy while not increasing the computational cost. Besides, the improvements for VGGNet and ResNet are much more significant than EfficientNet. It is because EfficientNet's architecture is already well-designed, compact, and contains SE mod- ules. Thus, our method may bring some redun- dancies, but it still improves the accuracy by 1.8%. Sceing from the results (Table on the more challenging benchmark, ImageNet, our method still performs well. Moreover, using our RMN, ResNer-50 can outperform EfficientNet-BD For the CIFAR- 100 benchmark, although the image resolution is small (32 x 32) our method can still gain considerable improvements. Moreover, ResNet- 18pe--88-pp 18-based RMN achieves impressive accuracy and even outperforms the original ResNet-50.
Model
Asc.
*Purams
*FLOPs
VGG-16
17
63, 16%
135.0M
14,10B
RosNes-18
61.78%
2.258
ResNee-50
67.28%
23.9M
5,25B
ElficietNee-B0 8D
67.14%
4.3M
0.478
RMN (VGG-16)
66.57%
237.8M
14.118
RMN (ResNet-18)
66.53%
89.5M
2,268
RMN (ResNes-50)
RMN (FfficionNNss-B0)
0.478
Table Evaluation on InnylmmaeNNet. Ten runs average results (skip when occurring PU death).
Model
Aoc.
4Params
AFLOPs
Model
Acc.
#Parems
AFLOPS
VGG-16
72.93%
138.3M
15.518
VOG-16 52
73.16%
33.8M
0.43B
ResNet-13
13
H
70.72%
11.7M
1.8IB
ResNet- 18
75.62%
11.2M
0.56B
5e
76.06%
ResNet-
77.53%
1,31B
Nol
76.46%
0.308
FfficienNet-B
78.04%
4.2M
0.128
RMN (VGG-116
RMN (VGGG-16
75.74%
RMN (ResNet-18)
73.74%
89.0M
1.828
RMN (ResNe- (8)) 77.94%
88.4M
0.57B
RMN (ResNet.50)
78.31%
190 7M
4.138
RMN (ResNer-S0)
79.06%
184.6M
1,32B
(BffcicenNei-BO)
78.08%
(EfiiceenNee-B0))
78. 88.88%
0.128
Table 3: Evaluation on ImageNet. Three runs average
Table 4: Evaluation on CIFAR-100. Ten average results (skip when occurring PU death).
results (skip when occurring PU death).
4.4 Comparisons with Related Work
RNR 28 and DeepMOR 38
the two related works. Our motivations and methods different, and we discuss and make comparisons in this subsection.
RNR learns RNN-based routing network by reinforcement learning dynamically select experts. trains the routing network and the backbone network alternately. DeepMOE learns a CNN- decision network to activate specific layers in the backbone network dynamically. Unlike these methods, our RMN solves routing in a new perspective, routing by memory, rather than introducing complicated routing networks. It is more straightforward and elegant. Besides, since the routing is based on non- con-parmttriic nearest memory search, RMN is lightweight, easy to train, and requires no modification in the back-propagation nule. Here make the experimental comparisons with RNR.
For RNR, we follow codes to align our baselines' training details and then retrain our RMN on CIFAR-100 Based on ResNet-18 18 backbone, our RMN (N 4, without CA) achieves 78.96%

---

accuracy while costs 14.2ms per image (using Xeon 8255C CPU), which is superior to RNR's (#branch=4) 7esults-78.42% accuracy with 59.8ms.
For DeepMOF, we make comparisons on respective baselines. DeepMOE improves ResNet-50's accuracy ImageNet from 76.15% 77.12%, while RMN improves ResNet-50 from 76.08% to 77.78% (N = 8, without CA). Besides, our RMN is more efficient since does not introduce extra decision networks.
Conclusion
In this paper, we proposed specific mechanism, routing by memory, for conventional feed-forward networks. We integrated with the existing CNN architectures and built the Routing by Memory Network (RMN). Specifically, introduces the Procedural Unit (PU) to the CNNs, which consists of a memory (a representative feature) with procedure (some convolutional blocks). We employed memories to forward different features to their expert PUs. Networks with the proposed mechanism can be trained efficiently using a four-step training strategy. According to the results on Tiny ImageNet, ImageNet, and CIFAR-IO0 100, our RMN significantly improves VGG-16 16, ResNet-18 ResNet-50, and EfficientNet-Bi while not increasing the computational cost.
6 Limitation Analysis
The main limitations of our method are extra parameters, larger training batch sizes, unstable training, and parallel inference efficiency.
First, since our method clones the experts multiple times, introduces extra parameters and consumes additional memory and storage. But, discuss in Section 4.1.3 the model parameters consume little memory. Besides, in many real-world scenarios, the model storage cost negligible. other words, our method trade redundant memory and storage for accuracy improvement.
Second, since our network is multi each training iteration. Thus, our method requires a larger training batch size (4x for 8-way RMN). But V100 (32GB) enough to train most existing CNN models. Besides, some techniques can use larger batch sizes in limited memory, such gradient accumulation, memory-saving CNN training framework (e.g., MXNet), and dynamic memory allocation.
multi-branch, branch, each branch (expert) requires enough training samples in
Third, the model training is unstable since some PUs may die, especially when using not good training parer-parameters. If some PUs die, the accuracy will usually be lower than that of the baseline.
Finally, since the data flow in our method is multi-path, its parallel inference efficiency is lower than that of the baseline when inference batch size is greater than
Last but not least, our rule of assigning similar features to the same expert is mixed blessing. is explainable, non-paraman aanpaaametric, and fast. But compared with parametric routing, our rule may impede the model itself from achieving better results.
Acknowledgments and Disclosure of Funding
This work supported by JST AIP Acceleration Research Grant Number JPMICR20U1 and JSPS KAKENHI Grant Number JP20H04205, Japan. This work is also supported by Japan JSPS Research Fellowship for Young Scientists (DC) with JSPS KAKENHI Grant Number JP2IJ13152.
References
[1] Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network experts for large- scale image categorization. In European Conference on Computer Vision, pages 516-531 Springer, 2016.
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 3366-3375, 2017.
10

---

[3] Yoshua Bengio, Nicholas Léonard. and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
[4] Shaofeng Cai, Yao Shu, and Wei Wang. Dynamic routing networks. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pages 3588-3597, January 2021.
[5] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11030-11039, 2020.
[6] Zhourong Chen, Yang Li, Samy Bengio, and Si. You look twice: Gaternet for dynamic filter selection in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1172-9180, 2019.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[8] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient spansity, arXiv preprint arXiv:2101.03900 2021.
[9] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1039-1048, 1048, 2017.
[10] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint arXiv: 1706.000677777111 2017.
[11] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 1rXiv:1410.5401, 2014.
[12] Sam Gross, Marc' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6865-6873, 2017.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98): 1780, 1997.
[15] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1744v.170..0461,, 2017.
[16] Jie Hu, Li Shen, and Gang Sun. saneece-and-excitatiee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[17] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[18] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[19] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.

---

[20] Zequn Jie. Peng Sun, Xin Li, Jiashi Feng, and Wei Liu. Anytime recognition with routing convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 2019.
[21] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1105, 2012.
[24] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7, 2015.
[25] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang. and Jian Sun. Learning dynamic routing for semantic segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 8553-8561 2020.
[26] Mason McGill and Pietro Perona. Deciding how to decide:
Dynamic routing in artificial neural networks. In International Conference Machine Learnngg, pages 2363-2372. PMLR, 2017.
[27] Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Specialized dynamic architectures for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8080-8089, 2018.
[28] Yongming Rao, Jiwen Lu, Lin, and Jie Zhou. Runtime network routing for efficient image classification. IEEE transactions on pattern analysis and machine intelligence, 41(1))22291- 2304, 2018
[29] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv: 2017.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):2112522 2015.
[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experrs layer. arXiv preprint 1XXIv:7701.06300 2017.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
[33] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9,
2015.
[34] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 4105-6114 PMLR, 2019.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[36] Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-18, 2018.
[37] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 409-424, 2018.
12

---

[38] Xin Wang. Fisher Yu, Lisa Dunlap. Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. Deep mixture of experts via shallow embedding. In Uncertainty in Artificial Intelligence, pages 552-562. PMLR, 2020.
[39] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8817-8826, 2018.
[40] Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. arXiv preprint arXiv: 1904.0497, 2019.
[41] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
[42] Kaipeng Zhang, Zhanpeng Zhang, Hao Wang. Zhifeng Li, Yu Qiao, and Wei Liu. Detecting faces using inside cascaded contextual enn. Proceedings of the IEEE International Conference on Computer Vision, pages 3171-3179, 2017.
13


Reviews:
[{"review_id": "ovHRsZ7xSRA", "reviewer": "Reviewer_vPSw", "summary": "Routing strategy is used to select process units in convolutional neural networks. Memory is one interesting idea, which is used to record the latent information of inputs. Because different samples have different memory footprints, routing-by-memory provides one optimized execution path。", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The work provides an interesting idea named routing-by-memory. The memory guides different blocks tackle different features in a divide-and-conquer fashion.\n2. The experimets show the effectiveness of the method.\n\nWeaknesses:\n1. The meaning of memory in CNNs are poorly explained. Which kinds of information are recorded by the memory? Although some visualization are providied in the supplementary material.\n2. The number of process units for different stages are same. Will the results be imporoved if different stages have different number of PUs?\n3. The proposed method leads  to minor increase of FLOPs and major increase of memory.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The main limitations of our method are extra parameters and larger training batch sizes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hour."}, "extra_text": {}}, {"review_id": "dXx5L9pThKw", "reviewer": "Reviewer_oRgf", "summary": "This paper proposes a new routing method for MoE. The main idea is to construct multi-convolution blocks (called procedure in the paper) and assign similar features to the same blocks. It can improve the accuracy while not increasing the computational cost. Practically, it introduces memory features with the nearest neighbors algorithm in the routing process of MoE. It updates memory by moving average fashion in the training phase.\n\nCompared with existing MoE methods using sub-network in the routing process, the proposed routing mechanism is non-parametric, efficient, and explainable. And it achieves convincing experimental results on computer vision tasks, including image classification, semantic segmentation, and face recognition.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed memory-based routing mechanism is novel and exciting. It offers a new non-parametric expert selection way to MoE. Compared with existing methods, it is more efficient and explainable. Besides, its network is easier to train using a standard gradient descent algorithm.\n2. The idea of assign similar features to the same experts is simple and allows explainable routing results while existing methods generate routing results in a black-box fashion. This paper may open a new line of research, non-parametric routing, into MoE.\n3. The experimental results are convincing. The proposed method can improve the baseline models' results on several image classification benchmarks and downstream tasks (semantic segmentation and face recognition) while not increasing the computational cost. The authors also compare their method with RNR and DeepMoE, two related MoE-based CNN methods, and show superior results.\n4. MoE got great success in the field of NLP in recent months, but few researchers about its effectiveness in training CNN models. This paper provides a good try and may raise many research interests.\n\nWeaknesses:\n1. Due to using multi-experts, a large number of parameters and training batch sizes are two common weaknesses for MoE methods. Though these weaknesses are acceptable for MoE methods, similar to existing methods, this paper still does not provide a practical solution and just claims that these two weaknesses are not critical or solve them using other techniques.\n2. The number of epochs in the first training stage (stem network training) should be an important hyper-parameter. It directly affects the memory initialization. But no experiments about this hyper-parameter in the paper.\n3. The paper properly discusses the relations to existing MoE-based CNNs. However, there are few discussions about MoE-based NLP methods while MoE is more popular in NLP. The author should add some related NLP methods to the section of related works.\n\n\nSuggestions:\n1. Due to the global average pooling, the authors use a vector as the memory. Thus, the proposed method can also be applied to many NLP models (e.g., rnn-based and transformer-based models). The author may have a try to see if it also work well in NLP.\n2. In memory, the authors use global average pooling to reduce costs. But it will erase some spatial information, which should be critical to computer vision tasks. The author may try different pooling sizes, like 2x2,4x4, and 8x8. The costs of them are also little.\n3. In recent months, MoE methods successfully trained huge and state-of-the-art NLP models (e.g., Switch Transformer). Though they are Arxiv papers, adding discussion with them may improve the paper.\n4. Since there is only a single feature in each memory, the memory may difficult to represent the underlying structure of the overall feature space, especially for ImageNet with diverse images. The authors may try multi-features in memory.\n5. Visualizing attention maps learned from the CA modules is helpful to understand the motivation of CA.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see Weaknesses and Suggestions in the Main Review.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "boZjRi1H5qb", "reviewer": "Reviewer_pRRP", "summary": "This paper proposes a dynamic network mechanism, routing-by-memory mechanism, to existing CNN networks for improving network performance with negligible extra computational cost. This mechanism adds multiple parallel Procedural Units (PUs) in each stage of the network, where each PU consists of a memory (a representative feature) and a procedure (some conventional blocks). By employing the memory for similarity calculation, the network can adaptively route different samples to different PUs in each stage to tackle various samples better. The experimental results on three public benchmarks demonstrate the effectiveness of the proposed dynamic routing mechanism.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\tThis paper presents a new dynamic network mechanism, routing-by-memory mechanism, which is inspired from the classical Mixture of Experts idea, and its effectiveness can be validated by the experimental results on three public benchmarks.\n\tThe paper is well written, and its structure is organized clearly.\n\nWeakness:\n\tConsidering the novelty, the presented dynamic routing mechanism itself is not quite novel. There already exists some literature about dynamic network structure based on each input sample with similar Mixture of Experts idea [1,2,3]. The authors need to provide more comparison to these work, and demonstrate the advantages of this work compared to previous ones.\n\tAs for the CA (Conditional Attention) module, there is some ambiguity about the mechanism of this module. As claimed in the paper, the same channels of features outputted from different PUs can capture different semantic meanings, which can increase the learning difficulty. However, the CA module is running in each PU separately, so how does this module help relieve this feature learning inconsistency problem. The authors are expected to provide more analysis into this module.\n\tIn the paper, the memory is initialized by clusters’ center of intermediate features, and the paper claimed that in this way, different memories can dominate different kinds of features in the same stage. However, this claim should be validated by empirical results. Authors can provide some visualization results or data analysis to show that different PUs in each stage can be specialized to tackle different kinds of input samples with the help of different memories. For example, the authors can show that whether the images in the same category are more likely to be assigned to the same PU in each stage. More experimental results like this can help validate that the trained PUs are specialized experts for certain functions, which is important.\n\n[1] Shaofeng Cai, Yao Shu, and Wei Wang. Dynamic routing networks. In WACV, 2021\n[2] William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers:Scaling to Trillion Parameter Models with Simple and\nEfficient Sparsity. arXiv e-prints, 2021\n[3] Huaxiu Yao, Yingbo Zhou, Mehrdad Mahdavi, Zhenhui Li, Richard Socher, Caiming Xiong, Online Structured Meta-learning, in Proceeding of the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020), virtual, Dec. 2020\n\n\n\n------- \n\nThe authors' rebuttal addressed my comments. Now I would like to upgrade my grading. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NIL", "review_text": "", "extra_scores": {"ethical_concerns": "NIL", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ARlEQ0c6rTH", "reviewer": "Reviewer_Svzq", "summary": "The authors propose to use clustering of features and nearest neighbor routing to create a modular network architecture. This greatly simplifies the routing algorithm, which usually relies on complicated, learned mechanisms. It improves the accuracy of various networks on several image classification benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To my best knowledge, this routing method is original.\n\n### Significance\nThe significance of the method is threefold:\n - It greatly simplifies the routing mechanism. Usually, modular networks require RL, EM, or other non-trivial training methods to train the router. This paper sidesteps that elegantly.\n - The method creates new modules by cloning the existing ones and making them specialized. This can be easily extended to dynamically creating modules in the course of training.\n - It offers an easy performance boost on certain tasks\n\n### Positives\n  - The method is simple and general\n  - The method might be useful more generally, for example for dynamically creating modules.\n\n### Negatives\n - The effect of the increased batch size is not measured on the baselines. Thus, it is not clear how much of the gain comes from the increased batch size.\n - The authors report only a single seed per experiment, without standard deviations.\n\n### Summary\nThe idea is nice, but I think the current version of the paper seems to be rushed and it does not meet the bar yet. I'm willing to improve my score if the above negatives are addressed.\n\n### Clarity\n  - Some parts of the text are hard to read and required me few re-reads to understand.\n  - Please have your submission proofread for English style and grammar issues.\n  - Line 46: \"we split the network into different stages by downsampling layers\". This suggests that you are designing the architecture and use the downsampling layers for splitting. But you are modifying the existing architectures. Does this mean that you are using the existing downsampling layers as boundaries of the processing units?\n  - Line 57 and 212: from the text it seems like you are using a fixed, relatively small number of epochs for the stem network training stage, but you mention early stopping here. Please clarify what do you mean by \"(i.e., early stopping)\"\n  - In formula (6), how is the selected module (R(m_i, f_{i-1})) represented? Is it one-hot?\n  - Line 218: \"we also clone the momentum\" - is it more generally the optimizer state (for example for RMSprop experiments) or just the momentum? Please clarify. If the latter, please clarify why just the momentum.\n  - Line 281-283, 310-311: why do you multiply the batch by N/2 and not N? That would result in the original batch size in expectation for all modules (if we assume that the clusters are equally likely). You also talk about imbalanced data in this sentence. Can you please clarify?\n\n### Redundancies:\n - Line 162, 165: redundant description of the GAP\n - Section 4.2.3 is redundant with the paragraph \"Momentum in Memory Updating\" paragraph of Section 4.1.3\n\n### Comments regarding the text\n  - Line 21: the \"sensory processor\" should not be considered as a part of \"human memory\"\n  - Line 37: increasing the number of procedures?\n  - Line 35-18: the end of this sentence is hard to understand\n  - Line 40: does not bring - does not require?\n  - Line 51: I found calling the Squeeze-and-Exite module [14] as CA module confusing.\n   - Line 54: use a colon instead of a comma after \"four training steps\"\n   - Line 56: by -> for?\n   - Line 61-62: consider clarifying \"continue the network training as the first step in the routing-based training step\"\n   - Line 128: \"mainly\" - the paper only focuses on image classification, so please remove\n   - Line 140-141: sentence hard to read\n   - Line 158: we use *the term* N-way?\n   - Line 190: scalar vector?\n   - Line 325: input's\n\n### What could make the paper nicer - optional\n - A bit more analysis. Maybe analyze how active are the modules for the target classes to see how they map to semantical categories\n - Analyze (and show) the usage statistics of the modules. Are they some which dominate or some that die out? How correlated are the module selections of different stages?\n\n### Citations\n  - Regarding the CNNs, please consider citing some earlier work, for example, LeCun et al: Handwritten digit recognition with a back-propagation network\n  - The first paper successfully training very deep CNNs are Highway Networks from Srivastava et al", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new dynamic routing-by-memory mechanism for MoE which is simple and elegant.   The mechanism effectiveness is validated by the experimental results on three public benchmarks. 

All the reviewers agreed the proposed memory routing is of interest to community thanks to its simplicity and good empirical results.


Decision: Accept (Poster)