Neural Routing by Memory
Authors: Kaipeng Zhang
Zhenqiang Li
Zhifeng Li
Wei Liu
Yoichi Sato
Venue: NeurIPS 2021 Poster (2021)
Keywords: convolutional neural networks
routing
memory

Abstract: Recent Convolutional Neural Networks (CNNs) have achieved significant success by stacking multiple convolutional blocks, named procedures in this paper, to extract semantic features. However, they use the same procedure sequence for all inputs, regardless of the intermediate features.
This paper proffers a simple yet effective idea of constructing parallel procedures and assigning similar intermediate features to the same specialized procedures in a divide-and-conquer fashion. It relieves each procedure's learning difficulty and thus leads to superior performance. 
Specifically, we propose a routing-by-memory mechanism for existing CNN architectures. In each stage of the network, we introduce parallel Procedural Units (PUs). A PU consists of a memory head and a procedure. The memory head maintains a summary of a type of features. For an intermediate feature, we search its closest memory and forward it to the corresponding procedure in both training and testing. In this way, different procedures are tailored to different features and therefore tackle them better.
Networks with the proposed mechanism can be trained efficiently using a four-step training strategy. Experimental results show that our method improves VGGNet, ResNet, and EfficientNet's accuracies on Tiny ImageNet, ImageNet, and CIFAR-100 benchmarks with a negligible extra computational cost.

Neural Routing by Memory
Kaipeng Zhang!? 1.2
Zhenqiang Li
Zhifeng Li
Wei Liu
Yoichi Sato
'Intiitute of Industrial Science, The University of Tokyo Tennent Data Platform
(kpzhang, liq] taa.suu-tton u-tokyo.ac. ac. michaelefli@tencent com v122238columbia, edu
yeato@iis.u-tokee ac.
Abstract
Recent Convolutional Neural Networks (CNNs) have achieved significant success by stacking multiple convolutional blocks, named procedures in this paper, to extract semantic features. However, they use the same procedure sequence for all inputs, regardless of the intermediate features. This paper proffers simple yet effective idea constructing parallel procedures and assigning similar intermedi- ate features to the same specialized procedures in divide-and-conque It relieves each procedure's learning difficulty and thus leads to superior perfor- mance. Specifically, we propose routing- routing-by-memon by-memory mechanism for existing CNN architectures. In each stage the network, we introduce parallel Procedural Units (PUs). A PU consists of memory head and procedure. The memory head maintains a summary of a type of features. For an intermediate feature, we search its closest memory and forward it to the corresponding procedure in both training and testing. In this way, different procedures are tailored to different features and therefore tackle them better. Networks with the proposed mechanism can be trained efficiently using four-step training strategy. Experimental results show that our method improves VGGNet, ResNet, and EfficientNet's accuracies on Tiny ImageNet, ImageNet, and CIFAR-100 benchmarks with negligible extra computational cost.
fashion.
Introduction
Human memory is often understood informational processing system. It plays essential role in humman intelligence and comprises short-term memory and long-term memory, inspiring many well-known machine learning models, such Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) [14, and Neural Turing Machine (NTM) 11.. Episodic memories, a type of long- term memory, are the collection of past personal experiences. They be retrieved and exploited by the brain when tackling problems that have been encountered before. Different memories activate different neurons the brain, directing us to perform specific procedures that have done before. Inspired by this observation, we introduce the routing-by-memon mechanism to the neural network. uses memory (a summary of seen features) to guide networks process different features with different procedures in a divide--ndd-ongeeee better performance. This paper applies the mechanism to the feature extraction in CNNs and refers to networks employing it as Routing-by -y-mmoorry Networks (RMNs). In the following, we will introduce conventional CNNs and our RMN.
manner, casing the difficulty of learning and achieving
Recent Convolutional Neural Networks (CNNs) achieve state-of-the-an results on computer vision tasks by stacking convolutional blocks (e.g., residual block (13), inception block (33], and Dense block (17), named procedures in this paper, to extract semantic features. However, they use the same
Co-corresponding authors
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Feature ft
Procedure
Feature
fi+1
(a) Typical feedforward neural network
Procedural Unit
Memory 7,,0
Procedure
3,00g
Feature f
R(fa. m))
Memory 1,1
Procedure 11.
Feature fi+1
Memory 1,
Procedure 12.
(b) Routing by memory network
Figure 1: (a) Conventional feed-forwaid network. It processes the features by stacked procedures. (b) Our proposed Routing by Memory Network (RMN). It employs parallel procedures (i.e., multi-branch network) and processes features in a divide-and-conquel manner. Different procedures are tailored to different types of features. is routing function, and we use the nearest neighbor algorithm for it. m, P, and & denote the memory, procedure architecture, and the parameters of the procedure, respectively. Memory indicates which type of features the corresponding procedure can handle. Given feature that a previous stage yields, it searches its most similar memory and forwards the feature to the corresponding procedure. For example, T0 1,0, 11.1 and m. represent scenery, food, and animal, respectively. Given lion's features, R will forward them to the third procedure.
stacked blocks to process all intermediate features, despite large variances of those features. Since similar intermediate features can be processed in the same way, propose simple yet effective idea of introducing parallel procedures (i.e., multi-branch CNN) and assigning similar intermediate features the same specialized procedures (i.e., the same branch). is a divide-and-conque structure, sceing Figure [ for an illustration. In this way, we can improve the model capacity and performance while not increasing the computational cost since forward an intermediate feature to only one procedure.
In RMN, introduce simple yet effective mechanism, routing by memory. Briefly, introduce the Procedural Unit (PU) to process features (see Figure and Figure for illustrations). It consists of memory summary/tepresentaive feature) with a procedure (some convolutional blocks). We use the memory to identify which type of features the corresponding procedure is expected to handle. Specifically, we split the network into different stages by downsampling layers. There are multiple PUs in each stage. All procedures within a stage use the same architecture but different parameters. In stage, given an intermediate feature produced from the previous stage, we will search its nearest memory and forward it to the corresponding procedure. In this way, different PUs are specialized to handle different types of features. Besides, the procedures, we introduce a routing -ependeent Squeeze-and -aeee-nnd---teeeee (SE)-like feature attention module, dubbed Conditional Attention (CA), to improve the performance further.
How to initialize and update memory and procedures is the main challenge when training our RMN, We propose an nos-lto-mpleement training strategy that includes four training steps: stem network training, procedure cloning, memory initialization, and routing-based training. In the stem network training step, we train a conventional CNN (c.g.. ResNet and EfficentNet) as a stem network. Then in the procedure cloning step, we generate multiple procedures (i.c., multi--banch CNN) in each stage of the network by cloning all learned procedures preset number of times. In the memory initialization step, we first extract intermediate features for all stages of the network. Then we use representative features as initialized memory. This paper clusters features each stage of the network and uses cluster centers as representative features. Finally, we continue the network training in the routing-basee training step, which similar to the first step, but the data flow inside the network is decided by routing results. In this step. the memory is updated in moving average fashion while other components are updated by the original optimization method used in the first stage. The overall training strategy is plug-and and--lad -lay to existing CNNs' training strategics.

---

This paper takes VGGNet. ResNet, and EfficientNet as backbones to train our proposed RMN. According to the experimental results, RMN significantly improves original models' results on some benchmarks while not increasing the computational cost. We summarize our contributions as follows:
We propose a novel divide-and-conquee mechanism called routing by memory, which constructs multi-branch CNN and assigns similar features to the same specialized branch. It can lead to better performance while not increasing the computational cost.
The proposed mechanism is plug-and-play to existing CNN architectures by virtue of proposed effective four-step training strategy.
We apply our mechanism to VGGNet, ResNet, and EfficientNet and achieve significant improvements in the accuracies on Tiny ImageNet, Imagenet, and CIFAR-100 benchmarks.
2 Related Work
2.1 Convolutional Neural Network Architecture
In 2012, AlexNet 23 outperforms handcrafted features engineering by a large margin in the ImageNet image classification competition. It combines feature extraction and classification in a deep CNN model to extract discriminative semantic features. After that, CNNs gradually dominate most computer vision tasks, such as image classification, object detection, and semantic segmentation. Classic CNN architecture (e.g., AlexNet 23 ZFNet [41], and VGGNet (32)) extract features by stacking convolutional layers. In 2016, ResNet 13], the most successful CNN architecture recent years, proposed to train very deep CNNs using skip connections and stacked residual blocks. After that, more and more efficient and effective stacked block-based CNNs (c.g., MobileNet[15), SENet [16), and EfficientNet (34) were proposed. However, they use the same block sequence for all inputs in feature extraction, regardless of the intermediate features. This paper introduces the plug-and-play routing-by-i seetinng-y-mennee memoor mechanism existing CNN architectures. It uses memory to guide different blocks to tackle different features in a divide-and -nnc-uue fashion, which boosts the accuracy while not increasing the computational cost.
2.2 Conditional Computation
Conditional computation 33 refers to activating only some of the modules in a network in an input- dependent fashion. Recent researches have introduced it to CNNs in order to accelerate network inference. AIG (36,, BlockDrop (39], and SkipNet (37] propose to learn the subset of blocks needed
to process given input. Since easy examples may not require deep layers' features to make the classification, SACT [9], Inside Cascaded CNN, 44 and Dynamic Routing (26) propose to do input-dependent early stopping at the stage of network inference. Routing Convolutional Network (RCN) (20) introduces a routing network that can perform anytime recognition. These methods aim to reduce the models' computational costs while maintaining comparable accuracies.
2.3 Mixture of Experts
Mixture of Experts (MoE) (19) 21 was proposed three decades ago. It is related to conditional
computation and introduces multiple experts (leamnes) divide the problem space into homogeneous regions. Given an input, MoE often uses a routing module to select the corresponding experts in a
hard soft selection fashion. In the deep learning era, sparsely- gatsedd mixture-of-expprts layer [i first introduces MoE to LSTM for language modeling. Lately, Switch Transformer (8] introduces
MoE to Transformer [35] to train a trillion parameter model and achieves great success in natural language processing, As for computer vision, there are two categories of MOE, dynamic parameter and dynamic architecture, which are compatible. Dynamic parameter methods 440
[ 6 learn
input-dependent convolutional kernels' parameters using parameters generation networks. Dynamic architecture methods employ multiple sub-networks and build dynamic computational graph by decision networks. NOEF [I] and HydraNets [27] divide the class label space and assign different sets of class labels to different sub-networks for handling. DeepMoE (38] proposes to increase the kernels and select each layer's features' channels by a decision network. Runtime Routing (28) proposes to learn an RNN-based decision network by reinforcement learning for multi-path networks. Akin to the Runtime Routing, DRNet 4 uses small CNN the decision network. Dynamic Routing (25).
3

---

ResNet-18
Lonv' & Pool
Block 2
Block
Block x
Block
Catt
Pr.0,
55,
2..9
ReoNet- 18- based RMN
Conv & Pool
mys
Block
mic
Block
Wag. Block
CA
Pa. Ago
CA
CA
Py.
CA
Maa
Block
m2
f
Sy.
Stage
Stage
Stage
Stage
Feature extraction
Feature processing
Classification
memory stage
blocks (ie, (1l)
Conditional (CA) module
Block m
Procedure
Procedure Unit (PU)
Figure 2: An illustration of Re and ResNet- 18-based 2-way RMN. We divide the network into three phases: (preliminary) feature extraction, feature processing, and classification. In the feature processing phase, split the network into four stages by downsampling layers. RMN introduces the routing-by-mente mechanism to each stage. It uses memory to guide forwarding different features their corresponding specialized procedures. Different colors denote different stages. Different shades of color represent different kinds of features, memories, and procedures within stage.
Routing Networks [29]. ExperGate [2], and HardMoE [12 extend MoE to semantic segmentation, multi-task learning, weakly-supervin learning, and lifelong learning. respectively. These decision networks increase the training difficulty and bring extra computations. Besides, some of them require modifying relevant gradient descent algorithms employing reinforcement learning to train the decision networks.
Our RMN solves routing in perspective, routing by memory, rather than designing complicated decision networks to decide the routing path. is more straightforward and elegant. Besides, since the routing is based non-parametric nearest memory search, RMN is lightweight, easy train, and requires no modification in the tack-propagation rule.
Routing by Memory Network
This section elaborates the architecture and training strategy of our proposed RMN.
3.1 Network Architecture
Our RMN is built by integrating the routing-by by-memory mechanism into an existing CNN architecture. In this paper, describe methods with the task of image classification. According to the functions, we treat the processing a common CNN as three separated phases: (preliminary) feature extraction, feature processing, and final classification. Our RMN enhances the intermediate features the feature processing phase by incorporating the suuting-by-meoooe mechanism. In the following sections, take ResNet- 18 [13 routing-by boagg--b-mmmrrr mechanism into a conventional CNN(see Figure] 2).
example backbone to present how integrate the
3.1.1 ResNet
In this section, we elaborate on the ResNet-18 18. Given input image X, in the first (preliminary) feature extraction phase, its feature maps fo are generated by a series of operations Po consisting of convolution, pooling, and batch normalization, which can be formulated follows:
fo Po 0n),
where Gg denotes the learnable parameters in P. Low-level features such as edges and corners are represented in these preliminary feature maps.
In the feature processing phase, multiple residual blocks are utilized to process feature map fo for extracting features in higher semantic levels. We group the blocks into different stages according to their output feature maps' dimensions and resolutions (i.e., divide the stages by down-sampling operations). For example, shown in Figure 2 divide blocks into four stages by grouping

---

two adjacent blocks as their feature maps' shapes are the same. We represent the operations (i.e., convolution blocks) in the idh stage as P. with its parameters 6, The feature maps yielded from the 7t stage are ft. Specifically, we formulate the processing of the tth stage as follows:
J = P:(11,,,,
== 1,23,44
(2)
The ResNet ends with a global average pooling layer and a 1000-way (the number of classes, 1000 in ImageNet) fully-connected layer with argmax (softmax in the training phase) function. They are used to classify one given image in the classification phase. We formulate this phase as follows:
y
arg max (GAP(fa)W + b),
DE cE(0.I1....c-1))
where GAP refers to the global average pooling in the classification phase. W and b are learnable weights and biases of the conneccd layer (i.e., classifier), respectively. e is the class index, and is the total number of classes. y denotes the output class label.
3.1.2 Res RMN
In this section, we take the ResNet-18 as a backbone network to introduce our RMN. In the Resnet- 18-based RMN, we propose Procedural Units (PU) in the feature processing phase. There are multiple PUs in each stage, and we use the term N-way RMN to represent how many PUs per stage in our RMN. All PUs within stage share the same architecture but learn different parameters due the routing mechanism. Each PU consists of two modules, including memory and procedure. The memory is a representative feature that is learned from features by a moving average fashion in the training phase (see Section 3.2 for details). We use global average pooling for memory to reduce the storage cost. In the 1th stage, given the feature fi-1 and memories my = ma,0: Mi,1, NNN we first do the routing R by nearest neighbor searching as follows: arg min (D(ms.j. GAP(Ji-))), R(m;, JE(O.I....NN-11)
(4)
where m denotes the j*h PU's memory in the jth stage. To reduce the storage cost, we apply global average pooling GAP to f before routing. D refers to a distance measurement metric, and we use Euclidean distance in this paper. Since D is non- non-parametric and fast to compute, the computational cost of routing is negligible and can be omitted.
After the routing, we forward the features to the corresponding procedures. A procedure consists of residual blocks and optional Conditional Attention (CA) module. The CA module is optional module introduced to improve the accuracy further by a little additional computational cost. Without the CA module, RMN re-formulates the jth stage of the feature processing phase as follows:
f. 04
i i= 1, 4,
(5)
where 02.0 O,,,,.., - refers to N sets of parameters for P.:
We then introduce the CA module. The same channels of features produced from different procedural units may represent different seman- tic meanings. For example, the first convolution kernel of procedure may focus on animals' fur. However, the first convolution kernel of an- other procedure may focus on furniture's ' texture. The inconsistent semantic meaning of different features increases the learning difficultly. In- spired by the position-coding in ViT 77 we in- troduce the CA module to do routing-dependent channel-wise attention to the features, relieving the inconsistency adaptively. The CA module consists of conditional input, a routing result, and a Squeeze-and ----ann-tttxttinn (SE) module [16]
PiCfi (LRmms
GAP
RRmm, fi-))
ReLU
Sigmoid
Scale
Figure 3: An illustration of Conditional Atten- tion (CA) module. The CA is depended bobeeze-and-Exciittiin (SE)-like module. (see Figure 3 for an illustration of the CA module). Specifically, it first computes the scalars
8 = [$1, 82, sx] that has the same number of channels as fi = [fr, Sea, Jix] This operation can be formulated as:
= CraciiteConcat(SSqueeee R(m,, f(-1))),
(6)
5

---

where Squeeze denotes using a global average pooling followed with a fully-connected layer to squeeze the feature dimension. Ercite denotes using fully-connected layer and a Sigmoid function to expand the feature dimension and compute attention values. Concal denotes channel-wise concatenation operation. can also use different coding methods for the routing results, such as one-hot coding, which sometimes makes the training more stable. Please note that all CA modules share parameters within a stage.
Then we scale the features fi according to the scalar vector s by channel-wise multiplication (we use to represent it). The CA module can be formulated as follows:
CA(A)) ft-
In summary, when using the CA module, RMN re-formulates the thh stage of the feature processing phase as follows:
St = P: 1))s i= 1,2,3,4.
3.2 Training Strategy
In this section, we take the MeANet-18-basel RMN example to introduce the four-step training strategy. Unlike the conventional training pipeline, after a few training epochs, we insert two steps to expand procedures and initialize memories. In the last step, we update memories in moving average fashion. The training strategy is auug-nnd-play to existing CNNs' training approach since can follow standard gradient descent algorithms in optimization.
3.2.1 Stem Network Training
In the first step, we aim to train a conventional CNN (c.g., VGG, ResNet, and EfficientNet), named stem network in this paper, extract reasonable features for memory initialization. Specifically, in this step, we train the standard ResNet-18 18 for few epochs and will resume the training in the fourth step. If using CA modules, denotes the discrete uniform distribution.
random routing results ~ U(1, N) for each CA module, where
3.2.2 Procedure Cloning
We clone the procedures in the second step. Specifically, clone the parameters of all procedures by N times (i.e., assign 0; to 1:. In this way, we can build branch procedural units in each stage of the feature processing phase. Please note that we also clone the optimizer state of the gradient in this step to maintain the training's 's consistency in the fourth phase.
3.2.3 Memory Initialization
We initialize the memories using representative features. In this paper, compute the representative features by cluster analysis. Specifically, in the jth stage of the feature processing phase, we extract each training sample's ft. Then we apply the Fuclidean distance-based K-means clustering algorithm to obtain N clusters. Finally, we initialize each memory in m,+1 by each cluster' center. In this way, different memories can dominate different kinds of features.
3.2.4 Routing-Bart -Base Training
In this step. we resume the model training. Inspired by the update of mean value in Batch Normal- ization (BN) [8]. we update the memory in moving-average fashion. Specifically, in training iteration and in the jth feature processing stage, given / samples that are routed to the jth PU, we have their features as fi, .... 1. We update the memory follows:
ΣH GAP(S!)
maj am,, + (1 (1
(9)
H
where a is Syper-parameter and denotes the momentum.
In this step, the computational graph (i.e., data flow) is dynamic due to the routing mechanism. But, the
mechanism uses the non-paramen nonaparaittric nearest neighbor algorithm to select the procedures, and the memory is updated in moving-average fashion. Thus, use the original

---

stochastic gradient descent algorithm with softmax cross-entropy loss function to train our RMN as the original ResNet-18. We also let each routing operation have a 0.05 probability of doing random routing, which can train procedures more stable.
4 Experiments
In this section, will first introduce our experimental setup. Then we present ablation experiments on our proposed main components and syper-parameters, Finally, we show our results on some image classification benchmarks.
4.1 Experimental Setup
In this section, we first introduce the datasets used. Then we introduce the network architectures. Finally, we present the training details of our RMN.
4.1.1 Datasets
In this paper, we take three image classification benchmarks, Tiny ImageNet (4), ImageNet 2012 [30], and CIFAR-100 [22] to evaluate method. ImageNet 2012 consists of 1.2M training images and 50,000 validation images for 1000 classes. Tiny ImageNet is a subset of ImageNet. It consists of 200 classes, and each class has 500 training images and 50 validation images. CIFAR-100 consists of 100 classes, and each class has 500 training images and 100 validation images. We use Tiny ImageNet for ablation experiments in Section 4.2 In Section 4.3 benchmarks.
show results on all three
4.1.2 Network Architectures
Our RMN can be applied to most existing CNNs architectures, and we take some widely-used architectures, i..., VGG, ResNet, and EfficientNet, in experiments. In the ablation experiments, we take ResNet-18 as the backbone network to train our RMN. In the section of evaluations on all benchmarks, present the results of using all architectures. For VGG-16, pooling layers to different stages and build the PUs. We also use batch normalization for VGG- 16. Since the image resolutions of Tiny ImageNet and CIFAR- -000 are 64 64, and 32 32, respectively, we drop the first two down-sampling operations for ResNet and EfficientNet when training on these two benchmarks. For VGGNet, we drop the first down-sampling operations Tiny ImageNet but keep them on CIFAR- 100. For the ImageNet, we use the resolution of 224 x 224.
split the network by
4.1.3 Training Details
Learning rate, We first follow the warmup strategy [00 to increase the learning rate from le-5 to 0.48 in the first five epochs. Then use the cosine learning rate strategy for the rest of the epochs, and we decrease the learning rate to le-5 at the final epoch.
The number of training epochs. For CIFAR-100 110 and Tiny ImageNet, the total numbers of training epochs for VGG-16, ResNet-18, ResNet-50, and EfficientNet-Bo are 120, 120, 200, and 300, respectively. For ImageNet, we use 160, 160, 220, and 400 epochs for them, respectively.
The number PUs. In this paper, N denotes the number of PUs per feature processing stage. Increasing N can improve the model capacity but brings in more parameters. Seeing Section 4.2 for the ablation experiments on N. We set N 8 for the accuracy and cost trade-off in other experiments. Please also note that, compared with memory consumption consume little memory (less than 1% overall consumption when doing inference on ImageNet with batch size 100). Hence, the extra GPU memory consumption by method is negligible. Besides, extra parameters bring an extra storage cost. But storage is abundant in real-world applications since disks are cheap.
feature extraction, the parameters
Batch Size. We use batch size 256 for all baselines. For our RMN, each feature processing stage, there are multiple PUs, and different features from the same batch are fed to different PUs. So, the batch size of each PU is much smaller than the total batch size, and accordingly, we have to enlarge the batch size to train our RMN. Simply, we multiply the original batch size (256) by N/2 though the

---

67
67
Accuracy (%)
66
66
66
65
Accuracy
65
Accuracy
65
64
64
64
63
63
63
62
62
62
61
10 12 14
16
512
1024 1536
2048
61
0.0
0.2
0.4 0.6 0.8
The number of PUs (a)
Batch Size (b)
Momentum in Memory Updating (c)
Figure 4: Experimental results Tiny ImageNet. The blue dashed line denotes the baseline ResNet- 18 with CA modules (without conditional routing inputs). (a) More PUs bring higher accuracy, and the accuracy got converged on 8 PUs, so set it to 8 in other experiments. (b) Since RMN assigns different samples to different PUs, it requires a large batch size to maintain the training stability. We set it 1024 in other experiments. (c) Higher momentum makes the memory updating smoother and more stable and leads to better results.
numbers of data instances among different PUs are imbalanced, which means that we use N 8 and batch size 1024 in this paper. Please see Section 4.2 for the ablation experiment of batch size.
Momentum in Memory Updating. The momentum is hyper-parameter used to make the memory updating smooth and stable. However, too high momentum will make the memory out of date, while too small momentum leads the memory updating unstable. We it to 0.9 in our paper. Please see Section 4.2 for its ablation experiment.
stochastic gradient descent with momentum 0.9 and weight decay le-5 for ResNet and VGGNet. We use the RMSProp optimizer for EfficientNet. We use Synchronized Batch Normalization (SyncBN) supported by the Nvidia APEX library. For CA modules, we use the reduction ratio 16. For data augmentation, we use random augmentation introduced in the ResNet [ paper. We 8x V100 (32GB memory version) with PyTorch.
Others. Regarding other training details,
4.2
Ablation and Hyper-Parameterr Experiments
4.2.1 Number of PUs
The key component in our RMN is the PU. So N, the number of PUs, is a critical hyper-parameter: Increasing the number of PUs will improve the model's capacity and lead higher accuracy. From Figure (1), (1), we can see that the accuracy increased along with larger N and got converged when N 8. However, larger N increases the number of parameters and requires larger batch size (batch = 256 x N/2). We make an accuracy and cost trade-off and set N to 8 in other experiments.
4.2.2 Batch Size
Our RMN introduces multiple branches with different PUs in the network, and the examples are assigned to the different branches. So, RMN requires a larger batch size than conventional CNNs. According the results in (b) too small batch sizes (128 or 256) lead to even lower accuracy than the baseline, which is because the memory updating and routing are unstable. Thus, network training becomes more difficult. Moreover, with the 128 batch size, we found about 40% PUs died after 50 training epochs, which means that these PUs' memories are far from the distribution of the current features, and thus no data can be assigned to these PUs. We consider that the small number of data instances assigned to the PUs will make the memory updating unstable and draw the corresponding memory features to the outliers. Too large batch size (2048) also leads to not good results. Simply, multiply the original batch size (256) by N/2 though the numbers of data instances among different PUs are imbalanced, which means that we set the batch size to 1024 in other experiments.
We also increased the batch size to 1024 baselines and reported the results for fair comparisons. We find that there is clear overfitting using batch size 1024 for the baselines, especially for ResNet-50 and EfficientNet-BH These results are lower than the results of our method. Due the limited pages, please the results our supplemental material.

---

4.2.3 Momentum in Memory Updating
We use the moving average to update the memories. Thus the momentum is key hyper- syper-parameter. Too small a will lead the memories to change quickly. The higher is, the more stable and smoother the memory updating becomes, while too large causes a very slow updating. According to Figure (c), we set a to 0.9 in other experiments.
4.2.4 Conditional Attention
The Conditional Attention (CA) module a SE- like modules to further improve accuracy by in- troducing a little extra cost. According to the results in Table it improves the ResNet-18's accuracy by 2.2% on Tiny ImageNet.
|
#Farmms
RMN RMN (ResNes- 18)
66.5%
89.5M
2.208
Table Ablations of the CA module on Tiny ImageNet.
4.3 Evaluation on Benchmarks
We evaluate methods three image clas- sification benchmarks: Tiny ImageNet, Ima- geNet, and CIFAR-100. According the eval- uation results ( on Tiny ImageNet, our RMN significantly improves the accuracy while not increasing the computational cost. Besides, the improvements for VGGNet and ResNet are much more significant than EfficientNet. It is because EfficientNet's architecture is already well-designed, compact, and contains SE mod- ules. Thus, our method may bring some redun- dancies, but it still improves the accuracy by 1.8%. Sceing from the results (Table on the more challenging benchmark, ImageNet, our method still performs well. Moreover, using our RMN, ResNer-50 can outperform EfficientNet-BD For the CIFAR- 100 benchmark, although the image resolution is small (32 x 32) our method can still gain considerable improvements. Moreover, ResNet- 18pe--88-pp 18-based RMN achieves impressive accuracy and even outperforms the original ResNet-50.
Model
Asc.
*Purams
*FLOPs
VGG-16
17
63, 16%
135.0M
14,10B
RosNes-18
61.78%
2.258
ResNee-50
67.28%
23.9M
5,25B
EfficietNee-B0 80
67.14%
4.3M
0.478
RMN (VGG-16)
66.57%
237.8M
14.118
RMN (ResNet-18)
66.53%
89.5M
2,268
RMN (ResNes-50)
RMN (Ffficion(Nss-B0)
0.478
Table Evaluation on InnylmmaeNNet. Ten runs average results (skip when occurring PU death).
Model
Aoc.
4Params
AFLOPs
Model
Acc.
#Parems
AFLOPS
VGG-16
72.93%
138.3M
15.518
VOG-16 52
73.16%
33.8M
0.43B
ResNet-13
13
H
70.72%
11.7M
1.8IB
ResNet- 18
75.62%
11.2M
0.56B
76.06%
ResNet-
77.53%
1,31B
Nol
76.46%
0.308
FfficienNet-B
78.04%
4.2M
0.128
RMN (VGG-116
RMN (VGGG-16
75.74%
RMN (ResNet-18)
73.74%
89.0M
1.828
RMN (ResNe- (8)) 77.94%
88.4M
0.57B
RMN (ResNet.50)
78.31%
190 7M
4.138
RMN (ResNer-S0)
79.06%
184.6M
1,32B
(BffcicenNei-BO)
78.08%
(EfiiceenNee-B0))
78. 88.885
0.128
Table 3: Evaluation on ImageNet. Three runs average
Table 4: Evaluation on CIFAR-100. Ten average results (skip when occurring PU death).
results (skip when occurring PU death).
4.4 Comparisons with Related Work
RNR 28 and DeepMOR 38
the two related works. Our motivations and methods different, and we discuss and make comparisons in this subsection.
RNR learns RNN-based routing network by reinforcement learning dynamically select experts. trains the routing network and the backbone network alternately. DeepMOE learns a CNN- decision network to activate specific layers in the backbone network dynamically. Unlike these methods, our RMN solves routing in a new perspective, routing by memory, rather than introducing complicated routing networks. It is more straightforward and elegant. Besides, since the routing is based on non- con-parmttriic nearest memory search, RMN is lightweight, easy to train, and requires no modification in the back-propagation nule. Here make the experimental comparisons with RNR.
For RNR, we follow codes to align our baselines' training details and then retrain our RMN on CIFAR-100 Based on ResNet-18 18 backbone, our RMN (N 4, without CA) achieves 78.96%

---

accuracy while costs 14.2ms per image (using Xeon 8255C CPU), which is superior to RNR's (#branch=4) 7esults-78.42% accuracy with 59.8ms.
For DeepMOF, we make comparisons on respective baselines. DeepMOE improves ResNet-50's accuracy ImageNet from 76.15% 77.12%, while RMN improves ResNet-50 from 76.08% to 77.78% (N = 8, without CA). Besides, our RMN is more efficient since does not introduce extra decision networks.
Conclusion
In this paper, we proposed specific mechanism, routing by memory, for conventional feed-forward networks. We integrated with the existing CNN architectures and built the Routing by Memory Network (RMN). Specifically, introduces the Procedural Unit (PU) to the CNNs, which consists of a memory (a representative feature) with procedure (some convolutional blocks). We employed memories to forward different features to their expert PUs. Networks with the proposed mechanism can be trained efficiently using a four-step training strategy. According to the results on Tiny ImageNet, ImageNet, and CIFAR-IO0 100, our RMN significantly improves VGG-16 16, ResNet-18 ResNet-50, and EfficientNet-Bl while not increasing the computational cost.
6 Limitation Analysis
The main limitations of our method are extra parameters, larger training batch sizes, unstable training, and parallel inference efficiency.
First, since our method clones the experts multiple times, introduces extra parameters and consumes additional memory and storage. But, discuss in Section 4.1.3 the model parameters consume little memory. Besides, in many real-world scenarios, the model storage cost negligible. other words, our method trade redundant memory and storage for accuracy improvement.
Second, since our network is multi each training iteration. Thus, our method requires a larger training batch size (4x for 8-way RMN). But V100 (32GB) enough to train most existing CNN models. Besides, some techniques can use larger batch sizes in limited memory, such gradient accumulation, memory-saving CNN training framework (e.g., MXNet), and dynamic memory allocation.
multi-branch, branch, each branch (expert) requires enough training samples in
Third, the model training is unstable since some PUs may die, especially when using not good training parer-parameters. If some PUs die, the accuracy will usually be lower than that of the baseline.
Finally, since the data flow in our method is multi-path, its parallel inference efficiency is lower than that of the baseline when inference batch size is greater than
Last but not least, our rule of assigning similar features to the same expert is mixed blessing. is explainable, non- aan-aaametric, and fast. But compared with parametric routing, our rule may impede the model itself from achieving better results.
Acknowledgments and Disclosure of Funding
This work supported by JST AIP Acceleration Research Grant Number JPMICR20U1 and JSPS KAKENHI Grant Number JP20H04205, Japan. This work is also supported by Japan JSPS Research Fellowship for Young Scientists (DC) with JSPS KAKENHI Grant Number JP2IJ13152.
References
[1] Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network experts for large- scale image categorization. In European Conference on Computer Vision, pages 516-531 Springer, 2016.
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 3366-3375, 2017.
10

---

[3] Yoshua Bengio, Nicholas Léonard. and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
[4] Shaofeng Cai, Yao Shu, and Wei Wang. Dynamic routing networks. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pages 3588-3597, January 2021.
[5] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11030-11039, 2020.
[6] Zhourong Chen, Yang Li, Samy Bengio, and Si. You look twice: Gaternet for dynamic filter selection in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1172-9180, 2019.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[8] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient spansity, arXiv preprint arXiv:2101.0390 2021.
[9] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1039-1048, 1048, 2017.
[10] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint arXiv: 1706.000677777111 2017.
[11] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 1rXiv:1410.5401, 2014.
[12] Sam Gross, Marc' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6865-6873, 2017.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98): 1780, 1997.
[15] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1744v.170..0461,, 2017.
[16] Jie Hu, Li Shen, and Gang Sun. saneece-and-excitatiee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[17] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[18] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[19] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.

---

[20] Zequn Jie. Peng Sun, Xin Li, Jiashi Feng, and Wei Liu. Anytime recognition with routing convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 2019.
[21] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 1105, 2012.
[24] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7, 2015.
[25] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang. and Jian Sun. Learning dynamic routing for semantic segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 8553-8561 2020.
[26] Mason McGill and Pietro Perona. Deciding how to decide:
Dynamic routing in artificial neural networks. In International Conference Machine Learnngg, pages 2363-2372. PMLR, 2017.
[27] Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Specialized dynamic architectures for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8080-8089, 2018.
[28] Yongming Rao, Jiwen Lu, Lin, and Jie Zhou. Runtime network routing for efficient image classification. IEEE transactions on pattern analysis and machine intelligence, 41(1))22291- 2304, 2018
[29] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv: 2017.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):2112522 2015.
[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experrs layer. arXiv preprint 1XXIv:7701.06300 2017.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
[33] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9,
2015.
[34] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 4105-6114 PMLR, 2019.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[36] Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-18, 2018.
[37] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 409-424, 2018.
12

---

[38] Xin Wang. Fisher Yu, Lisa Dunlap. Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. Deep mixture of experts via shallow embedding. In Uncertainty in Artificial Intelligence, pages 552-562. PMLR, 2020.
[39] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8817-8826, 2018.
[40] Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. arXiv preprint arXiv: 1904.0497, 2019.
[41] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
[42] Kaipeng Zhang, Zhanpeng Zhang, Hao Wang. Zhifeng Li, Yu Qiao, and Wei Liu. Detecting faces using inside cascaded contextual enn. Proceedings of the IEEE International Conference on Computer Vision, pages 3171-3179, 2017.
13


Reviews:
[{"review_id": "kD3LLUuKWzl", "reviewer": "Reviewer_cGyg", "summary": "This paper proposes a conditional computation model based on memory (not sure this is an appropriate term; see below). The routing-by-memory mechanism saves representative features for each procedure and later compares them with input features to select the most appropriate procedure. To stabilize routing-by-memory networks, a four-step training strategy is also introduced. The experiments show that routing-by-memory networks outperform their non-RMN baselines on image classification tasks with the similar number of flops.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is the novelty. The deep models with conditional computation have been around for years, and using representative features for routing is not new either [1, 2]. I think the only possible novelty here is adjusting representative features during the training. But it is unclear if this way is better than fixing routes with a pre-trained feature extractor in the beginning because the authors did not compare RMNs to any conditional computation baselines.\n\nThe fact that accuracy improved only a little while the number of parameters increased by several folds is disappointing ([3] observed a similar pattern in their work). It is not practical in terms of computational cost, because if the memory requirement becomes larger than a single GPU/TPU capability, the communication overhead will be involved.\n\nAlso, I am not sure if using the term memory is correct here. In this paper, the memory content is modified between training steps. I think the memory commonly refers to a \"scratchpad\" that can save temporary information during a single inference pass.\n\n[1] Park, David Keetae, et al. \"Megan: Mixture of experts of generative adversarial networks for multimodal image generation.\" arXiv preprint arXiv:1805.02481 (2018).\n\n[2] Mullapudi, Ravi Teja, et al. \"Hydranets: Specialized dynamic architectures for efficient inference.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\n[3] Ramachandran, Prajit, and Quoc V. Le. \"Diversity and depth in per-example routing models.\" International Conference on Learning Representations. 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": ".", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KQwpcCrQ72A", "reviewer": "Reviewer_bRNA", "summary": "This paper proposes a routing mechanism to dynamically adjust the DNN forwarding path conditioned on the input. This relieves the learning difficulty of a model by processing different features with specialized procedures. A four-step training strategy is introduced to support this dynamic routing process. Experiments on various backbone networks validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe idea of enforcing divide-and-conquer in DNN forwarding based on feature grouping looks new to me, although the internal technical components (e.g. k-means clustering, moving average, squeeze-and-excitation, etc) are commonly used tools.\n\nQuality:\nThe proposed method is technically sound. It is intuitive that networks specialized to handle certain groups of features can relieve the learning complexity and potentially improve performance.\nMy concern comes from the validation of the feature grouping. The visualization of the memory analysis has been put in the supplementary material. The separation of different features in different PUs (Fig. 1 in supplementary) is not obvious to me. Can you show more visualizations on more architectures and datasets to validate this specialized feature handling claim?\nI also wonder about the statistics of the usage of the routing dynamics, e.g. how many times on average the branch-switch happens across a dataset? what is the frequency of branch-switch at different stages in a model? any comparison between forcing the usage of different branches?\n\nClarity:\nThe paper is clear and easy to follow.\n\nSignificance:\nI think this dynamic architecture routing idea is beneficial to this community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "J-5tqMnAah", "reviewer": "Reviewer_fdXd", "summary": "The paper proposes to use extra network blocks (called memory units by the authors) to route different features in a network to different conv layers based on the k-nearest neighbors computation. Additional attention layers are also used to further scale the routing results as the input for the next layer. The proposed routing approach is applied to VGGNet, ResNet, and EfficientNet and shows accuracy improvement but with significant increases in memory/parameter cost.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* Thorough literature survey in the field of Mixture of Expert and Routing.\n* Figure 1 & 2 are helpful in comprehending the proposed modification to conventional routing architecture.\n* Detailed hyperparameter configurations are provided for the reproducing purpose.\n\nCons:\n* The major change to the conventional routing-based architectures in the proposed method is to replace the Gumbel Softmax routing with the k-nearest neighbors routing. However the authors only provide experimental results w/ and w/o the proposed routing method on three different backbone models. It is critical to understand why non-parametric routing by k-nearest neighbors is more preferable than the Gumbel Softmax routing. The comparison with RNR [2] and DeepMOE [3] in the appendix are not based on Gumbel Softmax routing.\n* As mentioned in the limitation section, the proposed routing algorithm requires significantly more parameters compared to the corresponding backbone networks. Specifically, on ImageNet, the proposed RMN requires 7.6 times the number of parameters (ResNet-18 from 11.7M increases to 89.9M parameters) in the resulting network architecture. The number of parameters in the proposed routing-based ResNet-18 is already more than 4 times the number of parameters in ResNet-1202. To me this does not seem to be an effective usage of a network’s capacity.\n* The routing concept is essentially doing differentiable architecture search but without the final path selection and final re-training. How is the proposed routing compared to recent DART [A1] based approaches? Do we really need to keep all the branches in the routing block? If not, can we simply perform branch selection and re-training after the selection is done?\n\n[A1] DARTS: Differentiable Architecture Search. ICLR 2019.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciate the authors discussing the limitation in terms of extra parameters used in the proposed routing approach. However, without addressing this issue, it is hard to justify why the proposed routing method is more preferred compared to the recent architecture search methods that performs branch selection and then re-training.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "D2MdtMb3iXh", "reviewer": "Reviewer_Qanu", "summary": "This work proposes a nonparametric method for routing in convolutional network image classifiers, where routing is the selection of a subset of modules to execute on a given input. Routing by memory associates each module with a feature, the \"memory\", that is scored against the input feature to select the module with the nearest memory. The routing is done at each stage of the network, and selects a single memory and its associated module, so the resulting architecture is always a chain from input to output (and not more general DAG). The memories are updated by the mean of the features assigned to it as a moving average during training. The combination of nearest neighbor matching and momentum updates yields a routing rule with few parameters, as it has only those of the embedding for memories and an optional squeeze-and-excitation-like adaptor for each module. This approach does not require reinforcement learning or stochastic optimization tricks like the gumbel-softmax to learn the representation or routing. This routing scheme requires a phased training approach: 1. learn a feature representation with a standard network, 2. widen the network by duplicating the modules in each stage, 3. cluster features to initialize a memory for each module, 4. continue updating the modules by gradient descent and the memories by clustering and momentum. In total, this routing scheme increases the number of parameters without increasing inference computation time. Experiments on standard architectures, specifically VGG-16, ResNet-{18,50}, and EfficientNet-B0, show higher accuracy with the proposed routing than without. Accuracies improve by 1.5-4 points with 2x-8x the parameter count. Ablations examine the methods parts and hyperparameters on the minor Tiny ImageNet dataset: both the memory routing and squeeze-and-excitation-like adaptor help and the number of parallel modules for routing, batch size, and momentum for memory updates are all effective hyperparameters. No experiments compare against routing baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The baseline architectures are diverse: VGG, ResNets, and EfficientNet(-B0).\n- The choices of number of modules (PUs), batch size, and momentum over the nonparametric parts (memories) are ablated.\n- The proposed method is essentially \"free\" at inference time in terms of FLOPs, since only one module is routed at each depth. (However, there is no GPU runtime profiling. Often FLOPs reduction does not convert to time reduction.)\n\nWeaknesses\n- No competing method is compared against. Instead, all results compare standard baseline networks with and without the proposed neural routing by memory. Comparisons are promised in the supplement, but only one of many methods is reproduced for fair comparison, and the accuracy on this single experiment differs by only 0.5 point.\n- Routing is restricted to a single choice of procedure, so routing always results in a \"line\" architecture that cannot run more than one module in each step. Other work, such as CondConv, has \"soft\" routing that can make use of multiple modules at the same depth.\n- The training is more complicated, in that it has to be divided into four stages. Other routing methods, like MoE methods, can simply be trained in one go.\n- Claims are not justified. \"Routing is based on non-parametric nearest memory\" does not imply the method is \"easy to train\" or prove that alternatives require \"complicated decision networks\" CondConv [36] for example is quite simple, and does not require multi-stage training. For the ease of training, it is unclear a priori if routing by nearest neighbor will be easier or harder to train, since the nearest neighbor could switch from one iteration to the next, which might radically change gradients. This claim needs an experiment that measures training, by for instance examining the entropy of routing decisions and the task accuracy across epochs.\n- Experiments are not controlled. The accuracy improvement is potentially confounded by (1) change in batch size and (2) momentum updates to the memories in the RMN. Batch size is well-known to be a sensitive hyperparameter, and momentum updates/weight averaging are both the subject of research for improved optimization (see Polyak averaging, weight averaging as in \"Averaging Weights Leads to Wider Optima and Better Generalization\" UAI'18, etc.). Although larger batch sizes rarely help in this regime, it nevertheless needs confirmation for controlled experimentation.\n\nRelated Work: Dynamic/predicted parameter methods [4, 5, 36] are discussed but not compared against. Why not? They share the goal of increasing parameter counts to thereby increase accuracy while controlling the amount of computation for inference. Note there are more such methods in existence, like Dynamic Filter Nets [NeurIPS'16] and Involution [CVPR'21, after submission deadline].\n\nClarity: The vocabulary of the paper is unique to this work, and so presents an unnecessary barrier to comprehension. To give a few examples: \"procedure\", \"stem network\", \"conditional\" attention. Is a procedure in any way different than the more common usage of the word \"block\"?\n\nDecision: The contribution cannot be gauged as a research or engineering contribution  without comparison to related work. The experiment design throughout this work denies comparison by only evaluating totally standard baseline networks and the proposed routing scheme. That is, no alternative routing scheme is tried. Nor are other interventions increasing parameter count and/or batch size tried as controls for two main effects of the proposed routing scheme. These empirical limitations along with missing related work drive the decision for rejection.\n\nFor Rebuttal:\n- Please compare results with MoE and related methods, such as CondConv, in order to have relevant routing-based comparisons.\n- Please report results of control experiments on batch size and weight averaging/momentum on the parameters of standard networks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of increased parameter counts and complicating training are addressed satisfactorily in the conclusion. This work has no particular potential negative social impacts, only having the general possible impacts of an alternative deep network architecture for visual recognition, and so no special comment is needed.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has received 4 expert reviews, with a single reviewer providing a slightly favorite rating (but a less informative review), 
and 3 extremely critical and informative reviews. The reviewers raised the following points:

= Unconvincing evaluations, as comparisons are done with own baselines only, i.e. without the contributions of the paper, and no comparison with a competing method is in the main paper. One comparison was discussed in the discussion phase, after reception of the author's response, but it was found to be unconvincing. As a summary, the results in accuracy are quite close and only one method is reproduced by running the official code.

= The advantages of the method are unclear wrt to the state of the art

= Experiments are not controlled.

The author's response has been found to be handwavy in many respects, iterating claims of the paper instead of attempting to answer the issues raised by reviewers.

The AC judges that the paper is not yet ready.
Decision: Accept (Poster)