Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a new routing method for MoE. The main idea is to construct multi-convolution blocks (called procedure in the paper) and assign similar features to the same blocks. It can improve the accuracy while not increasing the computational cost. Practically, it introduces memory features with the nearest neighbors algorithm in the routing process of MoE. It updates memory by moving average fashion in the training phase.

Compared with existing MoE methods using sub-network in the routing process, the proposed routing mechanism is non-parametric, efficient, and explainable. And it achieves convincing experimental results on computer vision tasks, including image classification, semantic segmentation, and face recognition.
main_review: Strengths:
1. The proposed memory-based routing mechanism is novel and exciting. It offers a new non-parametric expert selection way to MoE. Compared with existing methods, it is more efficient and explainable. Besides, its network is easier to train using a standard gradient descent algorithm.
2. The idea of assign similar features to the same experts is simple and allows explainable routing results while existing methods generate routing results in a black-box fashion. This paper may open a new line of research, non-parametric routing, into MoE.
3. The experimental results are convincing. The proposed method can improve the baseline models' results on several image classification benchmarks and downstream tasks (semantic segmentation and face recognition) while not increasing the computational cost. The authors also compare their method with RNR and DeepMoE, two related MoE-based CNN methods, and show superior results.
4. MoE got great success in the field of NLP in recent months, but few researchers about its effectiveness in training CNN models. This paper provides a good try and may raise many research interests.

Weaknesses:
1. Due to using multi-experts, a large number of parameters and training batch sizes are two common weaknesses for MoE methods. Though these weaknesses are acceptable for MoE methods, similar to existing methods, this paper still does not provide a practical solution and just claims that these two weaknesses are not critical or solve them using other techniques.
2. The number of epochs in the first training stage (stem network training) should be an important hyper-parameter. It directly affects the memory initialization. But no experiments about this hyper-parameter in the paper.
3. The paper properly discusses the relations to existing MoE-based CNNs. However, there are few discussions about MoE-based NLP methods while MoE is more popular in NLP. The author should add some related NLP methods to the section of related works.


Suggestions:
1. Due to the global average pooling, the authors use a vector as the memory. Thus, the proposed method can also be applied to many NLP models (e.g., rnn-based and transformer-based models). The author may have a try to see if it also work well in NLP.
2. In memory, the authors use global average pooling to reduce costs. But it will erase some spatial information, which should be critical to computer vision tasks. The author may try different pooling sizes, like 2x2,4x4, and 8x8. The costs of them are also little.
3. In recent months, MoE methods successfully trained huge and state-of-the-art NLP models (e.g., Switch Transformer). Though they are Arxiv papers, adding discussion with them may improve the paper.
4. Since there is only a single feature in each memory, the memory may difficult to represent the underlying structure of the overall feature space, especially for ImageNet with diverse images. The authors may try multi-features in memory.
5. Visualizing attention maps learned from the CA modules is helpful to understand the motivation of CA.
limitations_and_societal_impact: Please see Weaknesses and Suggestions in the Main Review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Routing strategy is used to select process units in convolutional neural networks. Memory is one interesting idea, which is used to record the latent information of inputs. Because different samples have different memory footprints, routing-by-memory provides one optimized execution path。
main_review: Strengths:
1. The work provides an interesting idea named routing-by-memory. The memory guides different blocks tackle different features in a divide-and-conquer fashion.
2. The experimets show the effectiveness of the method.

Weaknesses:
1. The meaning of memory in CNNs are poorly explained. Which kinds of information are recorded by the memory? Although some visualization are providied in the supplementary material.
2. The number of process units for different stages are same. Will the results be imporoved if different stages have different number of PUs?
3. The proposed method leads  to minor increase of FLOPs and major increase of memory.
limitations_and_societal_impact: Yes. The main limitations of our method are extra parameters and larger training batch sizes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a dynamic network mechanism, routing-by-memory mechanism, to existing CNN networks for improving network performance with negligible extra computational cost. This mechanism adds multiple parallel Procedural Units (PUs) in each stage of the network, where each PU consists of a memory (a representative feature) and a procedure (some conventional blocks). By employing the memory for similarity calculation, the network can adaptively route different samples to different PUs in each stage to tackle various samples better. The experimental results on three public benchmarks demonstrate the effectiveness of the proposed dynamic routing mechanism.
main_review: Strengths:
	This paper presents a new dynamic network mechanism, routing-by-memory mechanism, which is inspired from the classical Mixture of Experts idea, and its effectiveness can be validated by the experimental results on three public benchmarks.
	The paper is well written, and its structure is organized clearly.

Weakness:
	Considering the novelty, the presented dynamic routing mechanism itself is not quite novel. There already exists some literature about dynamic network structure based on each input sample with similar Mixture of Experts idea [1,2,3]. The authors need to provide more comparison to these work, and demonstrate the advantages of this work compared to previous ones.
	As for the CA (Conditional Attention) module, there is some ambiguity about the mechanism of this module. As claimed in the paper, the same channels of features outputted from different PUs can capture different semantic meanings, which can increase the learning difficulty. However, the CA module is running in each PU separately, so how does this module help relieve this feature learning inconsistency problem. The authors are expected to provide more analysis into this module.
	In the paper, the memory is initialized by clusters’ center of intermediate features, and the paper claimed that in this way, different memories can dominate different kinds of features in the same stage. However, this claim should be validated by empirical results. Authors can provide some visualization results or data analysis to show that different PUs in each stage can be specialized to tackle different kinds of input samples with the help of different memories. For example, the authors can show that whether the images in the same category are more likely to be assigned to the same PU in each stage. More experimental results like this can help validate that the trained PUs are specialized experts for certain functions, which is important.

[1] Shaofeng Cai, Yao Shu, and Wei Wang. Dynamic routing networks. In WACV, 2021
[2] William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers:Scaling to Trillion Parameter Models with Simple and
Efficient Sparsity. arXiv e-prints, 2021
[3] Huaxiu Yao, Yingbo Zhou, Mehrdad Mahdavi, Zhenhui Li, Richard Socher, Caiming Xiong, Online Structured Meta-learning, in Proceeding of the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020), virtual, Dec. 2020



------- 

The authors' rebuttal addressed my comments. Now I would like to upgrade my grading. 


limitations_and_societal_impact: NIL
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose to use clustering of features and nearest neighbor routing to create a modular network architecture. This greatly simplifies the routing algorithm, which usually relies on complicated, learned mechanisms. It improves the accuracy of various networks on several image classification benchmarks.
main_review: To my best knowledge, this routing method is original.

### Significance
The significance of the method is threefold:
 - It greatly simplifies the routing mechanism. Usually, modular networks require RL, EM, or other non-trivial training methods to train the router. This paper sidesteps that elegantly.
 - The method creates new modules by cloning the existing ones and making them specialized. This can be easily extended to dynamically creating modules in the course of training.
 - It offers an easy performance boost on certain tasks

### Positives
  - The method is simple and general
  - The method might be useful more generally, for example for dynamically creating modules.

### Negatives
 - The effect of the increased batch size is not measured on the baselines. Thus, it is not clear how much of the gain comes from the increased batch size.
 - The authors report only a single seed per experiment, without standard deviations.

### Summary
The idea is nice, but I think the current version of the paper seems to be rushed and it does not meet the bar yet. I'm willing to improve my score if the above negatives are addressed.

### Clarity
  - Some parts of the text are hard to read and required me few re-reads to understand.
  - Please have your submission proofread for English style and grammar issues.
  - Line 46: "we split the network into different stages by downsampling layers". This suggests that you are designing the architecture and use the downsampling layers for splitting. But you are modifying the existing architectures. Does this mean that you are using the existing downsampling layers as boundaries of the processing units?
  - Line 57 and 212: from the text it seems like you are using a fixed, relatively small number of epochs for the stem network training stage, but you mention early stopping here. Please clarify what do you mean by "(i.e., early stopping)"
  - In formula (6), how is the selected module (R(m_i, f_{i-1})) represented? Is it one-hot?
  - Line 218: "we also clone the momentum" - is it more generally the optimizer state (for example for RMSprop experiments) or just the momentum? Please clarify. If the latter, please clarify why just the momentum.
  - Line 281-283, 310-311: why do you multiply the batch by N/2 and not N? That would result in the original batch size in expectation for all modules (if we assume that the clusters are equally likely). You also talk about imbalanced data in this sentence. Can you please clarify?

### Redundancies:
 - Line 162, 165: redundant description of the GAP
 - Section 4.2.3 is redundant with the paragraph "Momentum in Memory Updating" paragraph of Section 4.1.3

### Comments regarding the text
  - Line 21: the "sensory processor" should not be considered as a part of "human memory"
  - Line 37: increasing the number of procedures?
  - Line 35-18: the end of this sentence is hard to understand
  - Line 40: does not bring - does not require?
  - Line 51: I found calling the Squeeze-and-Exite module [14] as CA module confusing.
   - Line 54: use a colon instead of a comma after "four training steps"
   - Line 56: by -> for?
   - Line 61-62: consider clarifying "continue the network training as the first step in the routing-based training step"
   - Line 128: "mainly" - the paper only focuses on image classification, so please remove
   - Line 140-141: sentence hard to read
   - Line 158: we use *the term* N-way?
   - Line 190: scalar vector?
   - Line 325: input's

### What could make the paper nicer - optional
 - A bit more analysis. Maybe analyze how active are the modules for the target classes to see how they map to semantical categories
 - Analyze (and show) the usage statistics of the modules. Are they some which dominate or some that die out? How correlated are the module selections of different stages?

### Citations
  - Regarding the CNNs, please consider citing some earlier work, for example, LeCun et al: Handwritten digit recognition with a back-propagation network
  - The first paper successfully training very deep CNNs are Highway Networks from Srivastava et al
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper presents a new dynamic routing-by-memory mechanism for MoE which is simple and elegant.   The mechanism effectiveness is validated by the experimental results on three public benchmarks. 

All the reviewers agreed the proposed memory routing is of interest to community thanks to its simplicity and good empirical results.


Decision: Accept (Poster)