Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper investigates multi-branch network structures from the aspect of information/feature space disentanglement, aiming to represent each class through certain path in a multi-branch network. To achieve it, the authors propose to assign classes to subNNs via a coding scheme. Output activations are calibrated by the aid of energy normalization which is regularized with a coding loss and a dropout operation. Experiments conducted on multiple datasets show the ability of the method on improving ResNeXt accuracy as well as the subNNs to the assigned class set.
strengths_and_weaknesses: Strengths:

1. The motivation of disentangling feature space of multiple classes in multiple branches is interesting and novel.  Several reasonable strategies are developed to realize the purpose, i.e., energy normalization to stabilize the flow energy in the network, coding loss to surpass the inactive subNNs, as well as dropout on subNNs to reduce feature correlation.

2. The overall framework is simple and easy to implement. 

3. The paper is well written and easy to follow.

4. Some empirical analysis on the ability of subNN specialization is provided beyond classification accuracy comparison. 

Weaknesses:

1. The method assigns classes via coding vectors while indeed apply a random assignment without consideration of either semantic correlation or visual similarity between classes, which is a bit counterintuitive. A family of related work investigate and leverage the class correlation to improve classification performance, e.g., [1] and [2]. The authors need to discuss the insight behind random assignment and analyze the relationship to these related works. 

2. Some important ablation studies are missing in the experiments. Please refer to the ``questions”.

3. It seems contribute little to interpret representation learning in a multi-branch network. If the focus is instead the ability of pruning network to be a binary classifier, its generalization performance in downstream tasks (e.g., detection or few shot learning) is important to verify the reasonability of feature space splitting.

[1] J. Deng et al., Fast and Balanced: Efficient Label Tree Learning for Large Scale Object Recognition. In NeurIPS 2011.
[2] J. Deng et al., Large-Scale Object Classification Using Label Relation Graphs. In ECCV 2014.

questions: 1. Inputs go through different subNNs in a soft-combined manner, i.e., no hard selection of certain path for each class is performed during inference. How is the performance if using hard manner instead? And how is the performance compared to assigning classes in semantic or visual correlation (which can be computed via other pretrained networks)?

2. It would be valuable to provide activation distribution of different classes on subNNs for better understanding the behavior of the method.

3. Since this method enforces the separation of features shared between classes, have the authors tested if it can improve performance on adversarial examples?
limitations: The insight behind the method needs to be further analyzed and empirically studied.  The method aims to improve multi-branch networks, which is not restricted to certain downstream cv tasks. It seems like no potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: An algorithm is used to construct a coding scheme per-class over resblock subNNs, with a sparser coding scheme in later layers.  An auxiliary loss is used to encourage the activations to match the *fixed* coding scheme.  This specialization can then be used to construct binary classifier sub-networks from a trained model.  It also improves overall accuracy.  I think these results are interesting and on the whole the paper is well written, but the nature of the algorithm could be presented more clearly from the beginning (especially to give a flavor of what the coding scheme is).  

Notes: 

  -Interested in interpretable feature.  Encourage class-specific information to flow through specific paths.  

  -Each path can then be used as its own lightweight binary model.  

  -Many architectures have a multi-branch design, such as inception, resnext.  This is usually justified in terms of downstream performance rather than specialization.  

  -Binary codeword equal with length equal to number of branches is matched to each class, indicating branches to activate per class.  

  -Possible to make half-size model which works per-class.  

  -Intermediate layers can make early predictions.  

  -Encourage zero energy for the de-activated blocks.  

  -Fixed sparsity level over the different codes (such as 3/10).  Earlier layers have less sparsity than later layers.  
strengths_and_weaknesses: 
Strengths: 


  -The inactive block removal (Figure 2a) is quite nice.  


  -The construction of the sparse binary classifier is also nice.  


Weaknesses: 


  -After checking appendix, the coding scheme seems to be a fixed and not data driven algorithm.  While the full algorithm can be in the appendix, I feel like a high-level description of how this works should be presented much earlier in the paper.  


  -The writing of the paper could do a lot more to explain how it works early on.  


questions: 
  -Scaling to very large number of classes?  

  -The way that coding is induced still feels a bit ad-hoc, for example the use of 4th power in equation 5.  Does it not work to optimize something like a binary gumbel-softmax for each code and then try to encourage only a few to be active for each class?  

  -I'm still a bit confused about how the coding scheme is set per class.  Is it random?  If not, is it some kind of top-k sparse softmax with learned parameters per-class?  

limitations: Limitations and ethics are well-discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes Coded ResNext for interpretability in the classification task where each subNNs are dedicated to specific classes. Each class have distinct binary codes and subNNs are trained by the coding loss with binary codes. Each subNNs can be used as a binary classifier and early prediction is possible. In Cifar-10, 100, ImageNet, Coded ResNeXt outperforms ResNeXt.
strengths_and_weaknesses: **Strengths**
- Coded ResNeXt idea is very novel and interpretability in Coded ResNeXt was quite interesting. (I'm not the expert in this area. Therefore, I have to check the other reviews.)
- SubNetworks trained in the coded ResNext can be used as a binary classifier and early prediction is possible with the intermediate layer results. 
- There are many attractive properties. 

**Weaknesses**  
- The main concern is that the main architecture is restricted to ResNeXt. The alation studies on the other network should be included. Even though the justification for ResNeXt is provided in section 4.3.1, I believe other multi-branch networks which aggregate the outputs by summation can substitute ResNeXt. Sub blocks for ResNet, MobileNetV2, and Efficient Net (MBConv) might substitute subNN of ResNeXt.
- The performance of this method seems to depend on the binary code. An ablation study for this randomness on the binary code for classes is required to support the superiority of this method.
- In section 3.2.1, the authors state that the coding scheme is an NP-Hard problem. However, this kind of problem can be polynomial as in [1] where the sparsity of each binary code is fixed and the pairwise term which corresponds to the hamming distance sets the binary codes apart as far as possible.  Please check section 4.2.1 and section 4.3 in [1]. 

[1] Efficient-end-to-end learning for quantizable networks
questions: Please check weaknesses. 

1) I believe the justification of ResNeXt by the authors is not enough and an ablation study is required.

2) Randomness on the binary code should be analyzed. 

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a novel way to guide information flow through a network allowing different paths to be followed for different predictions.  In the presented instantiation of the idea, the paper uses the ResNeXt network that follows different and disentangled paths for different output classes. The paper highlights the use of such a network for interpretability, as well as computational savings that can be obtained via this process. The results show that the network can achieve disentangled information paths without sacrificing any accuracy in the ImageNet-1K and CIFAR datasets. 
strengths_and_weaknesses: # Strengths

- **[S1]: Original, Interesting Idea:** The main idea in the paper is original and very interesting. Without using semantics from any classes, the proposed work can guide the information flow for different classes through different subnetworks without sacrificing overall accuracy (Though for this POC, lots of tricks need to be applied). This is a very significant innovation.

- **[S2]: Significant to a broad audience:** The paper highlights different ways where this might be useful; This reviewer agrees with all of them, and actually thinks that the significance might be broader still. A strong contender is out of distribution generalization and mitigation of spurious correlations. The paper notes that 

> Interestingly, when removing inactive ones [sub-networks], the accuracy tends to increase.

I find this result to be highly encouraging towards better disentanglement, OOD generalization, and mitigation of spurious correlations. Similarly, the computational savings aspects of this network are also not sufficiently addressed here (not a weakness, just a note), but a combination of early exit possibility with well-learned subnetworks, can make this a powerful tool for extracting a single-purpose lightweight network out of a bigger network. Finally, as noted by the a paper, actually exploiting similarities/semantics/other information about the classes (which this work does not) can further improve this method in all fronts including better disentanglement, OOD generalization, accuracy etc. 


# Weaknesses

- **[W1]: Some presentation-related woes:** There are some issues with clarity. Especially, I found the discussion coding schemes construction a bit hard to follow. The effects of r_l and different choices for it (including extreme cases, where a high r_l is used for most layers -- with the extreme case being the regular ResNeXt, and/or low r_l used throughout) are also not fully explored. Also missing is a bit more thorough analysis of specialization as a function of depth. However, all of these are "nice to haves" and does not really detract from the main contributions of the paper. 

- **[W2]: Some breadth missing in literature review:** While the idea presented is undoubtedly novel, I do think there are similarities with adjacent areas of research, especially in out-of-distribution literature. I have listed a few below:

    - R1: Early exit networks: https://arxiv.org/abs/2004.12814 
    - R2: Early exit networks used for different information processing for different samples: https://arxiv.org/abs/2204.02426
    - R3: Subnetworks used to improve OOD detection: https://arxiv.org/abs/2106.02890


# Overall

Overall, this is a very exciting work; While the scope of experiments is a bit limited (I'd have like to see more, as noted above), this is hardly an actual weakness, just something the authors/others can expand in the future. Overall, the successful demonstration of disentangled information flow for different classes is already exciting, but the potential for this idea in other applications (noted above) is also equally exciting. I would definitely like to see this at the main conference. 

questions: # Comments

C1: I would have loved to see a bit more analysis on how the disentanglement in the subnets translates to disentanglement (if any) in the feature space. Even a qualitative example by selecting a few distinct classes, and higher-level features that are shared/different for those classes would be a welcome addition. 

# Questions

Q1. Could different coding schemes be used to allow networks to learn subnets based on other factors (e.g., metadata, additional information, hierarchical classes, etc. ). How difficult is this?

Q2: What is the effect of a larger/smaller amount of networks shared with all classes? Can you share any insights about what this "shared" portion learns?

limitations: The paper adequately addresses limitations.
review_title: null

Meta-review: metareview: The authors propose a modification to ResNeXt where each sample is routed to a subset of the network. The aim is to activate only a subset of the network (subNN), in fact extract a binary classifier for each class from the trained larger network, with a significantly smaller parameter footprint. The main idea is conceptually simple -- in each ResNeXt block, pick a subset of paths for each class (e.g. 3 sub-blocks for each class) precomputed in a data-independant way via a simple binary coding scheme. To ensure that these blocks specialize to a given class a "coding loss" is added to the classic cross-entropy loss. The coding loss forces the mean energies of the subNNs (sum of squared activations, summed across C, H, W) inactive for class k to zero and those of the active subNNs to positive values. In addition, the authors add a stochastic dropout-like operation where the output of a subNN can be zeroed-out. The authors empirically validate the idea on multiple datasets and show that (1) It can improve ResNeXt accuracy, and (2) that competitive class-specific binary classifiers can be extracted post-hoc. The ablation studies clearly demonstrate that a high degree of specialization is enforced.

The reviewers appreciated the novelty and the conceptual simplicity. The method also seems practically significant, especially if it can be extended to other neural architectures. The ablation studies were well-received. After the discussion phase there are still some questions related to the rather specific choice of the 4th power for the coding loss, as well as positioning with respect to works building on the early exit ideas. Nevertheless, the reviewers agreed that this idea is novel and interesting for the larger crowd and seems practically significant. Please update the manuscript as discussed.
confidence: Certain
Decision: Accept