Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose ParaDis, a parallel and distributed version of universally slimmable networks. They use the fundamentals from this work, plus a few additions, to train models that can be switched between different parallel factors with only minor accuracy degradation. They evaluate using ResNet and MobileNet on ImageNet against models trained solely for a given configuration to show that their technique matches and sometimes exceeds this baseline accuracy. They finish the paper with an ablation study that examines the impact of their additions to the training procedure. 
main_review: === Strengths ===

Figure 1 was very helpful in giving context to different types of distributed inference. Yet, on my first reading it wasn't clear whether the image was split spatially or channel-wise. This was clear though given my previous understanding of slimmable networks.

The ablation study was useful in confirming the importance of IPKD-A on top of US networks.

The inclusion of IPKD-A seems like a strong modification from the US networks.

The paper honestly discussed its limitations and difficulties scaling up to 8 switches.

=== Weaknesses ===

The major evaluation figures seem to not be highlighting the correct aspects of the method. In my understanding, this method should strive to maintain the accuracy of the single configurations as much as possible, while improving the model latency through parallelization. This is shown in the figures. Then, the comparison to US networks could either show improvements over previous work, or equality with improvements in latency or other aspects. This part seems to be lacking and currently from the evaluation itself it isn't clear why ParaDis would be preferred to US networks. US networks would also have decreased latency from their reduced width.

Ideally, the evaluation should include latency results since this is the biggest advantage of parallelization. This isn't directly captured by the FLOPS.

The ImageNet results are important, but there should be at least one table confirming that ParaDis works across other architectures. I imagine that the "information split" that ParaDis causes could have widely different effects in different networks. Residual branches, bottlenecks, and pointwise convolutions could possibly affect this.

Given the relatively poor single configuration results on some switches, these data points may want to be run multiple times and listed with error bars. It is hard to believe that cutting the network in half improves the ResNet accuracy up to two points.

I struggle to understand the statement: "within ParaDis networks, there might be potentially much more possible switches than in slimmable or US Nets." In US Networks, there is an ordering on the channels so that lowest index channels are the primary ones. This leads to up to C different configurations, since each switch only adds an extra channel. It seems like this should be true for ParaDis too, where the leftmost channels of the network are considered primary. If the network has 4 channels, then the possible configurations are [1], [.5, .5], [.5, .25, .25], and [.25, .25, .25, .25], and [.25, .25, .5] for example would not be valid. In that case, ParaDis also has at most C switches in general. In reality, just as in the US paper which limits the minimum channels to .25C, there would need to be a minimum sub-model size, leading to fewer than C switches.


=== Questions ===

Why is the [.5, .25, .25] configuration left untrained? I assume this was to demonstrate that it doesn't have to be explicitly trained to perform well, as long as its submodels are trained in other switches.

Other types of networks were mentioned in the related works, e.g. early exit models. How do these other types compare to ParaDis in terms of accuracy at different latencies?

If ParaDis and US Networks perform similarly for given compute, what situations would favor ParaDis models?

How are residual branches handled during the splits in ResNet?

What are the tradeoffs of updating all the switches vs. a subset of the switches per epoch? Did you choose to update all because in this case there were only 4 (or 3 if one of them wasn't trained)?

For the negative results for 8 switches, what is the minimum size of the channels in the split? I would be worried about the bottleneck of ResNet being split too much for example. Do these issues go away with a larger scaled width?
summary_of_the_review: This paper provides an interesting variant on the idea of slimmable networks that allows the technique to be applied to distributed inference. The idea is well-motivated and described in detail, yet the training procedure seems slightly ad-hoc and the evaluation falls short in multiple ways. The paper needs more baseline techniques, more model architectures, and larger parallelization factors for me to fully support acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed to train several parallel sub-networks in the slimmable networks framework, which could run in parallel on different devices subject to runtime hardware configurations to reduce latency.
main_review: * Strength: Similar to slimmable networks, this approach could train all the networks at the same time and retraining is not necessary when deploying.
* Weakness: This approach fails to address several important problems in distributed embedded computing and the detailed advantages on hardware latency is missing.

* Reasons to accept:
  * The parallel distribution to speed up model deployment can help reduce latency.
  * The approach to train the ParaDiS network is computation-efficient and the performance is good.

* Reasons to reject:
  * Only a few experiments have been performed on classification task, without covering detection, segmentation etc. The ablation study could not show significant advantage of distillation.
   * Lack of specific information about how much latency is saved via parallel computing.

### Questions
* From Figure2, the second layer seems to depend on all the neurons of first layer, so how is it distributed in parallel? Is the computation done in a mixed serial-parallel style, or is the computation repeated?
* Many modern DLA have on-chip memory for storing intermediate results to save DDR bandwidth. Parallel computing across devices will introduce extra copies from and to DDR in this case. What's the cost model to decide when to distribute and when not to ?



summary_of_the_review: The paper improves upon the slimmable network framework by introducing parallel sub-networks to reduce latency. However, the improvement has not been well empirically evaluated and justified. The main discussed scenario is distributed embedded computing, but important topics in this scenario like mapping between different traits of sub-network and the capabilities of devices, has not been covered. Not to mention runtime behaviors like load balancing and allocation of shared resources like CPU and memory. The paper may need be expanded significantly to include sufficient contents on the topic.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an algorithm that distill from a larger teacher to a set of parallely distributable student models each of which only compute a fraction of channels in the network. The combined output of the student models can be used to compute the final class prediction.  In this way, student models can be deployed in a set of mobile or edge devices. 
main_review: Strengths
1. The experiment section is well written with clear description and thorough studies. The experimental results provide sound evidence to support claims made in the paper. 

2. The proposed distillation algorithm is described clearly and straightforward to re-implement. 

3. Described the corner cases like batch norm layers and proposed solutions to address them.

Weaknesses
1.  I hope this application of this work can be beyond the convolutional models in the image classification domain. Sharing all parameters across devices posed a serious constraints to those transformer-like models whose number of model parameters is much larger but not necessarily needs more compute flops per image.
 
2. The proposed framework required knowledge of all devices in advance before training. And it's not scalable when the list of switches is large. 

3. The practice of distilling a bigger teacher to the student model is well established in the literature (like noisy student). It's better to add more related work along this line.

4. Another interesting baseline will be an ensemble of smaller models, each of which will be deployed on a separate device. In this way, the whole ensemble network can still work even after the number of devices in the network changes.


summary_of_the_review: A careful study of distillation algorithm for the mobile/edge device deployment. The whole paper is well written with experimental results supporting the main contribution claimed by the authors. However, the significance of this approach is a bit limited due to the scalability issue (only works with convolutional models and  a very small set of switches.)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes ParaDiS, a slimmable network that can be executed across multiple devices, and be transferred across different devices without re-training. To train ParaDis, the authors propose to distill from a wider network, and combine activation knowledge distillation to improve the performance. While the idea of parallel execution is promising, there is little detailed discussion on that point.
main_review: 
Strengths:

1. It is a practical and helpful extension to execute the slimmable networks in a parallel way, which has the potential to leverage the computation power of multiple devices.

2. The paper is well written and easy to follow.

3. Comprehensive studies are listed to verify each component in the proposed approach.

Weakness:

1. Methodologically there is no fundamental differences from slimmable networks. The paper proposes 1) distilling from wider networks and 2) distilling feature maps, both of which are not new. 

2. There is little details on how to execute the parallel inference across multiple devices, which I think is a major issue of this paper. For instance:
- Does it mean there are three devices holding each part of the [0.5, 0.25, 0.25] configuration? If so, there is inevitable communication among devices.
- It would be more interesting to show the strategy to split the model across different devices, depending on their resources (memory or computation). 

3. No simulation results on parallel execution of ParaDis. I may still worry about the latency overhead comparing with the inference of a singe device, especially if different devices have different computation power, or there is any communication among devices.

4. It seems the width configurations are not scalable and sensitive to width configurations as discussed in Section 6. From to Figure 1, is it necessary that all neurons in the first width configuration are fully connected (in terms of the performance)? 


Detailed Comments:

1. It is still not clear to me the following statement: "Indeed, since the BN statistics are computed via a forward pass during the calibration phase, the statistics of a particular sub-model are independent of the switch it belongs to. In other words, BN statistics of the last 0.25-width sub-model will be the same, whether they are computed within [0.5, 0.25, 0.25]× switch or within [0.25, 0.25, 0.25, 0.25]× switch. To summarize, the calibrated BN statistics of a given sub-model are switch-independent."

As the last 0.25-width receives input from different paths of previous network, what does the word "independent/same statistics" mean? 


2. Figure 4: what about Slimmable and Slimmable Wide IPKD for MobileNet experiments?

3. What is the training cost compared with vanilla training or US training, if all switches are enumerated in each iteration?
summary_of_the_review: Marginally below the acceptance threshold.
review_title: null

Meta-review: This paper presents an approach for distilling a larger teacher model into a set of students that can run in parallel at lower cost. The main strengths are that the approach appears conceptually sound and reasonably well executed. The main weaknesses are that the differences relative to previous work is fairly slim, and the experimental results are overly idealized. While the main benefit of the approach is improvement in latency, the experiments evaluate in terms of FLOPS. There was some back and forth between the authors and reviewers about these points. Authors added some additional results and seem to acknowledge the limitations, e.g., saying “Our time and hardware constraints did not allow us to perform experiments in a realistic deployment.” However, for a paper primarily concerned with reducing latency, reviewers were unconvinced that this evaluation was sufficient.
Decision: Reject