Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Parameter sharing can reduce memory footprint of neural networks and memory bandwidth requirements, but existing methods require manually tuning the sharing strategy. This paper uses a small phase of training to cluster the learned layer representations by groups. This allows networks to be scaled from small to large parameter counts (to be clear, number of trainable parameters) without changing the model architecture. Importantly, this procedure does not change the number of FLOPs in the model. 

Experiments across a wide set of tasks and networks compare this approach with either (1) SWRN from Savarasese & Maire, 2019), or (2) existing hand-tuned parameter scaling from families of networks such as EfficientNet, DenseNet, or ALBERT.


main_review: Strengths

I commend the authors on extensive evaluation across multiple datasets, models, and tasks are convincing of the broad applicability of this approach. Not an expert in the literature, but during my investigation, the PNAS approach seems like a reasonably novel mechanism for parameter allocation and automatically learning which weights to share. Methodologically interesting approach, with implications for studies on generalizability. 

Weaknesses

The practical impact of this approach is overclaimed in this paper. For example, while the SSN reach lower error compared to the EfficientNet family as the same # of trainable parameters, the EfficientNet family (and ALBERT, WRN, etc.) are designed to have lower inference time, which SSN does not achieve.

Figure 1 marks both the author's method (NPAS) and previous work in cross-layer sharing (presumably Savarasee & Maire, 2019) as "make training more efficient", but the effects are dramatically different. NPAS's efficiency is in lower memory footprint, whereas prior work reduces the FLOPs and therefore the training time (Table 4, Savarase & Maire, 2019). 

While shared parameters does reduces the communication during data parallel training, the authors provide no data or reasoning for the magnitude of that effect on training time. Given that CNNs already have relatively low parameter count, I suspect that the effect on the model sync time during distributed training is minimal. 

A meta-comment: parameter counts may be the wrong metric to compare model capacity in these studies, as the methods you compare against (e.g. the EfficientNet_B0 family) have reduce parameters by removing from network, whereas NPAS reduced by sharing weights. For example, would path norm be more appropriate from a generalization studies perspective, as hinted in your conclusion? This is admittedly out of scope for this paper, as the paper's main claims are on efficiency.

My score could be improved by:
* Figure 1: breaking out "makes training more efficient" into "Reduces FLOPs" and "reduces memory footprint" rows, and marking the methods appropriately. Citing the paper for prior work directly in the column or the caption.
* In the discussion of results, acknowledging that these other methods reduce FLOps and training time, whereas the NPAS approach does not.
* Providing data on the actual effect of this reduced number of trainable parameter counts on (1) model training times in a distributed training setting, and (2) memory usage on the GPU. Are these savings practically realizable?
summary_of_the_review: The proposed method is methodologically an interesting contribution, however the practical impacts on training efficiency and model efficiency are minimal and overstated, which limits my recommendation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to solve an interesting but meaningful task, i.e., learning to allocate parameters between layers; in other words, searching for the parameter sharing strategy in a neural network. Parameter sharing between serial layers is useful for saving parameter numbers, which is beneficial for issues including memory consumption, bandwidth, etc. 
main_review: The topic introduced in this paper is interesting and meaningful, which provides a novel idea for saving parameters. The proposed parameter upsampling and downsampling methods are promising.

Here are some questions to be answered.
1. The parameter number is indeed saved. However, how about the real performance on the hardware platforms, e.g., bandwidth, memory consumption, training speed? It is essential to report these results for evaluating the real value of the method.
2. How much benefit does the NPAS method bring? It is necessary to perform an experiment similar to "random search" in NAS to validate the effectiveness. Noting that in Fig. 4 (b), differences between the manual and auto one are not so evident, where it seems only the downsampling layers require unique parameters as in the manual one.
3. How are the parameters allocated in depthwise layers, e.g., in EfficientNet? Is there any difference between depthwise and plain convolutions?
4. Is the proposed method specified to tasks? It is interesting to explore whether parameters are allocated differently in different tasks.
summary_of_the_review: The topic is interesting but the concerns listed above need to be cleared. The main concerns lie in the real-application value and real benefit of the methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work investigates how to automate allocating parameters within a fixed budget to each layer and generating its weights with assigned parameters, which is a generalized version of parameter sharing. The proposed method is validated with leading performances and efficiencies on multiple datasets across multiple tasks.
main_review: #### **Strong points:**
1. This work is well-motivated. Parameter sharing is one of the most promising directions to reduce training and inference costs, but not addressed in AutoML community. So, the aim of the work is timely.
2. The proposed method is effective and efficient. Also, the authors validate the proposed method across not only multiple datasets but also multiple tasks. Moreover, the authors show that other techniques to reduce the inference cost are orthogonally applicable.

#### **Weak points:**
1. The proposed method contains many hyperparameters to be tuned, such as the window size, embedding dimension, and # partition groups. How sensitive is the performance of the proposed method to them? Also, do all tasks and datasets share the same set of hyperparameters?
2. The proposed method (LB-SSN) seems to reduce only the number of parameters. However, in the practitioner’s point of view, reducing inference time may be more critical. Are other pruning methods also applicable to the network configuration obtained by LB-SSN? It would be great to add the result to Table 4.

#### **Detailed comments:**
It is not easy to understand the proposed method with Figure 2 at a glance. Simplifying Figure 2 or replacing it with an abstractive algorithm description may be better.
summary_of_the_review: This work is well-motivated, and the experimental results support claims well. However, major concerns lie in the practices; the proposed method additionally introduces hyperparameters and cannot help practitioners reduce inference time.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method to automatically select parameters to share between layers.
It proposes to use a shape shifter network to either increase or decrease the number of parameters in the model.
The parameters are mapped into parameter groups through a preliminary training step and k-mean cluster the layers.
Layers in the same group share parameters. It will generate weights by downsampling or upsampling depending on the layer needs.
The method is tested in Low Budget and High Budget regimes and on different tasks.
It also shows that the method can be used together with distillation and pruning.

main_review: Figure 1. shows comparison between distillation and pruning which seems to be a different class of methods. It is mentioned to be complementary to NPAS. It would be better to show competitive analysis on prior work or similar approaches.

Slimmable networks is also a work presents a network that change parameters sizes automatically

NPAS adds training cost per epoch, but the parameter sharing enables faster convergence. Demonstrating that the effective train time to reach same accuracy would improve the paper.

The explanation flow in section 3 seems non-intuitive, because the method uses section 3.2 first and then it uses section 3.1.
Adding pseudo-code or algorithm would also improve clarity of the method.

page 8, typo: covolutional


summary_of_the_review: The paper demonstrates SSN and NPAS and presents experimental results on various benchmarks.
The text could be improved for clarity and organization.
review_title: null

Meta-review: The reviewers were mostly concerned about the practical impact/implications of the proposed methods. There was a long discussion across multiple threads of the benefits of the approach proposed in CNNs vs larger language models, dissecting the benefits in terms of training time (as opposed to memory or FLOPs, which may have a non-linear impact on running time). Overall, the authors did a good job of putting their contribution into context and addressing the reviewer concerns.
Decision: Accept (Poster)