Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors present a very interesting idea of training a large network and a small network simultaneously with an interesting new loss function. The authors show that this can lead to a much smaller network with good accuracy (compared to the original network); thus this may be a good technique for sparsification.

**Theory**

Please clarify what induced L2 penalty means. The use of primes to indicate differentiation was ambiguous. Please be explicit about what you are differentiating with respect to in the notation.

Can you motivate why we should focus on the second order term during training?

Also if at all possible would be good to show the relevant quantities during training to see if the intuition is correct.

**Experimental Evidence**

Error bars from multiple runs to get a sense of variation relative to the difference being measured in Table 1 seems critical. 

In addition, showing data from the sparsity literature on what other methods are able to accomplish (particularly weight magnitude pruning) would be very useful. For example, [Blalock and Gonzalez: What is the State of Neural Network Pruning?] seems to have data for ResNet-50, but it may not exactly apply, so the ideal thing would be to provide data by doing the baseline comparison yourself. (That paper also has a checklist of things to be careful of when doing pruning research.)

Also, a claim in the introduction is that training is faster this way since pruning and finetuning don’t need to be iterated. Would be good to back this up with actual numbers.    

**Writeup**

The write-up could do with some more attention e.g. use \citep instead of \citet in most places. The mask M should be a tensor not a matrix. Also in your context, “speed” could refer to training time or inference latency. Would be nice to use these more canonical terms instead of just speed.

**Future Work**

One question that occurred to me while reading your paper (particularly the theory section) is what happens if you do the joint training as you describe, but instead of the adjoint loss, you use some sort of regular weight loss and different amounts of regularization penalties for the common weights and the weights unique to the large model. Would that give similar results?

For practical use of course, ResNet50 may not be the best target for sparsification since EfficientNets and MobileNets are already so much smaller. Please see the [Blalock and Gonzalez] paper (Sec 3.3) for perspective on this. 

**Post Rebuttal**

Thanks for the response. I still didn't get an answer for why the second order term and not the first order term is of interest in the Taylor expansion. Also the paper does not seem to be updated with the promised changes, particularly error bars. So I am leaving my score as is. 
review_title: Promising work but some gaps

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors introduce the concept of "Adjoined Network" training, where a CNN network is trained concurrently to a much smaller network that is composed of a subset of the parameters of the larger network. The parameters are shared between the two, and the loss --aside from the standard cross-entropy of the larger network-- also incorporates the KL-divergence between the outputs of the two architectures. This is so the trained small network will be able to simulate the larger one with a fraction of the parameters, which makes for less memory and faster inference at deploy time.

The paper additionally shows that the concurrent training also acts as a regularizer for the larger network, and empirically demonstrates to be more effective than dropout.

In general, I like the general idea and how this adjoined training encourages the large network to be more careful at optimizing a specific subset of its weights. The improvement on the accuracy of the large network when trained adjoined is also a nice byproduct.

However, I have the following concerns:
- From a practical standpoint, this approach requires to train the large network alongside the small one. So, at least as described, this approach cannot be used with existing pre-trained networks which is the more typical scenario. It'd be interesting to see if a similar approach could be applied to obtain a small network by fine-tuning a pre-trained architecture.
- The results are marginally better than teacher-student, but the only architecture size being presented is the one for which the performance is comparable to that of the full network. I believe it's important to understand how the small adjoined network compares to the teacher-student network as a function of changing the desired size.

Minor: it would help the reader to add a paragraph describing intuitively which pieces of the larger CNN are shared by the smaller one.  From what I understand, in each layer, a small subset of the output channels is selected and additionally a random subset of such weights is not used?


review_title: An interesting approach to obtain a small CNN network while training a larger one.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a method for training two networks jointly. This resembles the teacher-student approach where the teacher (large) network is trained first and used as an instructor for training the student network (smaller). The proposed paper implements the training of the two networks in a joint procedure and proposes a new kind of loss function "adjoint loss" that consists of two terms: (1) the prediction loss of the teacher network, and (2) the Kullback-Leibler divergence between the predictions of the student and the teacher. Moreover, the weights of the student are a subset of those of the teacher. 

The method is similar to classical knowledge distillation approach (student-teacher paradigm by Hinton et al 2015). The key difference seems to be in weight sharing across the two networks. This idea is partly novel, but should refer to the 2018 paper "Rocket Launching: A Universal and Efﬁcient Framework for Training Well-Performing Light Net" by Zhou et al, who also consider weight sharing but use a different (but similar) loss function.

The results should be clarified. The impression is that the comparison is primarily against the conventional teacher-student setting (Hinton, 2015). This comparison is not relevant, as there are over 4500 citations to the original work. Also in more general terms, the proposed method should better reflect the already lengthy history of knowledge distillation, and position the current work better in the landscape.
review_title: New approach for teacher-student joint training

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a training policy for jointly learning parameters of both supernets (or big teacher models) and subnets (or small student models). The student models have the similar architectures with teacher models, but have different numbers of convolution filters. For jointly training teacher models and student models, an adjoint loss consisted of cross-entropy loss and KL-div is defined. The experiments are conducted on several image benchmarks using ResNet-18 and ResNet-50.

Strengths:
+: The experimental results show the proposed method can significantly reduce model complexity with slight performance loss. 
+: The proposed method seems simple and easy to implement.  


Weaknesses:
-: Lacking more clarification on significances of this work.
(1) No state-of-the-art model compression method is compared for verifying the effectiveness of the proposed method. For filter pruning, many recently proposed methods [r1, r2, r3, r4] are presented to compress different sizes of CNN models (e.g., VGG, ResNet18, ResNet34, ResNet50, ResNet101, ResNet152) on various image benchmarks (e.g., CIFAR-10, CIFAR-100 and ImageNet). The corresponding discussion and comparison are missing in this work.
[r1] Discrimination-aware Channel Pruning for Deep Neural Networks. NIPS, 2018.
[r2] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks. NIPS, 2019.
[r3] Channel Pruning via Automatic Structure Search. IJCAI, 2020.
[r4] HRank: Filter Pruning using High-Rank Feature Map. CVPR, 2020.

(2) The idea of training subnets from supernets shares similar philosophy with one-shot NAS. Particularly, one-shot NAS based on knowledge distillation has been studied [r5], which supernets and subnets are jointly trained. Meanwhile, [r5] also shows the searched small student models can achieve comparable or better performance than big teacher models. Therefore, what are advantages of the proposed method over [r5]?
[r5] Neural Architecture Search by Block-wisely Distilling Architecture Knowledge. CVPR, 2020.

(3) I wonder that why the results (73.41%) of ResNet-50 on ImageNet in this paper is inferior to those of other works (~75.1%). Besides, why the authors did not report the results of ResNet-18 on ImageNet?

(4) For fully verifying the effectiveness of the proposed method, the authors would better report the results using more CNN models, rather than those on various benchmarks. In particular, Imagewoof shares similar philosophy with ImageNet.

-: The writing is too colloquial, and there are too many typos. 
(1) $13.7x$ -> $13.7 \times$
(2) kl -> KL
(3) the the smaller -> the smaller
(4) dropouts -> Dropout

-: I wonder that the meaning of Table 3. For the common settings, ResNet does not adopt Dropout. Therefore, is it fair or necessary for comparing ResNet with Dropout?

-: How about the transfer (generalization) ability of the proposed adjoined networks?  

review_title: Review for Better Together: Resnet-50 accuracy with  13x fewer parameters and at  3x speed

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes “Adjoined networks” as a training approach that can regularize and compress any CNN-based neural architecture in a manner of one-shot learning paradigm. The proposed technique deals with the main issues of current compression approaches. That is (1) Requiring the availability of special hardware to support fast inference and (2) Requiring large training time as they alternate between pruning and fine-tuning.   

1)	The section 2 of Adjoined networks is not clearly illustrated. The author says that the small network selects a fraction of the convolution filters of the large one as its filters. However, the channel of random selection filters may be inconsistent with the feature maps of the deeper layers. How to implement the convolution in small network needs to be further illustrated?
2)	The author needs to clarify whether the M is optional expect the setting as 16/8/4/2 in the work. A binary mask matrix M is used to compress parameters. However, the M is fixed before training and is not learnt. In the experiments, M is such a matrix that the first 16/8/4/2 rows are all 1 and the rest 0. In table 9, with M zeros-outing parameters incrementally, the results do not follow the same trendy. 
3)	The proposed “Adjoint loss” is similar with the loss of existing knowledge distillation architectures. The author needs to illustrate the difference between them.

review_title: A practical training approach that can regularize and compress any CNN-based neural architecture.

Meta-review: The authors proposed to train a large network and a small network simultaneously with a new loss function. The parameters are shared between the two networks, and the loss also incorporates the KL-divergence between the outputs of the two models. In this way, the authors claim that one can train a small network with similar accuracy to the large one, while using less memory and having faster inference speed.

The reviewers think the papers is at the borderline. It has some interesting results, however, it also has quite a few problems:
1)	The technical novelty of the paper as compared to the teacher-student models is not adequate. 
2)	There are many missing references and baselines, since model compression has been a long-studied problem.
3)	Experiments on more different NN models are preferred in order to verify the generalization ability of the proposed approach
4)	The compatibility with the pretraining framework is not very clear

The authors provided their rebuttals to the review comments. However, according the discussions among the reviewers, their concerns were not fully addressed yet and most of them would like to stand on their original scores. As a result, we do not think the paper should be accepted in its current form.

Decision: Reject