Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of paper:
The paper studies the concept of codistillation in data parallel distributed training. In this setting, the standard minibatch SGD algorithm requires exchange of models in every update of every node. Recent work in distributed training has studied "local SGD", where models are exchanged at frequent (usually periodic) intervals after a bunch of local updates. This paper studies an alternative, called "codistillation". The idea is that at a given node, say node $i$, the local model updates are regularized by the most recent models at nodes $\{j, j \neq i\}$ through an appropriately modified loss term. Specifically, the loss term bias the model at node $i$ towards having similar classification outcomes on the training data as the (most recent) local estimate of the model at nodes $\{j, j \neq i\}.

The paper argues that this codistillation approach has  a regularizing effect, and if the other regularization approaches are combined with codistillation in a balanced manner so as to avoid over-regularization, then the performance can be similar to minibatch SGD, with much lower global communication.

Pros:
--> The paper's results are certainly intriguing and likely to lead to further investigations of codistillation as an alternative or complementary approach to local SGD for reducing communications in distributed training. 
 --> The results are promising and the experimental results seem quite comprehensive. 

Cons:
--> I am not sure completely understand/buy the reasoning provided in the paper. The regularization in imagenet - as I understand it - biases towards chosing models with lower weights (e.g., sparser models). The regularization effect of co-distillation is towards chosing models that are not too far away from the initial point - indeed 1(c) shows that the codistilled models are not too far away from the original models.  However, the paper (e.g, Sections 3.1, 3.2) seems to suggest these two forms of regularization are similar/equivalent, and one form can be increased and the other form can be lowered.  I find this explanation not very convincing. 
--> I think the regularization parameters (e.g., what exactly is the constant L2 regularization, what is the factor of 10^{-4} in Section 3.2) needs to be explained in more detail in the paper for completeness...at this point, the reading requires knowledge of the details of Goyal (2017) for the reader to know what exactly is implemented by the authors.
--> The result that codistillation with local updates can lead to lower loss as compared to minibatch SGD is almost too surprising to be true. Common intuition suggests that the best achievable performance in terms of accuracy per update that any distributed approach can get is that of minibatch SGD, which is what one would do if all the data was centralized at one node. So I am suspcious of the improvements. I think there are two possible reasons, that the authors might clarify (or suggest other reasons)
a) The benchmark minibatch SGD model is not regularized properly, or something else is ineffectively done in this benchmark estimate (e.g., perhaps hyper parameters are not as well chosen).
b) Perhaps I do not understand what exactly is done in the minibatch SGD setting, e.g., is the minibatch SGD done with one round of global communication in every model update, or is there some adjustment made to make the communication costs comparable?

Summary:
Although not all aspects of the paper are convincing,  the results of the paper are intriguing and it is likely to lead to further investigations along this topic.  I am on the borderline on this paper, perhaps leaning slightly more towards acceptance.
 

====== Comments after author response ====

Thanks for the detailed response. Having read the update and the other reviews, I have lowered my score. 
 I am still left unconvinced on (at least) two aspects.

--> It is unclear to me as to how a method can improve upon minibatch-SGD can actually have better generalization. While nothing theoretically rules this out, it possible means that that either (A) the model architecture was sub-optimal and there is room for improvement, or (B) the optimization found a minimum that is not very good. In either case, there appears to be a different "lesson"  than the one described in the paper. This is certainly worth exploring in more detail.

--> I agree with some of the other reviewers that the experimental results can be improved, specifically, the usage of just 2 replicas (now clarified in the paper) is limiting, and a more detailed analysis regarding how to tune the regularization parameters.

I do think that the paper is on track towards an interesting discovery, but I would like to see a deeper/more detailed analysis to be convinced.  

review_title: Interesting Results on Codistillation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to have a closer look at the role of codistillation for distributed training. Authors provided an answer with their empirical observations. That is, codistillation acts as a regularizer, since the distance between the learned model and the initialization is smaller than sync SGD without codistillation. Then, the authors claim that the codistillation may over-regularize and study how to modify the training configurations to avoid it. There are further discussions on the overfitting and robustness to hyper-parameters in sec 4 and sec 5. 


Comparing to sync SGD, codistillation bring in auxillary loss enforcing all the local workers learn from each other in some degree. Thus, it is intuitively a regularization term. Deep understanding on how it takes effect will help us better trade off the bias and variance in distributed training. Unfortunately, this paper does not achieve this goal. 

Most of the claims in this paper are not surprising, with limited insights. 1) It is common sense that, regularization may “over-regularize” and can “reduce overfitting”. The empirical observations to demonstrate these are not necessary in sec 3.1 and sec 4. 2) To “bridge the (over-regularize) gap” (In sec 3.2), authors report the empirical behavior of distributed training with codistillation, with difference choice of learning rate, batch size, concise learning rate schedule, tasks (NMT), and update frequency (in sec 5). The proposed modification of training is minor for a specific setting, i.e., to decreasing learning rate from 10^{-4} to 10^{-5} on ImageNet for ResNet50. 

Authors ignore the factors which are more direct to trade off bias and variance. 1) The coefficient of the auxiliary loss \alpha can adjust the regularization. I have not seen the discussion on it. What is the value for it in the experiments? Did I miss anything? 2) For me, the number of local workers may alleviate or enlarge the regularization of codistillation. It seems that, most of the experiments are conducted with 2 models/workers. 

Overall, this paper is easy to follow, but the results (from the experiments) are non-surprising, with limited insights. The modification proposed is minor and did’t get significant improvement of performance.


Overall, this paper proposed methods to make CNN to be symmetric as (NS euqation) dynamics. The technical contribution need be highlighted.

review_title: Non-surprising explanation about the role of codistillation in distributed training with empirical observations

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work analyzes the effect of co-distillation for distributed training under moderate batch sizes. Using distillation-like techniques to improve synchronous SGD training is an interesting direction. And the paper carefully analyzed this setting while using the same amount of compute, which is not done by prior work to my knowledge. In addition, the writing is good and easy to follow.

However, I think the work can be improved in the following aspects:

W1: The setup of the work is not very realistic. 

The authors proposed to analyze distributed training (using two machines) with very small batch sizes (e.g., 256). These two constraints don't appear to be commonly used for ML tasks where distributed training help. E.g., for CV, Exploring the Limits of Weakly Supervised Pretraining (Dhruv et al., 2018) was already using batch sizes of 8,064 back in 2018; modern NLP tasks use significantly larger batch sizes (up to 4M in GPipe, Huang et al., 2018). Even if hardware is a limiting factor, showing how results scale with more than 2 model copies would help a lot in demonstrating the potential value of this work in distributed settings.

W2: The contribution of this work seems to be incremental.

Considering distillation / co-distillation's regularization effect is not exactly a novel direction. This line of thoughts originated at least from Distilling the Knowledge in a Neural Network (Hilton et al., 2014) and was discussed in other papers like Collaborative Learning for Deep Neural Networks (Song et al., 2018).

W3: Experiments could be done on top of stronger baselines. 

The authors proposed that co-distillation benefits models with higher capacity more (e.g., 3.2: ResNeXt10(1) may benefit more from codistillation due to having more capacity than ResNet50), but there are no CV experiments demonstrating that this scales with more parameters done on more complex architectures (e.g., SENet, AmoebaNet, etc).

W4: In the NLP/NMT case, there are no experiments showing that this benefits self-attention architectures besides a remark in Section 4. For small datasets like IWSLT'14 DE->EN, it's known that using fewer parameters  (e.g., 256d) vs the default 1024 in transformer-big produces much better results. See e.g., Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder (AAAI'19). Comparing the regularization effect with 6 layers / 1024 dimension as a baseline setup feels like cherry picking.  For NMT it's also much more common to report BLEU vs perplexity.

W5: if the paper's goal is to demonstrate co-distillation as a viable distributed training alternative, it would be also valuable to compare it with other related training methods like gossip SGD etc.


====== Edit after author response ====

I have read authors' responses and have decided to keep my score as is.

The additional experiments haven't addressed problem formulation issues (W1, W2, W5) if the paper is positioned as more of a theoretical work; if the paper is positioned as a more of an experimental work, the baselines used (W3, W4) need to be improved with proper hparams settings.

review_title: Incremental empirical work on co-distillation for distributed training with weak evaluation

final_score: rating: 4: Ok but not good enough - rejection
review_content: Update: I have read the authors' response, and have decided to keep my score as-is

Claims:

1) Codistillation achieves accuracy close to synchronous gradient descent if the model's built-in regularizers (e.g. L2) are decreased to zero while training

2) Codistillation therefore acts as a regularizer, and should be explicitly accounted for during training

3) Codistillation works for moderate batch sizes, not just large-batch

Pros:

-Interesting observations about codistillation's empirical behavior, which contributes to the overall literature on distributed training

-Codistillation imposes much lower network costs since model parameters only have to be exchanged once every few thousand updates (rather than every mini-batch of updates)

Cons:

-There are problems with the systems setting, which undermine the strength and validity of the claims:

1) The setting is not fully clear: given that only n=2 codistillation replicas are used throughout the paper, how are the GPUs arranged? My guess is that there are 2 physical machines with 8 GPUs each, and within each machine, the GPUs exchange their updates synchronously. This information should be clearly presented.

2) The overheads of synchronous distributed training increase with higher machine/replica count, so codistillation should be really be studied with many replicas, which is where it has the greatest potential to have an advantage. Note that, depending on the model, network and GPU hardware, synchronous distributed training on just 2 replicas/machines may not even incur any network overhead, which completely eliminates the overhead-reducing advantages of codistillation. Hence, the paper needs to study codistillation with at least n=4 to 16 replicas, verifying that the regularization effect of codistillation still holds with more replicas, and that the proposed means of controlling regularization effects (namely, reducing the built-in regularizers) is still effective. Thought experiment: what happens if the regularization effect of having 16 replicas is so strong (because there are 16 terms in the auxiliary loss) that reducing the built-in regularizers is no longer sufficient?

3) The increased compute overhead of codistillation (due to auxiliary loss needing to evaluate every of the n model replicas every update) is not investigated, so it is unclear how it stacks up against the decrease in network costs. Runtime benchmarks versus all-reduce could have been provided and analyzed.

-Experiments are performed on only two models, ResNet and Transformer. Comparable papers that focus on empirical evidence usually test their hypotheses on five or more models, in order to demonstrate that the technique is truly generic. Testing on non-residual CNN architectures (e.g. VGG) and RNN architectures would have made the results more convincing.

-No systematic method is proposed/evaluated for reducing the model's built-in regularizers. Even if no theory is presented, I imagine the paper could have presented at least a rate function with controllable hyperparameter (similar to learning rate), and systematically conducted experiments for different values of the hyperparameter. Currently, each model's builtin regularizers are reduced in an ad hoc fashion, so the evidence is only of anecdotal quality.

Suggestions for improvement:

-In Section 4, the authors may want to clarify whether the model's built-in regularizers (L2 etc.) are or are not being reduced - it was not clearly stated, and I am guessing it is the latter case.

-The paper would not be complete without a discussion of Federated Learning (McMahan et al., 2017), which provides another means of reducing the network overhead of synchronous distributed training, among other benefits. Even if experiments against Federated Learning are not conducted, it should be discussed in the related work.


review_title: The paper investigates codistillation as an alternative to distributed synchronous gradient descent, by comparing on batch sizes and learning rates that are typical for the latter. A hypothesis is put forward that codistillation acts as a regularizer, and evidence is provided showing that gradually decreasing the model's built-in regularizers (e.g. L2) over training epochs allows codistillation to more closely match the achieved accuracy of synchronous distributed training.

Meta-review: This paper proposes a novel and interesting approach called co-distillation for distributed training. The main idea is to add a regularizer in order to encourage local models to be consistent with the global objective. Although the idea is a promising alternative to local-update SGD, the approach is mostly empirical. The claim that co-distillation helps reduce overfitting could be better justified by theoretical analysis, in addition to the experimental results. I hope that the reviewers' constructive comments will help improve the paper for future re-submission.
Decision: Reject