Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an activation quantization method, AC-SGD, for the fine-tuning step on language models with slow network set-up. AC-SGD compresses the change of the activations instead of activation values directly. This paper shows this compression method can converge well and it can lead to throughput improvement under slow network systems. 
strengths_and_weaknesses: ### Strengths
- This paper doesn’t aim at compressing activations directly. Instead, by compressing the changes of the activation, the proposed algorithm can achieve efficient convergence.
- With a slow network system (i.e., the portion of communication overheads is quite big during end-to-end distributed fine-tuning process), this algorithm can achieve higher throughput with smaller communication overhead.


### Weaknesses
As weaknesses of this paper, I have two concerns as below:
- I’m very confused about what slow networks are. It seems to be hard for me to imagine which scenario this paper assumes, V100 multi-gpus with 100Mbps network.  The only network I can imagine is that single GPUs are distributed and inter-connected with slow ethernet systems. It is so weird. I don’t understand why we should assume extremely slow interconnection between high-end GPUs for fine-tuning large PLMs. 
- For the GPT results in Figure 4, I think showing training loss is not sufficient to prove that the compression method works well for generative language models, especially for fine-tuning steps. Since the pre-trained model consists of an enormous number of parameters and is trained with a large dataset, it is hard to trust the train loss and validation loss for fine-tuning dataset. It tends to be easily overfitted and there is rare correlation between loss results and measured scores. Since there is a limitation on scale of BERT-like models, this paper should show the inference results on generation tasks and large generation models. 

questions: - How about model (tensor) parallelism (ex. Megatron)? 

limitations: - As I mentioned, this paper should describe why they assume this slow network system in detail. As far as I know, it seems to be not common. 
- The experimental results are not sufficient to show that this method works well.
- Showing profiling results of the fine-tuning process with this method and various speeds of the network would be better for understanding.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors proposed a differential activation quantization scheme with proven convergence, for distributed model-parallel training.  

strengths_and_weaknesses: [+] Clear presentation in general

[+] Real-world demonstration

[-] Limitation of practical usefulness from storage requirement

[-] Lack of stability study

questions: - A biggest limitation of differential compression is the necessity for storage of the entire epoch of activations, and second biggest, the uselessness in online learning scenarios where data never repeat.
- Numerical stability probably needs more validation.  Though more stable than DirectQ, AC-SGD did present non-monotonic training loss, e.g. Fig. 5b.  Given that these are fine-tuning of pre-trained LMs, which are known to be already close in parameter space to convergence, it is questionable whether AC-SGD was stable at initialization for training from scratch.
- Following the same thought, as the authors mentioned in the manuscript, near-convergence training dynamics is probably more tolerant to aggressive activation quantization.
- Though SGD convergence proof is in the order, numerically it is usually more complicated in practice when diverse regularizers and different optimizers were used.  Is there any commonly used regularization or optimizers that do not agree with AC-SGD? 
- What is the $Q(\cdot)$?  Does it involve affine calibration from the differential data statistics?  If so, what's the procedure? 
- What if activation is already in low precision, say in FP16 in mixed-precision training?  Is AC-SGD still better off?  What about FP8?  
- "Activation quantization" is perhaps more explicit and accurate than "activation compression".

limitations: See above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work examines the feasibility to compress the activations for models trained with pipeline parallelism. To this end, the scheme AC-SGD is proposed, which aims to make pipeline-parallel training more communication-efficient over slot networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AC-SGD compresses the changes of the activations. The most novel insight is that: one can still achieve O(1/pT) convergence rate for non-convex objectives under  activation compression, without making assumptions on gradient unbiasedness that  do not hold for deep learning models with non-linear activation functions. The this work shows that AC-SGD can be optimized and implemented efficiently, without additional end-to-end runtime overhead. The evaluations on AC-SGD are shown on fine-tuned language models with up to 1.5 billion parameters, compressing activations to 2-4 bits. AC-SGD provides up to 4.3⇥ end-to-end speed-up in slower networks, without sacrificing model quality.
strengths_and_weaknesses: Strengths:
- An interesting insight on pipeline-parallel training.
- The solution is effective and the optimization space is covered.
- Extensive evaluation results.
- Source codes available

Weaknesses:
- The work may need more rationales upfront to motivate the problems (i.e. slow networks)
- More breakdown analysis to showcase the effectiveness of the approach

questions: - Can you provide more evidence to back up your simulation setups on the slow work?
- How your optimizations benefit the overall system (separately and collectively)? 
limitations: - I think the paper has good potentials, and it can be strengthened by addressing the two points (raised in the weakness part).
- I assume the geo-distributed network is a more realistic scenario for this proposed approach. Hence, I would suggest the authors to get more literature on this part, and back up both their motivation and simulation setup better (both qualitatively and quantitatively).
- Also, I believe breaking down the optimizations in AC-SGD can add more values in this work, so that the readers can better understand how essential your optimizations are.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Large Language models are trained or finetuned using a combination of pipeline and data parallelism methods. Pipeline parallelism leads to activation and activation gradient being communicated between devices, while data parallelism leads to weight gradients being communicated between devices. A lot of work has focussed on the later, but the former has received less focus. Compressing activations is non trivial as compressing it in a stochastic unbiased way will lead to biases in the gradient  and breaks the unbiasedness assumption made by most gradient compression results. Previous work either failed to provide theoretical guarantees 

This paper proposes AC-SGD, an activation compression algorithm for communication efficient pipeline parallelism. This method helps accelerate pipeline parallelism over slow networks. They provide theoretical guarantees to prove convergence and show that the method can be implemented with minimum runtime overhead (albeit at a huge memory cost). They compare their method with recent work and show better accuracy at lower precision. Depending on the communication BW used as baseline, they can achieve 4.3x end to end speedup.
strengths_and_weaknesses: Strengths:

1. The claims in the paper are evaluated against strong baselines
2. The method comes with a theoretical guarantee of convergence 

Weakness:

1. The idea relies on data being repeated over multiple epochs. This can limit the applicability of this techniques to more important workloads like pre-training and limit to the idea to only fine-tuning of LLM. This reduces the potential impact of the work.

2. The requirement to store activations for new samples in the finetuning datasets makes it harder to scale this technique with larger finetuning datasets
questions: Questions -

1. What was the storage requirement for each of the benchmarks evaluated in the paper?
2. What motivated the path to explore compressing the delta of activation per sample rather than the activation itself

limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes
review_title: null

Meta-review: metareview: In this paper, authors propose to speed up the fine tuning of large models over slow networks by compressing *deltas* of activations (vs activations themselves), so as to reduce the computation cost.

Original reviews were mixed, but at the end of the discussion period, all reviewers are leaning towards acceptance. The main issues that were raised are:
* The motivation for training very large models over slow networks
* The limited amount of metrics to validate the quality and robustness of the optimization process
* Concerns about the scalability of the method (storage requirements) and its applicability to the online setting

I consider that these concerns have been mostly addressed during the discussion period by the authors, who also remained honest about some of the limitations of their method.

In my opinion, the pros of this work (a practically useful idea) outweigh the cons (it may only be useful in somewhat niche settings), and I thus recommend acceptance.
confidence: Less certain
Decision: Accept