Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: CL system has the problem of communication deficiency caused by limited bandwidth.
Other solution (model slicing, matrix factorization) has may degrade the model accuracy if compressing features.
This paper propose new quantization scheme, called SGQ (Stripe-wise Group Quantization).
SGQ captures both channel and spatial-level similarity in pixels, and hierarchically encodes the features in these two levels to achieve a much higher compression ratio.
strengths_and_weaknesses: *strengths

The idea is straightforward, and the paper is easy to follow.
The observation of motivation is good, and the solution makes sense.

*weaknesses

In main paper, Cifar-10, Cifar-100  and mini-ImageNet are too small datasets.
questions: Why didn't you experiment with large dataset (ex. ImageNet)?
limitations: CL system has the problem of communication deficiency caused by limited bandwidth.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In order to improve the communication efficiency in collaborative learning systems, this paper proposes to consider not only the pixel similarities but also channel information by channel-attention grouping. Theoretical analysis is provided to calibrate gradients of both features and CAG blocks in the backward process. Experiments demonstrate higher traffic reduction and image processing speed while keeping comparable accuracy.
strengths_and_weaknesses: Strengths:
1. The observation this work is based on is that some channels hold quite different features when the corresponding filters are orthogonal to each other. This phenomenon is quite interesting and makes it reasonable to design the channel-attention grouping block. 
2. The analysis of gradients in the backward process is thorough and sound.
Experiments on NVIDIA Jetson Nano and HUAWEI Atlas 200DK demonstrate high efficiency and practical values of the proposed method with comparable performance with the baseline.
Weaknesses:
From the first few sections and figure 2, we would consider channel-attention grouping is in front of the stripe-wise group quantization, but these two methods are arranged in the opposite order in the methodology section. It seems to be confusing to readers.
questions: 1. What is the definition of $g_x$ in Theorem 1?
2. Since additional fully connected layers are added to learn the channel information, how about the increased computation cost when the number of channels and devices is high?

limitations: The number of clusters $k$ is tied to the bit, then the performance degradation of extreme cases including 1-bit or 2-bit is still a problem.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Network bandwidth poses challenges to the collaborative learning system. The existing communication-optimized methods either are insufficient in reducing the traffic or may degrade the model performance. This work proposed a hierarchical compression algorithm for feature data, called Stripe-wise Group Quantization (SGQ). SGQ refactors feature based on similarity in both channel and spatial levels. This work also presented the theoretical analysis for the convergence order of SGQ. Experiments show that SGQ offers a higher reduction ratio without accuracy loss compared to state-of-the-art methods such as Product Quantization (PQ).
strengths_and_weaknesses: Strengths:
+ This work tried to tackle the communication challenges of the collaborative learning system deployed in reality.  Experiments show that the proposed method SGQ is very practical:
   + SGQ offers a sufficient reduction ratio in the data traffic
   + SGQ maintains the model accuracy and the convergence is proved
   + the overhead of SGQ is only around 8-9% of the total runtime, which is acceptable compared to the communication overhead, and thus SGQ is able to offer 9.22X to 11.37X speedup.
+ The paper is well organized and easy to follow.

Weaknesses:
- The evaluation section only shows the traffic reduction ratio and traffic size. It would be better to show the tradeoff between accuracy and actual measured latency and throughput.
- It is unclear the contribution of each part of SGQ. It would be better to have the ablation study on different parts of SGQ.

questions: Please refer to the above weakness part.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a structured strip-wise quantization method, named Strip-Wise Group Quantization (SGQ), to implement the scalable collaborative learning systems on tiny edge devices through a hierarchical compression on refactored feature structures.
strengths_and_weaknesses: Strengths
1. The novelty of capturing such channel-dimension structured information to compress feature size is good, which is different from the viewpoint of the conventional methods.
2. The authors deliver a clear set of challenges and motivations for deploying CL systems on tiny edge devices.

Weaknesses
1. This article addresses the communication bottleneck and builds a novel communication-efficient CL system; it is better to bring out a well-through analysis in the experiment section. There are not enough demonstrations regarding communication efficiency.
2. The analysis of the sublinear convergence order of SGQ is not clear in Section 3.
3. The ablation study is based on the performance of the final model with different quantization bits, which shows the impact of quantization bits. I think it would be better if the performance of other components of SGQ could be added to analyze their different effects on the final model.
questions: 1. Dose K-means used for dividing the feature maps into several clusters take a long time? As per my understanding, K-means tend to be way slow for grouping, and the cluster number is also hand-tuned. Now that the Channel-Attention Grouping block is employed to measure and group the entire features, I was wondering if attention could further support dividing them into smaller groups based on pixel similarity instead of using K-means.
2. Is it possible to employ extra attention, like spatial attention for spatial dimension, as the channel-attention block for channel dimension in this work? If not, why could we do attention to the channel dimension but not the spatial dimension?
limitations: None
review_title: null

Meta-review: metareview: This paper proposes a novel communication-efficient learning method that significantly reduces feature size and communication traffic. The rebuttal solved the reviewers concerns about the dataset size and accuracy / latency trade off. 
confidence: Less certain
Decision: Accept