Hierarchical Channel-spatial Encoding for Communication-efficient Collaborative Learning
Authors: Qihua Zhou
Song Guo
LIU Yi
Jie Zhang
Jiewei Zhang
Tao GUO
Zhenda XU
XUN LIU
Zhihao Qu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Collaborative Learning
Feature Encoding
Traffic Saving

Abstract: It witnesses that the collaborative learning (CL) systems often face the performance bottleneck of limited bandwidth, where multiple low-end devices continuously generate data and transmit intermediate features to the cloud for incremental training. To this end, improving the communication efficiency by reducing traffic size is one of the most crucial issues for realistic deployment. Existing systems mostly compress features at pixel level and ignore the characteristics of feature structure, which could be further exploited for more efficient compression. In this paper, we take new insights into implementing scalable CL systems through a hierarchical compression on features, termed Stripe-wise Group Quantization (SGQ). Different from previous unstructured quantization methods, SGQ captures both channel and spatial similarity in pixels, and simultaneously encodes features in these two levels to gain a much higher compression ratio. In particular, we refactor feature structure based on inter-channel similarity and bound the gradient deviation caused by quantization, in forward and backward passes, respectively. Such a double-stage pipeline makes SGQ hold a sublinear convergence order as the vanilla SGD-based optimization. Extensive experiments show that SGQ achieves a higher traffic reduction ratio by up to 15.97 times and provides 9.22 times image processing speedup over the uniform quantized training, while preserving adequate model accuracy as FP32 does, even using 4-bit quantization. This verifies that SGQ can be applied to a wide spectrum of edge intelligence applications.

Hierarchical Channel-spatial Encoding for Communication-effiient Collaborative Learning
Qihua Zhou', Song Guo Yi Liu', Jie Zhang', Jiewei Jiewei Zhang', Tao Guo', Zhenda Xu², Xun Liu', Zhihao Qu²
Department of Computing, The Hong Kong Polytechnic University
School of Computer and Information, Hohai University {esqzhou, csyiliu, casgzhoo,,csyiiu,,oo ccow pwwpppollueeeee edu hk
{j (jieaa. zhang, cocotao.... guo, jackal...... xxu, compxun. Sconnectt.tt......mmmmmm polyu...pp....... . song. guo@polyu. edu. hk, quzhihao@hhu. edu.
Abstract
It witnesses that the collaborative learning (CL) systems often face the performance bottleneck of limited bandwidth, where multiple low-end devices continuously generate data and transmit intermediate features to the cloud for incremental training. To this end, improving the communication efficiency by reducing traffic size is of the most crucial issues for realistic deployment. Existing systems mostly compress features at pixel level and ignore the characteristics feature structure, which could be further exploited for more efficient compression. In this paper, we take new insights into implementing scalable CL systems through hierarchical compression
features, termed Stripe- wise Group Quantization (SGQ). Different from previous unstructured quantization methods, SGQ captures both channel and spatial similarity in pixels, and simultancously encodes features in these two levels to gain a much higher compression ratio. In particular, we refactor feature structure based inter-channel similarity and bound the gradient deviation caused by quantization, in forward and backward passes, respectively. Such a double-stage pipeline makes SGQ hold a sublinear convergence order as the vanilla SGD-based optimization. Extensive experiments show that SGQ achieves a higher traffic reduction ratio by up to 15.97 and provides 9.22x image processing speedup over the uniform quantized training, while preserving adequate model accuracy FP32 does, even using 4-bit quantization. This verifies that SGQ can be applied to wide spectrum of edge intelligence applications.
Introduction
Recent years have great prospects of deploying vision tasks on tiny edge devices by using their always-on microprocessors, embedded sensors and neural chips. Considering the realistic environment that new data is continuously generated on user devices that cannot be aggregated at once due to the privacy and energy concerns, it comes to the rise of collaborative learning (CL) paradigm [1, 22, 15). Considering the resource constraints in CL systems, it is expected partition the models between multiple edge devices and the cloud, and coordinate the two sides during the training procedure [32, 7]. In the forward pass of CL, the shallow cul layers deployed on edge devices are used for low-level feature extraction and the intermediate features will be transmitted to cloud for the subsequent processing on deep remaining layers. Also, in the backward pass, the cloud returns the derivative flows of the transmitted features to the devices for model updating. By adopting such learning paradigm, devices can update parameters and evolve models continuously.
"Correspondng author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Feature Size W*H*C
Channel
Width (w) Height (H)
C=1
Structured Stripe-wise
Unstructured Pixel-wise
Figure I: Visualization the structured stripe-wise quantization (the left, ours, maintaining 4-channel structure) v.s. the unstructured pixel-wise quantization (the right, e-g,, UQ [41, 14, 40], flattening original structure as channel), which are with the 3 x and 6 pixel plane, respectively. The batch size is set as for illustration convenience.
Prior work and limitations. In general, deploying efficient CL system needs to address com- munication deficiency caused by limited bandwidth [15, 27]. where reducing feature size is the key to improve communication performance. Existing communication-opiiiii
methods, including progressive model slicing (e.g.. CLIO [12]), matrix factorization (e.g.. Poseidon [37]) and top-k filter (e.g., Async-opt [3]), either hold an insufficient traffic reduction ratio or may degrade the model accuracy compressing features. Besides, existing quantization methods cannot directly be adopted to feature compression they are specifically designed for weights [19, 41, 38] or activations [4], which hold different distribution characteristic from the features [8]. Simply applying these methods cannot fundamentally reduce feature size verified by our experiments (see $4.3), which is not applicable to realistic CL systems. This motivates us to design a new quantization scheme that provides a desired compression ratio on feature maps while not sacrificing the model quality.
Observations and challenges. Actually, the feature regions often hold specific pixel similarity across channels when the kernels aim at extracting interrelated features, especially in shallow layers [8, 23, 39, 26, 2]. We could leverage this property to quantize each pixel along all the channels in structured manner (i.e., the stripe wise in Figure 1). rather than the "flat" perspective (i.e., the pixel wise in Figure 1) of Uniform Quantization (UQ) [41, 14, 40J. However, by analyzing the value distribution of each channel inside the feature, observe that some channels hold quite different features when the corresponding filters are orthogonal to each other. This indicates that simply adopting the vanilla Product Quantization (PQ) [30, 5] along the whole channel dimension will introduce a significant representation error on features and finally degrade the model accuracy. We need reorganize the features into groups based on their channel-level similarity. instead of treating the features as a whole or roughly partitioning them into successive subsets. Therefore, capturing such thannel-dimension structured information the key to fundamentally compress feature size, which is often omitted by conventional quantization methods designed for parameters, activations or gradients. As each group, we need to find a collection of representative pixels, each of which can replace other pixels similar to it.
Our solution. We achieve the above target by proposing the Stripe-wise Group Quantization (SGQ, $2.1) method, which captures both channel and spatial-level similarity in pixels, and hierarchically encodes the features in these two levels achieve much higher compression ratio. Specifically, we introduce the Channel-atienion Grouping (CAG, $2.2) block to measure the per-channel significance and reorganize the entire features along the channel dimension into several groups, each of which holds similar inter-channel texture and pixel intensity reflected by channel significance. In each group, we employ K-means to divide the feature maps into several clusters based on pixel similarity along the stripe (i.e., vector) and represents all the pixels belonging to each cluster by its centroid. As all the pixels are clustered along the channel dimension simultaneously, more volume of the features can be quantized over the conventional unstructured pixel-wise quantization, yielding much smaller pixel plane. Thus, given the same quantization bits, SGQ could achieve higher compression ratio, especially for features with a large channel number, which is common in modern CNNs. For example, as to the 4-channel 3 x 3 feature map shown in Figure 1, SGQ provides a 4x higher compression ratio over the unstructured pixel-wise UQ using same quantization bits. We implement our idea into a novel CL framework, which could automatically insert the SGQ block into different models and adjust clustering hyper- paramttts according to model characteristies in

---

Device Side:
Channel-attentin Grouping
Stripe-wise Group Quantization
Cloud Side:
Backward Pass
Gradient Culibration
Original Features
K-means
Clustering
Derivative Flows
0000000
Encoding
Forward Pass
Channel Attention
Quantized Code&Codehookk Features
Remaining Layers
Cut Layers
Feature Groups
Figure 2: The overview of our framework across edge devices and the cloud.
the forward pass. To preserve model accuracy, we also design the Gradient Calibration (GC, $2.3) module to adjust the corresponding gradients of quantized features and CAG block in the backward pass, which holds a sublinear convergence order as the vanilla SGD. This double-pas pipeline makes SGQ communication--eficienn method enable model evolution on multiple edge devices without sacrificing much model accuracy as the FP32 training.
Ad Advantages and contributions. Evaluation based NVIDIA Jetson Nano [24] and HUAWEI Atlas 200DK [13] shows that SGQ could effectively alleviate the communication overhead for feature transmission and provide comparable model quality the original FP32 training, supporting various CNN models. Thus, SGQ can serve as a lightweight module for the resource-constrain CL environments. Overall, the key contributions of our work
follows:
Scalable collaborative learning framework: We propose a scalable CL framework that enables model evolution multiple edge devices and match the requirements of continuous analytics. It will be open-source" and will support commodity edge devices (e.g., NVIDIA Jetson Nano and HUAWEI Atlas 200DK), thus can be easily deployed in realistic scenarios.
General feature compression method: We address the communication bottleneck by capturing the structured pixel similarity in both channel and spatial levels, and propose the SGQ method ($2.1) to hierarchically encode the features in these two levels for much higher compression ratio over existing methods. SGQ can serve as general quantization block supporting different CNN models, without sacrificing much model accuracy as the FP32 training.
We formulate the impact of quantized features the training process and present the theoretical analysis to bound the gradient deviation in backward pass, making SGQ-based training holds sublinear convergence order ($3) the vanilla SGD algorithm.
Efficient convergence order:
To the best of our knowledge, the proposed SGQ is the first general traffic reduction method exploiting the quantization feasibility in both spatial and channel levels for building communication ti efffccien CL systems. It achieves much higher feature compression ratio the pertinent existing methods while not sacrificing model accuracy even in 4 bits, thus providing great advantages for real-world implementation. We believe SGQ could constantly contribute to the further development of edge intelligence applications.
2
Methodology
The key of our collaborative learning framework is to quantize feature maps for traffic reduction and fetch the corresponding gradients for model updating in forward and backward passes, respectively. As shown in Figure 2, this target is resolved by the Stripe-wise Group Quantization (SGQ) based on Grouping (CAG) block and the Gradient Calibration (GC) method.
Channel-are
Source codes will be shared Github after the double-bline review.
3

---

2.1 Stripe-wise Group Quantization
SGQ is used for compressing the feature map size at the end of device's cut layers by converting the
FP32 values to low-bit format (e.g., INT8). It contains two key steps: (1) feature discretization and (2) pixel encoding.
Step #1: Feature discretization. In this step, all the pixels of the feature maps belonging to a same group are categorized into several clusters, which transfers the "continuous" FP32 values to the discrete ones. The number of clusters is called the quantization level and directly impacts the data representation precision. Pixels will be covered by 2 clusters if we use the n-bit quantization, where is usually set as 8 or 4.
This procedure can be handled the K-means clustering under given bits. Considering the computa- tional overhead of K-means, we need to downsamples the feature maps first and generate the K-means clustering model based these samples, instead of the whole feature maps. Thus, the overhead can be restricted within 6% of the forward pass time by using the suggested 4-bit quantization (details are in $4.5). Given the feature maps X {I) -2, ... .mm generated at the end of device's 's cut layers, we categorize X into & (k = 2") clusters {cI (2,,02, corresponding clustering centroid matrix is U {ui, u2
,.k.. where each æ; is a pixel of X and the
u. ... ux Therefore, the cost function of
K means clustering can be formulated as:
J ΣΣ
r 1, uyll2.rs E Rmxk
(1)
==
[1, 1o
xS E cj.
Ti
else.
where R is the pixel mapping matrix generated by K-means clustering, reflecting whether æ; belongs to Cj. For each pixel æ;, we can calculate its clustering centroid y; as:
= arg min jE(1,2,- .k]
||æ; - u2l12-
(3)
By restricting the partial derivatives of Eq. (1) as O, we can figure out the latest centroids as:
En m T
Ug E Ti,
The above procedure will repeat until all the centroids stable enough to form the clustering model and we finally formulate the feature discretization as Y xx11 = D(Xmxc), where represents the discretization function that maps each pixel æ; the one-hot cluster label y (y: Y).
Step #2: Pixel encoding. This step represents each pixel by the centroid of the cluster it belonging to, such that all the pixels are encoded as the unique index of the corresponding clustering centroid, which can be covered by n bits. The entire feature maps is compressed to 32 of the original FP32 size. For brief, such encoding procedure that reflects the mapping function between the original pixel and the centroid is called the codebook.
This procedure can be handled by a series of matrix transformation. Given the pixel mapping matrix Rmxk k and the clustering centroid matrix U, each row ry of the pixel mapping matrix R represents the cluster label of yj in the one-hot form. Therefore, we can get the transformation relation between Y mxx and Raak as onchot(Y).. Then, the pixel encoding process can be formulated as:
Q(a,) = Eniuuy X, j=1
(5)
Q(X) R. onchot (Y) U.
(6)
By approximating the one-hot Y via the softmax function, we can get each encoded pixel i;; as:
Ej=1 uee--(= (0. u()
कं Q(x;)
Σ)=1 -.,a B E Q(X).

---

where the distance between e; and is minimized. In summary, the above two steps constitute the basic function of SGQ and we can get the final quantized feature map Q(X).
Traffic analysis. Different from the existing quantization methods that flats the entire feature maps and conduct quantization in the pixel wise, SGQ quantizes all the pixels along with the channel dimension (i.e., the stripe wise), thus providing a much higher compression ratio of feature maps. As shown in Figure 1, we make a comparison of the traffic size by using conventional Uniform Quantization (UQ) [41, 20, 14] and SGQ, under the same quantization bits.
As the original model is split between edge devices and the cloud, the cloud needs both quantized feature maps and codebook to recover the intermediate results generated by the cut layers, so as to promote the computation of remaining layers. The major network traffic is dominated by the quantized feature maps and codebook. Note that the codebook is still represented in the FP32 data format while the quantized feature maps only requires n bits. Given G groups, each of which holds C channels, the traffic size Ssao by using SGQ can be described as:
SscQ =
(n N..WH- 32 32 Cy), Ein 1=1
feature
codebook
where W, H represent the with and height of the pixel plane, respectively. Correspondingly, the UQ can be regarded special version that entire feature map is flatted as channel, thus the traffic size Suq based on UQ is described as:
G Suq = n WH. Σ 32.22.2
codebook
feature
To make SGQ generate less traffic size, we require Ssaq < Stiq and this inequation can be simplified as:
WH
Σ i= C -
(10)
2n+5
E C G
As G is usually far smaller than - C₁ in practice, Eq. (10) is easy to satisfy in common CNN models. For example, if we cut ResNet18 at CONV1 with 4-bit and 10-group SGQ, we will have the quantitative relation that 4x112x
G
2 >>
64-1 64-10 Therefore, SGQ can provide a much higher
compression ratio over conventional UQ thus effectively reducing traffic size of feature transmission.
2.2
Channel-attention Grouping Block
Recall that SGQ makes feature discretization based on K-means clustering, which closely replies on the distance measurement between pixels. Actually, not all channels are equally important the representation of feature maps. Assigning each channel with equal distance weighting cannot well capture the characteristics of the entire feature map. natural idea is to precisely reflect per-channel significance and "pay attention" to the most significant channels for more efficient clustering. More seriously, channels may hold similar texture or orthogonal to each other. Simply conducting the vanilla Product Quantization (PQ) [30, 5] to pixels along the channel dimension will introduce a significant representation error because it is hard or even impossible for PQ to find the proper cluster centroids that can replace other pixels, thus finally degrading the model accuracy. We need to reorganize the features into groups based on their channel- similarity, instead of treating the features as whole or roughly partitioning them into successive subsets.
Consequently, we design the Channel-attenme attention Grouping (CAG) block to capture the channel- dimension structured information for more precise clustering. As shown in Figure 2, the key of CAG block is to form one-dimension vector with C elements, each of which represents the distance weighting of corresponding channel. This function can be abstracted as a series of affine transforma- tion A(X) that converting the original feature maps into the C-element channel significance vector V Vixc, i.e. V A(X).
5

---

In practice, we could employ two-layer fully-com colly-coneected network to approximate this procedure, which is inspired by the self-attention mechanism [6, 34, 11, 35]. beginning, we use common downsampling methods (e.g., AVG pooling) shrink each channel of the feature map into one pixel, thus the feature map compressed as a C-element vector Vixc. Then, the two-layer fully-connected block conducts an affine transformation to extract channel-level significance and the transformed vector V represents the distance weighting of each channel. We can use this vector to adjust the distance measurement for K-means, which is described as:
ΣΣ THJ < uj
11 U. E..
(1I)
Consequently, the CAG block helps SGQ extract the most informative channels and preserves the spatial characteristics of feature maps after pixel encoding.
2.3 Gradient Calibration
As the feature maps generated by the cut layers have been quantized in the forward pass, need to adjust the corresponding derivative flows in the backward pass, so as to figure out the correct gradients of both feature maps and CAG block for preserving model convergence. We achieve this target by designing the Gradient Calibration (GC) module. The theoretical analysis of model convergence by employing GC module will be discussed in $3. Here, will highlight the gist of how GC works to feature maps and CAG block, respectively.
Gradients of quantized feature maps. Based on the formulation of SGQ function in Eq. (7), can calculate the theoretical gradients of quantized feature maps as:
oQ(x:;) Ox;
2Σ 11=1 u (ug ui)p,Pt
(12)
Σj=1 E1=1 P.P
20. - U+ UU, _ x;
(13)
where Py = (m= and U the pseudoinverse of U. Note that U, is the dot product matrix, where the i-th element u ug. As the dot product of clustering centroid matrix u, may introduce extra computational overhead in Eq. (13), we can store these clustering matrices and conduct the calculation on the cloud side. Then, the cloud only needs to return the intermediate derivative flow of (Q(aas) to edge devices during the backward pass, which will not yield much traffic. Full derivation Eq. (13) is in the supplementary material of $ B.
dæ:
Gradients of CAG block. The CAG block serves an independent branch to measure the channel- level significance and the transformed vector V is merged to the SGQ method for distance calculation. Therefore, the gradient calculation of CAG block involves the operations of branch merging and duplication. Based on the derivative flow generated by the backward pass of quantized feature maps, the gradients of branch merging is described as:
0A(X) = OA(X)
OQ(X)
aQ(x) 0x
(14)
0X
aQ(x) 0X
where * denotes the element-wise multiplication that broadcasts each E V to the corresponding position of (Q(X) @x Also, the gradients of branch duplication can be regarded as the sum of derivative flows. Thus, the final gradients of CAG block is described as:
OA(X) V
aQ(x)
aQ(x)
(15)
0x
OX
OX
Note that CAG block only holds a slight computational overhead (less than 2%) in both forward and backward passes. The detailed analysis will be discussed in $4.5.
3 Convergence Analysis
Theorem 1. Assuming each follows ||0||2 S B, the upper bound of the approximate gradients can be described as
(Q(x)
lla 2B². Meanwhile, given pixels and C channels, the upper bound dw

---

FM,
NobioNet-VL, CF10.
oNotx11.0 MI,
100
100
80
Accursty (N4)
75
75-
Accurssy (90)
5 &
ALcuracy [N]
60
Accurscy
50
50
40
Ug ALC
ACC
25
25
20-
20
Acc
Epochs
Figure Comparison of convergence curves using different benchmarks and baselines.
Table 1: Summary of average model accuracy (%) using 4-bit compression, compared with FP32.
Method
VGGII, FM
MobileNet- V1, CF10 ResNet34, MI
ShuffleNet-xl 00 MI
FP32 (Upper Bound)
97.55
94,74
80,31
78.73
UQ
95.12
92.41
36.89
53.15
PQ
95.94
92.67
69.61
70.16
CLIO
21.02
19.16
13.06
11.10
SGQ
96.57
93.45
74.37
74.86
of real gradients is described as C
I 1g S C. Therefore, the approximate gradients generated
by GC follows
[Q(x) 1|2 S
2B
9 12.
Bi
C
Theorem indicates that the E norm of - the gradient derived under quantized feature map is bounded by the product of a constant and the l norm of the original gradient. Under the widely applied assumption bounded gradient and the conclusion of Doublesqueeze [31, 29]. the proposed SGQ approach holds the same convergence order the SGD method without quantization feature maps. The detailed proof can be found in the supplementary material of & C.
4 Experiments
4.1 Experiment Setting
Devices. To match the edge environment, we evaluate SGQ on two types of devices: (1) NVIDIA Jetson Nano series [24], and (2) HUAWEI Atlas 200DK [13], both of which are connected to the NVIDIA RTX 2080Ti server through 10GbE network.
Benchmarks. Our benchmarks are image classification tasks based on the training of AlexNet [18],
VGG-11 [28], ResNet- 8/34 [9], ShuffleNet- V2- 12-1.0x4 1.0x/0.5x [21]. and MobileNet- V1 [10], with the
18/34
1FFAR-10/100 (CF10/100) [17]. Fashion MNIST (FM) [36] and mini- Imai-ImaeeNet (MI) [33] datasets. As to MI, the batch size is 32 with the Adam [16] optimizer. All of these benchmarks are implemented via PyTorchh-1177 17..1 [25].
SGD optimizer. As to CF and FM, the batch size is 100 with
Baselines. We inspect the proposed SGQ method with four pertinent baselines: (1) the vanilla full-precision training (FP32), (2) the uniform quantization (UQ) [41], (3) the product quantization (PQ) [30] and (4) the progressive-slicnng CLIO [12], which uses the top 32 slices corresponding to
X
the same compression ratio of rebit UQ. Note that our SGQ under same representation bits provides
a much higher compression ratio over a The details of traffic saving are in $4.3.
32
4.2 Convergence Efficiency
We inspect the training convergence curves SGQ and other baselines. As shown in Figure we can observe that SGQ (red) achieves the highest accuracy over other baselines using different benchmarks, verifying SGQ general feature compression method and can be applied the training of most CNNs. Even training on the small-scale FM and CF10 datasets, the CLIO-based training (gray) cannot converge under the same compression ratio, although its Top-k like variants are widely used for compressing parameters and gradients in distributed model training. This phenomenon indicates that simply dropping a certain proportion of pixels or channels in feature maps will cause significant information loss and finally destroys the training convergence. Meanwhile, UQ (green) cannot preserve an acceptable model accuracy when training on the large-scale MI datasets, with severe fluctuation along the epochs. This is because UQ needs to flat the entire feature maps before
7

---

quantization, thus losing the spatial characteristics of the pixels among different channels. Inversely, both SGQ and PQ (pink) can maintain good model accuracy as their quantization schemes well maintain the feature structure after pixel encoding. However, our SGQ achieves a faster convergence rate and holds higher model accuracy over the PQ method, because the channel-agnostic PQ cannot capture the difference of orthogonal channels and roughly quantizing features via successive division will introduce quantization errors. For a more straightforward comparison, we summarize the average model accuracy of different method using the same compression ration of 4 bits in Table 1. It is clear that SGQ outperforms other baselines in different training configurations and does not much sacrifice the model accuracy FP32 does. Remarkably, Figure 3 and Table only display the results of 4-bit training, our SGQ method can also achieve good training performance even in extremely low quantization bits (e.g.. bits), which well matches the resource constraints of edge devices.
4.3
Traffic Saving
FM, Alextiet. C:304, 9:32
CF100, VGG11, C:256, B:100
ResNet18/44, C128, 8:32
ShuffleNee-xx1.0. C.100
Traffic Reduction Pato
80
Trlffc Dedluctin Rablo
50-
WLSL
Traffic Reduction Ratio
25
Wil FP32
Traffic Reduction Ralio
35
30
RIL
50
40-
20
25
30-
15
20
20
30
20
10
10
Quantization Bits
Quantization Bits
Quantization Bits
Quantization Bits
Figure 4: Average traffic reduction ratio by using SGQ.
As shown in Figure 4, by presenting two pertinent cases, we highlight SGQ's traffic reduction ratio, over the FP32 training, CLIO (using top 25% slices) and UQ where
Traffic Reduction Ratio Baselines's Traffic
SGQ's Traffic
We did not compare the results under 2 and bits because UQ and CLIO cannot achieve stable model accuracy in such configuration. With the hierarchical channel-spatial encoding. SGQ can compress feature maps more efficiently, thus providing prominent traffic reduction ratios over the baselines. Specifically, we can observe that SGQ's s reduction ratio increases when (1) using lower quantization bits (e.g., 4 bits), (2) cutting at layers with more channels (e.g., C:256), and (3) using larger batch size (e.g., 8. 1:0) In these three cases, the feature maps follow the long strip shape, where conducting channel-level clustering based lower bits will yield smaller quantized features and corresponding codebook, thus significantly reducing the traffic size for transmitting feature maps. However, UQ cannot quantize the pixels along the channel dimension simultaneously, limiting its effectiveness of realistic deployment. Such a degree of reduction ratio makes SGQ an adequate feature compression method for the low-bandwidtt edge environment. Overall, the superiority of SGQ over existing methods can be quickly understood by checking the results in Figure SGQ explicitly outperforms existing methods by achieving better trade-off between model accuracy and traffic size.
100
V9611, FM
100
Mobiihhes VL CFLD
R0
Rosher34, M
Shuffeleedd.. 0
50
92
380
560
2700
U0
250
50
2700
0J1
CUD
B40
40
20
10
Traffic (N))
10
30
Traffic Size (MB)
25
10
20
20
40
50
20
40
50
100
Figure 5: SGQ significantly outperforms existing methods in both model accuracy and traffic size.
4.4 Ablation Study
Impact of quantization bits. We also compare the final model accuracy of SGQ by using different quantization bits and training benchmarks in Figure 6. As the quantization bits directly impact
8

---

GF10, Batch:100
Ml. Batch:32
VGGL1.
100
75-
Acursoy 1661
Accuracy
Accuracy
Acturscy
65
80
Aochhtt
ResNet18 Shufflehet-1.00 Shutteeket-0.k
70
Nesheti8
20
Sita
581
20
5.b
6Bb
NobileNeeiV
4B85
60
50
0
Quantization
Quantization
Epocto
Epochs
Figure 6: Average model accuracy by using different bits, where the 4-bit configuration is recom- mended for SGQ's deployment practice. The detailed convergence curves of each model under different bits are provided in the supplementary material of $ D.
the cluster numbers in feature discretization, lower bits will lead more accuracy degradation. In most cases, SGQ achieves acceptable accuracy under different bits, even for the 2-bit training. (e.g.. MobileNet and ShuffleNet). This property makes SGQ can effectively save traffic size for transmitting feature maps. Note that the training quality is also related to the splitting position of cut layers. Generally, given the same quantization bits, cutting at the shallower layers holding relatively larger feature section (i.e., the product of width and height) but smaller channel number can achieve higher model accuracy. However, the extreme case that features are with wide channels (e.g., 512) while with small pixel plane (e.g., 4 x 4), SGQ's performance may degrade as there is not enough clustering space to conduct feature discretization. Although we can reshape the feature maps to reduce channels and expand the pixel plane, the compression ratio of feature size will also decrease. As the 4-bit SGQ sufficient to reduce communication traffic while maintaining good model accuracy, we recommend to use this as the default training configuration in practice.
FP32
4-bit SGQ
4-bit PQ
Figure The heat maps of the output features generated by different training schemes, which highlight the class-discriminative regions of model's prediction results.
Visualization of quantized features. Based on the training of ShuffleNet- 1.0x with mini- mini- -mae-em under different baselines, employ the Grad-CAMH. [2] method to generate the heat maps of output features and inspect which feature regions impact the model's decision most. As shown in Figure
observe that SGQ holds compact heat map as FP32 does, with similar texture and focal points. However, PQ's heat map skews far away from FP32's focal points with fuzzy texture. This indicates that SGQ can correctly localize significant regions while PQ may make mistakes, thus achieving higher model accuracy over PQ.
4.5 System Overhead
Table 2: Average system overhead proportion (%) of computational time in different SGQ modules.
# Bits
SGQ
CAG's FP
CAG's BP
GC
Total
8 bits
31.93
1.09
1.88
8.67
43.57
6 bits
14.86
1.06
1.76
4.51
22.19
bits (Suggested)
5.51
1.05
1.75
1.62
9.89
2 bits
4.12
1.04
1.73
1.24
8.14

---

Controlling system overhead is also a crucial issue to deploy SGQ on realistic edge devices. As shown in Table we inspect the overhead proportion (%) of computational time cost in different modules during the training procedure. SGQ is the most fundamental module in the forward pass, where the vanilla K-means clustering dominates the computational time. By employing the pixel sampling based on average pooling, we could restrict the overhead proportion within 5.51% % when using the suggested 4-bit quantization. Meanwhile, by employing softmax-base based approximation and affine transformation, the overhead of GC module is well bounded within 1.62% during backward pass, Note that the CAG block involves both forward and backward passes as its FC layers requires parameter updating. CAG block holds slight overhead in these two stages and is independent to quantization bits, thus providing a good extensibility to general CNN models. Based on the three modules, the total overhead of SGQ is controlled in an acceptable range that matches the on-device computational capacities. In practice, we suggest using 4-bit quantization to make a balance between system overhead and model accuracy. In such setting, SGQ method could improve the image processing speed (images/sec) and achieve good speedup from 9.22x to 11.37 x, on average, over the baselines.
5 Conclusion
This work develops new insights into traffic saving to build a communication-efficii collaborative learning paradigm. Unlike previous methods aiming at improving bandwidth utilization or using an unstructured pixel-wise compression, we jointly capture the channel and spatial-level feature redundancy, and conduct a hierarchical compression in these two levels achieve much higher traffic reduction ratio. Specifically, we propose the Stripe-wise Group Quantization (SGQ) method to better leverage the pixel similarity by reorganizing the features into groups based on channel significance, handled by the Chunnel-attention Grouping (CAG) block in forward pass. Meanwhile, we calibrate the gradients of quantized features with a comprehensive theoretical analysis of the convergence rate. Evaluations show that SGQ provides significant traffic reduction methods while not sacrificing much model accuracy under different quantization bits, achieving good training flexibility and communicational efficiency. We believe SGQ can contribute to the further development of edge intelligence applications.
existing
Acknowledgements
This research was supported by fundings from the Key-Area Research and Development Program of Guangdong Province (No. 2021 002180001400003,, Hong Kong RGC Research Impact Fund (No. R5060-19), General Research Fund (No. 152221/19E, 152203/20E, and 152244/21E), the National Natural Science Foundation of China (61872310), and Shenzhen Science and Technology Innovation Commission UCCI120200109114208667
References
[1] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2020.
[2] Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839-847. IEBE Computer Society, 2018
[3] Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Józefowicz. Revisiting distributed synchronous SGD. arXiv preprint, abs/1604.00981 2016.
[4] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. arXiv preprint, atss/8005.0ml
2018
[5] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint, abs/1412.6115,, 2014.
10

---

[6] Jingcai Guo, Shiheng Ma, Jie Zhang, Qihua Zhou, and Song Guo. Dual-view attention networks for single image super-resolution. In Proceedings of ACM International Conference on Multimedia (MM), pages 2728-2736, 2020.
[7] Otkrist Gupta and Ramesh Raskar. Distributed learning of deep neural network over multiple agents. J. Netw. Comput. Appl., 116:1-8, 2018.
[8] Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo. Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1577-1586. IEEE, 2020.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[10] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint, ab4/1704.0 2017.
[11) Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciaation networks. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 7132-7141, 2018.
[12] Jin Huang. Colin Samplawski. Deepak Ganesan, Benjamin M. Marlin, and Heesung Kwon. CLIO: enabling automatic compilation of deep learning pipelines across iot and cloud. In Pro- ceedings of Annual International Conference on Mobile Computing and Networking (MobiCom), pages 58:1-58:12, 2020.
[13] HUAWEI. Atlas 200dk ai developer kit. https://e hww.wi.......................... com/ussproducct/
cloud- cowputingneceeeeeeeee /c/ attas/ attaaaalsssppaa 200, 2020.
[14] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-arithmeticonnn inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2704-2713, Salt Lake City, USA, 2018
[15] Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor N. Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 615-629, 2017.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, USA, 2015.
[17] Alex Krizhevsky, Learning multiple layers of features from tiny images. University of Toronto, 2009.
[18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 1 1116,1114, Lake Tahoe, USA, 2012.
[19] Yue Li, Wenrui Ding, Chunlei Liu, Baochang Zhang, and Guodong Guo. TRQ: networks with residual quantization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 8538-8546. AAAI Press, 2021.
temary neural
[20] Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
[21] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Proceedings of the European Conference Computer Vision (ECCV), volume I 11218 of Lecture Notes in Computer Science, pages 122-138. Springer, 2018.
11

---

[22] Bradley McDanel, Sai Qian Zhang. H. T. Kung, and Xin Dong. Full-stack optimization for accelerating cnns using powers-of-two weights with FPGA validation. In Proceedings of the ACM International Conference on Supercomputing (ICS), pages 449-460, Phoenix, USA, 2019.
[23] Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. In Proceedings of the Advances in Neural Information Processing Systems (NeurlPS), 2020.
[24] NVIDIA. Jetson nano developer kit. https://developper wttp://ellperr....aiiiaaa.a..aaa .widia. com/ com/embedded/ iatsoo-nano-devvllle kit, 2021.
[25] PyTorch. Pytorch: An open source machine learning framework. https://pytorch..rg//
2021.
[26] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of IEEE International Conference on Computer Vision (ICCV),
pages 618-626, IEEE Computer Society, 2017.
[27] Wenqi Shi, Yunzhong Hou, Sheng Zhou, Zhisheng Niu, Yang Zhang, and Lu Geng. Improving device-edge cooperative inference of deep learning via 2-step pruning. In Proceedings of IEEE Conference on Computer Communications (INFOCOM), pages 1-6, 2019.
[28] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego USA, 2015.
[29] Sebastian U. Stich, Jean- BanntBapistt Cordonnier, and Martin Jaggi. Sparsified SGD with memory. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 4452-4463, 2018.
[30] Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
[31] Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang, and Ji Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass crop-ccomeestted compression. In Proceedings of the International Conference on Machine Learning (ICML), volume 97, pages 6155-6165, 2019.
[32] Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, and Seyit Camtepe. Splitfed: When federated learning meets split learning, arXiv preprint, abs/2004. 1088, 2020.
[33] Oriol Vinyals, Charles Blundell, Tim Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 3630-3638, 2016.
[34] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6450-6458, 2017.
[35] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. CBAM: convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), volume 11211, pages 3-19, 2018.
[36] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint, abs/1708.07747, 2017.
[37] Hao Zhang, Zeyu Zheng, Shizhen Xu, Wei Dai, Qirong Ho, Xiaodan Liang, Zhiting Hu, Jinliang Wei, Pengtao Xie, and Eric P. Xing. Poseidon: An efficient communication architecture for distributed deep learning on GPU clusters. In Proceedings of USENIX Annual Technical Conference (ATC), pages 181-193, 2017.
12

---

[38] Kang Zhao, Sida Huang. Pan Pan, Yinghan Li, Yingya Zhang, Zhenyu Gu, and Yinghui Xu. Distribution adaptive INT8 quantization for training enns. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), pages 3483-3491. AAAI Press, 2021.
[39] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of IEFF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2921-2929. IHEE Computer Society, 2016.
[40] Qihua Zhou, Song Guo, Zhihao Qu, Jingcai Guo, Zhenda Xu, Jiewei Zhang, Tao Guo, Boyuan Luo, and Jingren Zhou. Octo: INT8 training with loss-aware compensation and backward quantization for tiny on-device learning. In Irina Calciu and Geoff Kuenning. editors, Proceed- ings of the USENIX Annual Technical Conference (USENIX ATC), pages 177-191. USENIX Association, 2021.
[41] Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified INT8 training for convolutional neural network. arXiv preprini, abs/1912. 12607, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discussed the extreme cases
that may degrade our method's performance in $4.4.
(c) Did you discuss any potential negative societal impacts of your work? [No] We did not discuss the potential negative societal impacts because we believe our research is meaningful to deploy edge intelligence and will bring benefits to industry.
(d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We provided all the assumptions of our theoretical results.
(b) Did you include complete proofs of all theoretical results? [Yes] We provided the detailed proof of our theoretical results in Appendix (supplemental material).
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We provide our demo codes in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We discussed the experimental setting in $4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [N/A] We mainly focused on the results based on average performance.
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, or cloud provider)? [N/A] We discussed the device configuration and benchmarks in $4.1.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All the datasets, models and baselines used our experiments are correctly cited.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yeiWpkG-QWi", "reviewer": "Reviewer_HVqg", "summary": "This work proposes a structured strip-wise quantization method, named Strip-Wise Group Quantization (SGQ), to implement the scalable collaborative learning systems on tiny edge devices through a hierarchical compression on refactored feature structures.", "questions": "1. Dose K-means used for dividing the feature maps into several clusters take a long time? As per my understanding, K-means tend to be way slow for grouping, and the cluster number is also hand-tuned. Now that the Channel-Attention Grouping block is employed to measure and group the entire features, I was wondering if attention could further support dividing them into smaller groups based on pixel similarity instead of using K-means.\n2. Is it possible to employ extra attention, like spatial attention for spatial dimension, as the channel-attention block for channel dimension in this work? If not, why could we do attention to the channel dimension but not the spatial dimension?", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The novelty of capturing such channel-dimension structured information to compress feature size is good, which is different from the viewpoint of the conventional methods.\n2. The authors deliver a clear set of challenges and motivations for deploying CL systems on tiny edge devices.\n\nWeaknesses\n1. This article addresses the communication bottleneck and builds a novel communication-efficient CL system; it is better to bring out a well-through analysis in the experiment section. There are not enough demonstrations regarding communication efficiency.\n2. The analysis of the sublinear convergence order of SGQ is not clear in Section 3.\n3. The ablation study is based on the performance of the final model with different quantization bits, which shows the impact of quantization bits. I think it would be better if the performance of other components of SGQ could be added to analyze their different effects on the final model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qKgzmHhUElB", "reviewer": "Reviewer_B9sp", "summary": "Network bandwidth poses challenges to the collaborative learning system. The existing communication-optimized methods either are insufficient in reducing the traffic or may degrade the model performance. This work proposed a hierarchical compression algorithm for feature data, called Stripe-wise Group Quantization (SGQ). SGQ refactors feature based on similarity in both channel and spatial levels. This work also presented the theoretical analysis for the convergence order of SGQ. Experiments show that SGQ offers a higher reduction ratio without accuracy loss compared to state-of-the-art methods such as Product Quantization (PQ).", "questions": "Please refer to the above weakness part.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This work tried to tackle the communication challenges of the collaborative learning system deployed in reality.  Experiments show that the proposed method SGQ is very practical:\n   + SGQ offers a sufficient reduction ratio in the data traffic\n   + SGQ maintains the model accuracy and the convergence is proved\n   + the overhead of SGQ is only around 8-9% of the total runtime, which is acceptable compared to the communication overhead, and thus SGQ is able to offer 9.22X to 11.37X speedup.\n+ The paper is well organized and easy to follow.\n\nWeaknesses:\n- The evaluation section only shows the traffic reduction ratio and traffic size. It would be better to show the tradeoff between accuracy and actual measured latency and throughput.\n- It is unclear the contribution of each part of SGQ. It would be better to have the ablation study on different parts of SGQ.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YPBdEC5hY7w", "reviewer": "Reviewer_jfAG", "summary": "CL system has the problem of communication deficiency caused by limited bandwidth.\nOther solution (model slicing, matrix factorization) has may degrade the model accuracy if compressing features.\nThis paper propose new quantization scheme, called SGQ (Stripe-wise Group Quantization).\nSGQ captures both channel and spatial-level similarity in pixels, and hierarchically encodes the features in these two levels to achieve a much higher compression ratio.", "questions": "Why didn't you experiment with large dataset (ex. ImageNet)?", "limitations": "CL system has the problem of communication deficiency caused by limited bandwidth.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*strengths\n\nThe idea is straightforward, and the paper is easy to follow.\nThe observation of motivation is good, and the solution makes sense.\n\n*weaknesses\n\nIn main paper, Cifar-10, Cifar-100  and mini-ImageNet are too small datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Er8fFzLXub9", "reviewer": "Reviewer_ofcH", "summary": "In order to improve the communication efficiency in collaborative learning systems, this paper proposes to consider not only the pixel similarities but also channel information by channel-attention grouping. Theoretical analysis is provided to calibrate gradients of both features and CAG blocks in the backward process. Experiments demonstrate higher traffic reduction and image processing speed while keeping comparable accuracy.", "questions": "1. What is the definition of $g_x$ in Theorem 1?\n2. Since additional fully connected layers are added to learn the channel information, how about the increased computation cost when the number of channels and devices is high?\n", "limitations": "The number of clusters $k$ is tied to the bit, then the performance degradation of extreme cases including 1-bit or 2-bit is still a problem.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The observation this work is based on is that some channels hold quite different features when the corresponding filters are orthogonal to each other. This phenomenon is quite interesting and makes it reasonable to design the channel-attention grouping block. \n2. The analysis of gradients in the backward process is thorough and sound.\nExperiments on NVIDIA Jetson Nano and HUAWEI Atlas 200DK demonstrate high efficiency and practical values of the proposed method with comparable performance with the baseline.\nWeaknesses:\nFrom the first few sections and figure 2, we would consider channel-attention grouping is in front of the stripe-wise group quantization, but these two methods are arranged in the opposite order in the methodology section. It seems to be confusing to readers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept