Task-aware Distributed Source Coding under Dynamic Bandwidth
Authors: Po-han Li
Sravan Kumar Ankireddy
Ruihan Zhao
Hossein Nourkhiz Mahjoub
Ehsan Moradi Pari
ufuk topcu
Sandeep P. Chinchali
Hyeji Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: Data Compression
Distributed Source Coding
Semantic Communication
Multi-sensor Networks
Bandwidth Allocation
Information Theory
TL;DR: We design a distributed compression framework which learns low-rank task representations and efficiently distributes bandwidth among sensors to provide a trade-off between performance and bandwidth.

Abstract: Efficient compression of correlated data is essential to minimize communication overload in multi-sensor networks. In such networks, each sensor independently compresses the data and transmits them to a central node. A decoder at the central node decompresses and passes the data to a pre-trained machine learning-based task model to generate the final output. Due to limited communication bandwidth, it is important for the compressor to learn only the features that are relevant to the task. Additionally, the final performance depends heavily on the total available bandwidth. In practice, it is common to encounter varying availability in bandwidth. Since higher bandwidth results in better performance, it is essential for the compressor to dynamically take advantage of the maximum available bandwidth at any instant. In this work, we propose a novel distributed compression framework composed of independent encoders and a joint decoder, which we call neural distributed principal component analysis (NDPCA). NDPCA flexibly compresses data from multiple sources to any available bandwidth with a single model, reducing compute and storage overhead. NDPCA achieves this by learning low-rank task representations and efficiently distributing bandwidth among sensors, thus providing a graceful trade-off between performance and bandwidth. Experiments show that NDPCA improves the success rate of multi-view robotic arm manipulation by 9% and the accuracy of object detection tasks on satellite imagery by 14% compared to an autoencoder with uniform bandwidth allocation.

Task-aware Distributed Source Coding under Dynamic Bandwidth
Po-han Li
Sravan Kumar Ankireddy!
Ruihan Zhao
Hossein Nourkhiz Mahjoub
Ehsan Moradi-Pari"
Ufuk Topeu
Sandeep Chinchali
Hyeji Kim
The University of Texas at Austin
"Honda Research Institute USA
Abstract
Efficient compression of correlated data is essential to minimize communication overload in multi-sensor networks. In such networks, each sensor independently compresses the data and transmits them to central node. A decoder at the central node decompresses and passes the data to pre-trained machine learning- based task model to generate the final output. Due to limited communication bandwidth, it is important for the compressor to learn only the features that are relevant the task. Additionally, the final performance depends heavily on the total available bandwidth. In practice, it is common encounter varying availability in band- width. Since higher bandwidth results in better performance, it is essential for the compressor dynamically take advantage of the maximum available bandwidth any instant. In this work, we propose novel distributed compression framework composed of independent encoders and a joint decoder, which call neural dis- tributed principal component analysis (NDPCA). NDPCA flexibly compresses data from multiple sources to any available bandwidth with a single model, reducing compute and storage overhead. NDPCA achieves this by learning low-rank task representations and efficiently distributing bandwidth among sensors, thus provid- ing graceful trade-off between performance and bandwidth. Experiments show that NDPCA improves the success rate of multi-view robotic arm manipulation by 9% and the accuracy of object detection tasks satellite imagery by 14% compared to an autoencoder with uniform bandwidth allocation.
?
Introduction
Efficient data compression plays pivotal role in multi-sensor networks to minimize communication overload. Due to the limited communication bandwidth of such networks, it is often impractical to transmit all sensor data central server, and compression of the data is necessary. It is important for the sensors to compress the respective data independently, to avoid communication overload in the network. Information theory literature refers to this setting distributed source coding (T). where the goal to recover the original data with minimal distortion. In many cases, the data collected by the sensors is only processed by downstream task model, e.g., an object detection model, but not by humans, and hence the original distributed source coding goal of minimizing reconstruction error is no longer applicable. Instead, the goal should be to maximize the performance of the downstream task model. Additionally, in practice, data collected by multi-sensor networks is often correlated e.g. stereo cameras with overlapping fields of view. To improve communication efficiency. is
"gqual contribution; order decided randomly. Correspondence to (pohanli, sravan.ankireddy @utexas.edu. https///gihub..cmm//AAuuiisssss htt.a//wwwe......eee.oeeeeeeeeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

X1
E
71
DPCA
Task loss
2
8
2
X
X2
B2
DPCA
Inconsistent bandwidth
DPCA
D
Task +
2
8
Frozen weights
Trainable
Xk
Ex
Zz
DPCA
Linear projection Backpropagation
Figure 1: Task-aware distributed source coding with NDPCA. XI...., Xx are correlated data sources. Encoders Ei,...; Eg independently compress data latent representations Z1,-... Zx- Using linear matrices, the DPCA module projects the representations to any lower dimension at the encoder and projects them back to the original data space at the decoder, which allocates the bandwidth of sources based on the importance of the task Q. The goal is to find the optimal encoders and decoder that minimize the final task loss.
important for the compression framework to take advantage of the correlation and avoid transmission of redundant data. Combining both objectives, the final goal of the distributed compression framework is to learn relevant features that maximize the task performance, while avoiding the transmission of redundant features by exploiting the correlation between sources. Together, we refer the distributed compression of task-relevant features as task-aware distributed source coding.
However, existing compression methods fail to combine the following three aspects: 1. Existing distributed compression methods perform poorly in the presence of task model. Although neural networks have been shown to be capable of compressing stereo images 2 and correlated images [.. existing methods focus on reconstructing image data, but not for downstream tasks. 2. Existing task-aware compression methods cannot take advantage of the correlation of sources. Previous works only consider compressing task-relevant features of single source (5+2). but not multiple correlated sources. All existing methods for 1 & 2, especially those based on neural networks, only compress data to a fixed level of compression but not to multiple levels. Thus, they cannot operate in environments with different demands of compression levels and require separate model trained for each compression level. Here, we note that we use the term bandwidth to indicate the information bottleneck in the dimension of transmitted data. Based on the choice of quantization, is straightforward to convert the latent dimension to other popular metrics such as bits per pixel (bpp) in the case of image sources. Additionally, we consider the scenario of total bandwidth constraint for the uplink, which is typical for wireless networks [10].
We design neural distributed principal component analysis (NDPCA)-a distributed compression framework that can transmit task-relevant features at multiple compression levels. We consider the case where task model the central node requires data from all sources and the bandwidth in the net- work not consistent over time, as shown in Fig. In NDPCA, neural encoders E1, Eg...., Ex first independently compress correlated data X1, X1, X2....... ......... .X to latent representations Z1, Zz...... ................................ Zg. A proposed module called distributed principal component analysis (DPCA) further compresses these representations any lower dimension according to the current bandwidth and decompresses the data at the central node. Finally, a neural decoder at the central node decodes the representations 21, 22..... Zi to 82, ... Xx and feeds them into task. Task-aware compression aims to minimize task loss, defined as the difference task outputs with and without compression, such as the difference in object detection results. Due to the significant training cost involved, we avoid training the task model, which is usually a large pre-trained trained neural network.
To highlight our proposed method, NDPCA learns task-relevant representations with a single model multiple compression levels. includes a neural autoencoder generate uncorrelated task-relevant representations in a fixed dimension. It is desirable to learn uncorrelated representations to prevent the transmission redundant information. It also includes a module for linear projection, DPCA, to allocate the available bandwidth among sources based on the importance of the task, by observing the respective principal components, and then further compressing the representations to any desired dimension. By harmoniously combining the neural autoencoder and the linear DPCA module, NDPCA generates representations that are more compressible in limited bandwidths, providing a graceful trade-off between performance and bandwidth.

---

Contributions: Our contributions are three-fold: First, we formulate a task-aware distributed source coding problem that optimizes the compression for a given task instead of reconstructing the sources (Sec.2). Second, we provide a theoretical justification for the framework by analyzing the case of linear compressor and a linear task (Sec. 3}. Finally, we propose task-aware distributed source coding framework, NDPCA, that learns a single model for different levels of compression to handle any type of source and task(Sec. 4). We validate NDPCA with tasks of CIFAR- 10 image denoising, multi-view robotic arm manipulation, and object detection of satellite imagery (Sec.5) NDPCA results in a 1.2dB increase in PSNR, a 9% increase in success rate, and a 14% increase in accuracy compared to autoencoder with uniform bandwidth allocation, for the respective experiments mentioned above.
Problem Formulation
We now define the problem statement more formally. Consider a set of K correlated sources. Let Tú E R. denote the sample from source i where E {1, 2,...,,'' K}. Samples from each source are compressed independently by encoder E, to latent representation that Σ m, m, where is the total bandwidth available. A joint decoder D receives the representations {21, 22,..., } and reconstructs the sources {&1, #2......... 5x). In the setting without task, the goal find of encoders and a decoder recover the inputs { 22, ...,,...; Jh with minimal loss:
E mRm such
K Σ (t) aTask-agnostic distributed source coding),
argmin E1, Ba... DA.D t=1
(1)
where fiee is the reconstruction loss, e.g., the mean-squared error loss.
In the presence of a task Φ. it takes the reconstructed inputs to compute the final output The goal is to find a set of encoders and decoder such that the task loss Crask is minimized, where (( I2....,.... T.k) ] is the task output computed without compression. We refer to this problem as task-aware distributed source coding, which is the main focus of this paper: ((D(E) E2(2)),............ Ex
&g). :2.
argmin Leas ( (D(1), Fg. D.
(Task-aware distributed source coding),
where Crask is the task loss, e.g., the difference of bounding boxes when the task is object detection.
Bandwidth allocation: In the previous formulations,
assume that the output dimensions of encoders are known a priori. However, the dimensions are related to the compression quality of each encoder, which is also design factor. That is, given the total available bandwidth m. first need to obtain the optimal m, for each source i, then, we can design the optimal encoders and decoder accordingly. Finding the optimal set of bandwidths for given task is long-standing open problem in information theory (TT), even for the simple task of a modulo-two sum of two binary sources (12). Also, existing works [4] largely assume a fixed latent dimension for sources and train different models for different total available bandwidth m. which is, of course, suboptimal. In this paper, framework provides heuristics the underlying key challenge of optimally allocating available bandwidth, i.e., deciding m,, while adapting to different total bandwidths m with single model.
3 Theoretical Analysis
We start with a motivating example of task-aware distributed source coding under the constraint of linear encoders, a decoder, and a linear task. We first solve the linear setting using our proposed method, distributed principal component analysis (DPCA). We then describe how DPCA compresses data to different bandwidths and analyze the performance of DPCA. In this way, we gain insights into combining DPCA with neural autoencoders later Sec. [
DPCA Formulation: We consider a linear task for two sources, defined by the task matrix & E Rex( where the sources .i E R. and Zi E Te= are of dimensions ni and nı, respectively, and the task output is given by y dxr E IRP, where assume the sources to be zero-mean. Now, we have observations of two sources X, Ra. xN and
[sr xx2 Without loss of generality, we
X2 1 and their corresponding task outputs Y (xX) e x where X [X] x
3

---

We aim to design the optimal linear encoding matrices (encoders) E1 E Rm Egg E R and the decoding matrix (decoder) D E + (m)+ma) that minimizes the task loss defined as the Frobenius norm of ((X) ((X), ), where X is the reconstructed X. We only consider the non-trivial case where the total bandwidth is less than the task dimension, m my + my < p. i.e. the encoders cannot directly calculate the task output locally and transmit it to the decoder. For now, we assume that m, and mg are given, and we discuss the optimal allocation later in this section.
Letting Z = E X1 E RR,, XN and Z2 = X2 E R denote the encoded representations and M = DD denote the product of the task and decoder matrices, we solve the optimization problem:
E1, E M = argmin MM
| - M
(3a)
Z1
X 22
s.t.
Z
7/2
(3b)
12
(3c)
X1 X2
Y = DD MZ, Y +b
(3d)
Note that solving M is identical to solving the decoder D) since we can always convert M to D by the generalized inverse of task क. The encoders F₂ and Foz project the data representations 7 and Zz in (b). We constrain the representations to be orthonormal vectors in (3c) as in the normalization in principal component analysis (PCA) for the compression of single source (15). This constraint
lets us decouple the problem into subproblems later in (5). Finally, in (3d), the decoder D) decodes Z1 and Zg to X1 and X2 and passes the reconstructed data to task Φ.
We now solve the optimization problem in (3) For any given Fr, Fig (thus, given Z), we can optimally obtain M = YZ (ZZT )-1 = Y by linear regression. Now, we are left to find the optimal encoders Eg, Egg. First, a preprocessing step removes the correlation part of X₃ from X₂ by subtracting the least-square estimator (X1))
Solution:
X2 X2 X2(X)) X2 X2X (x1x)
The orthogonality principle of least-square estimators [16 p.386] ensures that X 0 x We decouple the objective in Baa) with respect to E1, Eg by the orthogonality principle and (3c
min Y 2 = |Y E1,E2
max E1
|| = |
max E1
X E 1 max
1
E 12
E2
(5)
where Y = = = 1 22 |X X2
Vi + Y2. We then have two subproblems from
E argmax
1 X, X
E 11
E argmax
|| Eg 12
Ei
(6)
Fa
(7)
S.L.
X ET
Tmi
s.t.
E,X,X, E = lmz
The two subproblems are the canonical correlation analysis [17, which be solved by whitening
E, X,X, E2X2 and singular value decomposition (see [17] for details).
Dynamic bandwidth: So far, we solved the case for fixed bandwidths m. and m2. We now describe ways to determine the optimal bandwidth allocation given current total bandwidth m. To do so, DPCA solves (6) and 7 with mi = ni and m2 = n2 and obtains Ej, Eý and all pairs of canonical directions and correlations. Canonical directions and correlations can be analogized to more general case of singular vectors and values. Similar to PCA, the sums of squares of canonical correlations are the optimal values of (6) and 77) so DPCA sorts all the canonical correlations in descending order and chooses the first m pairs of canonical correlations and directions. These canonical correlations determine the optimal encoders ET, EE and decoder Dr which indirectly solves mi and m2. Intuitively, the canonical correlations indicate the importance of a direction to the task, and prioritize the transmission of directions by importance. For simplicity, we only consider the case of 2 sources. DPCA can easily compress more sources simply by constraining all Zs to be independent and thus decoupling the original problem (3) more subproblems.
Performance analysis of DPCA: When DPCA compresses new data matrices with encoder E and Eg, the preprocessing step is invalid the encoders cannot communicate with each other. So for

---

To summarize, uncorrelated data matrices XI..... .x are desired for DPCA. If so, DPCA opti- mally decides the bandwidths of all sources based on the canonical correlations, representing their DPCA to perform optimally while skipping the step, the two data matrices need to be uncorrelated, namely, (2) 0, because in such case the preprocessing step removes nothing from the data sources. Given that correlated sources lead to suboptimality of DPCA, we characterize the performance between the joint compression, PCA, and the distributed compression, DPCA, under the same bandwidth in Lemma 3.1 with the simplest case of reconstruction, namely, = lp. this setting, the canonical correlation analysis is relaxed to the singular value decomposition, which is later used for NDPCA in Sec.
Lemma 3.1 (Bounds of DPCA Reconstruction). Given a zero-mean data matrix and its covariance,
X
X1 X
E t Xx
Covn 0
O
Cov12
CoV22
Cov21
Xdion
AX
assume that AX relatively smaller than XX", and X is positive definite with distinct eigenval- ues. For PCA's s encoding and decoding matrices EPCA, DrCA and DPCA's 's encoding and decoding matrices EDPCA DDPCA, the difference of the reconstruction losses is bounded by
Ri R1+ Σ i=m+1
0 5 1 - -DPCA (D)C - ||X -DPcxEPcA(X)E
Aoe AXe:
where Di and e; are the i-th largest eigenvalue and eigenvector of XXT, Tr is the trace function, and m. is the dimension of the compression boutleneck.
The proof of Lemma 3.1 in Appendix A.I Note that AX is the correlation of sources, so as || gets smaller, the difference of PCA and DPCA is closer to 0. That is, as the covariance decreases, DPCA performs more closely to PCA, which the optimal joint compression.
importance for the task. One application of DPCA is that encoders can use the remaining unselected canonical directions improve compression when the available bandwidth is higher later.
4 Neural Distributed Principal Component Analysis
The theoretical analysis in the previous section indicates that DPCA has two drawbacks: it only compresses data optimally if sources are uncorrelated, and it only works for linear tasks. However, DPCA dynamically allocates bandwidth to sources based on their importance. On the other hand, neural autoencoders are shown to powerful tools for compressing data to a fixed dimension but cannot dynamically allocate bandwidth. This contrast motivates us to harmoniously combine neural autoencoder to generate representations and then pass them through DPCA compress and find the bandwidth allocation. We refer to the combination of a neural autoencoder and DPCA as neural distributed principal component analysis (NDPCA). With a single neural autoencoder and matrix at each encoder and decoder, NDPCA adapts to any available bandwidth and flexibly allocates bandwidth to sources according to their importance to the task.
Outline: NDPCA has two encoding stages, as shown in Fig. [ First, the neural encoder at each k-th source encodes data X₁ to a fixed-dimensional representation Zμ for E [K]. Then the DPCA linear encoder adapts the dimension of Z₁ via linear projection according to the available bandwidth and the correlation among the sources as per (6). Similarly, the decoding of NDPCA is also performed in two stages. First, the DPCA linear decoder reconstructs the K need-dimensional representations Z1, Z2..... ŽK, based on which the joint neural decoder generates the estimate of data X1, X2 ...,,... These estimates are then passed to the neural task model & to obtain the final task output Y. Note that since have a non-linear task model here, envision that the neural encoders generate non-linear embedding the sources, while the DPCA mainly adapts the dimension appropriately as needed; the role of the DPCA here is to reliability reconstruct the embedding 2s, which corresponds the case described in Lemma 3.1 with the task matrix & identity.
Training procedure: During the training of NDPCA, the weights of the task are always frozen because it is usually large-scale pre-trained model that is expensive to re-train. We aim to learn the
5

---

K neural encoders and the joint neural decoder which minimize the loss function:
Liot = Atack | YI
+rrec (1121 x111x + [x, - X21l J| xx 11?))
(8)
taak
reconstruction loss
In the task-aware setting when Aver = O, the neural autoencoder fully restores task-relevant features, which is the main focus of this paper. When Atask = O, the neural autoencoder learns to reconstruct the data X, which the task-agnostic setting later compared in Sec.
We now discuss how to encourage NDPCA to work well under various available bandwidths with DPCA during the training phase. We begin by making observations on the desired property of the neural embeddings arising from the limitations of the DPCA: (I) uncorrelatedness: Lemma 3. shows that DPCA more efficient when the correlation among the intermediate representations less. (2) linear compressibility: we encourage the neural autoencoder to generate low-rank representations, which can be compressed by only few singular vectors, making them more bandwidth efficient.
We tried to explicitly encourage the desired properties with additional terms in (8. but they all adversely affect the task performance. To obtain uncorrelated representations, tried penalizing the cosine similarity between the representations. We also tried similar losses that penalize correlation, as per 21]. but none improves the task performance. We observed that the autoencoder automatically learns representations with small correlation, and any explicit imposition of complete uncorrelatedness too strong. For linear compressibility, we tried penalizing the convex low-rank approximation-the nuclear norm-oo of the representations, as per [22] 233 However, we observe similar trend in the final task performance as the network tends to minimize the nuclear norm while harming the task performance. For the comparison of the resulting performance, see Appendix F.1
In this regard, propose novel linear compression module that allows adapt to DPCA during training rather than using additional terms the loss. We introduce a random-dimension DPCA projection module to improve performance in lower bandwidths. It projects representations Z to a low dimension randomly chosen, simulating projections in various available bandwidths during inference. It can be interpreted as a differentiable singular value decomposition with random dimension, described in Alg. | For encoding, it first normalizes the representations and performs singular value decomposition randomly selects the number of vectors to use for projection. For decoding, it decodes with the selected singular vectors again and denormalizes the data. Note that during training, we only run Alg. D on a batch. This module helps to improve the overall performance over range of bandwidths, and we show the ablation study of this module in Appendix
all sources. Then, it sorts the vectors by the singular values and
With the training data, the DPCA projection module first saves the mean of representations Z and the encoder and decoder matrices in the maximum bandwidth. It only needs to save for the maximum bandwidth because its rows and columns are already sorted by the singular values, which represent the importance of each corresponding vector. During inference, when the current bandwidth is m, it chooses the top m rows and columns of the saved encoders and decoder matrices to encode and decode representations. No retraining needed for different bandwidths. Only the storage of neural autoencoder and a linear matrix at each encoder and decoder is needed.
Inference:
Robust task model: We pre-train the task model with randomly cropped and augmented images to make the model less sensitive to noise in the input image space, namely, the model has a smaller Lipschitz constant. This augmentation trick is based [8]. A robust task model has smaller Lipschitz constant, so is less sensitive to the input noise injected by decompression when we concatenate it with the neural autoencoder. For a detailed analysis of the performance bounds between robust task and task-aware autoencoders, see Appendix A.2
5 Experiments
We consider three different tasks to test our framework: (a) the denoising CIFAR-10 images 24 (b) multi-vien robotic arm manipulation (25), which we refer to as the locate and lift task, and (c) object detection on satellite imagery [26). Across the experiments, we assume that there are two data sources, referred to as views, each containing partial information relevant to the task. We present our results based on the testing set and refer to our proposed method, task-aware NDPCA, as NDPCA for simplicity. NDPCA includes single autoencoder with large dimension of representations

---

(a) CIFAR-IO: 10: view 1 less corrupted and thus contains more information about the original images.
(b) Locate and lift: Side-view (column 1) faintyy captures the ab- solute position of objects, in con- trast the am-view (column 2).
(c) Airbus detection: view and view observe different parts of the complete view with overlap.
Figure 2: Datasets: (column 1) view 1. (column 2) view 2. In all experiments, both views are correlated, but view is more important than the other as contains more information relevant the task.
Z E R 2-mmax It then compresses representations and allocates bandwidth via DPCA, as discussed previously. We show that NDPCA can bridge the performance gap between distributed autoencoders and joint autoencoders, defined below, to allocate bandwidth and avoid transmitting task-irrelevant features. We also provide experiments of NDPCA with more than 2 data sources in Appendix C demonstrate NDPCA's capability in more complicated settings.
Baselines: We compare Algorithm Projection into a random low dimension using DPCA
NDPCA against three ma- jor baselines. First is the aask- joint autoen- coder (JAE), where a sin- gle pair of encoder and decoder compresses both views. JAE is considered an upper bound of NDPCA since it can leverage the correlation between both views while avoiding en- coding redundant informa- tion. Next the task- aware vanilla distributed autoencoder (DAE), where two encoders independently encode the corresponding views to equal bandwidths and joint decoder decodes
Input: A size b batch of latent representations Z; E Rxxm from each source i, min and max bandwidth Mmin, Mmax
Output: Compressed representation 2," of each souree, reconstructed
representation 2 for all sources function ENCODE(2 Z,, Y/lmin; Mimm) for each source do
Z, + Z, - Mean 2.) s;, V/, Hi + SVD(Zi)
Normalize representations Singular value decomposition
end for
8, V <- Cat(si), Cat(V.) Concatenate singular values and vectors m < Rand Mmax) b Randomly choose projection dimension
10:
s" V + arg max(s V). > Select the top values of
for each source do VATh (VIV V Vi} Select m vectors from sources Project Z, to lower dimensions
12:
13:
7:TH = x V.T
14: end for
return Zum + Cat(2.")
Return Compressed representation
end function
17:
function DECODE(Z.")
18:
for each source do
the data, DAE is considered a lower bound of NDPCA since both encoders utilize the same bandwidth regard- less of the importance of the views for the task, while NDPCA allocates bandwidths in a task-aware manner. Last the task-agnostic NDPCA which differs from NDPCA in the training loss of reconstructing the original views. Due to the novelty of the problem formulation, we cannot make a fair comparison with any of the existing approaches. For instance, 4 14) perception, whereas 6 7] focus on task-oriented compression but are limited for a single source. Additionally, none of the previous works consider datasets of unequal importance, again making any performance comparison unfair. Hence we focus mainly on an ablation study style of comparison of NDPCA, clearly highlighting and validating the advantages of our approach.
19:
Z, + 2, x C Zi 2, Mean(Z.) end for
> Decompressed representation
20:
Denormalize representations
21:
22:
return 2 + (2i)
Return reconstructed representations
23:
end function
focus purely on distributed compression of images for reconstruction and human
CIFAR-16 10 denoising: We first consider simple task of denoising CIFAR-10 images using two noisy observations of the same image, shown in Fig. (a). We use CIFAR- 10 as toy example to clearly highlight the advantage of NDPCA the presence sources with unequal importance the task. Due to the simplistic nature of the classification task. which only requires 4 bits (digit 0-9) as
7

---

NDPCA (aurs)
DAE
-B-
JAE
Task-agnostic NDPCA
Uncompressed
CIFAR
Locate and Lift
Airbus Detection
23
0.8
0.8
0.7
21
0.6
0.6
MAP50
20
Success
0.5
0.4
1.9
18
0.3
0.2
16
20
20
10
Dimension of Latent
Dimension Latent
Dimension Latent Space
(a)
25
Distrbution bandwicthh
80
Distribution of bandwiddhh
Desribubbon of bandwidth
50
40
15
30
20
10
20
10
16 32 48
64
16 Dimension Latent Space
20
24
12
20
24
32
40
Dimension of Lalent (d)
Dimension
Latent Space
(e)
Figure 3: Top: Performance Comparison for 3 different tasks. Our method achieves equal or higher performance than other methods. Bottom: Distribution of total available bandwidth (latent space) among the two views for NDPCA (ours). The unequal allocation highlights the difference in the importance of the views for given task.
the information bottleneck, we choose denoising as our "task", making it more suitable to showcase the performance across range of available bandwidth. Here, the importance of each observation, or view, for the task is simply the noise level. For view 1, we consider an image corrupted with additive white Gaussian noise (AWGN) with a variance of 0.12. And view is highly corrupted by AWGN with a variance 1. All the images were normalized to [0. 1] before adding the noise. We compressed the noisy observations and passed the reconstructed images through pre-trained denoising network. We then computed the final peak signal-to-B tonlltto-oise ratio (PSNR) with respect to the clean image. Since the noise levels of both views unequal, the importance of the task is unequal well. The optimal bandwidth allocation should not be equal, thus showing the advantage of NDPCA. Although view 1 contains more information, not all bandwidth should be allocated to view 1. This problem is called the CEO problem [27 28 In fact, even if one view is highly corrupted, we should still leverage that view and never allocate O bandwidth it. We discuss why is the in Appendix B
Locate and lift: For the manipulation task, we consider scenario which a simulated 6 degrees-of- freedom robotic arm controlled by reinforcement learning agent inputs two camera views to locate and lift a yellow brick. We call the view from the robotic arm "arm-var "arm-view" and the one recording the whole desk "side-view", task, details discussed in Appendix F3 We trained the agent in a supervised-lean -earning manner. We collected dataset of observation and action pairs (29] and trained agent from the dataset. Then, we defined task loss as the L2 norm actions from images with and without compression and trained NDPCA minimize the task loss through the agent. Literature calls this training method "behavior cloning" (30] as it learns from demonstrations. Behavior cloning causes a drop in performance, but this paper only focuses the performance degradation caused by compression, behavior cloning agent with uncompressed views as the upper bound of our method.
shown in Fig. ( The two views
complementary to completing the
treat the
This task considers using satellite imagery to locate Airbuses. Satellites observe overlapping images of an airport and transmit data to Earth through limited bandwidth, as shown in Fig. 2 (c). We crop all images in the dataset into smaller pieces (224 x 224 pixels). The two data sources are the upper 160 pixels (source 1) and the lower 104 pixels of the image (source 2) with 40 pixels overlapped. Our object detection model follows the paper "You Only Look Once" (Yolo) 31]. The task loss here the difference between object detection loss with and without compression.
Airbus detection:
Our key results are: (1) Task-aware NDPCA outperforms task-agnostic NDPCA, and (2) bandwidth allocation should be related to the importance the task. Across experiments, shown in Fig. Bl ())()), that task-aware NDPCA performs much better than task-agnostic NDPCA
Results:

---

and DAE, which equally allocates bandwidths. We see from Fig. that task-aware NDPCA provides a graceful performance degradation with respect to available bandwidth, with no additional training or storage of multiple models. On the other hand, DAE and JAE require retraining for every level of compression, so every sample point the plot is a different model.
Fig. 3(a) shows the results of denois- ing CIFAR-10 with NPDCA trained at Mminn mmax) = (8, 64). Although view 1 is more important than view 2, DAE can only equally allocates band- width to both sources. NDPCA com- presses the data and flexibly allocates bandwidths, as shown in d)) where we can see that Z1 has more band- width than Zz. NDPCA results in 1.2 dB gain in PSNR compared to DAE when m = 64.
Fig. Bb) shows the results of the lo- cate and lift task with NPDCA trained at (Mmin- Mmax) = (8, 48). We set the length of an episode as 50 time steps and measure the success rate in 100 episodes. We show the upper bound, a behavior cloning agent with- out compression, in gray dotted lines. The arm view is more important as captures the precise location of the brick, and as expected, NDPCA allo- cates more bandwidth to the arm-view (Z2), as seen in Fig. 3(c.. We see that NDPCA has a 9% higher success rate compared to DAE when m 24.
Figure 4: Task-aware v.s. task-agnostic: Ground truth bounding boxes are (row 1), while detected boxes of task aware are yellow (row 2). Nothing is detected the task-agnostic setting (row 3). Task-agnostic images are perceptible to human eyes, while task- aware images capture task-relevant features, thus imperceptible to human eyes.
Fig. (ie) shows the results of the Airbus detection with NPDCA trained at (Mminn Mmmxx = (8, 40). We measured the mean average precision (mAP) with 40% confidence score and 50% intersection over the union as the thresholds. We show the uncompressed upper bound in gray dotted lines. NDPCA results in up to 14% gain in mAP50 compared to DAE. In Fig. 3 (f), we plotted the ratio of the areas of both views, while equally splitting the overlapping part, a dashed black line. Surprisingly, NDPCA's empirical allocation of bandwidth is highly aligned with the theoretical ratio, supporting that it captures the importance of the task and allocates bandwidth according to it.
Comparison of NDPCA with JAE: JAE uses the information from both views simultancously to capture the best joint embedding for the task. In an ideal scenario. JAE will be the upper bound for the performance and hence easily performs better than DAE across all the experiments. Interestingly. in Fig. b) and (c), we see that NDPCA outperforms not only DAE but also JAE as well. We attribute it to the better representations present in higher-dimension latent space. It turns out that learning a high- high-dimensional representation and then projecting to lower dimension space, like NDPCA, is more efficient compared to directly learning a low-dimensional representation, like JAE. This projection from higher dimensional to lower dimensional similar to pruning large neural networks to identify effective sparse sub-networks. [32 33 We also note that Low-Ran Adaptation (LoRA) 34] technique for large language models can be thought of as a similar approach.
Task-aware v.s. task-agnostic: We plotted the reconstructed images of task-aware (Arec = 0) and task-agnostic (Atask = 0) NDPCA in Fig. Task-aware images are imperceptible to human eyes since they restore features of a non-linear task model, aligning with the results in [8] Fig. 4]. For discussion of non-zero Atasi and Arecs we refer readers to Appendix E.5
In general, autoencoders are poor at generalizing to out-of-distribution data and the drawback translates NDPCA as well. When the testing set is noticeably different from the training set, the performance of NDPCA can get noticeably lower. Additionally, during training. DPCA performs the singular value decomposition the training set. The decomposition operation can become ill-conditioned and unstable if the batch size is too small. An alternative approach could be
Limitations:

---

parametric low-rank decomposition such as LoRA [34] or using adapter networks 35 although the complexity increases and the compatibility with DPCA remains be explored.
Related Work
Information theoretic perspective: Slepian and Wolf et al. are the first to obtain the minimum bandwidth of distributed sources to perfectly reconstruct data [36). However, they use exponentially complex compressors while assuming that the joint distribution of sources is known, which is impractical. In the presence of a task, finding the rate region of two binary sources has remained an open problem, even for modulo-two
tasks [12]. In terms of imperfectly reconstructing data with neural autoencoders, previous works consider compression of the original data to a fixed dimension while our work focuses compressing data to any bandwidth with task model.
13 37..
Real-world data, such as images or audio, are ubiquitous and high- dimensional, while downstream tasks that input the data only utilize certain features for the output. downstream task. Previous works analyze linear task (5), image compression [6-8 future prediction (9. and data privacy 39 40). while ours compresses distributed sources under limited bandwidth.
Task-aware compression:
Task-aware compression aims to compress data while maximizing the performance
Neural autoencoder: Previous works show the ability of neural autoencoders to generate meaningful and uncorrelated representations. Instead of adding additional loss terms during training like (18] 21 41 we use random projection module to help a neural autoencoder learn uncorrelated and
compr-compeessible representations. Other works focus on designing new neural architectures for multi-view image compression [ 14), while ours focuses on the framework to compress data to different compression levels. We choose autoencoders instead of variational autoencoders [42 43
because we focus on the compression of fixed representations rather than generative tasks from latent distributions. Also, autoencoders are more compatible with DPCA than variational autoencoders.
7 Conclusion and Future Work
We proposed theoretically grounded linear distributed compressor, DPCA, and analyzed its per- formance compared to the optimal joint compressor. Then, we designed a distributed compression framework called NDPCA by combining neural autoencoder and DPCA to allocate bandwidth according to their importance to the task. Experiments on CIFAR-IO denoising, locate and lift, and Airbus detection showed that NDPCA near-optimally outperforms task-agnostic or equal-bandwidth compression schemes. Moreover, NDPCA requires only one model and does not need to be retrained for different compression levels, which makes suitable for settings with dynamic bandwidths.
Avenues for future research include settings where the information flow is not unidirectional but bidirectional, such that the encoders and the decoder can communicate compress data better. Discovering representations in a more complex space using kernel PCA instead of linear PCA and exploration of more complex non-linear correlations are also left as interesting future work. Another interesting direction to expand the work would be analyzing the robustness of the representations, both in the latent space with respect corruption such as additive white Gaussian noise (AWGN) as well as with respect to the downstream task model. The current framework learns task-relevant features that tied the task model but the performance is expected drop significantly when the task model is updated or changed. Hence, it is desirable to incorporate robust and transferable properties into the features learned.
Acknowledgement
This work was supported in part by the National Science Foundation 2133481, NASA 8ONSSC2 M0071, ARO Award W91 1NF2310062. ONR Award N00014-21-1-2373 NSF Award CNS-2008824, and Honda Research Institute through 6G@UT center within the Wireless Networking and Communications Group (WNCG) at the University of Texas at Austin, Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
10

---

References
[1] A. E. Gamal and Y.-H. Kim. Network Information Theory. Cambridge University Press, 2011.
[2] J. Ballé, V. Laparra, and E. P. Simoncelli. "End-to-end optimized image compression". In: arXiv preprint 1xii::611.001000 (2016).
[3] J. Ballé, D. Minnen, S. Singh, al. "Variational image compression with scale hyperprior".
In: arXiv preprint arXiv: arXiv:/802.00 petp:0/22.033000 (2018).
[4] X. Zhang, J. Shao, and J. Zhang. "LDMIC: Learning- boaning-bbssdd Distributed Multi-view Image Coding". In: The Eleventh International Conference on Learning Representations. 2023.
[5] J. Cheng, S. Chinchali, and A. Tang. "Task-Aware Network Coding Over Butterfly Network".
In: arXiv preprint arXiv: 2201. 11917 (2022).
[6] R. Ji, H. Yao, W. Liu, et al. "Task- Tassk-eepndeett Visual-Codebook Compression". In: IEEE
Transactions
Image Processing 21.4 (2012), pp. 2282-2293.
[7] J. Choi and B. Han. "Task-Aware Quantization Network for JPEG Image Compression". In:
European Conference on Computer Vision. 2020. ISBN: 978-3-030-58565--.
[8] M. Nakanoya, S. S. Narasimhan, S. Bhat, et al. "Co-design of communication and machine inference for cloud robotics". In: Autonomous Robots (2023).
[9] J. Cheng, M. Pavone, S. Katti, et al. "Data Sharing and Compression for Cooperative Net- worked Control". In: Advances in Neural Information Processing Systems. 2021.
[10] P. Liu, J. Jiang, G. Zhu, al. "Training time minimization in quantized federated edge learn- ing under bandwidth constraint". In: 2022 IEEE Wireless Communications and Networking Conference (WCNC). IEEE. 2022, pp. 530-535.
[11] A. El Gamal and Y.-H. Kim. Network information theory. Cambridge university press, 2011.
[12] J. Korner and K. Marton. "How to Encode the Modulo-two Sum of Binary Sources". In: IEEE Transactions on Information Theory 25.2 (1979), pp. 219-221.
[13] J. Whang, Acharya, H. Kim, and A. Dimakis. "Neural Distributed Source Coding". In: (2021).
arXiv preprint arXiv:2
[14] N. Mital, E. Özyılkan, A. Garjani, and D. Gündüz. "Neural Distributed Image Compression with Cross-Attention Feature Alignment". " In: IEEE/CVF Winter Conference on Applications of C'CCmutee Vision. 2023.
[15] I. Jolliffe. "Principal Component Analysis", In: International Encyclopedia of Statistical Science. Springer Berlin Heidelberg, 2011, pp. 1094-1096. ISBN: 978-3-642-04898-2
[16] S. M. Kay. Fundamentals of Statistical Signal Processing: Estimation Theory. Prentice-Hall,
Inc., 1993.
[17] D. R. Hardoon, Szedmak, and J. Shawe-Taylor, "Canonical Correlation Analysis: An Overview with Application Learning Methods". In: Neural Computation 16.12 (2004), pp. 2639-2664.
[18] A. Bardes, J. Ponce, and Y. LeCun. "VICReg: Variance-Invarten for Self SeperSup Learning". In: International Conference on Learning Representations.
Regularization
2022.
[19] A. Singh, E. Garza, A. Chopra, al. "Decouple-and-Samme Protecting Sensitive Information in Task Agnostic Data Release".. In: European Conference on Computer Vision. 2022. [20] K. Bousmalis, G. Trigeorgis, N. Silberman, et al. "Domain Separation Networks". In: Advances in Neural Information Processing Systems. 2016.
[21] R. T. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. "Isolating Sources of Disentanglement in Variational Autoencoders". In: Advances in Neural Information Processing Systems. 2018.
[22] M. Salzmann, C. H. Ek, R. Urtasun, and T. Darrell. "Factorized Orthogonal Latent Spaces". In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 2010.
[23]
M. Fazel. "Matrix rank minimization with applications". PhD thesis. Stanford University,
2002.
[24] A. Krizhevsky. "Learning Multiple Layers of Features from Tiny Images". PhD thesis. Univer- sity of Toronto, 2009.
[25) A. Zhan, R. Zhao, I.. Pinto, et al. "A framework for Efficient Robotic Manipulation". In: Deep
RL Workshop at Advances in Neural Information Processing Systems. 2022.

---

[26] A. Defense and S. Intelligence. Airbus Aircraft Detection, https: / wWW kaggle com/ / datasets airbusgeo/ airbus aircrafts sample dataset [Online; accessed 03- April-2023). 2021.
[27] T. Berger, Z. Zhang, and H. Viswanathan. "The CEO problem". In: IEEE Transactions Information Theory 42.3 (1996), Pp. 887-902.
[28] V. Prabhakaran, D. Tse, and K. Ramachandran. "Rate region of the quadratic Gaussian CEO problem". In: International Symposium on Information Theory. 2004.
[29] R. Zhao, U. Topcu, S. Chinchali, and M. Phielipp. "Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighhor-Cuiide (2023).
Explorations". In: arXiv preprint arXiv: 2302. 14242
[30] F. Torabi, G. Warnell, and P. Stone. "Behavioral Cloning from Observation". In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018. ISBN: 9780999241 127.
[31] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. "You Only Look Once: Unified, Real-Time Object Detection". In: IEEE Conference on Computer Vision and Pattern Recognition. 2016.
[32] J. Frankle and M. Carbin. "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks". In: International Conference on Learning Representations. 2019.
[33] M. Ye, C. Gong. L. Nie, et al. "Good subnetworks provably exist: Pruning via greedy forward selection". In: International Conference on Machine Learning. 2020.
[34] E. J. Hu, yelong shen, P. Wallis, et al. "LoRA: Low-Rank Adaptation of Large Language Models". In: International Conference on Learning Representations. 2022.
[35] N. Houlsby, A. Giurgiu, S. Jastrzebski, al. "Paaramete--efficient transfer learning for NLP". In: International Conference on Machine Learning. 2019.
[36] D. Slepian and J. Wolf. "Noiseless coding of correlated information sources". In: IEEE Transactions on Information Theory (1973).
[37] E. Diao, Ding, and V. Tarokh. "DRASIC: Distributed Recurrent Autoencoder for Scalable Image Compression". In: Data Compression Conference. 2020.
[38] Y. Dubois, B. Bloem-Reddy K. Ullrich, and C. Maddison. *Lossy Compression for Lossless
Prediction". In: Advances in Neural Information Processing Systems. 2021.
[39] P.-h. Li, S. P. Chinchali, and U. Topcu. "Differentially Private Timeseries Forecasts for Net-
worked Control". In: American Control Conference. 2023.
[40] J. Cheng, A. Tang, and S. Chinchali. "Task-aware Privacy Preservation for Multi-dimensiona Data". In: Proceedings of the 39th International Conference Machine Learning. 2022.
[41] W. F. Lo, N. Mital, H. Wu, and Gündüü. "Collaborative Semantic Communication for Edge Inference". In: IEEE Wireless Communications Letters (2023).
[42] D. P. Kingma and M. Welling. "Auto-Encoding Variational Bayes". In: arXiv preprint (2013).
arXiv:1312.6
[43] I. Higgins, L. Matthey, A. Pal, et al. VAE: Learning Basic Visual Concepts with a Con- strained Variational Framework". In: International Conference Learning Representations. 2017.
[44]
F. Rellich and J. Berkowitz. Perturbation theory of eigenvalue problems. CRC Press, 1969.
[45] P. Pauli, A. Koch, J. Berberich, et al. "Training Robust Neural Networks Using Lipschitz Bounds". In: IEEE Control Systems Letters 6 (2022), pp. 121-126. ISSN: 2475-1456.
[46] Y. Huang, H. Zhang, Y. Shi, et al. "Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds". In: Advances Neural Information Processing Systems. 2021.
[47] Ultralytics YOLOv8 Docs. https://docs ultralytics. com/ [Online; accessed 16-April- 2023]. 2023.
[48] A. Buslaev, V. Iglovikov, E. Khvedchenya, al. "Albumentations: Fast and Flexible Image Augmentations". In: Information 1 1.. (2020). ISSN: 2078-2489.
12


Reviews:
[{"review_id": "Py5z8wWBGq", "reviewer": "Reviewer_V3iA", "summary": "This paper proposes a distributed task-specific compression method called NDPCA, composed of both a neural network autoencoder and a linear PCA reconstruction. Given multiple sources of data, NDPCA first compresses the information separately using different independent neural network encoders. Next, it applies a linear distributed encoder based on PCA, which further bottlenecks the information. For each neural encoded source, it projects it into a PCA subspace, such that the total number of dimensions used is equal to a predefined bandwidth $m$. Each source is allocated a different number of dimensions, based on the ranking in the top $m$ singular values. The information is decoded twice, first reprojected back from the PCA subspaces and secondly using a neural network decoder that goes back to the original sources space. The reconstructed sources are input to a task specific network. The authors propose to learn the autoencoders in NDPCA using task-aware losses, that is the task loss should be the same with the reconstructed sources as with the original sources. One of the benefits of the proposed approach is that, because of the use of PCA as an intermediate reconstruction step, one can dynamically choose the total number of eigenvectors used (and, consequently, the bandwidth) without a need for retraining the model. The authors provide a theoretical analysis of their approach and multiple experiments on a number of different tasks that support the methodological choices.", "questions": "1. By using SVD in training, how much overhead does it introduce? In terms of memory/time vs when not using random DPCA in training.\n2. How important is the batch size? You state in the limitations that SVD can become unstable for small batch sizes, but do the results improve with a higher batch size?", "limitations": "The limitations are addressed by the authors.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is written clearly, well structured and easy to follow.\n2. The proposed method, while simple, is novel and achieves good results. The random DPCA module is a nice idea.\n3. The experimental evaluation is done on multiple different tasks, which demonstrated the applicability of the proposed framework.\n4. I like the negative results discussion provided in the paper, relating to (1) uncorrelatedness and (2) linear compressibility.\n", "weaknesses": "1. While the framework presented is described to work for any number of input sources, all the experiments are conducted by considering only 2. I would have loved to see how the method behaves for a larger number of sources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our paper.\n\nWeakness:\n\nBoth our DPCA and NDPCA frameworks have been designed to accommodate multiple sources effectively. In particular, our DPCA linear formulation demonstrates its effectiveness with multiple sources, as outlined in lines 137-139 of the original submission. \n**We also add one additional experiment with NDPCA compressing 4 sources with different sizes of views.** Again, its performance was compared to that of a task-aware vanilla distributed autoencoder (DAE) in the same setting. The results, presented in the global review's attached PDF, reveal NDPCA's superiority over DAE in the Airbus dataset, and NDPCA with two sources slightly outperformed the one with four sources. The distribution of bandwidth to sources is unequal because each source has different importance to the task, which is related to the size of the view.\n\nQuestions: \n1. The memory and computational time overhead of using SVD during training is neglectable compared to the backpropagation of deep neural networks. The reason is that SVD is only a series of matrix operations and the dimension of matrices in SVD is “batch size x dimension of representations”, which makes it easy to compute.\n2. Yes, a small batch size may cause the SVD to be ill-conditioned and unstable as the matrix might have multiple 0 singular values. For a higher batch size, **we do not observe any significant improvement compared to a sufficiently large (stable) batch size**.\na "}, {"review_id": "ws1nBDIQDN", "reviewer": "Reviewer_SHcb", "summary": "This paper presents the neural distributed principal component analysis (NDPCA) method that compresses features from multiple sensor sources with a given total bandwidth limit. NDPCA carries the following novelties. First, it is task-aware. The algorithm trains the compression networks by minimizing the final task. Second, the single model can compress the features at different bandwidth limits. This is achieved by training the networks at the maximum available bandwidth and picking the largest components based on the available bandwidth at inference time.\n\nThe authors evaluated the NDPCA method on three different tasks: 1) denoising of CIFAR-10 images, 2) multi-view robotic arm manipulation, and 3) object detection from satellite imagery. They compared NDPCA against the following baselines: 1) vanilla distributed PCA with equal bandwidth allocation, 2) joint PCA that jointly encodes all sources, and 3) task-agnostic NDPCA. The evaluation result shows NDPCA achieves much better task performance than vanilla distributed PCA and can even outperform joint PCA.", "questions": "It will be useful to provide some references to support the assumption of bandwidth limit.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This paper presents a sound method to compress features from multiple sources given limited bandwidth. The method outperforms the baselines by doing the compression in a task-aware manner.\n\n* The evaluation used three real-world applications.", "weaknesses": "* I have some doubts about the problem setup used in the paper. The paper assumes there is a total availabe bandwith that is shared by all sensor sources, and there is no per-source bandwidth limit (i.e., one can allocate all the bandwith to one single source). It will be useful to provide some references to support this assumption. My concern with this setup is that, to have the communication bottleneck shared by all sources, those are likely very close to each other. If the sensors are close to each other, one can build a joint encoder that encodes the features from all sources jointly, which makes joint PCA a realistic option.\n\n* The evaluation contains only setups with two sources. It will be useful to provide results showing the method's performance with more sources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for their valuable feedback, and we will address each point in detail.\n\nWeakness:\n\n1-1. Our study was inspired by the scenario presented in the introduction, involving multiple distant satellites transmitting data to an Earth-based mission control center. Employing independent encoders for the satellites and a joint decoder for the control center, we mimicked this setup. Given the upcoming IoT era where scattered sensors gather data, our research focused on a similar challenge: effectively utilizing compressed data from individual sensors for joint decoding and processing, which holds significance for future applications.\n\n1-2. Regarding the reviewer's concern about the per-source bandwidth limit, **we would like to highlight that our proposed framework, NDPCA, is adaptable to this scenario.** NDPCA is capable of measuring the importance, represented by singular values, of each dimension from all sources. As a result, we can allocate bandwidth to each source effectively using greedy allocating algorithms, prioritizing the most important available source per source constraints.\n\n2. Both our DPCA and NDPCA frameworks have been designed to accommodate multiple sources effectively. In particular, our DPCA linear formulation demonstrates its effectiveness with multiple sources, as outlined in lines 137-139 of the original submission. **We also add one additional experiment with NDPCA compressing 4 sources with different sizes of views.** Again, its performance was compared to that of a task-aware vanilla distributed autoencoder (DAE) in the same setting. The results, presented in the global review's attached PDF, reveal NDPCA's superiority over DAE in the Airbus dataset, and NDPCA with two sources slightly outperformed the one with four sources. The distribution of bandwidth to sources is unequal because each source has different importance to the task, which is related to the size of the view.\n\nQuestion:\n\nWe examine a scenario where there's an uplink bandwidth limitation, essentially a collective constraint shared by all sensors. For instance, in an IoT setup with 100 sensors and a single central node, this constraint translates to a reception bandwidth limitation for the central node, less than or equal to the combined bandwidth of all 100 nodes. Such a situation is typical in wireless sensor networks.\n\nReference: P. Liu *et al.*, \"Training Time Minimization in Quantized Federated Edge Learning under Bandwidth Constraint,\" *2022 IEEE Wireless Communications and Networking Conference (WCNC)*.\n\nFurthermore, in the satellite network scenario described in our introduction section and our object detection experiments, the bandwidth available to satellites is not on par with that of 5G wireless networks. Satellite network providers like Starlink offer internet speeds ranging from 50 to 250 Mbps, whereas 5G networks can reach speeds of up to Gbps.\n\n"}, {"review_id": "CQT77Se2Gy", "reviewer": "Reviewer_A8qh", "summary": "This paper proposes a solution for multi-view machine learning with distributed computing and limited bandwidth. Different views of data are encoded and then compressed to be transmitted to the decoder for learning tasks. It assigns higher bandwidth for data with better quality to make a tradeoff between different quality of data. Experiment results show higher psnr or reconstruction rate or accuracy than prior art.", "questions": "Why for CIFAR10, only the reconstruction PSRN is reported, not the accuracy of the classification? If this is the case, we could use any image dataset, not necessarily classification dataset, such as DIV2K for image reconstruction.", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem this paper tries to solve is important. The multi-view distributed learning will improve over single-view learning and the communication can be a bottleneck for the system.\n\n", "weaknesses": "The experimented model and data is somehow outdated. For example, CIFAR10 is an old dataset and the paper does not provide the details on the classification network or the accuracy for classification. Detection dataset (airbus) and model (YoloV1) should be updated to larger and modern ones, such as COCO and YoloV6, V7or V8. Though interesting, the method seems, at least to me, to be working on toy examples and models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. \n\nWeakness:\n\nRegarding the use of CIFAR10, we acknowledge its age but included it as a toy example for quick iteration and sanity checks due to its manageable size. Thus, we can try different methods to improve uncorrelatedness and linear compressibility, as presented in the ablation study in the appendix.\n\nRegarding the airbus detection with Yolov1, we also experimented with Yolov8 but found no significant difference between the two. Additionally, we opted for Yolov1 due to its transparency and ease of modification for our setting, supported by abundant open-source resources available online. It is important to note that our paper primarily focuses on **the methodology of data compression for multiple sources**, rather than showcasing state-of-the-art computer vision models.\n\nQuestions:\n\nIn short, CIFAR10 PSNR was selected for its convenience in quick iteration and sanity checks during our research. We use CIFAR-10 as a toy example to demonstrate the use of NDPCA in the presence of sources with unequal importance to the task. Due to the **simplistic nature of the classification task, which only requires 4 bits (digit 0-9) as the information bottleneck**, we choose reconstruction as our “task”, making it more suitable to showcase the performance across a range of available bandwidth. Moreover, we utilized image reconstruction to showcase the ability of NDPCA in task-agnostic settings. While choosing other image datasets is feasible, we already demonstrated the results of NDPCA in the computer vision domain with our Airbus experiments. "}, {"review_id": "o9QAI5W1s4", "reviewer": "Reviewer_dRdb", "summary": "This work targets compressing the correlated data to be communicated in a multi-sensor network. The multi-sensor network pipeline is defined as the following steps: (1) each edge sensor compresses the data and transmits it to a central node, and (2) the central node decompresses the data and passes it to a machine learning task for the final output. Specifically, the authors first formulate a task-aware distributed source coding problem based on the target application. Then, they provide a theoretical justification for the formulated problem and propose a task-aware distributed source coding framework. The experiments on CIFAR-10 denoising, multi-view robotic arm manipulation, and satellite image object detection validate that the proposed framework can achieve better accuracy as compared to an autoencoder baseline under the same compression ratio.\n\n====Post Rebuttal===\nThanks for the rebuttal, I have read the author’s rebuttal. The rebuttal addressed my concern on \"Scalability to different scenarios\" by adding more explanation on Locate and Lift tasks. For the other two concerns (e.g., \"Need more motiving applications\" and \"Lack of compression with recent neural compression methods for images\"), I was not fully convinced by the rebuttal.", "questions": "> + What is the cost of training the proposed NDPCA as compared to methods without training?", "limitations": "See \"Weaknesses\"", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "> + Clear problem formulation: the problem formulation with visual-friendly figures can help the readers easily understand the target application, \n> + Providing theoretical analysis and algorithm table: the provided theoretical analysis and algorithm table helps the readers understand the proposed method in a more structured way.\n> + Comprehensive analysis of experiment results: the experiment settings are well described, and comprehensive analyses are conducted to better show figures and tables.", "weaknesses": "> + Need more motiving applications: although pure theory work can also have a huge impact, it would be great to add more real-world motiving applications for justly that compressing data for multi-sensor networks is an important question, and the proposed method is the key enabler. For example, with more and more convenient Internet access, will the varying and limited bandwidth still be a problem for existing edge devices? Why assume the data will not be decoded for humans but only for computers? What is the key metric to benchmark the multi-sensor network can achieve decent performance? > 99% accuracy on some specific applications or do the machine learning task in a real-time manner? Why is the improvement by the proposed method meaningful? Will the whole pipeline with the proposed method achieve some specific common-agreed metric (e.g., 30FPS  real-time), and the pipeline without the proposed method cannot?  \n> + Lack of compression with recent neural compression methods for images: although this work does not only target compression for images, but at least two of the three benchmark datasets are pure image datasets. Since there exist some works that target encoding images into neural networks (e.g., https://arxiv.org/abs/2006.09661), it will be great to add the comparison and discussion with those works.\n> + Scalability to different scenarios: In the task of “Locate and Lift”, the trends of different methods are not the same trend as the trends on the other two datasets. Specifically, the proposed NDPCA does not always achieve a higher success rate as compared to the baseline DAE. Is it caused by the task type being quite different from image denoising and object detection? Thus, the scalability of different scenarios is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weakness:\n1. In our study, we consider a scenario where satellites send data to a mission control center on Earth, using independent encoders due to their distance apart, with the control center having a joint decoder. Similarly, as the IoT era nears, factories will use sensors in distant locations, demanding efficient machine-learning models to process compressed data from these sensors jointly.\nAs shown in Fig. 4 and the appendix, **our method can trade off human perception and computer features with a weighted loss, but in our main focus, we show results that are fully task-aware.** \nOur primary emphasis stems from the limited communication bandwidth. **To effectively execute machine learning tasks, the bandwidth should prioritize transmitting task-aware features.**\n The key metric in the multi-sensor network setting is the performance of the task with limited bandwidth as we argue that bandwidth is the bottleneck of such settings. In other settings, real-time inference may be essential, but it is not the focus of the paper. The NDPCA framework only adds negligible computation which we elaborate on later.\n\n2. As stated in the related work section, to the best of our knowledge, our work is the only one **focusing on designing a framework to compress data to different compression levels by the same models.** Other previous works have centered on designing new neural architectures for multi-view image compression, using different neural network layers. Given this distinction in focus and setting, we chose not to compare NDOCA with other works and only compared it with JAE and DAE.\n\n3. The Locate and Lift experiment is a reinforcement learning behavior cloning task that is **highly influenced by the initial environmental conditions during both the training and testing phases**. This is also why we showcase the performance of runs from multiple random seeds. The sensitivity is evident from the substantial variance in performance observed in various RL papers, such as the soft-actor-critic paper by Haarnoja et al. in 2018 (Figure 1). In the realm of RL research, researchers typically compare the average performance of different methods across multiple random seeds. In this paper, **the NDPCA method consistently outperforms DAE on average in our specific experimental setting.**\n\nQuestions:\n\nNDPCA has additional DPCA modules compared to vanilla DAE. The DPCA module performs singular value decomposition (SVD) on a batch of data. The memory and computational time overhead of using SVD during training are neglectable compared to the backpropagation of deep neural networks. The reason is that SVD is only a series of matrix operations and the dimension of matrices in SVD is “batch size x dimension of representations”, which makes it easy to compute.\n"}, {"review_id": "vapE5nW8JU", "reviewer": "Reviewer_uwj4", "summary": "This paper studies compression in a distributed computing setting, named neural distributed principal component analysis(NDPCA). The proposed NDPCA can adapt to available bandwidth and flexibly allocates bandwidth to multiple sources according to their contribution to the final task. Experiments demonstrate the effectiveness of NDPCA on bandwidth allocation.", "questions": "In lines 198-199, “We observed that the autoencoder 199 automatically learns representations with small correlation”. Does this observation still exist when there are more encoders (>2)?", "limitations": "No particular limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "By dynamically distributing bandwidth among sensors, NDPCA implements a graceful trade-off between performance and bandwidth, enabling adaptive resource allocation. The experiments conducted in the paper demonstrate that NDPCA significantly improves the success rate of multi-view robotic arm manipulation by 9% and the accuracy of object detection tasks on satellite imagery by 14% compared to an autoencoder with uniform bandwidth allocation.", "weaknesses": "The proposed method is mainly a control algorithm that can adapt to multiple streams of data, which sound like a resource allocation algorithm. The neural processing of data is not essential here.\n\nThe experiments may be too simplistic to be convincing. For example, the three experiments are all in a two-view setting and may not provide a realistic assessment of bandwidth allocation capabilities. Besides, there is no comparison with prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback.\n\nYes, our NDPCA can be interpreted as a resource allocation algorithm, but the key focus of our study is to **harmonically combine linear DPCA modules and neural networks**, leveraging their different purposes. The linear DPCA module measures the importance of sources using singular values, while the neural networks process complex, high-dimensional real-world data, making them essential for real-world applications.\n\nAs stated in the related work section, to the best of our knowledge, our work is the only one focusing on **designing a framework to compress multi-sourced data to different compression levels using the same model.** Other previous works have centered on designing new neural architectures for multi-view image compression, using different neural network layers. Given this distinction in focus and setting, we chose not to compare NDPCA with other works and only compared it with JAE and DAE. **We also add one additional experiment with NDPCA compressing 4 sources with different sizes of views.** Again, its performance was compared to that of a task-aware vanilla distributed autoencoder (DAE) in the same setting. The results, presented in the global review's attached PDF, reveal NDPCA's superiority over DAE in the Airbus dataset, and NDPCA with two sources slightly outperformed the one with four sources. The distribution of bandwidth to sources is unequal because each source has different importance to the task, which is related to the size of the view.\n\nQuestions: \n\nYes, similar trends of small correlations also exist in our additional experiments with 4 sources\n"}, {"review_id": "wIRjTpIfTP", "reviewer": "Reviewer_84yk", "summary": "This paper proposed a task-aware distributed source coding framework called NDPCA (Neural Distributed Principal Component Analysis). This framework aimed to solve the problem of efficient compression of correlated data in multi-sensor networks. In section 2 and 3, the authors provided a formulation of the problem and solved the problem in a linear setting with their proposed method DPCA. They also described ways to determine bandwidth allocation and analyzed the bound of DPCA reconstruction loss in practical conditions. In section 4, the authors proposed NDPCA to generalize their DPCA method to nonlinear tasks by combining a neural autoencoder. They also discussed their training and inference methods to avoid re-training neural networks for different bandwidths. In section 5, the experiments showed performance of proposed framework in three tasks. All experiments assumed two data sources and compared the framework with three other baseline methods. The results showed that task-aware NDPCA performed similar to or better than all baselines in three tasks and had a graceful tradeoff between performance and bandwidth.", "questions": "1.What is the performance of NDPCA when there are more than 2 data sources?\n\n2.What is the performance of NDPCA when using testing dataset? If it is too low, is there an insight why or any possible solutions?\n\n3.In section 5, it is very interesting that NDPCA outperforms JAE, but the reason is only simply explained in the paper. Could you please elaborate a bit?\n\n4.Could you provide a simple theoretical analysis or insights explaining why difference of DPCA reconstruction losses does not harms the result of non-linear tasks?", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The idea of compressing data from distributed sources into different sizes according to their importance is novel and effective. The proposed formulation of the problem and its solution in linear setting serve as a good insight for nonlinear version of the problem.\n\n2.The proposed training and inference methods of NDPCA do not require retraining for different bandwidth, which is convenient and saves computing and storage resources.\n\n3.The experiments in this paper compare the proposed framework to three baseline methods in three different tasks. The authors comprehensively analyze the results to show the advantages of task-aware NDPCA in different practical situations.", "weaknesses": "1.The theoretical analysis in section 2 and 3 only covers situations where encoder, decoder and tasks are linear, which is not a strong support for NDPCA that works for non-linear situations.\n\n2.The experiments assume only two data sources, which is a heavy restriction. The performance of NDPCA when there are more than 2 (or a very large number of) data sources are not presented or analyzed.\n\n3.As described in the last paragraph of section 5, the autoencoders are poor at generalizing out-of-distribution data, which is also the weakness of NDPCA. This paper does not offer a solution to this problem, which will possibly limit the usefulness of NDPCA in practical scenarios where fresh data is generated and transmitted in real-time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments.\n\nWeakness:\n\n1. The key focus of our study is to **harmonically combine linear DPCA modules and neural networks**, leveraging their different purposes. The linear DPCA module is designated to measure the importance of sources with singular values, and the neural networks are designed to process complicated real-world data and work harmonically with the DPCA modules. While there are no theoretical guarantees for their combination, we tested NDPCA on real-world datasets to demonstrate its capability.\n\n2. (also question 1) Both our DPCA and NDPCA frameworks have been designed to accommodate multiple sources effectively. In particular, our DPCA linear formulation demonstrates its effectiveness with multiple sources, as outlined in lines 137-139 of the original submission. **We also add one additional experiment with NDPCA compressing 4 sources with different sizes of views**. Again, its performance was compared to that of a task-aware vanilla distributed autoencoder (DAE) in the same setting. The results, presented in the global review's attached PDF, reveal NDPCA's superiority over DAE in the Airbus dataset, and NDPCA with two sources slightly outperformed the one with four sources. The distribution of bandwidth to sources is unequal because each source has different importance to the task, which is related to the size of the view.\n\n3. (and also question 2) **The results presented in the paper are based on testing sets**, emphasizing the case where fresh data is generated and transmitted in real time. We used data augmentation (using the albumentation package as described in the appendix) to aid the neural networks in adapting to unseen data better, bridging the gap of generalization.\n\nQuestions: \n\n3. We find the results interesting as well. Two perspectives explain the performance difference between NDPCA and JAE. First, NDPCA has two encoders, making its model parameters slightly larger than JAE, which may contribute to better performance. Second, NDPCA uses DPCA modules to compress representations Z, which can effectively remove noise injected in the data, giving it an advantage over JAE in noise removal and performance.\n\n4. The reason is that after DPCA reconstruction, NDPCA uses a decoder to decode the data back to its original space. During training, the encoders and decoder are exposed to the DPCA reconstruction, while the other methods we tried in the appendix section do not, so the DPCA reconstruction only slightly harms the result of non-linear tasks. Of course, it is impossible to not harm the result of non-linear tasks as data is always compressed.\n"}]

Author rebuttal:
This is the attached file to show the latest results from our additional experiments with 4 data sources.

Meta-review: The paper proposes a neural distributed principal component analysis (NDPCA). It's a distributed compression framework composed of independent encoders and a joint decoder. NDPCA can compress data from multiple sources to any available bandwidth. Reviewers were generally positive about the submission. The concern raised by the more concerned reviewer were according to the opinion of the AC addressed in the author feedback.
Decision: Accept (poster)