SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now discuss how to encourage NDPCA to work well under various available bandwidths with DPCA during the training phase. We begin by making observations on the desired property of the neural embeddings arising from the limitations of the DPCA: (1) uncorrelatedness: Lemma3.1 shows that DPCA is more efficient when the correlation among the intermediate representations is less. (2) linear compressibility: we encourage the neural autoencoder to generate low-rank representations, which can be compressed by only a few singular vectors, making them more bandwidth efficient.
We tried to explicitly encourage the desired properties with additional terms in [8], but they all adversely affect the task performance. To obtain uncorrelated representations, we tried penalizing the cosine similarity between the representations. We also tried similar losses that penalize correlation, as per [18,21], but none improves the task performance. We observed that the autoencoder automatically learns representations with small correlation, and any explicit imposition of complete uncorrelatedness is too strong. For linear compressibility, we tried penalizing the convex low-rank approximation—the nuclear norm—of the representations, as per [22,23]. However, we observe a similar trend in the final task performance as the network tends to minimize the nuclear norm while harming the task performance. For the comparison of the resulting performance, see Appendix F.1
In this regard, we propose a novel linear compression module that allows us to adapt to DPCA during training rather than using additional terms in the loss. We introduce a random-dimension DPCA projection module to improve performance in lower bandwidths. It projects representations  $Z$  to a low dimension randomly chosen, simulating projections in various available bandwidths during inference. It can be interpreted as a differentiable singular value decomposition with a random dimension, described in Alg. For encoding, it first normalizes the representations and performs singular value decomposition on all sources. Then, it sorts the vectors by the singular values and randomly selects the number of vectors to use for projection. For decoding, it decodes with the selected singular vectors again and denormalizes the data. Note that during training, we only run Alg. on a batch. This module helps to improve the overall performance over a range of bandwidths, and we show the ablation study of this module in Appendix F.2.
Inference: With the training data, the DPCA projection module first saves the mean of representations  $Z$  and the encoder and decoder matrices in the maximum bandwidth. It only needs to save for the maximum bandwidth because its rows and columns are already sorted by the singular values, which represent the importance of each corresponding vector. During inference, when the current bandwidth is  $m$ , it chooses the top  $m$  rows and columns of the saved encoders and decoder matrices to encode and decode representations. No retraining is needed for different bandwidths. Only the storage of a neural autoencoder and a linear matrix at each encoder and decoder is needed.
Robust task model: We pre-train the task model with randomly cropped and augmented images to make the model less sensitive to noise in the input image space, namely, the model has a smaller Lipschitz constant. This augmentation trick is based on [8]. A robust task model has a smaller Lipschitz constant, so it is less sensitive to the input noise injected by decompression when we concatenate it with the neural autoencoder. For a detailed analysis of the performance bounds between robust task and task-aware autoencoders, see Appendix A.2
We consider three different tasks to test our framework: (a) the denoising of CIFAR-10 images [24], (b) multi-view robotic arm manipulation [25], which we refer to as the locate and lift task, and (c) object detection on satellite imagery [26]. Across all the experiments, we assume that there are two data sources, referred to as views, each containing partial information relevant to the task. We present our results based on the testing set and refer to our proposed method, task-aware NDPCA, as NDPCA for simplicity. NDPCA includes a single autoencoder with a large dimension of representations
(a) CIFAR-10: view 1 is less corrupted and thus contains more information about the original images.
Figure 2: Datasets: (column 1) view 1. (column 2) view 2. In all experiments, both views are correlated, but one view is more important than the other as it contains more information relevant to the task.
(b) Locate and lift: Side-view (column 1) faintly captures the absolute position of objects, in contrast to the arm-view (column 2).
(c) Airbus detection: view 1 and view 2 observe different parts of the complete view with overlap.
$Z \in \mathbb{R}^{2 * m_{\max}}$ . It then compresses representations and allocates bandwidth via DPCA, as discussed previously. We show that NDPCA can bridge the performance gap between distributed autoencoders and joint autoencoders, defined below, to allocate bandwidth and avoid transmitting task-irrelevant features. We also provide experiments of NDPCA with more than 2 data sources in Appendix C to demonstrate NDPCA's capability in more complicated settings.
Baselines: We compare NDPCA against three major baselines. First is the task-aware joint autoencoder (JAE), where a single pair of encoder and decoder compresses both views. JAE is considered an upper bound of NDPCA since it can leverage the correlation between both views while avoiding encoding redundant information. Next is the task-aware vanilla distributed autoencoder (DAE), where two encoders independently encode the corresponding views to equal bandwidths and a joint decoder decodes the data. DAE is considered a lower bound of NDPCA since both encoders utilize the same bandwidth regardless of the importance of the views for the task, while
Algorithm 1 Projection into a random low dimension using DPCA
1: Input: A size  $b$  batch of latent representations  $Z_{i} \in \mathbb{R}^{b \times m_{i}}$  from each source  $i$ , min and max bandwidth  $m_{\mathrm{min}}, m_{\mathrm{max}}$   
2: Output: Compressed representation  $Z_{i}^{m}$  of each source, reconstructed representation  $\hat{Z}$  for all sources  
3: function ENCODE( $Z_{i}, m_{\mathrm{min}}, m_{\mathrm{max}}$ )  
4: for each source  $i$  do  
5:  $\bar{Z}_{i} \gets Z_{i} - \mathrm{Mean}(Z_{i})$  ▷Normalize representations  
6:  $s_{i}, V_{i}, H_{i} \gets \mathrm{SVD}(\bar{Z}_{i})$  ▷ Singular value decomposition  
7: end for  
8:  $s, V \gets \mathrm{Cat}(s_{i}), \mathrm{Cat}(V_{i}) \triangleright$  Concatenate singular values and vectors  
9:  $m \gets \mathrm{Rand}(m_{\mathrm{min}}, m_{\mathrm{max}}) \triangleright$  Randomly choose projection dimension  
10:  $s^{m}, V^{m} \gets \arg \max([s, V], m)$  ▷ Select the top  $m$  values of  $s$   
11: for each source  $i$  do  
12:  $V_{i}^{m} \gets \{V | V \in V^{m}, V \in V_{i}\}$  ▷ Select  $m$  vectors from sources  
13:  $Z_{i}^{m} = \bar{Z}_{i} \times V_{i}^{m}$  ▷ Project  $Z_{i}$  to lower dimensions  
14: end for  
15: return  $Z_{\mathrm{low}} \gets \mathrm{Cat}(Z_{i}^{m})$  ▷ Return Compressed representation  
16: end function  
17: function DECODE( $Z_{i}^{m}$ )  
18: for each source  $i$  do  
19:  $\hat{Z}_{i} \gets Z_{i}^{m} \times \mathrm{Cat}(V_{i}^{m})^{\top}$  ▷ Decompressed representation  
20:  $\hat{Z}_{i} \gets \hat{Z}_{i} + \mathrm{Mean}(Z_{i})$  ▷ Denormalize representations  
21: end for  
22: return  $\hat{Z} \gets \mathrm{Cat}(\hat{Z}_{i})$  ▷ Return reconstructed representations  
23: end function
NDPCA allocates bandwidths in a task-aware manner. Last is the task-agnostic NDPCA which differs from NDPCA in the training loss of reconstructing the original views. Due to the novelty of the problem formulation, we cannot make a fair comparison with any of the existing approaches. For instance, [2,4,14] focus purely on distributed compression of images for reconstruction and human perception, whereas [6,7] focus on task-oriented compression but are limited for a single source. Additionally, none of the previous works consider datasets of unequal importance, again making any performance comparison unfair. Hence we focus mainly on an ablation study style of comparison of NDPCA, clearly highlighting and validating the advantages of our approach.
CIFAR-10 denoising: We first consider a simple task of denoising CIFAR-10 images using two noisy observations of the same image, shown in Fig.2(a). We use CIFAR-10 as a toy example to clearly highlight the advantage of NDPCA in the presence of sources with unequal importance to the task. Due to the simplistic nature of the classification task, which only requires 4 bits (digit 0-9) as
Figure 3: Top: Performance Comparison for 3 different tasks. Our method achieves equal or higher performance than other methods. Bottom: Distribution of total available bandwidth (latent space) among the two views for NDPCA (ours). The unequal allocation highlights the difference in the importance of the views for a given task.
the information bottleneck, we choose denoising as our "task", making it more suitable to showcase the performance across a range of available bandwidth. Here, the importance of each observation, or view, for the task is simply the noise level. For view 1, we consider an image corrupted with additive white Gaussian noise (AWGN) with a variance of  $0.1^2$ . And view 2 is highly corrupted by AWGN with a variance of 1. All the images were normalized to  $[0, 1]$  before adding the noise. We compressed the noisy observations and passed the reconstructed images through a pre-trained denoising network. We then computed the final peak signal-to-noise ratio (PSNR) with respect to the clean image. Since the noise levels of both views are unequal, the importance of the task is unequal as well. The optimal bandwidth allocation should not be equal, thus showing the advantage of NDPCA. Although view 1 contains more information, not all bandwidth should be allocated to view 1. This problem is called the CEO problem [27][28]. In fact, even if one view is highly corrupted, we should still leverage that view and never allocate 0 bandwidth to it. We discuss why it is the case in Appendix B
Locate and lift: For the manipulation task, we consider a scenario in which a simulated 6 degrees-of-freedom robotic arm controlled by a reinforcement learning agent inputs two camera views to locate and lift a yellow brick. We call the view from the robotic arm "arm-view" and the one recording the whole desk "side-view", as shown in Fig. 2(b). The two views are complementary to completing the task, details discussed in Appendix F.3. We trained the agent in a supervised-learning manner. We collected a dataset of observation and action pairs [29] and trained an agent from the dataset. Then, we defined task loss as the  $L_{2}$  norm of actions from images with and without compression and trained NDPCA to minimize the task loss through the agent. Literature calls this training method "behavior cloning" [30] as it learns from demonstrations. Behavior cloning causes a drop in performance, but this paper only focuses on the performance degradation caused by compression, so we treat the behavior cloning agent with uncompressed views as the upper bound of our method.
Airbus detection: This task considers using satellite imagery to locate Airbuses. Satellites observe overlapping images of an airport and transmit data to Earth through limited bandwidth, as shown in Fig.2(c). We crop all images in the dataset into smaller pieces  $(224\times 224$  pixels). The two data sources are the upper 160 pixels (source 1) and the lower 104 pixels of the image (source 2) with 40 pixels overlapped. Our object detection model follows the paper "You Only Look Once" (Yolo) [31]. The task loss here is the difference between object detection loss with and without compression.
Results: Our key results are: (1) Task-aware NDPCA outperforms task-agnostic NDPCA, and (2) bandwidth allocation should be related to the importance of the task. Across all experiments, shown in Fig. 3(a)-(c), we see that task-aware NDPCA performs much better than task-agnostic NDPCA
and DAE, which equally allocates bandwidths. We see from Fig. 3 that task-aware NDPCA provides a graceful performance degradation with respect to available bandwidth, with no additional training or storage of multiple models. On the other hand, DAE and JAE require retraining for every level of compression, so every sample point in the plot is a different model.
Fig. 3(a) shows the results of denoising CIFAR-10 with NPDCA trained at  $(m_{\mathrm{min}}, m_{\mathrm{max}}) = (8, 64)$ . Although view 1 is more important than view 2, DAE can only equally allocate bandwidth to both sources. NDPCA compresses the data and flexibly allocates bandwidths, as shown in 3(d), where we can see that  $Z_1$  has more bandwidth than  $Z_2$ . NDPCA results in 1.2 dB gain in PSNR compared to DAE when  $m = 64$ .
Fig. 3(b) shows the results of the locate and lift task with NPDCA trained at  $(m_{\mathrm{min}}, m_{\mathrm{max}}) = (8, 48)$ . We set the length of an episode as 50 time steps and measure the success rate in 100 episodes. We show the upper bound, a behavior cloning agent without compression, in gray dotted lines. The arm view is more important as it captures the precise location of the brick, and as expected, NDPCA allo
Figure 4: Task-aware v.s. task-agnostic: Ground-truth bounding boxes are red (row 1), while detected boxes of task-aware are yellow (row 2). Nothing is detected in the task-agnostic setting (row 3). Task-agnostic images are perceptible to human eyes, while task-aware images capture task-relevant features, thus imperceptible to human eyes.
cates more bandwidth to the arm-view  $(Z_{2})$ , as seen in Fig. 3(e). We see that NDPCA has a  $9\%$  higher success rate compared to DAE when  $m = 24$ .
Fig. 3(c) shows the results of the Airbus detection with NPDCA trained at  $(m_{\min}, m_{\max}) = (8, 40)$ . We measured the mean average precision (mAP) with  $40\%$  confidence score and  $50\%$  intersection over the union as the thresholds. We show the uncompressed upper bound in gray dotted lines. NDPCA results in up to  $14\%$  gain in mAP50 compared to DAE. In Fig. 3(f), we plotted the ratio of the areas of both views, while equally splitting the overlapping part, in a dashed black line. Surprisingly, NDPCA's empirical allocation of bandwidth is highly aligned with the theoretical ratio, supporting that it captures the importance of the task and allocates bandwidth according to it.
Comparison of NDPCA with JAE: JAE uses the information from both views simultaneously to capture the best joint embedding for the task. In an ideal scenario, JAE will be the upper bound for the performance and hence easily performs better than DAE across all the experiments. Interestingly, in Fig. 3(b) and (c), we see that NDPCA outperforms not only DAE but also JAE as well. We attribute it to the better representations present in higher-dimension latent space. It turns out that learning a high-dimensional representation and then projecting to a lower dimension space, like NDPCA, is more efficient compared to directly learning a low-dimensional representation, like JAE. This projection from higher dimensional to lower dimensional is similar to pruning large neural networks to identify effective sparse sub-networks. [32, 33]. We also note that Low-Rank Adaptation (LoRA) [34] technique for large language models can be thought of as a similar approach.
Task-aware v.s. task-agnostic: We plotted the reconstructed images of task-aware  $(\lambda_{\mathrm{rec}} = 0)$  and task-agnostic  $(\lambda_{\mathrm{task}} = 0)$  NDPCA in Fig.4. Task-aware images are imperceptible to human eyes since they restore features of a non-linear task model, aligning with the results in [8] Fig. 4]. For discussion of non-zero  $\lambda_{\mathrm{task}}$  and  $\lambda_{\mathrm{rec}}$ , we refer readers to Appendix E.5
Limitations: In general, autoencoders are poor at generalizing to out-of-distribution data and the drawback translates to NDPCA as well. When the testing set is noticeably different from the training set, the performance of NDPCA can get noticeably lower. Additionally, during training, DPCA performs the singular value decomposition in the training set. The decomposition operation can become ill-conditioned and unstable if the batch size is too small. An alternative approach could be a
parametric low-rank decomposition such as LoRA [34] or using adapter networks [35], although the complexity increases and the compatibility with DPCA remains to be explored.
Information theoretic perspective: Slepian and Wolf et al. are the first to obtain the minimum bandwidth of distributed sources to perfectly reconstruct data [36]. However, they use exponentially complex compressors while assuming that the joint distribution of sources is known, which is impractical. In the presence of a task, finding the rate region of two binary sources has remained an open problem, even for modulo-two sum tasks [12]. In terms of imperfectly reconstructing data with neural autoencoders, previous works consider compression of the original data to a fixed dimension [13, 37], while our work focuses on compressing data to any bandwidth with a task model.
Task-aware compression: Real-world data, such as images or audio, are ubiquitous and high-dimensional, while downstream tasks that input the data only utilize certain features for the output. Task-aware compression aims to compress data while maximizing the performance of a downstream task. Previous works analyze linear task [5], image compression [6-8, 38], future prediction [9], and data privacy [39, 40], while ours compresses distributed sources under limited bandwidth.
Neural autoencoder: Previous works show the ability of neural autoencoders to generate meaningful and uncorrelated representations. Instead of adding additional loss terms during training like [18-21,41], we use a random projection module to help a neural autoencoder learn uncorrelated and linear-compressible representations. Other works focus on designing new neural architectures for multi-view image compression [4,14], while ours focuses on the framework to compress data to different compression levels. We choose autoencoders instead of variational autoencoders [42,43] because we focus on the compression of fixed representations rather than generative tasks from latent distributions. Also, autoencoders are more compatible with DPCA than variational autoencoders.
We proposed a theoretically grounded linear distributed compressor, DPCA, and analyzed its performance compared to the optimal joint compressor. Then, we designed a distributed compression framework called NDPCA by combining a neural autoencoder and DPCA to allocate bandwidth according to their importance to the task. Experiments on CIFAR-10 denoising, locate and lift, and Airbus detection showed that NDPCA near-optimally outperforms task-agnostic or equal-bandwidth compression schemes. Moreover, NDPCA requires only one model and does not need to be retrained for different compression levels, which makes it suitable for settings with dynamic bandwidths.
Avenues for future research include settings where the information flow is not unidirectional but bidirectional, such that the encoders and the decoder can communicate to compress data better. Discovering representations in a more complex space using kernel PCA instead of linear PCA and exploration of more complex non-linear correlations are also left as interesting future work. Another interesting direction to expand the work would be analyzing the robustness of the representations, both in the latent space with respect to corruption such as additive white Gaussian noise (AWGN) as well as with respect to the downstream task model. The current framework learns task-relevant features that are tied to the task model but the performance is expected to drop significantly when the task model is updated or changed. Hence, it is desirable to incorporate robust and transferable properties into the features learned.
This work was supported in part by the National Science Foundation 2133481, NASA 80NSSC21M0071, ARO Award W911NF2310062, ONR Award N00014-21-1-2379, NSF Award CNS-2008824, and Honda Research Institute through 6G@UT center within the Wireless Networking and Communications Group (WNCG) at the University of Texas at Austin. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.