Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper "Lossy Compression for Lossless Prediction" considers the problem of data compression for downstream predictive tasks, rather than preserving the perceptual quality of reconstructions, in order to achieve significantly higher compression rate with negligible or no performance loss in  downstream tasks. In particular, the paper theoretically characterizes the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentation, and derives two corresponding variational objectives, VIC and BINCE, for training end-to-end neural compressors. The method is validated empirically on image tasks, both training from scratch and adapting an existing SOTA vision transformer model with an entropy bottleneck, and is shown to achieve superior compression rates than both traditional and learned image compression baselines.
main_review: **Originality**

Although the paper is most similar in spirit to "End-to-End Learning of Compressible Features" (Singh et. al., 2020), and in fact Singh et. al.'s method  can be seen as a special case of the proposed VIC method (where there is only one downstream task of interest and thus no data augmentation), the current paper broadens the scope considerably by considering lossy compression for a *collection* of related invariant tasks, and is therefore a novel and original contribution.


**Quality and Clarity**

The paper is clearly written and well-organized. The theoretical results appear to be rigorously derived (in the supplementary material). However, the discussions in the experiments section leave a few things to be desired. In particular,
1. In section 5.2, the worse performance of VIC (e.g., its inability to achieve zero distortion) compared to BINCE can benefit from a better explanation; and more generally, a comparison between the two formulations and discussions on their pros and cons would be beneficial. 
The worse performance of VIC was attributed to the channel averaging operation of the ResNet18 encoder used; however, since BINCE does not perform reconstructions (unlike VIC) and thus does not suffer from the poor encoder, it would make a more fair comparison by evaluating VIC performance using the compressed Z representation (with a separately trained predictor) as done for the evaluation of BINCE.  It would be reassuring to see that VIC is able to achieve comparable performance to BINCE, e.g., by increasing the expressivity of the variational family.

2. It's not clear how the performance of zero-shot compressor using a pre-trained SSL model compares to end-to-end training with the proposed method (either VIC or BINCE). The already good zero-shot performance from fitting an entropy model to the latent representations of a pre-trained model (in section 5.2) seems to render end-to-end training with VIC or BINCE unnecessary. 

3. It's not clear how the pretraining of CLIP fits into the BINCE framework (e.g., it's not clear what kind of data augmentation transform $A$ is used during training).

**Significance**

As far as I'm aware, this paper is the first to consider the problem of learned lossy compression in a multi-task setting, and lays a solid foundation for future work (both in theory and methods), thus represents a significant contribution. 

limitations_and_societal_impact: **Limitations:** See the various points in the "Quality and Clarity" section above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes methods to introduce a self-supervised approach to neural image compression for classification tasks.
VIC is a modified neural compressor in which inputs are augmented but target reconstructions are not.
BINCE is a modified neural compressor same as VIC but trained by an entropy bottleneck and an InfoNCE [15] objective. This paper also introduce and evaluate a zero-shot compressor using pre-trained self-supervised models CLIP [38].
main_review: I think it is a novel idea to introduce a self-supervised approach to neural image compression for classification tasks and theoretical background is well described.

But I am curious about the following points.
- I don't know the unit of "bit-rate" in table 3 (it not seems to bit per pixel, is it bit per image?). Please write a clear definition.
-  I don't think this paper (the main paper) adequately describes how the quantizer is handled. However, the Appendix describes the quantizer process with a linear transformation using the learnable parameter. Also, you are probably using a uniform noise quantizer [24], but I think these need to be explained in the main paper.
limitations_and_societal_impact: The caveats are well described in this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The author proposes a compression algorithm by extracting invariant information from the data source, which significantly reduces the file size for different downstream tasks that share similar structures while maintaining high prediction accuracy.
main_review: pros:
1. The idea is interesting and novel. Compared to other compression work that is aimed at reconstruction quality, the author focus on extracting critical (and of course compressible) information from source data; the extracted data can be applied to different downstream tasks without further fine-tuning.
2. The author proposes 2 invariant neural compressors for this compression scheme; both of them show competitive rate-distortion(or rate-accuracy) results on different tasks.
3. Theoretical Justification

questions:
1. In table 3. Higher $\beta$ should correspond to lower distortion, but the table shows higher distortion and lower rate?
2. I wonder how the augmentation methods can influence the rate-distortion performance? The tasks seem to be highly correlated to the augmentation methods we use. For image classification, we can easily use different transforms, but for tasks like image captioning, it could be hard to find the right way to augment the data.
3. The pseudocode for Algorithm 1 is not clear for me. line 5- line 7: $\tilde{\textbf x}$ and $\textbf x$ are not the same thing? To make sure if I understand correctly, $A(x)$ and $A(x^-_i)$ use the same augmentation method and a $x^+$ is augmented differently.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes lossy compression algorithms that only compress task-relevant information in order to obtain a better bit-rate while maintaining the performance of the downstream task. Specifically, the paper proposes to augment the training samples before encoding, and the decoder aims to reconstruct the original image (another proposed model uses contrastive loss to do this). In this way, the model can learn representations that are invariant of the input augmentations, which could lead to smaller bit-rate.
main_review: Given that transfer learning is of great importance in machine learning, the task of compressing data only for certain downstream tasks is definitely worth exploring. However, though the paper made a good effort to formulate this problem, I think the experiments used in the paper might be insufficient to demonstrate the effect of their proposed technique — performing data augmentation during training leads to better lossy compressors:

 - The author claimed in the introduction that they aim to “ensure good performance on any future tasks of interest”. This setup was only tested in the CLIP experiment in Section 5.3, where the authors trained a compressor and validated on 8 datasets unseen during training. However, the good results may simply come from the fact that the latent representations of CLIP (a large pre-trained model analogous to BERT) already have a low bit-rate. That is, training any lossy compressor with the latent representations as input could lead to similar rate gains.

To address this concern, I would recommend the authors to at least compare the bit-rate also with other modern lossy compressors trained on latent representations generated by CLIP. It would also be great if the authors to demonstrate the performance gain provided by different data-augmentation techniques. In general, only comparing with JPEG does not provide useful information about how good the proposed model is since the goal of JPEG is to compress images while not changing its visual appearance by too much.

 - Need to justify the proposed models is not simply compressing the labels. For example, in the experiment in Section 5.2, performance drop and compression gain are compared. However, even by simply compressing the labels, one can get zero decreases in test accuracy and achieve a much better compression gain. The proposed models (VIC, BINCE) are of course not doing this explicitly. However, I am a bit concerned that this could be accidentally done implicitly. For example, consider an MLP with hidden layer size 32 - 16 - 8. If the bottleneck is set at the layer with 32 hidden neurons, the bit-rate could be worse than when the bottleneck is set at the last layer (8 hidden neurons). 

I think the authors should design a better evaluation metric to avoid the above trivial case.

 - Overall, I like the high-level idea of the paper, but it seems hard for me to separate the cause of the bit-rate gain: is it caused by the good latent representations learned by SSL models, or the proposed invariant compression technique plays a critical role there? For example, if we have a powerful SSL model trained to classify ImageNet samples, then CIFAR10 can be faithfully solved only by looking at the 1000-class imagenet label. I believe this can be a good paper if this concern is addressed, and I would be willing to increase my score if the authors convince me during rebuttal.
limitations_and_societal_impact: Limitations are discussed in Section 7. This includes the methods' irrecoverable loss of information, and instability when facing task changes.
review_title: null

Meta-review: The paper analyzes "supervised" data compression for downstream predictive tasks, aiming to achieve higher compression rates with negligible or no performance loss. It theoretically characterizes the bitrate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentation, and derives two corresponding variational objectives, VIC and BINCE. The method is validated empirically on image tasks, both training from scratch and adapting an existing SOTA vision transformer model with an entropy bottleneck.

The method can be seen as a generalization of "End-to-End Learning of Compressible Features" (Singh et al., 2020) but significantly expands its scope by considering lossy compression for a collection of related invariant tasks and by providing a rigorous information-theoretical justification of the approach. Original weaknesses pointed out by the reviewers were sufficiently addressed in the review period. Experiments are thorough and exhaustive. 
Decision: Accept (Spotlight)