Lossy Compression for Lossless Prediction
Authors: Yann Dubois
Benjamin Bloem-Reddy
Karen Ullrich
Chris J. Maddison
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Compression
Invariances
Information Theory
Machine Learning
Self-Supervised Learning

Abstract: Most data is automatically collected and only ever "seen" by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, we characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations. Based on our theory, we design unsupervised objectives for training neural compressors. Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than 1000x on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.

Lossy Compression for Lossless Prediction
Yann Dubois
Benjamin Bloem-Reddy
Vector Institute yamndubois99@gmiil.ccmmm
The University of British Columbia benbrāstat.uke citiitat.bbb.c. ubc.ca ca
Karen Ullrich
Chris J. Maddison
Facebook Al Research
University of Toronto Vector Institute cmaddis@cs. toronto.
karenuêfb. com
Abstract
Most data is automatically collected and only ever "seen" by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations. Based unsupervised objectives for training neural compressors. Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than x on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.
theory, we design
Introduction
Progress in important areas requires processing huge amounts of data. For climate prediction, models still data-limited [1. despite the Natl. Center for Computational Sciences storing 32 million gigabytes (GB) of climate data (2). For autonomous driving, capturing a realistic range of rare events with current methods requires around trillion GB of data At these scales, data are only processed by task issk-specific algorithms, and storing data in human mumann-raae readable formats can be prohibitive. We need compressors that retain only the information needed for algorithmic execution of downstream tasks.
Existing lossy compressors are not up to the chal- lenge, because they aim reconstruct the data for hu- man perception [0) However, much perceptual information is not needed perform the tasks that we care about. Consider classifying images, which require about I MB store. Classification is typ- ically invariant under small image transfornations. such rescalings or rotations, and could instead be performed using representation that discards such information (see Fig. T.. The amount of unnecessary perceptual information is likely substantial, as illus- trated by the fact that typical image classification can be performed using detailed caption, which requires only about kB to store (1000x fewer bits).
4
Source
Standard rec.
Our rec.
Figure 1: Our unsupervised coder improves compression by only keeping information nec- essary for typical tasks. (left) source aug- mented MNIST digit: (center) neural percep- tual compressor achieves 130 bit-rate; (right) our invariant compressor achieves 48 bit-rate.
Kalra and Paddock (3] estimated that autonomous vehicles would have drive hundreds of billions of miles demonstrate reliability rare events, At TB hour miles hour. this 3e12 GB.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Our goal is to quantify the bit-rate needed to ensure high performance on collection of prediction tasks, In the simple case of a single supervised task, the minimum bit-rate is achieved by compressing predicted labels, and essentially corresponds to the Information Bottleneck (IB: [II.. Our challenge, instead, is to ensure good performance on any future tasks of interest, which will rarely be completely known at compression time, or might be too large to enumerate.
We overcome this challenge by focusing on sets of tasks that are invariant under user-defined transformations (e.g., translation, brightness, cropping), as is the case for many tasks of interest to humans I22 [3]. This structure allows us to characterize worst-case invariant task, which bounds the relative predictive performance on all invariant tasks. As result, the bit-rate required to perform well on all invariant tasks exactly the rate compress the worst-case labels. At high level, the worst-case task is to recognize which examples are transformed versions of one another, and rate savings come from discarding information from those transformations.
We also provide two unsupervised neural compressors to target the optimal rates. One is similar a variational autoencoder [14] that reconstructs canonical examples (Fig. Ip. Our second is
simple modification of contrastive self-supervised learning (SSL; (15)), which allows us to convert
pre-trained SSL models into powerful, generie compressors. Our contributions are:
We formalize the notion of compression for downstream predictive tasks.
We characterize the bits needed for high performance any task invariant augmentations.
We provide unsupervised objectives to train compressors that approximate the optimal rates. We show that compressor outperforms JPEG by orders of magnitude was never trained (i.e., zero-shot). E.g., on ImageNet [16], it decreases the bit-rate by 1000
datasets which it
2 Rate-distortion theory background
The goal of lossy compression theory is to find the number of bits (bit-rate) required to store outcomes r of a random variable (r.v.) X, so that it can be reconstructed within a certain tolerance. This is accomplished in [Shannon [17] rate-distortion (RD) theory by mapping X into r.v. Z with low mutual information I[X; 21. Specifically, given a distortion measure D[X, 2], the RD theory characterizes the minimal achievable bit-rate for a distortion threshold ó by
Rate( = min 1[X; Z] such that D[X, Z| S 8.
p(2 x)
In lossy compression, Z is usually a reconstruction of X, i.e., it aims to faithfully approximate X. As a result, typical distortions, e.g., the mean squared error (MSE), assume that the sample spaces X Z of both v.v.s are the same. This assumption not required. Indeed, any distortion d X x - Rgo of the form D X, Z| Ex(x.2 2) [d[X Z)]. where there exists = E Z such that D[X;=] is finite, is valid choice [8]. This shows that RD theory can be used outside of reconstructions. In the following
we refer to Z as compressed representation of X to distinguish it from a reconstruction.
3 Minimal bit-rate for high predictive performance
In this section, we characterize the bit-rate needed to represent X to ensure high performance on downstream tasks. Our argument has three high-level steps: (i) define distortion that controls downstream performance when predicting from Z instead of X; (ii) simplify and validate this distortion when desired tasks satisfy an invariance condition; (iii) apply RD theory with the valid distortion. For simplicity, our presentation is relatively informal; formal proofs are in Apps. A and
3.1 A distortion for worst-case predictive performance
Suppose is an image. Potential downstream tasks might include Yuog. whether the image displays dog; or Yıs, whether the image is hand-drawn. Formally, these and other downstream tasks are expressed as T = (Yáng: Yia,...,, J. a set of random variables that are jointly distributed with X. Let R[Y X] denote the Bayes (best possible) risk when predicting Y from X. For ease of presentation in the main paper, we consider only classification tasks T and Bayes risk of the standard log loss R[Y X] := inf, Ep(x,r) !- log q [X]]. We deal with MSE and regression in Appx. B.6

---

x
(ERM)
1
(() |||||
M(x)
(42)
M(a)
f(x)
|=]
(a) Rotation
(b) Scaling
(c) Any transformation
7
7
8
X
7
M()) Σ ()
((a)
( unauumenddss)
camon(())
(d) Permutation
(e) Graph isomorphism
(I) Any data augmentation
Figure 2: Maximal invariants (X) are representatives cquivalence classes. Example Ass include the: (a) Euclidean norm for rotations; (b) unit vector for scaling; (c) f when equivalence classes pre-images by f: (d) empirical measure for permutations; (e) canonical graph for graph isomorphisms; (f) unaugmented input for data augmentations.
In this setting, meaningful distortion [,[X, Z| quantifies the difference between predicting any E T from the compressed Z, as opposed to using X. This the worst-case excess risk,
D,(X, Z] = sup YET
R[Y Z RY
(2)
[x, = O, is possible achieve lossless prediction: performing as well using Z as using X. More generally, bounding D, by 6 ensures that R|Y Z R X] < for all tasks in T. However, there are two issues that need be addressed before Eq. (2) can be used. First, it is not clear whether D, is valid distortion for RD theory. Second, the worst excess-risk D, assumes access all downstream tasks of interest during compression, which is unrealistic in general.
3.2 Invariant tasks
The tasks that we care about are arbitrary, and often share structure. One such structure is invariance to certain pre-specified transformations of input data. For example, computer vision tasks often invariant to mild transformations such as brightness changes. Such invariance structure is common in realistic tasks, as seen by the wide-spread use of data augmentations (13) in machine learning (ML), which encourage predictions be the same for an unaugmented and an augmented xt Motivated by this we focus on sets of invariant tasks T.
We consider general notion of invariance, namely invariance specified by equivalence relation ~ on x The equivalence induces partition of A' into disjoint equivalence classes, and we are interested in tasks whose conditional distributions constant within these classes.
Definition 1. The of invariant tasks of interest with respect equivalence (X²,,), denoted T.,, is all random variables Y such that & ~ xt - p(Y x)) = p(Y |x) ) for any x, xt E X..
3.3 Rate-distortion theory for invariant task prediction
The key to simplifying Dr. is the existence of (non- (non-unique) worst-case invariant task, denoted M(X. Such task contains all and only information to which tasks Y E T., are not invariant; we call them maximal invariants. A maximal invariant M with respect is any function satisfying
M(x) = M(x*) for any x, e X.
(3)
2 As reminder, an equivalence relation 2 ~ x' + ~u æ, and (transitivityy) x ~ x' and x' x" = T x"
x, a' x
N: (reflexivity) æ (symmetry)
"This extends the definition of maximal invariants [9] beyond invariances group actions.

---

A maximal invariant removes all information that tasks are invariant to, as it maps equivalent inputs to the same output, i.e. MM (z) = (M Yet, it retains the minimal information needed to perform invariant tasks, by mapping non-equivalent inputs of x to different outputs M(x) * Mx="). In other words, (x) indexes the equivalence classes. For example, the Euclidean norm is a maximal invariant for rotation invariance, as all vectors that are rotated versions of one another can be characterized by their radial coordinate. For data augmentations, the canonical (unaugmented) version of the input is maximal invariant. Other examples are shown in Fig.
We prove in Appx. B.2 that under weak regularity conditions, maximal invariant tasks exist in T and that they achieve the supremum in Eq. (2) This allows us to show that Dr., reduces to the Bayes risk of predicting M (X) from Z and that it is valid distortion measure. Crucially, this allows us to quantify downstream performance without enumerating invariant tasks.
Let (X', be an equivalence relation and M a maximal invariant that takes at most countably many values, with H M (X)]
Proposition 1.
< bc. Then D,- 2 with log loss a valid distortion and
Dr_ Z| = (XM 2]
(4)
Here we used [[M(X) 0, as M is deterministic function. Also, note that the countable requirement holds when tasks are invariant to some rounding of the input, as is typically the case due to floating-poin point storage. We accommodate the uncountable case for squared-error loss in Appx. B.6
With a valid distortion in hand, we invoke the RD theorem with D,,, to obtain our "Rate- IntaraInvarienee (RI) theorem. The RI theorem characterizes the bit-rate needed to store X while ensuring small log-loss on invariant tasks. We obtain analogous results for squared-error loss.
Theorem Invaiannvar, Assume the conditions of Prop. 1
Let d 2 0, and Rate(5) denote the minimum achievable bit-rate for transmitting Z such that for any Y E T. we have R|Y Z] R[Y| X] S Then Rate(0) = 0 if 2 HX)( and otherwise it is finite and
Rote(() = (X)(
8
H[X]
H[X (MX) - 8. (5)
information needed to prodict T~
acceptable decrease prodictive loss
standard compression
gains due to invarninces
To ensure lossless prediction, i.e., Rate(O), our theorem states that we require a bit-rate of (XM( Intuitively, this is because (X) contains the minimal information needed to predict losslessly any Y E T_ Furthermore, the theorem relates compression and prediction by show- ing that allowing a 8 decrease in log-loss performance on all tasks can save exactly ö bits. Intuitively, this is linear relationship, because expected log-loss is measured in bits. On the right of Eq. (5) we further decompose H[M (X)) into two terms to provide another interpretation: (i) H[X], which, for discrete X, the bit-rate required to losslessly compress X, and (ii) H[X M (X)J, which quantifies the information removed due to the invariance of desired tasks. Importantly, removing this information does not impact the best possible predictive performance. See Fig.
BIX
H[ M(X) gains due to invarancc
H[M[
increasing threshold 8
decreases by 8
Retel&)
HMM(X)
Distortion threshold 5
Figure 3: Rate-Invariance function.
The bit-rate gains can be substantial, depending on the invariances. Consider compressing sequence of n i.i.d. fair coin flips. Suppose one only interested in predicting permutation invariant labels. Then instead of compressing the entire sequence in H = n H[x] = n bits, one could compress the number of heads, which is maximal invariant for permutation invariance, in Olog n) bits.) As more interesting examples, we recover in Appx. B.4fresults from (i) unlabeled graph compression [0 (ii) multiset compression [21]: (iii) single task compression (IB: [T]. The equivalence ~ can be induced by any transformations, such as transforming an image to its caption. We use this idea in Sec. 5.3 to obtain >1000x x compression on ImageNet without sacrificing predictive performance.
"We prove in Appx. B.5 that Hate(O) (X)) for any losses used in ML.
'The number of heads K follow a binomial distribution so H[K] E Ologgn). Here K is also a minimal sufficient statistic for x More generally, if P(X') is invariant to ~ and ~ is the coarsest such relation, then minimal sufficiency coincides with maximal invariance. In practice, however, P(X') will rarely be ~ invariant.

---

Entropy bottleneck eusureee compresed &
VIC distortion www.eeieeee...
BINCE distortion
5
5
Augment
Encode
reconstructs unangmented
discriminates augmented
X
0
(2|x)
Distortion
data qe (M)
data
1o(2,
X
Batch
Figure 4: Our unsupervised objectives for invariant image compression under data augmentation the same encoder, but differ in their approximation to the invariance distortion. Both models encode the augmented data, pass the representation through an entropy bottleneck which ensures that they are compressed, and use a distortion to retain the information about the identity of the original data. The models differ in how they retain that information: (VIC) by reconstructing unaugmented inputs; (BINCE) by recognizing which inputs come from the same original data.
Unsupervised training of invariant neural compressors
In this section, we design practical, invariant neural compressors that bound optimal rates. Derivations are in Appx. C In particular, we are interested the arg min encoders p( ( of the RD function (Eq. (D) under the invariance distortion D To accomplish this, we can optimize the following equivalent (i.e., it induces the same RI function) Lagrangian, where /3 takes the role of 6
arg min x(4x)
I[x; 2] + B R[M(X) |2)
(6)
In ML, the maximal invariant M is often not available. Instead, invariances are implicitly specified by sampling a random augmentation from A, applying to a datapoint X, and asking that the model's prediction be invariant between X and A(X). For example, invariance to cropping can be enforced by randomly cropping images while retaining the original label. We show in Appx. C that in such case, can treat the augmented A(X) as the new source, Z as the representation of A(X), and the unaugmented X as the maximal invariant task M (A))).. Indeed, R[M (A(X)) 2] is equal to R[X Z] up to constant, so we can rewrite Eq. a6) as the following equivalent objective,
arg min p(Z(A(X))
T[A(X;; + , R[X|[] 2).
Such reformulation is possible if random augmentations retain the invariance structure X A(X) but "erase" enough information about equivalent inputs, specifically, if X (LL(()) (X). We discuss the second requirement in Appx. Cut note that it will likely not be practical issue if the dataset is small compared the support [D] << [ With this, we have an objective whose r.v.s. are easy to sample from. However, both terms in Eq. (7) are still challenging to estimate.
In the following, by stochastic gradient descent 23 over the encoder's ' parameters. Both approximations use the standard lossy neural compression bound T[Z; A A(X)] S H[2] S ming F-p(z)( log go [Z)] where go (Z) is called an entropy model (or prior) 24
develop two practical variational bounds to Eq. (7), which can be optimized
25 This has the advantage that the learned go can be used for entropy coding Z 26 27). See Ballé et al. (28) for possible entropy models. Our two
approximations differ in how they upper bound R[X| Z). The first uses reconstruction loss, which attempts to reconstruct the unaugmented input æ E DD from A(z). The second uses a discrimination loss, which attempts to recognize which examples augmented versions of the input.
4.1 Variational Invariant Compressor (VIC)
Our first model is modified neural compressor in which inputs augmented but target reconstruc- tions are not. We refer to it as a variational invariant compressor (VIC). See See 4 for an illustration. VIC has encoder P.p [Z) A(X)),
entropy model go (Z),,, and decoder 4s (X Z). Given data sample æ E D, we apply random augmentation A(x), and encode it to get a representation Z. The decoder then attempts reconstruct the unaugmented x from Z. This leads to the objective,
(0, := Σ (21A))) [log (Z) + 3 log qa (x Z)]
ZED
"As the RI function is not strictly convex Fig. 3 it should be beneficial to use R R(M(X) 212 to ensure that sweeping over / is equivalent to sweeping over à 12] We did not see any difference in practice.
5

---

The term log 9o (2) is an entropy bottleneck, which bounds the rate I[A(X);; 2] and ensures that unnecessary information removed. The term log (2 2) bounds the distortion R[X 2] Σ Fx(x,2)(- lo [X] [] and ensures that VIC preserves the information needed for invariant tasks.
4.2
Bottleneck InfoNCE (BINCE)
Our second compressor retains all predictive information without reconstructing the data. It has two components: an entropy bottleneck and an InfoNCE [15] objective, which is the standard in contrastive SSL. We refer to this as the bottleneck InfoNCE (BINCE), see Fig. 4 BINCE has an advantage over VIC in that it avoids the problem of reconstructing possibly high dimensional data.
Algorithm
shows how train BINCE, where each call to A returns an independent augmentation of its input. As with VIC, for every datapoint x D, we obtain representation Z by applying an aug- mentation A(z) and passing through the encoder PZ A(X)). We then sample a "positive" exam- ple Z by encoding a different augmented version of the same underlying datapoint z. Finally, we sample n "negative" examples z by encoding aug- mentations AarT) ) of datapoints x; E 'D that are different from This results in a sequence Z For conciseness we will denote
Algorithm BINCE's s forward pass for x
Require: Pv, go: fv. D, A, 8, n,
1: ã < sample( A(x))
b Augment b Encode
2: f- sample(p, (Z)
3: rate_
log 20 (0
4: {-, H==1 + select (D) {x}} times
5: x + sample([A(z), A(r A(2,))
6: z + cample(pp(2 x))
7: = z10]
esp lo
(z+,Zz ...., Z, the above sampling procedure as (Z, Z A, D,r). The final loss uses a discriminator f that is opti- mized to score the equivalence of two representation,
8: softmax (E'' exp (+)
9: distortion_los + - log (sofmmax)
10: return rate_ loss + B distortion loss
exp fz Σzez exp fo (Z', 2)
CaNcc (p,, 0, = Σ EpLAA,, log go( + B -log RED
(9)
BINCE retains the necessary information by classifying (as seen by the softmax) which Z is associated with an equivalent example X. Both VIC and BINCE give rise to efficient compressors by passing X through pp (X) and entropy coding using qo (Z). In theory they can both recover the optimal rate for lossless predictions, i.e., H[M (X)). ]. in the limit of infinite samples ([D) (DD,,n) and unconstrained variational families. In practice, BINCE has the advantage over VIC of (i) not requiring a high dimensional decoder; and (ii) giving (for suitable fw) representations that are approximately linearly
separable (1 and thus easy to predict from [15 32 The disadvantages of BINCE are that
(i) does not provide to reconstructions diminishes interpretability; and (ii) has a high bias, unless the
number of negative samples n is large 33 34 which computationally intensive.
Experiments
We evaluated our framework focusing on two questions: (i) What compression rates can our frame- work achieve at what cost? (ii) Can we train general purpose predictive image compressor? For all experiments, we train the compressors, freeze them, train the downstream predictors, and finally evaluate both test set. For classical compressors, standard neural compressors (VC) and our VIC, used either reconstructions X as inputs to the predictors representations Z. As BINCE does provide reconstructions, we predicted from the compressed Z using multi-layer perceptron (MLP). We used ResNet18 (35) for encoders and image predictors. For entropy models we used Ballé et al. 28 hyperprior, which uses uniform quantization. We optimized syper-parameters on validation using random search. For classification tasks, we report classification error instead of log-loss. The former is more standard and gave similar results Aspx F.2). For experimental details see see Appx. E For additional results see Appx. F Code is giggtu........ com www.//o///aaaaaee /tw///aaaabbb//ossllssssss
5.1 Building intuition with toy experiments
To build an visual intuition, we compressed samples from a banana source distribution (36]. assuming rotation invariant tasks, e.g., classifying whether points are in the unit circle. We also
compressed MNIST digits as in Fig. 1] Digits are augmented (rotations, translations, shearing, scaling) both at train and test time ensure that our invariance assumption still holds.

---

Ou WC Newer Caroo 1K3
14
12a
Bic-Rate
(b) VC high rate
(c) VC
(d) VC low rate
6
4
2-
2:
2
Invarinnaee
(c) VIC high rate
(f) VIC
(g) VIC low rate
Figure 5: Compression rates of Banana source [3 can be decreased when downstream tasks are rotation invariant. (Left) Our invariant compressor (VIC, blue) outperforms neural compressors (VC, orange). 5 runs with standard errors in gray. (Right) VIC quantizes the space using disks to remove unnecessary angular information. Pink lines are quantization boundaries, dots are code vectors with size proportional to learned probabilities. Low rates correspond to low /3 in Eq. (7)
Where do our rate gains come from? For rotation invariant tasks, our method (VIC) discards unnecessary angular information by learning disk-shaped quantizations (Fig. S bottom right). Specif- ically, VIC retains only radial information by mapping all randomly rotated points (disks) back to maximal invariants (pink dots). In contrast, standard neural compressors (VC) attempt to reconstruct all information, which requires a finer partition (Fig. 5] top right). As a result (Fig. Sa). VIC needs smaller bit-rate (y-axis) for the same desired performance (D r.. x-axis). The area under the RD curve (AURD) for VIC is 35.8:t4.2 against 48.1 1 for VC, i.e., expected bit-rate gains are around 70%. Similar gains are achieved for augmented MNIST in Fig. 6 by reconstructing canonical digits.
2
250
>
Bit-late
2
C
50
7
2
/
/
4
0.6
0.0
Classification Error Distartion
(a) Rate-EEnnn Error curve
(b) Reconstructions that allow 99% downstream accuracy
Figure 6: (Left) By reconstructing prototypical digits our VIC (blue) achieves higher compression of augmented MNIST digits than standard neural compressors (VC, orange) without hindering downstream classification. 5 runs. (Right) The source examples (first row) as well as reconstructions for the non non-invariant (second row) and invariant compressor (last row).
Can we recover the optimal bit-rate? We investigated whether our losses can achieve the optimal bit-rate for lossy prediction by using supervised augmentations, i.e., A(x) randomly samples a train example z that has the same label. For MNIST the single-task optimal bit-rate is H[Y] = log(10) se 3.3 bits. VIC and BINCE respectively achieve 5.7 and 5.9 bits, which shows that our losses relatively good despite practical approximations. Details in Appx. F2
What is the impact of the choice of augmentations? The choice of augmentation A implicitly defines the desired task-set T, i.e.. the set of all tasks for which does not remove information. As result Theorem 2 can be rewritten as Rate(8) = I[X; A(X)] 6, so the rate decreases when A removes more information from X. illustrate this we trained our VIC using three augmentation sets on MNIST, all of which keep the true label invariant but progressively discard more X information. VIC respectively achieves a rate of 185.3, 79.0, and 5.7 bits, which shows the importance of using augmentations that remove X information. Details and BINCE results are in Appx. F2

---

5.2 Evaluating our methods with controlled experiments
To investigate our methods, we compressed the STL110 dataset 337 We augment (flipping, color jittering, cropping) the train and test set, to ensure that the task invariance assumptions are satisfied. We focus on more realistic settings in the next section. In each experiment, combinations of cyppe-paarameent to ensure equal computational budget across models and baselines.
sampled 100
Table 1: Invariant compressors (BINCE, VIC) outperform classical (PNG, JPEG, WebP) and neural
(VC) compressors on STL10. BINCE achieves lossless prediction but compresses 121 x better.
PNG
38
JPEG
WehP
40
VC X
VIC X
VIC Z
BINCE
Decrease lest acc. Compression gains
0
0.7
1.1
21.0
25.1 269x
16.1
0.0
1x
3x
13x
63x
175x
121x
How do our BINCE and VIC compare to standard compressors? In Table 1 we compare com- pressors at the lowest downstream error that they achieved. As benchmark, we use PNG's lossless compression. Predicting from PNG corresponds to standard image classification, and obtains a rate of 1.42e4 bits per image for 80.8% accuracy. Classical lossy methods (JPEG, WebP) achieved up to 13x bit-rate gains with little drop in performance. In comparison, our BINCE method achieved 121x compression gains with no impact on predictions. Both our invariant (VIC) and standard (VC) neural compressors significantly decreased classification accuracy, which we believe can be explained by the encoders architecture (ResNet18) that we use for consistency (see Appx. F.3).
Should we predict from representations Z reconstructions X In Table we analyzed the impact of predicting from Z instead of X for VIC and see that this increases accuracy by 9%. In contrast, predicting from Z for VC decreases performance by 12% (see (Apx. F.3). This suggests that invariant reconstructions X might not be easy to predict from with standard image predictors.
Are we learning invariant compressors? Invariant compressors should provide RD curves that are robust test distribution shift in the desired augmentations. We thus trained our VIC by applying the augmentations 50% of the time but varying that probability at test time. In Appx. F.3 we show that this distribution shift have negligible influence on RD curves.
5.3 A zero-shot compressor using pre-trained self-supervised models
BINCE includes standard contrastive SSL loss. So, we investigated whether existing pre-trained SSL. models
32 41 can be used to build generic compressors. In particular, we investigated whether CLIP [41 could be quickly turned into a powerful task-centric compressor for computer vision. the introduction, we motivated large compression gains by noting that typical image classification tasks can be predicted from detailed captions instead of images (around 1000 more bits). CLIP is a vision transformer 42 pre-trained on 400M pairs of images and text (Zimages =tt,)) using contrastive loss. The "augmentation" A is then a function that maps Timage to its associated xiter and vis-versa. This will partition the images and texts into sets, each of which are associated directly or by transitivity in CLIP's dataset. This suggests that CLIP is retaining the image information that corresponds to detailed caption, and may be turned into a generic compressor for image classification.
CLIP can essentially be seen as a BINCE model with an image-to text augmentation, but without an entropy bottleneck. (For details about the -LIP-BINCE relation see Appx. C.5) We thus constructed an approximation of our desired image-to-text BINCE compressor by two simple steps. First, we downloaded and froze CLIP's parameters. Second, we trained, on the small MSCOCO dataset [R), an entropy bottleneck to compress CLIP'' representation. The latter step can be done by training any lossy compressor on CLIP's representations, we did so using Ballé et al. (28) hyperprior entropy model with a learned rounded precision. We then evaluated our resulting compressor on 8 datasets (various classification tasks and image shapes) that were never seen during training (zero-shot), by training MLP for downstream predictions each dataset. One can this as a multi-task setting (each dataset is a distinct task). We investigate the case of multiple labels per images in Appx. F.5)
Can we use pretrained SSL to obtain a generic compressor? Table 2 shows that we can exploit existing state-of-the-art (SOTA) SSL models to get a powerful image compressor, which achieves 1000x bit-rate gains on ImageNet compared to JPEG (at the quality level used for storing ImageNet). The bit-rate gains (1St row) are significant across pero-shot datasets, even for biological tissues (PCam; [4]). Importantly, these gains come at little cost in test performance. Indeed, the test
8

---

Table 2:
Converting a pretrained SSL model into a zero-shot compressor achieves substantial bit-rate gains while allowing test accuracies similar supervised models predicting from raw images.
ImageNet
STL
PCam
Cars
CIFARIO
Food
Pets
Caltech
Rate gains vs JPEG
110x
35x
64x
131x
7x
109x
150x
126x
Our Acc. [%6]
76.3
98.7
80.9
79.6
95.2
88.3
89.5
93.4
Supervised Acc. [%]]
76.1
99.0
82.6
49.1
96.7
81.8
90.4
94.5
accuracies of MLPs from our representations (2td row) is similar to a near SOTA model trained on the uncompressed images (3td row is from Radford al. [4]). These results are not surprising as JPEG is optimized to retain perceptual rather than classification information. Note that the large variance in rate gains come from JPEG rates due to different images shapes (see Table 3).
Our CLIP compressor retains all the information needed to get 0 error for those tasks. Table provides the test performance for MIPs, while our theory discusses Bayes risk, which is independent of specific predictors and generalization. We estimated the excess Bayes risk for our datasets by counting the images (in train and test) that get compressed to the same Z but have different labels. We found that we are in the lossless prediction regime for those datasets.
Table 3: Our entropy bottleneck (EB) CLIP improves compression of representations up 17x with little impact on predictions. The same compressor is used across datasets. Rates are per image.
ImageNet
STL
PCam
Cars
CIFAR10
Food
Pcts
Caltech
JPEG
1.49e6
4.71e4
9.60e4
1.92e5
1.05e4
1.54e5
1.81e5
1.69e5
Bit-Rate
CLIP
1,52c4
1.52c4
1.52c4
1.52c4
1,52c4
1.52c4
1.52c4
1.52c4
+EB high 8
2.47c3
2.46c3
2.61e3
2.59c3
2.53e3
2.39c3
2.33e3
2.46c3
+EB B
1.35e3
1.34e3
1.49e3
1.47c3
1.41e3
1,27c3
1.21e3
1.34c3
+EB low /
9.63e2
9.52e2
1.09e3
1.07e3
1.02e3
8.89e2
8.35e2
9.53e2
Test Acc.
CLIP
76.5
98.6
84.5
80.8
95.3
88.5
89.7
93.2
+EB high 3
76.6
98.7
82.7
80.4
95.3
88.5
89.6
93.5
+EB /
76.3
98.7
80.9
79.6
95.2
88.3
89.5
93.4
+EB low /
76.0
98.7
80.1
78.9
94.8
87.6
88.6
92.9
What is the effect of the entropy bottleneck? In Table 3 we compare the pretrained CLIP, to our CLIP compressor with an entropy bottleneck (EB) trained at different values for B. When trained with high our EB improves bit-rates by an average of 6x without impacting predictions. For our compressor from Table 2 (CLIP+EB (8) the gains increase to 11 x with little predictive impact. The sacrifice in predictions is more clear for 16x bit-rate gains (low (). This shows that CLIP's raw representations retain unnecessary information as it not explicitly trained to discard information.
How would end-to-end BINCE compare staggered training? Compression gains likely be larger by end-to-end training of BINCE, which would require access to CLIP's original dataset To get idea of potential gains we compared end-to-end and staggered BINCE on augmented MNIST We found sigmificant rate improvements (358 to 131 bits) for similar test accuracy.
in Appx. F.2
Our CLIP compressor is simple to use. In Appx. E.7, we provide minimal script (150 lines) to train a generic compressor in less than five minutes on a single GPU. The script contains an efficient entropy coder for model (200 images/second), which shows its practicality. As usual in SSL, the compressed representations are also more computationally efficient to work with than standard compressors. compressors. In our minimal script we achieve the desired performance (98.7% STL) using linear model that is trained in one second, which 1000 faster than the baseline in Table 2] This shows that our pipeline can improve computational efficiency in addition storage efficiency.
What augmentations to use for SSL compression? Table commares two ResNet50 pretrained with contrastive learning using invariance to text-image (CLIP) standard image augmentations (SimCLR 32) such as cropping or flipping. We see that CLIP's augmentation usually give better compression and downstream performance, which shows the importance of the choice of augmentations. This also supports our motivation of using text-image augmentations, which are likely preservig for a vast amount of tasks but discard large amounts of unnecessary information.
We investigated finetuning CLIP MSCOCO but it suffered from catastrophic forgetting.

---

Table 4: Text-image invariance is better than invariance to standard augmentations for image classifi- cation. CLIP and SimCLR are both ResNet50 pretrained with InfoNCE but different augmentations.
ImageNet
STL
PCam
Cars
CIFARIO
Food
Pets
Caltech
Rate
CLIP+EB
2108
1962
1949
2421
2111
1991
1867
1968
SimCLR+EB
2811
2732
2769
2751
2950
2077
2839
2502
Aec.
CLIP+EB
63.2
92.0
78.6
68.0
65.5
74.1
81.8
83.0
SimCLR+EB
62.8
91.9
81.4
29.6
78.6
60.0
78.9
79.0
6 Related work
In Appx. D we discuss more related work, including invariances in compression and the link SSL.
Task-centric compression. To our knowledge, our paper is the first to formalize compression only for predictions. IB [ uses task-centric distortion, but is not used for compression as it requires supervised training, so there are no advantages compared to compressing predicted labels. low frequencies for classification [45] or high frequencies for segmentation [46]. Other authors have incorporated predictive errors to perceptual distortions 44 48), but cannot compress without the perceptual distortion for the same reason as IB. One exception is Weber et al.[s 49 compressor; which (when removing their perceptual distortion) minimizes MSE in the hidden layers of pretrained classifier. Even more related is Singh et al.Js is [50] work on compressing pretrained features for transfer learning, which is practice similar to our compression of SSL features. Their work do not provide theoretical justifications, and are constrained to tasks that are similar to those used for pretraining.
Some authors used heuristics to bypass the supervised issue, e.g., focusing
7 Discussion and Outlook
Given the ever increasing amount of data that is processed by task-specific algorithms, it is necessary to rethink the current task-agnostic compression paradigm. We formalized the first compression framework for retaining only the information necessary for high performance desired tasks. Using our theory, we provide two unsupervised objectives for training neural compressors. Experimentally. show that these compressors ImageNet) smaller than standard image compressors without losing predictive performance.
achieve bit-rates that are orders of magnitude (1000x on
There are number of caveats that should be addressed. First, to achieve better rates, our theory requires an irrecoverable loss of information. This can be an issue if the set of desired tasks changes. For example, if one uses text-image invariances then it may be impossible to perform image segmentation from the compressed representations. One solution would be to keep an original copy and use invariant compression for duplicated data, e.g., for the thousands copies of ImageNet. A second issue is the interpretability of the compressed representations. This can be partially addressed by reconstructing prototypical data in Fig. (post-hoc decoders could be trained for BINCE). third caveat is that the compressed representations may be harder to learn from, e.g., neural networks may struggle to predict from representations even if the information is retained. Although our experiments actually showed the opposite, this should be addressed theoretically, e.g., using decodable information [51] (2).. Finally, successful of our framework requires access label- preserving augmentations A that discard significant information about X. Finding such an A may be challenging for some tasks. Given that augmentations are ubiquitous in ML, the community will hopefully continue developing task-specific augmentations which we could take advantage of.
Nevertheless, we achieved orders of magnitude improvements in compression for predictions, and we believe that our improvements are just the beginning. For example, many tasks can be answered by referencing detailed natural language description of the data. In these cases, the improvements can be very large, potentially 1Mx for videos In the long-term, we hope that abandoning perceptual reconstructions will enable individuals process data at scales that are currently only possible large institutions, and our society to take advantage of large data sources in a more sustainable way.
A movie takes around 10GB to store, but the information relevant to humans (e.g.. "what happened to the house?", "how did the movie end?") likely be stored in detailed movie script and would require 100KB.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Alex Alemi, David Duvenaud, Andriy Mnih, Emile Mathieu, Jonah Philion, Yangjun Ruan, and llya Sutskever for their helpful feedback and encouragements. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. BBR acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC); RGPIN-2020-04995, DGECR-2020-00344
References
[1] D. Rolnick, P. L. Donti, L. H. Kaack, K. Kochanski, A. Lacoste, K. Sankaran, A. S. Ross, N. Milojevic-
N. Jaques, Waldman-Brown. Sherwin, S. K. Mukkavilli, K. P. Körding, C. P. Gomes, A. Y. Ng, D. Hassabis, J. C. Platt, Creutzig. T. Chayes, and Y. Bengio, "Tackling Climate Change with Machine Learning," CoRR, vol. abs/1906.05433 2019, _eprint: 1906.05433. [Online]. Available:
Luccioni, T. Maharaj, E. D.
http:///wwi..orr/ab//9900..000
[2] M. Z. Zgurovsky and Y. P. Zaychenko, Big Data: Conceptual Analysis and Applications. Springer, 2020.
[3] N. Kalra and S. M. Paddock, "Driving to safety: How many miles of driving would it take demonstrate autonomous vehicle reliability?" Transportation Research Part A: Policy and Practice, vol. 94, pp. 182-193, 2016, publisher: Elsevier.
[4] J. Yeong, G. Heansco-,ernaddez,, J. Barry, J. Walsh, and others, "Sensor and sensor fusion technology in autonomous vehicles: a review," Sensors, vol. 21, no. 6, p. 2140, 2021, publisher: Multidisciplinary Digital Publishing Institute.
[5] J. Johnston, "Transform coding of audio signals using perceptual noise criteria," IEEE Journal on Selected Areas in Communications, vol. 6, no. 2, pp. 314 323, Feb. 1988, conference Name: IEEE Journal on Selected Areas in Communications.
[6] D. J. Heeger and P. C. Teo, "A model of perceptual image fidelity," in Proceedings 1995 International Conference on Image Processing, Washington, DC, USA, October 23-26. 1995. IEEE Computer Society, 1995, pp. 343-345. [Online]. Available: Wdoi.org/10..1 109///CI..1995553333
https
[7] J.-S. Lee and T. Ebrahimi, "Perceptual Video Compression: A Survey," leee Journal Of Selected Topics In Signal Processing, vol. 6, no. 6, p. 684, 2012, number: ARTICLE Publisher: leee-Inst Electrical Electronics Engineers Inc. [Online]. Available:
[8] Y. Blau and T. Michaeli, "Rethinking Lossy Compression: The Hatte-iistorion-Perreee Tradcoff," " in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- June 2019, Long Beach, California, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. 675-685. [Online]. Available: http://proceeeiings.mm wtt.sssm....../////////a...
html
[9] Y. Pan, "Digital Audio Compression," Digital Technical Journal, 1993.
[10] F. Mentzer, G. Toderici, M. Tschannen, and E. Agustsson, "High-Fidelity Generative Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-
Image Compression,"
12. 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin. Eds., 2020. [Online]. Available:
[11] N. Tishby, F. N. Pereira, and W. Bialek, "The information bottleneck method," CoRR, vol. physics/0004057, 2000. [Online]. Available: http://arnivvarrrasssyyyysccooop
[12] J. Heaton, "lan Goodfellow, Yoshua Bengio, and Aaron Courville: Deep learning The MIT Press, 2016, 800 pp. ISBN: 0262035618" Genet. Program. Evolvable Mach., vol. 19, no. 1-2, pp. 305-307, 2018. [Online]. Available:
11

---

[13] C. Shorten and T. M. Khoshgoftaar, "A survey on Image Data Augmentation for Deep Learning"" Journal of Big Data. vol. 6, no. 1, p. 60, Jul. 2019. [Online]. Available: https///oi.orr//I0 10. 8666400377-09--0197--)
[14] D. P. Kingma and M. Welling, "Auto-Encoding Variational Bayes," in 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2014. [Online]. Available: http://arxiv.org/bb//111...2
[15] A. v. d. Oord, Y. Li, and O. Vinyals. "Representation Learning with Contrastive Predictive Coding," arXiv: 1807.03748 [cs, stat], Jan. 2019, arXiv: 1807.03748. [Online]. Available:
http://arxiv.orr/bbb///aa 1807 037748
[16] Deng, W. Dong, R. Socher, I..J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2009, pp. 248-255, iSSN: 1063-6919.
[17] C. E. Shamnon, "Coding theorems for discrete source with a fidelity criterion," IRE Nat. Conv. Rec, vol. no. 142-163, p. 1, 1959.
[18] T. Berger, "Rate distortion theory for sources with abstract alphabets and memory," Information and Control, vol. 13, no. 3, pp. 273, Sep. 1968. [Online]. Available:
[19] M. L. Eaton, "Group Invariance Applications in Statisties," Regional Conference Series in Probability and Statistics, vol. 1, pp. 1-13,, 1989, publisher: Institute of Mathematical Statistics. [Online]. Available: https//wwww...bbrrrr//aaae441111
[20] N. Rashevsky, "Life, information theory, and topology." The bulletin of mathematical biophysics, vol. 17, no. 3. pp. 229-235, Sep. 1955. [Online]. Available: https://doi.org// 100 1007/BF02477866
[21] L. R. Varshney and V. K. Goyal, "Benefiting from Disorder: Source Coding for Unordered Data," arXiv:0708.22 2310 [cs, math), Aug. 2007, arXiv: 0708.2310. [Online]. Available:
http://axxiv.orrr/b//008..33331
[22] A. Kolchinsky, B. D. Tracey, and S. V. Kuyk. "Caveats for information bottleneck in deterministic scenarios," in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. [Online]. Available:
http://openeevie..ee/fommmm hit.:e..eee.ooommmdddeeeeellll.
[23] L. Bottou, "Large-Scale Machine Learning with Stochastic Gradient Descent," in 19th International Conference on Computational Statistics, COMPSTAT 2010, Paris, France, August 22-27, 2010 Keynole, Invited and Contributed Papers, Y. Lechevallier and G. Saporta, Eds. Physica- Verlag, 2010, pp. 177-186. [Online]. Available:
http://doo.../100..00000
978-3-7908 2604-3 16
[24] J. Ballé, V. Laparra, and E. P. Simoncelli, "End-to-end Optimized Image Compression," " in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. [Online]. Available: http://oppneeiiewwwee///otttttdd
[25] L. Theis, W. Shi, A. Cunningham, and Huszár, "Lossy Image Compression with Compressive Autoencoders." in 5th International Conference on Learning Representutions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. [Online]. Available: http://pppeeeeeeeeeeefoo/umddddd
[26] J. Rissanen, "Generalized Kraft Inequality and Arithmetic Coding," IBM Journal of Research and Development, vol. 20, no. 3, pp. 198-203, May 1976, conference Name: IBM Journal of Research and Development.
[27] J. Duda, "Asymmetric numeral systems," CoRR, vol. abs/0902.0271, 2009, _pprint: 0902.0271. [Online]. Available:
[28] Ballé, D. Minnen, Singh, S. Hwang, and N. Johnston, "Variational image compression with a scale hyperprior," in 6th Internutional Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.ne,, 2018. [Online]. Available: http://ppoueevoe...e//oooiiiele
12

---

[29] N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar, "A Theoretical Analysis of Contrastive Unsupervised Representation Learning." in Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. 5628-5637. [Online]. Available: http://proceediings mir, presss//se////aaumm
19a.htm]
[30] C. Tosh, A. Krishnamurthy, and D. Hsu, "Contrastive learning, multi-view redundancy, and linear models," " in Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide, ser. Proceedings of Machine Learning Research, V. Feldman, K. Ligett, and S. Sabato, Eds., vol. 132. PMLR, 2021, pp. 1179-1206. [Online]. Available:
[31] J. D. Lee, Q. Lei, N. Saunshi, and J. Zhuo, "Predicting What You Already Know Helps: Provable Self-Supervised Learning," " arXiv:2008 01064 [cs, stat], Aug. 2020, arXiv: 2008.01066 [Online]. Available: hutp:/arxivv.rrg http://arxiioor/abb//0000 01064
[32] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, "A Simple Framework for Contrastive Learning of Visual Representations," in Proceedings of the 37th International Conference on Machine Learning, 1020, 13-18 July 2020, Virtual Event, ser. Proceedings of Machine Learning Research, vol. 119, PMLR, 2020, pp. 1597-1607. [Online]. Available: http://proceedingg.. mir, w 19/chen20j.hmml
[33] B. Poole. S. Ozair, A. v. d. Oord, A. Alemi, and G. Tucker, "On Variational Bounds of Mutual Information." in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 2019, pp. 5171-5180. [Online]. Available:
[34] J. Song and S. Ermon, "Multi-label Contrastive Predictive Coding," arXiv:2007.00852 [cs, stat], Jul. 2020, arXiv: 0099509952. [Online]. Available: http://aaxiv.or//aas/000..0000
[35] K. He. X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Las Vegas. NV, USA: IEEE, Jun. 2016, pp. 770-778. [Online]. Available: http
[36] J. Ballé, P. A. Chou, D. Minnen, S. Singh, N. Johnston, E. Agustsson, S. J. Hwang, and G. Toderici, "Nonlinear Transform Coding," " CoRR, vol. abs/2007.03034, 2020, _eprint: 2007.03034 [Online]. Available: https://arrv.orr/////00....3.
[37] A. Coates, A. Ng. and H. Lee, "An analysis of single-layer networks in unsupervised feature learning," in Proceedings of the fourteenth international conference artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 215-223.
[38] N.
Graphies httppp/wwwwgggppp//////eseiiiiip
(PNG), ISOЛIC 15948,
2003,
published:
[39] P. E. Group (JPEG), ITU-T 7.81, ITU-T T.83, ITU-TT.84, ITU-T T.86, ISOЛEC 10918, 1992. published: www.jpeg.org/ipeg//
[40] WebP, Google, 2018, published: developers wwwpoeerr.gooll..ooo
wttp:esssgoogee.ooo//e/////e/ee cww/speed/wwbp..
[41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning Transfcrable Visual Models [cs], Feb. 2021, arXiv: 2103.00020.
From Natural Language Supervision," arXiv:2103.
[Online]. Available: http://aakiv.oog//ab///1...000
[42] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani,
M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale," CoRR, vol. abs/2010.11929, 2020, .print: 2010. 1 1929. [Online]. Available: httpps///arxiv.or//aa//////
[43] T.-Y. Lin, M. Maire, Belongie, L. Bourdev, R. Girshick. J. Hays, Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, "Microsoft COCO: Common Objects in Context," arXiv:1405.088 0311
[cs], Feb. 2015, arXiv: 1405.0312. [Online]. Available: http://arri..or//as//14......1
[44] B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling, "Rotation Equivariant CNNs for Digital Pathology," " arXiv: 00962 [cs, stat], Jun, 2018, arXiv: 1806.03962. [Online].
Available: http:://rxiv.orgg///// wttp://///wbb....0.0//./36
13

---

[45] Z. Liu, T. Liu, W. Wen, L. Jiang, J. Xu, Y. Wang, and G. Quan, "DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework," arXiv: [cs], Mar. 2018, arXiv: 307.00588.. [Online]. Available: http://arki..or//ass//88.....
[46] Z. Liu, x. Xu, T. Liu, Q. Liu, Y. Wang. Y. Shi, W. Wen, M. Huang, H. Yuan, and J. Zhuang, "Machine Vision Guided 3D Medical Image Compression for Efficient Transmission and Accurate Segmentation in the Clouds," arXiv:1904.088/ [cs, stat], Apr. 2019, arXiv: 1904.08487. [Online]. Available: http://aaxxiv.org/bbb// 1940////9900088887777
[47] D. Liu, D. Wang, and H. Li, "Recognizable or Not: Towards Image Semantic Quality Assessment for Compression." " Sensing and Imaging, vol. 18, no. p. 1, Dec. 2016. [Online].
Available: https://doi. http://ddoo.o//1OO.. 1ttp:/1011000011211110000 1000000111222000---05555555
[48] D. Liu, H. Zhang, and 7. Xiong, "On The
Tradeoff," in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, d' Alché Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 1204 1213. [Online]. Available: httpss//proceedinss.euuii.ss... 40a8 wew.cc/ccee0ele9aeeeee Basalll9st22--Aattttttat Abstract.hhml
2aper/2019hhssh6c22999
[49] M. Weber, C. Renggli, H. Grabner, and C. Zhang, "Observer Dependent Lossy Image Compression," in Pallern Recognition - 42md DAGM German Conference, DAGM GCPR 2020, Tübingen, Germany, September 28 October 1, 2020, Proceedings, ser. Lecture Notes in Computer Science, Z. Akata, A. Geiger, and T. Sattler, Eds., vol. 12544. Springer, 2020, pp.
140. [Online]. Available: https://doi.org/ 10.1997
[50] S. Singh, S. Abu-El-Haija, N. Johnston, J. Ballé, A. Shrivastava. and G. Toderici. "End-to-End Learning of Compressible Features," in IEEE International Conference on Image Processing, ICIP 2020, Abu Dhabi, United Arab Emirates, October 25-28. 2020. IEEE, 2020, pp. 3349-3353., [Online]. Available: hntips://doi.or 10. 1009//11194077 2020.9190860
[51] Y. Xu, S. Zhao, J. Song, R. Stewart, and Ermon, "A Theory of Usable Information under Computational Constraints." in 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. [Online].
Available:
[52] Y. Dubois, D. Kicla, D. J. Schwab, and R. Vedantam, "Learning Optimal Representations with the Decodable Information Bottleneck," in Advances in Neural Information Processing Systems 33: Annual Conference
Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin, Eds., 2020. [Online]. Available: http://cwcecciii..s.eeiiicccpppp
d8ea5f53c haw.bbbooooo222232353333555 Abbsssstt Abstract.html
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Sec. 7
(c) Did you discuss any potential negative societal impacts your work? [Yes] In Sec. we mention the potential decrease in interpretability.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? Yes] See Appx.
A.2
(b) Did you include complete proofs of all theoretical results? [Yes] See Appx. B
you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? [Yes] The code to train our main compressor is in Appx. E.7, the code to replicate all our results is at anonymous (b) Did you specify all the training details (e-g., data splits, hyperparameters, how they were chosen)? [Yes] The most important training details can be found at AApx. E] Minor hyperpa- rameters can be found in our code at anonymous
(c) Did you report егrоr bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We report error bars for MNIST (Appx. F.2) and Banana (Appx. F.I). For larger experiments we sampled fixed number of hyperparameters for each model and bascline (see Appx. E). and report the best result.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In Appx. E
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or URL? [Yes] Our clip compressor in [Appx. E.7annd and our code at anonymous
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [No] We not propose any new dataset. The main possible issues come from the CLIP dataset 41,, for which the data collection procedure is discussed in details in
their paper.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We do not propose any new dataset. The main possible issues come from the CLIP dataset 4], for which the data collection procedure is discussed details in their paper.
15


Reviews:
[{"review_id": "zo65jGcs0kU", "reviewer": "Reviewer_XSke", "summary": "This paper proposes methods to introduce a self-supervised approach to neural image compression for classification tasks.\nVIC is a modified neural compressor in which inputs are augmented but target reconstructions are not.\nBINCE is a modified neural compressor same as VIC but trained by an entropy bottleneck and an InfoNCE [15] objective. This paper also introduce and evaluate a zero-shot compressor using pre-trained self-supervised models CLIP [38].", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think it is a novel idea to introduce a self-supervised approach to neural image compression for classification tasks and theoretical background is well described.\n\nBut I am curious about the following points.\n- I don't know the unit of \"bit-rate\" in table 3 (it not seems to bit per pixel, is it bit per image?). Please write a clear definition.\n-  I don't think this paper (the main paper) adequately describes how the quantizer is handled. However, the Appendix describes the quantizer process with a linear transformation using the learnable parameter. Also, you are probably using a uniform noise quantizer [24], but I think these need to be explained in the main paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The caveats are well described in this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "PkybgSQOLQl", "reviewer": "Reviewer_YGFN", "summary": "This paper proposes lossy compression algorithms that only compress task-relevant information in order to obtain a better bit-rate while maintaining the performance of the downstream task. Specifically, the paper proposes to augment the training samples before encoding, and the decoder aims to reconstruct the original image (another proposed model uses contrastive loss to do this). In this way, the model can learn representations that are invariant of the input augmentations, which could lead to smaller bit-rate.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Given that transfer learning is of great importance in machine learning, the task of compressing data only for certain downstream tasks is definitely worth exploring. However, though the paper made a good effort to formulate this problem, I think the experiments used in the paper might be insufficient to demonstrate the effect of their proposed technique — performing data augmentation during training leads to better lossy compressors:\n\n - The author claimed in the introduction that they aim to “ensure good performance on any future tasks of interest”. This setup was only tested in the CLIP experiment in Section 5.3, where the authors trained a compressor and validated on 8 datasets unseen during training. However, the good results may simply come from the fact that the latent representations of CLIP (a large pre-trained model analogous to BERT) already have a low bit-rate. That is, training any lossy compressor with the latent representations as input could lead to similar rate gains.\n\nTo address this concern, I would recommend the authors to at least compare the bit-rate also with other modern lossy compressors trained on latent representations generated by CLIP. It would also be great if the authors to demonstrate the performance gain provided by different data-augmentation techniques. In general, only comparing with JPEG does not provide useful information about how good the proposed model is since the goal of JPEG is to compress images while not changing its visual appearance by too much.\n\n - Need to justify the proposed models is not simply compressing the labels. For example, in the experiment in Section 5.2, performance drop and compression gain are compared. However, even by simply compressing the labels, one can get zero decreases in test accuracy and achieve a much better compression gain. The proposed models (VIC, BINCE) are of course not doing this explicitly. However, I am a bit concerned that this could be accidentally done implicitly. For example, consider an MLP with hidden layer size 32 - 16 - 8. If the bottleneck is set at the layer with 32 hidden neurons, the bit-rate could be worse than when the bottleneck is set at the last layer (8 hidden neurons). \n\nI think the authors should design a better evaluation metric to avoid the above trivial case.\n\n - Overall, I like the high-level idea of the paper, but it seems hard for me to separate the cause of the bit-rate gain: is it caused by the good latent representations learned by SSL models, or the proposed invariant compression technique plays a critical role there? For example, if we have a powerful SSL model trained to classify ImageNet samples, then CIFAR10 can be faithfully solved only by looking at the 1000-class imagenet label. I believe this can be a good paper if this concern is addressed, and I would be willing to increase my score if the authors convince me during rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in Section 7. This includes the methods' irrecoverable loss of information, and instability when facing task changes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "LJ7eApXqBh3", "reviewer": "Reviewer_x5kJ", "summary": "The paper \"Lossy Compression for Lossless Prediction\" considers the problem of data compression for downstream predictive tasks, rather than preserving the perceptual quality of reconstructions, in order to achieve significantly higher compression rate with negligible or no performance loss in  downstream tasks. In particular, the paper theoretically characterizes the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentation, and derives two corresponding variational objectives, VIC and BINCE, for training end-to-end neural compressors. The method is validated empirically on image tasks, both training from scratch and adapting an existing SOTA vision transformer model with an entropy bottleneck, and is shown to achieve superior compression rates than both traditional and learned image compression baselines.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nAlthough the paper is most similar in spirit to \"End-to-End Learning of Compressible Features\" (Singh et. al., 2020), and in fact Singh et. al.'s method  can be seen as a special case of the proposed VIC method (where there is only one downstream task of interest and thus no data augmentation), the current paper broadens the scope considerably by considering lossy compression for a *collection* of related invariant tasks, and is therefore a novel and original contribution.\n\n\n**Quality and Clarity**\n\nThe paper is clearly written and well-organized. The theoretical results appear to be rigorously derived (in the supplementary material). However, the discussions in the experiments section leave a few things to be desired. In particular,\n1. In section 5.2, the worse performance of VIC (e.g., its inability to achieve zero distortion) compared to BINCE can benefit from a better explanation; and more generally, a comparison between the two formulations and discussions on their pros and cons would be beneficial. \nThe worse performance of VIC was attributed to the channel averaging operation of the ResNet18 encoder used; however, since BINCE does not perform reconstructions (unlike VIC) and thus does not suffer from the poor encoder, it would make a more fair comparison by evaluating VIC performance using the compressed Z representation (with a separately trained predictor) as done for the evaluation of BINCE.  It would be reassuring to see that VIC is able to achieve comparable performance to BINCE, e.g., by increasing the expressivity of the variational family.\n\n2. It's not clear how the performance of zero-shot compressor using a pre-trained SSL model compares to end-to-end training with the proposed method (either VIC or BINCE). The already good zero-shot performance from fitting an entropy model to the latent representations of a pre-trained model (in section 5.2) seems to render end-to-end training with VIC or BINCE unnecessary. \n\n3. It's not clear how the pretraining of CLIP fits into the BINCE framework (e.g., it's not clear what kind of data augmentation transform $A$ is used during training).\n\n**Significance**\n\nAs far as I'm aware, this paper is the first to consider the problem of learned lossy compression in a multi-task setting, and lays a solid foundation for future work (both in theory and methods), thus represents a significant contribution. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Limitations:** See the various points in the \"Quality and Clarity\" section above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "-Pm_oxP0u6", "reviewer": "Reviewer_QdsL", "summary": "The author proposes a compression algorithm by extracting invariant information from the data source, which significantly reduces the file size for different downstream tasks that share similar structures while maintaining high prediction accuracy.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros:\n1. The idea is interesting and novel. Compared to other compression work that is aimed at reconstruction quality, the author focus on extracting critical (and of course compressible) information from source data; the extracted data can be applied to different downstream tasks without further fine-tuning.\n2. The author proposes 2 invariant neural compressors for this compression scheme; both of them show competitive rate-distortion(or rate-accuracy) results on different tasks.\n3. Theoretical Justification\n\nquestions:\n1. In table 3. Higher $\\beta$ should correspond to lower distortion, but the table shows higher distortion and lower rate?\n2. I wonder how the augmentation methods can influence the rate-distortion performance? The tasks seem to be highly correlated to the augmentation methods we use. For image classification, we can easily use different transforms, but for tasks like image captioning, it could be hard to find the right way to augment the data.\n3. The pseudocode for Algorithm 1 is not clear for me. line 5- line 7: $\\tilde{\\textbf x}$ and $\\textbf x$ are not the same thing? To make sure if I understand correctly, $A(x)$ and $A(x^-_i)$ use the same augmentation method and a $x^+$ is augmented differently.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes "supervised" data compression for downstream predictive tasks, aiming to achieve higher compression rates with negligible or no performance loss. It theoretically characterizes the bitrate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentation, and derives two corresponding variational objectives, VIC and BINCE. The method is validated empirically on image tasks, both training from scratch and adapting an existing SOTA vision transformer model with an entropy bottleneck.

The method can be seen as a generalization of "End-to-End Learning of Compressible Features" (Singh et al., 2020) but significantly expands its scope by considering lossy compression for a collection of related invariant tasks and by providing a rigorous information-theoretical justification of the approach. Original weaknesses pointed out by the reviewers were sufficiently addressed in the review period. Experiments are thorough and exhaustive. 
Decision: Accept (Spotlight)