Inverse Problems Leveraging Pre-trained Contrastive Representations
Authors: Sriram Ravula
Georgios Smyrnis
Matt Jordan
Alex Dimakis
Venue: NeurIPS 2021 Poster (2021)
Keywords: Inverse Problems
Representation Learning
Contrastive Learning
Robustness

Abstract: We study a new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on clean images, like CLIP. The problem is to recover the representation of an image R(x), if we are only given a corrupted version A(x), for some known forward operator A. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Using a linear probe on our robust representations, we achieve a higher accuracy than end-to-end supervised baselines when classifying images with various types of distortions, including blurring, additive noise, and random pixel masking. We evaluate on a subset of ImageNet and observe that our method is robust to varying levels of distortion. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators. 

Inverse Problems Leveraging Pre-trained Contrastive Representations
Sriram Ravula"
Georgios Smyrnis" The University of Texas at Austin Electrical and Computer Engineering
The University of Texas at Austin Electrical and Computer Engineering sriram. rawaaaauteaaaaa.aa.acoo. edu
gamyrnis@utexas.cm edu
Matt Jordan
Alexandros G. Dimakis
The University of Texas at Austin Computer Science mj mjordan@cs.o utexas. edu
The University of Texas at Austin Electrical and Computer Engineering dimakis@aus tin. utexas edu
Abstract
We study new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on clean images, like CLIP. The problem is recover the representation of an image R(x), if we are only given a corrupted version A(x), for some known forward operator A. We propose supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Using linear probe a higher accuracy than end-to-end supervised baselines when classifying images with various types of distortions, including blurring, additive noise, and random pixel masking. We evaluate on a subset of ImageNet and observe that our method is robust varying levels of distortion. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.
robust representations, achieve
Introduction
Modern representation learning networks like CLIP 35 image classification, even for zero-shot problems with labels not seen during training. Training these encoders comes at staggering cost and requires datasets and computing resources only available to very few organizations. In this paper show how leverage this pretrained power for new family of problems in the presence of image corruptions or other types of measurements.
are showing incredible performance for
Inverse problems involve reconstructing an unknown vector x from measurements Typically, the forward operator A corrupts the unknown vector x and reduces its dimension, i.c. the observations y live in lower-dimensional space compared to In the special case of linear inverse problems, the forward operator is simply a matrix and the measurements are in the form y Ar + noise. Special cases of linear inverse problems include image denoising, inpainting, super-resoluttio, compressed sensing used medical tomography, seismic geological imaging and many others, e.g. [31] for recent overview.
A(x).
"qqual contribution.
Code available
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Original
Random Mask
Our Prediztions
Baseline Predictions
62.1% Carbonara
51.5% Gila Monster
Mixing Bool
Hognose
9.8% Dutch Oven
10 1% Rottwailer
Original
Goussian Noise
Our Predictions
Raoeline Predictions
60.9% Doberman
71.0% Red Fox
Tenier
10.2% Head Cablagg
17.9% Chihuahua
8.7% Gyromitra
Original
Gaussian Blur
Our Predictions
Raseline Predictions
Rock
Football
21.9% Dung Beetle
25 0% Goose
16.8% - Fiddler Crab
15.1% Standard Poodle
Figure I: Predictions of models supervised baselines for corrupted images. Our robust encoders observe highly corrupted images and use simple linear probe to classify ImageNet-100 100 labels. present the top 3 classes from our models as well as those from the end-to-end supervised baselines trained with same amount of labeled data, for select images. For three different types of forward operators (90 percent missing pixels, strong Gaussian noise and high Blurring) robust encoders classify correctly and also produce reasonable top 3 alternatives. On the contrary, the supervised baselines completely fail even though they fine-tuned on exactly this task to classify corrupted images, starting from powerful ImageNet pretrained ResNet-101. We also expect that most humans would fail to classify such highly corrupted images more examples are included in the Appendix.
In this paper, we introduce the study of new family of inverse problems: reconstructing the representation of an image given a corrupted measured input. Formally, if a (clean) image is and its CLIP representation is R(x), we would like to obtain that representation by only observing highly corrupted input Azx). This is impossible if the forward process A removes information needed to obtain the representation. Surprisingly, we show that we can recover representations that useful for downstream tasks, even from extremely corrupted versions of the image.
We introduce robust encoder that is trained imitate the behavior of the pretrained CLIP encoder acting on clean images и. However, the input to the robust encoder is only corrupted images A(x) that are created by applying the forward operator on æ. Our approach is illustrated Figure The teacher encoder is the pretrained CLIP, and the student encoder is our robust encoder operating on corrupted images. Formally, the robust encoder S(A(z)) is trained approximate R(x) using contrastive loss.
Many applications, such as object recognition with low-cost cameras, remote sensing and aerial imaging rely noisy or blurry data and can face occlusions sensor corruptions. As we demonstrate the Appendix, normal CLIP fails on highly corrupted images. Our procedure allows us to transfer the power of CLIP to heavily corrupted images in downstream tasks, with relatively little extra training.
our experiments
1.1 Results
We show that our method is able to obtain useful representations even under extreme corruptions such as removing 90% of pixels as shown in the top panel of Figure The highly corrupted images enter our robust encoder and the obtained representation is used in linear classifier to produce ImageNet- 100 labels. Our main result that our method outperforms pretrained ResNet (of the same size as robust encoder) fine-tuned end-to-end on labeled distorted images.

---

bmbbeddingge
Teacher
Copy
Batch
Cuntoncion
Dister
Nakay Embeddangs
Figure 2: Overview of our proposed method. We initialize student and teacher model from a pretrained CLIP encoder. Clean image batches are fed to the teacher while distorted versions of those images are fed to the student. The student is trained using contrastive loss which makes student and teacher representations of the same original images more similar while making their representations different images less similar.
Using less labeled data For some corruption levels, we are able to outperform end-to-end fine-tuned ResNets using as little as 10% of labeled samples. This is even when the fine-tune sunetuned baseline uses 100% of ImageNet- 100 labels for training. The primary advantage of our model is that our robust encoder observes representations from the pretrained CLIP encoder, which was trained with much bigger dataset compared to ImageNet. Still, the fact that this implicit advantage the representations and 10% of labeled data is sufficient to outperform a supervised trained ResNet fine-tuned with 10 times more labeled data very surprising and illustrates the power and versatility of pretrained representation learners.
Robustness to noise and data shifts Our method is very robust changes in the forward operators, data statistics and label shifts. We experiment with three classes of forward operators: random pixel masking, additive Gaussian noise, and Gaussian blurring distortions. For each, we train and test on wide-range of distortion levels, from slight to severe corruption. We show that our robust encoder produces useful representations even when the level corruption is outside its training domain.
We show that our representations are useful for a wide range of tasks, without requiring knowledge of the task when the robust encoder is trained. We illustrate excellent classification accuracy across five datasets, frequently outperforming end-to-end supervised baselines trained with knowledge of the target task. Our experiments include chest X-ray COVID pneumonia task which has very different morphology compared to ImageNet. Surprisingly, the same universal representations, combined with a custom linear probe are very successful across all tasks.
Contrastive versus MSE training We formulate the contrastive student training method as a regular- ization upon the simple mean squared error loss between student embeddings of distorted image and teacher embeddings of the clean version of that image. We analyze the effects of this regularization on the training dynamics. Our results empirically show that simple MSE is worse in most cases, further strengthening our argument for the usefulness of contrastive learning in this setting.
Related Work
Robust Image Recognition has been shown that classifiers that perform well on clean image tasks are not robust to common image distortions [21]. Several datasets have been proposed specifi- cally to benchmark generalization of classifier performance to natural distortions [8] 38). Related to our work, (45 and (12) fine-tune pretrained classifiers distorted data, which seems yield better performance than end-to-end training. However even under these training processes, modern classifiers exhibit inferior performance human vision distorted data [2]
Inverse problems There significant recent literature on solving inverse problems including denoising, inpainting and deconvolution for deblurring. While classical techniques rely on sparsity
based priors e.g. 30 37 recent techniques include data-driven deep-learning methods [1] 10 13
as well as combinations of sparsity and generative methods [I] and untrained deep nets [6] Our work focuses on recovering the representation of an image, as opposed to an image itself, so (23) However, approach is fundamentally different from all
also related to task-aware sensing
3

---

previous inverse problems since instead of trying to reconstruct the image we aim to reconstruct representation R(z), which lives different space. The other important distinction that even though our corruption processes are linear in the pixel space, they are non-linear with respect to the representation vector we try to recover. In effect, we are solving a non-linear inverse problem in a supervised way using contrastive loss. In the Appendix, we compare to methods that attempt to solve the inverse problem in pixel space, then apply a classifier on the recovered image.
Contrastive Representation Learning Self-supervised representation learning has recently ex- ploded in popularity, largely due to the success of constrastive losses in learning representations from unlabeled data 4 6 15
77 43). These techniques able to generate highly general embeddings of images that are effective for many types of downstream tasks, even on domains that were not explicitly considered in training 127]. Contrastive losses generally operate based on a simple push-pul puul principle: images desired to be close in embedding space are pushed together, while unrelated images pulled apart. One particularly popular choice of contrastive loss is the InfoNCE loss, derived from techniques for noise contrastive estimation [1]. and popularized in the self-supervised setting in [3). Several works have considered adversarial training frameworks yield representations that are more [2
robust to adversarial attacks [2 25.. However these works only consider adversarial robustness and not robustness to common corruptions. Our approach is similar to that of (24), which employs
variant of the InfoNCE loss for a supervised setting. This differs from our work in that we exclusively focus on robustness natural corruptions. Moreover, our contrastive training step can be performed without any task sask-specific labeled data. This is made possible by the use of the powerful embeddings provided by CLIP, and allows the embeddings be used for downstream tasks on multiple datasets.
Knowledge Transfer Methods Our work is closely related to prior works which aim to distill,
reduce. or transfer the knowledge from one network to another for a specific task B 19 29
44]. Of
note is [40], where the authors use contrastive objective to transfer representations from teacher network student network. Our work diverges in that we do not transfer from larger, more powerful teacher to a smaller student, but rather transfer between a teacher and student of the same architecture initialized from the same weights. In addition, although the authors test on cross-modal transfer tasks such as transferring between color channels, transferring representations between clean and distorted images is a different task: we try to extract the same high-level information from less data as opposed to different, but related data.
3 Method
Our problem is to recover the embeddings of clean images when we only have access to highly corrupted versions of the images. The encoder R(-) ) is assumed to yield high-quality representations for a variety of domains. The distortion process A(-) is assumed to be a known forward operator that greatly distorts images. We assume A()) is sufficiently severe to inhibit the performance of the encoder R(.), but not so severe that recovery is impossible. From a collection of input images {xx N we are only given access to the distorted images and the representations of the clean images {A(x;), R(x;)) Our approach is to learn a student function S so that S (((()) is equally useful as the teacher representation R(-). We measure the utility of a representation by the performance on an unspecified downstream supervised learning task.
Least squares Loss
One potential approach for the task of recovering the teacher's embeddings from the corrupted inputs is to minimize the expected fz distance in embedding space between the clean teacher embeddings R(.) and the predicted student representations S(A()). If both R(-) and (-)) are constrained have normoaale outputs, then the empirical least squares loss becomes:
N Σ (S((a)))) R(xe))
(S A)
N
where we have expanded and subtracted out constant terms. However, this process may not yield the most effective embedding of the corrupted data. Due to either limitations in the training process, the severity of the distortion process, or the generalization properties of the approximate minimizers of AMSEE the learned embeddings S (A(()) may not be as useful for downstream tasks as embeddings learned using the other losses we consider.

---

Contrastive Loss Inspired by recent advances in self-supervised representation learning, we learn by minimizing contrastive loss. We consider the following variant on the popular InfoNCE loss:
(xp(K(i,))/)) Σ Exp((
(contr (S;
log
N
where (i := (S(A(z;)), R(r,)) measures the similarity between the learned embedding of A(x:) and the clean embedding of Tj, and 7 is temperature hyperparameter. We follow [2]42] and rewrite _contr in terms of explicit 'pull' and "push" terms as
1 L R, Lunit (S; R, A),
conttr (S: A)
(3)
where the second term, referred to as the uniformity term, is defined as:
Gunif (S; R. A) =
R(JJ)/T
(4)
The first term of Geantr is simply LSS which encourages alignment of ((A(z)) with R(x;), and the Lunif term encourages the learned representations for corrupted data be dissimilar from all other representations of clean data.
The primary difference between Equation learning the choice of the similarity measure -., Without access to specified target embeddings, (it is typically chosen to be the inner product between projections of the embeddings of x; and xj. (or between x; and a positive example in the case of K K(i, i)). Here, the uniformity term is necessary prevent representation collapse, where in our setting LMS alone suffices to prevent this degenerate case. We note that alternative choices for ((,,-) -) in Lunif may be employed. For example, we could also contrast the student embeddings S(A(-)) across two different images. We ablate against other choices in the experimental section and find that our choice of ((.,)) is one of several that exhibits comparable performance.
and and the InfoNCE loss commonly used in self- sulf-suue
Effect of uniformity term To examine the effect of the uniformity term of the loss on the training dynamics, we consider the gradients of Lcontr with respect to the parameters of the encoder S. We first decompose Gniif to consider the contributions of each individual data point:
Gunit
1 N
Cunit T,; R,
Lynif (S; R, A) log
KU.
j=1
Then the gradient of ccnntr with respect to the parameters of S may be written as
N
N
V eontt (S;T, R, = v AMSE (S; R, A)
where
wi(i) exp exx==exxxxxi))) (K(i-))--- Cinit (S; R,,
This can be interpreted as follows. The weighting of the first term by -- balances the gradients V T (i, i) and Σ; VgK(i.)) as is common with other choices of contrastive losses (c.f. Theorem in (41). ). Noting that for each i, the weights wr,() sum to 1, the gradient of each individual uniformity term Éunif is convex combination of the gradients of the similarity terms K(i..). The individual similarity terms are weighted exponentially proportionally to each (ii,j's contribution to the total uniformity loss. As decreases, the weights place greater emphasis on terms that are most similar. As commonly chosen to be less than 1, the dynamics automatically reflect the influence of the "hardest" negative examples.
5

---

When is perfect recovery possible? Finally, we describe conditions which allow for perfect recovery of the clean embeddings from the corrupted images in the training set. The first condition that the corruption process should not be too destructive: it suffices to assume that the implication A(r²) = A(r,) = R(x;)= = R(x;) holds for all Ti, in the training set, i.e., there exists function which attains exact recovery. In this case, any S that minimizes ÊMSE has 5* (A(x:)) R(z:) for all z; in the training set. To argue for the same recovery guarantees when optimizing Lconer we need to assume that the teacher R(-) provides a well-separated embedding of the training data.
Proposition 1. If R is a minimizer of the uniformity term
N
N
(ft.),
arg min
log
exp
j=1
then any encoder S" E arg mingg Connt (S;T, R, A) exactly recovers the target embedding, (*(xi))) R(z;) for all zi in the training set.
Training procedure Our final method consists of (1) the contrastive step, in which the student learns representations for the distorted images and (2) a fine-tuning step, where we train linear classifier on top of the learned representations. During the second step, the student encoder is kept frozen.
Experiments
We show that the proposed training process recovers useful representations from corrupted inputs for a variety of forward operators, evaluating several different classification tasks. We start by evaluating the representation quality when the distortion process and data distribution are the same during both training and inference, examining the label-efficiency of our approach.
Then evaluate approach when the severity of the distortions changes test time, when the test data distribution differs from the one used during training, and when the labels are shifted. Finally, ablate against alternative formulations of our loss function.
For all experiments, we perform contrastive training for the robust encoder using a 100-class subset of ImageNet, which we refer to as ImageNet-100 [36] [9] to reduce computational resources. Our target representations are attained from the CL.IP ResNet-101. We find robust representations with contrastive learning and evaluate by training linear classifier top of the frozen representations. In all experiments, the distortions are applied randomly to each batch of images, and independently for each image in the batch. Where applicable, we report our results over 10 different instances of random corruptions applied on the evaluation images. Our baselines are built on a ResNet- 101 initialized with weights from supervised training the full ImageNet dataset. The final fully-connecte layer is replaced with a 100-dimensional output. We fine-tune the whole model in a supervised fashion using distorted inputs and their correct labels from ImageNet- 100. The baseline is trained for 25 epochs with batch size of 64. Our robust encoder is trained for 25 epochs with a batch size of 256, and the linear probe top of is trained for 10 epochs. Additional experiments, as well further details on training and hyperparameter choices are discussed in the appendix and in our provided code.
4.1 Known Data Distribution and Forward Operator
In this section, evaluate the quality of the learned robust representations for classifying images from the validation set of ImageNet- 100, using the same distortions during training and inference. These experiments demonstrate the usefulness of our method for vision inverse problems where the data distribution and forward operator are both known at training time. We also demonstrate the label-efficiency of our approach by evaluating our learned representation when only few labeled samples are available to train the linear classifier.
Setup We train the robust contrastive model and the baseline as described above. Bight different distortion processes are examined: Gaussian blur with (kernel size, standard deviation) of (21, 5) and (37, 9); additive Gaussian noise with standard deviations 1. 0.3, and 0.5; and random pixel mask with 50%, 75,, and 90% of the pixels missing. We evaluate our method against the baseline in top-1

---

70
Top Accuray
Ilaidon mask, wut
20
Randon mask, buacine
Gaustian
Gausiion bosei
0.375 0.5%
57C Fraction Labels
15
10%
100%
Figure 3: Accuracies using a varying fraction of labeled samples to train a linear probe. We train robust encoders on images with 90% random pixel masking and additive Gaussian noise with standard deviation 0.5, and fit linear classifier the learned representations using varying fractions of labeled training samples. We compare a supervised baseline that uses all of the labeled training samples. Results are averaged over 10 random instantiations of corruptions on the ImageNet- validation dataset. We omit error bars as standard error is insignificant.
Table 1: Top-1 accuracy (percent) on ImageNet-100..
best accuracy for each distortion bolded. Each model trained using images with a fixed type of distortion. We train our robust CI.IP encoder contrastively, then fit a linear probe on the learned representations using either all or 10% of the labeled training samples. We report the mean and standard error for accuracy over 10 random instantiations of distortions on the ImageNet- 100 validation dataset (Gaussian blur is deterministic, n corresponds the length of blur kernel.
do not include standard error values). For Gaussian blur,
Supervised Baseline
Ours
Distortion
Ours
(10% labeled data)
Random Mask 50%
77.53:10.06
85.87:10.08
19.0000
Random Mask 75%
0.06
83.99
80.36:10.09
Random Mask 90%
74.124:0.09
82.96:10.08
79.87=0.12
Gaussian Noise 0.1
82.23-10.04
84.46:10008
80.99=0.09
Gaussian Noise a 0.3
75.78110.08
81.30:10.07
18.10:10.07
Gaussian Noise 0.5
71.431:..1
7..2
73.14:00.08
Gaussian Blur n 21
76.40
83.24
80.94
Gaussian Blur n
68.94
77.80
74.84
accuracy on the validation set of ImageNet- 100,, under the same distortion used to train each model. To demonstrate the label-efficiency of our method, also train linear probe with only 10% of the labeled data, and for two of our models we train linear probes using various amounts of labeled data.
that training linear probe top of the representations learned by procedure greatly improves accuracy compared to the supervised baseline. This further solidifies our original motivation. Furthermore, our model to outperform the baseline most cases. More fine-grained label-efficiency results can be seen in Figure where show that in two of these cases, using just 5% of the labeled data makes our model outperform or be competitive with the baseline trained using all the labels.
Results In Table T
that using only 10% of labeled samples is sufficient for
4.2 Known Data Distribution and Unknown Forward Operator
In many settings, the data distribution and the type of distortion will be known training time, but the severity of the distortion will be unknown. We consider the case where the severity of the distortion test-time is greater than what was seen during training.
Setup First, we train a model using training images with between 50 and 95% of pixels randomly masked. In addition, we train model using images with additive Gaussian noise with random standard deviation between 0.1 and 0.3. Once fully trained, we fit a linear classifier on top of the learned representations for each model using the same distortions. We also train two supervised baselines end-to-end with the same distortions. We evaluate the models trained with pixel masking on images with fixed level of to 99% percent missing pixels, and the networks trained with noise on images with additive Gaussian noise using fixed standard deviation between 0.35 and 0.5.
7

---

Top- Accuraty
30
28
Top
Froction Missing Poels
Noise Standard Deviztion
Figure 4: Accuracies for images with varying corruption levels using models trained range of levels. In the left figure, we compare robust model with baseline, both trained images with 50% 95% random pixel masking. the right figure, each model trained on images with additive Gaussian noise with random standard deviation from 0.1 to 0.3. We evaluate the models on images with more severe corruptions than applied during training. Results are averaged over 10 random instantiations of corruptions on the ImageNet-100 validation dataset. We omit error bars as standard error is insignificant.
Table 2: Top-1 accuracies for transfer learning. We fit linear classifier for each dataset on top of the representations learned by the models from ImageNet- 100. RM means the model was trained with random missing pixels, and GN means was trained with additive Gaussian noise. Results are mean and standard errors over 10 realizations of the distortions during evaluation.
Model
CIFAR-10
CIFAR-100
STL-10
COVID X-ray
ImageNet-100B
Baseline (RM)
7.08
55.10=0.08
81.09=0.05
82.0710.23
67.45110.09
Ours (RM)
80.93
0.04
58.55 0.09
89.44 0.05
84.01 + 0.20
80.09
Baseline (GN)
7.052=0.07
52.00=0.05
82028=0.07
83.54=0.15
69.49:10.1
Ours (GN)
7.019=0.09
52.51
86.30-0.08
79.74=#0.26
78.66:10.13
Results In Figure 4 we see that with an increase in noise levels, the accuracy of the models does decrease. However, the linear probe trained the entire dataset achieves better results than the baseline end-to-end supervised model. This shows that our model is more robust, even when the distortions are greater than those expected during training.
4.3 Unknown Data Distribution and Known Forward Operator
In this section we evaluate how well ImageNet pretraining with distortions allows the learned representations to transfer to different datasets. We use the same forward operator during training and inference to isolate the quality of the embeddings learned during the contrastive step even in the presence of distortion.
Setup Five datasets are chosen to evaluate transferability of robust representations. (1) CIFAR-10
and (2) CIFAR- 100 28), and (3) STI-110 g (4) The COVID-19 Chest X-ray dataset 9. (5) We generate another random 100-class subset of ImageNet (36] from the remaining 900 classes we did not use for ImageNet-106 100, which we refer to as ImageNet-100B.
We train the same models under the same distortions as outlined in Section 4.2 then fit linear classifiers for the new datasets on top of the fixed representations. Preprocessing details for each dataset may be found in the Appendix. We calculate top-1 accuracy of each model for classifying distorted images from the validation or test set of each of the new datasets. The images are distorted using the same forward operators used to train the networks.
Results Table 2 shows that our approach can achieve good results in a variety of datasets. In CIFAR-I0 and CTFAR-100, we get results comparable to the baseline. In STL-10, we have greater top 1 accuracy for both noise settings. For the COVID X-ray dataset, we get mixed results, where we beat the baseline for the random masking model, but we lose for the additive Gaussian noise model. The most surprising result is the vast increase in accuracy for the alternative ImageNet-1000 dataset.

---

Table 3: Label Shift: Out Distribution ImageNet Classes and their New Labels. the five ImageNet classes taken from outside of ImageNet-100 and the new labels given to them based on similar classes from within ImageNet-100 100.
ImageNet Class
New ImageNet- 100 Label
Cup
Cocktail Shaker
Dungeness Crab
Rock Crab
Mountain Bike
Moped
Wood Rabbit
Hare
French Bulldog
American Staffordshire Terrier
Our Base Ine
40
Top Accurssy (%)
600
Dure Badin
50%
10%
100%
Fraction
Missin Proels
Noise Standard Devistion
Figure 5: Accuracies for varying noise levels on unseen classes using label shift. On the left, we see the model trained on 50% to 95% random masking of pixels, while on the right the model trained on Gaussian noise with standard deviation from 0.1 to 0.3. Both models are evaluated on the unseen classes, using the chosen reference classes from ImageNet100 the targets, shown in the label shift table. Results averaged over 10 random instantiations of corruptions. We omit error bars in the left figure standard error insignificant.
This shows that due the supervised training of the baseline, some information from the labels leaks into the representation, leading to worse performance on related, but ultimately different dataset.
4.4 Label Shift
An important factor in determining the robustness a model is how gracefully it fails in the presence of unseen data modalities at inference time. For instance, if network trained to distinguish dogs from cars shown an image of a cat, the network should produce a embedding which is more likely to be classified a dog than car. This is even more important for inverse problems: single distorted image may result from the same forward operator being applied to any number of original images. If a distorted image from class outside of the training classes makes network output an unexpected or uninformative representation, then the network is likely also brittle to shifts in the data distribution within the classes knows at inference time. We evaluate the quality of representations produced by our method for distorted images from classes outside of the training dataset.
Setup We use the same models described in Section 4.2 To simulate images from unseen classes, we identify classes from the validation set of full ImageNet that were not used in our training data, but are similar to classes within training data. We replace the labels of these new images with the label of the classes in our training data that are similar to them, as seen in Table 3] Evaluation is done on images from these replacement classes, under fixed levels of distortion, with 50% to 99% missing pixels on the random masking model and 0.05 to 0.5 standard deviation for the Gaussian noise model.
Results In Figure 5] we can see that, for the random pixel mask case, we consistently outperform the baseline across all noise levels. For the Gaussian noise case, our model has slightly lower top-1 accuracy for small noises. This can be explained by the fact that small distortions do not drastically alter the image, so the baseline is bolstered by its ImageNet pretraining. However, as the noise level increases, the baseline model degrades much more quickly.

---

Tops Accuracy (%)
Stucent Teacher
Stureet Stuceet
Stucnni Buth
N1-Xar
MSE
00%
92%
94%
95%
68%
100%
Froction of Missing Powes
Figure 6: Ablation study. We evaluate the model trained on 50% to 95% random masking of pixels, using several variants of the uniformity term in the contrastive loss. Results averaged over 10 random instantiations of corruptions the ImageNet-100 validation dataset. We omit error bars standard error is insignificant.
4.5 Ablation: Training Loss
Finally, we ablate against various choices for the uniformity term in the contrastive loss. We consider variants of Gunif where we compare representations of i) the student and teacher, ii) the student and the student, iii) sum of both of the above. We also consider CMS i.c. no uniformity term, as well as one variant where every representation is contrasted with every other representation (irrespective of student or teacher), which we note is the same formulation as NT--entt as used in SimCLR [6]
Setup We use the same random masking model described in Section 4.2 with a similar evaluation on fixed masking levels of 90-99%. Explicit formulations of each of the losses is in the appendix.
Results We draw two main conclusions from Figure 6 First, the MSE loss performs worst, indicating the benefit of the uniformity term in the loss. Even though we have access to the pre- trained representations, it is not simple for the model to exploit the encoded information. If this were the case, MSE would be as effective as our contrastive loss. Second, we see that all student comparisons perform roughly equivalently, but are clearly more effective than MSE and NT-Xent.
5 Conclusion
In this work, we propose method for training image representation networks which are robust to various distortions on the input data. Our method has potential for improving the practical applications of powerful pre-trained models. Indeed, images in real-world settings are rarely exemplary: every stage of the imaging process, from capture to storage to transmission and display, can introduce noise or distortions in the images. Moreover, our process helps reduce the cost of training such models, both with respect to computation since we can add robustness to a pre-trained model instead of training one from scratch, as well as with respect to label efficiency since our method mainly relies on large amounts of inexpensive unlabeled data.
Several significant open problems emerge from this work. As we can see in Appendix D.8 our method requires some prior knowledge of the type of distortions that will occur to the image. This is due to the fact that our method relies on training the student to match the representations of the teacher, which significantly more difficult when the type of corruption (and thus, the representation itself) changes drastically. As future work, we aim to extend our method beyond this limitation, possibly by finetuning the teacher as well as the student, to produce representations that are more easily matched under different types of distortions. In addition, another interesting research direction how well this method performs under adversarially-chosen inputs.
Acknowledgements
This research has been supported by NSF Grants CCF 1763702, 1934932, AF 1901281, 2008710 10, 2019844 the NSF IFMI. 201 2019844 award as well as research gifts by Western Digital, Interdigital, WNCG and MLL, computing resources from TACC and the Archie Straiton Fellowship.
10

---

References
[1] M. Asim, A. Ahmed, and P. Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. arXiv preprint arXiv: 1672, 2019.
[2] A. Buades, B. Coll, and J.-M. Morel. Non-local means denoising. Image Processing On Line, 1:208-212, 2011.
[3] C. Bucila, R. Caruana, and A. Niculescu-Mizil. Model compression. In KDD '06, 2006.
[4] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of
visual features by contrasting cluster assignments. arXiv preprint arXtv:2006.09882, 2020.
[5] T. Chen and L. Li. Intriguing properties of contrastive losses. CoRR, Utthttps: penttps:/aarxxt //axxiv./////vve/// orgg/rgg.....oo. htrggabb/20111 02803
2020.
[6] T. Chen, S. Korblithh, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations, ArXiv, abs/2002.05709, 2020.
[7] T. Chen, S. Kormblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self self-suuprvised models are strong semi-supervised learners. ArXiv, abs/2006. 10022000610029, 2020.
[8] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In G. Gordon, D. Dunson, and M. Dudík, editors, Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FI., USA, 11-13 Apr 2011. mlr. httpa/eres/////cctteee1111 http://oaeesaaaaaaaaaaaaa
PMLR. URL htL:tttp//p/oceediiii
[9] J. P. Cohen, P. Morrison, and L. Dao. Covid-19 image data collection. arXiv 2003.11597, 2020. URL https://gi http://gttttuutte cwt/:b/ee/eee2e22.. wwwp:/eee00000/oidodooo cht.:///s//oooddeecctaaaaaaaaaaa dataset
[10] M. Z. Darestani, A. Chaudhari, and R. Heckel. Measuring robustness in deep learning based compressive sensing, arXiv preprint arXiv:2102.0611 2021.
[11] M. Dhar, A. Grover, and S. Ermon. Modeling sparse deviations for compressed sensing using generative models. arXiv preprint arXiv:1807.01444 2018.
[12] S. Dodge and L. Karam. A study and comparison of human and deep learning recognition performance under visual distortions. In 2017 26th International Conference on Computer Communication and Networks (ICCCN), pages 1-7, 2017. doi: 10. 109///CCC.2001.8800000
[13] L. Gómez, A. Eftekhari, and V. Cevher. Fast and provable admm for learning with generative priors. arXiv preprint 1907v:1907.033 2019.
[14] M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Y. W. Teh and M. Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia. Italy, 13-15 May 2010. PMLR. httppttt::/prroceedaa iml..mlr............mmmmmm.mm.. www.mmrrr........mmmm.mmmmmmmmle /vass//v//a gutmann 1entmannn0a.... 10a. html
[15] K. Hc. H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[16] R. Heckel and P. Hand. Deep decoder: Concise image representations from untrained non- convolutional networks. arXiv preprint arXiv: 1810.0398, 2018
[17] O. Henaff. Data-efficient image recognition with contrastive predictive coding. and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 1 119 of Proceedings of Machine Learning Reseurch, pages 4182-4192, PMLR, 13-18 Jul 2020. 2/tp./ut/tttpp//p/o/eeeeeeeeeeee httttp://pproeedingg.s.. mlr. https/////11////eaaf222220 html
III
[18] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. In International Conference on Learning Representations, 2019, URL https://openreviexx www..net/oormmidd--eeeitiii
[19] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http:///axxpxi.....o org/
abs/1503. 11111033331100111111111111111111
[20] C.-H. Ho and N. Vasconcelos. Contrastive learning with adversarial examples. Oct. 2020.
11

---

[21] H. Hosseini, B. Xiao, and R. Poovendran. Google's cloud vision api is not robust to noise. 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 101-105, 2017.
[22] Z. Jiang, T. Chen. T. Chen, and Z. Wang. Robust pre-training by adversarial contrastive learning.
In NeurlPS, 2020.
[23] M. Kabkab, P. Samangouei, and R. Chellappa. Task-aware compressed sensing with generative adversarial networks. In Pncrrt--econd AAAI Conference on Artificial Intelligence, 2018.
[24] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and
D. Krishnan. Supervised contrastive learning. Apr. 2020.
[25] M. Kim, J. Tack, and S. J. Hwang. Adversarial Self-Supervised contrastive learning. June 2020.
[26] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980,
2015.
[27] K. Kotar, G. Ilharco. L. Schmidt, K. Ehsani, and R. Mottaghi. Contrasting contrastive Self- Supervised representation learning models. Mar. 2021.
[28) A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[29] D. Lee. Pseudo-label The simple and efficient semi cemi-supervised learning method for deep
neural networks. 2013.
[30] M. Lustig, D. Donoho, and J. M. Pauly. Sparse mri: The application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 58(6): (0000):1121955.11 2007.
[31] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett. Deep learning
techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39-56, 2020.
[32] A. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding.
ArXiv, abs/1807.00 807.03748, 2018.
[33] P. Pandit, M. Sahraee- S. Rangan, P. Schniter, and A. K. Fletcher. Inference with deep generative priors in high dimensions. arXiv preprint arXiv: 1911999110099 2019.
[34] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Desmaison, Kopf, E. Yang. De Vito, M. Raison, A. Tejani, S. CCilamkurthy,, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019. 9015- an- ann-inperative-ttyee hhighhpperrrrranneeeeeeeeeaaaaaa library, pdf
nwwrips. httpappppppppppprreeeeeeppppp/pp
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krüger, and I. Sutskever. Learning transferable visual models from
natural language supervision. https: pettpa:/oopeaai http://paaa..cooooooo///o//// www.//o/ooooooiiieeecoooooooooce cllp// 2021.
[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. S. Bernstein, A. Berg, and L. Fei-Fei. Imagenet large scale visual recognition
challenge. International Journal of Computer Vision, 115:211-252, 2015.
[37] J. Scarlett and V. Cevher. Limits on support recovery with probabilistic models: An information- theoretic framework. IEEE Transactions on Information Theory, 63(1):593-620, 2016.
[38] D. Temel, J. Lee, and G. Alregib. Cure-or: Challenging unreal and real environments for object recognition. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 137-144, 2018. doi: 10.1 1109//CML.22018000000
[39] Y. Tian, D. Krishnan, and P. Isola. 12Xiv:190.05844 2019.
Contrastive multiview coding. arXiv preprint
[40] Y. Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020. URL http::////ppeeereiie http:ppewrrreee.eeeeeeeeeee
id=SkgpBJrtvS)
[41] Y. Tian, L. Yu, X. Chen, and S. Ganguli. Understanding self-supervised leam learning with dual deep networks. CoRR, abs/2010.00578, 2020. URL httptt/t/axxii....o. org/.....sb../// aa/aabbb//000 2et.//00000008888
12

---

[42] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In JCML, 2020.
[43] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[44] Xic, E. Hovy, M.-T. Luong, and Q. Le Self-training with noisy student improves imagenet classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2020.
[45] Y. Zhou, S. Song, and N.-M. Cheung. On classification of distorted images with deep convo- lutional neural networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1213-1217, 2017. doi: 10. 1011099/CAS99220777922999999
13


Reviews:
[{"review_id": "L-_7OEqy-XV", "reviewer": "Reviewer_N8iG", "summary": "This works suggests a novel method on how to harness the representational power of models such as CLIP. In this task the input is distorted with a known function, such as missing pixels or gaussian noise, blurring. This works proposes training a student contrastively by matching the clip representation of the clean image with student's representation of the distorted image. They propose different matching functions and explain why in their setup it will not collapse if one does not compare against every other. \nThe experimental setup is extensive and shows much better label efficiency when working with distorted images, they also show better noise extrapolation and better dataset tranferability in compare to baseline. Their baseline is a resnet pretrained on imagenet supervised trained on the distorted imagenet-100.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea here, contrastively matching clean clip representations, is novel for this specific task. similar ideas have been used for adversarial robustness as well as related work explained in the paper. \n\nOne question is the choice of CLIP as the teacher. It would be interesting to know how much of the gain is to just from matching \"A\" clean representation and how much of it is from \"The best\" clean representation. An experiment with for example a simclrV2 pretrained as the teacher can give some insight whether just having access to the clean images during training gives the same gain. \n\nThey also propose a specific similarity measure and explain how it prevents collapse and tends toward hardest negatives. Their ablation also suggests that their proposed formulation is slightly better than simple MSE or typical InfoNCE loss (simCLR style). \n\nThe main portion of the paper is dedicated to experiments. They have extensive evaluation which shows their method consistently outperforms their baseline. The only baseline that they consider is a resnet pretrained on imagenet, finetuned supervised on the distorted dataset. Another important baseline which is missing is passing the distorted images through clip directly and finetuning a linear prob. \n\nApart from the label efficiency experiment results which are super impressive, I was also impressed by their results on OOD class accuracy and robustness. \n\nIn general, this is a significant step in utilizing the representation power of giantly trained models such as clip for computer vision tasks. The writeup is dense specially in the method section, but the experimental setup and results are properly presented and discussed. Unfortunately, the limitations (conclusion) section is forced to appendix.\n\n----Post rebuttal\n\nThank you for running the experiment with CLIP on corrupted images. The results further strengthens the claims in this paper, therefore I'm increasing my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have discussed the main limitations in the appendix. This is a strong paper and I will increase the score with the CLIP on distorted results. Most probably this work outperforms just running CLIP on the distorted images directly, but currently it is an importantly missed baseline.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "H11N6VDIm9", "reviewer": "Reviewer_9uEJ", "summary": "This paper proposes to recover the feature representation, pre-trained from CLIP, of a clean input image given a corrupted version of the image. They propose to recover the features, instead of the clean image, as their goal is to use these features for downstream tasks, e.g. classification. Furthermore, this allows for the transfer of knowledge from CLIP, which reduces the amount of labeled data needed for the downstream task. To recover the features, they train a student model via L2-Loss and contrastive loss given the clean feature from CLIP. Empirically, they evaluate on a subset of ImageNet (with only 100 classes) and consider three types of corruptions, random masking, Gaussian noise, and Gaussian blur. Empirical comparison with a baseline method, trained end-to-end with corrupted images, demonstrates their approach is more robust and requires less labeled data.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# A. Originality\n\nThe task presented in this work is interesting, i.e., the reconstruction of the cleaned pre-trained features from CLIP. Prior works in inverse problems tend to focus on recovering ``good visual quality’’, however, when applying to downstream tasks, direct recovery of features seems to be a good strategy. However, the proposed approach’s method is based on existing work on contrastive loss, and fairly similar to Supervised contrastive learning (NeurIPS 2020) [23]. While the authors explained\n \n> propose a fundamentally unsupervised approach where we have no knowledge of labels when learning the representations.\n \nI find the difference minor, specifically, instead of label supervision, their approach requires pre-trained features from CLIP. Additionally, labels are required for the explored down-stream tasks. \n \n\n# B. Quality\nThe writing of the paper is adequate, and the approach section is easy to follow. Below are some questions and concerns I have with the experiment setup and compared baseline.\n \n\n## B1. Experiment Setup\n\n* B1A) ImageNet Subset\nThe experiments are conducted using a subset of ImageNet using 100 classes. The authors cited [36] which uses the ImageNet100 dataset. However, keep in mind that in [36] they did report the full 1000-class ImageNet result. Reporting the full ImageNet result is beneficial as it allows for a fair comparison.\n \n* B1B) ImageNet-C Benchmark. \nIn the main paper, no result for ImageNet-C (Hendrycks and Dietterich 2019) is presented. In the supplementary materials, results with a subset of Image-NetC is presented; however, once again, the subset is not ideal. The whole point of ImageNet-C is to\n \n> establish rigorous benchmarks for image classifier robustness.\n \nThis will allow a fair comparison with prior works and more thoroughly evaluate the proposed approach.\n \n* B1C) The reported error bar is on the randomness of the corruption at test-time.\nThe paper could be strengthened by also reporting the error-bar over the randomness during training, i.e., train a model multiple times with different random seed.\n \n\n## B2. Baseline Model\n* B2A) More details on how pre-training is done?\nAt Line 179, it states\n> we pretrain the robust encoder using a 100-class subset of ImageNet\nI couldn’t find more details on how the pre-training is implemented. Supplemental seems to describe the training details.\n \n* B2B) The baseline model not pre-trained on ImageNet-100\nAt Line 182, it states\n> Our baselines are built on a ResNet-101 183 initialized with weights from supervised training on the full ImageNet dataset\n \nIt would be better to also pre-train the baseline models on ImageNet-100, as it is the dataset of evaluation. Also, this would make the baseline method consistent with the proposed method. Otherwise, the effect of pre-training is not ablated from the proposed approach.\n \n* B2C) Hyperparameter tuning on the reported dataset?\nFrom the text, it seems like hyperparameters are searched on the same set as the reported result. Is there a separate validation set for tuning the hyperparameters? \n \n* B2D) Please report the full hyperparameter search space \nIn supplemental Line 122\n \n> We choose the hyperparameters for each model using a linear search over several values for each hyperparameter. \n \nFor completeness, please report all the searched values.\n \n\n# C. Clarity\n## C1. Title\nI find the title not suitable for this paper. Specifically, the paper is more about building a robust classifier, as all the downstream tasks are related to classification and less about \"inverse problems\". Furthermore, the result on ImageNet100-C suggests that the approach isn’t beneficial in a few cases of corruption, hence the general catch all name of \"inverse problems\" seems to be an overclaim. \n \n## C2. Motivation\nThe introduction of the paper could be improved. I recommend motivating why one would want to recover the clean pre-trained features; Currently, the motivation comes towards the end of Sec. 1.\n \n\n# D. Significance\nOverall, the paper demonstrates some potential of the proposed approach. However, there remain some concerns regarding the experiment setup and compared baseline as stated above.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "DaE5U1MIgT", "reviewer": "Reviewer_oqRY", "summary": "The paper consider the following problem: Let $x$ be an image, let $R$ be a feature map or representation obtained through the CLIP network, and let $A$ be a distortion process. The representation takes as input an image and yields a feature representation. The distortion process also takes an image as input, and yields a distorted image (e.g., it deletes pixels).\n\nThe paper assumes we are given the representations and corresponding distorted images, i.e., a collection $\\{A(x_i), R(x_i)\\}$, and the goal is to learn a student function $S(A( \\cdot ))$ that is equally useful as the original representation $R(x)$ for a classification task. The paper learns the student $S$ by minimizing a contrastive loss, and the utility of the representation obtained by the student $S$ is measured by the performance on a supervised learning task, specifically on ImageNet-100. \n\nThe paper provides simulation results demonstrating that the method learns useful representations. The paper's main result is to show that the method 'outperforms a pretrained ResNet, of the same size as the robust encoder, fine-tuned end-to-end on labeled distorted images'. In addition to this result, the paper provides a number of ablations studies and compares to two reasonable baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity: The paper is well written and well organized.\n\nQuality: The paper provides good baselines and comparisons: The paper's main result is that the proposed method using the contrastive loss ''outperforms a pretrained ResNet, of the same size as the robust encoder, fine-tuned end-to-end on labeled distorted images''. Here, the paper compares to a ResNet that is trained on the noisy image end-to-end. That's a reasonable baseline; and another interesting baseline is to first denoise/recover the images and then train a classifier on the denoised images. This second baseline is studied in Appendix E.3. \n\nMotivation/Significance: The motivation to study the problem is not entirely clear to me; the paper writes 'in this paper we introduce the study of a new family of inverse problems', but it is not clear to me where the problem considered arrises. Would it be possible to give a real-world example application where the problem considered occurs?\n\nIn summary, the paper introduces a new problem and provides a well-performing approach to address it, and also provides good baselines and comparisons. On the other hand, the motivation for the problem is entirely unclear to me which is a major negative point.\n\n\nMinor comments:\n- The paper provides as baseline a ResNet-101 which has the same number of parameters as the representation it learns, and the network is trained on the noisy examples. That's a good baseline; but would it be also possible to state the numbers for the same method (ResNet-101) trained on the original representation, to have an idea on how bad the effect of the respective perturbations is?\n- The paper 'pretrains the robust encoder using the 100-class subset of ImageNet'. Would it be possible to add ablation studies showing how important the pretraining is for learning good representations?\n- Regarding the setup; does the paper use the same noise and random pixel mask for each image, or a new one for each image? I assume a different one, but this wasn't clear to me from the main body, apologies if I missed something.\n- The title 'inverse problems leveraging pre-trained contrastive representations' is misleading as it suggest the paper is concerned with solving an inverse problem. Yet, the paper is about learning a feature map from a disturbed version of an image.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "7yrEZdhd-F5", "reviewer": "Reviewer_4CMh", "summary": "This paper proposes an inversion method that exploits a pre-trained representation for regularization. Given only corrupted images, instead of working in the pixel space, it attempts to find their matching representation pairs in the pre-trained representation space that is trained with clean images by using contrastive objective function. Assuming that one has an access to a powerful representation that is already available, this method provides a robust representation against various types and levels of distortions and train/test time discrepancy. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The organization of the paper needs to be revised. As a current form, it looks incomplete. The conclusion is missing, and the results are sparsely spread out over the paper. Considering the high academic standards, this is not acceptable, and I highly recommend the authors to revise the manuscript in a simpler and clearer manner. \n\n- Other than the organization, my main concern is whether this is indeed a novel approach as the authors say. Please clarify the contribution and difference more against the works related to GAN inversion, deep generative compressed sensing (such as Bora et al. 2018), and [37].\n  - Although it is not exactly the same, there have been several previous works in the field of GAN inversion such as deep generative prior that exploit the pre-trained GAN to find matching latent representations, which are eventually used for recovering the clean image given the (corrupted) measurements. On the other hand, AmbientGAN attempts to find a clean data distribution given only the corrupted measurement. In this regard, the proposed method is one branch of the existing problems sharing the motivation, and thus for me, the statement that this is a new inverse problem is an overclaim. However, I understand that their final goals are different in that the previous works aim for reconstructing an image and exploit the representation as a tool to regularize the solution space, while the proposed method aim for learning a new representation itself and use it for classification. \n  - Line 105 – Line 107, “Our work diverges in that we do not transfer from a larger, more powerful teacher to a smaller student, but rather transfer between a teacher and student of the same architecture initialized from the same weights.” Isn’t this a too minor difference?\n  - Line 30, “Surprisingly, we show that it can be very well approximated even from extremely corrupted versions of the image.” For me, this is also not surprising. Because a model has an access to the pre-trained representation of clean images of similar (sometime even the same) kinds, it can exploit the encoded information. It is indeed impressive that the representation is helpful even for classifying totally different data distribution such as medical data, this has been also well known as a typical advantage of having the pre-trained weights (a.k.a. warm starting). \n\n- Line 44, “For some corruption levels, we are able to outperform end-to-end fine-tuned ResNets using as little as 10% of labeled samples. This is even when the fine-tuned baseline uses 100% of ImageNet-100 labels for training.” \nIs it legitimate to say that the current method is using less supervision? CLIP is trained on massive amounts of data and these are all encoded in its representation, which means that the method is implicitly using/inheriting the encoded information. This needs some more discussions. \n\n- Line 211, “We consider the case where the severity of the distortion at test-time is greater than what was seen during training.” \nWhat happens when it is less severe at test-time? \n\n- Only the classification is used as a downstream task to show the superiority of the learned representation. However, since the representation is learned against corrupted image measurements, it would be more interesting to see image reconstruction as a downstream task (how much it recovers the original image) using the learned representation. \n\n- What is the payoff for having more robust representation against extreme corruptions such as removing 90% of pixels? In the related work, the authors said that adversarial training frameworks are robust to adversarial attacks but not to common corruptions. On the other way around, how’s this method against the adversarial robustness? Won’t this result in a bad adversarial robustness? \n\n\n- Line 17, In this, paper -> In this paper, \n- Line 98, Our approach is similar that -> similar to/with\n- Line 177, Finally we -> Finally, we\n- Line 178, Punctuation dot is missing at the end.\n\n---\n\nUpdate after rebuttal, the authors have addressed my concerns, and I increase my score to 6. However, as the other reviewers have pointed out, I also agree that the authors need to clarify the main contributions and differences of the proposed model against the related work with extra care. The organization of the manuscript also needs some changes. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comment", "review_text": "", "extra_scores": {"ethical_concerns": "No comment", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers criticize the quality of writing and the unusual terminology, the artificial nature of the proposed problem, they seem to be be in agreement about the novelty and significance of the results.
Decision: Accept (Poster)