Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes to recover the feature representation, pre-trained from CLIP, of a clean input image given a corrupted version of the image. They propose to recover the features, instead of the clean image, as their goal is to use these features for downstream tasks, e.g. classification. Furthermore, this allows for the transfer of knowledge from CLIP, which reduces the amount of labeled data needed for the downstream task. To recover the features, they train a student model via L2-Loss and contrastive loss given the clean feature from CLIP. Empirically, they evaluate on a subset of ImageNet (with only 100 classes) and consider three types of corruptions, random masking, Gaussian noise, and Gaussian blur. Empirical comparison with a baseline method, trained end-to-end with corrupted images, demonstrates their approach is more robust and requires less labeled data.
main_review: # A. Originality

The task presented in this work is interesting, i.e., the reconstruction of the cleaned pre-trained features from CLIP. Prior works in inverse problems tend to focus on recovering ``good visual quality’’, however, when applying to downstream tasks, direct recovery of features seems to be a good strategy. However, the proposed approach’s method is based on existing work on contrastive loss, and fairly similar to Supervised contrastive learning (NeurIPS 2020) [23]. While the authors explained
 
> propose a fundamentally unsupervised approach where we have no knowledge of labels when learning the representations.
 
I find the difference minor, specifically, instead of label supervision, their approach requires pre-trained features from CLIP. Additionally, labels are required for the explored down-stream tasks. 
 

# B. Quality
The writing of the paper is adequate, and the approach section is easy to follow. Below are some questions and concerns I have with the experiment setup and compared baseline.
 

## B1. Experiment Setup

* B1A) ImageNet Subset
The experiments are conducted using a subset of ImageNet using 100 classes. The authors cited [36] which uses the ImageNet100 dataset. However, keep in mind that in [36] they did report the full 1000-class ImageNet result. Reporting the full ImageNet result is beneficial as it allows for a fair comparison.
 
* B1B) ImageNet-C Benchmark. 
In the main paper, no result for ImageNet-C (Hendrycks and Dietterich 2019) is presented. In the supplementary materials, results with a subset of Image-NetC is presented; however, once again, the subset is not ideal. The whole point of ImageNet-C is to
 
> establish rigorous benchmarks for image classifier robustness.
 
This will allow a fair comparison with prior works and more thoroughly evaluate the proposed approach.
 
* B1C) The reported error bar is on the randomness of the corruption at test-time.
The paper could be strengthened by also reporting the error-bar over the randomness during training, i.e., train a model multiple times with different random seed.
 

## B2. Baseline Model
* B2A) More details on how pre-training is done?
At Line 179, it states
> we pretrain the robust encoder using a 100-class subset of ImageNet
I couldn’t find more details on how the pre-training is implemented. Supplemental seems to describe the training details.
 
* B2B) The baseline model not pre-trained on ImageNet-100
At Line 182, it states
> Our baselines are built on a ResNet-101 183 initialized with weights from supervised training on the full ImageNet dataset
 
It would be better to also pre-train the baseline models on ImageNet-100, as it is the dataset of evaluation. Also, this would make the baseline method consistent with the proposed method. Otherwise, the effect of pre-training is not ablated from the proposed approach.
 
* B2C) Hyperparameter tuning on the reported dataset?
From the text, it seems like hyperparameters are searched on the same set as the reported result. Is there a separate validation set for tuning the hyperparameters? 
 
* B2D) Please report the full hyperparameter search space 
In supplemental Line 122
 
> We choose the hyperparameters for each model using a linear search over several values for each hyperparameter. 
 
For completeness, please report all the searched values.
 

# C. Clarity
## C1. Title
I find the title not suitable for this paper. Specifically, the paper is more about building a robust classifier, as all the downstream tasks are related to classification and less about "inverse problems". Furthermore, the result on ImageNet100-C suggests that the approach isn’t beneficial in a few cases of corruption, hence the general catch all name of "inverse problems" seems to be an overclaim. 
 
## C2. Motivation
The introduction of the paper could be improved. I recommend motivating why one would want to recover the clean pre-trained features; Currently, the motivation comes towards the end of Sec. 1.
 

# D. Significance
Overall, the paper demonstrates some potential of the proposed approach. However, there remain some concerns regarding the experiment setup and compared baseline as stated above.

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This works suggests a novel method on how to harness the representational power of models such as CLIP. In this task the input is distorted with a known function, such as missing pixels or gaussian noise, blurring. This works proposes training a student contrastively by matching the clip representation of the clean image with student's representation of the distorted image. They propose different matching functions and explain why in their setup it will not collapse if one does not compare against every other. 
The experimental setup is extensive and shows much better label efficiency when working with distorted images, they also show better noise extrapolation and better dataset tranferability in compare to baseline. Their baseline is a resnet pretrained on imagenet supervised trained on the distorted imagenet-100.
main_review: The main idea here, contrastively matching clean clip representations, is novel for this specific task. similar ideas have been used for adversarial robustness as well as related work explained in the paper. 

One question is the choice of CLIP as the teacher. It would be interesting to know how much of the gain is to just from matching "A" clean representation and how much of it is from "The best" clean representation. An experiment with for example a simclrV2 pretrained as the teacher can give some insight whether just having access to the clean images during training gives the same gain. 

They also propose a specific similarity measure and explain how it prevents collapse and tends toward hardest negatives. Their ablation also suggests that their proposed formulation is slightly better than simple MSE or typical InfoNCE loss (simCLR style). 

The main portion of the paper is dedicated to experiments. They have extensive evaluation which shows their method consistently outperforms their baseline. The only baseline that they consider is a resnet pretrained on imagenet, finetuned supervised on the distorted dataset. Another important baseline which is missing is passing the distorted images through clip directly and finetuning a linear prob. 

Apart from the label efficiency experiment results which are super impressive, I was also impressed by their results on OOD class accuracy and robustness. 

In general, this is a significant step in utilizing the representation power of giantly trained models such as clip for computer vision tasks. The writeup is dense specially in the method section, but the experimental setup and results are properly presented and discussed. Unfortunately, the limitations (conclusion) section is forced to appendix.

----Post rebuttal

Thank you for running the experiment with CLIP on corrupted images. The results further strengthens the claims in this paper, therefore I'm increasing my score.
limitations_and_societal_impact: Yes, they have discussed the main limitations in the appendix. This is a strong paper and I will increase the score with the CLIP on distorted results. Most probably this work outperforms just running CLIP on the distorted images directly, but currently it is an importantly missed baseline.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an inversion method that exploits a pre-trained representation for regularization. Given only corrupted images, instead of working in the pixel space, it attempts to find their matching representation pairs in the pre-trained representation space that is trained with clean images by using contrastive objective function. Assuming that one has an access to a powerful representation that is already available, this method provides a robust representation against various types and levels of distortions and train/test time discrepancy. 
main_review: The organization of the paper needs to be revised. As a current form, it looks incomplete. The conclusion is missing, and the results are sparsely spread out over the paper. Considering the high academic standards, this is not acceptable, and I highly recommend the authors to revise the manuscript in a simpler and clearer manner. 

- Other than the organization, my main concern is whether this is indeed a novel approach as the authors say. Please clarify the contribution and difference more against the works related to GAN inversion, deep generative compressed sensing (such as Bora et al. 2018), and [37].
  - Although it is not exactly the same, there have been several previous works in the field of GAN inversion such as deep generative prior that exploit the pre-trained GAN to find matching latent representations, which are eventually used for recovering the clean image given the (corrupted) measurements. On the other hand, AmbientGAN attempts to find a clean data distribution given only the corrupted measurement. In this regard, the proposed method is one branch of the existing problems sharing the motivation, and thus for me, the statement that this is a new inverse problem is an overclaim. However, I understand that their final goals are different in that the previous works aim for reconstructing an image and exploit the representation as a tool to regularize the solution space, while the proposed method aim for learning a new representation itself and use it for classification. 
  - Line 105 – Line 107, “Our work diverges in that we do not transfer from a larger, more powerful teacher to a smaller student, but rather transfer between a teacher and student of the same architecture initialized from the same weights.” Isn’t this a too minor difference?
  - Line 30, “Surprisingly, we show that it can be very well approximated even from extremely corrupted versions of the image.” For me, this is also not surprising. Because a model has an access to the pre-trained representation of clean images of similar (sometime even the same) kinds, it can exploit the encoded information. It is indeed impressive that the representation is helpful even for classifying totally different data distribution such as medical data, this has been also well known as a typical advantage of having the pre-trained weights (a.k.a. warm starting). 

- Line 44, “For some corruption levels, we are able to outperform end-to-end fine-tuned ResNets using as little as 10% of labeled samples. This is even when the fine-tuned baseline uses 100% of ImageNet-100 labels for training.” 
Is it legitimate to say that the current method is using less supervision? CLIP is trained on massive amounts of data and these are all encoded in its representation, which means that the method is implicitly using/inheriting the encoded information. This needs some more discussions. 

- Line 211, “We consider the case where the severity of the distortion at test-time is greater than what was seen during training.” 
What happens when it is less severe at test-time? 

- Only the classification is used as a downstream task to show the superiority of the learned representation. However, since the representation is learned against corrupted image measurements, it would be more interesting to see image reconstruction as a downstream task (how much it recovers the original image) using the learned representation. 

- What is the payoff for having more robust representation against extreme corruptions such as removing 90% of pixels? In the related work, the authors said that adversarial training frameworks are robust to adversarial attacks but not to common corruptions. On the other way around, how’s this method against the adversarial robustness? Won’t this result in a bad adversarial robustness? 


- Line 17, In this, paper -> In this paper, 
- Line 98, Our approach is similar that -> similar to/with
- Line 177, Finally we -> Finally, we
- Line 178, Punctuation dot is missing at the end.

---

Update after rebuttal, the authors have addressed my concerns, and I increase my score to 6. However, as the other reviewers have pointed out, I also agree that the authors need to clarify the main contributions and differences of the proposed model against the related work with extra care. The organization of the manuscript also needs some changes. 
limitations_and_societal_impact: No comment
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper consider the following problem: Let $x$ be an image, let $R$ be a feature map or representation obtained through the CLIP network, and let $A$ be a distortion process. The representation takes as input an image and yields a feature representation. The distortion process also takes an image as input, and yields a distorted image (e.g., it deletes pixels).

The paper assumes we are given the representations and corresponding distorted images, i.e., a collection $\{A(x_i), R(x_i)\}$, and the goal is to learn a student function $S(A( \cdot ))$ that is equally useful as the original representation $R(x)$ for a classification task. The paper learns the student $S$ by minimizing a contrastive loss, and the utility of the representation obtained by the student $S$ is measured by the performance on a supervised learning task, specifically on ImageNet-100. 

The paper provides simulation results demonstrating that the method learns useful representations. The paper's main result is to show that the method 'outperforms a pretrained ResNet, of the same size as the robust encoder, fine-tuned end-to-end on labeled distorted images'. In addition to this result, the paper provides a number of ablations studies and compares to two reasonable baselines.
main_review: Clarity: The paper is well written and well organized.

Quality: The paper provides good baselines and comparisons: The paper's main result is that the proposed method using the contrastive loss ''outperforms a pretrained ResNet, of the same size as the robust encoder, fine-tuned end-to-end on labeled distorted images''. Here, the paper compares to a ResNet that is trained on the noisy image end-to-end. That's a reasonable baseline; and another interesting baseline is to first denoise/recover the images and then train a classifier on the denoised images. This second baseline is studied in Appendix E.3. 

Motivation/Significance: The motivation to study the problem is not entirely clear to me; the paper writes 'in this paper we introduce the study of a new family of inverse problems', but it is not clear to me where the problem considered arrises. Would it be possible to give a real-world example application where the problem considered occurs?

In summary, the paper introduces a new problem and provides a well-performing approach to address it, and also provides good baselines and comparisons. On the other hand, the motivation for the problem is entirely unclear to me which is a major negative point.


Minor comments:
- The paper provides as baseline a ResNet-101 which has the same number of parameters as the representation it learns, and the network is trained on the noisy examples. That's a good baseline; but would it be also possible to state the numbers for the same method (ResNet-101) trained on the original representation, to have an idea on how bad the effect of the respective perturbations is?
- The paper 'pretrains the robust encoder using the 100-class subset of ImageNet'. Would it be possible to add ablation studies showing how important the pretraining is for learning good representations?
- Regarding the setup; does the paper use the same noise and random pixel mask for each image, or a new one for each image? I assume a different one, but this wasn't clear to me from the main body, apologies if I missed something.
- The title 'inverse problems leveraging pre-trained contrastive representations' is misleading as it suggest the paper is concerned with solving an inverse problem. Yet, the paper is about learning a feature map from a disturbed version of an image.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: While the reviewers criticize the quality of writing and the unusual terminology, the artificial nature of the proposed problem, they seem to be be in agreement about the novelty and significance of the results.
Decision: Accept (Poster)