Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors mitigate the data-hungriness of the CLIP model. The authors propose three directions: single-modality self-supervision; multi-view multi-modality contrastive learning, and nearest-neighbor supervision. With the proposed three components, the authors can achieve better or comparable results with CLIP with more than 4x fewer data.
main_review: Pros:
- Clear advantage of the proposed framework on data efficiency.
- Simple and straightforward methodology.
- Carefully designed experiments. (Comparison to CLIP with 15M data and ablation on computation time.)

Cons:
- Limited novelty in terms of the method:
    - The self-supervision and multi-view supervision are very similar to what has been used in Yuan et al.
    - The Nearest-neighbor supervision was used in Dwibedi et al. and Van Gansbeke et al.
- Experiments:
    - Missing experiments:
        - One of the biggest advantages of CLIP is its robustness to domain shift. Can the authors also provide performance on datasets like ImageNet-R, ImageNet Sketch etc.?
        - No results on image text retrieval.
        - The CLIP reports on 27 downstream datasets. Is there any specific reason for not comparing on all these datasets?
    - While the authors compare with training time, another big difference is in memory consumption. With the additional views, it is equivalent to doubling the batch size. A fairer comparison should be doubling the batch size for CLIP training and train for the same number of iterations or the same amount of time. Since large batch size is important in contrastive learning, it is worth-doing to me.

Questions:
- I believe CLIP also use random crop during training. How do the authors design the small local view (how small)? Can authors provide ablations on the effect of small size?
    - In addition, is the augmented view the same size (in terms of network input) as the original view but just more zoomed in?
- Did the authors do careful duplication removal?
summary_of_the_review: While the novelty is limited, the authors are able to achieve good results with a much smallerset data size. Potentially, with more data, this method can achieve much more superior performance than CLIP. However, I think the insights conveyed from this paper are limited so I will choose borderline accept.


Post-rebuttal update:
- I am very happy with the additional results, which address most of my concerns on fair comparison with CLIP. I will remain the original rating due to the limited "novelty". But I would give a 7 if there is a 7 because I think this paper should be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes DeCLIP with three additions to the original CLIP model: (1) single-modal self supervised pre-training; (2) cross-modal contrastive pretraining across multiple views and (3)  cross-modal contrastive pretraining across nearest neighbors in feature space. DeCLIP is able to achieve higher performance on most of downstream datasets with much less pre-training data (80M vs. 400M). 
main_review: Strength:
- Extensive experiments show DeCLIP achieves competitive improvements over CLIP while uses much less pre-training data, which also results in much less training time. With the same amount of pre-training data, DeCLIP achieves significant improvements over CLIP (+7%)
- The three additions are simple yet effective. The ablation study shows that all three of them contribute to the final performance improvements.
- Visualization show that nearest neighbor pairs can provide reasonable similar supervision signal compared to the original pair.
- The paper is well-written and easy to follow.


A few questions about the paper:
- For nearest neighbor pairs, are them all sampled from the same dataset as the original image-text pair? If so, are there different FIFO queue Q for different datasets during pre-training?
- What about other distance measures to compute nearest neighbors, for example cosine similarity? 
- Why not also sample nearest neighbor images? If added, it should add 8x more supervision than the original CLIP.

summary_of_the_review: This paper provides three simple yet effective additions to CLIP and show its strong performance with much less pre-training data. Overall, I feel the paper is clearly written, with extensive experimental results to support its claim.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes DeCLIP to further utilize the data potential by adding three training objectives to CLIP pre-training: 1) inspired by SimSiam and BERT, self-supervised objectives are added to both image and text; 2) they generate different views for both images and text, and apply contrastive objectives; 3) they sample neighbor text as additional positive examples.

DeCLIP improves data efficiency. With web-crawled data, DeCLIP outperforms CLIP counterparts with 4.5x smaller amount of data. In addition, while introducing addition objectives and especially different views increases per-batch compute time by 1.5x, the authors show that DeCLIP still outperforms CLIP when given the same compute time budget.


main_review: 
Pro:
This is the first work that I can think of that successfully combines self-supervision and supervised training, at such a large scale. While self-supervised training has shown promising results in CV, the experiments are usually done in an unsupervised/self-supervised setting where the model is trained with only raw images and then tested with linear probes. It remains less explored how one could combing self-supervision and supervised image-text pairs data and whether self-supervision is still helpful when a large amount of supervised data are available. This work provides strong empirical evidence that self-supervision can be combined with supervised data.

   

Con:
Table 2 shows that DeCLIP with 88M data can outperform CLIP with 400M. However, the 88M data are collected by the authors while the 400M data are collected by CLIP authors. As shown in Table 8 in the appendix, how one process/selects the data can result in a large performance difference. 

I cannot safely conclude that the data efficiency/performance improvement shown in Table 2 is completely attributed to DeCLIP. It could be attributed partially to that the 88M data are a high-quality subset of the 400M data. I would recommend adding a baseline of CLIP trained with 88M data.

The authors provide fair baselines of CLIP in Figure 7 and Table 8 (in the appendix) but the dataset scale is limited. As one of the core claims is the benefit of DeCLIP under large-scale pre-training, I think providing fair baselines in Table 2 is important.

Minor points:

   1. Analysis lacking on why we need self-supervision signals / nearest-neighbor supervision. 

      As noted in Pro, this paper touches upon an important problem: the relationship between self-supervision and supervised data. 

      However, the authors only show empirical evidence of self-supervision can be helpful even when a large amount of supervised data are available. It would be helpful to provide some discussions. For example, does cross-view self-supervision provide signals simply not available in supervised data? Does DeCLIP do better in certain types of tasks?

   2. I am under the impression that nearest neighbors are usually used to provide "hard negatives" in contrastive learning (e.g., in the Image Retrieval task in UNITER (Chen et al., 2019)). However, here the nearest neighbors are used as "positive" examples. I wonder what causes the different practices.


summary_of_the_review: **After rebuttal: I have read the authors' rebuttal and it resolves my concerns. I will raise my score to between 6 and 8.**
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a data-efficient CLIP (DeCLIP) by employing three additional objectives:

- (1) self-supervision within each modality (e.g., vision domain => SimSiam, text domain => masked language model as BERT)
- (2) Multi-view supervision (e.g., resize random crop for vision modality and EDA augmentation -- synonym replacement, random insertion, random swap, and
random deletion -- for text modality)
- (3) Nearest-neighbor supervision (e.g., using close plausible neighbors to pseudo-positives)

Table 2 shows that DeCLIP shows better zero-shot ImageNet top-1 accuracies with smaller dataset sizes (400M => 88M), and Table 3 shows that DeCLIP shows better overall linear probe performances on various datasets.

Also, Table 4 and Figure 7 support that the proposed three additional objectives make the CLIP training much efficient than the vanilla version.
main_review: ## Strengths

- (+) The proposed DeCLIP shows better ImageNet zero-shot performances than the original CLIP models with the same parameter size but smaller training data size.
- (+) The ablation study (Table 4) supports that the proposed three objectives are important to the final performances. Also, Figure 7 shows that the convergence speed of DeCLIP is faster than the original CLIP loss.

## Weaknesses

### CLIP numbers and DeCLIP numbers would not be comparable; the CLIP training environment is different from the DeCLIP training environment

I fully understand that CLIP results are almost impossible to reproduce with ordinary infrastructures. It is too expensive to reproduce the original ResNet50x64 results (18 days with 592 V100 GPUs). It takes more than one week for ResNet and ViT DeCLIP training even with 80 V100 GPUs (Appendix C), and the largest RegNet DeCLIP takes three weeks with 160 V100 GPUs. Hence, I fully understand that it is very difficult to report many numbers.

However, since this paper argues that "CLIP is data-hungry and DeCLIP can learn strong representation as much as CLIP with much smaller dataset size", I think that the CLIP results should be the reproduced numbers.

- The original CLIP paper did not investigate the impact of dataset size (except ablation study on the YFCC-15M dataset), and there is no justification of the dataset size of 400M. Hence, I think that arguing "CLIP needs 400M data points" needs to be justified by the following studies (namely, in this submission). In other words, I would like to see "reproduced CLIP results" in Figure 1 using the authors' implementation. Since DeCLIP needs more resources than CLIP (because of additional data augmentations, computation costs, memory size, ...), I think the authors can report CLIP results for 15M / 29M / 56M / 88M as DeCLIP.
- I presume that the reproduced CLIP environment is different from the original CLIP implementation (which is not publicly accessible yet). Table 8 amplifies my assumption; the reproduced CLIP performance is 4.6% better than the original CLIP (31.3 => 35.9). In my opinion, it could be very critical if the performance gains are originated from the DeCLIP implementation itself, not from the proposed objective functions. Hence, as I suggested in the previous bullet point, I would like to see the reproduced CLIP numbers for various dataset sizes.
- Third, we do not have enough knowledge on **88M DeCLIP full data** collected by the authors. I wonder whether the performance improvements are from the data collection, not the proposed method. For example, 88M full data quality could be better than the original 400M dataset. As described in Section C, additional filtering processes, such as Chinese caption filtering, would improve the dataset quality compared to CLIP. It is well-known that learning with fewer clean data often outperforms learning with a large noisy dataset. To avoid such concerns, I think Figure 1, Table 2, and 3 should need additional rows "CLIP training + 88M training data" for each model. In my opinion, because the training datasets are different, CLIP 400M models and DeCLIP 88M models are not directly comparable in zero-shot and linear probe benchmarks.
- Finally, CLIP did not use any data augmentation on the language domain, while DeCLIP uses the EDA augmentation for the multi-view supervision. It is not clear whether the performance improvement by MVS in Table 4 is from the EDA augmentation or the multi-view supervision. Because DeCLIP uses more data points than CLIP (Algorithm 1: Line 10), I wonder if the performance gain is from the data augmentation. CLIP + EDA augmentation experiments may answer my question.

Similarly, I would like to suggest reporting RegNetY-64GF CLIP results in Table 2.

### Additional analyses may need quantitative comparisons

This weakness is not very critical, but the submission will be stronger if the authors can address this issue.

Figure 8 and Figure 9 show the CAM visualizations and nearest neighbor search results by CLIP and DeCLIP. However, these results are very easy to cherry-pick; hence, these results should be supported by the quantitative comparisons.

For CAM comparisons, I would recommend weakly-supervised object localization (WSOL) benchmarks. WSOL does not require any additional training but only CAM score maps. I recommend using Choe et al. https://github.com/clovaai/wsolevaluation for comparing CLIP, and DeCLIP CAM results more concretely.

Nearest neighbor search performances can be easily measured by cross-modal retrieval benchmarks (in a zero-shot manner). I would recommend famous COCO or Flickr caption retrieval benchmarks. If the authors need more dense benchmarks, please refer Parekh et al. https://github.com/google-research-datasets/Crisscrossed-Captions.

Since these benchmarks are available without additional expensive training, I strongly recommend reporting the results using the suggested benchmarks.

## Questions

- Why are there no zero-shot results for other benchmarks? I only can see the linear probe verifications (Figure 2, Table 3).
- In Table 3, DeCLIP ViT-B/32 shows worse linear probe results on ImageNet than CLIP VIT-B/32, despite DeCLIP showing better zero-shot ImageNet results in Table 2. What is the guess by the authors for the results?
- It seems that the proposed DeCLIP needs more computation / memory resources compared to CLIP. Could the authors provide the memory consumption comparison between CLIP and DeCLIP? I presume that with the same GPUs, DeCLIP cannot use the same batch size as CLIP.

## List of suggested experiments

I fully understand that the rebuttal period is too short to produce the following numbers. At least, I would expect 88M CLIP results (Table 2) for ResNet-50 and ViT-B/32.

- I am willing to revise my score if these results show the same tendency as the paper claim
  - Figure 1. Reproduced CLIP ImageNet zero-shot top-1 results for 15M / 29M / 56M / 88M (15M is already in Table 8) -- ResNet-50
  - Table 2. Reproduced CLIP ImageNet zero-shot top-1 results for 88M with ResNet-50, (ResNet-101 -- not necessary), ViT-B/32, and RegNETY-64GF
  - Table 3. Reproduced CLIP linear probe results on 11 downstream tasks
- These numbers are not mandatory, but I think these results will make the submission stronger
  - Figure 8. WSOL benchmark (https://github.com/clovaai/wsolevaluation)
  - Figure 9. Cross-modal retrieval benchmarks (COCO Caption, Flickr Caption, Crisscrossed Caption, ...)
  - CLIP + EDA augmentation results on 15M / 29M / 56M / 88M datasets (ImageNet zero-shot top-1)
summary_of_the_review: This paper shows a strong empirical contribution; DeCLIP shows better results than CLIP with a smaller dataset size. However, I think the weakness of this paper slightly overweighs the strengths.

In particular, I am not fully convinced that DeCLIP is data-efficient and CLIP is data-inefficient. This argument should be supported by empirical comparisons, e.g., the reproduced CLIP results should be reported in Figure 1, Table 2, and 3. I listed why I think the DeCLIP results and the CLIP results are not directly comparable in the weakness section.

All technical components of DeCLIP are hard to say novel; self-supervision uses SimSiam and masked language model, multi-view supervision uses the EDA augmentation, nearest neighbor supervision can be viewed as a self-distillation method. However, the combination of known techniques can be novel if the empirical contribution is significant. I think this paper has a limited borderline novelty, but the empirical contribution can be a strength. For example, if this paper outperforms "ViT-L/14@336px" with much fewer data points, I think the empirical contribution exceeds its weakness. (disclaimer: I do not request ViT-L/14@336px results. I know that it is very expensive and unachievable by ordinary infrastructures) As of now, I think the empirical contribution may need more verifications, especially for the vanilla CLIP results.

Overall, I recommend "marginally below the acceptance threshold" for my initial recommendation.

-----------------------------
Post rebuttal comment.

After reading the responses (for raised concerns by all reviewers) and the revised paper, I think the authors address my concerns very well in general. I would like to encourage the authors includes the results in A3, A4 (not completed yet), A5, A6 in the paper. Also, I encourage the authors to add zero-shot results (Q7) in the final revision, which was not possible at the submission time (I missed it).

After the revision, the arguments are generally well-supported (revised my score for correctness has been updated to 4 from 3). Now, I recommend "6: marginally above the acceptance threshold" for this paper.
review_title: null

Meta-review: This paper aims at improving the data efficiency of pretraining in CLIP. This is a practically meaningful research direction. The proposed method is simple, even kind of straightforward and has limited innovations. It combines self-supervision within each modality, multi-view supervision across modalities, and nearest-neighbor supervision from other similar pairs. Such a combination showed strong empirical results: achieved better performance using seven times fewer data. The rebuttals resolved most critical concerns on experiments, such as fair comparisons with the original CLIP work.
Decision: Accept (Poster)