Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of representation learning for image retrieval and follow-ups on recent self-supervised methods that use cluster discrimination to handle the limitations of instance discrimination. These methods however either require to perform iterative clustering, or online clustering to avoid multiple iterations over the entire dataset but that still suffer from the collapsing problem. The authors propose a method that performs random selections along class and feature to alleviate the inter-class conflict and to improve feature compactness that only requires a single pre-step of off-line clustering.
strength_and_weaknesses: Pros
- The idea and motivation behind the method is presented very well by pointing out the limitations of currents methods regarding their clustering approach and proposing a possible solution. 
- This idea is also very simple, which makes it widely applicable, not only to this problem/setting.
- Experiments are also extensive and thorough.

Cons:
- The fact that you need access to a large pre-trained model it makes its applicability more limiting and not directly comparable to OPEN-CLIP. I see the approach as a way of further improving the results of a given base model, CLIP in this case.
- Related to my previous point, and given that we have assume that we have access to CLIP and that we are going to train the same backbone architecture (is this true?), would it make sense to initialize the backbone using CLIP's weights? Would this speed up training without impacting performance? This might address the problem of needing to have access to a large cluster of GPUs.
- It would have been interesting to see this applied to other large pre-trained models used as base, eg DINO or MAE.
- In table 8, I did not understand what are the different columns below Unicom. Could the authors elaborate please?
clarity,_quality,_novelty_and_reproducibility: - The paper is well written and easy to understand.
- This code is easily reproducible because the idea is simple, but the experiments only if you have access to 128 V100 GPUs
- The idea of subsampling classes and features might not be novel in general, but it is when applied to this context.
summary_of_the_review: The paper is well written and the idea and motivation behind the method is presented very well. This idea, although simple (this also makes it widely applicable) it is novel in this context. I also think that the methods has some limitations, like the fact that one assumes access to a large pre-trained model.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use random sampling for samples in clustering classes to avoid inter-class conflict and a random dropout mechanism for features to generate compact features.
strength_and_weaknesses: The method is easy to understand and implement. The experiments about using VL models for zero-shot/transfer learning tasks are exhaustive.

clarity,_quality,_novelty_and_reproducibility: The novelty of the paper is not significant. The sampling dimension of the feature is not too much different from Dropout method[1], which has been widely used in CV/ML pipelines.



[1] Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15.1 (2014): 1929-1958.

summary_of_the_review: As I mentioned above, although the method is easy to implement, the novelty of this paper is not significant. Especially for feature random selection part. For class selection, although it is easy to implement, it lacks further detailed analysis and improvement. For instance, we can use another clustering method for each cluster to exclude the potential inter-class conflict.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a self-supervised image representation approach that leverages using pre-existing CLIP image-text embedding. The approach first clusters images using CLIP embedding, and then learns image feature with softmax loss using cluster ids as class ids. In addition, random dimension sampling is employed during learning to achieve low dimensionality for retrieval tasks. The authors conducted extensive results on benchmark datasets to demonstrate the effectiveness of proposed approach in various settings.
strength_and_weaknesses: Strength 
+ Intuitive idea and easy to reproduce
+ Extensive experimental results + ablation studies

Weakness
- Novelty is a bit low. Sampled softmax is a commonly used technique. 
- Dimension sampling is a bit strange. A more natural approach to me would be to directly learn a projection to low dimensional feature space. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Overall paper is well written. Only non-clear part is that, during softmax learning, whether the entire image encoder is learned from scratch, or fine-tuned from the pre-existing CLIP image embedding. Would be good to compare the two in ablation studies.

Novelty: low (see above)

Reproducibility: approach is very intuitive and easy to reproduce
summary_of_the_review: While approach is appealing and practically useful, the novelty seems slightly below the bar of ICLR.  In additional, the dimension sampling part seems unnecessary to me. I feel directly learning a feature projection would be more intuitive and work better.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to learn representations for image retrieval with two main ideas: 1) during pre-training, automatically clustering a large image-text dataset and randomly selecting inter-class prototypes to avoid noisy assignments, and 2) to reduce embedding dimensionality, randomly select feature dimensions to be deactivated. Experiments are performed on standard datasets in the area of image retrieval, besides on standard image classification datasets, comparing against unsupervised and supervised methods.
strength_and_weaknesses: Strengths:

S1) A well-motivated idea to improve unsupervised learning via cluster discrimination. Cluster discrimination may suffer from inter-class conflicts of the prototypes, and the authors propose a very simple and efficient method to deal with it: randomly sample the negative prototypes. Ablation experiments show large improvements when using this idea.

S2) The proposed pre-training method outperforms other CLIP training methods on image classification linear probe experiments and on unsupervised image retrieval experiments. The comparison is strict against the OPEN-CLIP work, where the exact same dataset is used.

S3) The results on the supervised image retrieval tasks outperform other recent competitor methods.

S4) Overall, the experiments are comprehensive, including ablations and even extra comparisons on GLDv2 and the recent Google Universal Embedding Challenge.

Weaknesses:

W1) Random selection of class prototypes seems very simple and effective, but I am wondering whether something smarter can be even better. For example, discarding “hard negative prototypes” directly instead of only sampling randomly. The paper does not provide such comparisons, which would make the proposal stronger.

W2) I wonder if it is fair to call the proposed method “unsupervised”. The technique works on a large dataset of images and associated text, which could be considered already some form of weak supervision (it has more details than the Instagram hashtag dataset, for example). The CLIP text embeddings are even directly used to perform clustering. Generally, “unsupervised” methods work directly on images only and nothing else.

W3) I don’t see a direct connection between equations (1) and (2) in section 3.1. How is equation (2) actually maximizing the log-likelihood function? Do you mean that p(x_i;\theta) would be the probability of drawing the augmented image? Please explain. Similar comment for equations (3) and (4).

W4) Figure (3)-a is confusing. By “class number”, do the authors mean the ID of the class? By “photo number”, do you mean the number of photos in that class?

W5) There is a very straightforward connection between the random feature subsampling and the concept of Dropout. This connection is not made, but it should be.

W6) The random feature subsampling method works comparably to PCA or the addition of an FC layer to reduce dimensionality in the network. I don’t see a concrete benefit of using it, compared to a simple FC layer to reduce dimensionality, which is more principled.

W7) The story of the paper is inconsistent. There are many experiments on image classification datasets (section 4.2), but the paper is directed at the image retrieval problem.

Minor details:

M1) In the caption of Fig 1: the word “dataset” is repeated on the second line.

M2) Page 3: “cluttering” -> “clustering”

M3) Section 3.1: the set X  is repeated twice in the same first sentence, please remove one of them to make it more concise.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is generally clear and the reader can understand what the authors are proposing. However, the story is not very clear/consistent, as discussed above.

Quality: The idea of improving cluster discrimination via random negative prototype selection is shown to work clearly, and is simple. Experiments are well done. However, I see several issues as listed in the weaknesses above.

Novelty: Novelty on improving cluster discrimination as mentioned above. The random feature subsampling is also novel, but doesn’t sound well motivated to me.

Reproducibility: The approach is generally simple, I don’t foresee reproducibility issues. The authors also promise to release code and models to ensure reproducibility.
summary_of_the_review: This is a paper with interesting ideas, some of which are simple and shown to work well, for an important problem in computer vision. Experiments are comprehensive. However, I also see several issues at this point, as described in the weaknesses above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper aims to improve the representations for image retrieval with automatic clustering of a large image-text dataset and randomly selecting inter-class prototypes to avoid noisy assignments, and reduction of embedding dimensionality by randomly selecting feature dimensions to be deactivated. Extensive experiments on standard image retrieval datasets, compared with unsupervised and supervised methods, have shown promising results. Overall the intuitive ideas and their rationale are presented well, and shown to work well in benchmarks.
justification_for_why_not_higher_score: relatively simple random sampling schemes that work well but not profound to be highlighted yet
justification_for_why_not_lower_score: good to share with the ICLR community as its implementation is easy and the approach is widely applicable.
Decision: Accept: poster