Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The work follows the GenRep paper and performs representation learning based on a generative model trained on data. They extends the GenRep by switching the fixed latent transformation to a learnable transform. They formulates COP-Gen as an instance-specific latent space navigator which minimizes the mutual information between the generated positive pair, and trains the encoder by maximizing the mutual information between positive pairs. It is clearly written and the theoretical part is correct. The idea is easy to follow and widely used in other frameworks such as GAN model. The performance is improved compared to the GenRep paper, but to be expected.
strengths_and_weaknesses: Strengths: 
- In previous works, people tries to improve the performance of contrastive learning by creating hard negative, while in this paper, the authors uses the InfoMin criterion to create optimal positives. The authors provided solid mathematical results to support the idea, and the proposed method is described and illustrated clearly.

Weaknesses: 
- The performance is highly limited by the trained BigBiGAN model, although in theory with a perfect GAN model we could expect a comparable or even better results compared to SimCLR paper. 

- Minimax training objectives might struggles to converge to the optimal point, but this paper did not touch on this point. It would be helpful if more details about the training process is provided.

- Some non-contrastive paper, such as BYOL and SimSiam should be included in references.

- Minor point: 
  Typo: Border Impact ---> Broader Impact
questions: - Why do you choose 224 as batch size? It might be better to align with other people's work and use 256 instead.

- Could we combine this hard positive technique with non-contrastive learning frameworks, such as BYOL and SimSiam? Are there any experiment results on a small dataset, if the time is limited?

- Why not use a Lagrange multiplier to contain the semantic distance between z and z'?

- Normally training the model in a minimax way takes a longer time (epochs) to converge, or even struggles to converge. Could you provide a training/test loss plot with respect to the epochs?
limitations: The authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes an approach to generate optimal positives for contrastive learning. The positive pairs are generated by finding an instance-specific latent space navigator $T_z$ by minimizing the mutual information between the positive pairs. 
The synthetic data thus generated is used for self-supervised contrastive learning where the proposed approach outperforms the prior work on the Imagenet dataset and on the downstream task of object detection on the Pascal-VOC dataset.
strengths_and_weaknesses: + The paper is easy to follow. The paper is well-written and motivated. The contributions of the work wrt to the prior work and finding instance-specific optimal positives for augmentation in self-supervised contrastive learning is clear. 

+ The experimental results outperform prior work trained on synthetic images.  The experimental results in line with prior work, demonstrate the proposed approach outperforms the prior approaches. The experiments are also performed on VOC object detection to show the applicability of the approach in downstream tasks.

+ Ablations are thorough. The ablations conducted in Tables 2,3 and 4 show the benefits of the data augmentations as well as of the transformation applied in the latent space. 

- There are no major weaknesses, apart from the missing discussions on the choice of certain parameters (see Questions below) and comparison to prior work in terms of computational overhead. 

- The manuscript can be revised to fix the following typos: 
   *Equation 1, function sim not defined.
   *Line 90: "behind by formulating" behind our approach?
   *Line 44: "computational" to computationally.
   *Line 239: "as did in [60, 15, 28]" to as done in.
   "Border Impact" to Broader Impact
  

questions: 1. The choice of a good $\delta$ in equation 5. The experiment section does not discuss the choice of $\delta$ used in the paper. How is it chosen and what impact does training with different $\delta$ have on the experiments in the paper. When increasing the number of samples in Figure 3 for training does a lower $\delta$ mean redundant samples? It would be good to analyze the performance of the approach of different $\delta$ and the different number of samples.

2. In line 39: "in contrast to representation space transformation and non-domain-knowledge-guided data augmentation"; what kind of domain knowledge would be better preserved in this case?

3. How the method compares to prior work in terms of computational efficiency (training time, resources) should be discussed in the main paper. Even though in the checklist, the method is claimed to be computationally expensive. The augmentation with the optimal positives would be useful to the community if it does not add too much computational overhead.

4. The experimental results in table 1 are well within the standard deviation of the prior work. While its understandable that the experiments are time-consuming, the effect of different random seeds should be studied to establish the usefulness and significance of the results. 

limitations: The paper adequately discusses the limitations of the approach and its dependence on pre-trained models. The discussion on how the biases are amplified with the proposed augmentation approach can be discussed in further detail. For example, what happens when increasing the number of positive pairs in case of a biased dataset?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on how to  generate Contrastive Optimal Positives (COP-Gen) for self-supervised contrastive learning and propose a new method based on pretrained generative models. Then, this paper presents theoretical analysis to illustrate that the proposed method can generate positives that removes as much nuisance information as possible while preserving the semantics. Experimental results show the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1. This paper is novelty, and the main idea is interesting.
2. The experimental results is good.

Weaknesses:
1. The technical contribution is incremental.
2. The theoretical analysis is vague. I think the learned transformations can not remove the task-independent semantic information only based on the Definition 2.1 and Proposition 3.1.
3. The compared contrastive learning methods are outdated.
questions: 1. A major problem faced by GANs is the mode collapse problem. This can be understood as learning only part of the semantic information. Therefore, using GAN for contrastive learning may lead to the problem of lack of semantics, so that it cannot better serve downstream tasks. How to solve this problem?
2. I am very confused about Lemma 3.3. If Lemma 3.3. holds, then the learned semantic features are related to the input $Z$. However, $Z$ is a random noise and contains no semantic information.
3. The caption of Figure 2 is puzzling.
4. I hope the authors can provide the code as soon as possible. I want to verify this method in imagenet-1K dataset.
5. How does the method perform on transfer tasks and semi-supervised learning tasks? I hope the author can give the corresponding experimental results.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new framework -  Contrastive Optimal Positives Generation (COP-Gen) for self-supervised contrastive learning which aims to learn instance-specific latent transformations that minimizes the mutual information between the generated positive pair subject to the semantic consistency constraint.
strengths_and_weaknesses: ### Strengths
1.  The paper is well written and easy to follow.

2. The idea of using a pre-trained generative network to obtain the positive samples is novel.

### Weaknesses
My only concern about this paper is in the experimental part.

1. I observed that the authors adopt a resolution of 112x112  images during the pertaining stage, and the reported performance for other methods is much lower than the numbers in the original papers. I think strictly following the setting from the original paper will be a fairer comparison. (e.g. 100 Epoch of SimCLR should reach 62.8% Top-1 Accuracy)

2. Does the quality of the generative model affect the final result?

3. What is the extra training cost if we include an additional generative model?
questions: See weaknesses above
limitations: adequately addressed
review_title: null

Meta-review: metareview: This is a theory-oriented paper for contrastive representation learning. It extends the GenRep by replacing the fixed latent transformation to a learnable transform. The results on ImageNet-1K image classification linear probing and VOC object detection demonstrate the effectiveness of the proposed methods.

The paper receives unanimous accept from all reviewers, leading to an ``Accept'' decision. However, the impact of this paper could be higher (by attracting more attention), if it can compare with absolute SoTA methods in the leaderboard [*] in computer vision community.


[*] https://paperswithcode.com/sota/self-supervised-image-classification-on
confidence: Less certain
Decision: Accept