Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a contrastive learning scheme where not the feature representations of individual samples, but an aggregated feature representation of multiple samples (a set) is considered. The authors claim improvements over a SimCLR baseline on CIFAR, STL, ImageNet-100 and full ImageNet.
strength_and_weaknesses: 
**Major Weaknesses**

- The proposed framework (and claims) are quite general, while the actual experiments only test a very specific subset of "set contrastive learning": The number of features to aggregate is set to $K = 2$, and it is unclear if larger sets actually improve or degrate the performance (Figure 3). I wonder if having a "set" is the crucial detail here, or whether "pairs" of representations are sufficient.
  - In addition, I am unsure if the very general paper title "Set discrimination contrastive learning" is justified. Right now, the authors mainly show improvements when using image pairs rather than sets.
- The experimental validation is quite sparse. The paper is mostly evaluated on small scale data. A full ImageNet experiment with comparisons to multiple baselines is missing. Table 3 should be extended with more self-supervised learning algorithms, including non-contrastive ones, like mentioned in the appendix.
- MoCo is mentioned in the conclusions, but only included in Table 1. Either this claim should be dropped, or Tables 2 and 3 should include MoCo as a comparison method as well. In general, Table 8 should be included in the main paper as it shows the important limitations that other self-supervised learning schemes outperformed set contrastive learning. Also, some methods, e.g. DINO (Caron et al., 2021) are missing which demonstrate accuracies of up to 75.3%. Similarly, training MoCo-v2 longer alo gives another boost to 71.1%, which is also hidden in this table.
- The conclusion mentions increased "robustness": Where is this demonstrated in an experiment?

**Minor Weaknesses**

- The algorithm 1 should be better typeset. Right now there is a lot of confusion between math and pseudocode notation. Either is fine, as long as the syntax is consistent. It might be useful to add comments to the pseudo-code.
- There is no statement on hyperparameter selection.
- The bar plots are highly misleading as the y axis does not start at zero. Figure 3, 4 can be better represented by a lineplot (or a table), ideally with error bars to check which differences are meaningful.
- Even though small scale experiments are used, no error bars / standard deviations are reported.
- The authors claim "signficant" improvements at least in one location in the paper, but do not perform a suitable statistical test to back up this claim. The wording should be adapted (e.g., "considerable") or a test needs to be performed.

**Additional Questions**

- Did you ran your experiments using $K = 1$ to verify the baseline implementation? This setup should exactly match the baseline, correct?
- A number of works learn prototypes for contrastive learning, which effectively summarize multiple features into a single representation. Could you outline why set contrastive learning would yield improvements over such methods? For example, in Table 3 (ImageNet-1K), MoCo-V2 would outperform Miracle (68.6 vs. 71.1).
- Is it possible that the method increases the effective batch size by a factor of 2? This is a sensitive parameter in e.g. SimCLR, I wonder whether this might be a confounder for the experimental results.
- Is there a potential advantage to the method besides a boost in classification performance that I should be aware of and have not considered in my review?
- Did you start from a particular reference implementation of SimCLR? If so, could you state this in the paper/supplement to facilitate reproducibility?


clarity,_quality,_novelty_and_reproducibility: **Clarity/Quality**

- The paper is easy to follow, and contains a lot of numerical results, albeit on small scale datasets. A lot of crucial results to judge the impact/limitations of the method are however hidden in the supplement, e.g. the comparison to other methods on ImageNet-1k where set contrastive learning falls short. 

**Novelty**

- There might be similar methods that aggregate feature representations prior to contrastive learning. Especially the relationship to prototype-based methods (Swav, Moco) could be better discussed, as e.g. a momentum encoder probably has a similar effect on the feature representations.
- Besides this, I believe that the loss can be considered "novel", although in the used form with only two representations it looks more like an incremental improvement to SimCLR.

**Reproducibility**

- The authors do not state if a reference implementation will be publicly available.
- From the description in the paper, it is probably possible to re-implement the method.
summary_of_the_review: 
The method proposed by the paper is interesting, but the execution and empirical validation is insufficient. Specifically, only a very particular (and not very interesting) special case of the set contrastive learning setup is explored, and mostly validated on small scale datasets. The method seems to be effective and improves the baseline, but falls short in comparison with reference results from the literature. The authors should focus to better position their work within the self-supervised learning literature, and also include more commonly used contrastive and non-contrastive models to compare to in Table 7. There is a lot of follow up work on SimCLR (which is the baseline here) and it is very likely that other "additions" to SimCLR have similar effect to the set-contrastive learning scheme proposed here.

Since the learning approach is overall intersting, I am willing to adapt my score based on the discussion with authors and other reviewers if an interesting application/advantage of the method beyond SOTA results can be demonstrated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to introduce the set permutation invariance constraint in training self-supervised learning approaches. The paper also investigates the effect of using different aggregation function that encourages the permutation invariance among images. Empirical results demonstrates certain benefits of the proposed scheme in comparison to the SOTA SSL methods. 
strength_and_weaknesses: Strength:

The paper is well written. The paper proposes to impose the permutation invariance as constraint in training self-supervised learning methods. Inspired by set-based feature learning, the paper propose to aggregate set features from individual sample features by a symmetric function. Empirical results demonstrates superiority of the proposed method. 


Weakness:

W1: M is the number of times the approach shuffle the input. This incurs an issue of fairness. That being said, the proposed miracle method is exposed to the training data which is M-1 times (larger batchsize) more than its baselines. This empirical comparison seems unfair to the baselines. What happened to the conventional methods such as SimCLR if they also see the same amount of training data (with the same batchsize), i.e., also are trained with those additional permutated data (but without permutation invariance)? 

W2: I am also confused why permutation invariance is necessary on traditional image classification task, too. Unlike 3D point applications,  images are independent to each other, I am not convinced why keeping set permutation invariance for images in a set manner should be beneficial for image classification task, since classification prediction is independently deployed on each individual test images. This is counter intuitive to me. Please kindly address this issue during the rebuttal. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The proposal of set contrastive learning is novel and new to me. But I do not see the significance of the proposal on the image classification task, since the permutation invariance is kind of counter intuitive to me. Also see above weakness in terms of this issue.  
summary_of_the_review: Owing to the fairness issue, I am afraid I cannot recommend acceptance at this time. I might increase my score after rebuttal, if the fairness issue and the intuition behind the set invariance constraint can be properly addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Starting from the limitation that unnecessary information could be introduced into the pretext task learning in contrastive learning, this work proposed a method to learn shared features by maximizing set mutual information and circumventing instance discrimination. The set pooling model $g(\phi( ))$ process multiple output of encoder. This function itself has been proved to have permutation invariant properties. This work generates a set of data first and applies the aggregation function to a set of outputs. Since randomly grouping data in a mini-batch to a set will reduce the number of positive set pairs and negative set pairs, the work proposed to permute input multiple times to increase the number of set pairs.      
strength_and_weaknesses: The paper is well-written and easy to follow. The review is clear and the discussion between the existing work and this work is provided. 

One question I have is what’s the difference between permutating $B$ batch $M$ times and letting the model process your data $B*M$ times? If a baseline model runs $M$ times the number of epochs, would the performance increase? I am not convinced about the experimental results by comparing all methods for 200 epochs since this method feeds the data M times compared with other methods. 

Another point is that the theory behind the work is limited. Theorem 1 in the paper does not provide enough insight to explain the proposed method (basically a standard way to show the upper bound, but no explanation of the usage).
clarity,_quality,_novelty_and_reproducibility: Clear, and well-written. Though the idea comes from previous work such as the pooling model, it seems not sufficiently novel in theory, but it improves the performance in practice. The authors didn’t mention code releasing, however, the Algorithm is well explained. 
summary_of_the_review: The lack of theory and the unclear comparison makes me not quite sure about the real contribution of the paper. This is a good application of some existing methods though.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new way to do contrastive learning (CL). Unlike prior CL methods that focus on instance-level representation, this work contrast sets of representation. Experiments of CIFAR-10/-100 and Imagenet-100 show the proposed method's effectiveness. 
strength_and_weaknesses: Pros:

1. The paper is easy to follow. The paper has nice figures and beautiful tables.

2. Author provides a nice ablation of the proposed method. 

Cons:

1. Theoretical interpretation is vague. 
- 1(a) Theorem 1: as shown in the proof, theorem 1 basically says "InfoNCE(Z,Z') <= I(Z; (X1,X2))". The theorem is very general and not specific to the proposed algorithm. For example, the theorem holds no matter whether data augmentation is used or not. However, data augmentation in CL contributes a lot to the success of CL
- 1(b) The author use theorem 1 to argue that "If the encoder learns features that are specific to only one instance of the set, it does not increase the mutual information with any of the other instances of that set." Such an argument is not precise and clear. First, there is no such thing called "mutual information with instance". Mutual information is defined between random variables, not instances. Second, by symmetry, it is easy to see that I(Z; X1) = I(Z; X2) since X1 and X2 are two independent random samples from data. Thus there is no such case where I(Z; X1) is high while I(Z; X2) is low. 
- 1(c) hard negatives mining: The logic in the paragraph "Set discrimination from hard negatives perspective." does not make sense to me. I do not see the definition of "hard negative" and do not see why "set construction" is a strategy to produce hard negatives. 

2. Questions on experiments
- 2(a) Uncommon evaluation metrics: Most of the contrastive learning paper report "linear probing" results, e.g., the baselines used in this work, SimCLR, MoCoV2, DCL, FlatNCE. I suggest the author use linear probing instead of KNN as their main evaluation metric for CIFAR-10/-100, and STL-10 experiments. 
- 2(b) Number inconsistency from prior works: I find the baseline results reported by the author are not consistent with that reported in the original paper. In this paper's table 1, CIFAR-10, batchsize=256, the KNN accuracy of DCL is 82.7. However, in the original DCL[1] paper, the accuracy is 84.2 (Table 2).

[1] Yeh, Chun-Hsiao, et al. "Decoupled contrastive learning." arXiv preprint arXiv:2110.06848 (2021).
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and has nice clarity. The work has nice originality. It is interesting to see that set-based representation learning is more effective than instance-level representation learning in the context of contrastive learning. 

However, the underlying mechanism of why it works remains unknown. The author tries to provide some explanation through theoretical analysis. But I am afraid that the current theoretical analysis is superficial and not satisfying. Together with the issue of the experiment section mentioned above, I find the paper's quality needs to be improved. 
summary_of_the_review: Overall, considering the paper's originality and quality, I think the current version of the paper does not reach the bar of being published in ICLR. However, I will reconsider my score if all my concerns get addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a contrastive self-supervised learning framework based on the set encoding of the positive and negative. Specifically, the proposed framework performs contrastive learning using the set encodings, where the positive sets are constructed by applying different augmentations to the same set of instances, while the negative sets are constructed from different instances. The authors theoretically show that set-discrimination prioritize common features across instances over low-level features from each instance, and experimentally validate the proposed method method on multiple benchmark datasets, against multiple contrastive learning baselines, which it outperforms. 

The reviewers considered the idea of introducing the set encoding to self-supervised learning as novel, the idea of generating different permutations and enforcing permutation invariance to the different set encodings as reasonable, and the paper well-written.

However, at the same time, the reviewers had concerns about weak experimental validation, as the experiments were done mostly with small set size (K=2), on small-scale datasets, against only a few baselines, missing out most state-of-the-art self-supervised learning methods. The reviewers were also concerned about the lack of analysis on how the proposed set permutation helps with the generalization. There were also concerns regarding unfair comparison as the reviewers misunderstood that the set permutation requires more computations and exposes the model to a larger amount of data.

During the discussion period the misunderstandings on the fairness of the experimental validation were cleared away. However, the reviewers kept their unanimous weak reject ratings even after the discussion period, since they had remaining concerns on unclear motivation as to how the set-encoding helps with generalization, and weak experimental validation. The authors are advised to focus on addressing these two critical weaknesses in their revision. 
justification_for_why_not_higher_score: The paper needs to undergo another round of major revision - the current sets of analyses and experimental results do not explain how the method works and are insufficient for verifying whether the proposed method is effective or not.
Decision: Reject