Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper empirically claims that the overlap among the augmentation distributions from a similar category is much higher than those from dissimilar categories. Based on this discovery, this work establishes the semantic relationship between samples in an unsupervised manner based on the similarity of augmentation distributions. Technically, the paper proposes a new self-supervised loss function via maximizing the similarity of samples from the same augmentation distribution and making those from different augmentation distribution as orthogonal as possible for further enhancing the representation learning. Theoretically, this work is intended to explain why contrastive learning can effectively work with its proposed Augmentation Component Analysis.
strength_and_weaknesses: Strengths:
+	The perspective of augmentation overlaps is interesting for contrastive learning. Especially for those samples from the same category, the discovery of augmentation overlaps is much useful for representation learning in an unsupervised way. 
+	The loss functions proposed in this paper are verified their effectiveness for self-supervised representation learning. Especially, a projection loss is designed for compacting the representation from the same samples, which shares similar sprit with the prototype learning in supervised learning. The difference lies in the projection loss is mainly for the augmentation distribution of a single sample.

Weaknesses:
-	In this exploration of augmentation overlaps, the similarity between different samples don’t seem to be explained very well in this paper. And, there also seems to have no corresponding loss function or theoretic derivation can prove the model could pull the semantically similar samples close, as showed in the left of Figure 1.
-	The analysis of dimension reduction with the idea of PCA is not suitable to derive the similarity constraint in Eq. (4). Besides, the ACA-PC loss is similar to the function appeared in [1].
-	Performance gain of ACA-PC can be very marginal to prove the effectiveness of the proposed ACA. 
-	There exist many sentences in this paper that are hard to digest. E.g., “we claim that it is the similarity between the augmentation distributions of samples, …, that reveals the sample-wise similarity better” (page 1), “It seems that we can obtain the solution directly without further learning, however, not only the elements in the augmentation feature are hard to calculate, but also such high-dimensional target embeddings are impractical to use” (page 1),, “In addition, the resemblance between the objectives of ACA and traditional contrastive loss may explain why the latter can learn semantic-related embeddings” (page 2).

[1] HaoChen J Z, Wei C, Gaidon A, et al. Provable guarantees for self-supervised deep learning with spectral contrastive loss[J]. Advances in Neural Information Processing Systems, 2021, 34: 5000-5011.

clarity,_quality,_novelty_and_reproducibility: The novelty of this paper can be sufficient and it can be easy to reimplement the proposed ACA.
summary_of_the_review: The paper need further clarify the mentioned items before it is accepted for publishing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Augmentation Component Analysis (ACA), which employs the idea of PCA to perform dimension reduction on augmentation features. ACA reformulates the steps of extracting principal components of the augmentation features with a contrastive-like loss. With the learned principal components, another on-the-fly loss embeds samples effectively. ACA learns operable low-dimensional embeddings theoretically preserving the augmentation distribution distances.
strength_and_weaknesses: **Strength** 
1. The idea of augmentation component analysis is novel and interesting.
2. The theoretical analysis of this work is promising.

**Weaknesses** 
The experimental results are too weak.

1. Self-supervised learning aims to transfer the learned representations or whole network parameters into various downstream tasks. However, I do not see any transfer learning experiments in this paper. Could you provide more transfer learning experiments, for example, linear evaluation and fine-tuning in fine-grained classification tasks, semi-supervised learning, and object detection/segmentation?

2. The improvements of this method are very marginal. From Table 1 and Table 2, ACA-Full only surpasses the second-best performance 0.5% in most cases, which needs to be more convincing.

3. The comparison methods in Table 1 and Table 2 are outdated. I highly recommend the author compare ACA-Full with the latest contrastive learning methods. For example, [1][2][3][4][5][6][7]. Moreover, the convergence rate and final accuracy highly depend on the methods. For ImageNet experiments, the authors should at least train the model for 200 epochs (or even longer) to make sure all methods are fully converged.

[1] With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
[2] Solving Inefficiency of Self-supervised Representation Learning
[3] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
[4] Mean Shift for Self-Supervised Learning
[5] Ressl: relational self-supervised learning with weak augmentation
[6] Barlow Twins: Self-Supervised Learning via Redundancy Reduction
[7] AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations From Self-Trained Negative Adversaries


clarity,_quality,_novelty_and_reproducibility: The code and pre-trained models are desired. After reading this paper, it is not easy for me to reproduce the results. I acknowledge the novelty of this paper, but the poor experimental results need to be more convincing.
summary_of_the_review: See Strength and Weaknesses above
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new method for self-supervised representation learning based on data augmentation. Unlike prior work in this line of research, the proposed method considers semantic similarity (e.g., class equivalence) between different instances as well as semantic invariance between different views of the same instance. The key idea is that if two training instances are semantically close to each other, their augmented versions (i.e., different views) will be largely overlapped. Conceptually, the idea can be implemented by comparing posterior probabilities of augmented examples given original ones, called augmentation features, which however is computationally intractable due to the prohibitively large number of potential augmented examples and the difficulty of estimating such probabilities. The authors thus propose to simulate principal component analysis (PCA) of augmentation features through a neural network encoder trained by two loss functions; they also prove that the loss functions allow the encoder to produce PCA projection results without computing the augmentation features explicitly. The proposed method outperforms prior work in the same direction and provides theoretical interpretation of contrastive methods for self-supervised representation learning. 
strength_and_weaknesses: [Strengths]
- The main idea about relations between semantic similarity and similarity between augmentation distributions is new in self-supervised representation learning and sounds reasonable. The idea also has been justified empirically. 
- The way of simulating the direct yet impractical implementation of the idea using a neural network and two loss functions is interesting, solid (its validity is proven in the appendix), and worth to be introduced to the community. 
- The proposed method achieved best scores on multiple benchmarks for self-supervised representation learning. 


[Weaknesses]
- Clarity issues: (1) The third paragraph of Section 1 is hard to grasp; it will be useful if a figure that conceptually illustrates the main idea is added. (2) The notion of augmented sample x is not clear and seems not consistent in Section 3 and 4. In Section 3, x is introduced as if it is a new view of a natural image (i.e., an augmented version of the natural image), but in Section 4 it is coupled with all natural images to form posterior probabilities. 
- Lack of experiments: The proposed method needs to be compared with latest approaches to self-supervised representation learning. Also, I would recommend evaluating the proposed method on other epochs (e.g., 200, 400, 800) like prior work for the ImageNet linear classification benchmark. Moreover, advantages of the learned model have to be also validated for downstream tasks other than image classification, i.e., transfer learning setting for object detection and instance segmentation.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The manuscript has some clarity issues as commented above, but overall it is written clearly. 
- Reproducibility: The paper elaborates on technical details for reproducing the reported results.
- Novelty: The main idea of this paper is new and validated both theoretically and empirically. 

Overall, I believe the paper meets the standard of ICLR in terms of quality. 

summary_of_the_review: The paper presents a new idea for self-supervised representation learning, which enables us to consider both semantic similarity between different instances and semantic invariance between different views of the same instance. The implementation of the idea is solid theoretically and achieves outstanding performance on multiple public benchmarks. These positive points outweigh my concerns on lack of experiments, and thus I believe the paper is worthy to be introduced to the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presented a self-supervised learning method based on the augmentation distribution of the image. Specifically, the paper is motivated by the intuition that semantically similar images tend to have similar augmented views, and proposed a method called ACA to learn the augmentation principal components of data. The method is theoretically deduced to maintain and measure the discrepancy between the augmentation distribution across samples. The method is validated on many benchmarks (cifar, stl, imagenet), and shows competitive performance when compared with several existing models.
strength_and_weaknesses: Strength
- The proposed method is theoretically sound and complete. The deduced loss function seems new and interesting.
- The paper provided interesting pilot study and also interesting small experiments for motivation purposes.
- The evaluation experiments are in general complete and nicely conducted. The method is validated on many datasets, with sufficient benchmarking.


Weaknesses
1. Major concern: model performance
- The model performance on several datasets seems to be lower than previously reported results. Specifically, check [1, 2] where many numbers are greater than the presented method. Unfortunately, as the major experimental result of the presented method is framed to be about surpassing previous state-of-the-art (that is the only experiment), the presented numbers are just not satisfactory enough, which greatly weakens the paper’s contribution.

2. The related works and the references lack more recent works.
- There are many contrastive learning/self-supervised learning methods in 2021 and 2022 that are not included in related works. For example, check [2-5].

3. The writing and organization of the paper could be improved.
- The abstract could be improved. Specifically, I suggest rewording the first sentence and the 4th sentence.
- Section 4.1 and section 4.2 are a bit wordy and might be simplified. The computation overload is mentioned many times in two adjacent sections.
- Some of the preliminary in section 3 could be moved into related works, especially the second paragraph, which has much overlapping information with section 2.


4. Minor comments:
- I suggest rewording Section 3 first sentence.
- Related works the Self-Supervised learning section should mention non-contrastive methods as well, like BYOL. Instead, the paper put non-contrastive methods inside the contrastive learning section and implicitly infer them as Contrastive Learning methods without using negative samples, which is a bit controversial because BYOL is not always referred to as contrastive methods though some studies suggest that they intrinsically are.
- What is the connection between this method and Barlow Twins?
- It might be more beneficial if the authors could explore other use cases of the presented learning method rather than just performing classification. For example, if the proposed loss function could provide an “augmentation spectrum” of certain data, could that provide more informative latent space based on some other metrics when compared to other methods?


[1] Ermolov, Aleksandr, et al. "Whitening for self-supervised representation learning." International Conference on Machine Learning. PMLR, 2021.

[2] Azabou, Mehdi, et al. "Mine your own view: Self-supervised learning through across-sample prediction." arXiv preprint arXiv:2102.10106 (2021).

[3] Zbontar, Jure, et al. "Barlow twins: Self-supervised learning via redundancy reduction." International Conference on Machine Learning. PMLR, 2021.

[4] Tsai, Yao-Hung Hubert, et al. "A note on connecting barlow twins with negative-sample-free contrastive learning." arXiv preprint arXiv:2104.13712 (2021).

[5] Kalantidis, Yannis, et al. "Hard negative mixing for contrastive learning." Advances in Neural Information Processing Systems 33 (2020): 21798-21809.

clarity,_quality,_novelty_and_reproducibility: The clarity is okay, there does exist room for improvement.

The quality is above average.

The novelty is okay, there do exist novel theoretical contributions, but neither the motivation nor the presented method is super novel.

summary_of_the_review: I think this paper is borderline and I lean toward borderline rejection. The paper is based on valid motivation, the method seems to be relatively novel and sound, and the experiments are in general nicely organized. However, the paper lacks sufficient experimental results that support the superiority of the presented method, as the majority of the reported performance is lower than previously reported results. Unfortunately, this is the majority component of the experiments section, and as the paper is framed to present a method that could surpass/ provide comparable performance w.r.t existing methods, the experimental results outplay the methodology contribution imho.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose a novel self-supervised learning method that makes the distance in the embedding space of samples proportional to the distance of the augmentation distribution, starting from the assumption that semantically similar samples have similar sets of views/augmentations. All four reviewers agreed that the idea is interesting and novel and the proposed method is theoretically sound and complete. In the original review, there were concerns about presentation clarity and limited experimentations, but through the rebuttal period, the authors successfully addressed these concerns, and there was no disagreement in accepting the paper in the discussion.
justification_for_why_not_higher_score: I'm not against spotlight. However, considering the review scores and the the quality of the submission, I think the poster is appropriate.
justification_for_why_not_lower_score: all reviewers are positive and I also agree with reviewers assessments.
Decision: Accept: poster