Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new self-supervised learning objective that uses a notation of multi-dimensional Hirschfeld-Gebelein-Rényi (HGR) maximal correlation as the similarity measure for different views of the same sample. The paper gives some theoretical insights and experimental study on some relatively small datasets.
strength_and_weaknesses: For the strength:

- To the best of my knowledge, the idea seems to be new (although it sitll falls into the large category of non-contrastive learning, e.g., BYOL). The introduction of adversarial training seems interesting as well.

- The formulation is interesting, especially considering the similarity / correlation measure is a variational formulation which involves a constrained maximization as an inner optimization. 

- The experiments show some improvement on a number of tasks including tabular data, vision data and audio data.


For the weakness:

- The novelty is incremental given that both the introduction of multi-dimensional Hirschfeld-Gebelein-Rényi (HGR) maximal correlation and adversarial traininng can not be well justified (both theoretically and empirically). It seems to be the idea is directly connected to Barlo Twins [https://arxiv.org/pdf/2103.03230.pdf] since the final objective is essentially similar. I think a proper comparison to Barlo Twins is necessary.

- The theoreical part does not inform why the proposed method is better than the others (say cross-entropy).

- Althoguh the paper presents a number of tasks, they are a bit toy-ish. It could much convincing if the authors can conduct some experiments on larger datasets (say ImageNet).

clarity,_quality,_novelty_and_reproducibility: - For the clarity, the paper is generally well-written with clear structures. It is quite easy to read.

- For the originality, it is a bit incremental in the sense that it simply replaces the widely used correlation / similarity measure (say cross-entropy) with another one. The introduction of adversarial training seems new, but its effectiveness needs more ablation study to justify.


summary_of_the_review: I generally enjoy reading the paper and find the idea an interesting one. In terms of technical novelty, it is a bit incremental since the introduction of the new correlation measure is not well justified, as well as the adversarial training. In terms of empirical significance, I find it hard to evaluate based on the current toy-ish experiments.

My overall concerns are listed as follows:

- Since the final objective is a lagrangian relaxation, whether the final objective can well approximate the HGR correlation is unclear. I think the natural to evaluate this is to compute the individual loss terms in the objective and see how each term is optimized. For example, if the network output indeed tends to be element-wise independent, then the constraint for HGR correlation can be satisfied.

- For the novelty, as I mentioned above, it is a bit incremental. Just to add more details, I think the underlying motivation of the paper is very similar to Barlo Twins, but differently, the paper introduces a stronger regularization for the encoder network, ie, the network output is element-wise independent, which is essentially requiring the network to conduct something conceptually similar to nonlinear ICA. This is already a difficult task, and I don't understand why this will be beneficial for general data.

- For the experiments, I highly suggest the authors to conduct experiments on ImageNet and compare the results to Barlo Twins. Ideally, you can directly follow the settings of Barlo Twons, which can serve as a fair comparison.

Post-rebuttal:

Based on the comparison to Barlo Twins, the gain can be barely observed. I can see there are differences to Barlo Twins, but the authors do not really justify why these differences are generally beneficial to generalizability, and most importantly, the empirical evidence does not support the authors' argument. Given the above reason, I am still leaning towards rejection for the time being.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a domain-agnostic method to learn augmentations in self-supervised learning.
strength_and_weaknesses: Strength:
1. The paper proposes a new paradigm other than contrastive learning for self-supervised learning.
2. The proposed algorithm is theoretically justified.
3. The paper is written and easy to follow in general.

Weakness:
1. The algorithm still requires domain-dependent hyperparameters (e.g., epsilon and L_p norm). I wonder if there is a way to automate the choices from the data which makes the whole algorithm agnostic.
2. The attack model is limited to additive attacks. However, some natural augmentation may require a very epsilon to realize using an additive model. For example, suppose a model would like to learn a translation-invariant representation (a shift in image/audio leads to the same semantic). In that case, a proper augmentation may be shifting or cropping if it is domain-specific, which is hard to realize by an additive model.
3. The theory seems a bit useless to me. The error bound depends on some factors that are not measurable in practice (e.g., alpha and lambda). I wonder if it is possible to obtain some numerical bound that measures whether the learned model succeeds or not (before evaluating it on test dataset).
clarity,_quality,_novelty_and_reproducibility: The paper is original as far as I know and clearly written. 
summary_of_the_review: The paper proposed a simple and novel paradigm for self-supervised learning. Unlike contrastive learning, the proposed method can learn augmentation without domain knowledge. However, the paradigm still suffers from some limitations, as detailed in Weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for self-supervised learning based on adversarial sample generation. The proposed method is based on the probabilistic interpretation of PLS and/or CCA based on graphical models and does not require class labels or classifiers. It performs best among the domain-agnostic methods on tabular, image, and audio datasets. They also provide theoretical discussions as well as experimental comparisons.
strength_and_weaknesses: Strengths
1. The proposed method is a simple but straightforward design based on probabilistic CCA and PLS and performs better than related methods.
1. Experiments have been conducted on data sets from multiple modalities, which is sufficient to demonstrate versatility.
1. Rather than ending with a heuristic proposal of a method, there is also a theoretical discussion.

Weaknesses
1. The abstract and introduction also mention data augmentation, but the experiments are limited to comparisons among self-supervised learning methods. For example, in the case of images, Cutmix, RandAugment, and Random Erasing are used in addition to Mixup, but no comparison with such data expansion methods is made. The reviewer agrees that there is no de-facto standard data augmentation for tabular data, but some comparisons using image and audio datasets would be possible.
1. Approaches that maximize correlation, such as CCA and PLS, are also reasonable, while recent trends, such as (Verma et al., 2021) and (Ho and Nvasconcelos, 2020), use contrast learning. A contrastive loss with original features x, x+δ, and features x' with different semantics seems natural. The reviewer would like to know if there is any motivation for the proposed method not to employ contrastive learning.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality
- Clarity and quality are adequate. There is no description of ζ around Eq. (7). Is there any relationship between the given bounds and ζ?

Novelty
- Modules to generate perturbed examples and train encoders under CCA/PLS setting are not novel. Their combination of self-supervised learning and theoretical discussion would be new.

Reproducibility
- How is the hyperparameter γ tuned? It appears to be an important parameter that controls whether it is CCA-like or PLS-like, but there seem to be no sensitivity experiments.
- The experimental setting is unclear. For example, the network architecture is described by citing Nakkal & Salzmann (2021), but the authors should try to keep the description self-contained for reproducibility.
summary_of_the_review: Overall, the reviewers are inclined to judge this paper as acceptable. The score will be improved more if the above weaknesses and questions are adequately answered.

----

**Updated review**

Responses from the authors and additional experiments were sufficient to answer the reviewers' questions, thus improving the score. If there is room in the paper, the following responses from the authors could be added, even in summary.

> The fundamental motivation behind contrastive learning is maximizing the mutual information between two augmented views (Oord et al., 2018). However, it is inconsistent with the actual behavior in practice, e.g., optimizing a tighter bound on mutual information can lead to worse representations (Tschannen et al., 2019). Theoretical understanding of contrastive loss is still elusive. Analysis based on the assumption of latent classes provides nice theoretical insights (Saunshi et al., 2019), but the theoretical result that representation quality suffers with more negative samples is inconsistent with empirical results in SimCLR and popular contrastive learning methods. A recent study interprets contrastive loss as alignment and uniformity on the hypersphere (Wang et al, 2020), but they do not provide any theoretical guarantees on the downstream performance of the learned representation.

> For the reasons presented above, we decide not to employ contrastive learning and develop a self-supervised learning method based on well-known statistical learning methods such as CCA and PLS. Our goal is not to beat SOTA performance. We want to give some insights into why training models with unlabeled data using certain loss functions leads to useful representations in the downstream task. We believe such insights are very important for the development of unsupervised representation learning in the long term.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a domain-agnostic self-supervised learning algorithm using adversarial perturbations in the latent space. The paper argues that existing self-supervised learning algorithms either have domain-dependent pretext tasks or have domain dependent augmentations. For instance, the masked image/language models approach requires data to be tokenized while the contrastive learning approaches require domain-dependent augmentation. Thus, to propose a domain-agnostic approach, the authors devise a pre-text task where the objective is to reconstruct the latent representation of a clean sample from the perturbed sample. For perturbation, instead of using an augmentation the paper rather uses adversarial perturbation, where the perturbation is produced by a generator. The proposed algorithm iteratively switches between learning a generator to produce the adversarial perturbation and learning the encoder to predict the latent representation of the clean sample from the adversarially perturbed sample. To avoid learning the trivial solution by the encoder, latent reconstruction is done with an orthogonality constraint where the encoders project all input data into a constant vector. To show the generality of their approach, the authors present results over tabular data, audio data, and image data.



strength_and_weaknesses: Strength - 

1. The paper is generally easy to follow along and well-motivated. 
2. The approach also show theoretical guarantees on the linear probe error for their approach.
3. Empirically the results are good across different domains and datasets.

Weakness - 

1. I feel the paper's approach is very similar to BYOL, which also has a similar pretext task of reconstructing the latent representation of one augmented sample from another augmented image. The main difference is that BYOL uses domain augmentations while the proposed approach uses adversarial perturbation. However, there is no comparison with BYOL, at least for the image data. Further to avoid learning a trivial solution BYOL uses a predictor network on top of only one encoder(and hence asymmetric). In contrast, the paper injects an orthogonality constraint on the latent reconstructions.  How do these two approaches differ? Can authors do an ablation study between these two approaches? In particular, why would one want to use the orthogonal projection constraint as mentioned in the paper over the asymmetric networks approach as done in BYOL?

2. For learning the adversarial perturbation, the paper learn a generator to produce the perturbation. However, we could also instead just use the gradient of L_adv to compute the perturbation direction, similar to single-step PGD attack[1] done for adversarial attacks. This is much simpler than learning a generator. Can the authors comment on this, and show some ablation to validate that learning a generator is indeed better?

3. The proposed approach is still not domain agnostic as it cannot be applied over the text domain as it would be hard to define a perturbation over the input. This questions the domain-agnostic claim made in the paper. Masked learning approaches in contrast can be applied over both text and image data[2]. Can the authors comment on this?

References - 
[1] Towards Deep Learning Models Resistant to Adversarial Attacks. Madry et al.
[2] Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks. Wang et al.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow along. The idea proposed is slightly novel. 
summary_of_the_review: While I like the direction followed in the paper about proposing a domain-agnostic self-supervised learning approach I have concerns about the experimental setup, particularly proper comparisons with BYOL and also the method for doing adversarial perturbation. Furthermore, the proposed approach is still difficult to apply over text-domain and hence it is not domain-agnostic.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new self-supervised learning objective and its application to domain-agnostic SSL by using adversarial view generation. The proposed method is grounded by HGR maximal correlation, which interpolates between CCA and PLS. Then, they propose a novel SSL objective that has a theoretical guarantee for downstream generalization. Together with adversarial view generation by training auxiliary generator, they empirically validate the effectiveness of the proposed method on various domain-agnostic SSL benchmarks on vision, tabular data, and speech datasets. 

Strength:
- This paper proposes a novel non-contrastive self-supervised learning framework derived from maximal HGR correlation appended with theoretical support. (All reviewers agreed)
- Marginal improvement in domain-agnostic self-supervised learning baselines on tabular data, vision, and audio datasets. (All reviewers agreed)

Weakness: 
- Missing the relationship/comparison between the proposed SSL objective and existing SSL objectives. See the below comment. (Reviewer 3qmD, Hv4e, RgE2)
- The theoretical analysis seems vacuous (Reviewer 3qmD, JiWn)
- Lack of large-scale experiments, e.g., ImageNet (Reviewer 3qmD) 

justification_for_why_not_higher_score: While the motivation of self-supervised learning inspired by HGR maximal correlation is new and supported by theoretical guarantees, the contribution seems only incremental to the existing self-supervised learning literature. The Lagrangian-driven final objective resembles existing non-contrastive objectives, e.g., Barlow Twins [1] and Spectral Contrastive Loss [2]. In particular, one can easily check that the proposed objective is simply the Generalized Spectral Contrastive Loss [3] with EMA on the alignment term. Furthermore, the theoretical analysis of the guarantees for downstream generalization is quite similar to that of [2] (e.g., Assumption 4.1 & 4.2 to Assumption 3.6 & 3.7 in [2]). However, the paper does not contain such a comparison between other contrastive objectives, especially missing the comparison with spectral contrastive loss, which seems the most relevant reference. Also, as Reviewer 3qmD mentioned, the proposed objective does not show significant gain over other SSL methods, e.g., Barlow Twins, and the theoretical guarantee does not guarantee better generalizability. 

The paper has demonstrated the empirical advantage of the proposed method in various domain-agnostic self-supervised learning tasks, however, the empirical gain is not significant or the proposed baselines are not enough to claim the superiority of the proposed method. 
For example, as Reviewer 3qmD mentioned, the paper considers only small-scale dataset while most of the self-supervised literature consider large-scale ImageNet dataset for evaluation. 

Moreover, there are some missing baselines in domain-agnostic SSL. For example, I-MIX [4] is a mixup-based contrastive learning method that has shown its effectiveness in domain-agnostic SSL on vision, tabular, and speech datasets. Furthermore, MAE [5] is also a popular self-supervised learning method that can be used in a domain-agnostic way as illustrated in [6, 7]. 

Overall, the paper proposed an interesting direction for self-supervised learning objectives and demonstrated its effectiveness on domain-agnostic self-supervised learning tasks, but AC thinks the limited technical contribution and empirical findings are not significant enough for publication. 

[1] Zbontar, Jure, et al. "Barlow twins: Self-supervised learning via redundancy reduction." International Conference on Machine Learning. PMLR, 2021.

[2] HaoChen, Jeff Z., et al. "Provable guarantees for self-supervised deep learning with spectral contrastive loss." Advances in Neural Information Processing Systems 34 (2021): 5000-5011.

[3] HaoChen, Jeff Z., et al. "Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations." arXiv preprint arXiv:2204.02683 (2022).

[4] Lee, Kibok, et al. "i-mix: A domain-agnostic strategy for contrastive representation learning." arXiv preprint arXiv:2010.08887 (2020)

[5] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[6] Tamkin, Alex, et al. "Dabs: A domain-agnostic benchmark for self-supervised learning." arXiv preprint arXiv:2111.12062 (2021).

[7] Tamkin, Alex, et al. "DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision." Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022.
Decision: Reject