Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper illustrates method (MAST) on how to transfer data augmentation to a downstream task for Self-Supervised Learning (SSL) methods.

From their paper, their conclusion is that their method can show important role of data augmentation in creating useful invariance priors during SSL.

Here are their contribution:
- introduce MAST to make SSL representations disentangled and uncertainty-aware to
effectively encode different augmentation invariances for good generalization.
•show MAST is efficient, is resistant to feature suppression, and achieves state-of-the-art
downstream performance on diverse vision tasks without presuming any task information
during pre-training.
• provide interesting insights about how different augmentations are related in SSL and
how uncertainty reflects learning difficulty and impacts learning adaptively.

strength_and_weaknesses: Strength:
- Their paper has pointed out the problem of current deep learning field is that current model or data augmentation lacks of generalization.

- Weakness:

In their data augmentation part of 3.2, they set K=5 and K=15. Is there any performance trend of the different number of K?

clarity,_quality,_novelty_and_reproducibility: Their paper is in good clarity and quality.
summary_of_the_review: This paper mainly contributes a MAST by proposing a unified framework to embed all the distinct priors of invariance into
feature representations that can be readily used for different tasks. Generally, it is trying to solve a problem of generalization and downstream task. Generally, their method has shown a good direction on self supervised learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised learning method, Masked Augmentation Subspace Training (MAST), which learns more generalizable representations by modeling augmentation-specific invariance properties. This paper shows that MASK outperforms existing SSL methods under various downstream tasks, including semi-supervised learning, detection/segmentation, and fine-grained classification.

strength_and_weaknesses: Strengths
- This paper aims at learning more generalizable representations, which is an important problem in the SSL literature.
- The proposed method outperforms previous SSL methods under various downstream tasks.

Weaknesses
- Method explanation is unclear: how can Eq (2) learn augmentation-specific masks $m_i$? For example, how to know $m_1$ is related to color jittering, and $m_2$ is related to Gaussian blur, as shown in Figure 3a? Eq (2) does not force any relationship between a mask $m_i$ and an augmentation operation $o_j$.
  - I'm wondering whether MAST uses $K$ forward passes for each training iteration as LooC [1] did. If not, how to learn augmentation-specific embedding spaces? This could be critical because Eq (2) may cause misunderstanding.
- Recent MAE-based approaches do not learn augmentation-invariant representations. What is the advantage of this paper over them? This paper should be compared with the MAE methods.
- The hyperparameter choices of λ = 25d/K, λ1 = 600/(dK) and λ2 = 25 seem to be carefully chosen. So, I think ``work well for all experiments without the need of careful tuning.'' is not true. If the authors want to claim that the sentence is true, they should conduct experiments with varying λ, λ1, and λ2 and also should verify that performance is not sensitive to the hyperparameter choices.
- There exist many fine-grained classification tasks, but Table 4 only provides two benchmarks, CUB and Flowers. More experiments with the tasks should be provided (see SimCLR [2], BYOL [3], or AugSelf [4] papers).
  - This paper should be compared with [4]: [4] achieves 45% top1 accuracy on CUB-200 and 88% on Flowers-102.
- There is no analysis of the correlation between each augmentation-specific embedding space and augmentation-related information (e.g., see Table 1 & Figure 3 in LooC [1] or see Table 8 & Figure 5 in AugSelf [4]).
- There is no analysis of uncertainty. Such an accuracy comparison in Table 5 does not explain how the uncertainty affects learning representations. Also, this paper should provide some empirical evidence verifying that the uncertainty is successfully learned.

[1] Xiao et al., What should not be contrastive in contrastive learning, ICLR 2021 \
[2] Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020 \
[3] Griall et al., Bootstrap Your Own Latent: A New Approach to Self-supervised Learning, 2020 \
[4] Lee et al., Improving Transferability of Representations via Augmentation-Aware Self-Supervision, NeurIPS 2021

clarity,_quality,_novelty_and_reproducibility: The detailed comments are described in the previous section. In summary,
- Clarity :: The method description is unclear.
- Quality :: The empirical results seem strong, but some important comparisons are missing.
- Novelty :: I think the proposed idea is somewhat novel.
- Reproducibility :: All the hyperparameters are well-described.

summary_of_the_review: Although this paper successfully achieves strong empirical results in various downstream tasks, the method descriptions are unclear, and some important comparisons and analyses are missing. I think they are critical, so I vote for rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose learning embedding in augmentation subspaces via masking, which specifies which dimensions are relevant for which augmentation.  In the loss function, the mask is applied to the embeddings before comparison.  Also, the loss function includes a L1 norm term that encourages  masks to be sparse (with fewer features used).   To model uncertainly in the embeddings, the approach learns the mean (replacing the embedding previously) and covariance matrix.  The loss function is normalized by the covariance.   To stablize the learning of the uncertainty model, they use a regularization term to consistency between the Gaussian embeddings before masking via symmetric KL divergence.

They compare with 13 algorithms on ImageNet via linear evaluation and semi-supervised learning.  They also evaluate representations learned on ImageNet for downstream tasks on 5 different datasets via linear classification, object detection, and instance segmentation.  Their results indicate that their approach generally compares favorably.  In their ablation study, they look at the effect of disentangled embedding, scheduling, learned mask, and Gaussian uncertainty.  They analyzed the correlation between masks and add more uncorrelated masks than the more correlated ones.   They found 




strength_and_weaknesses: Strengths:

1.  Learning embeddings in augmentation subspaces via masking is interesting.

2.  Their results indicate that their approach generally compares favorably. 

3.  The paper is generally well written.

Weaknesses:

1.  While performance improves, further discussion/analysis on which subset of the masked embeddings are beneficial to different tasks would be interesting.   
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written.   A minor comment:

Eq 1: Would using Z,Z' be more appropriate than V,V' since I think the loss is with respect to the embeddings, not views?


The proposed approach was compared with a number of algorithms on a few tasks.  

Reproducibility seems ok.

summary_of_the_review: Learning embeddings in augmentation subspaces via masking is interesting.   Their results indicate that their approach generally compares favorably.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose a method that tend to learn task-independent representations. Specifically, the proposed method uses multiple masks to generate different subspace representations to avoid feature suppression. The proposed method also employs the uncertainty modeling technique against variance from strong augmentations. The experiments demonstrate that the proposed method outperforms SOTA baselines in various downstream tasks.
strength_and_weaknesses: Strength:

1. How to learn task-independent representations is a key research problem in SSL.
2. The proposed method shows remarkable improvements in various settings.
3. The paper is well organized and easy to follow. 

Weakness:

1. The proposed method is too complex, with 5 hyper-parameters in the loss function. Although authors claim all hyper-parameters do not need to be careful turn, only results on ImageNet-100/1K are provided. How about on other datasets, e.g., CIFAR-10/100?
2. The proposed method shows significant improvements on ImageNet-1K compared with MoCo V2, over 3%, but on ImageNet-100, the improvements drop to below 1%. Why does it happen? 
3. The authors do not provide the results of the short-term training, such as 200 epochs. Does the proposed method require long-term training to fully explore the strong augmentations? 

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, novelty, and reproducibility are all good for this paper.
summary_of_the_review: Although the proposed method is complex and more experiments should be added, it still shows a new way to learn task-independent representations. I recommend this work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agree that the paper is to mitigate an important research problem that current model or data augmentation lacks of generalization. The idea of learning embeddings in augmentation subspaces via masking is interesting and its effectiveness has also been validated by empirical results.
justification_for_why_not_higher_score: Based on all the reviews, this paper is a solid accept but the contribution and novelty has not met the bar for spotlight.
justification_for_why_not_lower_score: All reviewers agree to accept the paper due to its technical contributions.
Decision: Accept: poster