Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper suggests dynamic adversarial contrastive learning which gradually anneals from a strong augmentation to a weak augmentation. Further, the authors propose fast post-processing stage for adapting it to classification tasks which boost the robustness. From this simple and effective strategy, DynACL reduces the gap between supervised AT and selfsup AT. 
strength_and_weaknesses: **Strength**

- This paper proposes a simple and effective scheduling method for selfsup-AT which could surprisingly improve the robustness.
- The paper is well-written and shows convincing motivations.
- This paper suggests novel aspects of the relationship between augmentation and adversarial training in selfsup-AT.
- This paper demonstrates extensive experimental results including diverse ablation experiments and semi-supervised settings.

**Concerns and Questions**
- I hope the authors describe the difference between the DynACL++ and AdvCL post-processing which pseudo labels the unlabeled examples with k mean clustering. I think it is overemphasized claims that fast post-processing is a novel idea.
- Does proposed technique also can be adapted to RoCL? And also boost the performance of this method too?
- Does dynamic scheduling also benefit the performance of vanilla simCLR?
- The proposed dynamic scheduling which gradually decreases the strength of the augmentation seems a somewhat empirical approach seems bit simple approach without any convincing motivation or evidence. It is understandable that we should control the strength of augmentation but I am not sure proposed method is the most effective design of data augmentation. Because if there is some standard to find the adequate strength of the augmentation at a certain training stage of self-AT (i.e., classwise distance, or MMD), an adaptive strength controller could show better robustness than the gradual scheduling.
- I am not quite sure this approach could be generalized to different architectures, or different frameworks. Since the approach is a naive approach, hyperparameter search may be always needed for optimal performance. And the current gradual scheduling could not be the best option for some circumstances (where may need a longer step size). I think adaptive data augmentation scheduling [1,2] could also be applied to self-AT which could improve the current approach more intuitively and make the approach to be more universal to diverse circumstances.

[1] ADAAUG: LEARNING CLASS- AND INSTANCE- ADAPTIVE DATA AUGMENTATION POLICIES, ICLR 2022

[2] MetaAugment: Sample-Aware Data Augmentation Policy Learning, AAAI 2021
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is easy to understand and well-written.

**Quality:** The presentation of the paper is good and well-organized.

**Novelty:** The dynamic gradual design of the data augmentation seems not very novel to me but it is simple and effective.

**Reproducibility:** The paper has well reproducibility which elaborates well on the details.
summary_of_the_review: Overall, I recommend acceptance to this paper. This is well written and has intuitive motivation with empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first reveals a dilemma about the augmentation strength that either strong or weak data augmentations are harmful to self-supervised adversarial training (self-AT). To resolve the dilemma,  the paper proposes a simple remedy named DynACL (Dynamic Adversarial Contrastive Learning). DynACL adopts a dynamic augmentation schedule along the training process that gradually anneals the strength from strong to weak, then adopts a fast post-processing stage for adapting it to downstream tasks. 
strength_and_weaknesses: Strength:

a)	The paper reveals the reason behind the robustness gap between self-AT and sup-AT and investigates the effect of data augmentation strategy on self-AT.

b)	The proposed DynACL shows a significant improvement in clean accuracy and robustness over existing self-AT methods.

c)	The DynACL is also more computationally efficient.

d)	This paper is well written.

Weaknesses:

a)	In Table 1, the standard accuracy (SA%) of DynACL on STL-10 is significantly lower than other self-AT methods, which is not explained in the paper.

b)	The metric RA(%) in Table 2 is not explained.

clarity,_quality,_novelty_and_reproducibility: The paper is well organized, but the presentation has minor details that could be improved. 
The technicality of the paper appears to be sound, and the theoretical knowledge is explicit.
The idea to propose a dynamic augmentation strategy along the training to improve the robustness of self-AT is novel.
The code is not available, but the experimental details are sufficient. It doesn't seem difficult to reproduce.

summary_of_the_review: The paper is well written, the contribution is clearly presented and the experimental results are thoroughly executed. I would tend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the effects of data augmentation in adversarial contrastive learning (ACL). The authors show that stronger data augmentation can result in a larger distribution gap between training data and test data, and a smaller classwise distance. Therefore, the authors propose a piecewise decay augmentation schedule for ACL, namely DynACL. The empirical results validate that DynACL can achieve comparable and even better adversarial robustness than supervised adversarial training (Madry et al., Towards deep learning models resistant to adversarial attacks, 2017).
strength_and_weaknesses: Strength
+ This paper provides an interesting and insightful investigation of the effects of data augmentation in ACL. The empirical investigation validates that stronger data augmentation can be harmful to ACL. 
+ The proposed method is compatible with variants of ACL method and has no extra computational consumption. The experiments validate DynACL can significantly improve adversarial robustness compared to previous self-supervised adversarial training methods.

Weaknesses
- The investigation totally depends on the empirical phenomenon. Say the classwise distance. The paper only shows the phenomenon. But it seems that the paper does not provide more (theoretically) analyses of the reason for this phenomenon.  
- The proposed dynamic scheduling is somewhat naïve. And, it is strange that the training procedure still uses the most aggressive data augmentation at the final stage, which has been pointed out as harmful to ACL.


clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and written. I can easily follow most of the parts. The novelty of the proposed method seems to be fair. This paper has good reproducibility since the authors have provided the algorithm in the appendix and illustrated the hyper-parameters in detail.
summary_of_the_review: Overall, the empirical results solidly support the effectiveness of the proposed method. However, the analysis of some investigation (e.g., why a more aggressive data augmentation can incur a smaller classwise distance?) is not very clear. Therefore, I tend to borderline accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studied the strength of data augmentation in self-supervised adversarial training (self-AT). In particular, the authors proposed a simple remedy named DynACL (Dynamic Adversarial Contrastive Learning) by adopting a dynamic augmentation schedule along the training process. Besides the empirical results about the impact of data augmentation on self-supervised adversarial training, the authors also tried to include some theoretical analyses.
justification_for_why_not_higher_score: The authors only evaluate the proposed algorithm on smaller datasets, but either contrastive learning or adversarial robustness have already nowadays been evaluated on larger dataset imagenet, or at least tiny imagenet dataset. Also, though the authors provided some theoretical analyses, its connection with the algorithm is quite weak. 
justification_for_why_not_lower_score:  The authors showed extensive experiments with surprisingly good performance to support their observation on the data argumentation within self-supervised adversarial training. Though the proposed algorithm could be heuristic and its connection with the theory could be a bit weak, the empirical findings in this paper could be inspiring.
Decision: Accept: poster