Self-Supervised Adversarial Training via Diverse Augmented Queries and Self-Supervised Double Perturbation

Authors: ruize  zhang
sheng  tang
juan  cao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:46:26

Abstract: Recently, there have been some works studying self-supervised adversarial training, a learning paradigm that learns robust features without labels. While those works have narrowed the performance gap between self-supervised adversarial training (SAT) and supervised adversarial training (supervised AT), a well-established formulation of SAT and its connections with supervised AT are under-explored. Based on a simple SAT benchmark, we find that SAT still faces the problem of large robust generalization gap and degradation on natural samples. We hypothesize this is due to the lack of data complexity and model regularization and propose a method named as DAQ-SDP (Diverse Augmented Queries Self-supervised Double Perturbation). We first challenge the previous conclusion that complex data augmentations degrade robustness in SAT by using diversely augmented samples as queries to guide adversarial training. Inspired by previous works in supervised AT, we then incorporate a self-supervised double perturbation scheme to self-supervised learning (SSL), which promotes robustness transferable to downstream classification. Our work can be seamlessly combined with models pretrained by different SSL frameworks without revising the learning objectives and helps to bridge the gap between SAT and AT. Our method also improves both robust and natural accuracies across different SSL frameworks. Our code is available at https://github.com/rzzhang222/DAQ-SDP.

Main paper: 
 introduction: 
 Deep neural network has shown its power in various machine learning tasks. In spite of its beneficial properties in optimization and generalization, deep neural network is vulnerable to adversarial attack as samples with carefully designed tiny perturbations may cause significant deviations of model The ACL methods above were restricted to the CL framework and could not generalize to other SSL methods. Previously, Zhang et al. [38] proposed to disentangle the task of SSL and AT into two stages of learning, which first trains SSL models with natural data and then enables AT under the pseudo-supervision of the naturally trained SSL features. This learning framework brings consistent performance boost for both contrastive and non-contrastive methods. Moreover, the feature pseudo-supervision in this framework takes a form similar to the supervision in supervised and semi-supervised AT, thus providing a unified perspective to analyze AT paradigms under different supervision. In this paper, we regard this learning framework as a simple benchmark for SAT.While previous works on ACL and SAT have achieved remarkable performance that is comparable to supervised AT, there is neither a well-established formulation nor an analysis of the general learning process as in the supervised counterpart. To better understand the limitations of this learning paradigm, we start from the SAT benchmark proposed by Zhang et al. [38] , which advocates an SAT process that is much more efficient than previous ACL works (100 vs. 1000 epoches of adversarial training). As shown in Figure 1 , we record finetuned train-set and test-set accuracies after SAT pretraining on CIFAR-10 with ResNet-34 as backbone and find that there is a robust generalization gap of around 19% and clean accuracy gap of around 6%. The results are similar for CIFAR-100, which gives a robust generalization gap of around 22% and clean accuracy gap of around 9%. The results show that there exists a large robust generalization gap and natural performance degradation, similar to the AT counterpart.Given that both SSL and AT are hard tasks [38] , we hypothesize these problems to be caused by insufficient data complexity and model regularization. In order to solve the problems, we propose a method termed as DAQ-SDP (Diverse Augmented Queries Self-supervised Double Perturbation) and handle the problems from two aspects. First, while previous papers [24, 38] concluded that complex augmentation techniques are crucial for SSL but harmful for SAT robustness, we argue that strong and diverse augmentations could help SAT if used properly. Specifically, we find that every training distribution is worth one set of BatchNorm layers in SAT and propose an Augmentation-Adversary (Aug-Adv) Pairwise-BatchNorm adversarial training method. It turns out that naturally trained SSL models with a single set of batch norm layers can provide effective guidance for multi-branch adversarial training. Second, while previous works on SAT focused on designing sample adversarial perturbation for SSL, we find that self-supervised model perturbation also contributes to downstream robustness with a proper training scheme. Despite the task mismatch from pretraining to finetuning, there exists a cross-task transferability of robust generalization. Our work finds a general method that can be directly applied to different SSL frameworks without revising learning objectives, providing insights that contribute to the understanding of SAT.As far as we know, we are the first to analyze SAT as a general learning paradigm in an SSLframework-agnostic way and solves its problems by revealing its traits related to clean performance degradation and robust generalization. The main contributions are summarized as follows:• In contrast to the conclusion of previous works [38, 24] , we suggest that strong and diverse augmentations can boost self-supervised robustness and propose an Aug-Adv Pairwise-BatchNorm technique for better robust generalization and less natural degradation.• Different from previous works [22, 20, 14, 38, 18, 24, 37, 23] that focused on introducing sample adversarial perturbation from supervised AT to SSL, we use model perturbation in SSL pre-training to boost downstream robustness. We then propose a self-supervised double perturbation scheme in the later stage of SAT to improve robust generalization without affecting the learning of natural features.• We conduct experiments on CIFAR-10 and CIFAR-100, the commonly used datasets in previous works. On CIFAR-100, our proposed method improves over 2% on AutoAttack [7] and clean data results with ResNet-34. On CIFAR-10, our method improves over 1% on AutoAttack [7] and over 2% on clean data results with ResNet-34. The experimental results demonstrate the effectiveness of our method across SSL frameworks, models and datasets.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We claim that we find a general method to solve the robust generalization and clean accuracy reduction problem for different SSL frameworks. This claim is accurately reflected and supported in the analysis and experiments.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 The task of SAT integrates AT into SSL and aims at learning robust feature representations that enable efficient finetuning. Here we will give a brief summary of previous works related to this field.Supervised Adversarial Training (supervised AT) Given deep neural network's vulnerability to adversarial perturbations, many defense methods have been proposed. Among them adversarial training, originally proposed by Goodfellow et al. [16] , has become a prevalent method. Adversarial training simulates a min-max process that finds the perturbation with largest distortion and then minimize the training loss over the adversarial data. Madry et al. [25] proposed a representative adversarial training method and uses random initialized multi-step projected gradient descent to generate adversarial samples. Many following works [21, 10, 39, 32, 40] further revised the adversarial learning framework and applied more advanced techniques such as logits pairing, boundary guidance and consistency regularization for improving robustness while keeping the accuracy on clean data.Self-Supervised Learning (SSL) Self-supervised learning is the task of learning feature representations with no label available. In this case, a discriminative self-supervised method generally relies on a pretext task for pretraining to learn useful information. Previously, many pretext methods have been proposed [15, 11, 28] . One of the most successful pretext task is instance discrimination [6, 4] . Contrastive learning defines a instance discrimination task and learns the features using similarities between positive and negative pairs. The recent development from contrastive learning methods MoCo [6] and SimCLR [4] to non-contrastive methods BYOL [17] and SimSiam [5] has revised the contrastive loss to a positive-pair loss and further simplified the framework. Moreover, some works [29, 13, 19] combined the contrastive framework with some more advanced techniques including positive and negative pair mining, prototypes and customized contrastive view crafting to enrich the information learned by the contrastive framework for effective and efficient learning.The development of self-supervised learning provides a new direction for acquiring robust features. In self-supervised learning, the emergence of instance discrimination as the new state of art self-supervised pretext task provides a natural setting for adversarial robustness to fit in. While constructing reliable decision boundary using ground truth labels is not feasible, previous works [22, 20, 14, 24, 37] proposed to exploit contrastive loss adversarial training for promoting robustness. The assumption is that if the feature space near the data sample is smooth enough, the feature prediction of adversarially perturbed data will be consistent with its clean counterpart. Gupta et al. [18] suggested that contrastive learning has intrinsic sensitivity to adversarial perturbations and proposed a simple method to remove false negative pairs. This method enhances robustness in SSL without revising AT and is orthogonal to ours. Xu et al. [37] proposed to use causal reasoning in ACL and used adversarial invariance regularization to enhance ACL. While achieving an impressive performance, the methods above were still restricted to the contrastive learning framework.Later, Some works have started to explore robustness in the broader SSL picture. Zhang et al. [38] formulated SAT as a two-stage framework that first trains an SSL model and then uses the learned features as guidance for AT. This work has set a strong baseline for this task and can be directly generalized to different SSL frameworks. Thus we take this two-stage SAT framework as the baseline to explore robustness in the broader picture of SSL. Kim et al. [23] proposed an interesting idea that carefully crafted targeted adversarial perturbations can help enhancing robustness for non-contrastive SSL methods. However, the improvements on contrastive frameworks are not as consistent as in the non-contrastive case. Moreover, some works [38, 24] suggested that complex augmentations are crucial for SSL but destructive for SAT. In this work, we approach the task of SAT by studying its learning process and drawing an analogy to supervised AT, with the goal of acquiring a better understanding of the difference and similarity between these two learning paradigms. We then propose a method to tackle the potential problems in SAT. In the following sections, we will introduce the problem statement and then describe our motivation and method.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We mentioned in the introduction that to distill diverse and generalizable information from clean model, we need to have a backbone model that has sufficient capacity.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Self-Supervised Learning As the most representative contrastive learning framework, SimCLR [4] uses data in the same batch as negative pairs and optimizes the following objective:EQUATIONIn the equation above,τ 1 (x), τ 2 (x) are two augmented views of the same image.z i = g • f (τ i (x))is the projected feature of the corresponding view. z i and z j are a positive pair. N is the number of negative samples.Contrastive SSL relies on large batch size or extra maintained queue for negative pairs and can be computationally expensive. In contrast, positive-only frameworks only include positive pairs. The learning objective of SimSiam [5] , a representative positive-only method, can be formulated as:EQUATIONEQUATIONIn the equation above, τ 1 (x), τ 2 (x) are two augmented views of the same image.z i = g • f (τ i (x))and p i = h • z i are the projected and predicted feature of the corresponding view, in which g is a projector helps to preserve instance dicriminative features and h is a predictor helps to prevent model callapse.Adversarial Contrastive Learning Based on the framework of contrastive learning (CL), multiple previous works have proposed adversarial contrastive learning (ACL) methods, which aims at improving the robustness of the learned features. Pevious works have adopted such a learning framework but each had some revision of the loss term [22, 20, 14] . In general, those methods can be formulated as:EQUATIONwhereEQUATIONIn the equations above, ℓ adv CL is the adversarial contrastive loss with an extra variable for the adversarial view. It is often calculated as the average of the pairwise contrastive loss [14] . x adv is the adversarial sample generated with this loss.To find a general method that can improve robustness for different SSL frameworks, we start from a basic SAT framework [38] :EQUATIONandEQUATIONwhereEQUATIONThis SAT framework separates the adversarial training process into two stages. In the first stage, an SSL model f 1 is trained with clean data. Then the features predicted by the clean model are used as pseudo-supervision for adversarial training of f 2 in the second stage. The clean samples providing guiding features can be regarded as queries that help to distill useful information from the clean model. Compared to ACL methods mentioned above, this type of method is more general and also more computationally efficient. In our work, we adopt this framework as the baseline.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
method: 
 In this paper, we aim at finding a method that can solve the problem of robust generalization and clean accuracy degradation in SAT. Note that our work proposes a general method that can be directly combined with different pretrained SSL models for improvements instead of requiring adversarially re-training the models from scratch. Figure 2 demonstrates the overall framework of our method. In the following sections, we will introduce each part of our method.Previously in supervised AT, some works have discussed about the effects of complex augmentation strategies on robustness [31, 30, 1] . The idea is to fit the model to the labels on the generalized sample distributions to reduce overfitting. However, in SAT, there is no ground truth label to provide supervision on training or generalized data distributions. We hypothesize that SSL models trained with natural data, especially instance discrimination based ones, already contain certain level of generalization capability as the feature space is learned with large amount of data under strong and complex augmentations. However, this capability of generalization can be lost during AT. Thus we propose to use the diversely augmented clean features as supervision for AT.In the field of ACL and SAT, previous works [38, 24] have concluded that strong and diverse augmentations are essential for SSL but harmful for robustness. Thus Luo et al. [24] proposed to gradually reduce the strength of augmentation during the training and Zhang et al. [38] proposed to only use random resized crop and horizontal flip in adversarial training. In this paper, we argue that diversely augmented samples help SAT given that the model has sufficient capacity, as it's actually essential for the robust model to distill rich information to keep the capability of generalization.Note that data augmentation strategies including AutoAugment [9] and RandAugment [8] require labels to calculate validation accuracy in the process of searching for optimal augmentation policies, thus can not be directly applied to SAT. In this paper, we propose to use TrivialAugment [27] , which is a dataset-independent and search-free method that randomly samples the augmentation policy and strength. Since we need to improve the generalization while also specialize on the testing distribution, we need to fit the adversarial model with clean model features both on the diversely augmented and basic augmented distributions. Inspired by previous work in supervised AT [1] , we adopt a multi-stream structure that takes samples from different distributions as inputs. In the previous ACL and SAT works, there were no clear conclusion on the usage of BatchNorm layers. While some works [22, 14] suggested to use different BatchNorm layers for adversarial sample, Zhang et al. [38] The formulation of our method is:EQUATIONwhereEQUATIONandEQUATIONandEQUATIONIn the equations above, Sim(., .) is the cosine similarity between two features and aug i corresponds to the augmentation type mentioned above. f 2-augi-adv and f 2-augi-clean are the student model with the corresponding pairwise-BatchNorm layers. Our method forces the pseudo-supervised adversarial model to inherit the generalization capability from clean model with diversely augmented queries. The AugAdv pairwise-BN helps features of each training distribution to fit the clean feature counterparts without interfering with each other. The experimental results demonstrate that diverse and complex augmentations can improve SAT robustness. This finding helps to narrow the gap between improving SAT and supervised AT.Previous works in ACL and SAT have borrowed the idea of adversarial sample perturbation from supervised AT to SAT and made revisions either on the specific training loss term [20, 22] or the effective way of generating sample perturbations [14, 23] . However, whether more advanced ideas in supervised AT can bring improvements in SAT is under-explored. In this section, we suggest that adversarial weight perturbation can be introduced into SSL pretext task for downstream robustness. Note that in the method proposed by Wu et al. [35] , adversarial weight perturbation is applied to different supervised AT frameworks including vanilla PGD [25] , TRADES [39] , RST [3] and MART [34] . However, all those methods require full or partial labels and are based on classification loss.The adversarial weight perturbation can be expressed as:EQUATION)where µ is the perturbation size of the model weight.In contrast, in our work the weight perturbation is introduced into the SSL pretext task. Thus the perturbation doesn't regularize the weight classification-loss landscape, but works on the feature similarity loss in a label-free paradigm instead. This transition of learning paradigm makes it interesting to see whether such perturbations respect to the SSL objective can benefit downstream robust generalization. Specifically, the self-supervised weight adversarial perturbation perturbs the model weight in the direction of enlarging the self-supervised cosine similarity loss and increases the smoothness of this similarity loss landscape. The weight perturbation can be formulated as:EQUATION)The weight perturbation finds the "worst" adversarial scenario which is beneficial for model robustness. However, this extra adversarial component also further increases the difficulty of the task. In supervised AT, the existence of ground truth labels helps the model to converge despite the enlarged difficulty. However, both AT and SSL are difficult tasks. In the early stage of SAT, regulating the model to learn this rather difficult objective with respect to insufficiently learned adversarial features could impede the learning of natural features and we propose to apply this weight adversarial perturbation only in the later stage of learning when the pseudo-supervised learning of clean and adversarial features is stabilized. Without our weight self-perturbation scheme, there is a clean accuracy drop of 0.7% and PGD robust accuracy drop of 0.8% on CIFAR-10 with ResNet-34.The adversarial weight perturbation calculated on weakly augmented data is combined with sample adversarial perturbations over the four sample distributions in the previous section. The overall learning objective is:EQUATIONwhereEQUATIONand ℓ adv = Sim(f θ2-augi-adv (x adv aug i ), f θ2-augi-clean (x augi )). As shown in Figures 1 and 3 , the robust generalization gap is reduced by around 3% and the test clean accuracy improves by more than 1.5% on average with our proposed method, which means properly regulating the smoothness of weight SSL-loss landscape in pre-training can improve the robust generalization of downstream classification despite the lack of labels.Despite the task difference between the learning paradigms, our method steps forward to an unified understanding of SAT and supervised AT by revealing their similar characteristics with respect to generalization. In this work, We also provide a "higher" level of perspective than previous narrower methods that focus on SAT with single SSL framework. We look forward to seeing future works with general approaches that improve adversarial training across different supervision settings. 
experiment: 
 In this section, we demonstrate the effectiveness of our method. First, we evaluate the effectiveness when DAQ-SDP is plugged into contrastive and positive-pair only SSL frameworks. Then we conduct an ablation study for each part of our method. We also visualize the representation learned by our method through t-SNE [33] in Appendix. Experimental Setup: We apply our method on top of SimCLR [4] , SimSiam [5] and BYOL [17] to evaluate the performance improvements across SSL frameworks. We also conduct extensive experiments on ResNet-18, ResNet-34 and ResNet-50. We find that the improvements are less significant on the smaller model ResNet-18. The rationale may be that simply increasing data complexity for ResNet-18 by using one extra strongly-augmented view could decrease the training set clean accuracy from 54.45% to 52.78% on CIFAR-100, suggesting insufficient model capacity to distill richer information while fitting well on clean data. This is understandable as model capacity takes a significant role in adversarial robustness and many techniques in supervised AT [30, 1, 35] require models with sufficient capacity to be effective. Given the complexity of AT and SSL, we expect a model larger than ResNet-18 is needed to learn rich information from the training data.All SSL models in our method are first trained with clean data for 1000 epochs, then adversarially trained with 5-step PGD attack with the epsilon size of 8/255. Methods in previous works are adversarially trained for 1000 epoches as mentioned in their papers. The robustness is evaluated with 1 and Table 2 , our method contributes to significant improvements on both clean and robust accuracy on both CIFAR-100 and CIFAR-10 with SimCLR [4] . Note that TARO [23] is an adversarial sample generation method that needs to be combined with specific SAT baselines. In this work we combine it with the same baseline framework we use for better performance and fair comparison. Although TARO [23] gives slightly better clean accuracy on CIFAR-10, our method outperforms TARO [23] on robust accuracy by a large margin. On CIFAR-100, both our clean and robust accuracy outperforms TARO [23] . Also note that methods except TARO [23] , DecoupledACL [38] and ours are contrastive based methods and don't generalize to positive-pair only SSL frameworks. In Table 3 , we compare our method with previous works on SimSiam [5] and BYOL [17] . As shown in the results, our work provides a consistent improvements for different SSL frameworks. This is because we treat differently trained models as the teacher that provides supervision for the clean data feature space. Once we obtain the supervision, we take an SSL-framework agnostic process for robustness improvements.b) We then conduct experiments on ResNet-18 and ResNet-50. From Table 4 and Table 5 , our method shows improvement across model sizes.c) We also provide cross-dataset transfer learning results. Table 6 shows transfer learning from CIFAR-100 to CIFAR-10, in which our work outperforms other SAT methods. In Table 7 , we provide the transfer learning SLF results from CIFAR-10 to STL-10, which shows that our method can transfer well across datasets from more different domains. In this section we evaluate each part of our method. Table 8 shows the results on CIFAR-100. With our Diverse Augmented Query, the clean accuracy increases by 1.23% and the PGD robust accuracy increases by 1.43%. With our Diverse Augmented Query and Weight Self-Perturbed Scheme, the clean accuracy increases by 2.10% and the robust accuracy increases by 2.41% . As shown in Figures 1 and 3 , both robust generalization and clean accuracy are improved compared to the baseline. These improvements brought by our proposed method demonstrate the effectiveness of the enhanced sample complexity and model regularization with the SSL loss term.The smoothness of the weight loss landscape has been shown to be important for robust generalization in supervised AT [35] . In Figure 4 we also analyze the effect of our method to downstream weight loss landscape through 1D visualization as previous work in supervised AT [35] did. From Figure 4 we can see that the regularizing effect transferred to downstream loss landscape is actually much attenuated compared with directly regularizing classification loss in supervised AT [35] , showing difficulty of such a transferred improvement from SSL pretext tasks. 
conclusion: 
 In this paper, we observe that SAT has the similar problem of large robust generalization gap and clean accuracy degradation as in supervised AT. We then propose a general method to solve this problem, which can be directly combined with different pretrained SSL models without further changing learning objectives. We first challenge the previous conclusion that diverse and strong augmentations harms SAT and propose a diversely augmented query based method with Aug-Adv Pairwise-BatchNorm to distill generalizable and diverse information from clean model. Second, different from previous works that focused on introducing sample perturbation to the SSL pretext task, we suggest that regulating the smoothness of the SSL loss landscape by adversarial weight self-perturbation boosts robust generalization transferable to downstream classification. Our method not only improves the performance across different SSL frameworks, but also provides insights for narrowing the gap between the study of these two adversarial learning paradigms.A Appendix / supplemental material A.1 Feature VisualizationIn this section, we visualize the the learned features of the test set of CIFAR-10 with t-SNE [33] . Each data point is colored with its label. As shown in 5, the features of different classes learned by our DAQ-SDP has a clearer boundary than the baseline method.  
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We included the compute resources in the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Our work helps to build robust models under no label conditions, which can be useful in real life.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The contents used in this paper are cited Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The model and code will be released if got accepted.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our work is useful for pretraining robust models with no labels. However, it generated one more adversarial data and perturbed weights, which takes more computation. So it can cause more consumption of energy and pollution. Despite this limitation, we believe our method is still beneficial for the society and promotes model robustness in real life.NeurIPS Paper ChecklistJustification: We conformed to the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This work analyzes and improves self-supervised adversarial training (SAT) as it relates to adversarial training (AT). The results improve across a variety of self-supervised schemes and the reviewer consensus is positive. Importantly this is an informative study at a higher "level" than prior narrower studies of SAT, in its proposal of a general approach across specific self-supervision methods, which can promote more research on adversarial training without supervision. The AC sides with acceptance.

Four reviewers with expertise in domain adaptation, self-supervision, distribution alignment, and adversarial vote for acceptance (mcfF: 7) and borderline acceptnace (dErS: 5, 6QGn: 5, iqHN: 5). The authors provide a rebuttal, all reviewers engage with the rebuttal, and the authors and reviewers discuss across multiple threads and posts.

To summarize, the main argument for acceptance is the improved understanding and improved empirical results for SAT relative to AT which are delivered by a general approach consisting of the novel coordination of data and model perturbations alongside the factorization of model statistics. While elements exist in prior work, they are appropriately credited, and the proposed method is sufficiently novel. Lastly the empirical findings counter past evaluations and therefore provide valuable signal. On the other hand, the weaknesses raised during review are (1) issues in notation and lacking ablation (mcfF), (2) questions of clarity (mcfF, dErS, 6QGn), (3) concern for novelty vs. two-stage AT (dErS), and (4) lacking coverage across model sizes like a smaller ResNet-18 in particular and more generally a question of limitations (iqHN). Each of the weaknesses was satisfactorily resolved by the rebuttal and discussion, and during the AC-reviewer discussion phase no additional issues were raised.

The productive discussion and clarifications led to higher scores for mcfF and 6QGn and iqHN. The thread with iqHN is significant, because it was the most negative, but it ultimately resulted in support of borderline acceptance. The area chair monitored this thread closely, and commends the authors for preserving a factual and respectful tone, and acknowledges the reviewer for the respectful and thoughtful conclusion of the thread.

Miscellaneous feedback:

- There is a typo in the abstract for "generation gap" vs. "generalization gap".
- This is a matter of style, but the number of significant digits in plots such as Figures 1 & 3 are perhaps distractingly high for the range of results, so consider lesser precision for presentation.

Reviews: review: paper_summary: paper_summary: This paper proposes a method to solve the robust generalization problem for self-supervised adversarial training in general. Starting by showing the generalization gap in existing self-supervised adversarial training framework, it proposes to solve the problem from the aspects of data complexity and model regularization and provides self-supervised remedies. This paper reduces the gap between the understanding for traditional adversarial training and self-supervised adversarial training by demonstrating the connection of their characteristics and helps to build a “unified perspective for adversarial training” under different scenarios. Robust accuracy and clean accuracy are improved across multiple self-supervised learning frameworks.
strength_weakness: strength_weakness: 1.To illustrate the usage of the “pairwise-BatchNorm” technique, the authors may need to provide ablation study that uses normal BatchNorm instead for comparison.
2.Some notations lack explanations in the paper. As an example, the notation related to theta are inconsistent in equation (10) and (11). Which theta in equation (10) is used in equation (11)?
3.There are typos or missing signs in the formulas, e.g. the adversarial data generation in equation (11).
questions: questions: 1.Correcting the typos in the formulas and explain the important notations are needed.
2.Additional ablation results for the “pairwise-BatchNorm” are needed.
limitations: limitations: The authors addressed the requirement for sufficient model capacity based on the complexity of this adversarial training task. This is acceptable. There is no negative societal impact.
paper_summary: paper_summary: The paper proposes a method to improve self-supervised adversarial training. This method consists of two stages. First, a standard self-supervised model is trained on clean images to learn a feature extractor network F-1. In the second stage, a robust feature extractor F_2 is trained based on the features generated by F_1. The total loss is a linear combination of standard self-supervised loss, where F-2 learns to mimic the features generated by F_1, and an adversarial loss. The adversarial loss follows adversarial training framework where the targets are provided by F_1. The method applies weak and strong augmentations in both stages. However, in order to process different streams of data, the method uses four sets of batch-norm layers for weak and strong augmentations and adversarial and clean images. Moreover, the method applies adversarial weight perturbation for downstream robustness.
strength_weakness: strength_weakness: The proposed approach does not include substantial novelty, where the main idea is following the two stage adversarial training in [38]. The addition of weight perturbation from [35] does not prompt a creative combination of ideas. 

The presentation clarity can be substantially improved. There are several terms that are never defined in the equations such as aug_i and  f2_augi_clean in eqs 9,10,11, and g and h in lines 144, 150. In eqs 3 and 4 l_CL is a three-variable function, whereas it is defined as a two-variable function in eq 1. There are several typos such as line 171 "we aims", 185 "In the filed of", 242 "loss landscape respect". 

The claim of unifying self-supervised and supervised adversarial training in section 4.3 is not supported empirically or theoretically.
questions: questions: There are no experiments to show the outcome of weight perturbation on the loss landscape of the proposed method. Overall the paper is missing qualitative experiments.
limitations: limitations: The authors discuss aspects of limitations of their method in introduction, however potential negative societal impact is not included.
paper_summary: paper_summary: The paper introduces the DAQ-SDP (Diverse Augmented Queries Self-supervised Double Perturbation) method to solve the problem of large robust generalization gap and clean accuracy degradation in self-supervised adversarial training. The experimental results demonstrate the effectiveness of the DAQ-SDP method.
strength_weakness: strength_weakness: 1. Assumptions lack reasonable basis. (In line 56)
2. There are some typos:
	- Incorrect citation and reference:
		- Using \citet is better. (In lines 39, 122, 132, etc.）
		- In lines 314 and 315, "Figure 1" is linked to Table 1.
	- Inconsistent names:
		- "Pairwise-BatchNorm" (in line 63) and "pairwise BatchNorm" (in line 209)
		- "resNet34" (in caption of Table 1) and "ResNet34" (in caption of Table 2)
		- "Taro" (in lines 297, 300, 301, 302) and "TARO" (in Tables 1, 2, 5)
	- Some typos:
		- "clean" instead of "lean" in Eq. 8.
		- "note" instead of "Note" in Line 302.
		- "where" instead of "Where" in Line 234.
		- "a result" ? in Line 308.
3. Figures seem to be a bit fuzzy.
questions: questions: 1. Could you share the reason why you chose these hyperparameters (In lines 284-288) and whether you have done experiments with other hyperparameters? 
2. How does DAQ-SDP compare to DecoupledACL and TARO in terms of runtime?
3. Can other models be used to prove the effectiveness of the approach?
limitations: limitations: The author mentioned that this approach is only applicable to a sufficiently large model (ResNet34).
paper_summary: paper_summary: The authors aim to further close the gap between the clean generalization gap and the robust generalization gap for SSL models. Several engineering improvements have been made to the SAT pipeline, such as including strong data augmentation, adversarially perturbing weight, and using separate BN. The resulting algorithm exhibits non-trivial improvements on benchmark datasets.
strength_weakness: strength_weakness: Major:

(1) Baseline + Weight Self-Perturbed Scheme should be included in Table 4.

(2) The authors claim that " diversely augmented samples help SAT given that the model has sufficient capacity". I am wondering whether the difference between ResNet-18 (used in previous papers) and ResNet-34 (used in this paper) is large enough for the claim to hold. How about using even larger models, like ResNet-101 or Wide-ResNet-28-10? 

(3) Also, I believe the ResNet-18 results should not be entirely excluded. Previous works only tuned their hyper-parameters on ResNet-18, while the hyper-parameters for the proposed algorithm are tuned on ResNet-34. The comparison may not be fair. 

(4) The claims made in Section 4.3 are not well supported by the results, especially the "unified understanding of SAT and AT". I cannot see why applying AWP to SAT can lead to such "unified understanding".  Besides, "robustness and robust generalization can be transferable across SSL and classification" is also evidenced by previous works.


Minor:

(1) Figure 1 and Figure 3 seem blur. Are you using the JPEG images directly?

(2) Typo errors: the caption of Table 1. Equation 8.
questions: questions: (1) How does the algorithm perform on ResNet-18?
limitations: limitations: See weakness.

Comments: title: Thanks for the rebuttal
Thanks for the rebuttal
further clarification for rebuttal
further clarification for the rebuttal
response to question of reviewer iqHN
Thanks for the rebuttal
comment: Dear reviewer iqHN,

Thanks for your reply. We will incorporate all the suggestions from reviewers into the revised version. 

Best Regards.
Dear authors:

I acknowledge your diligent effort during the rebuttal period. After reading the latest rebuttal as well as comments from other reviewers, I believe my main concerns are mostly resolved. 

I have adjusted my rating accordingly and I hope that the extra experiments, discussion, and clarification will be incorporated into the revised version.

All the best
Dear reviewer iqHN,

Thanks for your time and efforts in this discussion. We understand your worries about our motivation and writing clarity. We still really hope you can consider the following explanations:

---

1.	We humbly explain why **at least we didn’t deliberately hide the limitation of the method**. In the initial submission, we claimed that our method “helps SAT given that the model has sufficient capacity”, “ResNet18 … has insufficient model capacity to distill richer information” and “a model larger than ResNet18 is needed” for our method. By saying this, we tried to **make the main claim that the method has good performance on models larger than ResNet18**. All the experiments were used to support this point. **At least in our point of view**, **this description of our improvement contains no exaggeration**. Maybe this claim of limitation is still insufficient, but at least we didn’t intend to lead the reader to ignore this limitation. Otherwise, we would not repeat this claim several times in the paper. Actually, we mentioned this limitation during rebuttal again because we thought the limitation is already claimed and the claim in the paper is not exaggerated. 

---

2.	We hope to argue that **BYOL and SimSiam were written as “non-contrastive”** in [41] (ICML 2021) and [23] (NIPS 2023). Both of these works analyze the difference between non-contrastive methods (specifically BYOL and SimSiam) and contrastive methods.

---

3.	We realize that we wrote long rebuttal paragraphs that contain too many unnecessary details and cause confusion. We feel very sorry for this. If possible, we hope to **give a very brief explanation for the clarity problem**:
- In “self-perturbed AWP …”, we are comparing **clean and PGD accuracy** of our weight perturbation scheme (**applying AWP after 60 epochs**) and applying weight perturbation for **the whole training process**. 
- As you point out, the “robust overfitting” defined as “reduction of test accuracy during adversarial training” in supervised AT is not observed in SAT. However, at least based on our experimental result, there still exists a related **training-testing robust generalization gap problem**. This paper is trying to solve this problem.

---

We thank again for your efforts in reviewing our paper. Your suggestions about writing clarity and formatting paragraphs are very helpful. We really appreciate it. 

Best Regards,

Authors

[23] Minseon Kim, Hyeonjeong Ha, Sooel Son, and Sung Ju Hwang. Effective Targeted Attacks for Adversarial Self-Supervised Learning. In NeurIPS, pages 56885–56902, 2023.

[41] Yuandong Tian, Xinlei Chen and Surya Ganguli. Understanding Self-Supervised Learning Dynamics without Contrastive Pairs. In ICML 2021.
Thanks for your response. My concerns have been solved. I raised my score.
Dear authors,

First I would like to express my gratitude for you to add further clarification which I find to be helpful. However, there are three main concerns that I feel worrying after reading the continued rebuttal.

----

**Academic Integrity**

As a reviewer, I care more about your motivation behind leaving out the ResNet-18 result despite the model being adopted in most (if not all) previous works. How can the readers know the performance of your algorithm on smaller models? It is not ethical to exclude any important results, especially those related to the main claims, and write several unjustified claims to explain why you do so.

----

**Writing Clarity**

Reading sentences like these is pure pain:

"the self-perturbed AWP scheme provides a 0.70% and 0.80% improvements over self-perturbed AWP for the whole training process

- "a 0.70% and 0.80% improvements": This is not grammatically correct.

- "self-perturbed AWP ... improve over ... self-perturbed AWP ...": Why are you comparing the same method?

"The related robust overfitting phenomenon is studied a lot in supervised AT but not studied in the SSL context"

- because SSL doesn't robustly overfit at all!

----

**Writing correctness***

The authors write that "non-contrastive methods BYOL and SimSiam".

However, it appears that the primary reason that algorithms like BYOL and SimSiam work is that it is doing a form of contrastive learning—just via an indirect mechanism. I personally view them as contrastive learning with advanced tricks, as it is written by the SimSiam authors "so it (SimSiam) can also be thought of as 'SimCLR without negative pairs'".

----

For the above reasons, I will keep my score at 3. I do not recommend acceptance.
We further clarify the runtime of DecoupledACL, TARO and our method DAQ-SDP: 
DecoupledACL takes 2min52s per epoch. TARO takes 2min54s per epoch. Our method DAQ-SDP takes 5min24s in the first 60 epochs and 6min29s in the last 40 epochs.
I would like to acknowledge that I have read the authors' rebuttal. The new experiments and visualizations address some of my concerns. Thus, I increased my rating.
Please also note that although the results with ResNet18 and ResNet50 are not included the original submission, we have clearly stated the limitation of the method with respect to model size from line 275 to line 284 in the Experimental Setup section.
Sorry for missing the ablation study results in the previous reply. The ablation study result on CIFAR100 with SimCLR is here:
| Method | Clean | PGD |
|---|---|---|
|Baseline | 51.44 | 30.68 |
|DAQ(single-BN) | 52.27 | 31.56 |
|Diverse Augmented Query | 52.67 | 32.11 |
|Weight Self-Perturbed Scheme | 51.56 | 32.37 |
|DAQ-SDP(ours) | 53.54 | 33.09 |
We are glad to discuss **your questions about the claim of “unified perspective” and “robust generalization”** in this part. Indeed, the whole adversarial self-supervised learning literature is about robustness and generalization transferable to downstream tasks. But **there is a difference between previous works and ours**. And **in this part we hope to clarify the following points**:

---

1.	Since you have asked about the difference between our work and previous works that explored robustness and generalization in self-supervised AT. In this paragraph we hope to first explain  there are actually two types of generalization in the self-supervised AT context. **The first is the generalization between tasks from the SSL perspective**. For instance, SSL robustness generalizing to other datasets and downstream classification tasks. At least to the best of our knowledge, all previous works explore the extent of this generalization. In this work we are actually curious about **the robust generalization of SSL methods from the adversarial training (training-testing) perspective, which is related to the robust overfitting concept in supervised AT**. Since SSL operates on features and usually directly finetuned on downstream datasets, this generalization/overfitting during pretraining is often overlooked in previous works. This problem can be seen clearer if we conduct pretraining and finetuning on the train/test split of the same dataset. (This setting is called self-task transfer in previous works) Although we don’t observe decreased validation robustness during training (called **robust overfitting**) as in supervised AT, we do **observe a similar large robust generalization gap** (in Figure 1 of the paper). Our method **helps to solve this generalization problem** (in Figure 3) and **improves the results from this AT perspective for both contrastive and non-contrastive SSL** (shown in the same dataset and cross-dataset downstream classification results). The related robust overfitting phenomenon is studied a lot in supervised AT but not studied in the SSL context. **We believe the identification and analysis of such traits of self-supervised AT create a space for further improvements.** That is what we mean by “providing a unified perspective across AT under different supervisions”. We look forward to your suggestions about making these phrases clearer. $\newline$

---

2.	In this paragraph we would like to discuss **why our adversarial self-perturbed weight is not that trivial or straightforward and how is that related to our motivation**. In supervised AT, phenomenon of robust overfitting is prevalent and AWP helps to improve robustness because there is an obvious drop of validation robust accuracy in training. However, **whether AWP works depends on the trade-off between reduced generalization gap and decreased training accuracy.** Since there is a difference in the difficulty of supervised AT and self-supervised AT (which is “a challenging problem due to its two mixed challenging goals” and “clearly can be even more challenging than the semi-supervised AT setting” in [38]), **it’s not obvious whether the robustness generalization gap for self-supervised AT is large enough to compensate for the trade-off for improvement**. It’s also interesting that using the self-perturbed AWP scheme provides a 0.70% and 0.80% improvements over self-perturbed AWP for the whole training process on CIFAR10. Without the scheme there is a 0.88% improvement on PGD acc. but 0.81% reduction on clean acc. compared with the result without self-perturbed AWP. This can also be related to the traits of self-supervised AT. 

---

3.	In this paragraph we further explain why is the improvement not that straightforward. **Transferring preferrable AT properties except robustness itself, e.g. loss landscape smoothness, from SSL to classification is not considered in previous works and not as easy as one might think**. In Figure 4 of the uploaded one-page pdf, we show the 1d visualization of resulting loss landscape of the baseline and our method. Although regularizing SSL objective smooths downstream loss landscape, the transferred regularization effect is much attenuated than directly regularizing classification loss as in [35], showing the difficulty of such an improvement.
Dear reviewer iqHN,

We have updated the long paragraphs with new format, highlight and summary:

In this paragraph, we hope to argue that **comparison between our method and previous ones on ResNet34 are fair** and **the hyperparameters are finetuned to our best effort or used appropriately**. **First**, DynACL[24]+AIR[37] results are the same as reported in their original paper. **Second**, DecoupledACL[38] hyperparameters on larger models are finetuned for clean-robust trade-off. The hyperparameter to tune is weight decay in adversarial training. On CIFAR100, weight decay of 2e-4 gives 51.44% (clean) and 30.68% (PGD) but a larger regularization with weight decay of 5e-4 gives 49.94% and 31.99%. On CIFAR10, weight decay of 5e-4 gives 82.46 (clean) and 56.86% (PGD) but a larger regularization of 1e-3 gives 80.57% and 56.48%. **Third**, TARO[23] is an attack method that can be combined with different frameworks. The initial one-stage training results reported in the paper is much lower and we combined it with our model for improved results. The hyperparameter that matters is the batch size for choosing target to attack. We use the same batch size of 512. On ResNet50, the hyperparameter is finetuned in the same manner. An increase of weight decay from 5e-4 to 6e-4 on CIFAR10 typically causes a robustness reduction of around 1.00% in these methods.
Dear reviewer iqHN,

Thanks for your reply.

First, we would like to argue that in the initial submission, we already pointed out that the performance of the method is more obvious on larger models than on ResNet18, that's because our method requires larger model capacity for better performance. **This claim is already included in the original submission.** You can find this from line 275 to line 284 in Experimental Setup section. **So we are not hiding results.** In the rebuttal, you require us to provide the results on other models, so we provided them. 

Second, the response to Q4 is trying to solve the worry that our experiments are unfair. We are arguing that **we finetuned the parameters on larger models and used appropriate batchsize.**

Third, the response to Q5 is trying to explain our motivation. We first argued that **the generalization/overfitting problem from adversarial training perspective in self-supervised AT is overlooked** in previous works. But the identification of this problem creates a space for improvement. Then we argued that **the improvement brought by the self-perturbed adversarial weight technique is not that straightforward or easy.**

Best Regards,

Authors
Dear authors,

First I would like to express my gratitude for you to conduct the extra experiments, which I find to be helpful. However, there are two main concerns that I feel worrying after reading the rebuttal.

----

**Academic Integrity**

The authors state in the rebuttal that "The results of ResNet18 are not included in our initial submission for it is not as obvious as in the larger model.", which reveals that the authors have the intention to **exclude certain experiment results simply because they are not in favor of the authors' claim** rather than faithfully acknowledge the results as a limitation or give intuitive/rigorous explanation for the results in the initial submission.  

According to the NeurIPS Code of Conduct, "Scientific misconduct including fabrication, falsification, or plagiarism of paper submissions or research presentations, including demos, exhibits or posters.", I personally believe the authors' motivation of **hiding** the results on smaller models worrying.

----

**Writing Clarity**

The responses to Q4 and Q5 have no format, no highlight, and no summarization. Even after spending 15 minutes trying to understand the long paragraphs, I still cannot understand the core logic behind the authors' responses. Even though the original paper is written well, I still find the unclearness in the rebuttal unsatisfactory.

----

For the above reasons, I have lowered my score to 3. I would like to reconsider my rating if the authors can provide more clarification.