SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- RoCL [23]. RoCL is SimCLR [6] based adversarial self-supervised learning methods. We experiment with the official code $\dagger$ . To make a fair comparison, we set the attack step to 10 as other baselines. We train the model with 1,000 epochs under the LARS optimizer with weight decay  $2e - 6$  and momentum with 0.9. For the learning rate schedule, we also followed linear warmup with cosine decay scheduling. We set a batch size of 512 for all datasets (CIFAR-10, CIFAR-100, STL10). For data augmentation, we use a random crop with 0.08 to 1.0 size, horizontal flip with a probability of 0.5, color jitter with a probability of 0.8, and grayscale with a probability of 0.2 for RoCL training.
- ACL [22]. ACL is SimCLR [6] based adversarial self-supervised learning methods. We conduct the experiment with the official code $\ddagger$ . To make a fair comparison, we set the attack step to 10 as other baselines. We train the model with 1,000 epochs. We set a batch size of 512 for STL10 dataset. For CIFAR-10, and CIFAR-100, we use the official pretrained checkpoints. For data augmentation, we use a random crop with 0.08 to 1.0 size, horizontal flip with a probability of 0.5, color jitter with a probability of 0.8, and grayscale with a probability of 0.2 for ACL training. We set PGD dual mode which calculates both clean and adversarial during the training.
- BYORL [15] BYORL is BYOL [17] based adversarial self-supervised learning methods for low label regime. Since there is no official code for BYORL we implement the BYORL by ourselves. We implement based on BYOL from a self-supervised learning library  $\S$ . We use the same CIFAR-10 setting in the library except for normalization. We exclude normalization in the data augmentation. To make a fair comparison, we implement on the ResNet18 with attack step 10 of PGD. As shown in supplementary materials in [15], when the model is trained with 10 steps in ResNet34 it shows  $37.88\%$  of robustness. We conjecture that we have a different performance from the original paper because the original paper employs 40 steps of PGD in WideResNet34 to obtain the reported robustness which requires extraordinary computation power.
- AdvCL [12]. AdvCL is SimCLR [6] based adversarial self-supervised learning which employ pseudo labels from the model that is pretrained on ImageNet [26] data. Even though the outstanding performance of AdvCL, we exclude this model as our baseline because the proposed methods require the model that is trained with the labels of ImageNet which we assume to have no label information for training.
B Detailed description of experimental setups.
B.1 Resource description.
All experiments are conducted with a two NVIDIA RTX 2080 Ti, except for the experiments with CIFAR-100 experiments. For CIFAR-100 experiments, two NVIDIA RTX 3080 are used. All experiments are processed in Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz.
B.2 Training detail.
For all methods, we train on ResNet18 [18] with  $\ell_{\infty}$  attacks with attack strength of  $\epsilon = 8 / 255$  and step size of  $\alpha = 2 / 255$ , with the number of inner maximization iterations set to  $K = 10$ . For the optimization, we train every model for 800 epochs using the SGD optimizer with the learning rate of
$\dagger$ https://github.com/Kim-Minseon/RoCL  
 $\ddagger$ https://github.com/VITA-Group/Adversarial-Contrastive-Learning  
 $\S$ https://github.com/vturrisi/solo-learn
Algorithm 1 Targeted Attack Robust Self-Supervised Learning (TARO) for contrastive-based SSL
Input: Dataset  $\mathcal{D}$  , transformation function t, model  $f$  , parameter of model  $\theta$  , target score function S   
for iter  $\in$  number of iteration do   
for  $x_{i}\in$  miniBatch  $B = \{x_{1},\ldots ,x_{m}\}$  do   
for n in 2 do Transform input  $\mathbf{t}_n(x_i)$  Find target images  $\mathbf{t}_n(x_k)$  from  $\mathcal{S}(\mathbf{t}_n(x_i)$  , batch) Generate targeted adversarial examples  $\mathcal{L}_{\mathrm{cont - attack}} = \mathcal{L}_{\mathrm{nt - xent}}(\mathbf{t}_1(x_i),\{\emptyset \} ,\{\mathbf{t}_1(x_i)_{\{\mathrm{neg}\}}\}) + \mathcal{L}_{\mathrm{simplarity}}(\mathbf{t}_1(x_i),\mathbf{t}_2(x_k))$ $\delta^{t + 1} = \Pi_{B(0,\epsilon)}\Bigl (\delta^t +\alpha \mathrm{sign}\Bigl (\nabla_{\delta^t}[\mathcal{L}_{\mathrm{cont - attack}}]\Bigr)\Bigr)$ $\mathbf{t}_n(x_i)^{adv} = \mathbf{t}_n(x_i) + \delta^t$    
end for Calculate training loss   
 $\mathcal{L}_{\mathrm{TARO}} = \mathcal{L}_{\mathrm{nt - xent}}(\mathbf{t}_1(x_i),\{\mathbf{t}_2(x_i),\mathbf{t}_1(x_i)^{adv}\} ,\{\mathbf{t}_1(x_i)_{\{\mathrm{neg}\}}\})$    
end for   
 $\theta \gets \theta -\beta \nabla_{\theta}\mathcal{L}_{\mathrm{TARO}}$    
end for
0.05, weight decay of  $5\mathrm{e} - 4$ , and the momentum of 0.9. For data augmentation, we use a random crop with 0.08 to 1.0 size, horizontal flip with a probability of 0.5, color jitter with a probability of 0.8, and grayscale with a probability of 0.2. We exclude normalization for adversarial training. We set the weight of adversarial similarity loss  $w$  as 2.0. We use batch size 512 with two GPUs.
In the score function, we calculate the similarity score term and the entropy term as shown in Equation 11. First, to exclude the positive pairs' similarity score we set the similarity score between positive pairs to  $-1$ . Then, to calculate the overall score, after obtaining the similarity score and entropy of each sample, we normalize each component with Euclidean normalization to balance each component to the score function. Further, the detailed algorithm of TARO for contrastive SSL is described in Algorithm 1 and Eq. 14.
B.3 Evaluation details.
PGD  $\ell_{\infty}$  attack. For all PGD  $\ell_{\infty}$  attacks used in the test time, we use the projected gradient descent (PGD) attack with the strength of  $\epsilon = 8 / 255$ , with the step size of  $\alpha = 8 / 2550$ , and with the number of inner maximization iteration set to  $K = 20$  with the random start.
AutoAttack. We further test against a strong gradient-based attack, i.e., AutoAttack (AA) [8]. AutoAttack is an ensemble attack of four different attacks (APGD-CE, APGD-T, FAB-T [9], and Square [1]). AGPD-CE is an untargeted attack, APGD-T and FAB-T are targeted attacks. The Square is a black-box attack. We use an official code to test models $\mathbb{I}$ .
Self-supervised learning. For self-supervised learning, we denote linear evaluation when we use only clean images to train the fully connected (fc) layer after the pretraining phase. When we denote robust linear evaluation, we train the fc layer with adversarial examples. While ACL uses partial fine-tuning to obtain their reported accuracy and robustness, to make a fair comparison, we freeze the encoder and train only the fc layer. Robust fine-tuning is training all parameters including parameters of the encoder with adversarial examples. For linear evaluation, we followed the baseline hyperparameters for each model. We train the baseline models with 150 epochs, 25 epochs, and 50 epochs for RoCL, and ACL, respectively. We also followed their learning rate of 0.1, 0.1, and  $2 \times 10^{-3}$  for RoCL, and ACL, respectively. On the other hand, we train our model with 100 epochs with a learning rate of 0.5 for linear evaluation. We use AT loss for robust linear evaluation except for ACL. For ACL, we use TRADES loss as the official code.
C Experimental Details of Analysis.
Analysis the distribution of target class. To analyze the target from the score function  $(S)$ , we employ an adversarially supervised trained model. We calculate the score function that is trained with our TARO on SimSiam. We use a train set. For each class, we calculate the mean predict probability, which is the average of all softmax outputs of target images from the supervised trained
$\mathbb{q}$  https://github.com/fra31/auto-attack
model. Further, we also count the number of samples that are predicted for each class. In Figure 2, the results are target images of the airplane as a base image. There is a similar tendency even though we change the base class to other classes as shown in the following Figure 4.
(a) Distribution of class of target of automobile
(b) Mean predict probability of automobile
(c) Distribution of class of target of deer
Figure 4: Analysis of target distribution in different classes
(d) Mean predict probability of deer
Visualization of embedding space. To visualize the embedding of our targeted attack and untargeted attack, we use t-Distributed Stochastic Neighbor Embedding (t-SNE) [5] with the cosine similarity metric. Our TARO model is trained on CIFAR-10 as a feature extractor. We sample a few examples and conduct two types of attack, the untargeted attack and the targeted attack. To visualize more effectively we ignore the other seven classes in CIFAR-10. We visualize clean examples from three classes and then visualize adversaries that are generated with our targeted attack and untargeted attack, respectively, with dark blue.
D Additional Experiment
Contrastive based adversarial self-supervised learning with TARO. Our TARO could be also applied to positive pairs in contrastive-based adversarial self-supervised learning (e.g., RoCL [23], ACL [22]). We applied our TARO in instance-wise attack of the contrastive-based approaches as follow,
$$
\mathcal {L} _ {\text {a t t a c k}} = \mathcal {L} _ {\text {n t - x e n t}} (x, \{\emptyset \}, \{x _ {\text {n e g}} \}) + \mathcal {L} _ {\text {s i m i l a r i t y}} (x, \{x _ {j _ {\text {T A R D}}} \}) \tag {14}
$$
where attack loss is consists of original attack loss nt-xent loss [6] and similarity loss. The similarity loss additionally constrains the positive pairs as the TARO that maximize the similarity between the  $x$  with the  $j^{th}$  index images which is searched by our TARO score function. Overall, we generate adversarial examples that maximizes the  $\mathcal{L}_{\mathrm{attack}}$  loss. Surprisingly, when we apply TARO on the contrastive learning based approach, previous work could achieve marginally better clean accuracy and robustness. This shows that our empirical assumption also holds on contrastive-based SSL but since there is (1/batch size) effects on the total loss the gain could be marginal.
Robustness against black box attack We conduct black box attack to verify our model is robust to gradient free attacks. We generate black box adversaries with AT [29] model, RoCL [23] model and our models. Then, we test adversaries to each other. As show in the table, our model is able to defend the black box attack from AT model than the RoCL model. Moreover, our model generates stronger black box adversaries than RoCL since AT model shows more weak robustness.
Table 9: Results of black box attack. Models on the row are the tested models. Models on the columns are the source models to generate black box adversaries.
AT	RoCL	Ours
AT	-	59.73	60.92
RoCL	70.40	-	57.98
Ours	69.97	54.99	-
Robustness against diverse attacks We tested our approach against diverse types of adversarial attacks, including the Carlini-Wagner (CW) attack [2], black-box attack, i.e., Pixle [33], and Patchattack, i.e., PIFGSM [13], as shown in Table 10. Since our approach already showed improved performance against Autoattack, which includes black
Table 10: Results against diverse attacks.
Method	PGD	CW	Pixle	PIFGSM
RoCL	42.89	76.45	67.32	43.23
+TARO	45.37	72.75	68.40	44.56
SimSiam	32.28	68.14	54.56	28.31
+TARO	44.97	73.87	67.22	46.37
box Square attacks, our approach is able to consistently demonstrates enhanced robustness against both the CW attack and black-box attacks.
E Proof of the Theorem
Let us consider the problem as a simple binary task using a linear layer model to demonstrate our theoretical motivation. The dataset  $\mathcal{D} = X$ ,  $\cdot$  consists of training examples, where  $x \in X$  represents a training example without any class label. We assume there is a single positive pair and a single negative pair. The linear model is denoted as  $f(\cdot)$ . The adversarial perturbations generated using both losses are as follows:
$$
x _ {\mathrm {n t - x e n t}} ^ {\mathrm {a d v}} = x + \arg \max  _ {\delta} \left\{- \log \frac {\left(\exp \left(\frac {f (x + \delta)}{\| f (x + \delta) \|} \cdot \frac {f (x)}{\| f (x) \|} / \tau\right)\right)}{\exp \left(\frac {f (x + \delta)}{\| f (x + \delta) \|} \cdot \frac {f \left(x _ {\mathrm {n e g}}\right)}{\| f \left(x _ {\mathrm {n e g}}\right) \|} / \tau\right)} \right\} \quad \text {s u b j e c t t o} \quad \| \delta \| \leq \epsilon \tag {15}
$$
where we approximate the cosine similarity distance loss into  $\ell_1$  distance function. In both cases, a  $\delta$  maximizes the respective loss, subject to the constraint that the norm of  $\delta$  is less than or equal to  $\epsilon$ . The objective in positive-only SSL is to make the perturbed and original samples dissimilar as follows,
$$
\delta_ {\mathbf {s s}} = \underset {\delta} {\arg \max } | f (x) - f (x + \delta) |, \tag {16}
$$
$$
\delta_ {\mathrm {n t - x e n t}} = \underset {\delta} {\arg \max } | f (x) - f (x + \delta) | - | f \left(x _ {n e g}\right) - f (x + \delta) |. \tag {17}
$$
The range of adversarial attack of each loss is then calculated as follow,
$$
\begin{array}{l} \| \delta_ {\mathbf {s s}} \| = \| \operatorname * {a r g m a x} _ {\delta} | f (x) - f (x + \delta) \| \\ = \arg \max  _ {\delta} (| f (x) - f (x + \delta) |) ^ {2} \tag {18} \\ \end{array}
$$
$$
\begin{array}{l} \| \delta_ {\mathrm {n t - x e n t}} \| = \| \operatorname * {a r g m a x} _ {\delta} \left(| f (x) - f (x + \delta) | - | f (x _ {\mathrm {n e g}}) - f (x + \delta) |\right) \| \\ = \underset {\delta} {\arg \max } \left(| f (x) - f (x + \delta) | ^ {2} - 2 | f (x) - f (x + \delta) | \cdot | f (x _ {\text {n e g}}) - f (x + \delta) |\right. \\ + \left| f \left(x _ {\text {n e g}}\right) - f (x + \delta) \right| ^ {2}) \\ \approx \underset {\delta} {\arg \max } \left(| f (x) - f (x + \delta) | ^ {2} - 2 | \delta | \cdot | f (x _ {\mathrm {n e g}}) - f (x + \delta) | + | f (x _ {\mathrm {n e g}}) - f (x + \delta) | ^ {2}\right) \\ \approx \underset {\delta} {\arg \max } \left(| f (x) - f (x + \delta) | ^ {2} + | f (x _ {\text {n e g}}) - f (x + \delta) | ^ {2}\right) \quad \because \delta \leq \epsilon \\ \geq \underset {\delta} {\arg \max } | f (x) - f (x + \delta) | ^ {2}. \tag {19} \\ \end{array}
$$
If there are more negative pairs, the difference in perturbation range between positive-pair-only attacks and contrastive attacks could become more pronounced.
Theorem E.1 (Perturbation range of self-supervised learning loss). Given a model trained under the positive-only distance loss, the adversarial perturbations  $\delta_{ss}$  are likely to be smaller than those perturbations  $\delta_{nt - xent}$  from a model trained under the positive-pair and negative-pair distance loss. Formally,  $\| \delta_{ss}\|_{\infty} < \| \delta_{nt - xent}\|_{\infty}$ .
When applying a random targeted attack within the positive-pair-only self-supervised learning framework, we can effectively increase the range of perturbations. Let us assume that the target instance  $x_{\mathrm{target}}$  is different from the original instance  $x$ , and the distance between them is greater than the threshold  $\delta$ . The perturbations generated through the targeted attack are as follows:
$$
\delta_ {\text {t a r g e t e d - a t t a c k}} = \underset {\delta} {\arg \max } | f (x + \delta) - f (x _ {\text {t a r g e t}}) |. \tag {20}
$$
Let us denote target instance  $x_{\text{target}}$  as  $x'$  for simple equations,
$$
\begin{array}{l} | f (x) - f (x + \delta) | <   | f \left(x ^ {\prime}\right) - f (x + \delta) | \\ \because | x ^ {\prime} - x | > \delta \\ = \left| f (x + \delta) - f \left(x ^ {\prime}\right) \right| \tag {21} \\ \therefore \| \arg \max  _ {\delta} | f (x) - f (x + \delta) | \| <   \| \arg \max  _ {\delta} | f (x + \delta) - f \left(x ^ {\prime}\right) \| \\ \end{array}
$$
The random targeted attack, which targets instances that are at a greater distance than  $\delta$  from the original input, can potentially increase the perturbation range and ultimately enhance overall robustness.
Theorem E.2 (Perturbation range of targeted attack). Given a model trained under the  $\mathcal{L}_{\text{targeted-attack}}$  loss, the adversarial perturbations  $\delta_{\text{targeted-attack}}$  are likely to be larger than those from a model trained under the  $\mathcal{L}_{ss}$ . Formally,  $\|\delta_{\text{targeted-attack}}\|_{\infty} > \|\delta_{ss}\|_{\infty}$ .
The pursuit of adversarial robustness against malicious attacks within deep neural networks remains an unsolved, yet fundamental area of deep learning research. To date, several self-supervised adversarial training approaches have been proposed, primarily based on the contrastive learning framework. However, the attainment of robustness via a 'positive-pair only' self-supervised learning approach is still under-explored. Consequently, self-supervised frameworks have evolved from large batch contrastive learning to a focus on single 'positive-pair only' learning paradigms. The area of self-supervised learning that we are targeting aims to delve into the robustness of these new learning frameworks through our tailored attacks. Furthermore, we believe that achieving superior robustness in self-supervised learning is a crucial research path towards achieving authentic robustness in representation. We hope that our work will inspire more research aimed at achieving generalizable robustness in unseen domains and datasets by leveraging the potential of various self-supervised frameworks.