SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} - \eta_ {t} \left\langle \nabla_ {\boldsymbol {w}} R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}), \frac {1}{M} \sum_ {i \in \mathcal {I} _ {B t}} \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} ^ {t}, x _ {i}) \right\rangle \\ = - \frac {\eta_ {t}}{M} \left\langle \nabla_ {\boldsymbol {w}} R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}), \sum_ {i \in \mathcal {I} _ {B ^ {t}} \cap \mathcal {I} _ {(j, x _ {0}, r)}} \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} ^ {t}, x _ {i}) \right\rangle \\ - \frac {\eta_ {t}}{M} \left\langle \nabla_ {\boldsymbol {w}} R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}), \sum_ {i \in \mathcal {I} _ {B ^ {t}} \backslash \mathcal {I} _ {(j, x _ {0}, r)}} \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w} ^ {t}, x _ {i}) \right\rangle \\ \leq - \frac {\beta \eta_ {t} \xi_ {t}}{M} \left\| \nabla_ {\boldsymbol {w}} R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) \right\| ^ {2} + \frac {(M - \xi_ {t}) \eta_ {t} G ^ {2}}{M} \\ \leq - \frac {2 \beta \mu \eta_ {t} \xi_ {t}}{M} \left(R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) - R _ {(j, x _ {0}, r)} ^ {*}\right) + \eta_ {t} G ^ {2}, \\ \end{array}
$$
choose  $\eta_t = \frac{1}{\beta \mu t}$ , due to  $T \leq \frac{L_1}{2 \beta \mu}$ ,  $\eta_t G^2 \leq \frac{L_1 \eta_t^2 G^2}{2}$ .
Note that  $\xi_t \sim \text{Hypergeometric}\left(n, \left|\mathcal{I}_{(j,x_0,r)}\right|, M\right)$ , by Hoeffding's inequality [59], we have
$$
\mathbb {P} \left(\xi_ {t} - \left| \mathcal {I} _ {(j, x _ {0}, r)} \right| \leq - M \varepsilon\right) \leq e ^ {- 2 M \varepsilon^ {2}},
$$
take  $\frac{\delta}{T} = e^{-2M\varepsilon^2}$ , we have
$$
\frac {\xi_ {t}}{M} \geq \frac {\left| \mathcal {I} _ {(j , x _ {0} , r)} \right|}{M} - \sqrt {\frac {\ln (T / \delta)}{2 M}}
$$
holds simultaneously for all  $t \in \{1, 2, \dots, T\}$  with probability at least  $1 - \delta$ , which means
$$
\begin{array}{l} - \frac {2 \beta \mu \eta_ {t} \xi_ {t}}{M} \left(R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) - R _ {(j, x _ {0}, r)} ^ {*}\right) \\ \leq - 2 \beta \mu \eta_ {t} \max  \left\{\left(\frac {\left| \mathcal {I} _ {(j , x _ {0} , r)} \right|}{M} - \sqrt {\frac {\ln (T / \delta)}{2 M}}\right), 0 \right\} \left(R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) - R _ {(j, x _ {0}, r)} ^ {*}\right). \\ \end{array}
$$
$$
\begin{array}{l} R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t + 1}) - R _ {(j, x _ {0}, r)} ^ {*} \\ \leq R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) - R _ {(j, x _ {0}, r)} ^ {*} \\ \leq \left(1 - \frac {2}{t} \max  \left\{\left(\frac {\left| \mathcal {I} _ {(j , x _ {0} , r)} \right|}{M} - \sqrt {\frac {\ln (T / \delta)}{2 M}}\right), 0 \right\}\right) \left(R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) - R _ {(j, x _ {0}, r)} ^ {*}\right) + \frac {L _ {1} G ^ {2}}{2 \beta^ {2} \mu^ {2} t ^ {2}} \\ = \left(1 - \frac {2}{t} \max  \left\{\left(\frac {\rho_ {(j , x _ {0} , r)} \pi^ {d / 2} r ^ {d}}{\Gamma \left(\frac {d}{2} + 1\right) M} - \sqrt {\frac {\ln (T / \delta)}{2 M}}\right), 0 \right\}\right) \left(R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) - R _ {(j, x _ {0}, r)} ^ {*}\right) + \frac {L _ {1} G ^ {2}}{2 \beta^ {2} \mu^ {2} t ^ {2}} \\ \end{array}
$$
holds simultaneously for all  $t\in \{1,2,\ldots ,T\}$  with probability at least  $1 - \delta$
Under Assumption 1, due to the Lipschitz continuous gradient, for  $\forall (x_i,y_i)\in S$ , we have
$$
\ell (\boldsymbol {w}, x _ {i}) \geq \ell (\boldsymbol {w}, x) + \left\langle \nabla_ {x} \ell (\boldsymbol {w}, x), x _ {i} - x \right\rangle - \frac {L _ {2}}{2} \| x _ {i} - x \| ^ {2}, \tag {4}
$$
$$
\ell (\boldsymbol {w}, x _ {i}) \leq \ell (\boldsymbol {w}, x) + \left\langle \nabla_ {x} \ell (\boldsymbol {w}, x), x _ {i} - x \right\rangle + \frac {L _ {2}}{2} \| x _ {i} - x \| ^ {2} \tag {5}
$$
$$
\ell (\boldsymbol {w}, x) \leq \ell (\boldsymbol {w}, x _ {i}) + \left\langle \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}), x - x _ {i} \right\rangle + \frac {L _ {2}}{2} \| x _ {i} - x \| ^ {2} \tag {6}
$$
holds for  $\forall x\in \mathcal{X}$ . By (4), choose  $x = x_{i} - r\frac{\nabla_{x}\ell(\boldsymbol{w},x)}{\|\nabla_{x}\ell(\boldsymbol{w},x)\|}$ , which satisfies  $\| x_i - x\| \leq r$ , then we have
$$
\begin{array}{l} \ell \left(\boldsymbol {w}, x _ {i}\right) \geq \ell \left(\boldsymbol {w}, x _ {i} - r \frac {\nabla_ {x} \ell (\boldsymbol {w} , x)}{\| \nabla_ {x} \ell (\boldsymbol {w} , x) \|}\right) + r \| \nabla_ {x} \ell (\boldsymbol {w}, x) \| - \frac {L _ {2} r ^ {2}}{2} \\ \geq r \left\| \nabla_ {x} \ell (\boldsymbol {w}, x) \right\| - \frac {L _ {2} r ^ {2}}{2} \\ = r \left\| \nabla_ {x} \ell (\boldsymbol {w}, x) - \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) + \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \right\| - \frac {L _ {2} r ^ {2}}{2} \\ \geq r \left(\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \| - \| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) - \nabla_ {x} \ell (\boldsymbol {w}, x) \|\right) - \frac {L _ {2} r ^ {2}}{2} \\ \geq r \left(\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \| - L _ {2} \| x _ {i} - x \|\right) - \frac {L _ {2} r ^ {2}}{2} \\ \geq r \left\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \right\| - \frac {3 L _ {2} r ^ {2}}{2}, \\ \end{array}
$$
$$
\left\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \right\| \leq \frac {\ell (\boldsymbol {w} , x _ {i})}{r} + \frac {3 L _ {2} r}{2}. \tag {7}
$$
Use (5), we have
$$
\begin{array}{l} \ell (\boldsymbol {w}, x _ {i}) - \ell (\boldsymbol {w}, x) \leq \left\langle \nabla_ {x} \ell (\boldsymbol {w}, x), x _ {i} - x \right\rangle + \frac {L _ {2}}{2} \| x _ {i} - x \| ^ {2} \\ \leq r \left\| \nabla_ {x} \ell (\boldsymbol {w}, x) \right\| + \frac {L _ {2} r ^ {2}}{2} \\ \leq r \left(\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \| + \| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) - \nabla_ {x} \ell (\boldsymbol {w}, x) \|\right) + \frac {L _ {2} r ^ {2}}{2} \\ \leq r \left\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \right\| + \frac {3 L _ {2} r ^ {2}}{2}, \\ \end{array}
$$
then use (6), we have
$$
\begin{array}{l} \ell (\boldsymbol {w}, x) - \ell (\boldsymbol {w}, x _ {i}) \leq \left\langle \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}), x - x _ {i} \right\rangle + \frac {L _ {2}}{2} \| x _ {i} - x \| ^ {2} \\ \leq r \left\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \right\| + \frac {L _ {2} r ^ {2}}{2} \\ \end{array}
$$
$$
\frac {\left| \ell (\boldsymbol {w} , x _ {i}) - \ell (\boldsymbol {w} , x) \right|}{r} - \frac {3 L _ {2} r}{2} \leq \| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \|. \tag {8}
$$
Combining (7) and (8), we get
B Details of Pre-training Embeddings
In actual experiments, it was found that if the embedding is not pre-trained first, the model convergence speed will be quite slow. We tried to freeze the parameters of the embedding layer after pre-training multiple epochs, but found that even with the same number of epochs each pre-training, the internal structures between the embedding vectors are very different, and there may be clustering between different class embedding (as shown in Figure6), the distance between multiple classes is very close, which leads to redundant information from many other classes in embedding.
To address this issue and enable the generator's embeddings to better reflect class information, we employ the following loss, utilizing the feature outputs of the source model as guidance for embedding pretraining:
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {M}} = \sum_ {i, j} \overline {{M}} _ {i, j} ^ {S _ {\mathrm {e u c}}} \log \frac {\overline {{M}} _ {i , j} ^ {S _ {\mathrm {e u c}}}}{\overline {{M}} _ {i , j} ^ {E _ {\mathrm {e u c}}}} + \sum_ {i, j} \overline {{M}} _ {i, j} ^ {E _ {\mathrm {e u c}}} \log \frac {\overline {{M}} _ {i , j} ^ {E _ {\mathrm {e u c}}}}{\overline {{M}} _ {i , j} ^ {S _ {\mathrm {e u c}}}} \\ + \sum_ {i, j} \overline {{M}} _ {i, j} ^ {S _ {\mathrm {c o s}}} \log \frac {\overline {{M}} _ {i , j} ^ {S _ {\mathrm {c o s}}}}{\overline {{M}} _ {i , j} ^ {E _ {\mathrm {c o s}}}} + \sum_ {i, j} \overline {{M}} _ {i, j} ^ {E _ {\mathrm {c o s}}} \log \frac {\overline {{M}} _ {i , j}}{\overline {{M}} _ {i , j}}. \\ \end{array}
$$
However, directly optimizing the  $\mathcal{L}_M$  leads to the issue of loss collapse. We analyse each component of the loss. First, note that for  $\forall i,j\in [K], - 1\leq M_{i,j}^{E_{\mathrm{cos}}}\leq 1, - 1\leq M_{i,j}^{S_{\mathrm{cos}}}\leq 1.$  Then
$$
\begin{array}{l} \overline {{M}} _ {i, j} ^ {E _ {\mathrm {c o s}}} = \frac {\exp \left(M _ {i , j} ^ {E _ {\mathrm {c o s}}}\right)}{\sum_ {k = 1} ^ {K} \exp \left(M _ {i , k} ^ {E _ {\mathrm {c o s}}}\right)} = \frac {1}{1 + \sum_ {k \neq j} \exp \left(M _ {i , k} ^ {E _ {\mathrm {c o s}}} - M _ {i , j} ^ {E _ {\mathrm {c o s}}}\right)} \geq \frac {1}{1 + (K - 1) \exp (2)}, \\ \overline {{M}} _ {i, j} ^ {S _ {\mathrm {c o s}}} = \frac {\exp \left(M _ {i , j} ^ {S _ {\mathrm {c o s}}}\right)}{\sum_ {K = 1} ^ {K} \exp \left(M _ {i , k} ^ {S _ {\mathrm {c o s}}}\right)} = \frac {1}{1 + \sum_ {k \neq j} \exp \left(M _ {i , k} ^ {S _ {\mathrm {c o s}}} - M _ {i , j} ^ {S _ {\mathrm {c o s}}}\right)} \geq \frac {1}{1 + (K - 1) \exp (2)}. \\ \end{array}
$$
Under the settings in this paper,  $K = 10$ , so both terms analyse above are positive with lower bounds. In the optimization process, their ratio will not approach zero. As for  $\overline{M}_{i,j}^{S_{\mathrm{euc}}}$ , it is a deterministic positive constant. Therefore, when it serves as the numerator, it will not lead to a ratio close to zero inside the logarithm. However, for  $\overline{M}_{i,j}^{E_{\mathrm{euc}}}$ , if it becomes close to zero when used as the numerator, it will cause loss collapse. Note that
$$
\overline {{M}} _ {i, j} ^ {E _ {\mathrm {e u c}}} = \frac {\exp \left(M _ {i , j} ^ {E _ {\mathrm {e u c}}}\right)}{\sum_ {k = 1} ^ {K} \exp \left(M _ {i , k} ^ {E _ {\mathrm {e u c}}}\right)} = \frac {1}{1 + \sum_ {k \neq j} \exp \left(M _ {i , k} ^ {E _ {\mathrm {e u c}}} - M _ {i , j} ^ {E _ {\mathrm {e u c}}}\right)} \geq \frac {1}{K \max _ {1 \leq k \leq K} \exp \left(M _ {i , k} ^ {E _ {\mathrm {e u c}}} - M _ {i , j} ^ {E _ {\mathrm {e u c}}}\right)}.
$$
Figure 6: The top row displays the result of directly trained for 9 epochs, while the bottom row shows the result of directly trained for 49 epochs. The numbers  $0 - 9$  represent different class labels in sequential order. The two heatmaps on the left depict the Euclidean distance and cosine similarity between the embeddings of each class. The two images on the right display the distances after PCA projection onto a 2-dimensional plane (normalized and unnormalized, representing cosine similarity and Euclidean distance, respectively).
To prevent  $\overline{M}_{i,j}^{E_{\mathrm{euc}}}$  from becoming too small, we need to ensure:
$$
\frac{1}{K\max_{1\leq k\leq K}\exp\left(M_{i,k}^{E_{\mathrm{euc}}} - M_{i,j}^{E_{\mathrm{euc}}}\right)} >0,
$$
which means we need to ensure that
$$
\max  _ {1 \leq k \leq K} \exp \left(M _ {i, k} ^ {E _ {\mathrm {e u c}}}\right) \neq + \infty , \forall 1 \leq i \leq K,
$$
since  $M_{i,k}^{E_{\mathrm{euc}}}, M_{i,j}^{E_{\mathrm{euc}}}$  are non-negative. Also, note that  $M_{i,k}^{E_{\mathrm{euc}}} = |e^i - e^k| \leq |e^i| + |e^k|$ . Therefore, by imposing the following constraint in the optimization:
$$
\left\{ \begin{array}{l} \min  \mathcal {L} _ {\mathcal {M}} \\ s. t.   \left\| e ^ {i} \right\| \leq U, 1 \leq i \leq K \end{array} \right.
$$
We can ensure that  $M_{i,k}^{E_{\mathrm{euc}}} \leq 2U$ , which in turn guarantees  $\max_{1 \leq k \leq K} \exp M_{i,k}^{E_{\mathrm{euc}}} \neq +\infty, \forall 1 \leq i \leq K$  thus avoiding the issue of  $\overline{M}_{i,j}^{E_{\mathrm{euc}}}$  becoming too small and causing loss collapse.
By explicitly incorporating the above constraint as a regularizer into the optimization objective, we obtain the final manifold matching loss:
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {M}} = \sum_ {i, j} \overline {{M}} _ {i, j} ^ {S _ {\mathrm {e u c}}} \log \frac {\overline {{M}} _ {i , j} ^ {S _ {\mathrm {e u c}}}}{\overline {{M}} _ {i , j} ^ {E _ {\mathrm {e u c}}}} + \sum_ {i, j} \overline {{M}} _ {i, j} ^ {E _ {\mathrm {e u c}}} \log \frac {\overline {{M}} _ {i , j} ^ {E _ {\mathrm {e u c}}}}{\overline {{M}} _ {i , j} ^ {S _ {\mathrm {e u c}}}} \\ + \lambda_ {1} \left[ \sum_ {i, j} \overline {{M}} _ {i, j} ^ {S _ {\cos}} \log \frac {\overline {{M}} _ {i , j} ^ {S _ {\cos}}}{\overline {{M}} _ {i , j} ^ {E _ {\cos}}} + \sum_ {i, j} \overline {{M}} _ {i, j} ^ {E _ {\cos}} \log \frac {\overline {{M}} _ {i , j}}{\overline {{M}} _ {i , j}} \right] + \lambda_ {2} \sum_ {i = 1} ^ {K} \left\| e ^ {i} \right\|. \\ \end{array}
$$
In our experiments, we use the AdamW optimizer with a learning rate of  $1.5e - 5$  for 15000 epochs to optimize the embeddings, while setting  $\lambda_{1}$  to 5 and  $\lambda_{2}$  to 0.01. The results, as shown in Figure 7, demonstrate a significant improvement in the clustering of embeddings between different classes.
The trained embedding is denoted as  $e_j \in \mathbb{R}^{d_1}, j \in [K]$ , where  $e_j \in \mathbb{R}^{d_1}, j \in [K]$ . For the utilized Resblock, let the input and output channels be  $C_{in}$  and  $C_{out}$  respectively. The fully connected layer
Figure 7: The results obtained by optimizing  $\mathcal{L}_{\mathcal{M}}$
$\mathrm{MLP}(\cdot) \in \mathbb{R}^{d_1} \to \mathbb{R}^{C_{out}}$ . Firstly, we employ the initial convolution kernel  $\mathrm{Conv}_1$  to map the raw data from the input  $h$  of the current layer, transitioning it from  $[C_{in}, H, W]$  to  $[C_{out}, H, W]$ . Then we perform an addition operation between this mapped result and the embedding transformation obtained through the MLP. Formally expressed as:
$$
h ^ {\prime} = \mathrm {C o n v} _ {1} (h) + \mathrm {M L P} (e _ {y})
$$
where  $y$  is the label of the input. Subsequently, we apply a second convolution to this output, followed by a LayerNorm operation, and finally, pass it through an attention operation before adding it to the residual. Formally stated as:
$$
\operatorname {o u t p u t} = \operatorname {A t t e n t i o n} (\operatorname {L N} \left(\operatorname {C o n v} _ {2} \left(h ^ {\prime}\right)\right) + h)
$$
C Additional Ablation Studies
To investigate the impact of different settings during the training phase, we conduct several experiments. First, we investigate the output consistency of the model for different values of  $q$  used for sample screening, as shown in Figure 8. A smaller  $q$  implies a stricter selection, which leads to an increase in output consistency (in terms of mean), but also results in a larger standard deviation. In this case,  $q = 5$  is a relatively suitable choice.
Then we validate the impact of different choices of  $d(\cdot, \cdot)$  on targeted transferability. The results are shown in Figure 9, where  $d(\cdot, \cdot)$  was chosen as the Smooth L1 loss, which significantly improves the targeted transferability compared to L1 and L2 losses.
In the exploratory experiments conducted in Section 2, the three different network architectures we employed are depicted in Table 4.
Table 4: Architectures of NN Classifiers
Model	Hidden Layer 1	Hidden Layer 2	Hidden Layer 3
Model 1	Linear(2,500)ReLU	Linear(500,500)ReLU	Linear(500,2)
Model 2	Linear(2,50)ReLU	Linear(50,100)ReLU	Linear(100,150)ReLULinear(150,2)
Model 3	Linear(2,20)ReLU	Linear(20,20)ReLU	Linear(20,20)ReLULinear(20,2)
For the experiments on CIFAR10 mentioned in Section 2, we used the PyTorch framework. The optimizer is SGD, with an initial learning rate of 0.1, weight decay of  $5e - 4$ , and momentum of 0.9. We apply cosine annealing learning rate decay, with a maximum number of epochs ( $T_{max}$ ) set to 200. Tolerance of Early-stopping is set to 30. Additionally, we used a batch size of 128 during training and applied the following data augmentations:
- Randomly crops the input image to a size of  $32 \times 32$  pixels with padding of 4 pixels.
Figure 8: Density plots of output similarity between models under different  $q$  selections. For the screened data, we calculate the cosine similarity between the output of the source model and the output of all target models, and take the average to represent the output similarity of the data across different models. "q=all" indicates no screening, and the dashed line represents the average for different values of  $q$ .
Figure 9: Targeted transferability (Src:Res50) with different chosen of  $d(\cdot, \cdot)$ .
- Randomly flips the input image horizontally with a probability of 0.5.
After data augmentation, the input is then transformed into a tensor and normalized with a mean of (0.4914, 0.4822, 0.4465) and a standard deviation of (0.2023, 0.1994, 0.2010).
For the experiments in Section 4, we used the structure of following models: ResNet-50, VGG19bn, DenseNet-121, ResNet-152 from the torchvision library, and others from the timm library. The torchattacks library is utilized for MIM, SI-NIM, TIM, and DIM.
E.1 Ensemble Model as Source
In Table 5, we report the results of the case of an ensemble model as the source model. The selected ensemble model are consist of three models: Res-50, Inc-v3, and Inc-v4, while the other models
<https://github.com/pytorch/vision>
$^{2}$ https://github.com/huggingface/pytorch-image-models
<sup>3</sup>https://github.com/Harry24k/adversarial-attacks-pytorch
not involved in the ensemble are treated as target models. ESMA still demonstrates outstanding adversarial transferability.
Table 5: Targeted transfer success rates. The source model is an ensemble of Res-50, Inc-v3, and Inc-v4.
Attack	Src: Ensemble
→VGG19bn	→Dense121	→Res152	→IncRes-v2	→ViT	AVG
MIM	1.76%	3.40%	3.29%	1.07%	0.22%	1.95%
TIM	9.96%	16.13%	14.76%	8.27%	1.11%	10.05%
DIM	10.27%	16.31%	15.22%	8.20%	0.87%	10.17%
SI-NIM	1.16%	1.67%	1.49%	0.93%	0.13%	1.08%
Po-TI-Trip	14.22%	21.76%	18.69%	12.09%	1.62%	13.68%
SI-FGS2M	15.72%	24.84%	20.79%	12.20%	1.53%	15.02%
S2I-SI-TI-DIM	17.82%	21.07%	21.36%	16.38%	2.38%	15.80%
Logit	16.09%	27.56%	17.51%	36.96%	2.20%	20.06%
DTMI-Logit-SU	18.11%	30.16%	18.73%	38.64%	2.64%	21.71%
RAP-LS	48.98%	72.91%	70.67%	26.53%	4.22%	44.66%
HGN	55.10%	67.37%	63.54%	49.85%	26.78%	52.53%
TTP(129257.2)	60.02%	71.33%	67.44%	55.42%	30.04%	56.85%
ESMA(112303.5)	62.51%	74.11%	72.61%	62.33%	32.70%	60.85%
E.2 Models Trained on Different Distributions as Victim
Given that our discussion is based on the condition that the source and target models are trained on the same distribution data, to explore the transferability between models trained on training data with different distribution, we conducted additional transfer attacks on two adversarially trained models, Inc-v3-adv and IncRes-v2-ens. The results are reported in Table 6. ESMA's performance is slightly weakened but still competitive. Considering that this validation is not directly related to the viewpoints, conclusions, and statements of our paper, we will further discuss the transferability on target models trained on data with distribution shift in our future work.
Table 6: Targeted transfer success rates. "Src" indicates the source model.
Src	Target	MIM	TIM	DIM	SI-NIM	Po-TI-Trip	SI-FGS2M	S2I-SI-TI-DIM	Logit	DTMI-Logit-SU	RAP-LS	HGN	TTP	ESMA(Ours)
Res50	→Inc-v3adv	0.11%	0.36%	0.18%	0.11%	0.22%	0.24%	0.76%	0.20%	0.27%	0.38%	2.12%	3.53%	3.23%
→IncRes-v2ens	0.11%	0.22%	0.18%	0.02%	0.38%	0.30%	0.89%	0.18%	0.22%	0.27%	2.70%	4.53%	3.98%
Src	Target	MIM	TIM	DIM	SI-NIM	Po-TI-Trip	SI-FGS2M	S2I-SI-TI-DIM	Logit	DTMI-Logit-SU	RAP-LS	HGN	TTP	ESMA(Ours)
Dense121	→Inc-v3adv	0.13%	0.13%	0.18%	0.11%	0.18%	0.22%	0.56%	0.20%	0.20%	0.22%	1.88%	3.18%	1.73%
→IncRes-v2ens	0.09%	0.18%	0.13%	0.11%	0.31%	0.20%	0.84%	0.22%	0.20%	0.18%	2.39%	3.56%	1.83%
Src	Target	MIM	TIM	DIM	SI-NIM	Po-TI-Trip	SI-FGS2M	S2I-SI-TI-DIM	Logit	DTMI-Logit-SU	RAP-LS	HGN	TTP	ESMA(Ours)
VGG19bn	→Inc-v3adv	0.13%	0.11%	0.16%	0.11%	0.11%	0.22%	0.16%	0.13%	0.13%	0.18%	0.20%	0.49%	0.31%
→IncRes-v2ens	0.04%	0.07%	0.09%	0.01%	0.11%	0.13%	0.22%	0.07%	0.07%	0.09%	0.53%	0.60%	0.22%
Src	Target	MIM	TIM	DIM	SI-NIM	Po-TI-Trip	SI-FGS2M	S2I-SI-TI-DIM	Logit	DTMI-Logit-SU	RAP-LS	HGN	TTP	ESMA(Ours)
Res152	→Inc-v3adv	0.13%	0.20%	0.20%	0.04%	0.40%	0.27%	0.93%	0.24%	0.26%	0.24%	4.26%	5.96%	2.31%
→IncRes-v2ens	0.07%	0.36%	0.27%	0.07%	0.56%	0.33%	1.02%	0.27%	0.31%	0.38%	4.55%	5.96%	3.68%
Src	Target	MIM	TIM	DIM	SI-NIM	Po-TI-Trip	SI-FGS2M	S2I-SI-TI-DIM	Logit	DTMI-Logit-SU	RAP-LS	HGN	TTP	ESMA(Ours)
Ensemble	→Inc-v3adv	0.20%	0.60%	0.40%	0.13%	0.69%	0.60%	2.51%	0.53%	0.53%	0.58%	22.67%	24.44%	17.00%
→IncRes-v2ens	0.11%	0.96%	0.42%	0.09%	0.87%	0.66%	2.51%	0.20%	0.24%	0.58%	21.25%	23.33%	20.82%
F Fooling Google Lens
We show that adversarial examples perturbed using ESMA can deceive Google Lens in image search, as shown in Figure 10 and 11.
(a) French Bulldog
(b) Google Lens: French Bulldog
(c) ESMA (Target:Goose)
(d) Google Lens: Goose
(e) ESMA (Target:Hippo)
(f) Google Lens: Hippo
Figure 10: The original image is of a French Bulldog, which can be intentionally perturbed to yield different search results, such as a goose in the second row and a hippopotamus in the third row.
(a) Model T
(b) Google Lens: Model T
(c) ESMA (Target:Cannon)
(d) Google Lens: Cannon
(e) ESMA (Target:Grey Owl)
(f) Google Lens: Grey Owl
Figure 11: The original image is of a Ford Model T, which can be intentionally perturbed to yield different search results, such as a cannon in the second row and a gray owl in the third row.