SlowGuess/NeurIPS_2023_OCR huggingface dataset:
According to Table 1, punctuation-level attacks require considerable cost to deploy. Even considering only the single punctuation attack, it is still time-consuming. Taking the mode of Insertion as an example, its original time complexity is  $\mathcal{O}(kn)$ . Therefore, it is essential to decrease the search cost. If we have reduced the search cost of a single punctuation attack, the time complexity of multiple-punctuation attacks can be reduced by using either a greedy algorithm or a beam search. The  $n$  is determined by the target text  $x$ , which is not feasible to optimize. As  $k$  increases, the search space expands, resulting in a higher fooling rate of Insertion attacking. Therefore, to achieve a higher fooling rate, we should not reduce the value of  $k$ .
As we cannot decrease  $k$  and  $n$ , we focus on a specific process: text function  $f(\pmb{x})$ . For example, in a text classification task, function  $f(\pmb{x}) = \text{softmax } f_{fe}(\pmb{x})$ , where the processing of feature extraction is denoted by the function  $f_{fe}$  and softmax is used as the final classification layer. The classification cost  $t_0$  can be divided into two parts: the feature extraction consumption  $t_1$  and the final classification consumption  $t_2$ . The classification time  $t_2$  is transient compared to  $t_1$  and therefore can be ignored. Consequently, we have  $t_0 \approx t_1$ .
Embedding the adversarial candidate text  $\pmb{x}_{adv}$  perturbed by various configurations requires multiple queries, and thus it is time-consuming. Therefore, we propose TPPE to quickly and reasonably embed the adversarial candidate text  $\pmb{x}_{adv}$ . According to Eq. (5), a single punctuation attack consists of three components: input text  $\pmb{x}$ , perturbed position, and candidate punctuation. To reduce the cost of embedding processing for adversarial text  $\pmb{x}_{adv}$ , we refine its embedding process by combining the embeddings of text, position, and punctuation. Again, we take the Insertion attack mode as an example. When we insert punctuations  $p$  into the  $i$ -th position of the target text, we query text function  $f(\pmb{x})$  (white-box) or substitute function  $f_{sub}(\pmb{x})$  (black-box), obtaining  $E_{text}$  as the embedding of text and  $f_{fe}(\pmb{p})$  as the embedding of punctuation  $p$ . The embedding of punctuation  $p$  only needs to be queried once. For the embedding of the position, we adopt the embedding of the static position based on the calculation of sine and cosine from the Transformer [40], present in Eq. (7). The variable  $pos$  represents the position and the variable  $i$  represents the dimension.
$$
P E _ {(p o s, 2 i)} = \sin \left(p o s / 1 0 0 0 0 ^ {2 i / d _ {\mathrm {m o d e l}}}\right), P E _ {(p o s, 2 i + 1)} = \cos \left(p o s / 1 0 0 0 0 ^ {2 i / d _ {\mathrm {m o d e l}}}\right). \tag {7}
$$
After calculating embeddings of input text, punctuation, and position, we calculate candidate attacking text embeddings using two steps: 1) concatenating three embeddings as candidate attacking text embedding  $\mathbf{E}_{all} = \text{Concat}(\mathbf{E}_{text}, \mathbf{E}_{pos}, \mathbf{E}_{punc})$  and, 2) calculating candidate attacking text embedding using mathematical operations presented in Table 3.
Table 4: The quantitative results of single punctuation attack in the T2I task. "Ori-text0" indicates the original text "a professional photograph of an astronaut riding a triceratops", and "Ori-text1" indicates the original text "a corgi is playing piano, oil on canvas".
dataset	pokemon-blip-captions
all	train	test	Ori-image	Adv-image	Ori-text	Adv-text
Ori-text	0.3273	0.3272	0.3278	Ori-text0	0.3281	0.2484	1	0.9782
Adv-text	0.2591	0.2586	0.2610	Ori-text1	0.4040	0.3468	1	0.9843
The Insertion mode inserts punctuation  $p$  into the  $i$ -th position of input text, so we regard  $E_{text} + E_{pos} + E_{punc}$  as the embedding of  $x_{adv}$ . The Displacement mode displaces punctuation  $p$  from position  $i$  to position  $j$  in the input text, so we regard  $E_{text} - E_{pos}^{i} + E_{pos}^{j} + E_{punc}$  as the embedding of  $x_{adv}$ . The Deletion mode deletes punctuation  $p$  from the  $i$  th position of the input text, so we compute  $E_{text} + E_{pos} - E_{punc}^{i} + E_{punc}^{j}$  as the embedding of  $x_{adv}$ . The Replacement mode replaces punctuation  $p_i$  with  $p_j$  in the input text, so we take  $E_{text} + E_{pos} - E_{punc}^{i} + E_{punc}^{j}$ . TPPE requires only single-shot embedding of the input text  $x$ , which means we decrease the query time complexity from  $\mathcal{O}(kn)$  of Insertion,  $\mathcal{O}(nt)$  of Displacement,  $\mathcal{O}(t)$  of Deletion, and  $\mathcal{O}(kt)$  of Replacement, to  $\mathcal{O}(1)$  under single punctuation attack.
In our study, we assume the worst-case of applying punctuation-level attacks: the victim model is a black-box model where only prediction labels are available instead of scores from the function. In this scenario, we train a substitute function  $f_{sub}$  [30] to transform the black-box scenario into a white-box scenario by collecting part of training datasets  $X$ . Specifically, we query the text function and derive the prediction labels  $y_{pre}$ , then train the substitute function  $f_{sub}$  using the input text and the label  $y_{pre}$  as paired data. After training  $f_{sub}$ , we transform the black-box scenario into a white-box scenario.
Directly querying  $f_{sub}$  to determine which punctuation should be used is also time-consuming for multiple queries. We iteratively attack the input text  $x$  and quickly gain the adversarial text  $x_{adv}$  after training the classification model from the TPPE of  $x_{adv}$  to the label of  $x_{adv}$  by  $f(x)$ . Additionally, we propose a search method called TPPEP to achieve a single-shot attack. We analyze the worst-case scenario for TPPEP: zero query, black-box function, hard-label output, single punctuation limitation, and single-shot attack. The specific pseudo code of TPPE and TPPEP are presented in Alg. 1 and Alg. 2 in the Appendix.
To achieve the goal of zero query and gain more information, we train a substitute function  $f_{sub}$  to fit the text function  $f$ . Then, we query the substitute function  $f_{sub}$  to gain the embedding of text  $E_{text}$ . We apply the TPPE method to gain the embedding of adversarial candidate text  $x_{adv}$ , denoted as  $E_{x_{adv}}$ . We transform the attacking task into a paraphrasing task. Specifically,  $E_{x_{adv}}$  and  $E_{text}$  are concatenated as input data, and the result of the attack (the successful attack denotes label 1; else the label 0) is the predicting label. The adversarial candidate text with the highest paraphrasing score calculated by the TPPEP method is chosen to deploy the attack.
3.3 Analysis of TPPEP
In this section, we discuss a specific and practical form of classification function that can deceive the text model. We assume that there is a function  $f(\pmb{x})$  in hypothesis space  $\mathcal{F}$ . Since  $f(\pmb{x}) = \text{softmax}(f_{fe}(\pmb{x}))$ , we hypothesize that there exists a function  $f_1$  that fits the function softmax and satisfies  $|f_1(\pmb{x}) - \widehat{y}_\pmb{x}| < \epsilon_1$ . A simple functional form of  $f_1$  is a function of arg max softmax. Furthermore, we propose the hypothesis that there exists a function  $f_2$ , which can predict  $\widehat{y}_{\pmb{x}_{adv}}$  from TPPE  $(\pmb{x}_{adv})$  and satisfy  $|f_2(\pmb{x}_{adv}) - \widehat{y}_{\pmb{x}_{adv}}| < \epsilon_2$ . We also define a function  $f_3$ ,  $f_3(\pmb{x}_{adv}, \pmb{x}) = \begin{cases} 1 & \text{if } |f_2(\pmb{x}_{adv}) - f_1(\pmb{x})| \geq 1 \\ 0 & \text{otherwise} \end{cases}$ . This means we can predict the result of an attack by  $|f_2(\pmb{x}) - f_1(\pmb{x})|$ . We calculate the absolute difference between  $f_2(\pmb{x}_{adv})$  and  $f_1(\pmb{x})$  and compare the absolute difference between  $|f_2(\pmb{x}_{adv}) - \widehat{y}_{\pmb{x}_{adv}}|$  and  $|\widehat{y}_{\pmb{x}_{adv}} - \widehat{y}_\pmb{x}|$  in Eq. (8) as
$$
\begin{array}{l} \left| f _ {2} \left(\boldsymbol {x} _ {a d v}\right) - f _ {1} (\boldsymbol {x}) \right| = \left| f _ {2} \left(\boldsymbol {x} _ {a d v}\right) - \widehat {y} _ {\boldsymbol {x} _ {a d v}} + \widehat {y} _ {\boldsymbol {x} _ {a d v}} - f _ {1} (\boldsymbol {x}) + \widehat {y} _ {\boldsymbol {x}} - \widehat {y} _ {\boldsymbol {x}} \right| \\ = \left| \left[ f _ {2} \left(\boldsymbol {x} _ {a d v}\right) - \widehat {y} _ {\boldsymbol {x} _ {a d v}} \right] + \left[ \widehat {y} _ {\boldsymbol {x} _ {a d v}} - \widehat {y} _ {\boldsymbol {x}} \right] + \left[ \widehat {y} _ {\boldsymbol {x}} - f _ {1} (\boldsymbol {x}) \right] \right| \tag {8} \\ \leq \varepsilon_ {1} + \varepsilon_ {2} + \left| \widehat {y} _ {\boldsymbol {x} _ {a d v}} - \widehat {y} _ {\boldsymbol {x}} \right|. \\ \end{array}
$$
According to Eq. (8), when  $\widehat{y}_{\boldsymbol{x}_{adv}} = \widehat{y}_{\boldsymbol{x}}$ , the actual label of TPPEP is 0 and  $|f_{2}(\boldsymbol{x}_{adv}) - f_{1}((\boldsymbol{x}))| \leq \varepsilon_{1} + \varepsilon_{2}$ . Since  $\varepsilon_{1}$  and  $\varepsilon_{2}$  are sufficiently small,  $f_{3}(f_{2}(\boldsymbol{x}_{adv}), f_{1}(\boldsymbol{x}))$  will be 0, allowing us to classify them correctly. As  $f_{1}(x)$  and  $f_{2}(x)$  fit well  $\widehat{y}_{\boldsymbol{x}_{adv}}$  and  $\widehat{y}_{\boldsymbol{x}}$ , when  $\widehat{y}_{\boldsymbol{x}_{adv}}$  and  $\widehat{y}_{\boldsymbol{x}}$  are not equal,  $f_{3}(f_{2}(\boldsymbol{x}_{adv}), f_{1}(\boldsymbol{x}))$  will be 1. Thus, we have found a specific fooling function  $f_{3}(f_{2}(\boldsymbol{x}_{adv}), f_{1}(\boldsymbol{x}))$ , which implies that there is at least one fooling function in the hypothesis space  $\mathcal{F}$ . Moreover, we discuss that  $f_{3}(f_{2}(\boldsymbol{x}_{adv}), f_{1}(\boldsymbol{x}))$  can be learned based on the Universal Approximation Theorem [16]. According to Eq. (1), we can train  $F_{1}$  and  $F_{2}$  to fit  $f_{1}$  and  $f_{2}$ , which satisfy  $|F_{1}(\boldsymbol{x}) - f_{1}(\boldsymbol{x})| < \epsilon_{3}$  and  $|F_{2}(\boldsymbol{x}_{adv}) - f_{2}(\boldsymbol{x}_{adv})| < \epsilon_{4}$ . We analyze the difference between  $|f_{2}(\boldsymbol{x}_{adv}) - f_{1}(\boldsymbol{x})|$  and  $|F_{2}(\boldsymbol{x}_{adv}) - F_{1}(\boldsymbol{x})|$  in Eq. (9). If the difference between them is less than 1, then  $f_{3}(F_{2}(\boldsymbol{x}_{adv}), F_{1}(\boldsymbol{x}))$  and  $f_{3}(f_{2}(\boldsymbol{x}_{adv}), f_{1}(\boldsymbol{x}))$  will predict the same label.
$$
\begin{array}{l} \left. \left| \left[ F _ {2} \left(\boldsymbol {x} _ {a d v}\right) - F _ {1} (\boldsymbol {x}) \right] - \left[ f _ {2} \left(\boldsymbol {x} _ {a d v}\right) - f _ {1} (\boldsymbol {x}) \right] \right| = \left| \left[ F _ {1} (\boldsymbol {x}) - f _ {1} (\boldsymbol {x}) \right] + \left[ f _ {2} \left(\boldsymbol {x} _ {a d v}\right) - F _ {2} \left(\boldsymbol {x} _ {a d v}\right) \right] \right| \right. \tag {9} \\ \leq \left| F _ {1} (\boldsymbol {x}) - f _ {1} (\boldsymbol {x}) \right| + \left| F _ {2} \left(\boldsymbol {x} _ {a d v}\right) - f _ {2} \left(\boldsymbol {x} _ {a d v}\right) \right| \leq \varepsilon_ {3} + \varepsilon_ {4}. \\ \end{array}
$$
According to Eq. (9), we compare the difference between  $|f_{2}(\pmb{x}_{adv}) - f_{1}(\pmb{x})|$  and  $|F_{2}(\pmb{x}_{adv}) - F_{1}(\pmb{x})|$ . As  $\varepsilon_{3}$  and  $\varepsilon_{4}$  are sufficiently small,  $|F_{2}(\pmb{x}) - F_{1}(\pmb{x})|$  will approximate  $|f_{2}(\pmb{x}) - f_{1}(\pmb{x})|$ , suggesting that we can fit  $|\widehat{y}_{\pmb{x}_{adv}} - \widehat{y}_{\pmb{x}}|$  by  $|f_{2}(\pmb{x}) - f_{1}(\pmb{x})|$ . Based on the above analysis, we prove that a feasible solution exists in the hypothesis space and can be trained.
Tasks and Datasets. We have implemented punctuation-level attacks and TPPEP in the context of three distinct text-to-label tasks, utilizing three diverse datasets. Furthermore, we have reported the outcomes of these experiments concerning text summarization, semantic-similarity-scoring, and text-to-image tasks, focusing particularly on the single-shot punctuation attack. The tasks that underwent single-shot attacks and TPPEP perturbations encompassed text classification (TC), paraphrasing, and natural language inference (NLI) tasks. The TC task takes a text as input, and the text model generates a predicted label for the input text. For our TC task, we have opted to utilize the CoLA dataset [43]. CoLA is the dataset of a binary classification task that aims to predict whether the input sentence is "correct" from a linguistic perspective. As for the paraphrase task, the input consists of a pair of texts, and the objective of the text model is to ascertain the semantic similarity between the two texts. Our choice for the paraphrasing task is the QQP dataset [21]. Furthermore, the NLI task involves predicting the relationship between pairs of input sentences and we employ the Wanli dataset [25].
Experiment Setup. We have applied both punctuation-level attacks and TPPEP to target pretrained SOTA models, including ELECTRA [7], XLMR [37], DistilBERT1 [36], RoBERTa [27], and DeBERTa [14]. In numerous real-world text API application scenarios, the API solely provides the predicted label for the input text. Therefore, our experiments are tailored for black-box attacks, where the text model exclusively generates categorical labels in response to queries. The conventional perturbation budget of character-/word-/sentence-level attacks is typically assessed based on the number of altered words. In contrast, our punctuation-level attacks perturb zero word, and we define the perturbation budget as the number of perturbed tokens. In the experiment, we set the strictest perturbation limit, specifically focusing on the perturbation of a single punctuation. Thus, we discuss the results in the case of a single punctuation attack. In addition, to make our TPPEP work in hard-label attacks, we first train a substitute function  $f_{sub}$  [30] by querying the text function and deriving the prediction labels  $y_{pre}$ . Then, we apply the proposed TPPEP method to search for the most vulnerable candidate text.
Text classification task, paraphrase task, and natural language inference task. The result of the TC task is presented in Table 5. Our punctuation-level attack shows encouraging results on ELECTRA [7] model and Cola dataset with  $90.80\%$  and  $93.67\%$  fool rates respectively by single-shot Insertion attack. On all three datasets, Insertion consistently obtains the highest fool rate due to the largest search space compared to other attacks such as Deletion, which has the smallest search space and thus obtains the lowest fool rate. The fool rate of Displacement and Replacement depends more on the specific datasets and text models. The results of the traversal
Table 5: The results on Cola, QQP, and Wanli datasets. Top-1 is the fool rate of single-shot and single punctuation attack. The Top-3 and Top-5 are the fool rates of three and five single punctuation attacking candidate texts. Traversal is the fool rate of a single punctuation attack by traversal, which is the maximum value of a single punctuation attack.  $p_1$  is the ratio of TOP-1 to traversal. Average search space (ASP) is the mean of candidate texts.
Cola ELECTRA [7] XLMR [37] mode
Top-1	Top-3	Top-5	Traversal	\(p_1\)	Top-1	Top-3	Top-5	Traversal	\(p_1\)	ASP
Insertion	67.40%	73.06%	73.83%	90.80%	74.23%	28.76%	52.64%	63.57%	93.67%	30.70%	362.62
Displacement	36.05%	66.35%	73.44%	80.44%	44.82%	43.05%	60.12%	76.03%	80.73%	53.33%	11.59
Deletion	5.18%	5.85%	5.94%	5.94%	87.21%	4.89%	5.85%	5.85%	5.85%	83.59%	1.15
Replacement	24.64%	36.82%	44.77%	74.59%	33.03%	6.62%	9.88%	12.37%	20.23%	32.72%	41.49
QQP DistilBERT1 [36] DistilBERT2 [36]
Insertion	14.72%	18.76%	22.68%	47.18%	31.20%	8.67%	10.43%	11.73%	48.23%	17.98%	957.72
Displacement	8.52%	15.05%	18.86%	26.78%	31.81%	7.21%	12.43%	15.57%	23.44%	30.76%	36.57
Deletion	3.94%	5.93%	6.02%	6.03%	65.34%	5.06%	6.86%	6.95%	6.96%	72.70%	2.53
Replacement	7.59%	10.04%	12.18%	19.70%	38.53%	16.70%	20.97%	22.65%	29.65%	56.32%	90.91
Wanli RoBERTa [27] DeBERTa [14]
Insertion	8.44%	19.22%	26.20%	66.74%	12.65%	15.28%	29.20%	37.40%	80.14%	19.07%	1161.12
Displacement	5.12%	9.14%	12.26%	26.14%	19.59%	10.28%	16.60%	20.34%	38.40%	26.77%	53.94
Deletion	3.22%	5.84%	6.14%	6.16%	52.27%	5.74%	8.58%	8.96%	8.98%	63.92%	2.94
Replacement	8.48%	15.96%	19.80%	45.82%	18.51%	6.92%	13.08%	16.88%	54.76%	12.64%	105.88
Table 6: The results of the semantic-similarity-scoring task
Sentence-BERT	Distilbert
STS-B	Pearson	Spearman	Pearson	Spearman
Without Attack	0.7990	0.6988	0.8056	0.7257
TOP-1	0.7874	0.6862	0.7902	0.7035
TOP-3	0.7760	0.6738	0.7759	0.6990
TOP-5	0.7654	0.6626	0.7649	0.6745
Traversal	0.6992	0.5832	0.6994	0.6048
search imply that a single punctuation attack can fool those text models. The results of Top-1 present the result of single-shot and single punctuation attack. Our experiments show that single-shot and single punctuation attacks are more effective for datasets with small average search space (ASP) such as the CoLA dataset. In the case of ELECTRA model and CoLA dataset, single-shot and single punctuation attack obtains  $67.40\%$ ,  $36.05\%$ ,  $5.18\%$ , and  $24.64\%$  respective fool rate by Insertion, Displacement, Deletion, and Replacement attack. The fool rate of the single-shot and single punctuation attack decreases with increasing search space but still achieves a fool rate of  $8.44\%$  when the ASP is 1161.12 in the Wanli dataset.
Semantic-similarity-scoring task, summarization task, and text-to-image task. In addition, we have expanded the scope of single punctuation attacks to encompass semantic-similarity-scoring (sss), summarization, and text-to-image tasks (T2I). Datasets from STS-B [26] and gigaword [20] datasets were chosen for the sss and summarization tasks. Results of single punctuation attacks are tabulated in Tables 6 and 7. In the sss task, the Insertion punctuation attack leads to reductions of Pearson and Spearman correlations for Sentence-BERT and Distilbert, resulting in reductions of 0.0998, 0.1156, 0.1062, and 0.1209, respectively. For the summarization task, we utilize the ROUGE-1 metric as the dependent variable, denoted as  $y$ . Then we formulate a predictive model to estimate the ROUGE-1 score for the test set. Subsequently, we select the candidate attack text with the lowest predicted ROUGE-1 value. The Insertion punctuation attack decreases the ROUGE-1 score [24] of gigaword [20] by 6.47. In the sss task,  $y$  represents the semantic similarity between the text before and after the attack. To model this, we have developed a predictive model to estimate semantic similarity values for the test set. This significant decline demonstrates the susceptibility of sss and summarization models to single punctuation attack.
In the T2I task, we determine the CLIP score denoted as  $y$  and select the candidate attack text with the lowest predicted CLIP score. Stable Diffusion V2 [33] is selected as the victim model, and the results are illustrated in Fig. 2. When we input the sentence "a professional photograph of an astronaut riding a triceratops", the Stable Diffusion V2 generates the "correct" image containing both the astronaut and triceratops. However, after inserting a period into the input text, Stable Diffusion V2 generates an image without triceratops. This dramatic change also happens in another input sentence. Furthermore, we have expanded the section dedicated to quantitative experiments in T2I applications,
Table 7: The results of text to image and summarization task
Task	Metric	Without Attack	TOP-1	TOP-3	TOP-5	Traversal
Text to image	CLIP score	0.3278	0.3176	0.3069	0.3022	0.2610
Summarization	ROUGE-1	11.69	10.91	9.65	9.11	5.22
Ori-output
Adv-output
Ori-output
Figure 2: The results of single punctuation attack in the text-to-image task. "Ori-text" indicates the original text and "Adv-text" indicates the adversarial text under a single punctuation attack. "Ori-output" and and "Adv-output" indicate the images generated by the original text and the adversarial text respectively. The results reveal that the text-to-image model is fooled by inserting a comma or period into the input text.
utilizing the CLIP score as our evaluation metric. The experimental results are displayed in Table 4, with "ori-text1" denoting the sentence "a corgi is playing the piano, oil on canvas." By inserting a comma as a punctuation-level attack, "adv-text1" is derived from "ori-text1". Significantly, despite a high semantic similarity of 0.9843 between "ori-text1" and "adv-text1", the CLIP score decreases from 0.4040 to 0.3468 after the attack. Furthermore, to evaluate the resilience of the Stable Diffusion model against punctuation-level attacks, we utilize the "pokemon-blip-captions" dataset. Following the punctuation insertion attacks, the overall CLIP score decreases from 0.3273 to 0.2591. This reduction is consistently observed in both the training and testing subsets, highlighting the model's susceptibility. In the present the results of our endeavor to extend punctuation-level attacks, TPPE, and TPPEP algorithms to encompass three distinct tasks: summarization, semantic similarity scoring, and text-to-image generation show the effectiveness of TPPE and TPPEP.
In this work, we propose a punctuation-level attack based entirely on inserting, displacing, deleting, and replacing punctuation to fool text models. Our punctuation-level attack is more imperceptible to human beings and has less semantic impact compared to character-/word-/sentence-level attack. We introduce TPPE and TPPEP methods to reduce the search cost and achieve the time complexity of the query as  $\mathcal{O}(1)$ . Furthermore, we discuss the ultimate effectiveness of punctuation-level attacks and present the analysis of the TPPEP method. Experimental studies on representative datasets of SOTA victim models demonstrate the effectiveness of our proposed methods.
This work was supported in part by the National Key R&D Program of China (Grant No. 2022ZD0118100), in part by the National Natural Science Foundation of China (Grant No. 62372480), in part by the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012839), in part by ARC-Discovery grant projects (DP 190102261 and DP220100800), and in part by the Shenzhen Science and Technology Program (No. JSGG20220831093004008), and sponsored by CCF-Tencent Rhino-Bird Open Research Fund (No. CCF-Tencent RAGR20230118).