SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Punctuation-level Attack: Single-shot and Single Punctuation Attack Can Fool Text Models
Wenqiang Wang $^{1}$  Chongyang Du $^{1}$  Tao Wang $^{2}$  Kaihao Zhang $^{3}$
Wenhan Luo $^{1*}$  Lin Ma $^{4}$  Wei Liu $^{5}$  Xiaochun Cao $^{1}$
<sup>1</sup>Shenzhen Campus of Sun Yat-sen University <sup>2</sup>Nanjing University
$^{3}$ Australian National University  $^{4}$ Meituan  $^{5}$ Tencent
The adversarial attacks have attracted increasing attention in various fields including natural language processing. The current textual attacking models primarily focus on fooling models by adding character-/word-/sentence-level perturbations, ignoring their influence on human perception. In this paper, for the first time in the community, we propose a novel mode of textual attack, punctuation-level attack. With various types of perturbations, including insertion, displacement, deletion, and replacement, the punctuation-level attack achieves promising fooling rates against SOTA models on typical textual tasks and maintains minimal influence on human perception and understanding of the text by mere perturbation of single-shot single punctuation. Furthermore, we propose a search method named Text Position Punctuation Embedding and Paraphrase (TPPEP) to accelerate the pursuit of optimal position to deploy the attack, without exhaustive search, and we present a mathematical interpretation of TPPEP. Thanks to the integrated Text Position Punctuation Embedding (TPPE), the punctuation attack can be applied at a constant cost of time. Experimental results on public datasets and SOTA models demonstrate the effectiveness of the punctuation attack and the proposed TPPE. We additionally apply the single punctuation attack to summarization, semantic-similarity-scoring, and text-to-image tasks, and achieve encouraging results.
Deep Neural Networks (DNNs) have achieved tremendous success in the NLP community and have spawned a series of well-known applications such as ChatGPT [13]. However, more attention should be paid to the vulnerability of NLP models under adversarial attacks to protect them. Adversarial examples have been shown to have a devastating impact in the image domain [39, 12, 38]. Unlike the image domain, text is a discrete domain [45], which means that any small perturbations, such as sentence, word, or even character perturbations in text, would be easily perceived by humans. It is difficult to achieve imperceptible perturbations. Additionally, the text is a non-differentiable domain, which results in traditional optimization-based methods being ineffective in the NLP domain. Recent research on textual adversarial attacks can be mainly classified into character-level, word-level, sentence-level, and multi-level attacks [41]. Character-level attacks modify words by inserting, deleting, misspelling, replacing, or swapping characters. These attacks are easily detectable by humans and some apps can correct them automatically. However, character-level attacks require multiple queries to determine which characters to attack based on function gradient or score. Word-level attacks modify words by adding, deleting, or replacing important words in a text. Similar to character-level attacks, word-level attacks also require multiple queries to determine which words to
*Corresponding Author: <whluo.china@gmail.com >
Figure 1: The examples of punctuation-level attacks. The text model is fooled by inserting, displacing, deleting, and replacing punctuation from the original text. The predicted label of the input text is either acceptable or unacceptable. A single punctuation attack can significantly change the score of the predicted label.
attack. Both character- and word-level attacks are iterative and thus may be time-consuming in real NLP applications due to multiple queries required for each attack iteration. Sentence-level attacks modify entire texts by paraphrasing or adding meaningless sentences to input texts. The significant perturbations caused by sentence-level attacks may sometimes change the ground truth of input text. Perturbations in character-, word-, and sentence-levels can cause semantic gaps from the original text and are easily detectable by humans.
Considering the issues mentioned above, we propose punctuation-level attacks as a new method of textual adversarial attack. Compared with other attack methods, perturbations on punctuations are less perceivable than character- and word-level attacks, more difficult to detect than character-level attacks, and cause less impact on ground truth than other types of attacks. In previous research, little attention has been paid to punctuation; several researchers have inserted punctuations into words to create out-of-vocabulary (OOV) words, which is a specific method of character-level attacks. On the contrary, we first propose punctuation-level attacks that regard punctuation perturbations as a new level of attack similar to character-, word-, and sentence-level attacks. We define basic attacking methods and analyze why these methods can fool text functions. Unlike other types of attacks, punctuation-level attacks only modify punctuations by inserting, displacing, deleting, or replacing them without perturbing characters, words, and sentences. Punctuation-level attacks can deceive text functions by inserting, displacing, deleting, or replacing punctuations. Specific examples are presented in Fig. 1, showing that the single punctuation attack results in a dramatic change in the score of the predicted label. To deploy punctuation-level attacks, we need to consider which positions in the input text should be modified and which punctuations should be modified. For example, if there are  $k$  candidate punctuations and  $n$  tokens in the input text, there will be  $kn$  candidate attack texts when we insert one punctuation,  $k^2 n(n + 1)$  candidate attack projects when we insert two punctuations, and  $k^3 n(n + 1)(n + 2)$  candidate attack projects when we insert three punctuations. The time complexity of multiple punctuation attacks can be reduced using a greedy algorithm or a beam search. However, even considering only a single punctuation attack is still time-consuming.
To address this problem, we propose a method called Text Position Punctuation Embedding (TPPE) to reduce search cost. The feature extraction processing of input text  $\pmb{x}$  is also the embedding processing of text  $\pmb{x}$ , which is time-consuming for multiple queries. Therefore, we propose Text Position Punctuation Embedding (TPPE) to quickly and reasonably embed adversarial candidate text  $\pmb{x}_{adv}$ . Let us illustrate it with the Insertion attack mode. When we apply Insertion attack to punctuation  $p$  into the  $i$ -th position of the input text, we query the text function  $f(\pmb{x})$  or the substitute function  $f_{sub}(\pmb{x})$  and gain the  $f_{fe}(\pmb{x})$  as the embedding of text  $E_{text}$  and  $f_{fe}(\pmb{p})$  as the embedding of punctuation  $p$ . The embedding of punctuation  $p$  only needs to be queried once and can be used repeatedly. As for the embedding of position, we adopt the embedding of the static position based on the calculation of sine and cosine from Transformer [40]. And we define the embedding of the adversarial candidate text  $\pmb{x}_{adv}$  by concatenating or adding the embedding of text  $\pmb{x}$ , punctuation  $p$  and position  $i$ . We can iteratively attack the input text  $\pmb{x}$  and quickly obtain the adversarial text  $\pmb{x}_{adv}$  after training the classification model from the TTPE of  $\pmb{x}_{adv}$  to the prediction of  $\pmb{x}_{adv}$  by  $f(\pmb{x})$ . In addition, we propose a search method Text Position Punctuation Embedding and Paraphrase (TPPEP) to achieve a single-shot attack. We analyze the most information-lacking scenario for TPPEP: zero query, black-box text function, hard-label output, single punctuation limitation, and single-shot attack. To achieve the goal of zero query and gain more information, we train a substitute function  $f_{sub}$  to fit the text function  $f$ . Then, we query the substitute function  $f_{sub}$  to obtain the embedding of
text  $E_{text}$ . We apply the TPPE method to obtain the embedding of the adversarial candidate text  $x_{adv}$ , denoted as  $E_{x_{adv}}$ . We then transform the attacking task into the paraphrasing task.  $E_{x_{adv}}$  and  $E_{text}$  are concatenated as input data, and the result of attack (successfully attack, donated as label 1, otherwise, donated as label 0) is the predicting label. We choose the adversarial candidate text with the highest paraphrasing score calculated by the TPPEP method and deploy the single punctuation attack in a single-shot manner accordingly.
The contributions can be summarized as follows. (1) Punctuation-level attacks: We first propose punctuation-level attacks, which regard the perturbations of punctuation as a systematic attack like character-level, word-level, and sentence-level attacks. We propose four primary modes of punctuation-level attacks and explain punctuation-level attacks from the perspective of optimal perturbations. (2) TPPE: We first propose the TPPE embedding method to decrease the search cost. We reduce the query time complexity from  $\mathcal{O}(kn)$  of Insertion,  $\mathcal{O}(nt)$  of Displacement,  $\mathcal{O}(t)$  of Deletion, and  $\mathcal{O}(kt)$  of Replacement, to  $\mathcal{O}(1)$  under single punctuation attack. It can quickly and reasonably embed the adversarial candidate text  $x_{adv}$  using a single-shot query. (3) Single-shot and Single Punctuation Attack: To make our punctuation-level attack more imperceptible, we modify only one punctuation. Besides, we discuss single-punctuation attacks in the most challenging scenario: zero query, black-box function, hard-label output, one-punctuation limitation, and single-shot attack, which is the closest to the real-world scenarios. We correspondingly propose the TPPEP method and achieve promising experimental results.
Textual adversarial attack. Text adversarial attack methods in the literature can be broadly categorized into two classes: white-box attacks and black-box attacks. In white-box attacks, some methods [23, 35, 34, 1, 6] employ FGSM [12] to identify the words or characters with the most significant impact on text-related tasks. For example, TextFool [23] introduces a gradient loss function within the model to assess the influence of specific words or characters on task performance. Suranjana et al. [35] consider firstly trying to remove the adverb as a word-level attack, and use a gradient loss function in the model to achieve the attack. AdvGen [1] focuses on the similarity between the original words and their substituted counterparts to target neural machine translation (NMT) models. Moreover, some methods rely on directional derivatives [9, 8]. For instance, Hot-flip [9] manipulates text function by altering characters based on directional derivative gradients. Ebrahimi et al. propose controlled attacks, an extension of HotFlip that enables targeted attacks [8]. In black-box attacks, specific methods aim to insert nonsensical sentences into the input text with the intention of deceiving the text model [19, 42, 4]. Notable approaches include AddSent [19] and AddSentDiverse [42]. AddSent incorporates meaningless sentences into the input text to confound comprehension system models [19]. Meanwhile, AddSentDiverse broadens the pool of adversarial candidate texts by varying the placements of inserted sentences [42]. Other strategies involve the manipulation of words and sentences [3, 29, 11, 22, 28, 2, 5, 44, 31]. For example, Textbugger [22] alters words by substituting, removing, adding, or rearranging them to deceive the text model. Furthermore, certain methods are centered on paraphrasing the input text [18, 32]. SCPNs [18] employs an encoder-decoder architecture to generate new paraphrased versions of the input text.
Punctuation attacks. To the best of our knowledge, there is a scarcity of research in the existing literatures on punctuation attacks. Most related studies primarily concentrate on punctuation perturbation, specifically the placement of punctuation marks within or at the end of words. This approach can be classified as a character-level attack. For example, Hosseini et al. insert punctuation into words to transform them into OOV words [17]. In another work by Hofer et al. [15], they also regard inserting a single comma as a special character-level attack, and they insert a comma at the end of the important word to translate this important word into an OOV word. In addition, a special symbol text attacks (SSTA) method is proposed in [23], which is viewed as an adjunct of TextFool [10] by adding fifty tildes behind the text.
3.1 Punctuation-level Attacking
The Proposed Punctuation-level Attacks. We propose four modes of attacking at the punctuation-level and analyze their time complexity for traversal search. These modes are as follows:
Table 1: Punctuation-level attack modes and Complexity.  $k$  is the number of candidate punctuations,  $t$  is the number of punctuations in the input text and  $n$  is the number of input tokens.
mode	single punctuation	m times of punctuation attacks	m times of attacks by greedy algorithm
Insertion	O(kn)	O(km-1i=0(n+i))	O(0.5k(2n+m-1))
Displacement	O(nt)	O(nmtm)	O(mnt)
Deletion	O(t)	O(m-1i=0(t-i))	O(0.5m(2t+1-m))
Replacement	O(kt)	O((kt)m)	O(mkt)
- Insertion: Punctuation  $p$  is inserted into the target text to fool the text model.
- Displacement: Punctuation  $p$  is moved from position  $i$  to position  $j$  in the target text.
- Deletion: Punctuation  $p$  is removed from the target text.
- Replacement: Punctuation  $p_i$  is replaced by  $p_j$  in the target text.
We denote the number of candidate punctuations by  $k$ , the number of punctuations in the input text by  $t$ , and the number of input tokens by  $n$ . In Insertion attack, there are  $n$  candidate positions, and each position can be inserted by  $k$  candidate punctuations. So the complexity of Insertion by traversal search is  $\mathcal{O}(kn)$ . Regarding the  $m$  times of Insertion attacks, one text will generate  $kn$  candidate texts after a single punctuation attack, and the complexity of Insertion to each candidate is  $k(n + 1)$ , thus the complexity of multiple Insertion attacks is  $\mathcal{O}\left(k^{m}\prod_{i = 0}^{m - 1}(n + i)\right)$ . If the greedy algorithm is applied for the  $m$  times of Insertion attacks, the generated  $kn$  candidate texts after the single punctuation attack will be selected with the greatest change in predicted score, and the number of positions is increased to  $n + 1$ . Thus, the complexity of Insertion to this selected candidate is  $\mathcal{O}(0.5k(2n + m - 1))$ . The complexity of Displacement by traversal search is  $\mathcal{O}(nt)$  because there are  $t$  candidate punctuations and each position can be displaced by  $n$  candidate positions. As for the  $m$  times of Displacement attacks, one text will generate  $nt$  candidate texts after the single-shot attack, and the complexity of Displacement to each candidate is  $nt$ . So the complexity of  $m$  times of Displacement attacks is  $\mathcal{O}((nt)^{m})$ . If the greedy algorithm is applied for  $m$  times of displacement attacks, there are still  $t$  candidate punctuations and  $n$  candidate positions. So the complexity of  $m$  times of Displacement by the greedy algorithm is  $\mathcal{O}(mnt)$ . The complexity of Deletion by traversal search is  $\mathcal{O}(t)$  because there are  $t$  candidate punctuations that can be deleted. In the context of multiple occurrences of Deletion attacks, each attack leads to the removal of a candidate punctuation. Consequently, the complexity of  $m$  consecutive Deletion attacks can be expressed as  $\mathcal{O}\left(\prod_{i = 0}^{m - 1}(t - i)\right)$ , where  $t$  represents the total number of possible candidates. If the greedy algorithm is applied for  $m$  times of Deletion attacks, due to every Deletion attack, the text will lose one punctuation, thus the complexity of  $m$  times of Deletion by the greedy algorithm is  $\mathcal{O}(0.5m(2t + 1 - m))$ . The complexity of Replacement by traversal search is  $\mathcal{O}(kt)$  because there are  $t$  candidate punctuations and each punctuation can be replaced by  $k$  candidate punctuations. Regarding the  $m$  times of replacement attacks, there are still  $k$  candidate punctuations and  $t$  punctuations in the input text after a single-shot attack. Thus, the complexity of  $m$  times of Replacement attacks is  $\mathcal{O}((kt)^{m})$ . If the greedy algorithm is applied for  $m$  times of Deletion attacks, the complexity of  $m$  times of Replacement by the greedy algorithm is  $\mathcal{O}(mkt)$ . All complexity is shown in Table 1.
Single punctuation Attack. The target text  $\pmb{x}$  consists of  $n$  tokens, noted as  $\pmb{x} = [x_{1},\dots ,x_{n}]$ . According to the Universal Approximation Theorem [16], we can define a function  $F(\pmb{x})$  to fit the text function  $f(\pmb{x})$  by a continuous nonconstant, bounded, and monotonically increasing function  $\phi (\cdot)$  as
$$
F (\boldsymbol {x}) = \sum_ {m = 1} ^ {M} v _ {m} \phi \left(\boldsymbol {w} _ {m} ^ {\top} \boldsymbol {x} + b _ {m}\right), | F (\boldsymbol {x}) - f (\boldsymbol {x}) | <   \epsilon , \tag {1}
$$
where  $v_{m}, b_{m} \in \mathbb{R}$  and  $\pmb{w}_{m} \in \mathbb{R}^{D}$ . Universal Approximation Theorem demonstrates the fact that if our attack can result in vicious impact in  $f(\pmb{x})$ , it will also fool the text function  $F(\pmb{x})$ . In the
Table 2: The number of tokens affected by a single attack.  $k$  is the number of candidate punctuations,  $i$  is the position of attacking punctuation in the input text, and  $n$  is the number of input tokens.
Mode	Perturbed tokens	Perturbed words	Perturbed level
character-level attack	1	1	value of token
word-level attack	1	1	value of token
Insertion	n-i+2	0	value and position of token
Displacement	j-i+1	0	value and position of token
Deletion	n-i	0	value and position of token
Replacement	1	0	value of token
following, we will analyze how punctuation-level attacks influence the text function  $F(\pmb{x})$  based on the basic linear unit  $\pmb{w}_m^\top \pmb{x} + b_m$  of  $f(\pmb{x})$  as
$$
\boldsymbol {w} _ {m} ^ {\top} \boldsymbol {x} + b _ {m} = w _ {1} x _ {1} + w _ {2} x _ {2} + \dots + w _ {m} x _ {m} + b _ {m}. \tag {2}
$$
We compare character-level, word-level, and punctuation-level attacks under the constraint of a single-element attack, i.e., perturbing one character, one word, or one punctuation. In character-level attacks, a single-character attack impacts only one token  $x_{i}$ , making it OOV. In many text functions, the OOV token is embedded as an identical token  $x_{oov}$ . In this scenery,  $\boldsymbol{w}_{m}^{\top}\boldsymbol{x} + b_{m}$  can be written as
$$
\boldsymbol {w} _ {m} ^ {\top} \boldsymbol {x} + b _ {m} = w _ {1} x _ {1} + w _ {2} x _ {2} + \dots + w _ {i} x _ {o o v} + \dots + w _ {m} x _ {m} + b _ {m}. \tag {3}
$$
Similar to character-level attacks, a single-word attack impacts only one token  $x_{i}$ . Since there are few candidate tokens  $x_{can}$  that are semantically similar to the token  $x_{i}$  when considering the similarity of input text with and without a word-level attack,  $\boldsymbol{w}_{m}^{\top}\boldsymbol{x} + b_{m}$  can be rewritten as
$$
w _ {1} x _ {1} + w _ {2} x _ {2} + \dots + w _ {i} x _ {\text {c a n}} + \dots + w _ {m} x _ {m} + b _ {m}. \tag {4}
$$
Regarding the proposed punctuation attacks, we discuss single punctuation attacks in terms of Insertion, Displacement, Deletion, and Replacement respectively. Taking Insertion as an example, when we insert punctuation  $p$  into position  $i$  of the target text,  $\boldsymbol{w}_m^\top \boldsymbol{x} + b_m$  will change to
$$
w _ {1} x _ {1} + w _ {2} x _ {2} + \dots + w _ {i} p + w _ {i + 1} x _ {i} + \dots + + w _ {m} x _ {m - 1} + w _ {m + 1} x _ {m} + b _ {m}. \tag {5}
$$
According to Eq. (3) and (4), the character-level and word-level attacks only perturb one token and one word by the single punctuation attack. According to Eq. (5), the single Insertion attack does not perturb any words but perturbs the  $i$ -th token by substituting  $x_{i}$  with punctuation  $p$  and all tokens after the  $(i - 1)$ -th position of the input text by moving them one bit backward, which totally perturbs  $n - i + 2$  tokens. The result of Displacement, Deletion, and Replacement is presented in Table 2. According to Table 2, the punctuation-level attack can impact more tokens, indicating that it can cause greater perturbation than character- and word-level attacks. If we insert, displace, or delete punctuation in the appropriate position, it will result in a more significant impact on the basic linear  $\boldsymbol{w}_{m}^{\top}\boldsymbol{x} + b_{m}$  of  $f(\boldsymbol{x})$  compared to character-level and word-level attacks. Meanwhile, a single punctuation attack may apply a more vicious impact to  $f(\boldsymbol{x})$  and  $F(\boldsymbol{x})$ . Compared to words, punctuation is thinner, smaller, and more imperceptible. Due to the nature that punctuation-level attack does not perturb any word, semantically, the adversarial text is more imperceptible for human beings. Furthermore, the Insertion, Displacement, Deletion, and Replacement of the punctuation produces less impact on the semantic information by human beings understanding.
The Ultimate Effectiveness of Punctuation-level Attacks. According to Eq. (3), (4), (5), character- and word-level attacks under a single-element limitation may focus on perturbing token values. However, Insertion, Displacement, and Replacement attacks can not only perturb token values but also token positions which may result in a vicious impact on  $f(x)$ . This position perturbation almost has no impact on semantic information for human beings.
We discuss why multiple punctuation-level attacks can fool the text model by taking Insertion attacks as an example. In the text model, there exists the maximum length of the input text, and we denote it as  $N$ . If the length of the input text is greater than  $N$ , the text model  $f$  considers only the first  $N$  tokens. If we apply  $N$  Insertion attacks, there will be  $k^{N}$  candidate texts. The probability of fooling the text model is equal to 1 minus the probability of all  $k^{N}$  candidate texts failing to fool
Table 3: Text embedding in punctuation-level attack.
Mode	Concat	mathematical operation
Insertion	Eall	Etext + Epos + Epunc
Displacement	Eall	Etext - Ei pos + Ej pos + Epunc
Deletion	Eall	Etext - Ei pos - Epunc
Replacement	Eall	Etext + Epos - Ei punc + Ej punc
the text model. We denote the probability of fooling the text model as  $p_{\text{fool}}$  and the probability of failing to fool the text model by the  $i$ -th candidate text as  $p_i$ ,  $i \in [1, K^N]$ . And we denote the maximum value of  $p_i$  as  $p_{\text{max}}$ . We make an assumption about  $p_{\text{max}}$  here. Due to the fact that  $p_i$  is the probability of the  $i$ -th candidate text before querying the text model  $f$ , and the attacking result is unknown, we have  $0 < p_{\text{max}} < 1$ . Correspondingly, we have
$$
p _ {f o o l} = 1 - \prod_ {i = 1} ^ {k ^ {N}} P _ {i} \geq 1 - p _ {\max } ^ {k ^ {N}}. \tag {6}
$$
According to Eq. (6),  $1 - p_{\text{max}}^{k^N}$  will approach 1 as  $k$  increases. As a probability, we have  $1 - p_{\text{max}}^{k^N} \leq p_{\text{fool}} \leq 1$ . Consequently,  $p_{\text{fool}}$  will tend to 1 when  $k$  increases, which means multiple Insertion attacks can ultimately fool the text model with  $k$  increasing.
3.2 Embedding and Search Methods