Towards Irreversible Attack: Fooling Scene Text Recognition via Multi-Population Coevolution Search
Authors: Jingyu Li
Pengwen Dai
Mingqing Zhu
Chengwei Wang
Haolong Liu
Xiaochun Cao
Venue: NeurIPS 2025 poster ()
Keywords: adversarial attack
scene text recognition
evolution algorithm
computer vision
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose a pixel-level black-box attack method to fool STR models to predict more incorrect characters, using a novel multi-population coevolution search algorithm.

Abstract: Recent work has shown that scene text recognition (STR) models are vulnerable to adversarial examples.
Different from non-sequential vision tasks, the output sequence of STR models contains rich information. 
However, existing adversarial attacks against STR models can only lead to a few incorrect characters in the predicted text.
These attack results still carry partial information about the original prediction and could be easily corrected by an external dictionary or a language model.
Therefore, we propose the Multi-Population Coevolution Search (MPCS) method to attack each character in the image. 
We first decompose the global optimization objective into sub-objectives to solve the attack pixel concentration problem existing in previous attack methods.
While this distributed optimization paradigm brings a new joint perturbation shift problem, we propose a novel coevolution energy function to solve it.
Experiments on recent STR models show the superiority of our method.
The code is available at \url{https://github.com/Lee-Jingyu/MPCS}.

Towards Irreversible Attack: Fooling Scene Text Recognition via Multi-Population Coevolution Search
Jingyu Li Pengwen Dai Mingqing Zhu', Chengwei Wang', Haolong Liu', Xiaochun Cao School of Cyber Science and Technology,
Shenzhen Campus of Sun Yat-sen University, Shenzhen, Guangdong 518107, China (1ijy768, zhumq8, liuhlong7) 0maal2 sysaiiisssssu...... sysu. edu. cn,
wangchx230alummi sysu. edu. cn,
{daipu, caoxiaochun) @mail. sysu.edu.uc. edu.
Abstract
Recent work has shown that scene text recognition (STR) models vulnerable to adversarial examples. Different from non-sequential vision tasks, the output sequence of STR models contains rich information. However, existing adversarial attacks against STR models can only lead to a few incorrect characters in the pre- dicted text. These attack results still carry partial information about the original pre- diction and could be easily corrected by an external dictionary or a language model. Therefore, we propose the Multi-Population Coevolution Search (MPCS) method to attack each character in the image. We first decompose the global optimization objective into sub-objectives solve the attack pixel concentration problem exist- ing in previous attack methods. While this distributed optimization paradigm brings new joint perturbation shift problem, we propose novel coevolution energy function to solve it. Experiments on recent STR models show the superiority of our method. The code is available /tthtt:///giii com/Lee- Minee-JinnyMMPPCC
1 Introduction
Scene text recognition (STR) 29 20 30 has garnered increasing attention in the field of computer vision due to wide applications, such augmented reality [27]. visual question answering (3I), automatic driving (8), etc. Despite its great success, recent studies [32] 36 371 have shown that STR models are vulnerable adversarial examples, which deceive the models into making incorrect predictions by adding imperceptible perturbations on the input images. In practice, adversarial examples can be used prevent private text 4 37 in images from being recognized and exploited by malicious optical character recognition (OCR) systems. Therefore, adversarial attacks against STR models have become valuable research topic.
Existing adversarial attacks in the computer vision field mainly focus on non-sequential tasks 9 23 [X], such image classification, object detection, face recognition, etc. The output of these models is usually single label and contains limited information. In this scenario, once the model outputs a different label, the information conveyed by the output is completely altered. In contrast, the prediction of STR models can be seen as a series of character labels, which contain rich information. When attacking STR models, the altered prediction may retain partial information about the original prediction. With the retained information, the opponent could correct the mispredicted text through an extra dictionary 22 30 13
or large language models (LLMs) 34 [5), as shown Figure [
However, existing adversarial attacks against STR models consider merely altering the model predic- tion successful attack. For example, some works focus on attacking the text image with clean
Corresponding author: Pengwen Dai <daipwemail, wysll.syss..om edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

62
Text correction
2
8: Perturbation
singe
SINCE
since
Text prediction
STR model
LM Dict
SINCE
SINCE 8, X
satisfies X
Input
sifies
Figure Adversarial attack results with single incorrect character could be corrected through a language model or dictionary. The attack results become more irreversible as the number of perturbed characters increases.
These methods may change the background or fail to attack due to complex backgrounds when dealing with scene text images. Differently, RoLMA (38] attacks license plate images, whose backgrounds are more complicated, by adding some light spots on the background. These light spots will also destroy the visual continuity of the background when dealing with scene text images. Compared to license plate images, scene text images are harder to attack due to their more complex backgrounds and diverse text styles. Xu al. [36] develop an efficient optimization- based white-box attack method to fool STR models. Furthermore, in the black-box setting, A AD (37] narrows the search space in the differential evolution algorithm to attack STR models. These methods overlook the fact that the attack results still contain rich information about the correct text.
background
In this work, we propose a black-box pixel-level attack method aiming at attacking all characters in the scene text image. We first discuss the reason for previous pixel-level attacks failing attack more characters, i.e. the attack pixel spatial concentration problem in their global optimization. Then we decompose the global optimization objective into several sub-objectives and introduce spatial constraints to avoid the attack pixel concentration. However, optimizing sub-objectives may undermine the global feasibility, manifested as the joint perturbation shift problem. Therefore, we propose the Multi-Population Coevolution Search (MPCS) method, which utilizes multiple evolutionary adversarial pixel populations in different image subspaces individually search for each attack pixel in parallel. To solve the joint perturbation shift problem, MPCS introduces coevolution energy function that implicitly embeds the global constraint into the local objective function. Experimental results on recent STR models demonstrate that MPCS produces substantially more incorrect predicted characters than state-of- the--ar art attack methods.
2 Related works
2.1 Attacks common vision models
Existing adversarial attack methods mainly focus on non-sequential vision tasks, such as image classification, object detection, and facial recognition. These adversarial attack methods can be roughly categorized into white-box attacks and black-box attacks. White-box attacks use the whole information of the victim model, including model structure, parameters, and gradients. Fast gradient sign method (FGSM) [ is proposed to generate adversarial perturbation according to the sign of the gradient. Deepfool 23 proposes to find the nearest classification hyperplane of the victim model to generate adversarial examples. GAT 122 composes multiple attack methods and utilizes a doubly stochastic matrix to optimize the attack order.
Differently, black-box attacks can only obtain the output of the victim model. OnePixel 33 employs the Differential Evolution (DE) algorithm find few adversarial pixels attack image classification models. Transfer- based attacks (25] I utilize adversarial examples generated from attacking source model fool the target victim model. PRGF (5) introduces two prior-guides random gradient- free algorithms aimed at enhancing the efficiency of black-box adversarial attacks, which leverage transfer- based prior provided by the gradient
surrogate model. Furthermore, to enhance the transferability of adversarial perturbations, L.TTA [7] generates generalized adversarial perturbations through data and model augmentation, and optimizes the final perturbation update through meta- learning.

---

These methods aim at attacking models that output non-sequential results, such categories, posi- tions, and scores. However, text recognition models predict text sequences, which usually contain more information for attackers to manipulate. We need to alter more predicted characters to ensure irreversible attack results.
2.2 Attacks on text recognition models
Text recognition models output text sequences containing more information than non-sequential outputs, which makes adversarial attacks aimed them more challenging. [32 introduces an adversarial attack method targeting OCR systems. which involves making minor modifications to printed text images deceive the OCR system into recognizing words with meanings opposite to the original text. [5 explores the effectiveness of the transfer-based attack algorithm in attacking 4 involves manipulating the background of text images by replacing clean image backgrounds with random textures to disrupt the recognition by OCR systems. RoLMA [38] proposes a targeted attack method to fool license plate recognition models by adding some crafted light spots to images. Furthermore, Xu et al. (36] propose an optimization-bant method to update the perturbation iteratively, attacking both CTC-based (connectionist temporal classification) and attention-ban based scene text recognition models. Based on the DE algorithm, AD²r 37 proposes to narrow down the continuous searching space to a discrete space, which can speed up the searching process. generates adversarial examples by fixing a few pixels' values or 255 in the image.
real-world OCR systems. Another approach
These methods focus on changing the model output, which means attackers only need to change one character in the model predictions. However, text with a few incorrect characters is easy to restore via the correction of external dictionary or LLMs 34 [5]. Therefore, in this work, aim attack all characters in images to make text recognition models predict more incorrect characters.
3 Preliminary
3.1 Problem formulation
Given a scene text image X E Rxxux3 (h and w are the height and width of the image), and its ground truth label Y [y1, ./.. .... yn] (n means the character number), the existing paradigm of crafting an adversarial example Xaadu X d to add small perturbation to X change the output of the victim model F, such that,
F(X) F(X s.t. ||0|p S.
where represents the maximum perturbation.
However, when attacking scene text recognition models, merely altering the output not enough. We need to mislead the victim model to predict many erroncous characters possible, rendering the attack results more irreversible. So we formulate a new objective as follows,
8 arg max D (F(X), +)),, S.L. |60|p S,
where 8" is the best adversarial perturbation. D(:--) is the edit distance 19] between the two predicted texts, which is measured by counting the minimum number of character operations required transform one string into the other.
In the black-box attack settings, the structure, parameters, and gradients of the victim model are not accessible. The attacker query the victim model to get the model output, which could be either a probability distribution or a character sequence with confidence scores. In practice, most OCR systems provide only the latter, which contains less information for attackers. Considering practicality, we choose the latter attack scenario.
3.2 Objective decomposition
In pixel-level attacks, the final perturbation 8 consists of a series of pixel perturbations 85, i.e., = Σ 5,. And each pixel perturbation can be represented by single attack pixel p3 (#r, yi; v:). where 2; and y the coordinates, and is the pixel value. Former methods 33 37 optimize 8 in
3

---

M
M
x secyre
xi
secyre
6
X
adv,
secure
rav;
sedyre
82
X2 sedure
Xadv; sedure
M
M
(a) PriledAttack
(b) Successful Attack
Figure 2: Illustration the joint perturbation shift. (a) and (b) demonstrate two cases when combining two successful perturbations into one to attack text recognition models. Characters with underlines are incorrectly predicted.
the whole image space. This optimization paradigm is prone to fall into the local optimal solution where attack pixels are concentrated in the local space of the image. In this case, some characters will have nearby attack pixel and then be correctly recognized.
To solve this problem, we decompose the optimization objective 8 into multiple sub-objectives 67, turning it to be distributed optimization problem. Then we limit the perturbation location to specific region in each sub-optimization problem. Specifically, we divide the image into multiple non-overlapping subspaces {R,)x and find local optimal attack pixel p; in each subspace R,. In this way, a proper division (Ra J n could ensure that the attack pixels are not spatially concentrated.
3.3 The joint perturbation shift problem
In this distributed optimization problem, local optimization may undermine global feasibility. Even each local optimized perturbation o can successfully attack the target model F, the global perturbation 8 = 0 has a certain probability of failing to attack, i.e.
N
P F(X)= = = + 8*)1 A
+ +02) 0.
(3)
1=1
We term this phenomenon the joint perturbation shift problem, and give simple illustration of in Figure 2 Let M and M be the manifold area [T] where data samples can be correctly and incorrectly recognized by the target model, respectively. X is the original data sample with label "secure". Although Xqdr X dj and X X X + 82 are two successful adversarial samples located in M,, the joint perturbation 8 51 82 may shift the adversarial sample Xaavv X to M and lead to failed attack.
4 Multi-population coevolution search
Following the distributed optimization paradigm, we propose the Multi- Multi-Population Coevolution Search (MPCS) method, which divides the image horizontally into N equal regions {R+}.v and utilizes evolutionary populations to search in each region. Meanwhile, a coevolution energy function introduced address the joint perturbation shift problem. Next, we will elaborate on the MPCS algorithm.
Initialization. We first randomly initialize an adversarial pixel population P; in each region R₁, Each population P; contains M adversarial pixels {u} ] M whose coordinates randomly initialized and values are fixed to 6, which can be formulated as follows.
P
{p] (r/, v/. (/)( vt) E R,, o 0)
where ue! and ya are the pixel coordinates, of the pixel rgh value. Moreover, based on the finding from [37] that the adversarial pixel values of l maa (255, 255, 255) and Omin (0, 0, 0) have

---

higher possibility to successfully attack than other values, we set & as follows,
O mmazi
G(v)) S j.
(5)
min
G(v > μ.
where G(v,) denotes the gray value of uz. and is set to 127 (the median of the set of pixel values). After that, we calculate the prior energy E² of each adversarial pixel in populations as,
Ea (i, App))).
(6)
where A (-)) is the perturbation generated by replacing the original pixel with the adversarial pixel, and Dist(Y, denotes the distance between the predicted label and the ground truth label Y (detailed in Appendix D). Here, Eº is equal to the local objective function.
Coevolution. We let the N populations coevolve in the direction of higher energy following the differential evolution (DE) algorithm 3 (37
Briefly, each adversarial pixel p in population has a probability of mutating into new adversarial pixel p Among and the one with higher energy will be retained for the next generation.
However, as the existence of the joint perturbation shift problem, the prior energy function E cannot filter the best adversarial pixel. Therefore, we propose novel cocvolution energy function ES, which implicitly embeds the global constraint into the local objective function,
Ex Ev!)) Dist(Y, + A(())) (Pi)N-1)), s.t. A At.IskSN,KFl
(7)
where { is the set of adversarial pixels with the highest energy in the other N populations. Note that in the first generation, we select {pī N- according to their prior energy E". After the first generation, evolution of one population will influence the other populations, which is similar to the coevolution phenomenon in nature.
replace E" with E to calculate the energy of adversarial pixels. In this case, the
Early stop. We propose an early stop mechanism, which terminates the evolution of the populations whose maximum energy has not increased compared to the previous generation. When the maximum number generations Jmaz reached or when all populations have ceased evolution, we obtain the best adversarial pixels (p) N from each population.
Post process. If {p} JNv can successfully attack the victim model, we take them as the final attack pixels (pade) N;; otherwise, will employ pixel set population P. to search for better pixel values following DE algorithm,
P. (O= -'=
(zj v.))
(8)
where L is the population size of Pe. Q' is the t-th adversarial pixel set whose pixel coordinates frozen to be the same as {pā b and values are randomly initialized. The energy function of P, is calculated as,
E ("(Q') = ((x) A(Q'))
At the max generation Imar of Pp. we obtain the best adversarial pixel set Q gdest as the final attack pixels (pode N. With the final attack pixels, the adversarial example is generated as follows,
x + A(tp adv }x).
(10)
Analysis
We make theoretical analysis of the superiority of our proposed coevolutionary energy function E over the prior energy function E". For certain pixel pf in population P., define:
F(X A(d)) EP(!)) P(A=
(11)
where 1 is the indicator function whose value is 0 or 1, and the estimator EP(2)) approximately reflects the relative strength of the probability of A = 1. Similarly, we define:
B = =(F(X + F(X + A(lp) (p)N--1)) ( P(B 1),
(12)
5

---

where k & i and ( denotes the local optimal pixels in the other N 1 populations.
Our final goal is increase the expectation of Z, which is defined as,
7 = [(F(x) * F(X + ((P))))))
(13)
where {pt denotes the local optimal pixels in all N populations.
Next, according to the joint perturbation shift problem illustrated in Sec 3.3 we have:
P(Z 1 = 1) 1, P(Z = 0/4 = 1)
(14)
which means the conditional distribution P(ZIA 1) is not deterministic distribution. Therefore, the conditional entropy of Z given A = 1 is strictly positive, i...
H(ZIA= 1)= = Σ
= 1) =|A= 0.
(15)
#E(0.1)
On the other side, (x(r) I (v))) holds for all pixels in population P..
It means that if a successful attack can be achieved by the joint perturbation A((p)) (pt))=1) N-1), then the joint perturbation of all the local optimal pixels A ((pi) v) must also yield successful attack. Therefore:
P(Z 1 = =1 P(Z = 0/B 1)=
(16)
Similarly, we have H =) H(ZIA 1), which reveals that when B = 1 (joint perturbation succeeds), the prediction of the global attack result Z is completely deterministic, unlike when A ==1 (local perturbation succeeds). It means that the prior energy function E² has a certain degree of information loss when estimating the attack performance of pixel p/., while the coevolution energy function E fills the loss.
Experiments
6.1 Experiment settings
Datasets and models. We validate the effectiveness of our proposed attack method on four frequently- used datasets, including ICDARI3 [7] (horizontal regular scene text, 1015 images), SVTP (26] (perspective distorted text, 645 images), CUTESO (28 (curved scene text, 288 images), and IC- DAR15 [16] (irregular scene text, 2077 images). We fool three mainstream paradigms of scene text recognition, including the CTC-based paradigm (CRNN (29)). the attention-basee paradigm (ASTER
30
and SAR (20), and the multi-modal paradigm (IGTR (6). All the models are from MMOCR
18 or OpenOCR (24). If not specified, the experiments are conducted on the CRNN model and the
CUTESO dataset for convenience.
Metrics. We employ four metrics including the attack success rate before/after text correction (SR/SR*), the perturbation rate (PR), and the (L2) distance between the adversarial image and the original image. Specifically, we employ GPT-4o mini [14] to correct the mispredicted text results and then calculate the SR* metric. The PR metrie measures the proportion of the mispredicted characters,
PR D(Y,, F(x,"))
(17)
len(Y:)
where 5 represents the total number of successful adversarial examples, len(.) denotes the length of the text, and -). is the edit distance between the two predicted texts. The L2 metric measures the image perturbation degree.
Text correction. Many STR methods Language models can do the same job, and they are more flexible. So we choose to employ GPT-40 mini [14 for text correction and then calculate the SR metric. Further details are in Appendix C
extra dictionary correct their prediction 22
30
6.2 Comparison with pixel-level attacks
We replicate two state-of-the-art black-box pixel-level attack methods, AD k7 and OnePixel 33 which are also DE-based methods, and compare their attack performance with ours, shown

---

Table 1: Comparison with other pixel-level attack methods. SR* is the attack success rate after correction by GPT-4o mini [4]. PR denotes the character perturbation rate.
Model
Method
CUTES0
ICI3
SVTP
SR. 1
SR
PR. 1
L2.
SR. 1
SR 7
PR.
1.2. |
SR.
SR 1
PR. 1
1.2.
OnePixel
95.60
60.09
52.26
4.25
95.31
43.78
49.82
3,98
96.76
58.06
42.99
3.93
CRNN
ADR
95.60
49.73
29.75
4,20
96,31
36.24
29.77
3,97
96.99
42.61
23.50
3.91
MPCS
87.36
86.75
94.54
4.30
85.47
70.12
90.19
4.02
85.65
77.67
80.07
4.04
OnePixel
57.98
20.59
39.18
4.23
41.86
14.05
39.63
3.95
68.17
28.93
33.41
3.89
ASTER
AD
57.14
16.93
28.67
4.22
47.66
13.48
32,94
3,96
67.45
22.35
22.06
MPCS
51.26
27.77
74.93
4.32
43.31
21.60
83.05
3.98
57.17
42.14
51.79
3.95
OnePixel
65.98
19.26
43,40
4.20
42.52
16.16
41.40
3.91
58.45
18.89 89.89
28.67
3.87
SAR
AD*E
70.08
25.00
30.68
4.18
42.52
14,10
28.51
3.88
57.67
17.30
21.56
3.87
MPCS
58.61
33,76
66.02
4.19
34.22
15.77
55.31
3.82
48.35
25.96
45.01
3.83
OnePixel
41.38
24.29
61.55
4.35
24.27
17.19
55.66
4.02
41.41
24.04
54.22
4.03
IGTR
ADRE
42.24
25.00
36.04
4.31
23.95
14.95
33.14 14
3,97
42.29
18.94
30.28
4.02
MPCS
61.21
49.14
89.86
4.44
39.67
32.05
82.86
4.16
54.87
37.95
79.29
4.22
in Table In specific, we set the number of attack pixels to 10 (the same as ours) when reproducing these methods. As shown in Table our method achieves the highest perturbation rate (PR) on four mainstream STR models and three widely used datasets. For example, when attacking the SOTA STR model IGTR (6), our method achieves 89.86% 90,86, 82. 86%, and 79.29% PRs on the CUTES0, IC13, and SVTP datasets respectively, surpassing the second-best method by 28.31 %6 27.21 %, and 25.07%.
Furthermore, when attacking the CRNN, ASTER, and SAR models on the CUTES0 dataset, our method achieves 26.66%, 7.18%, and 14.5% higher SR* metrics than the second-best method, although our metrics slightly lower due to the joint perturbation shift. It demonstrates that our attack method is much more irreversible and that the PR metric can effectively evaluate the attack's s irreversibility.
Table 2: Comparison with other attack paradigms. Sparse-rs (2) is black-box attack method targeting image classification models. UDUP [4] is a transfer- based attack method targeting OCR systems. Xu et al. (6) propose white-box attack method targeting STR models.
CUTESO
IC13
SVTP
Model
Method
SR. f
PR.
1,2. +
SR.
PR.
L,2,
SR.
PR. f
L2, |
Sparse-rs
63.73
40.23
11.22
56.77
38.10
11.44
76.11
52.70
9.80
UDUP
2.76
82.86
0.63
0.23
13.33
0.14
0
0.19
CRNN
Xu
33.51
30.35
2.88
30.95
41.20
3.06
49.65
21.47
2.85
MPCS
87.36
94.54
4.30
85.47
90.19
4.02
85.65
80.07
Sparse-rs
12.55
7.65
11.41
9.50
6.84
11.46
22.18
11.27
9.88
UDUP
0,97
30.00
1.20
0.12
16.67
0.15
0.20
ASTER
Xu
36.26
35.38
2.85
32.84
38.70
3.04
$6.29
23.04
MPCS
51.26
74.93
4.32
43.31
83.05
3.98
57.17
51.79
3.95
Sparse-rs
19.23
11.65
11.43
12.47
9.54
11.45
24.77
12.03
9.76
SAR
UDUP 3.70
107.45
1.78
0)
0.20
0.18
16.67
0.17
MPCS
58.61
66.02
4.19
34.22
55.31
3.82
48.35
45.01
6.3 Comparison with other attack paradigms
To demonstrate the superiority of our proposed method in attacking STR models, we compare our method with several other attack paradigms, as shown in Table Xu al. (36) propose a white- box attack method that updates the perturbation iteratively by optmization. When reproducing its code, we set the upper limit of the L2 metrie to 10 to prevent image distortion caused by excessive perturbation, while our 1.2 metrics are less than 5. On the CRNN model and CUTE80 dataset, the SR and PR of this method are 53.88 53.85% and 64. 19% lower than ours. This because this method can hardly find successful perturbations under low perturbation limitations.

---

Table Ablation study of our method. MP means the multi multi- population mechanism, CEF denotes the coevolution energy function, MG represents that the population will evolve to the maximum generation (i.e. will not stop when the attack is successful), and ES is the early stop mechanism.
Method
MP
CEF
MG
ES
SR.
SR
PR.
L2. +
Baseline
95.6
55.58
29.75
4.20
76.37
52.95
52.37
4.32
/
88.46
75.42
71.43
4.30
MPCS
V
/
90.66
87.20
99.72
4.29
87.36
86.75
94.54
4.30
Besides, we compare our method with UDUP 4 a transfer-based attack method that fools OCR models by generating adaptive underpainting. As shown in Table 2 this method has extremely low SR when attacking STR models, up to 84.6% lower than ours. This because UDUP cannot successfully add underpainting to scene text images with complicated backgrounds. Furthermore, we adopt Sparse-rs 22, a black-box attack method designed for image classification models, for attacking text recognition models by modifying its victim models and loss function. Our method outperforms Sparse-rs with +23.63% SR, +54.31% % PR, and -6.92 1.2 on CUTESO and CRNN. These results demonstrate that attack methods tailored for image classification models are not suitable for attacking text recognition models.
6.4 Ablation study
In this section, we conduct comprehensive ablation experiments on the four main components of our proposed method. We reproduce AD (37] under our experiment settings and use it as our baseline. As shown in Table after replacing the single population the baseline with our proposed multi-population (MP), the PR increases by 22.62% while the SR decreases by 19.23% (due to the joint perturbation shift problem). As shown in Figure 3 with the spatial constraints imposed by MP, adversarial pixels within each population gradually converge to small regions that are evenly distributed across the entire image during the evolution process. Next, we employ the coevolution energy function (CEF) to get SR and PR increased by 12.09% and 19.06%, demonstrating its effectiveness.
Furthermore, in order to achieve a better performance, we force all the populations to evolve until the maximum generation (MG) rather than stopping evolution when the attack is successful. The fourth row shows that this solution gains an improvement of 2.2% and 28. 29% on SR and PR, respectively. Meanwhile, we propose the early stop (ES) mechanism balance the extra time consumption caused by MG. According to the last row in Table 3 it has small impact on the attack performance. And the average total number of evolutionary generations with ES is less than that without ES, as shown in Figure
SUBLEUE
GUBLLUE OODONN
"london"
LONDON
N
"lunican"
Geseration w/. ES Geveration PS
PR w/ ES
PR veoo FPS
110
HOND
910
100
90
92
50
92
9r
Cevration
S-
70
5
50
34
CUTER
IC13
IC15
SVIP
Figure 3: Population evolution visualization.
Figure 4: Effect of the early stop (ES).

---

Table Influence of the population number N and the population size M. We set up experiments on two datasets while keeping the sum of all population sizes constant.
CUTESO
IC13
N, M
SR.
PR. t
I22
SR.1
PR.
1.2. +
(4, 150)
75.27
61.84
2.75
69.60
56.19
2.59
(5, 120)
77.47
70.23
3.09
73.19
60.75
2.90
(6, 100)
84.61
71.83
3.37
74.41
63.93
3.17
(10, 60)
87.35
94.54
4.30
80.11
87.23
4.02
(20, 30)
91.20
124.52
6.03
88.27
133.77
5.72
6.5 Influence of hhper--pramne pper-parameters
Firstly, we investigate the influence the population number and the population size P,, as shown in Table 4 We keep the of all population sizes constant, ie., N x M 600, and adjust the combination of N and M. As increases to 20 on CUTES0, the SR, PR, and 1.2 rise to 91 20%0 124.52%, and 6.03, respectively. We adopt N 10, which also has promising attack performance with smaller perturbation.
6.6 Correction by the dataset dictionary
We also conduct experiments on a more traditional text correction method, i.e., through an extra dictionary. We use the test set vocabulary of the IC15 dataset as the dictionary to correct the results of successful attacks through our method exhibits much higher SR and PR compared to both OnePixel [33] and ADÒE (37]. These results demonstrate that our method still has better attack irreversibility under this setting.
shown Table among the attack results on the three STR models,
6.7 Effect range of adversarial pixels
conduct experiments to investigate the effect range of adversarial pixels. In specific, we restrict the location of adversarial pixels ensure that some characters do not have adversarial pixels around them. As shown in Table 6) when the adversarial pixels are restricted to the left half and right half of the image, the PR metric decreases by 20.42% and 13.68%, respectively. These results demonstrate that the adversarial pixel has higher possibility of affecting nearby characters rather than farther characters.
Table 5: The attack results the ICI5 dataset. SR* the attack success rate after correction by the dataset dictionary.
Table 6: The attack performance the CUTESO dataset when the perturbation area is limited the left and right half of the image, respectively.
CRNN
ASTER
SAR
Method
SR*,
PR. 1
SR
PR. 1
SR*, 1
PR.
Perturbation Area
SR. 1
PR. 1
I.2. |
OnePixel
42.15
77.93
19.19
62.18
21.10
65.20
Left 32.97
74.12
3.38
ADDE
29.63
50.31
18.32
51.17
17.23
50.24
Right
39.01
80.68
3.65
MPCS
71.19
94.48
35.53
78.27
27.84
73.87
All
87.36
94.54
4.30
7
Discussion and limitations
In this work, we propose pixel-level black-box attack method targeting at altering each character in STR models' prediction, termed as MPCS. By modifying only a few pixels the image, MPCS can mislead STR models to predict more incorrect characters than previous methods, while maintaining whole visual semantic information. We believe MPCS offers valuable applications while requiring careful consideration of its societal implications. On the positive side, the generated adversarial examples can strengthen STR models through adversarial training, improving their robustness against real-world perturbations. Additionally, MPCS enables effective privacy protection by preventing

---

unauthorized OCR extraction of sensitive text with minimal visual distortion. For example, when users upload images social platforms, we could add tiny perturbation pixels to the images so that the private texts may not be correctly recognized and collected by malicious OCR systems. Besides, the potential misuse of such adversarial attacks raises important concerns. Malicious actors could exploit MPCS to bypass security systems, manipulate automated text recognition, evade content moderation, highlighting the need for responsible research practices and countermeasures.
Meanwhile, although MPCS does not affect the viewing of text images by human eyes, the pixels it disturbs are still quite different from the original pixels. In future work, we will investigate finding better pixel value optimization method to minimize the deviation between adversarial and original pixel values, thus enhancing the attack imperceptibility.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (62302532, 62411540034), the Natural Science Foundation of Guangdong (2025A151501 100251515011224 1224), and Shenzhen Science and Technology Program (202206193000001, 202208162255223 KOTD2021100009559015
References
[1] Yanbo Chen and Weiwei Liu. A theory of transfer-based black-box attacks: Explanation and implications. In Advances in Neural Information Processing Systems, volume 36, pages 13887-13907, 2023.
[2] Francesco Croce, Maksym Andriushchenko. Naman D Singh, Nicolas Flammarion, and Matthias Hein. Sparse-rs: versatile framework for query-efficient sparse black-box ad- versarial attacks. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 36, pages 6437-6445, 2022.
[3] Swagatam Das and Ponnuthurai Nagaratnam Suganthan. Differential evolution: A survey of the state-of-the-art. IEEE Transactions on Evolutionary Computation, 15(1):4-31, 2010
[4] JiaCheng Deng, Li Dong, Jiahao Chen, Diqun Yan, Rangding Wang, Dengpan Ye, Lingchen Zhao, and Jinyu Tian. Universal defensive underpainting patch: Making your text invisible optical character recognition. In Proceedings of the ACM International Conference Multimedia, pages 7559-7568, 2023.
[5] Yinpeng Dong, Shuyu Cheng. Tianyu Pang, Hang Su, and Jun Zhu, Query-efficient black-box adversarial attacks guided by transfer- based prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44( 12):95333336) 954533666848,,, 2021.
[6] Yongkun Du, Zhineng Chen, Yuchen Su, Caiyan Jia, and Yu-Gang Jiang. Instruction eestruction-guided scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4):2723-2738. 2025.
[7] Shuman Fang, Jie Li, Xianming Lin, and Rongrong Ji. Learning to learn transferable attack. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 571-579, 2022.
[8] Alvaro Gonzalez, Luis M Bergasa, and J Javier Yebes. Text detection and recognition traffic panels from street-level imagery using visual appearance. IEEE Transactions on Intelligent Transportation Systems, 15(1):228 22-2288888.... 2013.
[9] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.68 412.6657,,222 2014.
[10] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the International Conference Machine Learning, page 369-376, 2006.
10

---

[11] Yanru He, Kejiang Chen, Guoqiang Chen, Zehua Ma. Kui Zhang, Jie Zhang, Huanyu Bian, Han Fang. Weiming Zhang, and Nenghai Yu. Protego: Protect text content against ocr extraction attack. In Proceedings of the ACM International Conference on Multimedia, pages 7424-7434, 2023.
[12] Lei Hsiung, Yun Tsai, Pin-Yu Chen, and Tsung-Yi Ho. Towards compositional adver- sarial robustness: Generalizing adversarial training to composite semantic perturbations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[13] Mingxin Huang, Dezhi Peng, Hongliang Li, Zhenghao Peng, Chongyu Liu, Dahua Lin, Yuliang Liu, Xiang Bai, and Lianwen Jin. Swintextspotter v2: Towards better synergy for scene text spotting. International Joumnal of Computer Vision, pages 1-21, 2025.
[14] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.212266 2024.
[15] Niddal H. Imam, Vassilios G. Vassilakis, and Dimitris Kolovos, Ocr post-correctio for detecting adversarial text images. Journal of Information Security and Applications, 66:103170 2022.
[16] Dimosthenis Karatzas, Lluis Gomez- Bainod,, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasckhar, Shijian Lu, et al. ledar 2015 competition on robust reading. In International Conference on Document Analysis and Recognition, pages 1156-1160, 2015.
[17] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Big- orda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. ledar 2013 robust reading competition. In International Conference on Document Analysis and Recognition, pages 1484-1493, 2013.
[18] Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue, Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao, Wenwei Zhang, Kai Chen, Wayne Zhang, and Dahua Lin. Mmoer: A comprehensive toolbox for text detection, recognition and understanding. In Proceedings of the ACM International Conference on Multimedia, page 3791-3794, 2021.
[19] Vladimir I Levenshtein al. Binary codes capable of correcting deletions, insertions, and reversals, In Soviel Physics Doklady, volume 10, pages 707-710, 1966.
[20] Hui Li, Peng Wang, Chunhua Shen, and Guyu Zhang. Show, attend and read: A simple and strong baseline for irregular text recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):86100 (100)))860--117, 2019.
[21] Jiquan Li, Junliang Guo, Yongxin Zhu, Xin Sheng, Deqiang Jiang, Bo Ren, and Linli Xu. Sequence-to-actionn Grammatical error correction with action guided sequence generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10974-10982, 2022.
[22] Canjie Luo, Lianwen Jin, and Zenghui Sun. Moran: A multi-object rectified attention network for scene text recognition. Pattern Recognition, 10:109-118, 2019.
[23] Seyed- Moosavi- -ezaail,,, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2574-2582. 2016.
https://githus
[25] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow: Transferability in machine learn- ing: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv: 2016.
[26] Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recognizing text with perspective distortion natural scenes. In Proceedings of the IEEE International Conference Computer Vision, pages 569-576, 2013.

---

[27] Rishi R. Abhishek S, Anudeep N, and V Vivek. A survey on advanced text recognition and projection in augmented reality. In International Conference on Advances in Computing. Communication Control and Networking, pages 1085-1089, 2022.
[28] Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for natural scene images. Expert Systems with Applications, 41(18):80277-044) 2014.
[29] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image- based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 39(11))2299) 2304, 2016.
[30] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Aster: An attentional scene text recognizer with flexible rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2035 2042000052288,,,.0 2019.
[31] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
[32] Congzheng Song and Vitaly Shmatikov. Fooling ocr systems with adversarial text images. arXiv preprint arXiv: 053885 2018.
[33] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 23(5):828 841,, 2019.
[34] Yujin Takahashi, Satoru Katsumata, and Mamoru Komachi. Grammatical error correction using pseudo learner corpus considering learner's error tendency. Proceedings of the Annual Meeting of the Association for Compulational Linguisties: Student Reseurch Workshop, pages 27-32, 2020.
[35] Shudeng Wu, Tao Dai, Guanghao Meng, Bin Chen, Jian Lu, and Shu-Tao Xia. Transferable adversarial attacks for deep scene text detection. In International Conference on Pattern Recognition, pages 8945-8955 2021.
[36] Xing Xu, Jiefu Chen, Jinhui Xiao, Lianli Gao, Fumin Shen, and Heng Tao Shen. What machines see not what they get: Fooling scene text recognition models with adversarial text images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12304-12314, 2020.
[37] Yikun Xu, Pengwen Dai, Zekun Li, Hongjun Wang, and Xiaochun Cao. The best protection is attack: Fooling scene text recognition with minimal pixels. IEEE Transactions on Information Forensics and Security, 18:1 550--1955, 2023.
[38] Mingming Zha, Guozhu Meng. Chaoyang Lin, Zhe Zhou, and Kai Chen. Rolma: practi- cal adversarial attack against deep learning-based Ipr systems. In Information Security and Cryptology, pages 101-117, 2020.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction in Sec. I clearly state the main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec. 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly describe the key steps in our method in Sec. settings are also detailed in Sec. 6
The experiment
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
We provide the code as the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htte://alllllgiiiee///oeeeuaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Sec.
d
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars, method has very significant improvement, as shown in T Also, the number of experiments is too large, and the cost is too high.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec 6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/ps///.cccc..cco/oop/////a
Answer: [Yes]
Justification: We have carefully checked the Ethics Guidelines to make sure our research is with
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impacts in Sec
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The documentation is provided in the code repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: LLM is employed to correct the attack results. The details could be found in See and Appendix C
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19

---

Appendix
A Implementation details
In our experiments, we set the population number N and the population size M to 10 and 60, respectively. For population Pr, we set the population size L to 600. The max generation Imaz and Imax are set to be 10 and for low query budget. The height h and width of the input image are 32 and 100, respectively. The experiments are conducted on workstation with a single GeForce RTX 3090 GPU, 2.90 GHz Intel(R) Xeon(R) Gold 6226R CPU, and 64G RAM, based on Pytorch v2.0.1.
B Visualization
We visualize our adversarial examples and their corresponding attack results, and compare them with other attacks, as shown in [ Furthermore, against the CRNN model under different settings in Table g as shown in Figure 6 Typically, the attack performance becomes better with the increasing N. Besides, we notice that only few pixels are needed to attack single character, as shown in the second and third rows. Therefore, we can dynamically adjust the attack pixel number according to specific images to weaken the perturbation in practice.
visualize the generated adversarial examples
Method
Origin
Xu al.
OnePixel
ADCE
Ours
carp
Carp
carp
carp
carp
Image
Label
"carp"
"calp"
"cajile"
"calp"
"doeilits"
Image
ISLAND'S
SLAND'S
ISLAN'Ss
ISBAND'S
ISEAND'S
Label
"islands"
"islan_s" =
"isgangys"
"ishands"
"ssganjdng"
Image
ESTD
ESTD
ESTP
ESTD
EST
Label
"estd"
"esti"
"esitin"
"estp"
"fiitria"
Tokyo
Tokyo
Tokyo
Tokyo
Tokyo
Image
Label
"tokyo"
"tokya"
"tokyje"
"tokye"
"iokiie"
Image
Label
"20"
"200"
"sscy"
"202"
"geallan"
Image
DREAMS
DREAMS
DREAMS
DREAMS
DREAMS
Label
"dreams"
"cheams"
"chopims"
"dreags"
"brering"
Figure 5: Visualization of the adversarial examples and their attack results. Characters with underlines incorrectly predicted. Compared other methods, our approach exhibits extraordinary attack performance. Zoom in to get a better view.
C Text correction by LLM
We employ GPT-4o mini [14] to correct the mispredicted text results and calculate the SR* metric. The prompt is formed follows:
"Correct the spelling of the following words:
slacrest
=>
20

---

N
6
10
20
Origin
Image
CLUB
CLUB
CLUB
CLUB
CLUB
CLUB
Label
"callb"
"callb"
"calld"
"labls"
"Tablies"
"club"
Image
14
14
Labell
"ad44"
"hally"
"nidely"
"manf"
"alail"
"14"
Image
Label
"cess"
"cod"
"cod"
"CeSS"
"geswl" "
Image
Pu PUBLICK
PUBLICK
PUBLICK
PUBLICK
BLICK
PUBLICK
Label
"rubblack"
"plistpek"
"pubabick"
"fuistrer"
"foisbantes"
"publick"
Figure 6: Visualization our adversarial examples against the CRNN model under different attack settings. N is the population number, which is equal to the attack pixel number. The sum of population sizes is 600 in all experiments.
peach =>
staron
The output of the model may be like:
"Here's 's the corrected list of words:
slacrest => Seacrest
peach => peach (unchanged)
staron station
After we get the model output, we first remove the annotations like "(unchanged)" to avoid their influ- ence. Then convert all words to lowercase and compare the predicted words with corresponding labels in lowercase. If the predicted word different from label, we treat as a failed attack; otherwise, it is successful attack.
D The calculation of distance
We employ CTC Loss 10 to calculate the distance Dist(Y, F(Xade)).
[y1, ...; yol (n
means the gt character number) is the ground truth label, and F F(X denotes the model output. Since we employ the black-box attack setting where the model output F(Xadv) includes character sequence (9, 92, ... 2m] (m means the predicted character number) and a confidence score sequence [$1, 52; ... 5m] (s [0, i)), need preprocess facilitate the CTC loss calculation.
Firstly, we define the character dictionary Dict. Next, we generate a m x (1 + 1) matrix T as,
Saz
Dicl(ūa) b,
T(a, 6]
(18)
Dicl(ga) # b,
1+1
where Dicl(ÿa) represents the index of character j. in the dictionary. Then we apply the log-softmax function along the second dimension of tensor T obtain the matrix 7 of logarithmic probabilities. After that, we can calculate the CTC loss between 7 and Y.
21

---

AD'E OnePixel
10
Dist
CUTE80
IC13
IC15
SVTP
Dataset
Figure 7: The average shortest distance between adversarial pixels on the CRNN model.
Method
Origin
OnePixel
AD²E
MPCS
Disneyland
Disneyland
disneyland
Disnayland
Image
CRNN
Label
"disneyland"
"disneylajnd"
"disneylend"
"orsuapiera"
Disneyland
Disneyllnd
Disneylang
disneyiand
Image
ASTER
L.abel
"disueyland"
"distegland"
"disneyfand"
"otskeyjand"
Disneyland
Disnayland
Disneyland
Disnaylan
Image
SAR
Label
"disucyland
"disgieylum lend"
"disucy fand"
2yl2du
chocolates
chocolates
chocolates
chocolates
Image
CRNN
Label
"chocolates"
"chojcololes"
"chooolgjes"
"dhacidloiee"
chocolates
chocolates
chocolates
chocolates
Image
ASTER
I.abel
"cocollates
"chocglates."
"chgoglates"
"chiacaiates"
chocolates
chocolotes
chocolates
chocolates
Image
SAR
Label
"chocolates"
"chacolates"
"chocglates"
"thetciates"
Figure 8: Visualized comparison the CRNN, ASTER, and SAR models. The underlined characters are incorrectly predicted. The red rectangles images outline the characters with no nearby adversarial pixel. Zoom in get a better view.
22

---

100
CUTE80
90
IC13
SVTP
80
70
CSR(%)
60
40
30
20
10
20
10
50
70
100
PR())
Figure 9: The relation between the correction success (CSR) by GPT-4o mini and the perturbation rate (PR) of texts.
E Attack pixel spatial concentration problem
In this section, we demonstrate the attack pixel spatial concentration problem that exists in previous
pixel-level attack methods [3 37
Firstly, we give quantitative analysis by calculating the average shortest distance between adversarial pixels. Given N adversarial pixels (PI, P......... Px), we calculate the distance between each adversarial pixel and nearest adversarial pixel to get (d1, d2. ..., dN), and then calculate the average value of d. As shown in Figure 7 our method has greater average distances on four datasets than previous methods. This is because our proposed multi-population mechanism restricts the adversarial pixel location in the horizontal direction.
Furthermore, we give qualitative comparison by visualizing the attack results of OnePixel and AD 37 with
the CRNN [29] ASTER (30) and SAR 20 models, respectively. The attack pixel number is 10. As shown in Figure both OnePixel and AD2E have the adversarial pixel local aggregation problem when attacking the three STR models. The attack pixels concentrate within localized regions of the image, resulting in some characters having attack pixels nearby. Consequently, these characters can correctly recognized by the model. Meanwhile, they can only alter or two predicted characters most cases. Differently, our method effectively solves the adversarial pixel local aggregation problem through the multi-population mechanism. The attack pixels generated evenly distributed across the entire image along the horizontal direction, ensuring that each character in the image has at least one nearby attack pixel.
The perturbation rate metric
To evaluate whether PR metrics can accurately reflect the irreversibility of attacks, we visualize relationship between the correction success rate (CSR) by GPT-4o mini and the perturbation rate (PR) of texts. As shown Figure on the CUTESO (28), IC13 [17] and SVTP (26) datasets, the overall correction success rate decreases the perturbation increases. It demonstrates the effectiveness of the perturbation rate.
G Transfer attack on commercial OCR system
implement transfer attacks on BaiduOCR (https://clous baidu. Coom/dcc/CRR, index html) using adversarial examples that can successfully attack SAR. As shown in Figure [0] the transfer attack achieves 53.44%, 63.51%. % and 62.65% LPR on IC13, IC15, and SVTP datasets,
respectively. Some qualitative results are also shown in Figure I0 These results demonstrate that
23

---

Dataset
IC13
IC15
SVTP
53.44
63.51
62.65
Image
for
Premier
KY RO
Pred
for +
premier
kyro -
Image
30 sGy Joymil
BEADS
Pred
30 -
soymilk - 50
beads beal
Figure 10: The result of conducting transfer attack on BaiduOCR using adversarial samples obtained by attacking the SAR model.
SEACREST
X:
Y: "seacrest"
EECREST
xadv
F(Xadv): "slagrese"
Figure 11: Metric computation example.
method has good transferability and is still capable of misleading the model to predict a significant number of incorrect characters during the transfer attack.
H Metric computation example
We provide a real metric computation example in this section. As shown in Figure I1 the original image X has a predicted label Y "seacreast", and the adversarial example image Xadoo is recog- nized by the STR model F as F(Xadv) = "slagrese".. Then, the edit distance between their predicted labels is D(Y, D(Y,F(Xxd1)) = 3, and the length of the ground truth is len(Y) = 9. Therefore, the perturbation rate (PR) of this adversarial example PR D(Y,F(X=du)) 33.33%. Besides, the len[Y) L2 metric evaluates the average L2 distance between the original image and the adversarial example image: L2 Ei ||X;
X X: ||2. In this example, the L2 distance is 3.62. As for the success rate before/after correction (SR/SR*), for example, if the number of samples is 1000, and 800 of them are successfully attacked, then the SR is 80%. Next, we use an LLM to correct the mispredicted texts. 200 of them cannot be corrected, then the SR is 20%.
I Evaluation on non-Latin scripts
To further examine the generality of our proposed method beyond Latin scripts, we additionally conducted experiments on a Chinese text recognition benchmark. Our main experiments rely on STR models provided by MMOCR and OpenOCR, both of which trained large-scale English datasets. Consequently, our primary evaluation focuses on four widely used English benchmarks.
However, method does not depend on any script-specific assumptions, expect it to generalize to non--attin scripts as well. To validate this, we performed a comparative study on the dataset. introduced in "Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study." Specifically, we randomly sampled 200 images from this benchmark and applied both our attack and the baseline method to the IGTR model, which trained on Chinese datasets by OpenOCR. Our method achieved success rate (SR) of 51.85% and perturbation rate (PR) of 34.44%, outperforming the baseline (SR: 38.89% PR: 12%).. These results demonstrate that our approach maintains strong effectiveness on non-Latin scripts, further confirming its generalizability across different languages and writing systems.


Reviews:
[{"review_id": "wMBSeMdy0d", "reviewer": "Reviewer_cRqd", "summary": "In this paper the authors present a pixel-level blackbox attacked method targeting  to manipulate scene text images. The authors use  a method named an MPCS to attack each character in the image to avoid being easily corrected by external dictionaries, and shows the effectiveness of their proposed method using four STR datasets.", "questions": "1.  The authors mention that  they used early stopping to balance the extra time caused by maximum generation (MG). Is it possible to quantify the average attack time per  sample text-line image with and without early stopping? \n2.  I have already mentioned it in the strength and weakness part above.  Since there are many Non-Latin scent text datasets, is there a specific reason for selecting the Latin script? Was it intentional?\n3. The authors provide different metrics such as SR/SR*, PR, and L2 distance, is it possible to provide sample real examples and compute the values using each metric,  the examples can be include in the supplemental part?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n- The paper is well written and easy to follow. \n- The experiment section provides detailed results with clear and concise reasoning for the hyperparameters and the methods chosen.\n\n**Weaknesses**\n- The method presented in this paper is evaluated solely on the Latin/English script, and it is unclear whether it can be extended to different scripts or it is script dependent. It would be nice if the authors could test additional Non-Latin scene texts.", "was_revised": true, "final_justification": "The questions i raised have been addressed, and the authors also presented a new experiment for testing their method on Non-latin scripts, which I had previously commented on as a weakness. So, I finalize my rating as accept.", "rebuttal": "**Q1: Average attack time with early stop.**\nWe thank the reviewer for this good question.\nThe attack time varies across different devices. \nOn our device with a 3090 GPU, the average attack time on CRNN with the CUTE80 dataset is 18.23 seconds w/. ES and 37.17 seconds w/o. ES.\nThis result demonstrates the effectiveness of the ES mechanism.\n\n**Q2: Datasets selection.**\nWe thank the reviewer for pointing this out. \nWe agree that evaluating on non-Latin scripts would further validate the generality of our method. \nOur current experiments rely on STR models from MMOCR and OpenOCR, which are trained on large-scale English datasets.\nHence, we have only conducted evaluations on four widely used English datasets.\nImportantly, our method itself does not rely on script-specific assumptions, and we expect it to generalize to non-Latin scripts. \nTo validate this, we conducted a set of comparative experiments on a Chinese text recognition benchmark (from “Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study”), comparing our method with $\\mathrm{AD}^2\\mathrm{E}$.\nDue to time constraints, we randomly select 200 images from this benchmark and apply both attacks on the IGTR model, which is trained on Chinese datasets by OpenOCR.\nOur method achieves a success rate (SR) of 51.85\\% and a perturbation rate (PR) of 34.44\\%, while $\\mathrm{AD}^2\\mathrm{E}$ achieves 38.89\\% SR and 21.12\\% PR.\nThese results suggest that our method generalizes well to non-Latin scripts.\n\n\n**Q3: Metric calculation example.**\nWe thank the reviewer for the suggestion. \n\n(1) The perturbation rate (PR) metric.\n\n\\begin{equation}\n    \\textbf{PR} = \\frac{1}{S}\\sum_{i=1}^{S}{\\frac{D(Y_i, \\mathcal{F}(X_i^{adv}))}{len(Y_i)}},\n\\end{equation}\n\nFor example, if the ground truth $Y_i$ is \"ABCD\", and the predicted label $\\mathcal{F}(X_i^{adv})$ is \"CDAB\", then the edit distance between them is $D(Y_i, \\mathcal{F}(X_i^{adv}))=2$.\nThe length of the ground truth is $len(Y_i)=4$.\nSo the perturbation rate of this sample is 50\\%.\n\n(2) The success rate before/after correction (SR/SR*).\nFor example, if the number of samples is 1000, and 800 of them are successfully attacked, then the SR is 80\\%.\nNext, we use an LLM to correct the mispredicted texts.\nIf 200 of them cannot be corrected, then the SR* is 20\\%.\n\n(3) The L2 distance metric.\nThis metric evaluates the average L2 distance between the original image $X_i$ and the adversarial example image $X_i^{adv}$:\n\\begin{equation}\n    \\textbf{L2} = \\frac{1}{S}\\sum_{i=1}^{S}||X_i - X_i^{adv}||_2.\n\\end{equation}\n\nWe will provide such examples in the supplementary material."}, {"review_id": "iU2QXM71GG", "reviewer": "Reviewer_1goM", "summary": "This paper aims to attack the STR model with irreversible output since text string is contextual and less wrong character can be corrected with dictionary or LLMs. They propose the Multi-Population Coevolution Search (MPCS) method to attack each character in the image. They  decompose the global optimization objective into sub-objectives to solve the attack pixel concentration problem existing in previous attack methods. To address the new combined perturbation shift problem, they propose a coevolution energy function and make the theoretical analysis. Experiments on CRNN, ASTER, SAR and IGTR demonstrate the results.", "questions": "See \"Weaknesses\".", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) Finding the phenomenon that text string attacked with few wrong characters is easily recovered is valuable since in many scenarios the dictionary is used and the LLMs' abilities progress rapidly.\n2) The proposed MPCS performs better in irreversible settings.\n3) The theoretical analysis guarantees the superiority of the proposed coevolutionary energy function.\n\nWeaknesses:\n1) The proposed method is complex and the SR rate decreases with large margins. SR* is important, but SR is also an important metric to evaluate different methods, the decrease from 90%+ to 80%+ demonstrates that the proposed method's success rate is too low. \n2) Equation 5, theta_max is not defined, and is it (255, 255, 255)?\n3) Tab.1, the results of SAR+IC13 show the proposed method is ineffective, please explain it in detail.\n4) Tab.2 lacks SR* results, please explain.\n5) Tab.3, ES gives the negative effect. In my opinion, the attack time overhead is not so important than that of SR.", "was_revised": true, "final_justification": "See my latest comment.", "rebuttal": "**Q1: Decreased SR.**\nWe thank the reviewer for highlighting this concern. \nThe decrease in SR mainly stems from two factors: (1) the combined perturbation shift phenomenon, and (2) the early-stop mechanism. \nTo address the first, we introduce the CEF, which effectively alleviates its impact and improves SR, as shown in Table 3. \nThe early-stop mechanism, on the other hand, is designed to reduce the generation number (Figure 4), representing a deliberate trade-off between SR and attack time.\n\nBesides, we would like to emphasize that our primary goal is to design **irreversible attacks**. \nIn STR scenarios, dictionary-based correction is a common post-processing step, so attacks that remain effective after such correction are of greater practical significance. \nThus, SR* (success rate after correction) and PR (perturbation rate) are more indicative of performance than SR (before correction). \nImportantly, the SR drop after correction (SR $-$ SR*) reflects attack reversibility: our method shows only a slight decrease (0.61\\%, 87.36\\% $\\rightarrow$ 86.75\\%), whereas the second-best method suffers a drastic drop (35.51\\%, 95.60\\% $\\rightarrow$ 60.09\\%).\n\n\n**Q2: Symbol definition.**\nWe thank the reviewer for pointing out this error.\nIndeed, the $\\theta_{max}$ in equation 5 is (255, 255, 255). \n\n**Q3: Lower performance.**\nWe thank the reviewer for highlighting this concern.\nWe believe there are three main factors contributing to the relatively lower performance on SAR+IC13.\n(1) The IC13 dataset consists of easy samples with normal fonts aligned horizontally, which increases the attack difficulty. \n(2) The SAR model is designed for irregular text recognition, which means it is naturally robust against adversarial examples. \n(3) **Most importantly**, OnePixel has no constraint on the attack pixel location, so it could allow many attack pixels to concentrate in a small region and change a single character. That explains its higher SR.\nHowever, we want to note that our perturbation rate (PR) is still the highest (55.31\\% vs. 41.40\\%), and the SR drop after correction (SR $-$ SR*) is also smaller than other methods (18.45\\% vs. 26.35\\%).\nAs we explained in **Q1**, these metrics are much more important for evaluating attack irreversibility, and the results demonstrate the best irreversibility of our attack method.\n\n**Q4: Missing metric in Table 2.**\nWe thank the reviewer for highlighting this concern.\nWe didn't calculate the SR* metric in Table 2 because it is redundant and expensive. \nAs shown in Table 2, compared to other attack paradigms except for UDUP, which has extremely low SRs, **our method achieves much higher SR (9.54\\% $\\sim$ 38.71\\%) and PR (27.37\\% $\\sim$ 67.28\\%)**. \nSince a higher SR (before correction) and PR (more incorrect characters) directly imply a higher SR* (after correction), we believe the SR* metric is not necessary in this table.\n\nDue to time constraints, we compute the SR* metric for the second-best method only (Sparse-rs), as a representative comparison.\nOn CRNN and CUTE80, Sparse-rs achieves an SR of 63.73\\% and a PR of 40.23\\%, which are 23.63\\% and 54.31\\% lower than our results, respectively.\nAfter applying text correction, its SR* drops to 22.68\\%, which is 64.07\\% lower than ours.\nThese results indicate that a lower PR typically leads to a more significant drop in success rate after correction (i.e., SR - SR*), further supporting the advantage of our method.\n\n**Q5: The ES mechanism.**\nWe thank the reviewer for highlighting this concern.\nThe ES mechanism provides a trade-off between attack time and SR. If the attackers want the best SR regardless of attack time, they can cancel the ES mechanism by setting the parameter \"early_stop\" to 0 in our provided code."}, {"review_id": "q1Kp0EdAlh", "reviewer": "Reviewer_1ydv", "summary": "This article proposes a novel black box adversarial attack method - MPCS, for STR models. There are two key issues with existing adversarial attack methods: Information residue, the predicted results after the attack still retain some of the original text information, which is easily corrected by language models; Pixel concentration leads to uneven spatial distribution of perturbed pixels, resulting in some characters not being effectively attacked. The core contribution of this article is to achieve a globally optimized distribution of perturbed pixels through horizontal partitioning and co evolutionary energy functions; Propose and solve the combined perturbation offset problem and prove the information lossless nature of the energy function.", "questions": "I am curious about how formula 7 balances local optimization and global optimization? Will this process fall into the dilemma of local optimality?\n\nTable 4 shows that when N=20, L2=6.03. Will non-uniform partitioning reduce distortion?\n\nIn fact, the disturbance points generated by this method are obvious, although the overall disturbance value is not high. Is there a way to make the disturbance points more hidden?\n\nCan the number of populations and the size of populations be automatically adjusted? This is more conducive to the optimization of this method.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper presents a systematic attack framework MPCS that tackles a critical challenge in adversarial attacks on STR models: irreversibility. The multi-population coevolution search is well-motivated and addresses the pixel concentration problem effectively.\n\n## Strengths :\n\nThe authors tested it on 4 standard datasets (ICDAR13/15, SVTP, CUTE80) and 3 STR model types (CTC-based, attention-based, multimodal). At the same time, it was evaluated against 6 baseline models (OnePixel, AD²E, Sparse-rs, UDUP, Xu et al., transfer attack). The perturbation rate (PR) was introduced to measure the irreversibility of the attack and verified by correction based on GPT-4.\n\nThis paper explicitly models irreversibility in STR attacks, going beyond simple misclassification. The authors formally define the combined perturbation shift problem and prove that the co-evolutionary energy function can avoid information loss. Meanwhile, MPCS ensures that the perturbation is evenly distributed. The early stopping mechanism balances effectiveness and efficiency.\n\nThe authors demonstrate that even state-of-the-art STR models (e.g., IGTR) are vulnerable to irreversible attacks and propose how adversarial examples can protect sensitive text (e.g., license plates, ID cards) from unauthorized OCR extraction.\n\n## Weaknesses：\n\nThe performance of the method proposed by the author is reduced for long texts. It is only tested on the CNN/RNN-based STR model, and no experiments are done on Vision Transformers or diffusion-based OCR models to verify the effect of the method. Besides, lack of exploration on the defense mechanism of this method.\n\nFormula 7 does not clearly describe the connection between local optimization and global optimization. Can it be theoretically guaranteed that the method finds the global optimum without falling into the local optimum?\n\nAt the same time, this article does not consider whether the current mainstream defense measures can defend against the method of this model. For example, can the adaptive STR model detect MPCS perturbations?\n\nAlthough this article emphasizes privacy protection, the scenario of malicious applications can be better discussed.\n\nWhat are the usage scenarios of this method and is there any possibility of its application in real-life scenarios?", "was_revised": true, "final_justification": "The responses have mostly addressed my concerns. Based on the feedback from the authors and the other reviewers' comments, I would like to keep my original score as BA.", "rebuttal": "**W1: Reduced performance on long texts.**\nWe thank the reviewer for highlighting this point.\nWe conduct some experiments on CRNN and CUTE80 to explore the performance on long texts and short texts under the same setting.\nWhen the text length is greater than 4, the success rate (SR) is 84.96\\% and the perturbation rate (PR) is 63.33\\%.\nIn contrast, for texts with length $\\leq 4$, the SR is 89.86\\% and PR is 134.29\\%.\nThis indicates that longer texts are indeed harder to attack, as the number of attack pixels per character decreases with text length.\nHowever, as shown in Table 4, increasing the total number of attack pixels can effectively improve performance on long texts.\n\n**W2: Attack on transformer or diffusion-based models.**\nWe thank the reviewer for this question.\nAs shown in Table 1, we have already conducted experiments on IGTR (TPAMI 2025), which is a multi-modal STR model with a transformer architecture. As for the diffusion-based STR model, to the best of our knowledge, there is no open-sourced diffusion-based STR model to implement experiments on.\n\n**W3: Against defense mechanism**\nWe thank the reviewer for this great question.\nAs our best-known, mainstream defense mechanisms against our attack include adversarial training, post-correction, and pre-processing.\nFirst, adversarial training adds adversarial examples to the training data to enhance model robustness. Our attack method could also benefit adversarial training by generating adversarial examples.\nSecond, post-correction is already included in our attack setting. Our method could effectively disable the post-correction.\nFinally, pre-processing such as image filtering could remove most of the perturbations, but it would also distort the image quality. \nAs for the \"adaptive STR model\" mentioned by the reviewer, we are not aware of a specific method with this name.\nWe would greatly appreciate it if the reviewer could provide more details, which would help us better understand and address this point.\n\n\n**W4/Q1: Question about the CEF.**\nWe thank the reviewer for this insightful question.\nFirst, regarding the coevolution energy function (CEF, Eq. 7), the \"global constraint\" refers to the information of the best pixels from the other $N-1$ populations. Specifically, we generate a combined perturbation $\\Delta(\\lbrace p_i^j \\rbrace \\cup \\lbrace p_{k}^* \\rbrace_{N-1})$ and evaluate its attack performance to determine the energy of the current attack pixel $p_i^j$.\nSecond, our method employs a DE algorithm within each population to search for attack pixel locations. As DE itself cannot guarantee finding the global optimum, our method also cannot ensure a global optimum.\n\n**W5: Application scenarios.**\nWe thank the reviewer for this valuable suggestion.\nA specific scenario in privacy protection is: when users upload images to social platforms, we could add tiny perturbation pixels to the images so that the private texts may not be correctly recognized and collected by malicious OCR systems.\nExcept for privacy protection, our attack method could potentially be applied in other contexts, such as poisoning training data, misleading OCR systems, or concealing toxic or prohibited content.\nWe will further elaborate on these potential scenarios and their implications in the camera-ready version.\n\n\n**Q2: Influence of the partition method.**\nWe thank the reviewer for this good question.\nFirst, when $N=20$, we use 20 attack pixels to generate adversarial examples, which leads to greater distortion than $N=10$ (default setting).\nSecond, the partitioning strategy affects the final locations of these attack pixels, which may cause slight variations in distortion—either an increase or a decrease—though the impact is generally minor. \n\n**Q3: Obvious disturbance points.**\nWe thank the reviewer for highlighting this concern.\nOur work mainly focuses on searching for the attack pixel locations while the attack pixel values are set to 0 or 255.\nIf the values of 0 or 255 can successfully attack the target model, we directly use these values; otherwise, we use DE again to search for other attack pixel values.\nTo improve imperceptibility, we are exploring better value-search strategies. \nA simple approach is to start from the original pixel value $v_0$ and iteratively try $v_0 \\pm 1$, $v_0 \\pm 2$, and so forth until a successful perturbation is found.\n\n**Q4: The population setting.**\nWe thank the reviewer for this good question.\nAs shown in Table 4, with the same total number of pixels ($N \\times M$), a larger $N$ yields better attack performance. Thus, we recommend using a larger $N$ with a smaller $M$. These parameters can also be adjusted automatically based on the text length; for example, if we aim to perturb each character by one pixel, $N$ can be set equal to the number of characters in the image."}, {"review_id": "VyKr9Q4pt9", "reviewer": "Reviewer_HCsA", "summary": "Against adversarial attacks on Scene Text Recognition (STR) models, existing attack methods generate erroneous text that is easily corrected by a dictionary or a Large Language Model (LLM), i.e., the attack is “reversible”. To address this problem, the authors aim to generate “irreversible” attacks, that is, to allow the model to predict as many incorrect characters as possible. Therefore, this paper proposes a black-box pixel-level attack method called “Multi-Population Co-Evolutionary Search” (MPCS). The authors first solve the problem of “spatial concentration of attack pixels” in previous attacks by decomposing the global optimization objective into multiple image subspaces. Then, in order to solve the emerging problem of “combined perturbation shift” due to the objective decomposition, the authors design an innovative “co-evolutionary energy function”, which enables the parallel pixel populations in different subspaces to co-evolve. Finally, the experimental results show that MPCS can significantly increase the number of mispredicted characters, thus realizing more destructive and irreversible attacks.", "questions": "1.\tIs “θmin = (0, 0, 0) and θmin = (0, 0, 0)” a wrong representation in the initialization of pixel values section? In addition, the authors skip the description of this paragraph only by citing the literature [38]. The unreasonable representation in both aspects makes it more difficult for readers to understand the methodology of this paper.\n\n2.\tI believe that the essence of the attack task is to find a trade-off between the conflicting goals of “maximizing the success rate of the attack” and “minimizing the image perturbation”. Therefore, why don't the authors choose a multi-objective optimization framework to obtain a series of Pareto-optimal solutions covering different preferences?\n\n3.\tFor a black-box attack algorithm, the computational cost is also crucial. However, the authors propose a complex framework containing multi-population, multi-generation evolution, whose computational cost is obviously huge. Can the authors provide relevant explanations and experimental analysis?\n\n4.\tThe coevolution energy function (CEF) relies on a single optimal individual for each other population. Is this robust enough for the evolutionary process? Have the authors considered other coevolution strategies, such as using Top-K individuals from each population to represent its evolutionary state?\n\n5.\tCan the authors provide more explanation on “why optimization-based white-box methods are not effective under low L2 paradigm constraints” when MPCS outperforms constrained white-box attack methods in their experiments?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe authors do not merely allow the model to produce incorrect predictions, but also introduce the concept of “irreversibility” of the attack. This provides a good point for the study of adversarial attacks against sequence prediction models.\n\nWeakness:\n\nThe contribution of this paper lies in solving two core problems: “pixel concentration” and “combined perturbation shift”. The former is just a well-known local optimization problem in evolutionary algorithms. The latter, i.e., the “Combined Perturbation Shift Problem”, which is emphasized by the authors, is extremely abstract and unclear. Although a definition is given in the paper, the concept is still not well understood by the reader according to the short description in Section 3.3 and Figure 2.\n\nThe high computational cost of the MPCS method is obvious, especially when evaluating the coevolutionary energy function, where each individual needs to be evaluated in combination with the optimal individuals of the other N-1 populations. The authors do propose an early stop mechanism to balance performance and efficiency. However, this is only a mitigation measure and does not address the fundamental problem of high computational costs associated with evolutionary algorithms.\n\nThe field of black-box attacks is rapidly developing, and it is possible that other more powerful or varied SOTA algorithms exist. However, it is hardly convincing that the authors chose only OnePixel and AD²E, two obsolete methods also based on evolutionary algorithms, as comparative methods for pixel-level attacks.", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Combined perturbation shift problem.**\nWe thank the reviewer for the valuable feedback.\nThe combined perturbation shift refers to the phenomenon: when we combine several successful perturbations $ \\delta_i^* $  to get a new perturbation $ \\delta^* = \\sum \\delta_i^* $, this new perturbation could possibly be a failed perturbation.\nThis is the key challenge we want to solve.\nDue to space constraints, our explanation in Section 3.3 and Figure 2 was brief; we will provide a more intuitive illustration and detailed discussion in the camera-ready version to make the concept clearer.\n\n**W2/Q3: The high computational cost.**\nWe thank the reviewer for the valuable feedback.\nThe computational cost mainly comes from querying the target STR model, while the DE algorithm itself only needs little computation.\nAlso, when evaluating the coevolutionary energy function $\\textbf{E}^c(p_i^j)$, we only query the model once using the combined perturbation $\\Delta(\\lbrace p_i^j \\rbrace \\cup \\lbrace p_{k}^* \\rbrace _{N-1})$, which has the same query cost as $\\textbf{E}^p(p_i^j)$.\nThus, the coevolutionary energy function does not introduce any external computational cost.\n\nBesides, we set the total attack pixel number consistent across different multi-population settings.\nFor example, when the population number $N=10$, the population size is $M=60$; when $N=1$, the population size is $M=600$. \nTherefore, no matter how many populations we used, the computational cost in one generation is consistent.\nIn our setting, the average attack time on CRNN with CUTE80 is only 18.23 seconds, which we consider acceptable.\n\n**W3: Comparative methods.**\nWe thank the reviewer for the constructive feedback. The field of black-box attacks indeed evolves rapidly. \nWe chose OnePixel and $\\mathrm{AD}^2\\mathrm{E}$ because they are well-established baselines for pixel-level black-box attacks, sharing the same evolutionary paradigm as our method. \nTo the best of our knowledge, $\\mathrm{AD}^2\\mathrm{E}$ remains the state-of-the-art among pixel-level black-box attacks in the STR domain.\nIn addition, we included recent SOTA black-box attacks such as Sparse-RS and UDUP in Table 2, although these methods are either not pixel-level or not directly tailored to STR tasks.\nTherefore, we believe our comparison covers both representative baselines within the same paradigm and other strong black-box attack strategies across paradigms.\n\n\n**Q1: Pixel value initialization.**\nWe thank the reviewer for highlighting this error.\nThe \" $\\theta_{min}=(0,0,0)$ and $\\theta_{min}=(0,0,0)$\" should be \"$\\theta_{min}=(0,0,0)$ and $\\theta_{max}=(255,255,255)$\".\nWe chose these two values because they are found to be most effective in the work [38].\n\n**Q2: Optimization framework.**\nWe thank the reviewer for this inspiring question.\nIndeed, the attack task consists of two objectives: maximizing the success rate and minimizing the image perturbation.\nTherefore, in future work, we can use a multi-objective optimization framework, such as NSGA-II, to find the best attack pixel values.\nSpecifically, after we obtain the attack pixel locations via MPCS, we can use NSGA-II to search for attack pixel values that are most imperceptible.\n**This approach complements rather than conflicts with our MPCS method**, and we plan to explore it as a promising direction in future work.\n\n**Q4: Coevolution strategy.**\nWe thank the reviewer for this insightful question.\nAs shown in Table 2, when introducing the CEF, the SR, SR*, and PR get a rise of 12.09\\%, 22.47\\%, and 19.06\\%, respectively.\nThis demonstrates its effectiveness and robustness.\nAs for the coevolution strategy mentioned by the reviewer, we believe it will enhance the attack performance with several times the amount of calculation.\nWith CEF, we calculate the energy of an individual by generating a combined perturbation $\\Delta(\\lbrace p_i^j \\rbrace \\cup \\lbrace p_{k}^* \\rbrace_{N-1})$ and use it to query the target model one time. \nIf we use the top-K individuals from other populations, there are two choices:\n\n(1) Using all top-K individuals to generate a combined perturbation $\\Delta(\\lbrace p_i^j \\rbrace \\cup \\lbrace \\lbrace p_{k}^s \\rbrace^K \\rbrace _{N-1})$. \nThis will increase the perturbation, making it not aligned with our final perturbation. \nThe increased perturbation will also bring a much higher SR.\nSo it is not a good choice.\n\n(2) Traversing the top-K individuals to generate combined perturbations one by one.\nThis may enhance the robustness, but since each combined perturbation needs a query time, the computational cost will increase several times.\n\n**Q5: Explanation on white-box methods.**\nWe thank the reviewer for this good question.\nWe reproduced the white-box method [37] and found that if we set no upper bound on its perturbation size, some of the adversarial examples generated by it could be severely distorted, but the average perturbation is still small.\nIn contrast, our method has a stable perturbation size for each adversarial example.\nFor a fair comparison, we have to set an upper bound on the perturbation size of [37]."}]

Author rebuttal:

Meta-review: (a) This paper introduces a novel black-box adversarial attack method for Scene Text Recognition (STR) models, named Multi-Population Coevolutionary Search (MPCS). The key contribution of the paper is the focus on "irreversibility” i.e., generating adversarial examples that are resistant to correction by dictionaries or Large Language Models. The method addresses the "pixel concentration" problem of prior attacks by dividing the image into horizontal regions and running parallel evolutionary searches. To solve the subsequent "combined perturbation shift" problem that this decomposition creates, the authors propose a co-evolutionary energy function to ensure the perturbations work in tandem. The paper's findings, validated across multiple STR models and datasets, show that MPCS significantly increases the number of mispredicted characters (Perturbation Rate), resulting in attacks that maintain a high success rate even after post-correction attempts.

(b) around the concept of "irreversibility," which reviewers found to be a valuable and significant contribution to the field. The proposed MPCS framework is well-motivated, systematically designed, and supported by strong theoretical analysis, particularly for the innovative co-evolutionary energy function that addresses a key challenge in multi-agent optimization. The experimental evaluation is another major strength, praised by reviewers for its thoroughness, with tests conducted on multiple standard datasets against a wide range of STR models and baseline attack methods, all of which clearly demonstrate the superiority of the proposed approach.

(c) The most significant weakness noted by reviewers was that the method was evaluated solely on English/Latin script, which raised questions about its generalizability to other languages. Some reviewers also found the method to be complex, raising concerns about the computational cost and the trade-off between the attack success rate (SR) and the more crucial irreversible success rate (SR*). Minor weaknesses included a lack of clarity on certain concepts like the "combined perturbation shift problem" and questions regarding the choice of baselines and specific experimental settings which was addressed by the authors in the rebuttal.

(d) The recommendation is to Accept this paper. The primary reason is that it introduces a novel and practically important concept—"irreversibility"—to the domain of adversarial attacks on STR models and provides a technically solid and well-validated method to achieve it. The authors' rebuttal was strong, directly addressing the main weakness by conducting a new experiment on a non-Latin (Chinese) dataset, which successfully demonstrated the method's generalizability and addressed the concern of reviewer cRqd. The final reviewer consensus is mostly positive with the only negative review coming from reviewer HCsA who has not been involved in the reviewal process despite few reachouts. The paper provides a clear contribution, advances the state-of-the-art in a meaningful way, and has review consensus and I am recommending a poster based on the above.

(e) The rebuttal period was well managed by authors mostly and demonstrated the authors' commitment to addressing reviewer feedback. The most critical point raised by Reviewer cRqd was the evaluation's limitation to Latin scripts. The authors directly addressed this by conducting a new experiment on a Chinese text recognition benchmark, showing that their method generalized well and outperformed a key baseline. This new evidence led the reviewer to finalize an "Accept" rating. Other concerns from Reviewers 1goM and 1ydv regarding performance trade-offs (SR vs. SR*), the method's complexity, and its performance on long texts were also thoroughly addressed with detailed explanations and supplementary experiments. The authors successfully clarified that the slight decrease in the standard success rate (SR) is a deliberate trade-off to achieve a much higher irreversible success rate (SR*), which is the paper's primary goal. This active and successful engagement solidified the positive opinions of the reviewers. Unfortunately, reviewer HCsA has not addressed the author’s rebuttal even though the authors provide responses to their key concerns.
Decision: Accept (poster)