Title: Towards Irreversible Attack: Fooling Scene Text Recognition via Multi-Population Coevolution Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: adversarial attack

scene text recognition

evolution algorithm

computer vision
Abstract: Recent work has shown that scene text recognition (STR) models are vulnerable to adversarial examples.
Different from non-sequential vision tasks, the output sequence of STR models contains rich information. 
However, existing adversarial attacks against STR models can only lead to a few incorrect characters in the predicted text.
These attack results still carry partial information about the original prediction and could be easily corrected by an external dictionary or a language model.
Therefore, we propose the Multi-Population Coevolution Search (MPCS) method to attack each character in the image. 
We first decompose the global optimization objective into sub-objectives to solve the attack pixel concentration problem existing in previous attack methods.
While this distributed optimization paradigm brings a new joint perturbation shift problem, we propose a novel coevolution energy function to solve it.
Experiments on recent STR models show the superiority of our method.
The code is available at \url{https://github.com/Lee-Jingyu/MPCS}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAgainst adversarial attacks on Scene Text Recognition (STR) models, existing attack methods generate erroneous text that is easily corrected by a dictionary or a Large Language Model (LLM), i.e., the attack is “reversible”. To address this problem, the authors aim to generate “irreversible” attacks, that is, to allow the model to predict as many incorrect characters as possible. Therefore, this paper proposes a black-box pixel-level attack method called “Multi-Population Co-Evolutionary Search” (MPCS). The authors first solve the problem of “spatial concentration of attack pixels” in previous attacks by decomposing the global optimization objective into multiple image subspaces. Then, in order to solve the emerging problem of “combined perturbation shift” due to the objective decomposition, the authors design an innovative “co-evolutionary energy function”, which enables the parallel pixel populations in different subspaces to co-evolve. Finally, the experimental results show that MPCS can significantly increase the number of mispredicted characters, thus realizing more destructive and irreversible attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe authors do not merely allow the model to produce incorrect predictions, but also introduce the concept of “irreversibility” of the attack. This provides a good point for the study of adversarial attacks against sequence prediction models.\n\nWeakness:\n\nThe contribution of this paper lies in solving two core problems: “pixel concentration” and “combined perturbation shift”. The former is just a well-known local optimization problem in evolutionary algorithms. The latter, i.e., the “Combined Perturbation Shift Problem”, which is emphasized by the authors, is extremely abstract and unclear. Although a definition is given in the paper, the concept is still not well understood by the reader according to the short description in Section 3.3 and Figure 2.\n\nThe high computational cost of the MPCS method is obvious, especially when evaluating the coevolutionary energy function, where each individual needs to be evaluated in combination with the optimal individuals of the other N-1 populations. The authors do propose an early stop mechanism to balance performance and efficiency. However, this is only a mitigation measure and does not address the fundamental problem of high computational costs associated with evolutionary algorithms.\n\nThe field of black-box attacks is rapidly developing, and it is possible that other more powerful or varied SOTA algorithms exist. However, it is hardly convincing that the authors chose only OnePixel and AD²E, two obsolete methods also based on evolutionary algorithms, as comparative methods for pixel-level attacks.\n\nQUESTIONS:\n1.\tIs “θmin = (0, 0, 0) and θmin = (0, 0, 0)” a wrong representation in the initialization of pixel values section? In addition, the authors skip the description of this paragraph only by citing the literature [38]. The unreasonable representation in both aspects makes it more difficult for readers to understand the methodology of this paper.\n\n2.\tI believe that the essence of the attack task is to find a trade-off between the conflicting goals of “maximizing the success rate of the attack” and “minimizing the image perturbation”. Therefore, why don't the authors choose a multi-objective optimization framework to obtain a series of Pareto-optimal solutions covering different preferences?\n\n3.\tFor a black-box attack algorithm, the computational cost is also crucial. However, the authors propose a complex framework containing multi-population, multi-generation evolution, whose computational cost is obviously huge. Can the authors provide relevant explanations and experimental analysis?\n\n4.\tThe coevolution energy function (CEF) relies on a single optimal individual for each other population. Is this robust enough for the evolutionary process? Have the authors considered other coevolution strategies, such as using Top-K individuals from each population to represent its evolutionary state?\n\n5.\tCan the authors provide more explanation on “why optimization-based white-box methods are not effective under low L2 paradigm constraints” when MPCS outperforms constrained white-box attack methods in their experiments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a novel black box adversarial attack method - MPCS, for STR models. There are two key issues with existing adversarial attack methods: Information residue, the predicted results after the attack still retain some of the original text information, which is easily corrected by language models; Pixel concentration leads to uneven spatial distribution of perturbed pixels, resulting in some characters not being effectively attacked. The core contribution of this article is to achieve a globally optimized distribution of perturbed pixels through horizontal partitioning and co evolutionary energy functions; Propose and solve the combined perturbation offset problem and prove the information lossless nature of the energy function.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a systematic attack framework MPCS that tackles a critical challenge in adversarial attacks on STR models: irreversibility. The multi-population coevolution search is well-motivated and addresses the pixel concentration problem effectively.\n\n## Strengths :\n\nThe authors tested it on 4 standard datasets (ICDAR13/15, SVTP, CUTE80) and 3 STR model types (CTC-based, attention-based, multimodal). At the same time, it was evaluated against 6 baseline models (OnePixel, AD²E, Sparse-rs, UDUP, Xu et al., transfer attack). The perturbation rate (PR) was introduced to measure the irreversibility of the attack and verified by correction based on GPT-4.\n\nThis paper explicitly models irreversibility in STR attacks, going beyond simple misclassification. The authors formally define the combined perturbation shift problem and prove that the co-evolutionary energy function can avoid information loss. Meanwhile, MPCS ensures that the perturbation is evenly distributed. The early stopping mechanism balances effectiveness and efficiency.\n\nThe authors demonstrate that even state-of-the-art STR models (e.g., IGTR) are vulnerable to irreversible attacks and propose how adversarial examples can protect sensitive text (e.g., license plates, ID cards) from unauthorized OCR extraction.\n\n## Weaknesses：\n\nThe performance of the method proposed by the author is reduced for long texts. It is only tested on the CNN/RNN-based STR model, and no experiments are done on Vision Transformers or diffusion-based OCR models to verify the effect of the method. Besides, lack of exploration on the defense mechanism of this method.\n\nFormula 7 does not clearly describe the connection between local optimization and global optimization. Can it be theoretically guaranteed that the method finds the global optimum without falling into the local optimum?\n\nAt the same time, this article does not consider whether the current mainstream defense measures can defend against the method of this model. For example, can the adaptive STR model detect MPCS perturbations?\n\nAlthough this article emphasizes privacy protection, the scenario of malicious applications can be better discussed.\n\nWhat are the usage scenarios of this method and is there any possibility of its application in real-life scenarios?\n\nQUESTIONS:\nI am curious about how formula 7 balances local optimization and global optimization? Will this process fall into the dilemma of local optimality?\n\nTable 4 shows that when N=20, L2=6.03. Will non-uniform partitioning reduce distortion?\n\nIn fact, the disturbance points generated by this method are obvious, although the overall disturbance value is not high. Is there a way to make the disturbance points more hidden?\n\nCan the number of populations and the size of populations be automatically adjusted? This is more conducive to the optimization of this method.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper aims to attack the STR model with irreversible output since text string is contextual and less wrong character can be corrected with dictionary or LLMs. They propose the Multi-Population Coevolution Search (MPCS) method to attack each character in the image. They  decompose the global optimization objective into sub-objectives to solve the attack pixel concentration problem existing in previous attack methods. To address the new combined perturbation shift problem, they propose a coevolution energy function and make the theoretical analysis. Experiments on CRNN, ASTER, SAR and IGTR demonstrate the results.

STRENGTHS AND WEAKNESSES:
Strengths:
1) Finding the phenomenon that text string attacked with few wrong characters is easily recovered is valuable since in many scenarios the dictionary is used and the LLMs' abilities progress rapidly.
2) The proposed MPCS performs better in irreversible settings.
3) The theoretical analysis guarantees the superiority of the proposed coevolutionary energy function.

Weaknesses:
1) The proposed method is complex and the SR rate decreases with large margins. SR* is important, but SR is also an important metric to evaluate different methods, the decrease from 90%+ to 80%+ demonstrates that the proposed method's success rate is too low. 
2) Equation 5, theta_max is not defined, and is it (255, 255, 255)?
3) Tab.1, the results of SAR+IC13 show the proposed method is ineffective, please explain it in detail.
4) Tab.2 lacks SR* results, please explain.
5) Tab.3, ES gives the negative effect. In my opinion, the attack time overhead is not so important than that of SR.

QUESTIONS:
See "Weaknesses".

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
In this paper the authors present a pixel-level blackbox attacked method targeting  to manipulate scene text images. The authors use  a method named an MPCS to attack each character in the image to avoid being easily corrected by external dictionaries, and shows the effectiveness of their proposed method using four STR datasets.

STRENGTHS AND WEAKNESSES:
**Strengths**:
- The paper is well written and easy to follow. 
- The experiment section provides detailed results with clear and concise reasoning for the hyperparameters and the methods chosen.

**Weaknesses**
- The method presented in this paper is evaluated solely on the Latin/English script, and it is unclear whether it can be extended to different scripts or it is script dependent. It would be nice if the authors could test additional Non-Latin scene texts.

QUESTIONS:
1.  The authors mention that  they used early stopping to balance the extra time caused by maximum generation (MG). Is it possible to quantify the average attack time per  sample text-line image with and without early stopping? 
2.  I have already mentioned it in the strength and weakness part above.  Since there are many Non-Latin scent text datasets, is there a specific reason for selecting the Latin script? Was it intentional?
3. The authors provide different metrics such as SR/SR*, PR, and L2 distance, is it possible to provide sample real examples and compute the values using each metric,  the examples can be include in the supplemental part?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03