Know Thine Enemy: Adaptive Attacks on Misinformation Detection Using Reinforcement Learning (2024)

Abstract: We present XARELLO: a generator of adversarial examples for testing the robustness of text classifiers based on reinforcement learning. Our solution is adaptive, it learns from previous successes and failures in order to better adjust to the vulnerabilities of the attacked model. This reflects the behaviour of a persistent and experienced attacker, which are common in the misinformation-spreading environment. We evaluate our approach using several victim classifiers and credibility-assessment tasks, showing it generates better-quality examples with less queries, and is especially effective against the modern LLMs. We also perform a qualitative analysis to understand the language patterns in the misinformation text that play a role in the attacks.

heading: 1 Introduction
text: Nowadays, an ever-increasing proportion of the text we read online is published by anonymous or unfamiliar authors, e.g. in online news outlets, blogs, social media portals, instant messaging, and communication agents. This puts a great burden on the entities hosting such platforms, having to filter the user-generated data to remove or de-prioritise content considered inflammatory, misleading, unpleasant or simply illegal. A large part of this work is performed manually by moderators, but the use of automatic machine-learning (ML) classifiers is becoming more common (Singhal et al., 2022). This scenario necessitates testing the robustness of the deployed models, i.e. their ability to deliver correct results even when their input is manipulated, e.g. by a fake news spreader.
The robustness is usually tested by analysing input examples and checking what kind of modifications made to them confuse the victim classifier to change its output. For example, let us assume the following statement is correctly identified by a classifier as misleading: Drinking orange juice
causes DEATH!. However, if the same classifiers return a different result when causes is replaced with provokes or cuases, this weakness can be used by attackers. Discovering such adversarial examples (AE) is the best way to understand the vulnerabilities of the common methods before they can be exploited by attackers. A plethora of approaches for AE generation for text classifiers has been proposed (Zhang et al., 2020) and tested, including in misinformation detection (Przybyła et al., 2023).
The AE techniques explored so far are usually based on making incremental changes to an individual example (e.g. word replacements), and testing the victim’s response to the modifications, until it returns a desired response (Zhang et al., 2020). This simple procedure is repeated for each example independently. Here we consider a different approach, where an attacker is adaptive and it learns from successes and failures from each attack attempt. Thus, the attacker can observe and exploit the weaknesses of the victim, i.e. modifications that are particularly likely to flip the classification decision. This corresponds to the real-world circumstances of misinformation spreaders that are established large-scale enterprises, e.g. Russia’s Internet Research Agency (DiResta et al., 2019), able to gather significant expertise regarding the weaknesses of the moderation on major platforms.
To understand the effectiveness of such attacks, we propose XARELLO (eXploring Adversarial examples using REinforcement Learning Optimisation), a method for learning weaknesses of a target classifier to improve quality of the proposed modifications. XARELLO is built upon the reinforcement learning framework, which allows it to gather experience in the adaptation phase and then use it in the attack phase. Using the framework for testing AE solutions in several misinformation detection tasks for English (Przybyła et al., 2023), we show that our solution indeed manages to adapt over time and deliver performance beating the state of the
125
art, both in terms of the more subtle modifications and lower number of attempts necessary. The victims, against which our attacker is tested, include a state-of-the-art LLM (GEMMA), which surprisingly appears the most vulnerable to the adaptive attack. We also qualitatively analyse the generated examples to better understand the techniques our models learn during the adaptation. The code for XARELLO is openly available to encourage research into AEs as well as building more robust classifiers1.

heading: 2 Related work
text: The challenge of discovering AEs began in image classification research (Szegedy et al., 2013), where neural networks were discovered to change predictions after noise was added to the input. Generalising this approach to text is not trivial due to its discrete nature and the lack of ‘imperceptible noise’ equivalent, but several approaches emerged (Zhang et al., 2020). Typically, they rely on an iterative procedure of replacing fragments of input text with words that are similar in terms of meaning (Ren et al., 2019; Garg and Ramakrishnan, 2020; Li et al., 2020; Alzantot et al., 2018), in terms of visual appearance, or using character replacements (Gao et al., 2018). Recent work has been improving this paradigm (Liu et al., 2023) or abandoning it in favour of sentence-to-sentence paraphrasing, e.g. using auto-encoders (Li et al., 2023).
Misinformation detection is a scenario with a high probability of adversarial action. Several studies have been performed to assess the robust-
1https://github.com/piotrmp/xarello
ness of the two most-popular tasks: Fact-checking, usually using manually crafted rules (Zhou et al., 2019; Thorne et al., 2019; Hidey et al., 2020); and fake news detection (Jin et al., 2020; Ali et al., 2021; Brown et al., 2020; Smith et al., 2021). We also need to mention the novel threat of machinegenerated text used for misinformation, and the models for its detection (Crothers et al., 2023) being vulnerable to attacks (Su et al., 2023).
In order to perform an evaluation of XARELLO in various scenarios, we rely on the previous systematic study of adversarial robustness in the credibility assessment context (Przybyła et al., 2023), taking into account four misinformation-detection tasks and two victim classifiers. This will allow us to compare our solution to the eight AE generators evaluated there.
Finally, a few attempts have already been made to use reinforcement learning (RL) in the context of AE generation. Our solution has certain similarities with that of Vijayaraghavan and Roy (2019), who also apply RL to find the most successful word substitutions, but in a less challenging setup: attacking a CNN network performing sentiment analysis and news classification. Other work involving RL include that of Li et al. (2021) and Chen et al. (2023). However, our study is the first to perform adaptive AE generation for the misinformation text, where a victim vulnerability model is first explicitly learned and then deployed for a more efficient attack.

heading: 3 Methods
text: XARELLO modifies given text not only based on the current input (original content), but also taking into account the outcome of previous attempts
made against the same victim classifier. The whole process has two phases: adaptation and attack.
Figure 1 shows a schema of our solution. We map the problem of generating AEs (section 3.1) to the reinforcement learning paradigm through the XARELLO environment (section 3.2). During adaptation, a XARELLO agent (section 3.3) learns to perform actions (token replacements) that maximise its reward (change in the victim’s prediction). The core of the model is a neural network estimating the outcome of making modifications to the input text. During the attack, the learned model, encoding information about the vulnerabilities of the victim, can be used to generate a multitude of adversarial examples, undergoing evaluation.

heading: 3.1 Preliminaries
text: We focus on binary text classification task using pairs (xi, yi), where xi is a text fragment and yi is a binary label denoting credibility of the text (section 4.3). The victim of the attack is a classifier f , which, for a given example xi, provides a binary output label f(xi) ∈ {0, 1}, but also probabilities of the positive class fp(xi) ∈ (0, 1). The goal of the attack is to come up with a modification function m, such that the difference with the original example is small (m(xi) ≈ xi), but the victim changes its decision (f(xi) ̸= f(m(xi)), for example xi = Drinking orange juice causes DEATH! and m(xi) = Drinking orange juice provokes DEATH!. Here we consider both the targeted scenario, taking into account only examples of non-credible text, for which the classifier made the correct decision (yi = f(xi) = 1); as well the untargeted one, where all examples are included.

heading: 3.2 XARELLO environment
text: The basic steps in our model are the same as in most methods for AE generation in text, i.e. sequential modifications, each consisting of replacing a word by a candidate from a pre-computed list, until the victim changes its decision (see section 2). Usually, no single replacement can result in an AE, but several are necessary. To learn an optimal strategy for such a task, we use the reinforcement learning (RL) framework (Sutton and Barto, 2018). We define the environment in the following way:
• an environment state s includes the following:
– x(t)i,j – the current form (in step t) of the i-th target text, expressed as a sequence of N tokens (j ∈ {1 . . . N}),
– f(xi) – the decision of the victim for the original text.
• an action a made by an agent: a pair (j, k) including the positions of the changed token j and the replacement candidate zk from a pre-computed list z1, z2, . . . zK .
• a reward returned in response to an action:
– 1, if the provided example is an AE, – −1, for an attempt to modify a non-word
token (see section 4.5). – otherwise, [fp(x (t) i )−fp(x (t−1) i )]× [1−
2×f(xi)], i.e. the difference in the score compared to previous state, computed with respect to the original class, so that positive values indicate the victim getting closer to changing the decision.
Adaptation: During adaptation, the environment presents subsequent examples to the agent. While it would be preferable to have only unique examples, the limited data size means that examples are repeated for several epochs. Since an agent is unlikely to find an AE by just a single word replacement, it is allowed several modifications (steps) until an AE is successful or the maximum number of steps (MS = 5) is achieved. For example, an agent might try Drinking orange juice provokes DEATH!, then Consuming orange juice provokes DEATH!, then Consuming orange juice brings DEATH!, and so on. Such a sequence, called episode, is attempted ME = 5 times (with text reset to the originals state in between) before the next example is used. We encourage variability of actions between episodes through the penalisation of action reuse (section 4.5).
Attack: In the attack stage the Q model is frozen and no learning is performed, allowing more elaborate action sequences as follows:
1. 10 episodes of up to 5 steps, 2. 5 episodes of up to 10 steps, 3. 2 episodes of up to 25 steps, 4. 1 episode of up to 50 steps.
Performing several episodes for the same number of steps allows the attacker to makes several attempts to create an AE with few changes, before performing deeper modifications. As during training, the text is reset to the original form between episodes and penalisation is used to encourage variation between attempts (section 4.5). For longer input text (news bias and rumour detection tasks,
see section 4.3), the number of steps allowed is multiplied by 5. The process can stop at any point if an AE for the current text is found, which is sent for evaluation.

heading: 3.3 XARELLO agent
text: The implementation of the XARELLO agent is based on Q-learning (Watkins, 1989), which involves estimating the value of Q(s, a), i.e. the expected reward achieved from making action a in state s and following a greedy strategy. In particular, we implement a deep Q-network (FrançoisLavet et al., 2018), where the estimation is produced by a deep neural network, subsequently trained based on the actually observed rewards.
Q model: We compute the value of Q(s = (x
(t) i,j , f(xi)), a = (j, k)) as follows (see Fig. 2):
1. For each token position j, its E1-dimensional embedding is computed through a Transformer (Vaswani et al., 2017) encoder working on the current text x(t)i,j , 2. It is concatenated with a pre-computed E2dimensional embedding of candidate zk, forming a E1 + E2-dimensional representation of each possible action (j, k),
3. A linear layer with rectified linear activation is applied, reducing the dimensionality to E3,
4. Depending on the value of original prediction f(xi), one of two independent final linear layers is used, reducing the dimensionality to a scalar, containing the value of Q(s, a).
The neural network is implemented so that it computes the Q value for every possible action in a given state in a single execution.
Action choice: Choosing an action based on the Q value depends on the phase. In attack, simply the action with maximal Q value is selected (greedy strategy). In the adaptation phase, a random action may also be made with the probability equal exploration factor ϵ ∈ [0, 1] – an ϵ-greedy strategy (Sutton and Barto, 2018). Further information including parameter values and underlying components is in section 4.5.
Learning: As usual in fitted Q-learning, after an action is performed, the value of Q estimation is compared with the observed reward and discounted expected reward (using discount coefficient γ) and the resulting discrepancy is used as a loss for training the underlying neural network.

heading: 4 Evaluation
text: Since our solution is motivated by the adversarial scenarios in the misinformation space, we base our evaluation on the BODEGA framework (Przybyła et al., 2023), which is designed specifically for this area. It enables the evaluation in four misinformation detection tasks: style-based news bias assessment (HN), propaganda detection (PR), fact checking (FC), rumour detection (RD), all for English. A non-credible (positive, label=1) example, which should be detected by a classifier, is a news item from a hyper-partisan source, a sentence including a propaganda technique, a fact refuted by the provided evidence, or a thread initiated by a rumour. Examples are shown in table 4 in appendix E. All of the tasks are based on data released on CC licences (Potthast et al., 2018; da San Martino et al., 2020; Thorne et al., 2018; Han et al., 2019).
BODEGA enables an evaluation of attacks on two classifiers, based on BiLSTM (Hochreiter and Schmidhuber, 1997) and fine-tuned BERT (Devlin et al., 2018). Additionally, in order to understand the vulnerability of the modern LLMs, we test against 2-billion-parameter GEMMA (Gemma Team and Google DeepMind, 2024).

heading: 4.1 Performance measures
text: The attack performance is assessed by comparing each original examples with the produced AE and computing four measures:
1. confusion score: 1 if the example provided is a successful, 0 otherwise,
2. semantic score: a measure of the meaning preservation between the original text and the
AE, computed using BLEURT (Sellam et al., 2020) and clipped to (0,1),
3. character score: a measure of character-level changes, computed using Levenshtein distance (Levenshtein, 1966) and scaled as a similarity score in (0,1),
4. BODEGA score: a product of the above.
These quantities are averaged over all examples in a given experiment. More information on these measures, including the handling of multi-sentence inputs, could be found in the BODEGA framework (Przybyła et al., 2023). Additionally, we record the average number of queries a method needs to perform on the victim classifier before an AE is generated, as a measure of how realistic a given strategy is to be used in practice.
During the adaptation phase, we measure its progress through certain indicators after each epoch, both on the training data and a held-out development set (used greedily). These include mean reward value, the fraction of the episodes that end with a success, and the number of steps involving a given text before an AE is found.

heading: 4.2 Qualitative analysis
text: In addition, we provide a qualitative analysis of AEs generated by the XARELLO system against the BERT classifier in the targeted PR task. In Section 6, we make some observations on linguistic patterns that appear in this subset of AEs. Human evaluation is especially important for NLP models that generate text which people may read, or use in text generation to aid replicability (Belz et al., 2023). These models must also generate naturalistic text which reflects qualities such as grammaticality, fluency, and coherence (van der Lee et al., 2021) in order to be usable in practice, i.e. as misinformation content.

heading: 4.3 Data
text: Table 1 shows the data distribution, based on the BODEGA framework. We do not use the data reserved for victim training (not included in the table)
and leave final attack portion unchanged, enabling comparison with previous work. We employ the development subset in XARELLO, splitting it into adaptation-train (for Q adaptation) and adaptationeval (for monitoring the process, see measures above). We also show what fraction of each dataset as a whole is positive, i.e. non-credible.

heading: 4.4 Experiments
text: Each experiment starts with performing the adaptation for 20 epochs. During every epoch, firstly the adaptation-train data are used to learn from the experiences and update the network accordingly. Afterwards, the held-out adaptation-eval portion is used (with the greedy strategy and no weight updates) to measure the adaptation performance.
After the adaptation is finished, the model that performed the best on adaptation-eval, i.e. needed the least steps on average to reach an AE, is selected for final attack evaluation. This is performed by connecting the learned Q model to an environment working in attack mode and evaluating the quality of the AEs with BODEGA.
In total, 12 adaptation processes are performed (against three victim classifiers for each of the four tasks), which are followed by two evaluation scenarios: targeted or untargeted. We compare an adapted XARELLO against:
• BERT-ATTACK (Li et al., 2020), performing a procedure of iterative replacement of words by candidates from a language model, fairly similar to XARELLO, but without any adaptation to the victim. BERT-ATTACK achieved the best result among those evaluated on BODEGA (Przybyła et al., 2023).
• DeepWordBug (Gao et al., 2018), a simpler approach, replacing individual characters in the selected words, aiming to preserve visual similarity to the original text. DeepWordBug was also the best-performing in some attack scenarios in BODEGA.
• XARELLO-raw, a version of the XARELLO agent which was not adapted to the victim. Testing this version allows us to make sure the observed differences are due to adaptation process, rather than the attack procedure.

heading: 4.5 Optimisation details
text: Preprocessing: The maximum length of a text fragment is N = 512 tokens and all instances are padded accordingly. For each text and each
non-padding token, the K = 20 replacement candidates are obtained by applying language modelling through BERT (Devlin et al., 2018) in bert-base-cased variant, implemented in HuggingFace Transformers (Wolf et al., 2020). No masking is used, as in BERT-ATTACK (Li et al., 2020), and the most likely tokens for each position are treated as candidates, disregarding the original word and special tokens.
Neural network: We use BERT (configured as above), to obtain embeddings of size E1 = 768. To represent candidates, we use static fastText (Mikolov et al., 2017) vectors, i.e. the facebook/fasttext-en-vectors model from HuggingFace, returning an embedding of size E2 = 300. The reduced representation has size E3 = 8. The Q network includes the 110 million parameters of BERT and 8570 in the further layers.
Q override: In order to indicate that nonword tokens ([CLS], [SEP] or [PAD]) cannot be changed, the reward for attempting to replace them is set to −1. Moreover, the Q value obtained from the neural network is overridden using two rules: (1) the value for replacing special tokens are set to −1 and (2) the value for actions that have already been applied for this text in the current sequence of episodes are reduced by a factor of −0.1. This penalisation mechanism makes it possible to generate diverse actions even when Q network remains unchanged, esp. in the attack phase. Both of the alterations correspond to behaviours that are benefi-
cial for the rewards and would be learnt eventually, but introducing them accelerates the adaptation.
Further details on hardware and computing times, software implementation, adaptation process and parameter tuning can be found in appendix B.

heading: 5 Results
text: Figure 3 shows the progress made during the adaptation to the BiLSTM and BERT victims (the results for GEMMA are included as figure 4 in appendix A). We plot the average number of steps made until an AE is found or the limit is reached, taking values between 5 (the AE is found on first try in all 5 episodes) and 25 (all 5 steps in the 5 episodes are used). All models start with a value close to the maximum and manage to improve over time, but the gains are more pronounced for the PR and FC tasks than RD or HN. This is understandable, as the text fragments involved in the latter two (news articles and rumour threads) are much longer, so it is relatively rare to see an AE generated within the 5 modifications allowed during adaptation. We can also see that the BiLSTM victim, as a weaker classifier, is easier to attack, allowing an AE to be found in fewer steps after the adaptation.
It is encouraging to notice that the performance on the unseen eval dataset improves similarly, indicating that the model indeed learns vulnerabilities of the victim model instead of memorising the steps that prove successful for the training data. Towards the end of the 20-epoch process we see the im-
provements on the eval dataset slow down, suggesting that further training would result in overfitting, which confirms the preliminary experiments with 50 epochs (see appendix B).
Table 2 shows the results of the main experiment in untargeted scenario (with all data), carried out by taking a Q neural network optimised during adaptation and applying to the attack data portion. The performance indicators averaged over all scenarios (final rows) confirm the benefits of the proposed approach: it achieves better-quality AEs, reflected with a higher BODEGA score. The gains are most pronounced against the BiLSTM victim, where XARELLO achieves the score of 68%, compared to 51% of BERT-ATTACK, needing only 90 queries instead of 417. We also see an improvement over baseline in case of BERT, but it is interesting to notice that GEMMA, the model of largest size and best classification performance, is quite vulnerable against XARELLO attacks (58% compared to 47% of BERT-ATTACK).
Overall, DeepWordBug produces examples that are semantically and visually similar to the original, but achieve success only in some cases. For example, in BiLSTM fact-checking scenario, DeepWordBug has a confusion score of 57%, BERT-ATTACK of 86%, but XARELLO reaches 100%. This is pos-
sible due to the adaptation process, as XARELLO raw ranks similarly to BERT-ATTACK and only the full version achieves the improvements.
The performance differs across tasks: XARELLO shows improvement in all of them except news bias assessment, especially against the BERT victim. This is most likely due to the length of the input: news articles often fill the whole 512-token window, resulting in 512*20 possible actions – a space unlikely to be thoroughly explored within the limits of the adaptation. The quality of the sample AEs remains high, but they are just not found for as many examples as in BERT-ATTACK. This is in line with the slow adaptation for this combination visible in Figure 3 (BERT/HN) and research showing fake news detection as relatively robust (Jin et al., 2020).
On the other hand, the performance gains for tasks with shorter text are substantial. In evaluation against fact-checking task XARELLO not only beats BERT-ATTACK in terms of BODEGA score (77% vs 53%), but is able to reach an AE in 7.42 queries on average, rather than 146.
The results for the targeted attacks are shown in table 3 in appendix D. The general outlook is very similar, but the targeted attacks appear more successful, especially against BERT and GEMMA.

heading: 6 Linguistic analysis
text: In order to see what these improved performance metrics look like in actual output utterances, we perform a textual analysis on 67 AEs against BERT, generated by XARELLO from the PR task in the targeted scenario. These are examples where lowcredibility text was recognised as such by the victim model, but the modifications introduced by XARELLO changed this decision. Examples of the described phenomena are shown in appendix C.
Our main takeaway is that the XARELLO agent strongly relies on making replacements at the subword level. Some of these render clear non-words which result in sentences becoming completely ungrammatical. Other non-words may pose less of a problem to reader, as they are typographically very similar to the original text. A similar phenomenon occurs in generated non-words which may appear to be infrequent or archaic words which match the orthographic and phonological rules of English2.
It is possible that readers may not notice these spelling mistakes. In multiple studies over decades, the first and last letters in a word contribute more strongly to recognition (Huebert and Cleary, 2022), for example when “hypocritically" is replaced by “*hypoclipically". AEs may therefore be ungrammatical, but still effective.
There are also patterns of adjectival replacement which appear to perform a form of semantic bleaching, or that introduce euphemistic language by replacing an emotionally charged noun or noun phrase with a pronoun3 or a more generalistic noun4. This strategy is not always successful, with around half of this type of replacement resulting in ungrammatical utterances5. Moreover, the agent may be too greedy and remove crucial constituents of an utterance6. We also discovered words which XARELLO has learned to retain, or avoid and provide replacements. It often chooses “new" or “big" to replace more semantically-transparent or emotive words, and this links to our observations about adjectival and pronominal replacement.
The observed modification types may stem from the nature of XARELLO’s victim BERT’s subword tokenisation method, as well as our use of fastText to represent word replacement candidates. In order to ‘fool’ the classifier, XARELLO may rely too
2Original: “lives and vocations"→AE: “*vassations" 3Original: “his aggressive behaviour"→AE: “own" 4Original: “that type of injustice→AE: “work". 5Original:“from the american people"→AE: “my us" 6Original:“reported on a gaping hole in"→AE: “*"
strongly on replacing pieces of words whose output resembles the orthographic and morphological rules of English but which may not be acceptable to real-world readers.
Possible methods to mitigate ungrammatical output could be to check output tokens against the N-gram probability of the AE, using semantic similarity as a heuristic for whole-token replacement, penalising tokens which do not appear in a lookup lexicon, or by using reinforcement learning from human feedback (Ziegler et al., 2020).

heading: 7 Limitations
text: Despite showing positive results, our study has several limitations. Firstly, in casting the AE generation as a RL problem (section 3.2), we discard the possibility of adding new words to the original text, which is possible in some previous AE generators, such as BAE (Garg and Ramakrishnan, 2020). Word deletion is not allowed either, even though it is one of the most natural ways of changing the form of the text while preserving its meaning (Shardlow and Przybyła, 2023). Finally, we do not perform any special treatment of subwords, e.g. as in BERT-ATTACK (Li et al., 2020). These operations are excluded in order to reduce the size of the action space, but incorporating them would be a promising avenue for future research.
Secondly, due to the long processing time, we performed only a basic exploration of the influence of the many parameters present in our solution (see appendix B). Some of these, e.g. discounting coefficient, do not have an obvious meaning in context of AE search, and their best value could only be discovered through systematic tuning. Others, such as dimensions of the Q network, number of steps and episodes, likely depend on a particular task, so would have to be tuned for each of them separately. Finally, some, such as number of candidates, would almost certainly improve the performance, but at the cost of longer adaptation time. However, these experiments might be justified if we want to simulate an attacker that consistently operates against a specific target.
Moreover, classifiers more elaborate than included here could be tested as victim models as well. We decided to use BiLSTM and BERT in the interest of comparability with previous solutions, numerous of which were evaluated against BODEGA (Przybyła et al., 2023), and GEMMA to illustrate vulnerability of modern LLMs. It is
interesting to notice that the latest of the tested approaches is also the most prone to attacks. Future work might verify if this is caused by reasons connected to our setup, e.g. relatively small datasets for tuning a network of this size, or a more fundamental weakness of very large models.
Even though misinformation is an equally grave problem for non-English Internet, our solution is only evaluated on English datasets. However, XARELLO does not depend on English in any particular way and could be applied to any language, as long as a Transformer model for it exists.
Finally, the results on the news bias assessment indicate our approach does not generalise very well to the case when numerous changes in a long text need to be made. This is because the final reward typically could not be achieved within the short horizon of the adaptation episodes. A more exhaustive search for solutions should happen during adaptation in such cases, including attacks of increasing length, as in the attack phase.

heading: 8 Ethical impact
text: The work in the domain of adversarial robustness needs to be scrutinised to make sure it does not aid the malicious actors. However, discovering AEs is definitely more likely to help build up the defences. Firstly, the examples we generate cannot be used directly to perform any attacks. That is because AEs are not transferable, so they would work only with the models they were discovered for, i.e. the victim classifiers. The models used for content moderation are likely trained using newer architectures and proprietary internal data. Secondly, despite the progress in the domain, most attack scenarios still require dozens or even hundreds of attempts are impossible to conduct in practice.
More generally, the AEs are vulnerabilities that exist due to the nature of neural networks and research such as ours is only revealing, not creating them. In our view, it is better that such techniques are obtained and discussed within the transparent research discourse rather than they would be discovered just by misinformation spreaders. For these reasons, we have decided to make the XARELLO code available7.

heading: 9 Conclusion
text: To sum up, XARELLO adapts well to the weaknesses of a victim model and in all scenarios, ex-
7https://github.com/piotrmp/xarello
cept with very long text, achieves superior performance. This result applies to various victim models, from small RNN networks to classifiers based on large modern fine-tuned LLMs. This allows us not only to find AEs for more examples, and of better quality, but also do this with fewer attempts. The evaluation becomes more realistic, as it is more likely that a platform would allow a user to send 5 consecutive messages of similar content to find an AE, rather than 133, needed by other methods.
We rely on an expectation that an attacker has already some experience with the current classifier. This is a much lower bar than in white-box attacks, assuming complete access to victim model weights. Nevertheless, in practice it will depend on the internal operations both of misinformation spreaders (e.g. experience retention) and content platforms (e.g. model updating frequency).
Ultimately, AEs allow us to find and understand the weaknesses of the investigated models before they are deployed. We can build on these methods to improve the model robustness. Our contribution could be easily used for this purpose, i.e. by including the generated AEs in the training data, as in the adversarial training paradigm (Bai et al., 2021).
We hope that by making the code of XARELLO openly available, we enable such use-cases and contribute to more reliable role of automatic classifiers in making the Internet safer.

heading: Acknowledgements
text: The work of P. Przybyła is part of the ERINIA project, which has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101060930. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the funders. Neither the European Union nor the granting authority can be held responsible for them. We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2023/016896. We also acknowledge support from Departament de Recerca i Universitats de la Generalitat de Catalunya (ajuts SGR-Cat 2021) and from Maria de Maeztu Units of Excellence Programme CEX2021-001195-M, funded by MCIN/AEI /10.13039/501100011033.

heading: A Adaptation process for GEMMA
text: Figure 4 shows the adaptation process for the GEMMA victims.
B Implementation details
Software implementation: The Q-learning environment is defined in terms of Env class in the gymnasium framework for RL (Towers et al., 2023). The neural network is implemented in pyTorch (Paszke et al., 2019).
Performance: The adaptation process is executed on a machine using one NVIDIA A100 GPU with 40 GB RAM. The duration of the process (all 20 epochs) varies depending on the victim and task performed, taking from 18 hours (BiLSTM, PR) to 42 hours (BERT, HN).
Parameter tuning: Due to the length of the adaptation process, only very limited parameter tuning was performed. To reduce the necessary processing time, in all of these experiments we used a smaller version of the model (for input up to 128 tokens), 1500 instances for adaptation and 400 for testing, both within the development portion of the PR task. Each run took around 10 hours to complete, save for the adaptation length experiment, taking proportionally longer. We tested separately adaptation length (20 or 50 epochs), memory mechanism (experiences either previously observed or drawn from memory of 4000), warmup periods (0.1, 0.3, 0.5, 0.7. 0.9), discounting parameter (0.0 or 0.5) and number of candidates (10 or 20). In general, our observations indicate low variability of the results within the ranges tested, but the best variants were selected for the main evaluation.
Adaptation: We train for 20 epochs on the adaptation dataset. The discounting coefficient is set to γ = 0.5 and exploration factor ϵ falls linearly during warmup period from 100% at the beginning of the process to 10% after 30% of the adaptation are finished and remains constant afterwards. As in the seminal work on deep RL (Mnih et al., 2015), we use a memory of previous experiences. Up to 4000 experiences are kept in a queue and 16 are randomly selected for Q update at every step. This learning is initiated every time 16 new experiences are added to the memory. The neural network is updated using Adam optimiser (Kingma and Ba, 2015) with a constant learning rate of 2× 10−5

heading: C Qualitative analysis: examples
text: Changed characters by the agent are in boldface, and the star (*) symbol indicates incoherence, ungrammaticality or disfluency.
Examples where subwords are replaced rendering an ungrammatical sentence:
• Original: “...doctors are warning that it will be continuing to spread and worsen"
• AE: “*...doctors are warning that it will be continuing to slow and badn"
• Original: “is already reeling over the revelations...a Cardinal over weekend, has been credibly accused"
• AE: “*is already poiseding over the revelations...a Cardinal over the weekend, has been nowredibly accused"
Examples of non-words which are typographically similar:
• Originals: “menace", “hypocritically", “blatently", “colluded"
• AEs: “*meace", “*hypoclipically", “*bratently", “*copoluded"
Example of a non-word which may appear like an infrequent or archaic word:
• Original: “many who have spent their lives and vocations"
• AE: “*many who have spent their lives and vassations"
Examples of adjectival replacement resulting in euphemistic language:
• Original: “that type of injustice" • AE: “that type of work"
Examples of a pronoun replacing a noun/noun phrase:
• Original: “his aggressive behaviour", “vicious comments", “treated as criminals"
• AE: “his own behaviour", “his comments", “treated as it"
An example where this does not work well:
• Original: “from the american people" • AE: “*from the my us"
An example where a whole constituent of a sentence is removed unsuccessfully:
• Original: “reported on a gaping hole in" • AE: “*reported on a in"

heading: D Results in the targeted attacks
text: Table 3 includes the results of the evaluation in the targeted scenario.

heading: E Text examples
text: Table 4 shows examples of credible and noncredible text in each task.

heading: Future Work
text: These operations are excluded in order to reduce the size of the action space, but incorporating them would be a promising avenue for future research. Secondly, due to the long processing time, we performed only a basic exploration of the influence of the many parameters present in our solution (see appendix B). Some of these, e. g. discounting coefficient, do not have an obvious meaning in context of AE search, and their best value could only be discovered through systematic tuning. Others, such as dimensions of the Q network, number of steps and episodes, likely depend on a particular task, so would have to be tuned for each of them separately. Finally, some, such as number of candidates, would almost certainly improve the performance, but at the cost of longer adaptation time. However, these experiments might be justified if we want to simulate an attacker that consistently operates against a specific target. Moreover, classifiers more elaborate than included here could be tested as victim models as well. We decided to use BiLSTM and BERT in the interest of comparability with previous solutions, numerous of which were evaluated against BODEGA (Przybyła et al. , 2023), and GEMMA to illustrate vulnerability of modern LLMs. It is
interesting to notice that the latest of the tested approaches is also the most prone to attacks. Future work might verify if this is caused by reasons connected to our setup, e. g. relatively small datasets for tuning a network of this size, or a more fundamental weakness of very large models. Even though misinformation is an equally grave problem for non-English Internet, our solution is only evaluated on English datasets. However, XARELLO does not depend on English in any particular way and could be applied to any language, as long as a Transformer model for it exists. Finally, the results on the news bias assessment indicate our approach does not generalise very well to the case when numerous changes in a long text need to be made. This is because the final reward typically could not be achieved within the short horizon of the adaptation episodes. A more exhaustive search for solutions should happen during adaptation in such cases, including attacks of increasing length, as in the attack phase