## 3 Materials and Methods
### Bot Strategy Model  
Reinforcement Learning (RL) has been widely applied in decision making in virtual adversarial environments. Proximal Policy Optimization (PPO), one of the most widely used RL models, has been applied to control one bot, successfully increased its influence, and evaded a detector simultaneously [10]. Thus, PPO with an MLP policy is applied to train our bot strategy model. The bot's action is controlled by two agents responsible for action selection and target selection, respectively. Observation space: Due to the scale of the experiment, the whole environment cannot be input and thus limits the bot strategy model's input to the bot's current state, i.e., the features used to train the RF-based bot detection model. The two agents share the same observation space of size \(n_{f}\). Action space: To enable bot interactions within the virtual environment, a set of actions mimicking typical Twitter user behavior is defined. The bot's possible actions include  
* Posting a tweet;
* Updating the profile;
* Retweeting, mentioning, or following an account;
* Creating or deleting a list;
* Waiting for a day. Thus, the action selection agent's action space is a discrete space of 8 actions. Since we did not generate new texts and focused on decision making, tweets posted by 10,000 bots that are most dissimilar from \(Seed_{F}\) were selected randomly to build tweet pools of 1000 for posting, retweeting, and mentioning, respectively. The \"wait for a day\" action is added to simulate the flow of time. This allows for the measurement of important features of an account, such as posting frequency and account age, which are commonly used in existing bot detection models. Moreover, this adds to the realism of the simulation and improves the practicality of the bot strategy model. As introduced in Section 3.4, the action space of the target selection agent is a discrete space consisting of 1000 actions. Each action corresponds to the selected user from one user cluster. Reward Formulation: The bots are categorized into two classes: fake accounts and influencer bots. A fake account is simply a bot that evades bot detection. Conversely, an influencer bot needs to escape bot detection and increase its influence to qualify as an influencer bot. Overall, we believe that fake accounts and influencer bots represent the majority of bot types. For instance, bots designed to manipulate votes and fake followers are essentially fake accounts, since their primary goal is to evade detection. Similarly, influencer bots embody the objectives of spam and political bots, as they aim to gain influence while avoiding detection. Two distinct rewards are developed to train the bot strategy model accordingly. The profit of a fake account is proportional to its survival time:  
\[P^{f}=l,D(s_{l})>0.5\quad\&\quad D(s_{l-1})<0.5 \tag{7}\]  
where \(l\) represents the time when the account is detected and suspended. Note that RF-based detectors' direct outputs and those from deep learning-based detectors typically range from 0 to 1, which are considered the probability of the input account being a bot, i.e., the bot score. A higher bot score indicates that the detector can classify the account as a bot with greater confidence. The profit of an influencer is quantified by counting the number of views of their posts, which is related to the number of followers and tweets. Let \(|N(u,t)|\) be the number of followers at time \(t\) and \(Post_{t}\in\{0,1\}\) represent whether the bot posts a tweet at time \(t\). Assuming that all followers would view all tweets posted after they follow the bot, the total number of views of the bot is  
\[P^{i}=\sum_{t=0}^{l}|N(u,t)|\times Post_{t} \tag{8}\]  
To mitigate the sparse reward problem, the reward function is designed by incorporating both the immediate profit and the dynamically monitored predictions from the bot detection model. \[R_{t}^{f}= \begin{cases}D(s_{t})-D(s_{t-1}),t<l\\ \frac{t}{t_{m}},t=l\end{cases}  \tag{9}\]  
\[R_{t}^{i}=