## 3 Methodology
### Cooperative Adversarial Training
If the perturbation is too large, the model does not encounter similar intensity perturbations during training, and the anti-interference ability of the model is not effectively enhanced. Here, we determine the amplitude of perturbations according to the parameter range of the embedding layer and select the best perturbation within this range. If a fixed perturbation value is used, determining the optimal parameter becomes difficult. Even small perturbations may sometimes cause greater loss, and the perturbations are irregular and diverse in the actual training process. In order to simulate this type of irregularity, we use the UCB multi-armed bandit method to select perturbations and obtain the dynamic balance between loss maximization and perturbation diversity to ensure that the model can resist both large and small perturbations. The multi-armed bandit method [59] is a classical reinforcement learning algorithm that effectively solves the balance problem of exploration and utilization. Figure 2 shows the basic process of the multi-armed bandit method. The traditional multi-armed bandit method has multiple rocker arms, and the user can obtain certain benefits by shaking each rocker arm. However, as the user does not know the specific values of the rewards, they must estimate the rewards of the rocker arm based on previous experience. Because our goal is to maximize loss, we regard each perturbation value as a rocker arm and we treat the loss generated by each perturbation value as a profit. We record the mean loss produced by each perturbation as the estimated reward, and the user selects the perturbation with the largest estimated reward. The number of times each rocker arm is used varies, and the more frequently used rocker arm tends to estimate gains more accurately; thus, we need to make each arm application as balanced as possible while ensuring that the maximum loss is selected. Moreover, in the actual training processes the model is subject to a variety of interference types. In order to obtain accurate estimated benefits and simulate the irregularity of perturbations, we use the UCB (upper confidence bound) algorithm to optimize the multi-armed bandit method. We set a factor \(\varepsilon\) to balance the loss maximization with the disturbance diversity. The balance formula is as follows:  
\[Estimated\_Q_{i}=Avg\_loss_{i}+\varepsilon\sqrt{\frac{logT}{2(t_{i}+1)}}, \tag{16}\]  
Figure 2: An illustration of the multi-armed bandit method. The agent can sense the state and obtain rewards by shaking the rocker arms. This process updates the estimated reward according to past rewards. where \(T\) is the total number of times all rocker arms are used, \(t_{i}\) denotes the number of times a single rocker arm is used, and \(Avg\_loss_{i}\) is the average loss value considering the previous instances. To enhance the exploratory ability of the algorithm and avoid obtaining local optimal solutions, we set a small factor, \(\delta\); when the probability is smaller than this factor, we randomly select the perturbation. When the probability is larger than this factor, we select the perturbation according to Formula (16). The perturbation selection process is shown in Algorithm 3. ```
Input: A set of perturbed values \(S\) and its set size \(K\)  A balance factor \(\varepsilon\) and a small random factor \(\delta\)  The total number of times all perturbations are used \(T\)  The set of times each perturbation is used \(t_{i}\), \(1\leq i\leq K\) Output: Perturbation value \(P\)
1 Initialize \(T=0\), \(t_{i}=0\), \(1\leq i\leq K\). 2\(q\leftarrow\) Generate a random number from 0 to 1 if\(q<\delta\)then
3 Random select \(s_{i}\) in \(S\) \(index\gets Get\_index(s_{i})\)
4 else
5 for\(s_{i}\in S\)do
6\(Estimated\_Q_{i}=Avg\_loss_{i}+\varepsilon\sqrt{\frac{logT}{2(t_{i}+1)}}\)
7 end for
8\(index\gets Get\_index(max(Estimated\_Q_{i})),1\leq i\leq K\)
9 end for
10\(t_{index}\gets t_{index}+1\)
11\(T\gets T+1\)
12\(P\gets s_{index}\) return \(P\)
```  
**Algorithm 3**Perturbation selection algorithm