# Fast-M Adversarial Training Algorithm for Deep Neural Networks
## 5 Fast-M Adversarial Training Algorithm
### Method  
Inspired by the ideas of [PERSON] et al. (2019) regarding the impact of PGD adversarial training iterations on model robustness, this paper proposes replacing the simple increase in the internal adversarial perturbation calculation in fast adversarial training with staged increments. Fast adversarial training can be seen as giving one adversarial perturbation calculation to the neural network model in each round of training, where each round of training has an \"even distribution\". However, adversarial training in the early stages of neural network model training has little effect on improving model robustness. Therefore, the proposed method removes the adversarial perturbation calculation iteration in the early stages of fast adversarial training to lower computational costs and increase the number of calculation iterations of the adversarial perturbation in the later stages of model training to obtain better adversarial samples and improve the robustness of the training model. Based on the above points, this paper proposes a Fast-M adversarial training algorithm using the staircase iteration strategy, where \(M\) denotes the predetermined number of stages. The core of this method is the strategy of calculating the number of iterations for adversarial perturbation calculation at different stages. The proposed method divides the total training rounds \(T\) of the model's adversarial training into \(M\) stages, and the number of iterations for adversarial perturbation calculation at each stage is \(m\). The method adopts the approach of incrementally increasing the number of adversarial perturbation calculations for each stage. If the current training round is \(t\), the number of iterations for adversarial perturbation calculation at the current stage is \(m\), which is calculated with the following formula:  
\[m=\lfloor(t\cdot(M/T))\rfloor,\ t\in[0,T-1] \tag{4}\]  
The difference between this strategy and the one in the Amata adversarial training method lies in the fact that the number of iterations \(m\) in this strategy starts from 0. That is, the model's adversarial training does not perform adversarial perturbation calculations in the first stage. Only the number of stages \(M\) needs to be set, and the iteration number of each stage is calculated using the floor function. Therefore, the range of \(m\) is \([0,M-1]\). The key to the Fast-M adversarial training algorithm is to use a fixed large attack step size and to increase the number of iterations \(M\) in stages for adversarial training. The specific process is given in Algorithm 4. ```
Input: Training rounds \(T\), training model \(\mathbf{f}_{\theta}\), training dataset \((x,y)\) and its size \(N\), perturbation radius \(\epsilon\), attack step size \(\mathbf{\mathcal{O}}\), number of stages \(M\) Initialize \(\mathbf{\theta}\)//Initialize model weight parameter \(\mathbf{\theta}\) for\(t=1\to T-1\)do \(m=\lfloor(t\cdot(M/T))\rfloor\) for\(i=1\to N\)do \(\delta=Uniform(-\epsilon,\epsilon)/\text{Randomly initialized perturbation, bounded by size }\pm\epsilon\) for\(j=0\to m\)do \(\delta=\delta+\alpha\cdot\text{sign}(\
abla_{s}L(f_{\theta}(x_{i}+\delta),y_{i}))\) \(\delta=\text{clip}(\delta,-\epsilon,\epsilon)\) end for \(\theta=\theta-\
abla_{\theta}L(f_{\theta}(x_{i}+\delta),y_{i})\) //Update model weight parameter \(\mathbf{\theta}\)  end for
```  
**Algorithm 4** Fast-M Adversarial Training  
A round of fast adversarial training can be split into a standard training round and a round of FGSM adversarial perturbation calculation with random initialization. The main increase in training time compared to standard training is the time used for backward propagation in the additional round of FGSM adversarial perturbation calculation. For one Epoch \(=\)\(T\) of fast adversarial training, the additional backward propagation count is \(T\) (assuming training on a complete dataset and ignoring the batch size).