## 4 DRLAttack in the White-Box Setting
### Optimization Process of Attack Strategies  
In the previous section, the Markov Decision Process (MDP) was used to simulate the interaction between attackers and the recommendation system. To address the challenges posed by high-dimensional state and action spaces, we utilize a Deep Q-Network (DQN) in the DRLAttack framework, enhanced with a Dueling Network architecture to further optimize network performance. A Dueling Network is a neural network design that decomposes the Q-function into two separate components [27]: the value function \(V(s)\) and the advantage function \(A(s,a)\). The purpose of this design is to better learn the relative advantage of choosing each action in a given state, while maintaining an overall estimate of the state value. In the Dueling Network, the calculation of the Q-function is accomplished by combining the value function and the advantage function as follows:  
\[Q(s,a)=V(s)+\left(A(s,a)-\frac{1}{|A|}\sum\limits_{a^{\prime}}A(s,a^{\prime})\right) \tag{5}\]  
where \(\frac{1}{|A|}\sum\limits_{a^{\prime}}(s,a^{\prime})\) represents the average of the advantage functions across all actions, which ensures that the contribution of the advantage function to the Q-value is zero when summed over all possible actions. This arrangement allows the value function \(V(s)\) to remain independent of the chosen action, while the advantage function \(A(s,a)\) represents the additional value relative to the average action. This separation facilitates a more focused learning of state values and the specific advantages of each action, improving the efficiency and effectiveness of policy optimization in complex environments. In the DRLAttack framework, the specific details of the network implementation include the input layer, hidden layers, and output layer, as shown in Figure 1. Input Layer: The input to the network is the representation of state \(s\), which in the context of recommendation systems is the sequence of behaviors of fake users. Hidden Layers: These consist of multiple fully connected layers (DNN) designed to extract deep features from the state. Output Layer: The value function and advantage function are combined according to Equation (5) to generate Q-value estimates for each action. During each iteration of the training phase, there are mainly two stages: the replay memory generation stage and the parameter update stage. In the replay memory generation stage, the agent selects actions from a mixed item set based on the current Q-network and \(\epsilon\)-strategy. After executing action \(a_{t}\), the environment returns a new state \(s_{t+1}\) and reward \(r_{t}\). In the parameter update stage, the agent randomly samples an experience tuple \((s_{t},a_{t},r_{t},s_{t+1})\) from the experience buffer, and then updates parameters by minimizing the defined loss as follows:  
\[\mathcal{L}_{\theta}=\mathbb{E}_{s,a,r,s^{\prime}}[(r+\gamma Q^{target}(s,a; \theta)-Q^{policy}(s,a;\theta))^{2}] \tag{6}\]  
where \(Q^{target}(s,a;\theta)\) is the output of the target network, used only for calculating the target value, and \(Q^{policy}(s,a;\theta)\) is the output of the current network, used to generate new actions. These two networks have the same architecture but different parameters. Every fixed number of timesteps, the parameters of \(Q^{target}(s,a;\theta)\) and \(Q^{policy}(s,a;\theta)\) are synchronized. Additionally, we introduced the Prioritized Experience Replay (PER) [28] mechanism into the DRLAttack framework. In PER, each experience tuple \(Q^{target}(s,a;\theta)\) and \(Q^{policy}(s,a;\theta)\) is assigned a priority that reflects the importance of learning from that experience.