## 4 DRLAttack in the White-Box Setting
### Deep Reinforcement Learning Architecture
For any two items \(a\) and \(b_{i}\), their cosine similarity [26] is calculated as follows:  
\[Similarity(a,b)=\frac{\sum_{i=1}^{n}a_{i}\times b_{i}}{\sqrt{\sum_{i=1}^{n} a_{i}^{2}\times\sqrt{\sum_{i=1}^{n}b_{i}^{2}}}} \tag{2}\]where \(a_{i}\) and \(b_{i}\) represent the ratings given by user \(i\) to items \(a\) and \(b\), respectively, and \(n\) represents the total number of users who rated the items. Additionally, if the action space only includes items similar to the target item, this may make the attack actions more conspicuous and easily recognizable by the recommendation system's anomaly detection mechanisms. On the other hand, an action space purely based on similar items might limit the agent's exploration range, affecting learning efficiency and the ultimate effectiveness of the attack. Therefore, we opt to combine \(n\) ordinary item and \(top\) n candidate items to form the action space. The redesign of action space \(A\) can make the behavior of fake users more akin to that of real users, reducing the risk of detection. The mix of candidate and ordinary items also enhances the diversity of the action space. **Design of state space 5:** The core of the state space is the sequence of items that the agent has chosen in the steps leading up to the current step \(t\). Existing data poisoning attack methods often fix the target item's identifier at the beginning of the state space and then base subsequent actions on the agent's learning. While this approach is straightforward, it generates a highly correlated sequence of fake user behaviors that can be easily detected and mitigated by the recommendation system's defensive mechanisms. Practical scenarios also require the consideration of limited attack resources, as each fake user can interact with only a limited number of items. Therefore, we define state space \(S\) as a sequence \(s_{1},s_{2},\ldots,s_{T}\), where \(s_{t}\) represents the sequence of items a fake user chooses to interact with at timestep \(t\). The length of the sequence is determined by the maximum number of items \(T\) that a fake user can manipulate. In the DRLAttack framework, the goal is to generate \(N\) independent fake user behavior sequences, each reaching the predetermined maximum length of \(T\). Therefore, the process is set up to proceed in parallel, where each fake user independently selects actions from action space \(A\) without interfering with others, until their respective sequences are complete. For each fake user \(u_{i}\) and at each timestep \(t\), the update rule for the behavior sequence \(S_{i,t}\) is \(S_{i,t}=S_{i,t-1}+[a_{i,t}]\), continuing until \(|S_{i,t}|=T\). This design facilitates the parallel generation of multiple independent fake user behavior sequences under limited attack resources, enhancing the stealth and efficiency of the attack. **Design of conditional transition probability:** The transition probability \(p(s_{t+1}|s_{t},a_{t})\) defines the likelihood that, following an action \(a_{t}\) taken by the attacker at timestep \(t\), the state transitions from the current state \(s_{t}\) to the next state \(s_{t+1}\). **Design of reward function \(R\):** In reinforcement learning, the reward provided by the environment offers immediate feedback to the agent. This paper focuses on push attacks in a targeted attack scenario, where the attacker aims to recommend the target item to as many users as possible. DRLAttack employs a rank-based reward function, where a positive reward is calculated whenever the target item \(v_{*}\) appears on user \(u_{i*}^{A}\in\mathcal{U}_{*}^{A}\subset\mathcal{U}^{A_{*}}\)s recommendation \(top\)-k list. Here, \(\mathcal{U}^{A}\) represents the set of all users, and \(  
recommendation lists \(L_{u}^{A}\), for the target users, the reward value \(R(s_{t},a_{t})\) is calculated as the sum of all HR@K values for the user set \(U_{*}^{A}\) as follows:  
\[