## 4 DRLAttack in the White-Box Setting  
### Deep Reinforcement Learning Architecture  
When constructing the DRLAttack, the core concept is to model the attack process as a framework compatible with reinforcement learning. In this framework, the attacker acts as an agent interacting with the environment. To achieve this, it is essential to define new types of state spaces, action spaces, and reward functions and ensure that the entire process complies with the requirements of the Markov Decision Process (MDP). This includes the state space \(S\), action space \(A\), conditional transition probabilities \(P\), reward function \(R\), and discount factor \(\gamma\) (i.e., \(S,A,P,R,\gamma\)). Here is a detailed introduction to the model architecture:  
**Environmental modeling:** In the DRLAttack framework, the environment comprises the structure and algorithmic parameters of the target recommendation system. This environment is capable of capturing user preferences and behavior patterns through data on interactions between users and items, and it uses this information to generate recommendation lists. **Agent design:** DRLAttack employs a Deep Q-Network (DQN) deep reinforcement learning algorithm as its agent. This agent improves its strategy by observing the current state, selecting appropriate actions, and receiving rewards provided by the environment. The goal of the agent is to maximize the cumulative reward. **Design of action space \(A\):** The action space defines all the possible actions that an attacker can perform, typically including the set of all items in the recommendation system. However, designing it this way often has a negative impact on the learning of the agent's attack strategies, as the number of items in a real recommendation system's training dataset is usually very large. It is challenging for the agent to identify correlations between users and items within a vast action space using limited attack resources, leading to a low success rate for attacks. To improve the design of the action space, we first analyze the training dataset before preparing the attack. Using similarity measurement methods, we identify other items similar to the target item \(v_{i}\) to form a candidate item set \(S_{v_{i}}=\{v_{1},v_{2},v_{3},v_{4},\cdots v_{x}\}\). For implicit datasets, we use the Jaccard coefficient to find candidate items. The Jaccard similarity [26] between any two items \(v_{a}\) and \(v_{b}\) is calculated as follows:  
\[Jaccard(v_{a},v_{b})=\frac{|v_{a}\cap v_{b}|}{|v_{a}\cup v_{b}|} \tag{1}\]  
where \(|v_{a}\cap v_{b}|\) represents the number of times two items are interacted with by the same user (i.e., the count of simultaneous interactions with both items by users), and \(|v_{a}\cup v_{b}|\) represents the total number of interactions where at least one of the items is interacted with by users (i.e., the count of interactions with at least one of the two items). The process of finding \(top\) n similar items first involves constructing a matrix \(R\) from the user-item implicit interaction data, where rows represent users and columns represent items. An element value of 1 indicates an interaction between a user and an item, while 0 indicates no interaction. Then, calculate the similarity: for target item \(v_{i}\), iterate through all other items in the matrix to compute the Jaccard similarity between each item and target item \(v_{i}\). Finally, rank all items based on the calculated Jaccard similarities and select the top \(n\) items with the highest similarity as the candidate set for target item \(v_{i}\). For explicit datasets (where users provide ratings for items), cosine similarity is used to identify other items most similar to the target item to form a candidate item set.