## 5 DRLAttack in the Black-Box Setting
### Introduction of LSTM Network  
After creating spy users and redesigning the reward function, we recognize that in the data poisoning attack scenario, attackers need to dynamically adjust their strategies based on feedback from the recommendation system, as both user behavior and system feedback are continuously changing. Traditional deep neural networks, while excellent at processing static data, have significant limitations in capturing long-term dependencies in time series data. Therefore, we incorporated Long Short-Term Memory networks (LSTM) into the DRLAttack attack framework under the black-box setting. LSTM [32] can handle sequences of user-item interactions and predict the next best recommendation item. Because LSTM takes into account past interaction sequences and can capture the dynamic changes in user behavior, it provides a more dynamic and flexible learning mechanism for the agent in the black-box setting data poisoning attack strategy. Specifically, at each timestep \(t\), the state--namely, the sequence of items that the user has chosen to interact with up to that point--is fed into the LSTM network. The network then outputs the next optimal action based on the current sequence state--deciding which item ID to interact with next. The network structure and decision process primarily include the following steps:  
**State representation:** At timestep \(t\), state \(s_{t}\) is represented as a sequence of items \(\tau_{t}=\{a_{0},a_{1},a_{3},\cdots a_{t-1}\}\) that the user has interacted with. This sequence is transformed into a vector through an embedding layer to be fed into the LSTM layer. **Defining the LSTM network:** The LSTM layer receives an embedding of the user and item sequences along with the previous cell state \(C_{t-1}\) and outputs the current timestep's hidden state and cell state as follows:  
\[(h_{t},C_{t})=LSTM(e_{u},e_{a_{0}},e_{a_{1}},e_{a_{3}},\ldots,e_{a_{t-1}},C_{t-1}) \tag{14}\]  
where \(h_{t}\) is the LSTM hidden state at timestep \(t\), which integrates the user's features with the features of all previously interacted items at all past timesteps. \(C_{t}\) is the LSTM cell state at timestep \(t\), acting as an internal memory mechanism that carries the flow of information between timesteps, responsible for storing and transferring critical information across different timesteps. \(e_{u}\) is the representation of the user's features. \(e_{a_{t-1}}\) is the feature representation of items interacted with prior to timestep \(t\). The size of all hidden layers \(|h|\) is set to match the size of the feature representations \(|e|\). This network design allows the LSTM to retain historical information and use it to predict future actions. **Action selection:** After establishing the LSTM network, the policy network samples the next action \(a_{t}\) based on the hidden state \(h_{t}\), which is the next item ID to interact with. This follows a multinomial distribution over the action space \(A\)[33], meaning each possible action has a probability associated with it. These probabilities are calculated based on \(h_{t}\) as follows:  
\[Q(s_{t},a_{t},\theta)\sim\pi(\cdot|s_{t})=\frac{\exp(D(h_{t})\cdot e_{i})}{ \sum_{i\in A}\exp(D(h_{t})\cdot e_{i})} \tag{15}\]  
where \(D(\cdot)\) is a deep neural network that employs a dual-layer DNN with ReLU activation functions, which helps capture complex nonlinear relationships and enhances the model's representational power. The size of the DNN layers is set to match the dimension of the item feature representation \(|e_{i}|\), ensuring that the network's output can be directly compared and computed with the item features. \(\theta\) represents the network parameters. \(D(h_{t})\cdot e_{i}\) denotes the dot product operation between two vectors. The entire process can be interpreted as follows: \(D(h_{t})\) generates a score for each possible action, and the dot product operation adjusts these scores, outputting the value of each potential action to reflect the advantage of specific actions.