# DDRCN: Deep Deterministic Policy Gradient Recommendation Framework Fused with Deep Cross Networks
## 3 Preliminaries  
Reinforcement learning includes the environment and agent, and the goal of maximizing the cumulative reward is achieved through the continuous interaction between them, which can be modeled as a Markov Decision Process (MDP), abstracted as a five-tuple \(<S,A,R,P,\gamma>.S\) denotes the set of states, \(A\) denotes the set of actions, \(R\) is the reward function, \(P\) is the transition probability, and \(\gamma\) is the discount coefficient. Therefore, the recommendation system is abstracted as an agent, the user is abstracted as the environment, the recommendation system performing a recommendation is abstracted as an action, and the user's behavior characteristics are abstracted as states. In each recommendation process, the recommendation system dynamically updates the action based on the user's status and feedback to ensure the maximum cumulative profit of the recommendation. The specific MDP framework of the recommendation process is as follows:  
* State space \(S\): \(S\) is the set of environment states, and \(s_{t}\in S\) represents the state of the agent at time \(t\), which is the interaction between the user and the recommendation system at time t.
* Action space \(A\): \(A\) is the set of actions that the agent can take, and \(a_{t}\in A\) represents the action taken by the agent at time t. In particular, actions here refer to action vectors.
* Reward \(R\): The recommendation system will recommend actions based on the user's state and behavior, and the user will provide feedback (click, rating, retweet, etc.). Recommendation systems receive instant rewards based on user feedback \(r(s,a)\).
* State transition \(P\): When the recommendation system recommends action at time step \(t\), the state of the user at this time is transferred from \(s_{t}\) to \(s_{t+1}\).
* Discount factor \(\gamma\): \(\gamma\in[0,1]\)is a discount factor used to indicate the importance of future rewards, with \(\gamma\) being close to 1 to consider long-term rewards, and \(\gamma\) being close to 0 to consider immediate rewards.  
Figure 2 illustrates the interaction process between the agent and the environment. The agent gives an action \(a_{t}\) according to the current state \(s_{t}\). After receiving the action of the agent, the environment converts the state from \(s_{t}\) to \(s_{t+1}\) and rewards the agent \(r_{t}\) for its behavior. The agent receives the reward \(r_{t}\) and the state \(s_{t+1}\), and takes the next action \(a_{t+1}\), and so on. In particular, the agent's action does not represent the recommended item or the recommendation sequence, but a continuous parameter vector. This vector is then the inner product with the item embedding to obtain the item's rating, and the specific item is recommended to the user according to the order of the rating. In this paper, we construct a recommendation framework through the Actor-Critic network. In the Actor-Critic network, the Actor-network is a policy network, and the Critic-network is a state value estimation network. Through the dual networks, the model acts toward high cumulative rewards. In contrast, Q-learning is performed by storing Q-values (values of state action pairs) in Q-tables and continuously going through the Q-tables to update them. This approach is not suitable for handling large-scale scenarios. SA2C [21] is a variant based on the Actor-Critic network for recommendation scenarios. It introduces supervised learning to simulate the Actor-network to generate correct actions. In this paper, the Actor-Critic network is trained by a deep deterministic policy gradient network. The model is converged by setting the Actor Target network and the Critic Target network.