## 3 Materials and Methods
Algorithm 1 is as follows:  
```
1: Initialize the policy \(\pi_{\varnothing}\), predictive model \(\bar{M}^{p}_{\theta_{p}}\), gradient model \(\bar{M}^{q}_{\theta_{p}}\), value function \(Q_{\varphi}\)
2: Initialize environment replay buffer \(D_{\textit{env}}\), model replay buffer \(D_{\textit{model}}\)
3:repeat
4: Take an action in real environment with policy \(\pi_{\varnothing}\), and add transition \((s,\ a,\ s^{\prime})\) to \(D_{\textit{env}}\)
5: Train predictive model \(\bar{M}^{p}_{\theta_{p}}\) and update \(\theta_{p}\) by maximizing \(J_{\bar{M}^{p}}(\theta_{p})\)
6: Train predictive model \(\bar{M}^{q}_{\theta_{p}}\) and update \(\theta_{g}\) by maximizing \(J_{\bar{M}^{q}}(\theta_{g})\)
7:for\(N_{\textit{rollouts}}\) steps do
8: Sample \(s_{t}\) uniformly from \(D_{\textit{env}}\)
9: Perform \(k\)-step model rollout on \(\bar{M}^{p}_{\theta_{p}}\) starting from \(s_{t}\) with policy \(\pi_{\varnothing}\)
10: Add samples to \(D_{\textit{model}}\)
11:endfor
12:\(D\gets\ D_{\textit{env}}\ \cup\ D_{\textit{model}}\)
13:for\(N_{\textit{updateQ}}\) steps do
14: Update \(\varphi\) using data from \(D:\ \varphi\ \leftarrow\ \varphi-\lambda_{Q}\
abla_{\varphi}J_{Q}(\varphi)\)
15:endfor
16:for\(N_{\textit{updatePolicy}}\) steps do
17: Sample trajectories of length \(H\) with policy \(\pi\) and predictive model \(\bar{M}^{p}_{\theta_{p}}\)
18: Update \(\varnothing\) using sampled trajectories and gradient model \(\bar{M}^{q}_{\theta_{p}}:\ \varnothing\ \leftarrow\ \varnothing+\lambda_{n}\
abla_{\pi}J_{\pi}(\varnothing)\)
19:endfor
20:until The policy performs well in real environment \(M\)
21:return Optimal policy \(\pi_{\varnothing}\). ```  
**Algorithm 1** Directional Derivative Projection Policy Optimization  
The core of algorithm 1 is to train two different models, namely the prediction model and the gradient model. The DDPPO algorithm uses different loss functions to train these two models, respectively, and uses them appropriately in strategy optimization. In general, the DDPPO algorithm takes into account the gradient error ignored by previous studies and effectively improves accuracy through dual-model training. However, the algorithm still does not consider deployment efficiency or time complexity. The specific differences will be compared in the fourth part. (5) Deployment-Efficient Reinforcement Learning Algorithm  
On the other hand, again for recommendation systems, researchers are focusing on deployment efficiency, trying to save time by reducing the number of tests before going online. The learning process of traditional (online) reinforcement learning (RL) can be summarized as a two-part cycle: one is to learn a policy based on collected data; the second is to deploy the policy to the environment for interaction and obtain new data for subsequent learning. The goal of reinforcement learning is to complete the exploration of the environment in such a cycle and improve the strategy until it is optimal. However, in some practical applications, the process of deploying the policy is very tedious, and the data collection process is relatively fast after deploying the new policy. For example, in the recommendation system, the strategy is the recommendation plan, and a good strategy can accurately recommend the content that users need. Considering user experience, a company usually conducts internal tests for a long time before launching a new recommendation policy to check its performance. Due to the large user base, a large amount of user feedback data can be collected within a short period of time after deployment for subsequent policy learning. In such applications, researchers are more likely to choose algorithms that require only a small number of deployments to learn good strategies. However, there is still a gap between existing reinforcement learning algorithms and theories and these real needs. In this paper, the researchers tried to fill this gap. From a theoretical perspective, researchers first provided a rigorous definition of deployment-efficient RL.