## 2 Research Background
### RL Algorithms  
The basic decision-making model of RL is MDP. When the state and action space is small, RL assigns a value to each state or state-action pair, where \(V^{\pi}(s)\) denotes the value of state \(s\). \[V^{\pi}(s)\dot{=}\mathbb{E}_{\pi}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_ {0}=s\Big{]}, \tag{1}\]  
\(Q^{\pi}(s,a)\) denotes the value of state-action pair \((s,a)\). \[Q^{\pi}(s,a)\dot{=}\mathbb{E}_{\pi}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s _{0}=s,a_{0}=a\Big{]}. \tag{2}\]  
By definition, the Bellman equation is obtained:  
\[V^{\pi}(s)=\mathbb{E}_{\pi}\Big{[}R(s,a,s^{\prime})+\gamma V^{\pi}(s^{\prime}) \Big{]}. \tag{3}\]  
Taking the state value function as an example, the Bellman equation can be defined by Equation (4). \[V^{\pi}(s) =\mathbb{E}_{\pi}\Big{[}\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_{0} =s\Big{]} \tag{4}\] \[=\mathbb{E}_{\pi}\Big{[}R(s,a,s^{\prime})+\gamma\sum_{t=0}^{\infty }\gamma^{t}r_{t}|s_{0}=s\Big{]}\] \[=\mathbb{E}_{\pi}\Big{[}R(s,a,s^{\prime})+\gamma V^{\pi}(s^{\prime })\Big{]}.\]  
The optimal strategy \(\pi^{*}\) satisfies \(\forall s\in S\), \(V^{*}(s)=\max_{\pi}V^{\pi}(s)\), where \(V^{*}\) is the optimal value function, which satisfies [PERSON]'s optimal equation. \[V^{*}(s)=\max_{a}\mathbb{E}\Big{[}R(s,a,s^{\prime})+\gamma V^{*}(s^{\prime})\Big{]}. \tag{5}\]  
Define the Bellman optimal operator \(\mathbb{T}:\mathbb{R}^{|S|}\rightarrow\mathbb{R}^{|S|}\). \[\mathbb{T}V(s)\dot{=}\max_{a}\mathbb{E}\Big{[}R(s,a,s^{\prime})+\gamma V(s^{ \prime})\Big{]}. \tag{6}\]  
The key step of the value iteration algorithm can be obtained: \(V_{n+1}=\mathbb{T}V_{n}\). Synthesizing the multi-step Bellman evaluation operator \(\mathbb{T}^{\pi}\), the operator \(\mathbb{T}^{\lambda,\pi}\) that satisfies the compression map can be obtained in \(\mathbb{T}^{\lambda,\pi}V\dot{=}(1-\lambda)\sum_{i=0}^{\infty}\lambda^{i}( \mathbb{T}^{\pi})^{i+1}V\), and thus get the \(\lambda\)-policy iteration algorithm. Therefore, policies can be expressed based on value functions, or can be defined directly. We summarize related RL algorithms based on tabular values, value function evaluation, and policy evaluation. RL Algorithms Based on Tabular Values: Policy Iteration and its improved algorithm continue to repeat the two stages of Policy Evaluation and Policy Improvement until the optimal Policy and the optimal value function [25] is converged and obtained. However, the original Policy Iteration algorithm requires the value function to converge to the optimal solution for each Policy Evaluation before making a Policy Improvement. In the process of Policy Evaluation, the Value Iteration algorithm performs only one time of Iteration. When the state space is large, it takes a long time to scan all states. In this case, Asynchronous Value Iteration improves efficiency by iterating only one sample of the state space at a time. In the stage of Policy Evaluation, an optimal solution can be approximated more quickly with \(\lambda\)-Policy Iteration synthesizing all the multi-step Bellman evaluation operators and doing only one iteration; While Q-learning uses [PERSON]'s optimal operator, which is \(\max_{a}\mathbb{E}[R(s,a,s^{\prime})+\gamma V(s^{\prime})]\approx\max_{a}[R(s, a,s^{\prime})+\gamma V(s^{\prime})]\), to approximate the optimal value function and perform learning control. This Policy is characterized by the difference between behavioral policy and learning policy, falling into the off-policy learning. RL Algorithms Based on Value Function Evaluation: Large-scale or continuous states and actions caused the curse of dimensionality, making the tabular value based RL algorithms ineffective. However, the ways to solve these this curse include: (i) to reduce the state and action space; (ii) to estimate the value function \(V\) or Policy by using parameters far smaller than the number of states and actions. The state and action space is mapped by function to a set of parameter dimensions that are much smaller than the size of the state and action space.