## 4 Methodology
### Preliminaries  
In reinforcement learning (RL), the agent continuously engages with its surroundings. The agent observes the environmental state \(s_{n}\) in the state space \(\mathcal{S}\) for each time slot n and then follows the policy \(\pi(a_{n}\mid s_{n})\) by selecting actions \(a_{n}\) from the action space A, and the environment then changes to the following state \(s_{n+1}\in\mathcal{S}\). According to the environment \(P(s_{n+1}\mid s_{n},a_{n})\), reward function \(R(s_{n},a_{n},s_{n+1})\) sends a reward signal \(r_{n}\) to the agent. Until the maximum number of iterations is reached or the agent notices a terminal state, this process keeps going indefinitely. This kind of process is usually expressed as MDP, which defines a quintuple \(\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma)\). The cumulative reward from state \(s_{n}\) to the future is defined as:  
\[R_{n}=\sum_{\ell=0}^{\infty}\gamma^{l}r_{n+\ell} \tag{17}\]  
where \(\gamma\in(0,1)\) is called the discount factor. \(r\) is the instantaneous reward. The value function \(v_{\pi}(s)=\mathbb{E}_{\pi}[R_{n}\mid s_{n}=s]\) is the expectation of the cumulative reward obtained by following the policy \(\pi\) in state \(s\). To choose the action in accordance with the policy in the state s and obtain the cumulative reward expectations, use the action value function \(q_{\pi}(s,a)=\mathbb{E}_{\pi}[R_{n}\mid s_{n}=s,a_{n}=a]\). They represent a good state and a good state-action pair, respectively, and are defined by \(v_{\pi}(s)=\sum_{a\in A}\pi(a\mid s)q_{\pi}(s,a)\) to convert each other. Finding the best policy \(\pi^{*}\) that maximizes predicted cumulative reward for any state in the state space is the aim of RL, as shown in Formula (18). \[v_{\pi^{*}}(s)=\underset{\pi}{maxv_{\pi}(s)}=\underset{a}{maxq_{\pi^{*}}(s,a )},s\in\mathcal{S} \tag{18}\]  
To approximate a policy or value function, DRL employs DNN. DNNs are used in value-based DRL techniques to simulate value functions. All strategies can be evaluated, allowing the best strategy to be chosen, if strategy is finite, meaning there are a finite number of states and actions. A value-based DRL method's main goal is to reduce the discrepancy between the value network and the real value function. In writing, the objective function is:  
\[L^{V}(\theta)=\mathbb{E}_{n}[v_{\pi^{*}}(s_{n})-v(s_{n};\theta)]^{2} \tag{19}\]  
where \(v(\cdot\,;\theta)\) is the value network and \(\theta\) is its parameter set. \(v_{\pi^{*}}\) represents the true value. \(\mathbb{E}_{n}[\ ]\) represents the empirical mean of a finite batch of samples in the optimal way of sampling. In order to find the best policy, policy-based DRL methods conduct a direct search in the policy space. Comparatively speaking to a method based on a value function, the policy search can directly optimize the strategy without a value function, and the parameterized strategy can handle continuous state and action. It operates by calculating a gradient estimator for policies. The most used gradient estimator is written as Formula (20). \[\
abla L^{\text{PG}}(\theta)=\mathbb{E}_{n}\big{[}\
abla_{\theta}\log\pi(a_{n }\mid s_{n};\theta)\hat{A}_{n}\big{]} \tag{20}\]  
where \(\pi\) is a random strategy and \(\hat{A}_{n}\) is the estimation function. Generalized advantage estimation (GAE), which creates an elasticity between variation and bias, is suggested as a solution to the issue of high variance. GAE estimations are calculable using a Formula (21). \[\hat{A}_{n}^{GAE(\gamma,\theta)}=\sum_{l=0}^{\infty}\left(\gamma\phi\right)^{ l}\eta_{n+l}^{v} \tag{21}\]where \(\phi\) is a hyperparameter that is used to compromise between variance and bias. \(\eta_{n}^{v}\) is TD error and defined as Formula (22). \[\eta_{n}^{v}=r_{n}+\gamma v(s_{n+1};\omega)-v(s_{n};\omega) \tag{22}\]  
To alleviate the problem of local optimization, it is necessary to introduce off-policy, that is, using different strategies for training and sampling and training methods to improve the exploration ability of policy-based methods in the training process.