# A Research on Manipulator-Path Tracking Based on Deep Reinforcement Learning
## 2 Method
### Soft Actor-Critic  
Although the DQN algorithm, which is a milestone in deep reinforcement learning, solves the problem of high-dimensional and continuous input states, which cannot be solved by classical reinforcement learning, it still cannot solve situations in which the output actions are high-dimensional and continuous (such as multi-degree-of-freedom manipulator). Although the other deep-reinforcement-learning algorithms (such as DDPG [25], TD3 [26], and other algorithms) can handle cases in which the output action is high-dimensional and continuous, they usually have high sample complexity and weak sample convergence, which lead to some additional hyperparameter tuning. The Soft Actor-Critic (SAC) algorithm [27; 28] is a reinforcement-learning algorithm that introduces the maximum-entropy theory. In the framework of the algorithm, the strategy not only needs to maximize the expected cumulative reward value, but also needs to maximize the expected entropy, as shown in Equation (3),  
\[\pi^{*}=\max\sum_{t=0}^{T}E_{(s_{t},a_{t})\sim p_{\pi}}[r(s_{t},a_{t})+\alpha H (\pi(\cdot|s_{t}))] \tag{3}\]  
where \(\alpha\) is the weight of the entropy term, which can determine the relative importance of the entropy term relative to the reward term, thereby controlling the randomness of the optimal strategy. The maximum-entropy framework uses a technique called \"Soft Policy Iteration\" to alternately evaluate and improve policies in order to accomplish this goal. In an environment where the state space is discrete, the method can obtain the soft \(Q\) value from the randomly initialized function \(Q:S\times A\to R\) and repeatedly apply the modified Bellman backup operator \(T^{\pi}\), as shown in Equation (4),  
\[T^{\pi}Q(s_{t},a_{t})=r(s_{t},a_{t})+\gamma E_{s_{t+1}\
eq s_{t}}[V(s_{t+1})] \tag{4}\]where  
\[V(s_{t})=E_{a_{t}\sim\gamma}[Q(s_{t},a_{t})-\log\pi(a_{t}|s_{t})] \tag{5}\]  
is the soft state-value function used to calculate the policy value in policy evaluation. In the continuous state, a neural network with parameters is first used to replace the soft \(Q\)-function \(Q_{\theta}(s_{t},\ a_{t})\), and then it is trained to minimize the Bellman residual, as shown in Equation (6),  
\[I_{Q}(\theta)=E_{(s_{t},a_{t})\sim D}\bigg{[}\frac{1}{2}\big{(}Q_{\theta}(s_{t},a_{t})-\big{(}r(s_{t},a_{t})+\gamma E_{s_{t+1}\sim p}[V_{\overline{\theta}}(s_ {t+1})]\big{)}\big{)}^{2}\bigg{]} \tag{6}\]  
which can also be optimized with stochastic gradients:  
\[\overset{\wedge}{\
abla}_{\theta}J_{Q}(\theta)=\
abla_{\theta}Q_{\theta}(s_{t },a_{t})(Q_{\theta}(s_{t},a_{t})-r(s_{t},a_{t})-\gamma V_{\overline{\theta}}(s_ {t+1})\big{)} \tag{7}\]  
where \(V_{\overline{\theta}}(s_{t+1})\) is estimated by the target network of \(Q\) and the Monte Carlo estimation of the soft state-value function sampled from the experience pool. The goal of policy improvement is to maximize the available reward, so the policy must be updated to the new soft \(Q\) function's exponential form and restricted to a parameterized distribution (such as the Gaussian distribution). The policy must then be projected back into the permissible policy space using an information projection defined in terms of the Kullback-Leibler (KL) divergence, as displayed in Equation (8),  
\[\pi_{new}=\underset{\pi^{\prime}\in\Pi}{\arg\min}D_{KL}\bigg{(}\pi^{\prime}( \cdot|s_{t})\text{P}\frac{\exp(Q^{\pi_{old}}(s_{t},\cdot))}{Z^{\pi_{old}}(s_{ t})}\bigg{)} \tag{8}\]  
where \(Z^{\pi_{old}}(s_{t})\) can be ignored because it has no effect on the gradient. Furthermore, the policy \(\pi_{\phi}(a_{t}|s_{t})\) is parameterized with a neural network that can output mean and variance to define a Gaussian distribution, and then the parameters of the policy are learned by minimizing the expected KL divergence, as shown in Equation (9).