# Autonomous Driving Decision-Making Method Based on Spatial-Temporal Fusion Trajectory Prediction
## 2 Method
### Decision Model Based on Reinforcement Learning  
The decision process for autonomous driving can be regarded as a model-free Markov Decision Process (MDP), which consists of a quintuple \((S,A,P,R,\gamma)\), where \(S\) represents the state space, which is similar to the vehicle's \"point of view\", representing the information about the surrounding environment that the vehicle should pay attention to, \(A\) is the action space, \(P\) is the state transfer probability, \(R\) is the immediate reward obtained by taking an action, and \(\gamma\) is the decay factor. Reinforcement learning is commonly used to solve [PERSON] decision problems, which is essentially a [PERSON] decision process, and the key to achieving optimal decisions lies in the design of the state space, action space, and reward function. The SAC algorithm is an approach based on the maximum entropy reinforcement learning framework, which achieves a good balance between exploration and exploitation by introducing an entropy regularization term that enhances the robustness and the exploration ability of the intelligences and prevents the strategy from converging to a local optimal point prematurely. The flowchart of the SAC algorithm is shown in Figure 3.  
Where the strategy entropy is calculated as:  
\[\mathcal{H}(\pi(\cdot|s_{t}))=-\mathbb{E}_{a_{t}\sim\pi}\log\pi(a_{t}|s_{t})=- \log\pi(\cdot|s_{t}) \tag{10}\]  
It denotes the degree of randomness of the strategy \(\pi\) in the state \(s\). As a result, the goal of maximum entropy reinforcement learning, in addition to maximizing the cumulative reward, should also be to consider how to make the policy more random. \[\pi^{*}=\text{argmax}_{\pi}\mathbb{E}_{\pi}\Bigg{[}\sum_{t}r(s_{t},a_{t})+ \alpha\mathcal{H}(\pi(\cdot|s_{t}))\Bigg{]} \tag{11}\]  
where \(\alpha\) is a regularization factor to control the importance of entropy. Larger values of \(\alpha\) indicate more encouragement for the exploration of the intelligent body. In addition, a strategy function \(\pi\) as well as two action value functions \(Q\) are mainly used in the SAC algorithm. The strategy function \(\pi_{\varphi}(\cdot|s)\) inputs the state and outputs the action taken in that state with parameter \(\varphi\). The action value function \(Q(s,a)\) inputs the state \(s\) and action \(a\), and outputs a \(Q\) value that reflects the goodness or badness of the currently taken action, with parameters \(\theta_{1}\) and \(\theta_{2}\), respectively. When the function is updated, the action value network selects the smaller of the two target \(Q\) values, thus alleviating the problem of overestimation of values. The objective function of any \(Q\) function is:  
\[J_{Q}(\theta)=\mathbb{E}_{(s_{t},a_{t})\sim D}\Bigg{[}\frac{1}{2}\big{(}Q_{ \theta}(s_{t},a_{t})-\hat{Q}(s_{t},a_{t})\big{)}^{2}\Bigg{]} \tag{12}\]  
Figure 3: Flowchart of the SAC algorithm. where:  
\[\hat{Q}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim p}\bigg{[} \gamma\Big{(}\min_{i=1,2}\!Q_{\theta_{i}^{-}}(s_{t+1},a_{t+1})-\alpha\log\pi(a_{ t+1}|s_{t+1})\Big{)}\bigg{]} \tag{13}\]  
The objective function of the policy network update is shown by Equation (14), which aims to improve the policy by minimizing the KL dispersion of the action distribution, so that the improved policy is closer to the exponential action distribution of the current Q-function, thus improving the overall policy performance. \[J_{\pi}(\varphi)=\mathbb{E}_{s_{t}\sim D}\bigg{[}D_{KL}\bigg{(}\pi_{\varphi}( \cdot|s_{t})\parallel\frac{\exp(Q_{\theta}(s_{t},\cdot))}{Z_{\theta}(s_{t})} \bigg{)}\bigg{]} \tag{14}\]  
where \(Z_{\theta}(s_{t})\) is denoted as the normalized distribution of the Q-function. Considering that the strategy network outputs \(a\) distribution of actions from which action a is sampled, it is not possible to derive a derivation for it, so the action is sampled using a reparameterization technique.