## 2 Related Works
### Reinforcement Learning  
Reinforcement learning stands out as a highly effective approach to addressing the decision-making challenges in autonomous driving. Its objective is to identify the optimal policy that maximizes cumulative rewards. Through interaction with the environment, the intelligent agent independently explores, executes actions based on its policy to impact the environment, receives feedback in the form of rewards, and gathers data samples for learning. Through continuous learning, the agent acquires the optimal strategy. The reinforcement learning process is formulated through the Markov Decision Process (MDP) [15], with its fundamental concept illustrated in Figure 1. During each discrete time step t (where t = 1, 2, 3, \(\ldots\)), the agent engages with the environment. At these intervals, the agent acquires a depiction of the environmental state \(S_{t}\in\) S, where S denotes the collection of all potential states, and chooses an action from \(A_{t}\in\) A (\(S_{t}\)), where A (\(S_{t}\)) represents the set of feasible actions in state \(S_{t}\). In the reinforcement learning paradigm, agents iteratively transition through state-action-reward sequences: executing action \(A_{t}\) in state \(S_{t}\) yields immediate reward \(Rt+1\in R\) and subsequent state \(St+1\). This progression forms episodic trajectories (\(S1\), \(A1\), \(S1\), \(R1\), \(\cdots\), \(St-1\), \(At-1\), \(St\), \(Rt-1\)), terminating at \(St\). The cumulative discounted reward \(Rt\) Equation (1) quantifies long-term performance by exponentially weighting future rewards:  
\[R_{t}=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}R_{t^{\prime}-t} \tag{1}\]  
The discount factor adjusts the weight distribution of future rewards through an exponential decay mechanism, giving higher priority to immediate rewards. The core objective of reinforcement learning is to find the optimal policy \(\pi\) that maximizes the cumulative discounted reward \(\mathrm{Rt}\). Based on this reward mechanism, the state value function \(V^{\pi}\) and the action value function \(Q^{\pi}\) are constructed, as shown in Equations (2) and (3). The former evaluates the long-term return of a specific state, while the latter measures the expected return of a state-action pair, as shown in Equation (3). As expressed in Equation (4), the state value function is recursively represented using the Bellman equation, and the optimal control policy \(\pi\ast\) is determined by maximizing the state value function, as mathematically formulated in Equation (5). \[V^{\pi}(s_{t})=E_{\pi}[R_{t}|s_{t},\pi] \tag{2}\]  
\[Q^{\pi}(s_{t},a_{t})=E_{\pi}[R_{t}|s_{t},a_{t},\pi] \tag{3}\]  
\[Q^{\pi}(s_{t},a_{t})=E_{\pi}[r_{t}+\gamma\underset{a_{t+1}}{\max}Q^{\pi}(s_{t+ 1},a_{t+1})] \tag{4}\]  
\[\pi(s_{t})=\underset{a_{t}}{\max}Q(s_{t},a_{t}) \tag{5}\]  
Q-learning [7] is an algorithm that employs value iteration updates to directly approximate the optimal action value function \(Q^{\ast}\). The Q-learning algorithm uses the update rule in Equation (6) to revise the projected value of the current action value function:  
\[Q^{\ast}(s,a)\gets Q(s,a)+\alpha(r+\gamma\underset{a^{\prime}}{\max}Q(s^{ \prime},a^{\prime})-Q(s,a)) \tag{6}\]  
The anticipated reward for an agent when executing action \(a\) in state \(s\) is denoted as \(Q(s,a)\). The instantaneous reward for choosing action \(a\) in state \(s\) is represented by \(r\). The learning rate is denoted by \(\alpha\), while \(\underset{a^{\prime}}{\max}Q(s^{\prime},a^{\prime})\) signifies the anticipated reward for selecting alternative actions \(a^{\prime}\) in the same state \(s^{\prime}\). \(Q(s,a)\) using Equation (6). The process will eventually converge to \(Q^{\ast}(s,a)\), representing the optimal action value function [16]. This iterative method for determining the optimal action value function is known as Q-learning. To improve its generalization ability and expand its applicability, Q-learning can be integrated with deep neural networks, forming what is known as a deep Q-network.