## 3 Methodology
### Reinforcement Learning Network  
This study uses reinforcement learning as an overall framework for implementing autonomous driving. It is a machine learning scheme that learns in interaction, similar to the process of human learning knowledge. The agent continuously learns knowledge based on the rewards or penalties it receives in its interaction with the environment in order to explore the correct behaviors adapted to the environment on its own. In this research, the agent is the experimental vehicle that explores unfamiliar environments through trial and error based on the established reward-penalty mechanism. Through continuous feedback, it self-corrects its behavior and ultimately accomplishes the intended autonomous driving tasks in the environment. Reinforcement learning is based on the Markov Decision Process (MDP) [27]. This process mainly contains the following elements: \(S\), \(A\), \(P\), \(R\), \(\gamma\). \"\(S\)\" is the state of the vehicle in the environment. \"\(A\)\" represents all the actions of the vehicle. \"\(P\)\" represents the probability of the vehicle transitioning from state \(s\) at time \(t\) to state \(s\)' at time \(t+1\) after executing action \(a\). A state corresponds to an action or the probability of an action. And with an action, the next state is determined. This means that each state can be described by a definite value. From this, it can be determined whether a state is good or bad. For example, if the self-driving car to the left hits an obstacle, then the state to the left is a bad state. Thus, the goodness of a state is equivalent to the expectation of a future return, and \"\(R\)\" represents the return that the state at a given time will have. \"\(R\)\" (reward) is a real value that represents reward or punishment. A positive number is returned when the vehicle performs the expected action, while a negative number is returned when the vehicle performs the wrong action. \"\(\gamma\)\" is discount factor that determines the importance of future rewards relative to immediate rewards. The purpose of reinforcement learning is to maximize long-term future rewards, which can also be expressed as finding the largest return _U. \"U\"_ represents the sum of rewards accumulated in all states after executing a set of actions, but the direct addition of rewards in an infinite time sequence will result in unbiased and infinite loops of states. Therefore, the concept of discount \(\gamma\), with a value range of zero to one, is introduced into this function, and the reward returned from the subsequent state is multiplied by the discount coefficient. This means that the current reward is more important than the reward of future feedback. The definition formula is as follows [28]:  
\[ \begin{array}{l}U(s_{0}s_{1}s_{2}\cdots)=\sum\limits_{t=0}^{\infty}\gamma^{ t}R(s_{t}),\ \ \ 0\leq\gamma<1\\ \leq\sum\limits_{t=0}^{\infty}\gamma^{t}R_{max}=\frac{R_{max}}{1-\gamma}\end{array}  \tag{1}\]  
The MDP is a cyclic process in which the agent takes actions to change its state in order to obtain rewards and interact with the environment. The strategy of the MDP depends entirely on the current transaction, which is a reflection of its Markovian nature. Based on the above MDP, this research implements a DDPG network to achieve autonomous driving tasks. This choice is motivated by the continuous nature of steering and throttle actions during vehicle operation. DDPG combines the advantages of deep learning and Q-learning, making it particularly suitable for handling continuous action spaces. DDPG, proposed by [PERSON] et al. in 2015, is an actor-critic algorithm that operates through the collaboration of two networks [7]. The actor network determines the optimal policy, while the critic network evaluates the value of the current policy. Its key characteristics include off-policy training and the use of deep neural networks to handle complex, high-dimensional state and action spaces.