## 3 Proposed Method  
In this section, we review the preliminaries of DQN and formulate the routing control problem by controlling the driving direction at intersections as a Markov decision process (MDP). Then, we present the proposed method featuring an efficient reward function design to solve the formulated MDP. The routing control problem can be defined as an MDP, which comprises a set of states \(S\), a set of actions \(A\), and an immediate reward \(r\). Given the current state \(s\in S\), the agent selects an action \(a\in A\) to maximize the long-term cumulative reward \(R\). Consequently, the agent aims to make optimal decisions in routing control. To address this problem, we utilized DQN, which is a value function-based DRL method that can effectively solve discrete action space problems. The network is trained by minimizing the loss function, which is denoted as \(L(\theta)\), as follows:  
\[L(\theta)=\mathbb{E}\Big{[}(y-Q(s,a|\theta))^{2}\Big{]}, \tag{1}\]  
where \(y=r+\gamma\)\(\underset{a^{\prime}}{\text{max}}\)\(Q(s^{\prime},a^{\prime}|\theta^{\prime})\) is the target Q-value, and \(Q(s,\ a|\theta)\) is the predicted Q-value. The DQN network learns by minimizing the mean squared error (MSE) between the target Q-value and predicted Q-value, as well as by periodically updating \(\theta^{\prime}\) to \(\theta\). The Q-function equation for DQN is defined as \(Q(s,a)=r+\gamma maxQ(s^{\prime},a^{\prime})\), where \(\gamma\) represents the discount factor when adjusting the value of future rewards. DQN has a high correlation because it collects the data sequentially over time in an environment. Therefore, to solve the problem, experience replay is applied, which is a method of storing the experience and configuring a mini batch to randomly select the experience to learn. This can reduce the high correlation between the data and increase the data reliability. The proposed method recognizes the traffic situation and determines the direction to optimize the route based on the traffic situation. Traffic environments are dynamic and complex, with traffic volumes changing in real time. Consequently, the routes of vehicles are constantly changing and difficult to predict. This is especially the case in mixed traffic environments, the continuous variation due to various factors leads to a high likelihood of many possible states occurring. To address this, we utilized DQN, which is capable of learning complex patterns and interactions in state and action spaces by using neural networks. Additionally, DQN utilizes experience replay to efficiently utilize past experiences in training, thereby enabling the effective utilization of training data and ensuring stable learning even in high-dimensional state and action spaces. Furthermore, DQN is well suited for discrete action spaces, thus making it suitable for determining discrete actions such as driving directions. The proposed method updates the routes to alleviate traffic congestion by distributing the driving directions based on real-time local information at each intersection. Each AV is considered a DRL agent and is defined by \(\,v\in\{v_{1},v_{2},\ldots,v_{n}\}\), which allows the AVs to interact with the environment and learn the policies to reach their destination. The goal of each AV agent is to drive in a direction that changes toward the destination as quickly and optimally as possible to achieve higher rewards. As shown in Figure 1, a decision zone with a specified distance was defined on the approach roads of each intersection. Within these zones, the driving direction of the AV at the intersection was determined. Therefore, when an AV is in the decision zone, it derives the observed state \(s_{n}\) from the current traffic situation to form state space \(\,S\). State \(s_{n}\) is entered into AV agent \(v_{n}\), which represents the current traffic observation. Subsequently, AV agent \(v_{n}\) performs action \(a_{n}\) based on the current state \(\,s_{n}\). Thereafter, the AV agent receives the reward \(r_{n}\) from the traffic environment.