## 2 Research Methodology
### Learning with DRL  
RL is based on algorithms for learning through experiences between agents and the environment without prior information. In RL, agents employed in an unknown environment interact with their environment and take appropriate actions to maximize their performance. Based on selected actions, the scalar reward (positive and negative) is obtained, and the agent continues to learn until it reaches the highest performance. The RL model can be expressed by a four-tuple (S, A, R, T), in which S, A, R, and T are the possible state space, action space, reward space, and the transition function of the model. At each time t, the agent selects an action a\({}_{t}\), takes a reward r\({}_{t}\), and moves from a state s\({}_{t}\) to a new state s\({}_{t+1}\). The core of the algorithm is a Bellman equation, using the weighted average of the old and the new information value. \[Q^{new}\left(s_{t},\ a_{t}\right)\ =\ Q^{old}\left(s_{t},\ a_{t}\right)\ +\ \alpha\ \left(\ r_{t}+\gamma\ Q^{max}\left(s_{t+1},\ a_{a}\right)-Q^{old}\left(s_{t}, \ a_{t}\right)\ \right) \tag{5}\]  
Here,  
* \(Q^{new}\left(s_{t},\ a_{t}\right)\), \(Q^{old}\left(s_{t},\ a_{t}\right)\), \(Q^{max}\left(s_{t},\ a_{t}\right)\) are the updated, old and optimum Q network value. * \(\alpha,\ \gamma\) are the learning rate and discount factor of the training network. In this paper, we used the DRL-based approach to optimize traffic signals. Based on the data collected from the network system, the trained agent minimizes the waiting time of all vehicles to optimize the signal lights. The agent receives the current state of the network, the action of traffic lights, and the reward of the recent action while training. These data (s\({}_{t}\), a\({}_{t}\), cr\({}_{t,t+1}\)) are stored in the experience replay and used while training. \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Value** \\ \hline Platoon size & 1, 2, and 3 vehicles \\ \hline Vehicle length & 4 m \\ \hline Initial speed & 5 m/s \\ \hline Max\_speed & 20 m/s \\ \hline Max\_acceleration & 2.5 m/s\({}^{2}\) \\ \hline Max\_deceleration & 3 m/s\({}^{2}\) \\ \hline Platoon gap & 15 m \\ \hline Catch\_up distance & 50 m \\ \hline Speed\_factor & 1.2 only for follower and catch-up modes. 1 for other modes \\ \hline tau (The desired minimum time headway) & 0.3 s for CVs and 1 s for RVs. \\ \hline minGap & 0.5 m for CVs and 2.5 m for RVs \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Attributes of platoons. The actions are selected based on different policies, where the agent can choose an action that does not have the highest Q value if it wishes to explore the environment. Alternatively, it can choose to exploit and choose an action with the highest Q value. The algorithm is expressed in Algorithm 1. ```
Input: neural network agent with random weights, replay memory size, minibatch size, epsilon (\(\epsilon\)), learning rate (\(\alpha\)), and discount factor (\(\gamma\)) Initialize replay buffer B in Memory M Initialize action-value function Q with \(\theta_{o}\) Initialize the action-value function \(Q_{n}\) with random parameters \(\theta_{n}\ =\ \theta_{o}\) while episode < Total Episodes: Episode = 1,... E do  Start Simulation with first step J, observe initials state \(s_{o}\) and action \(a_{o}\) For J = 1,... N do \(Action\ =\ \left\{ \begin{array}{ll}Arbitrary\ with\ probability\ \epsilon\\ \arg\max_{a}^{\prime}Q\ (s_{o},\ a_{o},\ \theta_{o}),\ \ 1-\epsilon \end{array} \right.\) Perform action \(a_{n}\) and observe the reward r, next state \(s_{n}\) Store experiences (\(s_{o},\ a_{o},\ r,\ s_{n}\)) in B. Sample random B experiences from M Calculate the loss L L = \(\left\{ \begin{array}{ll}0,&\text{ if terminated}\\ \text{r + max}_{a}\text{Q}_{n}\ (s_{n},\ a_{n},\ \theta_{n}),&\text{ otherwise}\end{array}