# A Reinforcement Learning-Based Congestion Control Approach for V2V Communication in VANET
## 3 An RL-Based Framework of Congestion and Awareness Control
### A Q-Learning Approach  
Traditional MDP models require knowledge of state-transition probabilities, which can be challenging to obtain in the context of V2V communication. As an alternative, Q-learning is a model-free algorithm in RL that allows an agent to select and perform actions without relying on a prior understanding of the state-transition probabilities. Instead, the agent learns an optimal policy by directly interacting with the environment. The proposed Q-learning-based congestion control algorithm is implemented in two stages as follows:  
* Stage 1: The Q-learning algorithm is implemented using observation data obtained from a simulation. The algorithm generates a Q-table that represents the optimal policies at each state, as demonstrated in Algorithm 1. * Stage 2: The Q-table generated in the first stage is utilized by the vehicle to determine the interval before the next BSM transmission takes place. This stage is explained in detail in Algorithm 2. Algorithm 1 provides an overview of our proposed Q-learning-based adaptive congestion control (QBACC) approach. The algorithm starts by initializing all values in the Q-table, which contains all the action-state pairs, to 0. Then at each time step \(t\), the vehicle selects an action \(a_{t}\), observes the environment and receives a reward \(r_{t}\), transitions to a new state \(s_{t+1}\), and updates the value of \(Q(s,a)\) with Equation (6):  
\[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha(r(s_{t},a_{t})+\gamma\max_{a}Q(s _{t+1},a)-Q(s_{t},a_{t})) \tag{6}\]  
In Equation (6), \(\alpha\) is the learning rate, where \(0<\alpha\leq 1\), and \(\gamma\) is the discount rate, where \(0<\gamma\leq 1\). \(Q(s_{t},a_{t})\) is the current value of \(Q(s,a)\), \(\max_{a}Q(s_{t+1},a)\) is the estimate of optimal future value of \(Q(s,a)\) and \(r(s_{t},a_{t})\) is the reward when agent takes action \(a\) in state \(s\) at time step \(t\). Figure 2 lists the flowchart of Algorithm 1. ```
0: step size \(\alpha\in(0,1]\), small \(\epsilon>0\), number of episodes
0: Q-table with values of each state-action pair
1: Initialize \(\mathcal{S}\), the set of states (which contains one state for each beacon rate)
2: Set \(\mathcal{A}(s)\) to be the set of actions that can be taken in state \(s\), which consists of the ten possible beacon rates regardless of the value of \(s\). 3: Initialize the Q-table \(Q\), where \(Q(s,a)=0\) for all \(s\in\mathcal{S},a\in\mathcal{A}(s)\)
4:for each episode do
5: Set \(s\) to be a random state in \(\mathcal{S}\)
6:for\(i=1\) to \(10\)do
7: Choose action \(a\) from \(\mathcal{A}(s)\) using the \(\epsilon\)-greedy algorithm (random chance of choosing the action with the highest value in the Q-table so far; otherwise choose a random action)
8: Compute the reward using Equation (5), where CBR is calculated using Equation (7), with the vehicle density as input
9: Update \(Q(s,a)\) using Equation (6) and the obtained reward
10: Take action \(a\) and move to the corresponding state
11: Set \(s\) to be the new state
12:endfor
13:endfor
```  
**Algorithm 1** Q-Learning-based Adaptive Congestion Control (QBACC)Figure 2: Flow Chart of QBACC. To generate the Q-table, in Algorithm 1, we created 12 different traffic models with different vehicle densities, varying from 0 to \(maxVD\) vehicles in a 100 m radius, where we set \(maxVD=50\). Simulations were run for each traffic model with our designed action space, i.e., using a transmission rate ranging from 1 to 10 BSMs per second, to get the observation data (CBR) for each action at each state. These data were then used to generate functions for curves of best fit, which represent the correlation between the average beacon rate used by vehicles and the average CBR experienced by the network. These functions were used to create Equation (7) to estimate the CBR for each transmission rate and density combination based on the known trends.