# A Reinforcement Learning-Based Congestion Control Approach for V2V Communication in VANET
## 3 An RL-Based Framework of Congestion and Awareness Control
### A Q-Learning Approach
We set a maximum return value of 0.92 to prevent densities greater than the tested range, yielding values greater than 1 since the change in CBR was found to be negligible at high densities. The Q-learning algorithm was then run with the observation data and Equation (7) below to generate the Q-table for each combination of vehicle density and transmission rate for the whole state space. In the following equation, \(VD\) is the current vehicle density, \(BR\) is the estimated average transmission rate used by surrounding vehicles, and \(estCBR\) is the estimated CBR when there are \(VD\) neighboring vehicles that use an average transmission rate of \(BR\) BSMs. \[estCBR(VD,BR)=\begin{cases}0.0101 VD+0.0301&if\;BR=1\\ \begin{cases}0.0189 VD+0.0703&if\;VD\leq 33\\ 0.2730\ln(VD)-0.2526&otherwise\\ 0.0249 VD+0.1194&if\;VD\leq 27\\ 0.1663\ln(VD)+0.2487&otherwise\\ 0.0314 VD+0.1500&if\;VD\leq 21\\ 0.0988\ln(VD)+0.5318&otherwise\\ 0.0379 VD+0.1818&if\;VD\leq 17\\ 0.0884\ln(VD)+0.5843&otherwise\\ 0.0686 VD+0.1425&if\;VD\leq 10\\ 0.0819\ln(VD)+0.6817&otherwise\\ 0.0772 VD+0.1688&if\;VD\leq 8\\ 0.0659\ln(VD)+0.6940&otherwise\\ 0.0843 VD+0.1972&if\;VD\leq 7\\ 0.0442\ln(VD)+0.7760&otherwise\\ 0.0891 VD+0.2289&if\;VD\leq 7\\ 0.0304\ln(VD)+0.8246&otherwise\\ 0.0930 VD+0.2602&if\;VD\leq 6\\ 0.0151\ln(VD)+0.8736&otherwise\\ \end{cases} \quad if\;BR=7\)  
By using this equation to predict the CBR of every combination of vehicle density and transmission rate, the Q-learning algorithm is able to find which combinations yield the best results and can generate the corresponding actions as a result. Each row in the Q-table corresponds to a combination of a vehicle density (ranging from 0 to 50) and an estimated average transmission rate used by the neighboring vehicles (an integer from 1 to 10), while each column represents a transmission rate that can be chosen by the current vehicle. In Algorithm 1, the first two steps aim to define the state and action space. Step 3 initializes the Q-table and sets each of its values to 0. Since there is no \"terminating state\", we set the number of episodes to be 80,000 in this paper. During each episode, the vehicle executes steps 4 to 13 to update the Q-table. The algorithm employs a combination of an optimal policy and a random action (with a probability of 0.1) to search for better policies, which is the exploration strategy in RL [13]. The discount factor \(\gamma\) and the learning rate \(\alpha\) are set to be 0.9 and 0.01, respectively. After 80,000 iterations, the difference between the new and old Q-table is negligible, indicating that the algorithm has converged. The final Q-table will be saved into a file for use in Stage 2. As noted in [37], the complexity of Algorithm 1 is \(O(n)\) in the general case with duplicate actions. ```
1:\(curCBR\) = Obtain current CBR
2:\(curVD\) = Obtain current vehicle density
3:if\(curVD>maxVD\)then
4:\(curVD=maxVD\)
5:endif
6:\(maxVal=-999\)
7:\(index=9\) (default value to ensure there is always a valid output)
8:for\(i=0\) to 9 do
9:if\(estCBR(curVD,i+1)\) using Equation (7) \(\geq curCBR\)then
10:\(index=i\)
11: break
12:endif
13:endfor
14:for\(i=0\) to 9 do
15:\(qVal=\) Obtain the entry at index \(i\) of the row in the Q-table corresponding to \(curVD\) and \(index\)
16:if\(qVal>maxVal\)then
17:\(maxVal=qVal\)
18:\(bestBeaconRate=i+1\)
19:endif
20:endfor
21:\(bestBeaconInterval=1/bestBeaconRate\)
22: Send beacon using \(bestBeaconInterval\)
```  
**Algorithm 2** Policy Application of QBACC in OMNeT++  
Once an optimal policy has been determined in Stage 1, a vehicle can apply this policy to select the BSM transmission rate in Stage 2 (Algorithm 2). In steps 1 and 2, a vehicle detects its environment in terms of CBR and vehicle density. If the measured vehicle density is found to be higher than \(maxVD\), it is set to \(maxVD\) (steps 3 and 4). In steps 6 and 7, the values of \(maxVal\) and \(index\) are initialized. \(maxVal\) is set to a very small number to ensure that the algorithm will come across a higher value in the Q-table.