# A Reinforcement Learning-Based Congestion Control Approach for V2V Communication in VANET
## 3 An RL-Based Framework of Congestion and Awareness Control  
In the field of VANET, congestion control is a vital challenge for ensuring the safety of communication over the limited bandwidth of the wireless channel. The objective of \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Algorithm** & \begin{tabular}{c} **Control** \\ **Parameter** \\ \end{tabular} & \begin{tabular}{c} **Rule-Based/** \\ **Learning-Based** \\ \end{tabular} & \
\begin{tabular}{c} **Metrics** \\ **Used** \\ \end{tabular} &
\begin{tabular}{c} **Message** \\ **Type** \\ \end{tabular} \\ \hline LIMERIC [15] & Tx. Rate & Rule-based & CBR & Beacon \\ \hline EMBARC [16] & Tx. Rate & Rule-based & Tracking error & Beacon \\ \hline D-FPAV [21] & Tx. Power & Rule-based & Probability of message reception & Beacon, Event messages \\ \hline SPAV [31] & Tx. Power & Rule-based & Beaconing load & Beacon \\ \hline SBAPC [32] & Tx. Power & Rule-based & Beacon error rate, CBR, IPD & Beacon \\ \hline DR-DCC [25] & Data Rate & Rule-based & PDR & Beacon \\ \hline CACC [30] & Hybrid & Rule-based & PDR, PCR, CBR & Beacon \\ \hline BH-MAC [33] & Hybrid & Rule-based & RAC & Beacon \\ \hline CPMRA-CDPRA [29] & Hybrid & Rule-based & CBR & Beacon \\ \hline ML-CC [34] & Tx. Rate & Learning-based & PLR, Average Delay & Beacon \\ \hline MDPRP [28] & Hybrid & Learning-based & CBR, PDR & Beacon \\ \hline SSFA [20] & Tx. Rate & Learning-based & CBR,PDR & Beacon \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of algorithms. congestion control is to alleviate channel congestion by reducing the number of transmitted safety messages. This can be achieved by increasing the time interval between transmissions. However, reducing the frequency of transmissions leads to a reduction in awareness, causing a decrease in the visibility of other vehicles and vice versa. This in turn can cause congestion to arise again when the density of vehicles increases. The vehicles must therefore balance between prioritizing congestion control and awareness control, which becomes particularly challenging in dynamic mobility environments. Making the correct decision in different scenarios is crucial for successful congestion control. For instance, when the vehicle density is low, it may be beneficial to increase the transmission power to achieve a larger transmission range, while still maintaining an acceptable channel load by adjusting the transmission rate. When the vehicle density is high, the transmission power and rate must be adjusted accordingly. This decision-making problem is influenced by various factors, including vehicle density, channel congestion, and packet delay, making it challenging to find an optimal combination of transmission parameters using traditional methods, especially since some constraints may conflict with each other. It is important to note that the decision-making process in VANET must take into consideration the current situation only. This is due to the Markov property, which states that the future is independent of the past given the present. This means that the vehicle's decision is based on the current state of the traffic flow and channel, rather than prior conditions. The Markov property can be formalized as:  
\[P[S_{t+1}|S_{t}]=P[S_{t+1}|S_{1},\ldots,S_{t}] \tag{1}\]  
Equation (1) shows that the state at the next moment, \(t+1\), is solely dependent on the state of the current moment, \(t\). Hence, we can model the problem with an MDP. RL is a good framework to find solutions to an MDP [13]. The main learning principle of a typical RL cycle in V2V communication can be described in Figure 1.  
Here, a vehicle and environment interact at each of a sequence of discrete time steps, \(t=0,1,2,3,\ldots\). At each time step \(t\), the vehicle receives some representation of the environment's state, \(S_{t}\in\mathcal{S}\), and based on the representation, it selects an action, \(A_{t}\in\mathcal{A}(s)\). After one time step, the vehicle will receive a numerical reward, \(R_{t+1}\in\mathcal{R}\), and it moves to a new state, \(S_{t+1}\).