# A Reinforcement Learning-Based Congestion Control Approach for V2V Communication in VANET
## 3 An RL-Based Framework of Congestion and Awareness Control
### Elements of the RL Framework for Congestion and Awareness Control  
The primary objective of this paper is to demonstrate the application of RL in selecting appropriate transmission parameters for V2V congestion control. The selection process is modeled as a MDP. This paper will cover the design of the elements of the MDP, considering the following factors:  
* Finite state and action space: In the VANET application layer, it is assumed that each vehicle can only choose from a finite set of actions at each state. As the state space is finite, Q-learning can be used to solve the problem. * Determination of neighboring vehicles: The number of neighboring vehicles is determined based on the interactions with the environment, as indicated by the BSMs received from these vehicles. * Experimental observations: All observations are calculated through experimental actions taken by the vehicle. * Independent decision-making: Each vehicle independently selects its actions based on its own observations, without exchanging any information with other vehicles except for BSMs. The decision-making problem is formalized using the framework of MDP. The following are the key elements of the RL framework used to solve the MDP for V2V congestion control:* The agent: A learning agent must have the capability to perceive the state of its environment and take actions that can alter it. In the context of this problem, the vehicle acts as the agent that makes the decision of which action to take [13]. * The goal: The objective is to select the optimal action for each state, with the aim of maximizing the reward. A well-defined goal is crucial for the agent (vehicle), such as reducing congestion or enhancing awareness. In this paper, the goal is to maximize the reward of actions that maintain the CBR below 0.6. * The environment: The environment represents the uncertain world in which the agent operates and interacts. The agent can interact with the environment and modify it through its actions, but it cannot change the rules or dynamics of the environment. In the context of VANETs, the environment encompasses the wireless channel and other vehicles. The uncertainty arises from factors such as dynamic traffic flow, such as changing vehicle velocity and density. The actions of the vehicles can impact the state of the environment (wireless channel status), but will not affect the density of vehicles on the road. * The action: Actions are the means by which the agent interacts with and influences its environment. In the VANET application layer, the most common actions include setting the transmission power, message transmission rate, or data rate of the messages to be transmitted. In this paper, for simplicity, we only consider the transmission rate. The maximum transmission rate in DSRC is 10 Hz. The action space is defined as 10 discrete transmission rates, ranging from 1 BSM per second to 10 BSMs per second, where \(a\in\mathbb{N},1\leq a\leq 10\), for each action \(a\). * The state (observation): The state of the environment in the V2V communication problem is a collection of information that identifies the current situation. This information includes the wireless channel status, such as CBR, BER, IPD, etc. Moreover, the vehicle density, or the number of neighbors of a given vehicle, which represents the dynamics of the environment, should be considered. Even with the same action, the state could be different when the vehicle density is different. In our case, we define the space as a 2-tuple including the CBR and vehicle density, denoted as \(s=(CBR,VD)\), \(CBR\in\mathbb{R}^{+}\), \(0\leq CBR\leq 1\), and \(VD\in\mathbb{N}\), \(1\leq VD\leq maxVD\). The CBR is a real number between 0 and 1 that represents the channel busy ratio and the vehicle density is the number of vehicles within a 100 m radius, and \(maxVD\) is the maximum vehicle density in this range. In this paper, we set \(maxVD\) to 50. For each vehicle density, there are 10 CBR values corresponding to the 10 transmission rates.