## 4 Our DRL-JCBRTP Framework
### MDP Specifications of Our DRL-JCBRTP  
MDPs provide a mathematical framework for sequential decision-making, where outcomes are influenced by the decision-maker's policy, making them highly suitable for the intricate beaconing dynamics. Utilizing this framework, an RL agent can effectively adjust the BR and TP to improve awareness and to maintain an appropriate channel load. The Markov property simplifies the decision-making process by associating future outcomes directly with the current state, eliminating reliance on past decisions. This framework is essential for developing strategies that adapt to present conditions, providing a cohesive approach to the challenges of beaconing. Our MDP formulation embodies this principle, emphasizing policy-driven decisions for the immediate scenario, with the specifics outlined as follows:Agent_, tasked with learning the optimal policy and executing actions, is represented by individual vehicles in the road environment. These vehicles observe their local state information to adjust their BR and TP accordingly, aiming to fulfill diverse objectives. This setup fosters a decentralized framework, where every vehicle independently makes a decision informed by its specific state information. Despite the decentralized nature of decision-making, we assume that the same underlying policy is used across all vehicles, ensuring a coherent and unified strategy for managing BR and TP on the road. 2. _Environment_ serves as the complex and dynamic setting in which the agent operates, providing critical feedback on the agent's actions and influencing its subsequent state. In addition, it plays a pivotal role in delivering essential state information and issuing rewards or penalties based on the agent's decisions. This feedback loop is essential for steering the agent toward optimal behaviors, enabling it to adeptly adjust to and manage the continuously changing road and traffic conditions. 3. _Actions_, allowing the agent to react to the environment, are designated as \(A_{t}=a\in\mathcal{A}\), where \(\mathcal{A}\) is the set of possible actions. At each frame, the agent evaluates its current state to decide on a beacon broadcast, thus affecting the BR and TP. These actions are structured as a pair \(a=(b,p)\) consisting of the decision to transmit a beacon (\(b\)) and the TP (\(p\)), where \(b=\{0,1\}\) indicates whether to broadcast a beacon frame (1) or not (0), and if \(b=1\), an appropriate TP \(p\) dBm is chosen to mitigate channel congestion. Notably, all these actions conform to access-layer standards, ensuring regulatory compliance [22; 24; 26; 27]. 4. The _state_ is a vector \(s\in S\) consisting of the internal parameters of the agent and its observation of the environment, where \(S\) represents the complete set of potential states. This state, forming the critical basis for action selection by the agent, is exclusively built from locally available information, which includes: * **Vehicle kinematics:** the speed (VS) and acceleration (VA) of the ego vehicle obtained from the vehicle kinematics estimator. * **CBR:** the ratio of channel occupancy at the previous frame obtained from the access radio. * **Number of neighbor vehicles (NNV):** number of distinct vehicles whose beacons are successfully received in the last 20 frames. * **Average Beacon Interval (ABI):** average beacon interval between successively received beacons over all neighbor vehicles, quantified in the unit of frame duration \(T\). Then, the state of the agent is represented by a 5-tuple vector: \[s=(VS,VA,CBR,NNV,ABI)\]
5. _Rewards_ play a pivotal role in guiding the learning process of the agent. Upon executing an action \(a\) at time \(t\), the agent transits from its current state \(s\) to a new state \(s^{\prime}\), during which it receives a reward \(R_{t+1}\). This reward, a critical form of feedback from the environment, is formulated as a function of both the state \(s\) and the action \(a\).