## 4 Our DRL-JCBRTP Framework
### Design Principle of Reward Functions  
In DRL, the choice between sparse and dense rewards is critical in determining how effectively an agent learns from its environment [30; 31]. Sparse rewards are given only when a specific goal or milestone is reached, making them suitable for tasks where the path to the goal is less important than the outcome. However, this approach can lead to slow learning, as the agent receives minimal feedback on its actions. Conversely, dense rewards provide continuous feedback at each time step, guiding the agent with incremental signals that can significantly accelerate learning, especially in tasks requiring ongoing adjustments. In the context of vehicular beaconing, dense rewards are particularly well-suited because of the need for continuous vehicle awareness to prevent vehicle collisions. Beaconing involves real-time detection and tracking of vehicles, where each decision can immediately impact safety and communication efficiency. By using a dense reward structure, the agent is encouraged to make ongoing improvements in maintaining awareness, optimizing detection and tracking, and managing channel congestion. This continuous feedback loop is crucial in dynamic traffic environments, ensuring that the agent adapts its behavior to minimize risks and maintain optimal performance. The design of our dense reward function is intricately crafted to address the core challenges in beaconing: awareness in terms of vehicle detection, vehicle tracking, and their minimum failure distances, timeliness, and channel congestion. This function serves as the cornerstone of our DRL framework, meticulously balancing the trade-offs between these objectives to guide the decision-making process of the agent. Our reward structure aims to foster a harmonious integration of these critical aspects by appropriately setting the weight assigned to each reward component. This thoughtful design not only meets the operational requirements of beaconing but also promotes the efficient and sustainable use of resources. Our reward function consists of five components, i.e., the mDFD, mTFD, MTE, beacon interval, and congestion costs:  
\[R_{t+1}=-mdfdCost(d)-mtfdCost(d)-mteCost-biCost-cgCost, \tag{3}\]  
where each reward component will be addressed as follows:  
* **mDFD cost \(mdfdCost(d)\)**: The mDFD cost is the penalty applied to the distance \(d\) to the nearest undetected vehicle. Since the ego vehicle is unaware of this undetected vehicle, a short mDFD no greater than the detection distance threshold \(\theta_{D}\) poses the most significant safety risk. To reflect this severity, the highest penalty value should be set to its weight \(\omega_{DF}\) in (4). Since the risk of V2V collision quickly decreases with the increase in distance between the colliding vehicles, we exponentially decrease the \(mdfdCost(d)\) if the distance exceeds the threshold \(\theta_{D}\), as follows: \[mdfdCost(d)= \begin{cases}\omega_{DF},&0\leq d\leq\theta_{D};\\ \omega_{DF}\cdot\exp\biggl{(}-\dfrac{d-\theta_{D}}{\eta\cdot\theta_{D}} \biggr{)},&\theta_{D}<d\leq D_{T},\end{cases} \] (4) where parameter \(\eta\) determines the exponential time constant, which controls how quickly the cost decreases as the distance to the colliding vehicle increases. * **mTFD cost \(mtfdCost(d)\)**: The mTFD cost is the penalty imposed on the distance \(d\) to the nearest untracked vehicle. Although this vehicle has already been detected, a short mTFD less than the tracking distance threshold \(\theta_{T}\) potentially yields a V2V collision. Similar to the mDFD, if the distance is no greater than \(\theta_{T}\), a relatively high constant weight \(\omega_{TF}\) is allocated to the mTFD cost; otherwise, it decreases exponentially with the relative distance, as follows: \[mtfdCost(d)= \begin{cases}\omega_{TF},&0\leq d\leq\theta_{T};\\ \omega_{TF}\cdot\exp\biggl{(}-\dfrac{d-\theta_{T}}{\theta_{T}}\biggr{)},&\theta _{T}<d\leq D_{T}.\end{cases}