## 5 Numerical Results
### Impacts of DRL-JCBRTP NN Architectures on Beaconing Performance
Similarly, Figures 12 and 13 reveal that two-layer LSTM maintains the lowest awareness failure probability (AFP) and mAFD, making it the most reliable model for situational awareness in dynamic vehicular environments, while the added complexity of three-layer LSTM does not significantly enhance tracking or awareness capabilities, which remain slightly inferior to those of the two-layer configuration. **Challenges in Learning Efficiency of DRL-JCBRTP**: The learning curves in Figure 14 provide a clear comparison of the average return for different NN architectures. Notably,  
Figure 11: mTFD vs. vehicle density. Figure 12: AFP vs. vehicle density. Figure 13: mAFD vs. vehicle density. two-layer LSTM shows a relatively stable and high average return, indicating robust learning and effective adaptation to the beaconing problem. Three-layer LSTM, while more complex, stabilizes at a lower return than two-layer LSTM, suggesting that the added complexity does not significantly enhance learning and may even introduce challenges in optimization. The MLP and one-layer LSTM models, on the other hand, exhibit considerable fluctuations and ultimately deteriorate in performance, particularly the MLP, which fails to maintain effective learning and collapses as training progresses. These results highlight the advantage of using a two-layer LSTM in capturing the temporal dependencies crucial for beaconing, where the simple MLP struggles due to its inability to effectively handle time-series data. In the context of V2X beaconing, unlike in more controlled environments such as robotics, the relationship between an agent's actions and the resulting rewards can be less direct due to the stochastic nature of the environment. First, even if the ego vehicle transmits a beacon, some neighbor vehicles may fail to receive it due to wireless channel impairments, leading to degraded detection and tracking performance. Second, the ego vehicle's contribution to the CBR is inversely proportional to the number of vehicles within its transmission range. This means that the impact of any single action on the overall network performance is diluted when there are many vehicles, making it harder for the agent to learn an effective action policy. These factors reduce the correlation between actions and rewards, complicating the training process and potentially leading to suboptimal policy learning. We hypothesize that these challenges suggest the loss function in the training process might be represented as a highly multi-modal nonlinear function in the weight space, making optimization particularly difficult. While we applied advanced techniques such as Xavier initialization to mitigate exploding and vanishing gradients, the ADAM optimizer for efficient convergence, and the Polyak averaging to stabilize the learning process, the optimality of the policies exhibits significant variability, indicating that the learned policies may not consistently converge to the true optimal policy. As expected, validating this hypothesis requires further research, as additional studies are needed to thoroughly investigate and confirm the underlying complexities of the objective function in such dynamic environments. Nonetheless, the use of LSTM networks, especially two-layer LSTM, proves to be more effective than the MLP, as evidenced by the superior performance in maintaining higher awareness levels, confirming the importance of considering temporal dependencies in such dynamic environments. Figure 14: Learning curve of DRL-JCBRTP.