## 4 Our DRL-JCBRTP Framework
### DRL-JCBRTP NN Architecture Design Based on SAC Algorithm  
RL involves optimizing actions for an agent to maximize cumulative discounted rewards, typically modeled by MDPs [28]. In this context, the policy \(\pi(a|s)\), value function \(V_{\pi}(s)\), and action-value function \(Q_{\pi}(s,a)\) are central concepts, with the goal being to learn an optimal policy \(\pi^{*}\) that maximizes expected return. RL categorizes control strategies into model-based and model-free mechanisms [28]. Model-based control requires agents to learn a model of the environment's dynamics and rewards for action planning, offering efficiency in terms of samples but depending on the model's accuracy. Conversely, model-free control bypasses the need to model environmental dynamics by learning directly from experience. This framework is further divided into on-policy and off-policy methods based on their policy update and evaluation mechanisms [28]: on-policy methods adjust policies during their execution, while off-policy methods aim to identify an optimal policy through exploration with a distinct policy. DRL extends RL using deep learning techniques, which are particularly useful in high-dimensional environments [29], and leverages deep NNs (DNNs) for function approximation, enabling direct learning from raw sensory data and the ability to handle extensive state spaces. Considering the environment model is not sufficiently defined, our DRL-JCBRTP framework adopts a model-free, off-policy SAC algorithm that balances exploration and exploitation via entropy maximization [21]. SAC uses actor and critic NNs for policy and value function approximation, respectively. Figure 4: Architecture of our beaconing agent in an MDP framework. **Actor NN:** The policy objective in the reparameterized form within the actor NN is  
\[J_{\pi}(\phi)=\mathbf{E}_{s_{t}\sim\mathcal{D},e_{t}\sim\mathcal{N}}[\alpha\log \pi_{\phi}(f_{\phi}(\epsilon_{t};s_{t})|s_{t})-Q_{\theta}(s_{t},f_{\phi}( \epsilon_{t};s_{t}))], \tag{1}\]  
where the expectation of state \(s_{t}\) is sampled from the replay buffer \(\mathcal{D}\), \(\alpha\) is the temperature parameter to balance between exploration and exploitation, and \(f_{\phi}(\epsilon_{t};s_{t})\) is the action from NN given sampled noise \(\epsilon_{t}\) from a normal distribution \(\mathcal{N}\) and state \(s_{t}\). The first term in the expectation represents the entropy of the action under the current policy \(\pi_{\phi}\), and the second term is the expected return associated with the chosen state-action pair. In the existing SAC algorithms, the actor NN typically uses a multi-layer perceptron (MLP) to generate actions at each time step, selecting the optimal action within the given action space [21; 29]. This approach is effective in environments where actions are determined based on immediate observations. However, in our beaconing problems, where the goal is to control the BR, the challenge extends to predicting and adjusting beacon intervals over time, requiring the modeling of temporal dependencies. Since BRC is fundamentally a time-series prediction problem, this paper deviates from the traditional use of MLPs and employs recurrent NNs (RNNs) to generate actions. Specifically, Long Short-Term Memory (LSTM) networks are introduced within the actor NN to capture and leverage sequential patterns in beaconing data. The RNN-based actor NN is designed to predict future beacon intervals by considering the temporal context provided by previous actions and states, enabling more accurate and adaptive control in dynamic environments. By integrating RNNs into the SAC framework, the proposed method directly addresses the time-dependent nature of the beaconing task, significantly enhancing the actor NN's ability to manage sequential data and ultimately improving the efficiency and safety of V2X communications.