## 4 Optimizing EE- and SE-Based Beam Training
### A Low-Complexity AB-HP
Action space: \(a_{n}=\{A_{1,n},\ \ldots\ldots\ldots,\ A_{N,n}\}^{T}\) is necessary to adjust all policies and to determine their improvement, thus they arrive at \(\mathcal{Q}^{\pi}(s,\omega)=\mathbb{E}\Big{[}y_{q}^{\pi}(s,\omega)\Big{]}\) and analogous distributional Bellman equation, that is  
\[y_{q}^{\pi}(s,a)\triangleq r(s,a)+\up y_{q}^{\pi}\big{(}s^{\prime},\omega^{ \prime}), \tag{7}\]  
where \(s^{\prime}\) and \(\omega^{\prime}\) are random nature of the next state-action pair after developing a policy, and \(A:\triangleq B\) pointed to a haphazard variable \(B\) probability law and \(A\) are similar. As a result, when the distributional Bellman operator's policy assessment behaves, \(T\) can be defined as  
\[\begin{split} T\,y_{q}^{\pi}(s,\omega):&\triangleq R (s,\omega)+\up y_{q}^{\pi}\Big{(}s^{\prime},\ \operatorname*{arg\!\max}_{\omega^{\prime}\in A}\mathbb{E}\Big{[}y_{q}^{\pi}(s ^{\prime},\omega^{\prime})\Big{]}\ \Big{)},\\ &\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quadfound in (9), and for SE, it has a similar structure. States (such as CSI and RF chain), actions (beam training), and rewards are all part of the DRL environment. To guarantee convergence, hyperparameters like the learning rate and the trade-off factor are carefully adjusted. Different system sizes, and channel conditions, are used to assess the method's scalability and real-world performance. To optimize long-term rewards, the agent is trained by gradient optimization to stabilize learning. The training process uses a target network with hyperparameters such as exploration rate, learning rate, and trade-off factor. The reward function is explicitly given, along with formulae designed to balance performance indicators, to guarantee reproducibility. The simulations yielded the penalty values. Thus, in the RBT-DRL-EE, the reward function is provided by  
\[R_{n}^{ee}=\alpha\delta E_{n}-\alpha(1-\delta)V_{n}-\lambda\left(\frac{M_{n}}{ M_{max}}+\frac{p_{n}}{p_{max}}\right)\,,\ \ 0\leq\delta\leq 1,\ \ n=1,2,\ldots \tag{9}\]  
where \(\delta\), also known as the trade-off factor, regulates the ratio of the achievable EE to the necessary beam training overhead. Training dynamics and convergence speed during DRL are influenced by \(\alpha\), which scales the reward function update according to the learning rate. The learning rate causes slower convergence but more stable learning, whereas a higher rate speeds up learning but can also contribute to instability. By controlling the relative value of long-term vs. immediate benefits, the discount factor \(\delta\) influences the agent's decision to prioritize lowering overhead over enhancing EE or SE. Optimizing EE or SE is favored by a greater \(\delta\), whilst cutting beam training time is given priority by a lower value. The design of the reward function is crucial to maintaining equilibrium between the training overhead and the achievable EE/SE. This function allows the agent to learn the best trade-off by incorporating penalties \(V_{n}\) to discourage excessive training time. The agent's capacity to adjust to different channel conditions and system sizes, as well as the framework's actual performance, are directly impacted by the sensitivity of these hyperparameters. \(E_{n}\) is the EE obtained using the \(n-th\) beam training method. The combined effect of the channel state, the quantity of RF chains employed, and the chosen beam training technique can be reflected in the EE or SE.