# AI-Assisted Multi-Operator RAN Sharing for Energy-Efficient Networks
## 5 Energy-Saving Features and AI-Assisted Radio Resource Management
### Network Optimization as an HRL Problem  
In this work, we implemented an AI system (Figure 15) consisting of four independent and hierarchically connected agents (A#) as seen in Figure 15. These agents were trained to optimize the energy efficiency of the wireless network under some throughput degradation constraints using the features described in Section 5.1. The environment (E) models the behavior of a wireless network and provides feedback to the agents regarding their actions through a reward system. During each episode, A1 initializes the environment with an input traffic demand which returns a feature vector corresponding to its current state. Based on this information, A1 selects an optimization strategy that it deems most beneficial for that particular state; this is then executed by lower-level agents A2, A3, and A4. Following the completion of their task, these agents receive rewards accordingly, depending on their effectiveness in improving the energy efficiency within the network. Subsequently, they update the weights of their value network using Bellman's Equation (18). \[Q^{(i+1)}(s,a)=R+\gamma\underset{s^{\prime}}{max}Q^{(i)}(s^{\prime},a^{\prime};\theta) \tag{18}\]  
A combination of three agents (A2, A3, and A4) perform the tasks of carrier adaptation and radio resource adaptation (RRA), respectively. Algorithm 2 presents the pseudocode of the implementation. Table 2 describes the five agents. \begin{table}
\begin{tabular}{p{113.8 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline \hline
**Agent** & **No. of Features** & **No. of Actions** & **Type of Agent** & **Description/Actions** \\ \hline A1: Strategy selector & 152 & 3 & Network-level & Identifies the optimal selection sequence of different optimization strategies for a given network scenario. Actions include calls to lower-level agents that perform user–cell association, carrier shutdown, or site-level RRA. & Identifies the optimal selection sequence of different optimization strategies for a given network scenario. Actions include calls to lower-level agents that perform user–cell association, carrier shutdown, or site-level RRA. & Identifies the optimal selection sequence of different optimization strategies for a given network scenario. Actions include calls to lower-level agents that perform user–cell association, carrier shutdown, or site-level RRA. & Identifies the sites where RRA could lead to energy savings without negatively impacting the user throughput. \\ A4: Site-level RRA or local configuration adapter & 22 & 13 & Site-level & Performs antenna and bandwidth adaptation at the site selected by A4 depending on the network scenario, the number of connected users, and other site-level metrics. & Performs antenna and bandwidth adaptation at the site selected by A4 depending on the network scenario, the number of connected users, and other site-level metrics. \\ \hline \hline \end{tabular}
\end{table} 
Table 2: RL Agent description. Figure 15: HRL implementation. Reinforcement Learning Algorithm  
Of the many reinforcement learning algorithms, we focused on a variant of traditional Q-learning using neural networks as function approximators. Traditionally, in DQNs, during the learning (or training) phase, a single Q-network is used to select and evaluate an action. However, in noisy environments, this could lead to overestimating the action values (or Q-values), resulting in a slower learning process. To resolve this issue and obtain an unbiased Q-value estimate, we implemented each agent as a double deep Q-Network (DDQN) consisting of two Q-networks: (i) a value network and (ii) a target network, as seen in Figure 4 in [25]. In this approach, two separate Q-value estimators are used in tandem to select an action (i.e., action selection) and for value evaluation (i.e., action evaluation). The update is similar to (18) and is given as (19).