## 3 DQN-Based Solution
### Policy Learning Process  
The training process is used to dynamically learn the decision-making policy \(\pi\) that the DQN agent uses when selecting the different actions as a function of the current state. For this purpose, the DQN agent executes the DQN algorithm of [16] but particularized to the state, action and reward signals presented above. In general, reinforcement learning (RL) algorithms aim at finding the optimal policy \(\pi^{*}\) that maximizes the discounted cumulative future reward (i.e., \(\sum\limits_{j=0}^{\infty}\tau^{k}r(t+j+1)\), where \(\tau\) is the discount rate ranging \(0\leq\tau\leq 1\)). In value-based RL algorithms, such as DQN, this is done by obtaining the optimum action-value function \(Q^{*}(\textbf{s,a})\), which is a scalar value representing the maximum expected discounted cumulative reward starting at time \(t\) from state \(\textbf{s}\), taking the action \(a\) and following the policy \(\pi\). This can be expressed in a recursive form by the Bellman equation as:  
\[Q^{*}(\textbf{s},\textbf{a})=E[r(t+1)+\tau\cdot\underset{a^{\prime}}{max}Q^{*} \left(\textbf{s}(t+1),\textbf{a}^{\prime}\right)|\textbf{s}(t)=\textbf{s}, \textbf{a}(t)=\textbf{a},\ \pi] \tag{4}\]  
Given \(Q^{*}(\textbf{s,a})\), the optimum policy is defined by greedily selecting the action \(a\) with the highest value for each state \(s\), that is:  
\[\pi^{*}=\underset{a}{\text{argmax}}Q^{*}(s,a) \tag{5}\]  
To determine the optimum action-value function, DQN approximates \(Q^{*}(\textbf{s,a})\) by a deep neural network (DNN) with weights \(\theta\), denoted as \(Q(\textbf{s,a},\theta)\), which is progressively updated during the training process. This DNN takes as input the state \(s\) and provides as output the value for each possible action \(a\) in accordance with the weights \(\theta\), which define the interconnections between the different neurons. For updating \(Q(\textbf{s,a,}\theta)\), the DQN agent is composed of the following elements:  
* Evaluation DNN \(Q(\textbf{s,a,}\theta)\): this is the main approximation of the optimum action-value function \(Q^{*}(\textbf{s,a})\). It is used to determine the decision-making policy \(\pi\) for selecting the actions, as: \[\pi=\underset{a}{argmax}\ Q(s,a,\theta)\] (6)* Target DNN \(Q(\textbf{s,}\textbf{a,}\ \theta^{\prime})\): this is another DNN with the same structure as the evaluation DNN but with weights \(\theta^{\prime}\). It is used to obtain the Time Difference (TD) target \(r(t+1)+\tau\underset{a^{\prime}}{max}Q(s(t+1),a^{\prime},\ \theta^{-})\) that is used for making successive updates of the evaluation DNN during the training. Moreover, this DNN is updated every \(P\) time steps (i.e., time windows) with the weights of the evaluation DNN, i.e., \(\theta^{-}=\theta\). * Experience dataset \(D\): this is a dataset of length \(l\) that stores the experiences of the DQN agent. The stored experience at time \(t\) is represented by the tuple \(<s(t)\), _a(t), \(r(t+1)\), \(s(t+1)>\)_, which captures the state at \(t\), the action taken, and the resulting reward and new state at time \(t+1\). The stored experiences are randomly selected during the training process to update the weights \(\theta\). At initialization, the weights of both the evaluation and target DNNs are randomly selected. Then, they are updated as a result of the training process of the DQN agent, which is divided in two parts: the data collection and the update of weights \(\theta\). Data collection consists in gathering experiences and storing them in the experience dataset \(D\). For each time \(t\), the DQN agent observes the state of the environment \(\textbf{s}(t)\) for a given UE and, accordingly, it triggers an action _a(t)_ based on an \(\varepsilon\)-Greedy policy that chooses actions according to the policy \(\pi\) in Equation (6) with probability \(1-\) and a random action with probability \(\varepsilon\). Then, the reward \(r(t+1)\) is collected and the experience tuple \(<s(t)\), _a(t), \(r(t+1)\)_, \(s(t+1)>\)_ is stored in the dataset.