## 4 Deep RL-Assisted Resource Allocation
### Basics of RL Algorithms
After obtaining a new experience due to the action taken, the QL  
Figure 2: Reinforcement learning for multi-cell OFDMA systems. algorithm updates the value of a Q-table based on the feedback learning agents. The QL update uses the following iterative approaches to train the QL algorithm:  
\[Q\Big{(}s^{(t)},a^{(t)}\Big{)}\gets Q\Big{(}s^{(t)},a^{(t)}\Big{)}+\alpha \Big{[}R^{(t+1)}+\gamma\underset{a^{\prime}}{\max}Q\Big{(}s^{(t+1)},a^{\prime} \Big{)}-Q\Big{(}s^{(t)},a^{(t)}\Big{)}\Big{]} \tag{14}\]  
Here, \(\gamma\in(0,\,1]\) is a factor that determines the priority of future rewards compared to the current reward and lies in the [0:1] range. A value of \(\gamma=1\) means that future rewards are more important than the current reward. Furthermore, \(\alpha(0<\alpha\leq 1)\) is a learning rate, which denotes the proportion of newly learned data related to the given action value. When \(\alpha=1\), the agent can learn the policy, and the newly trained data are the only significant information. For each of the current Q-value \(Q\Big{(}s^{(t)},a^{(t)}\Big{)}\), an estimate of the expected reward is tracked and an \(\epsilon\)-greedy policy is selected based on these estimates. After that an arbitrary action is chosen with probability \(\epsilon(0\leq\epsilon\leq 1)\) while the action with highest value is selected with probability \(1-\epsilon\). Since the QL algorithm updates according to the Bellman equation, the testing or implementation phases can be executed through online temporal difference learning. Traditional RL methods struggle to address real-world problems due to the inability to efficiently model high dimensional state space's objective. The limitation of the QL algorithm is that it adopts the Q-table to store the Q-values. However, in many problems of practical scenarios, the Q-table will be hard to use when the state space of the problem is too large. Consequently, it is important to use function approximation to handle the large state-action spaces. In this paper, we focus on the deep Q-learning network (DQN), a combination of deep neural networks (DNN) and traditional QL. Based on the QL algorithm, DQN uses DNN to approximate the optimal action-value function \(Q(s,a,\theta)\approx Q^{*}(s,a)\) on a discrete action space. Here, \(Q(s,a,\theta)\) denotes the DQN and \(\theta\) is the parameter of the neural network. Instead of using a table to store the Q-values, the DQN uses a replay memory \(\mathcal{D}\) to store the transition tuples \((s^{(t)},a^{(t)},R^{(t+1)},s^{(t+1)})\) at each time period. The memory stores experiences to avoid correlation between input data in successive updates. The DQN is trained with a minibatch sampled randomly from the replay memory and updates the targets of that minibatch. The mean squared error (MSE) of the Bellman equation is minimized by the iterative update, which is used to train the Q-network. Hence, an experience replays to minimize the MSE, denoted as [30]:  
\[L(\theta)=\sum_{\mathcal{D}}\bigg{[}R^{(t+1)}+\gamma\underset{a^{\prime}}{ \max}Q\big{(}s^{\prime},a^{\prime};\theta^{-}\big{)}-Q\Big{(}s^{(t)},a^{(t)}; \theta\Big{)}\bigg{]}^{2} \tag{15}\]  
where \(\theta^{-}\) denotes the parameter set of learning target action-value function, which is generated from the training DQN parameter \(\theta\) periodically and fixed for successive updates. Note that the target of Problem (15) is to train the DQN function for a random mini-batch \(\mathcal{D}^{(t)}\) each time \(t\), such that \(R^{(t)}_{DQN}(s^{\prime},a^{\prime})=r^{\prime}+\gamma\underset{a^{\prime}}{ \max}\hat{Q}\Big{(}s^{\prime},a^{\prime};\theta^{(t)}\Big{)}\).