## 4 Deep RL-Assisted Resource Allocation
### Basics of RL Algorithms  
The RL method addresses sequential decision-making by maximizing a cumulative reward function while interacting with the wireless environment, as shown in Figure 2. Assuming time series \(t\), the agent observes the environment and receives a state and feedback regarding the observed states. For each time step \(t\), the agent receives a state \(s^{(t)}\in S\) from a state space and then selects a specific action \(a^{(t)}\in A\), where \(S\) and \(A\) are a set of states and possible actions, respectively. After several executions, the agent receives a reward \(r^{(t)}\in R\) and sends an action \(a^{(t)}\) to the environment estimated by applying a certain policy \(\pi\) to the state \(s^{(t)}\), where \(R:S\times A\to R\) denotes the reward function. The policy \(\pi\) is essentially undertaken by the agent in a given state and best possible action. The agent then follows a policy \(\pi(a,s)\) and maps the state \(s^{(t)}\) to a probability distribution over a set of possible actions \(A\). Once the agent executes action \(a^{(t)}\), the environment changes its given state \(s^{(t)}\) to a next state \(s^{(t+1)}\in S\) in response to the agent's action. This scheme is repeated until the agent reaches the ending state and restarts. Q-learning (QL) is a basic form of RL algorithm aiming to evaluate the actions of the agent based on the current environment and maps the outputs in the form of rewards. The goal of QL algorithm is to find an optimal strategy \(\pi_{*}\) that maximizes the future cumulative rewards starting from given state \(s\), performing action \(a\), while following policy \(\pi\). Therefore, the future cumulative reward at time \(t\) is given by  
\[R_{t}=\sum_{a=0}^{\infty}\gamma^{k}r_{t+k} \tag{11}\]  
where \(\gamma\) denotes the discount factor for long returns. The agent uses this discount factor to adjust the importance of securing rewards over time \(t\). Multiple episodes are executed to train the QL algorithm, and the agent uses the \(\epsilon\)-greedy policy to estimate the optimal strategy \(\pi^{*}(s,a)\). We describe the Q-function \(Q^{\pi}(s,a)\) for policy \(\pi\), which is the expected reward beginning from current state \(s\), selecting best action \(a\), and thereafter following strategy \(\pi\). \[Q^{\pi}(s,a)=\mathbb{E}_{\pi}\Big{[}R^{(t)}\Big{|}s^{(t)}=s,\;a^{(t)}=a\Big{]} \tag{12}\]  
Similarly, the optimal strategy has an optimal Q-function, denoted as \(Q^{*}\) and define as \(Q^{*}(s,a)=\max\limits_{\pi}Q^{\pi}(s,a)\) for \(s\in S\) and \(a\in A(s)\). On the other hand, \(Q^{*}\) gives the maximum expected reward attainable by any strategy \(\pi\) for each possible state-action pair. So, the mathematically optimal Q-function can be expressed by the Bellman equation as:  
\[Q^{\pi}(s,a)=R(s,a)+\gamma\sum_{s^{\prime}}\bigg{(}P\big{(}s,a,s^{\prime}\big{)} \max\limits_{a^{\prime}}Q^{\pi}\big{(}s^{\prime},a^{\prime}\big{)}\bigg{)} \tag{13}\]  
where \(s^{\prime}\) and \(a^{\prime}\) denote the value of the new state and action, respectively. After state transition due to the actions taken and following a policy \(\pi\), the agent stores all the optimal Q-values in a tabular form. The QL algorithm uses a lookup table, also known as a Q-table, to save the Q-values of the optimal function. The Q-table is a matrix in which the number of rows represents the states and the columns corresponding to the actions. Once this Q-table is constructed, at each time-step \(t\), the agent chooses the best actions based on the \(\epsilon\)-greedy strategy and \(\epsilon\) is set to the value from the latest training step. Then, the agent executes a random action, and the action with the higher value is selected with probability \(1-\epsilon\) to avoid getting stuck at non-optimal strategies. According to the \(\epsilon\)-greedy policy, the agent either exploits the Q-table to obtain many rewards or explores the environment to select better action in the future.