# Deep Reinforcement Learning-Based Video Offloading and Resource Allocation in NOMA-Enabled Networks
## 4 Deep Reinforcement Learning-Based Algorithm
### JVFRS-TO-RA-DQN Algorithm  
In this section, we propose JVFRS-TO-RA-DQN to solve the problem of joint video frame resolution scaling, task offloading, and resource allocation. The proposed algorithm is based on DQN, which can study the offline historical data through the experience of a simulation without requiring full environmental knowledge. The detailed algorithm is shown in Figure 2.  
According to the state of the system at present, the DQN algorithm maximizes the predefined reward function by choosing an \(A_{m,t}\) from a limited sum of actions.  
In the process of training, apart from state \(S_{m,t}\), action \(A_{m,t}\), policy \(\pi\), and reward function \(r_{m,t}\), the state-action value function \(Q^{\pi}(S_{m,t}\), \(A_{m,t})\) determines the action \(A_{m,t}\) of the state \(S_{m,t}\) through a mapping function \(\pi(A_{m,t}\,|\,S_{m,t})\). If \(Q^{\pi}(S_{m,t}\), \(A_{m,t})\) is updated at each time step, then it is assumed that it will converge to the optimum state-action value function \(Q^{\pi^{\prime}}(S_{m,t},A_{m,t})\). According to the Bellman equation, the evaluation of the quality of a specific action in a given specific state can be expressed as  
\[Q^{\pi}(S_{m,t},A_{m,t})=(1-\lambda)Q^{\pi}(S_{m,t},A_{m,t})+\lambda(r_{m,t}+ \gamma\underset{A^{\prime}}{\max}Q(S_{m,t+1},A^{\prime})), \tag{20}\]  
where \(\lambda\) represents the learning rate which reflects the rate of the algorithm adapting to a new environment, where \(\lambda\in(0,1]\). Since the complexity of Equation (20) is exponentially related to the number of state-action pairs, it is more difficult to solve \(Q\) values when the state-action pairs increase. In order to accurately calculate the \(Q\) values, predicting the values of \(Q\) between different state-action pairs is significant, and also represents the hinge of the DQN algorithm.  
In contrast to using traditional tabular Q-learning for prediction, DQN has a special replay memory structure to store the data generated after each step, including every step. When the network is in training, it extracts some memory from the replay memory for experiential learning. The replay memory has enough training data to fit the \(Q\) values of different state-action pairs, which leads to  
\[Q^{\pi}(S,A)\approx\hat{Q}(S,A,\theta), \tag{21}\]  
where \(\hat{Q}(\cdot;\theta)\) is a deep learning network function denoted by \(\theta\). \(Q^{\pi}(S,A)\) is updated by minimizing the loss function, which is defined as  
\[L(\theta)=\mathbb{F}\Bigg{[}\bigg{(}r_{m,t}+\gamma\underset{A^{\prime}}{\max} Q(S_{m,t+1},A^{\prime})-Q(S,A,\theta)\bigg{)}^{2}\Bigg{]}, \tag{22}\]  
The gradient descent algorithm is utilized to minimize the loss in Equation (22) and therefore to update the weight \(\theta\), so as to make it possible to minimize the error between the evaluation and the target. As a result, the neural network can predict more accurately as the training process continues. The JVFRS-CO-RA-DQN algorithm is explained in Algorithm 1.  
Figure 2: An illustration of the JVFRS-TO-RA-DQN architecture.