## 5 DRL-Based Online Resource Allocation
### Process of DRL-Based Online Resource Allocation  
Following the conventional training pipeline of DRL, this section presents the training process of the proposed DRL-QoS-RA algorithm. As illustrated in Figure 2, the agent observes environmental information \(s_{t}\), including channel information, QoS requirement, data amount, and existing terminals' resource occupation. Then based on a certain policy \(\pi\), the resource allocation action \(a_{t}\) is determined, and then the current reward \(r_{t}\) is collected, with the environment changing to \(s_{t+1}\). Each time the agent goes through the above process, the experience \(\{s_{t},a_{t},r_{t},s_{t+1}\}\) is collected and put into the experience pool, from which the agent can periodically sample experience data and train the policy neural network to optimize long-term benefits. The converged strategy neural network promotes energy efficiency and transmission success rate with QoS requirements guaranteed. The goal of this paper is to optimize long-term reward. \[\max_{\pi}\sum_{t=0}^{T}r_{t} \tag{14}\]  
However, since the system operates continuously rather than in a limited time, RL's classical theory adds the discount factor \(\gamma\) to Equation (14) to make the long-term reward more meaningful. \(\gamma\in(0,1]\) is used to adjust the short-term and long-term impact [23], in other words, how far the agents consider when making decisions. In terms of intuitive perspective, it makes the agent pay more attention to the impact of its action on the near future state. From the perspective of the agent's training process, \(T_{len}=1/(1-\gamma)\) can be used to estimate the number of steps that the agent considers in the future when making decisions. For \(t>1/(1-\gamma)\), the discount parameter \(\gamma^{t}\) is almost 0. It means that the agent's action selection does not consider its influence on the state beyond the T range, which benefits the convergence of RL methods. Therefore, with discount factor \(\gamma\), the  
Figure 2: Framework of DRL_CAPC algorithm. optimization goal is \(V^{\pi}(s)=E\left[\sum\limits_{t_{0}}^{t_{0}+T_{lin}}\gamma^{t-t_{0}}\tau_{t} \right]\), which indicates that the agent optimizes the long-time revenue by optimizing the discounted reward of the future T steps at each time stick. As a result, the larger that \(\gamma\) is, the more steps the agent takes into consideration with more difficulty of the training process. The smaller that \(\gamma\) is, the more the agent pays attention to the immediate interests. The long-term revenue to be optimized can be denoted as:  
\[\max_{\pi}\,E\left[\sum\limits_{t=0}^{T}\gamma^{t}\tau_{t}\right] \tag{15}\]  
where \(E[\cdot]\) denotes expectation and \(\pi\) denotes a specific strategy. For a certain state, the optimization objective is defined as the state value function in RL theory \(V^{\pi}(s)\), which denotes the long-term reward of a specific strategy from state \(s\). Owing to Markov property, \(V^{\pi}(s)\) can be represented as  
\[V^{\pi}(s_{t})=r_{t}+\gamma\sum\limits_{s_{t+1}}P(s_{t+1}|s_{t},\pi(s_{t})\,)V (s_{t+1},\pi) \tag{16}\]  
In RL theory, the state-action value of action \(a_{t}\) and state \(s\) is also defined as \(Q(s_{t},a_{t})\), which is the accumulated reward if action \(a_{t}\) is chosen by the agent. \[Q^{\pi}(s_{t},a_{t})=r_{t}+\gamma\sum\limits_{s^{\prime}\in\mathcal{S}}\,P(s_{ t+1}|s_{t},a\,)V^{\pi}(s_{t+1}) \tag{17}\]  
The goal of RL agent is to learn to perform actions to maximize the sum of benefits received in the long term. To achieve this goal, RL methods use different ways to estimate the state value function \(V^{\pi}(s)\) or state-action value function \(Q^{\pi}(s,a)\). For example, Q learning, as a popular method of RL, adopts a \(Q\) value table to estimate \(Q^{\pi}(s,a)\). When the state space becomes huge, the value function approximation methods based on table form encounter a dimensional disaster problem and are thus no longer applicable.