## 4 Deep Reinforcement Learning-Based Resource Allocation Policy
### Deep Actor-Critic Reinforcement Learning Algorithm
#### 4.2.2 The Actor with a DNN  
The DNN in the actor is shown in Figure 7, which includes an input layer, two hidden layers, and an output layer. The input layer of the actor is the current state of the environment. There are two hidden layers in the actor, where each hidden layer is comprised of \(H_{A}\) neurons. The output layer of the actor provides the probabilities of selecting actions in a given state. Furthermore, the output layer utilizes the soft-max activation function [28] to compute the policy of each action in the action space, which is given as:  
\[\pi_{\mathbf{\theta}}\left(a\left|s\right.\right)=\frac{e^{z_{a}}}{\sum\limits_{ \mathbf{a}^{\prime}\in\mathbb{A}}e^{z_{a}^{\prime}}}, \tag{22}\]  
where \(z_{a}\) is the estimated value for the preference of choosing action \(\mathbf{a}\). In the actor, the policy can be enhanced by optimizing the state-value function as follows:  
\[ \begin{array}{l}\int\left(\pi_{\mathbf{\theta}}\right)=E\left[V^{\pi}\left(s \right)\right]\\ =\sum\limits_{s\in\mathbb{S}}d^{\pi}\left(s\right)V^{\pi}\left(s\right),\end{array}  \tag{23}\]  
Figure 6: The deep neural network in the critic. where \(d^{\pi}\left(s\right)\) is the state distribution. Policy parameter \(\theta\) can be updated toward the gradient ascending to maximize the objective function [39], as follows:  
\[\Delta\theta=\alpha_{a}\
abla_{\theta}J\left(\pi_{\theta}\right), \tag{24}\]  
where \(\alpha_{a}\) denotes the learning rate of actor network, and policy gradient \(\
abla_{\theta}J\left(\pi_{\theta}\right)\) can be computed by using the TD error [47]:  
\[\
abla_{\theta}J\left(\pi_{\theta}\right)=E\left[\
abla_{\theta}\log\pi_{ \theta}\left(s,\textbf{{a}}\right)\delta\left(t\right)\right]. \tag{25}\]  
It is worth noting that TD error \(\delta\left(t\right)\) is supplied by the critic. The training procedure of the proposed DACRL approach is summarized in Algorithm 2. In the algorithm, the agent interacts with the environment and learns to select optimal action in each state. The convergence of the proposed algorithm depends on number of steps per episode, the number of training episodes and the learning rate, which is discussed in the following section. ```
1:Input: \(\mathbb{S}\), \(\mathbb{A}\), \(\gamma\), \(\alpha_{a}\), \(\alpha_{c}\), \(\textbf{{e}}^{\textbf{{e}}}(t)\), \(\mu(t)\), \(E_{ca}\), \(\xi_{avg}\), \(T\), \(W\), \(\epsilon_{min}\), \(\epsilon_{max}\), \(\epsilon_{d}\). 2: Initialize network parameters of the actor and the critic: \(\theta,\omega\). 3: Initialize \(\epsilon=\epsilon_{max}\). 4:for each episode \(e=1,2,3,...,L\) :
5: Sample an initial state \(s\in\mathbb{S}\). 6:for each time step \(t=0,1,2,3...,T-1\) :
7: Observe current state \(s\left(t\right)\), and estimate state value \(V_{\omega}^{\pi}\left(s\left(t\right)\right)\). 8: Choose an action:
9:
10:\(\textbf{{a}}\left(t\right)=\left\{ \begin{array}{ll}\arg\max\pi_{\theta} \left(a\left(t\right)\left|s\left(t\right)\right.\right)&\text{w.p 1 -ffl}\\ \text{random action }a\left(t\right)\in\mathbb{A}&\text{otherwise}\end{array} \right.\)
11: Execute action \(\textbf{{a}}\left(t\right)\), observe current reward \(R\left(t\right)\). 12: Update epsilon rate \(\epsilon=\max\left(\epsilon.\epsilon_{d},\epsilon_{\min}\right)\)
13: Update next state \(s\left(t+1\right)\)
14:Critic Process:
15: Estimate next state value \(V_{\omega}^{\pi}\left(s\left(t+1\right)\right)\). 16: Critic calculates TD error \(\delta\left(t\right)\)
17:if episode is end at time slot \(t\):
18:\(\delta\left(t\right)=R\left(t\right)-V_{\omega}\left(s\left(t\right)\right).\)
19:else
20:\(\delta\left(t\right)=R\left(t\right)+\gamma V_{\omega}\left(s\left(t+1\right) \right)-V_{\omega}\left(s\left(t\right)\right).\)
21:endif
22: Update parameter of critic network \(\omega\leftarrow\omega+\Delta\omega\)
23:Actor Process:
24: Update parameter of actor network \(\theta\leftarrow\theta+\Delta\theta\)
25:endfor
26:endfor
27:Output: Final policy \(\pi_{t}^{*}\left(\textbf{{a}}\left(t\right)\left|s\left(t\right)\right.\right)\).