## 3 Reinforcement Learning-Based Solution
### Actor-Critic-Based Solution Scheme
Thus, the policy under the parameterized actor function \(\omega^{\eta}\) can be represented as \(J(\omega^{\eta})=\mathbb{E}\{Q(a)\}\), as in [27], where, \(a=\eta(s_{i})\). Since \(J(\omega^{\eta})\) is continuously differentiable with respect to \(\omega^{\eta}\), \(\omega^{\eta}\) can be updated by the chain rule to the expected return from the start distribution \(J\) with the gradient of \(J(\omega^{\eta})\) as follows:  
\[\Delta\omega^{\eta}=\lambda^{a}\frac{1}{B}\sum_{\mathcal{B}}\Delta_{a}Q(a) \big{|}_{a=\eta(s_{i})}\Delta_{\omega^{\eta}}\eta(s_{i}), \tag{27}\]  
where \(\lambda^{a}\) denotes the learning rate of the actor structure; \(\Delta_{a}Q(a)\big{|}_{a=\eta(s_{i})}\) is the derivative of \(Q(a)\) with respect to \(a\), where \(a=\eta(s_{i})\); and \(\Delta_{\omega^{\eta}}\eta(s_{i}|\omega^{\eta})\) is the derivative of \(\eta(s_{i})\)  
Figure 3: The actor-critic framework based on the DDPG scheme. with respect to \(\omega^{\eta}\). Then, the following equations (28) and (29) are constructed by the derivative scheme:  
\[\omega^{\eta^{\prime}} =\varsigma^{a}\omega^{\eta}+(1-\varsigma^{a})\omega^{\eta^{\prime}}, \tag{28}\] \[\omega^{Q^{\prime}} =\varsigma^{c}\omega^{Q}+(1-\varsigma^{c})\omega^{Q^{\prime}}, \tag{29}\]  
where \(\varsigma^{a}\ll 1\) and \(\varsigma^{c}\ll 1\). This allows the values of the target networks to be constrained to change slowly, which greatly improves the stability of the learning [25]. The DDPG-based actor-critic algorithm with the parameter updating processes is summarized in Algorithm 3. In this scheme, the evaluation network of the actor structure performs the action for each state in the given environment. A random mini batch of transitions are chosen in each learning process by sampling uniformly from the experience replay buffer, and they are input one by one to the D2D agent. With each input experience, the actor updates its parameterized function \(\omega^{\eta}\) of the evaluation network based on the policy gradient to maximize \(\mathbb{E}\{Q(a)\}\), and the critic updates its parameterized function \(\omega^{Q}\) of the evaluation network based on the loss function to minimize the TD error. For both the actor and critic, the parameterized functions of the target networks are soft updated instead of directly copying the weights with that of their evaluation networks. ```
Initialization:  Initialize the parameterized functions \(\omega^{\eta}\), \(\omega^{\eta\prime}\), \(\omega^{Q}\), and \(\omega^{Q^{\prime}}\) of the evaluation and target networks of the actor-critic structure and the experience replay buffer \(\mathcal{B}\). Parameter Setting and Updating: for each episodedo  Initial states \(s^{0}=\{s^{0}_{1},\ldots,s^{0}_{M}\}\) observed by all D2D agents for each time slotdo  Actions \(a(t)=\eta(s(t))\) with the parameterized function \(\omega^{\eta}\) is selected and executed  by all D2D agents ;  Perform observation of the rewards \(r(t)\) and the new state \(s^{\prime}\); ifthe transition number \(<B_{T}\)then  Save the transition \(\{s(t),a(t),r(t),s^{\prime}\}\) in the experience replay buffer; else  Restore the first stored transition with \(\{s(t),a(t),r(t),s^{\prime}\}\) in the experience replay buffer;  A mini-batch of \(B\) transitions is selected from the experience replay buffer;  Update the parameterized functions of the critic structure: \(\omega^{Q}\leftarrow\omega^{Q}+\Delta\omega^{Q}\); \(\omega^{Q^{\prime}}=\varsigma^{c}\omega^{Q}+(1-\varsigma^{c})\omega^{Q^{\prime}}\);  Update the parameterized functions of the actor structure: \(\omega^{\eta}\leftarrow\omega^{\eta}+\Delta\omega^{\eta}\); \(\omega^{\eta^{\prime}}=\varsigma^{a}\omega^{\eta}+(1-\varsigma^{a})\omega^{ \eta^{\prime}}\);
```  
**Algorithm 3**DDPG-based Actor-Critic Solution Scheme