## 3 Reinforcement Learning-Based Solution
### Actor-Critic-Based Solution Scheme  
To learn the optimal policy for the OP formulated in (4), an actor-critic framework based on the DDPG scheme [25] is adopted, as shown in Figure 3, with consideration of the continuous-valued state and action spaces. In the DDPG scheme, the D2D agent passes through the learning process and targets finding the optimal policy \(\pi\) by achieving the maximum long-term reward \(\mathbb{E}\{Q^{\pi}(s,a)\}\). The DDPG scheme combines deterministic policy gradients and deep Q-learning approaches to train the actor and the critic network. Similar to DQN, a replay memory and a separate target network are also utilized here. In order to form the actor-critic framework, two similarly structured deep neural networks (DNNs) with different parameter settings are considered, where an evaluation network is used for real-time parameter updating, and a target network is included for soft parameter updating. The utilization of experience replay buffer and batch normalization procedures not only helps to improve the performances of both the evaluation and the target network but also provides stable and faster convergence. Each transition \((s(t),a(t),r(t),s^{\prime})\) of the D2D agent is kept in the experience replay buffer. When the experience replay buffer becomes full, the first recorded transitions are overwritten by the new-coming transitions that initiate the learning process. A random mini batch of \(B\) transitions, denoted as \((s_{i},a_{i},r_{i},s^{\prime}_{i})\)\(\forall\)\(i=1,\dots,B\), is chosen during the learning process by sampling uniformly from the experience replay buffer to update the actor and the critic networks. In the critic structure, the evaluation and target networks are used to estimate the \(Q\)-value functions as follows:  
\[Q^{\pi}(s,a)=\mathbb{E}_{r,s^{\prime}\sim E}\big{[}r+\delta\mathbb{E}_{a^{ \prime}\sim\pi}Q^{\pi}(s^{\prime},a^{\prime})\big{]}. \tag{23}\]  
The critic structure parameter is updated as \(Q(a)\approx Q^{\pi}(s,a)\) during the learning period since the achievable \(Q\)-value function under the deterministic policy \(\pi\) (\(\pi:\mathcal{S}\rightarrow\mathcal{A}\)) is \(Q^{\pi}(s,a)\), where \(\pi(s)=\text{arg max}_{a}Q(s,a)\)[26]. Let \(\omega^{\mathbb{Q}}\) and \(\omega^{Q^{\prime}}\) be the parameterized functions; \(Q(\cdot)\) denotes the network function of the evaluation network, and \(Q^{\prime}(\cdot)\) represents the network functions of the target network for the critic structure. Then, the mini batch of \(B\) transitions is chosen during the learning process, and the weights (\(\omega^{Q}\)) are updated by minimizing the loss function stated as  
\[L(\omega^{Q})=\mathbb{E}\{(\xi_{i})^{2}\}, \tag{24}\]where \(\xi_{i}\) is the temporal-difference (TD) error, which denotes the deference between the estimated \(Q\)-value and the target \(Q\)-value, which can be written as follows:  
\[\xi_{i}=Q(a_{i})-(r_{i}+\delta Q^{\prime}(a^{\prime})). \tag{25}\]  
Let the loss function \(L(\omega^{Q})\) be continuously differentiable with respect to \(\omega^{Q}\). Then, the critic structure can update \(\omega^{Q}\) with the gradient of \(L(\omega^{Q})\) as follows:  
\[\Delta\omega^{Q}=\lambda^{c}\frac{1}{B}\sum_{\mathcal{B}}\xi_{i}\Delta_{ \omega^{Q}}Q(a_{i}), \tag{26}\]  
where \(\lambda^{c}\) denotes the learning rate of the critic structure, and \(\Delta_{\omega^{Q}}Q(a_{i})\) is the derivative of \(L(\omega^{Q})\) with respect to \(\omega^{Q}\). Let \(\omega^{\eta}\) and \(\omega^{\eta^{\prime}}\) be the parameterized functions; \(\eta(\cdot)\) denotes the network function of the evaluation network, and \(\eta^{\prime}(\cdot)\) represents the network function of the target network in the actor structure. The actor structure utilizes the policy gradient scheme to update the parameterized function \(\omega^{\eta}\) during the learning process to make \(\eta(s_{i})\) approximate the optimal policy \(\pi\) so that the maximum \(\mathbb{E}\{Q^{\pi}(s,a)\}\) can be achieved.