## 3 Reinforcement Learning-Based Solution
### Deep Q-Learning-Based Solution Scheme  
In this solution scheme, a deep neural network is used as a function approximator to estimate the action-value function, where the \(Q\)-value is determined as follows:  
\[Q(s,a;\omega)\approx Q^{*}(s,a), \tag{18}\]  
where \(\omega\) is the weighting (bias) parameter in the neural network. The optimal \(Q^{*}(s,a)\) is obtained by updating \(\omega\) using stochastic optimization methods, i.e.,  
\[\omega(t+1)=\omega(t)-\lambda\Delta_{\omega}L(\omega), \tag{19}\]  
where \(\lambda\) denotes the learning rate to update \(\omega\), and \(\Delta_{\omega}\) represents the gradient of the loss function \(L(\omega)\) with respect to \(\omega\). Here, \(L(\omega)\) is the difference between the predicted value and the actual target value of the neural network. The action-value function \(Q(s,a)\) with a parameter vector \(\omega\) is then optimized by minimizing the loss function for iteration \(i\), and this can be expressed as  
\[L_{i}(\omega_{i})=\mathbb{E}_{s,a}[(y_{i}-Q(s,a);\omega_{i}))^{2}], \tag{20}\]  
where \(y_{i}\) denotes the target value, which can be estimated as  
\[y_{i}=\mathbb{E}_{s^{\prime}\sim E}\bigg{[}r+\delta\max_{a^{\prime}}Q(s^{ \prime},a^{\prime};\omega_{i-1})|s,a\bigg{]}. \tag{21}\]  
In particular, the gradient update with respect to the Q-function parameters \(\omega\) is computed as  
\[\begin{split}\Delta_{\omega_{i}}L_{i}(\omega_{i})& =\mathbb{E}_{s,a;s^{\prime}\sim E}\bigg{[}r+\delta\max_{a^{\prime }}Q(s^{\prime},a^{\prime};\omega_{i-1})\\ &-Q(s,a;\omega_{i})\Delta_{\omega_{i}}Q(s,a;\omega_{i})\bigg{]}.\end{split} \tag{22}\]  
During the training phase of the Q-network, the learning from different batches of consecutive samples is challenging due to the correlation among these samples. This leads to inefficient learning. On the other hand, since the current Q-network parameters define the upcoming training samples, this can introduce bad feedback loops. To tackle these issues, the experience buffer method is utilized where the Q-network is trained on random mini batches of transitions \((s(t),a(t),r(t),s^{\prime})\) as experienced from the replay buffer, instead of consecutive samples. Let \(B_{T}\) be the total experiences that can be saved in the replay buffer. At each learning step, a random mini batch of \(B\) experiences is sampled uniformly from the replay buffer pool, and then the Q-network's weights are updated by the stochastic gradient descent scheme. The deep Q-learning-based solution scheme using experience replay is summarized in Algorithm 2.  
```
Initialization:  Initialize the Q-network and the target Q-network with the parameterized function \(\omega\) and the experience replay buffer \(\mathcal{B}\). Parameter Setting and Updating: for each episodedo  Initial states \(s^{0}=\{s^{0}_{1},\dots,s^{0}_{M}\}\) observed by all D2D agents for each time slotdo  Actions \(a(t)=\eta(s(t))\) with the parameterized function \(\omega^{\eta}\) is selected and executed by all D2D agents ;  Perform observation of the rewards \(r(t)\) and the new state \(s^{\prime}\); ifthe transition number \(<B_{T}\)then  Save the transition \(\{s(t),a(t),r(t),s^{\prime}\}\) in the experience replay buffer; else  Restore the first stored transition with \(\{s(t),a(t),r(t),s^{\prime}\}\) in the experience replay buffer;  A mini-batch of \(B\) transitions is selected from the experience replay buffer;  Update the gradient of the loss function \(L(\omega)\) with respect to the parameterized function \(\omega\) using (22)
```  
**Algorithm 2**Deep Q-learning Based Solution Scheme using Experience Replay