## 3 Reinforcement Learning-Based Solution
### Q-Learning-Based Solution Scheme  
The objective of the D2D agent is to find a policy \(\pi^{*}\) which maximizes the _cumulative_ discounted reward denoted as  
\[\sum_{\tau\geq 0}\delta^{\tau}r(t+\tau), \tag{9}\]  
where \(r(t)\) is the immediate reward for the D2D agent at time \(t\), \(\delta\in(0,1]\) represents the discount factor, and \(t\) denotes the time slot.  
The optimal policy \(\pi^{*}\), which maximizes the expected sum of rewards, can be written as  
\[\pi^{*}=\arg\ \max_{\pi}\ \mathbb{E}\Bigg{\{}\sum_{\tau\geq 0}\delta^{\tau}r(t+ \tau)|\pi\Bigg{\}}. \tag{10}\]  
Now, the value function at state \(s\) following the policy can be expressed as  
\[V^{\pi}(s)=\mathbb{E}\Bigg{\{}\sum_{\tau\geq 0}\delta^{\tau}r(t+\tau)|\pi,s=s(t) \Bigg{\}}. \tag{11}\]  
The \(Q\)-value function of considering action \(a\) in state \(s\) following the policy can be written as  
\[Q^{\pi}(s,a)=\mathbb{E}\Bigg{\{}\sum_{\tau\geq 0}\delta^{\tau}r(t+\tau)|\pi,s=s (t),a=a(t)\Bigg{\}}. \tag{12}\]  
Now, the optimal \(Q\)-value function \(Q^{*}\) from a given (state, action) pair can be represented as  
\[Q^{*}(s,a)=\max_{\pi}\ \mathbb{E}\Bigg{\{}\sum_{\tau\geq 0}\delta^{\tau}r(t+ \tau)|\pi,s=s(t),a=a(t)\Bigg{\}}. \tag{13}\]  
The \(Q^{*}\) that satisfies the following Bellman equation [24] is utilized to find the optimal policy \(\pi^{*}\) by considering the best action in any state:  
\[\begin{split} Q^{*}(s,a)&=\mathbb{E}_{s^{\prime} \sim E}\bigg{[}r+\delta\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})|s,a \bigg{]}\\ &=\sum_{s^{\prime},s^{\prime}}\mathcal{P}(s^{\prime},r|s,a) \bigg{[}r+\delta\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})\bigg{]}.\end{split} \tag{14}\]  
Now, the Bellman equation (14) is updated with an iterative approach as follows:  
\[Q_{i+1}(s,a)=\mathbb{E}\bigg{[}r+\delta\max_{a^{\prime}}Q_{i}(s^{\prime},a^{ \prime})|s,a\bigg{]}, \tag{15}\]  
where \(Q_{i}\) converges to \(Q^{*}\) as iteration \(i\to\infty\).  
The iterative update on the Bellman equation is then utilized to find the optimal \(Q^{*}(s,a)\), which can be written as  
\[Q^{*}(s,a)\leftarrow(1-\alpha)Q^{*}(s,a)+\alpha\bigg{[}r+\delta\max_{a^{ \prime}}Q^{\pi}(s^{\prime},a^{\prime})\bigg{]}, \tag{16}\]  
where \(\alpha\in(0,1]\) denotes the learning rate.  
In a Q-learning-based solution scheme, the \(\varepsilon\)-greedy scheme [24] is used to choose the action based on the current action-value estimation, which can be written as follows:  
\[\pi=\begin{cases}\arg\max_{a(t)\in\mathcal{A}}Q(s(t),a(t)),&\text{with probability }1-\varepsilon\\ \operatorname{random}\{a_{i}\}_{a_{i}\in\mathcal{A}},&\text{with probability } \varepsilon.\end{cases} \tag{17}\]The Q-learning-based solution scheme is provided in Algorithm 1. At each iteration step of this algorithm, each D2D agent selects an action and updates its own policy independently.  
```
Initialization:  Initialize the \(Q\)-value function \(Q(s,a)\) with random weights  Parameter Setting and Updating: for each episodedo  Initial states \(s^{0}=\{s^{0}_{1},\dots,s^{0}_{M}\}\) observed by all D2D agents for each time slotdo  Actions \(a(t)\) are selected using the \(\varepsilon\)-greedy scheme (17) and then executed by all D2D agents ;  Perform observation of the rewards \(r(t)\) and the new state \(s^{\prime}\);  Update \(Q(s(t),a(t))\) using (16);  Update \(\pi(s(t),a(t))\);
```  
**Algorithm 1**Q-learning Based Solution Scheme