## 5 Transmission Slot Selection Policy with Reinforcement Learning
```
0\(\leqslant\gamma\leqslant 1\), \(\alpha\in(0,1]\), small \(\varepsilon>0\), \(n\geq 0\)  Initialize \(Q_{i}[a]\leftarrow\forall a\in A_{i}\) and \(\forall i\in N\)  Update Q table \(Q_{i}[a]\leftarrow\mathbb{E}_{i}^{h}(t)\forall i\in TS\) for each time slot in frame (TS) do for each node in Cluster\({}^{k}\)do if\(random(n)\geq\varepsilon\)then  Choose exploitation  Select maximizing action \(a_{i}=\arg\max Q_{i}[:]\) else  Choose exploration  Select a random action \(a_{i}\) endif  node n selects action \(a_{i}\) forms the joint action \(\mathbf{a}\leftarrow\langle a^{1},\ldots,a^{N}\rangle\)  Receive local reward from environment \(R_{t}^{n}(a_{i})\) endfor  Return global reward from environment \(R_{t}^{N}(\mathbf{a})\) \(R_{t}(a_{i})\leftarrow\beta R_{t}^{N}(\mathbf{a})+(1-\beta)R_{t}^{n}(a_{i})\)  Update \(Q_{t}[a_{i}]\leftarrow\alpha*(R_{t}(a_{i})+\gamma*Q_{t}[a_{max}]-Q_{t-1}[a_{i}])\) endfor
```  
**Algorithm 1** Distributed Multi-Agent Learning Algorithm for Transmission slot selection  
Figure 2: Training paradigm of multi-agent reinforcement learning system. selected depending on the interaction of each sensor with its environment, which affects the available energy. Transmission slots are designed as discrete time intervals. Action is the choice of the time interval for data transmission. 3. Reward: The reward calculation contains the weighted sum of two parts, namely the mutual reward and the independent local reward. The mutual reward of a cluster member after selecting a transmission slot is the total number of data packets that arrive in the cluster head \(R_{CH}\), and it is broadcast to cluster members at the end of each frame. The weight coefficient \(\beta\) is fixed at 0.4. Reward function penalizes conflicted slot selection with the decreased amount of received packets. In case of conflict, nodes will wait for the next round to send their data packets. As a result, a reduction in the number of received data packets will lead to a decrease in the global reward, prompting nodes to adjust to their neighboring nodes' decisions. The total harvested energy of the cluster member serves as the agent's local reward, calculated as the overall energy gain excluding the selected time interval for transmitting a single data frame. In simpler terms, it represents the accumulated harvested energy within the cluster during the idle time of nodes in the previous frame. \[Reward_{n}=\beta\sum_{t=1}^{TS}R_{CH}+(1-\beta)\sum_{t=1}^{TS}E_{s_{n}}^{h}(t) \tag{18}\]  
We propose a multi-agent reinforcement learning algorithm for the optimal policy in the selection of time slots. Awareness of environmental conditions and observed energy fluctuations in ambient resources can be utilized in the determination of timing for data transmission; hence, this approach potentially alleviates the load on the cluster head. Furthermore, the selection of the best transmission slots for each node in the cluster is of mutual interest to all nodes in the system, thereby maximizing energy utilization in the cluster. With learning, each sensor can exploit the ambient resources to harvest energy to assure the maximal lifetime of the network. In our method, we consider round-based data collection. After one frame of data collection is complete, all nodes in the cluster take the same reward for the number of data packets received and a local reward for the sum of their harvested energy in the frame. For intra-cluster communication, nodes perform data transmission in their time slots that are designated by their cluster head and switch back to an idle state until the next frame. However, instead of centralized control of medium access, nodes determine their slots by learning the environmental conditions of harvesting parameters. In Figure 3, there is an example timeline showing our approach within a data frame in clustered networks. A round's duration is preset to guarantee that, on average, every node can function as both a cluster head and a non-cluster head several times over its lifetime [9].