## 5 Transmission Slot Selection Policy with Reinforcement Learning  
The underwater medium introduces uncertainties regarding water depth and environmental conditions, impacting the magnitude of underwater vibrations and movements and thereby influencing the harvested energy over time. Scheduling methods based on time slot allocation often overlook environmental factors that influence the availability of harvestable energy in the underwater environment when assigning time slots to nodes within the cluster. For this reason, nodes transmit within their scheduled time periods, regardless of available energy, resulting in a deficiency of the expected amount of harvested energy. In our model, we adapt a multi-agent independent Q-learning algorithm with cooperation running multi-armed bandit problem on each node to autonomously select their time slot per data frame and learn from their past decision-making choices influenced by ambient conditions. In the single-agent, multi-armed bandit problem, the agent makes a choice among \(k\) different actions modeled as arms of a slot machine [40]. In this slot machine, each time after an action is performed, the agent receives a reward depending on a probability distribution in arms. The value of the actions is recorded as the mean of rewards over time steps. The agent maintains the estimated value of each action \(a\) in each time step, as \(Q_{t}(a)\). The aim of the agent is to maximize the cumulative rewards over repeated actions in time steps. In a single data frame, \(R_{t}[a_{i}]\) denotes the reward received after action \(a_{i}\) is selected in time step \(t\), and the estimated value of action \(a_{i}\) after \(t-1\) is \(Q_{t}[a_{i}]\) defined in Equation (17). \[Q_{t}[a_{i}]=Q_{t-1}[a_{i}]+\frac{1}{t-1}(R_{t-1}[a_{i}]-Q_{t-1}[a_{i}]) \tag{17}\]  
On the other hand, multi-agent systems (MASs) focus on developing algorithms that enable agents to learn adaptively and optimize their behavior in changing environments. Multi-agent reinforcement learning (MARL) aims to educate multiple agents simultaneously within a common environment by employing reinforcement learning (RL) techniques. In a fully cooperative setting, agents work together toward a shared goal,like assembling machines [41]. Figure 2 represents the multi-agent system, \(n\) distributed agents as underwater sensor nodes, cooperatively play the same multi-armed bandit game. Each agent has its own record of the Q table as time slot arrays. Agents choose actions independently, constructing a joint action array from received packets at the cluster head. We model k arms to represent time slots, the same as cluster size (the total number of cluster members). Note that each frame consists of k, i.e., TS many time slots. To find the best global combination of arms, agents wish to maximize a global return, or, in other words, minimize the overall regret. Associated with arms, mutually independent sequences of i.i.d.-valued rewards are modeled. Specifically, the reward is the overall energy gained in the rest of the arms except the arm selected for transmission. We run our model for a predetermined duration of a time slot. A detailed explanation of our method is given in Algorithm 1.