## 5 Outer and Inner Loop Optimization Algorithm with Online Learning
### Outer Loop Optimization: Novel Dec-MPMAB Algorithm
#### 5.1.2 Decentralized-UCB (D-UCB) Algorithm  
The UCB is a renowned multi-armed bandit (MAB) algorithm that adeptly manages the exploration-exploitation trade-off. This is achieved by associating an upper confidence bound with each arm's estimated reward and choosing the arm with the highest upper confidence bound at each time step. The UCB systematically boosts the confidence in the selected actions by mitigating their uncertainty. Similar to the single-player multi-armed bandit (MAB) discussed earlier, in the multi-player MAB [26], there are two phases referred to as the _exploration phase_ and _exploitation phase_. Exploration phaseThe input parameters for the algorithm include the number of players (N), the total number of arms (JM), the exploration parameter (C), and the time horizon (T). To initialize the algorithm, each player i initializes the following. 1. An array of length JM to store the number of times that arm \(a\in a_{1},a_{2},\ldots,a_{MJ}\) has been selected, denoted as \(\mathbf{n}_{i,MJ}(t)\). 2. An array of length JM to store the sample mean of rewards from arm \(a\), denoted as \(\overline{\mathbf{X}}_{i,MJ}(t)\). 3. An array of length JM to store the distributed upper confidence bounds, denoted as \(\mathbf{D}\)-\(\mathbf{UCB}_{i,MJ}(t)\). From t = 1 until t = T, for each player i, the algorithm proceeds to select RIS m and RB j. A D-UCB index is defined at the end of frame t as follows:  
\[\text{D-UCB}_{i,a}(t):=\overline{\mathbf{X}}_{i,a}(t)+\sqrt{\frac{\text{Clog}( n_{i}(t))}{n_{i,a(t)}}} \tag{18}\]  
where \(\overline{\mathbf{X}}_{i,a}(t)\) denotes the sample mean of rewards from action \(a\) for player \(i\) at time \(t\); \(\sqrt{\frac{\text{Clog}(n_{i}(t))}{ni,a(t)}}\) represents the exploration term, where \(n_{i}(t)\) denotes the number of times that player \(i\) plays the game in frame \(t\); and \(n_{i,a}(t)\) denotes the number of times that player \(i\) selects action \(a\) up to time \(t\). The update of the estimated action value for the multi-armed bandit (MAB), denoted as \(\overline{X}_{i,a}(t)\), is calculated using the following formula:  
\[\overline{\mathbf{X}}_{i,a}(t)=\overline{X}_{i,a}(t)+(1/n_{i,a}(t))*[R_{i,a}(t )-\overline{X}_{i,a}(t)] \tag{19}\]The D-UCB algorithm is employed to select the corresponding action, and the design is articulated as follows:  
\[A_{i}(t)\equiv\left\{ \begin{aligned} & argmax(\textbf{D-UCB}_{i,a})\\ & randomly\ choose\ untried\ arm\ A\end{aligned} \right. \tag{20a}\]  
If all arms have been tried, the agent will follow (20a) to select the arm; otherwise, it will follow (20b). After selecting action \(A\) at time \(t\) and obtaining its corresponding reward \(\overline{X}_{i,a}\), the average achievable data rate \(\textbf{E}[\overline{X}_{i,a}]\) and selection count \(n_{i,a}(t)\) are updated in steps 11 and 12 of Algorithm 1. ```
1:Input: Number of agents N and arms A. 2:Initialization: Initialize the following variables:
3:for\(i=1\) to \(N\)do
4: Initialize array \(\overline{X}_{i,a}\), \(\textbf{n}_{i,a}\) and \(\textbf{D-UCB}_{i,a}\) (initialize to 0 for all arms)
5:endfor
6: Choose exploration parameter C = 2
7:for t = 1 to T do
8:for i = 1 to N do
9: Select the arm following the rules from Equation (20)
10: Execute arm \(A_{i}(t)\) and observe the reward \(R_{i,A}(t)\), where \(R_{i,A}(t)\) is obtained from the inner loop Algorithm 2
11: Update estimated mean reward \(\overline{X}_{i,A}(t)\) for the selected arm \(A_{i}(t)\) using Equation (19)
12: Update the selection number for arm \(A_{i}(t)\):
13:\(n_{i,A}=n_{i,A}+1\)
14: Calculate the \(D-UCB_{i,A}\) index using (18)
15:endfor
```  
**Algorithm 1** D-UCB Algorithm  
_Exploitation phase:_ After the exploration phase, which may occur after a certain time or a specified number of rounds, players transition to the exploitation phase. Each player \(i\) selects an arm a that maximizes the estimated mean reward, i.e., selects arm \(a^{*}=\text{argmax}(\overline{X}_{i,a}(t))\) among all available arms a.