## 4 DG-Moss
### DG-MOSS Algorithm Description  
DG-MOSS in Algorithm 3 uses dynamic grouping to ensure the balance between exploration and exploitation. In any particular environment, DG-MOSS needs to ensure a balance between exploration and exploitation. Whether the effect of exploration is better or worse than that of exploitation depends on the accurate value of the estimation, the uncertainty of the environment, and the number of remaining operations. Therefore, we use multiple feedback information to reduce the dependence of the algorithm on the data context. Aiming at balancing between exploration and exploitation, we propose a new exploration calculation method based on dynamic grouping in Algorithm 3 with some details explained below. ```
1:Select each arm \(k\) in the set \(A\) once of \(K\) possible arms. 2:episode number \(i=1\), episode length \(T_{i}=10\), number of iterations \(t=0\). 3:while\(j<T_{i}:\)do
4:\(|A_{k}|=0\)
5:for each arm \(k\) in \(A\):do
6:\(mis=\sqrt{\frac{3\log\left(S\right)}{s}}\)
7:if\(\left|X_{k-history}-X_{k-cur}\right|<=mis\)then
8:\(A_{k}\gets A_{k}\cup arm_{k}\)
9:endif
10:endfor
11:if\(|A_{k}|>=P_{1}K\)then
12: action \(\leftarrow\) select best arm in A
13:else
14:if already explore every arm then
15:\(V_{k,s}=\hat{X}_{k,s}+c*\sqrt{\frac{max\left(\log\frac{T}{k_{s}}\right)}{s}}\)
16:\(action\gets argmax(V_{k,s})+1\)
17:else
18: action \(\leftarrow\) select the arm with \(s=0\)
19:endif
20:endif
21:\(T_{i}=8\log\left(\frac{T*(T+1)^{2}}{(avg-mis)^{2}}\right)\)
22:\(t=t+T_{i-1}\)
23:endwhile
24:\(X_{k-history},X_{k-cur}\)
```  
**Algorithm 3** DG-MOSS  
Steps 2\(\sim\)8 are dynamic grouping processes. Steps 9\(\sim\)18 are the action selection processes, which keep the balance between exploration and exploitation. Steps 10, 14, and 16 represent three different action sections. Step 1 utilizes T to control the training process. In step 2, we use an array \(A_{k}\) to measure the current value of each arm. In step 4, \(mis\) is the threshold between the total number of arm pulls \(S\) and the number of the current arm pulled \(s\). In step 5, the absolute value is used to save the allowable error between the true reward and the expected value of each round in selecting an arm, and it is lower than the corresponding value in the _mis_. During each round, \(A_{k}\) records the error between the true reward and expected value of this round in arm pulling. In step 6, only arms within the allowable error are added to achieve grouping. The allowable error is defined as the error between the true reward and the expected value of the current round in arm pulling. Each arm is calculated separately at the end of the arm selection process. Steps 9\(\sim\)10 are to select the arm with the maximum reward value from A using the dynamic array \(A_{k}\). Steps 12\(\sim\)14 are to select the arm with the maximum reward value from \(V_{k,s}\). If steps 9\(\sim\)14 are not executed, then step 16 selects the arm with s = 0. Then, we choose exploration or exploitation based on whether the number of values in the array is greater than a certain threshold at this time. The threshold is related to the number of arms and can be regarded as a fixed parameter. The probability parameter \(P_{1}\) is a selectable probability related to \(K\). In the experiment of DG-MOSS, \(0<P_{1}<1\). Sufficient training is required for the full utilization of feedback information; hence, the reasonable reduction in training episodes should be taken into consideration. In Section 4.3, we designed the idea of \(T_{i}\) derived from the inference of the upper bound of the DG-MOSS regret, including \(f(u)\) and Equations (13) and (20). Based on the function \(f(u)=8\log\left(\sqrt{\frac{T}{K}}u\right)/u^{2}\), we introduce the total time step \(T\), grouping parameter _mis_, and feedback information \(avg\) to jointly affect the episodes in the calculation of the \(8\log\) confidence interval width.