# A Reinforcement-Learning-Based Distributed Resource Selection Algorithm for Massive IoT
## 4 TOW Dynamics-Based Strategy  
Many algorithms [15; 16; 17; 18] have been proposed to solve the MAB problem. In this paper, the strategy used to solve the distributed channel selection problem is referred to the tug-of-war (TOW) dynamics [9; 10; 11; 12]. Despite the simplicity, the high efficiency of TOW dynamics has been analytically validated in making a series of decisions for maximizing the total sum of stochastically-obtained rewards in an environment where the reward probability frequently changes [9; 10; 11; 12]. The main methodology of [9; 10; 11; 12] is summarized below. The essential element of the TOW dynamics is the consideration of a volume-conserving physical object, e.g., the incompressible liquid (blue) is assumed in a branched cylinder shown in Figure 3, which implies a non-local correlation between the terminal parts. Specifically, the volume increase in one part is immediately compensated by the volume decrease in the other part(s). Here, the \(n\)-machine MAB is considered. For each machine (branch) \(k\) at time \(t\), let \(X_{k}(t)\) correspond to the displacement of machine \(k\). Thus, the reward estimates of each machine can be obtained by Equation (1),  
\[Q_{k}(t)=N_{k}(t)-(1+\omega)L_{k}(t)= Q_{k}(t-1)+\Delta Q_{k}(t) \tag{1}\]  
where \(k\in(1,...,n)\). Here, \(N_{k}\) denotes the accumulated counts of selections of machine \(k\), while \(L_{k}\) denotes non-rewarded counts of machine \(k\) until time \(t\). \(\Delta\)\(Q_{k}(t)\) is given by Equation (2). \[\Delta Q_{k}(t)= \begin{cases}+1&\text{if rewarded}\\ -\omega&\text{if non-rewarded}\end{cases}  \tag{2}\]  
Accordingly, based on the conservation laws in TOW dynamics, the displacement of machine \(k\) can be estimated by Equation (3),  
\[X_{k}(t+1)=Q_{k}(t)-\frac{1}{n-1}\sum_{k^{\prime}\
eq k}Q_{k^{ \prime}}(t)+osc_{k}(t) \tag{3}\]  
where \(n\) is the total number of arms. The incompressible liquid oscillates autonomously according to Equation (4) [12]. There are many possibilities of adding the oscillations. In [11], the influence of \(osc\) on the efficiency of decision making as studied, which is outside of the scope of this paper. For the sake of simplicity, we used \(A=0.5\) for all machines in this paper. \[osc_{k}(t)=Acos(2\pi t/n+2(k-1)\pi/n) \tag{4}\]  
Figure 3: tug-of-war (TOW) dynamics for \(n\)-machine MAB. The estimated reward probability, denoted as \(p_{k}(t)\), can be obtained by Equation (5),  
\[p_{k}(t)=\frac{R_{k}(t)}{N_{k}(t)}. \tag{5}\]  
where \(R_{k}(t)=N_{k}(t)-L_{k}(t)\) counts the number of times of getting rewards when machine \(k\) is played \(N_{k}\) times until time \(t\). Consequently, the TOW dynamics-based strategy evolves according to a particular simple rule: if machine \(k\) is played at each time \(t\), \(+1\) or \(-\omega\) is added to \(X_{k}(t+1)\) when rewarded and non-rewarded, respectively. To explore the appropriate weight parameter of \(\omega\), the expected value of reward estimates of machine \(k\) can be obtained by Equation (6). \[\mathbb{E}[Q_{k}(t)]=\{p_{k}-\omega(1-p_{k})\}N_{k} \tag{6}\]  
Here, the highest and the second highest estimated reward probability among n machines can be obtained given by Equation (5), denoted as \(p_{1 st}\) and \(p_{2 nd}\), respectively. Accordingly, the highest and the second highest expected value of reward estimates among n machines can be obtained by Equation (6). To achieve always selecting the machine with the highest reward estimates, the following forms could be expressed,  
\[p_{1 st}-\omega(1-p_{1 st})>0 \tag{7}\] \[p_{2 nd}-\omega(1-p_{2 nd})<0\]  
and these expressions can be rearranged into the form:  
\[p_{1 st}<\frac{\omega}{1+\omega}<p_{2 nd} \tag{8}\]  
In other words, the weight parameter \(\omega\) should satisfy the above condition Inequality (8) so that the selection correctly represents the largest reward estimates. One way of ensuring Inequality (8) is to take an \(\omega\) that satisfies Equation (9).