## 4 Multi-Armed Bandit (MAB) for Optimal MPTCP Scheduling  
In this section, we first provide a short background of MAB and then employ MAB to model and solve the scheduling problem (11). Next, we improve MAB's efficiency with federated learning and opportunistic scheduling. MAB employs a sequential decision approach in which an actor must decide which arms of the bandits to draw in each epoch [27]. Arm draws are MAB's actions, and after every execution of the action, a reward (payoff) is to be observed. The goal of the MAB approach is to facilitate the scheduling at the TRA-Layer framework design for progressive decisions to maximize the overall reward earned through a series of actions during transmissions of packets. Due to the limited knowledge, the actor often needs to balance the exploitation of past acts that worked well and investigate/explore future acts that may provide more significant rewards [28]. The scheduling problem (11) can be modeled as a MAB with the multipath scheduler and links acting as the actor and arms, respectively. The selected subset of arms \(\mathbb{S}\) is referred to as a super arm, and the appropriate normalized reward is \(\mathbf{R}(\mathbf{w})=-\frac{1}{T(\mathbf{w})\mathbf{w}_{\in\mathbb{S}}}+1\in[0,1)\). As a result, the goal of (8) may be viewed as determining a temporal sequence of actions based on a specific policy that maximizes the cumulative reward, i.e.,  
\[\max_{\mathbf{w}\in\mathbb{S}}\mathbf{R}(\mathbf{w})\text{ }s.t.\text{ }\sum_{i}w_{t}=W. \tag{13}\]  
To this end, we quantify the action selection policies for the scheduler in terms of regret. In particular, the regret is estimated as the difference between the predicted reward of optimum actions (equilibrium windows, \(\mathbf{w}^{\star}\)) and the reward achieved by the proposed strategy (given windows, \(\mathbf{w}\)). It is worth noting that minimizing regret is the same as maximizing the overall gain. Let \(R_{i}(t)\) denote the normalized reward, then its expected reward of link \(t\) (arm \(t\)), \(\bar{R}_{t}\), can be estimated as  
\[\bar{R}_{t}=\mathbb{E}\Big{[}1-\frac{\Big{(}D_{i}(t)+\sum_{m=1}^{\text{tr}_{i}(t )}P_{m}^{t}(t)\Big{)}}{T(\mathbf{w}(t))}\Big{]}, \tag{14}\]  
and thus the desired optimal super arm will be  
\[\mathbb{S}^{\star}=_{\mathbb{S}\subset\mathcal{I},|\mathbb{S}|=n}\{\min_{t \in\mathbb{S}}\bar{R}_{t}\}. \tag{15}\]  
Given policy \(\pi\), the cumulative regret up to time \(\mathcal{T}\) can then be computed as  
\[\text{Regret}(\pi)=\mathcal{T}\bar{R}_{t}(\mathbb{S}^{\star})-\mathbb{E}\Big{[} \sum_{t=1}^{\mathcal{T}}\mathbf{R}(\mathbb{S}(t))\Big{]}, \tag{16}\]  
thus the multipath scheduling problem can be reformulated as  
\[\min_{\mathbb{S}(t),t\geq 1}\text{Regret}(\pi(\mathbb{S})). \tag{17}\]  
With relevant insights from [27; 28; 31], using (17), we have developed and implemented multipath scheduling based on _upper confidence bound policy_, referred to as MS-UCB, as shown in Algorithm 2. ```
1:procedureInitialize
2:while\(t=1,2\ldots\)do
3: Random Selection: Superarm \(\mathbb{S}(t)\) and \(|\mathbb{S}(t)|=n\)
4: Compute \(\mathbf{u}(t)\) and \(\mathbf{v}(t)\)
5:endwhile
6:endprocedure
7:procedureUpdate
8:while\(((\mathcal{T}-t)^{2}>\mathit{tolerance},\ \epsilon)\)do
9: Renew \(\mathbf{u}(t)\) and \(\mathbf{v}(t)\) using (14) and (15)
10: Compute \(\mathbb{S}(t+1)\) using (16)
11: Increment \(t\gets t+1\);
12:endwhile
13:endprocedure
```  
**Algorithm 2** MS-UCB multipath scheduling algorithm  
At a higher level, the _feedback observation module_ for each arm/path is at the core of this algorithm (instead of every super arm or action as a whole). The same arm might be seen with respect to multiple actions, and in order to make better recommendations in the future, we gather and use the experience of the rewards of one arm from several actions.