## 4 Multi-Armed Bandit (MAB) for Optimal MPTCP Scheduling
### With Federated Learning  
Looking ahead, we consider a decentralized MAB problem to resolve multiple subflows scheduling from \(x\) Internet users using the link/arm \(t\) who can communicate their data, in particular their local updates \((\mathbf{u}(t),\mathbf{v}(t))\), exclusively with their neighboring multi-path TCP users using the same link. Without FL, each user has a series of choices about picking the \(n\) arms/links but only has access to local feedback/evaluations of the actual reward/regret for all actions made. In addition, they are often biased due to the skewness of the underlying distributions, as reported in [29]. Such dependence only on local learning often leads to suboptimal behaviors because of unforeseen network environments and dynamics, whereas converging on a decentralized approach requires data sharing from neighbors. Motivated by the potential of FL and with relevant insights from [29], we propose to provide congestion information to new joining MPTCP users during the handshake process. In particular, we seek a solution that will allow multipath schedulers to share local experiences \((\mathbf{u}^{x}(t),\mathbf{v}^{x}(t))\) in the network, which can be shared as a private copy of the learned model with the neighbors.  
Different from Algorithm 2 (step 10), the multipath scheduler enters the main while loop to choose super arm \(\mathbb{S}(t+1)\), after every transmission round by  
\[\mathbb{S}(t+1) = \mathbb{S}_{\subset\mathcal{I},|\mathbb{S}|=n}\sum_{t\in \mathbb{S}}\Bigg{(}\underbrace{\sqrt{(n+1)\frac{\log(t+1)}{\bar{u}_{i}(t)}}}_{ \text{exploration}}+\underbrace{v_{i}(t+1)}_{\text{exploitation}}\Bigg{)}.\]  
where we use the _renewal reward theorem_ to compute the exploration and exploitation based on the data sets from all multipath TCP users sharing the arm/path \(t\). Considering \(P(x)\) as the stationary distribution of the underlying Markovian Process, \(\{(X_{t})\}\) for tracking the evolution of multiple multipath users sharing link \(t\), with a slight abuse of notation, (we take \(P(x)=1/x\) for simplification in our numerical analysis):  
\[\bar{u}_{i}(t) = \lim_{x\to\infty}\frac{\mathcal{U}_{x,i}(t)}{T_{x+1}}\approx \frac{\sum\limits_{x}P(x)u_{i}^{x}(t)}{\sum\limits_{x}P(x)x}; \tag{21}\] \[\bar{v}_{i(t+1)} = \lim_{x\to\infty}\frac{\mathcal{V}_{x,i}(t)}{T_{x+1}}\approx \frac{\sum\limits_{x}P(x)v_{x(t)}}{\sum\limits_{x}P(x)x}. \tag{22}\]  
**Remark 3** (Opportunistic Scheduling).: _The stochastic delay-aware scheduling proposed above allocates packet transmissions for an application amongst the active links so as to minimize the average delay. Moreover, it ensures with a high probability a multipath connection (with \(W\) packets) that guarantees low delay and high speed. However, we may sometimes be fortunate when the delays experienced on one or more links happen to be lower than usual. Observe that a different realization of random delay might also lead to scheduling on links yielding lower delays. Of course, we will discover the delay realization only after transmitting the packets; however, by transmitting additional redundant packets (packet cloning) over and above the minimum number \(W\), we can still opportunistically reduce the delivery delay by exploiting link delay realizations when they occur. Observe that we are trading off the link capacity (as we send more packets) for the likelihood of lower delay._  
Next, we explain our idea of cloning to jointly (i) consider erratic PHY/MAC errors into stochastic models, and (ii) perform opportunistic scheduling.