SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chi-Jen Lu
Institute of Information Science
Academia Sinica, Taiwan
cjlu@iis.sinica.edu.tw
We study the best arm identification problem in combinatorial semi-bandits in the fixed confidence setting. We present Perturbed Frank-Wolfe Sampling (P-FWS), an algorithm that (i) runs in polynomial time, (ii) achieves the instance-specific minimal sample complexity in the high confidence regime, and (iii) enjoys polynomial sample complexity guarantees in the moderate confidence regime. To the best of our knowledge, even for the vanilla bandit problems, no algorithm was able to achieve (ii) and (iii) simultaneously. With P-FWS, we close the computational-statistical gap in best arm identification in combinatorial semi-bandits. The design of P-FWS starts from the optimization problem that defines the information-theoretical and instance-specific sample complexity lower bound. P-FWS solves this problem in an online manner using, in each round, a single iteration of the Frank-Wolfe algorithm. Structural properties of the problem are leveraged to make the P-FWS successive updates computationally efficient. In turn, P-FWS only relies on a simple linear maximization oracle.
An efficient method to design statistically optimal algorithms solving active learning tasks (e.g., regret minimization or pure exploration in bandits and reinforcement learning) consists in the following two-step procedure. The first step amounts to deriving, through change-of-measure arguments, tight information-theoretical fundamental limits satisfied by a wide class of learning algorithms. These limits are often expressed as the solution of an optimization problem, referred in this paper to as the lower-bound problem. Interestingly, this solution specifies the instance-specific optimal exploration process: it characterizes the limiting behavior of the adaptive sampling rule any statistically optimal algorithm should implement. In the second step, the learning algorithm is designed so that its exploration process approaches the solution of the lower-bound problem. This design yields statistically optimal algorithms, but typically requires to repeatedly solve the lower-bound problem. This method has worked remarkably well for simple learning tasks such as regret minimization or best-arm identification with fixed confidence in classical stochastic bandits [Lai87, GC11, GK16], but also in bandits whose arm-to-average reward function satisfies simple structural properties (e.g., Lipschitz, unimodal) [MCP14, WTP21].
The method also provides a natural way of studying the computational-statistical gap [KLLM22] for active learning tasks. Indeed, if solving the lower-bound problem in polynomial time is possible, one
may hope to devise learning algorithms that are both statistically optimal and computationally efficient. As of now, however, the computational complexity of the lower-bound problem remains largely unexplored, except for simple learning tasks. For example, in the case of best policy identification in tabular Markov Decision Processes, the lower-bound problem is non-convex [AMP21] and its complexity and approximability are unclear.
In this paper, we leverage the aforementioned two-step procedure to assess the computational-statistical gap for the best arm identification in combinatorial semi-bandits in the fixed confidence setting. We establish that, essentially, this gap does not exist (a result that was conjectured in [JMKK21]). Specifically, we present an algorithm that enjoys the following three properties: (i) it runs in polynomial time, (ii) its sample complexity matches the fundamental limits asymptotically in the high confidence regime, and (iii) its sample complexity is at most polynomial in the moderate confidence regime. Next, after formally introducing combinatorial semi-bandits, we describe our contributions and techniques in detail.
Best arm identification in combinatorial semi-bandits. In combinatorial semi-bandits [CBL12, CTMSP+15], the learner sequentially selects an action from a combinatorial set  $\mathcal{X} \subset \{0,1\}^K$ . When in round  $t$ , the action  $\boldsymbol{x}(t) = (x_1(t), \ldots, x_K(t)) \in \mathcal{X}$  is chosen, the environment samples a  $K$ -dimensional vector  $\boldsymbol{y}(t)$  whose distribution is assumed to be Gaussian  $\mathcal{N}(\boldsymbol{\mu}, \mathbf{I})$ . The learner then receives the detailed reward vector  $\boldsymbol{x}(t) \odot \boldsymbol{y}(t)$  where  $\odot$  denotes the element-wise product (in other words, the learner observes the individual reward  $y_k(t)$  of the arm  $k$  if and only if this arm is selected in round  $t$ , i.e.,  $x_k(t) = 1$ ). The parameter  $\boldsymbol{\mu}$  characterizing the average rewards of the various arms is initially unknown. The goal of a learner is to identify the best action  $\boldsymbol{i}^\star(\boldsymbol{\mu}) = \operatorname*{argmax}_{\boldsymbol{x} \in \mathcal{X}} \langle \boldsymbol{x}, \boldsymbol{\mu} \rangle$  with a given level of confidence  $1 - \delta$ , for some  $\delta > 0$  while minimizing the expected number of rounds needed. We assume that the best action is unique and denote by  $\Lambda = \{\boldsymbol{\mu} \in \mathbb{R}^K : |\boldsymbol{i}^\star(\boldsymbol{\mu})| = 1\}$  the set of parameters satisfying this assumption. The learner strategy is defined by three components: (i) a sampling rule dictating the sequence of the selected actions; (ii) a stopping time  $\tau$  defining the last round where the learner interacts with the environment; (iii) a decision rule specifying the action  $\hat{\boldsymbol{u}} \in \mathcal{X}$  believed to be optimal based on the data gathered until  $\tau$ .
The sample complexity lower-bound problem. Consider the set of  $\delta$ -PAC algorithms such that for any  $\mu \in \Lambda$ , the best action is identified correctly with probability at least  $1 - \delta$ . We wish to find a  $\delta$ -PAC algorithm with minimal expected sample complexity  $\mathbb{E}_{\mu}[\tau]$ . To this aim, using classical change-of-measure arguments [GK16], we may derive a lower bound of the expected sample complexity satisfied by any  $\delta$ -PAC algorithm. This lower bound is given by  ${}^{1}\mathbb{E}_{\mu}[\tau] \geq T^{\star}(\mu)\mathrm{kl}(\delta, 1 - \delta)$ . The characteristic time  $T^{\star}(\mu)$  is defined as the value of the following problem
$$
T ^ {\star} (\boldsymbol {\mu}) ^ {- 1} = \sup  _ {\boldsymbol {\omega} \in \Sigma} \inf  _ {\boldsymbol {\lambda} \in \operatorname {A l t} (\boldsymbol {\mu})} \left\langle \boldsymbol {\omega}, \frac {(\boldsymbol {\mu} - \boldsymbol {\lambda}) ^ {2}}{2} \right\rangle , \tag {1}
$$
where  $\Sigma = \{\sum_{\boldsymbol{x} \in \mathcal{X}} w_{\boldsymbol{x}} \boldsymbol{x} : \boldsymbol{w} \in \Sigma_{|\mathcal{X}|}\}$ ,  $\mathrm{kl}(a, b)$  is the KL-divergence between two Bernoulli distributions with respective means  $a$  and  $b$ , and  $\mathrm{Alt}(\boldsymbol{\mu}) = \{\lambda \in \Lambda : i^{\star}(\boldsymbol{\lambda}) \neq i^{\star}(\boldsymbol{\mu})\}$  is the set of confusing parameters. As it turns out (see Lemma 1),  $T^{\star}(\boldsymbol{\mu})$  is at most quadratic in  $K$ , and hence the sample complexity lower bound is polynomial. (1) is a concave program over  $\Sigma$  [WTP21], and a point  $\omega^{\star}$  in its solution set corresponds to an optimal allocation of action draws: an algorithm sampling actions according to  $\omega^{\star}$  and equipped with an appropriate stopping rule would yield a sample complexity matching the lower bound. In this paper, we provide computationally efficient algorithms to solve (1) and show how these can be used to devise a  $\delta$ -PAC best action identification algorithm with minimal sample complexity and running in polynomial time. We only assume that we have access to a computationally efficient Oracle, referred to as the LM (Linear Maximization) Oracle, identifying the best action should  $\boldsymbol{\mu}$  be known (but for any possible  $\boldsymbol{\mu}$ ). This assumption, made in all previous work on combinatorial semi-bandits (see e.g. [JMKK21, PBVP20]), is crucial as indeed, if there is no computationally efficient algorithm solving the offline problem  $\operatorname{argmax}_{\boldsymbol{x} \in \mathcal{X}}\langle \boldsymbol{x}, \boldsymbol{\mu} \rangle$  with known  $\boldsymbol{\mu}$ , there is no hope to solve its online version with unknown  $\boldsymbol{\mu}$  in a computationally efficient manner. The assumption holds for a large array of combinatorial sets of actions  $[\mathrm{S}^{+}03]$ , including  $m$ -sets, matchings, (source-destination)-paths, spanning trees, matroids (refer to [CCG21b] for a thorough discussion).
<sup>1</sup>We present proof in Appendix K for completeness – see also [JMKK21].
$^2\Sigma_N$  denotes the  $(N - 1)$  dimensional simplex.
The Most-Confusing-Parameter (MCP) algorithm. The difficulty of solving (1) lies in the inner optimization problem, i.e., in evaluating the objective function:
$$
F _ {\boldsymbol {\mu}} (\omega) = \inf  _ {\boldsymbol {\lambda} \in \operatorname {A l t} (\boldsymbol {\mu})} \left\langle \omega , \frac {(\boldsymbol {\mu} - \boldsymbol {\lambda}) ^ {2}}{2} \right\rangle = \min  _ {\boldsymbol {x} \neq i ^ {*} (\boldsymbol {\mu})} f _ {\boldsymbol {x}} (\omega , \boldsymbol {\mu}) \tag {2}
$$
where  $f_{\pmb{x}}(\pmb{\omega}, \pmb{\mu}) = \inf_{\pmb{\lambda} \in \mathcal{C}_{\pmb{x}}} \langle \pmb{\omega}, \frac{(\pmb{\mu} - \pmb{\lambda})^2}{2} \rangle$  and  $\mathcal{C}_x = \{\pmb{\lambda} \in \mathbb{R}^K : \langle \pmb{\lambda}, i^\star(\pmb{\mu}) - \pmb{x} \rangle < 0\}$ . Evaluating  $F_{\pmb{\mu}}(\pmb{\omega})$  is required to solve (1), but also in the design of an efficient stopping rule. Our first contribution is MCP (Most-Confusing-Parameter), a polynomial time algorithm able to approximate  $F_{\pmb{\mu}}(\pmb{\omega})$  for any given  $\pmb{\mu}$  and  $\pmb{\omega}$ . The algorithm's name refers to the fact that by computing  $F_{\pmb{\mu}}(\pmb{\omega})$ , we implicitly identify the most confusing parameter  $\lambda^\star \in \arg \inf_{\pmb{\lambda} \in \mathrm{Alt}(\pmb{\mu})} \langle \pmb{\omega}, \frac{(\pmb{\mu} - \pmb{\lambda})^2}{2} \rangle$ . The design of MCP leverages a Lagrangian relaxation of the optimization problem defining  $f_{\pmb{x}}(\pmb{\omega}, \pmb{\mu})$  and exploits the fact that the Lagrange dual function linearly depends on  $x$ . In turn, this linearity allows us to make use of the LM Oracle. From these observations, we show that computing  $F_{\pmb{\mu}}(\pmb{\omega})$  boils down to solving a two-player game, for which one of the players can simply update her strategy using the LM Oracle. We prove the following informally stated theorem quantifying the performance of the MCP algorithm (see Theorem 3 for a more precise statement).
Theorem 1. For any  $(\omega, \mu)$ , the MCP algorithm with precision  $\epsilon$  and certainty parameter  $\theta$  returns  $\hat{F}$  and  $\hat{\mathbf{x}}$  satisfying  $\mathbb{P}_{\mu}[F_{\mu}(\omega) \leq \hat{F} \leq (1 + \epsilon) F_{\mu}(\omega)] \geq 1 - \theta$  and  $\hat{F} = f_{\hat{\mathbf{x}}}(\omega, \mu)$ . The number of calls to the LM Oracle is, almost surely, at most polynomial in  $K$ ,  $\epsilon^{-1}$ , and  $\ln \theta^{-1}$ .
The Perturbed Frank-Wolfe Sampling (P-FWS) algorithm. The MCP algorithm allows us to solve the lower-bound problem (1) for any given  $\mu$ . The latter is initially unknown, but could be estimated. A Track-and-Stop algorithm [GK16] solving (1) with this plug-in estimator in each round would yield asymptotically minimal sample complexity. It would however be computationally expensive. To circumvent this difficulty, as in [WTP21], our algorithm, P-FWS, performs a single iteration of the Frank-Wolfe algorithm for the program (1) instantiated with an estimator of  $\mu$ . To apply the Frank-Wolfe algorithm, P-FWS uses stochastic smoothing techniques to approximate the non-differentiable objective function  $F_{\mu}$  by a smooth function. To estimate the gradient of the latter, P-FWS leverages both the LM Oracle and the MCP algorithm (more specifically its second output  $\hat{\pmb{x}}$ ). Finally, P-FWS stopping rule takes the form of a classical Generalized Likelihood Ratio Test (GLRT) where the estimated objective function is compared to a time-dependent threshold. Hence the stopping rule also requires the MCP algorithm. We analyze the sample and computational complexities of P-FWS. Our main results are summarized in the following theorem (refer to Theorem 4 for details).
Theorem 2. For any  $\delta \in (0,1)$ , P-FWS is  $\delta$ -PAC, and for any  $(\epsilon, \tilde{\epsilon}) \in (0,1)$  small enough, its sample complexity satisfies:
$$
\mathbb {E} _ {\boldsymbol {\mu}} [ \tau ] \leq \frac {(1 + \tilde {\epsilon}) ^ {2}}{T ^ {\star} (\boldsymbol {\mu}) ^ {- 1} - \epsilon} \times H \left(\frac {1}{\delta} \cdot \frac {c (1 + \tilde {\epsilon}) ^ {2}}{T ^ {\star} (\boldsymbol {\mu}) ^ {- 1} - \epsilon}\right) + \Psi (\epsilon , \tilde {\epsilon}),
$$
where  $H(x) = \ln (x) + \ln \ln (x)$ ,  $c > 0$  is a universal constant, and  $\Psi (\epsilon ,\tilde{\epsilon})$  is polynomial in  $\epsilon^{-1}$ ,  $\tilde{\epsilon}^{-1}$ ,  $K$ ,  $\| \pmb {\mu}\|_{\infty}$ , and  $\triangle_{\mathrm{min}}^{-1}$ , where  $\triangle_{\mathrm{min}} = \min_{\pmb {x}\neq \pmb{i}^{\star}(\pmb {\mu})}\langle \pmb{i}^{\star}(\pmb {\mu}) - \pmb {x},\pmb {\mu}\rangle$ . Under P-FWS, the number of LM Oracle calls per round is at most polynomial in  $\ln \delta^{-1}$  and  $K$ . The total expected number of these calls is also polynomial.
To the best of our knowledge, P-FWS is the first polynomial time best action identification algorithm with minimal sample complexity in the high confidence regime (when  $\delta$  tends to 0). Its sample complexity is also polynomial in  $K$  in the moderate confidence regime.
We start by introducing some notation. We use bold lowercase letters (e.g.,  $\pmb{x}$ ) for vectors, and bold uppercase letter (e.g.,  $\mathbf{A}$ ) for matrices.  $\odot$  (resp.  $\oplus$ ) denotes the element-wise product (resp. sum over  $\mathbb{Z}_2$ ). For  $\pmb{x} \in \mathbb{R}^K$ ,  $i \in \mathbb{N}$ ,  $\pmb{x}^i = (x_k^i)_{k \in [K]}$  is the  $i$ -th element-wise power of  $\pmb{x}$ .  $D = \max_{\pmb{x} \in \mathcal{X}} \| \pmb{x} \|_1$  denotes the maximum number of arms part of an action. For any  $\pmb{\mu} \in \Lambda$ , we define the sub-optimality gap of  $\pmb{x} \in \mathcal{X}$  as  $\triangle_{\pmb{x}}(\pmb{\mu}) = \langle i^{\star}(\pmb{\mu}) - \pmb{x}, \pmb{\mu} \rangle$ , and the minimal gap as  $\triangle_{\min}(\pmb{\mu}) = \min_{\pmb{x} \neq i^{\star}(\pmb{\mu})} \triangle_{\pmb{x}}(\pmb{\mu})$ .  $\mathbb{P}_{\mu}$  (resp.  $\mathbb{E}_{\mu}$ ) denotes the probability measure (resp. expectation) when the arm rewards are parametrized by  $\pmb{\mu}$ . Whenever it is clear from the context, we will drop  $\pmb{\mu}$  for simplicity, e.g.  $i^{\star} = i^{\star}(\pmb{\mu})$ ,  $\triangle_{\pmb{x}} = \triangle_{\pmb{x}}(\pmb{\mu})$ , and  $\triangle_{\min} = \min_{\pmb{x} \neq i^{\star}} \triangle_{\pmb{x}}$ . Refer to Appendix A for an exhaustive table of notation.
2.1 The lower-bound problem
Classical change-of-measure arguments lead to the asymptotic sample complexity lower bound  $\mathbb{E}_{\pmb{\mu}}[\tau] \geq T^{\star}(\pmb{\mu}) \mathrm{kl}(\delta, 1 - \delta)$  where the characteristic time is defined in (1). To have a chance to develop a computationally efficient best action identification algorithm, we need that the sample complexity lower bound grows at most polynomially in  $K$ . This is indeed the case as stated in the following lemma, whose proof is provided in Appendix K.
Lemma 1. For any  $\pmb{\mu} \in \Lambda$ ,  $T^*(\pmb{\mu}) \leq 4KD\triangle_{\min}(\pmb{\mu})^{-2}$ .
We will use first-order methods to solve the lower-bound problem, and to this aim, we will need to evaluate the gradient w.r.t.  $\omega$  of  $f_{\pmb{x}}(\omega ,\pmb {\mu})$ . We can apply the envelop theorem [WTP21] to show that for  $(\omega ,\mu)\in \Sigma_{+}\times \Lambda$
$$
\nabla_ {\omega} f _ {x} (\omega , \mu) = \frac {(\mu - \lambda_ {\omega , \mu} ^ {\star} (x)) ^ {2}}{2},
$$
where  $\Sigma_{+} = \Sigma \cap \mathbb{R}_{>0}^{K}$ ,  $\lambda_{\omega, \mu}^{\star}(\boldsymbol{x}) = \operatorname*{argmin}_{\boldsymbol{\lambda} \in \mathrm{cl}(\mathcal{C}_x)} \langle \boldsymbol{\omega}, \frac{(\boldsymbol{\mu} - \boldsymbol{\lambda})^2}{2} \rangle$  and  $\operatorname{cl}(\mathcal{C}_x)$  is the closure of  $\mathcal{C}_x$  (refer to Lemma 19 in Appendix G.2).
2.2 The Linear Maximization Oracle
As mentioned earlier, we assume that we have access to a computationally efficient Oracle, referred to as the LM (Linear Maximization) Oracle, identifying the best action if  $\mu$  is known. More precisely, as in existing works in combinatorial semi-bandits [KWA+14, PPV19, PBVP20], we make the following assumption.
Assumption 1. (i) There exists a polynomial-time algorithm identifying  $\pmb{i}^{\star}(\pmb{v})$  for any  $\pmb{v} \in \mathbb{R}^{K}$ ; (ii)  $\mathcal{X}$  is inclusion-wise maximal, i.e., there is no  $\pmb{x}, \pmb{x}' \in \mathcal{X}$  s.t.  $\pmb{x} < \pmb{x}'$ ; (iii) for each  $k \in [K]$ , there exists  $\pmb{x} \in \mathcal{X}$  such that  $x_{k} = 1$ ; (iv)  $|\mathcal{X}| \geq 2$ .
Assumption 1 holds for combinatorial sets including  $m$ -sets, spanning forests, bipartite matching,  $s-t$  paths. For completeness, we verify the assumption for these action sets in Appendix J. In the design of our MCP algorithm, we will actually need to solve for some  $v \in \mathbb{R}^K$  the linear maximization problem  $\max \langle x, v \rangle$  over  $\mathcal{X} \setminus \{i^\star(\mu)\}$ ; in other words, we will probably need to identify the second best action. Fortunately, this can be done in a computationally efficient manner under Assumption 1. The following lemma formalizes this observation. Its proof, presented in Appendix J, is inspired by Lawler's  $m$ -best algorithm [Law72].
Lemma 2. Let  $\pmb{v} \in \mathbb{R}^{K}$  and  $\pmb{x} \in \mathcal{X}$ . Under Assumption 1, there exists an algorithm that solves  $\max_{\pmb{x}' \in \mathcal{X}: \pmb{x}' \neq \pmb{x}} \langle \pmb{v}, \pmb{x}' \rangle$  by only making at most  $D$  queries to the LM Oracle.
3 Solving the lower bound problem: the MCP algorithm
Solving the lower bound problem first requires to evaluate its objective function  $F_{\mu}(\omega)$ . A naive approach, enumerating  $f_{\pmb{x}}(\pmb{\omega}, \pmb{\mu})$  for all  $\pmb{x} \in \mathcal{X} \setminus \{i^{\star}\}$ , would be computationally infeasible. In this section, we present and analyze MCP, an algorithm that approximates  $F_{\mu}(\omega)$  by calling the LM Oracle a number of times growing at most polynomially in  $K$ .
3.1 Lagrangian relaxation
The first step towards the design of MCP consists in considering the Lagrangian relaxation of the optimization problem defining  $f_{\boldsymbol{x}}(\boldsymbol{\omega}, \boldsymbol{\mu}) = \inf_{\boldsymbol{\lambda} \in \mathcal{C}_{\boldsymbol{x}}} \langle \boldsymbol{\omega}, \frac{(\boldsymbol{\mu} - \boldsymbol{\lambda})^2}{2} \rangle$  (see e.g., [BV04, Vis21]). For any  $(\boldsymbol{\omega}, \boldsymbol{\mu}) \in \Sigma_+ \times \Lambda$  and  $\boldsymbol{x} \neq \boldsymbol{i}^\star$ , the Lagrangian  $\mathcal{L}_{\boldsymbol{\omega}, \boldsymbol{\mu}}$  and Lagrange dual function  $g_{\boldsymbol{\omega}, \boldsymbol{\mu}}$  of this problem are defined as,  $\forall \alpha \geq 0$ ,
$$
\mathcal {L} _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {\lambda}, \boldsymbol {x}, \alpha) = \left\langle \boldsymbol {\omega}, \frac {(\boldsymbol {\mu} - \boldsymbol {\lambda}) ^ {2}}{2} \right\rangle + \alpha \left\langle \boldsymbol {i} ^ {\star} - \boldsymbol {x}, \boldsymbol {\lambda} \right\rangle \quad \text {a n d} \quad g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x}, \alpha) = \inf  _ {\boldsymbol {\lambda} \in \mathbb {R} ^ {K}} \mathcal {L} _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {\lambda}, \boldsymbol {x}, \alpha),
$$
respectively. The following proposition, proved in Appendix C.1, provides useful properties of  $g_{\omega,\mu}$ :
Proposition 1. Let  $(\omega, \mu) \in \Sigma_{+} \times \Lambda$  and  $\pmb{x} \in \mathcal{X} \setminus \{\pmb{i}^{\star}(\mu)\}$ .
(a) The Lagrange dual function is linear in  $\pmb{x}$ . More precisely,  $g_{\omega,\mu}(\pmb{x},\alpha) = c_{\omega,\mu}(\alpha) + \langle \ell_{\omega,\mu}(\alpha),\pmb{x}\rangle$
where  $c_{\pmb{\omega}, \pmb{\mu}}(\alpha) = \alpha \left\langle \pmb{\mu} - \frac{\alpha}{2} \pmb{\omega}^{-1}, \pmb{i}^{\star}(\pmb{\mu}) \right\rangle$  and  $\ell_{\omega, \pmb{\mu}}(\alpha) = -\alpha \left( \pmb{\mu} + \frac{\alpha}{2} \pmb{\omega}^{-1} \odot (\mathbf{1}_K - 2\pmb{i}^{\star}(\pmb{\mu})) \right)$ .
(b)  $g_{\omega ,\mu}(\pmb {x},\cdot)$  is strictly concave (for any fixed  $\pmb{x}$
(c)  $f_{\pmb{x}}(\pmb{\omega}, \pmb{\mu}) = \max_{\alpha \geq 0} g_{\pmb{\omega}, \pmb{\mu}}(\pmb{x}, \alpha)$  is attained by  $\alpha_{\pmb{x}}^{\star} = \frac{\triangle_{\pmb{x}}(\pmb{\mu})}{\langle \pmb{x} \oplus \pmb{i}^{\star}(\pmb{\mu}), \pmb{\omega}^{-1} \rangle}$ .
(d)  $\| \ell_{\pmb{\omega},\pmb{\mu}}(\alpha_{\pmb{x}}^{\star})\|_{1}\leq L_{\pmb{\omega},\pmb{\mu}} = 4D^{2}K\| \pmb {\mu}\|_{\infty}^{2}\| \pmb{\omega}^{-1}\|_{\infty}.$
From Proposition 1 (c), strong duality holds for the program defining  $f_{x}(\omega, \mu)$ , and we conclude:
$$
F _ {\boldsymbol {\mu}} (\boldsymbol {\omega}) = \min  _ {\boldsymbol {x} \neq i ^ {*}} \max  _ {\alpha \geq 0} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x}, \alpha). \tag {3}
$$
$F_{\mu}(\omega)$  can hence be seen as the value in a two-player game. The aforementioned properties of the Lagrange dual function will help to compute this value.
3.2 Solving the two-player game with no regret
There is a rich and growing literature on solving zero-sum games using no-regret algorithms, see for example [RS13, ALLW18, DFG21, ZODS21]. Our game has the particularity that the  $\pmb{x}$ -player has a discrete combinatorial action set whereas the  $\alpha$ -player has a convex action set. Importantly, for this game, we wish not only to estimate its value  $F_{\mu}(\omega)$  but also an equilibrium action  $\pmb{x}_e$  such that  $F_{\mu}(\omega) = \max_{\alpha \geq 0} g_{\omega, \mu}(\pmb{x}_e, \alpha)$ . Indeed, an estimate of  $\pmb{x}_e$  will be needed when implementing the Frank-Wolfe algorithm and more specifically when estimating the gradient of  $F_{\mu}(\omega)$ . To return such an estimate, one could think of leveraging results from the recent literature on last-iterate convergence, see e.g. [DP19, GPDO20, LNP+21, WLZL21, APFS22, AAS+23]. However, most of these results concern saddle-point problems only, and are not applicable in our setting. Here, we adopt a much simpler solution, and take advantage of the properties of the Lagrange dual function  $g_{\omega, \mu}(\pmb{x}, \alpha)$  to design an iterative procedure directly leading to estimates of  $(F_{\mu}(\omega), \pmb{x}_e)$ . In this procedure, the two players successively update their actions until a stopping criterion is met, say up to the  $N$ -th iterations. The procedure generates a sequence  $\{(\pmb{x}^{(n)}, \alpha^{(n)})\}_{1 \leq n \leq N}$ , and from this sequence, estimates  $(\hat{F}, \hat{\pmb{x}})$  of  $(F_{\mu}(\omega), \pmb{x}_e)$ . The details of the resulting MCP algorithm are presented in Algorithm 1.
$\pmb{x}$  -player. We use a variant of the Follow-the-Perturbed-Leader (FTPL) algorithm [Han57, KV05]. The  $\pmb{x}$  -player updates her action as follows: