SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {x} ^ {(n)} \in \operatorname * {a r g m i n} _ {\boldsymbol {x} \neq \boldsymbol {i} ^ {\star}} \left(\sum_ {m = 1} ^ {n - 1} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x}, \alpha^ {(m)}) + \left\langle \frac {\boldsymbol {\mathcal {Z}} _ {n}}{\eta_ {n}}, \boldsymbol {x} \right\rangle\right) = \operatorname * {a r g m i n} _ {\boldsymbol {x} \neq \boldsymbol {i} ^ {\star}} \left(\left\langle \sum_ {m = 1} ^ {n - 1} \ell_ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\alpha^ {(m)}) + \frac {\boldsymbol {\mathcal {Z}} _ {n}}{\eta_ {n}}, \boldsymbol {x} \right\rangle\right),
$$
where  $\mathcal{Z}_n$  is a random vector, exponentially distributed and with unit mean ( $\{\mathcal{Z}_n\}_{n\geq 1}$  are i.i.d.). Compared to the standard FTPL algorithm, we vary learning rate  $\eta_{n}$  over time to get anytime guarantees (as we do not know a priori when the iterative procedure will stop). This kind of time-varying learning rate was also used in [Neu15] with a similar motivation. Note that thanks to the linearity of  $g_{\omega ,\mu}$  and Lemma 2, the  $\pmb{x}$ -player update can be computed using at most  $D$  calls to the LM Oracle.
$\alpha$ -player and MCP outputs. From Proposition 1,  $f_{\pmb{x}}(\pmb{\omega}, \pmb{\mu}) = \max_{\alpha \geq 0} g_{\pmb{\omega}, \pmb{\mu}}(\pmb{x}, \alpha)$ . This suggests that the  $\alpha$ -player can just implement a best-response strategy: after the  $\pmb{x}$ -player action  $\pmb{x}^{(n)}$  is selected, the  $\alpha$ -player chooses  $\alpha^{(n)} = \alpha_{\pmb{x}^{(n)}}^{\star} = \frac{\triangle_{\pmb{x}^{(n)}}(\pmb{\mu})}{\langle \pmb{x}^{(n)} \oplus \pmb{i}^{\star}(\pmb{\mu}), \pmb{\omega}^{-1} \rangle}$ . This choice ensures that  $f_{\pmb{x}^{(n)}}(\pmb{\omega}, \pmb{\mu}) = g_{\pmb{\omega}, \pmb{\mu}}(\pmb{x}^{(n)}, \alpha^{(n)})$ , and suggests natural outputs for MCP: should it stops after  $N$  iterations, it can return  $\hat{F} = \min_{n \in [N]} g_{\pmb{\omega}, \pmb{\mu}}(\pmb{x}^{(n)}, \alpha^{(n)})$  and  $\hat{\pmb{x}} \in \mathrm{argmin}_{n \in [N]} g_{\pmb{\omega}, \pmb{\mu}}(\pmb{x}^{(n)}, \alpha^{(n)})$ .
Stopping criterion. The design of the MCP stopping criterion relies on the convergence analysis and regret from the  $x$ -player perspective of the above iterative procedure, which we present in the next subsection. This convergence will be controlled by  $\ell_{\omega,\mu}(\alpha_x^\star)$  and its upper bound  $L_{\omega,\mu}$  derived in Proposition 1. Introducing  $c_\theta = L_{\omega,\mu}(4\sqrt{K(\ln K + 1) + \sqrt{\ln(\theta^{-1}) / 2}})$ , the MCP stopping criterion is:  $\sqrt{n} > c_\theta (1 + \epsilon) / (\epsilon \hat{F})$ . Since  $\sqrt{n}$  strictly increases with  $n$  and since  $\hat{F} \geq F_\mu(\omega)$ , this criterion ensures that the algorithm terminates in a finite number of iterates. Moreover, as shown in the next subsection, it also ensures that  $\hat{F}$  returned by MCP is an  $(1 + \epsilon)$ -approximation of  $F_\mu(\omega)$  with probability at least  $1 - \theta$ .
Algorithm 1:  $(\epsilon, \theta)$ -MCP( $\omega, \mu$ )
initialization:  $n = 1,\hat{F} = \infty ,c_{\theta} = L_{\omega ,\mu}\left(4\sqrt{K(\ln K + 1)} +\sqrt{\ln(\theta^{-1}) / 2}\right);$    
while  $(n = 1)$  or  $(n > 1$  and  $\sqrt{n}\leq c_{\theta}(1 + \epsilon) / (\epsilon \hat{F}))$  do Sample  $\mathcal{Z}_n\sim \exp (1)^K$  and set  $\eta_{n} = \sqrt{K(\ln K + 1) / (4nL_{\omega,\mu}^{2})};$ $\pmb{x}^{(n)}\gets \mathrm{argmin}_{\pmb {x}\neq \pmb{i}^{\star}(\pmb {\mu})}\Bigl (\sum_{m = 1}^{n - 1}g_{\omega ,\pmb {\mu}}(\pmb {x},\alpha^{(m)}) + \langle \pmb {Z}_n,\pmb {x}\rangle /\eta_n\Bigr)$  (ties broken arbitrarily);  $\alpha^{(n)}\gets \mathrm{argmax}_{\alpha \geq 0}g_{\omega ,\mu}(\pmb{x}^{(n)},\alpha)$  (uniqueness ensured by Proposition 1 (c)); if  $g_{\omega ,\mu}(\pmb{x}^{(n)},\alpha^{(n)}) <   \hat{F}$  then  $(\hat{F},\hat{\pmb{x}})\gets (g_{\omega ,\mu}(\pmb{x}^{(n)},\alpha^{(n)}),\pmb{x}^{(n)})$  .  $n\gets n + 1$  end   
return  $(\hat{F},\hat{\pmb{x}})$
3.3 Performance analysis of the MCP algorithm
We start the analysis by quantifying the regret from the  $x$ -player perspective of MCP before its stops. The following lemma is proved in Appendix C.3.
Lemma 3. Let  $N\in \mathbb{N}$ . Under  $(\epsilon ,\theta)$ -MCP  $(\omega ,\mu)$
$$
\mathbb {P} \left[ \frac {1}{N} \sum_ {n = 1} ^ {N} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x} ^ {(n)}, \alpha^ {(n)}) - \frac {1}{N} \min  _ {\boldsymbol {x} \neq \boldsymbol {i} ^ {\star}} \sum_ {n = 1} ^ {N} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x}, \alpha^ {(n)}) \leq \frac {c _ {\theta}}{\sqrt {N}} \right] \geq 1 - \theta .
$$
Observe that on the one hand,
$$
\frac {1}{N} \sum_ {n = 1} ^ {N} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} \left(\boldsymbol {x} ^ {(n)}, \alpha^ {(n)}\right) \geq \min  _ {n \in [ N ]} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} \left(\boldsymbol {x} ^ {(n)}, \alpha^ {(n)}\right) = \hat {F} \tag {4}
$$
always holds. On the other hand, if  $\pmb{x}_e \in \mathrm{argmin}_{\pmb{x} \neq \pmb{i}^*} \max_{\alpha \geq 0} g_{\pmb{\omega}, \pmb{\mu}}(\pmb{x}, \alpha)$ , then we have:
$$
\frac {1}{N} \min  _ {\boldsymbol {x} \neq \boldsymbol {i} ^ {\star}} \sum_ {n = 1} ^ {N} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x}, \alpha^ {(n)}) \leq \frac {1}{N} \sum_ {n = 1} ^ {N} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x} _ {e}, \alpha^ {(n)}) \leq \max  _ {\alpha \geq 0} g _ {\boldsymbol {\omega}, \boldsymbol {\mu}} (\boldsymbol {x} _ {e}, \alpha) = F _ {\boldsymbol {\mu}} (\boldsymbol {\omega}). \tag {5}
$$
We conclude that for  $N$  such that  $\sqrt{N} \geq \frac{c_{\theta}(1 + \epsilon)}{\epsilon\hat{F}}$ , Lemma 3 together with the inequalities (4) and (5) imply that  $\hat{F} - F_{\mu}(\omega) \leq \frac{c_{\theta}}{\sqrt{N}} \leq \frac{\epsilon\hat{F}}{1 + \epsilon}$  holds with probability at least  $1 - \theta$ . Hence  $\mathbb{P}\left[\hat{F} \leq (1 + \epsilon)F_{\mu}(\omega)\right] \geq 1 - \theta$ . From this observation, we essentially deduce the following theorem, whose complete proof is given in Appendix C.2.
Theorem 3. Let  $\epsilon, \theta \in (0,1)$ . Under Assumption 1, for any  $(\boldsymbol{\omega}, \boldsymbol{\mu}) \in \Sigma_{+} \times \Lambda$ , the  $(\epsilon, \theta)$ -MCP  $(\boldsymbol{\omega}, \boldsymbol{\mu})$  algorithm outputs  $(\hat{F}, \hat{\boldsymbol{x}})$  satisfying  $\mathbb{P}\left[F_{\boldsymbol{\mu}}(\boldsymbol{\omega}) \leq \hat{F} \leq (1 + \epsilon) F_{\boldsymbol{\mu}}(\boldsymbol{\omega})\right] \geq 1 - \theta$  and  $\hat{F} = \max_{\alpha \geq 0} g_{\boldsymbol{\omega}, \boldsymbol{\mu}}(\hat{\boldsymbol{x}}, \alpha)$ . Moreover, the number of LM Oracle calls the algorithm does is almost surely at most  $\left\lceil \frac{c_{\theta}^2 (1 + \epsilon)^2}{\epsilon^2 F_{\boldsymbol{\mu}} (\boldsymbol{\omega})^2} \right\rceil = O\left(\frac{\|\boldsymbol{\mu}\|_{\infty}^4 \left\|\boldsymbol{\omega}^{-1}\right\|_{\infty}^2 K^3 D^5 \ln K \ln \theta^{-1}}{\epsilon^2 F_{\boldsymbol{\mu}} (\boldsymbol{\omega})^2}\right)$ .
4 The Perturbed Frank-Wolfe Sampling (P-FWS) algorithm
To identify an optimal sampling strategy, rather than solving the lower-bound problem in each round as a Track-and-Stop algorithm would [GK16], we devise P-FWS, an algorithm that performs a single iteration of the Frank-Wolfe algorithm for the lower-bound problem instantiated with an estimator of  $\pmb{\mu}$ . This requires us to first smooth the objective function  $F_{\mu}(\omega) = \min_{x \neq i^{\star}} f_{x}(\omega, \mu)$  (the latter is not differentiable at points  $\omega$  where the min is achieved for several sub-optimal actions  $\pmb{x}$ ). To this aim, we cannot leverage the same technique as in [WTP21], where  $r$ -subdifferential subspaces are built from gradients of  $f_{x}(\omega, \mu)$ . These subspaces could indeed be generated by a number of vectors (here gradients) exponentially growing with  $K$ . Instead, to cope with the combinatorial
decision sets, P-FWS applies more standard stochastic smoothing techniques as described in the next subsection. All the ingredients of P-FWS are gathered in §4.2. By design, the algorithm just leverages the MCP algorithm as a subroutine, and hence only requires the LM Oracle. In §4.3, we analyze the performance of P-FWS.
4.1 Smoothing the objective function  $F_{\mu}$
Here, we present and analyze a standard stochastic technique to smooth a function  $\Phi$ . In P-FWS, this technique will be applied to the objective function  $\Phi = F_{\mu}$ . Let  $\Phi: \mathbb{R}_{>0}^{K} \mapsto \mathbb{R}$  be a concave and  $\ell$ -Lipschitz function. Assume that the set of points where  $\Phi$  is not differentiable is of Lebesgue-measure zero. To smooth  $\Phi$ , we can take its average value in a neighborhood of the point considered, see e.g. [FKM05]. Formally, we define the stochastic smoothed approximate of  $\Phi$  as:
$$
\bar {\Phi} _ {\eta} (\omega) = \mathbb {E} _ {\boldsymbol {\mathcal {Z}} \sim \operatorname {U n i f o r m} \left(B _ {2}\right)} \left[ \Phi \left(\omega + \eta \boldsymbol {\mathcal {Z}}\right) \right], \tag {6}
$$
where  $B_{2} = \{\pmb{v}\in \mathbb{R}^{K}:\| \pmb{v}\|_{2}\leq 1\}$  and  $\eta \in (0,\min_{k\in [K]}\omega_k)$ . The following proposition lists several properties of this smoothed function, and gathers together some of the results from [DBW12], see Appendix H for more details.
Proposition 2. For any  $\omega \in \Sigma_{+}$  and  $\eta \in (0, \min_{k \in [K]} \omega_k)$ ,  $\bar{\Phi}_{\eta}$  satisfies: (i)  $\Phi(\omega) - \eta \ell \leq \bar{\Phi}_{\eta}(\omega) \leq \Phi(\omega)$ ; (ii)  $\nabla \bar{\Phi}_{\mu, \eta}(\omega) = \mathbb{E}_{\pmb{Z} \sim \text{Uniform}(B_2)} [\nabla \Phi_{\mu}(\omega + \eta \pmb{Z})]$ ; (iii)  $\bar{\Phi}_{\eta}$  is  $\frac{\ell K}{\eta}$ -smooth; (iv) if  $\eta > \eta' > 0$ , then  $\bar{\Phi}_{\eta'}(\omega) \geq \bar{\Phi}_{\eta}(\omega)$ .
Note that with (i), we may control the approximation error between  $\bar{\Phi}_{\eta}$  and  $\Phi$  by  $\eta$ . (ii) and (iii) ensure the differentiability and smoothness of  $\bar{\Phi}_{\eta}$  respectively. (iii) is equivalent to  $\bar{\Phi}_{\eta}(\omega') \leq \bar{\Phi}_{\eta}(\omega) + \langle \nabla \bar{\Phi}_{\eta}(\omega), \omega' - \omega \rangle + \frac{\ell K}{2\eta} \| \omega - \omega' \|_2^2$ ,  $\forall \omega, \omega' \in \Sigma_+$ . Finally, (iv) stems from the concavity of  $\Phi$ , and implies that the value  $\bar{\Phi}_{\eta}(\omega)$  monotonously increases while  $\eta$  decreases, and it is upper bounded by  $\Phi(\omega)$  thanks to (i). The above results hold for  $\Phi = F_{\mu}$ . Indeed, first it is clear that the definition (2) of  $F_{\mu}$  can be extended to  $\mathbb{R}^{K}$ ; then, it can be shown that  $F_{\mu}$  is Lipschitz-continuous and almost-everywhere differentiable – refer to Appendices I and H for formal proofs.
4.2 The algorithm
Before presenting P-FWS, we need to introduce the following notation. For  $t \geq 1$ ,  $k \in [K]$ , we define  $N_{k}(t) = \sum_{s=1}^{t} \mathbb{1}\{x_{k}(s) = 1\}$ ,  $\hat{\omega}_{k}(t) = N_{k}(t)/t$ , and  $\hat{\mu}_{k}(t) = \sum_{s=1}^{t} y_{k}(s) \mathbb{1}\{x_{k}(s) = 1\} / N_{k}(t)$  when  $N_{k}(t) > 0$ .
Sampling rule. The design of the sampling rule is driven by the following objectives: (i) the empirical allocation should converge to the solution of the lower-bound problem (1), and (ii) the number of calls to the LM Oracle should be controlled. To meet the first objective, we need in the Frank-Wolfe updates to plug an accurate estimator of  $\mu$  in. The accuracy of our estimator will be guaranteed by alternating between forced exploration and  $FW$  update sampling phases. Now for the second objective, we also use forced exploration phases when in a Frank-Wolfe update, the required number of calls to the LM Oracle predicted by the upper bound presented in Theorem 3 is too large. In view of Lemma 1 and Theorem 3, this happens in round  $t$  if  $\| \hat{\mu} (t - 1)\|_{\infty}$  or  $\triangle_{\mathrm{min}}(\hat{\mu} (t - 1))^{-1}$  is too large. Next, we describe the forced exploration and Frank-Wolfe update phases in detail.
Forced exploration. Initially,  $\mathbb{P}\text{-FWS}$  applies the LM Oracle to compute the forced exploration set  $\mathcal{X}_0 = \{\pmb{i}^{\star}(\pmb{e}_k):k\in [K]\}$ , where  $\pmb{e}_k$  is the  $K$ -dimensional vector whose  $k$ -th component is equal to one and zero elsewhere.  $\mathbb{P}\text{-FWS}$  then selects each action in  $\mathcal{X}_0$  once. Note that Assumption 1 (iii) ensures that the  $k$ -th component of  $\pmb{i}^{\star}(\pmb{e}_k)$  is equal to one. In turn, this ensures that  $\mathcal{X}_0$  is a  $[K]$ -covering set, and that playing actions from  $\mathcal{X}_0$  is enough to estimate  $\pmb{\mu}$ .  $\mathbb{P}\text{-FWS}$  starts an exploration phase at rounds  $t$  such that  $\sqrt{t / |\mathcal{X}_0|}$  is an integer or such that the maximum of  $\triangle_{\min}(\hat{\pmb{\mu}} (t - 1))^{-1}$  and  $\| \hat{\pmb{\mu}} (t - 1)\|_{\infty}$  is larger than  $\sqrt{t - 1}$ . Whenever this happens,  $\mathbb{P}\text{-FWS}$  pulls each  $\pmb{x}\in \mathcal{X}_0$  once.
Frank-Wolfe updates. When in round  $t$ , the algorithm is not in a forced exploration phase, it implements an iteration of the Frank-Wolfe algorithm applied to maximize the smoothed function  $\bar{F}_{\hat{\mu}(t-1),\eta_t}(\hat{\omega}(t-1)) = \mathbb{E}_{\boldsymbol{Z} \sim \mathrm{Uniform}(B_2)}[F_{\hat{\mu}(t-1)}(\hat{\omega}(t-1) + \eta_t\boldsymbol{Z})]$ . The sequence of parameters  $\{\eta_t\}_{t \geq 1}$  is chosen to ensure that  $\eta_t$  chosen in  $(0, \min_k \hat{\omega}_k(t))$ , and hence  $\hat{\omega}(t-1) + \eta_t\boldsymbol{Z} \in \mathbb{R}_{>0}^K$ . Also note that in a round  $t$  where the algorithm is not in a forced exploration phase,
Algorithm 2: P-FWS  $\left(\left\{\left(\epsilon_{t},\eta_{t},n_{t},\rho_{t},\theta_{t}\right)_{t}\right)\right.$
initialization:
return  $\hat{\pmb{t}} = \pmb{i}^{\star}(\hat{\pmb{\mu}} (t));$
for  $k = 1,\ldots ,K$  do  $\begin{array}{rl} & {\mathcal{X}_0\gets \mathrm{argmax}_{\boldsymbol {x}\in \mathcal{X}}\langle \boldsymbol {e}_k,\boldsymbol {x}\rangle (\mathrm{tie~broken~arbitrary})}\\ & {\mathrm{end}}\\ & {\mathrm{Sample~}\boldsymbol {x}\in \mathcal{X}_0\mathrm{~in~a~round - robin~manner~for~}4|\mathcal{X}_0|\mathrm{~rounds};\mathrm{update~}\hat{\boldsymbol{\mu}} (4|\mathcal{X}_0|)\mathrm{~and~}\hat{\boldsymbol{\omega}} (4|\mathcal{X}_0|);}\\ & {\mathrm{for~}t = 4|\mathcal{X}_0| + 1,\dots \mathrm{~do}} \end{array}$    
end if  $\sqrt{t / |\mathcal{X}_0|}\in \mathbb{N}$  or max  $\{\triangle_{\min}(\hat{\mu} (t - 1))^{-1},\| \hat{\mu} (t - 1)\|_{\infty}\} >\sqrt{t - 1}$  then Sample each  $\boldsymbol {x}\in \mathcal{X}_0$  once, update  $\hat{\pmb{\mu}} (t)$  and  $\hat{\omega} (t)$  , and  $t\gets t + |\mathcal{X}_0| - 1$  else Compute  $\nabla \tilde{F}_{\hat{\mu} (t - 1),\eta_t,n_t}(\hat{\omega} (t - 1))$  by  $(\rho_t,\theta_t)$  -MCP algorithm;  $\boldsymbol {x}(t)\gets i^{\star}\Bigl {(}\nabla \tilde{F}_{\hat{\mu} (t - 1),\eta_t,n_t}(\hat{\omega} (t - 1))\Bigr);$  Sample  $\boldsymbol {x}(t)$  and update  $\hat{\pmb{\mu}} (t)$  and  $\hat{\omega} (t)$  .   
end if max  $\{\triangle_{\min}(\hat{\mu} (t))^{-1},\| \hat{\mu} (t)\|_{\infty}\} \leq \sqrt{t}$  then  $\hat{F}_t\gets (\epsilon_t,\delta /t^2)$  MCP  $(\hat{\omega} (t),\hat{\mu} (t))$  . if  $t\hat{F}_t > (1 + \epsilon_t)\beta \Big(t,\Big(1 - \frac{1}{4|\mathcal{X}_0|}\Big)\delta \Big)$  then break;
by definition  $\triangle_{\min}(\hat{\mu}(t-1)) > 0$ . This implies that  $\hat{\mu}(t-1) \in \Lambda$  and that  $F_{\hat{\mu}(t-1)}$  and  $\bar{F}_{\hat{\mu}(t-1),\eta_t}(\hat{\omega}(t-1))$  are well-defined. Now an ideal FW update would consist in playing an action  $i^\star(\nabla \bar{F}_{\hat{\mu}(t-1),\eta_t}(\hat{\omega}(t-1))) = \operatorname*{argmax}_{\boldsymbol{x} \in \mathcal{X}}\left\langle \nabla \bar{F}_{\hat{\mu}(t-1),\eta_t}(\hat{\omega}(t-1)), \boldsymbol{x}\right\rangle$ , see e.g. [Jag13]. Unfortunately, we do not have access to  $\nabla \bar{F}_{\hat{\mu}(t-1),\eta_t}(\hat{\omega}(t-1))$ . But the latter can be approximated, as suggested in Proposition 2 (ii), by  $\nabla \tilde{F}_{\hat{\mu}(t-1),\eta_t,n_t}(\hat{\omega}(t-1)) = \frac{1}{n_t}\sum_{m=1}^{n_t}\nabla f_{\hat{\boldsymbol{x}}_m}(\hat{\omega}(t-1) + \eta_t\mathcal{Z}_m,\hat{\mu}(t-1))$ , where  $\mathcal{Z}_1,\dots,\mathcal{Z}_{n_t} \stackrel{i.i.d.}{\sim}$  Uniform  $(B_2)$ ,  $\hat{\boldsymbol{x}}_m$  is the action return by  $(\rho_t,\theta_t)$ -MCP  $(\hat{\omega}(t-1) + \eta_t\mathcal{Z}_m,\hat{\mu}(t-1))$ . P-FWS uses this approximation and the LM Oracle to select the action:  $\boldsymbol{x}(t) \in i^\star\left(\nabla \tilde{F}_{\hat{\mu}(t-1),\eta_t,n_t}(\hat{\omega}(t-1))\right)$ . The choices of the parameters  $\eta_t, n_t, \rho_t$  and  $\theta_t$  do matter.  $\eta_t$  impacts the sample complexity and should converge to 0 as  $t \to \infty$  so that  $F_{\mu,\eta_t}(\omega) \to F_{\mu}(\omega)$  at any point  $\omega \in \Sigma_+$  (this is a consequence of Proposition 2 (i)(iv)).  $\eta_t$  should not decay too fast however as it would alter the smoothness of  $\bar{F}_{\mu,\eta_t}$ . We will show that  $\eta_t$  should actually decay as  $1/\sqrt{t}$ .  $(n_t,\rho_t,\theta_t)$  impact the trade-off between the sample complexity and the computational complexity of the algorithm. We let  $n_t \to \infty$  and  $(\rho_t,\theta_t) \to 0$  as  $t \to \infty$  so that  $\left\langle \nabla \tilde{F}_{\mu,\eta_t,n_t}(\omega) - \nabla \bar{F}_{\mu,\eta_t}(\omega), x \right\rangle \to 0$  for any  $(\omega,x) \in \Sigma_+ \times \mathcal{X}$ .
Stopping and decision rule. As often in best arm identification algorithms, the P-FWS stopping rule takes the form of a GLRT:
$$
\tau = \inf  \left\{t > 4 \mid \mathcal {X} _ {0} |: \frac {t \hat {F} _ {t}}{1 + \epsilon_ {t}} > \beta \left(t, \left(1 - \frac {1}{4 \mid \mathcal {X} _ {0} |}\right) \delta\right), \max  \left\{\triangle_ {\min } \left(\hat {\boldsymbol {\mu}} (t)\right) ^ {- 1}, \| \hat {\boldsymbol {\mu}} (t) \| _ {\infty} \right\} \leq \sqrt {t} \right\}, \tag {7}
$$
where  $\epsilon_t \in \mathbb{R}_{>0}$ ,  $\hat{F}_t$  is returned by the  $(\epsilon_t, \delta / t^2)$ -MCP  $(\hat{\omega}(t), \hat{\mu}(t))$  algorithm. The function  $\beta$  satisfies
$$
\forall t \geq 1, \quad \left(t F _ {\hat {\boldsymbol {\mu}} (t)} (\boldsymbol {\omega} (t)) \geq \beta (t, \delta)\right) \Rightarrow \left(\mathbb {P} _ {\boldsymbol {\mu}} [ \boldsymbol {i} ^ {\star} (\hat {\boldsymbol {\mu}} (t)) \neq \boldsymbol {i} ^ {\star} (\boldsymbol {\mu}) ] \leq \delta\right), \tag {8}
$$
$$
\exists c _ {1}, c _ {2} > 0: \forall t \geq c _ {1}, \beta (t, \delta) \leq \ln \left(\frac {c _ {2} t}{\delta}\right). \tag {9}
$$
Examples of function  $\beta$  satisfying the above conditions can be found in [GK16, JP20, KK21]. The condition (8) will ensure the  $\delta$ -correctness of P-FWS, whereas (9) will control its sample complexity. Finally, the action returned by P-FWS is simply defined as  $\hat{\pmb{i}} = \pmb{i}^{\star}(\hat{\pmb{\mu}}(\tau))$ . The complete pseudo-code of P-FWS is presented in Algorithm 2.3
Our Julia implementation could be found at https://github.com/rectzeng/NeurIPS2023-PerturbedFWS.
4.3 Non-asymptotic performance analysis of P-FWS
The following theorem provides an upper bound of the sample complexity of P-FWS valid for any confidence level  $\delta$ , as well as the computational complexity of the algorithm.
Theorem 4. Let  $\mu \in \Lambda$  and  $\delta \in (0,1)$ . If  $\mathbb{P}$ -FWS is parametrized using
$$
\left(\epsilon_ {t}, \eta_ {t}, n _ {t}, \rho_ {t}, \theta_ {t}\right) = \left(t ^ {- \frac {1}{9}}, \frac {1}{4 \sqrt {t | \mathcal {X} _ {0} |}}, \left\lceil t ^ {\frac {1}{4}} \right\rceil , \frac {1}{1 6 t D ^ {2} | \mathcal {X} _ {0} |}, \frac {1}{t ^ {\frac {1}{4}} e ^ {\sqrt {t}}}\right), \tag {10}
$$
then (i) the algorithm finishes in finite time almost surely and  $\mathbb{P}_{\pmb{\mu}}[\hat{\pmb{n}} \neq \pmb{i}^{\star}(\pmb{\mu})] \leq \delta$ ; (ii) its sample complexity satisfies  $\mathbb{P}_{\pmb{\mu}}\left[\lim_{\delta \to 0} \frac{T}{\ln \delta^{-1}} \leq T^{\star}(\pmb{\mu})\right] = 1$  and for any  $\epsilon, \tilde{\epsilon} \in (0,1)$  with  $\epsilon < \min\{1, \frac{2D^{2} \triangle_{\min}^{2}}{K}, \frac{D^{2} \| \pmb{\mu} \|_{\infty}^{2}}{3}\}$ ,
$$
\mathbb {E} _ {\boldsymbol {\mu}} [ \tau ] \leq \frac {(1 + \tilde {\epsilon}) ^ {2}}{T ^ {\star} (\boldsymbol {\mu}) ^ {- 1} - 6 \epsilon} \times H \left(\frac {1}{\delta} \cdot \frac {4 c _ {2}}{3} \cdot \frac {(1 + \tilde {\epsilon}) ^ {2}}{T ^ {\star} (\boldsymbol {\mu}) ^ {- 1} - 6 \epsilon}\right) + \Psi (\epsilon , \tilde {\epsilon}),
$$
where  $H(x) = \ln x + \ln \ln x + 1$  and  $\Psi (\epsilon ,\tilde{\epsilon})$  (refer to (34) for a detailed expression) is polynomial in  $\epsilon^{-1}$ ,  $\tilde{\epsilon}^{-1}$ ,  $K$ ,  $\| \pmb {\mu}\|_{\infty}$ , and  $\triangle_{\mathrm{min}}(\pmb {\mu})^{-1}$ ; (iii) the expected number of LM Oracle calls is upper bounded by a polynomial in  $\ln \delta^{-1}$ ,  $K$ ,  $\| \pmb {\mu}\|_{\infty}$ , and  $\triangle_{\mathrm{min}}(\pmb {\mu})^{-1}$ .
The above theorem establishes the statistical asymptotic optimality of  $\mathbb{P}\text{-FWS}$  since it implies that  $\lim \sup_{\delta \to 0} \mathbb{E}_{\pmb{\mu}}[\tau] / \ln(1/\delta) \leq (1 + \tilde{\epsilon})^2/(T^\star(\pmb{\mu})^{-1} - 6\epsilon)$ . This upper bound matches the sample complexity lower bound (1) when  $\epsilon \to 0$  and  $\tilde{\epsilon} \to 0$ .
Proof sketch. The complete proof of Theorem 4 is presented in Appendix D.
(i) Correctness. To establish the  $\delta$ -correctness of the algorithm, we introduce the event  $\mathcal{G}$  under which  $\hat{F}_t$ , computed by  $(\epsilon_t, \delta / t^2)$ -MCP  $(\hat{\boldsymbol{\omega}}(t), \hat{\boldsymbol{\mu}}(t))$ , is an  $(1 + \epsilon_t)$ -approximation of  $F_{\hat{\boldsymbol{\mu}}(t)}(\hat{\boldsymbol{\omega}}(t))$  in each round  $t \geq 4|\mathcal{X}_0| + 1$ . From Theorem 3, we deduce that  $\mathbb{P}_{\boldsymbol{\mu}}[\mathcal{G}^c] \leq \sum_{t=4|\mathcal{X}_0|+1}^{\infty} \delta / t^2 \leq \delta / 4|\mathcal{X}_0|$ . In view of (8), this implies that  $\mathbb{P}_{\boldsymbol{\mu}}[\hat{\boldsymbol{t}} \neq \boldsymbol{i}^\star(\boldsymbol{\mu})] \leq \delta$ .
(ii) Non-asymptotic sample complexity upper bound.
Step 1. (Concentration and certainty equivalence) We first define two good events,  $\mathcal{E}_t^{(1)}$  and  $\mathcal{E}_t^{(2)}$ .  $\mathcal{E}_t^{(2)}$  corresponds to the event where  $\hat{\mu}(t)$  is close to  $\pmb{\mu}$ , and its occurrence probability can be controlled using the forced exploration rounds and concentration inequalities. Under  $\mathcal{E}_t^{(1)}$ , the selected action  $\pmb{x}(t)$  is close to the ideal FW update. Again using concentration results and the performance guarantees of MCP given in Theorem 3, we can control the occurrence probability of  $\mathcal{E}_t^{(2)}$ . Overall, we show that  $\sum_{t=1}^{\infty} \mathbb{P}_{\pmb{\mu}}[(\mathcal{E}_t^{(1)} \cap \mathcal{E}_t^{(2)})^c] < \infty$ . To this aim, we derive several important continuity results presented in Appendix G. These results essentially allow us to study the convergence of the smoothed FW updates as if the certainty equivalence principle held, i.e., as if  $\hat{\mu}(t) = \pmb{\mu}$ .
Step 2. (Convergence of the smoothed FW updates) We study the convergence assuming that  $(\mathcal{E}_t^{(1)} \cap \overline{\mathcal{E}_t^{(2)}})$  holds. We first show that  $\bar{F}_{\mu, \eta_t}$  is  $\ell$ -Lipschitz and smooth for  $\ell = 2D^2\|\pmb{\mu}\|_{\infty}^2$ , see Appendices H and I. Then, in Appendix E, we establish that the dynamics of  $\phi_t = \max_{\omega \in \Sigma} F_\mu(\omega) - F_\mu(\hat{\omega}(t))$  satisfy  $t\phi_t \leq (t-1)\phi_{t-1} + \ell \left(\eta_{t-1} + \frac{K^2}{2t\eta_t}\right)$ . Observe that, as mentioned earlier,  $1/\sqrt{t}$  is indeed the optimal scaling choice for  $\eta_t$ . We deduce that after a certain finite number  $T_1$  of rounds,  $\phi_t$  is sufficiently small and  $\max \{\triangle_{\min}(\hat{\pmb{\mu}}(t))^{-1}, \|\hat{\pmb{\mu}}(t)\|_{\infty}\} \leq \sqrt{t}$ .