SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 3. Finally, we observe that  $\mathbb{E}_{\pmb{\mu}}[\tau] \leq T_1 + \sum_{t=T_1}^{\infty} \mathbb{P}_{\pmb{\mu}}\left[t\hat{F}_t \leq (1+\epsilon_t)\beta\left(t,\left(1-\frac{1}{4|\mathcal{X}_0|}\right)\delta\right)\right] + \sum_{t=T_1+1}^{\infty} \mathbb{P}_{\pmb{\mu}}\left[(\mathcal{E}_t^{(1)} \cap \mathcal{E}_t^{(2)})^c\right]$ , and show that the second term in the r.h.s. in this inequality is equivalent to  $T^\star(\pmb{\mu})\ln(1/\delta)$  as  $\delta \to 0$  using the property of the function  $\beta$  defining the stopping threshold and similar arguments as those used in [GK16, WTP21].
(iii) Expected number of LM Oracle calls. The MCP algorithm is called to compute  $\hat{F}_t$  and to perform the FW update only in rounds  $t$  such that  $\max \{\triangle_{\min}(\hat{\mu}(t))^{-1}, \| \hat{\mu}(t)\|_{\infty}\} \leq \sqrt{t}$ . Thus, from Theorem 3 and Lemma 1, the number of LM Oracle calls per-round is a polynomial in  $t$  and  $K$ . As the  $\mathbb{E}_{\mu}[\tau]$  is polynomial (in  $\ln \delta^{-1}$ ,  $K$ ,  $\|\pmb{\mu}\|_{\infty}$  and  $\triangle_{\min}^{-1}$ ), the expected number of LM Oracle calls is also polynomial in the same variables.
We provide an exhaustive survey of the related literature in Appendix B. To summarize, to the best of our knowledge, CombGame [JMKK21] is the state-of-the-art algorithm for BAI in combinatorial semi-bandits in the high confidence regimes. A complete comparison to P-FWS is presented in Appendix B. CombGame was initially introduced in [DKM19] for classical bandit problems. There, the lower-bound problem is casted as a two-player game and the authors propose to use no-regret algorithms for each player to solve it. [JMKK21] adapts the algorithm for combinatorial semibandits, and provides a non-asymptotic sample complexity upper bound matching (1) asymptotically. However, the resulting algorithm requires to call an oracle solving the Most-Confusing-Parameter problem as our MCP algorithm. The authors of [JMKK21] conjectured the existence of such a computationally efficient oracle, and we establish this result here.
In this paper, we have presented P-FWS, the first computationally efficient and statistically optimal algorithm for the best arm identification problem in combinatorial semi-bandits. For this problem, we have studied the computational-statistical trade-off through the analysis of the optimization problem leading to instance-specific sample complexity lower bounds. This approach can be extended to study the computational-statistical gap in other learning tasks. Of particular interest are problems with an underlying structure (e.g. linear bandits [DMSV20, JP20], or RL in linear / low rank MDPs [AKKS20]). Most results on these problems are concerned with statistical efficiency, and ignore computational issues.
We thank Aristides Gionis and the anonymous reviewers for their valuable feedback. The research is funded by ERC Advanced Grant REBOUND (834862), the Wallenberg AI, Autonomous Systems and Software Program (WASP), and Digital Futures.
$\left[\mathrm{AAS}^{+}23\right]$  Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, Kentaro Toyoshima, and Atsushi Iwasaki. Last-iterate convergence with full-and noisy-information feedback in two-player zero-sum games. In Proc. of AISTATS, 2023.
[AKKS20] Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. In Proc. of NeurIPS, 2020.
[ALLW18] Jacob Abernethy, Kevin A Lai, Kfir Y Levy, and Jun-Kun Wang. Faster rates for convex-concave games. In Proc. of COLT, 2018.
[AMP21] Aymen Al Marjani and Alexandre Proutiere. Adaptive sampling for best policy identification in markov decision processes. In Proc. of ICML, 2021.
[APFS22] Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On last-iterate convergence beyond zero-sum games. In Proc. of ICML, 2022.
[BBGS11] André Berger, Vincenzo Bonifaci, Fabrizio Grandoni, and Guido Schäfer. Budgeted matching and budgeted matroid intersection via the gasoline puzzle. Mathematical Programming, 2011.
[BGK22] Antoine Barrier, Aurélien Garivier, and Tomáš Kocák. A non-asymptotic approach to best-arm identification for gaussian bandits. In Proc. of AISTATS, 2022.
[BLM13] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013.
[BV04] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[CBL06] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[CBL12] Nicolo Cesa-Bianchi and Gábor Lugosi. Combinatorial bandits. Journal of Computer and System Sciences, 2012.
[CCG21a] Thibaut Cuvelier, Richard Combes, and Eric Gourdin. Asymptotically optimal strategies for combinatorial semi-bandits in polynomial time. In Proc. of ALT, 2021.
[CCG21b] Thibaut Cuvelier, Richard Combes, and Eric Gourdin. Statistically efficient, polynomial-time algorithms for combinatorial semi-bandits. Proc. of SIGMETRICS, 2021.
[CGL16] Lijie Chen, Anupam Gupta, and Jian Li. Pure exploration of multi-armed bandit under matroid constraints. In Proc. of COLT, 2016.
$\left[\mathrm{CGL}^{+}17\right]$  Lijie Chen, Anupam Gupta, Jian Li, Mingda Qiao, and Ruosong Wang. Nearly optimal sampling algorithms for combinatorial pure exploration. In Proc. of COLT, 2017.
$\left[\mathrm{CLK}^{+}14\right]$  Shouyuan Chen, Tian Lin, Irwin King, Michael R Lyu, and Wei Chen. Combinatorial pure exploration of multi-armed bandits. In Proc. of NeurIPS, 2014.
[CMP17] Richard Combes, Stefan Magureanu, and Alexandre Proutiere. Minimal exploration in structured stochastic bandits. In Proc. of NeurIPS, 2017.
[CTMSP+15] Richard Combes, Mohammad Sadegh Talebi Mazraeh Shahi, Alexandre Proutiere, et al. Combinatorial bandits revisited. In Proc. of NeurIPS, 2015.
[DBW12] John C Duchi, Peter L Bartlett, and Martin J Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization, 2012.
[DFG21] Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. In Proc. of NeurIPS, 2021.
[DKC21] Yihan Du, Yuko Kuroki, and Wei Chen. Combinatorial pure exploration with full-bandit or partial linear feedback. In Proc. of AAAI, 2021.
[DKM19] Rémy Degenne, Wouter M Koolen, and Pierre Ménard. Non-asymptotic pure exploration by solving games. In Proc. of NeurIPS, 2019.
[DMSV20] Rémy Degenne, Pierre Ménard, Xuedong Shang, and Michal Valko. Gamification of pure exploration for linear bandits. In Proc. of ICML, 2020.
[DP19] Constantinios Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. Proc. of ITCS, 2019.
[FKM05] Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proc. of SODA, 2005.
[FKV14] Eugene A Feinberg, Pavlo O Kasyanov, and Mark Voorneveld. Berge's maximum theorem for noncompact image sets. Journal of Mathematical Analysis and Applications, 2014.
[GC11] A. Garivier and O. Cappé. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proc. of COLT, 2011.
[GH16] Dan Garber and Elad Hazan. A linearly convergent variant of the conditional gradient algorithm under strong convexity, with applications to online and stochastic optimization. SIAM Journal on Optimization, 2016.
[GK16] Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Proc. of COLT, 2016.
[GPDO20] Noah Golowich, Sarath Pattathil, Constantinos Daskalakis, and Asuman Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. In Proc. of COLT, 2020.
$\left[\mathrm{H}^{+}16\right]$  Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization, 2016.
[Han57] James Hannan. Approximation to bayes risk in repeated play. Contributions to the Theory of Games, 1957.
[HK12] Elad Hazan and Satyen Kale. Projection-free online learning. In Proc. of ICML, 2012.
[Jag13] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Proc. of ICML, 2013.
[JMKK21] Marc Jourdan, Mojmír Mutny, Johannes Kirschner, and Andreas Krause. Efficient pure exploration for combinatorial bandits with semi-bandit feedback. In Proc. of ALT, 2021.
[JP20] Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification in linear bandits. In Proc. of NeurIPS, 2020.
[KCG16] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. JMLR, 2016.
[KK21] Emilie Kaufmann and Wouter M Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. JMLR, 2021.
[KLLM22] Daniel Kane, Sihan Liu, Shachar Lovett, and Gaurav Mahajan. Computational-statistical gap in reinforcement learning. In Proc. of COLT, 2022.
[KSJJ+20] Julian Katz-Samuels, Lalit Jain, Kevin G Jamieson, et al. An empirical process approach to the union bound: Practical algorithms for combinatorial and linear bandits. In Proc. of NeurIPS, 2020.
[KTAS12] Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. Pac subset selection in stochastic multi-armed bandits. In Proc. of ICML, 2012.
[KV05] Adam Kalai and Santosh Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences, 2005.
[KWA+14] Branislav Kveton, Zheng Wen, Azin Ashkan, Hoda Eydgahi, and Brian Eriksson. Matroid bandits: fast combinatorial optimization with learning. In Proc. of UAI, 2014.
[Lai87] Tze Leung Lai. Adaptive treatment allocation and the multi-armed bandit problem. The annals of statistics, 1987.
[Law72] Eugene L Lawler. A procedure for computing the k best solutions to discrete optimization problems and its application to the shortest path problem. Management science, 1972.
$\left[\mathrm{LNP}^{+}21\right]$  Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, et al. Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes. In Proc. of AISTATS, 2021.
[MCP14] Stefan Magureanu, Richard Combes, and Alexandre Proutiere. Lipschitz bandits: Regret lower bounds and optimal algorithms. In Proc. of COLT, 2014.
[Neu15] Gergely Neu. First-order regret bounds for combinatorial semi-bandits. In Proc. of COLT, 2015.
[Oka73] Masashi Okamoto. Distinctness of the eigenvalues of a quadratic form in a multivariate sample. The Annals of Statistics, 1973.
[PBVP20] Pierre Perrault, Etienne Boursier, Michal Valko, and Vianney Perchet. Statistical efficiency of thompson sampling for combinatorial semi-bandits. In Proc. of NeurIPS, 2020.
[Per22] Pierre Perrault. When combinatorial thompson sampling meets approximation regret. In Proc. of NeurIPS, 2022.
[PPV19] Pierre Perrault, Vianney Perchet, and Michal Valko. Exploiting structure of uncertainty for efficient matroid semi-bandits. In Proc. of ICML, 2019.
[RG96] Ram Ravi and Michel X Goemans. The constrained minimum spanning tree problem. In Scandinavian Workshop on Algorithm Theory. Springer, 1996.
[RS13] Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. In Proc. of NeurIPS, 2013.
[S⁺03] Alexander Schrijver et al. Combinatorial optimization: polyhedra and efficiency, volume 24. Springer, 2003.
[SN20] Arun Suggala and Praneeth Netrapalli. Follow the perturbed leader: Optimism and fast parallel algorithms for smooth minimax games. In Proc. of NeurIPS, 2020.
[Vis21] Nisheeth K. Vishnoi. Algorithms for Convex Optimization. Cambridge University Press, 2021.
[WLZL21] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In Prof. of ICLR, 2021.
[WTP21] Po-An Wang, Ruo-Chun Tzeng, and Alexandre Proutiere. Fast pure exploration via frank-wolfe. In Proc. of NeurIPS, 2021.
[ZODS21] Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. In Proc. of NeurIPS, 2021.