Title: Multi-play Combinatorial Semi-Bandit Problem
Primary area: reinforcement learning
Keywords: multi-armed bandit, combinatorial optimization

Abstract: In the combinatorial semi-bandit (CSB) problem, a player selects an action from a combinatorial action set and observes feedback from the base arms included in the action. While CSB is widely applicable to combinatorial optimization problems, its restriction to binary decision spaces excludes important cases involving non-negative integer flows or allocations, such as the optimal transport and knapsack problems.
    To overcome this limitation, we propose a general form of CSB, the multi-play combinatorial semi-bandit (MP-CSB) problem, where a player can select a non-negative integer action and observe multiple feedbacks from a single arm in each round. 
    We propose two algorithms for the MP-CSB. 
    One is a Thompson-sampling-based algorithm that is computationally feasible even when the action space is exponentially large with respect to the number of arms, and attains $O(\log T)$ distribution-dependent regret in the stochastic regime, where $T$ is the time horizon. We show that this algorithm can be much more memory-efficient than naively expanding existing algorithms.
    The other is a best-of-both-worlds algorithm, which achieves $O(\log T)$ variance-dependent regret in the stochastic regime and the worst-case $\tilde{\mathcal{O}}( \sqrt{T} )$ regret in the adversarial regime. 
    Moreover, its regret in adversarial one is data-dependent, adapting to the cumulative loss of the optimal action, the total quadratic variation, and the path-length of the loss sequence.
    Finally, we numerically show that the proposed algorithms outperform existing methods in the CSB literature.

TL;DR: We generalize the ordinary combinatorial semi-bandit problem and provide algorithms for it.