Title: From Contextual Combinatorial Semi-Bandits to Bandit List Classification: Improved Sample Complexity with Sparse Rewards
Venue: NeurIPS 2025
Accepted: yes
Keywords: multiclass

classification

bandit

combinatorial

online

pac

exploration
Abstract: We study the problem of contextual combinatorial semi-bandits, where input contexts are mapped into subsets of size $m$ of a collection of $K$ possible actions. In each round of the interaction, the learner observes feedback consisting of the realized reward of the predicted actions. Motivated by prototypical applications of contextual bandits, we focus on the $s$-sparse regime where we assume that the sum of rewards is bounded by some value $s \ll K$. For example, in recommendation systems the number of products purchased by any customer is significantly smaller than the total number of available products. Our main result is for the $(\varepsilon,\delta)$-PAC variant of the problem for which we design an algorithm that returns an $\varepsilon$-optimal policy with high probability using a sample complexity of $\widetilde{O}\big( (\mathrm{poly}(K/m) + sm / \varepsilon^2) \log (|\Pi|/\delta) \big)$ where $\Pi$ is the underlying (finite) class and $s$ is the sparsity parameter. This bound improves upon known bounds for combinatorial semi-bandits whenever $s \ll K$, and in the regime where $s = O(1)$, the leading terms in our bound match the corresponding full-information rates, implying that bandit feedback essentially comes at no cost. Our PAC learning algorithm is also computationally efficient given access to an ERM oracle for $\Pi$. Our framework generalizes the list multiclass classification problem with bandit feedback, which can be seen as a special case with binary reward vectors. In the special case of single-label classification corresponding to $s=m=1$, we prove an $O \big((K^7 + 1/\varepsilon^2)\log (|\mathcal{H}|/\delta)\big)$ sample complexity bound for a finite hypothesis class $\mathcal{H}$, which improves upon recent results in this scenario. Additionally, we consider the regret minimization setting where data can be generated adversarially, and establish a regret bound of $\widetilde O(|\Pi| + \sqrt{smT \log |\Pi|})$, extending the result of Erez et al. ('24) who consider the simpler single label classification setting.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the $(\\epsilon,\\delta)$-PAC learnability of multiclass list classification with semi-bandit feedback. In this model, at each round, the nature generates an instance $X$ and a list of $s$ labels $Y = \\{y_1,y_2,\\ldots,y_s\\}$, where $y_i \\in [K]$, from a distribution $D$ and shows the instance $X$ to the learner. Then, the learner predicts a list of $m$ labels $\\hat{Y}$ and gets the feedback $\\hat{Y} \\cap Y$. After $T$ rounds, the learner will output a concept $h \\in \\mathcal{H}$ that maps the instance $X$ to a list of $m$ labels. They investigate the sample complexity of learning the $\\epsilon$-optimal hypothesis with the probability of at least $1-\\delta$ and get an upper bound of $\\tilde{O}((\\text{poly}(K/m)+sm/\\epsilon^2)\\log(|\\mathcal{H}|/\\delta))$, when the concept class $\\mathcal{H}$ is finite. They also provide an $\\Omega(sm/\\epsilon^2)$ lower bound. \n\nFor the $s=m=1$ case, they get a $O((K^7+1/\\epsilon^2)\\log(|\\mathcal{H}|/\\delta))$ upper bound. For the regret minimization setting, they get an expected regret bound of $\\tilde{O}(|\\mathcal{H}|+\\sqrt{smT\\log|\\mathcal{H}|})$ for the adversarial setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The PAC learnability of multiclass classification with bandit feedback is an interesting and important problem. There are significant connections between this problem and the long-standing multi-armed bandit problem. This paper considers multiclass list classification with semi-bandit feedback, and the result can be applied to solve the contextual combinatorial semi-bandit problem. Their algorithm also provides the regret bound for the contextual combinatorial semi-bandit problem. \n2. The algorithm provided by this paper is computationally efficient with an efficient ERM oracle. Therefore, this result may be interesting in both the statistical learning theory community and the broader TCS community. \n3. The presentation of this paper is quite good.\n\nWeakness:\n\nThe result only works for finite hypothesis classes. It is not clear what will happen when the hypothesis class is infinite.\nThe assumption that every call of the ERM oracle only spends constant time is not that benign when there is an infinite hypothesis class.\n\nQUESTIONS:\nIn the contextual combinatorial semi-bandit problem, your algorithm works under the case that the reward function is additive, that is, $r(A\\cup B) = r(A) + r(B)$. What if the reward function is submodular, or more generally, any set function? Is it possible to get the same sample complexity?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the bandit multiclass list-classification problem, an extension of bandit multiclass classification when multiple labels can be predicted for each example. The paper considers how to build an algorithm that aims to approximate the best classification hypothesis out of a finite hypothesis class $|\\mathcal{H}|$. Formally, at each time $t$, the environment draws a sample $(x_t,y_t) \\sim \\mathcal{D}$, after observing $x_t$, the algorithm predicts a set of labels and receives feedback indicating which of the predicted labels were correct. The paper draws a strong connection between this classification problem and the contextual combinatorial semi-bandit (CCSB) framework, which is leveraged to design the algorithm to approximate the best hypothesis with a small number of samples. A sample complexity of $\\mathcal{O} \\left ( \\left (\\frac{K^9}{m^8} + \\frac{sm}{\\epsilon^2} \\right )\\log \\frac{|\\Pi |}{\\delta} \\right )$ is then proven for this algorithm. The paper then provides a lower bound which matches their sample complexity asymptotically for small $\\epsilon$. In the appendix, the paper provides an analysis of a similar setting with an objective of regret minimization and provides an upper bound on the regret of $\\mathcal{O} \\left ( \\sqrt{sm T \\log |\\Pi|} \\right ) $.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper proposes a technically interesting algorithm for list-based classification in a bandit setting, but a clearer exposition and complete proofs are required. Furthermore, this work would benefit from broader generalization. \n\n # Strengths \n\n- The paper is well-written and describes the related literature clearly. \n\n- The paper provides lots of insight into the motivation and key elements of their algorithm, as well as the main technical elements necessary to prove the results. \n\n # Weaknesses\n\n- The paper provides lower bounds for the problem but does not include a proof. While some very high-level intuition is provided, the proof does not appear in the appendices. Considering this, it is difficult to consider Theorem 2 as more than an intuition. This makes it difficult to gauge whether their approach is optimal or if more progress can be expected. \n\n- While the introduction and setting showcase the fact that the paper provides both a sample complexity and regret approach, only the sample complexity approach is addressed in the main paper. Consider presenting both the sample complexity and regret minimization settings (the algorithms and main results) in the main paper to clarify the full scope of contributions. If space is an issue, consider reducing the amount of technical details in the main paper and stick to high-level ideas and/or main differences with existing techniques, and provide further insight in the appendix. \n\n- The paper does not address the case of infinite hypothesis classes. Providing this kind of result, even with restricting assumptions on the classes of hypotheses, would greatly improve the value of the contribution. \n\n- Some literature was missed on combinatorial bandits with sparse rewards. Please consider extending the related work, especially since you claim to provide new results in that line of work. ( see [1] for instance)\n\n- The paper mostly uses the notation introduced when describing the Contextual Combinatorial Semi-Bandit setting. I understand that the algorithm used is based on CCSB literature, and I appreciate the fact that the link is explicitly stated, as it makes understanding the whole thing easier, yet it is counterintuitive to introduce the problem you are trying to solve and to switch notations right away afterwards. Furthermore, using different notation in the algorithms makes it difficult to follow. See, for instance, question 2. \n\n\n[1] Perrault, P., Valko, M., & Perchet, V. (2020, July). Covariance-adapting algorithm for semi-bandits with application to sparse outcomes. In Conference on Learning Theory (pp. 3152-3184). PMLR.\n\nQUESTIONS:\n1. What do you think is the limiting factor to applying the current types of results to infinite hypothesis classes? \n\n2. In the CCSB setting, and in algorithm 1, the main problem at hand is to choose the right policy $\\pi \\in \\mathcal{\\Pi}$. In the framework of the paper, my understanding is that it is related to the hypothesis class $\\mathcal{H}$, is that correct? Could you clarify that? Furthermore, it feels like in the paper, you sometimes switch notation from the ones of the classification problem to the CCSB. It is my understanding that the underlying assumption is that there is an equivalence between the two problem. I feel that having one set of homogeneous notation would make navigating the paper easier.\n\n3. Algorithm 1 uses a procedure that is based on two separate phases. The first phase explores uniformly, while the second one uses the previously acquired information to focus the exploration. Is there a particular reason for having two phases (in contrast with having an algorithm that starts by exploring uniformly but performs a focused exploration similar to phase 2 gradually, while incrementally refining its estimates )?  \n\n\n4. In the paper, you do not provide any assumption regarding the space $\\mathcal{X}$. It is my understanding that this comes at no additional cost from a discrete space $\\mathcal{X}$ because you assume that the hypothesis classes are finite. Essentially, using the fact that both $\\mathcal{H}$ and $\\mathcal{Y}$ are finite, the problem reduces to one with finite $\\mathcal{X}$. Does that seem correct?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the setting of bandit multi-label list classification from a learning-theoretic perspective and analyze it under the agnostic PAC and agnostic online learning setting. In addition, the authors show that their results for the agnostic online learning setting can also apply in the combinatorial semi-bandit setting.\n\nMulti-label list classification is understood as predicting a list of labels for each instance, and each instance is allowed to have multiple true labels. In the bandit version of this scenario (or semi-bandit as the authors note), the learner receives a response that explains which subset of the labels are correct but not necessarily the set of all true labels for that instance. The reward function is designed to be the count of number of true labels for a given instance that occur in the list predicted by the learner. In all settings considered, the hypothesis class $\\mathcal{H}$ is assumed to be finite with a finite label space $\\mathcal{Y}$ that has $|\\mathcal{Y}|=K$.\n\nIn agnostic PAC learning, the authors develop an algorithm (Algorithm 1) that returns an $\\epsilon$-optimal hypothesis with probability $1 - \\delta$ with a sample complexity on the order of $O(\\mathrm{poly}(\\frac{K}{m}) + \\frac{sm}{\\epsilon^2})$ and show a lower bound with $\\Omega(\\frac{sm}{\\epsilon^2})$. The main algorithmic technique consists of employing a learner to run a Frank-Wolfe procedure on a specially-designed convex potential whose gradient effectively estimates the variance of the reward estimator of any single policy. When the setting is simplified to single-label classification, the bound has an improved dependence on $K$ ($K^9$ to $K^7$) compared to previous work. In agnostic online learning, the authors devise an algorithm that has expected regret at most growing $O(\\sqrt{KT})$ where $T$ is the number of rounds.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This setting is analyzed in both agnostic PAC learning and agnostic online learning where upper and lower bounds are proven in the agnostic PAC setting.\n- The paper generalizes their work to the setting of contextual combinatorial semi-bandits with a bound that directly relies on the sparsity of rewards rather than the size of the label set.\n- The authors introduce a clever twist of constructing a convex potential whose gradient is exactly related to the variance of reward estimators. By employing the Franke-Wolfe procedure, the resulting mixed policy distribution has guarantees on the maximum value of the reward estimators allowing the algorithm to effectively estimate the reward estimators for all policies.\n\nWeaknesses:\n- While the usage of Franke-Wolfe leads to non-trivial sample complexity bounds, the use of Franke-Wolfe itself is not novel as prior work (citation [17]) employs a similar procedure in the case of single-label bandit classification. As a result, the innovation is more about tweaking the methodology rather than introducing a new optimization paradigm.\n- While the authors do provide upper and lower bounds in the agnostic PAC setting showing the importance of the $\\frac{sm}{\\epsilon^2}$ term, the hidden polynomial factor $(K^9)$ is pretty large signifying that there is still a somewhat sizeable gap between the two bounds.\n- The writing of the paper is at times hard to understand, especially for an audience not well-versed in this specific literature. The paper could benefit from having an improved exposition of the learning setting itself and provide a better proof sketch of Theorem $1$ that elaborates the core technical ideas in a clearer fashion.\n\nQUESTIONS:\n- For the case of agnostic online learning, there doesn't seem to be a lower bound on the regret showing that the regret achieved by Algorithm $4$ is in fact optimal or loose. Can the authors elaborate on the optimality of the regret bounds in the case of agnostic online learning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multiclass list classification where the learner only receives semi-bandit feedback. That is, it assumes for each object $x$, there exist a set of $s$ true labels. Upon receiving $x$, the learner is to predict a list of $m$ labels and receives all correct labels among the $m$ labels. The paper establishes a PAC learning variant of the problem and proposes an algorithm with sample complexity of $\\tilde{O}(\\text{poly}(K/m)+sm/\\epsilon^2)\\log(\\lvert{H}\\rvert/\\delta)$, improving on its previous work on a special case when $m=s=1$. The paper also considers a regret minimization setting with adversaries, and establishes a regret bound of $\\tilde{O}(\\lvert{H}\\rvert+\\sqrt{smT\\log{\\lvert{H}\\rvert}})$.\n\nSTRENGTHS AND WEAKNESSES:\nFirst of all, I have a question on whether it is necessary to pose this problem in the multiclass and list learning paradigms, especially with (semi)-bandit feedback, plus adversarial data and online learning scenarios. It feels too much effort to fit the problem in the PAC learning setting. Given the motivative application mentioned in this paper, it feels much more like a recommendation system problem or contextual combinatorial semi-bandit problems. Moreover, the algorithm assumes having access to an ERM oracle, which is also less a PAC learning paradigm. I am not convinced that this is a reasonable and well-posed problem. The paper is generally hard to read, with many terminology remain unexplained.\n\nQUESTIONS:\nCan the authors give a more clear vision on why a PAC learning framework for the proposed problem is necessary?\nIf the multiclass list classification with bandit (or semi-bandit) feedback is not explored before, why considering adversarial data in this paradigm is necessary in this study?\nHow important the ERM oracle is? What if such oracle does not exist? If without such an oracle, is bandit multiclass list learning possible?\n\nLIMITATIONS:\nThe authors shall discuss the necessity of the ERM oracle.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}