Generalized Prediction Set with Bandit Feedback
Authors: Zhou Wang
Xingye Qiao
Venue: Accepted by TMLR ()

Abstract: In high-stakes environments where uncertainties abound, set-valued prediction offers a cautious and robust mechanism by presenting multiple potential labels as the prediction for each test instance to mitigate the potential risk associated with prediction errors. Yet, integrating this paradigm with out-of-distribution (OOD) detection remains scarcely explored in such settings as online learning with bandit feedback. The bandit feedback mechanism informs the learner about the correctness of the pulled arm/action instead of the explicit ground truth label, leaving the true class label unknown when an incorrect action is taken. To address this challenge, we introduce BanditGPS which conducts set-valued prediction with OOD detection in the bandit feedback setting, using an estimation to the ground truth of class labels. BanditGPS achieves three objectives: render small/informative prediction sets, enhance the OOD detection performance, and control the recall for all normal classes to meet prescribed requirements. Our approach is characterized by the loss function, which trades off between high OOD detection and small prediction sets. Theoretically, we prove that the convergence rate of the regret is $\tilde{\mathcal{O}}(T^{-1/2})$. The empirical results further show that BanditGPS effectively controls the recalls with promising performances on OOD detection and informative prediction.

Published in Transactions on Machine Learning Research (05/2025)
Generalized Prediction Set with Bandit Feedback
Zhou Wang
2uung199@@bigghamoooo.ccc
Deportment of Mothematics and Statistics Binghamton University, the State University of New York
Xingye Qiao
Briigobbnngammoon.com edu
Department Mathematics and Statistics
Binghamton University, the State University of New York
Reviewed on OpenReview: Attps: // openreview. net/ met/ forum? id- IlqqIz118
Abstract
In high-stakes environments where uncertainties abound, set-valued prediction offers a cautious and robust mechanism by presenting multiple potential labels the prediction for each test instance to mitigate the potential risk associated with prediction errors. Yet, integrating this paradigm with out-of-distribution (OOD) detection remains scarcely explored in such scttings as online learning with bandit feedback. The bandit feedback mechanism informs the learner about the correctness of the pulled arm arm/action /actiio instead of the explicit ground truth label, leaving the true class label unknown when an incorrect action is taken. To address this challenge, we introduce BanditGPS which conducts sct-valued prediction with OOD detection in the bandit feedback setting, using an estimation to the ground truth of class labels. BanditGPS achieves three objectives: render mnull/informative prediction sets, enhance the OOD detection performance, and control the recall for all normal classes to meet prescribed requirements. Our approach is characterized by the loss function, which trades off bet ween high OOD detection and small prediction sets. Theoretically, we prove that the convergence rate of the regret is ((7-1/2). The empirical results further show that BanditGPS effectively controls the recalls with promising performances on OOD detection and informative prediction.
1 Introduction
Conventional single-valued prediction assigns only a single class label each instance without a provable confidence guarantee. This paradigm could be overly-confident for some data instances, and thereby lead erroncous docisions with severe consoquences. Such issue is particularly pronounced in critical domains where instances are characterized by high uncertainties. For example, in medical triage, based single-label prediction, an AT system might incorrectly categorize a patient with subtle symptoms of severe condition as low-risk, delaying the critical treatment and hence endangering the patient's 's life. Similarly, in financial fraud detection, single-valued prediction system might not adequately account for the nuanced behaviors of complex fraud schemes, resulting either over-blocking of legitimate transactions (causing inconvenience and potential loss of business) or under-detecting sophisticated fraud tactics (leading to financial losses). To reduce these associated risks from those observations with high uncertainty, set-valued classification, which reports multiple possible class labels, may be employed. This approach allows for human intervention in those difficult data instances, enabling further error reduction through follow-up investigations.
The literature on set-valued prediction is diverse, with several different types of met (1) Classifica- tion with Reject Option (CRO) (Herbei & Wegkamp, 2006; Bartlett Wegkamp 2008 Charoenphakdee et al., 2021) incorporates a rejector into the loss function, which is used to abstain from classification an instance is highly ambiguous. This is equivalent to assigning all labels to the instance, resulting in less informative prediction set.. Zhang et al. (018)) introduces refined option to produce more informative/smalll smaller

---

Published in Transactions on Machine Learning Research (05/2025)
prediction set. However, CRO and its variants, despite considering the uncertainty of instances, do not control the "accuracy" under the set-valued prediction paradigm, i.e., the probability of the true label being included in the prediction set. (2) In contrast, Conformal Prediction (CP) (Shafer & Vovk 2008 Vovk et al. 2005 offers a prodiction set with a thcorctical guarantee of accuracy control. CP, assuming data exchangeability, is a distribution-free framework that works alongside machine learning models, e.g., neural net works, to facilitate sct-valued prediction. The caveat with CP is that the prediction set may be less informative due to not explicitly minimizing the set size. [Romano al. (2020); AAngelopoulos al. (2021) have focused on developing score functions to reduce the prediction set size to alleviate this limitation. (3) Confidence Set
Learning (CST.) (Wang & Qiao 2018 2022a Sadinle et al. 2019) approaches set-valued prediction from a
constrained optimization perspective, explicitly minimizing the prediction set size while controlling accuracy.
Conversely, Denis & Hebiri 2017 2020) seek lo maximize the accuracy of the prediction given a size budget.
All the above three camps concentrate on set-valued prediction within a closod-world assumption, where each instance is guaranteed at least one known class label. Nevertheless, the dynamic nature of the open world presents the inevitability of encountering new, unknown classes over time. This reality necessitates the unified process of not only classifying known entities but also detecting novel classes, task termed out-of-distribution (OOD) detection or open-set recognition YYngg et al., 2021 Lee et al.) 2018). Traditional approaches to OOD detection encounter limitations akin to those of single-valued prediction systems. In an effort to bridge this gap, by admitting an empty set for potential atypical points, recent innovations have expanded set-valued classification to incorporate an OOD detection component, such as Cautious Deep Learning (CDL) (Hechtlinger et al. 2018). Balanced Conformal Prediction Set (BCOPS) Guan & Tibshirani
)002), ; Generalized Prediction Set (GPS) (Wang & Qiao 2023 2022bp. and Deep Generalized Prediction Set.
(ecppPPS) (Wang & Qiao] 2024a). CDL and BCOPS, rooted in Conformal Prediction (CP), unfortunately
inherit its predisposition towards larger prediction sets. Conversely, GPS and DeepGPS strive for informative prediction by explicitly minimizing prediction set sizes, aligning with the principles of Confidence Set Learning
(CSL).
However, these advancements predominantly address scenarios of offline learning with full feedback, where detailed label information for each instance is accessible even following an incorrect prediction. This starkly contrasts with the online bandit feedback setting, where learner's received feedback is limited to the binary outcome of an arm/ac ammacctions success, devoid of explicit insights across all labels. For example, within clinical trials, an adaptive trial design system (learner) might select a treatment (arm), e.g., type of drug, without knowing its efficacy beyond the absence of disease remission, thus lacking comprehensive feedback on optimal treatments. Although Bandit Class-specific Conformal Prediction (BCCP) (Wang & Qiao 2024b) represents a step towards accommodating set-valued classification in a closed world under bandit leedback, it lacks mechanisms for OOD detection. Given the limited research this domain, our study pioneers the exploration of generalizing the set-valued prediction to have the capacity of OOD detection under the bandit feedback setting.
In this article, we have made the following four major contributions. Methodologically, we introduce BanditGPS, an adaptation of the Generalized Prediction Set for online learning with bandit feedback, marking a novel intersection of these research areas. Without accessing the explicit label information, the proposed approach handles the bandit feedback by utilizing an estimation of the ground truth of the class label. Secondly, unlike the offline training rogime of DeepGPS (Wang & Qiao 2024a), Bandit dynamically adjusts to maintain class-specific recall (to be defined later) for normal classes with fewer manual parameter adjustments, leading to feasible deployment for real-world online learning applications. Thirdly, unlike its predecessors, the Bandit allows explicit control of the balance between informative prediction and OOD detection effectiveness, making it easy for practitioners to use given their specific business needs. Lastly, we theoretically prove a regret, convergence rate of (T221 which is competitive in the literature; the empirical study confirms the efficacy of Ban.
The rest of this article organized as follows. In Section 2 we delve into key notations and review works that lay the groundwork for our study. Section 3 articulates the problem formulation and introduces the Bas method with an algorithm. The convergence rate of regret for the algorithm is theoretically explored in Section 4 In Section 5 we present empirical evidence demonstrating the effectiveness of BanditGPS. A
2

---

Published in Transactions on Machine Learning Research (05/2025)
conclusion is offered in Section Additional materials, including detailed proofs and extended experiments, are provided in Section A
2 Preliminary
In this section, we discuss some notions and briefly review seminal works across three pivotal areas: set-valued classification, out-of-distribution detection, and the multi-armed Bandit problem.
Set-valued Classification: Let (X E X y be an observation generated from an unknown distribution, where y' = (1...., K). A set-valued classifier is map o: : X' + 2 to return prediction set ((X) for the instance X based on a certain rule. Commons metries (Vork et al., 2017] to evaluate the performance of a set-valued classifier include the (expected) prediction set size and the accuracy. The prediction set size is defined through the cardinality of the prediction set, i.e., 1Ø(X), which gauges the number of class labels predicted for instance X. Because both the classifier (b.) and the instance X are random, we typically focus on ( (X) The accuracy in the set-valued prediction paradigm is defined as the probability that the true label of an observation is included in its prediction set, i.e., IPY ((X)], which is also referred to as coverage probability in the CP literature (Vovk et al., 2005).
The accuracy Y E xxX) represents the overall performance of the classifier over all classes. For each class, we denote the (class-specific) recall as P[Y E ((X) Y k] to measure the classifier's performance on class k. Intuitively speaking, there is a trade-off between prediction set size and recall. Higher recall for normal classes often comes with a larger prediction set d(X) (because this allows an increased likelihood for the true class label being included of (X)). While CP-basod classificrs (Vovk et al.
2005 Romano et al. 2020
Angelopoulos et. al., 2021 offer a controlled recall guarantee, their prediction set
sizes are contingent on the
chosen score function. In contrast, CSL-based approaches (Sadinle et al. 2019 Wang & Qiao 2018
2022a
2023 strive to minimize prediction set sizes while maintaining the normal class recall.
Out-of-distributioo (OOD) Detection and Open-set Recognition (OSR): The goal of OOD and OSR (Bendale & Boult 2016 Lee et al., 2018: Charpentier et al., 2020; Yang et al., 2021] is to conduct single-valued prediction for instances in the normal classes in addition to rejecting atypical observations that potentially comes from a new or anomaly class that was not present in the training data. Recently, Selective Classification with OOD detection (SCOD) (Xia. & Bouganis 2022 Zhu et al., 2023) and Unified Open-set Recognition (UOSR) (Kim et al. 2023 Cen et al.] 2023) further consider identifying difficulty observations (hard to distinguish among existing normal classes) when they are not rejected as OOD points. The difficult observations and OOD points in both SCOD and UOSR can be respectively viewed as the ambiguous instances with prediction set sizes greater than 1 and the atypical instances with the empty prediction set in the set-valued prediction paradigm.
Multi-armed Bandit (MAB): The MAB framework (Lai & Robbins 1985 Auer et al. 2002 concerns docision-making scenario where a learner soquentially pulls an arm A from set fl,..., .,, K} to maximize cumulative rewards over time. To this end, various policies (could be either a probability distribution deterministic rule) are proposed to guide the arm pulling, e.g., epsilon-greedy (Sutton & Barto 2018), Upper Confidence Bound (Auer et al. 2002), and Thompson sampling strategies [Thompson] 1933), etc. The policy a and the pulled arm A in the MAB can be conceptually treated as a form of decision rule and a predicted label in the classification problem.
MAB problems are prevalent in various fields, such as online advertising, clinical trials, and recommendation systems, where context X informs personalized decision-making, making. Specifically, after receiving the context X from an environment, learner pulls an arm A that follows .|- X), and then receives a feedback 1{A= = Y} returned by the environment. Such bandit foodback only indicates the correctness of the pulled arm. If the feedback is 0, the learner does not know the ground truth and hence faces a challenge in optimizing reward strategies. While existing studies (Kakade et al. 2008 Wang et al.) 2010 Crammer & Gentile 2013 Abbasi- Yadkori et al. | Gollapudi et al.] [002]] van der Hoeven ct al.]
2021 have explored lincar and neural network-basee (Zhou al.) 2020 Jin al. 2021 Zhang et al.J 2021 Xu ct 2022) approaches to the bandit, problem, they seldom address set-valued classifice cassification and/or OOD detection within this framework.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Notably, while research (Taffq et al.) 2022; Zhang et al. 2023 Stanton et al. 2023) has explored set-valued prediction in reinforcement learning, their focus diverges from our bandit-centric investigation. The recent work BCCP (Wang & Qiao 2024b) is most aligned with our work, yet it does not directly consider the task of OOD detection, a practical requirement in applications. For instance, in social content moderation, a system needs to flag user-generated content into some categories (safe, offensive, etc), and detect new types of harmful content as well. Our study, therefore, introduces novel approach, considering sct-valued classification with OOD detection in the context of bandit feedback, addressing significant gap in the literature.
The learning framework operates under bandit feedback, where only correctnoss information for a selected action is observed. At cach time step t, given context X a learner first produces a set-valued prediction (xx),) with a certain theoretical guarantee, indicating a subset of potential labels. Subsequently, a learner takes an action A, (denoting a single label) according to the policy *e where A, is sampled from all possibilities. The environment provides a binary feedback signal that whether the pulled arm A, confirms the ground true label of X (scc Algorithm
Set-valued Prediction with Bandit Feedback:
3 Methodology
In this section, we develop and optimize our set-valued classifier, BanditGPS, for the online bandit. feedback environment. BanditGPS is engineered to accomplish three critical objectives: (1) generate small/informant prediction sets; (2) proficiently detect OOD queries; and (3) maintain the recall for each normal class to be above a threshold of 1 7,,, where is a tolerance level prescribed by practitioners. The algorithmic framework and operational flow of BanditGPS are outlined in Algorithm Throughout the article, we use [K] to denote the set {I,...,K] } for convenience.
Consider a sequence of i..d. samples with inaccessible labels (X,, X 1,... T generated from the environment. Within this setting, we assume K established normal classes known from historical data. Nonetheless, we anticipate the emergence of OOD instances over time, which may not fit within these predefined classes and potentially signify novel, yet-to-be-identiffe classes.
To navigate this task, we leverage a neural network as our hypothesis class F. Specifically, we de- fine a decision function vector fw E F, parameterized by weights W at time t, where fwr(X) == (owr (X).... f (X)) (X)) IRK encapsulates the decision functions for each of the K normal classes. For any given query X, revealed at time t, the prediction set is constructed as,
or(X,) = {k E [K] Shxx (X,) 20
(1)
Intuitively speaking, class ls is considered to be plausible label for query X, if the query falls into the acceptance region for class l defined by {x fiy (x) 2 0}. Note that the union of all the K many acceptance regions may not cover the entire space X' and regions may overlap with each other.
Prediction Set Size Minimization: Based on the above def- inition of the prediction set, the prediction set size for the query X, is naturally quantified as (x,)), == Σ 1( which ranges from 0 to K. In particular, size 0 denotes that the classifier believes X, is an OOD instance, indicating its distinc- tiveness from all normal classes. Conversely, size K signifies the maximal ambiguity associated with the query, suggesting that it blurs the lines across all normal classes.
3.0
1u=0) I1 + -l-
(X,) 2 0).
2.5
[1 -l-
2.0
1.5
1.0
Due to the discontinuous, and hence, computationally intractable 0-1 loss [u 2 0} involved in the definition of prediction set size, we employ a surrogate hinge loss (u) := [++u++ max(0, 1+u) a proxy for the 0-1 loss (sec Figure 1.. This substitution enables a practical way to return smaller (more informative) prediction set by minimiziing the loss
0.0
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
15
2.0
Figure 1: Surrogate loss functions
K ( (xi)) = Σ
K
+ (Xi)l+
(2)
k=1
4

---

Published in Transactions on Machine Learning Research (05/2025)
OOD Detection Maximization: A query instance X, is not flagged as OOD if and only if its prediction set is non-emply, i.e,, ((x > 0. This is equivalent to the existence of at least one class h E [K] such that (y 2 0, or equivalently, ] (X,) 2 0. Therefore, to enhance the OOD detection performance, it is imperalive to penalize the indicator function of the occurrence of this event, 1 [maxk<[]] fnr (X,) 2 0}. Given the similar challenge imposed by the 0-1 loss as mentioned above, to effectively tackle the OOD detection task, we instead minimize the following:
21 (max five (X:)) = [1 max f (X:)) RE(K]
kE[K]
Recall Control for Normal Classes: By relating the definition of prediction set (1) the class-specific recall is further expressed as [V, E (X,) y, = k] = P [A (X,) 2 0 Y2 = kj] Thus, controlling the class-specific normal class recall to the prescribed value at least - 7, that is,
P [ (1)) 2 0 Y, k:
E
[1[ (X,) 0} Y, k] 2 1 7; for & E [K]
is equivalent to E[ = k} [[
(X,) < 0} ~11 - 0, for k E [K] due to the Bayes theorem and
some algebra (see details in Appendix A.2 This inequality can be viewed as constraint in constrained
optimization problems. Equivalently, our objective function incorporates the left-hand side of this constraint
as additional regularization terms, that is,
B[1[Y: = k3 [ALAi < 03 - -1],, for [K].
In the bandit feedback setting, the ground truth Yt and hence the quantities L(Y, = kj,k E [K] are unobservable since the learner has no direct access to the ground truth of the label Y, once receives the query X, Instead, it only knows whether its pulled arm As based on a policy N, is correct or not, ie., the foedback 1 (A, = Y/}. Particularly, if the feodback O, the learner has no idca which label is the true class. To overcome this challenge, in practice, the ground truth 1 - k} (the first indicator function in 4 can be replaced by an unbiased estimation conditional on Y),,
A == 1(A k} (K X,) 1(A = YY),
(5)
due to the fact Ex, [Ar from the K-arm policy, with arms from [K] reviewed in Section 2 we utilize an augmented policy with an additional arm, namely OOD, corresponding to the case that the instance comes from the OOD class. The definition of policy म. is deferred to the next subsection.
1(,, k}, where n, governs the distribution of A given X,. Different
To the same token, the intractable 0-1 loss 1 {u < 0} in the second indicator function in (4 can be substitutod by the surrogate hinge loss [1 u|+ == max(0, - u). This amounts to replacing 1 {u < 0} y in (4) by
(u) = [1 u]+ 7. Thus, instead of working with the loss function in 44 we can minimize the empirical average of the following loss function in practice:
D (f (X.)) = De, [[1 f + -7]
(6)
Policy Design: In this article, our policy is constructed based on the scoring function fwx(x)) (x), ... f (T)) associated with the normal classes. Specifically, the policy is defined as,
exp(frrr fb (X,))
(r(k Xx) =
k E [K].
exp( maxje[K] P (X.)) + Σx= exp(Frr (x1))
exp(- ma Piyee (X,))
T.. (OOD Xx) ==
exp(- maxje|K] Pire (X.)) + D exp (X,))
Note that (a X.), a E (l...., K, OOD}, sum up to and hence a legitimate probability distribution for Ar. Under this policy, the probability of pulling arm & is proportional to the exponentiated score for class k;
5

---

Published in Transactions on Machine Learning Research (05/2025)
while the probability of pulling OOD is proportional to the exponentiated negative maximum score over all normal classes. Hence, a small score across all normal classes indicates strong likelihood of the query being from the OOD) class.
The Overall Objective Function: Finally, taking into consideration three goals, i.e,, minimizing prediction set size (2), enhancing OOD detection (3, and controlling class-specific recall (6) for normal classes, the unified risk function for each data instance that we aim to minimize is
K ((X;;W9) = K Ei (x,)) + (1 X)G S (xi)) E Ne, (Xi)), kE[K]
K
(7)
Jem
where the neural network fw is updated by the stochastic gradient descent (or its variants). Here À E [0 is a uscr-specified value to indicate the trado-off between prodiction set size and OOD detection performance. Note that we do not treat as a luning parameter and rather leave it to be specified by the user according to their unique needs.
As for the parameter Ae,., an improperly chosen de, may lead to two cases: a λt, that is too small might fail to control the recall for normal classes, while a λe that is too large can cause the empirical normal recall highly above the pre- requisite - y and hence enlarge the prediction sets and impair the OOD detection performance. Additionally, an appropriate value of λe, can vary across tasks, making it difficult to predefine an optimal value. To address these limitations, we employ the primal-dual algorithm to dynamically optimize the value of A (see Equation ( in Al- gorithm D. Specifically, if the current prediction is over control, i.c., (f (X.) < O, Equa- tion (8) will adaptively lead to smaller value of the parameter λ in the next iteration. Such adjustment helps potentially return smaller prediction sets. Similarly, when (2, (X.)) will lead to a larger value of λ4+1, in the next iteration. This automatic updating rule frees us from the burden of manually selecting the value of A411, with excessive effort.
Algorithm BanditGPS
Require: Initialize weight matrices W and X1,e, k E [K]. Given À and learning rates 171 1: for t = 1...3..... T do
2: Learner receives X, and returns a prediction set
(x, = [kr E [K] fir (X,) 2 20
3: Learner pulls an arm A₁ ~ n, and computes D.,
4: Update parameters:
W W -VVW((
AC+1.E Drr. + mo E2, ( (x.)]
(8)
end for
(X)) > 0, Equation
4 Main Theorems
This section provides the convergence rate analysis concerning the regret associated with prediction set size and OOD detection performance. Our analysis builds upon foundational work in over- over-paramnn paramcterized neural
networks (Du et al. 2019 Allen-Zhu al. 2019 Cao & Gu 2019 2020 Chen al. 2023). i.e., sufficiently wide neural networks.
Let the hypothesis class of ReLU neural networks with depth L and constant width m be
F: F:= R + RK fw(()) WLoL-1 (W, (Wr-10 (W,,
where W = (Vee(W/ W1)
T) denotes a concatenated long vector by vectorizing W1 E Rmxp, Wr E Rmxm for 2 < 1 < - 1, and W, E k The ReLU activation functions at, E [L - are plement-wisee applied on each layer. Additionally, we define the ball around the neural network's initialization B(W1,w) .) = (W: ||W,- w w, E [Z]]. In this article, the Frobenius norm of a matrix is denoted by || |l2 and this notation extends to tensors.
Consider a restricted hypothesis class that adheres to the empirical normal recall control within this ball whose radius scales in the order of m 1122
Ft = (TWED F; 1
=
T 1Y = k} (X)) S 0 E [K] and W e B(W1, Rm11/21} }.
6

---

Published in Transactions on Machine Learning Research (05/2025)
CIPAR1C0
fowers
nilun butber scures
Sadd
nedat-iadd
Frog
Dig
Dig P
1000
2000
3000
5000
6000
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
6000
Figure 2: Updated de, for normal class across three datasets when
where R> > 0 is a fixed constant. We define the loss function as {(fw(X)) == A Ek= (2(FW(
(1
X)G (max*e[K] fW (X)) which consists of the loss on prediction set size and the loss on OOD detection; they may be viewed as negative reward and both are parts of the objective function L(X; W) in Equation 7 In this context, regret is defined as the difference between the loss achieved by the algorithm and the least. loss that could have been achieved by the best possible action taken in every iteration. To derive the rate of convergence for the regret, we use the following three assumptions.
Assumption 1. We assume the entries in Wi,7 [I. 1] are initialized with Gaussian distribution N( ; and the entries in W1 are initialized with Gaussian distribution (( k))
Assumption 2. The instance X, E RP, t E [T] is normalized to have a unit norm, i.c., ||ella = =1 Assumption 3. Cx,Cs > 0, s.t. for cach & E [K], 1 E [T], Ar, & CI. SCs. Remark 4. Assumption [ and Assumption 2 are standard assumptions in the regime of over-parameterized neural networks. Cao & Gu (2020) exlends Assumption
2 to the one with bound norms.
The assumption of the bounded de, is empirically verified in Figure 2 for the case of == 1. It shows that Xt,k S 6 holds for all t and & across the three datasets work with. The assumption of the boundness of A.. is not strong either because ils upper bound is inversely related to the lower bound of (h X,) # 0 (see Equation
(5) which can be manually manipulated in practice.
Theorem 5.
Let P& == P(Y, = k] be the prior probability for normal class h E [K]. The average recall for any class over all time for the prediction sets returned by Algorithm 7 is guaranteed:
T Σ PlV (x
1
1Y, Y, k) 2
Cx TraPk
Vk.
The above probability is taken over all the randomness during the learning process. If the learning rate 5 implics that, on average, the accumulative normal recall approaches to the prescribed value 1 - with rate O(T=-1). Note that the recall bound may be less meaningful in a short-term regime. Intuitively, if pk is very small, means that we may not collect sufficient samples for class k within a finite time horizon, leading to potential failures in recall control. However, the denominator in Theorem 5 includes T, compensating for this issue in the long run.
1/2 = -(T-") with a E [0, 1), Theorem
Theorem 6. Define
optimal learner with controlled normal recalls from the hypothesis class F+ fw- = argmin fWEFI + +E= ((fw(X.)). There exists an absolute constant, satisfying that, for any constant R > 0, m 2 0( LL (logm)))))) and the learning rale 1/1 = 0( (((RRTm), with probability at. least O(TL²) exp(- (xp(-((mw=211)) L)) over the randomness of the initialization W the regret is bounded as
T Regrel(T) = + Σ ((Jwx - + Σ t=l
e(Sw ((.))(x))) 500 [=l
LRVK VT
+
LRRA/3 /oo((m)
where the constant c* :- ( +1- KKC,
1- VK
KKACCs.
Theorem 0 shows that when the width of the neural network is sufficiently large, Algorithm I leads to the regret convergence rate with the order 01/21/2 + m m-16 /Tog(m)). Particularly, if the width further satisfies
7

---

Published in Transactions on Machine Learning Research (05/2025)
CIFAR10
CIFAR100
SVHN
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
0.25
0.25
Acum_ Normaollllaa Max
Acum_ Normal Max
Acum_Normal Max
0.00
0.00
0.00
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
Acum_ _OO
0.25
Asum_ DOD
0.25
Acum OOO
Acum, Normal Acum_Nomail Min
Acum, Normal Min
Acum. Noraa, Min
0.00
0.00
0.00
0
2000
4000
6000
0
2000
4000
6000
0
2000
4000
6000
Prediction set size
Acum_ Card
Acum_Card
Acum Acum_Card
Acum Cand Card
Acum Cond_Card
Anum Cond Card
4
2
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
Method
BanditGPS
BCCP
Full-feedback
Figure 3: Comparison among methods based on accumulative performance.
m 2 O(TS), then the regret is close to the parametric rate Ó(T-1/2), which is similar to the convex online learning. Similar convergence rate results were previously shown in Allen-Zhne et al. (2019) Cao & Gu 2019 Chen al. (2023) despite different scttings.
5 Experiments
represents the most aligned approach to our context even though it lacks mechanisms for OOD detection. To establish a baseline for comparison, we extend BCCP to also output an empty set, indicating OOD instances. Additionally, we present the performance of our model under the full feedback scenario, where the model is trained based on the labeled data, in order to show the performance in an ideal setting as a benchmark.
Baselines: Due to the unique setting and task, BCCP (Wang & Qiao 2024b)
Sct-up: We compare the methods by evaluating them CIFAR10, CIFAR100, and SVHN datascts. For CIFAR10, we set (Bird, Cat, Deer, Dog, Frog, Horse) normal classes while all the remaining (Airplane, Car, Ship, Truck} as the OOD class; for CIFARI00, we treat 10 coarser classes {Aquatic mammals, Flowers, Fruit and vegetables, Natural outdoor scenes, Omnivores and herbivores, Medium-sized mammals, Invertebrates, Reptiles, Small mammals, Trees} as normal classes, and the remaining 10 coarser classes as the entire OOD class; for SVHN, we let Digits (1, 2, 4, 6, 7, 9) as the normal classes and the remaining Digits {0, 3, 5, 8) be the OOD superclass.
We let all the experiments have the same desired recall - 7 0.95 across datasets, utilizing ResNet (He) ct al.] 2016) as the backbone architecture, Adam for optimization, learning rate mh 10-4 for network updates, and 72 = (1) = 11/2 for optimizing λ To improve the computational efficiency, model updates employ batch data with a size of 256 in each iteration, with about 6000 total iterations.
Metrics: For cach iteration t E [T), we report several accumulative quantities (sce definitions in Table D Acum_OOm Acum Acum__ormal Normal _Nrrmal_Min(t),,_______ deum_Normal Max(t) show the OOD detection performance and the recall coltrol on normal classes while Acum Card((), Acum Cond Card(t) assess the prediction set.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Displayed Metries for Comparison
Accumulative metric
Metric on the hold-out dataset
@am__Normal _Mi((t)_iiii___t = min Acum_ (tum_recalltt, k)
Hdt Midt_Normal_Ming = minxe(k] Hdt_ ceft__ecaal(t_ k)
Acum Normal Max(t)- [a @oaum_reaal((+ k)
Hdt_ Mot_Nooalll_ Max(t) = [a Hdt Adt__recall(t, k)
Acum (am_OOD(()) Acum (ocall(t,O00)
Hdt_OOD(() = Hddt_ coit_recaal((,, OOD)
Ein Ex c0, (Xx000)
Hdt Cond Card 1d (X.) /O0D/
Acum Cond Ctrd(t) = >i x,ee, ((Y0000)
1(9/000))
Acum_Cardde (Crd((t) E,- Ex,C (x.)
Hdt (ird(()rr((() 11 (x;()
[8,
size. Additionally, we assess classifiers' performance on a fixed holdout labelod datasct after cach iteration. In Table 1 B, denotes the batch of data at iteration 8, and n denotes the sample size of the holdout. dataset, and
Acum rtcall(t,k)
Acum recall(t, OOD)
E Ex,eB, 1(%, = k & & E (x.)
s=1 Ex,eB, 1(y OOD & ( 0}
& E [K]
E Ex,eB. 1(Y, = k}
Ex=EEx, A(Y, OOD)
Hdi recall((,k) k)
Hdt recall(1,0OD)
k Y, E (x.)), Eh =k
E 1[Y, OOD & (x, 0}
E [K]
Σ 1[Y, = k}
E 1(Y,
CIFAR10
CIFAR100
SVHN
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
0.25
0.25
HdL, Normal Max
Hdl Normal Max
Hdl, Normal Max
0.00
0.00
0.00
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
Hot. OOD
0.25
Hdt, OOD
0.25
Hot_OOD Hot_ Normal Min
Normal _Min
Normal Min
0.00
0.00
0.00
2000
4000
6000
2000
4000
6000
2000
4000
6000
5
Prediction set size
Hd.. Card
Hat Card
Hdl. Card
Hot, Cond_Card
Hot, Cond_ Card
4
Hdt, Cond__odd__ea Card
2
2000
4000
6000
2000
4000
6000
2000
4000
6000
Method
BanditGPS
BCCP
Full-feedback
Figure4 4: Comparison among methods based on performance on the fixed holdout datasets.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Particularly, the first two rows in Table I show the extreme cases of recall control normal classes in each iteration. The third row displays the OOD detection performance. In terms of prediction informativeness, the fourth row reports the prediction set size exclusively on all normal classes while the last row presents the prediction set size on all obscrvations, including those from the OOD class.
Results:
For a fair comparison with the BCCP where OOD detection is not particularly targeting, we first set À = 1 in the risk function -(;-) of BanditGPS, implying that the optimization of the OOD section performance is not a priority either. The top panel in Figure 0 demonstrates that both BanditGPS and BCCP effectively manage the accumulative class-specific normal recall, as illustrated by the colored dashed curves showing the smallest recall over all normal classes, which approach 1 7 as iterations progress. On the other hand, GPS distinguishes itself from BCCP with superior OOD detection capabilities (solid curves in the top panel) and smaller prediction sets (bottom panel). Further assessment on a holdout labeled dataset (see Figure 4 reinforces BanditGPS's enhanced performance over BCCP, showcasing its robust and improved OOD detection and informative prediction set. With the full feedback information, metrics in Figures 3 and 4 quickly improve as shown in green curves.
Trade-offis between Prediction Set Size and OOD Recall Driven by x: The parameter À affects the trade-off between metrics on prediction set size and OOD recall. In this section, we study these two metrics by varying the values of X, i.e., 0, 0.25,, 0.5, 0.75, and 1. As shown in Figures and t the smaller the X is, the larger the prediction set will be, despite the improvement in the OOD recall.
CIFAR10
CIFAR100
SVHN
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
0.25
0.25
Acum_ Normal Max
Acum Normal Max
Acum Normal Max
0.00
0.00
0.00
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
Acum_0OK
0.25
Asum_OOD
0.25
Acum_ soum_ODD
Acum_ Laau__NomalaMMaa
Anum_ Normal Min
Acum_ @aaa__aaaa_Laaaa Minn
0.00
0.00
0.00
2000
4000
6000
0
2000
4000
6000
0
2000
4000
6000
6
Prediction set size
5
4
3
Anum_Cart
Acum_ Card
Anum, Card Anum_Card
Acum, Cond. Card
Acum, Cond Card
2
Acum, Cond Card
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
A
1
0.75
0.5
0.25
0
Figure 5: Trado-off among accumulative metrics when varying the value of A.
6
Discussions
This work introduces a novel online set-valued prediction framework with dustr--diittibution (OOD) detection under bandit foodback. While we provide thcoretical guarantoes and empirical validation, we acknowledge several modeling assumptions and algorithmic components that deserve further clarification and suggest directions for future work.
10

---

Published in Transactions on Machine Learning Research (05/2025)
CIFAR10
CIFAR100
SVHN
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
0.25
0.25
Hdt_ NornNoloaaaaa.. Max
Hdt_ Nor_No Max
Hot, Normal Max
0.00
0.00
0.00
2000
4000
6000
2000
4000
6000
2000
4000
6000
1.00
1.00
1.00
0.75
0.75
0.75
Recall
0.50
0.50
0.50
0.25
Hdt_ OOD
0.25
Hdt. DOD
0.25
Hdt. OOD
Hot Nomal Min
Hdt Normal Min
Hdt, Nomal Min
0.00
0.00
0.00
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
6
10
Prediction set size
Card
Hdt Card
Hdt Card
Hdit_ Card_Card Card
Hit Card_Card Card
Hdl. Cond Card
2
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
A
0.75
0.5
0.25
0
Figure 6: Trade-off among metrics for the fixed holdout dataset when varying the value of A.
First, the thcorctical analysis in Theorem 6 relics on Assumption B which requires the boundedness of both the importance- weighted estimator A,, and the Lagrange multiplier Ae In practice, A,, depends inversely on the action sampling policy (k(k| X,). To ensure A. remains bounded, a simple yet effective solution is to clip the policy from below by a constant p, which guarantees Dr, S max(1/p, 1/(1 Kp)}. This design consideration can be naturally incorporated into the algorithm to ensure numerical stability and theoretical validity.
On the other hand, the multiplier d is dynamically updated to enforce recall constraints for normal classes. While our method adopts primal-dual approach, its precise theoretical bound is not given in this work. A promising direction is to build upon frameworks from online constrained optimization (Yu et al.) 2017 Yu & Necly 2020), which offer high-probability or expectation-based bounds for Lagrange multipliers under mild conditions. We plan to incorporate these insights into future extensions of BanditGPS.
Lastly, our current formulation operates in a binary bandit feedback setting, where single arm is pulled after set-valued prediction. This separates the prediction step (generating set) from the foedback step (observing label). An interesting future direction is to unily these steps within combinatorial bandit framework, where the predicted set itself is treated as the action, and partial or structured feedback is received over the set. Such a generalization could enhance learning efficiency and better refloct real-world applications where multiple predictions are jointly evaluated.
7 Conclusions
In the high-risk scenarios exacerbated by uncertainties from ambiguous instances and the emergence of new classes in dynamic environments, we introducod BanditGPS, a novel sct-valued classification method tailored for online bandit feedback settings. BanditGPS navigates the uncertainties by offering set of plausible labels for ambiguous instances and empty label set for OOD observation.
11

---

Published in Transactions on Machine Learning Research (05/2025)
BanditGPS handles the challenge of inaccessible label information inherent in bandit feedback by utilizing estimation of the ground truth of class labels. Leveraging primal-dual algorithm, BanditGPS dynamically adjusts tuning parameters in response to its performance history, facilitating an adaptive and automatic risk management mechanism. Furthermore, by enabling the weight between informative prediction and OOD detection, BanditGPS empowers practitioners tailor the system based on their specific operalional requirements. The theoretical and empirical validations of BanditGPS affirm its efficacy in managing the challenges of providing informative predictions while accurately detecting OOD instances under the control of class-specific normal recall.
There are limitations in this initial work that warrant further investigation and improvement. Beyond those discussed in Section G one promising direction is to integrate BanditGPS with alternative multi-armed bandit strategies that offer theoretical performance guarantees, such as Thompson Sampling or the Upper Confidence Bound algorithm, further enhance its decision-making capabilities. Additionally, rather than treating all abnormal inputs belonging single super OOD class, adapting BanditGPS support class-incremental learning could enable more refined discovery and assimilation of newly emerging classes in dynamic environments.
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Yasin Abbasi- Yadkor,, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference Learning Representations, 2021. URL http://pppmmrreveeeeeeeee///mrom
DbM9
Peter Auer, Nicolo Cesa-Biaachi, and Paul Fischer. Finitc-time analysis of the multiarmed bandit problem. Machine learning, 47:235-2555 256, 2002.
Peter L Bartlett and Marten H Wegkamp. Classification with reject option using hinge loss. Journal of Machine Learning Research, ((Aug):1823-18 1840, 2008.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Yuan Cao and Quanquan Gu. Gencralization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neurol informalion processing systems, 32, 2019.
Yuan Cao and Quanquan Gu. Generalization ion error bounds of gradient descent. for learning over-parameterived deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3349 3356, 2020.
Jun Cen, Di Luan, Shiwei Zhang, Yixuan Pei, Yingya Zhang, Deli Zhao, Shaojie Shen, and Qifeng Chen. The devil is in the wrongly-classffid samples: Towards unified open-sct recognition. In The Eleventh International Conference on Learning Represendalions, 2023. URL http::://pparrevier..eefffoooooo
xLrOI xYGAs.
Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pp. 1507-1517. PMLR, 2021.
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemaan.. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems, 33:1356 1367, 2020.
Xinyi Chen, Edgar Minasyan, Jason D) Lee, and Elad Hazan. Regret guarantees for online deep control. In Learning for Dynamics and Control Conference, pp. 1032-1045. PMLR, 2023.
Koby Crammer and Clandio Gentile. Multiclass classification with bandit feedback using adaptive regulariza- tion. Machine learning, 0033)3447-38) 2013.
Christophe Denis and Mohamed Hebiri. Confidence sets with expected sizes for multiclass classification. The Journal of Machine Learning Research, 18(1):3571 3598, 2017.
Christophe Denis and Mohamed Hebiri. Consistency of plug-in confidence sets for classification semi- supervised learning. Journal of Nonparametric Statistics, 32(1):42-72, 2020.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- parameterized neural networks. In International Conference on Learning Representations, 2019. URL
Sreenivas Gollapudi, Guru Guruganesh, Kostas Kollias, Pasin Manurangsi, Renato Leme, and Jon Schneider. Contextual recommendations and low-regret cutting-plane algorithms. Advonces in Neural Information Processing Systems, 34:22498-22500 2021.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Leying Guan and Robert Tibshirani. Prediction and outlier detection in classification problems. Journal of the Royal Stalistical Sociely. Scries B, Stalistionl Methodology, 84(2):524, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Yotam Hechtlinger, Barnabás Póczos, and Larry Wasserman. Cautious doep learning. arXiv preprint Ski::1805.094660 2018.
Radu Herbei and Marten H Wegkamp. Classification with reject option. The Canadian Journal of Statislics Ta Revue Canadienne de Statistique, pp. 709-721, 2006.
Tianyuan Jin, Pan Xu, Jieming Shi, Xiaokui Xiao, and Quanquan Gu. Mots: Minimax optimal thompson sampling. In International Conference on Machine Learning, pp. 5074-5083. PMLR, 2021.
Sham M Kakade, Shai Shalev-Shwartee and Ambuj Tewari. Efficient bandit algorithms for online multiclass prodiction. In Proceedings of the 25th international conference on Machine learning, pp. 440-447, 2008.
Jihyo Kim, Jiin Koo, and Sangheum Hwang. A unified benchmark for the unknown detection capability of deep neural networks. Expert Systems with Applications, 229:120461, 2023.
Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4 22, 1985.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of- distribution samples and adversarial attacks. Advances in neurol information processing systems, 31, 2018.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581 3591, 2020.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223 234, 2019.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
Samuel Stanton, Wesley Maddox, and Andrew Gordon Wilson. Bayesian optimization with conformal prediction sets. In International Conference on Artificial Intelligence and Statislics, pp. 959 986. PMLR, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Muhammad Faaiz Taufiq, Jean-Francois Ton, Rob Cornish, Yee Whye Teh, and Arnaud Doucet. Conformal off- off-policy prediction in contextual bandits. Advances in Neural Information Processing Systems, 35: 31512-31524 2022.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4)2285 294, 1933.
Dirk van der Hoeven, Federico Fusco, and Nicolò Cesa-Bianchi. Beyond bandit. feedback in online multiclass classification. Advances in neural information processing systems, 34:13280-133 132911, 2021.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer
Science Business Modia, 2005.
Vladimir Vovk, Ili Nouretdinov, Valentina Fedorova, Ivan Petej, and Alex Gammerman. Criteria of efficiency for set-valued classification. Annals of Mathemalics and Arlificial Intelligence, 81(1):21-46 -46, 2017.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Shijun Wang, Rong Jin, and Hamed Valizadegan. A potential-based framework for online multi-class learning with partial feedback. In Proceedings of the Thirleenth International Conference on Artificial Intelligence and Statistics, pp. 900 907. JMLR Workshop and Conference Proceedings, 2010.
Wenbo Wang and Xingye Qiao. Learning confidence sets using support vector machines. Advances in Neural Information Processing Systems, 31, 2018.
Wenbo Wang and Xingye Qiao. Set-valued support vector machine with bounded error rates. Journal of the American Statistical Association, pp. 1-13, 2022a.
Zhou Wang and Xingye Qiao. Learning acceptance regions for many classes with anomaly detection. arXiv preprinl ar X2v: 09963, 2022b.
Zhou Wang and Xingye Qiao. Sct-valued classification with out-of-distribution detection for many classes. Journal of Machine Learning Research, 24(375):1 2023.
Zhou Wang and Xingye Qiao. Deep generalized prediction set classifier and its theoretical guarantees. Transactions on Machine Leorning Research, 2024a.
Zhou Wang and Xingye Qiao. Efficient. online set-valued classification ion with bandit feedback. In International Conference on Machine Learning, pp. 51328 51347. PMLR, 2024b.
Guoxuan Xia and Christos-Savvas Bouganis. Augmenting softmax information for selective classification with out-of-distribution data. In Proceedings of the Asian Conference on Computer Vision, pp. 1995-2012, 2022.
Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. In International Conference on Learning Representations, 2022. URL https
//openreview.nt//forrmm/ wttp://eeeooooooodddaaa/aao//o/o
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. Xiviv preprint 1rXiv:2110.11334, 2021.
Hao Yu and Michael J Neely. A low complexity algorithm with O(vT) regret and ((1) constraint violations for online convex optimization with long term constraints. Journal of Machine Learning Research, 21 (1): 24, 2020.
Hao Yu, Michael Necly, and Xiaohan Wei. Online convex optimization with stochastic constraints. Advances in Neural Information Processing Systems, 30, 2017.
Chong Zhang, Wenbo Wang, and Xingye Qiao. On reject and refine options in multicategory classification. Journal of the American Statistical Association, (1335)2):730-745 2018.
Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural thompson sampling. In International Conference on Learning Representations, 2021. URI. httpss:/ htt..eettn..ooorrttddttttttltll
Yingying Zhang, Chengehun Shi, and Shikai Luo. Conformal off- prediction. In International Conference on Artificial Intelligence and Statistics, pp. 2751-2768. PMLR, 2023.
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. Tn International Conference on Machine Learning, pp. 11492 11502. PMLR, 2020.
Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-Lin Liu. Openmix: Exploring outlier samples for mis- classification detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12074 12083, 2023.
15

---

Published in Transactions on Machine Learning Research (05/2025)
A Appendix
A.1 Experiments Compute Resources
All experiments are conducted an NVIDIA P100 GPU with CUDA 11.3. The empirical metrics presented
in all figures are computed and averaged over 4 runs, with the shaded regions representing the standard error.
A.2 Derive for Normal Recall Control
12 = (
E 1 Us (x.) Y, = <
E
= 50
E
1(2 = kj [1t <0 -
P|Y; -
0
E [L(Y, k3 [1 (X+) < 0 r]] 00
A.3 Proofs
Proof of Theorem
5 From the updating rule Equation
we have
A Dee + 72 x l2 (s (x.))1
2 Ae, + 72 22 (J (X:))
1 A de,
XT
Cx
(f <
1/2 1=1
172
1/2
By taking the expectation for both sides of the last inequality, we have
T
Cx
122
Cx
E
1 (Xi)) (ma)se ((x,, Yi)) W'll 5
1/2
T
Cx
E [I(Y, =k} (1tr S
(9)
1/2
Cx
Σ
T |Y,
1Y,
(10)
1/2P
Cx
[11 < <X)
k] v
(11)
72pk
T
Cx 72Pk
P (x.,
< Y; k S Tr +
where 99 holds due to the (conditional) unbiased estimator A 10 holds due to the Bayes theorem, and I holds due to the substitution with the surrogate hinge loss as mentioned in Therefore, the last
nequality concludes
C2
Σ PY, E (x.)
2 7- TipPk
7
16

---

Published in Transactions on Machine Learning Research (05/2025)
Our derivation for Theorem 6 highly relies some commonly used lemmas and propositions in the over- parameterized neural network ure We skip their proofs to avoid redundancy. We leave their details to the main literature Allen-Zhu et al. 2019 : Cao & Gu 2001 2020 for the reader's interest.
Additionally, consider two sequences {ant and {bn}. We employ some asymptotic notations as follows: On (b.,) lim supn_cc and On (bb,,) if lim inminfn-so
<
Do; On = (bb,)) to indicate an additional logarithmic factor within O();
da 0. bye
For fixed instance X,, the neural network can be re-written in the form of matrix products below:
fw) = W.D.L--Wi-:D..
Di, D W1X,
(12)
We define h1 = Dr, W - Dr.1 Or,, L 1], where
Dr,1= Diag L((Wihe,--)) 20). ((((Whh,,)))) (.m200) 20)
Thus, the gradient can be written as
if L if (L -
Vw.Jw(X.)
4-1 Dr,, w, D..) r=l+1
(13)
Dt,
where еμ E RK is the column vector with value 1 in k-th entry and 0 otherwise. When W in Equation 12 is particularly set as the initialization W the associated W,, D.,, in Equation 12 will be denoted by W accordingly.
Lemma 7. I < O Log(m)) with the probubility al least 1 - 1em log(m))) over the rundomness of W E [L], for any perturbation matrices W [L] with [| S = O(L-1.b), any diagonal motrices
D [-3, 3) mmm I E [L with al most (33 L) non-zero entries, we have
L
L II W/D 1=0+1
w 1/3 /m log(m)
II (w) + - 1=0+1
+ -
so
VK
Proof of Lemma
7
II (w) + (D) + +D.,)) 1=t11
II w
1=+11
W D 1 W/,,D, w() (D). +D"
... W.D.L-I ... WT DI. (D), + D (W/, WL ... .D). +Daa)
1=0t1
W W D WL, ll,
| |D (Wl + w
(D),a + D
Allen Zhnet of
2019 Lemma 7.4(a))
Allen Zhuee al 2019 Lemma 8.6(b))
+
W W
WT
(D) + + D ( + (D).a + +D
(=0+)
Allen Zhuet al.
2019 Lemma 7.4()))
Allen Zhact al.
2019 Lemma 8.6(b))
< L. <
2/ [.lo((m)
+ O..
m K
O(VL)
K
w2/3
/ v log(m)
w2/3 I,2 log(m)
-
+ +0
WT,3/2
m
=
VK
K
VK
17

---

Published in Transactions on Machine Learning Research (05/2025)
where the structure of 0/1 is same as the one in Allen-Zhu et al. (201 We direct the reader to the corresponding literature for any interest.
Lemma 8. Cao & Gu 2019, Lemma D.1) If w I OTTL) (Pp--(((m'' L)) over W for all W, W E B(W), ,)), [T] and 1 E [L. 11-11 1],
3/2),, then with probability at least
[ - Du,llo z (
We will use Lemmas
aadd g to prove the below Theorem
Theorem 9. Let B(W" ,w)= == (W |[ - Will2 & w, [4]] be an w-neighborhood of the initialization. There ewists an absolute constant & such that, with probability at least 1 - exp( (mm over the randomness of W for all E (T), and W,W' W' E B(W), with w S kT. (lg=m)) 3/2, holds uniformly thot
(Sw(x.) [ (x)) VV w -W)
00 50 [ < (mlogmm)
W)
VK
Proof of Theorem
For W, W E B(W ww) (W: | wille ww, [4]], we have
fw(x;)) - fwx(())) (V W W-
L-1
1-1 (WWWW--
= -Wehi,L-1- WLhi,--1 (-1,L-1 WL WL)
Σ WL
Ww
-
1--1
-
Σ Wi
II
Dr, Wr Dri Wi-i)ha--
1=1
r=1+1
Similar to Claim 11.2 in Allen-Zhu ot al. (2019) there exists D'11 such that
h-1
h-1
h h1.--1
Σ
II (D,, + W.,, (D., + (W Wi-1)h
1=1
r=1+1
where diagonal matrices D/,,, E [L - 1] has entries [-1,1] such that | & O() Therefore,
fw((ar)) (20( - (V W W- w3
II w,D (ar-))(W-1-- (Wi-1- Wi-1)
Σ II W. +
D(rr-)) (W
-=111
1=1
r=l+1
1-1
<
max
W, + D'yrrr))
II
W, D Dr
Σ
W - W 111,1-1112
r=1+1
1=1
1/3
m log(m)
4/3
m log(m)
o
-w =0 -0
K
K
where ① due to
I r=1+1
W, (D. DD
II
W, W.D
r=1+1
I.
WD
W br-1
+
II
W W 1
II W
r=111
r=+
1/3 122 m (m)
K
18

---

Published in Transactions on Machine Learning Research (05/2025)
where ② holds by using Lemma This is because W.W B(W", w). By Lemma we have | (rr--1
-illo I TT and [Dt,r- D - I O(m))/TT). Hence [[arr D G-- D D llo S
[Dur-1 D lll + | llo I (m L) for any re L --
The below Proposition 10 is directly derived based on Theorem by utilizing the convexity and Lipchitaness of a given function L. This proposition implics near conveX with respect to the neural network paramcter W, although the neural net work not convex.
Proposition 10. Let the f RK R be a convez and e-Lipschitz function. There erists an absolute constant such that, with probability at least 1 O(TI²) exp( (Sp(-(((m2A/3))) over the randomness of W1, for all E [T), and W, W B(W', ,)) with w S KL (oL--[lo((m)) -3// it holds uniformly that
(c///3 mm log(m)
(x.) ((fw)(x))) S (X,)(f W) +0
K
Proof of Proposition 10
((fW((()))( (X,, ffr (X,))
b/c is convex
(X.)) - (x, (X),
+e(fw (X.)) WV -W)
co0t L3 m log(m))
00
(Vwe(fw(XX))) W W)),
(14)
VK
where the last inequality holds due to the Lipschitzness of E(-) and the usage of Theorem
The below Theorem 11 shows the boundness for the gradient of the neural network with respect to the parameters W.
Theorem 11. There exists absolute constant such that, with probability at least 1 O(TL²) exp( 21m over the randomness of W for all [T], [L], and W E B(W2, .w with w S kL FKL- (logmm)) it holds uniformly that [V = /2/ /m log(m)) s OVLLKm).
Proof of Theorem 1 This theorem is adapted from Cao & Gu 20019
Lemma B.3). From Equation
we
have
Vw..fwaz))12 = Σ vw.Jk<>))1
=
Σ |e h -
= O(K),
19

---

Published in Transactions on Machine Learning Research (05/2025)
and for E [L 1],
K 10 Wt
L-1
| fw
II ==++1
Dr. wr Da.ll2
k=
K
4-1
h llo WL L
II Dur Wr Dr,llaa
K
0(1) |l
II W, W.D
k=
r=i+1
L II r=++
S <O(K)
||
<O(K)
II W. WtDrrr- r=/+1
II W.D -1112
II r=1+1
r=1+1
122
m mlog(m)
+ +0
mww2/3 log(m) (m)
SO(K)
(15)
kK
K
= =o k! 11/22/1113 V log(mm) I O(VKm),
where 15 holds due to Lemma and Lemma 7.4(a) in Allen-Zia et al. 2019 and the last equality holds due to the upper bound of w. Consequently,
Vwfw(X)))2== (0(x)))))2=x(((1)) (1312/1133377 5/2 log(m) 0 (VVLKM) (LLKm)
(16)
Proof of Theorem 0:
Note that the hinge loss & convex 1-Lipschitz function. It clear to see that ((u)
is convex with respect to u E R To show its Lipschitzness, take u, RR Thus,
((uu= E(ū)l
(4((4)) ((x)) + (1 A)(E) (max ux) - Li max ūx)) kE[K]
K
kE[K]
K
(6i (ux) l (ūx)) + (1--)
max (ux ux) KE(K]
X
||u ii|l + 2)11 u/l2
K
X +1- 1))lu
xا|l
VK
implics ((-) is a function with Lipschitz constant 2 + 1 - X, and hence [I(()))22 S VR
A +1 A. Additionally,
/K
assume Ae, S Cr, < CA for all E [T', []] thus W(X;W) is Lipschitz mapping with respect to (W( with Lipschitz constant A +1- CAC
VR
Recall the risk function then can be rewritten as
K (fwi + Σ de, (1
k=
20

---

Published in Transactions on Machine Learning Research (05/2025)
and the updating rule is For notation simplicity, substitute w in the first term of 140 as
K
W = - 7n (we(fvi(X)) + Σ A (
R mm
and
let
O=O (A/VK+
1) 13 Tog(m)
//
Therefore, for W E B(W1, ) we have 1 - BVI
vm and
T
1
((fwi(x)))
Σ 1=1
(ffww(x)))
T
(w((fw=<< W W +
b/c of Proposition 10
T
T K
( - W , - +
(e, De Vwle, W* W + v
MT
T
R=1
1
Σ (Il 11mw--nm 1 +/// -w-1- - 1ol-|l/r1 - w -
2mn
T
+
T
Vw - +
-W"
W2L
+1 V +
2m
2011 T
T
Σ
CxCA 12,OVVK [INIFi(X.))
V.w +
+
2
À
I
+1- VKC.SS ((KLm)
2
VK
27:I
o (VKLN 113 / /omm)
+CCCs
+
+1 -1 +1-X+VRR.CAY O
M1 KLm
LR
+
VK
2m T'
+CCA of 14/3 log(m)
(A/VK +1 L3 (m)
1/6
/K
where the last inequality is due to telescoping sum, Proposition 10 and the below fact
|| - w 112 v/
$ 9 (x,i(x Wi)
+1-A+ o
(KWRIS (KW2/3 log(m)
$ nं (
(17)
VK
where 17 holds due the Lipschitzness of the function L(X;; W with respect to fwi (X,) and the usage of 16
21

---

Published in Transactions on Machine Learning Research (05/2025)
Thus, by setting constant e² == VR have
+1 X+ VKCACC))))) + &
1-A VR
KKCACa and choosing 7/2
R VKTm
T
IRRV
L log(m)
{(fw
(Xi))
Σ
(Xi))
+
VT
m//
22


Reviews:
[{"review_id": "RgE8sN9x7K", "reviewer": "Reviewer_uFKt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The author first analyzes set-valued prediction with out-of-distribution (OOD) detection under bandit feedback and proposes an algorithm with theoretical guarantees.\n\n2. Experimental results further validate the efficiency of the proposed method.\n\nWeaknesses:\n\n1. The bandit feedback setting lacks clarity. In traditional prediction, an action corresponds to making a prediction, and bandit feedback implies observing only whether the prediction is correct. However, in the context of set-valued prediction, it is unclear how actions should be defined for general sets. Should a label be randomly selected from the set, or should the entire set be treated as an action, close to combinatorial bandits? Moreover, the nature of the observations in this setting remains ambiguous. Additionally, when an empty set is predicted for an OOD case, does the action correspond to an empty set or a specific OOD label? Furthermore, can the bandit setting determine whether the OOD decision is correct?\n\n2. In \"Recall Control for Normal Classes,\" the author transforms the loss function using Bayes' theorem. However, this approach may not be valid when another loss function is considered simultaneously, as the interaction between multiple loss formulations is not well justified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work examines set-valued prediction with out-of-distribution (OOD) detection under bandit feedback, where the agent can only observe the output for selected arms—meaning the label is only revealed when the current prediction is made. In this setting, the author proposes a novel algorithm that converges to the optimal policy at a rate of $1/\\sqrt{T}$. Experimental results further demonstrate the efficiency of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The current description of the bandit feedback setting is unclear and requires a more precise formulation. (See Weakness 1)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xFUEoOFMA1", "reviewer": "Reviewer_f3Wq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts.** The proposed method appears quite interesting and seems to tackle an important problem, extending prediction sets that also account for OOD detection. The paper appears well written. It is appropriately placed among contemporary literature, it is well motivated and clearly organized. The presentation of the contributions and experiments is quite clear. The experimental evaluation appears comprehensive and the results convincing. The authors conducted experiments using multiple datasets, comparing with competitive baselines while using many different metrics. They also conducted ablation on the choice of the lambda parameter controlling the trade-off between performance in OOD and the informativeness (size) of the prediction sets. \n\n**Weaknesses.** The recall results for accum_normal_min and accum_nomral_max look a bit cluttered and it makes it a bit difficult for the reader to compare. The results on the hold-out set that seem important as one could view this as the test time performance of the proposed method are in the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the problem of set-valued prediction that also allows for OOD under bandit feedback in multi-class classification settings. In particular, they propose training a neural network with a loss function aiming to satisfy each of the following objectives: minimizing prediction set size, guaranteeing class-conditional coverage, and detecting  as many OOD samples as possible. The authors use a threshold rule on the output of such a neural network to construct prediction sets satisfying the above objectives. They further show theoretically  that their algorithm achieves asymptotically class-conditional coverage as well as they provide a regret upper bound for their algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As the authors do have quite some space (the paper is less than 10 pages), they could bring in the main the additional experimental results on the hold-out set. In addition, they could provide the results for the different metrics in separate figures, so that the comparison can be easier for the reader.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2zOiyDQNkp", "reviewer": "Reviewer_WqAu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: It nice to see that this paper extends the generalized prediction set problem to the online setting with bandit feedback. The authors also conducted several experiments to empirically verify the effectiveness of the proposed algorithm.\n\nWeaknesses and Questions: \n1. I'm not sure if I fully understand why $\\lambda_{t,k}$ should change over time. It seems that risk function (Eq. (7)) you try to optimize is changing with respect to your algorithm (Algorithm 1), which is weird. \n2. Assumption 3 seems strong to me: both $\\lambda_{t,k}$ and $\\Delta_{t,k}$ are changing with respect to the proposed algorithm; their upper bounds should be proven, instead of assumed.\n3. It seems to me that many theoretical results can be vacuous: e.g., in Theorem 5, the value of $p_k$ can be very small which makes the recall meaningless. Similarly, it's not clear how large is the value of $c^*$ in Theorem 6 since it depends on $C_\\lambda$ and $C_{\\Delta}$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the generalized prediction set (GPS) problem to the online setting with bandit feedback. The authors propose an algorithm for this new setting and analyze its properties under certain assumptions. Experimental results show that the proposed algorithms outperform baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see my comments above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies set-valued prediction with out-of-distribution (OOD) detection under bandit feedback. The authors proposed BanditGPS algorithm which extends Generalized Prediction Set for online learning with OOD detection, analyzed its theoretical guarantee by proving a regret convergence rate of $O(T^{-1/2})$, and empirically validated its effectiveness.

The reviewers recognized the importance of problem, the interesting proposed algorithm, and the sound theoretical analysis. The reviewers also raised several concerns about unclear problem formulation, presentation issues, strong assumptions and confusing theoretical results. The authors addressed most of the concerns in response and revised paper, while one reviewer believes the concerns about strong Assumption 3 and explanation of Theorem 5 remain. Two reviewers are positive about the paper and one reviewer recommended rejection. Overall, AE shares the positive opinion about the paper while agreeing with reviewers ygBb's concern about assumption. Thus, the recommendation is 'Accept with minor revision'. The authors are suggested to further clarify in the camera-ready version about: 1) the limitation of Assumption 3 and potential theoretical solution of upper bounding $\lambda_{t,k}, \Delta_{t,k}$, and 2) relationship to combinatorial bandits and potential extension.
Decision: Accept (poster)