Contextual Combinatorial Bandits With Changing Action Sets Via Gaussian Processes
Authors: Andi Nika
Sepehr Elahi
Cem Tekin
Venue: Accepted by TMLR ()

Abstract: We consider a contextual bandit problem with a combinatorial action set and time-varying base arm availability. At the beginning of each round, the agent observes the set of available base arms and their contexts and then selects an action that is a feasible subset of the set of available base arms to maximize its cumulative reward in the long run. We assume that the mean outcomes of base arms are samples from a Gaussian Process \rev{(GP)} indexed by the context set ${\cal X}$, and the expected reward is Lipschitz continuous in expected base arm outcomes. For this setup, we propose an algorithm called Optimistic Combinatorial Learning and Optimization with Kernel Upper Confidence Bounds (O'CLOK-UCB) and prove that it incurs $\tilde{O}(\sqrt{\lambda^*(K)KT\gamma_{KT}(\cup_{t\leq T}\mathcal{X}_t)} )$ regret with high probability, where $\gamma_{KT}(\cup_{t\leq T}\mathcal{X}_t)$ is the maximum information gain associated with the sets of base arm contexts $\mathcal{X}_t$ that appeared in the first $T$ rounds, $K$ is the maximum cardinality of any feasible action over all rounds, and $\lambda^*(K)$ is the maximum eigenvalue of all covariance matrices of selected actions up to time $T$, which is a function of $K$. To dramatically speed up the algorithm, we also propose a variant of O'CLOK-UCB that uses sparse GPs. Finally, we experimentally show that both algorithms exploit inter-base arm outcome correlation and vastly outperform the previous state-of-the-art UCB-based algorithms in realistic setups.

Published in Transactions on Machine Learning Research (09/2025)
Contextual Combinatorial Bandits With Changing Action Sets Via Gaussian Processes
Andi Nikal
andinika@miissus..om
Mar Planck Institute for Software Systems
Germany
Sepehr Elahit
sepchr.clahi@@eppl..dl
Department of Computer and Communication Sciences
EPFL
Switzerland
Cem Tekin
cemtekin@ee. bilkent. edu.
Department of Electrical and Electronics Engineering
Bilkent University
Turkey
Reviewed on OpenReview: https: // openreview. net/ forum?
Abstract
We consider a contextual bandit problem with a combinatorial action set and time-varying base arm availability. At the beginning of each round, the agent observes the set of available base arms and their contexts and then selects an action that feasible subset the set of available base arms to maximize its cumulative reward in the long run. We assume that the mean outcomes of base the context set X, and the expected reward is Lipschitz continuous in expected base arm outcomes. For this setup, we propose an algorithm called Optimistic Combinatorial Learning and Optimization ion with Kernel Upper Confidence Bounds (O'CLOK-UCB) and prove that it regret with high probability, where (Kr(U))rr))) is the maximum information gain associated with the sets base arm contexts X, that appeared in the first T rounds, K the maximum cardinality of any feasible action over all rounds, and X (K) is the maximum eigenvalue of all covariance matrices of selected actions up to time T, which is function of K. To dramatically speed up the algorithm, also propose variant of O'CLOK-UCB that uses sparse GPs. Finally, we experimentally show that both algorithms exploit inter- base arm outcome correlation and vastly outperform the previous statc-of-the-art UCB-based algorithms in realistic setups.
samples from a Gaussian Process (GP) indexed by
incurs (((x-(K) (/V/X-KKKTTTaaa KKTTer(Uugrrr
1 Introduction
The multi-armed bandit (MAB) problem is a cornerstone of reinforcement learning, capturing the funda- mental challenge of sequential decision-making under uncertainty: an agent repeatedly chooses actions and refines its beliefs to maximize cumulative reward (Auer al., 2002b; Bubeck & Cesa-Bianchi, 2012; Auer et al., 2002a). Among its many extensions, two have been especially influential due to their broad practical relevanco-frov recommender systems to personalized torial semi-bandits. In contextual bandits, the agent observes side information (context) al the beginning of each round and uses it to guide its secision-making (Lu et al., 2010; Langford & Zhang, 2007; Slivkins,
namelly contextual bandits and combina-
Work done while authors were at Bilkent University.

---

Published in Transactions on Machine Learning Research (09/2025)
2011: Chu et al., 2011). In combinatorial semi-bandits, the agent selects subset of base arms and receives leedback on each selected arm, thereby enabling richer action spaces and outcome observat (Cesa-Bianch) & Lugosi, 2012; Chen et al., 2013; 2016a; Kveton et al., 2015c;a; Zong et al., 2016; Hiranandani et al., 2020). Importantly, in many applications, the set of available arms itself may vary over time so-called changing action sets (Chakrabarti al., 2008; Li et al., 2019; Bnaya et al., 2013; Kleinberg et al., 2010).
This paper focuses on contextual combinatorial multi-armed bandits with changing action sets (C3-MAB), framework that naturally captures repeated interactions in dynamic environments. In each round t, the agent observes the available base arms and their contexts, selects subset (a super arm), receives reward, and observes noisy outcomes for the chosen arms. We allow base arm availability to change arbitrarily across rounds, and thus analyze regret under any sequence of availability patterns. At the same time, for any given base arm and context, outcomes are drawn from a fixed distribution parameterized by that context. The central challenge is to maximize cumulative reward without knowledge of future context arrivals or the underlying outcome function, requiring the agent to adaptively balance exploration and exploitation in real time.
When the set of possible contexts is large, learning without additional structure becomes infeasible. To overcome this, we impose smoothness via Gaussian processes (GPs), which offer flexible yet powerful modeling assumption that enables generalization across contexts (Contal et al., 2013). Specifically, we assume the base arm outcome function is a sample from a GP with a known kernel, ensuring both tractability and expressiveness.
GP-based bandit algorithms have long history of addressing the exploration exploration-exxloitatiio exploitation dilemma, begin- ning with the seminal work of Srinivas et al. (2012). Their closed-form posterior mcan and variance yield high-probability confidence bounds on expected rewards, enabling principled action selection. A key advan- tage of GP-UCB over classical UCB methods lies in its ability to assign meaningful uncertainty estimates even to unseen actions, thanks to posterior smoothness. This property is particularly valuable in environments with time-varying action sets, as it naturally encodes the structure required for efficient learning.
1.1 Our contributions
Below we describe the main contributions of this paper.
We propose a new learning algorithm called Optimistic Combinatorial Learning and Optimization with Kernel Upper Confidence Bounds (O'CLOK-CCB), tailored to solve the C3-MAB problem over com- pact context spaces under the assumption that the expected base arm outcomes are samples from GP and the expected reward is Lipschitz for the expected base arm outcomes. O'CLOK-UCB incurs ò ((A**))KTT<rr(lee)))
regret in T rounds in the general case when the feasible action cardinalities
are bounded variables, where (Kr(UUTTX) is the maximum information gain with respect to the sequence of available context sets X,, to be explicitly defined later, over the feasible actions with cardinalities upper bounded by K N, and X (K) is the maximum eigenvalue of all covariance matrices of the selected actions up to time T.
Our work resolves new theoretical and technical difficulties introduced as result of simultaneously taking into consideration several bandit features.
The candit feedback necessitates batch update of all the base arm indices at once after all base arms are selected. This, in return, requires new way of upper-boundie substantially different. from previous related work (Srinivas et al., 2012). In order to account for a non-identity covariance matrix (due to dependencies among base arms in an action), we leverage the conditional entropy of Gaussians and propertics of cigenvalues (see Lemma 1). Therefore, our setting is not a straightforward extension of a GP bandit in KT rounds.
the expected regret,
We adapt the classical notion of maximum information gain to the C3-MAB setting (represented by TKr(UUsT which takes into account time-varying base arm availability, and provide new regret upper bounds based on this notion of maximum information gain.
2

---

Published in Transactions on Machine Learning Research (09/2025)
We perform semi-syuthetic crowdsourcing simulations real-world dataset and compare algorithm with the previous state-of-the-art. Moreover, we illustrate how the time complexity of O'CLOK-UCB can be improved by using the sparse approximation method, where we only use a subset of the past history of selected contexts to update the posterior. We experimentally show that our method out performss the previous stale-of-the-art UCB-based algorithuns in this setting.
1.2 Related work
The Combinatorial MAB has been widely studied and well understood in the last decade (Cesa-Bianchi & Lugosi, 2012; Chen et al., 2013; Combes et al., 2015; Kveton et al., 2015e; 2014; Chen et al., 2014), together with its various branches such as cascading bandits (Kveton ct al., 2015b), and combinatorial bandits with probabilistieally triggered arms (Chen et al., 2016b; Huyuk & Tekin, 2019).
In recent years, the Contextual Combinatorial aulti-armed Bandit (CCMAB) problem has been attracting a lot of interest (Chen et al., 2018; Qin et al., 2014). Chen et al. (2018) studies the CCMAB problem with cascading feedback; that is, the learning agent can observe base arm outcomes of only a prefix of the played super arm. Their model allows for position discounts and a wide range of reward functions. Under monotonicity and Lipschitz continuity of the expected reward, their algorithm, CS-UCB, incurs Ö(vKT) regret in T rounds, where K is the maximum cardinality of any feasible super arm.
The contextual combinatorial multi-armed bandits with changing action sets has been studied in (Chen et al., 2018; Nika et al., 2020). Chen et al. (2018) assumes the expected reward to be submodular, and the expected base arm outcomes are assumed to be Hölder continuous with respect to their corresponding contexts. Their proposed algorithm (CCMAB) addresses the changing arm sets over rounds by uniformly discretizing the context space into a predetormined number of hypercubes (which depends on the time horizon T), thus utilizing the outcome similarities between nearby contexts. CCMAB incurs ((T())))))())) regret where 1) is the Hölder constant and D the context space dimension.
A potential drawback of this approach is the fixed discretization of the context space, which implics limited exploitation of the arms' similarity information. Nika et al. (2020) addresses this issue by adaptively dis- cretizing the D-dimensional context space (I1,)-112) following a tree structure, under some mild structural assumptions on (X, (1-112) (where Il-I12 is the Euclidean norm) and Lipschitz continuity assumption both of the expected reward with respect to the expected base arm outcomes and the latter with respect to their associated contexts. Their algorithm (ACC-UCB) incurs O(T(D+))((+2)+)) regret for any <>> 0 where D is the approximate optimality dimension of the context space X, tailored to capture both the benignness of the base arm arrivals and the structure of the expected reward under combinatorial setup. In general, D < D, implies an improvement of the bound rates to the previous work.
Adaptive discretization has been utilized in the GP bandit setting by Shekhar & Javidi (2018) for black-box function optimization. Under the same structural assumptions of the arm space, (X, d) as in (Nika el al., 2020) (where d is a general metric associated with X), and under some Hölder continuity assumptions on their covariance function, they propose tree-based algorithm whose aim is to maximize the cumulative reward given fixed budget of samples. They provide both near-optimality dimension type and information type regret bounds. It is worth mentioning that when the context space is very large, and the arm set is time-varying, adaptive discretization has proved to be an efficient technique, yielding optimal regret bounds (Bubeck et al., 2011). Nevertheless, if we further assume that the number of arms that come in a round is finite, imposing a GP prior on the base arm outcomes removes both the need for adaptive discretization and Lipschitz continuity of the expected basc arm outcomes. The postcrior's smoothness encodes enough information to estimate with high probability the outcomes of any available base arm, regardless of its history of arrivals. In light of this, we approach the C3-MAB problem using a simple procedure under mild assumptions.
The main difference between adaptive discretization, and our GP-based method is that, while adaptive discretization discretizes the search space into regions and maintains statistics over each region, GPs offer a functional approach. Such an approach allows us to retrieve point-wise statistical information in a fine- grained way just by evoking the posterior mean and variance. On the other hand, adaptive discretization operates on lower resolution, maintaining the same statistical information for all points in the same region.
3

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Comparison with related works. Below, represents the fixed cardinality of feasible super arm; (K) denotes the maximum eigenvalue of all covarinnce matrices of selected actions up to time T; D is the approximate optimality dimension introduced in (Nika et al., 2020) and their bounds hold for any > 0; T is the Hölder constant and v is the Matérn parameter. The bounds are shown up to polylog factors.
Work
Context
Function
Smoothness
Oracle (approx.)
Contextual &
Regret bounds
space
assumption
Changing action sets
(Chen al., 2013)
Finite
Lipschitz
Explicit
(a,p)
No
VKT (KT(0+2)//B 0 [TTT-I//B4)))
(Chen et al,, 2018)
Infinite
Submodular
Explicit
(1 - 1/e)
Yes
(Nika et al., 2020)
Compact
Lipechitz
Explicit
Yes
Ours (Linear kernel)
Compact
Lipechitz
GP-induced
Yes
Ours (RBF keruel)
Compact
Lipechitz
GP-induced
Yes
Ô V (X(K)KDTlog")
Ours (Matérn kernel)
Compact
Lipschitz
GP UP-induced
a
Yes
Ô (X (X)(K(TD (K)T(D40//0020))
Consequently, adaptive discretization uses less computational resources, since it focuses only on 'relevant' regions and adaptively refines them based on historical information, while sacrificing resolution. In contrast, GPs offer a ligh-resolution picture of all relevant statistios over the search space, thus allowing for a more precise inference, with an additional computational cost.. We have resolved this latter issue by proposing a practical version of our algorithm. Furthermore, we have provided an experimental comparison between GPs and adaptive discretization, thus showcasing the benefit of using GPs as opposed to the latter.
More recently, Elahi et al. (2023) extended the C3-MAB setting by incorporating group constraints. In their work, base arms belong to groups, and the selected super arm must satisfy constraints based on rewards associated with these groups. They utilize a multi-output Gaussian Process to model distinct outcomes for the primary super arm reward and the group constraint evaluation. Their proposed TCGP-UCB algorithm uses a double- UCB approach to explicitly balance reward maximization and group constraint satisfaction. While tackling a different objective involving constraint satisfaction, their work also leverages GPs for C3- MAB and arrives at regret bound form similar to ours.
Other works which are closest to ours include (Sandberg et al., 2025). The problem they consider, namely the combinatorial volatile Gaussian process semi-bandit problem, is the same as ours. In addition to regret bounds which have similar dependence, we also adapt the classical notion of information gain to the C3MAB setting. Moreover, while they consider the notion of expected (Bayesian cumulative) regret, we provide high-probability upper bounds on the notion of a-approximation regret. Additionally, (Accabi al., 2018) and (Nuara et al., 2022) propose algorithms with similar rationale to O'CL.OK-CCB. However, while our algorithm is provably no-regret in the C3-MAB setting with (potentially) infinitely many context-dependent timo-varying arms, the setting considered in (Accabi et al., 2018) and (Nuara et al., 2022) is standard combinatorial bandit setting with context-free time-invariant finite arm set.
In Table 1, we compare characteristics and regret bounds between our work and previous work when the cardinality of feasible super arms is constant. Note that we achieve a substantial improvement from (Nika et al., 2020) and (Chen et al., 2018) on the order T in the case of linear and squared exponential kernels.
The rest of the paper is organized as follows. In Section 2 we lay down sctup and recall some definitions of Gaussian processes and their properties; in Section 3 we introduce proposed algorithm and explain how it works; in Section 4 we provide the regret bounds for OCCLOK-UCB; Section 5 contains the experimental results, and we conclude in Section 6.
2 Problem formulation
We first introduce the general notation to be used throughout the paper. Let us fix a positive integer K 2 1 and a D-dimensional vector space X. We write [K] = (1, K}. Vectors are denoted by bold lowercase letters. Furthermore, given N E N, Iy denotes the identity matrix NxN and IT-)) denotes the indicator function.
4

---

Published in Transactions on Machine Learning Research (09/2025)
2.1 Base arms and their outcomes, super arms and their rewards
Our setup involves base arms that are defined by their contexts, x, belonging to the context set X. Let r(z) represent the random outcome generated by the base arm with context z. We assume that there exists a function f X + IR, such that we have r(x) = f(x) + 7i, for every x E X,, where n~ ~ o0) We assume that 17 is mutually independent across base arms and observations.
A super arm S is a feasible subset of the base arm set and it is defined by the contexts of its base arms. Consider a super arm S associated with the context tuple æ = [1s) such that I'm E X, The corresponding outcome and expected outcome vectors (the latter is also called the expectation vector) are denoted by r(z:) [r(z))...., r(ri.) (=(ss) )1"" and f(x) = [f(a))..... -(.())))....,(((sss)))) (x]ss)]? We assume that the reward received from playing this super arm is a non-negative random variable denoted by U (r()))). Moreover, we assume that the expected reward of playing any super arm is a function only of the set of arms and the mean vector, and we define v(((x)) = ())()())) Again, this assumption is standard in combinatorial bandits (Chen al., 2013). In addition, we impose the following mild assumptions on u, which allow for a very large class of functions to fit our model such as multi-armed bandit with multiple plays, maximum weighted bipartite matching, and probabilistie maximum coverage (Chen et al., 2013).
Vm ........$17 |S|}.
Assumption 1. (Monotovicity) Let S be a feasible super arm. For any f = If.....,,.ss. fisil" E 12 and f' If. I's|"" E ERIS, if fm I fmm Vm E [S.,...|[S), then u(f) S u(f').
Assumption 2. (Tipschitz continuuly of the expected reward in expected outcomes) 3B > 0 such that for any feasible super arm S, f [f....., fis E gllsl and f = [f,,..., KsjI" E @RIS we have lu(f) u(f')
Assumption states that the expected reward is monotonically non-decreasing with respect to the expected outcome vector. Assumption 2 implies that the expected reward varies smoothly as function of expected basc arm outcomes.
2.2 The optimization and learning problems
We consider sequential decision-making problem with time-varying base arms that proceeds over T' rounds indexed by t E [T]. The agent knows perfectly but does not know f beforchand. In each round t, M, base arms indexed by the set MM = [Mr] are available. We assume max M, I M, for some integer M. The context of base arm m E MM is represented by It,m E X. We denote by X; = jea.mmeee the set of available contexts and by fe = [f(
the vector of expected outcomes of the available base arms in round t. We denote by S, the set of feasible super arms in round t and S = U221S, the overall feasible set of super arms. Note that S, depends on the structure of the optimization problem which is known by the learner, and if S E S,, then S S Mr. Furthermore, we assume that the budget (maximum number of base arms in super arm) does not exceed some fixed integer K N, that is, for any S E S, we have |S| I K. At the beginning of round t, the agent first observes M, and X. Then, it selects a super arm St from S.,
2.2.1 The optimization problem
For a moment, consider the hypothetical situation where f is known beforehand. Then the agent would have selected an optimal super arm given by S7 E argmax (( u(f(x;,))) and subsequently obtained an expected reward of (ft) = maxses, (x),s)) However, it is known that, in general, combinatorial optimization is NP-hard (Wolsey & Nemhauser, 2014), and thus, efficient computation of S7 is not possible, even if ʃ were known. Fortunately, for many combinatorial optimization problems of interest, there exist computationally efficient approximation oracles. Thus, we assume that the agent obtains S, from an a-approximation oracle, which when given as input fo and the specific structure of the particular optimization problem, returns a super arm ( such that u(f( (rraeett))))) 2 a x (ft(
Note that Oracle also takes as input the feasible set S,, but this is omitted from the notation for brevity.
5

---

Published in Transactions on Machine Learning Research (09/2025)
2.2.2 The learning problem
Since the agent does not know f in our case, calls the approximation oracle in each round & with an M-ddimensional parameler vector 0g to get S. = Oracle(O:). Here, 0, an approximation to ft calculated based on the history of observations. We will show in Section that our learning algorithm uses upper confidence bounds (UCBs) based on GP posterior mcan and variances as lt. It is important to note here that St is a-oplinal solution under O, but not necessarily under St- At the end of round 4, the agent collects the reward ((r(zt,s)) where it,S; = ["t,,,........ 11,,1111111 I" is the set of context vectors associated with super arm St and r( - (r ..... F...,,,(,,,,, ( )I2 is the outcome vector of super arm S., It also observes r(x),s)) as part of the semi- bandit feedback. The goal of the agent is to maximize its expected cumulative reward in the long run.
To measure the loss of the agent in this setting by round T for a given sequence of base arm availability {K we use the standard notion of a-approximation regret (referred to as the regret hereafter), which is given as (T) = a E opt(ft) - Ei u(f(xI,))).
2.3 Example applications of C3-MAB
In this section, we detail some applications of C3-MAB.
Dynamic maximum weighted bipartite matching. Let G.= (Le, R, E₁) represent the bipartite graph in round 4, where La and R are the set of nodes that form the parts of the graph such that [h,] 2 K and [R] = K (here, K represents the budget), and E, is the set of edges. Here, each edge (i,j) E E, such that E La and j E R, represents a basc arm. The weight of the edge (1,5) E E, is given by (I),())) In this problem, Se corresponds to the set of K-element matchings of G.. Hence, given S E St, U(r(xr,)) Emes (It..m) and u(f(x1,))
Emes f. St can be computed by the Hungarian algorithm or its variants (Kuhn, 1955) in polynomial computation time, and hence 1. This problem can model the dynamic assignment of |R| resources to |a available users in round f, where each resource can be assigned to at most one user. Anexample application is multi-user multi-channel communication (Gai el al., 2012).
Dynamic probabilistic maximum coverage. Let G. (U,, E., E,) represent the bipartite graph in round t, where U, and V2 are the set of nodes that form the parts of the graph, and E, is the of edges such that [U] 2 K. Each edge (i,j) E E, has coneeet-de activation probability given f (1)), i.e., when a node i E U, is chosen, it activates ils neighbor with probability ((x2 independent of the other neighbors of j. Here, the goal is to choose a subset of K nodes in Us that maximizes the expected number of activated nodes in V.. Although this problem is known to be NP-hard (Chen et al., 2016a), there exists a deterministic a = (1 - (jje)))pprxximation oracle (Nemhauser et al., 1978).
2.4 Putting structure on base arm outcomes via GPs
Since the agent does not have any control over M, and Xe, the expected outcomes of available base arms can vary greatly over the rounds. Since how well the learner performs depends on how well learns the unknown f, we need to impose regularity conditions on f. In this paper, we model as a sample from a Gaussian process, which is defined below.
Definition 1. A Gaussian Process with inder set X is collection (f(x)) of random variables which satisfies the property that n E N. The probability law of GP (f(x)) is uniquely specified by its mean function x + p(z) = E[f(x) and its covariance function x2) + k( x2) = ([(((a1) - p(x)))
f(I,)) is Gaussian random vector for all (z1..... .n. E X and
- ((x2)))
We assume that for every I E X, we have k(z, x) < 1. This is a general assumption that is widely used in the GP bandits literature (Srinivas et al., 2012). Now let us recall the closod-form expressions for the postcrior distribution of Ga-sampled functions, given a set of observations. Fix N N. Consider a finite sequence din] [NN] of contexts with the corresponding vector F[N] = r(mp)) [r(m))...,, of
6

---

Published in Transactions on Machine Learning Research (09/2025)
outcomes and corresponding vector fIN] have r(zn) (x)) + M/, where the is the noise that corresponds to this particular outcome.
of expected outcomes. For every n S N, we
The posterior distribution of f given M[N] is that of a GP with mean function Jon and covariance function by given by
(2( = (KIN) + IZ NN (x, = k(x,x')= - (kIN)(z)) (K(M + I 1k(N(((a)))))
(1)
(2)
(x) kN (x,
where (x) = [k(x1, r)..,, x)]" E R and KMM = [k( J.y di> particular, the posterior distribution of f(x) is N ((un(2), (x)) and that of the corresponding observation r(x) is ((ux(x), (x) o²).
2.5 Information gain
In Bayesian optimization, the informativeness of a finite sequence d[d is quantified by the information gain, defined as [TTNN] f(N)) -Brr) where H-)) denotes the entropy of a random variable and (-))fxn) denotes the conditional entropy of a random variable with respect to fIN]. The information gain gives us the decrease in entropy of f(N] given the outcomes r[N]- We define the maximum information gain as YN = maxa(x] [(MNni S(N))-
Note that IN is the maximum information gain associated with any N-tuple of elements of X. For large (potentially infinite) spaces this quantity can be very large, depending on the chosen kernel. On the other hand, the dynamically varying base arm availability in our setting does not require such information. We only need the maximum possible information coming from a fixed sequence of base arm (context) arrivals. Thus, such notion becomes redundant in this case. This motivates us relate the growth rate of the regret with the information content of the available base arms. Hence, we begin by adapting the definition of the maximum information gain so that it accommodates base the regret bounds. Given & 2 1, let Z, C 2 be the set of context vectors corresponding to the feasible set S, of super arms. Let 2; := Lt,s, for some S E S,, be an element of Z, and let 2[T] = [2T,..., 27]....7.. We define the maximum information gain associated with the sequence of context arrivals X1, Nr as
availability, and guarantees the tightness of
7KT (Xp)=
max E,
1 (((())))
(4)
where Ny. = U. The information gain from any sequence of T selected super arms is upper bounded by 7KT (X7). Our regret bounds depend on it. Note that (KT( is the usual notion of the maximum information gain applied onto the union of all available context sets over rounds.
3 The learning algorithm
Our algorithm is called Optimistic Combinatorial Learning and Optimization with Kernel Upper Confidence Bounds (O'CLOK-UCB), with pseudocode given Algorithm The procedure described as follows: At round f, observe available base and their contexts. For each available base arm, we maintain index which is upper confidence bound its expected outcome. Let , be sequence of super arms. Given base arm m with its associaled context Zt,m: we define its index based on the observations ...., as:
T
de (F = M[t-1] 11 (Fr,m) + 8/ T1- (Tt,,))
where Me=11 ()) and () stand for the posterior mean and variance (respectively) given the vector irg,.... r - of observations up to round t. Furthermore, Be is a parameter that depends on whose exact value will be specified later. We denote by (x(2),s) the vector "t.t....)....,((((a..... Ir Note that our definition of the index of arm with context, x in the beginning of round t depends on mi- (x) and
7

---

Published in Transactions on Machine Learning Research (09/2025)
Environment
Learner
Arms Contexts
Arms &
Compute indices
(21.1..... M, JeST
Pt.1 5t,MM
Additional Problem Structure
Problemn structure in (eg. bipartite graph
OCLOK-UCB
Orucle
Superarm St
Expected Arm Outcomes
Update posterior
((ra)) - MhST
Random arm outcomes reward
(russ,) and U(riss)))
Figure Illustration of the steps of our algorithm for round PMC stands for probabilistic maximum coverage.
(r) This is due to the combinatorial nature of the problem. Since, at the beginning of round t, the algorithm has obtained semi-bandit. feedback from Er |Spl context selections, the posterior mean and variance of f calculated at that round will depend the selected super arms, namely, ,,,...,S
Algorithm O'CLOK-UCB
Input: X, K, M; GP prior: Ho = μ4, Ko k.
Initialize: μ40 = j,, ko k. do
for t=
Observe base arms in M, and their contexts X..
for Zt.m m E .M, do
Calculate M and (1 as in equation 1 and equation 3.
Compute index (m) as in equation 5.
end for
Oracle(i (z,,,), (e))).),)).
Observe outcomes of base arms in St and collect the reward.
end for
After the indices of the available base arms (i.c., (de(Fa,m)) are computed, they given input to the approxination oracle in round t to obtain the super arm S, (st,I...., st,1s.)) S Mr that will be played in round t. Note that St is an (approximately) optimal solution under 0,, but not necessarily under ft- We assume a deterministic oracle in order to guarantee our theoretical results.
After observing the semi-bandit feedback from the selected arms, at the beginning of the next round, we update the posterior distribution of f based on S, according rules equation 1 and equation 3, which will be used to compute the indices for the next round. Figure illustrates the steps of our algorithm for round t.
4 Regret bounds
We start by stating main result. which gives high probability upper bound on the a-regret in terms of (x)). Detailed proofs of all the stated results be found in Appendix A.
Theorem 1. Let 5 10, T E N. Given 8, 2 2lg /36), the regret incurred by O'CLOK-UCB in T rounds is upper bounded as (T) I BC with probability at least 1 - 8, where C(K) 8B² + o²), and X (K) is the marimum eigenvalue of all covariance matrices of selected actions over T rounds.

---

Published in Transactions on Machine Learning Research (09/2025)
To prove Theorem 1, need to utilize new result that gives lower bounds (KT(AT) in terms of quantities that are similar to those that upper bound the expected regret.
Lemma 1. Let st,s, be the vector of selected conterts at time t 2 1. Given T 2 we have:
T
S
I (r(zm) 2 2(o-2X
o
(Tr,))
where 2(7] [2,,..., zrl" is the vector of all selected conterts until round T and (K) is the marimum eigenvolue of matrices (2))-1(214)
Remark 1. Since, in every round, the outcomes of the selected base arms observed only after the selection process is over, cannot reduce the problem to sequential decision-making in KT rounds, in which case, the outcomes would allow for the sequential update of the indices and let us directly use the formula of the mazimumm information gain from (Srinivas el al,, 2012). We solve this problem by lower bounding the information gain. Consequently, Lemma 1 takes into account the conteztual combinatorial nature of the problem.
Finally, using kernel-dependent explicit bounds on (KT(XT given in (Srinivas et al., 2012; Vakili el al., 2020), we state corollary of Theorem which gives similar bounds on the a-regret incurred by O'CLOK-UCB.
Corollary 1. Let 8 (0, 1), T, E and let X C R.D be compact and convex. Under the conditions of Theorem and for the following kernels, the a-regret incurred by O'CLOK-UCH in T rounds is upper bounded (up to polylog factors) with probobilily at least - 5 as follows:
For the linear kernel we have: (T( 50 ( (V((K)KKT)
For the RBF kernel we have: (T( 0 (V~(K))TT log" T)
For the Malérn kernel we have: (T) S ( where v > 1 is the Malérn
parometer.
5 Experimental results
We perform simulations on: (i) a crowdsourcing setup with computational time comparisons, (ii) synthetic dataset to show how our algorithm exploits arm dependence, and (iii) real-world movie recommendation setup. All simulations were run using Python, with source code provided on our GitHub, on PC running Ubuntu 16.04 LTS with an Intel Core 17-6800K CPU, an Nvidia GTX 1080Ti GPU, and 32 GB of 2133 MHz DDR4 RAM.
5.1 Simulation I: Crowdsourcing
We evaluate the performance of O'CLOK-UCB by comparing it with ACC-UCB (Nika et al., 2020) and AOM- MC (Chen et al., 2018), two UCB-based algorithms. We perform semi-synthetic crowdsourcing simulations using the Foursquare dataset. The Foursquare dataset (Yang et al., 2015) contains check-in data from New York City (227,428 check-ins) and Tokyo (573,703 check-ins) for a period of 10 months from April 2012 February 2013. Each check-in comes with location tag well as the time of the check-in. In our simulations, the TKY dataset because its locations more spread out than the NYC dataset.
5.1.1 Simulation Setup
We perform a crowdsourcing simulation where the goal is to assign workers to arriving tasks. our sim- ulation, 250 tasks arrive (i.c., T 250), each with location (normalized longitude and latitude) sampled
Link: http:://ethhoccco////eeeeicccooo

---

Published in Transactions on Machine Learning Research (09/2025)
uniformly at random from [0, 1/2 and difficulty rating, sampled from [0 each worker also has 2D location in [0, 1J2, sampled from the TKY dataset, and a battery status, sampled from [0, 1].
1] (0 is most difficult). Similarly,
In round t, the agent observes the set of available workers, the expected number of which is sampled from a Poisson distribution with mean 100. The set is composed of workers whose distance (Fuclidean norm) to the task is less than v0.5. Then, each worker-task pair is an arm, with three-dimensional context comprised of the scaled distance between the worker and task, 3 the task's difficulty, and the worker's battery. We also set fixed budget constraint of K = 5, which means that workers should be chosen for each task.
Then, given the task-worker joint context (i.e., arm context) and, noting that =²,x², and x represent the worker-task distance, task difficulty, and worker battery, respectively, we define the expected outcome of each base arm as f(x) = B[[(x)] = Ag (z') Vx² x², where g is Gaussian probability density function with mean 0 and standard deviation 0.4, and A = v2T -0.4 is the scaling constant. Note that f is decreasing in the worker-task distance and task difficulty, and increasing in the worker's s battery. Then, the random quality of each worker with joint context a is defined as r(x) f(x) + 7. where is zero mean noise with a standard deviation of 0.1.
Finally, in round and given chosen workers with joint task-worker context vector x [2 FK), we K
define our reward as U(r(#)) log 1+E Σ r(as)) E())) Notice that the log term reflects the diminishing relurn having multiple workers. Moreover, since log is an increasing function, the Oracle for this setup will be greedy one that picks the arms whose contexts have the highest performance estimates.
5.1.2 Algorithms
O'CLOK-UCB: We use the GPflow library (Matthews et al., 2017) for the sampling from and updating the Gaussian Process. We d 0.05 and use squared exponential kernel with both variance and lengthscale set to 1.
SO'CLOK-UCB: In this variation of our algorithm, we use the sparse approximation to the GP posterior described in (Titsias, 2009). In this sparse approximation, instead of using all of the arm contexts up to round t to compute the posterior, a small s element subset of them used, called the inducing points. The non-sparse O'CLOK-UCB requires updating the GP posterior at cach round. A standard, efficient implementation performs this sequentially using block matrix inversion, resulting in a total computational complexity of O(KTT) over T rounds. By using sparse approximation with inducing points, this is dramatically reduced to a total complexity of KT²). We use this sparse variation with inducing points uniformly at random from all of the contexts picked so far. In other words, in round f, we sample contexts from (II,1.-.- (f1,1:-..11,K Ki,K ... }- We run simulat ions with three different inducing points: 10, 20, and 50. We also set 0.05 and use the squared exponential kernel.
ACC-UCB: We set U1 v3,02 02 1,p 0.5, and N 8, as given in Definition 1 of (Nika et al., 2020). The initial (root) context cell, Xo.1, is three dimensional unit hypercube centered at (0.5, 0.5, 0.5).
Since we have a throe-dimensional context space, 300 tasks (rounds), and an exact Oracle, we set hr [300s/ 3, as given in Theorem 1 of (Chen et al., 2018). Hence, each hypercube has length of 1/hr = 3.
CC-MAB:
Benchmark: The benchmark greedily picks the arms with the highest expected outcome.
"To scale the distance we divide it by v0.5 because that is the maximum possible distance.
10

---

Published in Transactions on Machine Learning Research (09/2025)
5.1.3 Results
We ran independent runs and averaged the reward, regret, and time taken over these runs. We present the standard deviations of the runs as error bars. Figure 2 shows the average lask reward up to task i, divided by the benchmark reward; and Figure 3 shows the cumulative regret of each algorithm for different T. The two UCB-based algorithms perform around the same and are equal in performance with SO'CLOK-UCB with 10 inducing points. However, they perform substantially worse for a larger number of inducing points. Interestingly, only 100 inducing points were enough to achieve a very close performance to O'CLOK-UCB, which does not use any approximations for the posterior update. Finally, as expected, the smaller the number of inducing points, the worse the performance of SO'CLOK-UCB compared with the non-sparse O'CLOK-UCR algorithm. However, even with inducing points, SO'CLOK-UCB able outperform both CC-MAB and ACC-UCB by around 30%.
We also plot the time taken for each algorithm to process each round, given in Figure 4. We see that the running time of O'CLOK-UCB does not scale well with the number of rounds, taking more than 20 minutes just to pick workers and update the posterior for the final task. On the other hand, the sparse algorithms do much beller and are not only four to five times faster, but also scale linearly with l. The latter, coupled with the fact that SO'CLK-UUB'ss performance practically identical to that of CCLOK--UBB means that even though SO'CLOK-UCE is slower than the UCB-based algorithms, its performance scales well with time and so can be used for large simulations or online settings. It is worth noting that in Figure 4, the runtime curves for ACC-UCB and CC-MAB superimposed, reflecting their similar and highly efficient computational performance compared to the GP--basdd methods.
Average task reward divided by
1.0
X10
benchmark reward task
1.25
50 SO'CLOK UCB inducing pts.)
0.8
SO'CLOK-UC (20 inducing pts.
Cumulative regret
1.00
(30 inducing pt.))
0.6
SO OK-UCR (100 inducing
SO'CLOK-UCE (10 inducing pt.)
0.75
D'CLOK-UCE
(20 induinn pts)
0.4
SOCCLOK-UCE (50 inducing pta.)
0.50
CC-MAB
SO'CLOK UCB (100 inducing pos.)
OK-UCK
0.25
ACC UCD
0.0
0.00
150
200
50
100
150
200
250
Arriving task
Number of rounds (T)
Figure 2: Average reward of each algorithm di- vided by that of the benchmark Simulation I.
Figure 3: Cumulative regret of each algorithm for different number of rounds (T) Simulation I.
benchmark reward up to round t
SO CLOK-UCB (10 inducing pta)
CLOK-UCR (20 inducing ptx.
Average reward divided
1.00
CLOK UCB (50 inducing sta.
Time taken (8)
SO CLOK-UP UCB (100 inducing pts.)
0.75
ACC-UCB
0.50
500
SO'CLOK-UCR induring pts
0.25
SOCLLOK-UCB (2 inducing
SOYCLOK-UCR (4 incuring pts)
0.00
100
200
100
200
300
Arriving task (8)
Round number (t)
Figure 4: Time taken in seconds to process each round for each algorithm in Simulation I.
Figure 5: Average reward of each algorithm di- vided by that of the benchmark for the Movic- Lens DPMC setup of Simulation III.
11

---

Published in Transactions on Machine Learning Research (09/2025)
O'CLOK-UCB
50' SCB (20 inducing pts.)
SO'CLOK- (50 inducing pts.)
1.0
1.0
1.0
Average task revard divided
0.8
Average task revard divided
0.8
Average task reward divided benchmark revard up
0.8
benchmark revard
0.6
benchmark revard
0.6
0.6
Outaame berel
0.2
0.2
0.2
0.0
0.0
0.0
Dutwer keend 108
Dutamn
Dusomm
0..0
-0.2
-02
-0.2
Dutzam ferell
fareel
100
200
300
0
100
200
300
0
100
200
300
Round number (8)
Round number (1)
Round number (t)
SO'CLOK-UCB (100 inducing pts.)
ACC-UCB
CC-MAB
1.0
1.0
1.0
Average task reward divided
by benchmark reward to !
0.8
Average task reward divided
Average task reward divided
by benchmark reward up to
by benchmark reuard up
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
-0.2
Dusonn Dusamm
0.2
-022
100
300
100
300
0
300
Round number (t)
Round number (1)
Round number (t)
Figure 6: Average reward of each algorithm divided by that of the benchmark for different outcome kernel lengthscales (1) in Simulation II.
5.2 Simulation II: Varying Base Arm Codependency
The goal of this simulation is to evaluate the performance of our approach against baseline methods under varying levels of base arm codependency, demonstrating how our GP-based method effectively exploits base arm dependencies.
5.2.1 Setup
We perform 30 different simulations with evenly spaced number of rounds from T = 10 to T = 300 in order to be able to plot the cumulative regret. Note that this step is needed because ACC-UCB and CC-MAB's decisions are affected by the number of rounds, hence to plot the cumulative regret, we need to run the simulations with different T.
Similar Simulation the number of arms in each round is sampled from a Poisson distribution with a mean of 100 and the budget is K 5. This means that we will have 30000 base arms when T 300. Sampling this many points from a GP requires the creation and Cholesky decompasition of 900 million element matrix, which would not only take long time on our i7 6700K PC, but would also require large amount of RAM (more than 7.2 GB). In order to address this issue, we first generate 6000 3D contexts from jo, 113 and then sample the GP at those points. Then, during our simulation, we sample cach base arm's context x and the corresponding expected outcome f fx) from the generated sets. Just like in Simulation I, r(x) = f(x) + 77, where 17 ~ (00 0.12). Note that we first generate the dataset (i.e., arriving arm contexts and rewards in each round) for T' = 300 and then use the same dataset, but truncated, for smaller T..
The expected outcome of each arm is generated from zero mean GP with the squared exponential kernel with hscale 1, given as k(xx,²) exp (- E {0.01, 0.05, 0.1, 1}. Intuitively, the larger the length scale, the larger the covariance (and thus dependence) between any two arms with nonequal contexts.
2 || x' I)). We run our simulations for five different. lengthscales,
12

---

Published in Transactions on Machine Learning Research (09/2025)
x10
O O'CLK-UCE
x10 SO.CLOK-UCB (20 inducing pts.)
x10" SO'CLOK-UCB (50 inducing pts.)
3.0
Dutame lemel -MM.
3.0
3.0
berell
Ducone benel 1-00
80 Oersome
Dutzame b 0.00
-013
2.5
2.5
2.5
Cumulative regret
Duturre Sevel 100
Curmulativ regret
Oataume hareed in
Cumulative regret
Dutiare bereei in 140
2.0
2.0
2.0
1.5
1.5
1.5
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
100
200
300
100
200
300
0
100
200
300
Number of rounds (2')
Number of rounds (T)
Number of rounds (T)
+10"
50 CLOK-UCB (100 inducing pts.)
10
ACC-UCB
X10'
CC-MAB
3.0
Dutuare lerel
3.0
Optacees haral 88
3.0
Dusiare larel MMC
in
2.5
2.5
2.5
Cumulative regret
Cumulative regret
2.0
2.0
2.0
Cumulative
1.5
1.5
1-5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
200
300
100
200
300
100
200
300
Number rounds (T)
Number rounds (T)
Number of rounds (T)
Figure 7: Cumulative regret of each algorithm at the end of different number of rounds and outcome kernel lengthscales (1) in Simulation II.
Finally, in round t and given K chosen arms context vector [x1,..., æk], we define our reward K u(r(x)) Σ r(x:). Just like in Simulation I, each algorithm will select five arms (i.e., k= 5). Moreover, the Oracle for this setup will be greedy one that picks the arms whose contexts have the highest performance estimates.
5.2.2 Algorithms
We use the same algorithms as in Simulation I, detailed below.
O'CLOK-UCB: We use the same squared exponential kernel as in Simulation and set 6 = 0.05.
SO'CLOK-UCB: We run simulations with three different inducing points: 20, 50, and 100. We also set 8 0.05 and use the same squared exponential kernel as O'CLOK-UCB.
We set 01 = v3, 02 = 1,p = 0.5, and N = 8. The initial (root) context cell, X is a three dimensional unit hypercube centered at (0.5, 0,5, 0.5).
ACC-UCB
Since we have a three-dimensional context space, 300 tasks (rounds), and an exact Oracle, we set hr = [300 = 3. Hence, each hypercube has a length of 1/hr = 1
CC-MAB:
Benchmark: The benchmark greedily picks the arms with the highest expected outcome.
13

---

Published in Transactions on Machine Learning Research (09/2025)
5.2.3 Results
We run each simulation five times and average over them to get the results. We also visualize the standard deviation of the runs as error bars. Figure shows the average lask reward up to lask I, divided by the benchmark reward; and Figure shows the final cumulative regret of each algorithm for different T. None of the algorithms is able to learn and achieve high reward when I = 0.01, which is expected because different arm outcomes are almost independent of one another for this length scale. As the length scale increases, we see that all algorithms improve, but it is only the GP algorithms that manage to achieve performance close to that of the benchmark (i.e., approach > 0.8 in reward plots). SO'CLOK-UCB with 20 inducing points performed the worst among the GP algorithms, which is expected because is essentially trying to learn the entire [0, 1]s arm context space with just 20 randomly picked contexts. However, when the number of inducing points increases, the performance of CO'CLOK-UGH with both 50 and 100 inducing points is practically identical to the non-sparse O' O'CLOK-UCB's performance when the outcome kernel has / = 1. Interestingly, D'CLOK-UCB achieves near-optimal performance, as shown by its scaled reward being very close to one, when the outcome kernel has 1 = 0.5 or 1 7= 1, but the same cannot be said about SO'CLOK- UCB. Moreover, O'CLOK-UCB is the only GP algorithm that has better performance when / = 1 compared with = 0.5. This indicates that the sparse algorithms are not able to make use of the increased arm outcome covariance, even with 100 inducing points. That being said, the difference in the average reward between O'CLOK-UCB and CO'CLO-UCH with 100 inducing points is only about 5% when and 1% when 1= 0.5. Thus, SOCCLO--UC is very good practical allernative to O'CLOK-UCB.
5.3
Simulation III: Movie Recommendation
5.3.1 Setup & dataset
We use a movie recommendation setup where in each round a selection of K = 3 movies are to be shown to a group of users, with the goal being maximizing the number of users that watch any recommended movie. We use the MovieLens 25M dataset (Harper & Konstan, 2015). Each movie comes with genre metadata that indicates the genres of the movie from a set of 20 genres (action, adventure, comedy, etc.). Moreover, each rating is between 0.5 and 5.0, increasing in increments of 0.5.
In our experiment, we only consider ratings after 2015 and users who have rated at least 200 movies. Then, in each round 4, we sample L, movies from the movie set, where L, follows Poisson with mean 75. Then, from the users who reviewed the picked movies, we sample Rs of them, where RG follows Poisson with mean 200. If a user j rated a movic i, then there is an edge connecting them with context Tıy- We define this context to be zi, = (u,, 9t)10, where uj is the average of the genres of the movies that user j rated, weighed by their rating, 9a is the genre vector of movie i, and 10 is a normalizing factor. 4 Then, we pick the expected outcome of a base arm (i.e., edge) with context Vi,j to be f(x,,) = 2/(1 + e - 1, to model realistic non-linear relationship. Its S-shape captures natural saturation effect, where the outcome is most sensitive to mid-range context values but has diminishing returns at the extremes. Then, the random outcome (i.c., the chance of the user watching the recommended movic) is Bernoulli random variable with probability ((xsg) Finally, the reward is the number of users that watched at least one movie that they were recommended. Formally, the expected reward in round t and given a super arm S (i.e., the set of outgoing movie-user edges from the K movies to be recommended) is
R2 Σ j=1
II (1 - (0-((((()))))) E E))((as))
L
u(f(xr,)) =
where I((i,)) E S) indicates whether the edge connecting movic and user j is among the picked base arms. Notice that even if the learner knew all the edge probabilities, the problem of picking left nodes (i.e., movies) to maximize the expected number of activated right nodes (i.e., users) is NP-hard and thus computationally intractable (Chen et al., 2016a). Therefore, we use an approximate oracle for both the learning algorithms and the benchmark. We chose TIM+ (Tang et al., 2014), which is an (a, -a-appooxee oracle with
We divide by and 20 normalize the context because the maximum number of genres that a movie has in the dataset 10.
14

---

Published in Transactions on Machine Learning Research (09/2025)
-n a=1- and 3 = 1 3n 3n' where n is the total number of nodes. Although our setup assumes n-approximate oracle, our algorithm has no issue using an (a, -)-approximate oracle in practice, as we will see in the results. Note that the oracle knows the problem structure and thus knows the number of left and right nodes as well as the edges connecting them, but it does not know the edge probabilities and instead takes them as input.
5.3.2 Algorithms
We run the experiment on the sparse version of our algorithm, SO'CLOK-UCB, and ACC-UCB of (Nika et al., 2020). We excluded our non-sparse algorithm because its computations efficiency is very poor, especially in practice, and as we will see, the sparse algorithm manages to perform extremely well anyways. Morcover, CC-MAB of (Chen al., 2018) was excluded because its (1 1/e)-greedy oracle that approximately maximizes a reward function takes as input set, but in the dynamic probabilistic maximum coverage (DPMC) problem, the input to the reward function is vector. Below the parameters and configurations of each used algorithm:
We use the sparse variation our algorithm, as described in Simulation I. We use three different numbers of inducing points: 1, 2, and 4. We also set å = 0.05 and use the squared exponential kernel. We use the TIM+ oracle with 0.1 and 1.
SO'CLO--UCB:
We set 01 = 1, p, 0.5, and N = 2, given in Definition 1 of (Nika al., 2020). The initial (root) context cell, Xo.1, is one-dinensional unit hypercube (i.e., a line) centered at (0.5). ACC-UCB also uses the TIM+ oracle with e = 0.1 and / = 1.
ACC-UCB:
5.3.3 Results
We run the experiment for 400 rounds and repeat it 5 times, averaging over each run. Figure 5 shows the rumning average reward of each algorithm divided by that of the benchmark. Even with 2 inducing points, SO'CLOK-UCB manages to outperform ACC-UCB by around 5%. With 4 inducing points, SO'CLOK-US UCB reaches an average reward that is only 0.5% less than that of the benchmark and outperforms ACC-UCB by more than 8%. These results show that SO'CLOK-UE UCB achieves optimal performance in a realistic setting where the underlying problem cannot be solved by grcedily picking the base arms with the highest estimate outcomes.
6 Conclusion
We considered the contextual combinatorial multi-armed bandit with changing action set problem with semi-bandit feedback, where in each round, the agent has to play a feasible subset of the base arms in order to maximize the cumulative reward. Under the assumption that the expected base outcomes are drawn from Gaussian process and that the expected reward is Lipschitz continuous with respect to the expected base arm outcomes, we proposed OCCLOK-UCB which incurs (K (TTkk(UrgrT))) regret in I' rounds. In experiments, we showed that sparse GPs can be used to speed up UCB computation without significantly degrading the performance. Our comparisons also indicated that GPs can transfer knowledge among contexts better than partitioning the contexts into groups of similar contexts based on a similarity metric. potential limitation of our work is the assumption of access offline approximation oracles and the Lipschitz continuity of the expected regret.
An interesting future research direction involves investigating how dependencies between base arms can be used for more efficient exploration. For instance, when the oracle selects base arms sequentially, it is possible to update the posterior variances of the not yet selected based arms by conditioning on the selected but not yet observed base arms. Another interesting direction would be to investigate how tight the dependence of the regret bounds is on K.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Acknowledgments
This work was supported in part by the Scientific and Technological Research Council of Türkiye (TÜBITAK) under Grants 215E342 and 124E065. The work of C. Tekin was supported by the BAGEP Award of the Sci- ence Academy; by the Turkish Academy Sciences Distinguished Young Scientist Award Program (TÜBA- GEBIP-2023); by TÜBÍTAK 2024 Incentive Award.
References
Guglielmo Maria Accabi, Francesco Trovo, Alessandro Nuara, Nicola Gatti, and Marcello Restelli. When Gaussian processes meet combinatorial bandits: Gcb. In 14th European Workshop on Reinforcement Learning, pp. 1-11, 2018.
Peter Auer, Nicolo
and Paul Fischer. Finite-time analysis of the multiarmed bandit problem.
Mach. Learn., 47(2-3):233 256, 2002a.
Peter Auer, Nicolo Cesa-Bianchi Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal computing, 32(1):48-77 2002b.
Zahy Bnaya, Rami Puzis, Roni Stern, and Ariel Felner. Volatile multi-armed bandits for guaranteed targeted social crawling. In Workshops at AAAI, 2013.
Sébastien Bubeck and Nicolo
Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv preprint arXiv:1204.570 2012.
Sébastien Bubeck, Rémi Munos, Gilles Stoltz, and Csaba Szepesvári. X-armed bandits. Journal of Machine Learning Research, 12(5). 2011.
Niv Buchbinder and Moran Feldman. Deterministic algorithms for submodular maximization problems. ACM Thans. Alg., 14(3):1-20 20, 2018.
Nicolo Cesa-Bianchi and Gábor Lugosi. Combinatorial bandits. Journal of Computer and System Sciences, 7844):1404-1122, 2012.
Deepayan Chakrabarti, Ravi Kumar, Filip Radlinski, and Eli Upfal. Mortal multi-armed bandits. Proc. Adv. Newral Inf. Process. Syst., 21:273 280, 2008.
Lixing Chen, Jie Xu, and Zhuo Lu. Contextual combinatorial multi-armed bandits with volatile arms and submodular reward. In Proc. Adv. Neural Inf. Process. Sysl., pp.
2018.
Shouyuan Chen, Tian Lin, Irwin King, Michael R Lyu, and Wei Chen. Combinatorial pure exploration of multi-armed bandits. Advances in newral informalion processing systems, 27, 2014.
Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: Goneral framework and applications. In Proc. 30th Inl. Conf. Mach. Learn., pp. 151 159. PMLR, 2013.
Wei Chen, Wei Hu, Fu Li, Jian Li, Yu Liu, and Pinyan Lu. Combinatorial multi-armed bandit with general reward functions. In Proc. Adv. Neural Inf. Process. Syst., pp. 1659 1667, 2016a.
Wei Chen, Yajun Wang, Yang Yuan, and Qinshi Wang. Combinatorial multi-armed bandit and its extension probabilistcaall triggered arms. The Journal of Machine Learning Research, 17(1):1746 1778, 2016b.
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In AISTATS, pp. 208 214, 2011.
Richard Combes, Mohammad Sadegh Talebi Mazrach Shahi, Alexandre Proutiere, et al. Combinatorial bandits revisited. Advances in neural information processing systems, 28, 2015.
Emile Contal, David Buffoni, Alexandre Robiequet, and Nicolas Vayatis. Parallel Gaussian process optimiza- tion with upper confidence bound and pure exploration. Lecture Notes in Computer Science, pp. 225 220-240, 2013. ISSN 1611-3349.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Sepehr Elahi, Baran Atalar, Sevda Ögüi, and Cem Tekin. Contextual combinatorial multi-output GP bandits with group constraints. Transactions on Machine Learning Research, 06 2023.
Yi Gai, Bhaskar Krishnamachari, and Rahul Jain. Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations. TEEE/ACM Trans. Netu., 20(5):1466-1478, 2012.
F. Maxwell Harper and Joseph A. Konstan. The Movielens Datasets: History and Context, ACM Trans. Interact. Intell. Syst., 5(4), December 2015. ISSN 2160-6435. doi: 10.1145/2827872.
Gaurush Hiranandani, Harvineet Singh, Prakhar Gupta, lftikhar Ahamath Burbanuddin, Zheng Wen, and Branislav Kveton. Cascading linear submodular bandits: Accounting for position bias and diversity in online learning to rank. In UAI, pp. 722-732. PMLR, 2020.
Alihan Huyuk and Cem Tekin. Analysis of Thompson sampling for combinatorial multi-armed bandit with probabilistically triggered armss In AISTATS, pp. 1322-1330. PMLR, 2019.
Robert Kleinberg, Alexandru Ceuulesuu-Mizil and Yogeshwer Sharma. Regret bounds for sleeping experts and bandits. Mach. Learn., 80(2-3):245-277 2010.
Harold W Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83 97, 1955.
Branislav Kveton, Zheng Wen, Azin Ashkan, Hoda Eydgahi, and Brian Eriksson. Matroid bandits: Fast combinatorial optimization with learning. arXiv preprinl 1403:1403.500 2014.
Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learning to rank in the cascade model. In Proc. 32nd Int. Conf. Mach. Learn., pp. 767-776, 2015a.
Branislav Kvelon, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Combinatorial cascading bandits. Advances in Neural Information Processing Systems, 28, 2015b.
Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Tight regret bounds for stochastic combinatorial semi-bandits. In AISTATS, pp. 535-543. PMLR, 2015c.
John Langford and Tong Zhang, The gpocdy algorithm for contextual multi-armed bandits. Proc. Adv. Neural Inf. Process. Syst., 20(1):96-1, 2007.
Fengjiao Li, Jia Liu, and Bo Ji. Combinatorial sleeping bandits with fairness constraints. IEEE Trons. Netw. Sci. Eng., 7(3):1799 1813, 2019.
Tian Lin, Jian Li, and Wei Chen. Stochastic online greedy learning with semi-bandit foedbacks. In Proc. Adv. Neural Inf. Process. Syst., pp. 352 360, 2015.
Tyler Lu, Dávid Pâl, and Martin Pál. Contextual multi-armed bandits. In AISTATS, pp. 485 492, 2010.
Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke, Fujii, Alexis Boukouvalas, Pablo GPflow: A Gaussian process library using
León-Vile Villaaáll,,ré, Zoubin Ghahramani, and James Hensman.
TensorFlow. J. Mach. Learn. Res., 18(40):1-6, apr 2017.
George L Nemhauser, Laurence Wolsey, and Marshall L Fisher. An analysis of approximations for maxi- mizing submodular set functions-i. Mathematical programming, 14(1):265 2644 1978.
Andi Nika, Sepehr Elahi, and Cem Tekin. Contextual combinatorial volatile multi armei bandit with adaptive discretization. In AISTATS, volume 108, pp. 1486 1496, 26 28 Aug 2020.
Alessandro Nuara, Francesco Trovò, Nicola Gatti, and Marcello Restelli. Online joint bid/daily budget optimization of internet advertising campaigns. Artificial Intelligence, 305:103663, 2022.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. Contextual combinatorial bandit and its application diversified online recommendation. In Proceedings of the 2014 SIAM Tul. Conf. on Data Mining, pp- 461 469. SIAM, 2014.
Jack Sandberg, Niklas Åkerblom, and Morteza Haghir Chehreghani. Bayesian analysis of combinatorial Gaussian process bandits. In Interational Conference Learning Representations, 2025.
Shubhanshu Shekhar and Tara Javidi. Gaussian process bandits with adaptive discretization. Electronic Journal of Statistics, 12(2):3829-355 -8744, 2018.
Aleksandrs Slivkins. Contextual bandits with similarity information. In Proc. 24th Annu. Conf. Learn. Theory, pp. 679 702, 2011.
N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger. Information-theoretia regret bounds for Gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theory, 58(5):3220-33)) 222222200-25,,- 2012.
Youze Tang, Xiaokui Xiao, and Yanchen Shi. Influence maximization: Near-optimal time complexity meets practical efficiency. In Proc. ACM SIGMOD Int. Conf. Manage. Data, SIGMOD '14, pp. 75 86, New York, NY, USA, 2014. ISBN 9781450333765.
Michalis K Titsias. Variational learning inducing variables in sparse Gaussian processes. In AISTATS, pp. 567 574, 2009.
Sattar Vakili, Kia Khezeli, and Victor Picheny. On information gain and regret bounds in Gaussian process bandils. arXiv preprinl arXiv:2009006963 2020.
Laurence A Wolsoy and George L Nemhauser. Integer and combinatorial optimization. John Wiley & Sons, 2014.
Yang, D. Zhang, W. Zheng, and Z. Yu. Modeling user activity preference by leveraging user spatial temporal characteristics in Ibsns. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(1): 129-142, 2015.
Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation problems. arxiv preprint arxiv: 1603. 05359, 2016.
18

---

Published in Transactions on Machine Learning Research (09/202))
Appendix
A Missing Proofs
The proof of Theorem relies on the following auxiliary lemmas, which we state and prove hereon. First, let us denote by [[-1] the vector of observations made until the beginning of round t, that is,
T 1] = [r" (æ1,s;), ( (01-1,1..))
For any 1. 2 1, note that the posterior distribution of (xx given the observation vector "[t-1] is [1 (z), (11])())) (x)), for any x E Nr. Thus, applying the Gaussian tail bound, we obtain
p(1(()) Pa- ((-) > /1 --- (x)))(()(r-m1)) r--n) s 2 exp
- 2
(6)
for 8r 2 We will use this argument in order to prove our first result which gives high probability upper bounds on the deviation from the function value of the index of any available context.
Our first result guarantees that the indices upper bound the expected base arm outcomes with high proba- bility.
Lemma 2. For any 8 E (0, 1), the probability of the following event is at least 1 - 6:
F Ivt 2 1, VI E X: 1f(x) - (])- s / (z)}
where Be = 22oog M
Proof. We have:
1 1-F(F) =
[
3t 2 1, E &e 1f(z) Mp mp- [z(2 > 81122 o mp=r)(x))]
(7)
E
Σ
(f((z) - 1 (x)) 8: T. -22)
21 NE
1 (15(x)) -1-1 (z) 3. (-yllire ||m-=]]
Σ
Σ
E
(8)
12
mp-11(00] 81 ( |mmuu
ΣΣ
E
P
[f(z))
121
(=9)
2 exp
(9)
2
22 ZE
68
2M
2M
6
8
Σ
2-
21
where equation 7 follows from the fact that P(F) = E[[(F); equation 8 follows from the tower rule and the fact that the sets X₁, t 2 are fixed (thus there no randomness from there); equation 9 follows from equation 6 and the rest follows from substituting 8, and the fact that Σ 1-2 2-2 2
19

---

Published in Transactions on Machine Learning Research (09/2025)
Next, upper bound the gap of selected super arm in terms of the gaps of individual arms. From here on, unless otherwise stated, we denote by 2t,k the context St,s,, associated with the kth selected arm $1,k at time t for brevity.
Lemma 3. Given round t 2 1, let us denote by Si = (5 1 Then, the following holds under the event F:
the optimal super arm in round t. [
| a u(f((r(;;) u(f(zr,s))) - 2B
Σ o (Ixx)]
Proof. Let us first define G, = argmaxses, u(ir( Given that event F. holds, we have:
a u(((((r),s)) u(s( S a ((i((ar, - u(f(
(10)
S a uluuu(( - u(f(arrss))
(11)
S ( )) - (2)t,s)))
(12)
18.1
S B
(e((),,))) (x+,a)
(13)
[S.]
< B
1i f (Txx
|s
+ B
/ 01 Pp--1 11 (TaN)
(14)
1=1
|S
S 223
o (Tex))
(15)
k=
where equation 10 follows from monotonicity of and the fact that f(r,,,,)) ) S de( )) for k s | by Lemma 2: equation 11 follows from the definition of G:: equation 12 holds since S, is the super arm chosen by the a-approximation Oracle; equation 13 follows from the Lipschitz continuity of u: equation 14 follows from the definition of index and the triangle inequality; for equation 15 we use Lemma 2.
We next provide the full proof of Lemma 1. We first restate the result below for convenience.
Statement. Let 2, == It,s, be the vector of selected conterts at time t 2 1. Given T 2 1, we have:
T S
I (((em)); 2 2x + 1)
1
ΣΣ
2 of
(T.x)
where 2[T] = [zi,..., 2T| is the veclor of all selected contexts until round T and (*K is the maximmum eigencalue of matrices (1)))
Proof. By definition, we have
(r(#m)) = H (r(zm)) - H (f(mm)))((m))
H ((())))(((((1)))) - и (((m)))((Em))
H (r(zr) ((=(--))) H (r(#(-11)) - H (((m)))((mm)
Reiterating inductively we obtain:
I ((((m)):
Σ H ()(z))(((((1))) + H (r(z1)) - H (+(z))))((1))
20

---

Published in Transactions on Machine Learning Research (09/2025)
since r(2(1) r(z1). Let ni Mejssl Note that H ((((zn))(((1))) - (f(z1) + 7,,.. ((ZT) + nT (
- -H (I(TI,1) + ,,....... (T11))) + 71,19111- (Tr.1)
+TI.1..... "......,((TTT f(TT,15)) + 1T.1551 f(((T)))
T |5,1
(m,,a)
log 2Teo [ss
where the last equality follows from the entropy formula for Gaussian random variables. On the other hand, since any finite dimensional distributions associated with a Gaussian process are Gaussian random vectors and also, since 2s is deterministic given 2|t- the conditional distribution of r(z)) given r(zp-1)) is N (H1r-11(Z)), (1)) + o² I where (1t-1[(z))= = [mp- (Zr,1)... Me- 11 ss))l" and
(
E(t-1n(Z1)=
(T+ T
o i (T,, )
Here, ko(z, v) = k(z,y) and ( y) == (x y), where N = Σ S Before we proceed, we need to emphasize the fact that O'CLOK- UCB does not employ any randomisation subroutines. There exist many deterministic a-approximation oracles (Lin et al., 2015; Qin et al., 2014; Buchbinder Feldman, 2018), including greedy approximation oracles. We assume that the a-approximation oracle that our algorithm uses is deterministic, and hence, there is no randomness coming from our algorithm. Therefore, we obtain the following.
((((zm): H (r(z))(((p-))) + H (r(z1)) H (+(m))))(=1))
log 2ete (2)1-11(Zz) o² [(s
2
T
log 2 (Σo(z1) + + [1:1)]]]]
log 2 1ss
(16)
2
2
T
T
log 2Tee (Ege-1) (zt) + o² ((.))
Σ
log 2Tee 2 |
2
2
1=1
1=1
T
1
log 2TTeO² (0-2 [1--1 + [(st)]
log 2 2 I|s
1=1
log 2TTeO 2
log (0-22)0--11 (zt) [[s,))
1 T
log 2Tee 2
2 t=1
T
log -1-11111((2) - [[s:i
(17)
2
1=1
T
[Sa]]
log
II
2
da 1)
(18)
2
A=1
21

---

Published in Transactions on Machine Learning Research (09/2025)
S
1
log
1)
2
S
2 AR
(19)
2
A
+
1=
k=1
T
18
2 Ak
(20)
-A- 1)
1=
18
2 ( (T(,k)
(21)
(X) + 1)
1=1 km
where equation 16 follows from the formula of conditional entropy for Gaussian random vectors; for equa- tion 18, we make the observation that given a symmetric positive-defiitt by matrix A, we can write [A + I = II (A& + 1); ;); for equation 19, we use the fact that log x 2 xr for all x > 0; in equation 20 (K) denotes the maximal cigenvalue of all Ig)))(Zz), for t $ T; for equation 21 we use the fact that the trace of symmetric matrix equal the sum of ils eigenvalues.
Finally, we are ready to prove Theorem 1.
Statement. Let E (0, N. Given B, 2ololMM''' /38), the regret incurred by O'CLOK-UCE in T rounds upper bounded with probability at least 1-6 6 as follows:
Ra (T) S (BTKKTRT((T))
where C = (A)((A( + 02).
Proof. From Lemma 3 we have:
T Ra (T) a Σ opt(f))
Σ u(f(z)))))
T |&
< 2B
o
[T(,)]
(22)
[=1
km
using the fact that 8//2 is monotonically increasing in l. Taking the square of both sides, we have:
T S+
(T) < 4B
11 [zehh|]
T -
A- 1=1
2
18c
< 44B Σ
Σ
o
(23)
|5.1
<4B²3rT 3T
|
0
2
11
(Tt
(24)
1=1
1=1
T
<AB23TTK
11 (T
1=1
1m
T
SS
=BB 3r TKo 2
p 11 (1,kk
(25)
[=1 k=1
22

---

Published in Transactions on Machine Learning Research (09/2025)
(x()) S (K
s <8B23, (K) 1)o² (
(26)
(27)
where for equation 23 and equation 24, we have used the Cauchy Schwarz inequality twice; in equation 25, we just multiply by o² and o 2 equation 26 follows from Lemma 1; and for equation 27, we use the definition of TT-
Taking the square root both sides we obtain our desired result.
Finally, we prove Corollary
Statement. Let 5 E 1), T.K E N and lel X C RD be compact and conver. Under the conditions of Theorem and for the following kernels, the a-regret incurred by O'CLOK- UCB in rounds is upper bounded (up polylog factors) with probability least 8 as follows:
For the linear kernel we have: (T(T)) 50 (V((K)DKT)
For the RBF kernel we have: (T) S0 (xx(KKKTTog"'T
For the Matérn kernel we have: (T) S (VX1K]K1(0t/0--20)) where >> 1 is the Matérn parameter.
Proof. This is an immediate application the explicit bounds on 7T given Theorem (Srinivas et al., 2012), to the bound we obtained Theorem 1. For the Matérn kernel, we have used the tighter bounds from (Vakili et al., 2020).
23


Reviews:
[{"review_id": "thlV7bFtqR", "reviewer": "Reviewer_7ZVz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, but the writing can be improved, as well as the structure in some of its parts.\nI think that the authors overlooked some literature that proposed the same algorithm they are providing in this paper. Therefore, the main novelty lies in the theoretical derivation and its extension to varying arms.\n\nI am not totally convinced by the motivating examples. The authors show some problems, but fail to make the reader understand how they correspond to the provided setting.\n\nFinally, the experiment. I think they are somehow convincing; however, I think that they should be presented in a different way, first comparing the regular version of the algorithm they proposed with the sparse version, and then going more in-depth while discussing the results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of combinatorial semi-bandit in the case where the set of arms to be chosen varies over time. The authors provide an algorithm showing regret bounds and favourable empirical properties over synthetically generated data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- \\forall M \\leq |S| -> \\forall m \\in \\{1, \\ldots |S|\\}\n-Example on page 6 about MAB: this is an example of the setting that can be modeled as a C3MAB, but it does not provide a practical reason to include in the modeling a time-varying set of arms. I think you should provide stronger examples for motivating your setting. Please provide a more convincing motivation.\n-Examples about Online Shortest Path and Online stochastic ranking: not clear why you are presenting these examples\n- The algorithm in Algorithm 1, even if for a specific advertising problem, has already been presented in the literature. You should refer properly to the corresponding literature:\nNuara, Alessandro, et al. \"Online joint bid/daily budget optimization of internet advertising campaigns.\" Artificial Intelligence 305 (2022): 103663. or Accabi et al. \"When Gaussian Processes Meet Combinatorial Bandits: GCB\".\n- I do not understand the rationale behind the transforming function in Section 5.1.1\n- Simulation I: It is not clear why you tested your algorithm in an environment which is not reflect the assumptions you posed in the theoretical part of the paper. I would either select a problem which is more suitable to your setting, or also analyse the case in which we have an (\\alpha, \\beta)-approximate oracle.\n-Simulation I: Also, the choice of experimenting with only the sparse version of the algorithm is not motivated. I would also like to understand how the original algorithm is behaving empirically.\n- you should give some reference for the TIM+ oracle\n- \"Note that the reason ...\" I think this comment was left from a previous version of the paper. I would suggest polishing and checking for such errors in the entire paper.\n- The parameter setting in the second simulation for so'clok.-ucb is a repetition of the first one. Please rearrange so that you do not have to make so many repetitions in the paper.\n- Figures: it is hard to read the legend...the lines are too thin. Please consider improving the readability of the graphs\n- Figure 6: why not to compare the algorithms over the same environment? The choice of Figure 6 does not allow for comparison of the different algorithms.", "broader_impact_concerns": "There is not broad impact statement, but since the problem analysed is mainly theoretical I do not think it would be relevant to add one.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bmkUnlKdga", "reviewer": "Reviewer_WMHd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n* Overall, the paper is well-written and easy to follow, with multiple illustrative examples of the C3-MAB setting and clear explanations of the proposed algorithm.\n* The introduction of the maximum information gain over the sequence of observed contexts (of length $T$) is a reasonable and intuitive refinement. Considering the maximum over all possible contexts would be too conservative, especially in problems with large or infinite context spaces.\n* The regret bound improves upon previous results when using commonly applied kernels, such as the linear and RBF kernels, making the contribution practically relevant.\n\n### Weakness\n\nThe current manuscript may, at first glance, appear to be a straightforward adaptation of GP techniques. While the improved regret bounds are appreciated, the theoretical contribution seems somewhat incremental, as the core proof strategy largely follows established GP bandit analysis frameworks.\n\nThe primary theoretical novelty seems to lie in Lemma 3, which characterizes the maximum information gain over the observed context sequences. However, this result is established for a deterministic version of the proposed algorithm, which is computationally expensive and not scalable. Although the authors introduce a sparse variant to improve efficiency, this version incorporates additional randomness due to the selection of inducing points, which could impact the theoretical guarantees but is not fully analyzed in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the contextual combinatorial bandit problem, where the available action sets can vary across rounds. Solving such problems without any structural assumptions would be extremely challenging and, in many cases, impractical. To address this, the authors introduce a smoothness assumption using Gaussian Processes (GPs), which can flexibly model a wide range of function classes. The resulting regret bound incorporates a notion of maximum information gain, but instead of relying on the worst-case information gain over all possible (and potentially unreachable) action sets, it focuses on the maximum achievable information gain within the action sets that can actually appear during the learning process. This would lead to a more meaningful and potentially tighter regret characterization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Regarding the above weakness, it would be helpful to further elaborate on how introducing GPs differs from relying on adaptive discretization or structural assumptions as in previous work. While using GPs removes the need for adaptive discretization (which is known to be efficient, as the authors noted on p.3), it introduces significant computational overhead. Additionally, although SO'CLOK-UCB is more efficient than O'CLOK-UCB, it still incurs considerably higher computational cost compared to CC-MAB. Providing a more detailed explanation of the practical and theoretical benefits of using GPs, beyond the improved regret bound, would strengthen the case for the proposed approach (if possible).\n\n2. In Remark 1, the notation should be $\\tilde{O}$ rather than $O$, due to the logarithmic dependency on $T$ through $\\beta_T$.\n\n3. In Figure 4, the results for ACC-UCB are not clearly visible (possibly due to overlap with other lines). It would be helpful to use a different line style or marker to distinguish the ACC-UCB results, or to explicitly discuss their behavior in the main text to ensure clarity.\n\n---\n### Very minor points (e.g. typos)\n\n* In the definition of $\\mathbf{r}\\_{[N]}$ on p.7, I think $r(\\tilde{x})$ instead of $r\\_{[N]}(\\tilde{x})$.\n* There is a missing closing parenthesis \")\" in the definition of $K_{[N]}$ right after equation 3. \n* In Remark 2, it would be clearer to explicitly mention that Lemma 3 appears \"in the appendix.\"\n* In Lemma 3, the notation $\\Sigma$ is introduced without definitions. Although it is a conventional notation, it would be helpful to explicitly clarify that it refers to the covariance matrix for completeness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f237vZXueT", "reviewer": "Reviewer_yA9y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n- To my knowledge, Lemma 3 seems to be novel and is interesting.\n\n**Weakness**:\n- In contributions, the authors claim that $\\bar{\\gamma}\\_T$ takes into account the availability of time-varying action sets. However, if the contexts of base arms can take arbitrary elements in $\\mathcal{X}$, then this quantity $\\bar{\\gamma}\\_T$ has the same rate as $\\gamma\\_{KT}$. Indeed, the resulting explicit upper bound in Corollary 2 is given by using $\\gamma\\_{KT}$. Therefore, I think that if the authors want to claim the benefit from using the quantity $\\bar{\\gamma}\\_T$, the authors should provide more concrete analysis which shows $\\bar{\\gamma}\\_T$ has a rate strictly slower than $\\gamma\\_{KT}$ under some conditions of super arms.\n- In Section 1.2, the authors describe that ``while they take a Bayesian approach to tackling it, our approach is frequentist.'' However, in the GP-bandit literature, the frequentist setting implies that the setting where $f$ belongs to the reproducing kernel Hilbert space. On the other hand, this paper assumes that $f$ follows a Gaussian process, which is often called the Bayesian setting. I believe that the proposed method in this paper is a Bayesian approach. The difference from Sandberg et al., 2023 is whether the target is the high probability regret bound or the expected regret.\n- Lemma 3 seems to be novel. However, the maximal eigenvalue $\\lambda^* (K)$ can be $K$ in the worst case, as discussed in Remark 1. If we substitute $K$ for $\\lambda\\^*(K)$, a similar result to Lemma 3 can be obtained by Proposition 1 in Desautels et al., 2014 (Consider the case where the all observations in $y_{fb[t]+1:t-1}$ is on $x$). Indeed, the batched Bayesian optimization problem is similar to C3 MAB in this paper in the sense that users must choose action sets before observations are obtained. Although I think Lemma 3 has a certain value, a more careful discussion of the relationship between this paper's proof and the proof technique in batched Bayesian optimization literature is required.\n\n>Thomas Desautels, Andreas Krause, Joel W. Burdick; 15(119):4053−4103, 2014. Parallelizing Exploration-Exploitation Tradeoffs in Gaussian Process Bandit Optimization\n\n- Regarding Remark 1, if the covariance matrix is always diagonal, $\\gamma_T = \\Omega(T)$. Therefore, this discussion is misleading and should be removed.\n- The upper bound in Theorem 2 is obvious since the base arms in $\\mathcal{X}$. Furthermore, the lower bound seems to be false. The derivation in Eq. (31) does not hold. This is because the maximization is performed on $\\mathcal{X}$ in Eq. (31) but on $\\cup_t \\mathcal{X}_t$ in Eq. (30), and $\\mathcal{X}_t$ can be arbitrarily small compared with $\\mathcal{X}$. I think that Theorem 2 should be removed or changed to a lemma, a proposition, and so on.\n- To me, the derivation of Corollary 2 is unclear. For example, why does the upper bound in the RBF kernel contain $\\sqrt{D}$ term? Why does the upper bound in the Matern kernel contain $\\lambda^* (K)$ term (not $\\sqrt{\\lambda^* (K)}$)? Why does the upper bound in the Matern kernel not contain $K$?\n- Although authors claim that the computational complexity of O'CLOCK-UCB is $O(K^3 T^4)$. However, if we update the inverse kernel matrix using the inversion of the block submatrices, the resulting computation complexity is $O(K^3 T^3)$. Therefore, the experimental result of O'CLOCK-UCB on the computational time may be slow due to the implementation.\n\n\n**(Probably) Careless mistakes**:\n- Takeno et al., 2023ab did not perform the analyses based on adaptive discretization.\n- In the fourth contributions listed in page 2, $K$ in $\\gamma\\_{KT}$ is missing.\n- In Remark 1, is the word \"not\" in the statement \"If that was the case\" missed? Please rephrase for clarification.\n- In Corollary 1, the last phrase \"with cardinality $KT$\" seems to be a mistake of $\\bar{T}$.\n- The upper bounds in Corollary 2 in the main paper are different from the statement in the appendix.\n- In the experiments, the abbreviation DPMC is not defined.\n\n**Minor comments**:\n- The whole paper is not easy to read for me. In particular, Section 2 is unclear since the problem setup, assumptions, and example applications are discussed in an inconsistent order (The order seems to be Definition of $f$ -> definition of $u$ -> assumptions on $u$ -> Definition of bandit problem -> Example applications -> Assumption on $f$). Furthermore, Section 5 seems to contain redundant content. In addition, I cannot see the necessity of several slightly complex notations, such as tilde and bar on $x$ and $z$.\n- The description \"the expected reward is Lipschitz continuous in expected base arm outcomes\" may be confusing. Since we consider that $f$ is a sample path from a GP, the expected base arm outcomes can also be interpreted as $\\mu\\_{t-1}(x)$, not $f(x)$.\n- Regarding the description \"We assume that $\\eta$ is independent across base arms,\" the authors assume that $\\eta$ is mutually independent across all observations, not only across base arms.\n- Remark 2 is slightly unclear since Lemma 3 is not explained in detail in the main paper. Moreover, since a similar proof technique has been considered in the batched Bayesian optimization literature, \"a new approach\" may be overemphasized.\n- I do not think that Remark 3 is remarkable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper formulates combinatorial contextual multi-armed bandits with changing action sets (C3 MAB), where the underlying outcomes of base arms are sampled from a Gaussian process.\n- This paper proposes a UCB-based acquisition function named O'CLOCK-UCB, which achieves a sub-linear (regarding the number of iterations $T$) high-probability regret upper bound.\n- To obtain the regret upper bound, the authors show the lower bound of the maximum information gain (Lemma 3).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the above weaknesses and careless mistakes. This revision should include the modification (or removal) in Section 1 related to those. Furthermore, if the authors agree, please consider the revision addressing minor comments.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)