Bandits with Mean Bounds
Authors: Nihal Sharma
Soumya Basu
Karthikeyan Shanmugam
Sanjay Shakkottai
Venue: Accepted by TMLR ()

Abstract: We study a variant of the bandit problem where side information in the form of bounds on the mean of each arm is provided. We prove that these translate to tighter estimates of subgaussian factors and develop novel algorithms that exploit these estimates. In the linear setting, we present the Restricted-set OFUL (R-OFUL) algorithm that additionally uses the geometric properties of the problem to (potentially) restrict the set of arms being played and reduce exploration rates for suboptimal arms. In the stochastic case, we propose the non-optimistic Global Under-Explore (GLUE) algorithm which employs the inferred subgaussian estimates to adapt the rate of exploration for the arms. We analyze the regret of R-OFUL and GLUE, showing that our regret upper bounds are never worse than that of the standard OFUL and UCB algorithms respectively. Further, we also consider a practically motivated setting of learning from confounded logs where mean bounds appear naturally.

Published in Transactions on Machine Learning Research (11/2024)
Bandits with Mean Bounds
Nihal Sharma
wihaalsharmmaauueuuu...u eduu
The University of Texas at Austin
Soumya Basu Google
busussoumyaagyoggee.com
Karthikeyan Shanmugam
Google Meed
Sanjay Shakkottai
sanjoy.shaakkotttiiuuueuuu edu
The University of Teaas at Austin
Reviewed on OpenReview: https: //t::://ppeeee openreview. net/ forum? id=4T24D624f1
Abstract
We study a variant of the bandit problem where side information in the form of bounds the mcan of each arm is provided. We prove that these translate to tighter estimates of subgaussian factors and develop novel algorithms that exploit these estimates. In the linear setting, we present the Restricted-set OFUT. (R-OFUT.) algorithm that additionally uses the geometric properties of the problem to (potentially) restrict the set of arms being played and reduce exploration rates for suboptimal arms. In the stochastic case, we propose the non-optimistic Global Under-Explore (GLUU) algorithm which employs the inferred subgaussian estimates to adapt the rate of exploration for the arms. We analyze the regret of R-OFUL and GLUE, showing that our regret upper bounds are never worse than that of the standard OFUL and UCB algorithms respectively. Further, also consider practically motivated setting of learning from confounded logs where mean bounds appear naturally.
1 Introduction
We study the problem of bandits with mean bounds and bounded rewards where an agent is presented with a set of K arms, along with side-information in the form upper and lower bounds on the average reward for each of the arms. The agent is then asked successively choose arms based on previous observations in order to maximize the cumulative reward. As is standard in MABs, the agent's performance is compared to that of genie that always chooses the arm that gives the largest reward in expectation. We consider this in the commonly studied frameworks of stochastic Multi-Armed Bandits (MABs) and the Linear Bandit. In the former, rewards of each arm are drawn independently from its associated distribution and are uncorrelated with rewards of other arms. The linear setting couples the rewards from all arms through an unknown, but fixed latent vector that used to paramterize the mean rewards. We seek design arm selection policies that efficiently utilize the provided mcan bounds offering both improve regret performance and computational complexity.
Our setting is motivated by the problem of inferring efficacy of interventions from confounded logs. As concrete example, consider the following healtheare example: Interavenous tissue plasminogen (tPA) activators are known to be highly effective in treating acute ischemic strokes if administered within 3 hours of the onset of symptoms. Otherwise, a less-effective medical therapy is recommended, as tPA causes higher chances of adverse side effects and hemorrhages Powers et al. (2019). If log is then generated without recording the time since the onset of symptoms, it would present tPA be better than the alternative. Naively using such log to infer tPA to be the best intervention could lead to unfavourable outcomes in areas with poor access

---

Published in Transactions on Machine Learning Research (11/2024)
to healthcare, where patients take longer to reach the hospital after symptoms appear. Inferring the optimal decision in the presence partial contextual information is non-trivial in general, however, one can extract. bounds on the mean of the treatments using these logs.
The offline problem of extracting bounds on mean effects of interventions is well-studied by works such as Robins et al. (2000); Brumback et al. (2004); Richardson et al. (2014); Zhang & Bareinboim (2017); Yadlowsky et al. 2022). We are interested in using these bounds aid the online learning of optimal actions, also studied in Zhang & Bareinboim (2017): Combes et al.] (2017) ) for the Bernoulli MABs (the latter also treats gaussian rewards). In this work, we show that using the mean bounds in non-trivial ways can lead to improvements in regret performance over existing methods in the case of linear bandits and stochastic MABs
with this side information.
The key intuition for our improvements comes from the notion of subgaussian factors of bounded random variables. A random variable taking values in [0, is known to be su-gaasanssiann which is tight for a Bernoulli random variable with mean 3.- Given additional information about the location of its mean (in an interval, say), tighter estimates of the variance can be inferred. However, whether such information provides sharper subgaussian factor, which in itself is an upper bound to the variance, is not known. The worst-case factor above is commonly used in the bandit setting, to characterize the concentration behavior of (bounded) random variables around their mean. Further, the jump from estimation to decision-making means that estimates can be incorrect (e.g,, biased, poor confidence) so long as it does not alter the decision. In a bandit setting, this observation has been classically used to explore sub-optimal actions at lower rate than that for the best action; this leads to a worse accuracy bound for the reward estimates of sub-optimal actions but does not affect decision-making. In this paper, we go beyond this intuition: we show that, surprisingly, known bounds on mean rewards for some actions (side information) enable us to explore other possibly unrelated actions at lower-rates, thus improving overall cumulative regret. Our contributions are detailed below:
Contribution 1: Improved Subgaussian Factors with Mean Bounds: We provide a characterization of the subgaussian factor a S of any random variable bounded in [0, 1J when upper and lower bounds on its mean are known. Specifically, in Theorem 3 and Corollary 3.1 we show that when the mean is known to be towards either half of this interval, one can infer factors that are strictly less than 0.5. These immediately imply tighter concentration bounds for bounded random variables. This result could be of independent interest, however, we study the elfects of such information in bandit learning.
Contribution 2:
Linear Bandits: We present the Restricted-set Optimism in the Face of Uncertainty for Linear bandits (R-OFUL) algorithm for bounded rewards. R-OFUL first uses the structure imposed by the linear rewards to refine the given side information and produces the lightest possible mean bounds for each arm in the action set before online interaction. Then, at each time, it leverages the geometry of the problem to restrict the set of arms to be considered. Finally, it reduces exploration by using the sharp estimates of subgaussian factors above for arms in the restricted set and chooses actions much like the standard OFUL algorithm of Abbaaii-adkorii et al. (2011).
We show that the lower bounds are key to our improvements in the online phase of the algorithm. First, the restricted set is constructed as a cone around the arm with the largest lower bound. Combining the lower bounds of the arms that remain with the boundedness of rewards then leads to our reduced exploration rates. Our analysis in Theorem f shows that using side information, R-OFUL can improve over the regret guarantees of standard OFUL by a constant factor. It also improves the computational cost. due to the restriction of arms. To the best of our knowledge, this is the first investigation on Linear Bandits with Mean bounds.
Contribution 3: Stochastic MA We develop GLobal Under-Explore (GLUU) an index based policy which, unlike the UCB Auer et al. (2002) and kl-UCB Cappé et al. (2013) algorithms, is not optimistic; ie. the indices do not serve as high-probability upper bounds to true means of each arm. We use the fact that violating the upper bound property for the indices of suboptimal arms does not adversely affect the regret as long the property is maintained for the (unknown) best arm. In particular, our indices for an arm are formed using a quantity that can be strictly lesser than the true subgaussian factor of the arm only if the arm is sub-optimal. This causes the sub-optimal arms to be under-explored which leads to improvement in regret performance.
2

---

Published in Transactions on Machine Learning Research (11/2024)
This problem is a specific form of the Structured Bandit framework of Combes et al. 2017) where the authors develop OSSB, a non-optimistic algorithm to balance exploration across arms in structured spaces. In the case of Bernoulli rewards, OSSB reduces to the B-kI-UCB algorithm of ZZhang & Bareinboim (2017) that uses the kl-UCB index for cach arm truncated at the corresponding upper bound. In both B-kI-UCB and OSSB, the lower bounds on arm means are only used initially to prune away arms that can be identified as suboptimal, and the upper bounds are used to clip the arm indices at each time.
For general bounded rewards, the upper bound s.g-factor of the optimal arm can be inferred from the highest lower bound of arm means. Therefore, we define the exploration rates of the arms to be the minimum of the individual arm subgaussian factors and the aforementioned upper bound. Our analysis in Theorem B shows that in instances with non-informative mean bounds, we recover the performance of UCB. However, using our adapted rates in instances with rich side information leads to GLUE significant improvements over vanilla UCB. Empirically, we see that our performance is comparable to B-KT-UCB when is known that one of the arms has mean close to 1.
Contribution 4: Mean bounds from confounded logs: We develop techniques to extract upper and lower bounds on the means of arm rewards from partially confounded logged data. Specifically, we consider a dataset that has been collected by an oracle that observes the full context, takes the optimal action and receives the corresponding rewards. However, the log only contains some parts of the context along with the corresponding (action, reward) pair, generalizing the work of Zhang & Bareinboim (2017), where none of the contexts are recorded. In the stochastic case, using bounds on the gap between the means of the best and second best arms as observed by the oracle, as well as the corresponding gap between best and worst arms, we derive upper and lower bounds on the mean rewards of arms to be used by agent that acts only based only on the recorded parts of the context. We show that these are tight, i.e. there are instances that meet both the upper and lower bounds. We also show that these bounds can be inferred in linear setting without the knowledge of gaps. We validate our work through synthetic and semi-syuthetic experiments with the Moviclens 1M dataset Harper & Konstan (2015).
1.1 Related Work
Bandit problems have seen a lot of interest over the past few decades (see) Bubeck et al. (2012): ;
Lattimore
& Szepesvári (2020) for comprehensive surveys). A vein of generalization for the same has seen numerous advances in incorporating several forms of side information to induce further structure into this setting.
Notable among them are graph-structures information [Buccapatnam et al. 2014 Valko et al. 2014 Amin
et al. 2012), latent models (Li et al.] 2010 Bareinboim et al.] 2015 Lattimore el. al., 2016 [Sen et al. 2017),
expert models (Auer et al. 1995 Mannor & Shamir 2011). : smoothness of the search space (Kleinberg et al.,
2008 Srinivas et al.) 2010: Bubeck et al. 2011), among several others. We assume side information in the
Torm of mean bounds and study how such information affects the decision making process.
In the stochastic setting, our bandit problem has connections the works by Zhang & Bareinboim (0017) and Combes et al.] (2017). An in-depth comparison with these can be found in Section [.3] Along another thread, Bubeck et al. (2013) provide algorithms with bounded regret if the mean of the best arm and a lower bound on the minimum suboptimality gap is known. These techniques, however, do not apply in our setting as the side information we consider does not allow us to extract such quantities in general. In the linear setting, with time-varying action sets, the works Li et al.] (2010); Abbasi- Yadkori et al. (2011) are inspired by the upper confidence bound-type arguments of Auer et al. (2002) for the stochastic case. When action sets remain fixed over time, arm-climination type algorithms like ones Valko et al. (2014) improve dependence on the dimension of the arms. We study the novel selting of linear bandits with mean bounds and varying action sets in this work.
The extraction of mcan bounds from confounded logs has been studied in the context of estimating treatment effects in the presence of confounders. Here, actions are treatments, and the rewards capture the effects of this choice. A line of existing work performs sensitivity analysis by varying a model on the latents, measured variables, trcatments and outcomes in way that is consistent with the observed data (Robins et al. 2000 Brumback et al., 2004 Richardson et al.) (014)). Recently in (Yadlowsky et al. 2022 Zhao et al.] 2019). a universal bound the ratio of selection bias due to the unobserved confounder assumed. This means that.
3

---

Published in Transactions on Machine Learning Research (11/2024)
the treatment choice has bounded sensitivity the unknown context (i.c., mostly irrelevant). We deal with the other extreme, where we assume that. the outcomes in the log are recorded using an unknown oplimal policy (under complete information), and that the knowledge of worst case sub-optimality gaps for the given latent context space is known. Our assumption allows for strong dependence on the unknown context.
The use of logged data to improve online learning has been studied recently by (Zhang & Bareinboim 2017 Zhang et al. 2019 Ye et al., 2020). The first assumes that the log contains no information of the variables that affect reward generation, while the others assumes that all such variables are present. We consider the middle ground, by assuming that a fraction of these variables are included in the logs. the authors of Tennenholtz et al. (2020) studied a related lincar bandit problem where the agent is provided with partial observations collected oflline according to a fixed behavioral policy which can be sampled from. These are then used to aid online decision making after the agent observes the full context. In contrast, we consider the case where the agent can only observe the partial contert at each time and is provided with confounded logs collected from policy (to which we do not assume sampling access) that is optimal under the full context.
2 Bandits with Mean Bounds
We consider the round-based interaction of a learning agent with a stochastic environment through set of Ko actions (or arms). At each round, the agent chooses one of the provided arms and observes a stochastic reward. To aid its decision making, the agent is also provided with side information in the form of bounds on the mean reward of each arm. The goal of the agent is to choose arms such that the cumulative reward is competitive with respect that obtained by a genic that only chooses the 'best' arm in cach round.
In this work, we concentrate on two commonly studied formulations of this problem: Stochastic Multi-armed Bandits and Linear Bandits. We detail the notations, structure of rewards, and notions of the best arm for each of these below.
Stochastic Mulli-armed Bandil: The agent is provided with Kg arms indexed by the set [Ko] = (1, Ko), with each arm being associated with a fixed and unknown reward distribution supported over the interval [0. 1]. The mean reward of arm k [Ko] is denoted by Pa. For each arm & E [Ko]. the side information is given by a tuple (l,(t), (x) such that jk E [4 (t), uu (t)] and 4 (t), (t) E [0, 1]. These tuples thus specify upper and lower bounds the mean reward for cach of the arms and can be different for cach arm. With this knowledge, at round i, the agent chooses arm A, and observes a reward Y, sampled from the distribution associated with the chosen arm. We define k = arg (e Me be the (unique) best arm with p*+ μ and the genie chooses this arm at each round.
The agent thus aims to minimize its average cunulative regret, which at round T is given by Pr = Ex= - R[.] The expectation here is over the randomness of the rewards and the choice of arms of the agent. If we have that for all & [Ko], 4 = 0, un = 1, then our setting matches that of the standard Multi-armed Bandit with bounded rewards.
Linear Bandit: In this case, in each round f, the agent is provided (possibly different) set of action At S A sampled from a (possibly infinite) set of actions A such that ||A.1| = Ko Each arm a E A is a vector in Rd. At round t, the agent chooses an arm A& E A, and observes a reward Y, = (*) + ne where 0* E Rd is a fixed unknown vector. The noise The a conditionally (A,)) subbaususnnn random variable with respect to the filtration F, = (A1, .... At- Y,-1, At)). This arm-specífic subguassian factor is not revealed to the agent. As in the case above, for all rounds I, Y, [0, I] and the agent is provided with tuples (la, u.)) for each AA such that 8T a Ila, uo] and la, Ue E [0, 1]. We also assume that ||0*|| E [m, M] and that |||]| 1 -II-II is the Euclidean norm) for a E A.
As before, the agent aims to maximize its cumulative reward in order to remain competitive with a genie that chooses the arm with highest mean reward at each round. Equivalently, the agent aims to minimize the regret, which at round T is given by Ry = ΣT, max (a Ar, f*). In contrast to the setting above, Ry is a random variable due to the randomness in A.. With la O, ua 1 for a A, our setting becomes that of the standard Linear Bandit with bounded rewards.
4

---

Published in Transactions on Machine Learning Research (11/2024)
For both the settings above, we note three things: a) The agent is allowed to use the all historical information and accumulated rewards to inform future arm choices, b) The provided bounds do not restrict the range of observed rewards, but only their mean, c) Our discussion can be easily generalized to bounded rewards supported over any interval [a, b] by appropriate shifting and scaling. Our methods will also apply when bounds were known on the norm of the actions instead of the strict equality the linear bandit case.
In the standard versions of both the above settings, Optimism-based policies have been well-studied. In the stochastic case, the standard UCB algorithm in Auer al. (2002) and the KL-UCB algorithm in Cappé et al. (2013) achieve provably optimal performance for specific families of reward distributions. However, with non-trivial mean bounds, it is shown in Zhang & Barcinboim (2017) that one can outperform these methods by leveraging the information that is provided by these bounds. In the linear case, the OFUL algorithm of Ybdaoii et al. (2011) (or LinUCB of Li et al. (2010)) provides optimal regret guarantees up to logarithmic factors (sce Chapter 24 in Lattimore & Szepesvári (2020)). To the best of our knowledge, the linear bandit problem with mean bounds has not been considered before. These naive algorithms that do not use the knowledge of the provided side information will serve baselines to the methods that we develop.
Before we propose our methods, we will first spend time developing improved concentration bounds for empirical means of random variables given mean bounds. These concentrations will be crucial in analyzing our arm selection policies for both the settings above.
3 Improved Concentrations with Mean Bounds
z exp (f As a A random variable X is said to be subgauussian and only E - consequence, have the following Chernoff-Hoofflling Hoeffding concentration inequality for o-sggaaussian variables:
12
P(X 2 1) S exp
(- 20
Further, is well known that any random variable that bounded in an interval ba is - subguassssin. With additional information about this variable, we not improve this factor. Suppose the mean of the random variable were known. We ask the question 'Does this additional information lead to a tighter (< 4)) estimate of the subgaussian factor?'. This is important because tighter subgaussian parameter would lead to faster concentrations of the random variable.
Suppose now that the random variable X E [0, 1] has mean m that is known. Since is bounded, the variance of this random variable is bounded by that of a Bernoulli random variable with the same mean. Further, the square of the subgaussian factor for any random variable an upper bound on its variance (See Lemma 5.4 in Lattimore & Szepesvári (2020)). Therefore, we have that var(X) I m(1 mm) S sg sg(Benooll((m)) < where var(.) is the variance and sg(-) is the subgaussian factor of the argument. We will now show that there indeed exist a E (m(1- m), 2 such that any random variable X E [0. 1] with mean m E (0.1) is o-subgaussian (the cases when m = 0, 1 are trivial).
For this to hold, we require that
12002
E mm)
S m expp (s(1 - m)) + (1 m) exp (-ms) S exp
2²
2 z log (1 -mme ms (e
maxx RER
2
2
log
(1 (1-m) + (1 mm) -ma
ms
(1)
Thus, it is sufficient to prove that there exist unique maximizer to the RHS of Equation D and that the achieved maxima is ]/ Below, we give the steps involved in proving this. The full proof is deferred to
Appendix A
We will use sequence of lemmas to establish our result. For simplicity, we define fm(x) m1=(1mm) + +1 m)e mur and begin by proving the following facts:
Lemma For all m E 10. and x fm( > Further, fm(x) f1-m(-z) and
lim o log (f)( = m(1 m).
5

---

Published in Transactions on Machine Learning Research (11/2024)
Given these properties of fm (x), then prove the following Lemmas: Lemma 2. The sollowing are lrue:
When m E (0.5, 1), the function ntf log (f)( is not manimized at any x > 0.
2. For all E (0,0.5), O, we have that with 2ı defined in Lemma 15
a) For all x E (0, Aof log (fm(z) > m(1 m),
b) For all m. f log (Jm (x)) m(1 m).
Combining these two Lemmas, we have our final result: Theorem 3. Let X E [0, be random variable with mean (0, m # 0.5. Then, X is o o-subgaussian for o² = maX z2 log (me + (1 m))-m>>. Additionally, /m(1 - m) <a<)
2
Comparing Subgaussian factors and Bernuulii Variance
Value
0.05
Square of Improved Subgaussian Factor o²
Variance of Bernoulli
1/4
0.00
m
Figure 1: Improved Subgaussian factor vs. Bernoulli variance
The lower bound of mm(1 m) implies that it increases in (0, 0.5) and decreases in (0.5, 1). In Figure we plot the values of o² and m(1 m)) the variance of Bernoulli with mean m, for different values of m in 10.1) 1) to verify this trend. This leads to the following corollary:
Corollary 3.1 (Improved Subgaussian factors with mean bounds). Let X be random variable over (0, 1] such that [[X] E [1, u] for some 1, u [0. For any m e (0,1), let e(m) = max,eR 32 log(fm(s)) with Sm (x) m1=(11m)) m) + m1- mme-me Then, the following are true:
Ifl> 0.5, we have that / ((T)-subguussian
uf < 0.5, we have that is -suyauuuuasssian.
Thus, bounds on the mean of the random variable that do not contain the worst-case value of 0.5 always lead to improved estimates of its subgaussian factor. We call such mean bounds to be 'non-triviaf.
4 Restricted-set Optimism under Uncertainty for Linear Bandits with Mean Bounds
Now that we know the implications of non-trivial mean bounds on the concentration behavior of random variable, we move on to studying the effects of such information on the regret of a linear bandit agent. Recall that at each round f, the agent given an action set Ar of Ko vectors in Rd sampled from a set A and must learn to choose actions A, E A,. Obscrving Y, = (A,, ) + Th. where #* an unknown vector and π), a conditionally af subgauussnns random variable with respect to the filtration generated by all observations up to - 1 and the choice of arm A,, the agent competes with genie that always picks the in the

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1 Rrestricted--set Optinism in the Face of Uncertainty for Linear bandits with mean bounds
Inputs: Action set A, bound tuples (la, ua) for cach E Tightening bounds: Update the tuples (lay Ug) as in Equation
for i. 1,2 do
Receive set Ar, identify â and (t).
5: Restricting instantaneous action sets:
6: Prune out all arms a with Uh < Imaz((
7: Form set (,) defined in Equation 3
8: Reduced exploration:
Compute oe as in Equation
10: With & defined in Equation choose at to be arg (0,,E, (a, observe reward Y,.
11:
end for
provided set with largest mean reward. That is, the agent aims to minimize its regret at round T given by Ri = = max (a Ar, 0*), As is standard in Linear Bandits, this regret Ri is random due to the randomness in the choice of Ar and thus, we seek to minimize this regret with high probability. Specifically, we use ò to be the user-specified failure probability and develop a policy that minimizes regret with probability at least 1 1- 6.
The agent also provided with tuples (la, Uh) for each arm a E A which can be used to aid decision making. Further, it is known that ||0*1| E [m, M]] and that for all A, ||a]| = = We propose the Restricted-sed Optimism under Uncertainty for Linear Bandits (R-OFUL) algorithm (summarized in Algorithm I that uses these mean bounds and the linear structure of rewards to minimize regret. At each round, the agent performs three steps: a) Tightening the mean bounds, b) Restricting the instantaneous action set, and e) Invoking the update similar to the OFUL algorithm of Abbasi Yadkori et al. (2011) with potentially improved subgaussian -bbbasi-Yadkori
factors that leads to reduced exploration. We describe each of these steps below.
Tightening the Mcan Bounds: Since all rewards are obtained using the same parameter 8", bounds on the mean of some action give us non-trivial information about the mean reward of all other actions. This fact is used to tighten the provided upper and lower bounds arm means. Formally, define C, {0 ||0]| e [m, M), Va E A, (a, 0) [la, uo]} be the set of feasible parameter vectors. Then, the tight bounds lu ua (we abuse notation by representing both the provided and tighter bounds by the same variables) are computed for each arm E A as
1, + min (a, 0), ua < max 0) AEC,
(2)
BECs
After these tight bounds are computed, the set feasible vectors C. recomputed with these updated versions of the mean bounds.
Restricting instantancous action sets: This phase is carried out after receiving the set A time and chooses subset of arms to be considered at each round. First the agent prunes away deterministically suboptimal arms used Ima (t) = max.cA, la. This restricts the set of arms in consideration to Ap = {a ua 2 (t)}
Now we denote by à, arg mngmax,eA, la the 'most promising arm at time I and use ang(a, a') cos ((ala")/[=[lll'i) as the angle between the vectors a and a'. With a cos (Lman(1//ad), the restricted set of actions at time is given by
A, (a A,(t)) ang(a, àt) s 20r}
Reduced Exploration: In this phaso, the agent computes an upper bound on the s.g- s.g-factors of arms in the set Ar (t) and uses this reduce exploration in its arm selection strategy. Recall that for any y E (0, we use e(y) max,eR & log(S((s)) where fy (2r (1-1== + (1 y)e-ve For each E A, with (tightened)
7

---

Published in Transactions on Machine Learning Research (11/2024)
mean bounds la, Nos define
le > 0.5 c(4)
2 (la Ua)
(ua))
if tha < 0.5 otherwise
0.25
to be the function that maps the upper and lower bounds to the reduced s.g-factor as result of Corollary 3.1 Then, upper bound on the s.g-factor of any arm in (t is given by
ot = min {
(mm))))(3))) b), max vl ua) aEA-(1)
(4)
The agent forms the ridge regression estimate of the parameter 0* at time t as 0, =V (Eh
with V. = Vi=E a + Alg. Here Ia is the d-dimensional identity matrix and > is a rogularization parameter. It then defines the following set around 0%:
an (Enmian,,
e = -= co: 0 1000-000 -oll s 58-(()), VBr(0) = VAM + +7V 2 2
-
(1+
+ dloo
(5)
dd
Here 7 maxx51 nn and d is the user-specified failure probability. The choice of arm A₁ is given by
A arg max (a, 0).
66)
(E),(t))
This selection rule is similar to that of vanilla OFUL in Abbasi- Yadkori et al. (2011) The difference here, restrict a) the set C, with the feasible set Co. b) the set of arms that can be played at time t with (,) and c) the confidence width (6) using the tighter subgaussian estimate of y.
4.1 Key Ideas
1. Deriving the restricted set: The boundedness and linearity of rewards implies that if there exists arm that promises high mean (in our case, åz), its angle with the parameter 6* is upper bounded. This is captured by the quantity ay- Further, it also implies that the angle between the best arm in the set (,) and 0* can be no more than dĄ- Combining these, we get that the best arm lies in a cone of angle 20, around â 2. Tight s.g-factors: Since the goal to minimize regret, the worst-case arm that can be played (one with the lowest mcan reward) (r) one that at an angle 30, away from d". As rewards are bounded, this lower bound the mean reward of the worst arm can be translated into an upper bound the s.g-factor of arms in (t) using the function v. If the side information of arms in (,) can provide sharper estimates this upper bound, we use those instead. This is reflected in our definition of in Equation
4.2 Regret of R-OFUL
We now present our high-probability regret guarantees for Algorithm 1 Theorem 4. With probability at least - 6, the regret of R-OFUL suffers a worst-case regrel of
R₂ < 8dl (0) log 1 dA
Proof Sketch. First, we establish the following lemma:
Lemma 5. Let at arg max (a, 8*) be the best amm at time Then, of (t)... Further, for any arm At, the s.g-factor a(a) satisfies on I at-
8

---

Published in Transactions on Machine Learning Research (11/2024)
CHUL
100
orUL R.OFUL
R.OFUL
Cumulative Regret
60
Curulaative Regret.
Regret
00
60
60
80
40
20
20
1000
2000
3000
4000
5000
1000
2000
3000
4000
50.00
1000
2000
3C00
4000
5000
Rounds
Rounds
Rounds
(a) 5 armss per round
(b) 10 arms per round
(c) 15 Arms per round
Figure 2: Comparing R-OFUL (Algorithm [ with vanilla OFUL with arms in R and bounded rewards. Results are averaged over 200 and
bars for standard deviation displayed. R-OFUL restricts chooses between 2-3 arms per round on average and thus, its average regret is comparable over all three figures, while OFUL suffers regret that grows with the number of arms per round. We also observe that R-OFUL computes arm updates 6 6.7x faster on average.
and only
Using this lemma, generalize the concentration arguments of Abbasi- Yadkori al. 2011 to prove that with probability at least 1 6, the parameter vector satisfies 8* E Co. The result then follows using the standard arguments for bounding regret in linear bandits.
The full proof can be found in Appendix B
4.3
Comparisons and Discussions
The regret of the OFUT. algorithm is of the order Ô (Valto maz (d log(1/8)) with Omaz = (b-ol/2 as the universal upper bound on the s.g-factor of the arms in the set A. Tf no informative bounds are present (the worst case) our R-OFUL algorithm matches the performance of OFUL exactly. However, richer side information can lead a constant factor improvement in the finite- time regret guarantees as Y S Omax- Further, in cases where (t)l 1, the instance-deeenden regret of R-OFUL is while any lincar bandit algorithm that docs not perform this restriction will suffer non-zero regret on average.
R-OFUL vs OFUL:
We also improve on the computation complexity of OFUL. Specifically, to compute A, (Equation G), the usual practice is to solve the optimization problem for all a individually for all arms and then choose one with the maximum objective value. the mean bounds are such that [t) < [Ael, that is, the set of arms restricted by R-OFUL, then, so is the number of optimization sub-problems to solve, which speeds up computation.
We study generalization of OFUL which is known to be optimal up to logarithmic factors in the worst case. However, it is known that optimality in general linear bandit problems is achieved by complex non-optimistic policies. The study of regret lower bounds and policies that match them are left open.
Optimality:
Empirical Evaluations: We compare the regret performance of R-OFUL with the vanilla OFUL algorithm empirically in Figure 2 For this, we sample random set of 100 arms IR 10 and sample 5, 10, 15 of these in each round. We set as the vector [0, 10 10.--. 10. and normalize
To generate rewards, for each arm Ac, we first sample Gaussian random variable with mean (a, 0*) and variance 0.8, and clip this be in [0 ([)1,0) if (a,0) 0(5 0) to form are generated separately to be away from the mean by uniform random variable in [0, 0.5] and these are also clipped the same way the rewards. That is, arms with positive (non-positive) mcan reward are forced to have bounds in [0, 1] ([) The subgaussian upper bound for OFUL is provided as 0.5, the worst-case subgaussian factor for any random variable bounded in either [0, 1] or [-1,0] while R-OFUL computes tighter subgaussian factors based on the mean bounds and uses y as Equation 5 to choose arms. We also use M We average our results over 200 independent runs. We see that R-OFUT consistently achieves much lower regret than the vanilla variant. Further, also observe that.
bounded rewards. The upper and lower bounds on the rewards
the norm bounds on 8t
9

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm GLobal Under-Explos (GLUE) for MABs with Mean Bounds
1: Inputs: Upper and lower bounds for each arm & E [Ko]] ugs 4x-
2: Pruning Phase:
3: Define bmaz = m. 4 and eliminate all arms i e [Ko] with us < Imax.
Reindex the remaining arms to be in {1,2, 2,.,,.... K}.
Learning Phase:
Set UCB index scaling parameters th for each arm as in Equation 7
7: for f = 1,2, 3,... do
8: Play arm A, = arg ma] U& (t 1) and observe reward Y,
9: Increment TA (t) and update AA (1).
10: Update (t) as in Equation 8
11: end for
empirically, R-OFUL restricts arms and only chooses between 2-3 arms per round on average. This leads to a 6.7x computation speed up compared to vanilla OFUL.
5 Global -nper-exnnoration for Stochastic Multi-armed Bandits with Mean Bounds
Now we move on to the stochastic Multi-armed Bandit (MAB) setting. We recall that in this case, there is no linear structure on the rewards. At each time, the agent picks one of Ko arms, and observes reward that is randomly sampled from the distribution that is associated with this arm. The agent also access the tuples (4x, u&) for all arms & E [Ko] to inform its choices. At round T, the agent aims to minimize the expected cumulative regret RT Σi= Blu" 1ann--Y, Y.]-
Prior work in the Zhang & Bareinboim 2017 Combes et al. (0117) have investigated this problem before, the for Bernoulli rewards, and the latter as structured bandit problem. The B-KL-UCB algorithm of the former sets arms indices as the elipped version (using the provided bounds) of the the standard KT-UCB indices from Cappé et al. (2013). The latter proposes OSSB for general reward distributions where the agent semi-infinite optimization problem. The
decides arms at each round by solving authors also prove that OSSB achieves asymptotically optimal regret for Bernoulli and Gaussian rewards when reward distributions are known apriori. In the special case of Bernoulli rewards, OSSSB and B-KL-UCB are equivalent (see Appendix D), and are thus optimal.
instriuttionddppendee
However, it is often not practical to assume that the reward distributions are known apriori. Further, even with this knowledge, it is unclear how computationally tractable the per-round optimization problem of OSSB is. While the Bernoulli version in B-KL-UCB is still applicable in this case, the KL-UCB indices for each arm in each round are themselves optimization problems with no known closed form solution. As the number of arms grow, computing these indices efficiently poses challenge.
The UCB algorithm of Auer et al. 2002) provides an index- based solution that is inexpensive to compute, albeit at the cost of the optimal regret performance. Specifically, it is well known that in the vanilla. MAB setting, KL-UCB always outperforms UCB for bounded reward distributions, with larger gains when the mean rewards are closer to the extremities. This is mainly due to the fact that UCB chooses the worst-case exploration rate for any bounded distribution, while KT-UCB adapts its rate according to the (unknown) value of the mean rewards. Works such as Liu et al. (2018) address the computation issuc by using set of semiidistance functions to boost the UCB index. This leads to a trade-off between optimal performance and efficient compute.
We propose Global Under-Exploration (GLUE) for Stochastic MABs with Mcan Bounds in Algorithm 2 It draws inspiration from KL-UCB, adapting its exploration rale to the location of the arm means using the provided mean bounds but like UCB, provides arm indices that are inexpensive to compute. We note that unlike B-KI-UCB, GLUE is not the clipped version of vanilla UCB. In particular, GLUE is not optimistic, i.e its arm indices are not high-probability upper bounds to the true mean of the arm. The algorithm works in two phases:
10

---

Published in Transactions on Machine Learning Research (11/2024)
Pruning Phase: Before the start of the online learning process, GLUE examines the provided mean bounds of each of the Ko arms are prunes out any deterministically suboptimal arms. Specifically, we compute
Imox- = maX 4 and discard any arm k with ug < <max- Such a strategy is also followed by B-KL-UCB of Zhang & Barcinboim 20177)
Learning Phase: Tet [K] = (1, K} be the (re-indexed) set of pruned arms, with K s Ko. For each of the arm & [K], we compute
c(l+)
if 0.5
1k kkmar
Ikmcc > 0.5
ox
duk)
if < 0.5, otherwise
(7)
ok
otherwise.
0.25
Here, kmoz = arg [[[]] lu and cyy) = MAX,RR (y with fy(x) = yell-a)" + (1 - y)e v=. Let (i)
1 En= {1 k} be the number of plays and the empirical mean reward obtained from arm lo up to time n respectively. The index of arm & at time is then set to be
= En= 1(A, = k} and (nn(n)) Ti(m]
(t) min {
20 (L(( + 1)) Ti(())
with f(t) = tog (t).
(8)
We observe two key things here. First, for each armn k, (L) is set using the quantity the rather than the true subgaussian factor of When baa 0.5, th for all orms is set as the subgaussian factor of the arm with the largest lower bound kmaz which is strictly lower than of for + kmas- Thus in general, txx S o Second, arm indices are clipped at the respective upper bounds, which is sensible these indices are representation of our belief of true arm means. We also note that using th = 0.25 and setting U&(t) without clipping at us recovers the standard Upper Confidence Bound (UCB) algorithm in Auer et al. 2002)
5.1 Key Ideas
1. Sharper subgaussian factors: As we saw in Section 3. bounds on the mean of a random variable (here, the arm rewards), provide lighter estimates of its subgausssian factor (Corollary 3.1 This is reflected in our definition 0g in Equation| 7
2. Undercxploring suboptimal arms using vha Consider standard MAB instance with K arms where it is known that arm & provides rewards with a subgaussian factor of of. The standard UCB amgorithm would then explore arm & at the rate dictated by the corresponding subgaussian factor. Now suppose additionally, that it was known that the (unknown) best arm produced rewards with subgaussian factor o We construct the algorithm A which sets the index of arm k analogous to UCB, but with exploration rate dictated by min min, oest)) )- As a result, A potentially assumes incorrecl s.g-factors for suboptimal arms. Consequently, the indices of A are no longer upper confidence bounds to the arm mcans, i.c., A is not optimistic. The key difference in the dynamics of UCB and A is that the latter potentially explores suboptinal arms less frequently than the former. This would lead to A incurring lower regret than standard UCB.
In GLUE, min(o], de is analogous to th.. Specifically, when Imar 0.5, the side information allows to estimate upper bound on the subgaussian factor of the best arm. We note that this upper bound is computed using the arm kmazi the identity of the best arm is still unknown. Thus, the regret minimization problem remains non-trivial in this case.
5.2 Regret Upper Bounds for GLUE
In this section, we study the regret performance of GLUE We regard improvements from pruning as trivial and analyze regret for the set of arms that remain after pruning. We assume without loss of generality that after pruning the arms are indexed in non-decreasing order of mean, i.., μr" = fi > H2 2 ... 2 pK, where K < Ko. We also define the sub-optimality gap of an arm k as A = 1r¹ - p4. The following thcorem bounds the regret of GLUE, and also presents its asymptotic regret scaling which is an upper bound to R. This quantity captures the long-term (logarithmic) contribution of arms towards regret.
-
lim sup log(n))
11

---

Published in Transactions on Machine Learning Research (11/2024)
enproeciax EXLLUC3
imprueditx BALACC
D-C-UEB
bi
Cane
(a) bmax
0.5; Arm 3 is meta-pruned
(b) Non-Informative Bounds; Arm 2 is meta-pruned
Amm Spaciicttors
Cloped Un torn leverds
Bemoull Ranards
Jm SaccPiatoes
Cppoed auturm Rowerds
terros Rowerss
031
5-CC
Kich
B-L,UC3
D-0LUC3
BCC cCC
B.C. Cct
(v) Imar 0.5; ImprrvveUUCB and GLUE are equivalent
(d) Non-inlormalive bounds; Clipping only
Figure 3: Empirical validation for Stochastic MABs with Mean Bounds under Clipped Uniform and Bernoulli rewards: Each row corresponds to a different specification of arm means and mean bounds shown in the left subplot. Regret performance under clipped uniform and Bernoulli rewards are shown the latter two. The regret averaged over 200 runs and error bars of one standard deviation are shown. In Figure [a for each arm, thx 01 In Figure 3b the bounds reveal no non- non-trivial information, however, Arm 2 is meta-pruned. In Figure 3c we set th = 0% since bmar < 0.5 and thus, ImprovedUCCB and GLUE coincide. In Figure [3d] the bounds do not provide non-trivial information about subgaussian factors and are only used to clip rewards. B-UCB, UCBImproved and GLUE compute arm choices 11x faster than B-LLBUBB average.
Theorem 6. Let Ki(6) {k [K] uk 6 13 and K2(6) { E [K] nr S un + > 1}, for any 5 O. Then, the expected cumulative regret of Algorithm g satisfies R., S infs>o R., (6) with
Σ (a((1+((n)) + 5 + 2 log f(n)
(5) S Σ
+
(max (ō, 1rt 2x
A
(8)
RE k2))())
Here, the function q(n) O (log(/(m)33). Further, the asymplotic regret of GLUE salisfies
Ri log(n)
2
lim sup
(9)
A
(0)
Proof Skelch for Theorem 0 As is usual in the regret analysis of stochastic MABs, we upper bound the number of times suboptimal arm is played, which translates to bound on the cumulative regret. In contrast however, we can not rely on the UCB property of the indices any longer.
Recall that we only need lo consider the set of K arms that remain after pruning. Theorem 3 and Corollary 3.1 gives us that 0g is an upper bound on the true s.g- factor for arm k. We first show that the best arm kn is always -ubgaasausiin Then, we establish that the asymptotic contribution to regret of any arm in K (8) is a constant. We deem these arms to be "meta-pruned", For K2(0), we show that the use of aseudo-variance is sufficient to explore these arms at a logarithmic rate. The theorem then follows by combining these two results, with the full proof in Appendix
5.3 Comparison and Discussions
Regret Upper Bounds: GLUE vs Existing Algorithms: We compare the regret upper bound of GLUE to slew of basclines in Table [ Here, B-UCB the clipped version of the vanilla UCB algorithm of Auer et al. (2002). It equivalent to using GLUE with vix 0.25, the worst-case subgaussian factor for each arm k. Further, d(p,4) plog (7/n) (1 log ((-))/(1-)) is the Bernoulli Kulbback-Liebler divergence.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Asymptotic Regret Comparisons: Columns list the algorithm, the (bounded) distribution of rewards, whether it displays meta-pruning and the asymptotic regret upper bound.
Algorithm
Reward dist.
Mela- pruning
Asymptolic Regret
(bounded)
UCB
Any
No
velk
B-UCB
Any
Yes
FuRgion
KL-UCB
Any
No
AwINJA//
B-KL-UCB Any
Yes
>
OSSB
Bernoull
Yes
GLUE
Auy
Yes
>
The vanilla UCB and KL-UCB algorithms do not display meta-pruning as they do not use the mean bounds. The regret bound of OSSB Combes et al. (2017) matches that of B-KT.-UCB for Bernoulli rewards (see Appendix D and is optimal.
The UCB, B-UCB and GLUE algorithms require ((K)) compute per iteration in the worst-case (when no arms are pruned). KI-UCB, B-KL-UCB (and OSSB for Bernoulli rewards) require (O Ko) computational complexity, with Copp the cost of solving the index optimization problem for each arm.
Our focus in this work has been the case when the provided mean bounds hold with probability 1. The immediate follow-up would be the case where these bounds only hold for cach arm k with some probability Ps- < 1. Practically, such settings occur when we are given historical data about plays from each arm (this is the setting studied in Shivaswamy & Joachims (2012) The following corollary gives horizon-dependent regret bound for GLUE in this case.
Uncertain mean bounds:
Corollary 6.1. Let pe Te, ux] hold with probability (1 pk) for each arm k [K]. Then by time n, GLUE suffers on additional regret of n Σ ph over Theorem 6 KE[K]
If we know the horizon to be T and each arm is observed O T) times in the provided history, using standard Chernoffttype arguments, we easily see that ΣkE(K] Pk = (1/TT) Thus, the additional regret incurred by time T' is simply a constant. However, the cases when the number of samples for an arm are sub-logarithmicc in the horizon and that with an unknown horizon are left open.
Lack of global under-exploration in R-OFUL: In the stochastic MAB with bounds, we used GLUE to explore all arms at rate 2 the subgaussian factor of the best arm (when Imaz 0.5). This was possible because the estimate of the best arm remained a v/ž-subauusian random variable, and was thus explored at the correct rate. In the linear case, the estimates for each arm uses all samples collected so far in O,, the ridge regression estimate of f* The estimate of the best arm is thus no longer di?-subgaussian and under-exploring arms in the set. (r) using (0(maar b), for example, would lead to poor estimates even for the best arm and thus larger regret.
Empirical Comparisons: We compare GLUE with B-UCB, ImprovedUCB (GLUE with dg instead of the in (t) and B-KI-UCB for elipped uniform distributions (drawn from uniform distribution in an interval around the mcan that is fully contained in [0, 1)) and Bernoulli rewards. We chose not to compare with OSSB since it is unclear how the associated optimization problem can be solved for this distributions. We drop UCB and KL-UCB since the clipped variants in B-UCB and B-KL-UCB, respectively, are never worse then the vanilla counterprats. All plots are averaged over 200 independent runs.
When Imaz is close to 1, we see that GLUE outperforms the optimistic UCB-based baselines (B-UCB and ImprovedUCB) and is comparable to B - KL - UCB. We note that this is the case where KL variants enjoy maximum improvements over UCB variants, but GLUE competes with the optimal in this case. In the case when all upper bounds are lesser than 0.5, GLUE matches the performance of ImprovedUCB, thus improving over B-UCB However, since there is no information about the best in this case, B-KL-UCB outperforms
13

---

Published in Transactions on Machine Learning Research (11/2024)
GLUE. We also note that each of B-UCB, ImprovedUCB and GLUE compute each iteration 11x faster than B-KT.-UCB average.
6 A Use Case: Learning from Confounded Logs
Up to this point, we have assumed that the learning agent has access to a tuple of mean bounds for each arm before the start of the online learning process. In this section, we describe a practically motivated scenario which such mean bounds arise naturally. Specifically, we consider the task of extracting non-trivial bounds means from partially confounded data from optimal oracle. We show that under mild assumptions, this problem leads to the stochastic MAB problem (and Linear Bandit) with mean bounds.
6.1 Confounded Logs for Stochastic Multi-armed Bandits
The Oracle Environment: We consider a contextual environment, where nature samples context vector (z, u) E = Z
xu from an unknown but fixed distribution F. We assuine that sets Z and U are both discrete.
At each time any of the Ko actions from the set A = {1, 2.., Kn} can be taken. The reward of each arm & E A for a context (z, u) E has mean fk,z,a and support [0, (with appropriate shifting and scaling, this can be generalized for any finite interval (a, b)). For each context (2, u) C, let there be unique best arm ke and let 12 be the mean of this arm.
Confounded Logs: The oracle observes the complete context (2e, ue) E C at each time and also knows the optimal arm k k,, for this context. She picks this arm and observes an independently sampled reward Ve with mean value ",, = "= She logs the information in a data set while omitting the partial context up In particular, she creates the data set D {(E,, ke y1) :t
The Agent Environment: A new agent is provided with the oracle's log. In this paper, we consider the infinite data setting. The agent makes sequential decisions about the choice of arms having observed the context 2, E 2 at each time = (1.,,...) while the part of the context u E U is hidden from this agent. Let ar be the arm that is chosen, 2t be the context, and Y, be the reward at time t. Define the average reward of as Mk,s Eveu Mk.s, ((ll),)Pull)).
arm & E under the observed context
The optimal reward of the agent under context Z is defined 1 maXkEA Mk,z- The agent aims to minimize its cunulative regret for each context separately. The cunulative reward for each E Z at lime T is defined as: (T) = E E, I(2 2) - -Y)]
6.1.1 Transferring Knowledge through Bounds
The agent is interested in the quantities Ma,,, the mean reward of arm k under the partial context z, for all arms h E A and contexts & 2, to minimize its cumulative regret. As the oracle only plays an optimal arm after seeing the hidden context u, the log provided by the oracle is biased and thus Jk,, can not be recovered from the log in general. However, is possible to extract non-trivial upper and lower bounds on the average Mk.a- In a binary reward and action selting, similar observations have been made in Zhang & Bareinboim (017). Alternative approaches to this problem, that include assuming bounds on the inverse probability weighting among others, discussed in Scction
Our assumption below is different, and in a setting where there are more Ihan two arms. We specily that the logs have been collected using a policy that plays an optimal arm for each (z, u) E C, but do not explicitly we impose a scparation condition on the
impose conditions on the distributions. Instead, in Assumption I
means of the arms conditioned on the full context. We define:
Definition 1. Let us define é., 6g for each E Z as follows:
5,
min well
max A
Pk.2.u
8,
max well
Me,
min k
Thus, for each obscrved context 2, these quantities specify the sub-optimality gaps between the best and second best arms, as well the best and the worst arms, respectively and which hold uniformly over the hidden contexts u E W.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Assumption 1 (Separation Assumption). The vectors (6.,0g 2} are provided part of the log. Addilionnlly, for each a E 2, we have that 0g > 0.
Remarks on Model and Assumption
Relation to Gap in Agent Space: We note that these gaps in the latent space do not allow us to infer the gap in the agent space. However, due to optimal play by oracle, these gaps help in obtaining mcan bounds for all arms, even those which have not been recorded in the log (Theorem
The gap 6 gives us information about the hardest arm to differentiate in the latent space when the full context was observed (ōg analogously is for the easiest arm to discredit). We also note that our approach can still be applied when the trivial bound 8, 1, or universal bounds not depending on context. 8 82, 6, are used, leading to bounds that are not as light.
Interpretation:
Motivating Healthcare Example: Often, hospital medical records contain information about the patient, the treatment given k,, and the corrosponding reward Ye that is obtained (in terms of patient's 's health outcome). However, a good doctor looks at some other information uy (that is not recorded) during consultation and prescribes the best action hy under the full context (2t, u4). If one is now tasked with developing a machine learning algorithm (an agent) to automate prescriptions given the medical record 2, this agent algorithm needs to find the best treatment k(2) on average over P(u[z). Furthermore, the gaps on treatments effects can potentially be inferred from other data sets like placebo-controlled trials.
Existing alternate assumptions: Studies in Yadlowsky et al. 2022 Zhao et al. (2019) impose conditions bounded sensitivity of the effects with respect to the hidden/ unrecorded context (eftectively, that this context does not significantly alter the effect). In our work, we explore the other alternative where the treatment has been chosen optimally with respect to the hidden/unnen /unrecorded context, and allows strong dependence on this hidden context.
Quantities computed from log data: The following quantities can now be computed by the agent from the observed log for each arm & E [oo] and cach visible context E Z:
1. (k(k) The probability of picking arm k under each context is denoted as P=(k). Mathematically, P2(k) Σ P((())).
WEMER
2. M₂ The average reward observed under cach context z is defined as Ma- can be computed by averaging observed rewards for all the entries with context 2, This can be written as ps Σ p(ul))
weld
3. (k)::: The contribution from arm k to the average reward Mz. Thus we have that (z(k)
Σ P(lz))
4. (y(k2 2): Finally, we identify the set of arms with "large rewards" K>(k, = (K' E [Ko] k + (k) > (k') for each & E [K] and E Z.
To see that these quantities can indeed be inferred, assume that the logs are given as an infinite table with columns Z, A, Y. Now, we collapse rows that share Z = 2, = into a single row with Y now being the mean reward of all such rows in the original table.
With this new finite table, to compute (k), we fix all rows with Z = and compute the fraction of these that have A = k. (k) is the average reward in all rows that have Z 2, A k and μa, E M4 (k). Finally, the set K3 (ki, z) can be computed from ua (z(k), (k) and the upper bound on the latent suboptimality gap 8..
Bounds in terms of computed quantitics: Using the quantitics defined above, the following thcorem describes how one can compute lower and upper bounds on the arm rewards in the agent space.
Thcorem 7. The following statements are true for each k [Ko] and z E Z:
1. Upper Bound: pk,z Uk, = fz= -=-5 -p)(k))
15

---

Published in Transactions on Machine Learning Research (11/2024)
2. Lower Bound: Me, 2 14,s := Mz(k)+
+
Σ (k=(K)) - (k)] (k,2)
Note that these bounds can be provided for all arms &t E [Ko] and contexts & E Z. Specifically, bounds can also be extracted for the arms that are never played in the log. This comes as a result of the gaps defined in Definition 1 Proving this results requires a careful use of the total probability theorem which can be found in Appendix
Remarks:
1. Finile Log: While we study the case of infnite logs, our approach can be readily extended to finite logs by replacing the quantities /z./22k) (k), P=(A) with empirical estimates and using standard Chernoff bounds to augment the bounds in Theorem 7 to hold with a probability which is a function of the size of the log. Then, the performance of the learning algorithm is as Corollary 6.1 and the discussion that follows.
2. Distribulion shifls: Our results can also be used in the case where under a fixed visible context, the conditional distribution of the hidden contexts changes from oracle to the learner. In particular, for some 2, suppose that vz 2 maXueu (u]=)) (r(u])),, where P. and Pt are used to differentiate the oracle and learner environments respectively. Using 2.2, we can readily modify our results in Theorem 7 to produce upper and lower bounds for arm rewards under the context 2 when the distributions are no longer the same.
Now, we show the existence of instances for which our bounds are tight. We say an instance is admissible if and only satisfies all the statistics generated by the log data. The following proposition shows all the bounds defined in Theorem 7 are partially tight.
Proposition 8 (Tightness of Transfer). For any log with uk,z, kR, as defined in Theorem 7 the following stotements hold:
1. There ecists an admissible instance where upper bounds PR,a = un.. for all k E [Ko] and z E 2. 2. For each k E [Ku], there exists a (separate) admissible instance such that μk,, = Ik,= for each 2 E 2.
The learning procedure is then carried out by using the bounds of Theorem 7 to instantiate GLUE (Algorithm with regret in Theorem 6] for each partial context 2 E Z with 4n Uk,s and 4 = 12.2-
6.2 Confounded Logs for Linear Bandits
The Oracle Environment: Consider a linear bandit environment with K arms as = (as, agaa) for i [K] and (random) latent vector 0r (02, Tw) such that 0 is fixed. Suppose 0 R2, and for all k [K], au,k: Tw E R Further, we assume that T, is such that cach entry of the vector is always between [m,M], and
||aل|| = for all & E [K] (the equality can be generalized to bounds on the norm). At each round t, (t) is drawn independently from some distribution C and the full vector 0* (1) =
is revealed to the oracle, based on which the arm A, = arg max (ax, 0 (t)) is played. The oracle then observes the reward Y = (Ac, 8* (t)) + n where Te is the conditionally (Ac) subgaussian bounded random variable with respect to the filtration generated by the reward observations up to time L 1 and arm choices up to time I.
Confounded Logs: We define k(T) to be the index of the best arm when the random realization of T, was T, i.e., k(T)= = arg max 0*) The oracle records the tuple (k(T., ()),, Y.) at the end of each round and omits the explicit information about (t). These tuples are collected over an infinite horizon to form the dataset = (()(T.(t)) 8%, Yi) e N).
The Agent Environment:
The agent is provided with the infinite log generated by the oracle and needs to interact with the same environment with no other information about 0* being revealed at each time. Specifically, at each round f, the agent chooses an arm A, E {a+ : k E [K]} and observes Y, = (A,, 0*) + with (t) being drawn independently according to C and n)t the conditionally a (ubb)-subaaussinn noise as before. Since the latent vector 0* is hidden and is random at each round, the agent seeks to minimize the
average regret given by
T
RT
Σ max Elsan kE[K]
Ae, 0*}]
16

---

Published in Transactions on Machine Learning Research (11/2024)
max (ap, - A,,,, + E((as, Ae. Tull ke[k]
max (AK:2 - At.z, 02) + (aR.u Ac,u: 02)
kE K)
In the above, 8% us the average value of the vector Tn under the distribution and A1, Ar, are the parts of the arm At corresponding to parameters 0% 0g respectively. Note that the first. term in the sum is known fully to the agent as 0 is recorded in the oracle logs and thus, the learning problem of the agent now reduces to a Linear Bandit over d dimensions (the dimension of (*)) using the 'pscudo-rewards Y YY (A,, f2) for the chosen arm A.. The genie policy in this case is to always play the arm k = arg | (Ak.z 0.) + (ax,u: (W) We note that this definition averages out the randomness in Tu and in general can not be inferred by knowing ((T)) from the logs.
Mapping these back to our running healthcare example, the seen context 0 are the known effects of each intervention in (ax k E [K]} on the patient. that have been established using medical trials, while Tu represents the responses patient to the intervention based on biomarkers that have not been explored in the trials. Thus, 0% is the average of these unknown effects over the population of patients.
Further, the following quantities can be inferred from the logs:
1. pk: The probability that arm & was best in the oracle environment defined as Pk Pcc (R(T,(t))
2. V&: The average reward obtained when arm & was optimal. This can be written as v4 Y()((Tu(t) k)
Using these quantities, the following result suggests the mean bounds that the agent can infer from the logs:
Theorem 9. For all arms ax : [K], let Pk. Vk be as defined above and la the vector of all I's in d dimensions. Then, we have that
(ax. 0-)) VAPE - (1 (kx, 02) [md((1 a, on.,), Md d. ax.,
Using these bounds, the agent can then employ R-OFUL (Algorithm I in order to minimize its regret. Contrary to the stochastic setting, due to the linearity of rewards, we need not make any assumptions on the suboptimality gaps in the oracle environment to provide these mean bounds.
6.3 Empirical Validation
Confounded Logs for Stochastic MABs:
We present recommendation systems example where agent is tasked with learning the best movic to recommend to users in an online manner: the agent observes the user's occupation at each round and solves independent bandit instance for each occupation. To assist in its learning, logs collected from an oracle are provided to this agent. The oracle has access to more information about users and observes occupation, age and gender in order to recommend movies, but only records the occupation for each user. We assume that environments do not change between the oracle and the agent, i.e., users are drawn randomly from the same population in both cases.
For this set of experiments, we the Movielens 1M dataset of Harper & Konstan 2015 This data set consists of 6040 users, from whom over million ratings of 3952 movies are collected. Each user is associated with gender, age, occupation and zip-code. In this work, we ignore the zip-code. The ratings (or rewards) lic in the interval [1, 5). These are normalized to [o, 1] through normal shifting and scaling. The reward matrix of size 6040 x 3952 is then completed using the SoftImpute algorithm from Mazunder et al. (2010). Post matrix completion, we delete all users who's total reward across all movies is below 1500. We also further cluster the age attribute to have classes: 17, 25, 49, 50+ We also combine the occupation attributes appropriately to have classes namely: Student, Academic, Scientific, Office, Arts, Law, Retired and Others.
17

---

Published in Transactions on Machine Learning Research (11/2024)
After the above, the Occupation attribute is treated as the visible one, age and gender are hidden. We form meta-users by averaging all the users according to the tuple of attributes (gender,age). The reward matrix is also modified to now have each row represent a particular (gender, age) realization (for example, ('M', 0-17)). These reward matrices are separately computed for cach of the 8 occupations. We then sample a random collection of 15 movies for each occupation to be considered as arms. Theorem 7 is then employed order to form the confounded logs for each occupation class. The realized upper and lower bounds after this process are shown in Figure in the appendix. In Figure we provide the online learning behavior for two of these instances.
209
lipper
Rigged
Upoer
D-IIn Improvellic
Imeroveduce
Arms
Tiee
Time
(a) Visible Context: LAW
(b) Visible Context: ACADEMIC
Figure 4: Online Learning Behavior of two instances from the experiments using the Movielens 1M dataset. The visible context displayed the captions.
0%A
OfILL
APPO
2000
3000
$0.00
1000
3000 1000
4050
(a) Setting 1
(b) Setting
Figure 5: Online Learning Behavior of two instances from the synthetic Linear Bandit setup. In each of the figures, the left figure summarizes the inferred bounds on (0t, (x), for each & E [12]. The right figure displays the online experiment. Results averaged over 200 independent runs and one standard deviation In the first setting, the bounds do not provide any improved subgaussian estimates, however, the restriction helps improve regret. In the second setting, the bounds provide improved exploration rates.
bars displayed.
Confounded Logs for Linear Bandits: For this set of experiments, we use synthetic setup: We first fix 8%, 0% = 21,2,344 3,4,4,47 5) E RR_ We normalize 0% to have norm 1, and 0", to have norm 0.9. For the arms, for each of the 12 arms, we draw ak,z to from a Folded Normal Distribution ([X] folded normal if X is normally distributed)
R25, We set 01,a to be in a ball radius 0.1 around 0% at random and for all other k, ag set be a normalized 2-sparse vector in RS, With these, we generate logs by uniformly sampling (t) in ball of radius 0.1 around 8% independently for each f, then picking the best sample a noisy latent reward uniformly at random from symmetric interval around (T, ((), (ECT.())))) ensuring that is [0, 1]. The reward at each time is given by the sum of this noisy latent reward and (0), (fT.,tt.)))) We collect logs of size 10 to approximate the infinite logs.
for this Tt(t). Then
We invoke Theorem 9 in order to infer upper and lower bounds on each of the 12 arms. Then, spawn variants of the R-OFUL (Algorithm D and vanilla OFUL from Abbasi-Yadkori et al. (2011) and run an online linear bandit algorithm on our selting for 5000 iterations. The results are averaged over 200 independent runs. In Figure 5 we present out results. We see that when the bounds do no help in restricting the arms under consideration, R-OFUL matches vanilla. OFUL (improvements here are from clipping the optimisite indices for each arm). When non trivial information can be gathered from the mean bounds, R-OFUL significatntly perffrromms the vanilla variant.
18

---

Published in Transactions on Machine Learning Research (11/2024)
7 Conclusion
In this work, we treated the problem of bandit learning with mean bounds in the linear and stochastic settings. Beginning with the study of how mean bounds lead to inference of sharp subgaussian factors when rewards are bounded, we present the R-OFUT, algorithm for the linear setting and the GLUE algorithm for stochastic bandits that use these sharper factors to reduce exploration.
the linear case, by restricting the set of arms being considered at each round, we show that R-OFUL enjoys not only improved regret, but also increasod compute efficiency in comparison to vanilla OFUL. In the stochastic setting, GLUE can offer comparable performance to the optimal B-KL-UCB algorithm when rich side information is available, and is never worse than the vanilla UCB policy. Further, we studied the practical use case of learning from (infinite) confounded logs where such mean bounds on rewards of actions can be inferred under mild assumptions on the latent environment.
Several avenues future work were discussed over the course exposition, chief among which is the question of lower bounds for the linear setting. While we present a compute-eficciet baseline for the case of linear bandits with mean bounds, drawing from intuition in standard linear bandits, the lower bound is likely achieved by a complex, non-optimistic algorithm that further leverages the side information. Studying the optimality would also help characterize the environments in which the mean bounds improve regret. performance. Further, the case of learning from finitely many confounded observations, which manifests as a problem of lcarning from probabilistic mean bounds remains open. In Corollary 6.1, we provide a natural a regret upper bound to the performance of GLUE in this case (which can also be extended to R-OFUL for linear bandits) which serves as a baseline. Finally, our assumption of bounded rewards led to tighter subgaussian factors and regret improvements. Identifying other use cases and modes of side information with arbitrary, potentially unbounded, reward distributions from which such improved factors can be computed also serves an an interesting direction.
Acknowledgements
We sincerely thank all the reviewers and editors for the feedback that helped improve the quality of the paper. This research was partially supported by NSF Grants 1826320, 2019844, 2107037 and 2112471, ARO grant W911NF-17-1-082 US DOD grant H98230-18-D-0007, ONR Grant N00014-19-1-2566 and the US DoT supported D-STOP Tier University Transportation Center.
References
Yasin Abbasi-Yadkon Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in newral information processing systems, 24, 2011.
Kareem Amin, Michael Kearns, and Umar Syed. Graphical models for bandit problems. arxiv preprint arX 2012.
Peter Aucr, Nicolo Cesa-Biande Desa-Bianchi, Yoav Freund, and Robert Schapire. Gambling a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of IEEE 36th Annual Foundations of Computer Science, pp. 221. IEFF, 1995.
Peter Auer, Nicolo Cesa- Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-255 256, 2002.
Flias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In Advances in Neural Information Processing Systems, pp. 1342 1350, 2015.
Babette A Brumback, Miguel A Hernán, Sebastien JPA Haneuse, and James M Robins. Sensitivity analyses for unmeasured confounding assuming marginal structural model for repeated measures. Statistics in medicine, 23(5):749-767, 2004.
Sébastien Bubeck, Rémi Munos, Gilles Stoltz, and Csaba Szepesvári. X-armed bandits. Journal of Machine Learning Research, 12(May):1655-1660 1695, 2011.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Sébastien Bubeck, Nicolo Cesa-Bianchi et al. Regret analysis of stochastic and nonstochastic multi-armed bandit, problems. Foundalions and Trends@ in Machine Learning, 5(1):1-122) 2012.
Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet. Bounded regret in stochastic multi-armed bandits. In Conference Learning Theory, pp. 122-134, 134, 2013.
Swapna Buccapatnam, Atilla Eryilmaz, and Ness B Shroff. Stochastic bandits with side observations on networks. In The 2014 ACM international conference on Measurement and modeling of computer systems, pp. 289-300, 2014.
Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, Gilles Stoltz, et al. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Arnals of Slatistics, (3):1511 2013.
Richard Combes, Stefan Magureamu, and Alexandre Proutiere. Minimal exploration in structured stochastic bandits. In Advances in Newral Information Processing Systems, pp. 1763-1771, 2017.
F Maxwell Harper and Joseph A Konstan. The movielens dalasets: History and context. ACM Transactions on Interactive Intelligent Systems (THS), 5(4):1-19, 2015.
Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in metric spaces. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pp. 681-690, 2008.
Finnian Lattimore, Tor Lattimore, and Mark D Reid. Causal bandits: Learning good interventions via causal inference. In Advances in Newral Information Processing Systems, pp. 1181-1189, 2016.
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Fang Liu, Sinong Wang, Swapna Buccapatnam,, and Ness Shroff. Ucboost: boosting approach to tame complexity and optimality for stochastic bandits. arXiv preprint arXiv:1804.052: 2018.
Shie Mannor and Ohad Shamir. From bandits to experts: On the value of side-observations. In Advances in Neural Information Processing Systems, pp. 684-692, 2011.
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. incomplete matrices. Journal of machine learning research, 11(Aug):2877 2322, 2010.
Spectral regularization algorithms for learning large
William J Powers, Alejandro A Rabinstein, Teri Ackerson, Opeolu M Adeoye, Nicholas C Bambakidis, Kyra Becker, José Biller, Michael Brown, Bart M Demaerschalk, Brian Hoh, et al. Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: A guideline for healthcare professionals from the american heart Iosociation/american stroke association. Stroke, 50(12):c344-c418, 2019.
Amy Richardson, Michael G Hudgens, Peter B Gilbert, and Jason P Fine. Nonparametric bounds and sensitivity analysis of treatment effects. Statistical science: a review journal of the Institute of Mathematical Statistics, 29(4):596, 2014.
James M Robins, Andrea Rotnitzky, and Daniel O Scharfstein. Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models. In Statistical models in epidemiology, the environment, and clinical trials, pp. 1-94. Springer, 2000.
Rajat Sen, Karthikeyan Shanmugam, Murat Kocaoglu, Alex Dimakis, and Sanjay Shakkottai. Contextual bandits with latent confounders: An umf approach. In Proceedings of the 20th International Conference Artificial Intelligence and Statistics, pp. 518-527, 2017.
Pannagadatta Shivaswamy and Thorsten Joachims. Multi-armed bandit problems with history. In Artificial Intelligence and Statislics, pp. 1046 1046-1054, 2012.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 1015 1022, 2010.
Guy Tennenholtz, Uri Shalit, Shie Mannor, and Yonathan Efroni. Bandits with partially observable offline data. arXiv preprint arXiv:2006.0673: 2020.
Michal Valko, Rémi Munos, Branislav Kveton, and Tomáš Kocák. Spectral bandits for smooth graph functions. In International Conference on Machine Learning, pp. 46-54, 2014.
Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu, John Duchi, and Lu Tian. Bounds on the conditional and average treatment eflect with unobserved conlounding factors. The Annals of Stalistics, 50(5):2587-266) 2022.
Li Ye, Yishi Lin, Hong Xie, and John Lui. Combining offline causal inference and online bandit learning for data driven decisions. arXiv preprint 20X,:22001.0699, 2020.
Chicheng Zhang, Alekh Agarwal, Hal Daumé lii, John Langford, and Sahand Negahban. Warm-starting contextual bandits: Robustly combining supervised and bandit feedback. In International Conference Muchine Learning, pp. 7335 7344, 2019.
Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandit: A causal approach. Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pp. 1778 1778-1780, 2017.
Qingyuan Zhao, Dylan Small, and Bhaswar B Bhattacharya. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 81(4):735 761, 2019.
21

---

Published in Transactions on Machine Learning Research (11/2024)
A Extreme bounds imply tighter subgaussian bounds
Proof of Lemma 1 Since e and e(l-mje are both convex, so is f() (since m, 1 m 2 0). Thus, f is
minimized when
(x) m(1 m) (ell- 6 mx
4 c(1-m)z mae
(1 m)x -
- x ==0
Therefore, the minimum value f(0) 1. Now, we have
fi-m(x)= = (1-m)e(t-(1 + (1 (1 (1-)m= (1 m)e-m(-2))
m((=2) + me(l-m)(-z) = fmm)-
Finally, the limit at = can be computed by using L L'Hopital's rule.
We now prove each of the facts in Lemma separately. Together, these two facts and Lemma 1 we have that there exist a a < 0.5 for all m + 0.5.
A.1 Non-existence of Positive Maximizer for Large Means
We begin with the case when m >> 0.5 and prove some useful Lemmas that will help establish our result. Lemma 10. Let A = mar + m(1- m(l-m)a" B + log(m). Then, A> B for all E (0.5, 1) and E IR
Proof. We have that
m(1 m)x²
mr +
- (x + log(m)) > - m(1 m))² + 2(m 1)x - 2log(m) >
2
Call m(1 - m2 + 2mm 1)x - 2 mmggmm) = h(r). differentiating and setting it to O gives 2m - m) 2(1 - m) = + x 1. Thus, min, h(x)) = (2) m-1 2 log(m). Since log(1 + y) < for all y > -1,
with m > 0.5, we can write
1 2log(m) m
m -
2(m 1) (m 1)(1 2m) 0.
m
Lemma 11. With A and B as defined in Lemma 10 the function e - e is non-decreasing for m E (0.5,
and x. E im. 00 )
Proof. We have that
d(e B))
a - nonndecceassin
20
dr
dA
dB
A
2 0
dr
dx
= (m + m(1
m) m)x)e² e - の 2 0.
From Lemma
10
(m + m(1 - m)))))) e - E (m m m)x - 1)
:. e.eA -e nenoo-decreasing = e (m + m m) | 1) 0
m(1 - m)r - (1 m) 2 =
m
22

---

Published in Transactions on Machine Learning Research (11/2024)
Lemma 12. With A and B as defined in Lemma 10 E (0, )
the function e E is increasing for m (0.5, 1) and
Proof. We begin with
d(eA BB
c² - e incrcasing
0
dz
dA
dB
e
dr e 0
dr
(m + m
m (x) cA
c 0
eB ((m + m(1 m))) e - 1>>0 >0
= (m + m0 m) A A-B >1
B- A (m + m(1 m)))
Note that at 0, both sides of this equation compute to m. Thus, the inequality holds if A(c A) dim+m(l-m)a) for > O. We have that d(e B-A, d(B-A) (1 (1-m) mmr) (1-))) and d(m+m(1-m)a) de de B-A < m. Let k(x) = =(1 (1-mz)e-B2 B-A Then, we have that
dx
m(1 m). Thus, we require that (1 B-A
d d(cB-A) me B-A B-A ((1-m)(1 m1)) m) -1 101-mal mo)
dr
Note that (1 - m2)² decreases from 1 to 0 in (0, ). Therefore, (1-mx)² < x1(0 + mxr)² < m for m 0.5. Therefore, (1 m)(1 [0][1=)x=-- range.
I-m
0 in (0, h) Since eB-A 0, k(x) also decreasing this
Thus, we have that for m E (0, L):
- mo)e B-A max (1 - mx)e B-A = m.
z6 1.
Lemma 13. For all m E (0.5, 1) and x > 0, we have that log (fm (x)) < m[]-m]]²
2
Proof. With A, as defined in Lemma 10 we require that
mn1 m)n²
log (fm(
2
mar + m(1 m))a
me. + - m < exp
= =
= - < A me" = -+oog(m) e BB
From Lemmas and 12 have that e B is non-decreasing in (0, oo). Thus, e e min e m, giving the result.
"
Lemma 14 (Part of Lemma. 2 When (0.5,1), 1), the function 3 log (fm(()) is not marimized at any <>0 0.
Proof. Using 13 we have for any m E (0.5, 1),x 0.
2
2 1²
m - -m)x² m(1 m).
인 log (x)) <
However, Lemma shows that lim 2 log (fm (x)) m(1 m). Therefore, the function is not maximized when 2 > 0.
23

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Existence of Positive Maximizer for Small Means
We now move to the case when m < 0.5. As before, we will establish the result using some useful lemmas..
Lemma 15. Let A = mæ mL1 m)a² and B = x + log(m). Then, for m E (0, 0.5), we have that the function B is
2
a) decreasing for E (1, where I1 1 Vm(1-m)
b) increasing for
Proof. Part a:
We require that in the given range,
dA
dB B <0
dx
dx
(m m(1
m)x) B-A
Both sides of the above inequality compute to m at = 0. Therefore, the inequality holds for E (0,z1) if
d(m + m(1 m)z)
m(1 -m) < (1 m1 mx)e d(eB-A) That is, m < (1 - mr)eBBA. B-A
dæ
dx
Let k(x) (1 - mx) mx)e B-A B-A Then, we have k(x) = e ((1 m) - mz)² m) > 0
((1 m1 mx) m) O. Let p(z) (1 -m](1 mr)² m. Then, we have that p(z) = 0 at
= + mm(mm) Therefore, p(x) > O for x < 21.
Thus, k(x) is increasing in (0, æ1) and hence k(x) > k(0)= = m. This implies that (m + m(1 - m)x) < B-A
in 20, which leads to the result.
Part b.
We first note that at A= for all x > Thus, A> B for x
B
+ log(m)) since m 1. Further, AA m m++ mmmm 1)x 1 dB
2 m
de
Thus, we get that
dA
dB " (m + m(1 m > e (m m(1 m)æ 1) > dr
for m
dr
And thus, e B is increasing in this range.
Lemma 16 (Part 2 in lemma 2
For all m E (0,).5). x > O, we have that with as defined in Lemma
15
a) For all x E (0,z1), log (x)( m(1 m),
b) For all a > 2. & log (fm (x)) < m(1 - m).
Proof. Part a. For this, we require that in (0, x1),
m(1 m)x2
log (fm (x)) >
2
m(1 - m)xx
e mz (mE* + 1 - m) > exp
2
> B
24

---

Published in Transactions on Machine Learning Research (11/2024)
with A, B as in Lemma 15
Using part of this Lemma, the incquality holds since e e is decreasing in (0,z1) and is maximized at x = 0 with value 1 - m.
Part b.
To see this, we require that
log (Jm(x)) < m(1 - m)a²
1111 < e -
2
This holds since e 1 e is increasing when (Lemma with value (1 - m > (1 - m) since m < 1.
15 part b) and is hence minimized at
A.3 Putting it all together
The following lemma will help us prove final result.
Lemma 17. Define C = ma + x² 8 and D = x + log(m). Then, c² e is increasing in (0, 00) for any m E (0, 0.5).
Proof.
This hold if and only if dc e dD DD > 0 for all > 0. That is,
m 4 c p>
+ I exp( C).. 4
For x = 0, both sides of this inequality compute to m. Thus, the inequality holds
d (m + 7) d(D exp(D-)) 0) 1 > (1 (1-m m 7) exp(D C). de
dr
Since exp(D- C) > 0 for all æ, the inequality is trivially true for x > (1 - -m). To see that this is true for
x E (0, 411 m)), define k(x) = (1 m 2) exp(D - C). Since k(x)) = ((L m 4) = exp( - C),
have that
k((x) = + (1-m--
4)
x
= 0 + x 4(1 - m) 2
4
Therefore, in (0,4(1 m))) k(x) = 0 ( x= 4(1 - -m) - = 2(1 - 2m). Further, since - m $ > 0 for
x < 4(1 m), x = 2(1 2m) must be maximizer of k(z). We are now left to prove that k(2 2m)) < 1 for any m 2 For this we have
1- 1m2m
2(1 2mm + log(m)) 2m((1 - 2m) - (1-2m)²
A(2(1 - (1m)
1
exp
2
2
1
1 1 2m
exp
log(m) + (1 - 2m)
2 2m2m
2
2
m
(1
2m 2m)(3 2m)
exp
2
2
Let g(x)
r
exp
(1-20 (1-2m)] 23 2m)
We have that
2
2
2m) 2m)
1 r
(((x) = exp
+
(8r - 8)
2
4
(1 2m 2m)(3 2m)
4x
4r + 1
= exp
2
2
25

---

Published in Transactions on Machine Learning Research (11/2024)
(1 2x)²
( (11-2m)6)
(m 2m)
2 exp
2
> 0 dx
Thus, ((x) is increasing for all x. Specifically, we have that for any m E (0, 2), k(2(1-2m)) q(m) < =
Thus, k=2 2m)) 1/ for any E (0, 2 and x E (0,4(1 m)). Therefore, (1 m - exp(D 2 and thus, eC - e is increasing for all x 0.
We are now ready to present our final result.
Proof of Theorem Subgaussianity
Let (2) = -or log (x)). For m < 0.5, part b of Lemma 1 and Theorem 14 imply that the maximum of 9m (z) not achieved at any negative æ. Further, Theorem 16 implics that there exists æ > 0 9m(x) >
m(1 - m). Additionally, since 9m m(1 - m) for x > m we have that the function of interest maximized
some
Analogously, for 0.5, this function is maximized at some
=2 < 0.
m
Thus, for any random variable X E [0, 1] with mean m E (0, 1), + 0.5, we have E[exp (s(X - mm)] s
exp
3.0 2
Thus, X is o-subgaussian.
Bounds on
From Part 2 of Lemma 2 (Lemma 16 we have that
2
0² lin x
log(Sm(z)) = =m(1 vn).
Thus, we are left with showing that a < when m < 0.5. Exploiting part of Lemma implics the result for m < 0.5.
We require that max,eR z log (x)) < However, since we have already shown that this maximum is achieved at positive x, it sufficient show maxy>0 & log (fm(z)) 4.. That is, for all 0,
2 log (fm (x:)
1
x²
4
x² <
= log (fm(z))
8
x²
= me." + 1 m exp
mer +
8
a.2
1 exp
+
-exp( + log(m))
8
From Lemma 17 we have that the RHS is increasing in (0, oc) and thus minimized when x with minimum value This gives the result.
Linear Bandits with Mean Bounds
Now we concentrate on the setting of Section 4 anddd provide proofs for the result of Theorem 4 First, we will show that the tight bounds we propose in Equation 2 are valid and tight. Then, we prove that the best instantancous arm each time is always contained in the restricted set at cach time. Next, we show that the quantity o, is valid upper bound on the subgaussian factor for all arms in the restricted set. This culminates in our final regret result. We begin with the bounds.
26

---

Published in Transactions on Machine Learning Research (11/2024)
Claim 1. The solutions to the optimization problems in Equation 2 is tight, that is, E0' E Co such that for any A, (a, 00) < la or (a, (a,0")
Proof. First we note that the constraint set C. is non-empty since 0* E C. and thus the quantities 4as ua are well-defined for each E A. Suppose 30' C. with (a, 0*) < la for some action This implics that la mineec, (a,0) 0) which is a contradiction. A similar argument for the upper bound completes the proof. J
Note that this point, C. redefined be the {0 ||o]| E [m, M). Va E A, (a, [la ual}. We now show that the best arm is always in contention any time.
Lemma 18. Let oz = arg max (a, 0r) be the instantaneous best arm. Then, oj Ar).
Proof. First, we show that oj E Apt). Suppose that Uu; < Imaa = (a,0") S uo; that is, (a7,0" 0*) < (ā,, This is a contradiction to the definition of až and thus, oj E Ap(t).
< lmaz (t) S (āz, 0*)
To show that oz Aprune()), we first observe that
Imaz (t) S (ōc.
= | || (an(ang( 0*))
S M (angan, 0*))
110011 z M,
(mma (1)
ang(a,, 0*) S cos-1
ax.
Further, since lmaz (1) S (ai, 0*) we get that ang(at, 0*) S at. Combining these two, we get that ang(Ar, at) S 2ay and thus, of E Ar)t).
The following lemma proves that the quantity a is a valid upper bound to the s.g-factor of any arm in the restricted set.
Lemma 19. For any arm a E Ar(t), (a))= S or.
Proof. From Corollary 3.1 we have that (a)) S vb(la, Ua) for any arm A,)t). Therefore, it follows that (a) < maxx (((a.
We are only left to prove that (a) S m3b,)))) For this, we note that the arm in (,) with lowest mcan reward is at most 3a, away from 0" This happens when H is at an angle ag away from â, and there exists some arm abod at an angle ar on the opposite side of â,- For this fictitious arm abad, we have (abod, 0*) = ||aboal| ll [|]]]]]]]]]] (3a.) 2 m cos(3at). Now, there are two cases: 1. If I' c3s(3
Then, vimm (((mcos((3.)) b) (b-o)² and Va E Ar(1), S (mm cos() b)
follows by definition.
2. If m cos(3a,) 2 0.2178a + 0.7822b: Then any arm a E (r),, (a, 0-) 2 (abad, 0+) 2 m cos Using this as lower bound for the arm a, we can write o(a)= I +0m cos (m ua) = (om b). This completes the proof.
We now prove our final regret result.
Proof of Theorem Let S. En=r De where the random variable n is conditionally Oag Ca subgausssinnn on oe and thus, nte also conditionally o--subaaussian
From Lemma 19
We claim that Mx(x) = exp (1r - L
with Mo(x) = is a supermartingale with respect to the filtration F = o(a,, YI,..., ae-1, Yz-1, a.). see this, observe that
Th (2, and
| I
))
Mx(x) = exp Σ
+
2
27

---

Published in Transactions on Machine Learning Research (11/2024)
Th nnta, at) | 2
[M M1-( E exp
02
S (z)
d is conditionally o.-8.g
Now let == Ald and h NO, H-1). With some linear algebra, we can write
||
/ gd
M,
M((x)d((r)
det V exp
2
Since M,)a) is supermartingale, is M,. Therefore, using the Maximal incquality for non-negative supermartingales, get that
()) log
det V. (
1)
8 z P
3t:
|| 2 log
Ad
1
det V Ad
P
3t Than
2 72
llg
log
(10)
8
The last step uses the fact that Σn= naan
max,cc on En=t Mnn = 1
7 En= Un. Now, with V₂ =
Eiz an
T we have
10. 0-
V
ana + (V,'ve I2) 0r
V.
n=1
V.
+ VAB
(I-V.'vi) Q-
an
Vir
Maln
+ vx\10011 <
Σ ana
VAM
(11)
V.-
Combining Equations 10 and
we get that with probability at least 1 1- 6,
det V.
|10, -0011, S VAM + + 2og
+ log
Ad
det V, X
V d
(1
t
S (e)
trace
1 1
Ad
Ad
Now, with & {0: 110e 07| S (.) observe that C. = Cs 0 & Thus,
P(3L: 0* & C.) S TP 0* & &.) 6.
The result of the theorem follows by using the standard arguments of Theorem 3 in Abbasi- Yadkori et al.
2011
C GLUE for stochastic Multi-Armed Bandits with Mean Bounds
We begin with a few useful lemmas.
Lemma 20. Any arm k is a subgaussian. The best arm is always thy subgaussai..
28

---

Published in Transactions on Machine Learning Research (11/2024)
Proof. Both these are application of Corollary 3.1 The first one follows using bounds 4, ug for the suboptimal arm &. For the second, observe that p² E [man: Us+], which are used to form the
The next lemma proves that meta-pruned arms are only played constant number of times asymptotically, Lemma 21. Let arm Ki(0)= == {k: E [K] n 2 ux + + + l]. Then, for all 1,
5
B[Th(n]] S [max (Ó, 10 uk })2
Proof. Since k E (((6) A 0, we have that {A, k} S (t)(t) ug}. Thus,
n
[[Tx(n)]= = E
1 [[A b}
1 (1 1) s us)
S E
2109 log (i 1)
< E
(1) - 1² Suz 1²²
2007 lox(s(())
r
exp
201
10a
1=
(Using Union Bound, Lemma 20
r(1= ux)²
exp
f(t)
200
t=1
2
11
(1*
)2
f(()
5v/
(n* )x
50/
(m μ - }
where the last inequality follows from the assumption on up.
The next lemma is a restatement of Lemma 8.2 in Lattimore & Szepesvári (0220) for standard subgaussian variables. It will be used to bound the number of plays of a suboptimal arm that is not meta-pruned
Lemma 22 (Lemma 8.2 in Lattimore & Szepesvári 2020
Let (X,) be sequence of zero meon, independent 0, > O, 2ad-2 and
o-subyaussion random variables. Let ji +E
2aa
K
f4 + V t
2 d
2a 2 6
u +
Ar +
t
t= [c]]
Then, E(m] < [p['] < 28-2 (a Vorma + 02) for each 1.
Proof. This is restatement of the lemma for the general case of o-subgaussian random variables.
29

---

Published in Transactions on Machine Learning Research (11/2024)
Clearly, we have that E/x] I E[n']. Thus,
2a
}
B = E
Σ
1
Pr +
8
1
t= [u]
2a t
u +
IP
Pr
+
V
20)
t=
t
(8 v20f1) 2)
)
S
exp
(X; ~ a - subgaussian)
20
t=
t
(6 v20/1) ") 1e
<
exp
20²
Now, using =²= (6vl-v22')
we have
202
2020xx v2a)²
2V20²
(V20²x + v2a) dx
8²
82
Note that the definition of u
2a 83
gives t == 1
== 0. Thus, get
20
00
2V20² x²
E[n'] < +
+
(2202x + 22a
dx
2
82
S 1 +
2a
+
40²
DE
12
dx +
AV
e --²²
da
82
02
82
20
1
4Vo²a
=1+
+
x
+
8²
82
2
82
2
= 1 1+
2
a V TOO2 + +02)
8
Where we use that So xe ze-
x² do and lo e x² dr = vF/2.
Lemma 23. For the best orm, E Ei 1(U((t 1) S /r -e]l S
50 for any e > 0.
Proof. We have
n
a
E
1(Ui(t 1) S c) =E
1 [[iit 1) 1 ~² 10 -c3
2:0 log (t))
E
(t)(t) - 10" s -
Ti 1)
[=1
5
2
This follows by using the union bound and Lemma 20 for Arm 1 (See Theorem 8.1 in Lattimore Szepesvári (2000) for more details on the inequality).
Lemma 24. If k K2(6)= == (k E
[K] : p= S us + 5, + 1) for an arbitrary 8 > 0, then,
3 +
2 (vBlog(((m))) + v WRoff log(((m) - of + h(2)).
[[]( s1+
AZ
Where h(n) Ω (log([(n))22
30

---

Published in Transactions on Machine Learning Research (11/2024)
Proof. Since k #1, {A k) S - -cA k3 for any (0, (k.
Thus, we have that
T1
n
B[T((n)] SE
Σ
1t 1) 5p' -ct -c +E Σ
((Ux(t-1)>)) c, k}}
5
E
(t - p¹ -, KH
(Using Lemma 23
For the second term, following the steps in the Proof of Theorem 8.1 in Lattimore & Szepesvári 2020 and using Lemma 22 above, we get.
E
-1) n¹ - A, =1]]
2112 (tt)
E
Pu 1) 1)+1
Tx 1) 3 1 - - A:
2 log( log(s(n))
$ E
per Mr A. --
$1+
2
vit (og(((()))
/ Wgo log(s(n)) + tf
(A, - c)²
Here, Pur the empirical mean of t ii.d samples from arm The last incquality follows from Lemma 22 with a= 2 (n)), 8 (Ax e) and a oz (using Lemma 20 Substituting this into the original expression,
get
2
B[Tx(n)] <
+ +
t log(((n)) + Vo log(s(n)) + +o
6
(Ae- e)
5067
2
<
inf
+1
+1+ +
( + Wo + of
(12)
<E(),Ap)
2
( c)2
Now, let g(m) (vR log(s(n) + v 1 (g
log + oft) and 0 <<= ag(n)]/341 < As for a (2/507) 1/3
We have the following:
5
2
inf <
+
1+
(As (2)(n)
2
5
1/3 + +1)² + (ag(n)2/3+ +1)
$ 1 +
(ag(n)2/3
g(n)
A
A Q²g(n)2/3
2g(n) (ag(n)2/3+1)
1 +
(a²g(n)2/3
(ng 1/3 1)
A
A a2g(n)2/3
2g(n)
g(n)2/3
+ g(n)2/3
2
+
50
100ff a +
A
A
A
a²)
5
2g(n)
(20
+ 3g(n) 1/3
(BV2077 2/3
1 +
[1 + (20m)))
AZ
A
A
A
=h(n)
The result follows with h(n) being defined as the last two terms of the expression above.
We ready to prove the theorem.
31

---

Published in Transactions on Machine Learning Research (11/2024)
Proof of Theorem D
The theorem now follows immediately using Lemmas 21 and 24
This is because can
decompose regret as
Ra = Σ 1=1
Blu
Y = Σ ATE(TA(1)] + Σ (
((
K2(6)
where K((6) {k: E [K] n² > uk + k 1 and K2(0) {k [K] n² S up + 1}, for any å > 0.
The first part of the theorem follows by bounding the two summations using Lemmas 21 and 24 respectively. To prove the asymptotic part, we simply take 8 + 0 and n - oc in the above expression.
OSSB for MABs with mean bounds and Bernoulli Rewards
Here, we derive closed form expressions for the semi-infinite optimization problem in Combes et al. (211 in the case of Bernoulli rewards and bandits with mean bounds. In this section, follow the notation in their paper. An instance B is represented by vector of arm mcans (6), (x,x))))) We say an instance 0 is feasible if for each k E [K], we have that 0g E [4, ug]. Let & be the set of all feasible instances. With θ as above, v(a)) ~ Bernoulli(es) and the mapping k; 0 + ykk: f) as p(k,, 0z our setting of Bernoulli bandits with mean bounds can hence be viewed as a Structured Bandit.
Define for any a, a, O, D(a, 3, d(ay, Bx) where d(:--) is the Bernoulli kl-divergence function. Let p (0) maTzE[K](((;O))) Then, the optimization problem that determines the regret lower bound in case can be given as:
min v()20
Σ
m(kk (0) 0))
kE[K]
s.l.
Σ (, AYE
kE[K]
0, ((0 = (X 0.00 #\()))
We now present the solution of the above optimization for given instance 0.
For simplicity, consider the case with 2 arms: Arm and Arm k. Let the instance e θ be such that p(1,0) = (0). We now have two cases:
1² > us: this case, the set A(O) is empty since there can no other instance À E θ with p(1.)) p) = (0) and Arm as optimal (since p(k, 0) I u < (0)). Thus, the optimal solution to the optimization problem 17" = (0.0).
2. 1 < (0): In this case, note that to satisfy the constraint over all instances in A(0), it it sufficient, to set ry(k) minpett [() d.((k.)).)) [((()))),__))) This uses the fact that d(a,z) as a function of æ is increasing in [a, 1]. Thus, the optimal solution in this case is 11 (0, ((pkk))).)
Generalizing this to the case with K arms, we get that the solution to this problem for given O E θ is given
by
{
if (0)) > ue or if k: = k0)
17" (k)
((m(k.0),,p'0) otherwise
Finally, using this the optima of the problem above, we have that the value at this optima is given by u(k:
C(O) Σ (0) (10))-(kk,0) 0)
Σ d ()(kk))),r'00))
k0)) dp(kk, 0), (0)) kEKa(0)
Thus, the asymptotic regret of OSSB roduces to C(0) as G,7 + O. We note that our case the OSSB algorithm uses the solution the above optimization at the t-th round with f replaced with the truncated empirical means, i.e. (min(ue, max(le, (c())) Ak E [K]].
32

---

Published in Transactions on Machine Learning Research (11/2024)
E Confounded Logs
E.1 Stochastic Multi-armed Bandit Setting
We begin with the proof Theorem
Proof of Theorem 7 For the upper bound, we split the expectation into two parts: (z, u) where & is optimal, and (2, u) where i not optimal. We have
pk, = Σ (ule,)P(llz)
wEM
Σ
M P(ll))
P(ul2)
uelck=ki,.
uELik
(Splitting the sum into parts based on the optimality of k)
M P(ul))
Σ
(1 5.) P(ul2)
wEMik=kk
(If k # ke then its reward is at most p - 22)
Σ 12 1 Pul()) - -8 WEN
Σ
P(u|z)
NEUL-
μr 6z (k))
(Using the definitions of μs and P=(k))
The incquality follows since the logs are assumed to be collected under an optimal policy. This completes the proof of the upper bound.
To prove the lower bound, we fix an arbitratry & E [Ko] and z E Z. Recall that K (k) (K' k * k. (.() I (k)=(k))}. We note that
( K' *
k. (k') > 5_p=(k)) and let Kg (k,, 2) == [Ko] (KK)\K>( (k, these sets can be identified from the logged data because M4 (k') and (k) can be derived for any k and Now we define the sets U> (k, =), 2g (ks, 2) as u> (k: 2) [u E U kz E K (k, u)), = fu E U k e K (<(k,z) 2)}
We now expand the mean of the arm & under partial context a as follows.
Ph,2 =
Σ
((up() +
Σ
P(ul2)
uElik=ki,..
= (k) +
Σ
(lsa)P((ll)))))) +
Σ
Weily (b..)
wets (k,a)
2 [k(k)
Σ
(Mi (p)l(
welds (k,2)
= =bz(k)+ +
Σ
Hs (k') 8g
Σ (k))
MEKs (kiz)
KEKs
The inequality holds because, firstly, we have 1k,,, which we for (k, Secondly, have Pk,z,a 2 (HC... - -d)) by definition of 8z, which we use for u E Us (k, 2). Since this holds for any arbitrary k, this proves the lower bound.
Now we move on to the proof of our tightness claim
Proof of 8 Upper Bound Tightness:
For the tightness of the upper bound, notice that
Uk = Σ P(ul)) - 6_
Σ
P(u|2)
uEl
33

---

Published in Transactions on Machine Learning Research (11/2024)
Σ
(kzz)P(ulz)+ +
Σ
(n {.)P())=).
wEll:=ki
uEll:k
Which is the same as the quantity Pk,= Eueu (k.s,,PPPP)) when
Pkiza = 12, 82 Yu E u #
II the above equality holds for all & [Ko) and all E 2, we have that jk,z = wk.z- This instance is admissible since it maintains the means of the best arms are unchanged, and thus do not affect the quantities μt and Pa(k) for any k. Thus, there exists an admissible instance where the upper bounds for each arm is tight.
Lower Bound Tightness:
Fix an arm k, and a partial context 2. For the tightness of the lower bound for this particular arm & and partial context 2, we present an instance now. We assign Pk,.," 0 for all u E Ug (k, 2), and Mk,,,u = M2,, - 6- for all u E Ug (k, 2). For this particular choice, it is easy to see that the above lower bound is tight. We now need to prove that this instance is admissible.
1. By construction we know Mk,z,u 0 12, 2 0- for all partial contexts Us (k;, That Pkaa 1, is easy to check.
2. We know that (1i,a. Mk,z,u) S 5g by construction for all partial contexts u E (k.,2) and due to the fact that pi.u S 8g and pk,,, for all partial contexts Ug (k)
3. We know that (1i,0 Mk.s,u) - by construction for all partial contexts E Us (k, and due to the fact that V 2 d and Mk,,, = 0 for all partial contexts (k, Here, 1 2 5, due to non-negativity of the mean-rewards and by definition of d. (the smallest gaps).
4. Finally, & is never optimal for any partial contexts u Us (k. 2) (k, Indeed, for all the partial For all the partial contexts U (k, =) we
contexts u E Ug 2), we have Jk,z, = and 1h 2 6, > have Mk.z,u = (M - 52) < M2 Therefore, for this instance we never observe k for any partial contexts u E (K, ( z), which ensures the log statistics does not change.
This concludes that the instance is admissible, and proves that the lower bound is tight.
E.2 Confounded Logs for Linear Bandits with Fixed Arms
We now prove our result for mean bounds from confounded logs in the linear bandit case.
Proof of Theorem Let & = { R(Tw)) be the event that arm & was optimal under the full context 0 when Ta was drawn according the distribution By definition, have ps PEE)) and
Bl(as, 0*)] (ak,z, 0%) + (ax,a: Tu))
(ak.zr 0-) + pp13 [(ak [x] + BB)) [(ar TW
PAVK + (1 p&) ((ak, 03) E (a)k,, (l)
Since Tu is such that the values of each of its entries are in [m, M]. write dm Ok.a) E [Ak,a, [&] S dM ak.a). Rearranging the above equation then gives us the required result.
E.3 More Empirical Results
In Figure we present the instances extracted out of each of the visible contexts of Occupation. The bounds are calculated as suggested by Theorem 7
34

---

Published in Transactions on Machine Learning Research (11/2024)
CPTA
(a) Academic
(b) Art
(c) Law
(d) Office
(e) Others
(f) Retired
(g) Scientific
(h) Student
Figure 6: Instances with Occupation visible context: The occupations listed in the caption.
The data filtration and movic selection process is explained in Section 6.3
35


Reviews:
[{"review_id": "zyddNhgW47", "reviewer": "Reviewer_q8Xf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\n1. Detailed description of the contribution and thorough review of the related works\n2. Experimental evaluation of the presented methods and description of the scenario where the introduced framework naturally works\n3. The regret bounds corresponding to the introduced algorithms are derived and compared with alternatives\n\n**Weaknesses**\n\n1. No Discussion/Conclusion/Future work sections\n2. The application problem remains unclear from the description of the confounded logs extracted from the Movielens dataset. This data are typically used in recommender system tasks, do you solve rating prediction problem with your approach? Please provide more details on the domain problem that is converted to linear bandits and stochastic MAB problems.\n3. Missing discussion of the limitations for the suggested methods and potential improvements for the developed framework", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the MAB problem, where additional side information on the rewards distribution is available. This feature makes the mean estimate tighter and improves the method's overall convergence. Based on this approach, the authors propose a Restricted-set OFUL algorithm for the linear bandits setting and a Global Under-Explore algorithm for the stochastic setting. The corresponding regret bounds are analyzed and compared with alternatives that do not consider additional information. The presented approach is evaluated experimentally in the learning from confounded logs, where the target bounds on the mean values appear naturally.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some requests are listed inside the weakness list above, the others are presented below\n\n1. Please add the link to the source code (e.g., in GitHub repository) of the developed methods and scripts to reproduce the results of the experiments.\n2. Add the Conclusion and Future work sections, where the summary of the presented results and limitations & potential further improvements should be discussed.\n3. Please discuss the limitations of the presented approach. In particular, I see two directions:\n- what assumptions on the underlying reward distribution should be satisfied? and what happened if they are failed? Do the presented methods robust to such scenarios?\n-  is it possible to improve the presented bounds further? Is it possible to derive the lower bound for the mean interval corresponding to the discussed type of the additional information?\n4. Please add discussion of the domain-specific problem that can be solved with your approach and what is the interpretation of the additional bound for mean values.", "broader_impact_concerns": "No concerns about the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tlep2ShxWh", "reviewer": "Reviewer_bMxB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The problem is interesting and the methods developed are nicely connected to the existing literature and explored to an appropriate extent through a mixture of theoretical and empirical analysis. From what I can tell this theoretical work is conducted accurately, the empirical work shows good performance of the methods proposed and is reproducible. The authors have done a good job in many sections of pre-empting potential queries or points of confusion and justifying the choices made and/or highlighting the benefits of the methods proposed.\n\nWeaknesses: Section 1 is not especially accessible. It starts with quite a technical observation about sub-Gaussian random variables, which I believe is intended as a motivation to the work, but is confusing before defining bandit problems etc. While I appreciate this is a paper that ultimately will be interesting to a technical audience, it would still be good to offer some definitions/introductions to key concepts rather than using the terminology of bandits, regret, etc. and the notion of what confounded of logs are before discussing them. Later in the paper, these concepts are addressed successfully. The other main weakness is in the comparison of theoretical upper bounds to lower bounds for the mean-bounded setting. The paper defers these to further work, and I think for TMLR there is enough here without lower bounds, but it would have been nice to at least see a bit more of a flavour of what the key differences between established lower bounds and those specific to this setting are likely to be, or at least conjectured to be.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers a version of the stochastic multi-armed bandit and stochastic linear bandit problems where in addition to standard assumptions about the reward's bounds/distribution, the decision-maker is also made aware of some upper and lower bounds on the expected rewards of actions.\n\nThe paper contributes an adaptation of OFUL (Abbasi-Yadkori et al., 2011) to incorporate information provided by the bounds in the linear bandit setting, and shows that this can achieve a lower high-probability bound on the worst-case regret than OFUL when the bounds are useful. For the multi-armed bandit setting, an algorithm with features of both UCB1 and KL-UCB is proposed, but which ends up not necessarily behaving optimistically, which again can improve over the regret of traditional algorithms when the mean bounds provide useful capacity to rule out certain actions or explore them at a lower rate.\n\nThe work is underpinned by a result showing that knowledge of bounds for the mean of bounded variables can yield sharper sub-Gaussianity results than those achievable using the bounds on the outcome alone, and is motivated by an example in treatment prescribing where an ML/AI tool may have access to less information than a physician, due to the absence of sensitive or otherwise unobserved variables, and thus only know the mean benefit of a treatment up to some best/worse-case bounds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I'd like to see the introduction re-written to improve accessibility for readers unfamiliar (or rusty on) bandits and confounding/healthcare applications.\n\n2. p1: 'If a log is generated without... alternative': doesn't that depend on what the distribution of times since stroke are in the training/log sample?\n\n2. The figures at times present standard deviation bounds that fall below 0. I think it would be better to plot minima and maxima rather than plus/minus 1 standard deviation, as the mean-1sd falling outside the range suggests a skewed distribution of the regret.\n\n3. p6, I'd say the agent 'aims to minimize' their regret, rather than necessarily does so.\n\n4. The idea of approximating kl-UCB to offer a computational speed-up seems to have been explored elsewhere - I'd recommend referencing and making connections to work such as Lin, Wang, Buccapatnam and Shroff (2018, IJCAI) \"UCBoost: A Boosting Approach to Tame Complexity and Optimality for Stochastic Bandits\"\n\n5. p10: 'number of played' doesn't seem right grammatically\n\n6. p12: I think asymptotic regret is only really defined within Theorem 6, somewhat implicitly, may it be beneficial for the discussion here to have defined it unambiguously in the main text beforehand?\n\n7. There are some open questions and interesting points for further work identified throughout the paper. I think a concluding Section 7 would be beneficial to pull these together.", "broader_impact_concerns": "There is no statement, but I do not have concerns about direct impacts of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ypJICz6I38", "reviewer": "Reviewer_kenH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well-written and easy to follow. The formulation is new to the existing literature.\n\nWeaknesses:\nMy main concern is the technical contribution of the paper:\n\n1. The main technical novelty of the paper is Section 3. All the derivations in the following sections are plug-ins of the results in Section 3 to the existing proofs/analyses. I don't see much challenge in such plug-ins. Coming back to Section 3, I think the result here is something people are well aware of, and the derivations are straightforward and don't involve advanced analysis.\n\n2. The bound of the paper is only effective when the interval of the mean doesn't include 0.5 (as indicated in Corollary 3.1). This doesn't change the nature of the existing UCB, and in my view, its advantage over UCB is not essential. To see this, after the pruning step (which is trivial given this additional side information), the remaining arms have a lower bound larger than some constant, say, $l$, and an upper bound smaller than some constant, say $u$. For UCB algorithm, having all the mean rewards between $[l,u]$ is equivalent having all the mean rewards between $[0,1]$. We can't criticize the existing algorithms for them assuming the reward in $[0,1]$ but not $[l,u]$.\n\n3. As I mentioned in the first point, I don't get the technical challenge when combining the new results in Section 3 with the existing analysis of UCB for MAB and linUCB. This limits the technical contribution of the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "​The paper addresses stochastic multi-armed bandit (MAB) and linear bandit problems incorporating side information, where the side information refers to additional bounds on the mean return of each arm. It introduces two natural algorithms for MAB and linear bandits setting and derives the corresponding regret bounds.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "See above weaknesses.", "broader_impact_concerns": "Na.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers think that the manuscript suggests a novel approach for incorporating side information in the update rule in linear and MAB setup. Although there were concerns about technical novelty and the experimental evaluation of the proposed approach is perhaps limited, they overall agree that this is a solid contribution to TMLR.
Decision: Accept (poster)