Lifting the Information Ratio: An Information-Theoretic Analysis of Thompson Sampling for Contextual  Bandits
Authors: Gergely Neu
Julia Olkhovskaya
Matteo Papini
Ludovic Schwartz
Venue: NeurIPS 2022 Accept (2022)
Keywords: Thompson sampling
contextual bandits
information ratio

Abstract: We study the Bayesian regret of the renowned Thompson Sampling algorithm in contextual bandits with binary losses and adversarially-selected contexts. We adapt the information-theoretic perspective of Russo and Van Roy [2016] to the contextual setting by considering a lifted version of the information ratio defined in terms of the unknown model parameter instead of the optimal action or optimal policy as done in previous works on the same setting. This allows us to bound the regret in terms of the entropy of the prior distribution through a remarkably simple proof, and with no structural assumptions on the likelihood or the prior. The extension to priors with infinite entropy only requires a Lipschitz assumption on the log-likelihood. An interesting special case is that of logistic bandits with $d$-dimensional parameters, $K$ actions, and Lipschitz logits, for which we provide a $\tilde{O}(\sqrt{dKT})$ regret upper-bound that does not depend on the smallest slope of the sigmoid link function.

Lifting the Information Ratio: An Cformatiin-Theoretti Analysis of Thompson Sampling for Contextual Bandits
Gergely Neu Universitat Pompeu Fabra Barcelona, Spain gergely wwwwgeeil...ooom com
Julia Olkhovskaya Vrije Universiteit Amsterdam Amsterdam, the Netherlands julia. olkhovskaaa@@maiilll... com
Matteo Papini Universitat Pompeu Fabra Barcelona, Spain matteo. pat.eeopppiiiiiuueee edu
Ludovie Schwartz
Universitat Pompeu Fabra
Barcelona, Spain
wudborcc...scwwart//888g
Abstract
We study the Bayesian regret of the renowned Thompson Sampling algorithm in contextual bandits with binary losses and udversarialll-sseeeted contexts. We adapt the information -eemmation-thhorttic perspective of Russo and Van Roy [2016) to the contextual setting by considering lifted version of the information ratio defined in terms of the unknown model parameter instead of the optimal action or optimal policy as done previous works on the same setting. This allows us to bound the regret in terms of the entropy of the prior distribution through remarkably simple proof, and with no structural assumptions on the likelihood or the prior. The extension to priors with infinite entropy only requires a Lipschitz assumption on the log-likelihood. An interesting special case is that of logistic bandits with d-dimensional parameters, K actions, and Lipschitz logits, for which we provide a Ö(v regret upper-bound that does not depend the smallest slope of the sigmoid link function.
Introduction
Thompson sampling is one the most popular algorithms for sequential decision making under uncertainty. First proposed by Thompson 1933]. it has been rediscovered several times over the consequent decades and has been eventually popularized in the machine learning literature by Chapelle and Li [011] and [Scott] [2010). who pointed out its excellent empirical performance for solving contextual bandit problems. These empirical studies were followed by a sequence of breakthroughs the front of theoretical analysis, spearheaded by the works of Agrawal and Goyal 2012 2013a b), Kaufmann et al. (2012), and Russo and Van Roy (2014), Russo and Van Roy [2016]. of the gold-standard methods for solving multi-armed bandit problems. Indeed, in the last decade, several chompson-sampling--ee methods have been developed and analyzed for variety of problem settings.
Thanks to these successes, Thompson sampling has become
The variety of different analysis techniques applied to Thompson sampling is perhaps even larger than the variety of problem settings that it has been applied The first key tools for analyzing the Bayesian regret of Thompson sampling for multi-armed bandits have been developed by Russo and Van Roy [014], and our analysis naturally borrows several of these tools. The worst-case results developed in said work were refined Russo and Van Roy 2016), where they proved for the first
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

time cetermmationthhhoeetie bounds on the regret of TS that scale with the Shannon entropy of the optimal action A'" under the prior on the model parameters, This result has inspired range
of follow-up works, including extensions to uncountable action sets [Dong and Van Roy 2018). approximate implementations [Lu and Van Roy] 2017] Qin et al. [022], and even new algorithms based on the analysis technique itself [Russo and Van Roy| 2018 Kirschner and Krause 2018). One
limitation that this technique could not overcome so far is not being able to satisfyingly deal with context.
When considering i.i.d. contexts and finite policy classes, one can apply the theory of Russo and Van Roy] [2016] and treat policies as actions (as done by Liu et al. 2018 Bubeck and Sellke 2020). or adapt the information-ratio directly (as done by Kirschher et al. 2020 Hao al. 2022) to obtain regret bounds scaling with the entropy of the optimal policy. This approach, however, can lead to a polynomial dependence on the number of contexts when assuming no further structure about the rewards, as the entropy of the optimal policy may be as large as log [T|] = |X log [A]. Another variant of TThompsso sampling demonstrating a similar prior-dependent regret bound has been proposed by Li] 20033]. whose regret guarantees also suffer from suboptimal dependence
the number of rounds T. A much more satisfying solution has been recently given by Zhang [021] whose "feel-good Thompson sampling" method guarantees both frequentist and Bayesian regret bounds of the order v VKTlog N, where K is the number of actions, T the number of rounds, and N the support size of the prior distribution on model parameters 0- Under Lipschitzness assumption on the prior and the likelihood, Zhang proves a frequentist regret bound scaling with the log-prior- prior-probabille mass assigned to the true parameter 0", The techniques involved in proving these results drew substantial inspiration from the works of Foster and Rakhlin [2020), Foster and Krishnamurthy [2021] and in
fact results of similar flavor were also proved recently in Foster et al. 2021).
Our own approach can be seen as a reconciliation of the analysis style of Zhang [021] with the cittmaation-thhoreti methodology of Russo and Van Roy (2016). Our main conceptual contribution is proposing an adjustment of the now-classic notion of "information ratio" proposed by Russo and Van Roy (0166 that applies to contextual bandits. In its original definition, the information ratio quantifies the tradeoff between incurring low regret and gaining information about the optimal action. As we will argue, this notion of information gain is inappropriate for contextual bandits with non- i..d. contexts. We propose a variant that measures the amount of information gained about the true model parameter l instead of the optimal action (which may be context dependent). The complexity notion resulting from this extension is called the "lifted information ratio". Our analysis shows that the Bayesian regret of Thompson sampling can be bounded in terms of the lifted information ratio and the Shannon entropy of the hidden parameter 0*, which mirrors the result of Russo and Van Roy 20166) Along the way, we draw inspiration from the recently proposed analysis technique Zhang 2021 for contextual bandits, and in fact we show that our notion of lifted information ratio bridges the concept of "decoupling coefficient" proposed by Zhang with the information ratio of Russo and Van Roy
We state our main results in the context of K K-armed contextual bandits with binary losses. For countable parameter spaces, we prove that the Bayesian regret of Thompson sampling satisfies a bound of order / /KTH((*) (8*). where H denotes the Shannon entropy. This result is comparable to the bound of Foster and Krishnamurthy (2021) for the FasICB algorithm, which is of the order / KT log | and holds in frequentist sense. This is the best result we aware of for this setting. To demonstrate the flexibility of our technique, we provide an extension to logistic bandits with Lipschitz continuous logits, generalizing the well-studied setting involving logits that are linear functions of the context and the parameter 0. For this setting, we prove a regret bound of order / log.logN log.N1/c((()) III.)) where (e, II-II) is the E-covering number of θ under norm ||-I| and C is the Lipschitz constant of the logits. This implies regret bound of order / /KdT in the well-studied case of linear logits. Notably, the bound does not show any dependence on the smallest slope is of the sigmoid link function that almost all existing results for this setting suffer from (Filippi et al. 2010 Kveton et al. 2020 Abeille et | [202]] Faury et al. 2022). Indeed, this constant has plagued all regret bounds since the early work of Filippi et al. (010) and was only recently moved to lower-order terms
by the breakthrough work of Abeille et al. 2021 Bounds involving other potentially large problem-
2

---

dependent constants have also been proved in the Bayesian setting by Dong and Van Roy [2018] and Dong et al. 2019). To our knowledge, our bounds are the first entirely remove this factor
The rest of the paper is organized as follows. After introducing the necessary technical background in Section 2 we discuss matters of information gains, information ratios, and decoupling coefficients
in Section 3 We state our main results and instantiate them in variety of settings in Section 4 We
provide the key ideas of the analysis in Section 5 and conclude in Section 6
Notation. For a natural number n E N, [m] = (1. , denotes the set of the first n natural numbers. For x; y E Rdd (2.y) denotes the canonical scalar product of and y. and ||x]| the Euclidean norm of x. We denote the Shannon entropy of a discrete random variable Z with probability mass function P as (Z) = -E [log(P(Z))) We use 02 for a d-dimensional vector of zeros and Ia for the d x d identity matrix.
2 Preliminaries
We consider a parametric class of contextual bandits with parameter space θ context space X, and K actions. To each parameter 0 E è there corresponds a contextual bandit with loss distribution Po.z. for each context x E X' and action ((0,z, x a). We will dedicate special attention to the case where the losses are binary and thus Pa is a Bernoulli distribution with parameter E(0, xr, a). For the main part of our theoretical analysis, we will assume θ is either finite set or a bounded metric space.
A, with the mean of the loss distribution denoted by
We study the problem of regret minimization in the Bayesian setting. In this setting, the enviromment secretly samples a parameter 0* from a known prior distribution Q1 over e. We assume that the agent has full knowledge of the prior and the likelihood model Po. The agent interacts with the environment for 7 rounds as follows. At each round & E [7). an adaptive adversary selects a context X₂. possibly using randomization and taking into account the previous history of actions and losses, but not 0* The latter is common assumption [e.g.. Agrawal and Goyal (0013). After observing the context X₁, the agent selects an action A₁ E A (possibly using randomization) and incurs binary loss L ~ Po .X..Ar' The goal of the agent is to minimize the expected sum of losses. In the Bayesian setting, this is equivalent to minimizing the Bayesian regret, defined follows:
R E Σ (e(o*) X1, Ar) - E(0 X A7))
where A; = arg min,e e(0", Xe, a) is the optimal action for round t, and the expectation in [ is over all sources of randomness: the initial sampling of 6* from Q1, the agent's 's randomization over actions, and the randomness of the loss realizations.
Furthermore, let F, = o(X1, AI, Li...., .a....,,,, X1, Ar, L.) be the sigma-algebra representing the history of contexts, actions and losses observed by the agent up to time included. We use Q2 to denote the distribution of the unknown parameter 6* conditional on the past history Fi-1, and simply call it the posterior distribution. We denote by me (-(x)) the distribution over the agent's actions conditional on X, and F,-1, and call it the agent's 's policy. Finally, we will frequently use the shorthand notations E, [] = [ F. X,) and P. []] = P. X,).
This paper is dedicated to the study of the celebrated Thompson Sampling (TS) algorithm. defined as follows. At each round TS draws a parameter O, from the posterior distribution Q.. Then, selects the action that minimizes {(0,, Xx, ). Finally, it updates Q. via Bayes' rule, obtaining the new posterior Q.+1. The algorithm can be equivalently defined to their posterior probability of being optimal, that is: IP [A, = F.F, 1.F-1:X4, X,) = P [A] = alF,-1,, Xt). The pseudocode is shown Algorithm I
method that plays actions according
3 Regret, information ratio, and decoupling coefficient
The classic results of Russo and Van Roy 2016 establish that the regret of Thompson sampling in non-contextual multi-armes bandit problems can be upper bounded in terms of quantity called the
Despite our best efforts, we could not verify how the bounds of Zhang 2021 scale with factors in this setting, due to the heavy use of asymptotic notation in their proofs.
3

---

Algorithm Thompson Sampling (TS)
1: Input: prior Q1
2: for t- 1,,...., do
3: Sample 8 ~ Qi
4: Observe context X
5: Play action A, arg minae E(0,, a))
6: Incur loss L,
7: Update Q, with (X,: Ac: I4) to obtain Q++1 by Bayes' rule
8:
end for
information ratio. Informally, the information ratio measures the tradeoff' between achieving low regret and gaining information about the identity of the optimal action A (which is a deterministic function of O in the standard multi-armed bandit setting). The formal definition is given by
Pf (E, 15(0,000,,0000
[A4; (A,, (*)))
where [ (A,, L+)) denotes the mutual information between A' and the (L,, At), conditioned on the history Ft-1. Intuitively, having small information ratio implies that every time Thompson sampling suffers large regret, has to gain lot of information about the optimal action, which suggests that it should be possible to bound the total regret by the total amount of information that there is be gained. The result of Russo and Van Roy [2016] confirms this intuition by showing that the regret of Thompson sampling of the order V (A*), where p² max, pf and H is the Shannon entropy. The information ratio itself can always be upper bounded by K/2, but better bounds can be shown when the loss function has favorable structural properties (e.g., whenever the reward function is a d-dimensional linear function, the information ratio is at most d/2).
pair
While this result and the underlying information-theoreti framework is very elegant, it is inappro- priate for studying contextual bandit problems. The specific challenge is that the optimal action A; changes from round to round and gaining large amount of information about A for any given round may not necessarily be useful for predicting future actions. To see this, consider a stylized example with action set {1, 2.... K}, where there exists an action Greveal = 1 whose loss entirely reveals the identity of the optimal action a" (z) for context x: E(0", x Oseveal) = 10-=" (x), Taking this action provides large information gain about Ai, but results in large regret and reveals nothing about the future losses. Thus, in the contextual setting, one can keep following a policy that provides low information ratio while suffering linear regret. This issue necessitates an alternative definition that still permits an effective information- -theoreticc analysis.
Our proposition is to consider relaxed definition of the information ratio based the mutual information between the true parameter 8* and the observed loss. In particular, we define
(E, [e(0", X1, A.) E(0*, X,, xr, A;)))²
P:
Ie L+)
where [((): Lt)= = E, [DEL tween 0" and Le, conditioned on the history Fi-1 and the context-action pair X,, A₁. This quantity measures the information that the agent gains about 0+. High values of Iz(0;; L.) intuitively allow making better predictions about the future loss realizations for possible context sequences. Since A; is deterministic function of 0* given X,, the data processing inequality implies that the informa- tion gain about 6* is always greater than that about Aj, which in turn implics that Pr is smaller than what one would obtain by directly generalizing the definition of Russo and Van Roy (2016). As this notion of information gain measures the efficiency of inferring the identity of a hidden parameter, we refer to 1,00; L.) the lifted information gain, and Pt defined in Equation ( the lifted information ratio. As our analysis will establish, a bounded lifted information ratio guarantees low regret, and we will show that the ratio itself can be bounded reasonably under conditions similar to the ones required by the analysis of Russo and Van Roy (2016). despite the fact that all quantities are defined conditionally on sequence of non-i.i.d. contexts..
X. A. P|. A.]] is the mutual information be-
We are not the first to consider the lifted information ratio: very similar quantities have been considered in the literature
citemattion-iirecter sampling, initiated by Russo and Van Roy

---

2018). who have already introduced an the non-contextual counterpart of the same quantity in their
Section 9.2, The recent works of Kirschnee al. 2020 Hao al. [002] also consider versions
of the standard information ratio for contextual bandits, but their analysis crucially relies on the assumption that the contexts are i.i.d. Similarly, [uu et al. [021]. Liu al. [022] define versions of the information ratio using various targets for information-ddrected sampling, which recovers our definition of the lifted information ratio when setting the target as 0*. While these works do not distinguish between all these versions of the information ratio on the level of terminology, we believe that emphasizing the "lifted" nature of our information ratio provides more clarity in that indicates the specific features of contextual bandit learning with parametric losses in a transparent way. Thus, we will keep using this terminology throughout the paper.
Our lifted information ratio is also closely related to a quantity appearing in the analysis of Zhang [202T], called the "decoupling coefficient". Adapted to our Bayesian setting, this coefficient can be defined as the smallest constant å such that the following inequality holds:
8 p._
E, (1("",x,,40 E(0", X A7)1 I inf E, 420
p ( At) - E, [ A X,)) +
2 EEE (e(0+, Xx, At) - E. [1(0" Xt. Ar)) X,, A.))²
where the first line gives the original definition mirroring that of Zhang [2021 and the second line plugs in the choice of μ achieving the infimum. Reordering gives the value of the optimal 57:
(E, |E(0", Xx, Ac) e(0", X,, A7)])²
07
4E, (0(0-, A.) E, [ 2,,,4,, Ar) Xx, A.)
which matches definition of the lifted information ratio, up to the difference of replacing the mutual information by the root mean- mean-squared error in predicting the true parameter 0r. Notably, this definition essentially coincides with the lifted information ratio for the special case of Gaussian losses.
4
Main results
In this section, we state our main results concerning the Bayesian regret of Thompson sampling for contextual bandits. We will assume that the losses binary and the action space is finite, unless otherwise stated. However, several of our results can be generalized beyond this setting. We will illustrate this in Section 4.3 where we provide some additional results for the classic setting of Gaussian linear contextual bandits.
We begin by stating two general regret bounds in terms of the lifted information ratio defined in Equation 2). The reader that is not interested in the full generality of our theory may skip to Section f.2 for concrete regret bounds. Our first abstract bound applies to priors with finite entropy, the simplest example being finite parameter spaces.
Theorem 1. Assume Qı supported on the countable set O1 S O and that the lifted information ratio for all rounds t satisfies Pt. S for some p > 0. Then, the Bayesian regret of TS after T rounds can be bounded as
Rr S / (0H)
In particular if 01 is a finite set with |01| = N, the regret of TS satisfies
Rr S v log
The proof of this theorem is stated in Section 5.1 Unfortunately, the Shannon entropy can be
unbounded for distributions with infinite support, which is in fact the typical situation that one encouters in practice. To address this concern, we develop broader family of distributions. In the following, (θ, g) is a metric space with metric o: O + R. We make the following regularity assumption on the likelihood function Pa.zaai
general result, that holds for
Assumption 1. There exists constant C
O such that for any 0, 0 E O |log Pe.r.a (L) - log Per (L)) < Co(0, 0') holds for all x E X, a E A, and L E (0. 1).
Under this assumption, we can state variant of Theorem I that applies to metric parameter spaces:
5

---

Theorem 2. Assume (6, 2) is a metric space, and Q1 is supported on 01 S θ with E-covering number 1, 9). Let Assumption Thold, and assume the lifted information ratio for all rounds satisfies Pr. S p.for some p > O. Then, the Bayesian regret of TS after T rounds can be bounded as
Rr S ppl' (log (Ne 0)) + 2sCT).
The proof is based on a covering argument on top of the proof of the previous theorem, and is provided in Appendix A.22 To get better understanding of Assumption II it is useful notice that is satisfied in basic settings, like logistic bandits with Lipschitz logits. See Section 4.2 for details.
4.1 Bounding the lifted information ratio
At this point, some readers may worry that the lifted information ratio may be impossible to bound due to the lifting to the space of parameters + To address this concern, we now turn to showing bounds on the lifted information ratio. We first consider the unstructured case, that holds for arbitrary parameter spaces, likelihoods, and priors.
Lemma 1. Suppose that the losses are binary and |A| K. Then, the lifted information ratio of Thompson sampling satisfies Pt I 2K for all t 2 1.
The proof of this lemma (provided in Section 5.2) relies decoupling argument between the choice of the action and of the parameters at round I, inspired by Zhang (2021). Taking his argument one step further, we center our analysis around an application of convex conjugacy, which we believe may be applicable in a broader variety of settings. We wish to highlight that this proof technique is very different from the -nformation-theorrti theoretic methodology pioneered by Russo and Van Roy (2016].
Next, we consider the case of linear expected losses in Euclidean parameter spaces, which, in principle, allows for unbounded number of actions.
Lemma 2. Suppose that θ C IRd, the losses are binary, and the expected losses are linear functions of the form E(0, 200,0.00 a) (0,, sp(22 xr, a)), where yp: X A Rd is a feature map, such that E(0, xr, a) [0, for all x E X, a E A e O. Then, the lifted information ratio satisfies S d/2.
The proof of this result (deferred to Appendix B follows the arguments of Proposition 5 of Russo and Van Roy| 2016). with some small but important changes to account for the presence of contexts.
Notably, both of these results match the classic bounds Russo and Van Roy [2066 on the standard definition of the information ratio for these settings (cf. their Propositions 5 and 3), implying that lifting to the space of parameters does not substantially impact the ingormiinformation tradeoff.
4.2 Concrete regret bounds for Bernoulli bandits
We now instantiate our bounds in two well-studied settings for Bernoulli bandits. We start from the fully unstructured case, assuming finite actions and finitely supported prior. The following regret bound follows directly from Theorem and Lemma [
Theorem 3. Consider contextual bandit with K actions and binary losses, and suppose 1₁, the support of 11, is finite with |01| N. Then, the Bayesian regret of TS satisfies:
Rr S / 22gTlog N.
This result comparable to the best known regret guarantees for this problem due to Foster and Krishnamurthy [20T)] and /Zang [202T] and matches the minimax rate for unstructured contextual bandits with a policy class of size N [Beygelzimer et al.) [011 Dudík et al. [011]. However, we not aware of comparable result for the Thompson sampling algorithm in the literature, be it Bayesian or not.
Moving to Bernoulli bandits with structure, we consider well-studied setting known as logis- tie bandits. this model, the losses are generated by a Bernoulli distribution as Le(8, x. a) ~ Ber(a (z a))), where a(z) = 1/(1 + e e=*) is the sigmoid function. We just assumee that fø(r, a). called the logit function, C-Lipschitz 0, which directly implies that Assumption holds. Notice that our definition generalizes the commonly used notion of logistic bandits that consider linear logit functions of the form fo(2, a) = (0, p2x a)), where yp X' x - IRd is some feature map. Our result for logistic bandits is based on Theorem 2 and Lemma] [

---

Theorem 4. Assume O C. Rd and [|0]| S S for all 0 E Φ₁ Consider class of logistic bandits with K actions and C-Lipschitz logit function fo(2.a) a). Then, the Bayesian regret of TS after rounds can be bounded as:
RT S /2KT (d (dlog(2SCT +1)
The proof follows from an application of Theorem 2 and can be found in Appendix A.3 To knowledge, this is the first regret bound for logistic bandits with nonlinear logits.
We can further specialize our result to linear logits, the setting that is most commonly studied in the literature:
Corollary 1. Assume θ C Rd. ||0|| & S for all E Φ Consider a class of logistic bandits with
K actions and linear logit function fa(x, a) = (f(xx, a), 0), with [ a)ll S B for all x E X and a E [K]. Then, the Bayesian regret of TS after T rounds can be bounded as:
RT S /2KT (d loloo((2))) (2SBT + +1 = Ö(VaKT).
The proof of the corollary is stated at Appendix A.3 A remarkable feature of this bound is that it shows no dependence on the minimum derivative of the sigmoid link function, albeit at the price of VK factor the bound. Nevertheless, we believe this to be the first regret guarantee that entirely gets rid of this potentially enormous constant without very strong assumptions. Indeed, this constant
has been present in nearly all previous bounds we are aware of [Filippi et al.] 2010 Li et al., 2017]
Faury et al. 2020 Abeille et al.) [021] Faury et al. [022] a these results have the advantage
of holding in a frequentist sense. In the Bayesian setting, the works of and Van Roy [2018) and Dong et al. [2019) have proved variety of bounds on the regret of Thompson sampling for non-contextual logistic bandits, but none of them are directly comparable with our result above. Dong et al. [2019] prove a regret bound of order d T for a highly specialized setting with A = θ and a range of other bounds under variety of strong assumptions.
An improved bound for the many-actions setting that scales at most logarithmically with K remains an open problem. Its difficulty is testified by a set of negative examples provided by Doog et al. 2019]. and by a long-lived conjecture of [Dong and Van Roy) 2018 regarding the information ratio for logistic bandits, that, to our knowledge, has not yet been verified in theory.
4.3 Beyond binary losses: linear bandits with Gaussian noise
We now illustrate how our techniques (in particular the lifted information ratio) can be extended beyond the case of binary losses, and in particular consider the classic setting of Bayesian lincar contextual bandits, where the loss is linear function of d-dimensional feature map with additive Gaussian noise, and the prior is also Gaussian.
Lemma 3. Suppose 6* be d- d-dimensional normal random vector, so 0 ~ N (0s, XIa) for some >> O, the loss is a normal random variable L, (((@@*, X2, At), o²) for some standard deviation O, and the expected loss is linear, so f(0,2,,0 x. a) (0, sp(x, a)), where yp X x [K] Rdd is feature map. Then, the lifted information ratio satisfies Pt I min 2(1 + log K)}.
For this setting, the bound < has already been shown for the classic information ratio by Russo and Van Roy (0166). However, believe that our bound P: S 2(1 + log K) is new for any definition of information ratio. By combining this bound on the lifted information ratio with standard arguments for linear contextual bandits, we recover both of the well-known regret bounds of order d log T and
log (KT) for this seting, respectively due to Abbasi- Yadkori et al. [201]] and Chu et [001] See Corollary 2 in Appendix B for a rigorous statement and the proof. This result, although not surprising, indicates once again that our notion of lifted information ratio does not lead to compromises in performance, even when the losses are not binary.
5
Analysis
This section presents the key ideas of the proofs of our main results. We will just provide the proof of Theorem D and that of Lemma] T which we believe offer the most insight into our techniques. All
Whiile for the cases when d & log K the dependence on K is not present in the regret bound, our analysis is restricted to the setting with the finite number of actions. Still, using standard discretization argument, it is possible to extend the analysis infinite action spaces.
7

---

other proofs, included those of auxiliary lemmas, are deferred to Appendix A and B For sake of clarity, we focus on the relatively simple case where θ countable, so that (with a slight abuse of notation) we can write (+) to denote the posterior probability associated with 0. Note, however, that our full proofs also handle the case of general distributions (details in Appendix A.T
5.1 The proof of Theorem
Recalling the definition of the lifted information ratio (Equation 2 we first notice that the regret can be rewritten as follows:
T
T
E
(e(0", X1, Ad) - -(,
Az))
=
E
Eg
[e(0",
Xx, e10", (1, Xc.
T
T
E
Vpeti
Pe
E
1 L.)
(3)
where the first step uses the tower rule of expectation, the second step the definition of Pt. and the final step follows from the Cauchy-Schwarz inequality.
The key challenge is then to bound the sum of information- gain terms. The following lemma provides more tractable form of this sum:
Lemma 4. Under the assumptions of Theorem
T L( Le)
II p (Lr) E log Eo (1(0) II (L.)
E
The proof of this lemma is based on a classic "Bayesian telescoping" argument that we have learned from Grünwald [2012]. We provide the proof of Lemma 4 in Appendix A. Supposing that the prior has bounded entropy, we can easily bound the term appearing on the right hand side as follows:
I po- (L,)
Ih po (Le) Q1(0)) IT
log Σ. (1) (T'
SE
log
=E log Qı =
Po- .o.)((.)))))
This concludes the proof of the first statement. The second statement follows from the first using the trivial bound on the Shannon entropy of any finite-support distribution.
5.2 The proof of Lemma
We start by introducing some notation that will be useful for the proof. In particular, we use g to denote the binary relative entropy function defined for all p, E [0, 1] as
((ll4)) plog P + + p) log
1 1-p
1-q"
and use the convention 0 log0 = O. Furthermore, define the posterior mean loss as (e(x,a) a) R, [e(0", a)). These notations allow us to conveniently rewrite the information gain as T,(0;; T.) E, [DKL (PL. F..1...A-....AA PL.IF....
[g (e(O*, X1, Ao)l(E A
We will now prove generalization of Lemma Lemma 5. Under the assumptions of Lemma sampling satisfies < 2 EneA E, [fa(X,,a))
which will directly imply the original result: for all t, the lifted information ratio of Thompson
Proof. The proof is based on an application of the Fenchel- Young inequality, which requires the introduction of the Legendre-Fencher conjugate of with respect to its first argument. This function defined for all u E I.
u² g" (ullg) = sup {nu ((pllg)} = log(1 + q(e" 1)) s uu pE(0.1)
(5)
2

---

where the second equality and the inequality follow from set of straightforward calculations deferred to Appendix A.4 Turning the actual proof, we consider the instantaneous pseudo-regre regret
EiO0 X,, A.) E(0', X1, At) in fixed round / and write the following (for any 1) > 0):
Es [ri] - E, [e(0", Xx,, A,) 8(0", Xe,
= Es [e(0", X,, At) E(0,, Xt. At))
(using that A7) has the same conditional distribution as As))
Ee (X,, E(,,, A.)]
(by conditional independence of 0* and A,)
T) (a) Xx) 1) Xx) {(0,, Xe,a)
E.
l (x( Ar)
(A
BEA
ex(X Ar) + 11 Σ Ta(a) (
< E2
aEd
(-IHAA "JTTE / Xe)
((Xesaa
+ 9"
(by the Fenchel- Young inequality)
[(Ai=a) a()eaa) 1)x((a|x,)
< le(X Ar)
Σ (a/xx)
[(A= ((X2.a))
+ 217² (e)x)a
(by Equation 5
E,
Te(a| Xt, a) ()( + Σ ((X),a)
1
2n aEA
aEA
(by the tower rule of expectation and IE, [[<A1=a]] = Tee (e())
E,
Te(()X))g(((0", X,, a))l<(X,,.)) + aeAA
Σ ((X)aa) aeA
2n
(using again that Og has the same conditional distribution as 0*)
= =m/4(00;;;)) + 2n aEA
Σ E, [))
Choosing the value of 0 for which the latter expression is minimal, obtain E2 [rx] S V (0+: E, [fr( a)). The proof is completed by taking the square on both sides and rearranging.
6 Conclusion
We have presented a new theoretical framework for analyzing Thompson sampling in contextual bandits, resulting in new results that advance the state of the art the well-studied problem of logistic bandits. We believe that these results encouraging and that our analytic framework may find many more applications in the future.
As always, we leave many more questions open than what we have closed. One major question regarding logistic bandits is if is possible improve our new results by significantly toning down the dependence on the number of actions K. In light of existing hardness results for nonlinear bandit problems (e.g., for generalized linear bandits with ReLU activation, Dong et al. 2021 Foster et al. 202T) we suspect that this may not be possible. As a more modest goal, we are curious to find out if the lifted information ratio can be upper bounded in terms of the smallest-slope parameter k as done in many other works on logistic bandits since Filippi et al. 2010). We conjecture that O(k-2d)

---

bound on the lifted information ratio is indeed possible. but we were not able to prove it so far. This the case for the eluder dimension [Russo and Van Roy 2013], another complexity measure that has been used to hounr-bound the regret for contextual bandits. The eluder dimension for linear losses is O(d), but for nonlinear losses we know only of (xx=22) bounds for the generalized linear case.
More broadly, we believe that the most interesting immediate challenge is to extend our results to hold beyond the Bayesian setting. As counterexample by ZZhngg 2021] shows, this may not be possible in general, but we wonder if his "feel-good" adjustment of Thompson sampling could be analyzed with the techniques we introduced in this paper.
Throughout the paper. we have studied several different settings, some of which come with a wide range of possible choices for the form of the prior and the likelihood. In some of this scenarios, updating the posterior and sampling from it may be computationally challenging. We have ignored this aspect in order focus the pure online-decision aspects, and implicitly assumed that posterior sampling can be performed without approximations. In practice, several heuristics have been proposed, see for instance [Dumitrascu et al. 2018) on efficient TS for logistic bandits. It would be interesting to study how approximate sampling affects our regret guarantees, along the line of [Phan et al. 2019 Mazumdar al. 2020
Acknowledgments and Disclosure of Funding
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 950180)
References
Y. Abbasi-Yadkorti D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. In NeurlPS, pages 2312-2320, 2011.
M. Abeille, L. Faury, and C. Calauzènes. Instance-wise minimax-optimal algorithms for logistic bandits. In A/STATS, volume 130 of Proceedings of Muchine Learning Research, pages 3691-3699. PMLR, 2021.
Agrawal and N. Goyal. Analysis of Thompson sampling for the multi-armed bandit problem, In COLT, volume 23 of JMLR Proceedings, pages 39.1-39.26. JMLR.org, 2012.
Agrawal and N. Goyal. Further optimal regret bounds for Thompson sampling. In AISTATS, volume 31 of JMLR Workshop and Conference Proceedings, pages 99-107. JMI..org. 2013a.
Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. ICML (3), volume 28 of JMLR Workshop and Conference Proceedings, pages 127-135. JMLR.org, 2013b.
A. Beygelzimer, J. Langford, I.. Li, I. Reyzin, and R. F. Schapire. Contextual bandit algorithms with supervised learning guarantees. In AISTATS, volume 15 of JMLR Proceedings, pages 19-26. JMg, 2011.
S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic problems. Found. Trends Mach. Learn., 5(1): 2012.
bandit
Bubeck and M. Sellke. First-order bayesian regret analysis of thompson sampling. In Algorithmic Learning Theory, pages 196-233. PMLR, 2020.
O. Chapelle and Li. An empirical evaluation of Thompsoon sampling. NeurlPS, pages 2249-2257, 2011.
W. Chu, Li, I.. Reyzin, and R. E. Schapire. Contextual bandits with linear payoff functions. In AISTATS, volume 15 of JMLR Proceedings, pages 208-214. JMLR.org, 2011.
K. Dong, Yang, and Ma. Provable model-based nonlinear bandit and reinforcement learning: Shelve optimism, embrace virtual curvature. In NeurIPS, pages (6168-26182) 2021.
10

---

S. Dong and B. Van Roy. An information -heorretic analysis for Thompson sampling with many actions. In NeurIPS, pages 4161-4169, 2018.
Dong, T. Ma, and B. Van Roy. On the performance Thompson sampling on logistic bandits. In COLT, volume 99 of Proceedings of Machine Learning Research, pages 1158-1160. PMLR, 2019.
M. Dudik, J. Hsu, S. Kale, N. Karampatziakis, Langford, Reyzin, and Zhang. Efficient optimal learning for contextual bandits. In UAI, pages 169-178. AUAI Press, 2011.
B. Dumitrascu, K. Feng, and B. E. Engelhardt. PG-TS: improved Thompson sampling for logistic contextual bandits. In NeurIPS, pages 4629-4638, 2018.
L. Faury, M. Abeille, C. Calauzènes, and O. Fercoq. Improved optimistic algorithms for logistic bandits. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 3052-3060. PMLR, 2020.
L. Faury, M. Abeille, K. Jun, and C. Calauzènes. Jointly efficient and optimal algorithms for logistic bandits. CoRR, abs/2201.01983 2022.
S. Filippi, O. Cappé, A. Garivier, and C. Szepesvári. Parametrie bandits: The generalized linear case.
In NeurlPS, pages 586-594. Curran Associates, Inc., 2010.
D. J. Foster and A. Krishnamurthy, Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. In NeurlPS, pages 18907-18919, 2021.
D. J. Foster and A. Rakhlin. Beyond UCB: optimal and efficient contextual bandits with regression oracles. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 1199-3210 PMLR, 2020.
J. J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin. The statistical complexity of interactive decision making. CoRR, abs/21 12.13487, 2021.
Grünwald. The safe Bayesian learning the learning rate via the mixability gap. ALT, volume 7568 of Lecture Notes in Computer Science, pages 169-183. Springer, 2012.
B. Hao, T. Lattimore, and C. Qin. Contextual information-directee sampling. In Proceedings of the 39th International Conference on Machine Learning, pages 8446-8464, 2022.
E. Kaufmann, N. Korda, and R. Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In ALT, volume 7568 of Lecture Notes in Computer Science, pages 199-213. Springer, 2012.
Kirschner and A. Krause. Information directed sampling and bandits with heteroscedastic noise. In COLT, volume 75 of Proceedings of Machine Learning Research, pages 358-384 PMLR, 2018.
Kirschner. T. Lattimore, and A. Krause. Information directed sampling for linear partial monitoring. In Conference on Learning Theory, pages 2328-2369) 2020.
B. Kveton, M. Zaheer, C. Szepesvári, L. Li, M. Ghavamzadeh, and C. Boutilier. Randomized exploration in generalized linear bandits. In AISTATS, volume 108 of Proceedings of Machine Learning Research, pages 2066-2076. PMLR, 2020.
Lattimore and C. Szepesvári. Bandil Algorithms. Cambridge University Press, 2020.
L. Li. Generalized Thompson sampling for contextual bandits. CoRR, abs/1310.71633 2013.
L. Li, Lu, and D. Zhou. Provably optimal algorithms for generalized linear contextual bandits. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 2071-2080. PMLR, 2017.
Liu, Z. Zheng, and N. B. Shroff. Analysis of thompson sampling for graphical bandits without the graphs. In UAI, pages 13-22. AUAI Press, 2018.
Y. Liu, A. M. Devraj, B. Van Roy, and Xu. Gaussian imagination bandit learning. arXiv preprint arXiv:2001.01902, 2022.

---

X. Lu and B. Van Roy. Ensemble sampling. In Advances in neural information processing systems, volume 30, 2017.
X. Lu, B. Van Roy, Dwaracherla. M. Ibrahimi, I. Osband, and Z. Wen. Reinforcement learning, bit by bit. arXiv preprini
2021.
E. Mazumdar, A. Pacchiano, Y. Ma, M. I. Jordan, and P. L. Bartlett. On approximate Thompson sampling with langevin algorithms. ICML, volume 119 of Proceedings of Machine Learning Research, pages 6797-6807. PMLR, 2020.
M. Phan, Y. Abbasi- Yadkori, and J. Domke. Thompson sampling and approximate inference. In NeurlPS, pages
2019.
C. Qin, Z. Wen, X. Lu, and B. Van Roy. An analysis of ensemble sampling. CoRR, abs22033.03333, 2022.
D. Russo and B. Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In NeurIPS, pages 2256-2264, 2013.
D. Russo and Van Roy. Learning optimize via posterior sampling. Math. Oper. Res., 39(4): 1243, 2014.
D. Russo and B. Van Roy. An information-thhooreie analysis of Thompson sampling. The Journal of Machine Learning Research, 17(1):2442-244 2016.
D. Russo and B. Van Roy. Learning to optimize via cnermattionddrectte sampling. Oper. Res., 66 (L):230-252, 2018.
M. Schervish. Theory of Statistics. Springer Series in Statisties. Springer New York, 1996. ISBN 9780387945460.
S. L. Scott. A modern Bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry, 26(6):639-655 2010.
W. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Bulletin of the American Mathematics Society, 25:285-294, 1933.
T. Zhang. Feel-good Thompson sampling for contextual bandits and reinforcement learning. CoRR, abs/2110.008s 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] This is done in Section 6
(c) Did you discuss any potential negative societal impacts your work? [N/A] Our results of the theoretical nature, so they don't ton't have any social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are stated in Section[Jann and in the appendix of the paper.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasig new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "LQgcgp8jMzm", "reviewer": "Reviewer_piBD", "summary": "Previously, information ratio was used to bound regret of Thomspon sampling algorithm for multi-armed bandits. This paper provides examples that information ratio loses its usefulness in deriving bounds for contextual bandits. They introduce a new lifted information ratio which deals with information gained about parameter instead of best action. Using this ratio, the authors derive regret bounds for Thomspon Sampling. Using the bounds, they derive regret bounds under several interesting cases.", "questions": "Paper can benefit more form discussion on tractability.", "limitations": "Satisfied with authors responses.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nLifted information ratio is an intuitive yet interesting quantity which authors motivate quite well. They also provide tight regret bounds using the new quantity.\n\nWeaknesses:\nAs paper mentions, they don't provide any tractable algorithms and leave that task for future. It would be better if they can add more discussion on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5B_Zyv-zLky", "reviewer": "Reviewer_qsXa", "summary": "This paper focuses on parametrized contextual bandit problems, and their analysis via the Russo-Van Roy information ratio framework. The main contribution is to observe that one can analyze information gain on the \"lifted\" level of the \"underlying context\" instead of the \"optimal action\". This leads to nice new regret bounds for logistic bandits. The analyses are clean and remove dependence on problem-dependent constants which were common in previous work. At a high level, it seems that the \"lifted\" information ratio may often be the right way to study parametrized contextual bandits. ", "questions": "The most pressing suggestion is to incorporate the prior work on general learning targets. Assuming I'm not mistaken about the strong similarity, I suggest the authors make substantial revisions to acknowledge, compare with, and perhaps build upon this earlier work. For example, introducing the term \"lifted information ratio\" seems inappropriate if the idea already has a name. I think the submitted paper still has plenty of merit, and it seems plausible to me that the application to logistic bandits gives a more convincing application of this idea than the previous works I mentioned. I'm not familiar enough with the literature to say. If I'm confused about this point, please let me know and I will be happy to raise my score.\n\nI have a few other questions and comments:\n\n1. The authors write: \n\n\"When considering i.i.d. contexts and finite policy classes, one can apply the theory of Russo and Van Roy [2016] and treat policies as actions to obtain regret bounds scaling with the entropy of the optimal policy, but this can lead to a polynomial dependence on the number of contexts.\" \n\nI would appreciate some elaboration on this point. My understanding is that the authors have in mind the setting of Thompson sampling with changing feedback graphs G_t (which for contextual bandit would be a union of K vertex-disjoint cliques). I believe the regret bounds here are of order \\sqrt{TK H} where H is the entropy of the optimal policy (e.g. https://arxiv.org/abs/1805.08930 or section 8 of https://arxiv.org/abs/1902.00681). I don't see this polynomial dependence on the number of contexts anywhere and would like some clarification. It's quite possible that I am misunderstanding something basic and I apologize if this is the case.\n\n2. In line 101, I was a bit surprised that the adversary does not know \\theta^*. It would be nice to have some commentary on whether this is likely to be truly essential, when it is used, if it is required in related work, etc. On its face it seems to detract a little bit from the picture.\n\n3. At the end of page 7, I was confused about the Bayesian telescoping argument. In particular it looks to me like the usual entropy chain rule should suffice: the expected information-gain at time t is by definition the average decrease in the entropy of \\theta^*. Summing and using the fact that entropy is always non-negative, the upper bound of H(\\theta^*) should follow. So it seems like the calculation was overcomplicated a bit and I cannot tell why.\n\n4. Equation (1): missing a comma in the first \\ell(\\cdot) expression.\n\n5. A positive comment to finish: I appreciated that the authors focused on countable priors while making it clear that this is just for convenience. I thought this choice made things quite pleasant to read.\n\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the results and the writing is very nice. \n\nAs far as I can tell, the main problem is that the key idea is less novel than the authors seem to believe. In particular while the application to logistic bandits is probably new, at an abstract level the lifted information ratio seems to essentially be a special case of the notion of \"learning target\" discussed in https://arxiv.org/abs/2103.04047 and https://arxiv.org/abs/2201.01902. (The adversarial contexts may change things slightly, but it seems mostly inconsequential for my comment.) I expand on what to do about this in the \"Questions\" section. In short, I think the submission should probably be revised to take this literature into account, and then resubmitted to a future conference (where I would support its acceptance).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4XNPsoMJ4cg", "reviewer": "Reviewer_hEwR", "summary": "The authors study the Bayesian regret of Thompson sampling with binary losses and adversarial contexts. Adapting the information theoretic approach from Russo and Van Roy, through a definition of lifted information ratio, they provide a regret bound similar to the non-contextual case. This leads to state-of-the-art regret guarantees in many well-known cases covered by the proposed setting. Notably, they prove a regret bound for K actions, and Lipschitz logits which is independent on the slope of the link function. ", "questions": "- Please look at the weakness part. Addressing the first two points will improve the paper.\n", "limitations": "Being a theoretical work any adverse social impact is hard to foresee.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The lifting of information ratio is supported by the intuition that the information of the optimal action in one context may be useless in the other context. Thus substituting the mutual information between optimal action, and taken action and loss, with the mutual information between true parameter, and the loss resolves the problem.\n- The adaptation of the information theoretic framework for Bayesian regret bounds in adversarial contextual setting is new. This has the potential for generalization in many other settings where information theoretic framework is used.\n- The current approach results in state-of-art regret guarantees in many problems with adversarial contexts.\n- The relation to the “decoupling coefficient” proposed by Zhang is also interesting (but this raises some question as well).\n\n\nWeaknesses\n- Moving away from the mutual information between optimal action to true parameter suggests we may end up with worse scaling w.r.t. the number of actions. However, that does not seem to be the case as seen in Lemma 3 which provides a $log(K)$ regret bound for $K$ actions. Maybe the authors should expand on how that is achieved. \n\n- Although the intuition supports the lifted Information ratio approach, it will be good to more rigorously point out why the previous information ratio fails in the contextual case. \n\n- Zhang explicitly mentions that TS may not be efficient in some cases, and introduces a bias towards higher reward in the Feel Good TS  approach. In light of that, the current connection seems to suggest TS suffice, at least for Gaussian case. Is this special for Gaussian case, and will fail in more general situations?\n\n\n- Can there be an approach where we can judicially choose between the mutual information between optimal action, vs the true parameter while analyzing TS? It will be great if such an approach is found, as we can incorporate that in the analysis. Some comments will be interesting.\n\nPost rebuttal:\n- The current paper seems like a special case of the the notion of \"learning target\" discussed in https://arxiv.org/abs/2103.04047 and https://arxiv.org/abs/2201.01902 (as pointed out by another Reviewer).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept