Title: Follow-ups Also Matter: Improving Contextual Bandits via Post-serving Contexts
Venue: NeurIPS 2023
Accepted: yes
Keywords: linear stochastic bandits

online learning

partial information

contextual bandits
Abstract: Standard contextual bandit problem assumes that all the relevant contexts are observed before the algorithm chooses an arm. This modeling paradigm, while useful, often falls short when dealing with problems in which additional valuable contexts can be observed after arm selection. For example, content recommendation platforms like Youtube, Instagram, Tiktok receive much additional features about a user's reward after the user clicks a content (e.g., how long the user stayed, what is the user's watch speed, etc.). To improve online learning efficiency in these applications,  we  study a novel contextual bandit problem with post-serving contexts and design a new algorithm, poLinUCB,  that achieves tight regret under standard assumptions. Core to our technical proof is a robustified and generalized version of the well-known Elliptical Potential Lemma (EPL), which can accommodate  noise in data. Such robustification is necessary for tackling our problem, though we believe it could also be of general interest.
Extensive empirical tests on both synthetic and real-world datasets  demonstrate the significant benefit of utilitzing post-serving contexts as well as the superior performance of  our   algorithm over the state-of-the-art approaches.

Follow-ups Also Matter: Improving Contextual Bandits via Post-serving Contexts

Abstract

Standard contextual bandit problem assumes that all the relevant contexts are observed before the algorithm chooses an arm. This modeling paradigm, while useful, often falls short when dealing with problems in which valuable additional context can be observed after arm selection. For example, content recommendation platforms like Youtube, Instagram, Tiktok also observe valuable follow-up information pertinent to the user's reward after recommendation (e.g., how long the user stayed, what is the user's watch speed, etc.). To improve online learning efficiency in these applications, we study a novel contextual bandit problem with post-serving contexts and design a new algorithm, poLinUCB, that achieves tight regret under standard assumptions. Core to our technical proof is a robustified and generalized version of the well-known Elliptical Potential Lemma (EPL), which can accommodate noise in data. Such robustification is necessary for tackling our problem, and we believe it could also be of general interest. Extensive empirical tests on both synthetic and real-world datasets demonstrate the significant benefit of utilizing post-serving contexts as well as the superior performance of our algorithm over the state-of-the-art approaches.

1 Introduction

Contextual bandits represent a fundamental mathematical model that is employed across a variety of applications, such as personalized recommendations (Li et al., 2010; Wu et al., 2016) and online advertising (Schwartz et al., 2017; Nuara et al., 2018). In their conventional setup, at each round t , a learner observes the context x t , selects an arm a t ∈ A , and subsequently, observes its associated reward r t,a t . Despite being a basic and influential framework, it may not always capture the complexity of real-world scenarios (Wang et al., 2016; Yang et al., 2020). Specifically, the learner often observes valuable follow-up information pertinent to the payoff post arm selection (henceforth, the post-serving context ). Standard contextual bandits framework that neglects such post-serving contexts may result in significantly suboptimal performance due to model misspecification.

Consider an algorithm designed to recommend educational resources to a user by utilizing the user's partially completed coursework, interests, and proficiency as pre-serving context (exemplified in platforms such as Coursera). After completing the recommendation, the system can refine the user's profile by incorporating many post-serving context features such as course completion status, how much time spent on different educational resources, performances, etc. This transition naturally delineates a mapping from user attributes (i.e., the pre-serving context) to user's learning experiences and outcomes (i.e., the post-serving context). It is not difficult to see that similar scenarios happen in many other recommender system applications. For instance, in e-commerce platforms (e.g., Amazon, Etsy or any retailing website), the system will first recommend products based on the user's profile information, purchasing pattern and browsing history, etc.; post recommendations, the system can

3

then update these information by integrating post-serving contexts such as this recent purchase behaviour and product reviews. Similarly, media content recommendation platforms like Youtube, Instagram and Tiktok, also observe many post-serving features (e.g., how long the user stayed) that can refine the system's estimation about users' interaction behavior as well as the rewards.

A common salient point in all the aforementioned scenarios is that the post-serving context are prevalent in many recommender systems; moreover, despite being unseen during the recommendation/serving phase, they can be estimated from the pre-serving context given enough past data. More formaly, we assume that there exists a learnable mapping ϕ ⋆ ( · ) : R d x → R d z that maps pre-serving feature x ∈ R d x to the expectation of the post-serving feature z ∈ R d z , i.e., E [ z | x ] = ϕ ⋆ ( x ) .

Unsurprisingly, and as we will also show, integrating the estimation of such post-serving features can significantly help to enhance the performance of contextual bandits. However, most of the existing contextual bandit algorithms, e.g., (Auer, 2002; Li et al., 2010; Chu et al., 2011; Agarwal et al., 2014; Tewari and Murphy, 2017), are not designed to accommodate the situations with post-serving contexts. We observe that directly applying these algorithms by ignoring post-serving contexts may lead to linear regret, whereas simple modification of these algorithms will also be sub-optimal. To address these shortcomings, this work introduces a novel algorithm, poLinUCB. Our algorithm leverages historical data to simultaneously estimate reward parameters and the functional mapping from the pre- to post-serving contexts so to optimize arm selection and achieves sublinear regret. En route to analyzing our algorithm, we also developed new technique tools that may be of independent interest.

Main Contributions.

First, we introduce a new family of contextual linear bandit problems. In this framework, the decision-making process can effectively integrate post-serving contexts, premised on the assumption that the expectation of post-serving context as a function of the pre-serving context can be gradually learned from historical data. This new model allows us to develop more effective learning algorithms in many natural applications with post-serving contexts.

Second, to study this new model, we developed a robustified and generalized version of the well-regarded elliptical potential lemma (EPL) in order to accommodate random noise in the post-serving contexts. While this generalized EPL is an instrumental tool in our algorithmic study, we believe it is also of independent interest due to the broad applicability of EPL in online learning.

Third, building upon the generalized EPL, we design a new algorithm poLinUCB and prove that it enjoys a regret bound ˜ O ( T 1 -α d α u + d u √ TK ) , where T denotes the time horizon and α ∈ [0 , 1 / 2] is the learning speed of the pre- to post-context mapping function ϕ ⋆ ( · ) , whereas d u = d x + d z and K denote the parameter dimension and number of arms. When ϕ ⋆ ( · ) is easy to learn, e.g., α = 1 / 2 , the regret bound becomes ˜ O ( √ Td u + d u √ TK ) and is tight. For general functions ϕ ⋆ ( · ) that satisfy α ≤ 1 / 2 , this regret bound degrades gracefully as the function becomes more difficult to learn, i.e., as α decreases.

Lastly, we empirically validate our proposed algorithm through thorough numerical experiments on both simulated benchmarks and real-world datasets. The results demonstrate that our algorithm surpasses existing state-of-the-art solutions. Furthermore, they highlight the tangible benefits of incorporating the functional relationship between pre- and post-serving contexts into the model, thereby affirming the effectiveness of our modeling.

2 Related Works

Contextual bandits. The literature on linear (contextual) bandits is extensive, with a rich body of works (Abe et al., 2003; Auer, 2002; Dani et al., 2008; Rusmevichientong and Tsitsiklis, 2010; Lu et al., 2010; Filippi et al., 2010; Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Li et al., 2017; Jun et al., 2017). One of the leading design paradigm is to employ upper confidence bounds as a means of balancing exploration and exploitation, leading to the attainment of minimax optimal regret bounds. The derivation of these regret bounds principally hinges on the utilization of confidence ellipsoids and the elliptical potential lemma. Almost all these works assume that the contextual information governing the payoff is fully observable. In contrast, our work focuses on scenarios where the context is not completely observable during arm selection, thereby presenting new challenges in addressing partially available information.

Contextual bandits with partial information. Contextual bandits with partial information has been relatively limited in the literature. Initial progress in this area was made by Wang et al. (2016), who studied settings with hidden contexts. In their setup there is some context (the post-serving context in our model) that can never by observed by the learner, whereas in our setup the learner can observe post-serving context but only after pulling the arm. Under the assumption that if the parameter initialization is extremely close to the true optimal parameter, then they develop a sub-linear regret algorithm. Our algorithm does not need such strong assumption on parameter initialization. Moreover, we show that their approach may perform poorly in our setup. Subsequent research by Qi et al. (2018); Yang et al. (2020); Park and Faradonbeh (2021); Yang and Ren (2021); Zhu and Kveton (2022) investigated scenarios with noisy or unobservable contexts. In these studies, the learning algorithm was designed to predict context information online through context history analysis, or selectively request context data from an external expert. Our work, on the other hand, introduces a novel problem setting that separates contexts into pre-serving and post-serving categories, enabling the exploration of a wide range of problems with varying learnability. Additionally, we also need to employ new techniques for analyzing our problem to get a near-optimal regret bound.

Generalizations of the elliptical potential lemma (EPL). The EPL, introduced in the seminal work ofLai and Wei (1982), is arguably a cornerstone in analyzing how fast stochastic uncertainty decreases with the observations of new sampled directions. Initially being employed in the analysis of stochastic linear regression, the EPL has since been extensively utilized in stochastic linear bandit problems (Auer, 2002; Dani et al., 2008; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Li et al., 2019; Zhou et al., 2020; Wang et al., 2022). Researchers have also proposed various generalizations of the EPL to accommodate diverse assumptions and problems. For example, Carpentier et al. (2020) extended the EPL by allowing for the use of the X -p t -norm, as opposed to the traditional X -1 t -norm. Meanwhile, Hamidi and Bayati (2022) investigated a generalized form of the 1 ∧∥ φ ( x t ) ∥ 2 X -1 t -1 term, which was inspired by the pursuit of variance reduction in non-Gaussian linear regression models. However, existing (generalized) EPLs are inadequate for the analysis of our new problem setup. Towards that end, we develop a new generalization of the EPL in this work to accommodate noisy feature vectors .

3 Linear Bandits with Post-Serving Contexts

Basic setup. We hereby delineate a basic setup of linear contextual bandits within the scope of the partial information setting, whereas multiple generalizations of our framework can be found in Section 6. This setting involves a finite and discrete action space, represented as A = [ K ] . Departing from the classic contextual bandit setup, the context in our model is bifurcated into two distinct components: the pre-serving context, denoted as x ∈ R d x , and the post-serving context, signified as z ∈ R d z . When it is clear from context, we sometimes refer to pre-serving context simply as context as in classic setup, but always retain the post-serving context notion to emphasize its difference. We will denote X t = ∑ t s =1 x s x ⊤ s + λ I and Z t = ∑ t s =1 z s z ⊤ s + λ I . For the sake of brevity, we employ u = ( x , z ) to symbolize the stacked vector of x and z , with d u = d x + d z and ∥ u ∥ 2 ≤ L u . The pre-serving context is available during arm selection, while the post-serving context is disclosed post the arm selection. For each arm a ∈ A , the payoff, r a ( x , z ) , is delineated as follows:

where θ ⋆ a and β ⋆ a represent the parameters associated with the arm, unknown to the learner, whereas η is a random noise sampled from an R η -sub-Gaussian distribution. We use ∥ x ∥ p to denote the p -norm of a vector x , and ∥ x ∥ A := √ x ⊤ Ax is the matrix norm. For convenience, we assume ∥ θ ⋆ a ∥ 2 ≤ 1 and ∥ β ⋆ a ∥ 2 ≤ 1 for all a ∈ A . Additionally, we posit that the norm of the pre-serving and post-serving contexts satisfies ∥ x ∥ 2 ≤ L x and ∥ z ∥ 2 ≤ L z , respectively, and max t ∈ [ T ] sup a,b ∈A ⟨ θ ⋆ a -θ ⋆ b , x t ⟩ ≤ 1 and max t ∈ [ T ] sup a,b ∈A ⟨ β ⋆ a -β ⋆ b , z t ⟩ ≤ 1 , same as in (Lattimore and Szepesvári, 2020).

3.1 Problem Settings and Assumptions.

The learning process proceeds as follows at each time step t = 1 , 2 , · · · , T :

The learner observes the context x t .

An arm a t ∈ [ K ] is selected by the learner.

3. The learner observes the realized reward r t,a t and the post-serving context, z t .

Without incorporating the post-serving context, one may incur linear regret as a result of model misspecification, as illustrated in the following observation. To see this, consider a setup with two arms, a 1 and a 2 , and a context x ∈ R drawn uniformly from the set {-3 , -1 , 1 } with ϕ ⋆ ( x ) = x 2 . The reward functions for the arms are noiseless and determined as r a 1 ( x ) = x + x 2 / 2 and r a 2 ( x ) = -x -x 2 / 2 . It can be observed that r a 1 ( x ) > r a 2 ( x ) when x ∈ {-3 , 1 } and r a 1 ( x ) < r a 2 ( x ) when x = -1 . Any linear bandit algorithm that solely dependent on the context x (ignoring ϕ ⋆ ( x ) ) will inevitably suffer from linear regret, since it is impossible to have a linear function (i.e., r ( x ) = θx ) that satisfies the above two inequalities simultaneously.

Observation 1. There exists linear bandit environments in which any online algorithm without using post-serving context information will have Ω( T ) regret.

Therefore, it is imperative that an effective learning algorithm must leverage the post-serving context, denoted as z t . As one might anticipate, in the absence of any relationship between z t and x t , it would be unfeasible to extrapolate any information regarding z t while deciding which arm to pull, a point at which only x t is known. Consequently, it is reasonable to hypothesize a correlation between z t and x t . This relationship is codified in the subsequent learnability assumption.

Specifically, we make the following natural assumption - there exists an algorithm that can learn the mean of the post-serving context z t , conditioned on x t . Our analysis will be general enough to accommodate different convergence rates of the learning algorithm, as one would naturally expect, the corresponding regret will degrade as this learning algorithm's convergence rate becomes worse. More specifically, we posit that, given the context x t , the post-serving context z t is generated as 1

Here, ϵ t is a zero-mean noise vector in R d z , and ϕ ⋆ : R d x → R d z can be viewed as the post-serving context generating function, which is unknown to the learner. However, we assume ϕ ⋆ is learnable in the following sense.

Assumption 1 (Generalized learnability of ϕ ∗ ) . There exists an algorithm that, given t pairs of examples { ( x s , z s ) } t s =1 with arbitrarily chosen x s 's, outputs an estimated function of ϕ ⋆ : R d x → R d z such that for any x ∈ R d x , the following holds with probability at least 1 -δ ,

where α ∈ (0 , 1 / 2] and C 0 is some universal constant.

The aforementioned assumption encompasses a wide range of learning scenarios, each with different rates of convergence. Generally, the value of α is directly proportional to the speed of learning; the larger the value of α , the quicker the learning rate. Later, we will demonstrate that the regret of our algorithm is proportional to O ( T 1 -α ) , exhibiting a graceful degradation as α decreases. The ensuing proposition demonstrates that for linear functions, α = 1 / 2 . This represents the best learning rate that can be accommodated 2 . In this scenario, the regret of our algorithm is O ( √ T ) , aligning with the situation devoid of post-serving contexts (Li et al., 2010; Abbasi-Yadkori et al., 2011).

Observation 2. Suppose ϕ ( · ) is a linear function, i.e., ϕ ( x ) = Φ ⊤ x for some Φ ∈ R d x × d z , then e δ t = O ( ∥ x ∥ X -1 t · log ( t/δ ) ) .

This observation follows from the following inequalities ∥ ϕ t ( x ) -ϕ ⋆ ( x ) ∥ = ∥ ̂ Φ ⊤ t x -Φ ⋆ ⊤ x ∥ ≤ ∥ ̂ Φ t -Φ ⋆ ∥ X t · ∥ x ∥ X -1 t = O ( ∥ x ∥ X -1 t · log ( t δ ) ) , where the last equation is due to the confidence ellipsoid bound (Abbasi-Yadkori et al., 2011). We refer curious readers to Appendix D.2 for a discussion on other more challenging ϕ ( · ) functions with possibly worse learning rates α .

3.2 Warm-up: Why Natural Attempts May Be Inadequate?

Given the learnability assumption of ϕ ⋆ , one natural idea for solving the above problem is to estimate ϕ ⋆ , and then run the standard LinUCB algorithm to estimate ( θ a , β a ) together by treating ( x t , ̂ ϕ t ( x t )) as the true contexts. Indeed, this is the approach adopted by Wang et al. (2016) for addressing a similar problem of missing contexts z t , except that they used a different unsurprised-learning-based approach to estimate the context z t due to not being able to observing any data about z t . Given the estimation of ̂ ϕ , their algorithm - which we term it as LinUCB ( ̂ ϕ ) - iteratively carries out the steps below at each iteration t (see Algorithm 2 for additional details): 1) Estimation of the context-generating function ̂ ϕ t ( · ) from historical data; 2) Solve of the following regularized least square problem for each arm a ∈ A , with regularization coefficient λ ≥ 0 :

Under the assumption that the initialized parameters in their estimations are very close to the global optimum, Wang et al. (2016) were able to show the O ( √ T ) regret of this algorithm. However, it turns out that this algorithm will fail to yield an satisfying regret bound without their strong assumption on very close parameter initialization, because the errors arising from ̂ ϕ ( · ) will significantly enlarge the confidence set of ̂ θ a and ̂ β a . 3 Thus after removing their initialization assumption, the best possible regret bound we can possibly achieve is of order ˜ O ( T 3 / 4 ) , as illustrated in the subsequent proposition.

Proposition 1 (Regret of LinUCB-( ̂ ϕ )) . The regret of LinUCB-( ̂ ϕ ) in Algorithm 2 is upper bounded by ˜ O ( T 1 -α d α u + T 1 -α/ 2 √ Kd 1+ α u ) with probability at least 1 -δ , by carefully setting the regularization coefficient λ = Θ( L u d α u T 1 -α log ( T/δ )) in Equation 1.

Since α ∈ [0 , 1 / 2] , the best possible regret upper bound above is ˜ O ( T 3 / 4 ) , which is considerably inferior to the sought-after regret bound of ˜ O ( √ T ) . Such deficiency of LinUCB-( ̂ ϕ ) is further observed in all our experiments in Section 7 as well. These motivate our following design of a new online learning algorithm to address the challenge of post-serving context, during which we also developed a new technical tool which may be of independent interest to the research community.

4 A Robustified and Generalized Elliptical Potential Lemma

It turns out that solving the learning problem above requires some novel designs; core to these novelties is a robustified and generalized version of the well-known elliptical potential lemma (EPL), which may be of independent interest. This widely used lemma states a fact about a sequence of vectors x 1 , · · · , x T ∈ R d . Intuitively, it captures the rate of the sum of additional information contained in each x t , relative to its predecessors x 1 , · · · , x t -1 . Formally,

Lemma (Original Elliptical Potential Lemma) . Suppose (1) X 0 ∈ R d × d is any positive definite matrix; (2) x 1 , . . . , x T ∈ R d is any sequence of vectors; and (3) X t = X 0 + ∑ t s =1 x s x ⊤ s . Then the following inequality holds

where a ∧ b = min { a, b } is the min among a, b ∈ R .

To address our new contextual bandit setup with post-serving contexts, it turns out that we will need to robustify and generlaize the above lemma to accommodate noises in x t vectors and slower learning rates. Specifically, we present the following variant of the EPL lemma.

Lemma 1 (Generalized Elliptical Potential Lemma) . Suppose (1) X 0 ∈ R d × d is any positive definite matrix; (2) x 1 , . . . , x T ∈ R d is a sequence of vectors with bounded l 2 norm max t ∥ x t ∥ ≤ L x ; (3) ϵ 1 , . . . , ϵ T ∈ R d is a sequence of independent (not necessarily identical) bounded zero-mean noises satisfying max t ∥ ϵ t ∥ ≤ L ϵ and E [ ϵ t ϵ ⊤ t ] ≽ σ 2 ϵ I for any t ; and (4) ˜ X t is defined as follows:

Then, for any p ∈ [0 , 1] , the following inequality holds with probability at least 1 -δ ,

Note that the second term is independent of time horizon T and only depends on the setup parameters. Generally, this can be treated as a constant. Before describing main proof idea of the lemma, we make a few remarks regarding Lemma 1 to highlight the significance of these generalizations.

The original Elliptical Potential Lemma (EPL) corresponds to the specific case of p = 1 , while Lemma 1 is applicable for any p ∈ [0 , 1] . Notably, the (1 -p ) rate in the T 1 -p term of Inequality 2 is tight for every p . In fact, this rate is tight even for x t = 1 ∈ R , ∀ t and X 0 = 1 ∈ R since, under these conditions, ∥ x t ∥ 2 X -1 t -1 = 1 /t and, consequently, ∑ T t =1 ( 1 ∧ ∥ x t ∥ 2 X -1 t -1 ) p = ∑ T t =1 t -p , yielding a rate of T 1 -p . This additional flexibility gained by allowing a general p ∈ [0 , 1] (with the original EPL corresponding to p = 1 ) helps us to accommodate slower convergence rates when learning the mean context from observed noisy contexts, as formalized in Assumption 1.

A crucial distinction between Lemma 1 and the original EPL lies in the definition of the noisy data matrix ˜ X t in Equation 1, which permits noise. However, the measured context vector x t does not have noise. This is beneficial in scenarios where a learner observes noisy contexts but seeks to establish an upper bound on the prediction error based on the underlying noise-free context or the mean context. Such situations are not rare in real applications; our problem of contextual bandits with post-serving contexts is precisely one of such case - while choosing an arm, we can estimate the mean post-serving context conditioned on the observable pre-serving context but are only able to observe the noisy realization of post-serving contexts after acting.

Other generalized variants of the EPL have been recently proposed and found to be useful in different contexts. For instance, Carpentier et al. (2020) extends the EPL to allow for the X -p t -norm, as opposed to the X -1 t -norm, while Hamidi and Bayati (2022) explores a generalized form of the 1 ∧∥ φ ( x t ) ∥ 2 X -1 t -1 term, which is motivated by variance reduction in non-Gaussian linear regression models. Nevertheless, to the best of our knowledge, our generalized version is novel and has not been identified in prior works.

Proof Sketche of Lemma 1. The formal proof of this lemma is involved and deferred to Appendix B.1. At a high level, our proof follows procedure for proving the original EPL. However, to accommodate the noises in the data matrix, we have to introduce new matrix concentration tools to the original (primarily algebraic) proof, and also identify the right conditions for the argument to go through. A key lemma to our proof is a high probability bound regarding the constructed noisy data matrix ˜ X t (Lemma 2 in Appendix B.1) that we derive based on Bernstein's Inequality for matrices under spectral norm (Tropp et al., 2015). We prove that, under mild assumptions on the noise, ∥ x t ∥ 2 ˜ X -1 t -1 ≤ ∥ x t ∥ 2 X -1 t -1 with high probability for any t . Next, we have to apply the union bound and this lemma to show that the above matrix inequality holds for every t ≥ 1 with high probability. Unfortunately, this turns out to not be true because when t is very small (e.g., t = 1 ), the above inequality cannot hold with high probability. Therefore, we have to use the union bound in a carefully tailored way by excluding all t 's that are smaller than a certain threshold (chosen optimally by solving certain inequalities) and handling these terms with small t separately (which is the reason of the second O (log(1 /δ )) term in Inequality 2). Finally, we refine the analysis of sthe standard EPL by allowing the exponent p in

Algorithm 1 poLinUCB ( Linear UCB with post-serving contexts )

1: for t = 0 , 1 , . . . , T do

2: Receive the pre-serving context x t

3: Compute the optimistic parameters by maximizing the UCB objective

4: Play the arm a t and receive the realized post-serving context as z t and the real-valued reward

5: Compute ̂ w t,a using Equation 3 for each a ∈ A .

6: Compute the estimated post-serving context generating function ̂ ϕ t ( · ) using ERM.

7: Update confidence sets C ( w ) and C ( ϕ , x ) for each a based on Equations 6 and 5.

t ̂ t,a t ̂ t t 8: end for

(1 ∧ ∥ x t ∥ 2 ˜ X -1 t -1 ) p and derive an upper bound on the sum ∑ T t =1 (1 ∧ ∥ x t ∥ 2 ˜ X -1 t -1 ) p with high probability. These together yeilds a robustified and generalized version of EPL as in Lemma 1.

5 No Regret Learning in Linear Bandits with Post-Serving Contexts

5.1 The Main Algorithm

In the ensuing section, we introduce our algorithm, poLinUCB, designed to enhance linear contextual bandit learning through the incorporation of post-serving contexts and address the issue arose from the algorithm introduced in Section 3.2. The corresponding pseudo-code is delineated in Algorithm 1. Unlike the traditional LinUCB algorithm, which solely learns and sustains confidence sets for parameters (i.e., ̂ β a and ̂ θ a for each a ), our algorithm also simultaneously manages the same for the post-serving context generating function, ̂ ϕ ( · ) . Below, we expound on our methodology for parameter learning and confidence set construction.

Parameter learning . During each iteration t , we fit the function ̂ ϕ t ( · ) and the parameters { ̂ θ t,a } a ∈A and { ̂ β t,a } a ∈A . To fit ̂ ϕ t ( · ) , resort to the conventional empirical risk minimization (ERM) framework. As for { ̂ θ t,a } a ∈A and { ̂ β t,a } a ∈A , we solve the following least squared problem for each arm a ,

For convenience, we use w and u to denote ( θ , β ) and ( x , z ) respectively. The closed-form solutions to ̂ θ t,a and ̂ β t,a for each arm a ∈ A are

Confidence set construction. At iteration t , we construct the confidence set for ̂ ϕ t ( x t ) by

Similarly, we can construct the confidence set for the parameters ̂ w t,a for each arm a ∈ A by

where ζ t,a = 2 √ λ + R η √ d u log ((1 + n t ( a ) L 2 u /λ ) /δ ) and n t ( a ) = ∑ t s =1 1 [ a s = a ] . Additionally, we further define ζ t := max a ∈A ζ t,a . By the assumption 1 and Lemma 3, we have the followings hold with probability at least 1 -δ for each of the following events,

5.2 Regret Analysis

In the forthcoming section, we establish the regret bound. Our proof is predicated upon the conventional proof of LinUCB (Li et al., 2010) in conjunction with our robust elliptical potential lemma. The pseudo-regret (Audibert et al., 2009) within this partial contextual bandit problem is defined as,

in which we reload the notation of reward by ignoring the noise,

It is crucial to note that our definition of the optimal action, a ⋆ t , in Eq. 9 depends on ϕ ⋆ ( x t ) as opposed to z t . This dependency ensures a more pragmatic benchmark, as otherwise, the noise present in z would invariably lead to a linear regret, regardless of the algorithm implemented. In the ensuing section, we present our principal theoretical outcomes, which provide an upper bound on the regret of our poLinUCB algorithm.

Theorem 1 (Regret of poLinUCB) . The regret of poLinUCB in Algorithm 1 is upper bounded by ˜ O ( T 1 -α d α u + d u √ TK ) with probability at least 1 -δ , if T = Ω(log(1 /δ )) .

The first term in the bound is implicated by learning the function ϕ ⋆ ( · ) . Conversely, the second term resembles the one derived in conventional contextual linear bandits, with the exception that our dependency on d u is linear. This linear dependency is a direct consequence of our generalized robust elliptical potential lemma. The proof is deferred in Appendix B.2.

6 Generalizations

So far we have focused on a basic linear bandit setup with post-serving features. Our results and analysis can be easily generalized to other variants of linear bandits, including those with feature mappings, and below we highlight some of these generalizations. They use similar proof ideas, up to some technical modifications; we thus defer all their formal proofs to Appendix B.3.

6.1 Generalization to Action-Dependent Contexts

Our basic setup in Section 3 has a single context x t at any time step t . This can be generalized to action-dependent contexts settings as studied in previous works (e.g., Li et al. (2010)). That is, during each iteration indexed by t , the learning algorithm observes a context x t,a for each individual arm a ∈ A . Upon executing the action of pulling arm a t , the corresponding post-serving context z t,a t is subsequently revealed. Notwithstanding, the post-serving context for all alternative arms remains unobserved. The entire procedure is the same as that of Section 3.

In extending this framework, we persist in our assumption that for each arm a ∈ A , there exists a specific function ϕ ⋆ a ( · ) : R d x → R d z that generates the post-serving context z upon receiving x associated with arm a ∈ A . The primary deviation from our preliminary setup lies in the fact that we now require the function ϕ ⋆ a ( · ) to be learned for each arm independently. The reward is generated as

The following proposition shows our regret bound for this action-dependent context case. Its proof largely draws upon the proof idea of Theorem 1 and also relies on the generalized EPL Lemma 1.

Proposition 2. The regret of poLinUCB in Algorithm 1 for action-dependent contexts is upper bounded by ˜ O ( T 1 -α d α u √ K + d u √ TK ) with probability at least 1 -δ if T = Ω(log(1 /δ )) .

The main difference with the bound in Theorem 1 is the additional √ K appeared in the first term, which is caused by learning multiple ϕ ⋆ a ( · ) functions with a ∈ A .

̂

0

1000

2000

3000

4000

5000

Time steps

0

1

2

3

4

5

Cumulative Regret

1e5

ϕ

(

x

) =

W

⊤ (

x

2 ) +

b

0

200

400

600

800

1000

Time steps

0.0

0.2

0.4

0.6

0.8

1.0

1e6

ϕ

(

x

) =

W

⊤

(

x

) +

b

0

200

400

600

800

1000

Time steps

0.0

0.5

1.0

1.5

2.0

1e6

ϕ

(

x

) =

W

⊤

x

+

b

Random

LinUCB (

x

only)

LinUCB (

x

and

z

)

LinUCB (

ϕ

)

poLinUCB (ours)

6.2 Generalization to Linear Stochastic Bandits

Another variant of linear bandits is the linear stochastic bandits setup (see, e.g., (Abbasi-Yadkori et al., 2011)). This model allows infinitely many arms, which consists of a decision set D t ⊆ R d at time t , and the learner picks an action x t ∈ D t . This setup naturally generalizes to our problem with post-serving contexts. That is, at iteration t , the learner selects an arm x t ∈ D t first, receives reward r t, x t , and then observe the post-serving feature z t conditioned on x t . Similarly, we assume the existence of a mapping ϕ ⋆ ( x t ) = E [ z t | x z ] that satisfies the Assumption 1. Consequently, the realized reward is generated as follows where θ ∗ , β ∗ are unknown parameters:

Therefore, the learner needs to estimate the linear parameters ̂ θ and ̂ β , as well as the function ̂ ϕ ( · ) . We obtain the following proposition for the this setup.

Proposition 3. The regret of poLinUCB in Algorithm 1 for the above setting is upper bounded by ˜ O ( T 1 -α d α u + d u √ T ) with probability at least 1 -δ if T = Ω(log(1 /δ )) .

6.3 Generalization to Linear Bandits with Feature Mappings

Finally, we briefly remark that while we have so far assumed that the arm parameters are directly linear in the context x t , z t , just like classic linear bandits our analysis can be easily generalized to accommodate feature mapping π x ( x t ) and π z ( z t ) = π z ( ϕ ( x t ) + ε t ) . Specifically, if the reward generation process is r a = ⟨ θ ⋆ a , π x ( x t ) ⟩ + ⟨ β ⋆ a , π z ( z t ) ⟩ + η t instead, then we can simply view ˜ x t = π x ( x t ) and ˜ z t = π z ( z t ) as the new features, with ˜ ϕ ( x t ) = E ϵ t [ π z ( ϕ ( x t ) + ϵ t )] . By working with ˜ x t , ˜ z t , ˜ ϕ , we shall obtain the same guarantees as Theorem 1.

7 Experiments

This section presents a comprehensive evaluation of our proposed poLinUCB algorithm on both synthetic and real-world data, demonstrating its effectiveness in incorporating follow-up information and outperforming the LinUCB( ̂ ϕ ) variant. More empirical results can be found in Appendix D.1.

7.1 Synthetic Data with Ground Truth Models

Evaluation Setup. We adopt three different synthetic environments that are representative of a range of mappings from the pre-serving context to the post-serving context: polynomial, periodicical and linear functions. The pre-serving contexts are sampled from a uniform noise in the range [ -10 , 10] d x , and Gaussian noise is employed for both the post-serving contexts and the rewards. In each environment, the dimensions of the pre-serving context ( d x ) and the post-serving context ( d z ) are of 100 and 5 , respectively with 10 arms ( K ). The evaluation spans T = 1000 or 5000 time steps, and each experiment is repeated with 10 different seeds. The cumulative regret for each policy in each environment is then calculated to provide a compararison.

Results and Discussion. Our experimental results, which are presented graphically in Figures 1, provide strong evidence of the superiority of our proposed poLinUCB algorithm. Across all setups, we

observe that the LinUCB ( x and z ) strategy, which has access to the post-serving context during arm selection, consistently delivers the best performance, thus serving as the upper bound for comparison. On the other hand, the Random policy, which does not exploit any environment information, performs the worst, serving as the lower bound. Our proposed poLinUCB (ours) outperforms all the other strategies, including the LinUCB ( ˆ ϕ ) variant, in all three setups, showcasing its effectiveness in adaptively handling various mappings from the pre-serving context to the post-serving context. Importantly, poLinUCB delivers significantly superior performance to LinUCB ( x only), which operates solely based on the pre-serving context.

7.2 Real World Data without Ground Truth

Evaluation Setup. The evaluation was conducted on a real-world dataset, MovieLens (Harper and Konstan, 2015), where the task is to recommend movies (arms) to a incoming user (context). Following Yao et al. (2023), we first map both movies and users to 32-dimensional real vectors using a neural network trained for predicting the rating. Initially, K = 5 movies were randomly sampled to serve as our arms and were held fixed throughout the experiment. The user feature vectors were divided into two parts serving as the pre-serving context ( d x = 25 ) and the post-serving context ( d z = 7 ). We fit the function ϕ ( x ) using a two-layer neural network with 64 hidden units and ReLU activation. The network was trained using the Adam optimizer with a learning rate of 1e-3. At each iteration, we randomly sampled a user from the dataset and exposed only the pre-serving context x to our algorithm. The reward was computed as the dot product of the user's feature vector and the selected movie's feature vector and was revealed post the movie selection. The evaluation spanned T = 500 iterations and repeated with 10 seeds. 500 MovieLens

Results and Discussion. The experimental results, presented in Figure 2, demonstrate the effectiveness of our proposed algorithm. The overall pattern is similar to it observed in our synthetic experiments. Our proposed policy consistently outperforms the other strategies (except for LinUCB with both pre-serving and post-serving features). Significantly, our algorithm yields superior performance compared to policies operating solely on the pre-serving context, thereby demonstrating its effectiveness in leveraging the post-serving information.

8 Conclusions and Limitations

Conlusions. In this work, we have introduced a novel contextual bandit framework that incorporates post-serving contexts, thereby widening the range of complex real-world challenges it can address. By leveraging historical data, our proposed algorithm, poLinUCB, estimates the functional mapping from pre-serving to post-serving contexts, leading to improved online learning efficiency. For the purpose of theoretical analysis, the elliptical potential lemma has been expanded to manage noise within post-serving contexts, a development which may have wider applicability beyond this particular framework. Extensive empirical tests on synthetic and real-world datasets have demonstrated the significant benefits of utilizing post-serving contexts and the superior performance of our algorithm compared to state-of-the-art approaches.

Limitations. Our theoretical analysis hinges on a crucial assumption that the function ϕ ⋆ ( · ) is learnable, which may not always be satisfied. This is particularly a concern when the post-serving contexts may hold additional information that cannot be deduced from the pre-serving context, irrespective of the amount of data collected. In such scenarios, the function mapping from the preserving context to the post-serving context may be much more difficult to learn, or even not learnable. Consequently, a linear regret may be inevitable due to model misspecification. However, from a practical point of view, our empirical findings from the real-world MovieLens dataset demonstrate that modeling the functional relationship between the pre-serving and post-serving contexts can still significantly enhance the learning efficiency. We hope our approaches can inform the design of practical algorithms that more effectively utilizes post-serving data in recommendations.

Acknowledgement. This work is supported by an NSF Award CCF-2303372, an Army Research Office Award W911NF-23-1-0030, and an Office of Naval Research Award N00014-23-1-2802.

̂

Time steps

0

100

200

300

400

500

0

100

200

300

400

Cumulative Regret

Random

LinUCB (

x

only)

LinUCB (

x

and

z

)

LinUCB (

ϕ

)

poLinUCB (ours)

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Naoki Abe, Alan W Biermann, and Philip M Long. Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica , 37:263-293, 2003.

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19):1876-1902, 2009.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Alexandra Carpentier, Claire Vernade, and Yasin Abbasi-Yadkori. The elliptical potential lemma revisited. arXiv preprint arXiv:2010.10182 , 2020.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 2008.

Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems , 23, 2010.

Nima Hamidi and Mohsen Bayati. The elliptical potential lemma for general distributions with an application to linear thompson sampling. Operations Research , 2022.

F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) , 5(4):1-19, 2015.

Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems , 30, 2017.

Tze Leung Lai and Ching Zong Wei. Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. The Annals of Statistics , 10(1): 154-166, 1982.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Yingkai Li, Yining Wang, and Yuan Zhou. Nearly minimax-optimal regret for linearly parameterized bandits. In Conference on Learning Theory , pages 2173-2174. PMLR, 2019.

Tyler Lu, Dávid Pál, and Martin Pál. Contextual multi-armed bandits. In Proceedings of the Thirteenth international conference on Artificial Intelligence and Statistics , pages 485-492. JMLR Workshop and Conference Proceedings, 2010.

Alessandro Nuara, Francesco Trovo, Nicola Gatti, and Marcello Restelli. A combinatorial-bandit algorithm for the online joint bid/budget optimization of pay-per-click advertising campaigns. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Hongju Park and Mohamad Kazem Shirani Faradonbeh. Analysis of thompson sampling for partially observable contextual multi-armed bandits. IEEE Control Systems Letters , 6:2150-2155, 2021.

Yi Qi, Qingyun Wu, Hongning Wang, Jie Tang, and Maosong Sun. Bandit learning with implicit feedback. Advances in Neural Information Processing Systems , 31, 2018.

Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411, 2010.

Eric M Schwartz, Eric T Bradlow, and Peter S Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science , 36(4):500-522, 2017.

Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications , pages 495-517, 2017.

Ryan Tibshirani. Nonparametric regression (and classification). 2017.

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

André Uschmajew. Local convergence of the alternating least squares algorithm for canonical tensor approximation. SIAM Journal on Matrix Analysis and Applications , 33(2):639-652, 2012.

Huazheng Wang, Qingyun Wu, and Hongning Wang. Learning hidden features for contextual bandits. In Proceedings of the 25th ACM international on conference on information and knowledge management , pages 1633-1642, 2016.

Huazheng Wang, Haifeng Xu, and Hongning Wang. When are linear stochastic bandits attackable? In International Conference on Machine Learning , pages 23254-23273. PMLR, 2022.

Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 529-538, 2016.

Jianyi Yang and Shaolei Ren. Robust bandit learning with imperfect context. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 10594-10602, 2021.

Shangdong Yang, Hao Wang, Chenyu Zhang, and Yang Gao. Contextual bandits with hidden features to online recommendation via sparse interactions. IEEE Intelligent Systems , 35(5):62-72, 2020.

Yun Yang and David B Dunson. Bayesian manifold regression. 2016.

Fan Yao, Chuanhao Li, Denis Nekipelov, Hongning Wang, and Haifeng Xu. How bad is topk recommendation under competing content creators? International Conference on Machine Learning , 2023.

Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502. PMLR, 2020.

Rong Zhu and Branislav Kveton. Robust contextual linear bandits. arXiv preprint arXiv:2210.14483 , 2022.

A Algorithm and Regret Analysis of LinUCB( ̂ ϕ )

We present the details of the algorithm described in Section 3.2 and the proof of the regret bound.

A.1 Main Algorithm

Parameter learning. We consider solving the following regularized least squared problem for estimating { ̂ θ t,a } a ∈A and { ̂ β t,a } a ∈A for each arm a :

where λ ≥ 0 are penalty factors ensuring the uniqueness of minimizers ̂ θ t,a and ̂ β t,a .

In the same convention, we use w to denote ( θ , β ) , and u to denote ( x , ̂ ϕ ( x ) ) . The closed-form solutions for ̂ θ t,a and ̂ β t,a in this least squared problem then become:

where we reload the notations of A t,a and b t,a ,

Confidence set construction. At iteration t , we construct the confidence set for ̂ ϕ t ( x t ) by

The construction of the confidence set for ̂ w t,a will be different, as we are using the predicted value ̂ ϕ t ( · ) for linear regression. Consider the following,

Therefore, the confidence set will be enlarged due to the error introduced by ̂ ϕ t ( · ) . In the below, we derive the confidence set. To build the confidence set, we need to bound

Since both { ϵ s } t s =1 and { η s } t s =1 are i.i.d sub-Gaussian random variables, respectively, we can use the self-normalized inequality to bound the corresponding terms, i.e., the followings hold with probability at least 1 -2 δ ,

Algorithm 2 LinUCB-( ̂ ϕ ) ( Linear UCB adapted from Wang et al. (2016) with post-serving contexts; The differences with Algorithm 1 are highlighted in blue color. )

1: for t = 0 , 1 , . . . , T do

2: Receive the pre-serving context x t

3: Compute the optimistic parameters by maximizing the UCB objective

4: Play the arm a t and receive the realized post-serving context as z t and the real-valued reward

5: Compute the estimated post-serving context generating function ̂ ϕ t ( · ) using ERM.

6: Compute ̂ w t,a by solving Equation 10 for each a .

7: Update confidence sets C t ( ̂ w t,a ) and C t ( ̂ ϕ t , x t ) for each a based on Equations 12 and 5. 8: end for

Therefore, the confidence set is

where ζ t,a = √ 2( L 2 ϵ + R 2 η ) log ((1 + n t ( a ) L 2 u /λ ) / ( δ/ 2)) + L u ( ∑ t s =1 e δ/t s ) / λ + 2 λ . In comparison to the original confidence set, there is one additional term due to the generalization error introduced from ̂ ϕ s ( · ) . In the next section, we will provide a regret analysis, which following from the proof of LinUCB (Li et al., 2010). We simply have the following

In the next, we expand the term ζ 2 T ,

In the next, we bound the second term in the above equation under the learnability assumption 1,

Therefore, naively choosing the value of λ will lead to a linear regret due to the term T 3 / 2 -α in the equation. To minimize the upper bound, we can choose the value of λ to be

Then, we can bound ζ 2 T by

By plugging it in and following the simplication as in the proof of Theorem 1, we can get the regret is upper bounded by

The above result is summarized as the following proposition.

Proposition 1 (Regret of LinUCB-( ̂ ϕ )) . The regret of LinUCB-( ̂ ϕ ) in Algorithm 2 is upper bounded by ˜ O ( T 1 -α d α u + T 1 -α/ 2 √ Kd 1+ α u ) with probability at least 1 -δ , by carefully setting the regularization coefficient λ = Θ( L u d α u T 1 -α log ( T/δ )) in Equation 1.

B Missing Proofs

B.1 Missing Proofs in the Generalized Elliptical Potential Lemma

Lemma 1 (Generalized Elliptical Potential Lemma) . Suppose (1) X 0 ∈ R d × d is any positive definite matrix; (2) x 1 , . . . , x T ∈ R d is a sequence of vectors with bounded l 2 norm max t ∥ x t ∥ ≤ L x ; (3) ϵ 1 , . . . , ϵ T ∈ R d is a sequence of independent (not necessarily identical) bounded zero-mean noises satisfying max t ∥ ϵ t ∥ ≤ L ϵ and E [ ϵ t ϵ ⊤ t ] ≽ σ 2 ϵ I for any t ; and (4) ˜ X t is defined as follows:

Then, for any p ∈ [0 , 1] , the following inequality holds with probability at least 1 -δ ,

Proof. Our proof follows the high level idea for proving the original EPL. However, to accommodate the noises in the data matrix, we have to introduce new matrix concentration tools to the original (primarily algebraic) proof, and also identify the right conditions for the argument to go through. A key lemma to our proof is the following high probability bound regarding the noisy data matrix:

Lemma 2. Let x 1 , ..., x T ∈ R d be a fixed sequence of vectors, and ϵ 1 , ..., ϵ T ∈ R d are independent random variables satisfying max t ∥ x t ∥ 2 ≤ L x , max t ∥ ϵ t ∥ 2 ≤ L ϵ , and E [ ϵ t ϵ ⊤ t ] ≽ σ 2 ϵ I . Then the following hold with probability at least 1 -2 d exp ( -Tσ 4 ϵ 8 L 2 ϵ ( L ϵ + L x ) 2 ) ,

The Proof of Lemma 2 employs the Bernstein's Inequality for matrices (Tropp et al., 2015), which is technical; for ease of presentation, we defer its proof of Appendix B.1. By Lemma 2, we have that, for every t ∈ [ T ] , the following inequality holds with probability at least 1 -2 d exp ( -tσ 4 ϵ 8 L 2 ϵ ( L ϵ + L x ) 2 ) :

under which we have

To prove our Lemma 1, we need to apply union bound to guarantee the above hold simultaneously for every t ≥ 1 with high probability. Unfortunately, this turns out to not be true because when t is very small (e.g., t = 1 ), the above inequality cannot hold with high probability. Therefore, to obtain high-probability guarantee by the union bound, we will have to exclude these small t 's and apply the union bound for only the events from some t ′ ∈ [ T ] , as follows

where the last inequality uses the fact that σ 4 ϵ / ( L 2 ϵ ( L ϵ + L x ) 2 ) ≤ ( σ ϵ /L ϵ ) 4 ≤ 1 and 1 -e -x ≥ x/ 2 for any x ∈ [0 , 1] . By solving the following inequality,

we have,

Let T 0 denote the ceiling of the right-hand-side of the above term. Therefore, we have the following hold with high probability at least 1 -δ :

In the next, we are going to bound the second term in the above equation, whose proof can be adapted from the proof of the original elliptical potential lemma. Using the fact that for any z ∈ [0 , + ∞ ] , z ∧ 1 ≤ 2 ln(1 + z ) , we get

Additionally, by definition, we have

This implies the following relationship between the determinant,

Since the only eigenvalues of a matrix of the form I + yy ⊤ are 1 + ∥ y ∥ 2 and 1 , we have

By taking the power p for both sides and taking the sum, we have

Since p ∈ [0 , 1] , the function g ( x ) = x p is a concave function. Thus, we have

Therefore, we can conclude that

By combining the above results, we have the following hold with probability at least 1 -δ :

Invoking

we obtained the desired inequality with probability at least 1 -δ :

Lemma 2. Let x 1 , ..., x T ∈ R d be a fixed sequence of vectors, and ϵ 1 , ..., ϵ T ∈ R d are independent random variables satisfying max t ∥ x t ∥ 2 ≤ L x , max t ∥ ϵ t ∥ 2 ≤ L ϵ , and E [ ϵ t ϵ ⊤ t ] ≽ σ 2 ϵ I . Then the following hold with probability at least 1 -2 d exp ( -Tσ 4 ϵ 8 L 2 ϵ ( L ϵ + L x ) 2 ) ,

Proof. We will analyze the term-wise difference, denoted as

Moreover, since E [ ϵ t ] = 0 for any t ∈ [ T ] , the expectation of S t (over randomness of noise) can be lower bounded as

Since ∥ x t ∥ 2 ≤ L x and ∥ ϵ t ∥ 2 ≤ L ϵ , we know that S t ≼ (2 L ϵ L x + L 2 ϵ ) I with probability 1 . Thus S t is uniformly upper bounded under the spectral-norm denoted by ∥ · ∥ , or formally

Consider the 'centered' matrix sum Z T = ∑ T t =1 [ S t -E [ ϵ t ϵ ⊤ t ] ] , with mean 0 . Since the spectralnorm of S t is upper bounded by 2 L ϵ L x + L 2 ϵ , its variance V ( S t ) = ∥ ∥ E [ S t S ⊤ t ]∥ ∥ 2 is upper bounded by (2 L ϵ L x + L 2 ϵ ) 2 . Thus, the variance of Z T equals the variance of sum ∑ T t =1 S t , which is then upper bounded by T (2 L ϵ L x + L 2 ϵ ) 2 . By the Bernstein's Inequality for random matrices (Tropp et al., 2015), we have the following high probability upper bound for the spectral norm ∥ · ∥ of Z T :

Since Z T is a symmetric matrix, its spectral norm upper bounds the absolute value of any eigenvalue. Thus we can lower bound the smallest eigenvalues as follows:

where the second inequality is due to two facts: (1) A ≽ B implies λ min ( A ) ≥ λ min ( B ) where A = ( ∑ T t =1 S t -σ 2 ϵ I ) and B = ( ∑ T t =1 S t -E [ ϵ t ϵ ⊤ t ] ) ; and (2) the event λ min ( A ) ≤ -ι thus is included within the event λ min ( B ) ≤ -ι . Consequently, we have

or equivalently,

By choosing the value of ι = Tσ 2 ϵ , we get

Using the fact that σ ϵ ≤ L ϵ , we can further simplify the above equation by

B.2 Missing Proofs in the Regret Analysis

Theorem 1 (Regret of poLinUCB) . The regret of poLinUCB in Algorithm 1 is upper bounded by ˜ O ( T 1 -α d α u + d u √ TK ) with probability at least 1 -δ , if T = Ω(log(1 /δ )) .

Proof. In the next, we prove the regret bound. For each time step t , the immediate regret is

where the inequality (a) is due to the definition of UCB, and (b) and (c) are obtained using the Cauchy-Schwarz inequality. Therefore, the cumulative regret can be further upper bounded by

In the next, we bound each term separately. Firstly, we have the following hold with probability at least 1 -δ by using the union bound,

In the next, to bound the remaining term, we use the result from Lemma 1. We first group the sums based the arm,

By denoting n T ( a ) as the number of times that arm a is pulled, we can divide the arms into two groups,

Then, we can further decompose the r.h.s term of Equation 16 based on if the corresponding arm is in G 0 or G 1 . Then, by applying Lemma 1, we have the following holds, with probability at least 1 -δ ,

where the last inequality is due to Lemma 1 and apply the union bound on the K arms. To bound the remainder term, since α ∈ [0 , 1 / 2] , by the learnability assumption as stated in Assumption 1 and Lemma 1, we have

Therefore, the total regret bound is bounded by the following term with probability at least 1 -2 δ ,

By hiding the logarithmic terms, we can further simplify it to be

B.3 Missing Proofs in Generalizations

Proposition 2. The regret of poLinUCB in Algorithm 1 for action-dependent contexts is upper bounded by ˜ O ( T 1 -α d α u √ K + d u √ TK ) with probability at least 1 -δ if T = Ω(log(1 /δ )) .

Proof. Our proof follows from the proof of Theorem 1. The immediate regret at each time step t is

Recall that the definition of a ⋆ t ,

To bound the immediate regret, we have

By the definition of UCB, we further have

By rearranging the terms, we get

Bounding the first term 1 is the same as the proof in Theorem 1, while bounding the second term 2 will be slightly different, as we now have K such functions of ϕ ⋆ a ( · ) for a ∈ A to learn. By denoting the error for each estimate of ϕ ⋆ a ( · ) at iteration t as e δ t,a . Therefore, the contribution from the second term to the total regret can be bounded by

Hence, by following the remaining steps in the proof of Theorem 1, we can conclude that the regret is upper bounded by

√

where the only difference is the additional K appeared in the first term.

Proposition 3. The regret of poLinUCB in Algorithm 1 for the above setting is upper bounded by ˜ O ( T 1 -α d α u + d u √ T ) with probability at least 1 -δ if T = Ω(log(1 /δ )) .

Proof. This proof also follows from the proof of Theorem 1. The immediate regret at each time step t is

Recall that the definition of x ⋆ t ,

To bound the immediate regret, we have

By the definition of UCB, we further have

By rearranging the terms, we get

Since we only need to fit a single θ ⋆ , β ⋆ and ϕ ⋆ ( · ) . We thus have the following bound for the total regret,

C Technical Lemmas

Lemma 3 ( Confidence Ellipsoid , based on Theorem 2 of (Abbasi-Yadkori et al., 2011)) . Let w ⋆ ∈ R d , V 0 = λ I , λ > 0 . For any t ≥ 0 , let u 1 , · · · , u t ∈ R d , define r t = ⟨ u t , w ⋆ ⟩ + η t where η t is R η -sub-Gaussian and assume that ∥ w ⋆ ∥ 2 ≤ L w ; let V t = V 0 + ∑ t s =1 u s u ⊤ s and ˆ w t be the corresponding regularised least-square estimator. Then, for any δ > 0 and t ≥ 0 , with probability at least 1 -δ , w ⋆ lies in the set:

Furthermore, if for all t ≥ 1 , ∥ u t ∥ ≤ L u , then for any δ > 0 and t ≥ 0 , with probability at least 1 -δ , w ⋆ lies in the set:

Lemma 4 (Bernstein's Inequality for Matrices, Theorem 6.1.1 of (Tropp et al., 2015)) . Let X 1 , · · · , X n ∈ R d 1 × d 2 be independent and centered random matrices. Assume that for each i ∈ [ n ] , X i is uniformly bounded , that is:

where ∥ · ∥ denotes the spectral-norm distance here. Introduce the sum

and let V ( Z ) denote the matrix variance statistics of the sum Z :

where the asterisk ∗ denotes the conjugate transpose operation. Then, for every ϵ ≥ 0 , we have,

100

ConvergenceRate

10-1

Linear (α = 0.5): T-0.5

Holder (β = 0.25, d = 2): T+d

28

Holder (β = 0.5, d = 3): T-z

23

Holder (β =0.75,d = 3）: T-

Holder (β=1,d= 3）:T-

100

101

102

103

Number of Examples (T)

0

1000

2000

3000

4000

5000

0.0

0.2

0.4

0.6

0.8

1.0

Cumulative Regret

1e6

(

x

) =

W

(

x

2 ) +

b

Random

LinUCB (

)

poLinUCB (ours)

LinUCB (

x

and

z

)

LinUCB (

x

only)

0

200

400

600

800

1000

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Cumulative Regret

1e6

(

x

) =

W

(

x

) +

b

Random

LinUCB (

)

poLinUCB (ours)

LinUCB (

x

and

z

)

LinUCB (

x

only)

Time steps

Time steps

D Additional Results and Discussions

D.1 Additional Experiments

We further investigate the case where the true reward is ⟨ ϕ ⋆ ( x ) , β ⟩ . For this case, the reward's dependency on the post-serving context is noiseless. The results are presented in Figure 4. We observe that the algorithm adapted from Wang et al. (2016) still performs worse than our method, though the gap seems become smaller.

D.2 Additional Discussions on Assumption 1

Our regret analysis can accommodate different values of α in Assumption 1. The rate of 1 / √ T (when α = 0 . 5 ) is a commonly observed rate for many classical machine learning algorithms, including linear regression, logistic regression, and SVM with a linear kernel. This rate is rooted in the law of large numbers and the central limit theorem. For smaller α values, the generalization error will converge more slowly than 1 / √ T , indicating that the learning problem becomes increasingly difficult. In the extreme case when α = 0 , ϕ ⋆ function cannot be learned accurately, thus we will inevitably suffer a linear regret (simply due to model misspecification).

For standard linear functions, α = 0 . 5 and our regret bound in such situations is tight w.r.t. α . However, we intentionally made assumption 1 to be more general in order to accommodate other much more complex ways of estimating the ϕ functions such as manifold regression (see, e.g., Yang and Dunson (2016)), non-parametric ways like k-nearest-neighbors to estimate phi, or to estimate complex non-smooth function (e.g., functions in Holder spaces). For example, when ϕ is a function in Holder space H ( β ) , then the learning rate is T -2 β/ (2 β + d ) , which is generally slower than T -0 . 5 and depends on β as well as the data dimension d (see, e.g., the note of Tibshirani (2017)). A visual illustration can be found in Figure 3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a variant of linear contextual bandits where a post-serving context is provided to the learner *after* the learner selects an arm. The reward is linear in the context and the post-serving context (but thus may not be linear in the original context on its own). The post-serving context is specified by noisy perturbation of a fixed (but unknown) function of the pre-specified context. The paper abstracts away the process of learning the function: the function is assumed to be learnable with sufficiently many pairs of context and noisy post-serving context pairs, and the “complexity” of the function is parameterized by the error of this learning process. \n\nThe paper designs an algorithm based on LinUCB that creates a confidence set for the post-serving context in addition to the typical confidence set for the reward function parameters. The regret bound has a $O(T^{1-\\alpha})$ term that depends on the complexity $\\alpha$ of the function and a $O(\\sqrt{T})$ term that depends on the dimension. The paper builds on the classical analysis of LinUCB, but requires a novel elliptical potential lemma. \n\nThe paper further complements the theoretical analysis with simulations on synthetic data and real-world data (MovieLens dataset).\n\nSTRENGTHS:\nThe paper provides an intriguing model for additional information available to the learner beyond the reward. The idea of a post-serving context is natural and well-motivated. The approach of abstracting away the complexity of learning the function mapping contexts to means of post-serving contexts is also clean and elegant. \n\nThe LinUCB-based algorithm in the paper is simple and elegant. The paper also provides a detailed discussion of shortcomings of relying on a point estimate of the post-serving contexts (in Section 2.2) and the need to appropriately address uncertainty in phi via a confidence set when computing an upper confidence-bound. \n\nThe analysis of the algorithm is also technically interesting, with an introduction of the novel elliptical potential lemma. The paper clearly describes how the novel elliptical potential lemma accommodates slower learning rates and allows for noisy contexts, and furthermore highlights how the lemma is tight in several regimes. \n\nThe paper is very well-written and explains the key ideas clearly.\n\nWEAKNESSES:\nWhile the paper focuses solely on upper bounds, it could be interesting to investigate lower bounds. Is the dependence on alpha in the regret bound tight? (See question below.) That being said, this is a fairly minor weakness, since the upper bound analysis is already involved and interesting. \n\nAnother minor weakness is that in Observation 1, it seems that the result is restricted to linear bandit algorithms (which should probably be clarified in the statement too). This seems to be a bit of an unfair comparison because the model is misspecified, and the linear bandit algorithm was not designed to handle misspecified rewards.\n\nQUESTIONS:\nQuestions: \n- Is the dependence on alpha in the regret bound tight?\n\nTypos:\n- “Generlaize” -> “generalize” p.5\n- “Sketche” -> “sketch” p.6\n- “Sthe” -> “the” p.6\n- “Followings” -> “following” p.7\n\nLIMITATIONS:\nYes, the authors adequately addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks into contextual bandits in the scenario where only partial context is provided to the learner for making decisions. Specifically, this paper comes up with an environment setting where the full context consists of pre and post components which are revealed to the learner before and after she finalizes the action to take. Given the true reward is tied to the full context, the algorithm should include a post-context inferring step to get low regret. For the linear case, this paper proposes a low regret algorithm poLinUCB for this pre-post contextual bandit setting. The performance of poLinUCB is proved with theorems and evaluated by simulations, where a smaller regret rate is shown in contrast to other method/baselines.\n\nSTRENGTHS:\n1. Both the theorem and simulation show in agreement that poLinUCB outperforms counter-method (Wang et al.) and baselines that are mentioned in the paper. \n2. The theorem and its consequential sharper regret rate in $T$ compared to Wang et al. is built upon the generalized EPL, which may be considered as a new tool for other independent interests that take the uncertainties of context into account.\n\nWEAKNESSES:\n**Soundness of the problem setup** Though the math content from problem formulation to algorithm to theorems are sound, it’s of independent evaluation that whether the setup at the first place is worth studying, put in other words, does the reward model (under line 107) cover/match any real-world problem. For me, some major drawbacks are: \n- it’s unlikely in reality that z_t will be fully revealed to learner \n- the learnability assumption fails when z_t is not fully determined by x_t, which is what usually happens in the hidden contextual setting where the hidden part is influenced by multiple factors, including but not limited to the revealed part x_t.\n- given this setting where z_t is fully predictable by x_t, then isn’t kernel/neural bandit already a good solution for this problem? \n\n**Method proposed is pretty similar to existing one** It’s in doubt that the outperformance is gained because poLinUCB has any essential algorithmic improvement than that of Wang at el. **or the formance gain is a only matter of new problem setting**. Wang at el. is not designed for the specific problem setting brought up by this work. \n- I can see that the original algorithm by Wang at el will suffer larger regret under your reward setup where true reward has component $<z_t, \\beta_a^*>$. Since $z_t$’s from previous rounds are accessible to learner, the most accurate way for estimating \\beta^* is certainly to utilize $z_t $rather than $\\hat \\phi (x_t)$ in loss. poLinUCB is more like an alternation of exsisting method as on adapting it to a new problem.\n- But what if  true reward turns to have $<\\phi^*(x_t), \\beta_a^*>$, which is like a twin problem setup of this paper’s, then maybe Wang at el. will outperforms poLinUCB given $z_t$ is a noisy sample of $\\phi^*(x_t)$ thus casts extra error into $\\hat \\beta_a$, while  $\\hat \\phi (x)$ is ERM, which should have some noise reduction effect strengthened as $t$ goes up. Upper bound of regret in Wang at el. is inferior doesn’t necessarily mean the algorithm is less optimal, the performance may be underestimated by an upper bound that is not tight.\n\nQUESTIONS:\n- What I’m not sure of is whether the problem setting is of good value to study, hopefully the author will respond with more evidence that may justify the problem setting.  \n- Is the dependency on $d_u = d_x +d_z $able to further optimize, given $z_t$ is predictable by $x_t$, the effective dimension of this problem maybe smaller than $d_u$.\n\nLIMITATIONS:\nIdentified Limitations are listed in Weakness and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel contextual bandit problem with post-serving contexts and introduces a new algorithm, poLinUCB, that achieves tight regret under standard assumptions. The authors demonstrate the effectiveness of their approach through both synthetic and real-world experiments, showing that poLinUCB consistently outperforms other strategies in leveraging post-serving information. Overall, the paper makes contributions to the field of online learning and contextual bandits.\n\nSTRENGTHS:\n- Introduces a new contextual bandit problem with post-serving contexts and proposes a novel algorithm, poLinUCB, that achieves tight regret under standard assumptions.\n- Demonstrates the effectiveness of the proposed approach through both synthetic and real-world experiments.\n- Makes significant contributions to the field of online learning and contextual bandits, and has the potential to improve efficiency in a wide range of applications.\n\nWEAKNESSES:\n- The paper does not seem to be fully finished. There is no conclusion section. \n- The paper could benefit from a more thorough discussion of the limitations and assumptions of the proposed approach, particularly in the context of real-world applications.\n- The experimental evaluation is limited. It could be expanded to include more datasets and scenarios.\n- The paper could provide more details on the implementation and computational complexity of the proposed algorithm.\n\nQUESTIONS:\n- Could the authors provide more details on the choice of hyperparameters and the sensitivity of the proposed algorithm to their values?\n- Could the authors provide more insights into the practical implications and potential impact of the proposed approach? \n- How does the proposed algorithm handle noisy or incomplete post-serving context information, and what are the limitations of the approach in such scenarios?\n\nLIMITATIONS:\nSee comments above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and analyzes the problem of contextual multi-armed bandits with post-serving contexts, where additional reward-relevant information is revealed after the the algorithm makes its choice. It divides the traditional context into a pre-serving context which is known the the algorithm at decision-time and a post-serving context which revealed later, but for which the algorithm can learn to produce an accurate estimate of its mean based on the pre-serving context. It then shows that the standard elliptical potential lemma needs updating in order to handle this case, and provides an alternative version. The paper then provides a novel algorithm which estimates the post-serving context in order to use in the new EPL, and proves its regret rate. Finally the paper performs experiments validating the performance of the poLinUCB algorithm.\n\nSTRENGTHS:\n1) Makes a novel contribution to a more general version of the elliptical potential lemma which is well-motivated by their chosen application problem\n2) Introduces a novel but useful-seeming problem setting where additional reward-relevant information is revealed after the algorithm's decision\n3) Experiments clearly demonstrate the performance improvements benefits of their new algorithm, which nearly matches the performance of LinUCB which\n\nWEAKNESSES:\n1) The paper doesn't have a conclusion section\n\nQUESTIONS:\n1) Could you spell out an example of the setting in more detail? The paper does a good job of explaining situations where there is a post-serving context, but isn't that explicit about what the reward function is where the reward is revealed immediately after the decision but the post-serving context is clearly helpful for predicting performance\n2) Could you say a bit more about why poLinUCB outperforms LinUCB even when $\\phi$ is a linear function? If $z = Ax$, then $r = \\theta x + \\beta z = \\theta x + \\beta Ax = (\\theta + \\beta A)x$, and it is equivalent to a slightly different linear contextual bandits problem.\n\nLIMITATIONS:\nThe authors did not discuss the limitations of this work. While they would be similar to the limitations of normal bandit problems, adding the post-serving context could plausibly be more impactful than (for instance) a 20% performance improvement on a standard bandit problem. In particular, it seems plausible that the post-serving context could allow one to use more complex reward functions, which could be good or bad.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considered a novel contextual bandit problem with post-serving contexts and designed a new algorithm, poLinUCB. With a generalized version of Elliptical Potential Lemma (EPL), they provided tight regret under standard assumptions. Empirically, they showed on synthetic and real-world datasets, the proposed algorithms outperformed LinUCB with different types of contexts.\n\nSTRENGTHS:\n- The paper is well-written and organised. Easy to read and well-defined notation. \n- The proposed new setting of contextual bandits is valid, interesting and well-motivated. The analysis of natural attempts fails provided good support evidence for the new proposed algorithm. \n- A generalised elliptical potential lemma (EPL) is proposed and well-explained. Can be an independent interest. \n- A regret bound is provided with parameters introduced in assumptions, with a clear description that how the parameter can be realised for the proposed algorithm. \n- The generalisations to action-dependent contexts, Linear Stochastic Bandits and Linear Bandits with Feature Mappings are interesting and useful extensions of the proposed setting. \n- Comparison to related algorithms in empirical is provided on both synthetic and real-world datasets.\n\nWEAKNESSES:\nThe regret analysis replies to Assumption 1 -- Generalised learnability of phi^ast, and the regret bound depends on the value of alpha. It is not clear when phi is not a linear function, how alpha will change. For example, when alpha approaches 0, the regret bound will approach linear regret.\n\nQUESTIONS:\n- Can you provide a detailed discussion of how Assumption 1 will be realised when phi is not a linear function: how alpha will change, how regret bound would be? \n- Can you test in synthetic experiments for more types phi functions, to verify the discussion above? \n- would the form the phi function influence the parameter learning in Eq. (3)? \n- what if phi function depends on contexts? e.g. for different users, different types of contexts can be observed before and after serving, and those contexts can have different relationships.\n\nLIMITATIONS:\nAs mentioned by the authors in the appendix, this work heavily relies on Assumption 1 and the relationship between pre- and post-serving contexts. In practice, those assumptions may not hold.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}