SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chaoqi Wang<sup>1</sup> Ziyu Ye<sup>1</sup> Zhe Feng<sup>2</sup> Ashwinkumar Badanidiyuru<sup>3</sup> Haifeng Xu<sup>1</sup>  
<sup>1</sup>University of Chicago<sup>1</sup> Google Research<sup>2</sup> Google<sup>3</sup> {chaoqi, ziyuye, haifengxu}@uchicago.edu  
{zhef, ashwinkumarbv}@google.com
Standard contextual bandit problem assumes that all the relevant contexts are observed before the algorithm chooses an arm. This modeling paradigm, while useful, often falls short when dealing with problems in which valuable additional context can be observed after arm selection. For example, content recommendation platforms like Youtube, Instagram, Tiktok also observe valuable follow-up information pertinent to the user's reward after recommendation (e.g., how long the user stayed, what is the user's watch speed, etc.). To improve online learning efficiency in these applications, we study a novel contextual bandit problem with post-serving contexts and design a new algorithm, poLinUCB, that achieves tight regret under standard assumptions. Core to our technical proof is a robustified and generalized version of the well-known Elliptical Potential Lemma (EPL), which can accommodate noise in data. Such robustification is necessary for tackling our problem, and we believe it could also be of general interest. Extensive empirical tests on both synthetic and real-world datasets demonstrate the significant benefit of utilizing post-serving contexts as well as the superior performance of our algorithm over the state-of-the-art approaches.
Contextual bandits represent a fundamental mathematical model that is employed across a variety of applications, such as personalized recommendations (Li et al., 2010; Wu et al., 2016) and online advertising (Schwartz et al., 2017; Nuara et al., 2018). In their conventional setup, at each round  $t$ , a learner observes the context  $x_{t}$ , selects an arm  $a_{t} \in \mathcal{A}$ , and subsequently, observes its associated reward  $r_{t,a_{t}}$ . Despite being a basic and influential framework, it may not always capture the complexity of real-world scenarios (Wang et al., 2016; Yang et al., 2020). Specifically, the learner often observes valuable follow-up information pertinent to the payoff post arm selection (henceforth, the post-serving context). Standard contextual bandits framework that neglects such post-serving contexts may result in significantly suboptimal performance due to model misspecification.
Consider an algorithm designed to recommend educational resources to a user by utilizing the user's partially completed coursework, interests, and proficiency as pre-serving context (exemplified in platforms such as Coursera). After completing the recommendation, the system can refine the user's profile by incorporating many post-serving context features such as course completion status, how much time spent on different educational resources, performances, etc. This transition naturally delineates a mapping from user attributes (i.e., the pre-serving context) to user's learning experiences and outcomes (i.e., the post-serving context). It is not difficult to see that similar scenarios happen in many other recommender system applications. For instance, in e-commerce platforms (e.g., Amazon, Etsy or any retailing website), the system will first recommend products based on the user's profile information, purchasing pattern and browsing history, etc.; post recommendations, the system can
then update these information by integrating post-serving contexts such as this recent purchase behaviour and product reviews. Similarly, media content recommendation platforms like Youtube, Instagram and Tiktok, also observe many post-serving features (e.g., how long the user stayed) that can refine the system's estimation about users' interaction behavior as well as the rewards.
A common salient point in all the aforementioned scenarios is that the post-serving context are prevalent in many recommender systems; moreover, despite being unseen during the recommendation/serving phase, they can be estimated from the pre-serving context given enough past data. More formally, we assume that there exists a learnable mapping  $\phi^{\star}(\cdot): \mathbb{R}^{d_x} \to \mathbb{R}^{d_z}$  that maps pre-serving feature  $\pmb{x} \in \mathbb{R}^{d_x}$  to the expectation of the post-serving feature  $\pmb{z} \in \mathbb{R}^{d_z}$ , i.e.,  $\mathbb{E}[z|\pmb{x}] = \phi^{\star}(\pmb{x})$ .
Unsurprisingly, and as we will also show, integrating the estimation of such post-serving features can significantly help to enhance the performance of contextual bandits. However, most of the existing contextual bandit algorithms, e.g., (Auer, 2002; Li et al., 2010; Chu et al., 2011; Agarwal et al., 2014; Tewari and Murphy, 2017), are not designed to accommodate the situations with post-serving contexts. We observe that directly applying these algorithms by ignoring post-serving contexts may lead to linear regret, whereas simple modification of these algorithms will also be sub-optimal. To address these shortcomings, this work introduces a novel algorithm, poLinUCB. Our algorithm leverages historical data to simultaneously estimate reward parameters and the functional mapping from the pre- to post-serving contexts so to optimize arm selection and achieves sublinear regret. En route to analyzing our algorithm, we also developed new technique tools that may be of independent interest.
Main Contributions.
- First, we introduce a new family of contextual linear bandit problems. In this framework, the decision-making process can effectively integrate post-serving contexts, premised on the assumption that the expectation of post-serving context as a function of the pre-serving context can be gradually learned from historical data. This new model allows us to develop more effective learning algorithms in many natural applications with post-serving contexts.
- Second, to study this new model, we developed a robustified and generalized version of the well-regarded elliptical potential lemma (EPL) in order to accommodate random noise in the post-serving contexts. While this generalized EPL is an instrumental tool in our algorithmic study, we believe it is also of independent interest due to the broad applicability of EPL in online learning.
- Third, building upon the generalized EPL, we design a new algorithm poLinUCB and prove that it enjoys a regret bound  $\widetilde{\mathcal{O}}(T^{1 - \alpha}d_u^\alpha + d_u\sqrt{TK})$ , where  $T$  denotes the time horizon and  $\alpha \in [0,1/2]$  is the learning speed of the pre- to post-context mapping function  $\phi^{\star}(\cdot)$ , whereas  $d_u = d_x + d_z$  and  $K$  denote the parameter dimension and number of arms. When  $\phi^{\star}(\cdot)$  is easy to learn, e.g.,  $\alpha = 1/2$ , the regret bound becomes  $\widetilde{\mathcal{O}}(\sqrt{Td_u} + d_u\sqrt{TK})$  and is tight. For general functions  $\phi^{\star}(\cdot)$  that satisfy  $\alpha \leq 1/2$ , this regret bound degrades gracefully as the function becomes more difficult to learn, i.e., as  $\alpha$  decreases.
- Lastly, we empirically validate our proposed algorithm through thorough numerical experiments on both simulated benchmarks and real-world datasets. The results demonstrate that our algorithm surpasses existing state-of-the-art solutions. Furthermore, they highlight the tangible benefits of incorporating the functional relationship between pre- and post-serving contexts into the model, thereby affirming the effectiveness of our modeling.
Contextual bandits. The literature on linear (contextual) bandits is extensive, with a rich body of works (Abe et al., 2003; Auer, 2002; Dani et al., 2008; Rusmevichientong and Tsitsiklis, 2010; Lu et al., 2010; Filippi et al., 2010; Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Li et al., 2017; Jun et al., 2017). One of the leading design paradigm is to employ upper confidence bounds as a means of balancing exploration and exploitation, leading to the attainment of minimax optimal regret bounds. The derivation of these regret bounds principally hinges on the utilization of confidence ellipsoids and the elliptical potential lemma. Almost all these works assume that the contextual information governing the payoff is fully observable. In contrast, our work focuses on scenarios where the context is not completely observable during arm selection, thereby presenting new challenges in addressing partially available information.
Contextual bandits with partial information. Contextual bandits with partial information has been relatively limited in the literature. Initial progress in this area was made by Wang et al. (2016), who studied settings with hidden contexts. In their setup there is some context (the post-serving context in our model) that can never be observed by the learner, whereas in our setup the learner can observe post-serving context but only after pulling the arm. Under the assumption that if the parameter initialization is extremely close to the true optimal parameter, then they develop a sub-linear regret algorithm. Our algorithm does not need such strong assumption on parameter initialization. Moreover, we show that their approach may perform poorly in our setup. Subsequent research by Qi et al. (2018); Yang et al. (2020); Park and Faradonbeh (2021); Yang and Ren (2021); Zhu and Kveton (2022) investigated scenarios with noisy or unobservable contexts. In these studies, the learning algorithm was designed to predict context information online through context history analysis, or selectively request context data from an external expert. Our work, on the other hand, introduces a novel problem setting that separates contexts into pre-serving and post-serving categories, enabling the exploration of a wide range of problems with varying learnability. Additionally, we also need to employ new techniques for analyzing our problem to get a near-optimal regret bound.
Generalizations of the elliptical potential lemma (EPL). The EPL, introduced in the seminal work of Lai and Wei (1982), is arguably a cornerstone in analyzing how fast stochastic uncertainty decreases with the observations of new sampled directions. Initially being employed in the analysis of stochastic linear regression, the EPL has since been extensively utilized in stochastic linear bandit problems (Auer, 2002; Dani et al., 2008; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Li et al., 2019; Zhou et al., 2020; Wang et al., 2022). Researchers have also proposed various generalizations of the EPL to accommodate diverse assumptions and problems. For example, Carpentier et al. (2020) extended the EPL by allowing for the use of the  $X_{t}^{-p}$ -norm, as opposed to the traditional  $X_{t}^{-1}$ -norm. Meanwhile, Hamidi and Bayati (2022) investigated a generalized form of the  $1 \wedge \|\varphi(\boldsymbol{x}_{t})\|_{X_{t-1}^{-1}}^{2}$  term, which was inspired by the pursuit of variance reduction in non-Gaussian linear regression models. However, existing (generalized) EPLs are inadequate for the analysis of our new problem setup. Towards that end, we develop a new generalization of the EPL in this work to accommodate noisy feature vectors.
3 Linear Bandits with Post-Serving Contexts
Basic setup. We hereby delineate a basic setup of linear contextual bandits within the scope of the partial information setting, whereas multiple generalizations of our framework can be found in Section 6. This setting involves a finite and discrete action space, represented as  $\mathcal{A} = [K]$ . Departing from the classic contextual bandit setup, the context in our model is bifurcated into two distinct components: the pre-serving context, denoted as  $\pmb{x} \in \mathbb{R}^{d_x}$ , and the post-serving context, signified as  $\pmb{z} \in \mathbb{R}^{d_z}$ . When it is clear from context, we sometimes refer to pre-serving context simply as context as in classic setup, but always retain the post-serving context notion to emphasize its difference. We will denote  $\pmb{X}_t = \sum_{s=1}^t \pmb{x}_s \pmb{x}_s^\top + \lambda \pmb{I}$  and  $\pmb{Z}_t = \sum_{s=1}^t \pmb{z}_s \pmb{z}_s^\top + \lambda \pmb{I}$ . For the sake of brevity, we employ  $\pmb{u} = (\pmb{x}, \pmb{z})$  to symbolize the stacked vector of  $\pmb{x}$  and  $\pmb{z}$ , with  $d_u = d_x + d_z$  and  $\| \pmb{u} \|_2 \leq L_u$ . The pre-serving context is available during arm selection, while the post-serving context is disclosed post the arm selection. For each arm  $a \in \mathcal{A}$ , the payoff,  $r_a(\pmb{x}, \pmb{z})$ , is delineated as follows:
$$
r _ {a} (\boldsymbol {x}, \boldsymbol {z}) = \boldsymbol {x} ^ {\top} \boldsymbol {\theta} _ {a} ^ {\star} + \boldsymbol {z} ^ {\top} \beta_ {a} ^ {\star} + \eta ,
$$
where  $\pmb{\theta}_{a}^{\star}$  and  $\beta_{a}^{\star}$  represent the parameters associated with the arm, unknown to the learner, whereas  $\eta$  is a random noise sampled from an  $R_{\eta}$ -sub-Gaussian distribution. We use  $\| \pmb{x}\|_p$  to denote the  $p$ -norm of a vector  $\pmb{x}$ , and  $\| \pmb{x}\|_{\pmb{A}} \coloneqq \sqrt{\pmb{x}^\top\pmb{A}\pmb{x}}$  is the matrix norm. For convenience, we assume  $\| \pmb{\theta}_{a}^{\star}\|_{2} \leq 1$  and  $\| \beta_{a}^{\star}\|_{2} \leq 1$  for all  $a \in \mathcal{A}$ . Additionally, we posit that the norm of the pre-serving and post-serving contexts satisfies  $\| \pmb{x}\|_{2} \leq L_{x}$  and  $\| \pmb{z}\|_{2} \leq L_{z}$ , respectively, and  $\max_{t \in [T]} \sup_{a,b \in \mathcal{A}}\langle \pmb{\theta}_{a}^{\star} - \pmb{\theta}_{b}^{\star}, \pmb{x}_{t} \rangle \leq 1$  and  $\max_{t \in [T]} \sup_{a,b \in \mathcal{A}}\langle \pmb{\beta}_{a}^{\star} - \pmb{\beta}_{b}^{\star}, \pmb{z}_{t} \rangle \leq 1$ , same as in (Lattimore and Szepesvári, 2020).
3.1 Problem Settings and Assumptions.
The learning process proceeds as follows at each time step  $t = 1,2,\dots ,T$  ..
1. The learner observes the context  $\boldsymbol{x}_t$ .
2. An arm  $a_{t}\in [K]$  is selected by the learner.
3. The learner observes the realized reward  $r_{t,a_t}$  and the post-serving context,  $\mathcal{Z}_t$
Without incorporating the post-serving context, one may incur linear regret as a result of model misspecification, as illustrated in the following observation. To see this, consider a setup with two arms,  $a_1$  and  $a_2$ , and a context  $x \in \mathbb{R}$  drawn uniformly from the set  $\{-3, -1, 1\}$  with  $\phi^{\star}(x) = x^{2}$ . The reward functions for the arms are noiseless and determined as  $r_{a_1}(x) = x + x^2 / 2$  and  $r_{a_2}(x) = -x - x^2 / 2$ . It can be observed that  $r_{a_1}(x) > r_{a_2}(x)$  when  $x \in \{-3, 1\}$  and  $r_{a_1}(x) < r_{a_2}(x)$  when  $x = -1$ . Any linear bandit algorithm that solely depends on the context  $x$  (ignoring  $\phi^{\star}(x)$ ) will inevitably suffer from linear regret, since it is impossible to have a linear function (i.e.,  $r(x) = \theta x$ ) that satisfies the above two inequalities simultaneously.
Observation 1. There exists linear bandit environments in which any online algorithm without using post-serving context information will have  $\Omega (T)$  regret.
Therefore, it is imperative that an effective learning algorithm must leverage the post-serving context, denoted as  $\boldsymbol{z}_t$ . As one might anticipate, in the absence of any relationship between  $\boldsymbol{z}_t$  and  $\boldsymbol{x}_t$ , it would be unfeasible to extrapolate any information regarding  $\boldsymbol{z}_t$  while deciding which arm to pull, a point at which only  $\boldsymbol{x}_t$  is known. Consequently, it is reasonable to hypothesize a correlation between  $\boldsymbol{z}_t$  and  $\boldsymbol{x}_t$ . This relationship is codified in the subsequent learnability assumption.
Specifically, we make the following natural assumption — there exists an algorithm that can learn the mean of the post-serving context  $\boldsymbol{z}_t$ , conditioned on  $\boldsymbol{x}_t$ . Our analysis will be general enough to accommodate different convergence rates of the learning algorithm, as one would naturally expect, the corresponding regret will degrade as this learning algorithm's convergence rate becomes worse. More specifically, we posit that, given the context  $\boldsymbol{x}_t$ , the post-serving context  $\boldsymbol{z}_t$  is generated as<sup>1</sup>
post-serving context generation process:  $\pmb{z}_t = \phi^{\star}(\pmb{x}_t) + \epsilon_t$ , i.e.,  $\phi^{\star}(\pmb{x}) = \mathbb{E}[z|\pmb{x}]$ .
Here,  $\epsilon_{t}$  is a zero-mean noise vector in  $\mathbb{R}^{d_z}$ , and  $\phi^{\star}:\mathbb{R}^{d_x}\to \mathbb{R}^{d_z}$  can be viewed as the post-serving context generating function, which is unknown to the learner. However, we assume  $\phi^{\star}$  is learnable in the following sense.
Assumption 1 (Generalized learnability of  $\phi^*$ ). There exists an algorithm that, given  $t$  pairs of examples  $\{(\pmb{x}_s,\pmb{z}_s)\}_{s = 1}^t$  with arbitrarily chosen  $\pmb{x}_s$ 's, outputs an estimated function of  $\phi^{\star}:\mathbb{R}^{d_x}\to \mathbb{R}^{d_z}$  such that for any  $\pmb {x}\in \mathbb{R}^{d_x}$ , the following holds with probability at least  $1 - \delta$ ,
$$
e _ {t} ^ {\delta} := \left\| \widehat {\phi} _ {t} (\boldsymbol {x}) - \phi^ {\star} (\boldsymbol {x}) \right\| _ {2} \leq C _ {0} \cdot \left(\left\| \boldsymbol {x} \right\| _ {\boldsymbol {X} _ {t} ^ {- 1}} ^ {2}\right) ^ {\alpha} \cdot \log (t / \delta),
$$
where  $\alpha \in (0,1 / 2]$  and  $C_0$  is some universal constant.
The aforementioned assumption encompasses a wide range of learning scenarios, each with different rates of convergence. Generally, the value of  $\alpha$  is directly proportional to the speed of learning; the larger the value of  $\alpha$ , the quicker the learning rate. Later, we will demonstrate that the regret of our algorithm is proportional to  $O(T^{1 - \alpha})$ , exhibiting a graceful degradation as  $\alpha$  decreases. The ensuing proposition demonstrates that for linear functions,  $\alpha = 1 / 2$ . This represents the best learning rate that can be accommodated<sup>2</sup>. In this scenario, the regret of our algorithm is  $O(\sqrt{T})$ , aligning with the situation devoid of post-serving contexts (Li et al., 2010; Abbasi-Yadkori et al., 2011).
Observation 2. Suppose  $\phi(\cdot)$  is a linear function, i.e.,  $\phi(\pmb{x}) = \Phi^\top \pmb{x}$  for some  $\Phi \in \mathbb{R}^{d_x \times d_z}$ , then  $e_t^\delta = \mathcal{O}\left(\| \pmb{x} \|_{\pmb{X}_t^{-1}} \cdot \log(t / \delta)\right)$ .
This observation follows from the following inequalities  $\| \phi_t(\pmb {x}) - \phi^{\star}(\pmb {x})\| = \| \widehat{\Phi}_t^\top \pmb {x} - \Phi^{\star^\top}\pmb {x}\| \leq$ $\| \widehat{\Phi}_t - \Phi^\star \|_{X_t}\cdot \| x\|_{X_t^{-1}} = \mathcal{O}(\| x\|_{X_t^{-1}}\cdot \log \left(\frac{t}{\delta}\right))$  , where the last equation is due to the confidence ellipsoid bound (Abbasi-Yadkori et al., 2011). We refer curious readers to Appendix D.2 for a discussion on other more challenging  $\phi (\cdot)$  functions with possibly worse learning rates  $\alpha$
<sup>1</sup>We remark that an alternative view of this reward generation is to treat it as a function of  $\pmb{x}$  as follows:  $r_a(\pmb{x}) = \pmb{x}^\top \pmb{\theta}_a^\star + \phi^\star(\pmb{x})^\top \beta_a^\star + \mathrm{noise}$ . Our algorithm can be viewed as a two-phase learning of this structured function, i.e., using  $(\pmb{x}_t, \pmb{z}_t)$  to learn  $\phi^\star$  that is shared among all arms and then using learned  $\hat{\phi}$  to estimate each arm  $a$ 's reward parameters. This more effectively utilizes data than approaches that directly learns the  $r_a(\pmb{x})$  function, as shown in our later experimental section (e.g., see Figure 1).
To be precise, our analysis can extend to instances where  $\alpha > 1/2$ . However, this would not enhance the regret bound since the regret is already  $\Omega(\sqrt{T})$ , even with knowledge of  $z$ . This would only further complicate our notation, and therefore, such situations are not explicitly examined in this paper.
3.2 Warm-up: Why Natural Attempts May Be Inadequate?
Given the learnability assumption of  $\phi^{\star}$ , one natural idea for solving the above problem is to estimate  $\phi^{\star}$ , and then run the standard LinUCB algorithm to estimate  $(\theta_{a},\beta_{a})$  together by treating  $(x_{t},\widehat{\phi}_{t}(x_{t}))$  as the true contexts. Indeed, this is the approach adopted by Wang et al. (2016) for addressing a similar problem of missing contexts  $z_{t}$ , except that they used a different unsurprised-learning-based approach to estimate the context  $z_{t}$  due to not being able to observing any data about  $z_{t}$ . Given the estimation of  $\widehat{\phi}$ , their algorithm — which we term it as LinUCB  $(\widehat{\phi})$  — iteratively carries out the steps below at each iteration  $t$  (see Algorithm 2 for additional details): 1) Estimation of the context-generating function  $\widehat{\phi}_t(\cdot)$  from historical data; 2) Solve of the following regularized least square problem for each arm  $a\in \mathcal{A}$ , with regularization coefficient  $\lambda \geq 0$ :
$$
\ell_ {t} \left(\boldsymbol {\theta} _ {a}, \boldsymbol {\beta} _ {a}\right) = \sum_ {s \in [ t ]: a _ {s} = a} \left(r _ {s, a} - \boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} _ {a} - \widehat {\phi} _ {s} \left(\boldsymbol {x} _ {s}\right) ^ {\top} \boldsymbol {\beta} _ {a}\right) ^ {2} + \lambda \left(\| \boldsymbol {\theta} _ {a} \| _ {2} ^ {2} + \| \boldsymbol {\beta} _ {a} \| _ {2} ^ {2}\right), \tag {1}
$$
Under the assumption that the initialized parameters in their estimations are very close to the global optimum, Wang et al. (2016) were able to show the  $O(\sqrt{T})$  regret of this algorithm. However, it turns out that this algorithm will fail to yield an satisfying regret bound without their strong assumption on very close parameter initialization, because the errors arising from  $\widehat{\phi}(\cdot)$  will significantly enlarge the confidence set of  $\widehat{\theta}_a$  and  $\widehat{\beta}_a$ .<sup>3</sup> Thus after removing their initialization assumption, the best possible regret bound we can possibly achieve is of order  $\widetilde{O}(T^{3/4})$ , as illustrated in the subsequent proposition.
Proposition 1 (Regret of LinUCB-( $\widehat{\phi}$ )). The regret of LinUCB-( $\widehat{\phi}$ ) in Algorithm 2 is upper bounded by  $\widetilde{\mathcal{O}}\left(T^{1 - \alpha}d_u^\alpha +T^{1 - \alpha /2}\sqrt{K} d_u^{1 + \alpha}\right)$  with probability at least  $1 - \delta$ , by carefully setting the regularization coefficient  $\lambda = \Theta \big(L_{u}d_{u}^{\alpha}T^{1 - \alpha}\log \big(T / \delta \big)\big)$  in Equation 1.
Since  $\alpha \in [0,1 / 2]$ , the best possible regret upper bound above is  $\widetilde{\mathcal{O}} (T^{3 / 4})$ , which is considerably inferior to the sought-after regret bound of  $\widetilde{\mathcal{O}} (\sqrt{T})$ . Such deficiency of LinUCB-  $(\widehat{\phi})$  is further observed in all our experiments in Section 7 as well. These motivate our following design of a new online learning algorithm to address the challenge of post-serving context, during which we also developed a new technical tool which may be of independent interest to the research community.
4 A Robustified and Generalized Elliptical Potential Lemma
It turns out that solving the learning problem above requires some novel designs; core to these novelties is a robustified and generalized version of the well-known elliptical potential lemma (EPL), which may be of independent interest. This widely used lemma states a fact about a sequence of vectors  $\boldsymbol{x}_1, \dots, \boldsymbol{x}_T \in \mathbb{R}^d$ . Intuitively, it captures the rate of the sum of additional information contained in each  $\boldsymbol{x}_t$ , relative to its predecessors  $\boldsymbol{x}_1, \dots, \boldsymbol{x}_{t-1}$ . Formally,
Lemma (Original Elliptical Potential Lemma). Suppose (1)  $\mathbf{X}_0 \in \mathbb{R}^{d \times d}$  is any positive definite matrix; (2)  $\mathbf{x}_1, \ldots, \mathbf{x}_T \in \mathbb{R}^d$  is any sequence of vectors; and (3)  $\mathbf{X}_t = \mathbf{X}_0 + \sum_{s=1}^t \mathbf{x}_s \mathbf{x}_s^\top$ . Then the following inequality holds