SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {t = 1} ^ {T} 1 \wedge \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2} \leq 2 \log \left(\frac {\det  \boldsymbol {X} _ {T}}{\det  \boldsymbol {X} _ {0}}\right),
$$
where  $a\wedge b = \min \{a,b\}$  is the min among  $a,b\in \mathbb{R}$
To address our new contextual bandit setup with post-serving contexts, it turns out that we will need to robustify and generalize the above lemma to accommodate noises in  $\pmb{x}_t$  vectors and slower learning rates. Specifically, we present the following variant of the EPL lemma.
<sup>3</sup>Specifically, Wang et al. (2016) assume that the initial estimation of  $\widehat{\phi}(\cdot)$  is already very close to  $\widehat{\phi}^*(\cdot)$  such that the error arising from  $\widehat{\phi}(\cdot)$  diminishes exponentially fast due to the local convergence property of alternating least squares algorithm (Uschmajew, 2012). This strong assumption avoids significant expansion of the confidence sets, but is less realistic in applications so we do not impose such assumption.
Lemma 1 (Generalized Elliptical Potential Lemma). Suppose (1)  $\mathbf{X}_0 \in \mathbb{R}^{d \times d}$  is any positive definite matrix; (2)  $\mathbf{x}_1, \ldots, \mathbf{x}_T \in \mathbb{R}^d$  is a sequence of vectors with bounded  $l_2$  norm  $\max_t \| \mathbf{x}_t \| \leq L_x$ ; (3)  $\epsilon_1, \ldots, \epsilon_T \in \mathbb{R}^d$  is a sequence of independent (not necessarily identical) bounded zero-mean noises satisfying  $\max_t \| \mathbf{\epsilon}_t \| \leq L_\epsilon$  and  $\mathbb{E}[\mathbf{\epsilon}_t \mathbf{\epsilon}_t^\top] \succcurlyeq \sigma_\epsilon^2 \mathbf{I}$  for any  $t$ ; and (4)  $\widetilde{\mathbf{X}}_t$  is defined as follows:
$$
\widetilde {\boldsymbol {X}} _ {t} = \boldsymbol {X} _ {0} + \sum_ {s = 1} ^ {t} (\boldsymbol {x} _ {s} + \boldsymbol {\epsilon} _ {s}) (\boldsymbol {x} _ {s} + \boldsymbol {\epsilon} _ {s}) ^ {\top} \in \mathbb {R} ^ {d \times d}.
$$
Then, for any  $p \in [0,1]$ , the following inequality holds with probability at least  $1 - \delta$
$$
\sum_ {t = 1} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\widehat {\boldsymbol {X}} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq 2 ^ {p} T ^ {1 - p} \log^ {p} \left(\frac {\det  X _ {T}}{\det  X _ {0}}\right) + \frac {8 L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}}{\sigma_ {\epsilon} ^ {4}} \log \left(\frac {3 2 d L _ {\epsilon} ^ {2} \left(L _ {\epsilon} + L _ {x}\right) ^ {2}}{\delta \sigma_ {\epsilon} ^ {4}}\right) \tag {2}
$$
Note that the second term is independent of time horizon  $T$  and only depends on the setup parameters. Generally, this can be treated as a constant. Before describing main proof idea of the lemma, we make a few remarks regarding Lemma 1 to highlight the significance of these generalizations.
1. The original Elliptical Potential Lemma (EPL) corresponds to the specific case of  $p = 1$ , while Lemma 1 is applicable for any  $p \in [0,1]$ . Notably, the  $(1 - p)$  rate in the  $T^{1 - p}$  term of Inequality 2 is tight for every  $p$ . In fact, this rate is tight even for  $\boldsymbol{x}_t = 1 \in \mathbb{R}, \forall t$  and  $\boldsymbol{X}_0 = 1 \in \mathbb{R}$  since, under these conditions,  $\| \boldsymbol{x}_t\|_{\boldsymbol{X}_{t - 1}^{-1}}^2 = 1 / t$  and, consequently,  $\sum_{t = 1}^{T}\left(1\wedge \| \boldsymbol{x}_t\|_{\boldsymbol{X}_{t - 1}^{-1}}^2\right)^p = \sum_{t = 1}^{T}t^{-p}$ , yielding a rate of  $T^{1 - p}$ . This additional flexibility gained by allowing a general  $p \in [0,1]$  (with the original EPL corresponding to  $p = 1$ ) helps us to accommodate slower convergence rates when learning the mean context from observed noisy contexts, as formalized in Assumption 1.
2. A crucial distinction between Lemma 1 and the original EPL lies in the definition of the noisy data matrix  $\widetilde{\boldsymbol{X}}_t$  in Equation 1, which permits noise. However, the measured context vector  $\boldsymbol{x}_t$  does not have noise. This is beneficial in scenarios where a learner observes noisy contexts but seeks to establish an upper bound on the prediction error based on the underlying noise-free context or the mean context. Such situations are not rare in real applications; our problem of contextual bandits with post-serving contexts is precisely one of such case — while choosing an arm, we can estimate the mean post-serving context conditioned on the observable pre-serving context but are only able to observe the noisy realization of post-serving contexts after acting.
3. Other generalized variants of the EPL have been recently proposed and found to be useful in different contexts. For instance, Carpentier et al. (2020) extends the EPL to allow for the  $X_{t}^{-p}$ -norm, as opposed to the  $X_{t}^{-1}$ -norm, while Hamidi and Bayati (2022) explores a generalized form of the  $1 \wedge \| \varphi(\boldsymbol{x}_{t})\|_{X_{t-1}^{-1}}^{2}$  term, which is motivated by variance reduction in non-Gaussian linear regression models. Nevertheless, to the best of our knowledge, our generalized version is novel and has not been identified in prior works.
Proof Sketch of Lemma 1. The formal proof of this lemma is involved and deferred to Appendix B.1. At a high level, our proof follows procedure for proving the original EPL. However, to accommodate the noises in the data matrix, we have to introduce new matrix concentration tools to the original (primarily algebraic) proof, and also identify the right conditions for the argument to go through. A key lemma to our proof is a high probability bound regarding the constructed noisy data matrix  $\widetilde{\mathbf{X}}_t$  (Lemma 2 in Appendix B.1) that we derive based on Bernstein's Inequality for matrices under spectral norm (Tropp et al., 2015). We prove that, under mild assumptions on the noise,  $\| x_{t}\|_{\widetilde{X}_{t - 1}^{-1}}^{2}\leq \| x_{t}\|_{X_{t - 1}^{-1}}^{2}$  with high probability for any  $t$ . Next, we have to apply the union bound and this lemma to show that the above matrix inequality holds for every  $t\geq 1$  with high probability. Unfortunately, this turns out to not be true because when  $t$  is very small (e.g.,  $t = 1$ ), the above inequality cannot hold with high probability. Therefore, we have to use the union bound in a carefully tailored way by excluding all  $t$ 's that are smaller than a certain threshold (chosen optimally by solving certain inequalities) and handling these terms with small  $t$  separately (which is the reason of the second  $\mathcal{O}(\log (1 / \delta))$  term in Inequality 2). Finally, we refine the analysis of sthe standard EPL by allowing the exponent  $p$  in
Algorithm 1 poLinUCB (Linear UCB with post-serving contexts)
1: for  $t = 0,1,\ldots ,T$  do
2: Receive the pre-serving context  $\pmb{x}_t$
3: Compute the optimistic parameters by maximizing the UCB objective
$$
\left(a _ {t}, \widetilde {\phi} _ {t} (\boldsymbol {x} _ {t}), \widetilde {\boldsymbol {w}} _ {t}\right) = \underset {(a, \phi , \boldsymbol {w} _ {a}) \in [ K ] \times \mathcal {C} _ {t - 1} \left(\widehat {\phi} _ {t - 1}, \boldsymbol {x} _ {t}\right) \times \mathcal {C} _ {t - 1} \left(\widehat {\boldsymbol {w}} _ {t - 1, a}\right)} {\arg \max } \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi (\boldsymbol {x} _ {t}) \end{array} \right] ^ {\top} \boldsymbol {w} _ {a}.
$$
4: Play the arm  $a_{t}$  and receive the realized post-serving context as  $z_{t}$  and the real-valued reward
$$
r _ {t, a _ {t}} = \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \boldsymbol {z} _ {t} \end{array} \right] ^ {\top} \boldsymbol {w} _ {a _ {t}} ^ {\star} + \eta_ {t}.
$$
5: Compute  $\widehat{\pmb{w}}_{t,a}$  using Equation 3 for each  $a\in \mathcal{A}$
6: Compute the estimated post-serving context generating function  $\widehat{\phi}_t(\cdot)$  using ERM.
7: Update confidence sets  $\mathcal{C}_t(\widehat{\boldsymbol{w}}_{t,a})$  and  $\mathcal{C}_t(\widehat{\phi}_t,\boldsymbol{x}_t)$  for each  $a$  based on Equations 6 and 5.
8: end for
$(1 \wedge \| \boldsymbol{x}_t\|_{\widetilde{\boldsymbol{X}}_{t - 1}^{-1}}^2)^p$  and derive an upper bound on the sum  $\sum_{t = 1}^{T}(1 \wedge \| \boldsymbol{x}_t\|_{\widetilde{\boldsymbol{X}}_{t - 1}^{-1}}^2)^p$  with high probability. These together yield a robustified and generalized version of EPL as in Lemma 1.
5 No Regret Learning in Linear Bandits with Post-Serving Contexts
5.1 The Main Algorithm
In the ensuing section, we introduce our algorithm, poLinUCB, designed to enhance linear contextual bandit learning through the incorporation of post-serving contexts and address the issue arose from the algorithm introduced in Section 3.2. The corresponding pseudo-code is delineated in Algorithm 1. Unlike the traditional LinUCB algorithm, which solely learns and sustains confidence sets for parameters (i.e.,  $\widehat{\beta}_a$  and  $\widehat{\theta}_a$  for each  $a$ ), our algorithm also simultaneously manages the same for the post-serving context generating function,  $\widehat{\phi}(\cdot)$ . Below, we expound on our methodology for parameter learning and confidence set construction.
Parameter learning. During each iteration  $t$ , we fit the function  $\widehat{\phi}_t(\cdot)$  and the parameters  $\{\widehat{\theta}_{t,a}\}_{a\in \mathcal{A}}$  and  $\{\widehat{\beta}_{t,a}\}_{a\in \mathcal{A}}$ . To fit  $\widehat{\phi}_t(\cdot)$ , resort to the conventional empirical risk minimization (ERM) framework. As for  $\{\widehat{\pmb{\theta}}_{t,a}\}_{a\in \mathcal{A}}$  and  $\{\widehat{\beta}_{t,a}\}_{a\in \mathcal{A}}$ , we solve the following least squared problem for each arm  $a$ ,
$$
\ell_ {t} \left(\boldsymbol {\theta} _ {a}, \boldsymbol {\beta} _ {a}\right) = \sum_ {s \in [ t ]: a _ {s} = a} \left(r _ {s, a} - \boldsymbol {x} _ {s} ^ {\top} \boldsymbol {\theta} _ {a} - z _ {s} ^ {\top} \boldsymbol {\beta} _ {a}\right) ^ {2} + \lambda \left(\| \boldsymbol {\theta} _ {a} \| _ {2} ^ {2} + \| \boldsymbol {\beta} _ {a} \| _ {2} ^ {2}\right). \tag {3}
$$
For convenience, we use  $\boldsymbol{w}$  and  $\boldsymbol{u}$  to denote  $(\theta, \beta)$  and  $(x, z)$  respectively. The closed-form solutions to  $\widehat{\theta}_{t,a}$  and  $\widehat{\beta}_{t,a}$  for each arm  $a \in \mathcal{A}$  are
$$
\widehat {\boldsymbol {w}} _ {t, a} := \left[ \begin{array}{l} \widehat {\boldsymbol {\theta}} _ {t, a} \\ \widehat {\boldsymbol {\beta}} _ {t, a} \end{array} \right] = \boldsymbol {A} _ {t, a} ^ {- 1} \boldsymbol {b} _ {t, a}, \text {w h e r e} \boldsymbol {A} _ {t, a} = \lambda \boldsymbol {I} + \sum_ {s: a _ {s} = a} ^ {t} \boldsymbol {u} _ {s} \boldsymbol {u} _ {s} ^ {\top} \quad \text {a n d} \quad \boldsymbol {b} _ {t, a} = \sum_ {s: a _ {s} = a} ^ {t} r _ {s, a} \boldsymbol {u} _ {s}. \tag {4}
$$
Confidence set construction. At iteration  $t$ , we construct the confidence set for  $\hat{\phi}_t(x_t)$  by
$$
\mathcal {C} _ {t} \left(\widehat {\phi} _ {t}, \boldsymbol {x} _ {t}\right) := \left\{\boldsymbol {z} \in \mathbb {R} ^ {d}: \left\| \widehat {\phi} _ {t} (\boldsymbol {x} _ {t}) - \boldsymbol {z} \right\| _ {2} \leq e _ {t} ^ {\delta} \right\}. \tag {5}
$$
Similarly, we can construct the confidence set for the parameters  $\widehat{\pmb{w}}_{t,a}$  for each arm  $a\in \mathcal{A}$  by
$$
\mathcal {C} _ {t} \left(\widehat {\boldsymbol {w}} _ {t, a}\right) := \left\{\boldsymbol {w} \in \mathbb {R} ^ {d _ {x} + d _ {z}}: \| \boldsymbol {w} - \widehat {\boldsymbol {w}} _ {t, a} \| _ {\boldsymbol {A} _ {t, a}} \leq \zeta_ {t, a} \right\}, \tag {6}
$$
where  $\zeta_{t,a} = 2\sqrt{\lambda} + R_{\eta}\sqrt{d_u\log((1 + n_t(a)L_u^2 / \lambda) / \delta)}$  and  $n_t(a) = \sum_{s = 1}^t\mathbb{1}[a_s = a]$ . Additionally, we further define  $\zeta_t \coloneqq \max_{a\in \mathcal{A}}\zeta_{t,a}$ . By the assumption 1 and Lemma 3, we have the followings hold with probability at least  $1 - \delta$  for each of the following events,
$$
\phi^ {\star} \left(\boldsymbol {x} _ {t}\right) \in \mathcal {C} _ {t} \left(\widehat {\phi} _ {t}, \boldsymbol {x} _ {t}\right) \quad \text {a n d} \quad \boldsymbol {w} ^ {\star} \in \mathcal {C} _ {t} \left(\widehat {\boldsymbol {w}} _ {t, a}\right). \tag {7}
$$
5.2 Regret Analysis
In the forthcoming section, we establish the regret bound. Our proof is predicated upon the conventional proof of LinUCB (Li et al., 2010) in conjunction with our robust elliptical potential lemma. The pseudo-regret (Audibert et al., 2009) within this partial contextual bandit problem is defined as,
$$
R _ {T} = \operatorname {R e g r e t} (T) = \sum_ {t = 1} ^ {T} \left(r _ {t, a _ {t} ^ {*}} - r _ {t, a _ {t}}\right), \tag {8}
$$
in which we reload the notation of reward by ignoring the noise,
$$
r _ {t, a} = \left\langle \boldsymbol {\theta} _ {a} ^ {\star}, \boldsymbol {x} _ {t} \right\rangle + \left\langle \boldsymbol {\beta} _ {a} ^ {\star}, \phi^ {\star} (\boldsymbol {x} _ {t}) \right\rangle \quad \text {a n d} \quad a _ {t} ^ {\star} = \underset {a \in A} {\arg \max } \left\langle \boldsymbol {\theta} _ {a} ^ {\star}, \boldsymbol {x} _ {t} \right\rangle + \left\langle \boldsymbol {\beta} _ {a} ^ {\star}, \phi^ {\star} (\boldsymbol {x} _ {t}) \right\rangle . \tag {9}
$$
It is crucial to note that our definition of the optimal action,  $a_{t}^{\star}$ , in Eq. 9 depends on  $\phi^{\star}(\pmb{x}_t)$  as opposed to  $\pmb{z}_{t}$ . This dependency ensures a more pragmatic benchmark, as otherwise, the noise present in  $\pmb{z}$  would invariably lead to a linear regret, regardless of the algorithm implemented. In the ensuing section, we present our principal theoretical outcomes, which provide an upper bound on the regret of our poLinUCB algorithm.
Theorem 1 (Regret of poLinUCB). The regret of poLinUCB in Algorithm 1 is upper bounded by  $\widetilde{\mathcal{O}}\left(T^{1 - \alpha}d_u^\alpha +d_u\sqrt{TK}\right)$  with probability at least  $1 - \delta$ , if  $T = \Omega (\log (1 / \delta))$
The first term in the bound is implicated by learning the function  $\phi^{\star}(\cdot)$ . Conversely, the second term resembles the one derived in conventional contextual linear bandits, with the exception that our dependency on  $d_{u}$  is linear. This linear dependency is a direct consequence of our generalized robust elliptical potential lemma. The proof is deferred in Appendix B.2.
6 Generalizations
So far we have focused on a basic linear bandit setup with post-serving features. Our results and analysis can be easily generalized to other variants of linear bandits, including those with feature mappings, and below we highlight some of these generalizations. They use similar proof ideas, up to some technical modifications; we thus defer all their formal proofs to Appendix B.3.
6.1 Generalization to Action-Dependent Contexts
Our basic setup in Section 3 has a single context  $\boldsymbol{x}_t$  at any time step  $t$ . This can be generalized to action-dependent contexts settings as studied in previous works (e.g., Li et al. (2010)). That is, during each iteration indexed by  $t$ , the learning algorithm observes a context  $\boldsymbol{x}_{t,a}$  for each individual arm  $a \in \mathcal{A}$ . Upon executing the action of pulling arm  $a_t$ , the corresponding post-serving context  $\boldsymbol{z}_{t,a_t}$  is subsequently revealed. Notwithstanding, the post-serving context for all alternative arms remains unobserved. The entire procedure is the same as that of Section 3.
In extending this framework, we persist in our assumption that for each arm  $a \in \mathcal{A}$ , there exists a specific function  $\phi_a^\star(\cdot): \mathbb{R}^{d_x} \to \mathbb{R}^{d_z}$  that generates the post-serving context  $z$  upon receiving  $x$  associated with arm  $a \in \mathcal{A}$ . The primary deviation from our preliminary setup lies in the fact that we now require the function  $\phi_a^\star(\cdot)$  to be learned for each arm independently. The reward is generated as
$$
r _ {t, a _ {t}} = \left\langle \boldsymbol {\theta} _ {a _ {t}} ^ {\star}, \boldsymbol {x} _ {t, a _ {t}} \right\rangle + \left\langle \boldsymbol {\beta} _ {a _ {t}} ^ {\star}, \boldsymbol {z} _ {t, a _ {t}} \right\rangle + \eta_ {t}.
$$
The following proposition shows our regret bound for this action-dependent context case. Its proof largely draws upon the proof idea of Theorem 1 and also relies on the generalized EPL Lemma 1.
Proposition 2. The regret of poLinUCB in Algorithm 1 for action-dependent contexts is upper bounded by  $\widetilde{\mathcal{O}}\left(T^{1 - \alpha}d_u^\alpha \sqrt{K} +d_u\sqrt{TK}\right)$  with probability at least  $1 - \delta$  if  $T = \Omega (\log (1 / \delta))$
The main difference with the bound in Theorem 1 is the additional  $\sqrt{K}$  appeared in the first term, which is caused by learning multiple  $\phi_{a}^{\star}(\cdot)$  functions with  $a\in \mathcal{A}$ .
Figure 1: Cumulative Regret in three synthetic environments. Comparisons of different algorithms in terms of cumulative regret across the three synthetic environments. Our proposed poLinUCB (ours) consistently outperforms other strategies (except for LinUCB which has access to the post-serving context during arm selection), showcasing its effectiveness in utilizing post-serving contexts. The shaded area denotes the standard error computed using 10 different random seeds.
6.2 Generalization to Linear Stochastic Bandits
Another variant of linear bandits is the linear stochastic bandits setup (see, e.g., (Abbasi-Yadkori et al., 2011)). This model allows infinitely many arms, which consists of a decision set  $D_{t} \subseteq \mathbb{R}^{d}$  at time  $t$ , and the learner picks an action  $\boldsymbol{x}_t \in D_t$ . This setup naturally generalizes to our problem with post-serving contexts. That is, at iteration  $t$ , the learner selects an arm  $\boldsymbol{x}_t \in D_t$  first, receives reward  $r_{t,\boldsymbol{x}_t}$ , and then observe the post-serving feature  $\boldsymbol{z}_t$  conditioned on  $\boldsymbol{x}_t$ . Similarly, we assume the existence of a mapping  $\phi^{\star}(\boldsymbol{x}_t) = \mathbb{E}[z_t|\boldsymbol{x}_z]$  that satisfies the Assumption 1. Consequently, the realized reward is generated as follows where  $\boldsymbol{\theta}^{\ast},\boldsymbol{\beta}^{\ast}$  are unknown parameters:
$$
r _ {t, \boldsymbol {x} _ {t}} = \left\langle \boldsymbol {x} _ {t}, \boldsymbol {\theta} ^ {\star} \right\rangle + \left\langle \boldsymbol {z} _ {t}, \boldsymbol {\beta} ^ {\star} \right\rangle + \eta_ {t}.
$$
Therefore, the learner needs to estimate the linear parameters  $\widehat{\pmb{\theta}}$  and  $\widehat{\beta}$ , as well as the function  $\widehat{\phi}(\cdot)$ . We obtain the following proposition for the this setup.
Proposition 3. The regret of poLinUCB in Algorithm 1 for the above setting is upper bounded by  $\widetilde{\mathcal{O}}\left(T^{1 - \alpha}d_u^\alpha + d_u\sqrt{T}\right)$  with probability at least  $1 - \delta$  if  $T = \Omega (\log (1 / \delta))$ .
6.3 Generalization to Linear Bandits with Feature Mappings
Finally, we briefly remark that while we have so far assumed that the arm parameters are directly linear in the context  $\boldsymbol{x}_t, \boldsymbol{z}_t$ , just like classic linear bandits our analysis can be easily generalized to accommodate feature mapping  $\pi^x(\boldsymbol{x}_t)$  and  $\pi^z(\boldsymbol{z}_t) = \pi^z(\phi(\boldsymbol{x}_t) + \varepsilon_t)$ . Specifically, if the reward generation process is  $r_a = \langle \theta_a^\star, \pi^x(\boldsymbol{x}_t) \rangle + \langle \beta_a^\star, \pi^z(\boldsymbol{z}_t) \rangle + \eta_t$  instead, then we can simply view  $\tilde{\boldsymbol{x}}_t = \pi^x(\boldsymbol{x}_t)$  and  $\tilde{\boldsymbol{z}}_t = \pi^z(\boldsymbol{z}_t)$  as the new features, with  $\tilde{\phi}(\boldsymbol{x}_t) = \mathbb{E}_{\epsilon_t}[\pi^z(\phi(\boldsymbol{x}_t) + \epsilon_t)]$ . By working with  $\tilde{\boldsymbol{x}}_t, \tilde{\boldsymbol{z}}_t, \tilde{\phi}$ , we shall obtain the same guarantees as Theorem 1.
This section presents a comprehensive evaluation of our proposed poLinUCB algorithm on both synthetic and real-world data, demonstrating its effectiveness in incorporating follow-up information and outperforming the LinUCB  $(\widehat{\phi})$  variant. More empirical results can be found in Appendix D.1.
7.1 Synthetic Data with Ground Truth Models
Evaluation Setup. We adopt three different synthetic environments that are representative of a range of mappings from the pre-serving context to the post-serving context: polynomial, periodical and linear functions. The pre-serving contexts are sampled from a uniform noise in the range  $[-10, 10]^{d_x}$ , and Gaussian noise is employed for both the post-serving contexts and the rewards. In each environment, the dimensions of the pre-serving context  $(d_x)$  and the post-serving context  $(d_z)$  are of 100 and 5, respectively with 10 arms  $(K)$ . The evaluation spans  $T = 1000$  or 5000 time steps, and each experiment is repeated with 10 different seeds. The cumulative regret for each policy in each environment is then calculated to provide a comparaison.
Results and Discussion. Our experimental results, which are presented graphically in Figures 1, provide strong evidence of the superiority of our proposed poLinUCB algorithm. Across all setups, we
observe that the LinUCB  $(x$  and  $z)$  strategy, which has access to the post-serving context during arm selection, consistently delivers the best performance, thus serving as the upper bound for comparison. On the other hand, the Random policy, which does not exploit any environment information, performs the worst, serving as the lower bound. Our proposed poLinUCB (ours) outperforms all the other strategies, including the LinUCB  $(\hat{\phi})$  variant, in all three setups, showcasing its effectiveness in adaptively handling various mappings from the pre-serving context to the post-serving context. Importantly, poLinUCB delivers significantly superior performance to LinUCB  $(x$  only), which operates solely based on the pre-serving context.
7.2 Real World Data without Ground Truth
Evaluation Setup. The evaluation was conducted on a real-world dataset, MovieLens (Harper and Konstan, 2015), where the task is to recommend movies (arms) to a incoming user (context). Following Yao et al. (2023), we first map both movies and users to 32-dimensional real vectors using a neural network trained for predicting the rating. Initially,  $K = 5$  movies were randomly sampled to serve as our arms and were held fixed throughout the experiment. The user feature vectors were divided into two parts serving as the pre-serving context ( $d_x = 25$ ) and the post-serving context ( $d_z = 7$ ). We fit the function  $\phi(\boldsymbol{x})$  using a two-layer neural network with 64 hidden units and ReLU activation. The network was trained using the Adam optimizer with a learning rate of 1e-3. At each iteration, we randomly sampled a user from the dataset and exposed only the pre-serving context  $\boldsymbol{x}$  to our algorithm. The reward was computed as the dot product of the user's feature vector and the selected movie's feature vector and was revealed post the movie selection. The evaluation spanned  $T = 500$  iterations and repeated with 10 seeds. MovieLens
Results and Discussion. The experimental results, presented in Figure 2, demonstrate the effectiveness of our proposed algorithm. The overall pattern is similar to it observed in our synthetic experiments. Our proposed policy consistently outperforms the other strategies (except for Lin-UCB with both pre-serving and post-serving features). Significantly, our algorithm yields superior performance compared to policies operating solely on the pre-serving context, thereby demonstrating its effectiveness in leveraging the post-serving information.
Figure 2: Results on MovieLens.