SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {t = 1} ^ {T} \left(\log \left(1 + \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right)\right) ^ {p} = \sum_ {t = 1} ^ {T} \left(\log \det  \boldsymbol {X} _ {t} - \log \det  \boldsymbol {X} _ {t - 1}\right) ^ {p}.
$$
Since  $p \in [0,1]$ , the function  $g(x) = x^p$  is a concave function. Thus, we have
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \left(\log \det  \boldsymbol {X} _ {t} - \log \det  \boldsymbol {X} _ {t - 1}\right) ^ {p} \leq \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \log \det  \boldsymbol {X} _ {t} - \log \det  \boldsymbol {X} _ {t - 1}\right) ^ {p} = \frac {1}{T ^ {p}} \log^ {p} \left(\frac {\det  \boldsymbol {X} _ {T}}{\det  \boldsymbol {X} _ {0}}\right).
$$
Therefore, we can conclude that
$$
\sum_ {t = 1} ^ {T} 1 \wedge \left(\| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq 2 ^ {p} T ^ {1 - p} \log^ {p} \left(\frac {\det  \boldsymbol {X} _ {T}}{\det  \boldsymbol {X} _ {0}}\right).
$$
By combining the above results, we have the following hold with probability at least  $1 - \delta$ :
$$
\sum_ {t = 1} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\tilde {\boldsymbol {X}} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq T _ {0} - 1 + \left(\sum_ {t = 1} ^ {T} 1 \wedge \| \boldsymbol {x} _ {t} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq T _ {0} - 1 + 2 ^ {p} T ^ {1 - p} \log^ {p} \left(\frac {\det  \boldsymbol {X} _ {T}}{\det  \boldsymbol {X} _ {0}}\right).
$$
Invoking
$$
T _ {0} - 1 \leq \frac {8 L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\sigma_ {\epsilon} ^ {4}} \log \left(\frac {3 2 d L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\delta \sigma_ {\epsilon} ^ {4}}\right),
$$
we obtained the desired inequality with probability at least  $1 - \delta$ :
$$
\sum_ {t = 1} ^ {T} \left(1 \wedge \| \boldsymbol {x} _ {t} \| _ {\widetilde {\boldsymbol {X}} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {p} \leq 2 ^ {p} T ^ {1 - p} \log^ {p} \left(\frac {\det  \boldsymbol {X} _ {T}}{\det  \boldsymbol {X} _ {0}}\right) + \frac {8 L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\sigma_ {\epsilon} ^ {4}} \log \left(\frac {3 2 d L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\delta \sigma_ {\epsilon} ^ {4}}\right).
$$
Proof. We will analyze the term-wise difference, denoted as
$$
\boldsymbol {S} _ {t} := \left(\boldsymbol {x} _ {t} + \boldsymbol {\epsilon} _ {t}\right) \left(\boldsymbol {x} _ {t} + \boldsymbol {\epsilon} _ {t}\right) ^ {\top} - \boldsymbol {x} _ {t} \boldsymbol {x} _ {t} ^ {\top} = \boldsymbol {\epsilon} _ {t} \boldsymbol {x} _ {t} ^ {\top} + \boldsymbol {x} _ {t} \boldsymbol {\epsilon} _ {t} ^ {\top} + \boldsymbol {\epsilon} _ {t} \boldsymbol {\epsilon} _ {t} ^ {\top} \tag {14}
$$
Moreover, since  $\mathbb{E}[\epsilon_t] = 0$  for any  $t\in [T]$ , the expectation of  $S_{t}$  (over randomness of noise) can be lower bounded as
$$
\mathbb {E} [ \boldsymbol {S} _ {t} ] = \mathbb {E} [ \boldsymbol {\epsilon} _ {t} ] \boldsymbol {x} _ {t} ^ {\top} + \boldsymbol {x} _ {t} \mathbb {E} [ \boldsymbol {\epsilon} _ {t} ^ {\top} ] + \mathbb {E} [ \boldsymbol {\epsilon} _ {t} \boldsymbol {\epsilon} _ {t} ^ {\top} ] = \mathbb {E} [ \boldsymbol {\epsilon} _ {t} \boldsymbol {\epsilon} _ {t} ^ {\top} ] \succcurlyeq \sigma_ {\epsilon} ^ {2} \boldsymbol {I}.
$$
Since  $\| \pmb{x}_t\| _2\leq L_x$  and  $\| \epsilon_t\| _2\leq L_\epsilon$ , we know that  $S_{t}\preccurlyeq (2L_{\epsilon}L_{x} + L_{\epsilon}^{2})I$  with probability 1. Thus  $S_{t}$  is uniformly upper bounded under the spectral-norm denoted by  $\| \cdot \|$ , or formally
$$
\left\| \boldsymbol {S} _ {t} \right\| \leq 2 L _ {\epsilon} L _ {x} + L _ {\epsilon} ^ {2}.
$$
Consider the "centered" matrix sum  $\mathbf{Z}_T = \sum_{t=1}^T \left[ \mathbf{S}_t - \mathbb{E}[\boldsymbol{\epsilon}_t \boldsymbol{\epsilon}_t^\top] \right]$ , with mean 0. Since the spectral norm of  $\mathbf{S}_t$  is upper bounded by  $2L_\epsilon L_x + L_\epsilon^2$ , its variance  $\mathbb{V}(\mathbf{S}_t) = \left\| \mathbb{E} \left[ \mathbf{S}_t \mathbf{S}_t^\top \right] \right\|_2$  is upper bounded by  $(2L_\epsilon L_x + L_\epsilon^2)^2$ . Thus, the variance of  $\mathbf{Z}_T$  equals the variance of  $\sum_{t=1}^T \mathbf{S}_t$ , which is then upper bounded by  $T(2L_\epsilon L_x + L_\epsilon^2)^2$ . By the Bernstein's Inequality for random matrices (Tropp et al., 2015), we have the following high probability upper bound for the spectral norm  $\| \cdot \|$  of  $\mathbf{Z}_T$ :
$$
\mathbb {P} \left[ \| \boldsymbol {Z} _ {T} \| \geq \iota \right] \leq 2 d \exp \left(\frac {- \iota^ {2} / 2}{\mathbb {V} (\boldsymbol {Z} _ {T}) + (2 L _ {\epsilon} L _ {x} + L _ {\epsilon} ^ {2}) \iota / 3}\right). \tag {15}
$$
Since  $Z_{T}$  is a symmetric matrix, its spectral norm upper bounds the absolute value of any eigenvalue. Thus we can lower bound the smallest eigenvalues as follows:
$$
\begin{array}{l} \mathbb {P} \left[ \| \boldsymbol {Z} _ {T} \| \geq \iota \right] \geq \mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {S} _ {t} - \mathbb {E} \left[ \boldsymbol {\epsilon} _ {t} \boldsymbol {\epsilon} _ {t} ^ {\top} \right]\right) \leq - \iota \right] \\ \geq \mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {S} _ {t} - \sigma_ {\epsilon} ^ {2} \boldsymbol {I}\right) \leq - \iota \right] \\ = \mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {S} _ {t}\right) \leq T \sigma_ {\epsilon} ^ {2} - \iota \right]. \\ \end{array}
$$
where the second inequality is due to two facts: (1)  $A \succcurlyeq B$  implies  $\lambda_{\min}(A) \geq \lambda_{\min}(B)$  where  $A = \left(\sum_{t=1}^{T} \boldsymbol{S}_t - \sigma_\epsilon^2 \boldsymbol{I}\right)$  and  $B = \left(\sum_{t=1}^{T} \boldsymbol{S}_t - \mathbb{E}[\boldsymbol{\epsilon}_t \boldsymbol{\epsilon}_t^\top]\right)$ ; and (2) the event  $\lambda_{\min}(A) \leq -\iota$  thus is included within the event  $\lambda_{\min}(B) \leq -\iota$ . Consequently, we have
$$
\mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {S} _ {t}\right) \leq T \sigma_ {\epsilon} ^ {2} - \iota \right] \leq 2 d \exp \left(\frac {- \iota^ {2} / 2}{\mathbb {V} (\boldsymbol {Z} _ {T}) + (2 L _ {\epsilon} L _ {x} + L _ {\epsilon} ^ {2}) \iota / 3}\right),
$$
$$
\mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {S} _ {t}\right) \geq T \sigma_ {\epsilon} ^ {2} - \iota \right] \geq 1 - 2 d \exp \left(\frac {- \iota^ {2} / 2}{\mathbb {V} (\boldsymbol {Z} _ {T}) + (2 L _ {\epsilon} L _ {x} + L _ {\epsilon} ^ {2}) \iota / 3}\right),
$$
By choosing the value of  $\iota = T\sigma_{\epsilon}^{2}$ , we get
$$
\begin{array}{l} \mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {S} _ {t}\right) \geq 0 \right] \\ \geq 1 - 2 d \exp \left(\frac {- T ^ {2} \sigma_ {\epsilon} ^ {4} / 2}{\mathbb {V} (\boldsymbol {Z} _ {T}) + (2 L _ {\epsilon} L _ {x} + L _ {\epsilon} ^ {2}) T \sigma_ {\epsilon} ^ {2} / 3}\right) \\ \geq 1 - 2 d \exp \left(\frac {- T ^ {2} \sigma_ {\epsilon} ^ {4} / 2}{T (2 L _ {\epsilon} L _ {x} + L _ {\epsilon} ^ {2}) ^ {2} + (2 L _ {\epsilon} L _ {x} + L _ {\epsilon} ^ {2}) T \sigma_ {\epsilon} ^ {2} / 3}\right). \\ \end{array}
$$
Using the fact that  $\sigma_{\epsilon} \leq L_{\epsilon}$ , we can further simplify the above equation by
$$
\mathbb {P} \left[ \lambda_ {\min } \left(\sum_ {t = 1} ^ {T} \boldsymbol {S} _ {t}\right) \geq 0 \right] \geq 1 - 2 d \exp \left(\frac {- T \sigma_ {\epsilon} ^ {4}}{8 L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}\right).
$$
B.2 Missing Proofs in the Regret Analysis
Proof. In the next, we prove the regret bound. For each time step  $t$ , the immediate regret is
where the inequality (a) is due to the definition of UCB, and (b) and (c) are obtained using the Cauchy-Schwarz inequality. Therefore, the cumulative regret can be further upper bounded by
$$
\begin{array}{l} R _ {T} = \sum_ {t = 1} ^ {T} \Delta_ {t} \leq \sqrt {T \sum_ {t = 1} ^ {T} \Delta_ {t} ^ {2}} \\ \leq \sqrt {T \sum_ {t = 1} ^ {T} \left(\left\| \widetilde {\phi} _ {t} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| \left\| \widetilde {\boldsymbol {\beta}} _ {a _ {t}} \right\| + \left\| \left[ \begin{array}{l} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}} ^ {- 1}} \left\| \left[ \begin{array}{l} \widetilde {\boldsymbol {\theta}} _ {a _ {t}} - \boldsymbol {\theta} _ {a _ {t}} \\ \widetilde {\boldsymbol {\beta}} _ {a _ {t}} - \boldsymbol {\beta} _ {a _ {t}} \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}}}\right) ^ {2}} \\ \leq \sqrt {T \left(\sum_ {t = 1} ^ {T} 2 \left\| \widetilde {\phi} _ {t} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| ^ {2} \left\| \widetilde {\beta} _ {a _ {t}} \right\| ^ {2} + 2 \zeta_ {T} ^ {2} \left(1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}}} ^ {2}\right)\right)} \\ \leq \sqrt {T \left(\sum_ {t = 1} ^ {T} 2 \left\| \widetilde {\phi} _ {t} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| ^ {2} \left\| \widetilde {\beta} _ {a _ {t}} \right\| ^ {2} + 2 \zeta_ {T} ^ {2} \left(1 \wedge \left\| \left[ \begin{array}{l} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1 , a _ {t}} ^ {- 1}} ^ {2}\right)\right)} \\ \end{array}
$$
In the next, we bound each term separately. Firstly, we have the following hold with probability at least  $1 - \delta$  by using the union bound,
$$
\sum_ {t = 1} ^ {T} 2 \left\| \widetilde {\phi} _ {t} (\boldsymbol {x} _ {t}) - \phi^ {\star} (\boldsymbol {x} _ {t}) \right\| ^ {2} \cdot \left\| \widetilde {\beta} _ {a _ {t}} \right\| ^ {2} \leq 8 \sum_ {t = 1} ^ {T} \left(e _ {t} ^ {\delta / T}\right) ^ {2}.
$$
In the next, to bound the remaining term, we use the result from Lemma 1. We first group the sums based the arm,
$$
\sum_ {t = 1} ^ {T} 1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1, a _ {t}}} ^ {2} = \sum_ {a \in \mathcal {A}} \sum_ {t \in [ T ]: a _ {t} = a} 1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1, a}} ^ {2} \tag {16}
$$
By denoting  $n_T(a)$  as the number of times that arm  $a$  is pulled, we can divide the arms into two groups,
$$
\mathcal {G} _ {0} := \left\{a \in \mathcal {A}: n _ {T} (a) = \Omega (\log (1 / \delta)) \right\} \quad \text {a n d} \quad \mathcal {G} _ {1} := \mathcal {A} \backslash \mathcal {G} _ {0}.
$$
Then, we can further decompose the r.h.s term of Equation 16 based on if the corresponding arm is in  $\mathcal{G}_0$  or  $\mathcal{G}_1$ . Then, by applying Lemma 1, we have the following holds, with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \sum_ {t \in [ T ]: a _ {t} = a} 1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1, a} ^ {- 1}} ^ {2} \\ = \sum_ {a \in \mathcal {G} _ {0}} \sum_ {t \in [ T ]: a _ {t} = a} 1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1, a} ^ {- 1}} ^ {2} + \sum_ {a \in \mathcal {G} _ {1}} \sum_ {t \in [ T ]: a _ {t} = a} 1 \wedge \left\| \left[ \begin{array}{c} \boldsymbol {x} _ {t} \\ \phi^ {\star} (\boldsymbol {x} _ {t}) \end{array} \right] \right\| _ {\boldsymbol {A} _ {t - 1, a} ^ {- 1}} \\ \leq 2 K d _ {u} \log \left(1 + \frac {T L _ {u} ^ {2}}{\lambda d _ {u}}\right) + \frac {8 K L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\sigma_ {\epsilon} ^ {4}} \log \left(\frac {3 2 K d _ {u} L _ {\epsilon} ^ {2} (L _ {\epsilon} + L _ {x}) ^ {2}}{\delta \sigma_ {\epsilon} ^ {4}}\right). \\ \end{array}
$$
where the last inequality is due to Lemma 1 and apply the union bound on the  $K$  arms. To bound the remainder term, since  $\alpha \in [0,1/2]$ , by the learnability assumption as stated in Assumption 1 and Lemma 1, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(e _ {t} ^ {\delta / t}\right) ^ {2} \leq \sum_ {t = 1} ^ {T} C _ {0} ^ {2} \cdot \left(1 \wedge \| \boldsymbol {x} \| _ {\boldsymbol {X} _ {t - 1} ^ {- 1}} ^ {2}\right) ^ {2 \alpha} \cdot \log^ {2} \left(\frac {t T}{\delta}\right) \\ \leq 4 C _ {0} ^ {2} T ^ {1 - 2 \alpha} \log^ {2 \alpha} \left(\frac {\det X _ {T}}{\det X _ {0}}\right) \log^ {2} \left(\frac {T}{\delta}\right) \\ \leq 4 C _ {0} ^ {2} T ^ {1 - 2 \alpha} d _ {u} ^ {2 \alpha} \log^ {2 \alpha} \left(\frac {T L _ {u} ^ {2} / d + \lambda}{\lambda}\right) \log^ {2} \left(\frac {T}{\delta}\right) \\ \end{array}
$$
Therefore, the total regret bound is bounded by the following term with probability at least  $1 - 2\delta$
$$
\sqrt {T \cdot \left(8 C _ {0} T ^ {1 - 2 \alpha} \log^ {2 \alpha} \left(\frac {\det X _ {T}}{\det X _ {0}}\right) \log^ {2} \left(\frac {T}{\delta}\right) + K \zeta_ {T} ^ {2} \left(d _ {u} \log \left(1 + \frac {T L _ {u} ^ {2}}{\lambda d _ {u}}\right) + \frac {4 8 L _ {\epsilon} ^ {4} L _ {u} ^ {2}}{\sigma_ {\epsilon} ^ {4}} \log \left(\frac {1 9 2 K d _ {u} L _ {\epsilon} ^ {4} L _ {u} ^ {2}}{\delta \sigma_ {\epsilon} ^ {4}}\right)\right)\right)}
$$
By hiding the logarithmic terms, we can further simplify it to be
$$
\widetilde {\mathcal {O}} \left(T ^ {1 - \alpha} d _ {u} ^ {\alpha} + d _ {u} \sqrt {T K}\right)
$$
B.3 Missing Proofs in Generalizations
Proposition 2. The regret of poLinUCB in Algorithm 1 for action-dependent contexts is upper bounded by  $\widetilde{\mathcal{O}}\left(T^{1 - \alpha}d_u^\alpha \sqrt{K} + d_u\sqrt{TK}\right)$  with probability at least  $1 - \delta$  if  $T = \Omega (\log (1 / \delta))$ .
Proof. Our proof follows from the proof of Theorem 1. The immediate regret at each time step  $t$  is
$$
\Delta_ {t} = r _ {t, a _ {t} ^ {*}} - r _ {t, a _ {t}} \tag {17}
$$
Recall that the definition of  $a_{t}^{\star}$
$$
a _ {t} ^ {\star} := \underset {a \in \mathcal {A}} {\arg \max } \left\langle \boldsymbol {\theta} _ {a} ^ {\star}, \boldsymbol {x} _ {t, a} \right\rangle + \left\langle \boldsymbol {\beta} _ {a} ^ {\star}, \phi_ {a} ^ {\star} \left(\boldsymbol {x} _ {t, a}\right) \right\rangle . \tag {18}
$$
To bound the immediate regret, we have
$$
\Delta_ {t} = \left\langle \boldsymbol {\theta} _ {a _ {t} ^ {*}} ^ {\star}, \boldsymbol {x} _ {t, a _ {t} ^ {*}} \right\rangle + \left\langle \boldsymbol {\beta} _ {a _ {t} ^ {*}} ^ {\star}, \phi_ {a _ {t} ^ {*}} ^ {\star} \left(\boldsymbol {x} _ {t, a _ {t} ^ {*}}\right) \right\rangle - \left\langle \boldsymbol {\theta} _ {a _ {t}} ^ {\star}, \boldsymbol {x} _ {t, a _ {t}} \right\rangle - \left\langle \boldsymbol {\beta} _ {a _ {t}} ^ {\star}, \phi_ {a _ {t}} ^ {\star} \left(\boldsymbol {x} _ {t, a _ {t}}\right) \right\rangle . \tag {19}
$$
By the definition of UCB, we further have
$$
\Delta_ {t} \leq \langle \widetilde {\boldsymbol {\theta}} _ {t, a _ {t}}, \boldsymbol {x} _ {t, a _ {t}} \rangle + \langle \widetilde {\boldsymbol {\beta}} _ {t, a _ {t}}, \widetilde {\phi} _ {t, a _ {t}} ^ {\star} (\boldsymbol {x} _ {t, a _ {t}}) \rangle - \langle \boldsymbol {\theta} _ {a _ {t}} ^ {\star}, \boldsymbol {x} _ {t, a _ {t}} \rangle - \langle \boldsymbol {\beta} _ {a _ {t}} ^ {\star}, \phi_ {a _ {t}} ^ {\star} (\boldsymbol {x} _ {t, a _ {t}}) \rangle . \tag {20}
$$
By rearranging the terms, we get
$$
\Delta_ {t} \leq \underbrace {\left\langle \widetilde {\boldsymbol {\theta}} _ {t , a _ {t}} - \boldsymbol {\theta} _ {a _ {t}} ^ {\star} , \boldsymbol {x} _ {t , a _ {t}} \right\rangle + \left\langle \widetilde {\boldsymbol {\beta}} _ {t , a _ {t}} - \boldsymbol {\beta} _ {a _ {t}} ^ {\star} , \phi_ {a _ {t}} ^ {\star} \left(\boldsymbol {x} _ {t , a _ {t}}\right) \right\rangle} _ {①} + \underbrace {\left\langle \widetilde {\boldsymbol {\beta}} _ {t , a _ {t}} , \widetilde {\phi} _ {t , a _ {t}} \left(\boldsymbol {x} _ {t , a _ {t}}\right) - \phi_ {a _ {t}} ^ {\star} \left(\boldsymbol {x} _ {t , a _ {t}}\right) \right\rangle} _ {②} \tag {21}
$$
Bounding the first term  $\boxed{1}$  is the same as the proof in Theorem 1, while bounding the second term  $\boxed{2}$  will be slightly different, as we now have  $K$  such functions of  $\phi_a^\star (\cdot)$  for  $a\in \mathcal{A}$  to learn. By denoting the error for each estimate of  $\phi_{a}^{\star}(\cdot)$  at iteration  $t$  as  $e_{t,a}^{\delta}$ . Therefore, the contribution from the second term to the total regret can be bounded by
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(e _ {t, a _ {t}} ^ {\delta / t}\right) ^ {2} = \sum_ {a \in \mathcal {A}} \sum_ {t \in [ T ]: a _ {t} = a} \left(e _ {t, a} ^ {t / \delta}\right) ^ {2} (22) \\ \leq 4 K C _ {0} T ^ {1 - 2 \alpha} \log^ {2 \alpha} \left(\frac {\det X _ {T}}{\det X _ {0}}\right) \log^ {2} \left(\frac {T}{\delta}\right). (23) \\ \end{array}
$$
Hence, by following the remaining steps in the proof of Theorem 1, we can conclude that the regret is upper bounded by
$$
\widetilde {\mathcal {O}} \left(T ^ {1 - \alpha} d _ {u} ^ {\alpha} \sqrt {K} + d _ {u} \sqrt {T K}\right), \tag {24}
$$
where the only difference is the additional  $\sqrt{K}$  appeared in the first term.
Proposition 3. The regret of poLinUCB in Algorithm 1 for the above setting is upper bounded by  $\widetilde{\mathcal{O}}\left(T^{1 - \alpha}d_u^\alpha +d_u\sqrt{T}\right)$  with probability at least  $1 - \delta$  if  $T = \Omega (\log (1 / \delta))$
Proof. This proof also follows from the proof of Theorem 1. The immediate regret at each time step  $t$  is
$$
\Delta_ {t} = r _ {t, \boldsymbol {x} _ {t} ^ {*}} - r _ {t, \boldsymbol {x} _ {t}} \tag {25}
$$
Recall that the definition of  $x_{t}^{\star}$
$$
\boldsymbol {x} _ {t} ^ {\star} := \underset {\boldsymbol {x} \in D _ {t}} {\arg \max } \left\langle \boldsymbol {\theta} ^ {\star}, \boldsymbol {x} \right\rangle + \left\langle \boldsymbol {\beta} ^ {\star}, \phi^ {\star} (\boldsymbol {x}) \right\rangle . \tag {26}
$$
$$
\Delta_ {t} = \left\langle \boldsymbol {\theta} ^ {\star}, \boldsymbol {x} _ {t} ^ {\star} \right\rangle + \left\langle \boldsymbol {\beta} ^ {\star}, \phi^ {\star} \left(\boldsymbol {x} _ {t} ^ {\star}\right) \right\rangle - \left\langle \boldsymbol {\theta} ^ {\star}, \boldsymbol {x} _ {t} \right\rangle - \left\langle \boldsymbol {\beta} ^ {\star}, \phi^ {\star} \left(\boldsymbol {x} _ {t}\right) \right\rangle \tag {27}
$$
$$
\Delta_ {t} \leq \langle \widetilde {\boldsymbol {\theta}} _ {t}, \boldsymbol {x} _ {t} \rangle + \langle \widetilde {\boldsymbol {\beta}} _ {t}, \widetilde {\phi} _ {t} ^ {\star} (\boldsymbol {x} _ {t}) \rangle - \langle \boldsymbol {\theta} ^ {\star}, \boldsymbol {x} _ {t} \rangle - \langle \boldsymbol {\beta} ^ {\star}, \phi^ {\star} (\boldsymbol {x} _ {t}) \rangle . \tag {28}
$$
$$
\Delta_ {t} \leq \underbrace {\left\langle \widetilde {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta} ^ {\star} , \boldsymbol {x} _ {t} \right\rangle + \left\langle \widetilde {\boldsymbol {\beta}} _ {t} - \boldsymbol {\beta} ^ {\star} , \phi^ {\star} \left(\boldsymbol {x} _ {t}\right) \right\rangle} _ {①} + \underbrace {\left\langle \widetilde {\boldsymbol {\beta}} _ {t} , \widetilde {\phi} _ {t} \left(\boldsymbol {x} _ {t}\right) - \phi^ {\star} \left(\boldsymbol {x} _ {t}\right) \right\rangle} _ {②} \tag {29}
$$
Since we only need to fit a single  $\theta^{\star},\beta^{\star}$  and  $\phi^{\star}(\cdot)$ . We thus have the following bound for the total regret,
$$
\widetilde {\mathcal {O}} \left(T ^ {1 - \alpha} d _ {u} ^ {\alpha} + d _ {u} \sqrt {T}\right). \tag {30}
$$
C Technical Lemmas
Lemma 3 (Confidence Ellipsoid, based on Theorem 2 of (Abbasi-Yadkori et al., 2011)). Let  $\pmb{w}^{\star} \in \mathbb{R}^{d}$ ,  $\pmb{V}_{0} = \lambda \pmb{I}$ ,  $\lambda > 0$ . For any  $t \geq 0$ , let  $\pmb{u}_{1}, \dots, \pmb{u}_{t} \in \mathbb{R}^{d}$ , define  $r_{t} = \langle \pmb{u}_{t}, \pmb{w}^{\star} \rangle + \eta_{t}$  where  $\eta_{t}$  is  $R_{\eta}$ -sub-Gaussian and assume that  $\| \pmb{w}^{\star} \|_{2} \leq L_{\pmb{w}}$ ; let  $\pmb{V}_{t} = \pmb{V}_{0} + \sum_{s=1}^{t} \pmb{u}_{s} \pmb{u}_{s}^{\top}$  and  $\hat{\pmb{w}}_{t}$  be the corresponding regularised least-square estimator. Then, for any  $\delta > 0$  and  $t \geq 0$ , with probability at least  $1 - \delta$ ,  $\pmb{w}^{\star}$  lies in the set:
$$
\mathcal {C} _ {t} = \left\{\boldsymbol {w} \in \mathbb {R} ^ {d}: \| \hat {\boldsymbol {w}} _ {t} - \boldsymbol {w} \| _ {\boldsymbol {V} _ {t}} \leq \sqrt {\lambda} L _ {\boldsymbol {w}} + R _ {\eta} \sqrt {2 \log \left(\frac {\det \left(\boldsymbol {V} _ {t}\right) ^ {1 / 2} \det (\lambda \boldsymbol {I}) ^ {- 1 / 2}}{\delta}\right)} \right\}. \tag {31}
$$
Furthermore, if for all  $t \geq 1$ ,  $\| \pmb{u}_t \| \leq L_{\pmb{u}}$ , then for any  $\delta > 0$  and  $t \geq 0$ , with probability at least  $1 - \delta$ ,  $\pmb{w}^{\star}$  lies in the set:
$$
\mathcal {C} _ {t} = \left\{\boldsymbol {w} \in \mathbb {R} ^ {d}: \| \hat {\boldsymbol {w}} _ {t} - \boldsymbol {w} \| _ {\boldsymbol {V} _ {t}} \leq \sqrt {\lambda} L _ {\boldsymbol {w}} + R _ {\eta} \sqrt {d \log \left(\frac {1 + t L _ {\boldsymbol {u}} ^ {2} / \lambda}{\delta}\right)} \right\}. \tag {32}
$$
Lemma 4 (Bernstein's Inequality for Matrices, Theorem 6.1.1 of (Tropp et al., 2015)). Let  $\mathbf{X}_1,\dots ,\mathbf{X}_n\in \mathbb{R}^{d_1\times d_2}$  be independent and centered random matrices. Assume that for each  $i\in [n]$ ,  $\mathbf{X}_i$  is uniformly bounded, that is:
$$
\mathbb {E} \left[ \boldsymbol {X} _ {i} \right] = \mathbf {0} \quad a n d \quad \| \boldsymbol {X} _ {i} \| \leq B, \tag {33}
$$
where  $\| \cdot \|$  denotes the spectral-norm distance here. Introduce the sum
$$
\boldsymbol {Z} = \sum_ {i = 1} ^ {n} X _ {i}, \tag {34}
$$
and let  $\mathbb{V}(Z)$  denote the matrix variance statistics of the sum  $Z$ :
$$
\begin{array}{l} \mathbb {V} (\boldsymbol {Z}) = \max  \left\{\| \mathbb {E} \left[ \boldsymbol {Z} \boldsymbol {Z} ^ {*} \right] \|, \| \mathbb {E} \left[ \boldsymbol {Z} ^ {*} \boldsymbol {Z} \right] \| \right\} (35) \\ = \max  \left\{\left\| \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} \boldsymbol {X} _ {i} ^ {*} \right\|, \left\| \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} ^ {*} \boldsymbol {X} _ {i} \right\| \right\}, (36) \\ \end{array}
$$
where the asterisk \* denotes the conjugate transpose operation. Then, for every  $\epsilon \geq 0$  we have,
$$
\mathbb {P} \left(\| Z \| \geq \epsilon\right) \leq \left(d _ {1} + d _ {2}\right) \cdot \exp \left(\frac {- \epsilon^ {2} / 2}{\mathbb {V} (Z) + B \epsilon / 3}\right). \tag {37}
$$
Figure 3: Comparison of convergence under different  $\phi$  functions (e.g., linear and those in Holder space).
Figure 4: Comparison under the setup where the reward's dependency on the post-serving context is noiseless.
D Additional Results and Discussions
D.1 Additional Experiments
We further investigate the case where the true reward is  $\langle \phi^{\star}(\pmb {x}),\beta \rangle$  . For this case, the reward's dependency on the post-serving context is noiseless. The results are presented in Figure 4. We observe that the algorithm adapted from Wang et al. (2016) still performs worse than our method, though the gap seems become smaller.
D.2 Additional Discussions on Assumption 1
Our regret analysis can accommodate different values of  $\alpha$  in Assumption 1. The rate of  $1 / \sqrt{T}$  (when  $\alpha = 0.5$ ) is a commonly observed rate for many classical machine learning algorithms, including linear regression, logistic regression, and SVM with a linear kernel. This rate is rooted in the law of large numbers and the central limit theorem. For smaller  $\alpha$  values, the generalization error will converge more slowly than  $1 / \sqrt{T}$ , indicating that the learning problem becomes increasingly difficult. In the extreme case when  $\alpha = 0$ ,  $\phi^{\star}$  function cannot be learned accurately, thus we will inevitably suffer a linear regret (simply due to model misspecification).