SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hamish Flynn $^{1,2}$ , David Reeb $^{1}$ , Melih Kandemir $^{3}$ , Jan Peters $^{2,4}$
<sup>1</sup>Bosch Center for Artificial Intelligence, Renningen, Germany
$^{2}$ Technische Universität Darmstadt, Germany
<sup>3</sup>University of Southern Denmark, Odense, Denmark
<sup>4</sup>Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Germany
hamish@robot-learning.de, david.reeb@de.bosch.com,
kandemir@imada.sdu.dk, jan.peters@tu-darmstadt.de
We present improved algorithms with worst-case regret guarantees for the stochastic linear bandit problem. The widely used "optimism in the face of uncertainty" principle reduces a stochastic bandit problem to the construction of a confidence sequence for the unknown reward function. The performance of the resulting bandit algorithm depends on the size of the confidence sequence, with smaller confidence sets yielding better empirical performance and stronger regret guarantees. In this work, we use a novel tail bound for adaptive martingale mixtures to construct confidence sequences which are suitable for stochastic bandits. These confidence sequences allow for efficient action selection via convex programming. We prove that a linear bandit algorithm based on our confidence sequences is guaranteed to achieve competitive worst-case regret. We show that our confidence sequences are tighter than competitors, both empirically and theoretically. Finally, we demonstrate that our tighter confidence sequences give improved performance in several hyperparameter tuning tasks.
The stochastic linear bandit problem is a sequential decision-making problem where, in each round  $t$ , a learner chooses an action  $a_{t}$  and then receives a stochastic reward  $r_{t}$  for its choice of action. The expected value of each reward is a linear function  $\phi (a_t)^\top \pmb{\theta}^*$  of a known feature vector  $\phi (a_{t})$  associated with the corresponding action, while  $\pmb{\theta}^{*}$  is unknown. The linear bandit problem has attracted a great deal of attention because it is expressive enough to be a faithful model of many real-world decision-making problems, such as news recommendation (Li et al., 2010) and dynamic pricing (Cohen et al., 2020), yet it is simple enough to make theoretical analysis tractable.
A popular way to design algorithms for linear bandits is to follow the principle of optimism in the face of uncertainty. This principle states that we should choose actions as if the expected reward function is as nice as plausibly possible. For linear bandits, the principle can be instantiated with a confidence sequence  $\Theta_0, \Theta_1, \ldots$  for the parameter vector  $\pmb{\theta}^* \in \Theta$  of the expected reward function. A confidence sequence is a sequence of subsets of the full parameter space  $\Theta$ , which is built iteratively as data becomes available and is constructed such that with high probability over the observed data,  $\pmb{\theta}^*$  is contained in each confidence set  $\Theta_t$ . One can then run an upper confidence bound (UCB) algorithm, which at round  $t$  chooses an action  $a_{t+1}$  by maximising the UCB  $\max_{\pmb{\theta} \in \Theta_t} \{\phi(a)^{\top} \pmb{\theta}\}$  with respect to  $a$ .
The popularity of UCB algorithms stems from the fact that they come with worst-case regret guarantees and often perform well in practice. However, the performance of a UCB algorithm is intimately
Figure 1: Tighter upper and lower confidence bounds via tail bounds for martingale mixtures. The upper and lower confidence bounds of CMM-UCB (left), AMM-UCB (middle), and OFUL (Abbasi-Yadkori et al., 2011) (right) for a test function linear in random Fourier features. The bounds from CMM-UCB and AMM-UCB are visibly closer to the true function (dashed line) than those of OFUL. The CMM-UCB confidence bounds are slightly tighter than the ones of AMM-UCB.
tied to the size of the confidence sets it uses. The smaller the subsets in the confidence sequence, the better the regret bound and, perhaps more importantly, the better the algorithm performs in practice.
In this work, we develop a general-purpose tail bound for martingale mixtures, which can be used to construct confidence sequences. When we specialise our general results to the linear bandit problem, the maximisation problem to compute the UCB is a convex program. We maximise the UCB over actions via gradient-based methods, and investigate two procedures for computing the UCB along with its gradient: (a) Convex Martingale Mixture UCB (CMM-UCB): We employ a convex solver for the UCB maximisation and calculate its gradients via differentiable convex optimisation (Agrawal et al., 2019); (b) Analytic Martingale Mixture UCB (AMM-UCB): We exploit weak Lagrangian duality to obtain an analytic upper bound on the UCB, whose gradient can be computed in closed-form or via standard automatic differentiation procedures.
Fig. 1 highlights a key observation: both of our UCBs are tighter than those used in the state-of-the-art OFUL algorithm (Abbasi-Yadkori et al., 2011) for stochastic linear bandits. We verify this claim empirically in Sec. 8 and prove it in App. C.2. We evaluate CMM-UCB, AMM-UCB, OFUL and various other linear bandit algorithms in several hyperparameter tuning problems (Sec. 8). We find that our tighter UCBs result in linear bandit algorithms with better performance.
Algorithms with regret guarantees have been developed for several variants of the stochastic linear bandit problem. Dani et al. (2008), Abbasi-Yadkori et al. (2009) and Rusmevichientong & Tsitsiklis (2010) proposed algorithms for a linear bandit problem where the action set is a fixed, possibly infinite subset of a finite-dimensional vector space. Auer (2002) and Chu et al. (2011) proposed algorithms for linear bandit problems where the action set has finite cardinality, but may change over time. Abbasi-Yadkori et al. (2011) proposed the OFUL algorithm for linear bandit problems with a changing and possibly infinite action set, which is essentially the problem that we investigate. We consider stochastic linear bandit problems where the reward function is a composition of a possibly non-linear feature map and a linear function. This can be seen as a restricted version of the stochastic kernelised bandit problem, where the kernel feature map is finite-dimensional. Srinivas et al. (2010); Valko et al. (2013); Chowdhury & Gopalan (2017); Camilleri et al. (2021); Salgia et al. (2021) and Li & Scarlett (2022) proposed algorithms with regret guarantees for various kernelised bandit problems.
In the bandit literature, confidence sets and confidence bounds constructed from online (e.g. non-i.i.d.) observation points for unknown linear functions have been proposed by Dani et al. (2008); Rushevichientong & Tsitsiklis (2010) and Abbasi-Yadkori et al. (2011). Online confidence sets/bounds for unknown functions in separable Hilbert spaces and reproducing kernel Hilbert spaces (RKHSs) have been proposed by Srinivas et al. (2010); Abbasi-Yadkori (2012); Kirschner & Krause (2018) and Durand et al. (2018). Russo & Van Roy (2013) derived online confidence sets for unknown functions belonging to arbitrary function classes.
We use the term "mixture of martingales", or martingale mixture, to refer to a martingale of the form  $\mathbb{E}_{v\sim P}[M_t(v)]$ , where  $(M_{t}(v)|t\in \mathbb{N})$  is a collection of martingales indexed by the variable  $v\in \mathcal{V}$ . Martingale mixtures can be traced back to (Darling & Robbins, 1968; Robbins, 1970), and have been used to construct confidence sequences since at least the work of Lai (1976). Proofs of tail bounds for martingale mixtures typically use the method of mixtures, which was first used by Robbins & Siegmund (1970) and was later popularised by de la Peña et al. (2004, 2009). Methods for martingale mixtures have seen renewed interest in the sequential testing literature (Howard et al., 2020; Kaufmann & Koolen, 2021). Examples of confidence sequences for bandits that use martingale mixtures include the works of Abbasi-Yadkori et al. (2011); Abbasi-Yadkori (2012); Kirschner & Krause (2018); Durand et al. (2018); Neiswanger & Ramdas (2021). Unlike in these examples, we construct confidence sequences based on adaptive martingale mixtures  $(\mathbb{E}_{v\sim P_t}[M_t(v)]|t\in \mathbb{N})$ , where the mixture distribution  $P_{t}$  can be refined as more data is acquired at each time  $t$ .
3 Problem Statement and Background
We consider a problem in which a learner plays a game over a sequence of  $T$  rounds, where  $T$  may not be known in advance. In each round  $t$ , the learner observes an action set  $\mathcal{A}_t$  and must choose an action  $a_t \in \mathcal{A}_t$ . The learner then receives a reward  $r_t = \phi(a_t)^\top \pmb{\theta}^* + \epsilon_t$ . The feature map  $\phi: \mathcal{A} \to \mathbb{R}^d$  is a known function that maps actions to  $d$ -dimensional feature vectors, where  $\mathcal{A} = \bigcup_{t} \mathcal{A}_t$ .  $\pmb{\theta}^* \in \mathbb{R}^d$  is an unknown parameter with Euclidean norm bounded by some known  $B > 0$ , i.e.  $\|\pmb{\theta}^*\|_2 \leq B$ .  $\epsilon_1, \epsilon_2, \dots, \epsilon_T$  are conditionally zero-mean  $\sigma$ -sub-Gaussian noise variables. These assumptions on  $\pmb{\theta}^*$  and  $\epsilon_1, \epsilon_2, \dots, \epsilon_T$  are standard in the linear bandit literature, see e.g. (Abbasi-Yadkori et al., 2011). While our regret analysis applies to any action sets, our algorithms focus on the case where the action sets  $\mathcal{A}_t$  are continuous subsets of  $\mathbb{R}^{d_A}$ .
The goal of the learner is to choose a sequence of actions that maximises the total expected reward, which is equal to  $\sum_{t=1}^{T} \phi(a_t)^\top \pmb{\theta}^*$  after  $T$  rounds. We use cumulative regret, which is the difference between the total expected reward of the learner and the optimal strategy, to evaluate the learner. For a single round, we define the regret as  $\Delta(a_t) = \phi(a_t^*)^\top \pmb{\theta}^* - \phi(a_t)^\top \pmb{\theta}^*$ , where  $a_t^* = \operatorname{argmax}_{a \in \mathcal{A}_t} \{\phi(a)^\top \pmb{\theta}^*\}$ . After  $T$  rounds, the cumulative regret is  $\Delta_{1:T} = \sum_{t=1}^{T} \Delta(a_t)$ .
Confidence Sequences. For any level  $\delta \in (0,1]$ , a  $(1 - \delta)$ -confidence sequence for the parameter vector  $\pmb{\theta}^*$  is a sequence  $\Theta_1, \Theta_2, \ldots$  of subsets of  $\mathbb{R}^d$ , such that each  $\Theta_t$  can be calculated using the data  $a_1, r_1, \ldots, a_t, r_t$  and the sequence satisfies
$$
\mathbb{P}_{\substack{a_{1},a_{2},\ldots ,\\ r_{1},r_{2},\ldots ,}}[\forall t\geq 1:\boldsymbol{\theta}^{*}\in \Theta_{t}]\geq 1 - \delta .
$$
A confidence sequence  $\Theta_1, \Theta_2, \ldots$  is thus a sequence of data-dependent confidence sets such that with high probability over the random actions and rewards,  $\pmb{\theta}^{*} \in \Theta_{t}$  holds for all  $t \geq 1$  simultaneously. We remark that the confidence sets  $\Theta_{t}$  in this paper are random closed sets in the sense of Def. 1.1.1 of (Molchanov, 2005), which implies that the event  $\pmb{\theta} \in \Theta_{t}$  is actually measurable for all  $\pmb{\theta} \in \mathbb{R}^{d}$ .
4 UCB Algorithms for Linear Bandits
We describe here how to transform confidence sets for  $\theta^{*}$  into a UCB algorithm for the linear bandit problem. Such algorithms have appeared under various names, such as LinRel (Auer, 2002), LinUCB (Li et al., 2010) and OFUL (Abbasi-Yadkori et al., 2011). We refer to this meta algorithm as LinUCB, and give its pseudo-code in Algorithm 1. When run with our confidence sets, we call this algorithm CMM-UCB resp. AMM-UCB (see Sec. 6).
In each round  $t$ , the first step is to construct a confidence set  $\Theta_t$  from the previous observations  $\{(a_k, r_k)\}_{k=1}^t$ . If  $\pmb{\theta}^{*} \in \Theta_{t}$  with high probability, then for any action  $a$ ,
$$
\mathrm {U C B} _ {\Theta_ {t}} (a) := \max  _ {\theta \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} \tag {1}
$$
is an upper confidence bound (UCB) on  $\phi(a)^{\top} \pmb{\theta}^{*}$ . Taking  $\min_{\theta \in \Theta_t}$  in (1) yields the lower confidence bound  $\mathrm{LCB}_{\Theta_t}(a)$ . Once a confidence set  $\Theta_t$  has been constructed and the next action set  $\mathcal{A}_{t+1}$  has been observed, the LinUCB algorithm chooses the action
$$
a _ {t + 1} = \underset {a \in \mathcal {A} _ {t + 1}} {\operatorname {a r g m a x}} \left\{\mathrm {U C B} _ {\Theta_ {t}} (a) \right\}, \tag {2}
$$
Algorithm 1: LinUCB
for  $t = 0,1,2,\ldots$  do Construct a confidence set  $\Theta_t$  from  $\{(a_k,r_k)\}_{k = 1}^t$  Observe next action set  $\mathcal{A}_{t + 1}$  Play next action  $a_{t + 1} = \mathrm{argmax}_{a\in \mathcal{A}_{t + 1}}\{\mathrm{UCB}_{\Theta_t}(a)\}$  Observe next reward  $r_{t + 1}$    
end
which maximises the UCB. The remaining challenge lies in the construction of the confidence sets.
5 Confidence Sequences from Martingale Mixtures
In this section, we develop a general-purpose tail bound for adaptive martingale mixtures. We then specialise our general result to the stochastic linear bandit setting, described in Sec. 3, and construct confidence sequences for the parameter  $\theta^{*}$ .
5.1 General-Purpose Tail Bound for Adaptive Martingale Mixtures
We consider a general setting in which we are given a filtration  $(\mathcal{H}_t|t\in \mathbb{N})$ , a sequence of adapted random functions  $(Z_{t}:\mathbb{R}\to \mathbb{R}|t\in \mathbb{N})$ , and a sequence of predictable random variables  $(\lambda_t|t\in \mathbb{N})$ . For  $f_{t}\in \mathbb{R}$ , we define the conditional cumulant generating function  $\psi_t(f_t,\lambda_t)$  as
$$
\psi_ {t} \left(f _ {t}, \lambda_ {t}\right) := \ln \left(\mathbb {E} \left[ \exp \left(\lambda_ {t} Z _ {t} (f _ {t})\right) \mid \mathcal {H} _ {t - 1} \right]\right),
$$
where the expectation  $\mathbb{E}$  is over  $Z_{t}(f_{t})$  and  $\lambda_t$  (although  $\lambda_t$  is non-random when conditioned on  $\mathcal{H}_{t-1}$ ). We use the shorthand  $\pmb{f}_t \coloneqq (f_1, f_2, \dots, f_t)$  and  $\pmb{\lambda}_t \coloneqq (\lambda_1, \lambda_2, \dots, \lambda_t)$ . Let
$$
M _ {t} \left(\boldsymbol {f} _ {t}, \boldsymbol {\lambda} _ {t}\right) = \exp \left(\sum_ {k = 1} ^ {t} \lambda_ {k} Z _ {k} \left(f _ {k}\right) - \psi_ {k} \left(f _ {k}, \lambda_ {k}\right)\right). \tag {3}
$$
$M_{t}(\pmb{f}_{t},\pmb{\lambda}_{t})$  is a slight generalisation of the martingale used in App. B.1 of (Russo & Van Roy, 2013). One can show that for any sequence  $(f_t|t\in \mathbb{N})$ ,  $(M_{t}(\pmb{f}_{t},\pmb{\lambda}_{t})|t\in \mathbb{N})$  is a martingale and  $\mathbb{E}[M_t(\pmb{f}_t,\pmb{\lambda}_t)] = 1$  (App. A.1). We will now construct an adaptive martingale mixture.
We call a data-dependent sequence of probability distributions  $(P_{t}|t\in \mathbb{N})$  an adaptive sequence of mixture distributions if: (a)  $P_{t}$  is a distribution over  $\pmb {f}_t\in \mathbb{R}^t$  ; (b)  $P_{t}$  is  $\mathcal{H}_{t - 1}$ -measurable; (c) the distributions are consistent in the sense that their marginals coincide, i.e.  $\int P_t(\pmb {f}_t)df_t = P_{t - 1}(\pmb {f}_{t - 1})$  for all  $t$ . These conditions on the sequence of distributions ensure that the martingale mixture  $(\mathbb{E}_{\pmb {f}_t\sim P_t}[M_t(\pmb {f}_t,\pmb {\lambda}_t)]|t\in \mathbb{N})$  is in fact a martingale. In App. A.1, we verify this and show that  $\mathbb{E}[\mathbb{E}_{\pmb {f}_t\sim P_t}[M_t(\pmb {f}_t,\pmb {\lambda}_t)]] = 1$ . From here, we can use Ville's inequality for non-negative supermartingales (Ville, 1939) to obtain our general-purpose tail bound.
Theorem 5.1 (Tail Bound for Adaptive Martingale Mixtures). For any  $\delta \in (0,1)$ , any sequence of predictable random variables  $(\lambda_t|t \in \mathbb{N})$ , and any adaptive sequence of mixture distributions  $(P_t|t \in \mathbb{N})$ , the following holds with probability at least  $1 - \delta$ :
$$
\ln \left(\underset {\boldsymbol {f} _ {t} \sim P _ {t}} {\mathbb {E}} \left[ M _ {t} (\boldsymbol {f} _ {t}, \boldsymbol {\lambda} _ {t}) \right]\right) \leq \ln (1 / \delta) \quad f o r a l l t \geq 1.
$$
We provide a proof of this result in App. A.2. Note that if each  $\psi_k(f_k, \lambda_k)$  in (3) is replaced by an upper bound on  $\psi_k(f_k, \lambda_k)$ , the statement of the theorem still holds.
Thm. 5.1 is closely related to the general-purpose anytime PAC-Bayes bound in Thm. 3.1 of (Chugg et al., 2023). The main difference is that our inequality holds for adaptive sequences of mixture distributions or priors. PAC-Bayes bounds with somewhat similar adaptive sequences of mixture distributions/priors have recently been proposed by Haddouche & Guedj (2022, 2023).
5.2 Confidence Sequences for Stochastic Linear Bandits
We now specialise Thm. 5.1 to the stochastic linear bandit setting. For the filtration  $(\mathcal{H}_t|t\in \mathbb{N})$ , we set  $\mathcal{H}_t$  to be the  $\sigma$ -algebra generated by  $(a_{1},r_{1},\ldots ,a_{t},r_{t},a_{t + 1})$ . For reasons that will become clear, we choose  $Z_{t}(f_{t}) = (f_{t} - \phi (a_{t})^{\top}\pmb{\theta}^{*})\epsilon_{t}$ . Since  $Z(f_{t})$  is linear in the noise variable  $\epsilon_t$ ,  $\psi_t(f_t,\lambda_t)$  can be upper bounded using the sub-Gaussian property of  $\epsilon_t$ . We have
$$
\psi_ {t} \left(f _ {t}, \lambda_ {t}\right) = \ln \left(\mathbb {E} \left[ \exp \left(\lambda_ {t} \left(f _ {t} - \phi \left(a _ {t}\right) ^ {\top} \boldsymbol {\theta} ^ {*}\right) \epsilon_ {t}\right) \mid \mathcal {H} _ {t - 1} \right]\right) \leq \lambda_ {t} ^ {2} \sigma^ {2} \left(f _ {t} - \phi \left(a _ {t}\right) ^ {\top} \boldsymbol {\theta} ^ {*}\right) ^ {2} / 2. \tag {4}
$$
With this upper bound on  $\psi_t(f_t, \lambda_t)$ , Thm. 5.1 implies that, with probability at least  $1 - \delta$
$$
\mathbb {E} _ {\pmb {f} _ {t} \sim P _ {t}} \left[ \exp \left\{\sum_ {k = 1} ^ {t} \lambda_ {k} (f _ {k} - \phi (a _ {k}) ^ {\top} \pmb {\theta} ^ {*}) (r _ {k} - \phi (a _ {k}) ^ {\top} \pmb {\theta} ^ {*}) - \frac {\sigma^ {2}}{2} \lambda_ {k} ^ {2} (f _ {k} - \phi (a _ {k}) ^ {\top} \pmb {\theta} ^ {*}) ^ {2} \right\} \right] \leq \frac {1}{\delta}.
$$
Since  $Z_{f}(f_{t})$  is linear in  $f_{t}$ , this integral has a closed-form solution whenever the mixture distribution is a Gaussian  $P_{t} = \mathcal{N}(\pmb{\mu}_{t},\pmb{T}_{t})$ . Although there is a closed-form solution for any predictable sequence  $(\lambda_t|t\in \mathbb{N})$  (see App. B.1), we choose  $\lambda_t\equiv 1 / \sigma^2$ , which yields a relatively simple convex quadratic constraint for  $\theta^{*}$ . Collecting the feature vectors in  $\Phi_t\coloneqq [\phi (a_1),\ldots ,\phi (a_t)]^\top \in \mathbb{R}^{t\times d}$  and writing the reward vector  $r_t\coloneqq [r_1,\dots,r_t]^\top$ , we arrive at (see App. B.2)
$$
\left\| \Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t} \right\| _ {2} ^ {2} \leq \left(\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\mathbb {1} + \frac {\boldsymbol {T} _ {t}}{\sigma^ {2}}\right) ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}\right) + \sigma^ {2} \ln \det  \left(\mathbb {1} + \frac {\boldsymbol {T} _ {t}}{\sigma^ {2}}\right) + 2 \sigma^ {2} \ln \frac {1}{\delta}. \tag {5}
$$
This inequality has an attractive interpretation. At each step  $t$  of the bandit process, the (unknown) ground-truth reward vector  $\Phi_t^\top \pmb{\theta}^*$  lies within a sphere around the observed reward vector  $\pmb{r}_t$ , with squared radius equal to the RHS of (5). One can think of the mean vector  $\pmb{\mu}_t$  as a prediction of the reward vector  $\pmb{r}_t$ , given the previous data  $a_1, r_1, \ldots, a_{t-1}, r_{t-1}, a_t$ . The covariance matrix  $\pmb{T}_t$  can be thought of as the uncertainty associated with the prediction  $\pmb{\mu}_t$ . If  $\pmb{\mu}_t$  is a good prediction of  $\pmb{r}_t$ , then the quadratic "prediction error" term in (5) will be close to 0, and we can afford to choose  $\pmb{T}_t$  close to zero to minimise the log determinant term. In this situation, (5) can give a much tighter constraint than the naive bound  $\sim t\sigma^2$ , especially when  $\sigma$  is a pessimistic upper bound on the true sub-Gaussian parameter. This naive bound follows from the observation that  $\| \Phi_t \pmb{\theta}^* - \pmb{r}_t \|_2 = \| \pmb{\epsilon}_t \|_2$ , where  $\pmb{\epsilon}_t = (\epsilon_1, \dots, \epsilon_t)$ . Combining the constraint in (5) with our assumption  $\| \pmb{\theta}^* \|_2 \leq B$  yields our confidence sequence for  $\pmb{\theta}^*$ .
Corollary 5.2 (Martingale Mixture Confidence Sequence). For any adaptive sequence of mixture distributions  $P_{t} = \mathcal{N}(\pmb{\mu}_{t},\pmb{T}_{t})$ , it holds with probability at least  $1 - \delta$  that for all  $t\geq 1$  simultaneously,  $\pmb{\theta}^{*}$  lies in the set
$$
\Theta_ {t} = \left\{\boldsymbol {\theta} \in \mathbb {R} ^ {d} \Big | \| \Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t} \| _ {2} \leq R _ {\mathrm {M M}, t} \quad \text {a n d} \| \boldsymbol {\theta} \| _ {2} \leq B \right\}, \tag {6}
$$
where we define  $R_{\mathrm{MM},t}^2$  as the right-hand-side of Eq. (5).
The boundaries of the constraints in (6) are both ellipsoids in  $\mathbb{R}^d$ , which means that each  $\Theta_t$  is the intersection of (the interiors of) two ellipsoids.
6 Martingale Mixture UCB Algorithms
In this section, we describe our CMM-UCB and AMM-UCB algorithms, which are two different implementations of LinUCB (Algorithm 1) with our confidence sequence from Corollary 5.2.
6.1 UCB Computation and Optimisation
To run the LinUCB action selection rule with our confidence sequence, we need to be able to maximise  $\mathrm{UCB}_{\Theta_t}(a)$  with respect to  $a$ . The value of the UCB at the action  $a$  is the solution of
$$
\mathrm {U C B} _ {\Theta_ {t}} (a) = \max  _ {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} \phi (a) ^ {\top} \boldsymbol {\theta} \quad \text {s . t .} \| \Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t} \| _ {2} \leq R _ {\mathrm {M M}, t} \quad \text {a n d} \quad \| \boldsymbol {\theta} \| _ {2} \leq B. \tag {7}
$$
This is a convex optimisation problem, which can be efficiently solved via convex programming. If the action sets have finite cardinality,  $\mathrm{UCB}_{\Theta_t}(a)$  can be maximised by solving (7) for each  $a \in \mathcal{A}_t$  and then comparing the solutions. If the action sets are continuous subsets of  $\mathbb{R}^{d_{\mathcal{A}}}$ , then exact maximisation of  $\mathrm{UCB}_{\Theta_t}(a)$  is (in general) infeasible. For example, when the feature map  $\phi$  is linear in  $a$ ,  $\mathrm{UCB}_{\Theta_t}(\cdot)$  is the maximum over a set of linear functions, which is a convex function of  $a$  (see Eq. (3.7) in Sec. 3.2.3 of Boyd & Vandenberghe (2004)). Since maximisation of a convex function is (in general) NP-hard, exact maximisation of  $\mathrm{UCB}_{\Theta_t}(a)$  is also (in general) NP-hard. For this reason, when the action sets are continuous subsets of  $\mathbb{R}^{d_{\mathcal{A}}}$  (and  $\phi$  is differentiable), we approximately maximise  $\mathrm{UCB}_{\Theta_t}(a)$  via gradient-based local search.
6.2 Convex Martingale Mixture UCB
Our Convex Martingale Mixture UCB (CMM-UCB) algorithm is based on computing (7) using numerical convex (conic) solvers from the CVXPY library (Diamond & Boyd, 2016; Agrawal et al., 2018). Note that (7) is already stated in a conic form, which is favourable for conic solvers (Boyd & Vandenberghe, 2004). Solving (7) numerically gives the tightest UCBs that can be obtained from our confidence sequence. To compute the gradient of  $\mathrm{UCB}_{\Theta_t}(a)$  with respect to the action  $a$ , we use recently developed methods for differentiating conic programs at their optimum (Agrawal et al., 2019), which are implemented in the cvxpylayers library.
6.3 Analytic Martingale Mixture UCB
Our Analytic Martingale Mixture UCB (AMM-UCB) algorithm uses an analytic upper bound on the solution of (7). The resulting analytic confidence bounds are looser than the numerical confidence bounds used by CMM-UCB, but are cheaper to evaluate and maximise. Theorem 6.1 states our upper bound on the solution of (7).