SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stephen Pasteris
The Alan Turing Institute London UK
spasteris@turing.ac.uk
c.hicks@turing.ac.uk
Vasilios Mavroudis
vmavroudis@turing.ac.uk
In this paper we adapt the nearest neighbour rule to the contextual bandit problem. Our algorithm handles the fully adversarial setting in which no assumptions at all are made about the data-generation process. When combined with a sufficiently fast data-structure for (perhaps approximate) adaptive nearest neighbour search, such as a navigating net, our algorithm is extremely efficient - having a per trial running time polylogarithmic in both the number of trials and actions, and taking only quasi-linear space. We give generic regret bounds for our algorithm and further analyse them when applied to the stochastic bandit problem in euclidean space. We note that our algorithm can also be applied to the online classification problem.
In this paper we adapt the classic nearest neighbour rule to the contextual bandit problem and develop an extremely efficient algorithm. The problem proceeds in trials, where on trial  $t$ : (1) a context  $x_{t}$  is revealed to us, (2) we must select an action  $a_{t}$ , and (3) the loss  $\ell_{t,a_t} \in [0,1]$  of action  $a_{t}$  on trial  $t$  is revealed to us. We assume that the contexts are points in a metric space and the distance between two contexts represents their similarity. A policy is a mapping from contexts to actions and the inductive bias of our algorithm is towards learning policies that typically map similar contexts to similar actions. Our main result has absolutely no assumptions whatsoever about the generation of the context/loss sequence and has no restriction on what policies we can compare our algorithm to.
Our algorithm requires, as a subroutine, a data-structure that performs  $c$ -nearest neighbour search. This data-structure must be adaptive - in that new contexts can be inserted into it over time. An example of such a data-structure is the Navigating net [17] which, given mild conditions on our metric and dataset, performs both search and insertion in polylogarithmic time. When utilising a data-structure of this speed our algorithm is extremely efficient - with a per-trial time complexity polylogarithmic in both the number of trials and actions, and requiring only quasi-linear space.
As an example we will apply our methodology to the special case of the contextual bandit problem in which the context sequence is drawn i.i.d. from a probability distribution over the  $d$ -dimensional hypercube. In this case, for any policy  $\hat{y}$  with a finite-volume decision boundary, our algorithm achieves  $\tilde{\mathcal{O}}\left(\hat{\alpha} T^{d/(d+1)} K^{1/(d+1)}\right)$  regret w.r.t.  $\hat{y}$ , where  $\hat{\alpha}$  measures the magnitude of what is essentially the part of the decision boundary of  $\hat{y}$  that lies in the support of the probability distribution.
In the course of this paper we develop some novel algorithmic techniques, including a new algorithmic framework CANPROP and efficient algorithms for searching in trees, which may find further application.
We now describe related works. The bandit problem [20] was first introduced in [28] but was originally studied in the stochastic setting in which all losses are drawn i.i.d. at random [18], [1], [3]. However, our world is very often not i.i.d. stochastic. The work of [4] introduced the seminal Exp3 algorithm which handled the case in which the losses were selected arbitrarily. This work also
introduced the EXP4 algorithm for contextual bandits. In general this algorithm is exponential time but in some situations can be implemented in polynomial time - such as their EXP3.S algorithm, which was a bandit version of the classic FIXEDSHARE algorithm [14]. In [12] the EXP3.S setting was greatly generalised to the situation in which the contexts where vertices of a graph. They utilised the methodology of [8], [16] and [13] in order to develop extremely efficient algorithms. Although inspiring this work, these algorithms cannot be utilised in our situation as they inherently require the set of queried contexts to be known a-priori. In the stochastic case another class of contextual bandit problems are linear bandits [21], [5] in which the contexts are mappings from the actions into  $\mathbb{R}^d$ . Here the queried contexts need not be known in advance but the losses must be drawn i.i.d. from a distribution that has mean linear in the respective context. The  $k$  nearest neighbour algorithm was first analysed in [6]. The work [26] utilised the  $k$  nearest neighbour methodology and the works [9] and [18] to handle a stochastic contextual bandit problem. However, their setting is extremely more restricted than ours. In particular, the context/loss pairs must be drawn i.i.d. at random and the probability distribution they are sampled from must obey certain strict conditions. In addition, on each trial the contexts seen so far must be ordered in increasing distance from the current context and operations must be performed on this sequence, making their algorithm exponentially slower than ours. Our algorithm utilises the works of [22] and [7] as subroutines. It should be noted that the later work, which was based on [23], was improved on in [11] - we leave it as an open problem as to whether we can utilise their work in our algorithm.
Algorithms for contextual bandits in metric spaces have been studied in [15, 19, 24, 25, 27, 29, 30, 31, 2, 26] but as far as we know ours is the first work to give a non-trivial bound for our problem in general (with no additional assumptions). As far as we are aware our above example bound for the fully stochastic case in euclidean space (with finite decision boundary) is also novel - the works [30, 24, 26] scale as  $\tilde{\mathcal{O}}(T^{(d+1)/(d+2)})$  in general (and [24, 26] also require additional assumptions). As far as we are aware we are also the first work, stochastic or otherwise, to give a regret scaling as  $\tilde{\mathcal{O}}(T^{1/2})$  when the contexts are drawn from well-separated clusters (i.e. there is a positive distance between all pairs of clusters) in a finite-dimensional metric space, and the comparator policy is constant on each cluster (we do, however, conjecture that in the stochastic case [26] can obtain such a regret scaling here - but, as stated above, it is exponentially slower).
Let  $\mathbb{N}$  be the set of natural numbers not including 0. Given a natural number  $m\in \mathbb{N}$  we define  $[m]\coloneqq \{j\in \mathbb{N}\mid j\leq m\}$ . Given a predicate  $p$  we define  $\llbracket p\rrbracket := 1$  if  $p$  is true and  $\llbracket p\rrbracket := 0$  otherwise. We define  $\log (\cdot)$  and  $\ln (\cdot)$  to be the logarithms with base 2 and  $e$  respectively. Given sets  $\mathcal{A}$  and  $\mathcal{B}$  we denote by  $\mathcal{B}^{\mathcal{A}}$  the set of all functions  $f:\mathcal{A}\to \mathcal{B}$  and by  $2^{\mathcal{A}}$  the set of all subsets of  $\mathcal{A}$ . We write  $x\sim [0,1]$  to mean that the value  $x$  is drawn from the uniform distribution on  $[0,1]$ .
All trees in this paper are considered rooted. Given a tree  $\mathcal{J}$  we denote its root by  $r(\mathcal{J})$ , its vertex set by  $\mathcal{J}$ , its leaves by  $\mathcal{J}^{\star}$ , and its internal vertices by  $\mathcal{J}^{\dagger}$ . Given a vertex  $v$  in a tree  $\mathcal{J}$  we denote its parent by  $\uparrow_{\mathcal{J}}(v)$  and the subtree of all its descendants by  $\Downarrow_{\mathcal{J}}(v)$ . Given an internal node  $v$  in a (full) binary tree  $\mathcal{J}$  we denote its left and right children by  $\triangleleft_{\mathcal{J}}(v)$  and  $\triangleright_{\mathcal{J}}(v)$  respectively. Internal nodes  $v$  in a (full) ternary tree  $\mathcal{J}$  have an additional child  $\nabla_{\mathcal{J}}(v)$  called the centre child. Given vertices  $v$  and  $v'$  in a tree  $\mathcal{J}$  we denote by  $\Gamma_{\mathcal{J}}(v, v')$  the least common ancestor of  $v$  and  $v'$ : i.e. the vertex of maximum depth which is an ancestor of both  $v$  and  $v'$ . We will drop the subscript  $\mathcal{J}$  in all these functions when unambiguous. Given a tree  $\mathcal{J}$ , a subtree of  $\mathcal{J}$  is a tree whose edge set is a subset of that of  $\mathcal{J}$ .
3.1 The General Result
We consider the following game between Learner (us) and Nature (our adversary). We call this game the similarity bandit problem. We have  $K$  actions. Learning proceeds in  $T$  trials. A-priori Nature chooses a sequence  $\langle n(t) \mid t \in [T] \setminus \{1\} \rangle$  where for all  $t \in [T] \setminus \{1\}$  we have  $n(t) \in [t - 1]$ . A-priori Nature also chooses a sequence of loss vectors  $\langle \ell_t \mid t \in [T] \rangle \subseteq [0,1]^K$ , but does not reveal them to Learner. On the  $t$ -th trial the following happens:
1. If  $t > 1$  then Nature reveals  $n(t)$  to Learner.
2. Learner chooses some action  $a_{t}\in [K]$
3. Nature reveals  $\ell_{t,a_t}$  to Learner.
Intuitively, given a trial  $t \in [T] \setminus \{1\}$ ,  $n(t)$  is a similar trial to  $t$ . Our inductive bias is that if an action  $a \in [K]$  is good for trial  $t$  then it is likely that  $a$  will also be good for the similar trial  $n(t)$ . We will measure our performance with respect to any policy  $y$ , where a policy is defined as a vector in  $[K]^T$ . Specifically, we wish to minimise the  $y$ -regret, which is defined as the difference between the total cumulative loss suffered by Learner and that which Learner would have suffered if it had instead chosen  $a_t$  equal to  $y_t$  for all trials  $t$ . Formally, this quantity is defined as follows:
Definition 3.1. Given a policy  $\mathbf{y} \in [K]^T$  we define the  $\mathbf{y}$ -regret of Learner as:
$$
R (\boldsymbol {y}) := \sum_ {t \in [ T ]} \ell_ {t, a _ {t}} - \sum_ {t \in [ T ]} \ell_ {t, y _ {t}}.
$$
The following quantity quantifies how much a policy agrees with our inductive bias.
Definition 3.2. Given a policy  $\pmb{y} \in [K]^T$  we define the complexity of  $\pmb{y}$  by:
$$
\Phi (\boldsymbol {y}):= 1 + \sum_ {t\in [T]\backslash \{1\}} \llbracket y _ {t}\neq y _ {n (t)}\rrbracket .
$$
We now state our main result:
Theorem 3.3. Consider the similarity bandit problem described above. Our algorithm CBNN takes a single parameter  $\rho >0$  and, for all policies  $\mathbf{y}\in [K]^T$  simultaneously, obtains an expected  $\mathbf{y}$ -regret bounded by:
$$
\mathbb {E} [ R (\boldsymbol {y}) ] \in \tilde {\mathcal {O}} \left(\left(\rho + \frac {\Phi (\boldsymbol {y})}{\rho}\right) \sqrt {K T}\right)
$$
where the expectation is taken over the randomisation of the algorithm. CBNN needs no initialisation time and has a per-trial time complexity of:
$$
\mathcal {O} (\ln (T) ^ {2} \ln (K)).
$$
3.2 Bandits in a Metric Space
We consider the following game between Learner (us) and Nature (our adversary). We call this game the metric bandit problem. We have  $K$  actions and a metric space  $(\mathcal{C}, \Delta)$  where  $\mathcal{C}$  is a (possibly infinite) set of contexts and for all  $x, x' \in \mathcal{C}$  we have that  $\Delta(x, x')$  is the distance from  $x$  to  $x'$ . We assume that Learner does not necessarily know  $(\mathcal{C}, \Delta)$ -priors but has access to an oracle for computing  $\Delta(x, x')$  for any  $x, x' \in \mathcal{C}$ . Learning proceeds in  $T$  trials. A-priors Nature chooses a sequence of contexts  $\langle x_t | t \in [T] \rangle \subseteq \mathcal{C}$  and a sequence of loss vectors  $\langle \ell_t | t \in [T] \rangle \subseteq [0, 1]^K$ , but does not reveal them to Learner. On the  $t$ -th trial the following happens:
1. Nature reveals  $x_{t}$  to Learner.
2. Learner chooses some action  $a_{t} \in [K]$ .
3. Nature reveals  $\ell_{t,a_t}$  to Learner.
Here, our inductive bias is that if an action  $a \in [K]$  is good for a context  $x \in \mathcal{C}$  then it is likely also good for contexts that are near to  $x$  with respect to the metric  $\Delta$ . Our algorithm for this problem will be based on the concept of a  $c$ -nearest neighbour which is defined as follows.
Definition 3.4. Given some  $c \geq 1$ , a finite set  $S \subseteq C$ , and a context  $x \in C$  we have that some  $\hat{x} \in S$  is a  $c$ -nearest neighbour of  $x$  in the set  $S$  if and only if:
$$
\Delta (x, \hat {x}) \leq c \min  _ {x ^ {\prime} \in \mathcal {S}} \Delta (x, x ^ {\prime}).
$$
In order to utilise CBNN for this problem we need a data-structure for adaptive nearest neighbour search. This problem is as follows. We maintain a finite set  $S \subseteq \mathcal{C}$ . At any point in time we must either:
- Insert a new context  $x \in  \mathcal{C}$  into the set  $S$  and update the data-structure.
- Given a context  $x \in \mathcal{C}$ , utilise the data-structure to find a  $c$ -nearest neighbour of  $x$  in the set  $S$ .
An efficient example of such a data-structure is the navigating net [17].
We can now reduce the metric bandit problem to the similarity bandit problem as follows. On any trial  $t \in [T] \setminus \{1\}$  choose  $\hat{x}_t$  to be a  $c$ -nearest neighbour of  $x_t$  in the set  $\{x_{t'} \mid t' \in [t - 1]\}$ . Then choose  $n(t) \in [t - 1]$  such that  $x_{n(t)} = \hat{x}_t$ .
We will utilise the following definition in order to bound the complexity of policies when  $n(\cdot)$  is chosen in this way.
Definition 3.5. Given a function  $\hat{y}:\mathcal{C}\to [K]$  and a set  $\mathcal{X}\subseteq \mathcal{C}$  then for all  $x\in \mathcal{C}$  define:
$$
\gamma (x, \hat {y}, \mathcal {X}) := \min  \left\{\Delta \left(x, x ^ {\prime}\right) \mid x ^ {\prime} \in \mathcal {X} \wedge \hat {y} \left(x ^ {\prime}\right) \neq \hat {y} (x) \right\}.
$$
We can now bound the complexity of policies as follows, noting that by Theorem 3.3 this leads directly to a regret bound.
Theorem 3.6. Assume we have a sequence  $\langle x_t | t \in [T] \rangle \subseteq \mathcal{C}$  and a function  $\hat{y} : \mathcal{C} \to [K]$ . Define  $\mathcal{X} := \{x_t | t \in [T]\}$  and for all  $t \in [T]$  define  $y_t := \hat{y}(x_t)$ . Assume that for all  $t \in [T] \setminus \{1\}$  we have that  $x_{n(t)}$  is a  $c$ -nearest neighbour of  $x_t$  in the set  $\{x_{t'} | t' \in [t - 1]\}$ . Then  $\Phi(\mathbf{y})$  is no greater than the minimum cardinality of any set  $\mathcal{S} \subseteq \mathcal{C}$  in which for all  $x' \in \mathcal{X}$  there exists  $x \in \mathcal{S}$  with  $\Delta(x, x') < \gamma(x, \hat{y}, \mathcal{X}) / 3c$ .
A strength of our algorithm is that it can be combined with binning algorithms, where the contexts  $x_{t}$  are partitioned into sets called bins and, for each bin, all contexts in that bin are replaced by a single context called the centre of the bin. The advantage of binning is that  $\gamma(x, \hat{y}, \mathcal{X})$  can increase, so that by Theorem 3.6 we may have that  $\Phi(\boldsymbol{y})$  decreases. However, when a context  $x_{t}$  is binned (i.e. replaced by its bin centre) its label  $\hat{y}(x_{t})$  can change, increasing the final regret by  $\mathcal{O}(1)$ . In Section 3.3 we give an example of the utilisation of binning.
3.3 Stochastic Bandits in Euclidean Space
As an example we now consider the utilisation of the above algorithms for the problem of stochastic bandits in  $[0,1]^d$  for some arbitrary  $d\in \mathbb{N}$  which we view as a constant in our bounds. Here we will only focus on what happens in the limit  $T\to \infty$ . We focus on stochastic bandits for simplicity, but the same methodology can be used to study the limiting behaviour of adversarial bandits. In this problem we have an unknown probability density  $\tilde{\mu} :[0,1]^d\times [0,1]^K\to \mathbb{R}$ . We have  $T$  trials. On trial  $t$  the following happens:
1. Nature draws  $(z_{t},\ell_{t})$  from  $\tilde{\mu}$
2. Nature reveals  $z_{t}$  to Learner.
3. Learner chooses some action  $a_{t}\in [K]$
4. Nature reveals  $\ell_{t,a_t}$  to Learner.
To aid us in this problem we will first quantise the contexts  $z_{t}$  to a grid. Note that this is an example of binning. The grid is defined as follows:
Definition 3.7. Given  $q \in \mathbb{N}$  define  $\mathcal{G}_q^d$  to be the set of vectors in  $[0,1]^d$  in which each component is an integer multiple of  $1/q$ .
On each trial  $t$  we will first quantise the vector  $z_{t}$  by defining  $x_{t}$  to be its nearest neighbour (w.r.t. the euclidean metric) in  $\mathcal{G}_q^d$ , where  $q \coloneqq (T / K)^{1 / (d + 1)}$ . Note that this can be done in constant time per trial. As in Section 3.2 we then use CBNN to solve the problem by defining  $n(t)$  such that  $x_{n(t)}$  is a  $c$ -nearest neighbour (w.r.t. the euclidean metric) of  $x_{t}$  in the set  $\{x_{t'} \mid t' \in [t - 1]\}$ . We consider  $c$  as a constant in our bounds.
As before, we will compare our cumulative loss to that of a policy that follows a function  $\hat{y} : [0,1]^d \to [K]$ . Our regret bound will be based on the following quantities:
Definition 3.8. Let  $\mu$  be the marginal of  $\tilde{\mu}$  with respect to its first argument and let  $\Delta$  be the euclidean metric on  $[0,1]^d$ . For all  $\epsilon > 0$  define:
$$
\mathcal {E} (\mu , \epsilon) := \{x \in [ 0, 1 ] ^ {d} | \exists x ^ {\prime} \in [ 0, 1 ] ^ {d}: \mu (x ^ {\prime}) \neq 0 \land \Delta (x, x ^ {\prime}) \leq \epsilon \}
$$
which is the set of contexts that are within distance  $\epsilon$  of the support of  $\mu$ . Given  $\hat{y} : [0,1]^d \to [K]$  we make the following definitions. For any  $\delta > 0$  define:
$$
\mathcal {M} (\hat {y}, \mu , \epsilon , \delta) := \{x \in [ 0, 1 ] ^ {d} \mid \exists x ^ {\prime} \in \mathcal {E} (\mu , \epsilon): \Delta (x, x ^ {\prime}) \leq \delta \wedge \hat {y} (x) \neq \hat {y} (x ^ {\prime}) \}
$$
which is the set of contexts that are at distance no more than  $\delta$  from the intersection of the decision boundary of  $\hat{y}$  and  $\mathcal{E}(\mu, \epsilon)$ . We then define:
$$
\alpha (\hat {y}, \mu) := \lim  _ {\epsilon \rightarrow 0} \lim  _ {\delta \rightarrow 0} \frac {1}{\delta} \int_ {x \in \mathcal {M} (\hat {y}, \mu , \epsilon , \delta)} 1; \quad \tilde {\alpha} (\hat {y}, \mu) := \lim  _ {\epsilon \rightarrow 0} \lim  _ {\delta \rightarrow 0} \frac {1}{\delta} \int_ {x \in \mathcal {M} (\hat {y}, \mu , \epsilon , \delta)} \mu (x)
$$
which are essentially the volumes of the part of the decision boundary of  $\hat{y}$  that lies in the support of  $\mu$ , with respect to the uniform density and the density  $\mu$  respectively.
With these definitions in hand we now present our regret bound, which utilises Theorem 3.6 in its proof:
Theorem 3.9. Let  $q \coloneqq \lceil (T / K)^{1 / (d + 1)} \rceil$ . For all  $t \in [T]$  let  $x_{t}$  be the nearest neighbour of  $z_{t}$  in the set  $\mathcal{G}_q^d$ . For all  $t \in [T] \setminus \{1\}$  let  $n(t)$  be such that  $x_{n(t)}$  is a c-nearest neighbour of  $x_{t}$  in the set  $\{x_{t'} \mid t' \in [t - 1]\}$ . Given some  $\hat{y} : [0,1]^d \to [K]$  let  $y_{t} \coloneqq \hat{y}(z_{t})$  for all  $t \in [T]$ . Then when  $\rho \coloneqq q^{\frac{d - 1}{2}}$  CBNN gives us:
$$
\mathbb {E} [ R (\boldsymbol {y}) ] \in \tilde {\mathcal {O}} \left((1 + \alpha (\hat {y}, \mu) + \tilde {\alpha} (\hat {y}, \mu)) T ^ {\frac {d}{d + 1}} K ^ {\frac {1}{d + 1}}\right)
$$
as  $T\to \infty$
We note that varying  $q$  and  $\rho$  in Theorem 3.9 will allow us to trade off the values 1,  $\alpha(\hat{y}, \mu)$  and  $\tilde{\alpha}(\hat{y}, \mu)$  in different ways.
In this section we describe our algorithm CBNN, for solving the similarity bandit problem, and give the pseudocode for the novel subroutines. In the appendix we give a more detailed description of how CBNN works and prove our theorems.
Instead of working directly with trial numbers we create a sequence of distinct nodes  $\langle x_t \mid t \in [T] \rangle$  and define, for all  $t \in [T] \setminus \{1\}$ , the node  $n(x_t) := x_{n(t)}$ . Let  $\mathcal{X} := \{x_t \mid t \in [T]\}$ . We can now represent policies as functions from  $\mathcal{X}$  into  $[K]$ . Hence, given some  $y: \mathcal{X} \to [K]$ , we define the  $y$ -regret and the complexity of  $y$  as:
$$
R (y) := \sum_ {t \in [ T ]} \ell_ {t, a _ {t}} - \sum_ {t \in [ T ]} \ell_ {t, y (x _ {t})} \quad ; \quad \Phi (y) := 1 + \sum_ {x \in \mathcal {X} \backslash \{x _ {1} \}} [   [ y (x) \neq y (n (x)) ]   ]
$$
4.1 A Simple but Inefficient Algorithm
To give the reader intuition we first describe our initial idea - a simple algorithm which attains our desired regret bound but is exponentially slower - taking a per-trial time of  $\tilde{\Theta} (KT)$ . The algorithm is based on Exp4 [4] which we now describe. On every trial  $t$  we maintain a weighting  $\hat{w}_t:[K]^{\mathcal{X}}\to [0,1]$ . We are free to choose the initial weighting  $\hat{w}_{1}$  to be any probability distribution. On each trial  $t$  the following happens:
1. For all  $a \in [K]$  set  $p_{t,a} \gets \sum_{y \in [K]^{\mathcal{X}}} [[y(x_t) = a]] \hat{w}_t(y)$ .
2. Set  $a_{t} \gets a$  with probability proportional to  $p_{t,a}$ .
3. Receive  $\ell_{t,a_t}$ .
4. For all  $a\in [K]$  set  $\hat{\ell}_{t,a}\gets [[a = a_t]]\ell_{t,a_t}\| \pmb {p}_t\| _1 / p_{t,a_t}$
5. For all  $y \in [K]^{\mathcal{X}}$  set  $\hat{w}_{t+1}(y) \gets \hat{w}_t(y) \exp(-\eta \hat{\ell}_{t,y(x_t)})$ .
For us we choose, for all  $y: \mathcal{X} \to [K]$ , an initial weight of:
$$
\hat {w} _ {1} (y) := (1 / K) (T (K - 1)) ^ {- \Phi (y)} (1 - 1 / T) ^ {(T - 1 - \Phi (y))}.
$$
Of course, we don't know  $\Phi(y)$  a-priori, and hence we cannot implement Exp4 explicitly (and it would take exponential time even if we did know  $\Phi(y)$  a-priori). Our crucial insight is the following. For any  $t \in [T]$  let  $\mathcal{X}_t \coloneqq \{x_{t'} \mid t' \in [t]\}$  and for any  $y: \mathcal{X} \to [K]$  let  $y^t$  be the restriction of  $y$  onto  $\mathcal{X}_t$ . Then for any  $t \in [T]$  and for any  $y': \mathcal{X}_t \to [K]$  we have:
$$
\sum_ {y \in [ K ] ^ {\mathcal {X}}: y ^ {t} = y ^ {\prime}} \hat {w} _ {1} (y) \propto \prod_ {x \in \mathcal {X} _ {t} \backslash \{x _ {1} \}} \left(\frac {\llbracket y ^ {\prime} (n (x)) \neq y ^ {\prime} (x) \rrbracket}{T (K - 1)} + \llbracket y ^ {\prime} (n (x)) = y ^ {\prime} (x) \rrbracket \left(1 - \frac {1}{T}\right)\right). \tag {1}
$$
Note that, for all  $t\in [T]$  and  $a\in [K]$ , we can write  $p_{t,a}$  as follows:
$$
\begin{array}{l} p _ {t, a} = \sum_ {y \in [ K ] ^ {\mathcal {X}}} \llbracket y (x _ {t}) = a \rrbracket \hat {w} _ {1} (y) \prod_ {t ^ {\prime} \in [ t - 1 ]} \exp (- \eta \hat {\ell} _ {t, y (x _ {t})}) \\ = \sum_ {y ^ {\prime} \in [ K ] ^ {\mathcal {X} _ {t}}} \llbracket y ^ {\prime} (x _ {t}) = a \rrbracket \left(\prod_ {t ^ {\prime} \in [ t - 1 ]} \exp (- \eta \hat {\ell} _ {t, y ^ {\prime} (x _ {t})})\right) \sum_ {y \in [ K ] ^ {\mathcal {X}}: y ^ {t} = y ^ {\prime}} \hat {w} _ {1} (y). \\ \end{array}
$$
By substituting in Equation (1) we have now brought  $p_{t,a}$  into a form that can be solved via Belief propagation [23] over the tree with vertex set  $\mathcal{X}_t$  and in which, for all  $t' \in [t] \setminus \{1\}$ , the parent of  $x_{t'}$  is  $n(x_{t'})$ .
It is well known that for any  $y:\mathcal{X}\to [K]$ ,  $\mathrm{EXP4}$  attains a  $y$ -regret of at most  $\ln (\hat{w}_1(y)) / \eta +\eta KT / 2$ . By setting  $\eta \coloneqq \rho /\sqrt{KT}$  and noting our choice of  $\hat{w}_{1}$  we obtain our desired regret bound.
4.2 Cancellation Propagation
In the remainder of this section we describe our algorithm CBNN, which is based on the same idea as the simple algorithm of Section 4.1.
In this subsection we describe a novel algorithmic framework CANPROP for designing contextual bandit algorithms with a running time logarithmic in  $K$ . It is inspired by Exp3 [4], specialist algorithms [8] and online decision-tree pruning algorithms [10] but is certainly not a simple combination of these works. CBNN will be an efficient implementation of an instance of CANPROP. Although in general CANPROP requires a-priori knowledge, CBNN is designed in a way that, crucially, does not need it to be known.