SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We assume, without loss of generality, that  $K$  and  $T$  are integer powers of two. CANPROP, which takes a parameter  $\eta > 0$ , works on a full, balanced binary tree  $\mathcal{B}$  with leaves  $\mathcal{B}^{\star} = [K]$ . On every trial  $t$  each pair  $(v, \mathcal{S}) \in \mathcal{B} \times 2^{\mathcal{X}}$  has a weight  $w_{t}(v, \mathcal{S}) \in [0,1]$ . These weights induce a function  $\theta_{t}: \mathcal{B} \to [0,1]$  defined by:
$$
\theta_ {t} (v) := \sum_ {\mathcal {S} \in 2 ^ {\mathcal {X}}} [   [ x _ {t} \in \mathcal {S} ]   ] w _ {t} (v, \mathcal {S}).
$$
On each trial  $t$  a root-to-leaf path  $\{v_{t,j} \mid j \in [\log(K)] \cup \{0\}\}$  is sampled such that  $v_{t,0} := r(\mathcal{B})$  and, given  $v_{t,j}$ , we have that  $v_{t,(j+1)}$  is sampled from  $\{\triangleleft(v_{t,j}), \triangleright(v_{t,j})\}$  with probability proportional to the value of  $\theta_t$  when applied to each of these vertices. The action  $a_t$  is then chosen equal to  $v_{t,\log(K)}$ . Once the loss has been observed we climb back up the root-to-leaf path, updating the function  $w_t$  to  $w_{t+1}$ .
CANPROP (at trial  $t$ ) is given in Algorithm 1. We note that if  $w_{t+1}(v, \mathcal{S})$  is not set in the pseudocode then it is defined to be equal to  $w_t(v, \mathcal{S})$ .
In Appendix B we give a general regret bound for CANPROP. For CBNN we set:
$$
\eta := \rho \sqrt {\ln (K) \ln (T) / K T}
$$
Algorithm 1 CANPROP at trial  $t$
1:  $v_{t,0}\gets r(\mathcal{B})$  17:  $a_{t}\gets v_{t,\log (K)}$    
2: for  $j = 0,1,\dots ,(\log (K) - 1)$  do 18:  $\tilde{\pi}_t\gets \prod_{j\in [\log (K)]}\pi_t(v_{t,j})$    
3: for  $v\in \{\triangleleft (v_{t,j}),\triangleright (v_{t,j})\}$  do 19:  $\psi_{t,\log (K)}\gets \exp (-\eta \ell_{t,a_t} / \tilde{\pi}_t)$    
4:  $\theta_t(v)\gets \sum_{S\in 2^X}[[x_t\in S]]w_t(v,S)$    
5: end for   
6:  $z_{t,j}\gets \theta_t(\triangleleft (v_{t,j})) + \theta_t(\triangleright (v_{t,j}))$  20: for  $j = \log (K),(\log (K) - 1),\dots ,1$  do 21:  $\psi_{t,(j - 1)}\gets 1 - (1 - \psi_{t,j})\pi_t(v_{t,j})$    
7: for  $v\in \{\triangleleft (v_{t,j}),\triangleright (v_{t,j})\}$  do 22:  $\psi_{t,j}^{\prime}\gets \psi_{t,j} / \psi_{t,j - 1}$    
8:  $\pi_t(v)\gets \dot{\theta}_t(v) / z_{t,j}$  23: if  $v_{t,j} = \triangleleft (v_{t,j - 1})$  then 24:  $\tilde{v}_{t,j}\gets \triangleright (v_{t,j - 1})$    
9: end for   
10:  $\zeta_{t,j}\sim [0,1]$  25: else   
11: if  $\zeta_{t,j}\leq \pi_t(\triangleleft (v_{t,j}))$  then 26:  $\tilde{v}_{t,j}\gets \triangleleft (v_{t,j - 1})$    
12:  $v_{t,j + 1}\gets \triangleleft (v_{t,j})$  27: end if   
13: else 28: for  $S\in 2^X:x_t\in S$  do   
14:  $v_{t,j + 1}\gets \triangleright (v_{t,j})$  29:  $w_{t + 1}(v_{t,j},S)\gets w_t(v_{t,j},S)\psi_{t,j}'$    
15: end if   
16: end for 30:  $w_{t + 1}(\tilde{v}_{t,j},S)\gets w_t(\tilde{v}_{t,j},S) / \psi_{t,j - 1}$    
31: end for   
32: end for
and for all  $(v, S) \in \mathcal{B} \times 2^{\mathcal{X}}$  we set:
$$
w _ {1} (v, \mathcal {S}) := \frac {1}{4} \prod_ {x \in \mathcal {X} \backslash \left\{x _ {1} \right\}} \left(\sigma (x, \mathcal {S}) \frac {1}{T} + (1 - \sigma (x, \mathcal {S})) \left(1 - \frac {1}{T}\right)\right) \tag {2}
$$
$$
\sigma (x, \mathcal {S}) := [   [   [ x \in \mathcal {S} ]   ] \neq [   [ n (x) \in \mathcal {S} ]   ]   ].
$$
This choice gives us the regret bound in Theorem 3.3. We note that CBNN will be implemented in such a way that  $n$  need not be known a-priori.
4.3 Ternary Search Trees
As we shall see, CBNN works by storing a binary tree  $\mathcal{A}(v)$  at each vertex  $v\in \mathcal{B}$ . In order to perform efficient operations on these trees we will utilise the rebalancing data-structure defined in [22] which here we shall call a ternary search tree (TST) due to the fact that it is a generalisation of the classic binary search tree and, as we shall show, has searching applications. However, as for binary search trees, the applications of TSTs are more than just searching: we shall also utilise them for online belief propagation.
We now define what is meant by a TST. Suppose we have a full binary tree  $\mathcal{J}$ . A TST of  $\mathcal{J}$  is a full ternary tree  $\mathcal{D}$  which satisfies the following. The vertex set of  $\mathcal{D}$  is partitioned into two sets  $\mathcal{D}^{\circ}$  and  $\mathcal{D}^{\bullet}$  where each vertex  $s\in \mathcal{D}$  is associated with a vertex  $\mu (s)\in \mathcal{J}$  and every  $s\in \mathcal{D}^{\bullet}$  is also associated with a vertex  $\mu^{\prime}(s)\in \Downarrow (\mu (s))^{\dagger}$ . In addition, each internal vertex  $s\in \mathcal{D}^{\dagger}$  is associated with a vertex  $\xi (s)\in \mathcal{J}$ . For all  $u\in \mathcal{J}$  there exists an unique leaf  $\Upsilon_{\mathcal{D}}(u)\in \mathcal{D}^{\star}$  in which  $\mu (\Upsilon_{\mathcal{D}}(u)) = u$
Essentially, each vertex  $s \in \mathcal{D}$  corresponds to a subtree  $\hat{\mathcal{J}}(s)$  of  $\mathcal{J}$  where  $\hat{\mathcal{J}}(r(\mathcal{D})) = \mathcal{J}$ . Such a vertex  $s$  is a leaf of  $\mathcal{D}$  if and only if  $|\hat{\mathcal{J}}(s)| = 1$ . For each internal vertex  $s \in \mathcal{D}^\dagger$  the subtree  $\hat{\mathcal{J}}(s)$  is split at the vertex  $\xi(s)$  into the subtrees  $\hat{\mathcal{J}}(\triangleleft(s))$ ,  $\hat{\mathcal{J}}(\nabla(s))$ , and  $\hat{\mathcal{J}}(\triangleright(s))$  corresponding to the children of  $s$ . The process continues recursively.
For completeness we now describe the rules that a TST  $\mathcal{D}$  of  $\mathcal{J}$  must satisfy. We have that  $r(\mathcal{D})\in \mathcal{D}^{\circ}$  and  $\mu (r(\mathcal{D}))\coloneqq r(\mathcal{J})$  . Each vertex  $s\in \mathcal{D}$  represents a subtree  $\hat{\mathcal{J}} (s)$  of  $\mathcal{J}$  . If  $s\in \mathcal{D}^{\circ}$  then  $\hat{\mathcal{J}} (s)\coloneqq \Downarrow (\mu (s))$  and otherwise  $\hat{\mathcal{J}} (s)$  is the set of all descendants of  $\mu (s)$  which are not proper.   
descendants of  $\mu^{\prime}(s)$  . Given that  $s\in \mathcal{D}^{\dagger}$  this subtree is split at the vertex  $\xi (s)$  where if  $s\in \mathcal{D}^{\bullet}$  we have that  $\xi (s)$  lies on the path from  $\mu (s)$  to  $\mu^{\prime}(s)$  . The children of  $s$  are then defined so that  $\hat{\mathcal{J}} (\triangleleft (s)) =$ $\hat{\mathcal{J}} (s)\cap \Downarrow (\triangleleft (\xi (s)))$  and  $\hat{\mathcal{J}} (\triangleright (s)) = \hat{\mathcal{J}} (s)\cap \Downarrow (\triangleright (\xi (s)))$  and  $\hat{\mathcal{J}} (\nabla (s)) = \hat{\mathcal{J}} (s)\setminus (\hat{\mathcal{J}} (\triangleleft (s))\cup \hat{\mathcal{J}} (\triangleright (s))).$  i.e.  $\xi (s)$  partitions the subtree  $\hat{\mathcal{J}} (s)$  into the subtrees  $\hat{\mathcal{J}} (\triangleleft (s)),\hat{\mathcal{J}} (\triangleright (s))$  , and  $\hat{\mathcal{J}} (\nabla (s))$  . This process   
continues recursively until  $|\hat{\mathcal{J}} (s)| = 1$  in which case  $S$  is a leaf of  $\mathcal{D}$
For all binary trees  $\mathcal{J}$  in our algorithm we shall maintain a TST  $\mathcal{H}(\mathcal{I})$  of  $\mathcal{J}$  with height  $\mathcal{O}(\ln(|\mathcal{I}|))$ . Such trees  $\mathcal{J}$  are dynamic in that on any trial it is possible that two vertices,  $u$  and  $u'$ , are added to the tree  $\mathcal{J}$  such that  $u'$  is inserted between a non-root vertex of  $\mathcal{J}$  and its parent, and  $u$  is designated as a child of  $u'$ . We define the subroutine  $\text{REBALANCE}(\mathcal{H}(\mathcal{I}), u)$  as one which rebalances the TST  $\mathcal{H}(\mathcal{I})$  after this insertion, so that the height of  $\mathcal{H}(\mathcal{I})$  always remains in  $\mathcal{O}(\ln(|\mathcal{I}|))$ . The work of [22] describes how this subroutine can be implemented in a time of  $\mathcal{O}(\ln(|\mathcal{I}|))$  and we refer the reader to this work for details (noting that they use different notation).
4.4 Contractions
Define the quantity  $\phi_0\coloneqq 0$  and for all  $j\in \mathbb{N}\cup \{0\}$  inductively define:
$$
\phi_ {j + 1} := \left(1 - \frac {1}{T}\right) \phi_ {j} + \frac {1}{T} (1 - \phi_ {j}).
$$
At any trial  $t$  the contexts in  $\{x_s\mid s\in [t]\}$  naturally form a tree by designating  $n(x_{s})$  as the parent of  $x_{s}$ . However, to utilise the TST data-structure we must only have binary trees. Hence, we will work with a (dynamic) full binary tree  $\mathcal{Z}$  which, on trial  $t$ , is a binarisation of the above tree. The relationship between these two trees is given by a map  $\gamma :\mathcal{Z}_t\to \{x_s\mid s\in [t]\}$  where  $\mathcal{Z}_t$  is the tree  $\mathcal{Z}$  on trial  $t$ . For all  $x\in \{x_s\mid s\in [t]\}$  we will always have an unique leaf  $\tilde{\gamma} (x)\in \mathcal{Z}_t^\star$  in which  $\gamma (\tilde{\gamma} (x)) = x$ . We also maintain a balanced TST  $\mathcal{H}(\mathcal{Z})$  of  $\mathcal{Z}$ .
Algorithm 2 gives the subroutine  $\mathrm{GROW}_t$  which updates  $\mathcal{Z}$  at the start of trial  $t$ . Note that  $\mathrm{GROW}_t$  also defines a function  $d: \mathcal{Z} \to \mathbb{N}$  such that  $d(u)$  is the number of times the function  $n$  must be applied to  $\gamma(u)$  to reach  $x_1$ .
Algorithm 2 GROW $_t$  which works on  $\mathcal{Z}$
1:  $u\gets \tilde{\gamma} (n(x_t))$  10: else   
2:  $u^{*}\gets \uparrow (u)$  11:  $\triangleright (u^{*})\gets u^{\prime}$    
3:  $u^{\prime}\gets$  NEWVERTEX 12: end if   
4:  $u''\gets$  NEWVERTEX 13:  $\triangleleft (u^{\prime})\leftarrow u^{\prime \prime}$    
5:  $\gamma (u^{\prime})\gets n(x_{t})$  14:  $\triangleright (u^{\prime})\gets u$    
6:  $\gamma (u^{\prime \prime})\gets x_{t}$  15:  $d(u^{\prime})\gets d(u)$    
7:  $\tilde{\gamma} (x_{t})\gets u^{\prime \prime}$  16:  $d(u^{\prime \prime})\gets d(u) + 1$    
8: if  $u = \triangleleft (u^{*})$  then 17: REBALANCE(  $\mathcal{H}(\mathcal{Z}),u^{\prime \prime})$    
9:  $\triangleleft (u^{*})\gets u^{\prime}$
A contraction (of  $\mathcal{Z}$ ) is defined as a full binary tree  $\mathcal{J}$  in which the following holds. (1) The vertices of  $\mathcal{J}$  are a subset of those of  $\mathcal{Z}$ . (2)  $r(\mathcal{J}) = r(\mathcal{Z})$ . (3) Given a vertex  $u \in \mathcal{J}$  we have  $\triangleleft_{\mathcal{J}}(u) \in \Downarrow_{\mathcal{Z}}(\triangleleft_{\mathcal{Z}}(u))$  and  $\triangleright_{\mathcal{J}}(u) \in \Downarrow_{\mathcal{Z}}(\triangleright_{\mathcal{Z}}(u))$ . (4) Any leaf of  $\mathcal{J}$  is a leaf of  $\mathcal{Z}$ .
CBNN will maintain, on every vertex  $v \in \mathcal{B}$ , a contraction  $\mathcal{A}(v)$  as well as a TST  $\mathcal{H}(\mathcal{A}(v))$  of  $\mathcal{A}(v)$ . Given  $\mathcal{J}$  is one of these contractions, we also maintain, for all  $i, i' \in \{0,1\}$  and all  $u \in \mathcal{J}$ , a value  $\tau_{i,i'}(\mathcal{J},u) \in \mathbb{R}_+$ . Technically these quantities, which depend on the above function  $d$ , define a bayesian network on  $\mathcal{J}$  which is explained in Appendix C.3. For all  $i \in \{0,1\}$  and all  $u \in \mathcal{J}$  we also maintain a value  $\kappa_i(\mathcal{J},u)$  initialised equal to 1.
On each of our contractions  $\mathcal{I}$  we will define, on trial  $t$ , a subroutine  $\mathrm{INSERT}_t(\mathcal{I})$  that simply modifies  $\mathcal{I}$  so that  $\tilde{\gamma}(x_t)$  is added to its leaves. This subroutine is only called on certain trials  $t$ . Specifically, it is called on the contraction  $\mathcal{A}(v)$  only when  $v$  is involved in CANPROP on trial  $t$ . Although the effect of this subroutine is simple to describe, its polylogarithmic-time implementation is quite complex. A function that is used many times during this subroutine is  $\nu: \mathcal{Z} \times \mathcal{Z} \to \{\triangleleft, \triangleright, \triangleleft\}$  in which  $\nu(u, u')$  is equal to  $\triangleleft, \triangleright$  or  $\triangleleft$  if  $u'$  is contained in  $\Downarrow_{\mathcal{Z}}(\triangleleft_{\mathcal{Z}}(u))$ , in  $\Downarrow_{\mathcal{Z}}(\triangleright_{\mathcal{Z}}(u))$  or in neither, respectively. Algorithm 3 shows how to compute this function. Now that we have a subroutine for computing  $\nu$  we can turn to the pseudocode for the subroutine  $\mathrm{INSERT}_t(\mathcal{I})$  in Algorithm 4. In the appendix we give a full description of how and why this subroutine works.
Algorithm 3 Computing  $\nu(u, u')$  for  $u, u' \in \mathcal{Z}$
1:  $\mathcal{E}\gets \mathcal{H}(\mathcal{Z})$  20: return  $\triangleleft$    
2: if  $u = u^{\prime}$  then 21: else if  $\xi (s^{*}) = u\wedge \hat{s}^{\prime} = \triangleright (s^{*})$  then   
3: return  $\triangleleft$    
4: end if 23: end if   
5:  $\tilde{s}\gets \Upsilon_{\mathcal{E}}(u)$  24:  $s\gets \hat{s}$    
6:  $\tilde{s}^\prime \gets \Upsilon_\varepsilon (u^\prime)$  25: while TRUE do   
7:  $s^*\gets \Gamma_{\mathcal{E}}(\tilde{s},\tilde{s}^{\prime})$  26: if  $s\in \mathcal{E}^{\circ}$  then   
8: for  $s\in \{\triangleleft (s^{*}),\nabla (s^{*}),\triangleright (s^{*})\}$  do 27: return A   
9: if  $\tilde{s}\in \Downarrow (s)$  then 28: else if  $u = \xi (s)\wedge \triangleleft (s)\in \mathcal{E}^{\bullet}$  then   
10:  $\hat{s}\gets s$  29: return A   
11: end if 30: else if  $u = \xi (s)\wedge \triangleright (s)\in \mathcal{E}^{\bullet}$  then   
12: if  $\tilde{s}'\in \Downarrow (s)$  then 31: return B   
13:  $\hat{s} '\gets s$  32: end if   
14: end if 33: for  $s'\in \{\triangleleft (s),\nabla (s),\triangleright (s)\}$  do   
15: end for 34: if  $\tilde{s}\in \Downarrow (s')$  then   
16: if  $\hat{s}\neq \nabla (s^{*})$  then 35:  $s\gets s'$    
17: return A   
18: end if 36: end if   
19: if  $\xi (s^{*}) = u\wedge \hat{s}^{\prime} = \triangleleft (s^{*})$  then 37: end for   
38: end while
Algorithm 4 The operation  $\mathrm{INSERT}_t(\mathcal{I})$  on a contraction  $\mathcal{J}$  of  $\mathcal{Z}$  at trial  $t$
1:  $\mathcal{E}\gets \mathcal{H}(\mathcal{Z})$  29:  $u^{*}\gets \mu (s)$    
2:  $\mathcal{D}\leftarrow \mathcal{H}(\mathcal{J})$  30:  $u^{\prime}\gets \uparrow_{\mathcal{J}}(\hat{u})$    
3:  $s\gets r(\mathcal{D})$  31: if  $\hat{u} = \triangle J(u^{\prime})$  then   
4:  $u_{t}\gets \tilde{\gamma} (x_{t})$  32:  $\triangle J(u^{\prime})\gets u^{*}$    
5: while  $s\in \mathcal{D}^\dagger$  do 33: else   
6: if  $\nu (\xi (s),u_t) = \triangle$  then 34:  $\triangleright_{\mathcal{J}}(u^{\prime})\gets u^{*}$    
7:  $s\gets \triangle (s)$  35: end if   
8: else if  $\nu (\xi (s),u_t) = \triangleright$  then 36: if  $\nu (u^{*},\hat{u}) = \triangle$  then   
9:  $s\gets \triangleright (s)$  37:  $\triangle J(u^{*})\gets \hat{u}$    
10: else if  $\nu (\xi (s),u_t) = \triangle$  then 38:  $\triangleright_{\mathcal{J}}(u^{*})\gets u_{t}$    
11:  $s\gets \nabla (s)$  39: else   
12: end if 40:  $\triangleright_{\mathcal{J}}(u^{*})\gets \hat{u}$    
13: end while 41:  $\triangle J(u^{*})\gets u_{t}$    
14:  $\hat{u}\gets \mu (s)$  42: end if   
15:  $s\gets r(\mathcal{E})$    
16: while  $s\in \mathcal{E}^\dagger$  do 43: for  $i\in \{0,1\}$  do   
17: if  $\nu (\xi (s),u_t) = \nu (\xi (s),\hat{u})$  then 44:  $\kappa_i(\mathcal{I},u^*)\gets 1$    
18: if  $\nu (\xi (s),u_t) = \triangle$  then 45:  $\kappa_i(\mathcal{J},u_t)\gets 1$    
19:  $s\gets \triangle (s)$  46: end for   
20: else if  $\nu (\xi (s),u_t) = \triangleright$  then 47: for  $u\in \{u^{*},\hat{u},u_{t}\}$  do   
21:  $s\gets \triangleright (s)$  48:  $\delta (u)\gets d(u) - d(\uparrow_{\mathcal{J}}(u))$    
22: else if  $\nu (\xi (s),u_t) = \triangle$  then 49: end for   
23:  $s\gets \nabla (s)$  50: for  $(i,i^{\prime})\in \{0,1\} \times \{0,1\}$  do   
24: end if 51: if  $i = i^{\prime}$  then   
25: else 52:  $\tau_{i,i^{\prime}}(\mathcal{J},u)\gets 1 - \phi_{\delta (u)}$    
26:  $s\gets \nabla (s)$    
27: end if   
28: end while 56: end for   
57: REBALANCE(H(J),ut)
4.5 Online Belief Propagation
In this subsection we utilise the work of [7] in order to be able to efficiently compute the function  $\theta_t$  that appears in CANPROP.
Given a vertex  $u$  in one of our contractions  $\mathcal{I}$  we define  $\mathcal{F}(\mathcal{I}, u) := \{f \in \{0, 1\}^{\mathcal{I}} \mid f(u) = 1\}$  and then define:
$$
\Lambda (\mathcal{J},u):= \sum_{f\in \mathcal{F}(\mathcal{J},u)}\prod_{\substack{u^{\prime}\in \mathcal{J}\setminus \{r(\mathcal{J})\}}}\tau_{f(\uparrow_{\mathcal{J}}(u^{\prime})),f(u^{\prime})}(\mathcal{J},u^{\prime})\kappa_{f(u^{\prime})}(\mathcal{J},u^{\prime}).
$$
As stated in the previous subsection, when a vertex  $v \in \mathcal{B}$  becomes involved in CANPROP on trial  $t$ , CBNN will add  $\tilde{\gamma}(x_t)$  to the leaves of  $\mathcal{A}(v)$  via the operation  $\mathrm{INSERT}_t(\mathcal{A}(v))$ . In the appendix we shall show that for each such  $v$  we then have:
$$
\theta_ {t} (v) = \Lambda (\mathcal {A} (v), \tilde {\gamma} (x _ {t})) / 4.
$$
We now outline how to compute this efficiently, deferring a full description for Appendix D.3. First note that for all contractions  $\mathcal{I}$  and all  $u\in \mathcal{I}$  we have that  $\Lambda (\mathcal{I},u)$  is of the exact form to be solved by the classic Belief propagation algorithm [23]. The work of [7] shows how to compute this term in logarithmic time by maintaining a data-structure based on a balanced TST of  $\mathcal{I}$  - in our case the TST  $\mathcal{H}(\mathcal{I})$  . Whenever, for some  $i\in \{0,1\}$  and  $u^{\prime}\in \mathcal{I}$  , the value  $\kappa_{i}(\mathcal{J},u^{\prime})$  changes, the data-structure is updated in logarithmic time. We define the subroutine EVIDENCE  $(\mathcal{J},u^{\prime})$  as that which updates this data-structure after  $\kappa_{i}(\mathcal{J},u^{\prime})$  changes. We also make sure that the data-structure is updated whenever REBALANCE  $(\mathcal{H}(\mathcal{I}),\cdot)$  is called. We then define the subroutine MARGINAL  $(\mathcal{J},u)$  as that which computes  $\Lambda (\mathcal{J},u) / 4$  . Hence, the output of MARGINAL  $(\mathcal{A}(v),\tilde{\gamma} (x_{t}))$  is equal to  $\theta_t(v)$
4.6 CBNN
Now that we have defined all our subroutines we give, in Algorithm 5, the algorithm CBNN which is an efficient implementation of CANPROP with initial weighting given in Equation (2).
Algorithm 5 CBNN at trial  $t$    
1: GROWt 18: end if   
2:  $u_{t}\gets \tilde{\gamma} (x_{t})$  19: end for   
3:  $v_{t,0}\leftarrow r(\mathcal{B})$  20:  $a_{t}\gets v_{t,\log (K)}$    
4: for  $j = 0,1,\dots ,(\log (K) - 1)$  do 21:  $\tilde{\pi}_t\gets \prod_{j\in [\log (K)]}\pi_t(v_{t,j})$    
5: for  $v\in \{\triangleleft (v_{t,j}),\triangleright (v_{t,j})\}$  do 22:  $\psi_{t,\log (K)}\gets \exp (-\eta \ell_{t,a_t} / \tilde{\pi}_t)$    
6: INSERTt(A(v)) 23: for  $j = \log (K),(\log (K) - 1),\dots ,1$  do   
7:  $\theta_t(v)\gets \mathrm{MARGINAL}(\mathcal{A}(v),u_t)$  24:  $\psi_{t,(j - 1)}\gets 1 - (1 - \psi_{t,j})\pi_t(v_{t,j})$    
8: end for 25: if  $v_{t,j} = \triangleleft (v_{t,j - 1})$  then   
9:  $z_{t,j}\gets \theta_t(\triangleleft (v_{t,j})) + \theta_t(\triangleright (v_{t,j}))$    
10: for  $v\in \{\triangleleft (v_{t,j}),\triangleright (v_{t,j})\}$  do 26:  $\tilde{v}_{t,j}\gets \triangleright (v_{t,j - 1})$    
11:  $\pi_t(v)\gets \theta_t(v) / z_{t,j}$  27: else   
12: end for 28:  $\tilde{v}_{t,j}\gets \triangleleft (v_{t,j - 1})$    
13:  $\zeta_{t,j}\sim [0,1]$  29: end if   
14: if  $\zeta_{t,j}\leq \pi_t(\triangleleft (v_{t,j}))$  then 30:  $\kappa_1(\mathcal{A}(v_{t,j}),u_t)\gets \psi_{t,j} / \psi_{t,j - 1}$    
15:  $v_{t,j + 1}\gets \triangleleft (v_{t,j})$  31:  $\kappa_1(\mathcal{A}(\tilde{v}_{t,j}),u_t)\gets 1 / \psi_{t,j - 1}$    
16: else 32: EVIDENCE(A(vt,j),ut)   
17:  $v_{t,j + 1}\gets \triangleright (v_{t,j})$  33: EVIDENCE(A(  $|\tilde{v}_{t,j})$  ,ut)   
34: end for
We would like to thank Mark Herbster (University College London) for valuable discussions.
Research funded by the Defence Science and Technology Laboratory (Dstl) which is an executive agency of the UK Ministry of Defence providing world class expertise and delivering cutting-edge science and technology for the benefit of the nation and allies. The research supports the Autonomous Resilient Cyber Defence (ARCD) project within the Dstl Cyber Defence Enhancement programme.
[1] R. Agrawal. Sample mean based index policies by o(log n) regret for the multi-armed bandit problem. Advances in Applied Probability, 27:1054 - 1078, 1995.
[2] S. Arya and Y. Yang. Randomized allocation with nonparametric estimation for contextual multi-armed bandits with delayed rewards. Statistics & Probability Letters, 2020.
[3] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-256, 2002.
[4] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput., 32:48-77, 2002.
[5] W. Chu, L. Li, L. Reyzin, and R. E. Schapire. Contextual bandits with linear payoff functions. In International Conference on Artificial Intelligence and Statistics, 2011.
[6] T. M. Cover and P. E. Hart. Nearest neighbor pattern classification. IEEE Trans. Inf. Theory, 13:21-27, 1967.
[7] A. L. Delcher, A. J. Grove, S. Kasif, and J. Pearl. Logarithmic-time updates and queries in probabilistic networks. J. Artif. Intell. Res., 4:37-59, 1995.
[8] Y. Freund, R. E. Schapire, Y. Singer, and M. K. Warmuth. Using and combining predictors that specialize. In Symposium on the Theory of Computing, 1997.
[9] A. Garivier and O. Cappé. The kl-ucb algorithm for bounded stochastic bandits and beyond. In Annual Conference Computational Learning Theory, 2011.
[10] D. P. Helmbold and R. E. Schapire. Predicting nearly as well as the best pruning of a decision tree. Machine Learning, 27:51-68, 1995.
[11] M. Herbster, S. Pasteris, and F. Vitale. Online sum-product computation over trees. In NIPS, 2012.
[12] M. Herbster, S. Pasteris, F. Vitale, and M. Pontil. A gang of adversarial bandits. In Neural Information Processing Systems, 2021.
[13] M. Herbster and J. Robinson. Online prediction of switching graph labelings with cluster specialists. In Neural Information Processing Systems, 2018.
[14] M. Herbster and M. K. Warmuth. Tracking the best expert. Machine Learning, 32:151-178, 1995.
[15] S. M. Kakade, S. Shalev-Shwartz, and A. Tewari. Efficient bandit algorithms for online multiclass prediction. In International Conference on Machine Learning, 2008.
[16] W. M. Koolen, D. Adamskiy, and M. K. Warmuth. Putting bayes to sleep. In NIPS, 2012.
[17] R. Krauthgamer and J. R. Lee. Navigating nets: simple algorithms for proximity search. In ACM-SIAM Symposium on Discrete Algorithms, 2004.
[18] T. L. Lai and H. E. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6:4-22, 1985.
[19] J. Langford and T. Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. In NIPS 2007, 2007.
[20] T. Lattimore and C. Szepesvári. Bandit algorithms. 2020.
[21] L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In The Web Conference, 2010.
[22] K. Matsuzaki and A. Morihata. Balanced ternary-tree representation of binary trees and balancing algorithms. In Mathematical Engineering Technical Reports, 2008.