SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[23] J. Pearl. Reverend bayes on inference engines: A distributed hierarchical approach. Probabilistic and Causal Inference, 1982.
[24] V. Perchet and P. Rigollet. The multi-armed bandit problem with covariates. The Annals of Statistics, 2013.
[25] W. Qian and Y. Yang. Kernel estimation and model combination in a bandit problem with covariates. J. Mach. Learn. Res., 17:149:1-149:37, 2016.
[26] H. W. J. Reeve, J. C. Mellor, and G. Brown. The k-nearest neighbour ucb algorithm for multi-armed bandits with covariates. Algorithmic Learning Theory, 2018.
[27] P. Rigollet and A. J. Zeevi. Nonparametric bandits with covariates. In Annual Conference Computational Learning Theory, 2010.
[28] H. E. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58:527-535, 1952.
[29] Y. Seldin, P. Auer, F. Laviolette, J. Shawe-Taylor, and R. Ortner. Pac-bayesian analysis of contextual bandits. In NIPS, 2011.
[30] A. Slivkins. Contextual bandits with similarity information. Journal of Machine Learning Research, 2014.
[31] C.-C. Wang, S. R. Kulkarni, and H. V. Poor. Arbitrary side observations in bandit problems. Adv. Appl. Math., 34:903-938, 2005.
A Introduction to the Appendix
We now turn to the full description and analysis of our algorithm CBNN. In Appendix B we describe our novel algorithmic framework CANPROP. In Appendix C we describe contractions and bayesian networks on them, showing how CANPROP can be implemented with them. Finally, in Appendix D we describe TSTs and how they are used to perform our required operations efficiently. In Appendix E we prove, in order, all of the theorems stated in this paper.
Recall that we created a sequence of distinct nodes  $\langle x_t \mid t \in [T] \rangle$  and defined, for all  $t \in [T] \setminus \{1\}$ , the node  $n(x_t) \coloneqq x_{n(t)}$ . Let  $\mathcal{X} \coloneqq \{x_t \mid t \in [T]\}$ . Given some  $y: \mathcal{X} \to [K]$ , we defined the  $y$ -regret and the complexity of  $y$  as:
B Cancellation Propagation
B.1 The General CANPROP Algorithm
We now introduce a general algorithmic framework CANPROP for handling contextual bandit problems with a per-trial time logarithmic in  $K$ . Without loss of generality assume that  $K$  is an integer power of two. Let  $\mathcal{B}$  be a full, balanced binary tree whose leaves are the set of actions  $[K]$ . Let  $\mathcal{B}' := \mathcal{B} \setminus \{r(\mathcal{B})\}$ . CANPROP takes a parameter  $\eta \in \mathbb{R}_+$  called the learning rate. On each trial  $t$  CANPROP maintains a function:
$$
w _ {t}: \mathcal {B} ^ {\prime} \times 2 ^ {\mathcal {X}} \rightarrow [ 0, 1 ].
$$
The function  $w_{1}$  is free to be defined how one likes, as long as it satisfies the constraint that for all internal vertices  $v \in \mathcal{B}^{\dagger}$  we have:
$$
\sum_ {\mathcal {S} \in 2 ^ {\mathcal {X}}} \left(w _ {1} (\triangleleft (v), \mathcal {S}) + w _ {1} (\triangleright (v), \mathcal {S})\right) = 1.
$$
We now describe how CANPROP acts on trial  $t$ . For all  $v \in \mathcal{B}'$  we define:
$$
\theta_ {t} (v) := \sum_ {\mathcal {S} \in 2 ^ {\mathcal {X}}} [   [ x _ {t} \in \mathcal {S} ]   ] w _ {t} (v, \mathcal {S})
$$
and for all  $v\in \mathcal{B}^{\dagger}$  we define:
$$
\pi_ {t} (\triangleleft (v)) := \frac {\theta_ {t} (\triangleleft (v))}{\theta_ {t} (\triangleleft (v)) + \theta_ {t} (\triangleright (v))}; \quad ; \quad \pi_ {t} (\triangleright (v)) := \frac {\theta_ {t} (\triangleright (v))}{\theta_ {t} (\triangleleft (v)) + \theta_ {t} (\triangleright (v))}.
$$
As we shall see CANPROP needs only compute these values for  $\mathcal{O}(\ln (K))$  vertices  $v$ . CANPROP samples a root-to-leaf path  $\{v_{t,j}\mid j\in [\log (K)]\cup \{0\} \}$  as follows.  $v_{t,0}$  is defined equal to  $r(\mathcal{B})$ . For all  $j\in [\log (K) - 1]\cup \{0\}$ , once  $v_{t,j}$  has been sampled we sample  $v_{t,(j + 1)}$  from the probability distribution defined by:
$$
\mathbb {P} \left[ v _ {t, (j + 1)} = v \right] := \llbracket \uparrow (v) = v _ {t, j} \rrbracket \pi_ {t} (v) \quad \forall v \in \mathcal {B} ^ {\prime}
$$
noting that  $v_{t,(j+1)}$  is a child of  $v_{t,j}$ . We define:
$$
\mathcal {P} _ {t} := \left\{v _ {t, j} \mid j \in \left[ \log (K) \right] \cup \{0 \} \right\}.
$$
CANPROP then selects:
$$
a _ {t} := v _ {t, \log (K)}
$$
and then receives the loss  $\ell_{t,a_t}$ . The function  $w_{t}$  is then updated to  $w_{t + 1}$  as follows. Firstly we define,
$$
w _ {t + 1} (v, \mathcal {S}) := w _ {t} (v, \mathcal {S}) \quad \forall (v, \mathcal {S}) \in \left\{v ^ {\prime} \in \mathcal {B} ^ {\prime} \mid \uparrow (v ^ {\prime}) \notin \mathcal {P} _ {t} \right\} \times 2 ^ {\mathcal {X}}.
$$
We then define:
$$
\psi_ {t, \log (K)} := \exp \left(\frac {- \eta \ell_ {t , a _ {t}}}{\prod_ {j \in [ \log (K) ]} \pi_ {t} (v _ {t , j})}\right).
$$
Once we have defined  $\psi_{t,j}$  for some  $j\in [\log (K)]$  we then define:
$$
\psi_ {t, (j - 1)} := 1 - \left(1 - \psi_ {t, j}\right) \pi_ {t} \left(v _ {t, j}\right)
$$
$$
\beta_ {t} (v) := \frac {\llbracket v \in \mathcal {P} _ {t} \rrbracket \psi_ {t , j} + \llbracket v \notin \mathcal {P} _ {t} \rrbracket}{\psi_ {t , (j - 1)}} \quad \forall v \in \{\triangleleft (v _ {t, (j - 1)}), \triangleright (v _ {t, (j - 1)}) \}
$$
$$
w _ {t + 1} (v, \mathcal {S}) := \left(\llbracket x _ {t} \in \mathcal {S} \rrbracket \beta_ {t} (v) + \llbracket x _ {t} \notin \mathcal {S} \rrbracket\right) w _ {t} (v, \mathcal {S}) \quad \forall (v, \mathcal {S}) \in \left\{\triangleleft (v _ {t, (j - 1)}), \triangleright (v _ {t, (j - 1)}) \right\} \times 2 ^ {\mathcal {X}}.
$$
The regret bound of CANPROP is given by the following theorem.
Theorem B.1. Suppose we have a function  $y: \mathcal{X} \to [K]$ . For all  $v \in \mathcal{B}$  define:
$$
\mathcal {Q} (y, v) := \{x \in \mathcal {X} \mid y (x) \in \Downarrow (v) \}.
$$
Then the expected  $y$ -regret of CANPROP is bounded by:
$$
\mathbb {E} [ R (y) ] \leq \frac {\eta K T}{2} - \frac {1}{\eta} \sum_ {v \in \mathcal {B} ^ {\prime}} \llbracket \mathcal {Q} (y, v) \neq \emptyset \rrbracket \ln \left(w _ {1} (v, \mathcal {Q} (y, v))\right).
$$
B.2 Our Parameter Tuning
We now describe and analyse the initial weighting  $w_{1}$  and the learning rate  $\eta$  that we will use. Define  $\mathcal{X}' := \mathcal{X} \setminus \{x_{1}\}$ . For all  $(x, \mathcal{S}) \in \mathcal{X}' \times 2^{\mathcal{X}}$  define:
For all  $(v, S) \in \mathcal{B}' \times 2^{\mathcal{X}}$  we define:
$$
w _ {1} (v, \mathcal {S}) := \frac {1}{4} \prod_ {x \in \mathcal {X} ^ {\prime}} \left(\sigma (x, \mathcal {S}) \frac {1}{T} + (1 - \sigma (x, \mathcal {S})) \left(1 - \frac {1}{T}\right)\right).
$$
Given our parameter  $\rho$  we choose our learning rate as:
$$
\eta := \rho \sqrt {\frac {\ln (K) \ln (T)}{K T}}.
$$
Given this initial weighting and learning rate, Theorem B.1 implies the following regret bound.
Theorem B.2. Given  $w_{1}$  and  $\eta$  are defined as above, then for any  $y: \mathcal{X} \to [K]$  the expected  $y$ -regret of CANPROP is bounded by:
$$
\mathbb {E} [ R (y) ] \in \mathcal {O} \left(\left(\rho + \frac {\Phi (y)}{\rho}\right) \sqrt {\ln (K) \ln (T) K T}\right).
$$
C Implementation with Contractions
C.1 A Sequence of Binary Trees
For any trial  $t$  we have a natural tree-structure on the set  $\{x_{t'} \mid t' \in [t]\}$  formed by making  $n(x_{t'})$  the parent of  $x_{t'}$  for all  $t' \in [t] \setminus \{1\}$ . However, in order to utilise the methodology of [22] we need to work with binary trees. Hence, we now inductively define a sequence of binary trees  $\langle \mathcal{Z}_t \mid t \in [T] \setminus \{1\} \rangle$  where the vertices of  $\mathcal{Z}_t$  are a subset of those of  $\mathcal{Z}_{t+1}$ . We also define a function  $\gamma: \mathcal{Z}_T \to \mathcal{X}$ . This function  $\gamma$  has the property that for any  $t \in [T]$  and for any distinct leaves  $u, u' \in \mathcal{Z}_t^\star$  we have that  $\gamma(u) \neq \gamma(u')$ , and that:
$$
\left\{\gamma \left(u ^ {\prime \prime}\right) \mid u ^ {\prime \prime} \in \mathcal {Z} _ {t} ^ {\star} \right\} = \left\{x _ {t ^ {\prime}} \mid t ^ {\prime} \in [ t ] \right\}.
$$
We define  $\mathcal{Z}_2$  to contain three vertices  $\{r(\mathcal{Z}_2),\triangleleft (r(\mathcal{Z}_2)),\triangleright (r(\mathcal{Z}_2))\}$  where:
$$
\gamma (r (\mathcal {Z} _ {2})) := \gamma (\triangleleft (r (\mathcal {Z} _ {2})) := x _ {1} \quad ; \quad \gamma (\triangleright (r (\mathcal {Z} _ {2})) := x _ {2}.
$$
Now consider a trial  $t \in [T]$ . We have that  $\mathcal{Z}_{t+1}$  is constructed from  $\mathcal{Z}_t$  via the following algorithm:  $\mathrm{GROW}_{t+1}$ :
1. Let  $u$  be the unique leaf in  $\mathcal{Z}_t^\star$  in which  $\gamma(u) = n(x_{t+1})$  and let  $u^* := \uparrow(u)$ .
2. Create two new vertices  $u'$  and  $u''$ .
3. Set  $\gamma (u^{\prime})\gets n(x_{t + 1})$  and  $\gamma (u^{\prime \prime})\gets x_{t + 1}$
4. If  $u = \triangleleft(u^{*})$  then set  $\triangleleft(u^{*}) \gets u'$ . Else set  $\triangleright (u^{*}) \gets u'$ .
5. Set  $\triangleleft(u') \gets u''$  and  $\triangleright(u') \gets u$ .
We also define a function  $d: \mathcal{Z}_T \to \mathbb{N} \cup \{0\}$  as follows. Define  $d'(x_1) := 0$  and for all  $t \in [T] \setminus \{1\}$  inductively define  $d'(x_t) := d'(n(x_t)) + 1$ . Finally define  $d(u) := d'(\gamma(u))$  for all  $u \in \mathcal{Z}_T$ . Since for all  $t \in [T]$  we have that the vertices of  $\mathcal{Z}_t$  are a subset of those of  $\mathcal{Z}_T$  we have that  $d$  also defines a function over  $\mathcal{Z}_t$  for all  $t \in [T]$ .
For all  $t \in [T]$  we define  $u_t$  to be the unique leaf of  $\mathcal{Z}_t$  for which  $\gamma(u_t) = x_t$ .
C.2 Contractions
Our efficient implementation of CANPROP will have a data-structure at every vertex  $v \in \mathcal{B}'$ . However, to achieve polylogarithmic time per trial we can only update a polylogarithmic number of these data-structures per trial. This necessitates the use of contractions of our trees  $\{\mathcal{Z}_t \mid t \in [T] \setminus \{1\}\}$  which are defined as follows. A contraction of a full binary tree  $\mathcal{Q}$  is another full binary tree  $\mathcal{J}$  which satisfies the following:
- The vertices of  $\mathcal{I}$  are a subset of those of  $\mathcal{Q}$ .
$r(\mathcal{J}) = r(\mathcal{Q}).$
Given an internal vertex  $u\in \mathcal{J}^{\dagger}$  we have  $\triangleleft_{\mathcal{I}}(u)\in \Downarrow_{\mathcal{Q}}(\triangleleft_{\mathcal{Q}}(u))$  and  $\triangleright_{\mathcal{I}}(u)\in \Downarrow_{\mathcal{Q}}(\triangleright_{\mathcal{Q}}(u))$
- Any leaf of  $\mathcal{I}$  is a leaf of  $\mathcal{Q}$ .
Note that any contraction of  $\mathcal{Z}_t$  is also a contraction of  $\mathcal{Z}_{t + 1}$  and hence, by induction, a contraction of  $\mathcal{Z}_{t'}$  for all  $t' \geq t$ . Given a trial  $t$  and a contraction  $\mathcal{J}$  of  $\mathcal{Z}_{t - 1}$  we now define the operation  $\mathrm{INSERT}_t(\mathcal{J})$  which acts on  $\mathcal{J}$  by the following algorithm:
1. Let  $\hat{u}$  be the unique vertex in  $\mathcal{J} \setminus \{r(\mathcal{J})\}$  such that  $u_{t}$  is in the maximal subtree of  $\mathcal{Z}_{t}$  with  $\hat{u}$  and  $\uparrow_{\mathcal{J}}(\hat{u})$  as leaves.
2. Let  $u^{*} \coloneqq \Gamma_{\mathcal{Z}_{t}}(u_{t}, \hat{u})$ .
3. Add the vertices  $u^{*}$  and  $u_{t}$  to the tree  $\mathcal{I}$ .
4. Let  $u' \coloneqq \uparrow_{\mathcal{I}}(\dot{u})$ .
5. If  $\hat{u} = \triangleleft_{\mathcal{I}}(u')$  then set  $\triangleleft_{\mathcal{I}}(u') \gets u^*$ . Else set  $\triangleright_{\mathcal{I}}(u') \gets u^*$ .
6. If  $\hat{u} \in \Downarrow_{\mathcal{Z}_t}(\triangleleft_{\mathcal{Z}_t}(u^*))$  then set  $\triangleleft_{\mathcal{J}}(u^*) \gets \hat{u}$  and  $\triangleright_{\mathcal{J}}(u^*) \gets u_t$ . Else set  $\triangleright_{\mathcal{J}}(u^*) \gets \hat{u}$  and  $\triangleleft_{\mathcal{J}}(u^*) \gets u_t$ .
Later in this paper we will show how this operation can be done in polylogarithmic time. Note that after the operation we have that  $\mathcal{I}$  is a contraction of  $\mathcal{Z}_t$  and  $u_{t}$  has been added to it's leaves. From now on when we use the term contraction we mean any contraction of  $\mathcal{Z}_T$ .
C.3 Contraction-Based Bayesian Networks
Here we shall define a bayesian network over any contraction  $\mathcal{I}$  and show how it can be utilised to compute certain quantities required by CANPROP. First define the quantity  $\phi_0\coloneqq 0$  and for all  $j\in \mathbb{N}\cup \{0\}$  inductively define:
The algorithm must compute these quantities for all  $j \in [T]$ . However, for all  $t \in [T]$  we have that  $\phi_t$  doesn't have to be computed until trial  $t$  so computing these quantities is constant time per trial. Given a contraction  $\mathcal{I}$ , a vertex  $u \in \mathcal{I} \setminus r(\mathcal{I})$  and indices  $i, i' \in \{0, 1\}$  define:
$$
\tau_ {i, i ^ {\prime}} (\mathcal {J}, u) := \llbracket i \neq i ^ {\prime} \rrbracket \phi_ {(d (u) - d (\uparrow_ {\mathcal {I}} (u)))} + \llbracket i = i ^ {\prime} \rrbracket \left(1 - \phi_ {(d (u) - d (\uparrow_ {\mathcal {I}} (u)))}\right)
$$
which defines the transition matrix from  $\uparrow_{\mathcal{I}}(u)$  to  $u$  in a bayesian network over  $\mathcal{J}$ . We shall now show how belief propagation over such bayesian networks can be used to compute the quantities we need in CANPROP. Suppose we have a contraction  $\mathcal{I}$  and a function  $\lambda: \mathcal{I}^{\star} \to \mathbb{R}_{+}$ . This function  $\lambda$  induces a function  $\lambda': \mathcal{X} \to \mathbb{R}_{+}$  defined as follows. Given  $x \in \mathcal{X}$ , if there exists a leaf  $u \in \mathcal{I}^{\star}$  with  $\gamma(u) = x$  then  $\lambda'(x) = \lambda(u)$ . Otherwise  $\lambda'(x) = 1$ . For all  $S \in 2^{\mathcal{X}}$  define:
$$
\tilde {w} (\mathcal {J}, \lambda , \mathcal {S}) := \left(\prod_ {x \in \mathcal {S}} \lambda^ {\prime} (x)\right) \left(\prod_ {x \in \mathcal {X} ^ {\prime}} \left(\sigma (x, \mathcal {S}) \frac {1}{T} + (1 - \sigma (x, \mathcal {S})) \left(1 - \frac {1}{T}\right)\right)\right).
$$
For the CANPROP algorithm we will need to compute
$$
\sum_ {\mathcal {S} \in 2 ^ {\mathcal {X}}} [   [ \gamma (\hat {u}) \in \mathcal {S} ]   ] \tilde {w} (\mathcal {J}, \lambda , \mathcal {S}) \tag {3}
$$
for some leaf  $\hat{u} \in \mathcal{J}^{\star}$  and some function  $\lambda: \mathcal{J}^{\star} \to \mathbb{R}_{+}$ . We shall now show how we can compute this quantity via belief propagation on the bayesian network. In particular, we shall construct a quantity  $\hat{\Lambda}(\mathcal{J}, \lambda, u)$  equal to the quantity in Equation (3). To do this, first define the function  $\lambda^{*}: \mathcal{J} \to \mathbb{R}_{+}$  so that for all  $u \in \mathcal{J}^{\star}$  we have  $\lambda^{*}(u) = \lambda(u)$  and for all  $u \in \mathcal{J}^{\dagger}$  we have  $\lambda^{*}(u) = 1$ . For all vertices  $u \in \mathcal{J}$  and all indices  $i \in \{0,1\}$  define:
$$
\tilde {\kappa} _ {i} (\mathcal {J}, \lambda , u) := [   [ i = 0 ]   ] + [   [ i = 1 ]   ] \lambda^ {*} (u).
$$
For all  $\hat{u}\in \mathcal{I}$  define:
$$
\mathcal {F} (\mathcal {J}, \hat {u}) := \left\{f \in \{0, 1 \} ^ {\mathcal {J}} \mid f (\hat {u}) = 1 \right\}
$$
and then define:
$$
\tilde{\Lambda} (\mathcal{J},\lambda ,\hat{u}):= \sum_{f\in \mathcal{F}(\mathcal{J},\hat{u})}\prod_{u\in \mathcal{J}\backslash r(\mathcal{J}}}\tau_{f(\uparrow_{\mathcal{J}}(u)),f(u)}(\mathcal{J},u)\tilde{\kappa}_{f(u)}(\mathcal{J},\lambda ,u).
$$
The equality of this quantity and that given in Equation (3) is given by the following theorem.
Theorem C.1. Given a contraction  $\mathcal{I}$ , a function  $\lambda: \mathcal{I}^{\star} \to \mathbb{R}_{+}$  and some leaf  $\hat{u} \in \mathcal{I}^{\star}$  we have:
$$
\tilde {\Lambda} (\mathcal {J}, \lambda , \hat {u}) = \sum_ {\mathcal {S} \in 2 ^ {\mathcal {X}}} [   [ \gamma (\hat {u}) \in \mathcal {S} ]   ] \tilde {w} (\mathcal {J}, \lambda , \mathcal {S}).
$$
Note that  $\tilde{\Lambda} (\mathcal{I},\lambda ,\hat{u})$  is of the exact form to be solved via belief propagation over  $\mathcal{I}$ . However, belief propagation is too slow (taking  $\Theta (|\mathcal{I}|)$  time) - we will remedy this later.
C.4 Cancelation Propagation with Contractions
We now describe how to implement CANPROP with contractions. For each  $v \in \mathcal{B}'$  we maintain a contraction  $\mathcal{A}(v)$  and a function  $\lambda_v: \mathcal{A}(v)^\star \to \mathbb{R}_+$ . We initialise with  $\mathcal{A}(v)$  identical to  $\mathcal{Z}_2$  and  $\lambda_v(u) = 1$  for both leaves  $u \in \mathcal{Z}_2^\star$ . Via induction over  $t$  we will have that at the start of each trial  $t$  we have, for all sets  $S \in 2^{\mathcal{X}}$ , that:
$$
w _ {t} (v, \mathcal {S}) = \tilde {w} (\mathcal {A} (v), \lambda_ {v}, \mathcal {S}) / 4. \tag {4}
$$
On trial  $t$  we do as follows. First we update  $\mathcal{Z}_{t-1}$  to  $\mathcal{Z}_t$  using the algorithm  $\mathrm{GROW}_t$ . We will perform the necessary modifications to our contractions as we sample the path  $\mathcal{P}_t$ . In particular, we first set  $v_{t,0} := r(\mathcal{B})$  and then for each  $j \in [\log(K) - 1] \cup \{0\}$  in turn we do as follows. For each  $v \in \{\triangleleft(v_{t,j}), \triangleright(v_{t,j})\}$  run  $\mathrm{INSERT}_t(\mathcal{A}(v))$  and set  $\lambda_v(u_t) \gets 1$ . Since  $\lambda_v(u_t) = 1$  Equation (4) still holds and hence, by Theorem C.1, we have:
$$
\theta_ {t} (v) = \tilde {\Lambda} (\mathcal {A} (v), \lambda_ {v}, u _ {t}) / 4.
$$
After  $\theta_t(v)$  has been computed for both  $v \in \{\triangleleft(v_{t,j}), \triangleright(v_{t,j})\}$  we can now sample  $v_{t,j+1}$ .
Once we have selected the action  $a_{t}$  we then update the functions  $\{\lambda_v\mid \uparrow_{\mathcal{B}}(v)\in \mathcal{P}_t\}$  by setting  $\lambda_v(u_t)\gets \beta_t(v)$  for all  $v\in \mathcal{B}^{\prime}$  with  $\uparrow_{\mathcal{B}}(v)\in \mathcal{P}_t$  . It is clear now that Equation (4) holds inductively.
C.5 Notational Relationship to the Main Body
We now point out how the notation in this section relates to that of the main body. In particular, we have, for all  $v \in \mathcal{B}'$ , all  $u \in \mathcal{A}(v)$  and all  $i, i' \in \{0,1\}$ , that:
-  $\kappa_{i}(\mathcal{A}(v),u) = \tilde{\kappa}_{i}(\mathcal{A}(v),\lambda_{v},u)$ .
-  $\Lambda (\mathcal{A}(v),u) = \tilde{\Lambda} (\mathcal{A}(v),\lambda_v,u)$ .
We note that the function  $\lambda_v$  does not explicitly appear in our pseudocode since it can be inferred from  $\kappa_i(\mathcal{A}(v),\cdot)$ .
D Utilising Ternary Search Trees
There are now only two things left to do in order to achieve polylogarithmic time per trial - to make an efficient online implementation of the  $\mathrm{INSERT}_t(\cdot)$  operation and an efficient online algorithm to perform belief propagation over our contractions. In order to do this we will utilise the methodology of [22] which we now describe. However, we do not give the full details of the rebalancing technique and refer the reader to [22] for these details (noting that [22] uses different notation).
D.1 Ternary Search Trees
In this section we will consider a full binary tree  $\mathcal{I}$ . A (full) ternary tree  $\mathcal{D}$  is a rooted tree in which each internal vertex  $s \in \mathcal{D}^{\dagger}$  has three children denoted by  $\triangleleft(s)$ ,  $\nabla(s)$ ,  $\triangleright(s)$  and called the left, centre, and right children respectively. We now define what it means for a ternary tree  $\mathcal{D}$  to be a ternary search tree (TST) of  $\mathcal{J}$ . Firstly, the vertex set of  $\mathcal{D}$  is partitioned into two sets  $\mathcal{D}^{\circ}$  and  $\mathcal{D}^{\bullet}$ . Every vertex  $s \in \mathcal{D}$  is associated with a vertex  $\mu(s) \in \mathcal{J}$  and every  $s \in \mathcal{D}^{\bullet}$  is also associated with a vertex  $\mu'(s) \in \Downarrow_{\mathcal{J}}(\mu(s))^{\dagger}$ . The root  $r(\mathcal{D})$  is contained in  $\mathcal{D}^{\circ}$  and  $\mu(r(\mathcal{D})) := r(\mathcal{J})$ . Each internal vertex  $s \in \mathcal{D}^{\dagger}$  is associated with a vertex  $\xi(s) \in \mathcal{J}$ . If  $s \in \mathcal{D}^{\circ}$  then  $\xi(s) \in \Downarrow(\mu(s))^{\dagger}$  and if  $s \in \mathcal{D}^{\bullet}$  then  $\xi(s)$  lies on the path (in  $\mathcal{J}$ ) from  $\mu(s)$  to  $\uparrow(\mu'(s))$ . For all  $s \in \mathcal{D}^{\dagger}$  we have:
-  $\nabla(s) \in \mathcal{D}^{\bullet}$ ,  $\mu(\nabla(s)) \coloneqq \mu(s)$  and  $\mu'(\nabla(s)) \coloneqq \xi(s)$ .
-  $\triangleleft(s)$  satisfies:
- If  $s \in \mathcal{D}^{\circ}$  then  $\triangleleft(s) \in \mathcal{D}^{\circ}$  and  $\mu(\triangleleft(s)) := \triangleleft(\xi(s))$ .
- If  $s \in \mathcal{D}^{\bullet}$  and  $\mu'(s) \in \Downarrow(\triangleright(\xi(s)))$  then  $\triangleleft(s) \in \mathcal{D}^{\circ}$  and  $\mu(\triangleleft(s)) := \triangleleft(\xi(s))$ .
- Else  $\triangleleft(s) \in \mathcal{D}^{\bullet}$ ,  $\mu(\triangleleft(s)) := \triangleleft(\xi(s))$  and  $\mu'(\triangleleft(s)) := \mu'(s)$ .
-  $\triangleright(s)$  satisfies:
- If  $s \in \mathcal{D}^{\circ}$  then  $\triangleright(s) \in \mathcal{D}^{\circ}$  and  $\mu(\triangleright(s)) := \triangleright(\xi(s))$ .
- If  $s \in \mathcal{D}^{\bullet}$  and  $\mu'(s) \in \Downarrow(\triangleleft(\xi(s)))$  then  $\triangleright(s) \in \mathcal{D}^{\circ}$  and  $\mu(\triangleright(s)) := \triangleright(\xi(s))$ .
- Else  $\triangleright(s) \in \mathcal{D}^{\bullet}$ ,  $\mu(\triangleright(s)) := \triangleright(\xi(s))$  and  $\mu'(\triangleright(s)) := \mu'(s)$ .
Finally, for each leaf  $s \in \mathcal{D}^{\star}$  we have:
- If  $s \in \mathcal{D}^{\circ}$  then  $\mu(s)$  is a leaf of  $\mathcal{J}$ .
- If  $s \in \mathcal{D}^{\bullet}$  then there exists  $u \in \mathcal{J}^{\dagger}$  such that  $\mu(s) = \mu'(s) = u$ .
Intuitively, each vertex  $s \in \mathcal{D}$  is associated with a subtree  $\hat{\mathcal{J}}(s)$  of  $\mathcal{J}$ . If  $s \in \mathcal{D}^\circ$  then  $\hat{\mathcal{J}}(s) := \Downarrow(\mu(s))$  and if  $s \in \mathcal{D}^\bullet$  then  $\hat{\mathcal{J}}(s)$  is the subtree of all descendants of  $\mu(s)$  which are not proper descendants of  $\mu'(s)$ . For every  $s \in \mathcal{D}$  such that  $\hat{\mathcal{J}}(s)$  contains only a single vertex, we have that  $s$  is a leaf of  $\mathcal{D}$ . Otherwise  $s$  is an internal vertex of  $\mathcal{D}$  and its children are as follows. We say that  $\hat{\mathcal{J}}(s)$  is split at the vertex  $\xi(s) \in \hat{\mathcal{J}}(s)^\dagger$ . If  $s \in \mathcal{D}^\bullet$  we require that  $\xi(s)$  is on the path in  $\mathcal{J}$  from  $\mu(s)$  to  $\mu'(s)$ . The action of splitting  $\hat{\mathcal{J}}(s)$  at  $\xi(s)$  partitions  $\hat{\mathcal{J}}(s)$  into the subtrees  $\hat{\mathcal{J}}(\triangleleft(s)), \hat{\mathcal{J}}(\nabla(s))$  and  $\hat{\mathcal{J}}(\triangleright(s))$  defined as follows:
-  $\hat{\mathcal{J}} (\triangleleft (s))\coloneqq \Downarrow (\triangleleft (\xi (s)))\cap \hat{\mathcal{J}} (s).$
-  $\hat{\mathcal{J}} (\triangleright (s))\coloneqq \Downarrow (\triangleright (\xi (s)))\cap \hat{\mathcal{J}} (s).$
$$
\cdot \hat {\mathcal {J}} (\nabla (s)) := \hat {\mathcal {J}} (s) \backslash (\hat {\mathcal {J}} (\triangleleft (s)) \cup \hat {\mathcal {J}} (\triangleright (s))).
$$
Utilising the methodology of [22] we will maintain TSTs of  $\mathcal{Z}_t$  (at each trial  $t$ ) and the trees in  $\{\mathcal{A}(v) \mid v \in \mathcal{B}'\}$ , each with height  $\mathcal{O}(\ln(T))$ . Note that these trees are dynamic, in that vertices are inserted into them over time. [22] shows how, after such an insertion, the corresponding TST can be rebalanced, in time  $\mathcal{O}(\ln(T))$ , so that its height is still in  $\mathcal{O}(\ln(T))$ . This rebalancing is performed via a sequence of  $\mathcal{O}(\ln(T))$  tree rotations, which generalise the concept of tree rotations in binary search trees.
D.2 Searching
In this section we show how we can use our TSTs to implement the operation  $\mathrm{INSERT}_t(\mathcal{I})$  on any trial  $t$  and contraction  $\mathcal{J}$  of  $\mathcal{Z}_{t - 1}$ . To do this we need to perform the following two search operations: