Title: Nearest Neighbour with Bandit Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: Nearest Neighbours

Contextual Bandits
Abstract: In this paper we adapt the nearest neighbour rule to the contextual bandit problem. Our algorithm handles the fully adversarial setting in which no assumptions at all are made about the data-generation process. When combined with a sufficiently fast data-structure for (perhaps approximate) adaptive nearest neighbour search, such as a navigating net, our algorithm is extremely efficient - having a per trial running time polylogarithmic in both the number of trials and actions, and taking only quasi-linear space. We give generic regret bounds for our algorithm and further analyse them when applied to the stochastic bandit problem in euclidean space. A side result of this paper is that, when applied to the online classification problem with stochastic labels, our algorithm can, under certain conditions, have sublinear regret whilst only finding a single nearest neighbour per trial - in stark contrast to the k-nearest neighbours algorithm.

Nearest Neighbour with Bandit Feedback

Abstract

In this paper we adapt the nearest neighbour rule to the contextual bandit problem. Our algorithm handles the fully adversarial setting in which no assumptions at all are made about the data-generation process. When combined with a sufficiently fast data-structure for (perhaps approximate) adaptive nearest neighbour search, such as a navigating net, our algorithm is extremely efficient - having a per trial running time polylogarithmic in both the number of trials and actions, and taking only quasi-linear space. We give generic regret bounds for our algorithm and further analyse them when applied to the stochastic bandit problem in euclidean space. We note that our algorithm can also be applied to the online classification problem.

1 Introduction

In this paper we adapt the classic nearest neighbour rule to the contextual bandit problem and develop an extremely efficient algorithm. The problem proceeds in trials, where on trial t : (1) a context x t is revealed to us, (2) we must select an action a t , and (3) the loss ℓ t,a t ∈ [0 , 1] of action a t on trial t is revealed to us. We assume that the contexts are points in a metric space and the distance between two contexts represents their similarity. A policy is a mapping from contexts to actions and the inductive bias of our algorithm is towards learning policies that typically map similar contexts to similar actions. Our main result has absolutely no assumptions whatsoever about the generation of the context/loss sequence and has no restriction on what policies we can compare our algorithm to.

Our algorithm requires, as a subroutine, a data-structure that performs c -nearest neighbour search. This data-structure must be adaptive - in that new contexts can be inserted into it over time. An example of such a data-structure is the Navigating net [17] which, given mild conditions on our metric and dataset, performs both search and insertion in polylogarithmic time. When utilising a data-structure of this speed our algorithm is extremely efficient - with a per-trial time complexity polylogarithmic in both the number of trials and actions, and requiring only quasi-linear space.

As an example we will apply our methodology to the special case of the contextual bandit problem in which the context sequence is drawn i.i.d. from a probability distribution over the d -dimensional hypercube. In this case, for any policy ˆ y with a finite-volume decision boundary, our algorithm achieves ˜ O ( ˆ αT d/ ( d +1) K 1 / ( d +1) ) regret w.r.t. ˆ y , where ˆ α measures the magnitude of what is essentially the part of the decision boundary of ˆ y that lies in the support of the probability distribution.

In the course of this paper we develop some novel algorithmic techniques, including a new algorithmic framework CANPROP and efficient algorithms for searching in trees, which may find further application.

We now describe related works. The bandit problem [20] was first introduced in [28] but was originally studied in the stochastic setting in which all losses are drawn i.i.d. at random [18], [1], [3]. However, our world is very often not i.i.d. stochastic. The work of [4] introduced the seminal EXP3 algorithm which handled the case in which the losses were selected arbitrarily. This work also

introduced the EXP4 algorithm for contextual bandits. In general this algorithm is exponential time but in some situations can be implemented in polynomial time - such as their EXP3.S algorithm, which was a bandit version of the classic FIXEDSHARE algorithm [14]. In [12] the EXP3.S setting was greatly generalised to the situation in which the contexts where vertices of a graph. They utilised the methodology of [8], [16] and [13] in order to develop extremely efficient algorithms. Although inspiring this work, these algorithms cannot be utilised in our situation as they inherently require the set of queried contexts to be known a-priori. In the stochastic case another class of contextual bandit problems are linear bandits [21], [5] in which the contexts are mappings from the actions into R d . Here the queried contexts need not be known in advance but the losses must be drawn i.i.d. from a distribution that has mean linear in the respective context. The k nearest neighbour algorithm was first analysed in [6]. The work [26] utilised the k nearest neighbour methodology and the works [9] and [18] to handle a stochastic contextual bandit problem. However, their setting is extremely more restricted than ours. In particular, the context/loss pairs must be drawn i.i.d. at random and the probability distribution they are sampled from must obey certain strict conditions. In addition, on each trial the contexts seen so far must be ordered in increasing distance from the current context and operations must be performed on this sequence, making their algorithm exponentially slower than ours. Our algorithm utilises the works of [22] and [7] as subroutines. It should be noted that the later work, which was based on [23], was improved on in [11] - we leave it as an open problem as to whether we can utilise their work in our algorithm.

Algorithms for contextual bandits in metric spaces have been studied in [15, 19, 24, 25, 27, 29, 30, 31, 2, 26] but as far as we know ours is the first work to give a non-trivial bound for our problem in general (with no additional assumptions). As far as we are aware our above example bound for the fully stochastic case in euclidean space (with finite decision boundary) is also novel - the works [30, 24, 26] scale as ˜ O ( T ( d +1) / ( d +2) ) in general (and [24, 26] also require additional assumptions). As far as we are aware we are also the first work, stochastic or otherwise, to give a regret scaling as ˜ O ( T 1 / 2 ) when the contexts are drawn from well-separated clusters (i.e. there is a positive distance between all pairs of clusters) in a finite-dimensional metric space, and the comparator policy is constant on each cluster (we do, however, conjecture that in the stochastic case [26] can obtain such a regret scaling here - but, as stated above, it is exponentially slower).

2 Notation

Let N be the set of natural numbers not including 0 . Given a natural number m ∈ N we define [ m ] := { j ∈ N | j ≤ m } . Given a predicate p we define J p K := 1 if p is true and J p K := 0 otherwise. We define log( · ) and ln( · ) to be the logarithms with base 2 and e respectively. Given sets A and B we denote by B A the set of all functions f : A → B and by 2 A the set of all subsets of A . We write x ∼ [0 , 1] to mean that the value x is drawn from the uniform distribution on [0 , 1] .

All trees in this paper are considered rooted. Given a tree J we denote its root by r ( J ) , its vertex set by J , its leaves by J ⋆ , and its internal vertices by J † . Given a vertex v in a tree J we denote its parent by ↑ J ( v ) and the subtree of all its descendants by ⇓ J ( v ) . Given an internal node v in a (full) binary tree J we denote its left and right children by ◁ J ( v ) and ▷ J ( v ) respectively. Internal nodes v in a (full) ternary tree J have an additional child ▽ J ( v ) called the centre child. Given vertices v and v ′ in a tree J we denote by Γ J ( v, v ′ ) the least common ancestor of v and v ′ : i.e. the vertex of maximum depth which is an ancestor of both v and v ′ . We will drop the subscript J in all these functions when unambiguous. Given a tree J , a subtree of J is a tree whose edge set is a subset of that of J .

3 Results

3.1 The General Result

We consider the following game between Learner (us) and Nature (our adversary). We call this game the similarity bandit problem . We have K actions . Learning proceeds in T trials. A-priori Nature chooses a sequence ⟨ n ( t ) | t ∈ [ T ] \ { 1 }⟩ where for all t ∈ [ T ] \ { 1 } we have n ( t ) ∈ [ t -1] . A-priori Nature also chooses a sequence of loss vectors ⟨ ℓ t | t ∈ [ T ] ⟩ ⊆ [0 , 1] K , but does not reveal them to Learner. On the t -th trial the following happens:

If t > 1 then Nature reveals n ( t ) to Learner.

Learner chooses some action a t ∈ [ K ] .

Nature reveals ℓ t,a t to Learner.

Intuitively, given a trial t ∈ [ T ] \ { 1 } , n ( t ) is a similar trial to t . Our inductive bias is that if an action a ∈ [ K ] is good for trial t then it is likely that a will also be good for the similar trial n ( t ) . We will measure our performance with respect to any policy y , where a policy is defined as a vector in [ K ] T . Specifically, we wish to minimise the y -regret , which is defined as the difference between the total cumulative loss suffered by Learner and that which Learner would have suffered if it had instead chosen a t equal to y t for all trials t . Formally, this quantity is defined as follows:

Definition 3.1. Given a policy y ∈ [ K ] T we define the y -regret of Learner as:

The following quantity quantifies how much a policy agrees with our inductive bias.

Definition 3.2. Given a policy y ∈ [ K ] T we define the complexity of y by:

̸

We now state our main result:

Theorem 3.3. Consider the similarity bandit problem described above. Our algorithm CBNN takes a single parameter ρ > 0 and, for all policies y ∈ [ K ] T simultaneously, obtains an expected y -regret bounded by:

where the expectation is taken over the randomisation of the algorithm. CBNN needs no initialisation time and has a per-trial time complexity of:

3.2 Bandits in a Metric Space

We consider the following game between Learner (us) and Nature (our adversary). We call this game the metric bandit problem . We have K actions and a metric space ( C , ∆) where C is a (possibly infinite) set of contexts and for all x, x ′ ∈ C we have that ∆( x, x ′ ) is the distance from x to x ′ . We assume that Learner does not necessarily know ( C , ∆) a-priori but has access to an oracle for computing ∆( x, x ′ ) for any x, x ′ ∈ C . Learning proceeds in T trials. A-priori Nature chooses a sequence of contexts ⟨ x t | t ∈ [ T ] ⟩ ⊆ C and a sequence of loss vectors ⟨ ℓ t | t ∈ [ T ] ⟩ ⊆ [0 , 1] K , but does not reveal them to Learner. On the t -th trial the following happens:

Nature reveals x t to Learner.

Learner chooses some action a t ∈ [ K ] .

Nature reveals ℓ t,a t to Learner.

Here, our inductive bias is that if an action a ∈ [ K ] is good for a context x ∈ C then it is likely also good for contexts that are near to x with respect to the metric ∆ . Our algorithm for this problem will be based on the concept of a c -nearest neighbour which is defined as follows.

Definition 3.4. Given some c ≥ 1 , a finite set S ⊆ C , and a context x ∈ C we have that some ˆ x ∈ S is a c -nearest neighbour of x in the set S if and only if:

In order to utilise CBNN for this problem we need a data-structure for adaptive nearest neighbour search . This problem is as follows. We maintain a finite set S ⊆ C . At any point in time we must either:

Insert a new context x ∈ C into the set S and update the data-structure.

Given a context x ∈ C , utilise the data-structure to find a c -nearest neighbour of x in the set S .

An efficient example of such a data-structure is the navigating net [17].

We can now reduce the metric bandit problem to the similarity bandit problem as follows. On any trial t ∈ [ T ] \ { 1 } choose ˆ x t to be a c -nearest neighbour of x t in the set { x t ′ | t ′ ∈ [ t -1] } . Then choose n ( t ) ∈ [ t -1] such that x n ( t ) = ˆ x t .

We will utilise the following definition in order to bound the complexity of policies when n ( · ) is chosen in this way.

Definition 3.5. Given a function ˆ y : C → [ K ] and a set X ⊆ C then for all x ∈ C define:

̸

We can now bound the complexity of policies as follows, noting that by Theorem 3.3 this leads directly to a regret bound.

Theorem 3.6. Assume we have a sequence ⟨ x t | t ∈ [ T ] ⟩ ⊆ C and a function ˆ y : C → [ K ] . Define X := { x t | t ∈ [ T ] } and for all t ∈ [ T ] define y t := ˆ y ( x t ) . Assume that for all t ∈ [ T ] \ { 1 } we have that x n ( t ) is a c -nearest neighbour of x t in the set { x t ′ | t ′ ∈ [ t -1] } . Then Φ( y ) is no greater than the minimum cardinality of any set S ⊆ C in which for all x ′ ∈ X there exists x ∈ S with ∆( x, x ′ ) < γ ( x, ˆ y, X ) / 3 c .

A strength of our algorithm is that it can be combined with binning algorithms, where the contexts x t are partitioned into sets called bins and, for each bin, all contexts in that bin are replaced by a single context called the centre of the bin. The advantage of binning is that γ ( x, ˆ y, X ) can increase, so that by Theorem 3.6 we may have that Φ( y ) decreases. However, when a context x t is binned (i.e. replaced by its bin centre) its label ˆ y ( x t ) can change, increasing the final regret by O (1) . In Section 3.3 we give an example of the utilisation of binning.

3.3 Stochastic Bandits in Euclidean Space

As an example we now consider the utilisation of the above algorithms for the problem of stochastic bandits in [0 , 1] d for some arbitrary d ∈ N which we view as a constant in our bounds. Here we will only focus on what happens in the limit T →∞ . We focus on stochastic bandits for simplicity, but the same methodology can be used to study the limiting behaviour of adversarial bandits. In this problem we have an unknown probability density ˜ µ : [0 , 1] d × [0 , 1] K → R . We have T trials. On trial t the following happens:

Nature draws ( z t , ℓ t ) from ˜ µ .

Nature reveals z t to Learner.

Learner chooses some action a t ∈ [ K ] .

Nature reveals ℓ t,a t to Learner.

To aid us in this problem we will first quantise the contexts z t to a grid. Note that this is an example of binning. The grid is defined as follows:

Definition 3.7. Given q ∈ N define G d q to be the set of vectors in [0 , 1] d in which each component is an integer multiple of 1 /q .

On each trial t we will first quantise the vector z t by defining x t to be its nearest neighbour (w.r.t. the euclidean metric) in G d q , where q := ( T/K ) 1 / ( d +1) . Note that this can be done in constant time per trial. As in Section 3.2 we then use CBNN to solve the problem by defining n ( t ) such that x n ( t ) is a c -nearest neighbour (w.r.t. the euclidean metric) of x t in the set { x t ′ | t ′ ∈ [ t -1] } . We consider c as a constant in our bounds.

As before, we will compare our cumulative loss to that of a policy that follows a function ˆ y : [0 , 1] d → [ K ] . Our regret bound will be based on the following quantities:

Definition 3.8. Let µ be the marginal of ˜ µ with respect to its first argument and let ∆ be the euclidean metric on [0 , 1] d . For all ϵ > 0 define:

̸

which is the set of contexts that are within distance ϵ of the support of µ . Given ˆ y : [0 , 1] d → [ K ] we make the following definitions. For any δ > 0 define:

̸

which is the set of contexts that are at distance no more than δ from the intersection of the decision boundary of ˆ y and E ( µ, ϵ ) . We then define:

which are essentially the volumes of the part of the decision boundary of ˆ y that lies in the support of µ , with respect to the uniform density and the density µ respectively.

With these definitions in hand we now present our regret bound, which utilises Theorem 3.6 in its proof:

Theorem 3.9. Let q := ⌈ ( T/K ) 1 / ( d +1) ⌉ . For all t ∈ [ T ] let x t be the nearest neighbour of z t in the set G d q . For all t ∈ [ T ] \ { 1 } let n ( t ) be such that x n ( t ) is a c -nearest neighbour of x t in the set { x t ′ | t ′ ∈ [ t -1] } . Given some ˆ y : [0 , 1] d → [ K ] let y t := ˆ y ( z t ) for all t ∈ [ T ] . Then when ρ := q d -1 2 CBNN gives us:

as T →∞ .

We note that varying q and ρ in Theorem 3.9 will allow us to trade off the values 1 , α (ˆ y, µ ) and ˜ α (ˆ y, µ ) in different ways.

4 The Algorithm

In this section we describe our algorithm CBNN, for solving the similarity bandit problem, and give the pseudocode for the novel subroutines. In the appendix we give a more detailed description of how CBNN works and prove our theorems.

Instead of working directly with trial numbers we create a sequence of distinct nodes ⟨ x t | t ∈ [ T ] ⟩ and define, for all t ∈ [ T ] \ { 1 } , the node n ( x t ) := x n ( t ) . Let X := { x t | t ∈ [ T ] } . We can now represent policies as functions from X into [ K ] . Hence, given some y : X → [ K ] , we define the y -regret and the complexity of y as:

̸

respectively.

4.1 A Simple but Inefficient Algorithm

To give the reader intuition we first describe our initial idea - a simple algorithm which attains our desired regret bound but is exponentially slower - taking a per-trial time of ˜ Θ( KT ) . The algorithm is based on EXP4 [4] which we now describe. On every trial t we maintain a weighting ˆ w t : [ K ] X → [0 , 1] . We are free to choose the initial weighting ˆ w 1 to be any probability distribution. On each trial t the following happens:

For all a ∈ [ K ] set p t,a ← ∑ y ∈ [ K ] X J y ( x t ) = a K ˆ w t ( y ) .

Set a t ← a with probability proportional to p t,a .

Receive ℓ t,a t .

For all a ∈ [ K ] set ˆ ℓ t,a ← J a = a t K ℓ t,a t ∥ p t ∥ 1 /p t,a t .

For all y ∈ [ K ] X set ˆ w t +1 ( y ) ← ˆ w t ( y ) exp( -η ˆ ℓ t,y ( x t ) ) .

For us we choose, for all y : X → [ K ] , an initial weight of:

Of course, we don't know Φ( y ) a-priori, and hence we cannot implement EXP4 explicitly (and it would take exponential time even if we did know Φ( y ) a-priori). Our crucial insight is the following. For any t ∈ [ T ] let X t := { x t ′ | t ′ ∈ [ t ] } and for any y : X → [ K ] let y t be the restriction of y onto X t . Then for any t ∈ [ T ] and for any y ′ : X t → [ K ] we have:

̸

Note that, for all t ∈ [ T ] and a ∈ [ K ] , we can write p t,a as follows:

By substituting in Equation (1) we have now brought p t,a into a form that can be solved via Belief propagation [23] over the tree with vertex set X t and in which, for all t ′ ∈ [ t ] \ { 1 } , the parent of x t ′ is n ( x t ′ ) .

It is well known that for any y : X → [ K ] , EXP4 attains a y -regret of at most ln( ˆ w 1 ( y )) /η + ηKT/ 2 . By setting η := ρ/ √ KT and noting our choice of ˆ w 1 we obtain our desired regret bound.

4.2 Cancellation Propagation

In the remainder of this section we describe our algorithm CBNN, which is based on the same idea as the simple algorithm of Section 4.1.

In this subsection we describe a novel algorithmic framework CANPROP for designing contextual bandit algorithms with a running time logarithmic in K . It is inspired by EXP3 [4], specialist algorithms [8] and online decision-tree pruning algorithms [10] but is certainly not a simple combination of these works. CBNN will be an efficient implementation of an instance of CANPROP. Although in general CANPROP requires a-priori knowledge, CBNN is designed in a way that, crucially, does not need it to be known.

We assume, without loss of generality, that K and T are integer powers of two. CANPROP, which takes a parameter η > 0 , works on a full, balanced binary tree B with leaves B ⋆ = [ K ] . On every trial t each pair ( v, S ) ∈ B × 2 X has a weight w t ( v, S ) ∈ [0 , 1] . These weights induce a function θ t : B → [0 , 1] defined by:

On each trial t a root-to-leaf path { v t,j | j ∈ [log( K )] ∪{ 0 }} is sampled such that v t, 0 := r ( B ) and , given v t,j , we have that v t, ( j +1) is sampled from { ◁ ( v t,j ) , ▷ ( v t,j ) } with probability proportional to the value of θ t when applied to each of these vertices. The action a t is then chosen equal to v t, log( K ) . Once the loss has been observed we climb back up the root-to-leaf path, updating the function w t to w t +1 .

CANPROP (at trial t ) is given in Algorithm 1. We note that if w t +1 ( v, S ) is not set in the pseudocode then it is defined to be equal to w t ( v, S ) .

In Appendix B we give a general regret bound for CANPROP. For CBNN we set:

Algorithm 1 CANPROP at trial t

and for all ( v, S ) ∈ B × 2 X we set:

where:

JJ K J KK This choice gives us the regret bound in Theorem 3.3. We note that CBNN will be implemented in such a way that n need not be known a-priori.

4.3 Ternary Search Trees

As we shall see, CBNN works by storing a binary tree A ( v ) at each vertex v ∈ B . In order to perform efficient operations on these trees we will utilise the rebalancing data-structure defined in [22] which here we shall call a ternary search tree (TST) due to the fact that it is a generalisation of the classic binary search tree and, as we shall show, has searching applications. However, as for binary search trees, the applications of TSTs are more than just searching: we shall also utilise them for online belief propagation.

We now define what is meant by a TST. Suppose we have a full binary tree J . A TST of J is a full ternary tree D which satisfies the following. The vertex set of D is partitioned into two sets D ◦ and D · where each vertex s ∈ D is associated with a vertex µ ( s ) ∈ J and every s ∈ D · is also associated with a vertex µ ′ ( s ) ∈ ⇓ ( µ ( s )) † . In addition, each internal vertex s ∈ D † is associated with a vertex ξ ( s ) ∈ J . For all u ∈ J there exists an unique leaf Υ D ( u ) ∈ D ⋆ in which µ (Υ D ( u )) = u .

Essentially, each vertex s ∈ D corresponds to a subtree ˆ J ( s ) of J where ˆ J ( r ( D )) = J . Such a vertex s is a leaf of D if and only if | ˆ J ( s ) | = 1 . For each internal vertex s ∈ D † the subtree ˆ J ( s ) is split at the vertex ξ ( s ) into the subtrees ˆ J ( ◁ ( s )) , ˆ J ( ▽ ( s )) , and ˆ J ( ▷ ( s )) corresponding to the children of s . The process continues recursively.

For completeness we now describe the rules that a TST D of J must satisfy. We have that r ( D ) ∈ D ◦ and µ ( r ( D )) := r ( J ) . Each vertex s ∈ D represents a subtree ˆ J ( s ) of J . If s ∈ D ◦ then ˆ J ( s ) := ⇓ ( µ ( s )) and otherwise ˆ J ( s ) is the set of all descendants of µ ( s ) which are not proper descendants of µ ′ ( s ) . Given that s ∈ D † this subtree is split at the vertex ξ ( s ) where if s ∈ D · we have that ξ ( s ) lies on the path from µ ( s ) to µ ′ ( s ) . The children of s are then defined so that ˆ J ( ◁ ( s )) = ˆ J ( s ) ∩⇓ ( ◁ ( ξ ( s ))) and ˆ J ( ▷ ( s )) = ˆ J ( s ) ∩⇓ ( ▷ ( ξ ( s ))) and ˆ J ( ▽ ( s )) = ˆ J ( s ) \ ( ˆ J ( ◁ ( s )) ∪ ˆ J ( ▷ ( s ))) . i.e. ξ ( s ) partitions the subtree ˆ J ( s ) into the subtrees ˆ J ( ◁ ( s )) , ˆ J ( ▷ ( s )) , and ˆ J ( ▽ ( s )) . This process continues recursively until | ˆ J ( s ) | = 1 in which case s is a leaf of D .

̸

For all binary trees J in our algorithm we shall maintain a TST H ( J ) of J with height O (ln( |J | )) . Such trees J are dynamic in that on any trial it is possible that two vertices, u and u ′ , are added to the tree J such that u ′ is inserted between a non-root vertex of J and its parent, and u is designated as a child of u ′ . We define the subroutine REBALANCE ( H ( J ) , u ) as one which rebalances the TST H ( J ) after this insertion, so that the height of H ( J ) always remains in O (ln( |J | )) . The work of [22] describes how this subroutine can be implemented in a time of O (ln( |J | )) and we refer the reader to this work for details (noting that they use different notation).

4.4 Contractions

Define the quantity ϕ 0 := 0 and for all j ∈ N ∪ { 0 } inductively define:

At any trial t the contexts in { x s | s ∈ [ t ] } naturally form a tree by designating n ( x s ) as the parent of x s . However, to utilise the TST data-structure we must only have binary trees. Hence, we will work with a (dynamic) full binary tree Z which, on trial t , is a binarisation of the above tree. The relationship between these two trees is given by a map γ : Z t →{ x s | s ∈ [ t ] } where Z t is the tree Z on trial t . For all x ∈ { x s | s ∈ [ t ] } we will always have an unique leaf ˜ γ ( x ) ∈ Z ⋆ t in which γ (˜ γ ( x )) = x . We also maintain a balanced TST H ( Z ) of Z .

Algorithm 2 gives the subroutine GROW t which updates Z at the start of trial t . Note that GROW t also defines a function d : Z → N such that d ( u ) is the number of times the function n must be applied to γ ( u ) to reach x 1 .

A contraction (of Z ) is defined as a full binary tree J in which the following holds. (1) The vertices of J are a subset of those of Z . (2) r ( J ) = r ( Z ) . (3) Given a vertex u ∈ J we have ◁ J ( u ) ∈ ⇓ Z ( ◁ Z ( u )) and ▷ J ( u ) ∈ ⇓ Z ( ▷ Z ( u )) . (4) Any leaf of J is a leaf of Z .

CBNN will maintain, on every vertex v ∈ B , a contraction A ( v ) as well as a TST H ( A ( v )) of A ( v ) . Given J is one of these contractions, we also maintain, for all i, i ′ ∈ { 0 , 1 } and all u ∈ J , a value τ i,i ′ ( J , u ) ∈ R + . Technically these quantities, which depend on the above function d , define a bayesian network on J which is explained in Appendix C.3. For all i ∈ { 0 , 1 } and all u ∈ J we also maintain a value κ i ( J , u ) initialised equal to 1 .

On each of our contractions J we will define, on trial t , a subroutine INSERT t ( J ) that simply modifies J so that ˜ γ ( x t ) is added to its leaves. This subroutine is only called on certain trials t . Specifically, it is called on the contraction A ( v ) only when v is involved in CANPROP on trial t . Although the effect of this subroutine is simple to describe, its polylogarithmic-time implementation is quite complex. A function that is used many times during this subroutine is ν : Z×Z → { ◀ , ▶ , ▲ } in which ν ( u, u ′ ) is equal to ◀ , ▶ or ▲ if u ′ is contained in ⇓ Z ( ◁ Z ( u )) , in ⇓ Z ( ▷ Z ( u )) or in neither, respectively. Algorithm 3 shows how to compute this function. Now that we have a subroutine for computing ν we can turn to the pseudocode for the subroutine INSERT t ( J ) in Algorithm 4. In the appendix we give a full description of how and why this subroutine works.

Algorithm 3 Computing ν ( u, u ′ ) for u, u ′ ∈ Z

̸

Algorithm 4 The operation INSERT t ( J ) on a contraction J of Z at trial t

4.5 Online Belief Propagation

In this subsection we utilise the work of [7] in order to be able to efficiently compute the function θ t that appears in CANPROP.

Given a vertex u in one of our contractions J we define F ( J , u ) := { f ∈ { 0 , 1 } J | f ( u ) = 1 } and then define:

As stated in the previous subsection, when a vertex v ∈ B becomes involved in CANPROP on trial t , CBNN will add ˜ γ ( x t ) to the leaves of A ( v ) via the operation INSERT t ( A ( v )) . In the appendix we shall show that for each such v we then have:

We now outline how to compute this efficiently, deferring a full description for Appendix D.3. First note that for all contractions J and all u ∈ J we have that Λ( J , u ) is of the exact form to be solved by the classic Belief propagation algorithm [23]. The work of [7] shows how to compute this term in logarithmic time by maintaining a data-structure based on a balanced TST of J - in our case the TST H ( J ) . Whenever, for some i ∈ { 0 , 1 } and u ′ ∈ J , the value κ i ( J , u ′ ) changes, the data-structure is updated in logarithmic time. We define the subroutine EVIDENCE ( J , u ′ ) as that which updates this data-structure after κ i ( J , u ′ ) changes. We also make sure that the data-structure is updated whenever REBALANCE ( H ( J ) , · ) is called. We then define the subroutine MARGINAL ( J , u ) as that which computes Λ( J , u ) / 4 . Hence, the output of MARGINAL ( A ( v ) , ˜ γ ( x t )) is equal to θ t ( v ) .

4.6 CBNN

Now that we have defined all our subroutines we give, in Algorithm 5, the algorithm CBNN which is an efficient implementation of CANPROP with initial weighting given in Equation (2).

Algorithm 5 CBNN at trial t

5 Acknowledgments

We would like to thank Mark Herbster (University College London) for valuable discussions.

Research funded by the Defence Science and Technology Laboratory (Dstl) which is an executive agency of the UK Ministry of Defence providing world class expertise and delivering cutting-edge science and technology for the benefit of the nation and allies. The research supports the Autonomous Resilient Cyber Defence (ARCD) project within the Dstl Cyber Defence Enhancement programme.

References

R. Agrawal. Sample mean based index policies by o(log n) regret for the multi-armed bandit problem. Advances in Applied Probability , 27:1054 - 1078, 1995.

S. Arya and Y. Yang. Randomized allocation with nonparametric estimation for contextual multi-armed bandits with delayed rewards. Statistics & Probability Letters , 2020.

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47:235-256, 2002.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput. , 32:48-77, 2002.

W. Chu, L. Li, L. Reyzin, and R. E. Schapire. Contextual bandits with linear payoff functions. In International Conference on Artificial Intelligence and Statistics , 2011.

T. M. Cover and P. E. Hart. Nearest neighbor pattern classification. IEEE Trans. Inf. Theory , 13:21-27, 1967.

A. L. Delcher, A. J. Grove, S. Kasif, and J. Pearl. Logarithmic-time updates and queries in probabilistic networks. J. Artif. Intell. Res. , 4:37-59, 1995.

Y. Freund, R. E. Schapire, Y. Singer, and M. K. Warmuth. Using and combining predictors that specialize. In Symposium on the Theory of Computing , 1997.

A. Garivier and O. Cappé. The kl-ucb algorithm for bounded stochastic bandits and beyond. In Annual Conference Computational Learning Theory , 2011.

D. P. Helmbold and R. E. Schapire. Predicting nearly as well as the best pruning of a decision tree. Machine Learning , 27:51-68, 1995.

M. Herbster, S. Pasteris, and F. Vitale. Online sum-product computation over trees. In NIPS , 2012.

M. Herbster, S. Pasteris, F. Vitale, and M. Pontil. A gang of adversarial bandits. In Neural Information Processing Systems , 2021.

M. Herbster and J. Robinson. Online prediction of switching graph labelings with cluster specialists. In Neural Information Processing Systems , 2018.

M. Herbster and M. K. Warmuth. Tracking the best expert. Machine Learning , 32:151-178, 1995.

S. M. Kakade, S. Shalev-Shwartz, and A. Tewari. Efficient bandit algorithms for online multiclass prediction. In International Conference on Machine Learning , 2008.

W. M. Koolen, D. Adamskiy, and M. K. Warmuth. Putting bayes to sleep. In NIPS , 2012.

R. Krauthgamer and J. R. Lee. Navigating nets: simple algorithms for proximity search. In ACM-SIAM Symposium on Discrete Algorithms , 2004.

T. L. Lai and H. E. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6:4-22, 1985.

J. Langford and T. Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. In NIPS 2007 , 2007.

T. Lattimore and C. Szepesvári. Bandit algorithms. 2020.

L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In The Web Conference , 2010.

K. Matsuzaki and A. Morihata. Balanced ternary-tree representation of binary trees and balancing algorithms. In Mathematical Engineering Technical Reports , 2008.

J. Pearl. Reverend bayes on inference engines: A distributed hierarchical approach. Probabilistic and Causal Inference , 1982.

V. Perchet and P. Rigollet. The multi-armed bandit problem with covariates. The Annals of Statistics , 2013.

W. Qian and Y. Yang. Kernel estimation and model combination in a bandit problem with covariates. J. Mach. Learn. Res. , 17:149:1-149:37, 2016.

H. W. J. Reeve, J. C. Mellor, and G. Brown. The k-nearest neighbour ucb algorithm for multi-armed bandits with covariates. Algorithmic Learning Theory , 2018.

P. Rigollet and A. J. Zeevi. Nonparametric bandits with covariates. In Annual Conference Computational Learning Theory , 2010.

H. E. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58:527-535, 1952.

Y. Seldin, P. Auer, F. Laviolette, J. Shawe-Taylor, and R. Ortner. Pac-bayesian analysis of contextual bandits. In NIPS , 2011.

A. Slivkins. Contextual bandits with similarity information. Journal of Machine Learning Research , 2014.

C.-C. Wang, S. R. Kulkarni, and H. V. Poor. Arbitrary side observations in bandit problems. Adv. Appl. Math. , 34:903-938, 2005.

A Introduction to the Appendix

We now turn to the full description and analysis of our algorithm CBNN. In Appendix B we describe our novel algorithmic framework CANPROP. In Appendix C we describe contractions and bayesian networks on them, showing how CANPROP can be implemented with them. Finally, in Appendix D we describe TSTs and how they are used to perform our required operations efficiently. In Appendix E we prove, in order, all of the theorems stated in this paper.

Recall that we created a sequence of distinct nodes ⟨ x t | t ∈ [ T ] ⟩ and defined, for all t ∈ [ T ] \{ 1 } , the node n ( x t ) := x n ( t ) . Let X := { x t | t ∈ [ T ] } . Given some y : X → [ K ] , we defined the y -regret and the complexity of y as:

̸

respectively.

B Cancellation Propagation

B.1 The General CANPROP Algorithm

We now introduce a general algorithmic framework CANPROP for handling contextual bandit problems with a per-trial time logarithmic in K . Without loss of generality assume that K is an integer power of two. Let B be a full, balanced binary tree whose leaves are the set of actions [ K ] . Let B ′ := B \ { r ( B ) } . CANPROP takes a parameter η ∈ R + called the learning rate . On each trial t CANPROP maintains a function:

The function w 1 is free to be defined how one likes, as long as it satisfies the constraint that for all internal vertices v ∈ B † we have:

We now describe how CANPROP acts on trial t . For all v ∈ B ′ we define:

and for all v ∈ B † we define:

As we shall see CANPROP needs only compute these values for O (ln( K )) vertices v . CANPROP samples a root-to-leaf path { v t,j | j ∈ [log( K )] ∪ { 0 }} as follows. v t, 0 is defined equal to r ( B ) . For all j ∈ [log( K ) -1] ∪ { 0 } , once v t,j has been sampled we sample v t, ( j +1) from the probability distribution defined by:

noting that v t, ( j +1) is a child of v t,j . We define:

and then receives the loss ℓ t,a t . The function w t is then updated to w t +1 as follows. Firstly we define,

We then define:

CANPROP then selects:

Once we have defined ψ t,j for some j ∈ [log( K )] we then define:

The regret bound of CANPROP is given by the following theorem.

Theorem B.1. Suppose we have a function y : X → [ K ] . For all v ∈ B define:

Then the expected y -regret of CANPROP is bounded by:

̸

B.2 Our Parameter Tuning

We now describe and analyse the initial weighting w 1 and the learning rate η that we will use. Define X ′ := X \ { x 1 } . For all ( x, S ) ∈ X ′ × 2 X define:

̸

For all ( v, S ) ∈ B ′ × 2 X we define:

Given our parameter ρ we choose our learning rate as:

Given this initial weighting and learning rate, Theorem B.1 implies the following regret bound.

Theorem B.2. Given w 1 and η are defined as above, then for any y : X → [ K ] the expected y -regret of CANPROP is bounded by:

C Implementation with Contractions

C.1 A Sequence of Binary Trees

̸

For any trial t we have a natural tree-structure on the set { x t ′ | t ′ ∈ [ t ] } formed by making n ( x t ′ ) the parent of x t ′ for all t ′ ∈ [ t ] \{ 1 } . However, in order to utilise the methodology of [22] we need to work with binary trees. Hence, we now inductively define a sequence of binary trees ⟨Z t | t ∈ [ T ] \ { 1 }⟩ where the vertices of Z t are a subset of those of Z t +1 . We also define a function γ : Z T →X . This function γ has the property that for any t ∈ [ T ] and for any distinct leaves u, u ′ ∈ Z ⋆ t we have that γ ( u ) = γ ( u ′ ) , and that:

We define Z 2 to contain three vertices { r ( Z 2 ) , ◁ ( r ( Z 2 )) , ▷ ( r ( Z 2 )) } where:

Now consider a trial t ∈ [ T ] . We have that Z t +1 is constructed from Z t via the following algorithm GROW t +1 :

Let u be the unique leaf in Z ⋆ t in which γ ( u ) = n ( x t +1 ) and let u ∗ := ↑ ( u ) .

Create two new vertices u ′ and u ′′ .

Set γ ( u ′ ) ← n ( x t +1 ) and γ ( u ′′ ) ← x t +1 .

If u = ◁ ( u ∗ ) then set ◁ ( u ∗ ) ← u ′ . Else set ▷ ( u ∗ ) ← u ′ .

Set ◁ ( u ′ ) ← u ′′ and ▷ ( u ′ ) ← u .

We also define a function d : Z T → N ∪{ 0 } as follows. Define d ′ ( x 1 ) := 0 and for all t ∈ [ T ] \ { 1 } inductively define d ′ ( x t ) := d ′ ( n ( x t )) + 1 . Finally define d ( u ) := d ′ ( γ ( u )) for all u ∈ Z T . Since for all t ∈ [ T ] we have that the vertices of Z t are a subset of those of Z T we have that d also defines a function over Z t for all t ∈ [ T ] .

For all t ∈ [ T ] we define u t to be the unique leaf of Z t for which γ ( u t ) = x t .

C.2 Contractions

Our efficient implementation of CANPROP will have a data-structure at every vertex v ∈ B ′ . However, to achieve polylogarithmic time per trial we can only update a polylogarithmic number of these data-structures per trial. This necessitates the use of contractions of our trees {Z t | t ∈ [ T ] \ { 1 }} which are defined as follows. A contraction of a full binary tree Q is another full binary tree J which satisfies the following:

The vertices of J are a subset of those of Q .

r ( J ) = r ( Q ) .

Given an internal vertex u ∈ J † we have ◁ J ( u ) ∈ ⇓ Q ( ◁ Q ( u )) and ▷ J ( u ) ∈ ⇓ Q ( ▷ Q ( u )) .

Any leaf of J is a leaf of Q .

Note that any contraction of Z t is also a contraction of Z t +1 and hence, by induction, a contraction of Z t ′ for all t ′ ≥ t . Given a trial t and a contraction J of Z t -1 we now define the operation INSERT t ( J ) which acts on J by the following algorithm:

Let ˆ u be the unique vertex in J \ { r ( J ) } such that u t is in the maximal subtree of Z t with ˆ u and ↑ J (ˆ u ) as leaves.

Let u ∗ := Γ Z t ( u t , ˆ u ) .

Add the vertices u ∗ and u t to the tree J .

Let u ′ := ↑ J (ˆ u ) .

If ˆ u = ◁ J ( u ′ ) then set ◁ J ( u ′ ) ← u ∗ . Else set ▷ J ( u ′ ) ← u ∗ .

If ˆ u ∈ ⇓ Z t ( ◁ Z t ( u ∗ )) then set ◁ J ( u ∗ ) ← ˆ u and ▷ J ( u ∗ ) ← u t . Else set ▷ J ( u ∗ ) ← ˆ u and ◁ J ( u ∗ ) ← u t .

Later in this paper we will show how this operation can be done in polylogarithmic time. Note that after the operation we have that J is a contraction of Z t and u t has been added to it's leaves. From now on when we use the term contraction we mean any contraction of Z T .

C.3 Contraction-Based Bayesian Networks

Here we shall define a bayesian network over any contraction J and show how it can be utilised to compute certain quantities required by CANPROP. First define the quantity ϕ 0 := 0 and for all j ∈ N ∪ { 0 } inductively define:

The algorithm must compute these quantities for all j ∈ [ T ] . However, for all t ∈ [ T ] we have that ϕ t doesn't have to be computed until trial t so computing these quantities is constant time per trial. Given a contraction J , a vertex u ∈ J \ r ( J ) and indices i, i ′ ∈ { 0 , 1 } define:

̸

which defines the transition matrix from ↑ J ( u ) to u in a bayesian network over J . We shall now show how belief propagation over such bayesian networks can be used to compute the quantities we need in CANPROP. Suppose we have a contraction J and a function λ : J ⋆ → R + . This function λ induces a function λ ′ : X → R + defined as follows. Given x ∈ X , if there exists a leaf u ∈ J ⋆ with γ ( u ) = x then λ ′ ( x ) = λ ( u ) . Otherwise λ ′ ( x ) = 1 . For all S ∈ 2 X define:

For the CANPROP algorithm we will need to compute

for some leaf ˆ u ∈ J ⋆ and some function λ : J ⋆ → R + . We shall now show how we can compute this quantity via belief propagation on the bayesian network. In particular, we shall construct a quantity ˜ Λ( J , λ, u ) equal to the quantity in Equation (3). To do this, first define the function λ ∗ : J → R + so that for all u ∈ J ⋆ we have λ ∗ ( u ) = λ ( u ) and for all u ∈ J † we have λ ∗ ( u ) = 1 . For all vertices u ∈ J and all indices i ∈ { 0 , 1 } define:

For all ˆ u ∈ J define:

and then define:

The equality of this quantity and that given in Equation (3) is given by the following theorem.

Theorem C.1. Given a contraction J , a function λ : J ⋆ → R + and some leaf ˆ u ∈ J ⋆ we have:

Note that ˜ Λ( J , λ, ˆ u ) is of the exact form to be solved via belief propagation over J . However, belief propagation is too slow (taking Θ( |J | ) time) - we will remedy this later.

C.4 Cancelation Propagation with Contractions

We now describe how to implement CANPROP with contractions. For each v ∈ B ′ we maintain a contraction A ( v ) and a function λ v : A ( v ) ⋆ → R + . We initialise with A ( v ) identical to Z 2 and λ v ( u ) = 1 for both leaves u ∈ Z ⋆ 2 . Via induction over t we will have that at the start of each trial t we have, for all sets S ∈ 2 X , that:

On trial t we do as follows. First we update Z t -1 to Z t using the algorithm GROW t . We will perform the necessary modifications to our contractions as we sample the path P t . In particular, we first set v t, 0 := r ( B ) and then for each j ∈ [log( K ) -1] ∪ { 0 } in turn we do as follows. For each v ∈ { ◁ ( v t,j ) , ▷ ( v t,j ) } run INSERT t ( A ( v )) and set λ v ( u t ) ← 1 . Since λ v ( u t ) = 1 Equation (4) still holds and hence, by Theorem C.1, we have:

After θ t ( v ) has been computed for both v ∈ { ◁ ( v t,j ) , ▷ ( v t,j ) } we can now sample v t,j +1 .

Once we have selected the action a t we then update the functions { λ v | ↑ B ( v ) ∈ P t } by setting λ v ( u t ) ← β t ( v ) for all v ∈ B ′ with ↑ B ( v ) ∈ P t . It is clear now that Equation (4) holds inductively.

C.5 Notational Relationship to the Main Body

We now point out how the notation in this section relates to that of the main body. In particular, we have, for all v ∈ B ′ , all u ∈ A ( v ) and all i, i ′ ∈ { 0 , 1 } , that:

κ i ( A ( v ) , u ) = ˜ κ i ( A ( v ) , λ v , u ) .

Λ( A ( v ) , u ) = ˜ Λ( A ( v ) , λ v , u ) .

We note that the function λ v does not explicity appear in our pseudocode since it can be inferred from κ i ( A ( v ) , · ) .

D Utilising Ternary Search Trees

There are now only two things left to do in order to achieve polylogarithmic time per trial - to make an efficient online implementation of the INSERT t ( · ) operation and an efficient online algorithm to perform belief propagation over our contractions. In order to do this we will utilise the methodology of [22] which we now describe. However, we do not give the full details of the rebalancing technique and refer the reader to [22] for these details (noting that [22] uses different notation).

D.1 Ternary Search Trees

In this section we will consider a full binary tree J . A (full) ternary tree D is a rooted tree in which each internal vertex s ∈ D † has three children denoted by ◁ ( s ) , ▽ ( s ) , ▷ ( s ) and called the left , centre , and right children respectively. We now define what it means for a ternary tree D to be a ternary search tree (TST) of J . Firstly, the vertex set of D is partitioned into two sets D ◦ and D · . Every vertex s ∈ D is associated with a vertex µ ( s ) ∈ J and every s ∈ D · is also associated with a vertex µ ′ ( s ) ∈ ⇓ J ( µ ( s )) † . The root r ( D ) is contained in D ◦ and µ ( r ( D )) := r ( J ) . Each internal vertex s ∈ D † is associated with a vertex ξ ( s ) ∈ J . If s ∈ D ◦ then ξ ( s ) ∈ ⇓ ( µ ( s )) † and if s ∈ D · then ξ ( s ) lies on the path (in J ) from µ ( s ) to ↑ ( µ ′ ( s )) . For all s ∈ D † we have:

▽ ( s ) ∈ D · , µ ( ▽ ( s )) := µ ( s ) and µ ′ ( ▽ ( s )) := ξ ( s ) .

◁ ( s ) satisfies:

-If s ∈ D ◦ then ◁ ( s ) ∈ D ◦ and µ ( ◁ ( s )) := ◁ ( ξ ( s )) .

-If s ∈ D · and µ ′ ( s ) ∈ ⇓ ( ▷ ( ξ ( s ))) then ◁ ( s ) ∈ D ◦ and µ ( ◁ ( s )) := ◁ ( ξ ( s )) .

-Else ◁ ( s ) ∈ D · , µ ( ◁ ( s )) := ◁ ( ξ ( s )) and µ ′ ( ◁ ( s )) := µ ′ ( s ) .

▷ ( s ) satisfies:

-If s ∈ D ◦ then ▷ ( s ) ∈ D ◦ and µ ( ▷ ( s )) := ▷ ( ξ ( s )) .

-If s ∈ D · and µ ′ ( s ) ∈ ⇓ ( ◁ ( ξ ( s ))) then ▷ ( s ) ∈ D ◦ and µ ( ▷ ( s )) := ▷ ( ξ ( s )) .

-Else ▷ ( s ) ∈ D · , µ ( ▷ ( s )) := ▷ ( ξ ( s )) and µ ′ ( ▷ ( s )) := µ ′ ( s ) .

Finally, for each leaf s ∈ D ⋆ we have:

If s ∈ D ◦ then µ ( s ) is a leaf of J .

If s ∈ D · then there exists u ∈ J † such that µ ( s ) = µ ′ ( s ) = u .

Intuitively, each vertex s ∈ D is associated with a subtree ˆ J ( s ) of J . If s ∈ D ◦ then ˆ J ( s ) := ⇓ ( µ ( s )) and if s ∈ D · then ˆ J ( s ) is the subtree of all descendants of µ ( s ) which are not proper descendants of µ ′ ( s ) . For every s ∈ D such that ˆ J ( s ) contains only a single vertex, we have that s is a leaf of D . Otherwise s is an internal vertex of D and its children are as follows. We say that ˆ J ( s ) is split at the vertex ξ ( s ) ∈ ˆ J ( s ) † . If s ∈ D · we require that ξ ( s ) is on the path in J from µ ( s ) to µ ′ ( s ) . The action of splitting ˆ J ( s ) at ξ ( s ) partitions ˆ J ( s ) into the subtrees ˆ J ( ◁ ( s )) , ˆ J ( ▽ ( s )) and ˆ J ( ▷ ( s )) defined as follows:

ˆ J ( ◁ ( s )) := ⇓ ( ◁ ( ξ ( s ))) ∩ ˆ J ( s ) .

ˆ J ( ▷ ( s )) := ⇓ ( ▷ ( ξ ( s ))) ∩ ˆ J ( s ) .

Utilising the methodology of [22] we will maintain TSTs of Z t (at each trial t ) and the trees in {A ( v ) | v ∈ B ′ } , each with height O (ln( T )) . Note that these trees are dynamic, in that vertices are inserted into them over time. [22] shows how, after such an insertion, the corresponding TST can be rebalanced , in time O (ln( T )) , so that its height is still in O (ln( T )) . This rebalancing is performed via a sequence of O (ln( T )) tree rotations , which generalise the concept of tree rotations in binary search trees.

D.2 Searching

In this section we show how we can use our TSTs to implement the operation INSERT t ( J ) on any trial t and contraction J of Z t -1 . To do this we need to perform the following two search operations:

Find the unique vertex ˆ u ∈ J \ { r ( J ) } such that u t is in the maximal subtree of Z t with ˆ u and ↑ J (ˆ u ) as leaves.

Find u ∗ := Γ Z t ( u t , ˆ u ) .

To perform these tasks in polylogarithmic time we will utilise TSTs E and D of Z t and J respectively. Both the searching tasks utilise a function ν : Z 2 t →{ ▲ , ◀ , ▶ } defined, for all u, u ′ ∈ Z t as follows. If u ′ ∈ ⇓ Z t ( ◁ ( u )) or u ′ ∈ ⇓ Z t ( ▷ ( u )) then ν ( u, u ′ ) := ◀ or ν ( u, u ′ ) := ▶ respectively. Otherwise ν ( u, u ′ ) := ▲ . This can be computed as follows. If u = u ′ then ν ( u, u ′ ) := ▲ . Otherwise let ˜ s and ˜ s ′ be the unique leaves of E such that µ (˜ s ) = u and µ (˜ s ′ ) = u ′ . Let s ∗ := Γ E (˜ s, ˜ s ′ ) and let ˆ s and ˆ s ′ be the children of s ∗ which are ancestors of ˜ s and ˜ s ′ respectively. If ˆ s = ▽ ( s ∗ ) then we have ν ( u, u ′ ) = ▲ . If ξ ( s ∗ ) = u then we have ν ( u, u ′ ) = ◀ or ν ( u, u ′ ) = ▶ if ˆ s ′ = ◁ ( s ∗ ) or ˆ s ′ = ▷ ( s ∗ ) respectively. If ˆ s = ▽ ( s ∗ ) and ξ ( s ∗ ) = u then we perform the following process. Start with s equal to ˆ s . At any point in the process we do as follows. If s ∈ E ◦ then the process terminates with ν ( u, u ′ ) := ▲ . If s ∈ E · and u = ξ ( s ) then the process terminates with ν ( u, u ′ ) = ◀ or ν ( u, u ′ ) = ▶ if ◁ ( s ) ∈ E · or ▷ ( s ) ∈ E · respectively. If s ∈ E · and u = ξ ( s ) then we reset s as equal to the child of s which is an ancestor of ˜ s and continue the process.

̸

The vertex ˆ u can be found as follows. We construct a root-to-leaf path in D such that, given a vertex s in the path, the next vertex in the path is ◁ ( s ) , ▷ ( s ) or ▽ ( s ) if ν ( ξ ( s ) , u t ) is equal to ◀ , ▶ or ▲ respectively. Given that s ′ is the leaf of D that is in this path we have ˆ u = µ ( s ′ ) .

The vertex u ∗ can then be found as follows. We construct a root-to-leaf path in E such that, given a vertex s in the path, the next vertex in the path is found as follows. If ν ( ξ ( s ) , u t ) = ν ( ξ ( s ) , ˆ u ) then given ν ( ξ ( s ) , u t ) is equal to ◀ , ▶ or ▲ , the next vertex is equal to ◁ ( s ) , ▷ ( s ) or ▽ ( s ) respectively. Otherwise, the next vertex is ▽ ( s ) . Given that s ′ is the leaf of E that is in this path we have u ∗ = µ ( s ′ ) .

The fact that these algorithms find the correct vertices is given in the following theorem:

Theorem D.1. The above algorithms are correct.

D.3 Belief Propagation

Here we utilise the methodology of [7] in order to efficiently compute the function ˜ Λ that appears in the CANPROP implementation. i.e. given a contraction J , a function λ : J ⋆ → R + and some leaf ˆ u ∈ J ⋆ we need to compute ˜ Λ( J , λ, ˆ u ) . For brevity let us define, for all i, i ′ ∈ { 0 , 1 } , and all vertices u ∈ J \ { r ( J ) } , the quantities:

For simplicity of presentation we will utilise a tree J ′ which is defined as identical to J except with a single vertex added as the parent of r ( J ) . For all i, i ′ ∈ { 0 , 1 } we define ˆ κ i ( r ( J ′ )) := 1 and ˆ τ i,i ′ ( r ( J )) = J i = i ′ K . For all u ∈ J we will define ↑ ( u ) := ↑ J ′ ( u )

We will utilise a TST D of J by maintaining potentials on the vertices of D defined as follows. First, as in Section D.1, for any vertex s ∈ D we define the subtree ˆ J ( s ) of J to be equal to ⇓ J ( µ ( s )) if s ∈ D ◦ and equal to the subtree in J of all descendants of µ ( s ) which are not proper descendants of µ ′ ( s ) if s ∈ D · . For all s ∈ D ◦ and i ∈ { 0 , 1 } we define:

̸

̸

and for all s ∈ D · and i, i ′ ∈ { 0 , 1 } we define:

We have the following recurrence relations for these potentials. Suppose we have an internal vertex s ∈ D † and i, i ′ ∈ { 0 , 1 } . If s ∈ D ◦ we have:

If, instead, s ∈ D · then, by letting s ′ := ◁ ( s ) , s ′′ := ▷ ( s ) if ◁ ( s ) ∈ D · and s ′ := ▷ ( s ) , s ′′ := ◁ ( s ) otherwise, we have:

If, on a trial t , we perform the operation INSERT t ( J ) or change the value of λ ( u t ) these recurrence relations can be used (in conjunction with the tree rotations) to update the potentials in logarithmic time.

Now that we have defined our potentials we will show how to use them to compute ˜ Λ( J , λ, ˆ u ) in logarithmic time. To do this we recursively define the following quantities for i ∈ { 0 , 1 } . Let ω i ( r ( D )) := 1 . Given an internal vertex s ∈ D ◦ we define:

Given an internal vertex s ∈ D · define s ′ := ◁ ( s ) , s ′′ := ▷ ( s ) if ◁ ( s ) ∈ D · and s ′ := ▷ ( s ) , s ′′ := ◁ ( s ) otherwise. Then:

For s ∈ D ◦ , ω ′ i ( s ) is not required and hence is arbitrary. We inductively compute the values { ω i ( s ) , ω ′ i ( s ) | i ∈ { 0 , 1 }} for all s in the path from r ( D ) to the unique leaf ˆ s ∈ D ⋆ in which µ (ˆ s ) = ˆ u . We then have ˜ Λ( J , λ, ˆ u ) = ω 1 (ˆ s ) .

Since this is known methodology we do not include a proof in this paper and direct the reader to [7].

E Proofs

E.1 Theorem 3.3

This theorem is proved in appendices B to D and the theorems therein.

E.2 Theorem 3.6

For all x ∈ C define ˆ γ ( x ) := γ ( x, ˆ y, X ) .

Choose a set S ⊆ C in which for all t ∈ [ T ] there exists x ∈ S with ∆( x, x t ) < ˆ γ ( x ) / 3 c . For all trials t let S t be the set of all contexts x ∈ S in which there exists s ∈ [ t ] with ∆( x, x s ) < ˆ γ ( x ) / 3 c .

̸

Now consider a trial t ∈ [ T ] \ { 1 } in which ˆ y ( x t ) = ˆ y ( x n ( t ) ) and choose x ∈ S with ∆( x, x t ) < ˆ γ ( x ) / 3 c .

Assume, for contradiction, that x ∈ S t -1 . Then there exists s ∈ [ t -1] with ∆( x, x s ) < ˆ γ ( x ) / 3 c so that by the triangle inequality we have:

which implies that ∆( x t , x n ( t ) ) < 2ˆ γ ( x ) / 3 . By the triangle inequality we then have that:

̸

Since ∆( x, x t ) < ˆ γ ( x ) we have y ( x ) = y ( x t ) and hence that y ( x ) = y ( x n ( t ) ) . But this contradicts the fact that ∆( x, x n ( t ) ) < ˆ γ ( x ) .

We have hence shown that x / ∈ S t -1 . Since x ∈ S t we then have that |S t | ≥ |S t -1 | . Since |S 1 | ≥ 1 this implies that:

̸

as required.

E.3 Theorem 3.9

Let ϵ be such that:

̸

Since we are only interested in the behaviour as q → ∞ assume, without loss of generality, that √ d/q < ϵ/ 3 . Choose λ > 0 and λ ′ > 0 sufficiently small for this proof to work. For all x ∈ G d q let H ( x ) be the set of points x ′ ∈ [0 , 1] d such that x is the nearest neighbour (or one of them if the nearest neighbour is not unique) of x ′ in G d q . Let L be the set of all x ∈ G d q for which there exists x ′ , x ′′ ∈ H ( x ) ∩ E ( µ, ϵ ) with ˆ y ( x ′ ) = ˆ y ( x ′′ ) . Note that for all x ′ ∈ L we have H ( x ′ ) ⊆ M (ˆ y, µ, 2 ϵ, √ d/q ) so:

By considering the limit of this inequality as q →∞ , and noting that d is being treated as a constant, we then have, by Equation (5), that:

Now define:

Since H ( x ′ ) ⊆ M (ˆ y, µ, 2 ϵ, √ d/q ) for all x ′ ∈ L , we have:

By considering the limit of this inequality as q →∞ , and noting that d is being treated as a constant, we then have, by Equation (5), that:

̸

Let D be the set of all x ∈ G d q such that there exists x ′ ∈ H ( x ) with µ ( x ) = 0 . Now define ˆ y ′ : [0 , 1] d → [ K ] such that for all x ∈ [0 , 1] d we have:

̸

If x ∈ D \ L then ˆ y ′ ( x ) is the unique a ∈ [ K ] such that there exists x ′ ∈ H ( x ) with µ ( x ′ ) = 0 and a = ˆ y ( x ′ ) . Note that if there existed more than one such a then we would have x ∈ L which is a contradiction.

If x / ∈ D \ L then ˆ y ′ ( x ) = ˆ y ′ (ˆ x ) where ˆ x is the nearest neighbour of x in D\L

Let A be a finite set of points in R d such that for all x ∈ R d with ∆( x, 0) ≤ 1 there exists x ′ ∈ A with ∆( x, x ′ ) < λ . Define:

We now show that for all x ∈ R d with 1 /q ≤ ∆( x, 0) ≤ d there exists x ′′ ∈ A ′ with:

To show this choose any such x and let i ∈ N be such that 2 i -1 /q ≤ ∆( x, 0) < 2 i /q . By the assumption on x we have that i ∈ [ ⌈ log( qd ) ⌉ ] . Since ∆( xq 2 -i , 0) < 1 choose x ′ ∈ A such that ∆( xq 2 -i , x ′ ) < λ . Then we have:

so, since 2 i x ′ /q ∈ A ′ , we have proved that Equation (9) is true.

We now let S ′ be a finite set of points in R d such that for all x ∈ [0 , 1] d there exists some x ′ ∈ S ′ with ∆( x, x ′ ) < λ ′ ϵ . Now define:

Let X := { x t | t ∈ [ T ] } . Take any x ∈ X . We now show that there exists x † ∈ S with ∆( x, x † ) < γ ( x † , ˆ y ′ , X ) / 3 c . We have three cases:

̸

First consider the case that ∆( x, x ′ ) > ϵ/ 3 . Let x ′ be the nearest neighbour of x in [0 , 1] d with ˆ y ′ ( x ) = ˆ y ′ ( x ′ ) . Choose x † ∈ S ′ such that ∆( x, x † ) < λ ′ ϵ . Since ∆( x, x † ) < ϵ/ 3 we must have ˆ y ′ ( x † ) = ˆ y ′ ( x ) . Let x ′′ be the nearest neighbour of x † in X with ˆ y ′ ( x ′′ ) = ˆ y ′ ( x † ) . Since ˆ y ′ ( x † ) = ˆ y ′ ( x ) we must have that ˆ y ′ ( x ′′ ) = ˆ y ′ ( x ) so that ∆( x, x ′′ ) > ϵ/ 3 . By the triangle inequality we then have:

̸

Hence γ ( x † , ˆ y ′ , X ) > (1 / 3 -λ ′ ) ϵ so that:

as required.

Now consider the case that x ∈ L . In this case we trivially have the result with x † := x .

̸

Finally consider the case that x / ∈ L and ∆( x, x ′ ) ≤ ϵ/ 3 . Let x ′ be the nearest neighbour of x in [0 , 1] d with ˆ y ′ ( x ) = ˆ y ′ ( x ′ ) . Let ˆ x and ˆ x ′ be the nearest neighbours of x and x ′ in D\L respectively. Note that by definition of ˆ y ′ we have ˆ y ′ (ˆ x ) = ˆ y ′ ( x ) and ˆ y ′ (ˆ x ′ ) = ˆ y ′ ( x ′ ) . By definition of D we must have x ∈ D , so since x / ∈ L we have ˆ x = x . Noting that ∆( x ′ , ˆ x ′ ) ≤ ∆( x ′ , ˆ x ) we must then have that ∆( x ′ , ˆ x ′ ) ≤ ∆( x ′ , x ) which means, by the triangle inequality, that ∆( x, ˆ x ′ ) ≤ 2∆( x, x ′ ) . Since x, ˆ x ′ ∈ D choose z ∈ H ( x ) and z ′ ∈ H (ˆ x ′ ) such that µ ( z ) , µ ( z ′ ) = 0 . Since x, ˆ x ′ ∈ D \ L we have:

̸

̸

By the triangle inequality and above we have:

̸

so since µ ( z ) , µ ( z ′ ) = 0 we must have that the straight line from z to z ′ is entirely contained in E ( µ, ϵ/ 2) . Since ˆ y ( z ) = ˆ y ( z ′ ) we can then choose some z † in the line from z to z ′ that is on the decision boundary of ˆ y (i.e. any open set around z † contains some ˜ z with ˆ y (˜ z ) = ˆ y ( z † ) ). Since there exists an open set around z † that is entirely contained in E ( µ, ϵ )

Note that:

̸

̸

̸

we must now have, by definition of L , that there exists ˆ z ∈ L such that z † ∈ H (ˆ z ) . By the triangle inequality and Equation (10) we have:

Since x ∈ D\L we have ˆ y ′ (˜ z ) = ˆ y ′ ( x ) for all ˜ z ∈ H ( x ) and hence we must have x ′ / ∈ H ( x ) so that ∆( x, x ′ ) ≥ √ d/ (2 q ) . By Equation (11) this means that:

By Equation (9) choose z ′′ ∈ A ′ such that:

and define x † = ˆ z + z ′′ . Since ˆ z ∈ L we have x † ∈ S as required. Note that by equations (12) and (13) we have:

̸

Since 2 λ < 1 / 6 we now have, from equations (12) and (14), that ∆( x, x † ) < ∆( x, x ′ ) so that ˆ y ′ ( x ) = ˆ y ′ ( x † ) . Let x ′′ be the nearest neighbour of x † in X with ˆ y ′ ( x ′′ ) = ˆ y ′ ( x † ) . Since ˆ y ′ ( x † ) = ˆ y ′ ( x ) we must have that ˆ y ′ ( x ′′ ) = ˆ y ′ ( x ) so that ∆( x, x ′′ ) ≥ ∆( x, x ′ ) . By the triangle inequality and Equation (14) we then have:

̸

Hence γ ( x † , ˆ y ′ , X ) > (1 -12 λ )∆( x, x ′ ) so that by Equation (14) we have:

as required.

Let y ′ ∈ [ K ] T be such that y ′ t := ˆ y ′ ( x t ) for all t ∈ [ T ] . We have shown that for all x ∈ X there exists x † ∈ S with ∆( x, x † ) < γ ( x † , ˆ y ′ , X ) / 3 c . By equations (6) and (8) we have that:

so by Theorem 3.3 we have:

We also have:

̸

̸

But ˆ y ( z t ) = ˆ y ′ ( x t ) implies that x t / ∈ D \ L so since x t ∈ D we must have x t ∈ L which happens with probability p and hence, by Equation (7), we have:

Combining equations (15) and (16) gives us:

Since q := ⌈ ( T/K ) 1 / ( d +1) ⌉ we have now shown that:

as required.

E.4 Theorem B.1

For every trial t ∈ [ T ] define:

̸

Choose some arbitrary trial t ∈ [ T ] . From here until we say otherwise all probabilities and expectations (i.e. whenever we use P [ · ] or E [ · ] ) are implicitly conditional on the state of the algorithm at the start of trial t . Note first that we have:

̸

̸

For all j ∈ [log( K )] ∪ { 0 } let γ t,j be the ancestor (in B ) of y ( x t ) at depth j . Note that for all v ∈ X \ { γ t,j | j ∈ [log( K )] ∪ { 0 }} we have y ( x t ) / ∈ ⇓ ( v ) so that x t / ∈ Q ( y, v ) and hence, directly from the CANPROP algorithm, we have w t +1 ( v, Q ( y, v )) = w t ( v, Q ( y, v )) . By Equation (17) and the fact that Q ( y, v ) = ∅ for all ancestors v of y ( x t ) this implies that:

For all j ∈ [log( K )] define:

and:

Now choose some arbitrary j ∈ [log( K )] . If γ t, ( j -1) ∈ P t then γ t, ( j -1) = v t, ( j -1) so ↑ ( γ t,j ) = v t, ( j -1) and hence, since x t ∈ Q ( y, γ t,j ) , we have λ t,j = ln( β t ( γ t,j )) . By definition of β t ( γ t,j ) this means that:

and that:

Multiplying these two equations by P [ γ t,j ∈ P t | γ t, ( j -1) ∈ P t ] and P [ γ t,j / ∈ P t | γ t, ( j -1) ∈ P t ] respectively, and summing them together, then gives us:

If, on the other hand, γ t, ( j -1) / ∈ P t then ↑ ( γ t,j ) / ∈ P t so λ t,j = 0 . This means that:

Since the probability that γ t, ( j -1) ∈ P t is equal to ∏ j ′ ∈ [ j -1] π t ( γ t,j ′ ) we then have, by combining equations (19) and (20), that:

By substituting into Equation (18) (after taking expectations) we then have that:

Note that if a t = γ t, log( K ) then γ t,j = v t,j for all j ∈ [log( K )] . By definition of ψ t, log( K ) and the fact that γ t, log( K ) = y ( x t ) , Equation (21) then gives us:

For all ( v, a ) ∈ B × [ K ] define:

For all a ∈ [ K ] define:

noting that this is non-zero only when a ∈ ⇓ ( v ) . Suppose we have some v ∈ B \ { r ( B ) } and some a ∈ ⇓ ( v ) ∩ [ K ] . Then, since P [ a t = a | v / ∈ P t ] = 0 , we have:

Since p t,a ( v ) = 0 whenever a / ∈ ⇓ ( v ) , this implies that for all ( v, a ) ∈ B † × [ K ] we have:

We now take the inductive hypothesis that for all j ∈ [log( K )] ∪ { 0 } we have:

and prove this via reverse induction (i.e. from j = log( K ) to j = 0 ). Note that given a ′ := a t we have P [ a t = a ′ ] = ∏ j ∈ [log( K )] π t ( v t,j ) and hence:

̸

so the inductive hypothesis holds for j = log( K ) . Now suppose that we have some j ′ ∈ [log( K )] and that the inductive hypothesis holds for j = j ′ . We shall now show that it holds also for j = j ′ -1 . Let v ′ be the child of v t, ( j ′ -1) that is not equal to v t,j ′ . Note that a t / ∈ ⇓ ( v ′ ) and hence exp( -η ˆ ℓ t,a ) = 1 for all a ∈ ⇓ ( v ′ ) (i.e. whenever p t,a ( v ′ ) = 0 ) which implies:

For all a ∈ [ K ] , Equation (23) gives us:

Substituting Equation (24) and the inductive hypothesis into this equation (when summed over all a ∈ [ K ] ) then gives us:

Since π t ( v ′ ) + π t ( v t,j ′ ) = 1 we have, direct from the algorithm, that π t ( v ′ ) + π t ( v t,j ′ ) ψ t,j ′ = ψ t, ( j ′ -1) so the inductive hypothesis holds for j = j ′ -1 . We have hence shown that the inductive hypothesis holds for all j ∈ [log( K )] ∪{ 0 } and in particular for j = 0 . Since p t,a ( v t, 0 ) = P [ a t = a ] we then have:

Since exp( -z ) ≤ 1 -z + z 2 / 2 for all z ∈ R + we have, from Equation (25), that:

so since ln(1 + z ) ≤ z for all z ∈ R we have:

Noting that P [ a t = a ] ˆ ℓ t,a = J a t = a K ℓ t,a for all a ∈ [ K ] , we have:

and:

Substituting these equations into Equation (26) (after taking expectations) gives us:

which, upon substitution into Equation (22) gives us:

Note that this equation implies that the same equation also holds when the expectation is not implicitly conditional on the state of the algorithm at the start of trial t . Hence, we now drop the assumption that the expectation is conditional on the state of the algorithm at the start of trial t . Summing Equation (27) over all trials t ∈ [ T ] and then rearranging gives us:

Now consider a trial t . For all v ∈ B † let:

Now take any j ∈ [log( K ) -1] ∪ { 0 } and let v := v t,j . Note that:

so that by definition of π t ( ◁ ( v )) and π t ( ▷ ( v )) we have:

Without loss of generality assume that ◁ ( v ) ∈ P t . Then the above equation implies that:

so by definition of ψ t,j we have:

Note that this equation trivially holds for all v ∈ B † \ P t and hence holds for all v ∈ B † . Since for all such v and all S with x t / ∈ S we have w t +1 ( ◁ ( v ) , S ) = w t ( ◁ ( v ) , S ) and w t +1 ( ▷ ( v ) , S ) = w t ( ▷ ( v ) , S ) we then have:

so, by induction on t we have, for all t ∈ [ T +1] , that:

Hence, for all v ∈ B \ r ( B ) and S ∈ 2 X , we have w t ( v, S ) ∈ [0 , 1] . We have now shown that ∆ T +1 ≥ 0 so that Equation 28 gives us:

which, by definition of ∆ 1 , gives us the desired result.

E.5 Theorem B.2

The fact that the weighting w 1 is valid is given by the following lemma:

Lemma E.1. For all v ∈ B † we have:

Proof. We will show that for all v ∈ B ′ we have:

which directly implies the result. So take some arbitrary v ∈ B ′ . Define, for all t ∈ [ T ] , the sets:

and for all x ∈ X ′ t and f ∈ F t define the quantity:

̸

which is defined since n ( x ) ∈ X ′ t ∪ { x 1 } . For all t ∈ [ T -1] we have:

Given any f ∈ F t we have:

and hence by Equation (29) we have:

Since X ′ T = X ′ this implies, by induction, that:

Note that we have a bijection G : F T → 2 X defined by:

and that for all ( f, x ) ∈ F T ×X ′ we have:

Hence, Equation (30) shows us that:

This implies that:

which implies the result.

Now that we have shown that the weighting w 1 is valid we can utilise Theorem B.1 to prove our regret bound. For any set S ∈ 2 X define:

Note that for all v ∈ B † and S ∈ 2 X we have:

so since T ln(1 -1 /T ) ∈ O (1) we have:

As in the statement of Theorem B.1 define, for all v ∈ B , the set:

First note that the graph (with vertex set X ) formed by linking x to n ( x ) for every x ∈ X ′ is a tree so that Φ( y ) ≥ |{ y ( x ) | x ∈ X}| 1 . So since for all v ∈ B ′ we have Q ( y, v ) = ∅ if and only if v has a descendent in { y ( x ) | x ∈ X} and each element of { y ( x ) | x ∈ X} has log( K ) ancestors in B ′ we have:

̸

̸

̸

Now suppose we have some x ∈ X ′ . If y ( x ) = y ( n ( x )) then for all v ∈ B ′ we have x, n ( x ) ∈ Q ( y, v ) or x, n ( x ) / ∈ Q ( y, v ) and hence σ ( x, Q ( y, v )) = 0 . On the other hand, if y ( x ) = y ( n ( x )) then for any v ∈ B ′ with σ ( x, Q ( y, v )) = 1 we must have that either x ∈ Q ( y, v ) or n ( x ) ∈ Q ( y, v ) so v is an ancestor of either x or n ( x ) and hence there can be at most 2 log( K ) such v . So in any case we have:

̸

Hence we have:

Equation (31) gives us:

̸

Substituting in equations (32) and (33) then gives us:

̸

so by Theorem B.1 we have:

Since η = ρ √ ln( K ) ln( T ) /KT we obtain the result.

E.6 Theorem C.1

Recall our sequence of trees ⟨Z t | t ∈ [ T ] \ { 1 }⟩ noting that each of these trees is a contraction so that τ i,i ′ ( Z t , · ) is defined for all i, i ′ ∈ { 0 , 1 } . Let ϵ := 1 /T . Define λ ′ : X → R + as follows. Given x ∈ X , if there exists a leaf u ∈ J ⋆ with γ ( u ) = x then λ ′ ( x ) = λ ( u ) . Otherwise λ ′ ( x ) = 1 . Given t ∈ [ T ] define ˆ λ t : Z t → R + such that for all u ∈ Z t we have that ˆ λ t ( u ) := λ ′ ( γ ( u )) if u is a leaf of Z t and ˆ λ t ( u ) := 1 otherwise. For all t ∈ [ T ] and f : { x t ′ | t ′ ∈ [ t ] } → { 0 , 1 } define:

̸

and:

and:

̸

We now have the following lemma:

Lemma E.2. For all t ∈ [ T ] and f : { x t ′ | t ′ ∈ [ t ] } → { 0 , 1 } we have:

Proof. We prove by induction on t . Suppose the result holds for t = s (for some s ≥ 2 ). We now show that it holds for t = s +1 as well. Let f ∗ be the restriction of f onto the set { x t ′ | t ′ ∈ [ s ] } . Let u ∗ and u ′ be the unique leaves in Z ⋆ s +1 of which γ ( u ′ ) = n ( x s +1 ) and γ ( u ∗ ) = x s +1 . By the construction of Z s +1 these vertices are siblings. Let u ′′ be the parent (in Z s +1 ) of both u ∗ and u ′ . First note that:

Since, by the construction of Z s +1 , we have γ ( ↑ Z s +1 ( u ∗ )) = γ ( u ′′ ) = n ( x s +1 ) we also have that d ( ↑ Z s +1 ( u ∗ )) = d ( u ∗ ) -1 so that, since ϕ 1 = ϵ , we have:

̸

Equations (34) and (35) give us:

Now suppose we have some f ′ ∈ N ( f ) . We have γ ( u ′′ ) = γ ( u ′ ) and hence d ( ↑ Z s +1 ( u ′ )) = d ( u ′′ ) = d ( u ′ ) so since f ′ ( u ′ ) = f ( n ( x s +1 )) and ϕ 0 = 0 we have:

Since, by the construction of Z s +1 , we have ↑ Z s +1 ( u ′′ ) = ↑ Z s ( u ′ ) and (as above) we have d ( u ′′ ) = d ( u ′ ) , we also have:

Since f ′ ( u ∗ ) = f ( x s +1 ) and ↑ Z s +1 ( u ∗ ) = u ′′ we have:

Now let:

Define:

and:

Combining equations (37), (38) and (39) gives us:

For all u ∈ Z s +1 \ { u ∗ , u ′ , u ′′ } we have ↑ Z s +1 ( u ) = ↑ Z s ( u ) so that:

and hence, since f ( n ( x s +1 )) = f ′ ( u ′ ) , we have:

Substituting in Equation (40) gives us:

We have ˜ κ f ′ ( u ′′ ) ( Z s +1 , ˆ λ s +1 , u ′′ ) = 1 and for all u ∈ Z s we have ˜ κ f ′ ( u ) ( Z s +1 , ˆ λ s +1 , u ) = ˜ κ f ′ ( u ) ( Z s , ˆ λ s , u ) . Substituting into Equation (41) gives us:

Summing over all f ′ ∈ N ( f ) and noting that f ′ ( u ∗ ) = f ( x s +1 ) and that:

gives us:

By the inductive hypothesis we then have:

which, by Equation (36), is equal to ˆ w ( f ) . We have hence shown that if the inductive hypothesis holds for t = s then it holds for t = s +1 also. An identical argument shows that the inductive hypothesis holds for t = 2 . We have hence shown that the inductive hypothesis holds for all t ∈ [ T ] \ { 1 } .

We now define a bijection G : { 0 , 1 } X → 2 X by:

Note that for all f : X → { 0 , 1 } and all x ∈ X \ { x 1 } we have:

̸

and hence, by Lemma E.2, we have:

so that:

Since:

and all sets in this union are disjoint, the right hand side of Equation (42) is equal to:

Given a vertex u ∈ Z T \ { r ( Z T ) } define:

and for all f : H ( u ) →{ 0 , 1 } define:

and:

so that:

Lemma E.3. Given a vertex u ′ ∈ Z T \ { r ( Z T ) } and an index i ∈ { 0 , 1 } we have:

Proof. We prove by induction on the height of ⇓ Z T ( u ′ ) . If this height is equal to zero then H ( u ′ ) = { u ′ , ↑ Z T ( u ′ ) } and for all f : H ( u ) →{ 0 , 1 } we have:

Since:

we immediately have the result for the case that the height of ⇓ Z T ( u ′ ) is zero. Now suppose that the result holds whenever the height of ⇓ Z T ( u ′ ) is equal to j (for some j ∈ N ). We will now show that it holds whenever the height of ⇓ Z T ( u ′ ) is equal to j +1 which will prove that the result holds always. By the inductive hypothesis we have, for all i ′ ∈ { 0 , 1 } , that:

∑

0

f

∈{

,

1

}

H

(

u

′

)

J

KJ

K

Summing over i ′ ∈ { 0 , 1 } and noting Equation (44) then shows us the result holds for this case and hence, by induction, holds always.

Given u ′ , u ′′ ∈ Z T with u ′′ ∈ ⇓ Z T ( u ′ ) we define ˆ H ( u ′ , u ′′ ) to be the maximal subtree of Z T which has u ′ and u ′′ as leaves. Given, in addition, f : ˆ H ( u ′ , u ′′ ) →{ 0 , 1 } we define:

and:

We now have the following lemma.

Lemma E.4. Given u ′ , u ′′ ∈ Z T with u ′′ ∈ ⇓ Z t ( u ′ ) \ { u ′ } and indices i ′ , i ′′ ∈ { 0 , 1 } we have that:

̸

Proof. We prove by induction on the distance from u ′ to u ′′ in Z T . If this distance is one then we have u ′ = ↑ Z T ( u ′′ ) and ˆ H ( u ′ , u ′′ ) = { u ′ , u ′′ } so we have:

which immediately implies that the inductive hypothesis holds in this case. Now suppose that the inductive hypothesis holds whenever the distance from u ′ to u ′′ is j . We now consider the case

f

(

↑

(

′

u

)) =

i

f

(

u

′

) =

i

′

ˆ

ζ

(

′

u

, f

) =

τ

′

(

Z

, u

)

and so:

and hence:

is equal to

Z

T

i,i

T

that the distance from u ′ to u ′′ is j +1 . Let u ∗ be the child of u ′ that lies in ˆ H ( u ′ , u ′′ ) . Without loss of generality assume that u ′′ is a descendant of ◁ ( u ∗ ) . Now choose any i ∗ ∈ { 0 , 1 } . Given f : ˆ H ( u ′ , u ′′ ) →{ 0 , 1 } let:

and let f ′ and f ′′ be the restriction of f onto the sets ˆ H ( u ∗ , u ′′ ) and H ( ▷ ( u ∗ )) respectively. Note that

By Lemma E.3 and the inductive hypothesis we then have that the quantity:

̸

Summing over i ∗ ∈ { 0 , 1 } gives us the result. We have hence proved the result in general.

is equal to the quantity:

Suppose we have some f : J → { 0 , 1 } . Let:

Given u ∈ J we have that:

̸

is equal to τ f ( ↑ J ( u )) ,f ( u ) ( J , u ) and hence Lemma E.4 implies that:

so since, by the definition of a contraction, the edge sets of the subtrees in { ˆ H ( ↑ J ( u ) , u ) | u ∈ J \ { r ( J ) }} partition the edge set of Z T we have, by definition of ˜ ζ , that:

Since for all f ′ ∈ ˆ F ( f ) and for all u ∈ Z T \ J we have ˜ κ f ′ ( u ) ( Z T , ˆ λ T , u ) = 1 we have now shown that the quantity:

is equal to the quantity:

Summing over all f ∈ F ( J , ˆ u ) and noting Equations (42) and (43) gives us the result.

E.7 Theorem D.1

Lemma E.5. Given u, u ′ ∈ Z t the algorithm for computing ν ( u, u ′ ) is correct.

Proof. If u = u ′ then the proof is trivial. Otherwise we consider the following cases:

̸

Consider first the case that ˆ s = ▽ ( s ∗ ) . Without loss of generality assume ˆ s = ◁ ( s ∗ ) . Then we have u ∈ ⇓ ( ◁ ( ξ ( s ∗ ))) and since ˆ s ′ = ◁ ( s ∗ ) we have u ′ / ∈ ⇓ ( ◁ ( ξ ( s ∗ ))) . Hence u ′ / ∈ ⇓ ( u ) so ν ( u, u ′ ) = ▲ as required.

̸

If u = ξ ( s ∗ ) then ˆ s = ▽ ( s ∗ ) so either ˆ s ′ = ◁ ( s ∗ ) or ˆ s ′ = ▷ ( s ∗ ) . In the former case we have u ′ ∈ ⇓ ( ◁ ( ξ ( s ∗ ))) = ⇓ ( ◁ ( u )) so that ν ( u, u ′ ) = ◀ and similarly in the later case we have ν ( u, u ′ ) = ▶ as required.

̸

If ˆ s = ▽ ( s ∗ ) and u = ξ ( s ∗ ) then we invoke the process. Consider the vertex s at any stage in the process. By induction we have that if s ∈ E · then u ′ ∈ ⇓ ( µ ′ ( s )) . This is because if s ∈ E · then µ ′ ( s ) is an ancestor of µ ′ ( ↑ E ( s )) . This further implies that when s = ˆ s we have u ′ ∈ ⇓ ( µ ′ ( ↑ E ( s ))) . Now suppose that s ∈ E ◦ and without loss of generality assume s = ◁ ( ↑ E ( s )) . Then u ∈ ⇓ ( ◁ ( ξ ( ↑ E ( s )))) and µ ′ ( ↑ E ( s )) ∈ ⇓ ( ▷ ( ξ ( ↑ E ( s )))) so that, since u ′ ∈ ⇓ ( µ ′ ( ↑ E ( s ))) , we have u ′ / ∈ ⇓ ( u ) and hence ν ( u, u ′ ) = ▲ as required. Suppose now that s ∈ E · and that u = ξ ( s ) . If ◁ ( s ) ∈ E · then we have µ ′ ( s ) ∈ ⇓ ( ◁ ( ξ ( s ))) = ⇓ ( ◁ ( u )) so that, by above, u ′ ∈ ⇓ ( ◁ ( u )) and hence ν ( u, u ′ ) = ◀ as required. Similarly, if ▷ ( s ) ∈ E · then ν ( u, u ′ ) = ▶ as required. This completes the proof.

Lemma E.6. The algorithm correctly finds ˆ u .

Proof. By induction on the depth of s we have, for all vertices s in the constructed path, that:

If s ∈ D ◦ then u t lies in the maximal subtree of Z t containing µ ( s ) and having ↑ J ( µ ( s )) as a leaf .

If s ∈ D · then u t lies in the maximal subtree of Z t with ↑ J ( µ ( s )) and µ ′ ( s ) as leaves.

Let s ′ be the unique leaf of D that is on the constructed path. If s ′ ∈ D ◦ then µ ( s ′ ) is a leaf of J and hence also a leaf of Z t . So by above we have that u t lies in the maximal subtree of Z t with ↑ J ( µ ( s ′ )) and µ ( s ′ ) as leaves. If, on the other hand, s ′ ∈ D · then since s ′ is a leaf of D we have that µ ( s ′ ) = µ ′ ( s ′ ) and hence, by above, we have that u t lies in the maximal subtree of Z t with ↑ J ( µ ( s ′ )) and µ ( s ′ ) as leaves. In either case we have ˆ u = µ ( s ′ ) as required.

Lemma E.7. The algorithm correctly finds u ∗ .

Proof. By induction on the depth of s we have, for all vertices s in the constructed path, that:

If s ∈ E ◦ then Γ Z t ( u t , ˆ u ) lies in ⇓ Z t ( µ ( s )) .

If s ∈ E · then Γ Z t ( u t , ˆ u ) lies in the maximal subtree of Z t with µ ( s ) and µ ′ ( s ) as leaves.

Let s ′ be the unique leaf of E that is on the constructed path. If s ′ ∈ E ◦ then µ ( s ′ ) is a leaf of Z t and hence, by above, Γ Z t ( u t , ˆ u ) = µ ( s ′ ) as required. If s ∈ E · then µ ( s ) = µ ′ ( s ) and hence, by above, Γ Z t ( u t , ˆ u ) = µ ( s ′ ) as required.

̸

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers contextual bandits in a nearest-neighbor paradigm. In this paradigm, the contexts exist in a metric space, such that contexts that are close in the metric space are also likely to admit the same \"correct\" action. \nIn other words, the decision boundary of the optimal mapping from context to action is assumed to be small. \nIntuitively, in such a setting, a nearest-neighbor type strategy is reasonable, in which one decides on an action for the current context based on past history on nearby contexts. \n\nFirst, the paper considers contextual bandits in the adversarial setting, with the added property that at any time step t, one of the contexts for a previous time step is \"flagged\". Intuitively, this flag corresponds to it being similar to the current context, providing some advice to the algorithm. The paper presents an algorithm for this model, and analyzes its regret in terms of the similarity between the current context and the flagged past context (as measured in terms of the optimal policy's assignment for these contexts). Next, the paper hones in on the case in which the flagged context is a (c-approximate) nearest neighbor of the current context, and simplifies the regret bound to be a function of the parameter c as well as the distance of the input from the decision boundary of the optimal policy. \n\n\nThe algorithm presented in the paper seems to be a very intricate construction of search trees over the given action space, to support the given regret bounds as well as efficient calculation. \n\n\nComments:\n\nSection 4.2: I found the definition of the ternary tree to be technical and hard to understand. Perhaps give an intuitive explanation of the construction before delving into notation and formal definitions.\n\nSTRENGTHS:\nThe assumption that contexts exhibit closeness-based similarity is natural, and allows for meaningful bounds even when the space of possible contexts is infinite. \nThe algorithm itself is interesting and very nontrivial.\nThe paper is generally well-written.\n\nWEAKNESSES:\nThe paper is somewhat pseudocode and notation heavy. I would personally rather have more intuition about the various components and their role in the algorithm, even at the cost of not having a full description of every procedure in the main body of the paper.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

comments: SUMMARY:
The paper under review investigates the novel application of nearest neighbor search in the context of contextual bandit problems, which I find both new and intriguing. The main contribution lies in the derived result that bounds the regret using \Phi(y), a metric that quantifies the likelihood of disparate optimal choices among closely related contexts. I appreciate the generality and applicability of this metric, as it encapsulates contextual dependencies and has the potential for broad practical use. I am not familiar with the techniques used in the analysis part, but they look good to me.

STRENGTHS:
This paper studies the application of nearest neighbor search in contextual bandits, which is new and practical. And it gets some interesting theoretical results.
The paper is written very clearly and has a good structure.

WEAKNESSES:
Some notations in the second part of section 2 (those related to trees) are not very intuitive and hard to interpret, but perhaps it is not easy to design symbols for them.

QUESTIONS:
Just to make sure that there is no hidden assumption on the relationship between context and loss vector? (compared with linear bandits)
Is it the case that if each context has completely different loss vectors, then \phi(y) will be very large and the bound becomes vacuous?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

comments: SUMMARY:
This paper studies the contextual multi-armed bandit problem in the adversarial setting. The authors propose an algorithm, CanProp, which utilizes an adaptive approximate nearest neighbor data structure to select the arm to pull for a given context.

STRENGTHS:
The authors provide a novel algorithm for the contextual multi-armed bandit problem in the adversarial setting. The algorithm appears to be theoretically sound, and the authors provide regret guarantees, which they specialize to the stochastic case.

WEAKNESSES:
1. Practicality and implementability: The algorithm relies heavily on data structures which do not seem practically feasible. The authors do not provide any empirical results to support their claims that this can be faster than existing methods

2. Notation: the authors use significant notation, some of which may be required, but some of which is unnecessarily difficult to follow. For example, line 75, using $x$ to indicate a random element instead of $X$. This is confusing, as in the condition stated in line 128, the requirement is that the function of $x$, $P[y(X)=a|X=x]$, is Lipschitz in $x$.

QUESTIONS:
How central is computational efficiency to the novelty claims of this work? If exact nearest neighbors are trivially computed at each time step (resulting in $O(t)$ complexity in round $t$) is the regret guarantee trivial? That is to say, is the result in Appendix B already known and the benefit of this paper is in a computational speed up, or was the regret guarantee itself not previously known?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This work studies adversarial contextual bandits. The approach to solving this problem considered in this work is to use the nearest neighbor (NN)search sub-routine algorithm, and the regret bound depends on a term that characterizes the efficiency of the NN oracle. The main advantage of using an NN-based algorithm is that the per-trial computation time can be improved exponentially, compared with previous EXP-4 based algorithms.

STRENGTHS:
The proof is technically sound as far as I know. 

The improvement of the algorithm running time is significant, if the exponential improvement is provided firstly by this work. 

The presentation is clear.

WEAKNESSES:
Some typos. For example: 
- line 212, $a_t$ equal to $z_{t, \log K}$-> $v_{t, \log K}$

No experiments. The authors could provide some simulation results to suggest their CBNN approach indeed works in practice. I am particularly interested in the comparison between CBNN and EXP-4, from both the computation time comparison and regret comparison.

QUESTIONS:
Is it possible to provide some hardness results to show that the per-trial computational time (polylog) is actually optimal?

LIMITATIONS:
This work is a theoretical work. It does need to address the societal impact issue.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper considers non-stochastic contextual bandit problems in which the regret is defined with an arbitrary decision policy that maps contexts to arms.
For this problem,
a framework of algorithms based on the nearest neighbor rule is developed.
The paper provides generic regret bounds for this framework and show regret bounds for semi-stochastic settings in which contexts consist of $d$-dimensional vectors.

STRENGTHS:
- A general framework applicable to a wide range of contextual bandits is proposed.
- The proposed algorithm is superior in terms of computational efficiency due to the use of sophisticated data structures.
- The algorithmic procedures and theoretical results are clearly explained.

WEAKNESSES:
- Comparisons with existing studies (both in terms of approach and results) are limited.
-

QUESTIONS:
- Are there any regret lower bounds that can be compared to the results of this paper?
- This paper uses a general result (Theorem 3.2) to construct a bound for one specific example (Theorem 3.4). Theorem 3.2 appears to be so general that I expect to be able to derive nontrivial results for other specific examples examples as well (e.g., when the context is discrete). 
Can you think of any such examples?

LIMITATIONS:
I have no concerns about the limitations and potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26