Title: Almost Minimax Optimal Best Arm Identification in Piecewise Stationary Linear Bandits
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-armed bandits

piecewise stationary bandits

best arm identification

fixed-confidence
Abstract: We propose a novel piecewise stationary linear bandit (PSLB) model, where the environment randomly samples a context from an unknown probability distribution at each changepoint, and the quality of an arm is measured by its return averaged over all contexts. The contexts and their distribution, as well as the changepoints are unknown to the agent.
We design Piecewise-Stationary $\varepsilon$-Best Arm Identification$^+$ (PS$\varepsilon$BAI$^+$), an algorithm that is guaranteed to identify an $\varepsilon$-optimal arm with probability $\ge 1-\delta$ and with a minimal number of samples.
PS$\varepsilon$BAI$^+$ consists of two subroutines, PS$\varepsilon$BAI and Naïve $\varepsilon$-BAI (N$\varepsilon$BAI), which are executed in parallel. PS$\varepsilon$BAI actively detects changepoints and aligns contexts to facilitate the arm identification process.
When  PS$\varepsilon$BAI and N$\varepsilon$BAI are utilized judiciously in parallel,  PS$\varepsilon$BAI$^+$ is shown to have a finite expected sample complexity. 
By proving a  lower bound, we show the expected sample complexity of PS$\varepsilon$BAI$^+$ is optimal up to a logarithmic factor.
We compare PS$\varepsilon$BAI$^+$ to baseline algorithms using numerical experiments which demonstrate its   efficiency.
Both our analytical and numerical results corroborate that the efficacy   of PS$\varepsilon$BAI$^+$ is due to   the delicate change detection and context alignment procedures embedded in PS$\varepsilon$BAI.

Almost Minimax Optimal Best Arm Identification in Piecewise Stationary Linear Bandits

Abstract

We propose a novel piecewise stationary linear bandit (PSLB) model, where the environment randomly samples a context from an unknown probability distribution at each changepoint, and the quality of an arm is measured by its return averaged over all contexts. The contexts and their distribution, as well as the changepoints are unknown to the agent. We design Piecewise-Stationary ε -Best Arm Identification + (PS ε BAI + ), an algorithm that is guaranteed to identify an ε -optimal arm with probability ≥ 1 -δ and with a minimal number of samples. PS ε BAI + consists of two subroutines, PS ε BAI and NAÏVE ε -BAI (N ε BAI), which are executed in parallel. PS ε BAI actively detects changepoints and aligns contexts to facilitate the arm identification process. When PS ε BAI and N ε BAI are utilized judiciously in parallel, PS ε BAI + is shown to have a finite expected sample complexity. By proving a lower bound, we show the expected sample complexity of PS ε BAI + is optimal up to a logarithmic factor. We compare PS ε BAI + to baseline algorithms using numerical experiments which demonstrate its efficiency. Both our analytical and numerical results corroborate that the efficacy of PS ε BAI + is due to the delicate change detection and context alignment procedures embedded in PS ε BAI.

1 Introduction

In stochastic multi-armed bandits (MABs), an agent interacts with the environment at each time step. The agent pulls an arm and observes the corresponding return provided by the environment. The classical MAB framework assumes a stationary environment where the expected return of each arm remains unchanged over time. However, we usually face ever-changing environments in real life. For instance, in investment option selection and portfolio management, fund managers want to select a subset of good candidate portfolios. However, the market may be bullish, bearish, or in some other state. The transition between these states can be well-modelled as being stochastic. We wish to select portfolios that yield the best long-term option under such a piecewise stationary environment. Further examples such as one based on agriculture in the face of stochastically changing weather patterns are discussed in detail in Appendix A. These motivate us to formulate and investigate a piecewise stationary linear bandit (PSLB) model.

Our PSLB model is equipped with an arm set X , a context set Θ and a deterministic but unknown sequence of changepoints C . At each changepoint, the environment samples a context θ ∈ Θ from an

unknown probability distribution P θ , and the returns of arms may change when the context changes. The return of each arm under each context is determined by its feature x ∈ X and the context θ . In particular, the expected return of an arm is the weighted sum µ x = E P θ [ x ⊤ θ ] . While the sequence of changepoints, as well as the distribution and latent vectors of contexts are not known, the agent samples an arm and observes the corresponding return at each time step so that it can identify the best arm arg max x µ x up to some tolerance ε with probability ≥ 1 -δ and with as few samples as possible. The agent's behavior does not affect the sequence of contexts that is drawn from P θ .

Main Contributions. We are the first to study the fixed-confidence best arm identification (BAI) problem in piecewise stationary bandits (PSB). Given δ > 0 , we say the arm with the highest expected return µ ∗ is the best , and an arm is ε -optimal if its expected return is at least µ ∗ -ε . We seek to design an ( ε, δ ) -PAC algorithm which can identify an ε -optimal arm with probability ≥ 1 -δ in as few time steps as possible, i.e., with minimal sample complexity.

Our first contribution concerns the formulation of a novel PSLB model, where we measure the quality of an arm x according to its expected return µ x = E θ ∼ P θ [ x ⊤ θ ] for the following reasons. Consider that an arm is measured by its average return across time, which is a generalization of the definition in stationary bandits (SB). A notable feature of PSB models is that the context changes as time evolves, and hence the arm's average return across time also changes, in general. Hence, we aim to identify an arm whose average return across contexts is high, and benefits the agent for interacting with the environment in the long run after the arm identification task. We are thus inspired to introduce the distribution of contexts under the PSLB model, define the expected return µ x for each arm, and use this ensemble (non-time varying) statistic as the benchmark for what we seek to learn. The BAI task using this statistic is meaningful but challenging, as the agent needs to reliably estimate the context vectors, changepoints, and context distribution.

Secondly, we propose PIECEWISE-STATIONARY ε -BAI + (PS ε BAI + ), an algorithm designed to tackle the BAI problem in our PSLB model. We prove that it is ( ε, δ ) -PAC and bound its sample complexity. PS ε BAI + samples arms according to a suitably defined G-optimal allocation, and runs two algorithms: NAÏVE ε -BAI (N ε BAI) and PS ε BAI as subroutines in parallel to achieve efficiency and attain a bound on the sample complexity in expectation.

· Being a baseline but naïve algorithm, the complexity of N ε BAI grows linearly with the maximum length between two changepoints L max , motivating us to design a more efficient algorithm, PS ε BAI, to reduce the impact of L max .

· PS ε BAI is equipped with two delicately designed subroutines LINEAR-CHANGE DETECTION (LCD) and LINEAR-CONTEXT ALIGNMENT (LCA) to actively detect changepoints and align contexts with those observed in the previous time steps respectively. Concretely, in terms of the design , PS ε BAI determines whether samples from two intervals are under the identical context via a sliding window mechanism, and detects changepoints and aligns contexts accordingly; this facilitates the estimation of context vectors and their distribution. Combining these elements into the design of PS ε BAI and analyzing them requires some care. On the theoretical side , we prove PS ε BAI identifies an ε -optimal arm faster than N ε BAI with high probability. The success of PS ε BAI relies on the LCD and LCA subroutines, while a minor drawback is that they require a non-vanishing failure probability budget which does not allow us to bound the sample complexity of PS ε BAI in expectation. To achieve a complete theoretical understanding, we delicately design the PS ε BAI + algorithm whose efficiency is inherited via the LCD and LCA procedures in PS ε BAI as well as the effective utilization of running PS ε BAI and N ε BAI in parallel.

Thirdly, we derive a lower bound on the complexity of any ( ε, δ ) -PAC algorithm in PSLB models. To derive this bound, we first lower bound the complexity of an algorithm when the contextual information (and changepoints) are available, and then quantify the number of arm samples (and realized contexts) required to reliably infer an ε -best arm.We compare the upper bound of PS ε BAI + and this generic lower bound in several instances. The matching (up to logarithmic terms) of bounds illustrate that our PS ε BAI + algorithm is almost asymptotically optimal.

Lastly, we demonstrate the efficiency of PS ε BAI + with numerical experiments. The first half of our experiment shows that PS ε BAI + is ( ε, δ ) -PAC and with significantly lower sample complexity compared to N ε BAI, corroborating our theoretical findings. In the second half, we compare PS ε BAI + to N ε BAI, and two other benchmarks DISTRIBUTION ε -BAI (D ε BAI) and D ε BAI β .While contexts and changepoints are not available to PS ε BAI + and N ε BAI, they are observed by D ε BAI and D ε BAI β . Nevertheless, PS ε BAI + is still competitive compared to D ε BAI and D ε BAI β in our

empirical experiments. Hence, both experiments justify the necessity of the change detection and context alignment procedures for boosting the learning of contexts and their distributions, as well as the identification of the best arm. We also show empirically that misspecifications to the knowledge of L min and L max do not affect the performance of our algorithm significantly.

Related work. Best arm identification (BAI) and regret minimization (RM) are two fundamental problems in multi-armed bandits. In stationary linear bandits, [1, 2] focus on BAI and [3, 4] aim to solve the RM problem. An efficient algorithm can choose the G-optimal allocation or XY -allocation rule to quickly identify a good arm [1, 5]. Besides, [6, 7] focus on ε -optimal arm identification, which is a generalization of the standard BAI problem.

The BAI and RM problems are also studied in thr non-stationary bandits (NSB), where in contrast to the SB model, the context varies with time [8-10]. NSB can be largely divided into two classes: the drifting bandit (DB) model, where the context changes at each step [8], and PSB, where the context changes less frequently [10]. [11] provides an extensive discussion on the definition of NSB models.

On one hand, the RM problem have been investigated extensively in DB models [12, 13]. On the other hand, concerning the BAI task in DB models, [14] investigated BAI with a fixed-horizon , where the best arm has the highest average return over this horizon; [15] assumes the best arm remains unchanged after certain time step and explores the fixed-confidence setting. Besides, when the contextual information in NSB models is available, NSB models are known as contextual bandit (CB) models [16-18]. [16] showed that the contextual information accelerates the best arm identification process. More discussions on DB and CB models are presented in Appendix C.

Moreover, the context can drift dramatically in a DB model while it remains unvarying in a SB model. Straddling between the DB and SB models, PSB models assume there is an interim stationary interval between each two consecutive changepoints, where the context remains unchanged. The context changes can be characterized in different ways and affect the performance of proposed algorithms in PSB models. For instance, a changepoint signals the return of at least one arm shift as in [10], and indicates the best arm changes as in [19].

In PSB models, a large body of works focus on RM. While [10, 20-22] equip their algorithms with changepoint detection techniques to handle the context changes, [23] actively checks the quality of each arm. However, there is no existing work on the fixed-confidence BAI problem in a PSB model. To fill this gap in the literature, we design PS ε BAI + for ε -optimal arm identification in our proposed PSLB model. We show PS ε BAI + is almost asymptotically optimal by comparing its complexity to a generic lower bound of all algorithms, and validate its efficiency through numerical experiments.

2 Problem Setup

For m ∈ N , let [ m ] := { 1 , 2 , . . . , m } . For a finite set S , let ∆ S denote the | S | -dim probability simplex on S . Let A ( q ) := ∑ x ∈X q x xx ⊤ be the matrix induced by q ∈ ∆ X with X ⊂ R d . An instance of piecewise stationary linear bandit is a tuple Λ = ( X , Θ , P θ , C ) . Specifically, x ∈ R d is an arm (vector) and the arm set X ⊂ R d is composed of |X| = K arms that spans R d . The latent vector matrix Θ = ( θ ∗ 1 , . . . , θ ∗ N ) ∈ R d × N contains N latent column vectors where the j th column θ ∗ j is associated with context j ∈ [ N ] . For the sake of normalization, we assume | x ⊤ θ ∗ j | ≤ 1 for all x ∈ X , j ∈ [ N ] . Let P θ denote the distribution (probability mass function) of the latent vectors and p j = P θ [ θ ∗ j ] . We represent the probabilities of latent vectors as p = ( p 1 , . . . , p N ) ∈ ∆ N . The fixed but unknown sequence of changepoints C := ( c 1 , c 2 , . . . ) is an sequence of increasing positive integers 1 = c 1 < c 2 < . . . , characterizing all the changepoints (time steps).

The return of arm x under latent context j is a random variable Y = x ⊤ θ ∗ j + η , where η is a zero-mean random variable (noise) supported on [ -1 , 1] , and the expected return of arm x is µ x := E θ ∼ P θ [ x ⊤ θ ] = ∑ N j =1 P θ [ θ ∗ j ] x ⊤ θ ∗ j . The best arm , which we assume is unique, is denoted as x ∗ := arg max x ∈X µ x with mean µ ∗ := µ x ∗ . Given a slackness parameter ε > 0 , we define the set of ε -best arms X ε := { x ∈ X : µ x ≥ µ ∗ -ε } . For each pair of arms ( x, ˜ x ) ∈ X 2 , define the contextual mean gap between x and ˜ x under latent context j as ∆ j ( x, ˜ x ) := ( x -˜ x ) ⊤ θ ∗ j and the mean gap between x and ˜ x as ∆( x, ˜ x ) := µ x -µ ˜ x .

Given l ∈ N , the interval ( c l , . . . , c l +1 -1) is known as the l th stationary segment and its length is L l := c l +1 -c l . We assume L min ≤ L l ≤ L max . Let l t := max { l : c l ≤ t } denote the number of

stationary segments up to time step t . At time step t ∈ [ T ] (see Dynamics 1),

(i) If t ∈ C , the environment samples a latent vector θ ∗ j t according to P θ , that is, it generates a (latent) context sample with P θ ; otherwise the latent vector remains unchanged, i.e., θ ∗ j t = θ ∗ j t -1 . The contexts { θ ∗ j c l } l ∈ N are sampled i.i.d. from P θ at each changepoint { c l } l ∈ N .

(ii) The agent pulls an arm x t and observes the stochastic return Y t,x t = x ⊤ t θ ∗ j t + η t , where η t is drawn independently from a distribution supported on [ -1 , 1] .

The agent uses an online algorithm π := { ( π t , τ t , r t ) } t ∈ N to actively interact with the instance Λ and only has access to the arm set X , number of latent vectors N , the bounds on the length of each segment L min and L max , the slackness parameter ε , and the confidence parameter δ .

sampling rule π t : H π t := ( ( x π s , Y s,x π s ) ) s ∈ [ t -1] →X samples an arm x π t based on the observation history H π t and observe the corresponding random return Y t,x π t ;

stopping rule τ t : H π t +1 →{ STOP , CONTINUE } decides whether to stop or continue to execute given the observation history H π t +1 . The stopping time under algorithm π is denoted as τ π ;

The stopping time τ π is the sample complexity of the algorithm π under instance Λ . The expected sample complexity is E [ τ π ] , where the expectation is taken w.r.t. the random returns, the realization of the contexts governed by the latent vector distribution P θ , and the randomness of the algorithm π .

recommendation rule r τ : H π τ +1 →X recommends an arm ˆ x π based on H π τ +1 upon termination.

An algorithm π is ( ε, δ ) -PAC (Probably Approximately-Correct) if

Our overarching goal in this paper is to devise an ( ε, δ ) -PAC algorithm that minimizes τ π with high probability (w.h.p.) and in expectation.

3 Algorithms

3.1 A Naïve Baseline: N ε BAI

We first devise the NAÏVE ε -BEST ARM IDENTIFICATION (or N ε BAI) algorithm (presented in Algorithm 2). In the design of N ε BAI, only the choice of confidence radius ˜ ρ t takes the potential context changes into consideration. Even though N ε BAI does not attempt to detect potential changes in the context, it can identify an ε -optimal arm w.h.p. and is with a finite expected sample complexity.

̸

Proposition 3.1. Let ∆ min = min x = x ∗ ∆( x ∗ , x ) ,

The N ε BAI algorithm is ( ε, δ ) -PAC and its expected sample complexity is ˜ O ( T N V + T N D ) .

The upper bound in Proposition 3.1 (also see Appendix F) consists of two main terms. (i) As N ε BAI samples arms according to the G-optimal allocation (see Appendix D), the amount of samples needed to estimate the average of latent vectors ∑ t s =1 θ ∗ j s /t contributes to T N V . (ii) T N D quantifies how fast ∑ t s =1 θ ∗ j s /t converges to the expectation of context vectors ∑ t j =1 p j θ ∗ j .

The sample complexity of N ε BAI grows linearly with L max , but we surmise that the sample complexity of a close-to-optimal algorithm should have a reduced dependence on L max .

3.2 Piecewise-Stationary ε -Best Arm Identification

The algorithm PIECEWISE-STATIONARY ε -BEST ARM IDENTIFICATION (or PS ε BAI) is presented in Algorithm 1. By using a sliding window mechanism, PS ε BAI actively detects the changepoints and aligns the current latent context with contexts observed in the previous time steps via LINEAR-CHANGE DETECTION (or LCD) and LINEAR-CONTEXT ALIGNMENT (or LCA), which are presented in Algorithms 3 and 4 (see App. D.2.2), respectively. PS ε BAI consists of three phases: (i) Exploration phase (Exp): Estimate latent vectors and their distribution P θ (Lines 8 to 11 and 25);

(ii) Change Detection phase (CD): Detect changepoints (Lines 12 to 16);

(iii) Context Alignment phase (CA): Evaluate the current context and align it with the contexts observed in previous time steps (Lines 17 to 21).

At time t , we estimate Θ and p with ˆ Θ t = ( ˆ θ t, 1 , . . . , ˆ θ t,N ) and ˆ p t = (ˆ p t, 1 , . . . , ˆ p t,N ) ⊤ , respectively. 1 We denote the empirical mean gap between x and ˜ x under context j as ˆ ∆ t,j ( x, ˜ x ) := ( x -˜ x ) ⊤ ˆ θ t,j .

PS ε BAI first computes the G-optimal allocation [1] λ ∗ on the arm set X and its maximum possible stopping time τ ∗ (Line 2). It initializes CD sample and CA id . CD sample collects samples to detect changepoints and CA id maintains a dictionary of { latent context index : identification samples } pairs (Line 3); 2 CA id [ j ] is the sequence of CD samples used to identify latent context j . It also initializes T t,j , the collection of time indices in [ t ] in the Exp phases under estimated context j . Define

Algorithm 1 PIECEWISE-STATIONARY ε -BEST ARM IDENTIFICATION (PS ε BAI)

1: Input: arm set X , size of the set of latent vectors N , bounds on the segment lengths L min and L max , slackness parameter ε , confidence parameter δ , sampling parameter γ and window size w , threshold b .

2: Initialize : Compute the G-optimal allocation λ ∗ and τ ∗ = 38400 ln(80) NL max ε 2 ln N 2 KL max δε 2 .

3: Set CD sample = [ ] , CA id = { } . Set t CD = + ∞ .

4: Set T t,j = ∅ and initialize T t , T t,j , T t with (3.1) for all t ≤ τ ∗ , j ∈ [ N ] . w

6: CA id = { 1 : [( x s , Y s,x s )] w 2 s =1 } , ˆ j t = 1 .

5: Sample w 2 arms { x s } 2 s =1 ∼ λ ∗ and observe the associated returns { Y s,x s } w 2 s =1 , t = w 2 , t CA = w 2 .

7: while t ≤ τ ∗ do

8: t = t +1

9: Sample an arm x t ∼ λ ∗ and observe return Y t,x t .

̸

10: if mod ( t -t CA , γ ) = 0 then

̸

11: Update ˆ j t = ˆ j t -1 , T t, ˆ j t = T t -1 , ˆ j t ∪ { t } , T t,j = T t -1 ,j for j = ˆ j t .

12: else

13: CD sample = CD sample +[( x t , Y t,x t )] .

14:

15:

16:

17:

18:

19:

20:

21:

22:

23:

24:

25:

26:

27:

28:

29:

30:

31:

Update ˆ j t = ˆ j t -1 , T t,j = T t -1 ,j for all j ∈ [ N ] .

if | CD sample | ≥ w then if LCD ( X , w, b, CD sample [ -w : ]) then

end if end if end if

Update the estimates using (3.1), (3.2) and (3.3).

if

Condition (3.4) is met and then

else if t CD = | CD sample | -w 2 then Recommend arm ˆ x ε .

t

CD

= +

Record ˆ x ε = arg max x ∈X x ⊤ ˆ Θ t ˆ p t . t CD = | CD sample | .

break

32:

end if

33: end while

It then collects w 2 samples and stores them in CA id , which is then used to identify the first latent context (Lines 5 to 6).

In the Exp phase, PS ε BAI firstly samples an arm x t with λ ∗ and observes the return Y t,x t = x ⊤ t θ ∗ j t + η t (Line 9). It then updates the estimated context index and time collectors (Line 11). It also updates the estimates of value and probability of each context j (Line 25) with

∞

where ˆ θ t,j = 0 if T t,j = 0 . We define α t , ξ t , β t,j , and ˆ ∆ clip 2 t,j ( x, ˜ x ) in Appendix D.2.1. For each pairs of arms ( x, ˜ x ), the confidence radius of ∆( x, ˜ x ) at time step t is

PS ε BAI actively enters the CD phase every γ time steps (Line 12). It firstly adds a CD sample to CD sample (Line 13). Next , if there are sufficient CD samples (Line 15), the LCD subroutine (presented in Algorithm 3) is called and utilizes the most recent w CD samples to check whether a changepoint just occurred (Line 16). PS ε BAI steps into the CA phase if a changepoint is detected, and skips the CA phase otherwise, which is illustrated by Figures 1(b) and 1(a) respectively.

Context

active CD samples

Step 1: No change alarm is raised

Step 2: Check stopping rules

Step 3: Start a new Exp phase

Time step

Change Detection Phases

Exploration Phases

Active CD samples

Inactive CD samples

samples

in an Exp phase

(a)

Context

Step 3: Revert the statistics

Step 1: LCD raises a change alarm

Step 2: LCA aligns the context

Context Alignment Phases

Step 4: Start a new Exp phase

Time step

Change Detection Phases

Exploration Phases

Active CD samples

Inactive CD samples

(b)

In the CA phase, PS ε BAI starts by resetting CD sample (Line 17), updating the count of time steps and recording the ending time of this CA phase (Line 18). Thereafter , the CA subroutine (presented in Algorithm 4) is invoked, which estimates the current latent context index ˆ j t and updates CA id (Line 19). If ˆ j t = N +1 , i.e., PS ε BAI identifies N +1 latent contexts, which is incorrect under instance Λ , it terminates and fails to identify an ε -optimal arm (Line 20). Lastly , all empirical statistics are reverted to those from ( w ( γ +1) / 2) time steps ago, i.e., the most recent ( wγ/ 2) samples in the Exp phases are abandoned (Line 21).

The stopping rule is described in Lines 26 to 32. (I) If the following condition is satisfied (Line 26):

̸

where the empirical mean gap ˆ ∆ t ( x ∗ t , x ) := ( x ∗ t -x ) ⊤ ˆ Θ t ˆ p t and x ∗ t := arg max x ∈X x ⊤ ˆ Θ t ˆ p t , PS ε BAI records the arm with the highest empirical mean as ˆ x ε and the number of CD samples t CD (Lines 27 and 28) but does not terminate immediately. Besides, a mild forced arm pull procedure is in the second line of (3.4), which is inspired by Lemma E.1 and to ensure the performance of PS ε BAI. (II) PS ε BAI will execute for another ( wγ/ 2) time steps in which w/ 2 CD samples are collected; if no changepoint is detected with these w/ 2 CD samples, the recorded arm ˆ x ε is recommended and PS ε BAI terminates (Lines 29 to 30). Part (II) of the stopping rule assures PS ε BAI does not terminate when a changepoint has occurred but has not been detected, as PS ε BAI may fail to identify an ε -optimal arm otherwise.

We remark that even though PS ε BAI uses the knowledge of L max , our experiments show that the performance of PS ε BAI is robust to small misspecifications in L max (see Appendix O.2). Furthermore, the computational complexity of PS ε BAI is computed in detail in Appendix D.4. The derived computational complexity indicates the proposed algorithm depends in a natural manner on the problem parameters such as d, K, N , and γ . Lastly, thanks to the LCD and LCA subroutines, a slightly modified variant of PS ε BAI can also solve the ' ε -Best Arm Tuple identification problem', which aims to identify an ε -best arm under each context ; see Appendix Q for details.

3.2.1 Theoretical guarantee of PS ε BAI

To facilitate the analysis of PS ε BAI, we propose the following assumptions. Note that our PS ε BAI algorithm may still succeed to identify an ε -optimal arm w.h.p. when the assumptions do not hold.

̸

Assumption 1 (Distinguishability Condition) . The agent can choose w , γ and b such that (1) 2 b ≤ ∆ c where ∆ c := min θ ∗ j = θ ∗ ˜ j max x ∈X | x ⊤ ( θ ∗ j -θ ∗ ˜ j ) | is the minimum gap between two contexts; and (2) 3 wγ ≤ L min . A possible choice is

This assumption guarantees (i) PS ε BAI will not abandon all samples during the reversion procedure (Line 21 of Algorithm 1); (ii) each two latent vectors can be distinguished if the window size w is sufficiently large (e.g., L min / 6 ). We clarify that this assumption is only for the rigor of theoretical guarantees and it holds provided that each stationary segment is sufficiently long; this is a feature of PSB models and similar assumptions are also present in existing works for their analyses [21, 10, 22]. We demonstrate the robustness of PS ε BAI to these parameters using experiments in Section 6.

Theorem 3.2. Define the context distribution estimation (DE) hardness parameter

where ¯ H ( x ε , x ) := ( ∑ N j =1 √ min { 16 p j , 1 / 4 }| ∆ j ( x ε , x ) + ε | ) 2 . Under Assumption 1, with probability at least 1 -δ , PS ε BAI identifies an ε -optimal arm and its sample complexity is

̸

The upper bound comprises three terms which serve distinct purposes:

(i) Latent vector estimation (VE): ˜ O ( T V ( x )) quantifies the bulk of samples needed to obtain a good estimate of latent context vectors such that the returns of x ε and x can be distinguished, where x ε is an ε -best arm and x / ∈ X ε is a suboptimal arm. T V ( x ) recovers the sample complexity in the stationary linear bandits in [1], indicating that PS ε BAI estimates latent vectors efficiently.

(ii) Context distribution estimation (DE): ˜ O ( T D ( x ε , x )) characterizes the bulk of samples needed to learn the distribution of latent context vectors.

(iii) Residual estimation (RE): ˜ O ( T R ( x )) counts the remaining samples needed for VE and DE, in addition to ˜ O ( T V + T D ) .

Besides, the max operator is applied to exclude all suboptimal arms. We also see that T V ( x ) and T D ( x ε , x ) are similar to T N V and T N D in Proposition 3.1 respectively.

Firstly, the bound in (3.6) implies that, in an instance with smaller relaxed mean gap ∆( x ∗ , x ) + ε , PS ε BAI terminates after a larger number of time steps; in other words, it is more difficult to identify an ε -optimal arm. In difficult instances with small ∆( x ∗ , x ) + ε , the different orders of this term in T V ( x ) , T D ( x ε , x ) and T R ( x ) indicate that, T R ( x ) is small compared to T V ( x ) and T D ( x ε , x ) .

Secondly, DE solely utilizes context samples generated with P θ and they are generated only at changepoints in C , while all the observations in Exp phases facilitate VE. From this perspective, there are less samples that can be used for DE than for VE as PS ε BAI processes, and hence T D ( x ε , x ) is supposed to be with larger order than T V ( x ) .

Moreover, for the purpose of DE, PS ε BAI needs to observe context samples at ˜ O ( ¯ H ( x ε ,x ) (∆( x ∗ ,x )+ ε ) 2 ln 1 δ ) = ˜ O ( H DE ( x ε ,x ) L max ln 1 δ ) changepoints where L max is the maximum length of a stationary segment, leading us to T D ( x ε , x ) . Close examination of the definition of ¯ H ( x ε , x ) reveals that both the vectors and their probabilities influence the number of samples needed for DE. The comparison between T D ( x ε , x ) and T N D in Proposition 3.1 clearly indicates that PS ε BAI mitigates the influence of L max by detecting changepoints and aligning the detected context with observed ones, while N ε BAI does not do so.

3.3 PS ε BAI + = PS ε BAI ∪ N ε BAI

We have provided a high-probability result for PS ε BAI in Theorem 3.2. The design of PS ε BAI (Line 7 of Algorithm 1) indicates that PS ε BAI will not recommend any arm if it does not terminate at time τ ∗ . This result is nontrivial, as the high-probability result in Theorem 3.2 depends on the success of change detection (Algorithm 3) and context alignment (Algorithm 4), which requires a non-vanishing failure probability (e.g., δ/ 2 ). Thus, we cannot derive an upper bound on the expected sample complexity of PS ε BAI. We devise a solution by designing the Piecewise-Stationary ε -Best Arm Identification + (PS ε BAI + ) algorithm with a simple but effective trick.

The PS ε BAI + algorithm samples one arm with the G-optimal allocation λ ∗ at each time step, with which Algorithms 1 and 2 are executed in parallel (detailed in Algorithm 5). This is feasible since PS ε BAI and N ε BAI algorithms have the same sampling rule.

Theorem 3.3. The PS ε BAI + algorithm is ( ε, δ ) -PAC and its expected sample complexity is

̸

PS ε BAI + inherits the superiority of PS ε BAI to adapt to the piecewise stationary environment, and employs the stopping rule of N ε BAI to maintain a finite expected sample complexity. As a result, the expected complexity of PS ε BAI + in Theorem 3.3 is of the same order as the high-probability one of PS ε BAI in Theorem 3.2 and is not larger than the complexity of N ε BAI in Proposition 3.1. We show how our results particularize to the stationary linear bandits BAI problem, as well as additional discussions on the upper bound, in Appendix P.

4 Lower Bound on the Sample Complexity

Given Λ = ( X , Θ , P θ , C ) , define the alternative instance Λ ′ = ( X , Θ ′ , P θ ′ , C ) w.r.t. Λ , where Θ ′ = ( θ ′ 1 , . . . , θ ′ n ) ∈ R d × N , P θ ′ [ θ ′ j ] = P θ [ θ ∗ j ] , and there exists x ∈ X \ X ε , such that x ⊤ ε E θ ′ ∼ P θ ′ [ θ ′ ] < x ⊤ E θ ′ ∼ P θ ′ [ θ ′ ] -ϵ for all x ε ∈ X ε . Let Alt Θ (Λ) be set of all alternative instances (w.r.t. Λ ).

Theorem 4.1. For all ( ε, δ ) -PAC algorithm π , there exists an instance Λ = ( X , Θ , P θ , C ) such that

̸

where

Recall that c N C is the N C -th changepoint in the changepoint sequence C , which is lower bounded by N C L min and is N C L max in the worst case. To derive the lower bound in Theorem 4.1, we investigate two environments different from the one defined in Section 2 (and as in Dynamics 1):

Dynamics 2: the agent observes the index of current context j t (i.e., contextual linear bandits);

Dynamics 3: the agent observes the changepoints in C and context vector θ ∗ j t 's, and hence she solely needs to estimate the distribution of contexts.

We bound the sample complexity of an ( ε, δ ) -BAI algorithm in Dynamics 2 and 3 respectively, which when combined, yield the lower bound in Theorem 4.1; this is detailed in Appendix M.

Note that T ε (Λ) -1 in the lower bound generalizes [16] to the setting of linear bandits. In addition, Theorem 4.1 can be reduced to a bound in stationary linear bandits with one latent vector [24] (see the discussion leading to (M.15)).

5 On the Asymptotic Optimality of PS ε BAI +

To illustrate the efficiency of our PS ε BAI + algorithm, we compare the upper bound on its expected sample complexity in Theorem 3.3 and the generic lower bound in Theorem 4.1 under specific instances below and in Appendix N. We also gain further insight into our PS ε BAI + algorithm.

1

2

3

4

5

6

7

8

9

10

11

12

10

6

10

8

10

10

(a) Sample complexity of PS

ε

BAI

+

vs. N

ε

BAI.

0

10

20

30

40

50

60

70

80

0

100

200

300

400

500

600

700

800

900

Number of Context Samples

1

2

3

4

5

6

10

20

30

0

20

40

60

80

10 3

10 4

10 5

(b) Number of context samples needed for BAI.

Example 1. Instance Λ = ( X , Θ , P θ , C ) is with (i) 2 d -1 arms: x (1) = e 1 , x ( i ) = e i , x ( d + i -1) = e 1 cos ϕ + e i sin ϕ for all i ∈ { 2 , . . . , d } where ϕ ∈ [0 , π/ 4) , (ii) 2 d -2 contexts: θ ∗ j ± = e 1 cos ϕ ± e j +1 sin ϕ for all j ∈ [ d -1] , (iii) Context distribution: p j = 1 /N for all j ∈ [ N ] .

̸

Under the instance defined in Example 1, x ( i ) for all i = 1 is inferior to x (1) under all contexts and x ( i + d ) for all i ∈ [ d -1] is marginally better than x (1) by 1 -cos ϕ only under context θ ∗ i + and ∆( x (1) , x ( i + d ) ) = cos ϕ -cos 2 ϕ . We expect PS ε BAI + to discover this feature of the instances and quickly identify an ε -optimal arm with a course estimation of the context distribution.

Corollary 5.1. For the instance defined in Example 1, we have H DE ( x ε , x ) = ˜ O ( NL max ) for all ( x ε , x ) ∈ X ε × ( X \ X ε ) . In addition, if ε < (cos ϕ )(1 -cos ϕ ) , we have

where E [ τ ] ∗ is the minimal expected sample complexity over all ( ε, δ ) -PAC algorithms and f : R → R satisfies f ( ϕ ) → 0 as ϕ → 0 + . The upper bound in (5.1) is achieved by PS ε BAI + .

The order of H DE in Corollary 5.1 indicates that T D ( x (1) , x ( d +1) ) = ˜ O ( NL max ln(1 /δ )) is not dominating the sample complexity of PS ε BAI + , suggesting that a coarse estimation of the context distribution is sufficient when ϕ is small. In other words, PS ε BAI + exploits the feature of instances and utilize samples mostly for estimate context vectors, which is again expected.

Corollary 5.1 implies that under such instances, the upper and lower bounds on the sample complexity of PS ε BAI + match up to logarithmic factors, that is, the performance PS ε BAI + is near optimal. Besides, the bound of N ε BAI in Proposition 3.1 is with an extra additive term L max compared to the lower bound in Corollary 5.1, illustrating that N ε BAI is suboptimal and again emphasizing the significance of detecting changes and aligning contexts for PS ε BAI + to reduce the impact of L max .

6 Numerical Experiments

We now evaluate the empirical performance of PS ε BAI + . We utilize the instance defined in Example 1 with d = 2 , ϕ = π/ 8 , We generate a changepoint sequence C such that c l +1 = c l + L l with L min = 3 × 10 4 , L max = 5 × 10 4 , P [ L l = L min ] = 0 . 8 , P [ L l = L max ] = 0 . 2 , and fix it throughout the whole set of experiments. We set the confidence parameter δ = 0 . 05 and vary the slackness parameter ε from 0 . 04 to 0 . 6 (i.e., ε = 0 . 03 × 1 . 35 k for k ∈ [12] ). We set γ = 6 , the window size w = L min / (3 γ ) and compute b via (3.5) in Assumption 1. 3 For each choice of algorithm and instance, we run 20 independent trials. All the code to reproduce our experiments can be found at https://github.com/Y-Hou/BAI-in-PSLB.git .

We first compare PS ε BAI + and N ε BAI. Both algorithms succeed to identify an ε -optimal arm, while empirically, the complexity of PS ε BAI + is ≤ 1% of that of N ε BAI. The empirical averages and standard deviations of the sample complexities of both algorithms are presented in Figure 2(a).

Figure 2(a) illustrates that empirically, the termination and arm recommendation of PS ε BAI + are determined by the execution of PS ε BAI as a subroutine, suggesting that in Theorem 3.3, the first term resulting from PS ε BAI actually determines the complexity of PS ε BAI + .

Next, we test the efficacy of PS ε BAI + to learn and exploit the latent vectors and the distribution of contexts. Specifically, we run PS ε BAI + and N ε BAI under Dynamics 1 where neither the index nor the vector of current context is visible to the agent. We also run benchmark algorithms: D ε BAI and its variant D ε BAI β under Dynamics 3 where context vectors and changepoints are all observed; these two algorithms are detailed and analyzed in Appendix O.4.

As the changepoint sequence C is fixed in a given instance and t/L max ≤ l t ≤ t/L min for all t ∈ N , we regard the number of context samples l τ as a proxy of the sample complexity τ . We present the number of context samples need by PS ε BAI + , N ε BAI, D ε BAI and D ε BAI β for arm identification w.r.t. 1 / (∆ min + ε ) 2 in Figure 2(b).

Figure 2(b) contains three messages. First, the complexity of PS ε BAI + scales as 1 / (∆ min + ε ) 2 , corroborating Theorem 3.3. Second, although PS ε BAI + has access to neither context vectors nor changepoints, it needs roughly the same number of context samples as D ε BAI and D ε BAI β , suggesting that it is competitive compared to these algorithms that have oracle information about the environment. Third, D ε BAI β uses the confidence radius in (3.3) and terminates with fewer context samples compared to D ε BAI, implying that the confidence radius is well-designed.

Furthermore, when ε decreases from 0 . 03 × 1 . 35 12 to 0 . 03 × 1 . 35 9 , the complexity of PS ε BAI almost remains unchanged while that of N ε BAI increases rapidly as presented in instances 9 to 12 in Figure 2(a). Meanwhile, the number of context samples need by two algorithms are shown to be with the same pattern in Figure 2(b). This contrast indicates that the cost of distribution estimation ( T D in (3.6)) for PS ε BAI + has been significantly minimized compared to N ε BAI.

To summarize, we emphasize that the empirical superiority of PS ε BAI + over N ε BAI implies that the efficacy of PS ε BAI + is inherited from PS ε BAI. Our experiments show that actively exploiting the context information, via changepoint detection and context alignment (as in PS ε BAI + and PS ε BAI) facilitates identifying the ε -optimal arm efficiently.

Similar to many existing algorithms in piecewise-stationary bandits [21, 10, 22], our algorithm requires Assumption 1 and the knowledge of L max . These may not be available in practice. Thus, we conduct more experiments in Appendix O.2 to exhibit the robustness of PS ε BAI + . Specifically, in Appendix O.2, we conduct experiments for the case in which L max is misspecified . In Appendix O.3, we alter the change detection frequency γ so that w and b change accordingly. In both sets of experiments, the overall sample complexity of PS ε BAI + does not vary significantly and retains its superiority over N ε BAI. We conclude that PS ε BAI + is robust to slight misspecifications in these parameters, as long as Assumption 1 is not severely violated. Please refer to Appendix O for further details and experiments.

7 Conclusion and Future Work

We proposed a novel PSLB model and designed the PS ε BAI + algorithm to identify an ε -optimal arm with probability ≥ 1 -δ . The efficacy of PS ε BAI + has been demonstrated both empirically and theoretically. We argued that this is due to the embedded change detection and context alignment procedures. There are several directions for further exploration.

Firstly, our PS ε BAI + algorithm provides a fairly general framework for algorithm design. For instance, in addition to utilization the G-optimal allocation to sample arms as in PS ε BAI + , the XY -allocation and adaptive XY -allocation [1] can also be considered. In other words, our PS ε BAI + algorithm can be generalized to form an entire class of algorithms for BAI in PSLB models. In addition, deriving instance-dependent guarantees is also of great interest.

Secondly, most of the literature on piecewise-stationary bandits [21, 10, 22] make assumptions to provide theoretical guarantees. It would be interesting to remove or reduce these assumptions under our ε -BAI problem setup, and yet still be able to provide similar theoretical guarantees.

Finally, we believe that it is possible to adapt our PS ε BAI + algorithm to the fixed-budget setting, i.e., to identify an ε -optimal arm with high probability in a fixed time horizon in PSLB models.

Acknowledgements: This work is funded by the Singapore Ministry of Education AcRF Tier 2 grant (A-8000423-00-00) and Tier 1 grants (A-8000189-01-00 and A-8000980-00-00).

References

Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-arm identification in linear bandits. In Proceedings of the 27th Advances in Neural Information Processing Systems , volume 27, pages 828-836, 2014.

Junwen Yang and Vincent Tan. Minimax optimal fixed-budget best arm identification in linear bandits. Proceedings of the 36th Advances in Neural Information Processing Systems , 35:12253-12266, 2022.

Yasin Abbasi-yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Proceedings of the 24th Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011.

Marc Abeille and Alessandro Lazaric. Linear thompson sampling revisited. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , pages 176-184. PMLR, 2017.

Tanner Fiez, Lalit Jain, Kevin G Jamieson, and Lillian Ratliff. Sequential experimental design for transductive linear bandits. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Proceedings of the 32nd Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Marc Jourdan, Rémy Degenne, and Emilie Kaufmann. An ε -best-arm identification algorithm for fixed-confidence and beyond, November 2023. arXiv:2305.16041.

Rémy Degenne and Wouter M Koolen. Pure exploration with multiple correct answers. Proceedings of the 32nd Advances in Neural Information Processing Systems , 32, 2019.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Proceedings of the 27th Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In Jyrki Kivinen, Csaba Szepesvári, Esko Ukkonen, and Thomas Zeugmann, editors, The 22nd International conference on Algorithmic learning theory , pages 174-188, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg.

Yang Cao, Zheng Wen, Branislav Kveton, and Yao Xie. Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , pages 418-427. PMLR, 2019.

Yueyang Liu, Xu Kuang, and Benjamin Van Roy. A definition of non-stationary bandits, Feb 2023. arXiv:2302.12202.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Learning to optimize under nonstationarity. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1079-1087. PMLR, 2019.

Jonathan Shapiro, Carlos M Carvalho, and Pradeep Ravikumar. Thompson sampling in switching environments with bayesian online change point detection. In Proceedings of the 16th International Conference on Artificial Intelligence and Statistics , pages 442-450. Microtome Publishing, 2013.

Zhihan Xiong, Romain Camilleri, Maryam Fazel, Lalit Jain, and Kevin Jamieson. A/B testing and best-arm identification for linear bandits with robustness to non-stationarity. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238 , pages 1585-1593. PMLR, 2023.

Robin Allesiardo, Raphaël Féraud, and Odalric-Ambrym Maillard. The non-stationary stochastic multi-armed bandit problem. International Journal of Data Science and Analytics , 3:267-283, 2017.

Masahiro Kato and Kaito Ariu. The role of contextual information in best arm identification, 2021. arXiv:2106.14077.

Yoan Russac, Christina Katsimerou, Dennis Bohle, Olivier Cappé, Aurélien Garivier, and Wouter M Koolen. A/B/n testing with control in the presence of subpopulations. Proceedings of the 34th Advances in Neural Information Processing Systems , 34:25100-25110, 2021.

Andrea Zanette, Kefan Dong, Jonathan N Lee, and Emma Brunskill. Design of experiments for stochastic contextual linear bandits. Proceedings of the 34th Advances in Neural Information Processing Systems , 34:22720-22731, 2021.

Yasin Abbasi-Yadkori, András György, and Nevena Lazi´ c. A new look at dynamic regret for non-stationary stochastic bandits. Journal of Machine Learning Research , 24(288):1-37, 2023.

Jia Yuan Yu and Shie Mannor. Piecewise-stationary bandit problems with side observations. In Proceedings of the 26th International Conference on Machine Learning , ICML '09, page 1177-1184. Association for Computing Machinery, 2009.

Fang Liu, Joohyun Lee, and Ness Shroff. A change-detection based framework for piecewisestationary multi-armed bandit problem. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Lilian Besson, Emilie Kaufmann, Odalric-Ambrym Maillard, and Julien Seznec. Efficient change-point detection for tackling piecewise-stationary bandits. The Journal of Machine Learning Research , 23(1):3337-3376, 2022.

Peter Auer, Pratik Gajane, and Ronald Ortner. Adaptively tracking the best bandit arm with an unknown number of distribution changes. In Proceedings of the 32nd Conference on Learning Theory , pages 138-158. PMLR, 2019.

Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification in linear bandits. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Proceedings of the 34th Advances in Neural Information Processing Systems , volume 33, pages 10007-10017. Curran Associates, Inc., 2020.

Peng Zhao, Lijun Zhang, Yuan Jiang, and Zhi-Hua Zhou. A simple approach for non-stationary linear bandits. In The 23rd International Conference on Artificial Intelligence and Statistics , pages 746-755. PMLR, 2020.

Masahiro Kato, Masaaki Imaizumi, Takuya Ishihara, and Toru Kitagawa. Asymptotically optimal fixed-budget best arm identification with variance-dependent bounds, 2023. arXiv:2302.02988.

Chao Qin and Daniel Russo. On adaptivity and confounding in contextual bandit experiments. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021.

Yasin Abbasi-Yadkori, Peter Bartlett, Victor Gabillon, Alan Malek, and Michal Valko. Best of both worlds: Stochastic & adversarial best-arm identification. In Proceedings of the 31st Conference on Learning Theory , pages 918-949. PMLR, 2018.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Kwang-Sung Jun, Kevin G Jamieson, Robert D Nowak, and Xiaojin Zhu. Top arm identification in multi-armed bandits with batch arm pulls. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , pages 139-148, 2016.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best arm identification in multi-armed bandit models. Journal of Machine Learning Research , 17:1-42, 2016.

Marc Jourdan and Rémy Degenne. Choosing answers in epsilon-best-answer identification for linear bandits. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 10384-10430. PMLR, 17-23 Jul 2022.

Michael J. Todd. Minimum-Volume Ellipsoids: Theory and Algorithms . SIAM, 2016.

Appendices

The contents of the appendices are organized as follows:

In Appendix A, we provide further motivating examples for our problem.

In Appendix B, we discuss the limitations of our method.

In Appendix C, we review more related works on drifting and contextual bandits.

In Appendix D, we provide more details about our algorithms

Appendix D.1: pseudo-code of N ε BAI in Algorithm 2.

Appendix D.2: more details of PS ε BAI including the precise definition of the confidence radius ρ t and details of LCD and LCA subroutines.

Appendix D.3: pseudo-code of PS ε BAI + in Algorithm 5.

Appendix D.4: the computational complexity of PS ε BAI in Algorithm 1.

In Appendix E: we provide a useful lemma for estimating the expected return of any arm in linear bandits when the sampling rule is according to the G-optimal allocation.

In Appendix F: we proof the upper bound on the expected sample complexity of N ε BAI.

In Appendices G to K: we detail the analysis of PS ε BAI, i.e., we provide the proof of the upper bound on its complexity in Theorem 3.2.

Appendix G: outline of the proof.

Appendix H: analysis of the Change Detection (CD) and Context Alignment (CA) procedures.

Appendix I: analysis of the estimation error by decomposing it into three terms: VectorEstimation Error (VE), Distribution-Estimation Error (DE), and Residual Estimation Error (RE).

Appendix J: proof of Theorem 3.2 based on the analysis above.

Appendix K: proof of technical lemmas that are utilized in the analysis of PS ε BAI.

In Appendix L: we prove the upper bound on the expected sample complexity of PS ε BAI + based on the analysis of N ε BAI and PS ε BAI.

In Appendix M: we derive the lower bound on the expected sample complexity of any algorithm, i.e., proof Theorem 4.1.

In Appendix N: we provide more examples to compare the derived upper and lower bounds on the expected sample complexity, and illustrate the efficacy of our PS ε BAI + algorithm.

In Appendix O: we provide more details of numerical experiments.

In Appendix P: we provide more discussions on

-the related methods for BAI in nonstationary bandits,

-the instance-dependent upper bound,

-the connection between the piecewise-stationary linear bandits model to the stationary linear bandits model,

-the special case where N = 1 .

In Appendix Q: we provide analytical results on the 'Best Arm Tuple Identification Problem'.

A Further Motivating Examples

We elaborate on the some concrete real-life examples that motivate our problem setup of identifying the ensemble best arm in piecewise-stationary linear bandits.

In scenarios such as investment option selection and portfolio management also mentioned by [10, 20], there is a multitude of options for fund managers to choose from and typically, they want to find, in the initial pure exploration process, a small subset of candidate portfolios (or even the 'best' portfolio) based on various economic indicators and the market performance of individual stocks before further exploitation. In a bearish market, more portfolios tend to incur losses; while in a bullish market, more portfolios tend to generate gains. The transition between these two contexts can be effected by stochastic factors, e.g., the weather, or the outbreak of a pandemic, making the market conditions (contexts) stochastic. In the face of these uncertainties (in the contexts and rewards), we wish to design and analyze algorithms that selected portfolio to yield the best long-term option under such a piecewise-stationary environment.

Crop rotation is another example. Since crop yields can be influenced by various factors, such as weather conditions (analogous to our stochastically generated contexts), selecting the most suitable crop to grow and harvest from is crucial. Given several candidate crops, crops of similar types (e.g., potatoes and sweet potatoes) are correlated as they tend to favor similar conditions, thus, they can be

modelled by bandits with a linear reward structure. Contextual factors, like weather conditions, are well-modelled as being stochastic. A fixed weather condition will last a period of time and it will not change suddenly. Domain knowledge from historical data/records provides us with prior knowledge on L min and L max . These observations dovetail with our model. Our objective is to choose the crop that offers the near-highest yield potential over a long time period (an ensemble ε -best arm) and is adaptable to local environment factors, such as weather patterns.

B Limitations

Similar to existing works on piecewise stationary bandits [21, 10, 22], we introduced some assumptions to provide theoretical guarantees for PS ε BAI + although PS ε BAI + has shown to be robust even in the absence of these assumptions. Since an algorithm may not need to differentiate two contexts with close latent vectors for identifying an ε -optimal arm, we surmise it is possible to weaken these assumptions. For this purpose, we will consider clustering the contexts into a few classes based on the distances between their latent vectors and design an algorithm that only aims to detect the change of context class, instead of the change of context.

C More Related Work

In drifting bandits (DB), the regrets of algorithms are affected by the level of nonstationarity of the environment, which can be measured by various quantities, such as the total variation of the context sequence and the number of time steps when the return of at least one arm changes [23, 25].

In contextual bandits (CB), where the contextual information is visible to the agent, [16, 26, 17, 27] aim to identify the best arm with the assumption that the context changes at every time step according to a fixed distribution, and the return of an arm is averaged across all contexts. We see that the context distribution is involved to measure the quality of arms. However, while the agent in CB models can observe the context information, the agent has no access to the contexts but still aims to identify an ε -optimal arm in our piecewise stationary linear bandit (PSLB) model.

In adversarial bandits , existing works pertaining to the BAI problem only explored the fixed-horizon setting [28]. Due to the difference between the fixed-confidence and fixed-horizon setting and the difference between adversarial and piecewise stationary bandits, their results cannot be trivially extended to solve the fixed-confidence BAI problem in piecewise stationary bandits.

D More Details of Algorithms N ε BAI , PS ε BAI and PS ε BAI +

All proposed algorithms make use of the well known G-optimal allocation (design) [1, 29], which is widely used in the linear bandits literature. The G-optimal allocation minimizes the maximal mean-squared prediction error in all directions [1]. Given an arm set X , the G-optimal allocation λ ∗ is a distribution over the arm set, which is the minimizer of g ( λ ) = max x ∈X ∥ x ∥ 2 A ( λ ) -1 , where A ( λ ) = ∑ x ∈X λ ( x ) xx ⊤ and λ ∈ ∆ X . Interested readers may refer to Chapter 21 in [29].

D.1 The NAÏVE ε -BEST ARM IDENTIFICATION (N ε BAI) Algorithm

As presented in Algorithm 2, N ε BAI samples an arm with the G-optimal allocation at each time steps; its stopping rule is grounded on the property of G-optimal allocation (see Lemma E.1) and affected by the maximum length of a single stationary segment L max .

Algorithm 2 NAÏVE ε -BEST ARM IDENTIFICATION (N ε BAI)

1: Input: the arm set X , the phase length bounds L min , L max , the slackness parameter ε , the confidence parameter δ .

2: Initialize : Compute the G-optimal allocation λ ∗ .

4: Sample t ∗ arms { x s } t ∗ s =1 ∼ λ ∗ and observe the associated returns { Y s,x s } t ∗ s =1 . Let t = t ∗ .

3: Compute C 3 = ∑ ∞ n =1 n -3 and t ∗ = 3 d ln(6 dKC 3 /δ ) .

5: Compute

̸

6: while ˙ x ⊤ t ˜ θ t -˜ ρ t + ε < max x =˙ x t x ⊤ ˜ θ t + ˜ ρ t do

7: Sample an arm x t ∼ λ ∗ and observe return Y t,x t and let t = t +1 .

8: Update ˜ θ t , ˙ x t and ˜ ρ t with (D.1).

9: end while

10: Recommend arm ˙ x ε = ˙ x t .

D.2 Details about PS ε BAI

D.2.1 Confidence radius utilized in PS ε BAI

For each pair of arms ( x, ˜ x ), the confidence radius of ∆( x, ˜ x ) at time step t is

where a clip 2 := min { max { a, -2 } , 2 } denotes the value of a that is clipped to the interval [ -2 , 2] and

ζ t ( x, ˜ x ) minimizes the last summation. In the final theoretical upper bound on the sample complexity, we take ζ t ( x, ˜ x ) = ε for simplicity. In the experiment, we utilize

for a simple and effective analytic expression in the experiment.

D.2.2 Subroutines of PS ε BAI

In the LCD subroutine (Algorithm 3), two estimates ˜ θ 1 and ˜ θ 2 of the latent vectors are independently computed from the first and second halves of the input CD samples (Line 2). LCD only raises an alarm of changepoint (Line 4) when the difference between ˜ θ 1 and ˜ θ 2 is sufficiently large (Line 3) and indicates a changepoint occurs w.h.p.

The LCA subroutine (Algorithm 4) estimates the latent vector of current context and updates the dictionary CA id . Specifically, it firstly samples w/ 2 samples with λ ∗ (Line 2). It then checks

Algorithm 3 LINEAR-CHANGE DETECTION (LCD)

1: Input: arm set X , detection window w , threshold b and w arms with the associated observations [(˜ x 1 , Y s, ˜ x 1 ) , . . . , (˜ x w , Y w, ˜ x w )] .

3: if ∃ x ∈ X , s.t. | x ⊤ ( ˜ θ 2 -˜ θ 1 ) | > b then

2: Compute ˜ θ 1 = 2 w ∑ w 2 s =1 A ( λ ∗ ) -1 ˜ x s Y s, ˜ x s and ˜ θ 2 = 2 w ∑ w s = w 2 +1 A ( λ ∗ ) -1 ˜ x s Y s, ˜ x s .

4: Return True

5: else

6: Return False

7: end if

Algorithm 4 LINEAR-CONTEXT ALIGNMENT (LCA)

1: Input: arm set X , detection window w , threshold b and context ids CA id .

2: Sample w 2 arms { ˜ x s } w 2 s =1 ∼ λ ∗ and observe the associated returns { Y s, ˜ x s } w 2 s =1 , where

3: for j ∈ CA id do

5: CA id [ j ] = [(˜ x s , Y s, ˜ x s )] w 2 s =1 .

4: if not LCD ( X , w, b, [(˜ x s , Y s, ˜ x s )] w 2 s =1 +CA id [ j ]) then

6: Return j, CA id .

7: end if

8: end for

9: index = len(CA id ) + 1 .

10: CA id [index] = [(˜ x s , Y s, ˜ x s )] w 2 s =1 .

11: Return index , CA id .

whether the current latent context has been visited in previous time steps by scanning through CA id (Line 3). In order to learn if the current context can be aligned with context j , the LCD subroutine is called with the w/ 2 recent samples and CA id [ j ] as input:

(i) If the current context is aligned with CA id [ j ] (Line 4), the current latent context is j w.h.p. Thus the LCA subroutine updates CA id [ j ] and returns the index j (Lines 5 to 6), which would be ˆ j t in Line 19 of Algorithm 1. Note that in Lines 10 and 11 of Algorithm 1, all the collected samples will be assigned to context ˆ j t (until the next changing alarm) and will be used to estimate θ ∗ ˆ j t and all p j 's. Therefore, aligning contexts allows PS ε BAI to make good use of observation history.

(ii) If the current context is not aligned with any observed context, i.e., it has not been visited, CA id gets extended with a new index-samples pair and is returned along with the new index (Lines 9 to 11).

D.3 The PIECEWISE-STATIONARY ε -BEST ARM IDENTIFICATION + (PS ε BAI + ) Algorithm

To help understand PS ε BAI + algorithm, we highlight the differences between PS ε BAI and PS ε BAI + , and the differences between subroutines LINEAR-CONTEXT ALIGNMENT (LCA) and LINEAR-CONTEXT ALIGNMENT + (LCA + ).

,

̸

̸

̸

̸

Algorithm 6 ESTIMATE UPDATE (EU)

1: Input: time step t , vector ˜ θ t -1 , threshold t ∗ and constant C 3 .

3: if t ≥ t ∗ then

5: end if

6: Return ˜ θ t , ˙ x t , ˜ ρ t .

Algorithm 7 LINEAR-CONTEXT ALIGNMENT + (LCA + )

1: Input: arm set X , detection window w , threshold b , context ids CA id , time step t , vector ˜ θ t , threshold t ∗ and constant C 3 .

̸

2: Set s = 0 , ˙ x ε = -∞ . 3: for s ≤ w/ 2 do 4: s = s +1 , t = t +1 . 5: Sample an arm ˜ x s ∼ λ ∗ and observe return Y s, ˜ x s . 6: Set ( x t , Y t,x t ) = (˜ x s , Y s, ˜ x s ) . 7: Compute ˜ θ t , ˙ x t , ˜ ρ t = EU ( t, ˜ θ t -1 , t ∗ , C 3 ) . 8: if ˙ x ⊤ t ˜ θ t -˜ ρ t + ε ≥ max x =˙ x t x ⊤ ˜ θ t + ˜ ρ t then 9: ˙ x ε = ˙ x t , index = len(CA id ) + 1 . 10: Return index , CA id , ˙ x ε , ˜ θ t . 11: end if 12: end for 13: for j ∈ CA id do 14: if not LCD ( w,b, [(˜ x s , Y s, ˜ x s )] w 2 s =1 , CA id [ j ]) then 15: CA id [ j ] = [(˜ x s , Y s, ˜ x s )] w 2 s =1 . 16: Return j, CA id . 17: end if 18: end for 19: index = len(CA id ) + 1 . 20: CA id [index] = [(˜ x s , Y s, ˜ x s )] w 2 s =1 . 21: Return index , CA id , ˙ x ε , ˜ θ t .

D.4 Computational Complexity of PS ε BAI

We provide analysis for the computational complexity of PS ε BAI in this subsection.

We consider the number of these operations: arithmetic (addition and multiplication) operations, logic operations and comparison operations and we also regard ln( · ) , √ · and sampling from a distribution as one step of operation.

We decompose the main loop of Algorithm 1 as follows, where the lines with O (1) operations are omitted:

Exploration phase (Lines 8 to 11): O (1) .

Change Detection phase (Lines 12 to 16):

The LCD subroutine in Line 16 needs O ( wd 2 + Kd ) operations.

Context Alignment phase (Lines 17 to 21):

The LCA subroutine in Line 19 needs O (( wd 2 + Kd ) N ) operations, as the LCD subroutine will be invoked N times in the worst case in Line 4 of LCA .

The reversion procedure in Line 21 needs O ( γwd ) operations.

The updating procedure and the stopping rule checking (Lines 25 to 32):

-Updating ˆ θ t,j needs O ( d 2 ) operations, as we need to incorporate the latest sample into the estimate.

-Updating ˆ p t,j , j ∈ [ N ] requires N operations.

-Updating the confidence radius and the empirical best arm need O ( KN ) operations.

The stopping condition in Equation (3.4) requires O ( K ) operations.

We remark that some intermediate results can be stored to avoid repeated computations and we believe the algorithm is efficient overall. In particular, since the reward structure is linear, there are K arms, and N contexts, O ( d 2 ) , O ( K ) , and O ( N ) operations probably cannot be be avoided.

From the above analysis, in the decreasing order of the number of operations:

The Change Detection phase requires the most budget as it will be invoked every γ time steps.

The Context Alignment phase requires the second many operations. While it requires many operations when it is implemented, it will not be called during a stationary segment.

The updating procedure uses small portion of the operations, as w is usually much greater than K and N .

The exploration phase demands constant operations in each loop.

E Auxiliary results

Lemma E.1. Let X s be the arm drawn with the G-optimal allocation λ ∗ at time step s and Y s,x s = x ⊤ s θ ∗ j s + η s be the corresponding return with E η s = 0 , | x ⊤ s θ ∗ j s | ≤ 1 and | η s | ≤ 1 . Then we have

for all ϵ ≥ d n ln 2 δ + √ ( d n ln 2 δ ) 2 + 4 d n ln 2 δ . In addition, if n ≥ d 4 ln 2 δ , we have

Proof. Note that the arms are selected according to the G-optimal design, and thus E x ∼ λ ∗ [ xx ⊤ ] = A ( λ ∗ ) . Therefore, for all s ∈ [ n ] ,

and

where we make use of the property of the G-optimal allocation in the last inequality

Additionally,

where we make use of the fact that η t is zero-mean and is independent of other random variables in ( a ) ; | x ⊤ s θ ∗ j s | ≤ 1 and P [ η 2 s ≤ 1 ] = 1 in ( b ) ; x s ∼ λ ∗ in ( c ) ; and the property of the G-optimal allocation (E.3) in ( d ) .

According to the Bernstein's inequality with (E.1), (E.2) and (E.4),

In order to upper bound the error probability by δ , we need

In addition, if n ≥ d 4 ln 2 δ , we have

This finishes the proof.

F Analysis of N ε BAI

To analyze the theoretical performance of N ε BAI, we first show that it can identify an ε -optimal arm with probability 1 -δ and derive a high-probability upper bound on its stopping time in Lemma F.1.

Lemma F.1 (High-probability upper bound of N ε BAI) . With probability 1 -δ , the N ε BAI algorithm identifies an ε -optimal arm after at most

time steps, where ∆ min = min i = i ∗ ∆( x ∗ -x i ) .

̸

Next, we prove that after a sufficiently large number of time steps, the probability that N ε BAI does not terminate is small in Lemma F.2.

Lemma F.2. Let

with C 3 = ∑ ∞ n =1 n -3 . For t ≥ T 0 , the probability that N ε BAI does not terminate before t time steps is δ ( α -1) C 3 ( t/ 2) 2 .

Lastly, we apply Lemmas F.1 and F.2 to prove Proposition 3.1.

̸

Proposition 3.1. Let ∆ min = min x = x ∗ ∆( x ∗ , x ) ,

The N ε BAI algorithm is ( ε, δ ) -PAC and its expected sample complexity is ˜ O ( T N V + T N D ) .

The detailed analysis is presented as below.

F.1 Detailed analysis of N ε BAI

Proof of Lemma F.1. The result is to be proven with the following procedures.

First step : Prove ∣ ∣ ∣ x ⊤ E θ ∼ P θ θ -x ⊤ ˜ θ t ∣ ∣ ∣ can be bounded with high probability.

Let ¯ θ t = 1 t ∑ t s =1 θ ∗ j s . (i) For all ε > 0 , we have,

Since | x ⊤ E θ ∼ P θ θ | ≤ 1 and x ⊤ θ ∗ j c l | ≤ 1 for all l , we obtain (a) by applying Hoeffding's inequality. We obtain (b) from the fact that ∑ N t l =1 L l = t and L l ≤ L max for all l . In other words, for all δ > 0 ,

(ii) Besides, according to Lemma E.1, if t ≥ d 4 ln 2 δ , we have

(iii) For all t > 0 , all x ∈ X , since

we have where

For simplicity, we set α = 3 and write ˜ ρ t (3) as ˜ ρ t in the following analysis. We now show that t ≥ d 4 ln 4 KC α t α δ holds when t ≥ 3 d ln 6 dKC 3 δ with the following lemma (the proof of which is presented by the end of Appendix F.1).

Lemma F.3. Let a, b, c > 0 and ac ≥ 1 / 2 , then

With

Define event ˜ E t = ⋂ x ∈X {∣ ∣ ∣ x ⊤ E θ ∼ P θ θ -x ⊤ ˜ θ t ∣ ∣ ∣ ≤ ˜ ρ t ( α ) } for t ≥ t ∗ and ˜ E = ⋂ t ≥ t ∗ ˜ E t . We have

We assume ˜ E holds in the second and third steps in this proof.

Second step : Prove N ε BAI recommends an ε -optimal arm when it stops.

̸

If the algorithm terminates and returns ˙ x ε = x ∗ , we have

̸

̸

̸

implying that ˙ x ε ∈ X ε . Hence, we always have ˙ x ε ∈ X ε .

Third step : Derive the stopping time of N ε BAI.

̸

(i) Let x (2) = arg max x = x ∗ x ⊤ E θ ∼ P θ θ and ∆ min = min x = x ∗ ∆( x ∗ -x ) = ( x ∗ -x (2) ) ⊤ E θ ∼ P θ θ . We apply the following lemma to show that N ε BAI stops when ˜ ρ t is sufficiently small.

̸

Lemma F.4 ([30, Lemma 3]) . Denote by ˆ i the index of the item with empirical mean is i -th largest: i.e., ˆ w ( ˆ 1) ≥ . . . ≥ ˆ w ( ˆ L ) . Assume that the empirical means of the arms are controlled by ε : i.e., ∀ i, | ˆ w ( i ) -w ( i ) | < ε . Then,

Since

we have

̸

When ˜ ρ t ≤ ( ε +∆ min ) / 4 , we have

which will lead to the termination of N ε BAI.

(ii) According to the definition of ˜ ρ t , we have

̸

time steps.

Proof of Lemma F.2. To begin with, let

For any T ≥ T 0 , let ¯ T = ⌈ T/ 2 ⌉ .

(I) If N ε BAI terminates within ¯ T time steps, there is nothing to prove.

(II) Assume N ε BAI does not terminates within ¯ T time steps. According to the analysis in the proof of Lemma F.1, if N ε BAI does not terminate before T time steps, i.e., the stopping time τ satisfies ¯ T -1 ⋂ ˜

that τ ≥ T , then event t = ¯ T +1 E t does not hold. This indicates

(a) results from the fact that ¯ T ≥ T/ 2 .

Altogether , for T ≥ T 0 , the probability N ε BAI does not terminate before T time steps is δ ( α -1) C α ( T/ 2) α -1 .

Proof of Proposition 3.1. First, Tonelli's Theorem implies that

Next, we apply Lemma F.2 to show

Besides, Lemma F.1 indicates that N ε BAI can identify an ε -optimal arm with probability 1 -δ .

We apply Lemma F.3 to invert the last line above. With

Altogether , we show that with probability 1 -δ , N ε BAI identifies an ε -optimal arm after at most

Proof of Lemma F.3. For x > 0 , let

Since

Let d = 2 ac , x = d ln( dy ) , then x ≥ 2 ac ln x = d ln x ⇔ d ln( dy ) ≥ d ln( d ln( dy )) ⇔ ln( dy ) ≥ ln( d ln( dy )) ⇔ y ≥ ln( dy ) .

Since z 0 . 4 ≥ ln z for all z ≥ 1 , we have ln( dy ) ≤ ( dy ) 0 . 4 ≤ √ dy and

when yd ≥ 1 . Hence, when ac ≥ 1 / 2 , y ≥ d = 2 ac ≥ 1 , we have y ≥ ln( dy ) . Furthermore, for x such that x ≥ max { 2 a ln b, 4 ac ln(2 ac ) } , we have f ( x ) ≥ 0 .

G Proof Outline of Theorem 3.2

A proof outline of Theorem 3.2 is provided in this section. It consists of three steps:

Step 1 : PS ε BAI (Algorithm 1) depends on the success of change detection and context alignment (Algorithm 3 and 4). We firstly upper bound the failure probability of these two subroutines via Lemma G.1, Lemma G.2, Lemma G.3 and summarized in Lemma G.4. More details about these two subroutines are provided in Appendix D.2.2 and the proof of the Lemmas are postponed to Appendix H.

Step 2 : Subsequently, conditioned on their success, we provide a theoretical guarantee for the choice of the confidence radius ρ t ( x, ˜ x ) for ∆( x, ˜ x ) at time step t in Lemma G.5. The proof is detailed in Appendix I.

Step 3 : Lastly, utilizing the above elements, we provide a sufficient condition for the stopping rule, and upper bound the number of time steps in Exp phases T τ via Lemma G.6 whose proof is presented in Appendix J. As the total number of time steps τ is upper bounded by a constant multiple of T τ , the high-probability upper bound on the stopping time τ is obtained. This finishes the proof of Theorem 3.2 (please refer to Appendix J).

Step 1 : We borrow the terminology in hypothesis testing to define the errors. Let the null hypothesis be: the algorithm has undergone a changepoint within the last w CD samples . For Algorithm 3, we will characterize the type I error: the algorithm has experienced a changepoint within the last w CD samples but it fails to raise a changing alarm , and the type II error: a changepoint has not occurred but Algorithm 3 raises a changing alarm . We refer to the event Failed Alarm (FA) and the event False Alarm Error (FAE) as that a type I error occurs and that a type II error occurs, respectively:

FA l := { a type I error occurs at changepoint c l } , l ∈ N ,

FAE t := { a type II error occurs time step t } , t ∈ T FAE ,

where T FAE := { t : t in CD phase , t ≤ τ, [ t -wγ,t ] ∩ C = ∅} and τ is the stopping time. In term of Algorithm 4, define the event MI l := { Algorithm 4 misidentifies θ ∗ j t } , l ∈ N and MI := ⋃ { l : c l ≤ τ } MI l . Define the good event

We upper bound the failure probability of Goodin Lemma G.1, Lemma G.2, Lemma G.3 and conclude the results in Lemma G.4.

Lemma G.1. For any δ FAE ∈ (0 , 1) , with b ≥ 8 d 3 w ln 2 δ FAE + √ ( 8 d 3 w ln 2 δ FAE ) 2 + 24 w d ln 2 δ FAE , LCD makes no false alarm before the stopping time τ with probability at least

Lemma G.1 can also be stated as, fix b ,

which indicates we can always decrease the error probability by enlarging the window size w .

̸

Assume c l is observable, i.e., θ ∗ j c l = θ ∗ j c l +1 , and denote ˆ c l as the alarm time of c l from Algorithm 3.

Lemma G.2. Conditional on FAE c , for any δ FA ∈ (0 , 1) , if ∆ c -b 2 ≥ d w ln 2 δ FA +

In addition, P [ FA ∣ ∣ FAE c ] ≤ l τ δ FA .

Lemma G.2 guarantees a prompt change alarm will be raised within wγ 2 time steps after a changepoint occurs. This also explains why wγ 2 samples are abandoned at line 16 of Algorithm 1 so that T t,j only keeps samples from context j .

Lemma G.3. Conditional on FAE c and FA c , based on the conditions in Lemma G.1 and Lemma G.2,

The intuition of Lemma G.3 is simple: change alarm should not be raised when the samples are from the same context and, change alarms should be raised when the samples are from the other ( N -1) contexts. The error made by Algorithm 3 and 4 can be concluded as follows

Lemma G.4. Assume the instance satisfies Assumption 1,

Step 2 : To give an upper bound on T τ , we firstly prove that the estimate of ∆( x, ˜ x ) at time t is within distance ρ t ( x, ˜ x ) from the ground truth w.h.p. Define the event CI as the estimates of all the mean gaps locate in the high-probability Confidence Interval

and CI := ⋂ t CI t , where ρ t is defined in (3.3).

Lemma G.5. If T t ≥ 2 L max 9 ln 2 δ d,T t ,

In addition,

In addition to an estimation over the latent vector which is sufficient under the stationary case ( α t in ρ t ) , we also need to control the derivation between the empirical distribution ˆ p t and the ground truth p ( β t in ρ t ), as well as the interactions between the latent vectors and the distribution ( ξ t in ρ t ). This is reflected by Kδ v,T t , Nδ d,T t and KNδ m,T t which bound the the failure probability of Vector Estimation, Distribution Estimation and Residual Estimation, respectively.

Step 3 :

Lemma G.6. Conditional on Good and CI , the recommended arm ˆ x ε ∈ X ε and when Algorithm 1 terminates, the order of T t is upper bounded by (3.6) .

The detailed proof is presented in Appendix J.

H Analysis of PS ε BAI : Change Detection and Context Alignment

As the output of Algorithms 3 and 4 depends on random samples, the output may not meet our expectation with some probabilities. We will characterize the probabilities of the three 'bad' events: FAE in Appendix H.1, FA in Appendix H.2 and MI in Appendix H.3; and finally upper bound the failure probability of the good event Good in this section.

Lemma G.4. Assume the instance satisfies Assumption 1,

Some notations are introduced first

θ 1: ∞ := ( θ ∗ j t ) ∞ t =1 is the latent vector sequence.

ˆ c l indicates the time step when we raise alarm for the l th changepoint.

According to the dynamics (see Dynamics 1) of the problem, at a changepoint c l ∈ C , the next latent vector will be sampled from P θ . Therefore, it may happen that θ ∗ j c l = θ ∗ j c l -1 , i.e, this changepoint c l is hidden and these two consecutive stationary segments are observed as one stationary segment. In order to upper bound the errors made by Algorithm 3, we make the following assumption, which yields the worst-case scenario in terms of the error probability.

̸

Assumption 2. Given θ 1: ∞ and the changepoint sequence C , θ ∗ j c l = θ ∗ j c l +1 , ∀ l ∈ N .

In this section, we do not consider the randomness of θ 1: ∞ , i.e., we consider a realization of the sequence. 4 The results do not involve any parameter depending on the specific sequence and thus can be applied to any latent vector sequence.

H.1 False Alarm Error

Lemma G.1. For any δ FAE ∈ (0 , 1) , with b ≥ 8 d 3 w ln 2 δ FAE + √ ( 8 d 3 w ln 2 δ FAE ) 2 + 24 w d ln 2 δ FAE , LCD makes no false alarm before the stopping time τ with probability at least

Proof of Lemma G.1. Note that within a stationary segment of length t , the observations are i.i.d., and thus the time segments between two consecutive false alarm time { ˆ c l -ˆ c l -1 } t l =1 (where ˆ c 0 = 0 ) are i.i.d. i.e., it is a renewal process. Thus, it is sufficient to bound the error probability under the stationary case, i.e. C = ∅ and θ 1: ∞ = ( θ ∗ j t = θ ∗ j 1 ) ∞ t =1 .

Assume that we are under a stationary segment of length t . We wish to upper bound the error P [FAE] = P [ˆ c 1 ≤ t ] . This is given by Lemma H.1.

Lemma H.1. Given w CD samples from the same context [(˜ x s , Y s, ˜ x s )] w s =1 in a CD phase at time step t , Algorithm 3 makes a false alarm error with probability upper bounded by Kδ FAE , i.e.

̸

Assume γ divides t . In the case of a stationary segment, the input samples to Algorithm 3 are { [( x sγ , Y sγ,x sγ )] k + w -1 s = k } t γ k =1 if no changing alarm is raised (We ignore the initialization at Line 5 in Algorithm 1 for simplicity). Denote Γ i ( t ) := { kγ : k ≡ i mod w,kγ ≤ t } , i = 0 , 1 , . . . , w -1 . Given any i = 0 , . . . , w -1 , for any k 1 = k 2 ∈ Γ i ( t ) , the two list of samples, [( x sγ , Y sγ,x sγ )] k 1 + w -1 s = k 1 and [( x sγ , Y sγ,x sγ )] k 2 + w -1 s = k 2 , do not overlap (thus, they are independent). Therefore, P [ k ] := P [ˆ c 1 =

kγ + i, ˆ c 1 ∈ Γ i ( t )] is a geometric distribution with parameter upper bounded by Kδ FAE . We have

Hence, the cumulative false alarm error is

Proof of Lemma H.1. According to Algorithm 3, given w CD samples from the same context [(˜ x s , Y s, ˜ x s )] w s =1 and x ∈ X , we need to bound

The proof is similar to Lemma E.1. For any s ∈ [ w 2 ] and ˜ s = s + w 2

.

and

By making use of (E.4),

According to the Bernstein's inequality,

In order to upper bound the error probability by δ FAE , we need

if

c

-

2

∆

b

d

2

d

w

ln

δ

FA

+

w

ln assumed to be greater than b .

By the choice of b ≥ 8 d 3 w ln 2 δ FAE + √ ( 8 d 3 w ln 2 δ FAE ) 2 + 24 w d ln 2 δ FAE , we have

A union bound over the K arms yields the final result.

H.2 Failed Alarm

Lemma G.2. Conditional on FAE c , for any δ FA ∈ (0 , 1) , if ∆ c -b 2 ≥ d w ln 2 δ FA + √ ( d w ln 2 δ FA ) 2 + 4 d w ln 2 δ FA ,

In addition, P [ FA ∣ ∣ FAE c ] ≤ l τ δ FA .

Proof of Lemma G.2. Conditioned on FAE c , the detection at the changepoints is independent. Thus we can assume there is only one changepoint c 1 within a certain number of consecutive time steps.

Algorithm 3 is given w CD samples which are collected under two different context. Without loss of generality, we assume the sample selected at time c 1 is among the CD samples (otherwise, we can regard the time step of the first sample from the second context as c 1 ). We wish to bound

which is the probability of the event that, after a changepoint occurs, a changing alarm will be raised within wγ 2 time steps (or w 2 CD samples). Here, ˆ c 1 ≥ c 1 can be guaranteed as we are conditioning on that there is no false alarm error ( FAE c ).

The event c 1 ≤ ˆ c 1 ≤ c 1 + γw 2 indicates, at least one of the CD sample list in

will trigger the changing alarm in Algorithm 3. In particular, in Lemma H.2, we consider the failed arm probability when the CD samples are composed by exactly half samples from each contexts,

Lemma H.2. Given w CD samples from the two different contexts [(˜ x s , Y s, ˜ x s )] w s =1 in a CD phase, where [(˜ x s , Y s, ˜ x s )] w 2 s =1 is from latent vector θ ∗ j and [(˜ x s , Y s, ˜ x s )] w s = w 2 +1 is from latent vector θ ∗ ˜ j , Algorithm 3 raises a changing alarm with probability lower bounded by

1

ln

-

δ

2

δ

FA

FA

where

δ

2

FA

)

2

+

4

d

w

∆

c

x

∈X

|

x

⊤

(

θ

∗

j

-

θ

∗

˜

j

)

|

and it is

where ∆ c = max x ∈X | x ⊤ ( θ ∗ j -θ ∗ ˜ j ) | and θ ∗ j , θ ∗ ˜ j are the latent vectors. Hence,

Proof of Lemma H.2. According to the design of LCD, there exists an x ∈ X ,

H.3 Context Alignment

Lemma G.3. Conditional on FAE c and FA c , based on the conditions in Lemma G.1 and Lemma G.2, P [MI | FAE c , FA c ] ≤ l τ [( N -1) δ FA + Kδ FAE ] .

Proof of Lemma G.3. The error of the context alignment procedure can be derived from the FAE and FA analyses. Conditioned on FAE c , FA c and that the previous l -1 contexts are correctly identified, i.e., ⋂ l -1 k =1 MI c k , we have the following statements.

Firstly, according to Lemma H.1, the change alarm at Line 4 in Algorithm 4 will not be triggered with probability at least (1 -Kδ FAE ) if the current context is context j . This error has been taken into account in the FAE (see Remark H.3).

Secondly, if the current context is not j (which will occur at most N -1 times), a change alarm will be raised with probability at least 1 -δ FA by Lemma G.2.

Therefore, the error probability at c l is upper bounded by

and by a union bound, the cumulative error probability bounded by

Remark H.3. Note that (1) we will only bound the FAE when the CD samples are from the same context; (2) in the analysis of FAE , we bound the error probability on the whole horizon up to time t

as we assume there is no changepoint. In particular, there will be unused and redundant w 2 Kδ FAE errors budget for FAE at each changepoint, which accumulates to l τ w 2 Kδ FAE before time step τ . Therefore, the second error term in (H.1) , l τ Kδ FAE , can be covered by the unused l τ w 2 Kδ FAE error budget from FAE , and thus it can be neglected in (H.1) .

Proof of Lemma G.4. According to Lemma G.1, G.2, G.3, Assumption 1 and Remark H.3, given a time τ , the total failure probability is upper bounded by

In particular, when τ is upper bounded by τ ∗ in Line 2 of Algorithm 1

where c 0 = 3 · 6400 ln 6400 . The number of changepoints till τ is upper bounded by l τ ∗ ≤ τ ∗ L min .

By Assumption 1, when b = 8 d 3 w ln 2 δ FAE + √ ( 8 d 3 w ln 2 δ FAE ) 2 + 24 w d ln 2 δ FAE and δ FAE = γδ 4( τ ∗ ) 2 K , the conditions of Lemma G.1 are met. And we can upper bound τ γ Kδ FAE by δ 4 τ ∗ ≤ δ 4 .

By making use of Assumption 1 and setting δ FA = δ 4 Nl τ ∗ τ ∗ , we have δ FA > δ FAE and

Thus, Lemma G.2 can be applied. l τ ∗ Nδ FA can be upper bounded by δ 4 τ ∗ ≤ δ 4 .

Therefore, according to (H.2), P [Good] ≥ 1 -δ 2 τ ∗ ≥ 1 -δ 2 .

I Analysis of PS ε BAI : Estimation Error

Lemma G.5 is proved in this section. We will upper bound these three error terms (see (I.1)): VE error in App I.1, DE error in App I.2, RE error in App I.3, and finally we will prove Lemma G.5 which upper bounds the failure probability of CI at the end of this section.

Lemma G.5. If T t ≥ 2 L max 9 ln 2 δ

In addition,

Given any two arms x, ˜ x ∈ X , by the triangular inequality, the deviation between ˆ ∆ t ( x, ˜ x ) and ∆( x, ˜ x ) can be upper bounded by three terms: the Vector-Estimation Error (VE) term, the DistributionEstimation Error (DE) term, and the Residual Estimation Error (RE) term:

where a clip 2 := clip 2 ( a ) := min { max { a, -2 } , 2 } is a shorthand notation for the value of a that is clipped to the interval [ -2 , 2] . The reason the value ˆ ∆ t,j ( x, ˜ x ) is clipped is the ground truth | ∆ t,j ( x, ˜ x ) | ≤ 2 .

Recall the event CI

and the confidence radius where

and ζ t ( x, ˜ x ) ∈ R can be any value. In particular, it can be the value that minimizes ∑ N j =1 β t,j | ˆ ∆ t,j ( x, ˜ x ) + ζ t ( x, ˜ x ) | or it can be taken as ε . For simplicity, we will take ζ t ( x, ˜ x ) = arg min ζ t ( x, ˜ x ) ∈ R ∑ N j =1 β t,j ( ˆ ∆ t,j ( x, ˜ x )+ ζ t ( x, ˜ x )) 2 = -∑ N j =1 β t,j ˆ ∆ t,j ( x, ˜ x ) ∑ N j =1 β t,j for a simple and effective analytic expression in the algorithm.

I.1 Vector-Estimation Error

Lemma I.1. Given x ∈ X and T t ≥ d ln 2

Proof of Lemma I.1. By the definition of the estimators in (3.2),

By the property of conditional probability, we have the desired result.

Therefore, conditional on Good, with probability at least 1 -Kδ v,T t , VE ≤ 2 α t for any x, ˜ x ∈ X .

I.2 Distribution-Estimation Error

Lemma I.2. Given Good and T t ≥ 2 L max 9 ln 2 δ d,T t , for any j ∈ [ N ] ,

Additionally, with probability 1 -Nδ d,T t ,

where ζ t ( x, ˜ x ) can be any value.

Proof of Lemma I.2. Given any j ∈ [ N ] , and the stationary segment l , we denote X j,l := ˆ L l ✶ { θ ∗ j l = θ ∗ j } -p j ˆ L l , where ˆ L l is the total length of the Exp phases in the l th stationary segment. Note that E [ X j,l |{ ˆ L l } l t l =1 , Good] = 0 , | X j,l | ≤ L max a.s. and Var[ X j,l |{ ˆ L l } l t l =1 , Good] = p j (1 -p j ) ˆ L 2 l ≤ p j (1 -p j ) ˆ L l L max . By Bernstein's inequality,

As T t,j = ∑ l t l =1 ˆ L l { θ ∗ j l = θ ∗ j } , T t = ∑ l t l =1 ˆ L l and ∑ l t l =1 ˆ L l = T t , we have

As the last bound is independent of { ˆ L l } l t l =1 , we have

If we want to upper bound the above by δ d,T t ∈ (0 , 1) , i.e.,

In particular,

satisfies the condition, we have

A problem here is we do not have access to p j during the dynamics, therefore, we adopt Lemma K.1 to further upper bound ˜ β t,j by β t,j .

Lemma K.1. Given any j ∈ [ N ] and T t ≥ 2 L max 9 ln 2 δ d,T t , ˜ β t,j ≤ β t,j .

Therefore,

In addition, with probability at least 1 -Nδ d,T t , we can upper bound DE term as:

where ζ t ( x, ˜ x ) ∈ R can be any value.

I.3 Residual-Estimation Error

RE is composed by the product of two deviations, i.e., ( ∆ j ( x, ˜ x ) -ˆ ∆ clip 2 t,j ( x, ˜ x ) ) and (ˆ p t,j -p j ) , as the time step t becomes large, we expect it will converge to zero fast. Thus, it is sufficient to have a coarse estimation of it.

Lemma I.3. For any x ∈ X , conditional on that | ˆ p t,j -p j | ≤ β t,j , ∀ j ∈ [ N ] ,

where ξ t := 25 √ 2 NL max T t ln 2 δ m,Tt .

Proof of Lemma I.3. According to Lemma E.1, for any x ∈ X , j ∈ [ N ] we have

if T t,j ≥ d 4 ln 2 δ m,Tt . Note the fact that ∆ j ( x, ˜ x ) ∈ [ -2 , 2] , hence,

And | x ⊤ ( θ ∗ j -ˆ θ t,j ) | ≤ 3 d with probability 1 . Denote

We have 5

where ( a ) adopts (I.7) and ( b ) is obtained by the following derivations:

(1) ψ t,j, 1 > ˆ p t,j indicates ˆ p t,j < d 4 T t ln 2 δ m,Tt < 25 4 L max T t ln 2 δ d,T t , and thus β t,j ≤ 5 2 · 5 √ 2 2 L max T t ln 2 δ d,T t .In addition, 6 ,

(2) ψ t,j, 1 = ˆ p t,j indicates ˆ p t,j ≤ d 4 T t ln 2 δ m,Tt and ψ t,j, 2 > ˆ p t,j implies ˆ p t,j < 25 4 L max T t ln 2 δ d,T t , thus (I.6) can be applied and β t,j ≤ 5 2 · 5 √ 2 2 L max T t ln 2 δ d,T t .

Therefore, conditional on Good, with probability at least 1 -KNδ m,T t , RE ≤ 2 ξ t .

Proof of Lemma G.5. By Lemma I.1, I.2 and I.3, conditional on Good, with probability at least 1 -( Kδ v,T t + Nδ d,T t + KNδ m,T t ) , ∣ ∣ ˆ ∆ t ( x, ˜ x ) -∆( x, ˜ x ) ∣ ∣ ≤ ρ t ( x, ˜ x ) . This finishes the proof of the first result in Lemma G.5.

We then accumulates all the error probabilities. By the choices of δ v,T t = δ 15 KT 3 t , δ d,T t = δ 15 NT 3 t , δ m,T t = δ 15 KNT 3 t ,

Note that there is an reversion step at Line 18 Algorithm 1, for each T ∈ N , there are at most two time steps t 1 < t 2 with T t 1 = T t 2 . Therefore,

This proves the second statement.

J Upper Bound of PS ε BAI : Proof of Theorem 3.2

Theorem 3.2 is proved in this section by three steps.

Firstly, we show that the recommended arm ˆ x ε is an ε -best arm upon the termination of the algorithm in Lemma J.1.

Secondly, we present a sufficient condition for the termination of the algorithm in terms of T t in Lemma G.6.

Lastly, we show that T τ is bounded by a constant fraction of τ and thus, obtain an upper bound on τ .

Step 1:

According to Lemma G.5, CI t holds with probability 1 -( Kδ v,T t + Nδ d,T t + KNδ m,T t ) . Conditional on the good event Good in (G.1) and the event CI t (the mean gaps are well approximated at time step t ), we expect the recommended arm will be an ε -optimal arm. This is formalized in the following lemma.

Lemma J.1. Conditional on Good and CI t , if the algorithm stops at time step t , the recommended arm ˆ x ε = x ∗ t is an ε -best arm.

Proof of Lemma J.1. If ˆ x ε = x ∗ , the lemma holds.

̸

we have

If ˆ x ε = x ∗ , according to the termination condition (3.4)

̸

̸

where the first inequality holds due to CI t . This indicates ∆( x ∗ , ˆ x ε ) ≤ ε and thus the recommended arm ˆ x ε = x ∗ t is an ε -best arm.

Step 2:

Lemma G.6. Conditional on Good and CI , the recommended arm ˆ x ε ∈ X ε and when Algorithm 1 terminates, the order of T t is upper bounded by (3.6) .

̸

Proof of Lemma G.6. By Lemma J.1, ˆ x ε ∈ X ε . Note that for any x = ˆ x ε , when

̸

By utilizing CI t , we have

̸

̸

Therefore, if (J.1) hold for all x = ˆ x ε , the algorithm must have stopped, i.e., it is sufficient to have the following for any x = ˆ x ε :

̸

̸

Lemma J.2 gives an upper bound on the total number of time steps in Exp phases at which (J.1) must hold for any two arms x ε ∈ X ε , x ∈ X \ { x ε } .

̸

Lemma J.2. For any two arms ( x ε , x ) , where x = x ε , x ∗ , when T t = 6400 ln 6400 · d (∆( x ∗ , x ) + ε ) 2 ln Kd/ (∆( x ∗ , x ) + ε ) 2 δ +6400ln6400 · H DE ( x ε , x ) ln N H DE ( x ε , x ) δ + 3200 √ 2 ln 3200 √ 2 · NL max ∆( x ∗ , x ) + ε ln KNL max ∆( x ∗ ,x )+ ε δ (J.2) = ˜ O ( d (∆( x ∗ , x ) + ε ) 2 ln 1 δ +H DE ( x ε , x ) ln 1 δ + NL max ∆( x ∗ , x ) + ε ln 1 δ ) where H DE ( x ε , x ) := L max (∆( x ∗ , x ) + ε ) 2   N ∑ j =1 √ min { 16 p j , 1 4 } | ∆ j ( x ε , x ) + ε |   2 , we must have ρ t ( x ε , x ) ≤ ∆( x ∗ , x ) + ε 2

By taking the maximum of (J.2) over all ε -best arms and the suboptimal arms, we get

̸

̸

Proof of Lemma J.2. Let c v , c d , c m ∈ (0 , 1) be constants, satisfying c v + c d + 2 c m = 1 . By the definition of ρ t ( x ε , x ) in (3.3), it is sufficient to have

where we take ζ t ( x ε , x ) = ε for theoretical simplicity, but we adopt ζ t ( x, ˜ x ) = arg min ζ t ( x, ˜ x ) ∈ R ∑ N j =1 β t,j ( ˆ ∆ t,j ( x, ˜ x ) + ζ t ( x, ˜ x )) 2 = -∑ N j =1 β t,j ˆ ∆ t,j ( x, ˜ x ) ∑ N j =1 β t,j in the algorithm, which still enjoys the theoretical guarantee.

VE Term According to the definition of α t , we need to bound:

By a coarse estimation, it is sufficient to have

where ¯ c v = 6400 c 2 v ln 6400 c 2 v .

DE Term The difficulty lies at the DE term.

We will upper bound 2 ξ t by c m ∆( x ∗ ,x )+ ε 2 first; the second term will be included in the RE term and analyzed later. Therefore, it is sufficient to have

By the definition of β t,j in (I.2) and Lemma K.3, we get

Lemma K.3. If ψ t,j = ˆ p t,j , then

Therefore, in order for (J.4) to hold, it is sufficient to have

where d d

RE Term By the definition of ξ t in (I.2), it is sufficient to have

where ¯ c m = 400 √ 2 c m ln 400 √ 2 c m .

By taking c v = 3 4 , c d = 1 8 and c m = 1 8 , the upper bound on T t can be concluded as

Step 3:

Theorem 3.2. Define the context distribution estimation (DE) hardness parameter

where ¯ H ( x ε , x ) := ( ∑ N j =1 √ min { 16 p j , 1 / 4 }| ∆ j ( x ε , x ) + ε | ) 2 . Under Assumption 1, with probability at least 1 -δ , PS ε BAI identifies an ε -optimal arm and its sample complexity is

̸

Proof of Theorem 3.2. By Assumption 1, and the choice of the parameters, Lemma G.4 indicates Good is guaranteed with probability at least 1 -δ 2 .

Note that some arm pulls are not counted in T t :

In every γ time steps, Algorithm 1 enters the CD phase to select a CD sample.

When a changepoint is detected, Algorithm 1 steps into the CA phase, where w 2 arms are sampled in Algorithm 4.

During the CA phase, we abandon wγ 2 samples.

Therefore, when t is large (or after the first stationary segment)

As γ ≥ 2 and wγ are set to be upper bounded by L min 3 , so the fractions above is upper bounded by an absolute constant 4 , e.g., with γ = 2 and w = L min 6 , the constant is 4 . Therefore, t is of the same order as T t . By Lemma G.6, (3.6) holds with probability 1 -δ .

K Analysis of PS ε BAI : Technical Lemmas

Lemma K.1. Given any j ∈ [ N ] and T t ≥ 2 L max 9 ln 2 δ d,T t , ˜ β t,j ≤ β t,j .

The proof is scheduled in 2 steps.

Step 1 : Upper bound √ 2 p j (1 -p j ) L max T t ln 2 δ d,T t .

When L max 3 T t ln 2 δ d,T t ≤ √ 2 p j (1 -p j ) L max T t ln 2 δ d,T t , we have

As p j (1 -p j ) ≤ 1 4 , ∀ p j ∈ (0 , 1) , this nai¨ ve bound gives ˜ β t,j ≤ 5 2 √ 2 · 1 4 · L max T t ln 2 δ d,T t .

If T t,j > 0 , according to Lemma K.2,

thus we have

If T t,j = 0 , i.e., the latent context j has never been observed and ˆ p t,j = 0 , we have

Take the trivial bound into consideration and observe that ϕ t,j = min { 25 L max T t ln 2 δ d,T t , 1 4 } and that

Thus, (K.1) also holds for T t,j = 0 .

To conclude, we have

Step 2 : Simplify the bound. Recall that

As T t ≥ 2 L max 9 ln 2 δ d,T t , we have

According to the definition of ϕ , if ˆ p t,j ≥ 25 4 L max T t ln 2 δ d,T t , we have

If ˆ p t,j ≤ 25 4 L max T t ln 2 δ d,T t , we have

Thus

This gives us the desired result.

Lemma K.3. If ψ t,j = ˆ p t,j , then

Part I : Prove that P (˚ x ε / ∈ X ε ) ≤ δ .

Let event E 1 = 1 { τ 1 < τ 2 } ∩ 1 { PS ε BAI returns arm x ε when it terminates } . Recall that ˙ x ε denotes the recommended arm of N ε BAI when it terminates and ˚ x ε denotes the recommended arm of

Proof of Lemma K.3. Recall to the definition of ϕ t,j in (I.2) and ψ t,j in (I.8),

We only need to upper bound ˆ p t,j .

By (I.4),

Thus ˆ p t,j = ψ t,j ≤ 4 p j , and ϕ t,j ≤ min { 16 p j , 1 4 } .

L Upper Bound of PS ε BAI + : Proof of Theorem 3.3

Theorem 3.3. The PS ε BAI + algorithm is ( ε, δ ) -PAC and its expected sample complexity is

̸

Proof. We let τ 1 denote the stopping time of PS ε BAI, τ 2 denote the stopping time of N ε BAI, and τ denote the stopping time of PS ε BAI + when an identical sequence of samples and returns are applied to them. By the design of these algorithms, we have

PS ε BAI + when it terminates. The design of algorithms indicates that:

Moreover, with the performance guarantees of N ε BAI and PS ε BAI(shown in Proposition 3.1 and Theorem 3.2 individually), we have

Part II : Derive the expected sample complexity of PS ε BAI + .

Wefirst derive the conditional expectation of the stopping time of PS ε BAI . By the same argument as in (J.6), T t ≤ t ≤ 4 T t for any t . Therefore, it is sufficient to upper bound E [ T τ 1 ] and E [ τ 1 ] can be upper bounded by 4 E [ T τ 1 ] .

According to Lemma G.6 and Lemma G.6, when both events Good and CI occur, PS ε BAI terminates with T τ 1 satisfying T τ 1 ≤ ˜ T , where ˜ T is defined in (J.3) and as below:

̸

Given time step t with T t ≥ 2 ˜ T (thus t ≥ 8 ˜ T ), we will upper bound P [ τ 1 > t ] in the following. We firstly upper bound P [ T τ 1 > T t ] . Note that

If PS ε BAI fails to terminate with T τ 1 ≤ T t 2 , it implies that event Good ⋂(⋂ s ∈I CI s ) fails, where I = { s : T t 2 +1 ≤ T s ≤ T t } . Hence,

where ( a ) is obtained by applying Lemma G.5 to time steps in I ∩ T t (i.e., the Exp time steps in I ) and note that there are at most two time steps t 1 < t 2 with T t 1 = T t 2 due to the reversion step; ( b ) is derived by substituting δ v,T t = δ 15 KT 3 t , δ d,T t = δ 15 NT 3 t , δ m,T t = δ 15 KNT 3 t which are used to define the confidence radius ρ in (3.3). By using the fact that 4 T τ 1 ≥ τ 1 , for any T t ≥ 2 ˜ T , we have

This indicates for t ≥ 8 ˜ T , the probability that PS ε BAI does not stop after t time steps is δ 2 τ ∗ + 48 δ 5( t -4) 2 . Therefore, by Tonelli's Theorem, we have

Next, we bound E τ , the expected sample complexity of PS ε BAI + . E τ can be decomposed as below:

Since τ = min { τ 1 , τ 2 } and P [min { τ 1 , τ 2 } > t ] ≤ P [ τ 1 > t ] for all t , we have

Besides, since PS ε BAI will terminate after τ ∗ time steps, i.e., P ( τ 1 ≤ τ ∗ ) = 1 . We have

Lemma F.1 indicates that P [ τ 2 ≥ t ] ≤ δ ( α -1) C 3 ( t/ 2) 2 for all t ≥ T 0 , where

Since the order of T 0 is apparently larger than τ ∗ , we have T 0 ≤ τ ∗ . Then, with the same method as in the proof of Proposition 3.1, the conditional sample complexity of N ε BAI can be bounded as follows:

Substituting terms in (L.2) with (L.1), (L.3), (L.4) and (L.5), we have

Where (a) is shown in (F.1).

Besides,

Altogether , we have

M Analysis of Lower Bound: Proof of Theorem 4.1

The dynamics for under our PSLB model is displayed in Dynamics 1.

Dynamics 1 Dynamics for piecewise-stationary linear bandits

1: The instance: Λ = ( X , Θ , P θ , C ) 2: while the algorithm does not stop at time step t do 3: if t ∈ C then 4: The environment samples θ ∗ j t ∼ P θ 5: else 6: θ ∗ j t = θ ∗ j t -1 (the environment does not change) 7: end if 8: The agent samples an arm x t based on the history up to time t -1 . 9: The reward Y t,x t = x ⊤ t θ ∗ j t + η t is revealed to the agent. 10: end while 11: Recommend an ε -best arm ˆ x ε .

To derive the lower bound in Theorem 4.1, we investigate two environments different from the one defined in Section 2 (and as in Dynamics 1):

Dynamics 2: the agent observes the index of current context j t , and the environment reduces to contextual linear bandits; the definition of Dynamics 2 and the lower bound under it are detailed in Appendix M.1.

Dynamics 3: the agent observes the changepoints in C and context vector θ ∗ j t 's, and hence she solely needs to estimate the distribution of contexts; the definition of Dynamics 3 and the lower bound under it are detailed in Appendix M.2.

We first derive a lower bound for ( ε, δ ) -BAI algorithms in Dynamics 2, that is, in contextual linear bandits.

Corollary M.1. For any ( ε, δ ) -PAC algorithm π , there exists an instance Λ = ( X , Θ , P θ , C ) with Dynamics 2 such that

is defined in Theorem 4.1. In addition, when |X ε | = 1 ,

̸

This lower bound generalizes the result of [16] to the linear bandit setting.

We next study Dynamics 3. In this setting, the agent solely needs to estimate the distribution of contexts P θ with context samples. Once the agent obtains a good estimate of P θ , she can identify an ε -optimal arm w.h.p. Hence, the lower bound on the complexity of an ( ε, δ ) -BAI algorithm is the product of the minimum length of a stationary segment and the minimum number of context samples/changepoints needed for distribution estimation.

Corollary M.2. For any ( ε, δ ) -PAC algorithm π , there exists an instance Λ = ( X , Θ , P θ , C ) with Dynamics 3 such that E τ ≥ c N C , where c N C is the N th C changepoint in the changepoint sequence C .

Altogether, the sample complexity of an ( ε, δ ) -BAI algorithm in Dynamics 2 and 3 build up the lower bound in Theorem 4.1.

M.1 Lower Bound for ε -BAI in Contextual Linear Bandits

In this section, we consider a sub-problem where the index of the current context j t is revealed at each time step t . The original piecewise-stationary problem becomes a contextual problem whose dynamics is presented in Algorithm 2. As more information is provided to the agent, the lower

Dynamics 2 Dynamics for contextual linear bandits

1: The instance: Λ = ( X , Θ , P θ , C ) 2: while the algorithm does not stop at time step t do 3: if t ∈ C then 4: The environment samples θ ∗ j t ∼ P θ 5: else 6: θ ∗ j t = θ ∗ j t -1 (the environment does not change) 7: end if 8: Reveal j t to the agent. 9: The agent samples an arm x t based on the history up to time t -1 . 10: The reward Y t,x t = x ⊤ t θ ∗ j t + η t is revealed to the agent. 11: end while 12: Recommend an ε -best arm ˆ x ε .

bound for this sub-problem is smaller than the one for the original problem.

Corollary M.1. For any ( ε, δ ) -PAC algorithm π , there exists an instance Λ = ( X , Θ , P θ , C ) with Dynamics 2 such that

is defined in Theorem 4.1. In addition, when |X ε | = 1 ,

̸

For simplicity, we consider the noise model is the Clipped Gaussian Distribution CN (1) , i.e., η ∼ CN (1) or P η = CN (1) .

Definition M.3. A random variable x follows the Clipped Gaussian Distribution with parameter σ , denoted by x ∼ CN ( σ ) , if it has the probability distribution function

where Φ( x ) is the cumulative distribution function of the standard Gaussian distribution.

Some notations are introduced here:

Given an instance Λ = ( X , Θ , P θ , C ) , define the alternative instance Λ ′ = ( X , Θ ′ , P θ ′ , C ) with respect to Λ , where Θ ′ = ( θ ′ 1 , . . . , θ ′ n ) ∈ R d × N and P θ ′ [ θ ′ j ] = P θ [ θ ∗ j ] , s.t. there ∃ x ∈ X \ X ε , ∀ x ε ∈ X ε , s.t., x ⊤ ε E θ ′ ∼ P θ ′ < x ⊤ E θ ′ ∼ P θ ′ -ε . We denote the set containing all the alternative instance (w.r.t. Λ ) as Alt Θ (Λ) .

H t = ( x s , Y s,x s , j s ) t -1 s =1 is the observation history up to but not include time t .

kl( p, q ) = KL(Bern( p ) , Bern( q )) is the KL divergence between two Bernoulli distributions with parameters p and q .

N j,x ( t ) = ∑ t s =1 ✶ { x s = x, j s = j } is the number of times arm in which x is sampled under context j . And N j ( t ) := ∑ x ∈X N j,x ( t ) is the number of times in which context j appears.

Therefore, the probability of the observation history H t +1 = ( x s , Y s,x s , j s ) t s =1 is

Then the log-likelihood between the two instance up to time t , given the observed data, is

where ( a ) utilizes P θ [ θ ∗ j c l ] = P θ ′ [ θ ′ j c l ] , ( b ) makes use of the relationship between the arm and observation and in ( c ) ϑ j c l := θ ∗ j c l -θ ′ j c l . Thus, the expectation of the log-likelihood is (in the following, the expectation is taken under instance Λ and algorithm π )

As the lengths of all the stationary phases are upper bounded, i.e., L l ≤ L max , ∀ l ∈ N , then by Wald's Lemma,

The above also holds for t = τ where τ is a stopping time. According to Lemma 19 in [31],

where F τ = σ ( H τ +1 ). In addition, let E = { the recommended arm ˆ x ε / ∈ X ε } , as the algorithm π is δ -PAC, then

From (M.8), (M.9), (M.10) and (M.11), we conclude that

where

The solution to the above optimization problem is in general intractable, even for the stationary case [32]. We can establish a connection of the above problem to the stationary ε -best identification problem in linear bandits when we assume the change of the latent vectors are the same, i.e., ϑ 1 = . . . = ϑ N .

M.1.1 Connection with the Stationary Case

In the alternative instance Alt Θ (Λ) , ∃ x ∈ X \ X ε , for any arm x ε ∈ X ε , s.t.

Thus,

Alt

Define Alt(Λ) restricted ⊂ Alt Θ (Λ) , with the additional constraint that ϑ 1 = ϑ 2 = · · · = ϑ N . Then we have

Alt(Λ) restricted

Note that in stationary linear bandits, the instance can be characterized by the arm set X ⊂ R d and the latent vector θ ∈ R d . Define the alternative instance in linear bandits [32] for the instance ( X , E θ ∼ P θ θ )

Note that

where in ( a ) we denote ¯ v := ∑ N j =1 p j v j as a mixture of { v j } N j =1 . In other words, the max-min problem becomes the one for the ε -best arm identification problem in stationary linear bandits. According to [32], the solution to the last optimization problem above is in general intractable.

However, the optimization problem can be simplified under some simple cases, e.g., the set of ε -best arm is a singleton [32]. In the next subsection, a lower bound for the original problem with X ε being a singleton will be derived.

M.1.2 A Simple Case: |X ε | = 1

Assume that the set of ε -best arm is a singleton, i.e., X ε = { x ∗ } , we will solve the original optimization problem:

we extend the procedures in [1] to the piecewise-stationary setup.

Lemma M.4.

̸

̸

Proof. Note that Λ ′ differs from Λ in the context matrix Θ . By the derivations in (M.12), there exists arm x = x ∗ , s.t.,

Therefore, the optimization problem becomes

where a > 0 . The Lagrangian function for this problem is

Then

This indicates that

Let a → 0 , we have

̸

Due to (M.13), we only require there exists x = x ∗ , such that the constraint is satisfied, therefore,

̸

By Lemma M.4, the stopping time can be lower bounded as

̸

This lower bound indicates that, (1) a good algorithm should actively detects and makes use of the contextual information to facilitate the arm identification process. (2) our lower bound extends the result of [16] to the linear bandits case.

The above lower bound can be further lower bounded if we restrict v 1 = . . . = v N .

Lemma M.5. Let SPD( d ) := { A : A ∈ R d × d , A > 0 } denote the set of SPD matrices of dimension d × d . Given any x ∈ R d , define the function f : SPD( d ) → R , f ( A ) = x ⊤ A -1 x , then f is convex.

Proof of Lemma M.5. Given any A,B ∈ SPD( d ) , define

It is suffice to prove g is convex.

Therefore, g is convex.

where ¯ v := ∑ N j =1 p j v j ∈ ∆ X and the inequality becomes equality when v 1 = · · · = v n . Thus,

̸

̸

The last term mimics the lower bound in the stationary linear bandits with the latent vector E θ ∼ P θ θ .

̸

In addition, if we let p 1 = 1 and p j = 0 for all j = 1 in the instance, our lower bound can be simplified to the lower bound in stationary linear bandits with latent vector θ ∗ 1 [24]

̸

̸

M.2 Lower Bound on the Number of changepoints

In this section, we consider an even easier problem: all the contextual information is known to the agent, except for the distribution P θ . The dynamics is displayed in Dynamics 3.

Dynamics 3 Dynamics for an easier problem

1: The instance: Λ = ( X , Θ , P θ , C ) 2: while the algorithm does not stop at time step t do 3: if t ∈ C then 4: The agent acknowledges t ∈ C . 5: The environment samples θ ∗ j t ∼ P θ 6: else 7: θ ∗ j t = θ ∗ j t -1 (the environment does not change) 8: end if 9: The agent observes θ ∗ j t 10: end while 11: Recommend an ε -best arm ˆ x ε .

We are going to consider the alternative instances with respect to the distribution on θ , i.e., Λ ′ = ( X , Θ , P ′ θ , C ) . where ∃ x ∈ X \ X ε , ∀ x ε ∈ X ε , s.t., x ⊤ ε E θ ∼ P ′ θ < x ⊤ E θ ∼ P ′ θ -ε . Denote the set of all the alternative instance (w.r.t. Λ ) as Alt P (Λ) . According to the Pinsker's inequality, let E := { the recommended arm ˆ x ε / ∈ X ε } , then for any δ -PAC algorithm π ,

We will give an upper bound on the denominator. Note that

where Λ x is an alternative instance with distribution P x θ s.t. x ⊤ E θ ∼ P x θ θ -ε > x ⊤ ε E θ ∼ P x θ θ, ∀ x ε ∈ X ε

Given any x / ∈ X ε , we denote the shorthand notation q j = P x θ [ θ ∗ j ] . We have

Fix a > 0 which is sufficiently small, we have the following optimization problem:

If such alternative distribution q exists, let L denote the augmented Lagrangian function

or λ x ε = 0 for some x ε ∈ X ε , which indicates some conditions are not satisfied. The equations above give

.

By solving

which is a polynomial with N -1 degree. Thus, an explicit solution for any instance is not applicable. However, there are some cases where we can get an estimate of λ x ε

Lemma M.6. Denote j 0 = arg min j ∈ [ N ] ∆ j ( x ∗ , x ) and j 1 = arg max j ∈ [ N ] ∆ j ( x ∗ , x ) . When |X ε | = 1 , -∆ j 0 ( x ∗ , x ) -2 ε > ∆ j 1 ( x ∗ , x ) > ε and ∑ N j =1 p j (∆ j ( x ∗ , x ) + ε + a ) 3 ≤ 0 , the solution to (M.20) is upper bounded by

Proof of Lemma M.6. When the ε -best arm set is a singleton, (M.20) becomes (where a is sufficiently small)

As (M.19) is a probability distribution, we need 1 + λ x ∗ (∆ j 0 ( x ∗ , x ) + ε + a ) ≥ 0 , which indicates λ x ∗ ≤ -1 ∆ j ( x ∗ ,x )+ ε + a . By the condition on j 0 and j 1 , 0 < λ x ∗ (∆ j ( x ∗ , x ) + ε + a ) < 1 , ∀ j ∈ [ N ] .

Therefore, by using 1 1+ x = 1 -x + x 2 -x 3 + x 4 1+ x for | x | < 1 , we can expand the equation as follows

In general, when we get the { λ x ε } x ε ∈X ε and plug it in (M.19), the alternative distribution q is obtained. Finally, we let a → 0 .

This gives the lower bound for the number of changepoints that need to be observed. A coarse estimation of the KL divergence without solving (M.20) can be done as follows

Note that the solution of λ x ε depends on ∆( x ε , x ) (e.g. Lemma M.6), so the final solution is of order (∆( x ε , x ) + ε ) 2 for a given x / ∈ X . This is the solution to the inside minimization problem in (M.17). The final lower bound will be

The lower bound is

where λ x ε is the solution to (M.20).

With the setup in Lemma M.6, we have

and the lower bound is

̸

̸

Remark M.7. We give some comments on the existence and uniqueness of the solution to (M.20) . Existence:

It is possible that (M.20) does not have a solution. For instance, consider a three-arm instance: x (1) = (1 , 0 . 5) , x (2) = (0 . 5 , 1) , x (3) = (0 . 6 , 0 . 6) , θ ∗ 1 = (1 , 0) , θ ∗ 2 = (0 , 1) , P θ = (0 . 5 , 0 . 5) , ε = 0 . 1 . We have x (1) ⊤ E θ ∼ P θ = x (2) ⊤ E θ ∼ P θ = 0 . 75 and x (3) ⊤ E θ ∼ P θ = 0 . 6 , thus x (3) is not an ε -best arm. Furthermore, there does not exist an alternative distribution q , such that x (3) is the best arm and neither x (1) , x (2) is ε -best. Under such case, the lower bound on E [ l t ] is 0 (we regard min x ∈∅ f ( x ) = + ∞ by convention).

It is possible that (M.20) does not have a solution and it is unnecessary to estimate P θ . For instance, consider a two-arm instance: x (1) = (1 , 0 . 5) , x (2) = (0 . 5 , 0 . 1) , θ ∗ 1 = (1 , 0) , θ ∗ 2 = (0 , 1) , P θ = (0 . 5 , 0 . 5) , ε = 0 . 1 . Arm x (1) is better than arm x (2) under all contexts. Therefore, no matter what P θ is, arm x (1) is the ε -best arm. Under such cases, there may exist an algorithm and it is sufficient for it to determine the best arm if the context vectors are well-approximated. There is no need to estimate P θ .

A necessary condition for the existence of the solution of (M.20) is: there exists x / ∈ X ε , for any x ε ∈ X ε , there ∃ j ( x ε ) ∈ [ N ] , s.t. ∆ j ( x ε ) ( x ε , x ) < -ε . In other words, for each ε -best arm x ε , there is at least one context in which the alternative arm x is better than x ε by at least ε .

A sufficient condition for the existence of the solution of (M.20) is: there exists x / ∈ X ε and j ∈ [ N ] , s.t. ∆ j ( x ε , x ) < -ε , ∀ x ε ∈ X ε . In other words, x is better than any ε -best arm under context j . In the alternative instance, we can lift P ′ θ [ θ ∗ j ] close to 1 so that arm x becomes the ε -best arm and X ′ ε ∩ X ε = ∅ .

Uniqueness:

The uniqueness of the solution is not guaranteed, as the KKT conditions (M.18) is only a necessary condition for the solution. We need to look for the solution that minimizes the KL divergence.

A sufficient condition for the uniqueness of the solution is (if it exists, which indicates ∃ j ∈ [ N ] , s.t. ∆ j ( x ∗ , x ) < -ε ): |X ε | = 1 . Specifically, denote f ( λ x ∗ ) := ∑ N j =1 p j 1+ λ x ∗ (∆ j ( x ∗ ,x )+ ε + a ) , we have

N More Examples and Details

In this section, we firstly provide one more example to illustrate the tightness of our derived upper bound lower bounds, indicating the efficiency of our PS ε BAI + algorithm. In addition, we can observe how the upper and lower bounds are affected by the level of piecewise non-stationarity and whether our PS ε BAI + algorithm can reduce the influence manifested by L max . After that , we present a proof sketch of the results in Corollaries 5.1 and N.1 in Appendix N.1.

Example 2. Instance Λ = ( X , Θ , P θ , C ) is with

d arms: x ( i ) = e i , i ∈ [ d ] .

N = d contexts: θ ∗ 1 = ( a, 0 , 0 , . . . , 0) ⊤ , θ ∗ 2 = ( a, b, b, . . . , b ) ⊤ -b e j , j ≥ 2 , where b > a > ε, b -a > ε

Context distribution: p j = p, j ≥ 2 and p 1 = 1 -( N -1) p , where p ∈ (0 , a -ε ( N -2) b ) .

Under Example 2, we have

̸

Thus, (1) x (1) is the unique ε -best arm; (2) { x ( i ) } i ≥ 2 are equivalent, and ∆ min := ∆( x (1) , x ( i ) ) ; (3) for any i ≥ 2 , x ( i ) can be an ε -optimal arm under some alternative distributions.

Corollary N.1. Firstly, for the instances defined in Example 2, we have

̸

and the sample complexity of the PS ε BAI + is tight up to ( NL max /L min ) and logarithmic factors. We also further observe some specific instances: (i) when p → 0 + , with ∆ min = min x = x ∗ ∆( x ∗ , x ) , we have

(ii) When p → ( a -ε ) ( N -2) b ) -and ( a + ε ) 2 +( b -a -ε ) 2 = Ω(1) , we have H DE = NL max ( ∆( x (1) ,x ( i ) )+ ε ) 2 and

The upper bounds are achieved by the PS ε BAI + algorithm.

We can observe from Corollary N.1 that

In case (i), when p → 0 + , p 1 → 1 -( N -1)( a -ε ) ( N -2) b and p j → ( a -ε ) ( N -2) b ) , and thus the instance tends to be non-stationary and ∆ min → ε . We will obtain

indicating that our algorithm can also reduce the impact of L max .

In case (ii), the upper and lower bounds are with the same order, and the difference is solely manifested by a additive term L max -L min , suggesting that PS ε BAI + is near optimal and again, PS ε BAI + mitigates the impact of L max .

N.1 Analysis of examples

Recall the instance Λ = ( X , Θ , P θ , C ) in Example 1:

Example 1. Instance Λ = ( X , Θ , P θ , C ) is with (i) 2 d -1 arms: x (1) = e 1 , x ( i ) = e i , x ( d + i -1) = e 1 cos ϕ + e i sin ϕ for all i ∈ { 2 , . . . , d } where ϕ ∈ [0 , π/ 4) , (ii) 2 d -2 contexts: θ ∗ j ± = e 1 cos ϕ ± e j +1 sin ϕ for all j ∈ [ d -1] , (iii) Context distribution: p j = 1 /N for all j ∈ [ N ] .

Corollary 5.1. For the instance defined in Example 1, we have H DE ( x ε , x ) = ˜ O ( NL max ) for all ( x ε , x ) ∈ X ε × ( X \ X ε ) . In addition, if ε < (cos ϕ )(1 -cos ϕ ) , we have

where E [ τ ] ∗ is the minimal expected sample complexity over all ( ε, δ ) -PAC algorithms and f : R → R satisfies f ( ϕ ) → 0 as ϕ → 0 + . The upper bound in (5.1) is achieved by PS ε BAI + .

Proof of Corollary 5.1. As µ x (1) = cos ϕ , µ x ( i ) = 0 , µ x ( d + i ) = cos 2 ϕ for i = 2 , . . . , d and ε ∈ (1 -cos ϕ, cos ϕ ) , x (1) is the best arm and x ( i ) , i = 2 , . . . , d are not ε -best arms and x ( d + i ) , i = 2 , . . . , d are ε -best arms. ∆ min = cos ϕ -cos 2 ϕ .

When ε < cos ϕ -cos 2 ϕ = ∆ min , x (1) is the unique ε -best arm. By solving (M.14), we see that there exists a continuous function f ( ϕ ) such that f ( ϕ ) → 0 as ϕ → 0 and we can get the lower bound for the VE term as

As ( x (1) -x ( i ) ) ⊤ θ ∗ j ± > 0 for i = 2 , . . . , d, j ∈ [ d -1] , x ( i ) cannot be an ε -best arm under any alternative distribution, we only need to consider x ( d + i ) , i = 2 , . . . , d , which are equivalent. Given any x ( d + i ) , by solving (M.20), we can upper bound

and thus the number of change points or context samples can be lower bounded as

According to Theorem 3.3, the upper bound is

where

As ∆ min = cos ϕ -cos 2 ϕ = cos ϕ (1 -cos ϕ ) , thus 1 -cos ϕ = ∆ min cos ϕ ≤ √ 2∆ min , and H DE (∆( x (1) , x ( d +1) ) < 144 NL max for any choice of ε .

Hence, the upper bound is

As ϕ → 0 , the first term (the VE term) dominates and it matches the lower bound in (N.1). Therefore, the upper bound is asymptotically tight up to logarithmic terms.

Corollary N.1. Firstly, for the instances defined in Example 2, we have

̸

and the sample complexity of the PS ε BAI + is tight up to ( NL max /L min ) and logarithmic factors. We also further observe some specific instances: (i) when p → 0 + , with ∆ min = min x = x ∗ ∆( x ∗ , x ) , we have

(ii) When p → ( a -ε ) ( N -2) b ) -and ( a + ε ) 2 +( b -a -ε ) 2 = Ω(1) , we have H DE = NL max ( ∆( x (1) ,x ( i ) )+ ε ) 2 and

The upper bounds are achieved by the PS ε BAI + algorithm.

Proof of Corollary N.1. Under Example 2, we have

̸

Thus, (1) x (1) is the unique ε -best arm; (2) { x ( i ) } i ≥ 2 are equivalent, and ∆ min := ∆( x (1) , x ( i ) ) ; (3) for any i ≥ 2 , x ( i ) can be an ε -best arm under some alternative distributions.

For any i ≥ 2 , by solving (M.20) with x ε = x (1) , x = x ( i ) , we obtain λ x (1) = ∆( x (1) ,x ( i ) )+ ε ( a + ε )( b -a -ε ) and the alternative distribution

The lower bound on the expected number of changepoints is

Thus the time complexity is lower bounded by

Furthermore, by solving (M.14), we obtain the lower bound on the expected sample complexity when the context index j t is revealed:

̸

We get the lower bound on the expected sample complexity:

According to Theorem 3.3, the upper bound on the expected sample complexity is

where we utilize

By comparing the lower bound in (N.2) and the upper bound (N.3), we conclude that

the sample complexity of PS ε BAI + is tight up to NL max L min and logarithmic factors.

When the mean gap is small, the sample complexity of PS ε BAI + is dominated by the former term, i.e., the design of PS ε BAI.

When p → 0 + , then p 1 → 1 , p j → 0 for j ≥ 2 , so the instance tends to be stationary and ∆ min → a . The lower bound in (N.2) becomes

and the upper bound in (N.3) turns into

The vector estimation term dominates the sample complexity and our upper bound is tight.

When p → ( a -ε ) ( N -2) b ) -, then p 1 → 1 -( N -1)( a -ε ( N -2) b and p j → ( a -ε ( N -2) b ) , so the instance tends to be non-stationary and ∆ min → ε . The lower bound in (N.2) becomes

and the upper bound in (N.3) turns into

where H DE is upper bounded by (N.4).

(1) If ( ( a + ε ) 2 +( b -a -ε ) 2 ) = O (4 ε 2 + NL max ) , DE is independent of ε and VE increases as ε decreases, thus the expected sample complexity is dominated by vector estimation term when ε is small;

If ( ( a + ε ) 2 + ( b -a -ε ) 2 ) = Ω(1) , the expected sample complexity is dominated by

distribution estimation term.

Under both scenarios, our upper bound is tight.

O Experimental Details

For the computation of the G-optimal allocation, we adopt the Wolfe-Atwood Algorithm with the Kumar-Yildirim start introduced in [33], where the input to the function is the arm set and the output is the G-optimal allocation. All experiments are conducted via MATLAB R2021b on a MacBook Pro with Apple M1 Pro chip and 16 GB memory.

To shorten the execution time,

Before PS ε BAI stops, PS ε BAI and N ε BAI are conducted in parallel (i.e., we run PS ε BAI + ). After PS ε BAI stops, N ε BAI continues and is run in a batch manner, i.e., we sample L min samples according to the G-optimal allocation one time and update the statistics. As the sample complexity of N ε BAI is of order at least 10 7 and each stationary segment is of order 10 4 , the effect of this batch sampling procedure can be largely ignored.

D ε BAI and D ε BAI β are both conducted in segments, because the latent context vector can be observed by the two algorithms and the latent vector does not change within a stationary segment.

As the experiment in Section 6 illustrates that PS ε BAI outperforms N ε BAI and dominates the PS ε BAI + algorithm, we run PS ε BAI instead of PS ε BAI + for the addition experiments in Section O.2 and Section O.3.

To increase the robustness of the algorithm, the window size for LCD is doubled. Note that this will only influence an absolute constant in the sample complexity of the proposed algorithm and the order of the sample complexity remains.

O.1 Modification of Confidence Radii in PS ε BAI and PS ε BAI +

During the proof of the upper bound, we have relaxed the absolute constants in the confidence radii to simplify the proof and increase the readability. In the experiments, we utilize the tighter confidence radii to gain better empirical performance. Note that when these tighter confidence radii are utilized by our algorithm, it still enjoys the current theoretical guarantee. The choice of confidence radii are as follows:

α t : according to Lemma I.1 and Lemma E.1, α can be tightened to be

β t,j : according to (I.3) and Lemma K.1, β t,j can be tightened to be

ξ t : instead of using 25 √ 2 NL max T t ln 2 δ m,Tt , we turn to bound the residual error by (I.9):

1

2

3

4

5

6

7

8

9

10

11

12

10

6

10

8

10

10

where

˜ ξ alg t,j is obtained in a similar manner as α alg t . ξ alg t characterizes the confidence radii of each context at a finer level.

These finer confidence bounds can save a constant of 2 . 5 when t is large.

O.2 Misspecified L max and L min

As PS ε BAI + requires the knowledge of L max , we empirically test the robustness of the algorithm towards L max on the instances in section 6. We run PS ε BAI with ˜ L min = νL min and ˜ L max = νL max where ν = 0 . 8 or 1 . 2 . An ε -best arm is recommended in all experiments. The sample complexities are presented in Figure 3. The result indicates that PS ε BAI (thus PS ε BAI + ) is robust towards the knowledge of L max and its superiority over N ε BAI is maintained.

O.3 Robustness towards w and b

According to the distinguishability condition (Assumption 1), point (2) indicates we can set w = L min 3 γ , where γ is the change detection frequency, and (3.5) indicates w and b are coupled. We denote ˜ w := L min 18 .

To exam the robustness towards w and b , we choose to vary the choice of γ , thus, w and b will change accordingly. Specifically, we select γ ∈ { 2 , 3 , 6 , 12 } and the corresponding w ∈ { 3 ˜ w, 2 ˜ w, ˜ w, ˜ w 2 } . The other parameters remain unchanged. The experiment result is presented in Figure 4. When γ = 18 , w = ˜ w 3 , Assumption 1 is severely violated and the result is not desirable.

The result indicates that, while smaller γ and greater w can result in slightly greater sample complexity, the overall sample complexity does not vary much and the superiority of our algorithm over the naive uniform sampling algorithm N ε BAI is maintained. We conclude that our algorithm is robust against these choices, as long as Assumption 1 is not severely violated.

5

6

7

8

9

10

11

12

10

5

10

6

10

7

10

8

10

9

10

10

O.4 Benchmarks: D ε BAI and D ε BAI β

We present the Distribution ε -Best Arm Identification (D ε BAI) in Algorithm 8 and its variant D ε BAI β in Algorithm 9. According to Dynamics 3, the agent has access to the context vector θ ∗ j t and the changepoint t ∈ C . Thus only the distribution P θ remains unknown and the agent needs to estimate it via the observed contexts.

Algorithm 8 DISTRIBUTION ε -BEST ARM IDENTIFICATION (D ε BAI)

1: Input: the arm set X , the latent vector matrix Θ , the slackness parameter ε , the confidence parameter δ .

2: Initialize : Compute the G-optimal allocation λ ∗ .

4: Observe θ ∗ j 1 .

3: Compute C 3 = ∑ ∞ n =1 n -3 .

5: Compute

̸

6: while min x =¨ x t (¨ x t -x ) ⊤ Θ¨ p t -¨ ρ t < -ε or t / ∈ C do

10: Recommend arm ¨ x ε = ¨ x t .

7: Observe θ ∗ j t and ✶ { t ∈ C} update t = t +1 . 8: Update ¨ x t and ¨ ρ t with (O.1) and update l t if t ∈ C . 9: end while

The design is straightforward except for ζ (¨ x t , x ) . It minimizes the summation ζ (¨ x t , x ) := arg min y ∈ R ∑ N j =1 | ∆ j (¨ x t , x ) + ζ (¨ x t , x ) | . This trick has also been utilized in the design of PS ε BAI (see (I.5)). It helps to better exploit the structure of the latent vectors Θ and facilitates the identification process. For easy implementation, we choose a proxy ζ (¨ x t , x ) = arg min y ∈ R ∑ N j =1 (∆ j (¨ x t , x ) +

Algorithm 9 DISTRIBUTION ε -BEST ARM IDENTIFICATIONβ (D ε BAI β )

1: Input: the arm set X , the latent vector matrix Θ , the slackness parameter ε , the confidence parameter δ .

2: Initialize : Compute the G-optimal allocation λ ∗ .

4: Observe θ ∗ j 1 .

3: Compute C 3 = ∑ ∞ n =1 n -3 .

5: Compute

̸

7: Observe θ ∗ j t and { t ∈ C} and let t = t +1 .

6: while min x = ◦ x t ( ◦ x t -x ) ⊤ Θ ◦ p t -◦ ρ t < -ε do

✶ 8: Update ◦ x t and ◦ ρ t with (O.2). 9: end while

10: Recommend arm ◦ x ε = ◦ x t .

D ε BAI β utilizes the techniques we have used to bound the deviation between the true distribution p j and the estimated ones ◦ p t,j for all j ∈ [ N ] . Similarly, we let ζ ( ◦ x, x ) =

As the theoretical guarantee of D ε BAI β can be derived following a similar manner as the proof of the DE term of PS ε BAI in Appendix I.2 and in the proof of Lemma G.6, for simplicity, we just present the result and omit the proof here here.

Theorem O.1. D ε BAI β identifies an ε -best arm within

̸

time steps with probability at least 1 -δ and in expectation, where

We present a theorem, along with a proof sketch, for the theoretical guarantee of D ε BAI below.

Theorem O.2. D ε BAI identifies an ε -best arm within

̸

time steps with probability at least 1 -δ and in expectation, where ¨ H ( x ε , x ) := min ζ (¨ x ε ,x ) ∈ R ∑ N j =1 | ∆ j (¨ x ε , x ) + ζ (¨ x ε , x ) | .

Proof. For the sake of conciseness and simplicity, only a proof sketch is provided for this Theorem. Similar results can be found in the referred contents. The proof is composed by 4 steps.

Step 1 : bound the deviation of P θ [ θ ∗ j ] by Lemma O.3 and Remark O.4.

Step 2 : prove the recommended arm is an ε -best arm. Conditional on Step 1 , we can prove this following the procedures in Lemma J.1.

Step 3 : give a sufficient condition and then obtain a high probability upper bound. This can be seen from (J.1) and by solving

̸

The high probability result is obtained by maximizing the above over x ε ∈ X , x = x ε , x ∗ .

Step 4 : the expected result can be derived in a similar method as in the proof of N ε BAI in Appendix F. The expected sample complexity and the high-probability sample complexity is of the same order.

Lemma O.3. With probability at least 1 -δ , sup j ∈ [ N ] | p j -¨ p t,j | ≤ ¨ β t for all t ∈ N , where ¨ β t = √ 2 l t ln 2 δ l t and δ l t = δ C 3 l 3 t .

Proof. We may define the cumulative distribution function (CDF) and the empirical CDF for P θ as F θ ( j ) = ∑ j k =1 P θ [ θ ∗ j ] and ¨ F t ( j ) = ∑ l t l s =1 ✶ { c l s = c j , l s ≤ j } respectively for j ∈ [ N ] . According to the DKW inequality, we have

By the implication

we have that

This is equivalent to

A union bound gives an upper bound of the failure probability ∑ ∞ l t =1 δ l t = δ .

Remark O.4. The use of the DKW inequality gives a union bound over the deviation of the distribution estimation all contexts. This avoid the N factor in the logarithm in ¨ β t and is beneficial when the number of contexts N is large.

When there are a few contexts, it is also possible to directly bound the deviation for each context via Azuma-Hoeffding's inequality, i.e.,

where ¨ β t := √ 1 2 l t ln 2 C 3 Nl 3 t δ . A union bound gives that with probability at least 1 -δ , | p j -¨ p t,j | ≤ ¨ β t for all t ∈ N , j ∈ [ N ] . As this new confidence radius is the same as the one in Lemma O.3 up to constant and logarithmic terms, the sample complexity should be of the same order.

We adopt this confidence radius ¨ β t := √ 1 2 l t ln 2 C 3 Nl 3 t δ in the experiment.

P Additional Discussions

In this section, we provide additional discussions on

the related methods for BAI in the nonstationary bandits literature in subsection P.1,

the upper on the sample complexity in Theorem 3.2 and Theorem 3.3 in subsection P.2,

the connection between the piecewise-stationary linear bandits model to the stationary linear bandits model in subsection P.3,

the special case where the number of context N = 1 in subsection P.4,

P.1 Discussion on the Related Methods in Nonstationary Bandits

To the best of our knowledge, there is limited literature investigating the BAI in the nonstationary bandits setup (there is comparatively a much richer literature for regret minimization in non-stationary environments): [14] studies BAI in the fixed-horizon setup and [15] assumes the best arm remains unchanged after certain time step and explores the fixed-confidence setting. Both of the works are not directly comparable to our proposed piecewise-stationary setup.

Additionally, we provide strong baselines algorithms DεBAI and DεBAI β in Section 6, which are detailed in Section O.4. These two baselines are given oracle information about the context (see Dynamics 3), while PS ε BAI + is not. As indicated by Figure 2(b), PS ε BAI + is competitive compared to these strong baselines and is much better than the naive approach.

P.2 More Discussions on the Upper Bound

Firstly, we emphasize that the sample complexity in Theorem 3.2 and Theorem 3.3 are instancedependent, in particular, the term H DE characterizes the difficulty in estimating the distribution of the context. Furthermore, the presence of the gaps ∆( x, x ∗ ) also underscores that our upper bounds are functions of the instance.

Secondly, our algorithm adopts the G-optimal design, which is minimax optimal for the BAI in standard linear bandits problem [1, 2]. Although we have not proved it, we believe that the sample complexity of the proposed algorithm may not be instance-dependent optimal.

Lastly, we provide some remarks on the instance-dependent optimal algorithms:

(1) The G-optimal design is the cornerstone for the more efficient and adaptive rules like XY -allocation and XY -Adaptive Allocation in [1]. Therefore, our algorithm provides a framework for more sophisticated algorithms in the piecewise-stationary linear bandits problem with the BAI task. Empirically, one can attempt to replace the G-optimal design by the XY -allocation during implementation, which can possibly yield empirical benefits.

(2) The current lower bound is established with two simpler problems (see Section 4), which is sufficient to show our algorithm is minimax optimal. However, a tighter instance-dependent lower bound based on the original problem may be required if one wishes to show an algorithm is instancedependent optimal. This can be challenging because the distribution of the arms, the contexts and the changepoints are unknown, and the characterization of the "alternative instance" (i.e., the ε -best arm set is changed) is difficult.

To conclude, we believe an instance-dependent optimal algorithm is appealing and can lead to future research.

P.3 Connections to the Stationary Bandits

Firstly , we would like to clarify that, in the piecewise-stationary linear bandits model, the instance tends towards a stationary one when max j ∈ [ N ] p j → 1 , instead of the scenario in which L max →∞ .

When L min and L max are large, estimating each latent vector becomes easier since there are more observations in a single stationary segment. However, the overall reward of an arm also depends on P θ , the distribution of latent context vectors, which is also assumed to be unknown in the setup. In order to estimate P θ , a learning agent can get one (unobservable) sample from P θ only at the (unobservable) change points. In other words, L max charaterizes the 'sparsity' of the context samples: the larger L max is, the sparser the context samples are. A larger L max indicates that samples from P θ are likely to be generated less frequently, which will result in the increase of the overall sample complexity. Consider an extreme example of Dynamics 3 where the context vector is revealed at every time step and L min and L max are very large. If at least l τ context samples from P θ are required to identify the best arm, then a sample complexity of l τ L min is unavoidable in the our setup under this extreme case.

Secondly , while max j ∈ [ N ] p j is important in characterizing the (non)stationarity of the instance, we would like to emphasize that both the distribution P θ and latent context vectors are essential in characterizing the sample complexity, as shown in the term ¯ H ( x ϵ , x ) in Theorem 3.2. To further illustrate this, we provide a simple but illuminating instance as follows: The instance is composed by N = 3 contexts and K = 2 arms, and ∆ j denotes the mean gap between arm 1 and arm 2 under context j ∈ [ N ] (to be specified below) and ∆ := ∑ j ∈ [ N ] p j ∆ j denotes the weighted mean gap between arm 1 and arm 2 . The probability of context 1 is p 1 = max j ∈ [ N ] p j = 1 -p 2 -p 3 , and p 1 is close to 1 . Let ˆ x denote the empirical version of the statistics x in the following arguments.

In such a case, consider this question: if an algorithm has only detected context 1 in the first t time steps and t is large, should it stop or not?

As no change point has occurred, the current dynamics behaves similarly to that of a stationary bandit environment up till now. Thus the empirical probabilities of the 3 contexts are ˆ p 1 = 1 , ˆ p 2 = ˆ p 3 = 0 and the empirical mean gap under context 1 ˆ ∆ 1 is close to ∆ 1 . Consider the following two cases:

Case 1: ∆ 1 = p 2 and ∆ 2 = ∆ 3 = -p 1 . The true weighted mean gap between arm 1 and arm 2 is ∆ = ∑ j ∈ [ N ] p j ∆ j = -p 1 p 3 < 0 , and thus arm 2 is the best arm. This indicates that, in order to estimate ∆ 2 and ∆ 3 , an algorithm needs to observe contexts 2 and 3 despite their small probabilities. But currently ˆ p 2 = ˆ p 3 = 0 , ˆ ∆ = ˆ ∆ 1 ≈ ∆ 1 > 0 , so the algorithm should not terminate.

Case 2: ∆ 1 = 1 and ∆ 2 , ∆ 3 ∈ [ -2 , 2] . Thus ˆ ∆ ≈ ∆ ≥ p 1 -2 p 2 -2 p 3 > 0 . This indicates there is no need to observe contexts 2 and 3 , because arm 1 is the best even in the worst case ( ∆ 2 = ∆ 3 = -2 ). In this case, as long as the algorithm gets a good estimate of ∆ 1 , it can confidently terminate.

As these two cases indicate, in addition to P θ , the latent vectors (which determine the means of the arms) are equally important for the stopping time. Our algorithm takes both factors into account, as reflected by the summation term in equation (3.4) and ˆ ∆ t ( x ∗ t , x ) in (3.5) for the algorithm design, and by ¯ H ( x ϵ , x ) for the theoretical upper bound.

We conclude that

The unknown distribution P θ and the latent vectors jointly determine the sample complexity. ¯ H ( x ϵ , x ) in Theorem 3.2 characterizes their roles, where the contexts with larger probabilities have commensurately greater influence on the sample complexity of the algorithm.

̸

When max j ∈ [ N ] p j → 1 with other parameters fixed, the piecewise-stationary linear bandit instance reduces to a stationary one. H DE ( x ϵ , x ) in Theorem 3.2 becomes L max / 4 , which implies that a constant number of change points need to be observed, and thus T D ( x ϵ , x ) can be regarded as a constant. The dominant term in our upper bound, the T V ( x ) term, recovers the upper bound in the stationary bandits, that is, max x = x ∗ d (∆( x ∗ ,x )+ ε ) 2 ln(1 /δ ) [32].

P.4 Special Case: N = 1

We only consider N > 1 as the bandit model is only truely piecewise-stationary when N > 1 . Thus, we did not specifically derive upper and lower bounds for the extreme case where N = 1 . Nevertheless, our analysis can cover this case due to the following reasons:

The (minimax) lower bound in Theorem 4.1 is not directly applicable since it is not derived for this extreme instance. However, if we step back to the analysis equation (M.16), the feasible set in the optimization problem is empty, and thus the lower bound on E [ l t ] ( N C in Theorem4.1) is 0 . In this case, the lower bound in Theorem4.1 reduces to the lower bound in the stationary bandits.

Regarding the upper bound in Theorem 3.2, when N = 1 , we have p 1 = 1 and H DE = L max 4 . In this case, as H DE does not depend on the mean gaps and the mean gaps are among the fundamental quantities to characterize the difficulty of an instance, the T D term can be regarded as a constant. Therefore, the dominant term in the upper bound is T V . This recovers the bound in the stationary setup.

In addition, as we assume N is an input to our algorithm, when we are aware N = 1 , we can actually adopt the algorithms for the stationary setting, e.g., the G -allocation or XY -allocation rule [1].

Q ε -Best Arm Tuple Identification Problem

In the main paper, we consider the identification of an ε -best arm in terms of the 'ensemble' quality µ x := E θ ∼ P θ [ x ⊤ θ ] = ∑ N j =1 P θ [ θ ∗ j ] x ⊤ θ ∗ j . The curious reader may wonder whether we can identify the ε -best arm tuple X tuple ε := ( x ε 1 , . . . , x ε N ) , where x ε j is an ε -best arm under context j , i.e., x ε j ∈ { x : x ⊤ θ ∗ j ≥ max x ∈X x ⊤ θ ∗ j -ε } . Given the tools we developed in this manuscript, we answer this problem in the affirmative.

̸

Intuitions: Let x ∗ j := arg max x ∈X x ⊤ θ ∗ j and ∆ ∗ j := min x = x ∗ j ∆ j ( x ∗ j , x ) . We expect to have an upper bound taking the form of

where O ( d ( ∆ ∗ j + ε ) 2 L min ln 1 δ ) is an upper bound on the number of context samples j and context j will occur once among every 1 p j contexts in expectation.

Analysis of the problem: The change detection and context alignment subroutines are only effective within τ ∗ time steps (Line 2 in Algorithm 1). However, if a context is with small occurrence probability p j , it may not appear before τ ∗ time steps.

Regarding this scenario, we only expect to obtain a high-probability upper bound on the sample complexity, whereas the expected sample complexity requires more techniques beyond our parallel execution trick (which is used to design PS ε BAI + ) and is an interesting direction left for future research.

Besides, it is more feasible to consider the identification of ε -best arms under contexts with high occurrence probability, i.e., we aim to identify

where ¯ p is a threshold on the occurrence probability of contexts. Let Θ ¯ p = { θ ∗ j : p j > ¯ p } denote those context with high occurrence probability.

Goal: We aim to devise an algorithm with the minimum sample complexity (arm pulls) to ascertain either (1) an ε -best arm under context j , or (2) θ ∗ j / ∈ Θ ¯ p .

With the above goal, we expect to identify an ε -best arm for all j with θ ∗ j ∈ Θ ¯ p ; for those j with θ ∗ j / ∈ Θ ¯ p , either an ε -best arm is identified or θ ∗ j / ∈ Θ ¯ p is ascertained.

We propose Algorithm 10 for this problem, which is quite similar to Algorithm 1, except for a few changes:

Algorithm 10 PIECEWISE-STATIONARY ε -BEST ARM TUPLE IDENTIFICATION

1: Input: arm set X , size of the set of latent vectors N , bounds on the segment lengths L min and L max , slackness parameter ε , confidence parameter δ , sampling parameter γ and window size w , threshold b , probability threshold ¯ p .

2: Initialize : Compute the G-optimal allocation λ ∗ and τ ∗ = 38400 ln(80) NL max ε 2 ln N 2 KL max δε 2 and Flag = [ N ] and Hold = [ ] and Output = [ ] .

3: Set CD sample = [ ] , CA id = { } . Set t CD = + ∞ .

5: Sample w 2 arms { x s } w 2 s =1 ∼ λ ∗ and observe the associated returns { Y s,x s } w 2 s =1 , t = w 2 , t CA = w 2 .

4: Set T t,j = ∅ and initialize T t , T t,j , T t with (3.1) for all t ≤ τ ∗ , j ∈ [ N ] .

w

̸

On Line 7 , the algorithm stops when an ε -best arm or p j ≤ ¯ p is identified for all contexts.

On Line 25 , it computes the confidence radii for the arms in context j t as well as the confidence radii for the occurrence probabilities

̸

̸

̸

On Line 26 to 40 , the stopping rule is changed:

Stopping condition (I) on Line 26

̸

where x ∗ t,j := arg max x ∈X x ⊤ ˆ θ t,j .

Lines 27 to 35 : identify an ε -best arm or ascertain p j ≤ ¯ p among the remaining contexts, and record these observations in Hold for easy access in stopping condition (II) .

Lines 36 to 40 : the recommended ε -best arm is recorded, where we adopt NAN to flag those contexts with small occurrence probabilities.

Theorem Q.1. Given an instance Λ , with probability at least 1 -δ , Algorithm 10 can recommend an ε -best arm for all context j ∈ Θ ¯ p with sample complexity

which can be simplified as

Proof of Theorem Q.1. We provide a concise proof sketch for this theorem.

By adapting the stopping rule for best arm identification in stationary linear bandits to ε -best arm identification, we observe that the number of arm pulls needed for ε -best arm identification under context j is

The remaining problem is to determine how many context samples/changepoints are needed such that the above number of arm samples can be achieved.

Recall that ˆ p t,j = T t,j T t (3.2) and Lemma I.2, we have

In addition, we have an upper bound on β t,j as in (J.5), so there would be sufficient arm samples for ε -best arm identification under context j if

By solving this inequality in terms of T t , we obtain

As we aim to identify the ε -best arms under each context θ ∗ j ∈ Θ ¯ p , the upper bound is at least

even if the set Θ ¯ p is given.

Similarly, for those contexts with small occurrence probability θ ∗ j / ∈ Θ ¯ p , by Lemma I.2 and (J.5), we can identify them when

Careful readers may notice that the denominator depends on a 'probability gap', which can be very small. In practice, if we can actually identify the ε -best arm in those contexts, it is also acceptable. Therefore, we instead only choose not to identify the ε -best arm in those contexts θ ∗ j ∈ Θ ¯ p/ 2 , which yields an upper bound

In this case, the bound is at most ˜ O ( L max ln(1 /δ ) ¯ p ) . For the rest contexts θ ∗ j / ∈ Θ ¯ p/ 2 \ Θ ¯ p , either identifying the ε -best arm or ascertaining its small occurrence probability suffices, that is,

The above bound is upper bounded by ˜ O ( max { L max , d ε 2 } · ln(1 /δ ) ¯ p ) .

By taking the maximum of (Q.3), (Q.4) and (Q.5), we can obtain a high-probability problemdependent upper bound on the sample complexity. In addition, we can get a high-probability problem-independent upper bound

By setting the threshold ¯ p carefully, the algorithm is guaranteed to terminate before τ ∗ given the good event Good (G.1).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Empirical experiments are conducted to show that Assumption 1 is needed for theoretical proof and the proposed algorithm is robust to mild violations of the assumptions. The limitations are discussed in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The problem setting is clearly introduced in Section 2 and all theoretical results are accompanied with proofs or proof outlines.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The procedures of the algorithms are clearly presented. All experiment details are provided in Section 6 and Appendix O.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is released. Please refer to Section 6.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 6 and Appendix O.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars (standard deviations) are included in all experiment results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Appendix O.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work is mainly theoretical, thus there is no identifiable societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is

being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The work is mainly theoretical.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The MATLAB code for the computation of G-optimal allocation used existing code, and has been mentioned and cited in Appendix O.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nhe paper introduces a novel algorithm, PSεBAI+, designed for ε-Best Arm Identification in Piecewise Stationary Linear Bandits (PSLB). The algorithm addresses the challenge of identifying an arm with an average return close to the optimal under varying contexts and unknown changepoints. It incorporates change detection and context alignment mechanisms to adapt to the non-stationary nature of the environment efficiently. The paper provides theoretical guarantees for the algorithm's performance and supports these claims with empirical results.\n\nSTRENGTHS:\n1. The paper proposes a new algorithm that combines change detection and context alignment in a PSLB setting. The algorithm is optimized to identify ε-optimal arms with a minimal number of samples, which is crucial for practical applications.\n2. It offers a thorough theoretical analysis, including a proof of the algorithm's high-probability performance and sample complexity bounds.\n3.The paper backs up the theoretical findings with empirical experiments, demonstrating the algorithm's efficiency and robustness.\n\nWEAKNESSES:\n1. The paper does not provide a comparison with the current state-of-the-art methods, which is essential for establishing the novelty and superiority of the proposed approach.\n\n2. Parameter Sensitivity Analysis:\n- There is no ablation study on the parameters w and b, which, based on past experiences replicating reference [10], could have a significant impact on the algorithm's performance.\n\nQUESTIONS:\n1. About the context alignment:\n- When a new distribution is detected, how does the algorithm decide when to perform context alignment?\n- What methods are used to judge the similarity between the new context and existing ones? The concept of context alignment is common, but the paper lacks a detailed description of the design specifics.\n- How does the algorithm confirm the reliability of the estimated new context feature vectors?\n\n2. Could the authors elaborate on the specific application of the G-optimal design in the algorithm?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper introduces a piecewise stationary linear bandit (PSLB) model and presents the PSεBAI+ algorithm for identifying the best arm with minimal sample complexity. The PSLB model accounts for environments where contexts change at unknown points and are drawn from an unknown distribution. The PSεBAI+ algorithm, combining PSεBAI and NεBAI subroutines, aims to detect changepoints and align contexts. Theoretical analysis indicates that PSεBAI+ achieves an expected sample complexity close to the optimal. The paper also includes numerical experiments comparing PSεBAI+ with baseline algorithms.

STRENGTHS:
- Introduces a new framework for best arm identification in piecewise stationary environments.
- The proposed PSεBAI+ algorithm combines changepoint detection and context alignment.
- Provides proofs that the expected sample complexity of PSεBAI+ is nearly optimal.
- Includes numerical experiments showing the algorithm's performance relative to baseline methods.

WEAKNESSES:
- The justification for using a piecewise stationary model could be more robust, as real-world scenarios may involve more complex conditions.
- The discussion on computational complexity is limited. The authors should discuss the computation required for each major step of the proposed algorithm.

QUESTIONS:
- Is $\theta_j^*$ sampled IID from $P_{\theta}$ each time? If so, it should be clearly stated "IID"

LIMITATIONS:
Discussion on limitations can be improved, with designated section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new model called PSLB, where the underlying parameter $\\theta$ in the common linear bandit model is sampled from an unknown distribution at fixed but unknown time steps. The authors propose a naive algorithm and a sophisticated algorithm for this problem, along with their sample complexity guarantees for best arm identification or $\\epsilon$-best arm identification. A lower bound is also proposed, with a specific instance demonstrating its near-minimax optimality.\n\nSTRENGTHS:\nThe new model makes a significant step towards making linear bandits more general, in the sense that it can model non-stationary environments, which is often true for many real-world applications. The analysis is very comprehensive from many perspectives. For example, it provides a naive algorithm whose sample complexity serves as a good comparison with the later, more complicated algorithm. The sample complexity of the more sophisticated algorithm shows an interesting dependency on the environment parameter $L_{max}$.\n\nWEAKNESSES:\nTypo: dynamics --> algorithm?\n\nMotivating Example: I am not sure if investment is a reasonable example to motivate the model, as the model assumes fixed changing points, which clearly goes against the non-stationary market dynamics.\n\nPresentations: Perhaps the biggest concern of mine is the presentation. The algorithm is overly tedious, and I am still not fully clear on it even after reading it multiple times.\n\nIt is not clear to me what $\\tilde{O}$ hides or if it is intended to hide the same things throughout the paper.\n\nQUESTIONS:\n1. In most of the linear bandit best arm identification literature, for example, references 1-5, the sample complexities end up being instance-dependent quantities, which have been shown to be optimal in many cases. Is it fair to say that the sample complexity of this work is loose in the sense that it does not reveal much instance dependency? \n\n2. It seems very critical to assume bounded noise and normalized rewards. What would happen if we assume general sub-Gaussian noise? I am considering a case where for one of the contexts, $\\theta$ can result in a very high reward for certain arms, but the context has a very low probability of occurring. How would the algorithm behave in this scenario?\n\nLIMITATIONS:\nThe work does not have any limitations due to negative societal impact. The main limitation is clarity, which has been mentioned in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}