Title: Change point detection and inference in multivariate non-parametric models under mixing conditions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multivariate; Nonparametric; Change point inference; short range dependence; Long-run variance; Confidence interval.
Abstract: This paper addresses the problem of localizing and inferring multiple change points, in non-parametric multivariate time series settings. Specifically, we consider a multivariate time series with potentially short-range dependence, whose underlying distributions have Hölder smooth densities and can change over time in a piecewise-constant manner. The change points, which correspond to the times when the distribution changes, are unknown.
 We present the limiting distributions of the change point estimators under the scenarios where the minimal jump size vanishes or remains constant.  Such results have not been revealed in the literature in non-parametric change point settings. As byproducts, we develop a sharp estimator that can accurately localize the change points in multivariate non-parametric time series, and a consistent block-type long-run variance estimator.  Numerical studies are provided to complement our theoretical findings.

Change point detection and inference in multivariate non-parametric models under mixing conditions

Abstract

This paper addresses the problem of localizing and inferring multiple change points, in non-parametric multivariate time series settings. Specifically, we consider a multivariate time series with potentially short-range dependence, whose underlying distributions have Hölder smooth densities and can change over time in a piecewiseconstant manner. The change points, which correspond to the times when the distribution changes, are unknown. We present the limiting distributions of the change point estimators under the scenarios where the minimal jump size vanishes or remains constant. Such results have not been revealed in the literature in nonparametric change point settings. As byproducts, we develop a sharp estimator that can accurately localize the change points in multivariate non-parametric time series, and a consistent block-type long-run variance estimator. Numerical studies are provided to complement our theoretical findings.

1 Introduction

Given a time series { X t } T t =1 ⊂ R p , which is assumed to be an α -mixing sequence of random vectors with unknown marginal distributions { P t } T t =1 . To incorporate the nonstationarity of { X t } T t =1 , we assume that there exists K ∈ N change points, namely { η k } K k =1 ⊂ { 2 , ..., T } with 1 = η 0 < η 1 < . . . < η k ≤ T < η K +1 = T +1 , such that

̸

Our primary interest is to develop accurate estimators of { η k } K k =1 and study their limiting properties. We refer to Assumption 1 for detailed technical conditions of our non-parametric change point model.

Nonstationary multivariate data are frequently encountered in real-world applications, including biology (e.g. Molenaar et al. 2009, Wolkovich & Donahue 2021), epidemiology (e.g. Azhar et al. 2021, Nguyen et al. 2021), social science (e.g. Kunitomo & Sato 2021, Cai et al. 2022), climatology (e.g. Corbella & Stretch 2012, Heo & Manuel 2022), finance (e.g. Herzel et al. 2002, Schmitt et al. 2013), neuroscience (e.g. Gorrostieta et al. 2019, Frolov et al. 2020), among others.

Daren Wang

Yi Yu

Due to the importance of modeling nonstationary data in various scientific fields, we have witnessed a soaring growth of statistical change point literature, (e.g. Aue et al. 2009, Fryzlewicz 2014, Cho & Fryzlewicz 2015, Cho 2016, Wang et al. 2020, Padilla et al. 2022). However, there are a few limitations in the existing works on multivariate non-parametric settings. Firstly, to the best of our knowledge, temporal dependence, which commonly appears in time series, has not been considered. Secondly, there is no localization consistency result for data with the underlying densities being Hölder smooth with arbitrary degree of smoothness. Lastly and most importantly, the limiting distributions of change point estimators and the asymptotic inference for change points have not been well studied.

Taking into account the aforestated limitations, this paper examines change point problems in a fully non-parametric time series framework, wherein the underlying distributions are only assumed to have Hölder smooth continuous densities and can change over time in a piecewise constant manner. The rest of the paper is organized as follows. In Section 2, we explain the model assumptions for multivariate time series with change points in a non-parametric setting. Section 3 details the two-step change point estimation procedure, as well as the estimators at each step. Theoretical results, including the consistency of the preliminary estimator and the limiting distribution of the final estimator, are presented in Section 4. Section 5 evaluates the practical performance of the proposed procedure via various simulations and a real data analysis. Finally, Section 6 concludes with a discussion.

Notation. For any function f : R p → R and for 1 ≤ q < ∞ , define ∥ f ∥ L q = ( ∫ R p | f ( x ) | q dx ) 1 /q and for q = ∞ , define ∥ f ∥ L ∞ = sup x ∈ R p | f ( x ) | . Define L q = { f : R p → R , ∥ f ∥ q < ∞} . Moreover, for q = 2 , define ⟨ f, g ⟩ L 2 = ∫ R p f ( x ) g ( x ) dx where f, g : R p → R . For any vector s = ( s 1 , . . . , s p ) ⊤ ∈ N p , define | s | = ∑ p i =1 s i , s ! = s 1 ! · · · s p ! and the associated partial differential operator D s = ∂ | s | ∂x s 1 1 ··· ∂x sp p . For α > 0 , denote ⌊ α ⌋ to be the largest integer smaller than α . For any function f : R p → R that is ⌊ α ⌋ -times continuously differentiable at point x 0 , denote by f α x 0 its Taylor polynomial of degree ⌊ α ⌋ at x 0 , which is defined as f α x 0 ( x ) = ∑ | s |≤⌊ α ⌋ ( x -x 0 ) s /s ! D s f ( x 0 ) . For a constant L > 0 , let H α ( L, R p ) be the set of functions f : R p → R such that f is ⌊ α ⌋ -times differentiable for all x ∈ R p and satisfy | f ( x ) -f α x 0 ( x ) | ≤ L | x -x 0 | α , for all x, x 0 ∈ R p . Here | x -x 0 | is the Euclidean distance between x, x 0 ∈ R p . In non-parametric statistics literature, H α ( L, R p ) is often referred to as the class of Hölder functions. For two positive sequences { a n } n ∈ N + and { b n } n ∈ N + , we write a n = O ( b n ) or a n ≲ b n , if a n ≤ Cb n with some constant C > 0 that does not depend on n , and a n = Θ( b n ) or a n ≍ b n , if a n = O ( b n ) and b n = O ( a n ) . For a deterministic or random R -valued sequence a n , write that a sequence of random variable X n = O p ( a n ) , if lim M →∞ lim sup n →∞ P ( | X n | ≥ Ma n ) = 0 . Write X n = o p ( a n ) if lim sup n →∞ P ( | X n | ≥ Ma n ) = 0 for all M > 0 . The convergences in distribution and probability are respectively denoted by D → and P. → .

2 Model setup

Detailed assumptions imposed on the model (1) are collected in Assumption 1.

Assumption 1. The data { X t } T t =1 ⊂ R p are generated based on model (1) and satisfy the following. a. For each t = { 1 , . . . , T } , the distribution P t has a Lebesgue density function f t : R p → R , such that f t ∈ H r ( L, X ) with r, L > 0 , where X is the union of the supports of all f t , and X has bounded Lebesgue measure.

b. Let g t be the joint density of X 1 and X t +1 . It satisfies that ∥ g t ∥ L ∞ < ∞ .

The minimal spacing between two consecutive change points ∆ = min K +1 k =1 ( η k -η k -1 ) > 0 .

d. The minimal jump size between two consecutive change points κ = min k =1 ,...,K κ k > 0 , where κ k = ∥ f η k -f η k +1 ∥ L 2 denotes the jump size at the k th change point. e. The process { X t } t ∈ Z is α -mixing with mixing coefficients

The minimal spacing ∆ and the minimal jump size κ are two key parameters characterizing the change point phenomenon. Assumption 1 d. characterizes the changes in density functions through the function's L 2 -norm, enabling us to detect local and global changes in non-parametric settings.

The decay rate of α k in Assumption 1 e. imposes an upper bound on the temporal dependence. This is a standard requirement in the literature (e.g Abadi 2004, Merlevède et al. 2009).

Revolving the change point estimators, we are to conduct the estimation and inference tasks. For a sequence of estimators ̂ η 1 < . . . < ̂ η ̂ K ⊂ { 1 , . . . , T } , our first task is to show the localization consistency, i.e. with probability tending to one as the sample size T grows unbounded, it holds that

We refer to ϵ as the localization error in the rest of this paper.

With a consistent estimation result, we further refine { ̂ η k } ̂ K k =1 and obtain { ˜ η k } ̂ K k =1 with error bounds | ˜ η k -η k | = O p (1) and derive the limiting distribution of ( ˜ η k -η k ) κ p r +2 k .

We briefly summarize the contributions of our paper as follows.

We develop a multivariate non-parametric seeded change point detection algorithm detailed Algorithm 1, which is based on the seeded binary segmentation method (SBS), proposed in Kovács et al. (2020) in the univariate setting. To the best of our knowledge, we are the first to innovatively adapt SBS to a multivariate non-parametric change point model.

Under the signal-to-noise ratio condition in Assumption 3 that κ 2 ∆ ≳ log( T ) T p/ (2 r + p ) , we demonstrate that the output of Algorithm 1 is consistent, with localization errors ϵ ≍ κ -2 k T p/ (2 r + p ) log( T ) , for k ∈ { 1 , . . . , K } . This localization error is first obtained for α -mixing time series with a generic smoothness assumption, while the state-of-the-art method from Padilla et al. (2021) only focuses on Lipschitz smooth densities and under temporal independence.

Based on the consistent estimators { ̂ η } ̂ K k =1 , we construct the refined estimators { ˜ η k } ̂ K k =1 and derive their limiting distributions in different regimes, as detailed in Theorem 2. These results are novel in the literature of change point and time series analysis.

Extensive numerical results are presented in Section 5 to corroborate the theoretical findings. The code used for numerical experiments is available upon request prior to publication.

3 A two-step multivariate non-parametric change point estimators

In this section, we present the initial and refined change point estimators, both of which share the same building block, namely the non-parametric CUSUM statistic.

Definition 1 (Non-parametric CUSUM statistic) . For any integer triplet 0 ≤ s < t < e ≤ T , let the CUSUM statistic be

where F t,h is a kernel estimator of f t , i.e. F t,h ( x ) = K h ( x -X t ) with the kernel function

accompanied with the bandwidth h > 0 .

The CUSUM statistic is a key ingredient of our algorithm and is based on the kernel estimator F t,h ( · ) . We highlight that kernel-based change-point estimation techniques have been employed in detecting change points in non-parametric models in existing literature, as demonstrated in, for instance, Arlot et al. (2019), Li et al. (2019), Padilla et al. (2021).

Our preliminary estimator is obtained by combining the CUSUM statistic in Definition 1 with a modified version of SBS based on a collection of deterministic seeded intervals defined in Definition 2.

Definition 2 (Seeded intervals) . Let K = ⌈ C K log 2 ( T ∆ )⌉ , with some sufficiently large absolute constant C K > 0 . For k ∈ { 1 , . . . , K } , let J k be the collection of 2 k -1 intervals of length l k = T 2 -k +1 that are evenly shifted by l k / 2 = T 2 -k , i.e.

The overall collection of seeded intervals is denoted as J = ∪ K k =1 J k .

With the CUSUM statistics and the seeded intervals as building blocks, we are now ready to present our multivariate non-parametric seeded change point detection algorithm.

Algorithm 1 Multivariate non-parametric Seeded Binary Segmentation. MNSBS (( s, e ) , J , τ, h )

INPUT: Sample { X t } t = s ⊂ R , collection of seeded intervals J , tuning parameter τ > 0 and bandwidth h > 0 .

̸

Algorithm 1 presents a methodological approach to addressing the problem of estimating multiple change points in multivariate time series data. At its core, the algorithm leverages the strength of seeded intervals, forming a multi-scale search mechanism. To identify potential change points, the method recursively employs the CUSUM statistics. For the functionality of the algorithm, specific inputs are required. These include the observed data set, represented as X t T t =1 , the seeded intervals denoted by J , the bandwidth h that is crucial for constructing the CUSUM statistics, and a threshold, τ , which is instrumental in change point detection. We provide theoretical and numerical guidance for tuning parameters in Sections 4 and 5, respectively.

Delving deeper into the architecture of Algorithm 1, it becomes evident that the SBS functions as its foundational framework, while the nonparametric version of the CUSUM statistics acts as its functional units. The design of this algorithm is particularly tailored given its inclination toward nonparametric detection and its ability to identify multiple change points. The SBS is, in essence, an advanced version of the moving-window scanning technique. Its distinctive characteristic is its adaptability in handling the challenges posed by multiple change points that exhibit unpredictable spacing. Instead of being confined to a fixed window width, the SBS introduces versatility by incorporating a range of window width options. Each of these widths is methodically applied during a moving-window scan.

Based on the preliminary estimators { ̂ η k } ̂ K k =1 provided by Algorithm 1, we further develop a refinement procedure to enhance the localization accuracy. To be more specific, let

Then, the preliminary estimators { ̂ η k } ̂ K k =1 and ˜ h ≍ h produce an estimator of κ k as:

We then propose the final change points estimators as

where

with h 1 = c ̂ κ k ̂ κ 1 /r k and F ( s,e ] ,h 1 = 1 e -s ∑ e i = s +1 F i,h 1 for integers e > s .

If the initial change point estimators are consistent, i.e. (3) holds with probability tending to 1 , then the interval (ˆ η k -1 , ˆ η k +1 ) is anticipated to contain merely one undetected change point. By conservatively trimming this interval to ( s k , e k ) , we can safely any change points previously detected within (ˆ η k -1 , ˆ η k +1 ) . Consequently, the trimmed interval ( s k , e k ) contains only true change point η k with high probability. Due to the same reason, our choice of weight in Equation 4,1/10, is a convenient choice. In general, any constant weight between 0 and 1 / 2 would suffice. Inspired by Padilla et al. (2021), who proposed to use O ( κ k ) as an optimal bandwidth in the context of Lipschitz densities, we adopt

h

1

=

O

ˆ

κ

1

/r

k

as the bandwidth for our kernel density estimator. This choice incorporates the broader scope of our work, which studies a more general degree of smoothness. Notably, if the underlying density functions strictly adhere to the Lipschitz criterion and r = 1 , our bandwidth selection aligns with that recommended by Padilla et al. (2021). We would like to emphasize that while the procedure proposed by Padilla et al. (2021) required knowledge of the population quantities κ k , our approach is adaptive as we provide data-driven methods to estimate κ k accurately.

With our newly proposed estimators, in Theorem 2, we derive an improved error bound for the refined estimators { ˜ η k } ˜ K k =1 over the preliminary estimators { ̂ η k } ̂ K k =1 . We also study the limiting distributions of the refined estimators. Section 4.4 and Section 5 will discuss the theoretically justified rates and practical choices of tuning parameters, respectively.

The computational complexity of Algorithm 1 is O ( T 2 log( T )∆ -1 · Kernel) , where O ( T 2 log( T )∆ -1 ) is due to the computational cost of the SBS, and 'Kernel' stands for the computational cost of numerical computation of the L 2 -norm of the CUSUM statistics based on the kernel function evaluated at each time point. The dependence on the dimension p is only through the evaluation of the kernel function. The computational complexity of the final estimators (including estimating ̂ κ k 's) is O ( T · Kernel) . Therefore, the overall cost for deriving { ˜ η k } ̂ K k =1 is O ( T 2 log T ∆ -1 · Kernel) .

4 Consistent estimation and limiting distributions

To establish the theoretical guarantees of our estimators, we first state conditions needed for the kernel function K ( · ) .

Assumption 2 (The kernel function) . Assume that the kernel function K ( · ) has compact support and satisfies the following additional conditions.

a. For the Hölder smooth parameter r in Assumption 1 a , assume that K ( · ) is adaptive to H r ( L, R p ) , i.e. for any f ∈ H r ( L, R p ) , it holds that

for some absolute constant ˜ C > 0 and tuning parameter h > 0 .

b. The class of functions F K = {K ( x -· ) /h : R p → R + , h > 0 } is separable in L ∞ ( R p ) and is a uniformly bounded VC-class; i.e. there exist constants A,ν > 0 such that for any probability measure Q on R p and any u ∈ (0 , ∥K∥ L ∞ ) , it holds that N ( F K , L 2 ( Q ) , u ) ≤ ( A ∥K∥ L ∞ /u ) ν , where N ( F K , L 2 ( Q ) , u ) denotes the u -covering number of the metric space ( F K , L 2 ( Q )) .

c. Let m = ⌊ r ⌋ and it holds that ∫ ∞ 0 t m -1 sup ∥ x ∥≥ t |K ( x ) | m d t < ∞ , ∫ R p K ( z ) ∥ z ∥ d z ≤ C K , where C K > 0 is an absolute constant.

(

)

Assumption 2 is a standard condition in the non-parametric literature (e.g. Giné & Guillou 1999, 2001, Sriperumbudur & Steinwart 2012, Kim et al. 2019, Padilla et al. 2021) and holds for various kernels, such as the Triweight, Epanechnikov and Gaussian kernels, which are considered in Section 5.

4.1 Consistency of preliminary estimators

To establish the consistency of the preliminary estimators outputted by Algorithm 1, we impose the following signal-to-noise ratio condition.

Assumption 3 (Signal-to-noise ratio) . Assume there exists an arbitrarily slow diverging sequence γ T > 0 such that

We note that Assumption 3 is a mild condition, as it allows both the jump size κ to vanish asymptotically and/or the spacing ∆ between change points to be much smaller than T . The consistency of Algorithm 1 is established in the following theorem.

Theorem 1. Suppose Assumptions 1, 2 and 3 hold. Let { ̂ η k } ̂ K k =1 be the estimated change points returned by Algorithm 1 with tuning parameters τ = c τ T p/ (4 r +2 p ) log 1 / 2 ( T ) and h = c h T -1 / (2 r + p ) for sufficiently large constants c h , c τ > 0 . Then

where C ϵ and C p, K are positive constants only depending on the kernel and the dimension p .

4.2 Refined estimators and their limiting distributions

To develop refined estimators based on the preliminary estimators and study their limiting distributions, we would need to require a slightly stronger signal-to-noise ratio condition below.

Assumption 4 (Signal-to-noise ratio for inference) . Assume that there exists an arbitrarily slow diverging sequence γ T > 0 such that

Assumption 4 is slightly stronger than Assumption 3. This is because our refined estimators are based on a sequence of random endpoints, i.e. the preliminary estimators. This brings theoretical challenges in deriving limiting distributions and estimating the long-run variances. It is worth noting that a similar phenomenon has been observed in the study on conducted by Xu, Wang, Zhao & Yu (2022).

Theorem 2. Suppose that Assumptions 1, 2 and 3 hold. Let { ˜ η k } ̂ K k =1 be the refined change point estimators defined in Section 3, with the preliminary estimators { ̂ η k } ̂ K k =1 returned by Algorithm 1, the intervals { ( s k , e k ) } ̂ K k =1 defined in (4) , and ̂ κ k defined as in (5) . The following holds:

a. (Non-vanishing regime) Suppose the jump size at the change point η k satisfies lim T →∞ κ k → ϱ k for some absolute constant ϱ k > 0 . Then, as T →∞ , it holds that | ˜ η k -η k | = O p (1) and that

˜

where

Here ∗ denotes convolution and h 2 = c κ k κ 1 /r k for some absolute constant c κ k > 0 . b. (Vanishing regime) Suppose the jump size at the change point η k satisfies lim T →∞ κ k = 0 . Then, as T →∞ , it holds that | ˜ η k -η k | = O p ( κ -2 -p/r k ) and that

˜

where h 2 = c κ k κ 1 /r k for some absolute constant c κ k > 0 . Here

with B 1 ( r ) and B 2 ( r ) being two independent standard Brownian motions, and

Theorem 2 considers vanishing and non-vanishing regimes of the jump sizes. The upper bounds on the localization error in both regimes can be written as

Therefore, when the Hölder smoothness parameter r = 1 , our final estimator { ˜ η k } attains the minimax optimal convergence rate developed in Lemma 3 by Padilla et al. (2021). Furthermore, when r = 1 , our resulting rate is sharper than that in Theorem 1 in Padilla et al. (2021), as we are able to remove the logarithmic factors from the upper bound. Additionally, our method can achieve optimal rates with choices of tuning parameters that do not depend on the unknown jump sizes κ k .

Theorem 2 summarizes the limiting distributions of the refined estimators { ˜ η k } ̂ K k =1 . In the nonvanishing case, the resulting limiting distribution can be approximated by a two-sided random walk and the change points can be accurately estimated within a constant error rate. In contrast, in the vanishing regime, a central limit theorem under mixing conditions leads to a two-sided Brownian motion distribution in the limit, which quantifies the asymptotic uncertainty of { ˜ η k } ̂ K k =1 , enabling inference on change point locations, and allowing for the construction of confidence intervals.

4.3 Consistent long-run variance estimation

To obtain valid confidence intervals for change points using the limiting distributions in Theorem 2 b. , it is crucial to access robust estimators for the long-run (asymptotic) variances { ˜ σ 2 ∞ ( k ) } ̂ K k =1 defined in (8). We propose a block-type long-run variance estimator in Algorithm 2 to fulfill this task and demonstrate its consistency in the following theorem.

Theorem 3. Suppose Assumptions 1, 2 and 3 hold. Let { ˜ σ 2 ∞ ( k ) } ̂ K k =1 be the population long-run variance defined in (8) and ̂ σ 2 ∞ ( k ) the output of Algorithm 2 with R = O ( T ( p + r ) / (2 r + p ) /κ p/ (2 r )+3 / 2 k ) . Then it holds that

4.4 Discussions on MNSBS

Tuning parameters. Our procedure comprises three steps: (1) preliminary estimation, (2) local refinement, and (3) confidence interval construction with three key tuning parameters. For step (1), we use a kernel density estimator with bandwidth h ≍ T -1 / (2 r + p ) , which follows from the classical non-parametric literature (e.g. Yu 1993, Tsybakov 2009). The threshold tuning parameter τ is set to a high-probability upper bound on the CUSUM statistics when there is no change point, which reflects the requirement on the signal-to-noise ratio detailed in Assumption 3. For refined estimation in step (2) and long-run variance estimation in step (3), we set the bandwidth parameter h 1 ≍ ̂ κ 1 /r k . This choice of bandwidth is inspired by the minimax rate-optimal bandwidth used in Padilla et al. (2021).

Comparison with Padilla et al. (2021). Our main contribution is deriving the limiting distribution of multivariate non-parametric change point estimators. This problem has not been formally studied in the existing literature. Additionally, our Hölder assumption is more general than the Lipschitz assumption used in Padilla et al. (2021). Our Assumption 1 d specifies changes through the L 2 -norm of probability density functions, which is a weaker assumption than the L ∞ -norm used in Padilla

et al. (2021). Furthermore, our assumptions allow for temporal dependence captured by α -mixing coefficients, whereas Padilla et al. (2021) assumed independent observations.

Comparison with existing literature in nonparametric, online, and inference change point.

In the nonparametric change point literature, different kernel-based methods are adopted for change point localisation and testing. In the offline setting, the penalized kernel least squares estimator, originally introduced by Harchaoui & Cappé (2007), was explored by Arlot et al. (2012) for multivariate change point problems, and an oracle inequality was derived. An upper bound on the localization rate provided by this method was established by Garreau & Arlot (2018) and was computationally enhanced further in Celisse et al. (2018). With a focus on a so-called running maximum partition strategy, Harchaoui et al. (2008) formulated a kernel-based test statistic to ascertain the existence of a change-point. In a similar vein, Zou et al. (2014) investigated a problem where s out of n sequences are anomalous and devised a test statistic using the kernel maximum mean discrepancy.

In the online setting, Kifer et al. (2004) introduces a meta-algorithm comparing data from a "reference window" to current data using empirical measures. Desobry et al. (2005) detects shifts by comparing two descriptor sets from the signal's immediate past and future, using a dissimilarity metric resembling the Fisher ratio in Gaussian cases via a soft margin single-class SVM. Meanwhile, Liu et al. (2013) adopts density ratio estimation with a non-parametric Gaussian kernel model for change-point detection, updating its parameters online through stochastic gradient descent.

The core methodology is largely shared but with different goals and performance measurements regarding online and offline change point literature comparisons. How to conduct inference in the online change point context is also unclear.

Compared to the existing work, in this paper, we follow the suit of using kernel-based CUSUM statistics but incorporate temporal dependence, which is rarely seen in the literature. Most importantly, we are unaware of existing work on nonparametric change point inference, which is the main selling point of our paper.

Most change point inference work focuses on fixed-dimensional parameters as well as lacks tracking of many model parameters. Xu, Wang, Zhao & Yu (2022), in terms of style, is indeed the most closely related. But tackles high-dimensional linear regression, fundamentally distinct from our nonparametric density estimation.

5 Numerical Experiments

We refer to MNSBS as our final estimator, which is used for both localization and inference tasks. To evaluate its localization performance, we compare our proposed method against four competitors MNP (Padilla et al. 2021), EMNCP (Matteson & James 2014), SBS (Cho & Fryzlewicz 2015) and DCBS (Cho 2016) - across a wide range of simulation settings, using corresponding R functions in changepoints (Xu, Padilla, Wang & Li 2022), ecp (James et al. 2019) and hdbinseg (Cho

& Fryzlewicz 2018) packages. However, to the best of our knowledge, no competitor is currently available for the inference task.

Tuning parameters. For MNSBS implementation, we use the Gaussian kernel and the false discovery rate control-based procedure of Padilla et al. (2021) for τ selection. Preliminary estimators are set as h = 2 × (1 /T ) 1 / (2 r + p ) , while the second stage estimator has bandwidths respectively set as ˜ h = 0 . 05 and h 1 = 2 × ̂ κ 1 /r k . Selection of R = ⌊( max ̂ K k =1 { e k -s k } ) 3 / 5 ⌋ with { ( s k , e k ) } ̂ K k =1 is guided by Theorem 3 using { ( s k , e k ) } ̂ K k =1 from (4). For the confidence interval construction, we use { ̂ κ k } ̂ K k =1 and { ̂ σ 2 ∞ ( k ) } ̂ K k =1 to estimate the required unknown quantities. We evaluate L 2 based statistics in Change point estimation and Long-run variance estimation using the Subregion-Adaptive Vegas Algorithm 1 with a maximum of 10 5 function evaluations.

Evaluation measurements For a given set of true change points C = { η k } K +1 k =0 , to assess the accuracy of the estimator ̂ C = { ̂ η k } ̂ K +1 k =0 with ̂ η 0 = 1 and ̂ η T +1 = T + 1 f , we report (1) Misestimation rate: the proportion of misestimating K and (2) Scaled Hausdorff distance: d H ( ̂ C , C ) , defined by d H ( ̂ C , C ) = 1 T max { max x ∈ ̂ C min y ∈C {| x -y |} , max y ∈ ̂ C min x ∈C {| x -y |}} .

The performance of our change point inference is measured by the coverage of η k , defined as cover k (1 -α ) for significance level α ∈ (0 , 1) . For, k = 1 , . . . , K,

with ̂ q u ( α/ 2) and ̂ q u (1 -α/ 2) are the α/ 2 and 1 -α/ 2 empirical quantiles of the simulated limiting distribution given in (7), ̂ κ k is defined in (5), and k = 1 , . . . , K .

5.1 Localization

We consider three different scenarios with two equally spaced change points. For each scenario, we set r = 2 , and vary T ∈ { 150 , 300 } and p ∈ { 3 , 5 } . Moreover, we consider { Y t = ✶ {⌊ T/ 3 ⌋ < t ≤ ⌊ 2 T/ 3 ⌋} Z t + X t } T t =1 ⊂ R p with X t = 0 . 3 X t -1 + ϵ t .

Scenario 1 (S1) Let Z t = µ ∈ R p , where µ j = 0 for j ∈ { 1 , . . . , ⌈ p/ 2 ⌉} and µ j = 2 otherwise. Let { ϵ t } be i.i.d. N (0 p , I p ) .

Scenario 2 (S2) Let Z t |{ u t = 1 } = 1 . 5 × 1 p , Z t |{ u t = 0 } = -1 . 5 × 1 p , where { u t } are i.i.d. Bernoulli( 0 . 5 ) random variables. Let { ϵ t } be i.i.d. N (0 p , I p ) .

Scenario 3 (S3) Let Z t = 0 . 3 Z t -1 +0 . 5 × 1 p + ϵ ∗ t , where { ϵ ∗ t } ⊂ R p and { ϵ t } ⊂ R p are mutually independent. They are i.i.d. with entries independently follow Unif ( - √ 3 , √ 3) and the standardized Pareto (3 , 1) , respectively.

S1-S3 encompass a variety of simulation settings, including the same type of distributions, changed mean and constant covariance in S1 ; a mixture of distributions in S2 ; and change between light-tailed and heavy-tailed distributions in S3 . We conduct 200 repetitions of each experiment and present the results for localization in Figure 1. Our proposed method, MNSBS, generally outperforms all other methods in all scenarios, except for S2 , where ECP performs better. However, we observe that MNSBS achieves comparable performance to ECP for large T in S2 .

5.2 Inference

In this section, we focus solely on analyzing the limiting distribution obtained in Theorem 2. b. , which pertains to the vanishing regime. We explain this from two different perspectives. Firstly, in the non-vanishing regime (Theorem 2. a. ), the localization error is at the order of O (1) . As a result, the construction of confidence intervals, which is a direct application of the limiting distribution, is of little demand with such estimation results. Secondly, since the localization error is only at the order of O (1) , the universality cannot come into play to produce a useful limiting distribution.

0.19

0.38

0.56

0.75

Misestimation rate of K

T=150

T=300

p = 3

p = 5

p = 3

p = 5

0.25

0.50

0.75

1.00

Misestimation rate of K

T=150

T=300

p = 3

p = 5

p = 3

p = 5

0.25

0.50

0.75

1.00

Misestimation rate of K

T=150

T=300

p = 3

p = 5

p = 3

p = 5

0.06

0.12

0.19

0.25

Scaled Hausdorff distance

T=150

T=300

p = 3

p = 5

p = 3

p = 5

0.09

0.17

0.26

0.35

Scaled Hausdorff distance

T=150

T=300

p = 3

p = 5

p = 3

p = 5

0.09

0.17

0.26

0.35

Scaled Hausdorff distance

T=150

T=300

p = 3

p = 5

p = 3

p = 5

that our localization results are robust to the bandwidth parameters, yet sensitive to the smoothness parameter r . We thus set r = 1000 in our simulations, as the density function of a multivariate normal distribution belongs to the Hölder function class with r = ∞ . Table 1 shows that our proposed inference procedure produces good coverage in the considered setting.

6 Conclusion

Wetackle the problem of change point detection for short range dependent multivariate non-parametric data, which has not been studied in the literature. Our two-stage algorithm MNSBS can consistently estimate the change points in stage one, a novelty in the literature. Then, we derived limiting distributions of change point estimators for inference in stage two, a first in the literature.

Our theoretical analysis reveals multiple challenging and interesting directions for future exploration. Relaxing the assumption ∆ ≍ T may be of interest. In addition, in Theorem 2. a , we can see the limiting distribution is a function of the data-generating mechanisms, lacking universality, therefore deriving a practical method to derive the limiting distributions in the non-vanishing regime may be interesting.

Acknowledgments and Disclosure of Funding

HX was supported by Swiss National Science Foundation Postdoc Mobility fellowship. OHMM is partially funded by NSF DMS-2015489. YY's research is partially funded by EPSRC EP/V013432/1.

References

Abadi, M. (2004), 'Sharp error terms and neccessary conditions for exponential hitting times in mixing processes', The Annals of Probability 32 (1A), 243-264.

Arlot, S., Celisse, A. & Harchaoui, Z. (2012), 'A kernel multiple change-point algorithm via model selection', arXiv preprint arXiv:1202.3878 .

Arlot, S., Celisse, A. & Harchaoui, Z. (2019), 'A kernel multiple change-point algorithm via model selection', Journal of machine learning research 20 (162).

Aue, A., Hörmann, S., Horváth, L. & Reimherr, M. (2009), 'Break detection in the covariance structure of multivariate time series models', The Annals of Statistics 37 (6B), 4046-4087.

Azhar, M. A. R., Nugroho, H. A. & Wibirama, S. (2021), The study of multivariable autoregression methods to forecast infectious diseases, in '2021 IEEE 5th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE)', IEEE, pp. 83-88.

Boussama, F., Fuchs, F. & Stelzer, R. (2011), 'Stationarity and geometric ergodicity of bekk multivariate garch models', Stochastic Processes and their Applications 121 (10), 2331-2360.

Cai, X., Wang, X., Eichi, H. R., Ongur, D., Dixon, L., Baker, J. T., Onnela, J.-P. & Valeri, L. (2022), 'State space model multiple imputation for missing data in non-stationary multivariate time series with application in digital psychiatry', arXiv preprint arXiv:2206.14343 .

Celisse, A., Marot, G., Pierre-Jean, M. & Rigaill, G. (2018), 'New efficient algorithms for multiple change-point detection with reproducing kernels', Computational Statistics & Data Analysis 128 , 200-220.

Chan, K.-S. & Tong, H. (2001), Chaos: a statistical perspective , Springer Science & Business Media.

Cho, H. (2016), 'Change-point detection in panel data via double cusum statistic', Electronic Journal of Statistics 10 (2), 2000-2038.

Cho, H. & Fryzlewicz, P. (2015), 'Multiple-change-point detection for high dimensional time series via sparsified binary segmentation', Journal of the Royal Statistical Society: Series B (Statistical Methodology) 77 (2), 475-507.

Cho, H. & Fryzlewicz, P. (2018), hdbinseg: Change-Point Analysis of High-Dimensional Time Series via Binary Segmentation . R package version 1.0.1. URL: https://CRAN.R-project.org/package=hdbinseg

Corbella, S. & Stretch, D. D. (2012), 'Predicting coastal erosion trends using non-stationary statistics and process-based models', Coastal engineering 70 , 40-49.

Desobry, F., Davy, M. & Doncarli, C. (2005), 'An online kernel change detection algorithm', IEEE Transactions on Signal Processing 53 (8), 2961-2974.

Doukhan, P. (1994), 'Mixing: properties and examples. lect', Notes in Statisit 85 .

Fan, J. & Yao, Q. (2003), Nonlinear time series: nonparametric and parametric methods , Vol. 20, Springer.

Fan, J. & Yao, Q. (2008), Nonlinear time series: nonparametric and parametric methods , Springer Science & Business Media.

Frolov, N., Maksimenko, V. & Hramov, A. (2020), 'Revealing a multiplex brain network through the analysis of recurrences', Chaos: An Interdisciplinary Journal of Nonlinear Science 30 (12), 121108.

Fryzlewicz, P. (2014), 'Wild binary segmentation for multiple change-point detection', The Annals of Statistics 42 (6), 2243-2281.

Garreau, D. & Arlot, S. (2018), 'Consistent change-point detection with kernels'.

Giné, E. & Guillou, A. (1999), 'Laws of the iterated logarithm for censored data', The Annals of Probability 27 (4), 2042-2067.

Giné, E. & Guillou, A. (2001), 'On consistency of kernel density estimators for randomly censored data: rates holding uniformly over adaptive intervals', Annales de l'IHP Probabilités et statistiques 37 (4), 503-522.

Gorrostieta, C., Ombao, H. & Von Sachs, R. (2019), 'Time-dependent dual-frequency coherence in multivariate non-stationary time series', Journal of Time Series Analysis 40 (1), 3-22.

Han, F. & Wu, W. B. (2023), Probability inequalities for high-dimensional time series under a triangular array framework, in 'Springer Handbook of Engineering Statistics', Springer, pp. 849863.

Harchaoui, Z. & Cappé, O. (2007), Retrospective mutiple change-point estimation with kernels, in '2007 IEEE/SP 14th Workshop on Statistical Signal Processing', IEEE, pp. 768-772.

Harchaoui, Z., Moulines, E. & Bach, F. (2008), 'Kernel change-point analysis', Advances in neural information processing systems 21 .

Heo, T. & Manuel, L. (2022), 'Greedy copula segmentation of multivariate non-stationary time series for climate change adaptation', Progress in Disaster Science 14 , 100221.

Herzel, S., St˘ aric˘ a, C. & Tütüncü, R. (2002), A non-stationary multivariate model for financial returns , Chalmers University of Technology.

James, N. A., Zhang, W. & Matteson, D. S. (2019), ecp: An R Package for Nonparametric Multiple Change Point Analysis of Multivariate Data . R package version 3.1.2. URL: https://cran.r-project.org/package=ecp

Kifer, D., Ben-David, S. & Gehrke, J. (2004), Detecting change in data streams, in 'VLDB', Vol. 4, Toronto, Canada, pp. 180-191.

Kim, J., Shin, J., Rinaldo, A. & Wasserman, L. (2019), Uniform convergence rate of the kernel density estimator adaptive to intrinsic volume dimension, in 'International Conference on Machine Learning', PMLR, pp. 3398-3407.

Kirch, C. (2006), Resampling methods for the change analysis of dependent data, PhD thesis, Universität zu Köln.

Kovács, S., Li, H., Bühlmann, P. & Munk, A. (2020), 'Seeded binary segmentation: A general methodology for fast and optimal change point detection', arXiv preprint arXiv:2002.06633 .

Kunitomo, N. & Sato, S. (2021), 'A robust-filtering method for noisy non-stationary multivariate time series with econometric applications', Japanese Journal of Statistics and Data Science 4 (1), 373-410.

Li, S., Xie, Y., Dai, H. & Song, L. (2019), 'Scan b-statistic for kernel change-point detection', Sequential Analysis 38 (4), 503-544.

Liebscher, E. (2005), 'Towards a unified approach for proving geometric ergodicity and mixing properties of nonlinear autoregressive processes', Journal of Time Series Analysis 26 (5), 669-689.

Liu, S., Yamada, M., Collier, N. & Sugiyama, M. (2013), 'Change-point detection in time-series data by relative density-ratio estimation', Neural Networks 43 , 72-83.

Matteson, D. S. & James, N. A. (2014), 'A nonparametric approach for multiple change point analysis of multivariate data', Journal of the American Statistical Association 109 (505), 334-345.

Merlevède, F., Peligrad, M. & Rio, E. (2012), 'Bernstein inequality and moderate deviations under strong mixing conditions', arXiv preprint arXiv:1202.4777 .

Merlevède, F., Peligrad, M., Rio, E. et al. (2009), 'Bernstein inequality and moderate deviations under strong mixing conditions', High dimensional probability V: the Luminy volume 5 , 273-292.

Molenaar, P., Sinclair, K. O., Rovine, M. J., Ram, N. & Corneal, S. E. (2009), 'Analyzing developmental processes on an individual level using nonstationary time series modeling.', Developmental psychology 45 (1), 260.

Narasimhan, B., Johnson, S. G., Hahn, T., Bouvier, A. & Kiêu, K. (2022), cubature: Adaptive Multivariate Integration over Hypercubes . R package version 2.0.4.5. URL: https://CRAN.R-project.org/package=cubature

Nguyen, H. M., Turk, P. J. & McWilliams, A. D. (2021), 'Forecasting covid-19 hospital census: A multivariate time-series model based on local infection incidence', JMIR Public Health and Surveillance 7 (8), e28195.

Padilla, C. M. M., Wang, D., Zhao, Z. & Yu, Y. (2022), 'Change-point detection for sparse and dense functional data in general dimensions', arXiv preprint arXiv:2205.09252 .

Padilla, O. H. M., Yu, Y., Wang, D. & Rinaldo, A. (2021), 'Optimal nonparametric multivariate change point detection and localization', IEEE Transactions on Information Theory .

Schmitt, T. A., Chetalova, D., Schäfer, R. & Guhr, T. (2013), 'Non-stationarity in financial time series: Generic features and tail behavior', EPL (Europhysics Letters) 103 (5), 58003.

Sriperumbudur, B. & Steinwart, I. (2012), Consistency and rates for clustering with dbscan, in 'Artificial Intelligence and Statistics', PMLR, pp. 1090-1098.

Tsybakov, A. B. (2009), Introduction to Nonparametric Estimation , Springer series in statistics, Springer, Dordrecht.

van der Vaart, A. W. & Wellner, J. A. (1996), Weak Convergence and Empirical Processes: With Applications to Statistics , Springer New York, New York, NY. URL: https://doi.org/10.1007/978-1-4757-2545-2 3

Venkatraman, E. S. (1992), Consistency results in multiple change-point problems , Stanford University.

Wang, D., Yu, Y. & Rinaldo, A. (2020), 'Univariate mean change point detection: Penalization, cusum and optimality', Electronic Journal of Statistics 14 (1), 1917-1961.

Wolkovich, E. & Donahue, M. J. (2021), 'How phenological tracking shapes species and communities in non-stationary environments', Biological Reviews 96 (6), 2810-2827.

Xu, H., Padilla, O., Wang, D. & Li, M. (2022), changepoints: A Collection of Change-Point Detection Methods . R package version 1.1.0.

URL: https://github.com/HaotianXu/changepoints

Xu, H., Wang, D., Zhao, Z. & Yu, Y. (2022), 'Change point inference in high-dimensional regression models under temporal dependence', arXiv preprint arXiv:2207.12453 .

Yu, B. (1993), 'Density estimation in the l ∞ norm for dependent data with applications to the gibbs sampler', The Annals of Statistics pp. 711-735.

Zou, S., Liang, Y., Poor, H. V. & Shi, X. (2014), 'Nonparametric detection of anomalous data via kernel mean embedding', arXiv preprint arXiv:1405.2294 .

A Additional simulation results

A.1 Robustness to Kernel Selection and Bandwidth Parameters

In this subsection, we provide additional simulation results in Table 2 to show that our proposed localization method, i.e. MNSBS, is robust against both the choice of Kernel functions and the choice of bandwidth parameters.

We consider the setting with T = 150 and p = 3 of Scenario 1 . In addition to the Gaussian kernel used in the previous numeric experiments, we consider the Epanechnikov kernel and the Triweight kernel. We also let the bandwidth h = c h × (1 /T ) 1 / (2 r + p ) with r = 2 and c h ∈ { 1 , 2 , 5 , 10 , 20 } . Out of the 500 iterations for each case, the table below reports the proportion of the number of change points K and the averaged localization errors.

̸

A.2 Runtime and localization for Independent Data

Examination of Scenario 1 with independent data.

We examined Scenario 1 where p = 3 , n is in the set { 150 , 300 } , and X t is i.i.d. distributed as N (0 p , I p ) . Our results indicate that MNSBS excels in change point localization. The refinement process further enhances its performance. Refer to Table 3 for specifics.

Runtime Comparison.

We benchmarked the runtime of our method against others. The tests were conducted on a machine powered by an Apple M2 chip with an 8-core CPU. The parameters were set at p = 3 and n in the set { 150 , 300 } for the independent setting. Our method performs comparably at n = 150 . However, it is slower at n = 300 , attributed to the computational demands of CUSUM. Detailed findings are presented in Table 3.

̸

B Real data application

We applied our proposed change point inference procedure to analyze stock price data 2 , which consisted of daily adjusted close price of the 3 major stock market indices (S&P 500, Dow Jones and NASDAQ) from Jan-01-2021 to Jan-19-2023. After removing missing values and standardizing the raw data, the sample size was n = 515 and the dimension p = 3 .

We localized 6 estimated change points and performed inference based on them; results are summarized in Table 4. We also implemented the NMP and ECP methods on the same dataset, the estimated change points being presented below. Except for the time point Aug-24-2022 estimated by ECP, all other estimated change points were located in the constructed 99% confidence intervals by our proposed method.

The transformed real data is illustrated in the figure below. These data correspond to the daily adjusted close price, from Jan-01-2021 to Jan-19-2023, of the 3 major stock market indices, S&P 500, Dow Jones and NASDAQ. Moreover, in Table 4, we present the estimated change point by our proposed method MNSBS on the data before mentioned, together with their respective inference.

date

standardized daily close price

0

100

200

300

400

500

-2

-1

0

1

2

The result of the implementation of NMP and ECP methods on the same dataset are {April-01-2021, July-01-2021, Oct-19-2021, Jan-14-2022, April-21-2022, Oct-26-2022} and {April-08-2021, June-25-2021, Oct-18-2021, Jan-18-2022, April-28-2022, Aug-24-2022, Oct-27-2022} respectively.

C α -mixing coefficients

This paper focuses on multivariate time series that exhibit α -mixing behavior with exponential decay coefficients. This condition is denoted as Assumption 1 e . While the constant 2 c is present in the exponent of the exponential function, it plays a non-essential role in our theoretical framework. We include it solely for the sake of convenience during verification.

The α -mixing condition with exponential decay as specified in Assumption 1 e is a commonly held assumption in time series analysis. A broad spectrum of multivariate time series satisfies this condition, including linear/nonlinear V AR models [e.g. Liebscher (2005)], a comprehensive class of GARCH models [e.g. Boussama et al. (2011)], and various Markov processes [e.g. Chan & Tong (2001)]. To further elaborate, consider the p dimensional stationary VAR(1) model:

where A is the p × p transition matrix whose spectral norm satisfying ∥ A ∥ ∈ (0 , 1) and the innovations ϵ t are i.i.d. Gaussian vectors. Denote Σ = cov ( X 1 ) , and let λ max and λ min be the largest and smallest eigenvalues of Σ . Then by Theorem 3.1 in Han & Wu (2023), we have that for any k ≥ 0 , the α -mixing coefficient of the time series X t satisfying

where C > 0 is some constant depending only on √ λ max /λ min . In this example, the constant C log(1 / | A | ) corresponds to the constant 2 c in Assumption 1 e . Essentially, Assumption 1 e is useful to unlock several technical tools under temporal dependence, which include a Bernstein's inequality Merlevède et al. (2012), a moment inequality [see Proposition 2.5 in Fan & Yao (2003)], maximal inequalities (see Section G.1) and a central limit theorem (see Section G.2). For instance, we utilize the moment inequality to bound the autocovariances of a dependence process with all lags by α -mixing coefficients, thereby demonstrating the existence of the long-run variance, which is the sum of all the autocovariances.

D Proof of Theorem 1

In this section, we present the proof of theorem Theorem 1.

Proof of Theorem 1. For any ( s, e ] ⊆ (0 , T ] , let

For any ˜ r ∈ ( ρ, T -ρ ] , we consider

From Algorithm 1, we have that

Therefore, Proposition 2 imply that with

for some diverging sequence C λ , it holds that

and,

Now, we notice that,

since 2 -x < 1 for any x > 0 . In addition, there are K = O (1) number of change points. In consequence, it follows that

The rest of the argument is made by assuming the events in equations (10), (11) and (12) hold. By Remark 1, we have that on these events, it is satisfied that

Denote

where κ = min { κ 1 , . . . , κ K } . Since Υ k is the desired localisation rate, by induction, it suffices to consider any generic interval ( s, e ] ⊆ (0 , T ] that satisfies the following three conditions:

Here q = -1 indicates that there is no change point contained in ( s, e ] .

Denote

Observe that by assumption 3,

Therefore, it has to be the case that for any true change point η m ∈ (0 , T ] , either | η m -s | ≤ Υ m or | η m -s | ≥ ∆ -Υ max ≥ 3 4 ∆ . This means that min {| η m -e | , | η m -s |} ≤ Υ m indicates that η m is a detected change point in the previous induction step, even if η m ∈ ( s, e ] . We refer to η m ∈ ( s, e ] as an undetected change point if min { η m -s, η m -e } ≥ 3 4 ∆ . To complete the induction step, it suffices to show that MNSBS (( s, e ] , h, τ )

(i) will not detect any new change point in ( s, e ] if all the change points in that interval have been previously detected, and

(ii) will find a point D I ∗ in ( s, e ] such that | η m -D I ∗ | ≤ Υ m if there exists at least one undetected change point in ( s, e ] .

In order to accomplish this, we need the following series of steps.

Step 1. We first observe that if η k ∈ { η k } K k =1 is any change point in the functional time series, by Lemma 5, there exists a seeded interval I k = ( s k , e k ] containing exactly one change point η k such that

Even more, we notice that if η k ∈ ( s, e ] is any undetected change point in ( s, e ] . Then it must hold that

Since Υ max = O (log( T ) T p 2 r + p ) and by assumption 3, we have that Υ max < 1 10 ∆ . Moreover, η k -s k ≤ 9 10 ( η k -η k -1 ) , so that it holds that

and in consequence s k ≥ s . Similarly e k ≤ e . Therefore

Step 2. Consider the collection of intervals {I k = ( s k , e k ] } K k =1 in Step 1. In this step, it is shown that for each k ∈ { 1 , . . . , K } , it holds that

for some sufficient small constant c 1 .

Let k ∈ { 1 , . . . , K } . By Step 1 , I k contains exactly one change point η k . Since f t is a one-dimensional population time series and there is only one change point in I k = ( s k , e k ] , it holds that

̸

which implies, for s < t < η

Similarly, for η k ≤ t ≤ e k

Therefore,

Since ρ = O (log( T ) T p 2 r + p ) , by Assumption 3, we have that

so that η k ∈ [ s k + ρ, e k -ρ ] . Then, from (14), (15) and the fact that | e k -s k | < ∆ and | η k -s k | < ∆ , || ˜ f ( s k ,e k ] η k || L 2 = √ e k -η k ( e k -s k )( η k -s k ) ( η k -s k ) || f η k -f η k +1 || L 2 ≥ c 2 √ ∆ 3 4 κ k . (16)

Therefore, it holds that

where the first inequality follows from the fact that η k ∈ [ s k + ρ, e k -ρ ] , the second inequality follows from the good event in (10) and Remark 2, and the last inequality follows from (16). Next, we observe that by Assumption 3

and,

In consequence, since κ k is a positive constant, by the upper bound of λ on Equation (9), for sufficiently large T , it holds that

Therefore,

Therefore Equation (13) holds with c 1 = c 2 2 .

Step 3. In this step, it is shown that SBS (( s, e ] , h, τ ) can consistently detect or reject the existence of undetected change points within ( s, e ] .

Suppose η k ∈ ( s, e ] is any undetected change point. Then by the second half of Step 1 , I k ⊆ ( s, e ] , and moreover

where the second inequality follows from Equation (13), and the last inequality follows from Assumption 3 and the choice of τ = C τ ( log 1 2 ( T ) √ 1 h p ) . Therefore, M s,e = ∅ , since I k ∈ M s,e .

̸

Suppose there does not exist any undetected change point in ( s, e ] . Then for any I = ( α, β ] ⊆ ( s, e ] , one of the following situations must hold,

(a) There is no change point within ( α, β ] ;

(b) there exists only one change point η k within ( α, β ] and min { η k -α, β -η k } ≤ Υ k ;

(c) there exist two change points η k , η k +1 within ( α, β ] and

Observe that if (a) holds, then we have

Cases (b) and (c) can be dealt with using similar arguments. We will only work on (c) here. It follows that, in the good event in Equation (10),

where the second inequality is followed by Lemma 7. Therefore in the good event in Equation (10), for any I = ( α, β ] ⊆ ( s, e ] , it holds that

Then,

We observe that √ log( T ) h p = O ( log( T ) 1 / 2 √ 1 h p ) . Moreover,

Therefore, by the choice of τ , we will always correctly reject the existence of undetected change points, since

Thus, by the choice of τ , it holds that with sufficiently large constant C τ ,

As a result, MNSBS (( s, e ] , h, τ ) will correctly reject if ( s, e ] contains no undetected change points.

Step 4. Assume that there exists an undetected change point η ˜ k ∈ ( s, e ] such that

Then, M s,e = ∅ . Let I ∗ be defined as in MNSBS (( s, e ] , h, τ ) with

To complete the induction, it suffices to show that, there exists a change point η k ∈ ( s, e ] such that min { η k -s, η k -e } ≥ 3 4 ∆ and | b I ∗ -η k | ≤ Υ k . To this end, we consider the collection of change points of { f t } t ∈ ( α ∗ ,β ∗ ] We are to ensure that the assumptions of Lemma 12 are satisfied. In the following, λ is used in Lemma 12. Then Equation (68) and Equation (69) are directly consequence of Equation (10), Equation (11), Equation (12). By the narrowest of I ∗ ,

and by Step 1 with I k = ( s k , e k ] , it holds that

Therefore for all k ∈ { ˜ k : min { η ˜ k -s, e -η ˜ k } ≥ c 2 ∆ } ,

where the last inequality follows from Equation (13). Therefore (70) holds in Lemma 12. Finally, Equation (71) is a direct consequence of the choices that

Thus, all the conditions in Lemma 12 are met. So that, there exists a change point η k of { f t } t ∈I ∗ , satisfying and

and given that, we get,

̸

for sufficiently large constant C , where we have followed the same line of arguments as for the conclusion of (20). Observe that i) The change points of { f t } t ∈I ∗ belong to ( s, e ] ∩ { η k } K ; and

ii) Equation (21) and ( α , β ] ⊆ ( s, e ] imply that k =1 ∗ ∗

As discussed in the argument before Step 1 , this implies that η k must be an undetected change point of { f t } t ∈I ∗ .

E Proof of Theorem 2

In this section, we present the proof of theorem Theorem 2.

Proof of Theorem 2. Uniform tightness of κ 2+ p r k ∣ ∣ ∣˜ η k -η k ∣ ∣ ∣ . Here we show a.1 and b.1 . For this purpose, we will follow a series of steps. On step 1 , we rewrite (6) in order to derive a uniform bound. Step 2 analyses the lower bound while Step 3 the upper bound.

Step 1: Denote ˜ r = ˜ η k -η k . Without loss of generality, suppose ˜ r ≥ 0 . Since ˜ η k = η k + ˜ r , defined in (6), is the minimizer of ̂ Q k ( η ) , it follows that

where,

Observe that,

If ˜ r ≤ 1 /κ 2+ p r k , then there is nothing to show. So for the rest of the argument, for contradiction, assume that

Step 2: Finding a lower bound. In this step, we will find a lower bound of the inequality (24). To this end, we observe that,

Let

We consider,

From above, we have that,

We now analyze the order of magnitude of term I 1 . Then, we get a lower bound for the term -I 1 . In fact I 1 , has an upper bound of the form o p ( ˜ rk p r +2 ) , where we use that || f η k ∗ K h 2 -f η k || L 2 = o (1) and || f η k +1 ∗ K h 2 -f η k +1 || L 2 = o (1) . For the term I 2 , we consider the random variable,

In order to use Lemma 3, we need to bound E ( | Y i | 3 ) . For this, first we use Cauchy Schwartz inequality,

then, by Minkowski's inequality,

Therefore, by Assumption 2, we have

for any t ∈ ( η k , e k ] . Moreover, we have that

Therefore, by Lemma 3, we have that I 2 = o p ( √ ˜ rκ k κ -p 2 r k (log( ˜ rκ p r +2 k ) + 1) ) . Thus,

Step 3: Finding an upper bound. Now, we proceeded to get an upper bound of (24). This is, an upper bound of the following expression,

Observe that, this expression can be written as,

So that, where,

Now, we analyze each of the terms above. For U 1 , observe that

where,

To analyze I 3 , we rewrite it as follow,

where,

Now, we will get an upper bound for each of the terms above. The term I 3 , 1 = O p ( ˜ r 1 T log( T ) κ p r k ) , which is followed by the use of Remark 1 . Even more, by Assumption 4, we get

For the term I 3 , 2 , by Cauchy Schwartz inequality and triangle inequality,

and using basic properties of integrals || f [ η k +1 ,e k ] ∗ K h 2 -f [ s k +1 ,η k ] ∗ K h 2 || L 2 = O ( κ k ) . Therefore,

Now, we need to get a bound of the magnitude of

in order to get an upper for I 3 , 2 . This is done similarly to I 2 . We consider the random variable

In order to use Lemma 3, we observe that since || F t,h 2 -f η k +1 ∗ K h 2 || L 2 ≥ 0 ,

Therefore, using Lemma 3 and that E ( || F t,h 2 -f η k +1 ∗ K h 2 || L 2 ) = O ( κ -p 2 r k ) by (26), we get that

Thus, by Assumption 4 and above,

Consequently, I 3 has been bounded, and we only need to go over the term I 4 , to finalize the analysis for U 1 . To analyze I 4 , we observe that

where,

Now, we explore each of the terms I 4 , 1 , I 4 , 2 , and I 4 , 3 . First, I 4 , 1 can be bounded as follows, we add and subtract ⟨ F t,h 1 , F t,h 2 ⟩ L 2 , to get

which, by Hölder's inequality, is bounded by

since || F t,h 1 -F t,h 2 || L 2 = O ( | κ -̂ κ | 1 2 κ p 2 r + 1 2 k ) = O p ( 1 κ p 2 r + 1 2 k (( log( T ) ∆ ) 2 r 2 r + p + T p 2 r + p log( T ) κ ∆ ) 1 2 ) , for any t , see Remark 2 for more detail. Similarly, for I 4 , 2 , we have that adding and subtracting 2 ⟨ F t,h 1 , F ( s k ,η k ] ,h 2 ⟩ L 2 ,

and by Hölder's inequality and Remark 2, it is bounded by

Finally, for I 4 , 3 , we notice that, adding and subtracting ⟨ F ( s k , ̂ η k ] ,h 1 , F ( s k ,η k ] ,h 2 ⟩ L 2 , it is written as,

which, by Hölder's inequality and Remark 2, is bounded by

Then, by above and Assumption 4, we conclude

From (29), (30) and (31), we find that U 1 has the following upper bound,

Now, making an analogous analysis, we have that U 2 is upper bounded by,

In fact, we observe that

where,

Then, I 5 is bounded as follows

where,

The term I 5 , 1 = O p ( ˜ r 1 T log( T ) κ p r ) , using Remark 1. Even more, by Assumption 4, we get p

For the term I 5 , 2 , by Cauchy Schwartz inequality,

for any t ∈ ( η k , η k + ˜ r ] . By Remark 1, we have that || f ( η k ,e k ] ∗ K h 2 -F ( η k ,e k ] ,h 2 || L 2 = O p ( 1 √ T √ log( T ) κ p r k ) . Therefore,

Now, similarly to the bound for I 2 , we consider the random variable

In order to use Lemma 3, we observe

T

(

1

√

I

5

,

2

O

p

√

log(

)

)(

T

κ

p

r

To analyze I 6 , we observe that

where,

√

-

k

p

r

p

r

k

+2

-

p

2

r

k

O

p

(

rκ

˜

(log(

rκ

˜

) + 1)) +

O

p

(

κ

)

)

=

o

p

(

rκ

˜

p

r

k

+2

)

.

(35)

Then we bound each of these terms. First, we rewrite I 6 , 1 , as

which, by Hölder's inequality, is bounded by

since || F t,κ -F t, ̂ κ || 2 L 2 = O ( | κ -̂ κ | 1 2 κ p 2 r + 1 2 k ) = O p ( 1 κ p 2 r + 1 2 k ( log( T ) ∆ ) 2 r 2 r + p + T p 2 r + p log( T ) κ ∆ ) 1 2 )) , for any t , see Remark 2 for more detail. Similarly, for I 6 , 2 we have,

and by Hölder's inequality and Remark 2, it is bounded by

Now for I 6 , 3 , we write it as

which, by Hölder's inequality and Remark 2, is bounded by

By above and Assumption 4, we conclude

From, (34), (35) and (36), we get that U 2 is bounded by

Therefore, from (32) and (33)

Step 4: Combination of all the steps above. Finally, combining (24), (27) and (37), uniformly for any ˜ r ≥ 1 κ p r +2 k we have that

which implies, and complete the proofs of a . 1 and b . 1 .

Limiting distributions. For any k ∈ { 1 , . . . , K } , due to the uniform tightness of ˜ rκ p r +2 k , (24) and (37), as T →∞

satisfies

Therefore, it is sufficient to find the limiting distributions of Q ∗ ( η k + ˜ r ) -Q ∗ ( η k ) when T →∞ . Non-vanishing regime. Observe that for ˜ r > 0 , we have that when T →∞ ,

When ˜ r < 0 and T →∞ , we have that

Therefore, using Slutsky's theorem and the Argmax (or Argmin) continuous mapping theorem (see 3.2.2 Theorem van der Vaart and Wellner, 1996) we conclude

Vanishing regime. Vanishing regime. Let m = κ -2 -p r k , and we have that m → ∞ as T → ∞ . Observe that for ˜ r > 0 , we have that

Following the Central Limit Theorem for α -mixing, see Lemma 4, we get

where B ( ˜ r ) is a standard Brownian motion and ˜ σ ( k ) is the long-run variance given in (8). Therefore, it holds that when T →∞

Similarly, for ˜ r < 0 , we have that when n →∞

Then, using Slutsky's theorem and the Argmax (or Argmin) continuous mapping theorem (see 3.2.2 Theorem in van der Vaart & Wellner (1996)), and the fact that, E ( || f ( s k ,η k ] ∗ K h 2 -f ( η k ,e k ] ∗ K h 2 || 2 L 2 ) = O ( κ 2 k ) , we conclude that

which completes the proof of b . 2

F Proof of Theorem 3

In this section, we present the proof of theorem Theorem 3.

Proof of Theorem 3. First, letting h 2 = c κ κ 1 r k and R = O ( T 2 r 2 r + p κ p 2 r + 3 2 k ) , we consider

We will show that

in order to conclude the result. For (i), we use a 2 -b 2 = ( a + b )( a -b ) , to write,

Then, we bound each of the terms I 1 and I 2 . For I 1 , we observe that,

Then, adding and subtracting, ̂ κ p 2 r -1 k 〈 F h 1 ,i -f i ∗ K h 1 , ( f η k -f η k +1 ) ∗ K h 2 〉 L 2 and

we get that,

which can be written as,

Now, we bound the expression above. For this purpose, by triangle inequality, it is enough to bound each of the terms above. Then, we use Hölder's inequality. First,

Then, using (54), we have that | κ p 2 r -1 k -̂ κ p 2 r -1 k | = O p ( 1 κ 2 -p 2 r k (( log( T ) ∆ ) 2 r 2 r + p + T p 2 r + p log( T ) κ ∆ ) 1 2 ) , and using Remark 2, it follows that

and,

So that,

Now, in a similar way, we observe that

where equality is followed by noticing that

and then using Remark 2 and Assumption 1. Finally,

where equality is followed by Remark 2, Assumption 1 and Minkowski's inequality. Therefore,

k 2 2 ̂ k ̂ k +1 1 L 2

Then, as before, we bound each of the terms above using Hölder's inequality. We start with the term

where the second inequality is followed by (41). Similarly,

where the second inequality is followed by the (41). Finally, the term

was previously bounded by,

Therefore,

In consequences,

In order to conclude (i), we notice that by Assumption 4 and that S = O ( T -p 2 r + p κ p 2 r + 3 2 k ) , which implies,

Now, we are going to see that ∣ ∣ ∣ ˘ σ 2 ∞ ( k ) -˜ σ 2 ∞ ( k ) ∣ ∣ ∣ P -→ 0 , T →∞ . To this end, we will show that the estimator is asymptotically unbiased, and its variance → 0 as T →∞ . First, we notice that, by Hölder's inequality and Minkowsky's inequality,

Now, we analyze the Bias. We observe that,

and,

so that, the bias has the following form,

Now, we show that each of the above terms vanishes as T →∞ . We have that, by condition (2) and covariance inequality

where α l is the mixing coefficient. Then,

by condition (2), choice of S and Assumption 4. Therefore, we conclude that the Bias vanishes as T →∞ . To analyze the Variance, we observe that, if Y r = 1 S ( ∑ i ∈S r ˘ Y i ) 2

where, ˜ α l are the mixing coefficients of { Y r } r ∈ Z , which is bounded by the mixing coefficient α l . From here, we conclude the result (ii).

G Large probability events

In this section, we deal with all the large probability events that occurred in the proof of Theorem 1. Recall that, for any ( s, e ] ⊆ (0 , T ] ,

Proposition 1. For any x ,

Proof. We have that the random variables { Z t = K h ( x -X t ) } T t =1 satisfies

and,

Moreover, let

We observe that,

making µ = x -z h , the last inequality is equal to

Then, by proposition 2.5 on Fan & Yao (2008), | cov ( Z 1 , Z 1 + t ) | ≤ Cα ( t ) 1 h 2 p C 2 K . On the other hand,

Since by assumption Assumption 1 b we have that || g t || L ∞ < ∞ , and

we obtain that | cov ( Z 1 , Z 1+ t ) | . Therefore, ∑ 1 h -1 t =1 | cov ( Z 1 , Z t +1 ) | ≤ C 1 h and, using the mixing condition bound, inequality (2),

where the last inequity is followed by the fact e -x < 1 x for x > -1 . In consequence,

Then, by Bernstein inequality for mixing dependence, see Merlevède et al. (2009) for more details, letting

we get that,

in consequence,

Since kh p ≥ log ( T ) if k > ρ , and log 2 ( k ) = O ( k ) ,

It follows that,

Proposition 2. Define the events

and,

Then where R is a positive constant.

Proof. First, we notice that

Now we will bound each of the terms I 1 , I 2 . For I 1 , we consider

with ∪ x i ∈{ x 1 ,...,x ( R √ T/ √ h p h ) p } Rec ( x i , √ h p h √ T ) ⊃ D , where D is the support of K and Rec ( x i , √ h p h √ T ) are boxes centered at x i of size √ h p h √ T and R is the size of the boxe containing D . Then by Proposition 1, for any x i ,

by an union bound argument,

Let I 1 , 1 = { max ρ ⩽ k ≤ T -˜ r sup x ∈ A ∣ ∣ ∣ 1 √ k ∑ ˜ r + k t = ˜ r +1 ( K h ( x -X t ) -∫ K h ( x -z ) dF t ( z ) )∣ ∣ ∣ } . For any x ∈ R p , there exist x i ∈ A such that

The term I 1 , 2 is bounded as followed.

For the term I 1 , 3 , since the random variables {K h ( x -X t ) } T t =1 have bounded expected value for any x ∈ R p

Thus,

From here,

Finally, we analyze the term I 2 . By the adaptive assumption, the following is satisfied,

We conclude the bound for event A 2 . We conclude the bound for event A 2 . Next, to derive the bound for event A 1 , by definition of ˜ F s,e t,h and ˜ f s,e t , we have that

Then, we observe that,

Therefore,

Finally, letting λ = 2 C 1 √ log T h p + 2 C 2 √ p √ h p +2 C 2 √ Th r , we get that

where the last inequality follows from above. This concludes the bound for A 1 .

Remark 1. On the events ( A 1 ) c and, ( A 2 ) c , by Assumption 1, we have that

where ˜ C X is the volume of the set X . Moreover, using inequality (47) , we have that

H α -mixing condition

A process ( X t , t ∈ Z ) is said to be α -mixing if

The strong mixing, or α -mixing coefficient between two σ -fields A and B is defined as

Suppose X and Y are two random variables. Then for positive numbers p -1 + q -1 + r -1 = 1 , it holds that

Let { Z t } ∞ t = -∞ be a stationary time series vectors. Denote the alpha mixing coefficients of k to be

Note that the definition is independent of t .

H.1 Maximal Inequality

The unstationary version of the following lemma is in Lemma B.5. of Kirch (2006).

∞

Lemma 1. Suppose { y i } i =1 is a stationary alpha-mixing time series with mixing coefficient α ( k ) and that E ( y i ) = 0 . Suppose that there exists δ, ∆ > 0 such that

where D only depends on δ and the joint distribution of { y i } ∞ i =1 .

Proof. This is Lemma B.8. of Kirch (2006).

Lemma 2. Suppose that there exists δ, ∆ > 0 such that

and

Then it holds that for any d > 0 , 0 < ν < 1 and x > 0 ,

and

Then where C is some constant.

Proof. Let

Then Lemma 1 implies that

Therefore it holds that

Observe that

Therefore which gives

Lemma 3. Let ν > 0 be given. Under the same assumptions as in Lemma 1, for any 0 < a < 1 it holds that

where C is some absolute constant.

Proof. Let s ∈ Z + and T s = [ 2 s /ν, 2 s +1 /ν ] . By Lemma 3, for all x ≥ 1 ,

Therefore by a union bound, for any 0 < a < 1 ,

For any r ∈ [ 2 s /ν, 2 s +1 /ν ] , s ≤ log( rν ) / log(2) , and therefore

Equation (2) directly gives

H.2 Central Limit theorem

Below is the central limit theorem for α -mixing random variable. We refer to Doukhan (1994) for more details.

Lemma4. Let { Z t } be a centred α -mixing stationary time series. Suppose for the mixing coefficients and moments, for some δ > 0 it holds

where convergence is in Skorohod topology and W ( t ) is the standard Brownian motion on [0 , 1] .

I Additional Technical Results

Lemma 5. Let J be defined as in Definition 2 and suppose Assumption 1 e holds. Then for each change point η k there exists a seeded interval I k = ( s k , e k ] such that

I k contains exactly one change point η k ;

max { η k -s k , e k -η k } ≤ 9 10 ∆ ;

b. min { η k -s k , e k -η k } ≥ 1 16 ∆ ; and

Proof. These are the desired properties of seeded intervals by construction. The proof is the same as theorem 3 of Kovács et al. (2020) and is provided here for completeness.

Let k ∈ { 1 , ..., K } , where K is from definition 2. By construction of seeded intervals, we have that in J k , we can find an interval ( ⌊ ( i -1) T 2 -k ⌋ , ⌈ ( i -1) T 2 -k + T 2 -k +1 ⌉ ] , for some i ∈ 1 , ..., 2 k -1 , such that

and

where l k = T 2 -k +1 , is the size of each interval in J k . By the choice of K , there exist k , such that l k = 9 10 ∆ , from where the claim is followed.

Lemma6. Let { X i } T i =1 be random grid points sampled from a common density function f t : R p → R , satisfying Assumption 1a and b . Under Assumption (2) , the density estimator of the sampling distribution µ ,

The verification of these bounds can be found in many places in the literature. See for example Yu (1993) and Tsybakov (2009).

Remark 2. Even more, by Assumption 1,

with high probability. Therefore, given that

and (5) , by triangle inequality, (53) , and Lemma 5, we have that

From here, and Assumption 2, if h 1 = O ( κ 1 r ) and h 2 = O ( ̂ κ 1 r ) , we conclude that

satisfies,

In fact,

Now, we analyze the two following terms,

and

For I 1 , letting u = x -X t h 1 , we have that

and, letting v = x -X t h 2 , we have that

Therefore, by Assumption 2 and the Mean Value Theorem,

Similarly, we have,

I.1 Multivariate change point detection lemmas

We present some technical results corresponding to the generalization of the univariate CUSUM to the Multivariate case. For more details, we refer the interested readers to Padilla et al. (2021) and Wang et al. (2020).

Let { X t } T t =1 ⊂ R p a process with unknown densities { f t } T t =1 .

Assumption 5. We assume there exist { η k } K k =1 ⊂ { 2 , ..., T } with 1 = η 0 < η 1 < ... < η k ≤ T < η K +1 = T +1 , such that

̸

In the rest of this section, we use the notation

for all 0 ≤ s < t < e ≤ T and x ∈ R p .

Lemma 7. If [ s, e ] contain two and only two change points η r and η r +1 , then

Assume

Proof. This is Lemma 15 in Wang et al. (2020). Consider the sequence { g t } e t = s +1 be such that

For any t ≥ η k ,

So for t ≥ η k , || ˜ f s,e t -˜ g s,e t || L 2 ≤ √ η k -sκ k . Since sup s ≤ t ≤ e || ˜ f s,e t || L 2 = max { || ˜ f s,e η k || L 2 , || ˜ f s,e η k +1 || L 2 } , and that

where the last inequality follows form the fact that g t has only one change point in [ s, e ] .

Lemma 8. Suppose e -s ≤ C R ∆ , where C R > 0 is an absolute constant, and that

Denote

Then for any k -1 ≤ p ≤ k + q , it holds that

Proof. This is Lemma 18 in Wang et al. (2020). Since e -s ≤ C R ∆ , the interval [ s, e ] contains at most C R +1 change points. Observe that

where ∣ ∣ ∣ p 1 -p 2 ∣ ∣ ∣ ≤ C R +1 for any η p 1 , η p 2 ∈ [ s, e ] is used in the last inequality.

Lemma 9. Let ( s, e ) ⊂ (0 , n ) contains two or more change points such that

If η k -s ≤ c 1 ∆ , for c 1 > 0 , then √

Proof. This is Lemma 20 in Wang et al. (2020). Consider the sequence { g t } e t = s +1 be such that

For any t ≥ η r , it holds that

Thus,

where the first inequality follows from the observation that the first change point of g t in ( s, e ) is at η k +1 .

Lemma 10. Under Assumption 5, for any interval ( s, e ) ⊂ (0 , T ) satisfying

Let

Then b ∈ { η 1 , . . . , η K } . For any fixed z ∈ R p , if ˜ f ( s,e ] t ( z ) > 0 for some t ∈ ( s, e ) , then ˜ f ( s,e ] t ( z ) is either strictly monotonic or decreases and then increases within each of the interval ( s, η k ) , ( η k , η k +1 ) , . . . , ( η k + q , e ) .

Proof. We prove this by contradiction. Assume that b / ∈ { η 1 , . . . , η K } . Let z 1 ∈ arg max x ∈ R p ∣ ∣ ∣ ¯ f s,e b ( x ) ∣ ∣ ∣ . Due to the definition of b , we have

It is easy to see that the collection of change points { f t ( z 1 ) } e t = s +1 is a subset of the change points of { f } e t = s +1 . Then, from Lemma 2 . 2 in Venkatraman (1992) that

which is a contradiction.

Recall that in Algorithm 1, when searching for change points in the interval ( s, e ) , we actually restrict to values t ∈ ( s + ρ, e -ρ ) . We now show that for intervals satisfying condition SE from Lemma 1, taking the maximum of the CUSUM statistic over ( s + ρ, e -ρ ) is equivalent to searching on ( s, e ) , when there are change points in ( s + ρ, e -ρ ) .

Lemma 11. Let z 0 ∈ R p , ( s, e ) ⊂ (0 , T ) . Suppose that there exists a true change point η k ∈ ( s, e ) such that and

where c 1 > 0 is a sufficiently small constant. In addition, assume that

where c 2 > 0 is a sufficiently small constant. Then for any d ∈ ( s, e ) satisfying

it holds that

where c > 0 is a sufficiently small constant, depending on all the other absolute constants.

Proof. Without loss of generality, we assume that d ≥ η k and ˜ f η k ( z 0 ) ≥ 0 . Following the arguments in Lemma 2 . 6 in Venkatraman (1992), it suffices to consider two cases: (i) η k +1 > e and (ii) η k +1 ≤ e Case (i) . Note that

and

Therefore, it follows from (56) that

The inequality follows from the following arguments. Let u = η k -s, v = e -η k and w = d -η k . Then

The numerator of the above equals

as long as

Case (ii) . Let g = c 1 ∆ / 16 . We can write

where

and b = ˜ f ( s,e ] η k + g ( z 0 ) -˜ f ( s,e ] η k ( z 0 ) . To ease notation, let d -η k = l ≤ g/ 2 , N 1 = η k -s and N 2 = e -η k -g . We have

where

Next, we notice that g -l ≥ c 1 ∆ / 32 . It holds that

where c 1 l > 0 is a sufficiently small constant depending on c 1 . As for E 2 l , due to (59), we have

As for E 3 l , we have

where the second inequality follows from (57) and the third inequality follows from (58), c 3 l, 1 , c 3 l, 2 > 0 are sufficiently small constants, depending on all the other absolute constants. Combining (62), (63), (64) and (65), we have

where c > 0 is a sufficiently small constant. In view of (61) and (67), the proof is complete.

Consider the following events

Lemma 12. Suppose Assumption 5 holds. Let [ s, e ] be an subinterval of [1 , T ] with e -s ≤ C R ∆ , and contain at least one change point η r with min { η r -s, e -η r } ≥ cT for some constant c > 0 . Let κ s,e max = max { κ p : min { η p -s, e -η p } ≥ cT } . Let

For some c 1 > 0 , λ > 0 and δ > 0 , suppose that the following events hold

and that

If there exists a sufficiently small c 2 > 0 such that

then there exists a change point η k ∈ ( s, e ) such that

where c 3 is some sufficiently small constant independent of T .

Proof. Let z 1 ∈ arg max z ∈ R p ∣ ∣ ∣ ˜ f ( s,e ] b ( z ) ∣ ∣ ∣ . Without loss of generality, assume that ˜ f ( s,e ] b ( z 1 ) > 0 and that ˜ f ( s,e ] b ( z 1 ) as a function of t is locally decreasing at b . Observe that there has to be a change point η k ∈ ( s, b ) , or otherwise ˜ f ( s,e ] b ( z 1 ) > 0 implies that ˜ f ( s,e ] t ( z 1 ) is decreasing, as a consequence of Lemma 10. Thus, there exists a change point η k ∈ ( s, b ) satisfying that

where the second inequality follows from Lemma 10, the third because of the good event A , and fourth inequalities by (70) and Assumption 1, and c > 0 is an absolute constant. Observe that ( s, e ) has to contain at least one change point or otherwise sup z ∈ R ∣ ∣ ∣ ˜ f ( s,e ] η k ( z ) ∣ ∣ ∣ = 0 which contradicts (72).

Step 1 . In this step, we are to show that

Suppose that η k is the only change point in ( s, e ) . Then (73) must hold or otherwise it follows from (14) that √

which contradicts (72).

Suppose ( s, e ) contains at least two change points. Then arguing by contradiction, if η k -s < min { 1 , c 2 1 } ∆ / 16 , it must be the cast that η k is the left most change point in ( s, e ) . Therefore

where the first inequality follows from Lemma 9, the second follows from the assumption of η k -s , the third from the definition of the event A and the last from (70) and Assumption 1. The last display contradicts (72), thus (73) must hold.

Step 2 . Let

It follows from Lemma 11 that there exits d ∈ ( η k , η k + c 1 ∆ / 32 ) such that

We claim that b ∈ ( η k , d ) ⊂ ( η k , η k + c 1 ∆ / 16 ) . By contradiction, suppose that b ≥ d . Then

where the first inequality follows from Lemma 10, the second follows from (78) and the third follows from the definition of the event A . Note that (79) is a contradiction to the bound in (72), therefore we have b ∈ ( η k , η k + c 1 ∆ / 32 ) .

Step 3 . Let

and

By the definition of b , it holds that

where the operator P s,e ( · ) is defined in Lemma 21 in Wang et al. (2020). For the sake of contradiction, throughout the rest of this argument suppose that, for some sufficiently large constant C 3 > 0 to be specified,

We will show that this leads to the bound

which is a contradiction. If we can show that

then (81) holds. To derive (82) from (80), we first note that min { e -η k , η k -s } ≥ min { 1 , c 2 1 } ∆ / 16 and that ∣ ∣ ∣ b -η k ∣ ∣ ∣ ≤ c 1 ∆ / 32 implies that

As for the right-hand side of (82), we have

On the event A∩B , we are to use Lemma 11. Note that (57) holds due to the fact that here we have

where the first inequality follows from the fact that η k is a true change point, the second inequality holds due to the event A , the third inequality follows from (70), and the final inequality follows from (71). Towards this end, it follows from Lemma 11 that

Combining (84), (86) and (87), we have

The left-hand side of (82) can be decomposed as follows.

As for the term (I), we have

As for the term (II.1), we have

In addition, it holds that

where the inequality is followed by Lemma 8. Combining with the good events,

As for the term (II.2), it holds that

As for the term (II.3), it holds that

Therefore, combining (94), (96), (97), (88), (89) and (93), we have that (82) holds if

The second inequality holds due to Assumption 3, the third inequality holds due to (79) and the first inequality is a consequence of the third inequality and Assumption 3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of localization of multiple change points for offline multivariate time series. A non-parametric kernel-based CUSUM statistics is used together with the SBS algorithm. Moreover, a two-step estimation procedure is proposed where the initial estimate is further modified to the final estimate. The consistency result and the limiting distribution of the change point estimators are proved.\n\nSTRENGTHS:\nThe main strength of the paper is its theoretical findings. Although the SBS and the kernel CUSUM statistics are both well-known, the theoretical results on the consistency and especially the limiting distribution under the setting of multivariate time series seem to be novel.\n\nWEAKNESSES:\nThe presentation can still be improved, and the numerical results is a bit limited.\n\nQUESTIONS:\nIn the current theoretical results, the consistency result is proved for the initial estimate, while the limiting distribution is provided for the final estimate. It would be better to also present the consistency result for the final estimate \\tilde{\\eta}, and to elaborate more on whether or not we still have such limiting distribution for the initial estimate. \nAnd it would be better if the author could illustrate in the numerical results the advantage of the second step (refined estimators), i.e., to show the improvement in estimation accuracy after using the refined estimator.\n\nIn the numerical results, the standard deviation of the misestimation rate (fig1) should be provided. The report of std can be very helpful here for comparing different methods since the average misestimation rate of MNSBS (blue) seems to be larger than or only slightly smaller than the other four baseline methods in various scenarios.\n\nIn the final estimate, line 126 on page 4, there is a typo in the subscript of F_{s_k,\\eta_k}, \\eta_k should not appear here since it is the unknown true change-point.\n\nIt would be better if the author could provide a pictorial illustration of the SBS algorithm.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes algorithm to localize the multiple change points in nonparametric, short-term dependent time series. Assumptions required on the time series are certain mixing conditions and the smoothness in terms of density. The core idea of the algorithm is based on CUSUM and seeded binary segmentation. A two stage estimator is proposed, both of which are consistent, with the refined one in the second step achieving minimax rate under certain smoothness cases. Limiting distribution of the estimators are also given, which allows inference on the change points. Numerical results demonstrates the potential of the proposed method.\n\nSTRENGTHS:\nThis is a solid work with nice theoretical results. The problem tacked is a difficult one: localization of multiple change points in a nonparametric, locally dependent time series. The paper establishes not only consistent results, but also gives limiting distributions which can be beneficial for inference.\n\nWEAKNESSES:\nAlthough this is a solid work with nice theoretical results, I feel the related literature is not sufficiently reviewed which are crucial for determining the significance and novelty of this work. I only found in the last paragraph of page 1 a short summary of the existing literature, which I pasted below and raised some of my questions.\n\n\"Firstly, to the best of our knowledge, temporal dependence, which commonly appears in time series, has not been considered.\"\n-- I actually don't think this is true, as far as I know, there are a lot of existing work considering temporal dependence of time series (although I agree most of the exiting change point literature still assumes the iid setting). For example, I randomly searched for \"change point dependent process\" and there seems a bulk of related works, e.g., [1][2]. I believe the authors probably know better than I about these papers, and I wonder whether they could add some clarification on the difference between their setup and the one in this work.\n\n\"Secondly, there is no localization consistency result for data with the underlying densities being Hölder smooth with arbitrary degree of smoothness.\" -- it seems this is comparing to the work of Padilla et al. (2021) which only focuses on Lipschitz smooth densities, is my understanding correct?\n\n\"Lastly and most importantly, the limiting distributions of change point estimators and the asymptotic inference for change points have not been well studied.\" -- Do the authors specifically here refer to the papers on nonparametric, multiple change point literature? I believe the limiting distribution of change point estimators is one of the core questions in change point problems, and I am a bit surprised that the existing papers do not study them. Do the authors mean that most papers have been focusing on consistency results instead of the limiting distributions of change point estimators?\n\n[1] Ray, Bonnie K., and Ruey S. Tsay. \"Bayesian methods for change‐point detection in long‐range dependent processes.\" Journal of Time Series Analysis 23.6 (2002): 687-705.\n[2] Dehling, Herold, Aeneas Rooch, and Murad S. Taqqu. \"Non‐parametric change‐point tests for long‐range dependent data.\" Scandinavian Journal of Statistics 40.1 (2013): 153-173.\n\nQUESTIONS:\nMy questions have been raised in the \"weakness\" section.\n\nLIMITATIONS:\nWhile this work has nice and solid theoretical results, I wonder if practitioners can truly find the method useful, both due to the high computational complexity, and the somewhat complicated limiting distributions (which might prevent the full deployment of this methods' inference capability).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission studies offline multivariate non-parametric change point detection. \nThe submission proposes a method for this task by combining 1) CUSUM estimator/statistic with 2) seeded intervals and 3) a refining procedure. The proposed estimator has 1) an improved error bound with weaker assumptions and 2) a limiting distribution for inference.  \nBoth empirical and theoretical results are provided supporting the advantages of the proposed method.\n\nSTRENGTHS:\nThe submission extends the existing theoretical results from the setting with Lipschitz smooth densities and temporal independence to the case with Hölder smooth densities with $\\alpha$ mixing, and also with a limiting distribution for the non-parametric case. \n\nThe writing is clear, with notations well defined.\n\nWEAKNESSES:\nThe submission is not presented in a very motivating way. For example, SBS appears abruptly without the background or reasoning. The methodology is also presented without explaining the motivations. \n\nThe methodological contribution of the proposed method is not clear. Combining CUSUM  and SBS is nothing new and also mentioned in the original SBS paper. As a result, the refining step in the proposed method is important for the methodological novelty of the submission. However, unfortunately, there are not enough discussions to motivate this refining step and emphasizing the novelty of this refining step. \n\nThe experiments can also be improved. The figures are hard to read since it is hard to match the competing methods to different colors. The advantages of the prosed method are not clearly demonstrated: some bars are very close to each other without showing significant advantages of the proposed method.\n\nQUESTIONS:\nThe theoretical results seem to be a good improvement to the existing theory. However, I did not dig deep enough into the proof to evaluate the technical novelty. It is also unclear how much novelty and significance there is in the methodology. \n\n1. Would it be possible for authors to compare to an ablation method, which just combines CUSUM with SBS without the refining step? Or, is this ablation just the SBS method compared in the experiments? If so,  would it be authors to explain why this method (which is quite close to the proposed one just without the refining step) performs so badly? \n\n2. Would it possible for authors to provide stds of the experiment results to make sure that the proposed method really provides significant performance improvement?   \n\nAs a result, this submission is really on the borderline to me. I tend to accept the submission for the theoretical contributions.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider offline change point detection for multi-variate data where there could be multiple change points (specifically changes in marginal distributions from one time step to the next).  The marginal densities of the underlying generative model are assumed to be smooth (specifically Hölder continuous, which includes Lipschitz continuous functions as a special case).  Building on a procedure using binary-segmentation search with kernel density estimation and consistency (Padilla et al (2021)), the authors analogously show consistency of change-point estimators for time-dependent data.   Furthermore, the authors derive limiting distributions of the change point estimators, both for the situation where the minimal jump size vanishes and for the situation where it remains constant.\n\nSTRENGTHS:\n- The authors study a challenging problem, non-parametric estimation of multiple changepoints.  Many prior works considered independent data and univariate, as well as stronger assumptions on the generative distributions.  \n\n- The authors prove consistency and analyze limiting distributions of the estimated change points.  (also this is with less knowledge (such as of optimal bandwidths) than Padilla et al (2021))\n\n- The authors include experiments and show its performance against baselines (designed for independent data).\n\nWEAKNESSES:\nMy main concerns regard writing/presentation and discussion of related works.  I list a number of specific points below (many of which, by themselves, are fairly minor; list is not exhaustive).    \n\n#### Writing/presentation\n- There is no discussion on motivating the time-series model class considered ($\\alpha $-mixing sequences of random vectors with unknown marginal distributions), or even mention what types of common parametric classes belong to this class.   \n    -  The property of $\\alpha $-mixing is never explained formally let alone any discussion on what that implies at a high-level for the time-series.  There is a large literature on parametric time-series models – it would be valuable to mention example model classes that satisfy the assumptions here.\n    - What types of potential applications could this benefit (the time-series of which could plausibly be modeled with this class of models)?  Lines 20-24 mention application areas with time-series data, which is fine, but there should be more of a connection made between (some of) those applications and the specific problem  considered.  \n\n- Algorithm 1 and Section 3\n    - Provide discussion in the main text about the steps of Alg. 1 and intuition behind the design.  \n    - SBS is mentioned but is not described (a brief high-level description should suffice)\n    - There is notation used in the algorithm that is not mentioned in Section 3, explain what those are.  \n    - Include discussion about kernel properties in Section 3. Alg 1 uses kernel estimators implicitly in the line with $\\tilde{F}$, but up to this point no $\\mathcal{K}$ is defined or taken as input, or described in Section 3.\n    - (minor) Def 1 text $\\mathcal{K}$ has not been described (or listed in the def. set up)\n    - (4) and (5) give some verbal description of what this is doing (i.e. what is the intuition behind how you are improving the estimates, possibly with a line or two about the way(s) in which the MNSBS might give poor(er) estimates that leads to this design).  Presumably it has to do with that the choice of interval locations and sizing in MNSBS was using a simple pattern (oblivious to where the estimated change points would be).  Now you have ‘naturally defined’ intervals between the estimated change points (so size and location of the intervals are adapted to the initial estimates).\n    - line 124-125 talk about the choice of weights 9/10 and 1/10.  \n    - line 125 ‘an estimator of $\\kappa_k$’ I would suggest to add verbal descriptors for notation to remind the reader, such as ‘an estimator of the jump size $\\kappa_k$ at the $k$th change point’ and similarly for other notation.  Also, several lines later (to explain differences with prior work) there is mention that it corresponds to the ‘optimal bandwidth’ and ‘ we use $\\hat{\\kappa}_k$ as bandwidth for the kernel density estimator’ though in line 127 the bandwidth used is $h_1$ which depends on, but is not set equal to,  $\\hat{\\kappa}_k$.  A clarification of that part, and a couple sentences shortly after Definition 1 about the bandwidth $h$ to lead the reader to anticipate why $h_1$ in line 127 is good would help.\n    - while bandwidths are mentioned earlier (Def. 1 for instance) there is no discussion on bandwidth optimality.  Padilla et al (2021) have nice discussions regarding the choice of the bandwidth; I would suggest summarizing the key points to give intuition to the reader.\n    - line 132 ‘smaller interval’ – provide brief intuition about why the intervals here would be smaller than the\n\n\n#### Experiments\n- Include plots of the realizations of the time-series (to give intuition for how hard/easy the task is for those generative models)\n- Report the run-times for the proposed method and baselines.  With that also briefly describe the platform.\n- It would also be valuable to run experiments with independent data, to see whether the proposed method suffers compared to baselines designed for that setting specifically.  \n\n#### Related works\n- “Random Forests for Change Point Detection” by Londschien et al https://arxiv.org/abs/2205.04997  a multivariate nonparametric\nchange point detection methods for independent data (with an R package available) – in their experiments, their method generally was similar or better than ECP (the best baseline in your experiments) and had subquadratic time complexity\n\n- There are works non-parametrically estimating the location and size of structural breaks in non-parametric time-series regression models.  “Estimating change points in nonparametric time series regression models”  by Mohr and Selk (2020) https://doi.org/10.1007/s00362-020-01162-8,  “  Nonparametric inference on structural breaks “ by Delgado and Hidalgo (2000) https://doi.org/10.1016/S0304-4076(99)00052-4 among others.  Given some (at least superficial) relation in the problem considered and some methodological components collectively in that body of work and the present work, which extends methods and results for non-parametric changepoint detection for independent data (esp. Padilla et al (2021)) to allowing for dependencies, I think some discussion on the how the problems, methods, and results of changepoint detection for non-parametric   time-series regression relate would be helpful.  Xu et al (2022) is mentioned in lines 181-182, though what the paper actually studied and how it related was not described.  \n\n- A (brief) discussion of similarities and differences between the problem considered here and that of (non-parametric) online change point detection would be good. \n- One of the main contributions is characterizing the limiting distributions for multiple change point estimates in this non-parametric setting.  To my knowledge, that is novel for the specific problem considered.   However, similarities/differences in analyses and results (class of distributions that the limiting distribution belongs to) for past works that have investigated  limiting distributions of change point estimates in different by related settings are not well-discussed.  Unless I overlooked it, the only other work mentioned in the context of identifying limiting distribution of change point estimators is Xu et al (2022) in lines 181-182 and that was brief and implicit.\n    - There are prior works that studied limit distributions for non-parametric estimators for single change point detection with independent data (as well as works in the parametric setting), although from what I can tell under simpler changes (such as changes in the mean).  For example, “The Asymptotic Behavior of Some Nonparametric Change-Point Estimators” by Dumbgen (1991)  and Horváth and Kokoszka (1997) \"The effect of long-range dependence on change-point estimators.\"   Also “Optimal change-point estimation in time series” by Chan et al. 2021 for limiting distribution of single time-series change point (Bayes) estimators.   There is a recent work “On multiple structural breaks in distribution: An empirical characteristic function approach” by Fu et al (2022) that analyzes the limiting distributions for estimates of multiple change point. Also potentially relevant is “The asymptotic distribution of CUSUM estimator based on $\\alpha$-mixing sequences” by Gao et al (2022), though the analysis is for univariate non-negative $\\alpha$-mixing non-negative random variables.\n    - Limiting distributions of change point estimates have been studied in the parametric setting.  While those results do not diminish the significance of the results in this paper, there should be some mention and preferably a discussion on similarities/differences between the derived distributions.\n\nQUESTIONS:\n#### Questions\n- Assumption 3 and Theorem 1.  The $\\gamma_T$ does not explicitly show up in the Theorem 1 statement – which constants depend on $\\gamma_T$?  Given the assumption statement is only “arbitrarily slow diverging sequence” that seems to me impressive though almost too mild for anything other than statements about the limit $T \\to \\infty$ \n\n- For the theorems, is $\\Delta$ *required* to be growing as a function of $T$?  From the assumptions it looks like as long as the jump size is increasing fast enough, we could fix the location of $K$ changepoints and the assumptions would hold.  $\\kappa$ growing quickly should make detecting change points easier, but would there still need to be some minimum value of $\\Delta$?  \n\n- Contribution 1 (lines 84-87) –  The first major contribution claimed is the development of a novel algorithm and statement “To the best of our knowledge, we are the first to innovatively adapt SBS to a multivariate non-parametric change point model”.  From a (perhaps superficial) understanding, the Algorithm 1 proposed here seems to be an incremental adaptation of the procedure proposed in Padilla et al (2021), with the random binary segmentation used in the latter replaced with the recently proposed deterministic binary segmentation method SBS (Kovács et al. (2020)).  Perhaps the authors can add more discussion on why that change is not straightforward. \n\n\n#### Spelling, Grammar, etc.\n- Line 98 ‘A … estimators’\n- Theorem 2 statement – The formal statement could be shortened and I think easier to read if the notation (eg 188-189, 192- (7)) were introduced and discussed (add simple description of $P_k$’s formula when it is introduced) before the formal theorem statement.\n- For theorem 3, maybe $\\max_{1\\leq k \\leq K} \\dots$\n- line 291 ‘additional additional’\n- lines 291-293 – that is great you had further experiments; mention that earlier both in introduction and early on in Section 5.\n\nLIMITATIONS:\nIt is fine (theoretical contribution)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies non-parametric offline change-point detection with the assumption that the probability density functions are Holder continuous on some compact support, and the time series is $\\alpha$-mixing. The authors propose a two-stage algorithm, first roughly divide the time series into different segments, then refine the change-point locations. The asymptotic distribution of the estimated change-points are derived, and as a third step, the author also proposes an algorithm to compute a confidence interval for each change-point. Numerical experiments show the effectiveness of the proposed methods.\n\nSTRENGTHS:\n1. The non-parametric model used in this paper is Holder continuous class and is general. \n2. The asymptotic distribution of the estimated change-point location is derived for the vanishing and non-vanishing regime respectively.\n3. The time series can have temporal dependencies under mixing conditions.\n4. The writing and the structure of the paper is clear.\n\nWEAKNESSES:\n1. The authors claim their model is under mixing conditions but the theory parts' dependency on the $\\alpha$-mixing coefficient $c$ in Assumption 1.e is hardly discussed in the main text.\n\nMinor issue: \n1. The equation under line 126: should remove $\\arg\\min$. \n2. The equation under line 126: why is there $\\eta_k$? Isn't $\\eta_k$ unknown?\n3. Line 131: Why is $\\widetilde \\eta_k$ referred to as the `kernel density estimator'? From previous context it is the change-point estimator.\n\nQUESTIONS:\n1. In Assumption 2.a, is it for any $h>0$?\n2. In Assumption 2.b, the end of line 155, what is $v$ on the exponent? Do you mean $\\nu$?\n3. In Theorem 2.a the non-vanishing regime, do you need $f_{\\eta_k},f_{\\eta_{k+1}}$ to converge in distribution as well or they can be arbitrary as long as the jump size $\\kappa_k$ converges to a constant? How does the $\\alpha$-mixing works here? What is $F_{t,h_2}$ for $t<0$? Why the limiting distribution of $\\widetilde \\eta_k$ depends on $f_0,f_1,f_2$ when $\\eta_k$ is far from 0?\n4. In equation 7, should it be $\\kappa_k^{p/r+2}$ on the numerator or just $\\kappa_k^{p/r-2}$?\n\nLIMITATIONS:\nAs stated in the submitted paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}