Quickest change detection for multi-task problems under unknown parameters

1 INTRODUCTION . Quickest Change Detection ( QCD ) problems arise naturally in settings where a latent state controls observable signals ( Basseville et al. , 1993 ) . In biology , it is applied in genomic sequencing ( Caron et al. , 2012 ) and in reliable healthcare monitoring ( Salem et al. , 2014 ) . In industry , it finds application in faulty machinery detection ( Lu et al. , 2017 ; Martí et al. , 2015 ) and in leak surveillance ( Wang et al. , 2014 ) . It also has environmental applications such as traffic-related pollutants detection ( Carslaw et al. , 2006 ) . Any autonomous agent designed to interact with the world and achieve multiple goals must be able to detect relevant changes in the signals it is sensing in order to adapt its behaviour accordingly . This is particularly true for reinforcement learning based agents in multi-task settings as their policy is conditioned to some task parameter ( Gupta et al. , 2018 ; Teh et al. , 2017 ) . In order for the agent to be truly autonomous , it needs to identify the task at hand according to the environment requirement . For example , a robot built to assist cooks in the kitchen should be able to recognise the task being executed ( chopping vegetables , cutting meat , .. ) without external help to assist them efficiently . Otherwise , the agent requires a higher intelligence ( one of the cooks for instance ) to control it ( by stating the task to be executed ) . In the general case , the current task is unknown and has to be identified sequentially from external sensory signals . The agent must track the changes as quickly as possible to adapt to its environment . However , current solutions for the QCD problem when task parameters are unknown , either do not scale or impose restrictive conditions on the setting . ( i.i.d . observations , exponential family distributions , partial knowledge of the parameters , etc. ) . In this paper , we construct a scalable algorithm with similar performances to optimal solutions . For this purpose , we use the change detection delay under known parameters as a lower bound for the delay in the unknown case . This improves our estimations of the parameters and thus improves our change point detection . We consider the case where the data is generated by some Markovian processes as in reinforcement learning . We assess our algorithm performances on synthetic data generated using distributions parameterised with neural networks in order to match the complexity level of real life applications . We also evaluate our algorithm on standard reinforcement learning environment . 2 QUICKEST CHANGE DETECTION PROBLEMS . Formally , consider a sequence of random observations ( Xt ) where each Xt belongs to some observation space X ( say , an Euclidean space for simplicity ) and is drawn from fθt ( .|Xt−1 ) , where the parameter θt belongs to some task parameter space Θ and { fθ , θ ∈ Θ } is a parametric probability distribution family ( non trivial , in the sense that all fθ are different ) . The main idea is that , at almost all stages , θt+1 = θt but there are some “ change points ” where those two parameters differ . Let us denote by tk the different change points and , with a slight abuse of notations , by θk the different values of the parameters . Formally , the data generating process is therefore : Xt ∼ ∑K k=0 fθk ( .|Xt−1 ) 1tk≤. < tk+1 ( t ) . The overarching objective is to identify as quickly as possible the change points tk and the associated parameters θk , based on the observations ( Xt ) . Typical procedures propose to tackle iteratively the simple change point detection problem , and for this reason we will focus mainly on the simpler setting of a single change point , where K = 2 , t0 = 0 , t1 = λ and t2 = ∞ , where λ is unknown and must be estimated . For a formal description of the model and the different metrics , we will also assume that the parameters ( θ0 , θ1 ) are drawn from some distribution F over Θ . As a consequence , the data generating process we consider boils down to the following system 1 : θ0 , θ1 ∼ F and { Xt+1 ∼ fθ0 ( .|Xt ) if t ≤ λ Xt+1 ∼ fθ1 ( .|Xt ) if t > λ . ( 1 ) 2.1 CRITERIA DEFINITIONS . As mentioned , the objective is to detect change points as quickly as possible while controlling the errors . There exist different metrics to evaluate algorithm performances ; they basically all minimise some delay measurements while keeping the rate of type I errors ( false positive change detection ) under a certain level . We will describe later on different existing definitions of these criteria ( type I error and delay ) . In order to evaluate a probability of error and an expected delay , we obviously need to define relevant probability measures first . Traditionally , there are two antagonistic ways to construct them : the MIN-MAX and the BAYESIAN settings ( Veeravalli & Banerjee , 2014 ) . First , we denote by Pn ( resp . En ) the data probability distribution ( resp . the expectation ) conditioned to the change point happening at λ = n. This last event happens with some probability µ ( λ = n ) – with the notation that bold characters designate random variables– , and we denote by Pµ ( resp . Eµ ) the data probability distribution ( resp . the expectation ) , integrated over µ , i.e. , for any event Ω , it holds that Pµ ( Ω ) = ∑ n µ ( λ = n ) Pn ( Ω ) . In the following , we describe the major existing formulations of the QCD problem where the goal is to identify an optimal stopping time τ : BAYESIAN formulation : In this formulation , the error is the Probability of False Alarms ( PFA ) with respect to Pµ . The delay is evaluated as the Average Detection Delay ( ADD ) with respect to Eµ . PFA ( τ ) = Pµ ( τ < λ ) and ADD ( τ ) = Eµ [ ( τ − λ ) |τ > λ ] In this setting , the goal is to minimise ADD while keeping PFA below a certain level α ( as in Shiryaev formulation ( Shiryaev , 1963 ) ) . Formally , this rewrites into : ( SHIRYAEV ) { να = arg minτ∈∆α ADD ( τ ) ∆α = { τ : PFA ( τ ) < α } ( 2 ) Min-Max formulation : The MIN-MAX formulation disregards prior distribution over the change point . As a consequence , the error is measured as the False Alarm Rate ( FAR ) with respect to the worst case scenario where no change occurs ( P∞ ) . As for the delay , two possibilities are studied : the Worst Average Detection Delay ( WADD ) and the Conditional Average Detection Delay ( CADD ) . WADD evaluates this delay with respect to the worst scenario in terms of both the change point and the observations . CADD is a less pessimistic evaluation as it only considers the worst scenario in terms of change point . Mathematically they are defined as : FAR ( τ ) = 1 E∞ [ τ ] and { WADD ( τ ) = supn esssupXn En [ ( τ − n ) +|Xn ] CADD ( τ ) = supnEn [ ( τ − n ) |τ > λ ] where Xn designates all observation up to the nth one . In this setting , the goal is to minimise either WADD ( Lorden formulation ( Lorden et al. , 1971 ) ) ; or CADD ( Pollak formulation ( Pollak , 1985 ) ) while keeping FAR below a certain level α . Formally , these problems are written as follows : ( LORDEN ) { να = arg minτ∈∆α WADD ( τ ) ∆α = { τ : FAR ( τ ) < α } and ( POLLAK ) { να = arg minτ∈∆α CADD ( τ ) ∆α = { τ : FAR ( τ ) < α } ( 3 ) 3 TEMPORAL WEIGHT REDISTRIBUTION . Under known parameters , optimal solutions for the QCD consist in computing some statistics , denoted by Sθ0 , θ1n along with a threshold Bα . The stopping time να is the first time S θ0 , θ1 n exceeds Bα . We provide a more detailed description in Appendix A.1 . 3.1 ASYMPTOTIC BEHAVIOUR OF THE SOLUTION TO THE BAYESIAN FORMULATION . The SHIRYAEV algorithm ( Shiryaev , 1963 ) is asymptotically optimal in the i.i.d . case ( Tartakovsky & Veeravalli , 2005 ) . This result is extended to the non i.i.d . case under the following assumption : Hypothesis 1 Given fθ0 and fθ1 , there exists q ∈ R and r ∈ N such that for any k ∈ N : 1 n ∑k+n t=k fθ1 ( Xt+1|Xt ) fθ0 ( Xt+1|Xt ) r-quickly−→ n→+∞ q ( 4 ) Under Hypothesis 1 , and with exponential or heavy tail prior , the SHIRYAEV algorithm is asymptotically optimal ( Tartakovsky & Veeravalli , 2005 ) . In addition , the moments of ADD satisfy the following property for all m ≤ r ( Tartakovsky & Veeravalli , 2005 ) : Eµ [ ( να − λ ) m|τ > λ ] α→0∼ Eµ [ ( νSα − λ ) m ] α→0∼ ( | log ( α ) | q+d ) m ( 5 ) where νSα is the SHIRYAEV algorithm stopping rule and d = − limn→∞ logP ( λ > n+ 1 ) /n . Our approach to the QCD problem under unknown parameters relies on the asymptotic behaviour from Equation ( 5 ) up to the second order ( r = 2 ) . We use it as a lower bound for the detection delay , providing a natural segmentation to our data when approximating the parameters . We will discuss this more in details in what follows . 3.2 RATIONAL FOR TEMPORAL WEIGHT REDISTRIBUTION . In the following , we denote the true parameters by θ∗0 and θ ∗ 1 and the corresponding stationary distributions by Π∗0 and Π ∗ 1 ( that are well defined for irreducible Markov Chain ) . The purpose of this section is to devise an algorithm learning the parameters θ∗0 and θ ∗ 1 using the asymptotic behaviour of the detection delay under known parameters . The delay predicted in Equation ( 5 ) is a performance lower bound in our setting ( otherwise it would contradict the optimality of the SHIRYAEV algorithm ) . Intuitively , the idea is to learn θ∗1 using the last | log ( α ) | q+d observation and to learn θ ∗ 0 using previous observation . This simple technique happens to be efficient and computationally simple . Parameters optimisation : The optimal Sθ0 , θ1t – called the SHIRYAEV statistic – has a specific form ( see Equation ( 12 ) in Appendix ) and the QCD problem under unknown parameters is equivalent to : να = arg mint { t|S θ0 , θ1 t > Bα } where { θ0 = arg minθ f0 ( θ , θ 1 ) θ1 = arg maxθ f1 ( θ 0 , θ ) , ( 6 ) and ( θ0 , θ1 ) are random initialisation parameters , f0 ( θ0 , θ1 ) = E0 [ log fθ1 ( Xt+1|Xt ) fθ0 ( Xt+1|Xt ) ] , f1 ( θ0 , θ1 ) = E1 [ log fθ1 ( Xt+1|Xt ) fθ0 ( Xt+1|Xt ) ] , and Ek∈ { 0,1 } is the expectation with respect to the probability distribution Pk ( Xt , Xt+1 ) = Π ∗ k ( Xt ) fθ∗k ( Xt+1|Xt ) . In fact , the solution να to Equation ( 6 ) is the optimal stopping time under the true parameters . This is a consequence of the following Lemma 1 : Lemma 1 θ∗0 and θ∗1 verifies the following for any θ0 , θ1 : θ∗0 = arg minθ f0 ( θ , θ1 ) and θ ∗ 1 = arg maxθ f1 ( θ0 , θ ) ( 7 ) In order to solve the equivalent QCD problem of Equation ( 6 ) , a good approximation of the functions f0 and f1 given the observations ( Xt ) n0 is required . This implies the ability to distinguish pre-change samples from post-change samples and this is precisely why optimal solutions are intractable . They compute the statistics for any possible change point and consider the worst case scenario . Previous approximate solutions simplify the setting by using domain knowledge to infer the pre-change distribution and by using a sliding window to evaluate the post-change distribution . The window size w is an irreducible cost in the expected delay . With known parameters , optimal algorithms have an average detection delay proportional to the error threshold α and to the inverse of the KL divergence between the pre and post change distributions . Given the optimal stopping time να , it ’ s possible to evaluate the posterior distribution of the change point P ( λ = t|να = n ) , which in turn is a good classifier of the pre and post change observation : P ( Xt ∼ fθ0 |να = n ) ∝ P ( λ > t|να = n ) ; P ( Xt ∼ fθ1 |να = n ) ∝ P ( λ < t|να = n ) . ( 8 ) The objective of this section is to exploit this classifier to construct a family of functions that approximate well f0 and f1 . Consider the following family of functions : fn0 ( θ0 , θ1 ) = E n 0 [ log ( fθ1 fθ0 ) ( Xτ+1|Xτ ) ] and fn1 ( θ0 , θ1 ) = E n 1 [ log ( fθ1 fθ0 ) ( Xτ+1|Xτ ) ] , where both the observations Xt and the indicies τ are random variables . The observations are generated using Equation ( 1 ) with a finite horizon t2 . The indicies are sampled with respect to τ ∼ P ( λ > τ |να = n ) / ∑t2 i=0P ( λ > i|να = n ) in En0 while they are sampled with respect to τ ∼ P ( λ < τ |να = n ) / ∑t2 i=0P ( λ < i|να = n ) in En1 . The family of functions fnk are a re-weighted expectation of the log-likelihood ratio of the observations using P ( λ|να = n ) . Basically , under the assumption that the optimal detection happens at t = n , we associate to each observation Xτ a weight proportional to P ( λ > τ |να = n ) when estimating f0 ( and respectively proportional to P ( λ < τ |να = n ) when estimating f1 ) . In addition , this family of functions is practical as we can approximate asymptotically P ( λ|να = n ) using the theoretical delay behaviour of the SHIRYAEV algorithms presented in Equation ( 5 ) . Lemma 2 For any given parameters ( θ0 , θ1 ) , the following convergences hold : If λ =∞ , then : limn , t2→∞ |fn0 ( θ0 , θ1 ) − f0 ( θ0 , θ1 ) | = limn , t2→∞ |fn1 ( θ0 , θ1 ) − f0 ( θ0 , θ1 ) | = 0 If λ < ∞ , then : { limn , t2→∞ |fn0 ( θ0 , θ1 ) − f1 ( θ0 , θ1 ) | = 0 limn , t2→∞ |fn1 ( θ0 , θ1 ) − f1 ( θ0 , θ1 ) | = 0 limt2→∞ |f να 1 ( θ0 , θ1 ) − f1 ( θ0 , θ1 ) | = 0 For any integer n ∈ N , if t2 = λ+ n , then : limλ , t2→∞ |f να 0 − f0| = 0 . A major implication of Lemma 2 is that with enough observations , fνα0 and f να 1 will eventually converge to the functions f0 and f1 . In fact , f t0 approaches f0 up to να and then start degrading ( as it ends up converging to f1 ) whereas f t1 becomes a better and better approximation of f1 around να and improves asymptotically . As such , we are going to consider the following problem as proxy to the original QCD problem ( Equation ( 6 ) ) . ν̂α = arg mint { t|S θt0 , θ t 1 t > Bα } where { θt0 = arg minθ f t 0 ( θ , θ t−1 1 ) θt1 = arg maxθ f t 1 ( θ t−1 0 , θ ) . ( 9 ) The rational behind this choice is that around the change point , Sθ t 0 , θ t 1 t converges to S θ∗0 , θ ∗ 1 t . Having access to a perfect evaluation of the functions f0 and f1 guarantees the fastest possible detection . In fact , να – the optimal stopping time under known parameters – is a lower bound to the detection delay in this setting . In practice , only the first t observation are accessible to evaluate f t0 and f t 1 . This introduces approximation errors to the estimates θt0 and θ t 1 , thus delaying the detection . However , this is not detrimental to the evaluation of the stopping time . Indeed : Before να : θt0 is a good estimator of θ∗0 and θt1 is a bad estimator of θ∗1 . In fact , the fraction of prechange observations used to learn θ∗1 is of the order of min ( 1 , λ/t ) . This helps maintaining a low log-likelihood ratio ( equivalently maintain Sθ t 0 , θ t 1 t below the cutting threshold ) , as the estimation of θ∗1 will converge to a parameter close to θ ∗ 0 . After να : θt1 is a good approximation of θ∗1 , but θt0 is a noisy estimation of θ∗0 ( as post-change observations are used to learn it ) . The fraction of the data generated using θ∗1 but used to learn θ∗0 is proportional to t−να t . This favours – up to a certain horizon – higher log- likelihood ratio estimates ( equivalently an incremental behaviour of the sequence Sθ t 0 , θ t 1 t ) . We will discuss further the performance issues of solving Equation ( 9 ) in the following . Distribution Approximation : In order to exploit the previously introduced results , we need a good approximation of P ( λ|να = n ) . If the observation satisfy Hypothesis 1 , then the moments of P ( λ|να = n ) up to the rth order satisfy Equation ( 5 ) . When r =∞ , this is equivalent to the Hausdorff moment problem , which admits a unique solution . The generalised method of moments can be used to approximate this distribution . In the remaining of this paper , we will restrict ourselves to the case where r = 2 . There is an infinite number of distributions that satisfy Equation ( 5 ) in this case ; however , given a parametric family , we can compute analytically a solution of the problem so we consider the Logistic distribution . For a given θ0 , θ1 , we approximate P ( λ|να = n ) with fθ0 , θ1n = Logistic ( µ , s ) where µ = n− | log ( α ) | DKL ( fθ0 |fθ1 ) +d and s = √ 3 | log ( α ) |π ( DKL ( fθ0 |fθ1 ) +d ) . Verifying that fθ0 , θ1n satisfies Equation ( 5 ) for m ≤ 2 is straightforward . As a consequence , P ( τ > t|τ ∼ fθ0 , θ1n ) is a fair approximation of P ( λ > t|να = n ) . Limitations : Solving the problem given by Equation ( 9 ) unfortunately yields poor performances for extreme values of the error threshold α . This is due to a degraded log likelihood estimation both before λ and as t→∞ . In fact , the log-likelihood ratios Lt and L∗t defined as : Lt ( Xt+1|Xt ) = log fθt1 fθt0 ( Xt+1|Xt ) and L∗t ( Xt+1|Xt ) = log fθ∗1 fθ∗0 ( Xt+1|Xt ) , satisfy the following lemma : Lemma 3 The log-likelihood ratio Lt and L∗t admit the following asymptotic behaviours : lim λ→∞ Eλ [ L ∗ λ ] < 0 but Lλ a.s−→ λ→+∞ 0 ; lim t→∞ Eλ [ L ∗ t |λ < ∞ ] > 0 but Lt a.s−→ t→+∞ 0 ( 10 ) Practically , this means that before the change point , the log likelihood ratio is over-estimated ( as it should be smaller than 0 ) , while after the change point , the log likelihood ratio is eventually under-estimated ( as it should exceed 0 ) . This is a consequence of Lemma 2 . In fact , for t λ ( respectively for t ≤ λ ) , both functions f t0 and f t1 are approaching f1 ( respectively f0 ) . Thus in both cases θt0 and θ t 1 converge to the same value . However , combined with the behaviour of f t 0 and f t 1 around να from Lemma 2 , we can expect the Kullback–Leibler ( KL ) divergence DKL ( fθt0‖fθt1 ) to converge to 0 before and after the change point while peaking around the optimal stopping time . We exploit this observation to mitigate the problems highlighted with Lemma 3 . Annealing : The under-estimation of L∗t after the change point is due to the use of post change observations ( drawn from fθ∗1 ) when estimating θ ∗ 0 . In practice , this is particularly problematic when the error rate must be controlled under a low threshold α ( equivalently a high cutting threshold Bα ) or when the pre and post change distributions are very similar . When t approaches the optimal stopping time ( t ≈ να ) , the minimising argument of f t0 is converging to θ∗0 . As t grows , the approximation f t0 is degraded , while f t 1 is becoming a better approximation , thus θ t 1 starts converging to θ ∗ 1 . This leads to an increase in the KL divergence . Our proposition is to keep optimising the version of f t0 that coincides with this increase ( which in turn is fνα0 : the best candidate to identify θ ∗ 0 ) . As a fix , we propose : 1- to shift the probability P ( λ > τ |νSα = n ) by replacing it with P ( λ > τ −∆|νSα = n ) ( where ∆ is the ‘ delay ’ of the detection with respect to the Shiryaev stopping time : ( n−νSα ) + ) , and 2- anneal the optimisation steps of θt0 as the KL divergence increases . These tweaks correct the objective function used to learn θ∗0 and stop its optimisation when the observations start to deviate from the learned pre-change distribution . The delay can be formally introduced by replacing f t0 with f t−∆ 0 in Equation ( 9 ) . In practice , ∆ is a delay heuristic that increases as the KL divergence increases . This reduces the noise when learning θ∗0 . The second idea is to use a probability p0 = 1−∆ of executing a gradient step when learning the pre-change parameter . This anneals the optimisation of θ∗0 . Penalisation : The over-estimation of L∗t before the change point , is due to the exclusive use of pre change observation when estimating θ∗1 . This is particularly problematic for application where a high error threshold is tolerable ( equivalently a low cutting threshold for the used statistic ) . This is also an issue when the pre and post change distribution are very different . As a solution , we penalise the log-likelihood using the inverse KL divergence . Before the change point , both parameters converge to θ∗0 : this means that the KL divergence between the estimated distributions is around 0 . Inversely , after the change point , if the optimisation of θt0 is annealed , the KL divergence between the estimated parameters must hover around DKL ( fθ∗0‖fθ∗1 ) . Formally , penalising the log likelihood can be seen as using L̂t when computing the statistics Sn where L̂t = Lt − c/DKL ( fθt0‖fθt1 ) for some real c > 0 . We provide in the appendix an ablation analysis as well as experimental proof of the effectiveness of both annealing and penalisation . Algorithm : In practice , only the first t observations are available in order to evaluate the parameters ( θ∗0 , θ ∗ 1 ) . Hence , we design a stochastic gradient descent ( SGD ) based algorithm to solve Equation ( 9 ) . Consider the following loss functions , where I is a set of time indices : Ln0 ( I , θ0 , θ1 ) = ∑ t∈I P ( τ < t|τ ∼ fθ0 , θ1n ) log ( fθ1 fθ0 ) ( Xt+1|Xt ) Ln1 ( I , θ0 , θ1 ) = ∑ t∈I P ( τ > t|τ ∼ fθ0 , θ1n ) log ( fθ1 fθ0 ) ( Xt+1|Xt ) ( 11 ) The quantity Ltk is an estimator of f tk using the observations ( Xt ) t∈I . The rational previously discussed implies that by re-weighting random samples from the observations we can approximate the functions fk using the familly of functions f tk . A good approximation of θt0 and θ t 1 can be obtained with few SGD steps . These parameters are used to update the SHIRYAEV or the CUSUM statistic . The change point is declared once the threshold Bα is exceeded . The proposed procedure is described in Algorithm 1 . In addition to the classical SGD hyper-parameters ( number of epochs , gradient step size , momentum , . . . ) , we propose to control the penalisation and the annealing procedure using the coefficients c , ≥ 0 respectively . Algorithm 1 Temporal Weight Redistribution 1 : Procedure : TWR ( θ0 , θ1 , ( Xt ) t2t=0 , Ne , Bα , , c ) 2 : Initialise S ← 0 ; θ0 , θ1 ← θ0 , θ1 ; ∆← 0 ; and p0 ← 1 3 : for t ∈ [ 1 , t2 ] do 4 : for e ∈ [ 1 , Ne ] do 5 : Randomly sample I : a set of indices from [ 0 , t ] 6 : θ0 ← gradient update rule ( θ0 , ∇θ0Lt−∆0 ( I , θ0 , θ1 ) ) with probability p0 7 : θ1 ← gradient update rule ( θ1 , −∇θ1Lt1 ( I , θ0 , θ1 ) ) 8 : end for 9 : S ← statistic update rule ( S , fθ1fθ0 ( Xt+1|Xt ) − c DKL ( fθ0 |fθ1 ) 10 : If S > Bα then declare change point and break 11 : If DKL ( fθ0 |fθ1 ) > D̄ then ∆← ∆ + 1 and p0 ← p0 − 12 : D̄ ← t−1t D̄ + 1 tDKL ( fθ0 |fθ1 ) 13 : end for

Summary: The author(s) propose a quickest change detection technique under known parameter scenario. They use a Markovian dynamics to generate the pre and post change-point distributions and use a Shirayev test-statistic based on the asymptotic behavior of the optimal delay under know parameters. The proposed methodology is validated on synthetic data and a multitask reinforcement learning example. There are several issues which restricts the paper to reach an optimal level. These are highlighted below