Online false discovery rate control for anomaly detection in time series

1 Introduction . Online anomaly detection is critical for many monitoring applications in health care , IT operations , manufacturing , or retail ( e.g. , [ 11 , 4 , 20 ] ) . In such settings , observations of one or several metrics of interest ( naturally represented as time series ) are sent sequentially to a detector . It is tasked with deciding whether a given observation is anomalous or not . Accordingly , anomaly detection in timeseries data is a rich field that has been surveyed in several articles [ 7 , 6 ] , Foorthuis [ 9 ] proposes a typology of anomalies , and Sadik and Gruenwald [ 21 ] discusses open research questions in the field . Arguably one of the most common approaches in addressing anomaly detection is via employing an unsupervised probabilistic anomaly scorer that assigns a probability to each point . This translates the problem of anomaly detection into one of online multiple hypotheses testing . Specifically , at each time step t , we observe a new data point zt and emit the null hypothesis Ht that zt is not anomalous . This hypothesis is tested with a p-value provided by the scorer and may be rejected ( detection of an anomaly ) if there is enough evidence against it . As in any statistical testing problem , a decision threshold determines whether the null hypothesis is rejected or not . In the case of a single test , the most common convention is to report a discovery ( a point is anomalous ) if the associated p-value is smaller than some preselected threshold ↵ . The effect is said to be statistically significant at level ↵ and the probability to falsely declare the point anomalous is less than ↵ . Importantly , in the case of multiple hypotheses tests , the threshold ↵ does not provide similar guarantees and the fraction of false positives may be arbitrarily close to 1 . This particularly applies to anomaly detection because the proportion of alternative hypotheses tends to be very low . Hence , multiple hypotheses testing requires other mechanisms to set decision thresholds . In the case where all p-values are known in advance ( offline setting ) , classical methods typically shrink the threshold ↵ conservatively [ 2 , 3 , 27 ] . In the setting of anomaly detection , a natural quantity to control is the false discovery rate ( FDR ) as introduced by [ 2 ] , that is , the ratio of falsely labeled anomalies to total anomalies . Methods for ⇤Correspondence : quentin.rebjock @ epfl.ch . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sequential FDR control ( FDRC ) were pioneered by [ 10 ] who proposed the so-called ↵-investing strategy , which was later built upon and generalized [ 1 , 15 , 16 , 18 , 19 , 30 , 34 , 16 ] . Online FDRC methods are appealing for anomaly detection problems as controlling the false discovery rate is equivalent to maximizing recall given a lower bound on precision . Consequently this allows to trade-off precision and recall even in the absence of labelled data , which is another commonality of anomaly detection . All online FDRC methods allocate some threshold at every time step , and reject the hypothesis based on it . The main contribution of this article is the proposal of a novel method to overcome a common limitation of existing online FDRC methods that renders them unsuited for online anomaly detection tasks . When few rejections are made because the alternative hypothesis is extremely rare , as is the case in most anomaly detection problems , rejection thresholds of existing methods tend to zero . This phenomenon is referred to as ↵-death in the literature [ 18 ] and prevents any future discoveries causing a loss of power . To circumvent this problem , we propose a method based on memory decay , revisiting [ 18 ] , to ensure that rejection thresholds are lower-bounded by a non-zero value . This guarantees that we have power even in the case where alternatives ( anomalies ) are rare . We demonstrate that this method allows us to control the decaying memory FDR [ 18 ] , a version of the FDR metric adapted to infinite streams that progressively forgets past decisions . Our approach is fully general in the sense that it can enhance all of the most popular algorithms in the generalized ↵-investing class . Our second contribution is the adaptation of the local dependency framework [ 34 ] to memory decay . This allows to use our methods for practical applications where p-values are in general not independent . We evaluate experimentally the performances of the proposed algorithms and demonstrate that we can overcome the challenges occurring when alternative hypotheses are exceedingly rare . This paper is structured as follows . Section 2 formalizes the problem of multiple hypotheses testing for anomaly detection and defines the false discovery rate . Section 3 reviews the main methods for online FDRC and illustrates their limitations . Section 4 details our contributions to prevent ↵-death . Section 5 proposes an adjustment to make the algorithms robust to local dependency in the p-values . Finally , Section 6 shows experimentally that our method preserves power while controlling the false discovery rate in anomaly detection settings . We conclude in Section 7 . 2 Problem Formalization : Multiple hypotheses testing for anomaly detection . We formalize the problem statement of online anomaly detection via an unsupervised approach using probabilistic time series models . There is a rich and growing literature describing such probabilistic models ( e.g. , [ 5 , 14 , 24 , 23 , 26 , 25 , 8 ] ) well suited for anomaly detection . Deriving p-values . At each time step t , hypothesis Ht is tested based on a p-value that must be obtained from past observations only . The concept of anomaly is ambiguous and suffers from a subjective and imprecise definition [ 9 ] . We assume that anomalies are characterized by statistically rare events that deviate significantly from expectations . For this reason , it is natural to address anomaly detection through probabilistic forecasting . A probabilistic forecast is an estimation of the probability distribution of zt given the past , that is Prob { zt | zt0 , t0 < t } , or equivalently its c.d.f . Ft . Under the null hypothesis ( no anomaly ) , such a forecast should be a decent estimate of zt . From the forecast we can derive a p-value pt , which is the probability to observe an event in the tail of the distribution relative to zt under the assumption that Ht is null . Hence , a small value for pt is an indication that Ht must be rejected . We may derive two-sided p-values as pt = 2 min ( Ft ( zt ) , 1 Ft ( zt ) ) using the predicted c.d.f . Ft for symmetric distributions . With this principle , we can infer a sequence of p-values { pt } 1t=1 iteratively as new observations are made available . Instead of two-sided p-values , other scenarios are conceivable as for certain usages , it may make sense to define anomalies as values either extremely high or low compared to the median . For example , in a scenario where we monitor the resources usage of a compute service , only large values should result in an alert . Accordingly , one-sided p-values can be derived to account for only one end of the distribution . More generally , p-values need only to be stochastically larger than the uniform distribution under the null hypothesis , that is : if Ht is truly null then Prob { pt u } u for all u 2 [ 0 , 1 ] . ( 1 ) With this definition of anomalies , we can decompose the problem of anomaly detection into two distinct sub-problems : ( i ) obtaining p-values in line with the application via a forecasting model and ( ii ) using a criterion on the p-values to accept/reject hypotheses . We will assume time series models to be given and focus on ( ii ) , that is , the choice of decision thresholds . False discovery rate control . As we reject hypotheses we want to make sure that we do not falsely report too many anomalies . If there was a single hypothesis we could reject it if the associated p-value is smaller than some threshold ↵ : this ensures that the probability to report an anomaly when it is not is less than ↵ . This is an immediate consequence of the definition of the p-value in Equation ( 1 ) . However , this procedure does not provide similar guarantees for a set of hypotheses in general and for sequentially dependent p-values as in the case of online anomaly detection in time series in particular . For this reason , in multiple hypotheses settings rejections are based on procedures controlling metrics pertaining to a set of tests . One of these metrics is the family-wise error rate ( FWER ) [ 13 ] , which is the probability to make at least one false discovery . The probability to reject a truly null hypothesis approaches 1 when the number of tests grows . This makes the FWER metric too conservative , leading to very few discoveries for long ( possibly infinite ) streams of data . The false discovery rate ( FDR ) is the more natural quantity to control in our setting . Procedures controlling the FDR aim at maximizing the number of true discoveries subject to an upper bound on the expected proportion of false discoveries . This is equivalent to maximizing recall subject to a user-defined lower bound on precision . For a given set of hypotheses the FDR is formally defined as the expected ratio of false positives to total discoveries : FDR , E [ FDP ] , E |H 0 \R| |R| _ 1 , where H0 and R are the sets of truly null hypotheses and rejections respectively , and FDP is the false discovery proportion of the samples . This article is concerned with the problem of online FDR control where observations arrive sequentially and we want to make a decision immediately on whether to classify the observation as anomalous or not . Alternative settings have been investigated in the literature . Rather than making a decision at every time step , Zrnic et al . [ 33 ] considers testing sequences of hypotheses by batches of variable length . This takes advantage of a partial ordering of the p-values at the cost of a delay in the decision , which is not desirable in online anomaly detection . Another option is to aggregate a series of p-values [ 17 , 12 ] and perform FDR control consequently , either sequentially or based on multivariate [ 22 ] forecasts . Wang [ 31 ] propose to perform online anomaly detection by coupling a STL decomposition with a FDR method based on the distribution of z statistics or the residuals building on a method proposed by Sun and Cai [ 29 ] . 3 Background : Online FDR control . Online testing algorithms aim at controlling a time-dependent variant of the FDR that we define as follows . We let Rt denote the rejection indicator at time t and define the number of rejections R ( T ) and of the number of false positives V ( T ) up to time T as R ( T ) = TX t=1 Rt and V ( T ) = TX t=1 Rt1 { t 2 H 0 } . Naturally , we define the FDP and FDR at time T as FDP ( T ) = V ( T ) R ( T ) _1 and FDR ( T ) = E [ FDP ( T ) ] respectively . Online decision rules that control the FDR allocate a decision threshold ↵t to test hypothesis Ht at every time step t. The hypothesis is rejected if the associated p-value is less than ↵t , and the rejection indicator is Rt = 1 { pt ↵t } . We wish to pick the rejection thresholds { ↵t } 1t=1 in such a way that FDR ( T ) ↵ at any time T for some FDR target ↵ 2 [ 0 , 1 ] . Decision thresholds must depend solely on past observations , so that for any time t there is a function ft such that ↵t = ft ( R1 , . . . , Rt 1 ) 2 F t 1 , ( 2 ) where F t , ( R1 , . . . , Rt ) is the -field ( information known ) at time t. In this section , we make the standard conditional super-uniformity assumption that if Ht is truly null then Prob { pt u | F t 1 } u for all u 2 [ 0 , 1 ] , ( 3 ) which is satisfied if { pt } 1t=1 is a sequence of valid p-values and if null p-values are independent of all others . The independence assumption can be questionable for time series data in certain settings . We propose a method to relax it in Section 5 but we maintain it until then to simplify the exposition . Generalized ↵-investing . The first algorithm controlling ( a modified version of ) the FDR was introduced by [ 10 ] with the idea of ↵-investing . This algorithm is part of a broader class of algorithms known as generalized ↵-investing ( GAI ) rules [ 1 ] . Algorithms of the GAI class record a quantity wt called the wealth , that represents the allocation potential for future decision thresholds . They start with an initial wealth w0 2 ( 0 , ↵ ) and at every time step t , an amount t is spent from the remaining wealth in order to test the hypothesis Ht at level ↵t . When a rejection occurs , an amount of t is earned back , which induces the update wt = wt 1 t + Rt t. The wealth must always be non-negative , which imposes t wt 1 , and the ( modified ) FDR is controlled only if the quantities t are bounded adequately . The exact FDR can be controlled by algorithms called LORD and LOND [ 15 ] . These algorithms are part of the GAI framework , and satisfy a monotonic property , meaning that ft is a coordinate-wise non-decreasing function in Equation ( 2 ) . In fact , any monotonic GAI rule controls the FDR , under the assumption that p-values are independent [ 16 ] . Oracle rules . Monotonic GAI rules can be seen as a special case of decision rules relying oracle estimates of the FDP [ 18 , 19 , 30 ] . Maintaining the quantity FDP ? ( T ) = P tT , t2H0 ↵t R ( T ) _ 1 below ↵ at time T guarantees that FDR ( T ) ↵ . This oracle can not be evaluated as the set of null hypotheses H0 is not known . A simple way to upper-bound FDP ? is to remove the condition t 2 H 0 in the summation , which leads to a class of algorithms known as LORD [ 15 , 16 , 18 ] . More refined upper-bounds give rise to adaptive algorithms known as SAFFRON and ADDIS [ 19 , 30 ] . In short , LORD can be seen as an online version of the Benjamini-Hochberg procedure [ 2 ] , SAFFRON as an online version of [ 27 ] , and ADDIS is able to slightly improve on SAFFRON ’ s performances when null p-values are not exactly uniformly distributed , but stochastically larger than uniform . The algorithms SAFFRON and ADDIS are able to improve slightly on LORD ’ s power when the proportion of alternative hypotheses is large , typically more than 50 % , which is not the case in anomaly detection . We illustrate this in Figure 1 . For this reason we focus on LORD and provide a more detailed discussion about SAFFRON and ADDIS in Appendix B . The oracle estimate of LORD is given by dFDPLORD ( T ) = P tT ↵t R ( T ) _ 1 and offers a lot of flexibility . FDR control at time T holds for any sequence { ↵t } 1t=1 chosen such that dFDP ( T ) ↵ , provided that the functions ft ( in Equation 2 ) are coordinate-wise non-decreasing . We show here how to generate such a sequence as suggested in [ 18 ] . We let { t } 1t=1 denote a nonincreasing sequence summing to 1 , where we set t = 0 for all t 0 for convenience . We also let ⇢j = min { t 0 | Pt i=1 Ri j } denote the time of the jth rejection ( and 1 if it doesn ’ t exist ) . Then the sequence defined by ↵t = w0 ( t t ⇢1 ) + ↵ X j 1 t ⇢j ( 4 ) satisfies the FDR control requirements . Note that thresholds decrease to zero except when new rejections are made . Using the same notations , a special instance of ADDIS is given by the sequence ↵t = ( ⌧ ) ⇣ w0 ( S0 ( t ) S1 ( t ) ) + X j Sj ( t ) ⌘ ^ , ( 5 ) for some fixed parameters 1 > ⌧ > > 0 , and where we define Sj ( t ) , 1 { t > ⇢j } + t 1X i=⇢j+1 1 { < pi ⌧ } . The corresponding SAFFRON algorithm is recovered by setting ⌧ = 1 . Limitations of FDR rules . Anomalies are by definition rare events and the proportion of alternative hypotheses in the stream is expected to be very low , typically less than 1 % in practice . This induces two distinct problems for the discovery of anomalies : P.1 The lower the proportion of alternatives , the harder it is to distinguish them from nulls with extreme values . This difficulty is inherent to FDR control , be it offline or online , and makes anomaly detection challenging . P.2 In the online setting , the decision thresholds ↵t decreases monotonously while no rejection is made . Hence , if no rejection is made for a long time , the thresholds will be essentially null and no additional discovery will occur . This phenomenon is known as ↵-death and causes a loss of power in anomaly detection regimes . The power of these algorithms depends on { t } 1t=1 and on the nature of the data since they are general and make minimal assumptions . In the remaining , we use the default and robust sequences derived by [ 18 , 19 , 30 ] , that is , t / log ( t^2 ) t exp p log ( t ) for LORD . In experiments involving SAFFRON and ADDIS we also use standard parameters ; see Appendix B for more details . Figure 1 illustrates both of these problems . In the regime where the proportion of alternatives is low , LORD , SAFFRON and ADDIS all exhibit the same ( poor ) performances . Selecting a sequence { ↵t } 1t=1 going to zero as slowly as possible helps to overcome P.2 to some extent but worsens P.1 in contrast .

Summary: This paper discusses techniques by which some limitations of current online false discovery rate control algorithms can be overcome. Importantly, the sequential decision thresholds from current algorithms decay to zero in the absence of recent rejections of the null hypothesis -- a situation called alpha-death. A technique to overcome this is proposed.  ================ Update: The scope of contribution has been clarified in the rebuttal. The authors address the lack of real-world experiments somewhat, but more is desired to improve the strength of the paper.