Drift Detection in Episodic Data: Detect When Your Agent Starts Faltering

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms have recently demonstrated impressive success in a variety of sequential decision-making problems ( Badia et al. , 2020 ; Hessel et al. , 2018 ) . While most RL works focus on the maximization of rewards under various conditions , a key issue in real-world RL tasks is the safety and reliability of the system ( Dulac-Arnold et al. , 2019 ; Chan et al. , 2020 ) , arising in both offline and online settings . In offline settings , comparing the agent performance in different environments is important for generalization ( e.g. , in sim-to-real and transfer learning ) . The comparison may indicate the difficulty of the problem or help to select the right learning algorithms . Uncertainty estimation , which could help to address this challenge , is currently considered a hard problem in RL , in particular for modelfree methods ( Yu et al. , 2020 ) . In online settings , where a fixed , already-trained agent runs continuously , its performance may be affected ( gradually or abruptly ) by changes in the controlled system or its surroundings , or when reaching new states beyond the ones explored during the training . Some works address the robustness of the agent to such changes ( Lecarpentier & Rachelson , 2019 ; Lee et al. , 2020 ) . However , noticing the changes may be equally important , as it allows us to fall back into manual control , send the agent to re-train , guide diagnosis , or even bring the agent to halt . This is particularly critical in real-world problems such as health care and autonomous driving ( Zhao et al. , 2019 ) , where agents are required to be fixed and stable : interventions in the policy are often limited or forbidden ( Matsushima et al. , 2020 ) , but any performance degradation should be detected as soon as possible . Many sequential statistical tests exist for detection of mean degradation in a random process . However , common methods ( Page , 1954 ; Lan , 1994 ; Harel et al. , 2014 ) assume independent and identically distributed ( i.i.d ) samples , while in RL the feedback from the environment is usually both highly correlated over consecutive time-steps , and varies over the life-time of the task ( Korenkevych et al. , 2019 ) . This is demonstrated in Fig . 1 . A possible solution is to apply statistical tests to large blocks of time-steps assumed to be i.i.d ( Ditzler et al. , 2015 ) . Since many RL applications consist of repeating episodes , such a blocks-partition can be applied in a natural way ( Colas et al. , 2019 ) . However , this approach requires complete episodes for change detection , while a faster response is often required . Furthermore , naively ap- plying a statistical test on the accumulated feedback ( e.g. , sum of rewards ) from complete episodes , ignores the dependencies within the episodes and may miss vital information , leading to highly sub-optimal tests . In this work , we devise an optimal test for detection of degradation of the rewards in an episodic RL task ( or in any other episodic signal ) , based on the covariance structure within the episodes . Even in absence of the assumptions that guarantee its optimality , the test is still asymptotically superior to the common approach of comparing the mean ( Colas et al. , 2019 ) . The test can detect changes and drifts in both the offline and the online settings defined above . In addition , for the online settings , we suggest a novel Bootstrap mechanism to control the False Alarm Rate ( BFAR ) through adjustment of test thresholds in sequential tests of episodic signals . The suggested procedures rely on the ability to estimate the correlations within the episodes , e.g. , through a ” reference dataset ” of episodes . Since the test is applied directly to the rewards , it is model-free in the following senses : the underlying process is not assumed to be known , to be Markov , or to be observable at all ( as opposed to other works , e.g. , Banerjee et al . ( 2016 ) ) , and we require no knowledge about the process or the running policy . Furthermore , as the rewards are simply referred to as episodic time-series , the test can be similarly applied to detect changes in any episodic signal . We demonstrate the new procedures in the environments of Pendulum ( OpenAI ) , HalfCheetah and Humanoid ( MuJoCo ; Todorov et al. , 2012 ) . BFAR is shown to successfully control the false alarm rate . The covariance-based degradation-test detects degradation faster and more often than three alternative tests – in certain cases by orders of magnitude . Section 3 formulates the offline setup ( individual tests ) and the online setup ( sequential tests ) . Section 4 introduces the model of an episodic signal , and derives an optimal test for degradation in such a signal . Section 5 shows how to adjust the test for online settings and control the false alarm rate . Section 6 describes the experiments , Section 7 discusses related works and Section 8 summarizes . To the best of our knowledge , we are the first to exploit the covariance between rewards in posttraining phase to test for changes in RL-based systems . The contributions of this paper are ( i ) a new framework for model-free statistical tests on episodic ( non-i.i.d ) data with trusted referenceepisodes ; ( ii ) an optimal test ( under certain conditions ) for degradation in episodic data ; and ( iii ) a novel bootstrap mechanism that controls the false alarm rate of sequential tests on episodic data . 2 PRELIMINARIES . Reinforcement learning and episodic framework : A Reinforcement Learning ( RL ) problem is usually modeled as a decision process , where a learning agent has to repeatedly make decisions that affect its future states and rewards . The process is often organized as a finite sequence of timesteps ( an episode ) that repeats multiple times in different variants , e.g. , with different initial states . Common examples are board and video games ( Brockman et al. , 2016 ) , as well as more realistic problems such as repeating drives in autonomous driving tasks . Once the agent is fixed ( which is the case in the scope of this work ) , the rewards of the decision process essentially reduce to a ( decision-free ) random process { Xt } nt=1 , which can be defined by its PDF ( f { Xt } nt=1 : R n → [ 0 , ∞ ) ) . { Xt } usually depend on each other : even in the popular Markov Decision Process ( Bellman , 1957 ) , where the dependence goes only a single step back , long-term correlations may still carry information if the states are not observable by the agent . Hypothesis tests : Consider a parametric probability function p ( X|θ ) describing a random process , and consider two different hypotheses H0 , HA determining the value ( simple hypothesis ) or allowed values ( complex hypothesis ) of θ . When designing a test to decide between the hypotheses , the basic metrics for the test efficacy are its significance P ( not reject H0|H0 ) = 1−α and its power P ( reject H0|HA ) = β . A statistical hypothesis test with significance 1 − α and power β is said to be optimal if any test with as high significance 1− α̃ ≥ 1− α has smaller power β̃ ≤ β . The likelihood of the hypothesis H : θ ∈ Θ given data X is defined as L ( H|X ) = supθ∈Θp ( X|θ ) . According to Neyman-Pearson Lemma ( Neyman et al. , 1933 ) , a threshold-test on the likelihood ratio LR ( H0 , HA|X ) = L ( H0|X ) /L ( HA|X ) is optimal . In a threshold-test , the threshold is uniquely determined by the desired significance level α , though is often difficult to calculate given α . In many practical applications , a hypothesis test is repeatedly applied as the data change or grow , a procedure known as a sequential test . If the null hypothesisH0 is true , and any individual hypothesis test falsely rejects H0 with some probability α , then the probability that at least one of the multiple tests will reject H0 is α0 > α , termed family-wise type-I error ( or false alarm rate when associated with frequency ) . See Appendix K for more details about hypothesis testing and sequential tests in particular . Common approaches for sequential tests , such as CUSUM ( Page , 1954 ; Ryan , 2011 ) and α-spending functions ( Lan , 1994 ; Pocock , 1977 ) , usually require strong assumptions such as independence or normality , as further discussed in Appendix F . 3 PROBLEM SETUP . In this work , we consider two setups where detecting performance deterioration is important – sequential degradation-tests and individual degradation-tests . The individual tests , in addition to their importance in ( offline ) settings such as sim-to-real and transfer learning , are used in this work as building-blocks for the ( online ) sequential tests . Both setups assume a fixed agent that was previously trained , and aim to detect whenever the agent performance begins to deteriorate , e.g. , due to environment changes . The ability to notice such changes is essential in many real-world problems , as explained in Section 1 . Setup 1 ( Individual degradation-test ) . We consider a fixed trained agent ( policy must be fixed but is not necessarily optimal ) , whose rewards in an episodic environment ( with episodes of length T ) were previously recorded for multiple episodes ( the reference dataset ) . The agent runs in a new environment for n time-steps ( both n < T and n ≥ T are valid ) . The goal is to decide whether the rewards in the new environment are smaller than the original environment or not . If the new environment is identical , the probability of a false alarm must not exceed α . Setup 2 ( Sequential degradation-test ) . As in Setup 1 , we consider a fixed trained agent with recorded reference data of multiple episodes . This time the agent keeps running in the same environment , and at a certain point in time its rewards begin to deteriorate , e.g. , due to changes in the environment . The goal is to alert to the degradation as soon as possible . As long as the environment has not changed , the probability of a false alarm must not exceed α0 during a run of h̃ episodes . Note that while in this work the setups focus on degradation , they can be easily modified to look for any change ( as positive changes may also indicate the need for further training , for example ) . 4 OPTIMIZATION OF INDIVIDUAL DEGRADATION-TESTS . To tackle the problem of Setup 1 , we first define the properties of an episodic signal and the general assumptions regarding its degradation . Definition 4.1 ( T -long episodic signal ) . Let n , T ∈ N , and write n = KT + τ0 ( for non-negative integers K , τ0 with τ0 ≤ T ) . A sequence of real-valued random variables { Xt } nt=1 is a T -long episodic signal , if its joint probability density function can be written as f { Xt } nt=1 ( x1 , ... , xn ) = [ K−1∏ k=0 f { Xt } Tt=1 ( xkT+1 , ... , xkT+T ) ] · f { Xt } τ0t=1 ( xKT+1 , ... , xKT+τ0 ) ( 1 ) ( where an empty product is defined as 1 ) . We further denote µ0 : = E [ ( X1 , ... , XT ) > ] ∈ RT , Σ0 : = Cov ( ( X1 , ... , XT ) > , ( X1 , ... , XT ) ) ∈ RT×T . Note that the episodic signal consists of i.i.d episodes , but is not assumed to be independent or identically-distributed within the episodes . For simplicity we focus on one-dimensional episodic signals , although a generalization to multidimensional signals is straight-forward ( see Appendix G ) . In the analysis below we assume that both µ0 and Σ0 are known . In practice , this can be achieved either through detailed domain knowledge , or by estimation from the recorded reference dataset of Setup 1 , assuming it satisfies Eq . ( 1 ) . The estimation errors decrease as O ( 1/ √ N ) with the number N of reference episodes , and are distributed according to the Central Limit Theorem ( for means ) and Wishart distribution ( K. V. Mardia & Bibby , 1979 ) ( for covariance ) . While in this work we use up to N = 10000 reference episodes , Appendix E shows that N = 300 reference episodes are sufficient for reasonable results in HalfCheetah , for example . Note that correlations estimation has been already discussed in several other RL works ( Alt et al. , 2019 ) . Fig . 1 demonstrates the estimation of mean and covariance parameters for a trained agent in the environment of HalfCheetah , from a reference dataset of N = 10000 episodes . This also demonstrates the non-trivial correlations structure in the environment . According to Fig . 1b , the variance in the rewards varies and does not seem to reach stationarity within the scope of an episode . Fig . 1c shows the autocorrelation function ACF ( t2 − t1 ) = corr ( t1 , t2 ) for different reference times t1 . It is evident that the correlations last for hundreds of time-steps , and depend on the time t1 rather than merely on the time-difference t2− t1 . This means that the autocorrelation function is not expressive enough for the actual correlations structure . Once the per-episode parametersµ0 ∈ RT , Σ0 ∈ RT×T are known , the expectations and covariance matrix of the whole signal µ ∈ Rn , Σ ∈ Rn×n can be derived directly : µ consists of periodic repetitions of µ0 , and Σ consists of copies of Σ0 as T × T blocks along its diagonal . For both parameters , the last repetition is cropped if n is not an integer multiplication of T . In other words , by taking advantage of the episodic setup , we can treat the temporal univariate non-i.i.d signal as a multivariate signal with easily-measured mean and covariance – even if the signal is measured in the middle of an episode . The degradation in the signal X = { Xt } nt=1 is defined through the difference between two hypotheses . The null hypothesis H0 states that X is a T -long episodic signal with expectations µ0 ∈ RT and invertible covariance matrix Σ0 ∈ RT×T . Our first alternative hypothesis – uniform degradation – states that X is a T -long episodic signal with the same covariance Σ0 but smaller expectations : ∃ ≥ 0 , ∀1 ≤ t ≤ T : ( µ ) t = ( µ0 ) t − . Note that this hypothesis is complex ( ≥ 0 ) , where 0 can be tuned according to the minimal degradation magnitude of interest . In fact , Theorem 4.1 shows that the optimal corresponding test is independent of the choice of 0 . Theorem 4.1 ( Optimal test for uniform degradation ) . Define the uniform-degradation weightedmean sunif ( X ) : = W ·X , whereW : = 1 > ·Σ−1 ∈ Rn ( and 1 is the all-1 vector ) . If the distribution of X is multivariate normal , then a threshold-test on sunif is optimal . Proof Sketch . According to Neyman-Pearson Lemma ( Neyman et al. , 1933 ) , a threshold-test on the likelihood-ratio ( LR ) between H0 and HA is optimal . Since HA is complex , the LR is a minimum over ∈ [ 0 , ∞ ) . Lemma 1 shows that ∃s0 : sunif ≥ s0 ⇒ = 0 and sunif ≤ s0 ⇒ = ( sunif ) . The rest of the proof in Appendix J substitutes in both domains of sunif to prove monotony of the LR in sunif , from which we can conclude monotony in sunif over all R. Following Theorem 4.1 , we define the Uniform Degradation Test ( UDT ) to be a threshold-test on sunif , i.e. , ” declare a degradation if sunif < κ ” for a pre-defined κ . Recall that optimality of a test is defined in Section 2 as having maximal power given significance level . To achieve the significance α required in Setup 1 , we apply a bootstrap mechanism that randomly samples episodes from the reference dataset and calculates the corresponding statistic ( e.g. , sunif ) . This yields a bootstrap-estimate of the distribution of the statistic under H0 , and the α-quantile of the estimated distribution is chosen as the test-threshold ( κ = qα ( sunif |H0 ) ) . Note that Theorem 4.1 relies on multivariate normality assumption , which is often too strong for real-world applications . Theorem 4.2 guarantees that if we remove the normality assumption , it is still beneficial to look into the episodes instead of considering them as atomic blocks ; that is , UDT is still asymptotically better than a test on the simple mean ssimp = ∑n t=1Xt/n . Note that ” asymptotic ” refers to the signal length n→∞ ( while T remains constant ) , and is translated in the sequential setup into a ” very long lookback-horizon h ” ( rather than very long running time ) . Theorem 4.2 ( Asymptotic power of UDT ) . Denote the length of the signal n = K · T , assume a uniform degradation of size √ K , and let two threshold-tests τsimp on ssimp and UDT on sunif be tuned to have significance α . Then limK→∞P ( τsimp rejects H0 ∣∣HA ) = Φ ( q0α + T√ 1 > Σ01 ) ≤ Φ ( q0α + √ 1 > Σ−10 1 ) = limK→∞P ( UDT rejects H0 ∣∣HA ) ( 2 ) where Φ is the CDF of the standard normal distribution , and q0α is its α-quantile . Proof Sketch . Since the episodes of the signal are i.i.d , both ssimp and sunif are asymptotically normal according to the Central Limit Theorem . The means and variances of both statistics are calculated in Lemma 2 . Calculation of the variance of sunif relies on writing sunif as a sum of linear transformations of X ( sunif = ∑n i=1 ( Σ −1 ) iX ) , and using the relation between Σ and Σ0 . Appendix J shows that the inequality between the resulted powers is equivalent to a matrix-form of the means-inequality , and proves it by applying Cauchy-Schwarz inequality to Σ−1/20 1 and Σ 1/2 0 1 . Motivated by Theorem 4.2 , we define G2 : = ( 1 > Σ−10 1 ) ( 1 > Σ01 ) T 2 to be the asymptotic power gain of UDT , quantify it , and show that it increases with the heterogeneity of the spectrum of Σ0 . Proposition 4.1 ( Asymptotic power gain ) . G2 = 1 + ∑T i , j=1 wij ( λi−λj ) 2 , where { λi } Ti=1 are the eigenvalues of Σ0 and { wij } Ti , j=1 are positive weights . Proof Sketch . The result can be calculated after diagonalization of Σ0 , and the weights { wij } correspond to the diagonalizing matrix . See Appendix J for more details . This time , calculation of the optimal test-statistic through the LR yields a minimum over ( T m ) possible subsets of decreased entries , which is computationally heavy . However , Theorem 4.3 shows that if we optimize for small values of ( where optimality is indeed most valuable ) , a near-optimal statistic is spart , which is the sum of the m = p · T smallest time-steps of ( X − µ ) after a Σ−10 - transformation ( see formal definition in Definition I.11 ) . We define the Partial Degradation Test ( PDT ) to be a threshold-test on spart with a parameter p. Theorem 4.3 ( Near-optimal test for uniform degradation ) . Assume that X is multivariate normal , and let Pα be the maximal power of a hypothesis test with significance 1 − α . The power of a threshold-test on spart with significance 1− α is Pα −O ( ) . Proof Sketch . The expression that is minimized is a sum of two terms . One term is the sum of a subset of entries of Σ−1 ( X − µ ) , which is minimized by simply taking the lowest entries ( up to the constraint of consistency across episodes , which requires us to sum the rewards per time-step in advance ) . In Appendix J we bound the second term and its effects on the modified statistic and on the modified test-threshold . We show that the resulted decrease of rejection probability isO ( ) .

Summary: This paper considers the drift detection for episodic data, where data episodes are assumed to be i.i.d. but data within each episodic can be correlated. It is assumed that the pre-change (nominal) mean and covariance of each episodic is perfectly known or can be accurately estimated from reference data. The Uniform Degradation Test (UDT) and Partial Degradation Test (PDT) are proposed to detect the mean shift. Moreover, this paper uses bootstrap to control the false alarm rate by setting the threshold as empirical quantiles of the detection statistic computed from reference data.