Graph-Augmented Normalizing Flows for Anomaly Detection of Multiple Time Series

1 INTRODUCTION . Anomaly detection ( Pimentel et al. , 2014 ; Ruff et al. , 2018 ) is the task of identifying unusual samples that significantly deviate from the majority of the data instances . It is applied in a broad variety of domains , including risk management ( Aven , 2016 ) , video surveillance ( Kiran et al. , 2018 ) , adversarial example detection ( Grosse et al. , 2017 ) , and fraud detection ( Roy & George , 2017 ) . Representative classical methods for anomaly detection are one-class support vector machines ( Schölkopf et al. , 2001 ) and kernel density estimation ( Parzen , 1962 ; Kim & Scott , 2012 ) . These methods rely on handcrafted features and often are not robust for high-dimensional data ( e.g. , images , speech signals , and time series ) . In recent years , inspired by the success of deep learning for complex data , many deep anomaly detection methods were proposed and they are remarkably effective in applications ( Ruff et al. , 2018 ; Sabokrou et al. , 2018 ; Goyal et al. , 2020 ) . Apart from these demonstrated applications , increasing demand exists for the anomaly detection of even more complex data ; i.e. , multiple time series . They contain a set of multivariate time series that often interact with each other in a system . A prominent example of the source of multiple time series is the power grid , where each constituent series is the grid state over time , recorded by a sensor deployed at a certain geographic location . The grid state includes many attributes ; e.g. , current magnitude and angle , voltage magnitude and angle , and frequency . Time series readings from sensors at nearby locations are often correlated and their behavior may be causal under cascading effects . Anomaly detection amounts to timely identifying abnormal grid conditions such as generator trip and insulator damage . Anomaly detection of multiple time series is rather challenging , due to high dimensionality , interdependency , and label scarcity . First , a straightforward approach is to concatenate the constituent series along the attribute dimension and apply a detection method for multivariate time series . However , when the system contains many constituents , the resulting data suffer high dimensionality . Second , constituent series bear intricate interdependencies , which may be implicit and challenging to model . When an explicit graph topology is known , graph neural networks are widely used to digest the rela- ∗This work was done while E. Dai was an intern at MIT-IBM Watson AI Lab , IBM Research . †To whom correspondence should be addressed . tional information ( Seo et al. , 2016 ; Li et al. , 2018b ; Yu et al. , 2018 ; Zhao et al. , 2019 ) . However , a graph may not always be known ( because , for example , it is sensitive information ) and hence graph structure learning becomes an indispensable component of the solution ( Kipf et al. , 2018 ; Wu et al. , 2020 ; Shang et al. , 2021 ; Deng & Hooi , 2021 ) . Third , labeling information is often limited . Even if certain labels are present , in practice , many anomalies may still stay unidentified because labeling is laborious and expensive . Hence , unsupervised approaches are the most suitable choice . However , albeit many unsupervised detection methods were proposed ( Ruff et al. , 2018 ; Sabokrou et al. , 2018 ; Malhotra et al. , 2016 ; Hendrycks et al. , 2019 ) , they are not effective for multiple time series . In this work , we explore the use of normalizing flows ( Dinh et al. , 2016 ; Papamakarios et al. , 2017 ) for anomaly detection , based on a hypothesis that anomalies often lie on low density regions of the data distribution . Normalizing flows are a class of deep generative models for learning the underlying distribution of data samples . They are unsupervised and they resolve the label scarcity challenge aforementioned . An advantage of normalizing flows is that they are particularly effective in estimating the density of any sample . A recent work by Rasul et al . ( 2021 ) extends normalizing flows for time series data by expressing the density of a series through successive conditioning on historical data and applying conditional flows to learn each conditional density , paving ways to build sophisticated flows for multiple time series . We address the high dimensionality and interdependency challenges by learning the relational structure among constituent series . To this end , Bayesian networks ( Pearl , 1985 ; 2000 ) that model causal relationships of variables are a principled choice . A Bayesian network is a directed acyclic graph ( DAG ) where a node is conditionally independent of its non-descendents given its parents . Such a structure allows factorizing the intractable joint density of all graph nodes into a product of easyto-evaluate conditional densities of each node . Hence , learning the relational structure among constituent series amounts to identifying a DAG that maximizes the densities of observed data . We propose a novel framework , GANF ( Graph-Augmented Normalizing Flow ) , to augment a normalizing flow with graph structure learning and to apply it for anomaly detection . There are nontrivial technical problems to resolve to materialize this framework : ( i ) How does one inject a graph into a normalizing flow , which essentially maps one distribution to another ? ( ii ) How does one learn a DAG , which is a discrete object , inside a continuous flow model ? The solution we take is to factorize the density of a multiple time series along the attribute , the temporal , and the series dimensions and use a graph-based dependency encoder to model the conditional densities resulting from factorization . Therein , the graph adjacency matrix is a continuous variable and we impose a differentiable constraint to ensure that the corresponding graph is acyclic ( Zheng et al. , 2018 ; Yu et al. , 2019 ) . We propose a joint training algorithm to optimize both the graph adjacency matrix and the flow parameters . In addition to resolving the high dimensionality and interdependency challenges , an advantage of modeling the relational graph structure among constituent series is that one can easily observe the dynamics of the data distribution from the graph . For time series datasets that span a long period , one naturally questions if the distribution changes over time . The graph structure is a useful indicator of distribution drift . We will study the graph evolution empirically observed . We highlight the following contributions of this work : • We propose a framework to augment a normalizing flow with graph structure learning , to model interdependencies exhibited inside multiple time series . • We apply the augmented flow model to detect anomalies in multiple time series data and perform extensive empirical evaluation to demonstrate its effectiveness on real-world data sets . • We study the evolution of the learned graph structure and identify distribution drift in time series data that span a long time period . 2 RELATED WORK . Anomaly Detection . Anomaly detection is a widely studied subject owing to its diverse applications . Recently , inspired by the success of deep learning , several deep anomaly detection methods are proposed and they achieve remarkable success on complex data , such as individual time series ( Malhotra et al. , 2016 ) , images ( Sabokrou et al. , 2018 ) , and videos ( Ionescu et al. , 2019 ) . These methods generally fall under three categories : deep one-class models , generative model-based methods , and transformation-based methods . Deep one-class models ( Ruff et al. , 2018 ; Wu et al. , 2019 ) treat normal instances as the target class and identify instances that do not belong to this class . In generative model-based methods ( Malhotra et al. , 2016 ; Nguyen et al. , 2019 ; Li et al. , 2018a ) , an autoencoder or a generative adversarial network is used to model the data distribution . Then , an anomaly measure is defined , such as the reconstruction error in autoencoding . Transformationbased methods ( Golan & El-Yaniv , 2018 ; Hendrycks et al. , 2019 ) are based on the premise that transformations applied to normal instances can be identified while anomalies not . Various transformations such as rotations and affine transforms have been investigated . On the other hand , anomaly detection of multiple time series is under explored . Recently , Deng & Hooi ( 2021 ) study the use of graph neural networks in combination with structure learning to detect anomalies . Our method substantially differs from this work in that the learned structure is a Bayesian network , which allows density estimation . Moreover , the Bayesian network identifies conditional dependencies among the constituent series and induces a better interpretation of the graph as well as the data distribution . Normalizing Flows . Normalizing flows are generative models that normalize complex real-world data distributions to “ standard ” distributions by using a sequence of invertible and differentiable transformations . Dinh et al . ( 2016 ) introduce a widely used normalizing flow architecture— RealNVP—for density estimation . Various extensions and improvements are proposed ( Papamakarios et al. , 2017 ; Hoogeboom et al. , 2019 ; Kingma & Dhariwal , 2018 ) . For example , Papamakarios et al . ( 2017 ) view an autoregressive model as a normalizing flow . To model temporal data , Rasul et al . ( 2021 ) use sequential models to parameterize conditional flows . Moreover , graph normalizing flows are proposed to handle graph structured data and improve predictions and generations ( Liu et al. , 2019 ) . In contrast , normalizing flows for multiple time series are rarely studied in the literature . In this work , we develop a graph-augmented flow for density estimation and anomaly detection of multiple time series . 3 PRELIMINARIES . We first recall key concepts and familiarize the reader with notations to be used throughout the paper . 3.1 NORMALIZING FLOWS . Let x ∈ RD be a D-dimensional random variable . A normalizing flow is a vector-valued invertible mapping f ( x ) : RD → RD that normalizes the distribution of x to a “ standard ” distribution ( or called base distribution ) . This distribution is usually taken to be an isotropic Gaussian or other ones that are easy to sample from and whose density is easy to evaluate . Let z = f ( x ) with probability density function q ( z ) . With the change-of-variable formula , we can express the density of the x , p ( x ) , by : log p ( x ) = log q ( f ( x ) ) + log |det∇xf ( x ) | . ( 1 ) In practical uses , the Jacobian determinant in ( 1 ) needs be easy to compute , so that the density p ( x ) can be evaluated . Moreover , as a generative model , the invertibility of f allows drawing new instances x = f−1 ( z ) through sampling the base distribution . One example of such f is the masked autoregressive flow ( Papamakarios et al. , 2017 ) , which yields z = [ z1 , . . . , zD ] from x = [ x1 , . . . , xD ] through zi = ( xi − µi ( x1 : i−1 ) ) exp ( αi ( x1 : i−1 ) ) , ( 2 ) where µi and αi are neural networks such as the multilayer perceptron . A flow may be augmented with conditional information h ∈ Rd with a possibly different dimension . Such a flow is a conditional flow and is denoted by f : RD × Rd → RD . The log-density of x conditioned on h admits the following formula : log p ( x|h ) = log q ( f ( x ; h ) ) + log |det∇xf ( x ; h ) | . ( 3 ) We now consider a normalizing flow for time series . Let X = [ x1 , x2 , . . . , xT ] denote a time series of length T , where xt ∈ RD . Through successive conditioning , the density of the time series can be written as : p ( X ) = p ( x1 ) p ( x2|x < 2 ) · · · p ( xT |x < T ) , ( 4 ) where x < t denotes all variables before time t. When the conditional probabilities are parameterized , Rasul et al . ( 2021 ) propose to model each p ( xt|x < t ) as p ( xt|ht−1 ) , where ht−1 summarizes the past information x < t . For example , ht−1 is the hidden state of a recurrent neural network before accepting input xt . Then , a conditional normalizing flow can be applied to evaluate each p ( xt|ht−1 ) .

Summary: The paper addresses the task of detecting anomalies in multiple time-series, for the setting where multiple instances of a fixed time-window are available for learning. The authors propose a method that involves using a directed acyclic graph (DAG) to model the (contemporaneous) dependencies between the variables in the time-series. An RNN is applied to form a representative state for each time-series at each point in time. A mapping is applied using the DAG to process the states from the previous time-step and the ancestor states from the current time-step to construct a “dependency representation” vector for each time series. A normalizing flow is then applied to map to a base distribution so that the log-density of the instance (the observed multiple time-series) can be evaluated. The conditional densities for each time series can also be evaluated. The paper includes an evaluation of the performance of the approach on a proprietary dataset, with a comparison to state-of-the-art baselines. Some qualitative results are also included for the task of density estimation for a public traffic dataset.