Graph-Augmented Normalizing Flows for Anomaly Detection of Multiple Time Series

1 INTRODUCTION . Anomaly detection ( Pimentel et al. , 2014 ; Ruff et al. , 2018 ) is the task of identifying unusual samples that significantly deviate from the majority of the data instances . It is applied in a broad variety of domains , including risk management ( Aven , 2016 ) , video surveillance ( Kiran et al. , 2018 ) , adversarial example detection ( Grosse et al. , 2017 ) , and fraud detection ( Roy & George , 2017 ) . Representative classical methods for anomaly detection are one-class support vector machines ( Schölkopf et al. , 2001 ) and kernel density estimation ( Parzen , 1962 ; Kim & Scott , 2012 ) . These methods rely on handcrafted features and often are not robust for high-dimensional data ( e.g. , images , speech signals , and time series ) . In recent years , inspired by the success of deep learning for complex data , many deep anomaly detection methods were proposed and they are remarkably effective in applications ( Ruff et al. , 2018 ; Sabokrou et al. , 2018 ; Goyal et al. , 2020 ) . Apart from these demonstrated applications , increasing demand exists for the anomaly detection of even more complex data ; i.e. , multiple time series . They contain a set of multivariate time series that often interact with each other in a system . A prominent example of the source of multiple time series is the power grid , where each constituent series is the grid state over time , recorded by a sensor deployed at a certain geographic location . The grid state includes many attributes ; e.g. , current magnitude and angle , voltage magnitude and angle , and frequency . Time series readings from sensors at nearby locations are often correlated and their behavior may be causal under cascading effects . Anomaly detection amounts to timely identifying abnormal grid conditions such as generator trip and insulator damage . Anomaly detection of multiple time series is rather challenging , due to high dimensionality , interdependency , and label scarcity . First , a straightforward approach is to concatenate the constituent series along the attribute dimension and apply a detection method for multivariate time series . However , when the system contains many constituents , the resulting data suffer high dimensionality . Second , constituent series bear intricate interdependencies , which may be implicit and challenging to model . When an explicit graph topology is known , graph neural networks are widely used to digest the rela- ∗This work was done while E. Dai was an intern at MIT-IBM Watson AI Lab , IBM Research . †To whom correspondence should be addressed . tional information ( Seo et al. , 2016 ; Li et al. , 2018b ; Yu et al. , 2018 ; Zhao et al. , 2019 ) . However , a graph may not always be known ( because , for example , it is sensitive information ) and hence graph structure learning becomes an indispensable component of the solution ( Kipf et al. , 2018 ; Wu et al. , 2020 ; Shang et al. , 2021 ; Deng & Hooi , 2021 ) . Third , labeling information is often limited . Even if certain labels are present , in practice , many anomalies may still stay unidentified because labeling is laborious and expensive . Hence , unsupervised approaches are the most suitable choice . However , albeit many unsupervised detection methods were proposed ( Ruff et al. , 2018 ; Sabokrou et al. , 2018 ; Malhotra et al. , 2016 ; Hendrycks et al. , 2019 ) , they are not effective for multiple time series . In this work , we explore the use of normalizing flows ( Dinh et al. , 2016 ; Papamakarios et al. , 2017 ) for anomaly detection , based on a hypothesis that anomalies often lie on low density regions of the data distribution . Normalizing flows are a class of deep generative models for learning the underlying distribution of data samples . They are unsupervised and they resolve the label scarcity challenge aforementioned . An advantage of normalizing flows is that they are particularly effective in estimating the density of any sample . A recent work by Rasul et al . ( 2021 ) extends normalizing flows for time series data by expressing the density of a series through successive conditioning on historical data and applying conditional flows to learn each conditional density , paving ways to build sophisticated flows for multiple time series . We address the high dimensionality and interdependency challenges by learning the relational structure among constituent series . To this end , Bayesian networks ( Pearl , 1985 ; 2000 ) that model causal relationships of variables are a principled choice . A Bayesian network is a directed acyclic graph ( DAG ) where a node is conditionally independent of its non-descendents given its parents . Such a structure allows factorizing the intractable joint density of all graph nodes into a product of easyto-evaluate conditional densities of each node . Hence , learning the relational structure among constituent series amounts to identifying a DAG that maximizes the densities of observed data . We propose a novel framework , GANF ( Graph-Augmented Normalizing Flow ) , to augment a normalizing flow with graph structure learning and to apply it for anomaly detection . There are nontrivial technical problems to resolve to materialize this framework : ( i ) How does one inject a graph into a normalizing flow , which essentially maps one distribution to another ? ( ii ) How does one learn a DAG , which is a discrete object , inside a continuous flow model ? The solution we take is to factorize the density of a multiple time series along the attribute , the temporal , and the series dimensions and use a graph-based dependency encoder to model the conditional densities resulting from factorization . Therein , the graph adjacency matrix is a continuous variable and we impose a differentiable constraint to ensure that the corresponding graph is acyclic ( Zheng et al. , 2018 ; Yu et al. , 2019 ) . We propose a joint training algorithm to optimize both the graph adjacency matrix and the flow parameters . In addition to resolving the high dimensionality and interdependency challenges , an advantage of modeling the relational graph structure among constituent series is that one can easily observe the dynamics of the data distribution from the graph . For time series datasets that span a long period , one naturally questions if the distribution changes over time . The graph structure is a useful indicator of distribution drift . We will study the graph evolution empirically observed . We highlight the following contributions of this work : • We propose a framework to augment a normalizing flow with graph structure learning , to model interdependencies exhibited inside multiple time series . • We apply the augmented flow model to detect anomalies in multiple time series data and perform extensive empirical evaluation to demonstrate its effectiveness on real-world data sets . • We study the evolution of the learned graph structure and identify distribution drift in time series data that span a long time period . 2 RELATED WORK . Anomaly Detection . Anomaly detection is a widely studied subject owing to its diverse applications . Recently , inspired by the success of deep learning , several deep anomaly detection methods are proposed and they achieve remarkable success on complex data , such as individual time series ( Malhotra et al. , 2016 ) , images ( Sabokrou et al. , 2018 ) , and videos ( Ionescu et al. , 2019 ) . These methods generally fall under three categories : deep one-class models , generative model-based methods , and transformation-based methods . Deep one-class models ( Ruff et al. , 2018 ; Wu et al. , 2019 ) treat normal instances as the target class and identify instances that do not belong to this class . In generative model-based methods ( Malhotra et al. , 2016 ; Nguyen et al. , 2019 ; Li et al. , 2018a ) , an autoencoder or a generative adversarial network is used to model the data distribution . Then , an anomaly measure is defined , such as the reconstruction error in autoencoding . Transformationbased methods ( Golan & El-Yaniv , 2018 ; Hendrycks et al. , 2019 ) are based on the premise that transformations applied to normal instances can be identified while anomalies not . Various transformations such as rotations and affine transforms have been investigated . On the other hand , anomaly detection of multiple time series is under explored . Recently , Deng & Hooi ( 2021 ) study the use of graph neural networks in combination with structure learning to detect anomalies . Our method substantially differs from this work in that the learned structure is a Bayesian network , which allows density estimation . Moreover , the Bayesian network identifies conditional dependencies among the constituent series and induces a better interpretation of the graph as well as the data distribution . Normalizing Flows . Normalizing flows are generative models that normalize complex real-world data distributions to “ standard ” distributions by using a sequence of invertible and differentiable transformations . Dinh et al . ( 2016 ) introduce a widely used normalizing flow architecture— RealNVP—for density estimation . Various extensions and improvements are proposed ( Papamakarios et al. , 2017 ; Hoogeboom et al. , 2019 ; Kingma & Dhariwal , 2018 ) . For example , Papamakarios et al . ( 2017 ) view an autoregressive model as a normalizing flow . To model temporal data , Rasul et al . ( 2021 ) use sequential models to parameterize conditional flows . Moreover , graph normalizing flows are proposed to handle graph structured data and improve predictions and generations ( Liu et al. , 2019 ) . In contrast , normalizing flows for multiple time series are rarely studied in the literature . In this work , we develop a graph-augmented flow for density estimation and anomaly detection of multiple time series . 3 PRELIMINARIES . We first recall key concepts and familiarize the reader with notations to be used throughout the paper . 3.1 NORMALIZING FLOWS . Let x ∈ RD be a D-dimensional random variable . A normalizing flow is a vector-valued invertible mapping f ( x ) : RD → RD that normalizes the distribution of x to a “ standard ” distribution ( or called base distribution ) . This distribution is usually taken to be an isotropic Gaussian or other ones that are easy to sample from and whose density is easy to evaluate . Let z = f ( x ) with probability density function q ( z ) . With the change-of-variable formula , we can express the density of the x , p ( x ) , by : log p ( x ) = log q ( f ( x ) ) + log |det∇xf ( x ) | . ( 1 ) In practical uses , the Jacobian determinant in ( 1 ) needs be easy to compute , so that the density p ( x ) can be evaluated . Moreover , as a generative model , the invertibility of f allows drawing new instances x = f−1 ( z ) through sampling the base distribution . One example of such f is the masked autoregressive flow ( Papamakarios et al. , 2017 ) , which yields z = [ z1 , . . . , zD ] from x = [ x1 , . . . , xD ] through zi = ( xi − µi ( x1 : i−1 ) ) exp ( αi ( x1 : i−1 ) ) , ( 2 ) where µi and αi are neural networks such as the multilayer perceptron . A flow may be augmented with conditional information h ∈ Rd with a possibly different dimension . Such a flow is a conditional flow and is denoted by f : RD × Rd → RD . The log-density of x conditioned on h admits the following formula : log p ( x|h ) = log q ( f ( x ; h ) ) + log |det∇xf ( x ; h ) | . ( 3 ) We now consider a normalizing flow for time series . Let X = [ x1 , x2 , . . . , xT ] denote a time series of length T , where xt ∈ RD . Through successive conditioning , the density of the time series can be written as : p ( X ) = p ( x1 ) p ( x2|x < 2 ) · · · p ( xT |x < T ) , ( 4 ) where x < t denotes all variables before time t. When the conditional probabilities are parameterized , Rasul et al . ( 2021 ) propose to model each p ( xt|x < t ) as p ( xt|ht−1 ) , where ht−1 summarizes the past information x < t . For example , ht−1 is the hidden state of a recurrent neural network before accepting input xt . Then , a conditional normalizing flow can be applied to evaluate each p ( xt|ht−1 ) .

Summary: The authors propose a model for anomaly detection on multivariate, high-dimensional time series data, where there are statistical dependencies across the different time series (referred to as "constituents" in the paper). The authors model statistical dependence using a Bayesian network and temporal dependence using an RNN. One key feature of the proposed model is that it simultaneously learns the structure of the Bayesian network AND a representation of the temporal state. The authors combine these two representations, which then gets used as input to a normalized flow technique, which makes computation of the likelihood of the data mathematically and computationally feasible.  The authors show that the proposed model has improved performance for the anomaly detection task compared to recently-proposed neural-network anomaly detection methods.