Context-invariant, multi-variate time series representations

1 INTRODUCTION . Many modern applications in the physical and virtual world are equipped with sensors that measure the state of the application , its sub-components , and the environment . Examples can be found in the Internet-of-Things ( IoT ) or in the DevOps/AIOPs space like monitoring wind turbines or cloudbased applications ( Lu et al. , 2009 ; Lohrmann & Kao , 2011 ; Nedelkoski et al. , 2019 ; Li et al. , 2020 ; Krupitzer et al. , 2020 ) . Leveraging such time series to identify abnormal appliance behaviour is appealing ( see Figures 1b,1c for an overview of time series anomaly types ) , yet certain characteristics of these time series make them difficult to model with existing representation learning techniques . First , time series corpora are often highly multi-variate as illustrated in Figure 1a . Each appliance has several sensors1 associated with it that measure both exogenous signals from the environment as well as endogenous signals from the internal state of the appliance . Examples for exogenous variables include user behaviour/traffic in a web-based application or physical measurements such as temperature in an IoT context . Conversely , endogenous variables could include the CPU usage or the vibrations of a machine . Increased ( application-internal ) network traffic is expected with higher user load , and higher ambient temperatures naturally result in elevated temperature of a wind turbine . It is however important to understand when an application deviates from such expected patterns and exhibits unexpected behaviour relative to its environment . We call such effects contextual anomalies . In addition , a defining characteristic of such time series corpora is the sparsity and noisiness of their associated labels . A label could indicate time spans when an application was in an a-typical state . This sparsity may be due to diverse reasons ranging from practical ( e.g. , data integration or cost of labelling ) to fundamental ( internal system failures may be exceedingly rare ) . Noisiness stems from the fact that failures are often subjective and human insight is needed or alarms come from rulebased systems that are themselves overly noisy ( Bogatinovski et al. , 2021 ; Wu & Keogh , 2021 ) . Hence , unsupervised or self-supervised representations of time series are needed that take the characteristics of such modern time series corpora into account . However , while the field of representation learning for sequential data has received considerable attention in domains s.a. natural language processing ( NLP ) ( Lan et al. , 2020 ; Mikolov et al. , 2013 ; Fang et al. , 2020 ; Jaiswal et al. , 2021 ) , 1Each of these sensors may also measure multiple statistics of the signal ( e.g. , min , max , avg , std ) . similar work in the numerical time series domain remains rare . Specifically , rich , multi-purpose representations facilitating down-stream applications are common in NLP . Instead , feature extraction methods mostly dominate for time series ( Lubba et al. , 2019 ; Christ et al. , 2017 ) with Franceschi et al . ( 2019 ) providing a notable exception based on temporal convolution networks ( TCN ) . The main contribution of our paper is the extension of this TCN-based approach to cater for the aforementioned complications . We summarize our contributions as follows : 1 . We propose context-invariant embeddings that allow to identify representations of time series that are invariant to the exogenous variables . We achieve this by adapting domain adversity ( Ganin et al. , 2016 ) to the time series domain . 2 . We extend the TCN ( Franceschi et al. , 2019 ) model with ( i ) modern contrastive losses that we lift to the time series domain for the first time , ( ii ) data augmentation techniques , and ( iii ) considering time series simultaneously at multiple resolutions.2 3 . We conduct an empirical study in which we show the effectiveness of our approach . We provide a semi-synthetic DevOps data set that we contribute to the research community and consider an under-explored wind turbine dataset ( apart from classical synthetic and physical datasets ) . Our quantitative results show that context-invariant embeddings indeed represent time series data such that contextual anomalies can be identified in a label-effective way . The qualitative results show that the embeddings allow us to navigate complex data sets in an explorative manner ( e.g. , considering nearest/farthest neighbours of interesting time series snippets ) . 2 REPRESENTATION LEARNING WITH CONTEXT INVARIANCE . To motivate our approach , consider a simplified system where under normal operation a single endogenous variable y depends instantaneously on a single exogenous signal x via a function y = g ( x ) + ε , where ε is a noise term . The ideal signal to detect contextual anomalies ( those that break this relation ) is the residual δ : = y − g ( x ) . Under normal operation this signal carries no information about the exogenous variable and thresholding the magnitude of this residual signal can detect anomalies . Our approach is motivated by this setup but extends it to more complex situations where ( i ) exogenous & endogenous variables can be multivariate , ( ii ) the relation stochastic & highly non-linear , and ( iii ) may depend on the history of the system state . In this case we can not simply compute a ” residual signal ” , but instead we can try to learn unsupervised representations that are invariant to the exogenous variable . This means the embeddings should be independent of the driving signal as long as the endogenous variables respond in a typical manner , which captures some aspect of the residual signal of the toy example . In the following sections we formalize this intuition further and show that context invariance indeed helps detect such anomalies . Let Z = { zi ∈ D T } Ni=1 be a set of N equally spaced time series zi of length at most T ∈ N where D is a domain of numerical values . We do not assume time series to be of equal length . We assume 2Indeed , time series corpora typically consist of equally-spaced time series ( e.g. , time series with measurements in 1-min , 5-min or 10-min intervals ) . This allows us to reason at multiple resolutions . a decomposition Z = X ∪ Y such that time series in X allow to predict time series in Y . We call X the set of environmental/exogenous time series and Y the set of internal/endogenous time series . We assume that it is possible to predict Y from X , but we make no assumption on causality . The goal of this paper is to map sub-series of Z into a high-dimensional embedding space RM which preserves loosely defined properties such as : “ normal ” time series are close to each other and far away from “ abnormal ” states . This facilitates down-stream tasks such as time series classification or anomaly detection in a label sparse setting . In particular , our definition of “ normal ” should be context-invariant , that is , only changes in the dependency structure between Y and X should result in large distances in the embedding space . For these tasks , a limited number of labels is available that allows to identify a time span of abnormal behaviour . Typically , the amount of labels is such that a supervised approach is prohibitive and even evaluation may be a challenge . Our representation learning approach consists of two main components : a predictor network g that ties the endogenous and exogenous time series together ( either by predicting endogenous from exogenous variables , or vice-versa ) , and , an embedding network f which learns embeddings using contrastive losses . We can combine both in multiple ways . One extreme is a two-step approach where we learn embeddings on the residuals of the predictor network . The other extreme is an end-to-end approach , where we learn embeddings such that the distance between ( multivariate ) time series is adjusted based on the exogenous variables in a domain-adversarial way . Figure 2 depicts the main components of our approach . For the predictor network , we mainly resort to standard models , so we focus our exposition on the main ( novel ) components in the following . 2.1 CONTRASTIVE , SELF-SUPERVISED , LEARNING OF MULTI-RESOLUTION TCN NETWORK . The basic building block of our embedding network architecture ( Franceschi et al. , 2019 ) consists of stacked temporal dilated causal convolutions ( Bai et al. , 2018 ) . We have multiple such networks , one per time resolution . We illustrate in Figure 4 the effect of aggregation on the input time series . To obtain a consolidated representation , the concatenated representations are mapped through a neural network . These multi-resolution representations allow the network to encode patterns that are more pronounced in the higher resolutions of the time series in a way that is more effective than an encoder which only operates on a single resolution . We choose resolutions manually as the natural granularities corresponding to the base frequency of the time series we consider in our empirical studies ( e.g. , seconds , minutes , hours ) . Similar to ( Franceschi et al. , 2019 ) , we rely on a contrastive , self-supervised learning approach to train the embedding network . This crucially relies on a loss function and a careful selection of positive ( a , b ) p ∈ ( X , Y ) , reference ( c , d ) c ∈ ( X , Y ) and negative ( x , y ) n ∈ ( X , Y ) time series snippets on which to compute the loss terms ( depicted in Figure 2 ) . Similar time series should be close to each other and dissimilar time series distant from each other in the embedding space . For an embedding network fW with parameters W , the loss function takes the following general form : min W dist ( fW ( ( c , d ) c ) , fW ( ( a , b ) p ) ) + max W dist ( fW ( ( x , y ) n , ( a , b ) p ) ) ( 1 ) We choose ( a , b ) p , ( c , d ) c such that ( c , d ) c ⊇ ( a , b ) p while ( x , y ) n is such that x ∩ c ≈ ∅ , y ∩ d ≈ ∅ ( e.g. , time snippets at different times and from different elements in the batch ) . Note further that ( x , y ) n is constructed to explicitly break the dependency structure in Z by choosing x to be the exogenous variables at a different time than y . During training , we further augment the examples randomly before feeding them to the TCN network . In particular we apply random jittering , scaling , flipping direction , 2d rotation around a center , permuting random segments , magnitude or time warping ( Um et al. , 2017 ) and window slicing or wrapping ( Guennec et al. , 2016 ) . Equation ( 1 ) is designed to support a variety of contrastive loss functions . Apart from the loss discussed in Franceschi et al . ( 2019 ) , we rely on other , more recent losses which we describe in the following briefly . These losses , in particular the latter two , aim to avoid collapse of the embeddings while taking practical consideration ( e.g. , the size of the batch ) into account . The SimCLR ( Chen et al. , 2020 ) takes two random windows zA and zB of a time series and encodes it to get two representations hA and hB . It then maximizes the similarity between these two representations from the same time series and dissimilarity between others representations in the batch using the Normalized Temperature-Scaled Cross-Entropy loss ( Sohn , 2016 ) as the distance in ( 1 ) . Formally , for temperature parameter τ and mini-batch size of N for each pair we define the loss as : ℓA , B = − log exp ( dot ( hA , hB ) /τ ) ∑2N n=1 , n 6=A exp ( dot ( hA , hn ) /τ ) , ( 2 ) where dot is the dot-product between ℓ2-normalized vectors . This contrastive loss benefits from larger N which might not be feasible in the time series setting and thus we also explore other losses . Barlow Twins ( Zbontar et al. , 2021 ) is a loss operating on two batches of different windows from the same respective time series embeddings , ZA and ZB . It computes the cross-correlation matrix along the batch dimension and stores the result in a square matrix C. The final loss then encourages the diagonal terms in this matrix to be close to 1 and the off-diagonal terms be close to 0 . Formally , ℓ = ∑ i ( 1− Cii ) 2 + λ ∑ i ∑ i 6=j C2ij , ( 3 ) where λ > 0 trades off the contribution of the first and second term in the loss . Intuitively , this decorrelation reduces the redundancy between the output embeddings forcing them to contain nonredundant information about the time series . Unlike SimCLR , MoCo ( He et al. , 2020 ) uses two encoders to obtain representations for the two random windows from the same time series . The representations through the 2nd momentum encoder are preserved in a queue . During training , positive pairs in ( 2 ) are constructed from the current batch while negative pairs ( denominator of ( 2 ) are constructed from the queue of embeddings . The 2nd encoder is updated by linear interpolation of the two encoder with a momentum-based moving average of their weights during training . By using a queue with a slowly changing encoder , this loss attempts to construct large and consistent embeddings which better samples the continuous high dimensional space , independent of the batch size .

Summary: The paper presents a model for learning representations of time series. It addresses, in particular, the context of multivariate time series having exogenous and endogenous dimensions, and where the goal is to learn good representations independently from the context (exogenous dimensions). This work is an extension of existing work, the additions being the multivariate aspect and the context-invariant aspect. The context-invariant problem is treated thanks to an adversarial learning mechanism.  Experiments are conducted on synthetic and real datasets.