Context-invariant, multi-variate time series representations

1 INTRODUCTION . Many modern applications in the physical and virtual world are equipped with sensors that measure the state of the application , its sub-components , and the environment . Examples can be found in the Internet-of-Things ( IoT ) or in the DevOps/AIOPs space like monitoring wind turbines or cloudbased applications ( Lu et al. , 2009 ; Lohrmann & Kao , 2011 ; Nedelkoski et al. , 2019 ; Li et al. , 2020 ; Krupitzer et al. , 2020 ) . Leveraging such time series to identify abnormal appliance behaviour is appealing ( see Figures 1b,1c for an overview of time series anomaly types ) , yet certain characteristics of these time series make them difficult to model with existing representation learning techniques . First , time series corpora are often highly multi-variate as illustrated in Figure 1a . Each appliance has several sensors1 associated with it that measure both exogenous signals from the environment as well as endogenous signals from the internal state of the appliance . Examples for exogenous variables include user behaviour/traffic in a web-based application or physical measurements such as temperature in an IoT context . Conversely , endogenous variables could include the CPU usage or the vibrations of a machine . Increased ( application-internal ) network traffic is expected with higher user load , and higher ambient temperatures naturally result in elevated temperature of a wind turbine . It is however important to understand when an application deviates from such expected patterns and exhibits unexpected behaviour relative to its environment . We call such effects contextual anomalies . In addition , a defining characteristic of such time series corpora is the sparsity and noisiness of their associated labels . A label could indicate time spans when an application was in an a-typical state . This sparsity may be due to diverse reasons ranging from practical ( e.g. , data integration or cost of labelling ) to fundamental ( internal system failures may be exceedingly rare ) . Noisiness stems from the fact that failures are often subjective and human insight is needed or alarms come from rulebased systems that are themselves overly noisy ( Bogatinovski et al. , 2021 ; Wu & Keogh , 2021 ) . Hence , unsupervised or self-supervised representations of time series are needed that take the characteristics of such modern time series corpora into account . However , while the field of representation learning for sequential data has received considerable attention in domains s.a. natural language processing ( NLP ) ( Lan et al. , 2020 ; Mikolov et al. , 2013 ; Fang et al. , 2020 ; Jaiswal et al. , 2021 ) , 1Each of these sensors may also measure multiple statistics of the signal ( e.g. , min , max , avg , std ) . similar work in the numerical time series domain remains rare . Specifically , rich , multi-purpose representations facilitating down-stream applications are common in NLP . Instead , feature extraction methods mostly dominate for time series ( Lubba et al. , 2019 ; Christ et al. , 2017 ) with Franceschi et al . ( 2019 ) providing a notable exception based on temporal convolution networks ( TCN ) . The main contribution of our paper is the extension of this TCN-based approach to cater for the aforementioned complications . We summarize our contributions as follows : 1 . We propose context-invariant embeddings that allow to identify representations of time series that are invariant to the exogenous variables . We achieve this by adapting domain adversity ( Ganin et al. , 2016 ) to the time series domain . 2 . We extend the TCN ( Franceschi et al. , 2019 ) model with ( i ) modern contrastive losses that we lift to the time series domain for the first time , ( ii ) data augmentation techniques , and ( iii ) considering time series simultaneously at multiple resolutions.2 3 . We conduct an empirical study in which we show the effectiveness of our approach . We provide a semi-synthetic DevOps data set that we contribute to the research community and consider an under-explored wind turbine dataset ( apart from classical synthetic and physical datasets ) . Our quantitative results show that context-invariant embeddings indeed represent time series data such that contextual anomalies can be identified in a label-effective way . The qualitative results show that the embeddings allow us to navigate complex data sets in an explorative manner ( e.g. , considering nearest/farthest neighbours of interesting time series snippets ) . 2 REPRESENTATION LEARNING WITH CONTEXT INVARIANCE . To motivate our approach , consider a simplified system where under normal operation a single endogenous variable y depends instantaneously on a single exogenous signal x via a function y = g ( x ) + ε , where ε is a noise term . The ideal signal to detect contextual anomalies ( those that break this relation ) is the residual δ : = y − g ( x ) . Under normal operation this signal carries no information about the exogenous variable and thresholding the magnitude of this residual signal can detect anomalies . Our approach is motivated by this setup but extends it to more complex situations where ( i ) exogenous & endogenous variables can be multivariate , ( ii ) the relation stochastic & highly non-linear , and ( iii ) may depend on the history of the system state . In this case we can not simply compute a ” residual signal ” , but instead we can try to learn unsupervised representations that are invariant to the exogenous variable . This means the embeddings should be independent of the driving signal as long as the endogenous variables respond in a typical manner , which captures some aspect of the residual signal of the toy example . In the following sections we formalize this intuition further and show that context invariance indeed helps detect such anomalies . Let Z = { zi ∈ D T } Ni=1 be a set of N equally spaced time series zi of length at most T ∈ N where D is a domain of numerical values . We do not assume time series to be of equal length . We assume 2Indeed , time series corpora typically consist of equally-spaced time series ( e.g. , time series with measurements in 1-min , 5-min or 10-min intervals ) . This allows us to reason at multiple resolutions . a decomposition Z = X ∪ Y such that time series in X allow to predict time series in Y . We call X the set of environmental/exogenous time series and Y the set of internal/endogenous time series . We assume that it is possible to predict Y from X , but we make no assumption on causality . The goal of this paper is to map sub-series of Z into a high-dimensional embedding space RM which preserves loosely defined properties such as : “ normal ” time series are close to each other and far away from “ abnormal ” states . This facilitates down-stream tasks such as time series classification or anomaly detection in a label sparse setting . In particular , our definition of “ normal ” should be context-invariant , that is , only changes in the dependency structure between Y and X should result in large distances in the embedding space . For these tasks , a limited number of labels is available that allows to identify a time span of abnormal behaviour . Typically , the amount of labels is such that a supervised approach is prohibitive and even evaluation may be a challenge . Our representation learning approach consists of two main components : a predictor network g that ties the endogenous and exogenous time series together ( either by predicting endogenous from exogenous variables , or vice-versa ) , and , an embedding network f which learns embeddings using contrastive losses . We can combine both in multiple ways . One extreme is a two-step approach where we learn embeddings on the residuals of the predictor network . The other extreme is an end-to-end approach , where we learn embeddings such that the distance between ( multivariate ) time series is adjusted based on the exogenous variables in a domain-adversarial way . Figure 2 depicts the main components of our approach . For the predictor network , we mainly resort to standard models , so we focus our exposition on the main ( novel ) components in the following . 2.1 CONTRASTIVE , SELF-SUPERVISED , LEARNING OF MULTI-RESOLUTION TCN NETWORK . The basic building block of our embedding network architecture ( Franceschi et al. , 2019 ) consists of stacked temporal dilated causal convolutions ( Bai et al. , 2018 ) . We have multiple such networks , one per time resolution . We illustrate in Figure 4 the effect of aggregation on the input time series . To obtain a consolidated representation , the concatenated representations are mapped through a neural network . These multi-resolution representations allow the network to encode patterns that are more pronounced in the higher resolutions of the time series in a way that is more effective than an encoder which only operates on a single resolution . We choose resolutions manually as the natural granularities corresponding to the base frequency of the time series we consider in our empirical studies ( e.g. , seconds , minutes , hours ) . Similar to ( Franceschi et al. , 2019 ) , we rely on a contrastive , self-supervised learning approach to train the embedding network . This crucially relies on a loss function and a careful selection of positive ( a , b ) p ∈ ( X , Y ) , reference ( c , d ) c ∈ ( X , Y ) and negative ( x , y ) n ∈ ( X , Y ) time series snippets on which to compute the loss terms ( depicted in Figure 2 ) . Similar time series should be close to each other and dissimilar time series distant from each other in the embedding space . For an embedding network fW with parameters W , the loss function takes the following general form : min W dist ( fW ( ( c , d ) c ) , fW ( ( a , b ) p ) ) + max W dist ( fW ( ( x , y ) n , ( a , b ) p ) ) ( 1 ) We choose ( a , b ) p , ( c , d ) c such that ( c , d ) c ⊇ ( a , b ) p while ( x , y ) n is such that x ∩ c ≈ ∅ , y ∩ d ≈ ∅ ( e.g. , time snippets at different times and from different elements in the batch ) . Note further that ( x , y ) n is constructed to explicitly break the dependency structure in Z by choosing x to be the exogenous variables at a different time than y . During training , we further augment the examples randomly before feeding them to the TCN network . In particular we apply random jittering , scaling , flipping direction , 2d rotation around a center , permuting random segments , magnitude or time warping ( Um et al. , 2017 ) and window slicing or wrapping ( Guennec et al. , 2016 ) . Equation ( 1 ) is designed to support a variety of contrastive loss functions . Apart from the loss discussed in Franceschi et al . ( 2019 ) , we rely on other , more recent losses which we describe in the following briefly . These losses , in particular the latter two , aim to avoid collapse of the embeddings while taking practical consideration ( e.g. , the size of the batch ) into account . The SimCLR ( Chen et al. , 2020 ) takes two random windows zA and zB of a time series and encodes it to get two representations hA and hB . It then maximizes the similarity between these two representations from the same time series and dissimilarity between others representations in the batch using the Normalized Temperature-Scaled Cross-Entropy loss ( Sohn , 2016 ) as the distance in ( 1 ) . Formally , for temperature parameter τ and mini-batch size of N for each pair we define the loss as : ℓA , B = − log exp ( dot ( hA , hB ) /τ ) ∑2N n=1 , n 6=A exp ( dot ( hA , hn ) /τ ) , ( 2 ) where dot is the dot-product between ℓ2-normalized vectors . This contrastive loss benefits from larger N which might not be feasible in the time series setting and thus we also explore other losses . Barlow Twins ( Zbontar et al. , 2021 ) is a loss operating on two batches of different windows from the same respective time series embeddings , ZA and ZB . It computes the cross-correlation matrix along the batch dimension and stores the result in a square matrix C. The final loss then encourages the diagonal terms in this matrix to be close to 1 and the off-diagonal terms be close to 0 . Formally , ℓ = ∑ i ( 1− Cii ) 2 + λ ∑ i ∑ i 6=j C2ij , ( 3 ) where λ > 0 trades off the contribution of the first and second term in the loss . Intuitively , this decorrelation reduces the redundancy between the output embeddings forcing them to contain nonredundant information about the time series . Unlike SimCLR , MoCo ( He et al. , 2020 ) uses two encoders to obtain representations for the two random windows from the same time series . The representations through the 2nd momentum encoder are preserved in a queue . During training , positive pairs in ( 2 ) are constructed from the current batch while negative pairs ( denominator of ( 2 ) are constructed from the queue of embeddings . The 2nd encoder is updated by linear interpolation of the two encoder with a momentum-based moving average of their weights during training . By using a queue with a slowly changing encoder , this loss attempts to construct large and consistent embeddings which better samples the continuous high dimensional space , independent of the batch size .

Summary: This article deals with representation of multivariate signals in the case where we can distinguish endogenous & exogenous channels. This article mainly relies on (Franceschi et al.,2019), whose proposal consists in representing signals using Dilated-TCNN and to apply triplet loss on the sequences as in word2vec. Then the authors introduce 3 cases: normal behavior, anomalies where both endogenous & exogenous are impacted and partial anomalies. The encoded representation of a contextualized signal should be close to its temporal neigbors. Then, the authors tackle prediction & classification tasks on M5 and electricity datasets. The experimental part is a little bit confusing, as the datasets used in the different experiments are not the same. Generally speaking, results are very difficult to interpret.