Context-invariant, multi-variate time series representations
Authors: Stephan Rabanser
Tim Januschowski
Kashif Rasul
Oliver Borchert
Richard Kurle
Jan Gasthaus
Michael Bohlke-Schneider
Nicolas Papernot
Valentin Flunkert
Venue: ICLR 2022 Submitted (2022)
Keywords: time series
representation learning
contrastive learning
context invariance
anomaly detection
domain adversarial learning

Abstract: Modern time series corpora, in particular those coming from sensor-based data, exhibit characteristics that have so far not been adequately addressed in the literature on representation learning for time series. In particular, such corpora often allow to distinguish between \emph{exogenous} signals that describe a context which influences a given appliance and \emph{endogenous} signals that describe the internal state of the appliance. We propose a temporal convolution network based embedding that improves on the state-of-the-art by incorporating recent advances in contrastive learning to the time series domain and by adopting a multi-resolution approach. Employing techniques borrowed from domain-adversarial learning, we achieve an invariance of the embeddings with respect to the context provided by the exogenous signal. To show the effectiveness of our approach, we contribute new data sets to the research community and use both new as well as existing data sets to empirically verify that we can separate normal from abnormal internal appliance behaviour independent of the external signals in data sets from IoT and DevOps.

Under review as a conference paper at ICLR 2022
CONTEXT-INVARIANT, MULTI-VARIATE TIME SERIES REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Modern time series corpora, in particular those coming from sensor-based data, exhibit characteristics that have far not been adequately addressed in the litera- ture on representation learning for time series. In particular, such corpora often al- low to distinguish between exogenous signals that describe context which influ- ences a given appliance and endogenous signals that describe the internal state of the appliance. We propose temporal convolution network based embedding that improves on the satt--of-the-ar by incorporating recent advances in contrastive learning to the time series domain and by adopting a multi-resolution approach. Employing techniques borrowed from comain-adversariin learning, we achieve an invariance the embeddings with respect to the context provided by the exoge- nous signal. To show the effectiveness of our approach, we contribute new data sets to the research community and use both new as well as existing data sets to empirically verify that can separate normal from abnormal internal appliance behaviour independent the external signals in data sets from IoT and DevOps.
INTRODUCTION
Many modern applications in the physical and virtual world are equipped with sensors that measure the state of the application, its sub-components, and the environment. Examples can be found in the Internet-of-Thine Things (IoT)) or in the DevOps/AIOP: space like monitoring wind turbines or cloud- based applications (Lu et al. 2009 Lohrmann Kao 2011 Nedelkoski et al.) 2019: Li et al. 2020) Krupitzer et al. 2020). Leveraging such time series to identify abnormal appliance behaviour is ap- pealing (see Figures 1b Ic for an overview time series anomaly types), yet certain characteristics
of these time series make them difficult to model with existing representation learning techniques.
First, time series corpora are often highly multi-variate as illustrated in Figure La Each appliance has several sensors] associated with it that measure both exogenouy signals from the environment as well as endogenous signals from the internal state of the appliance. Examples for exogenous variables include user behaviour/traffic in a web-based application or physical measurements such as temperature in an IoT context. Conversely, endogenous variables could include the CPU usage or the vibrations of a machine. Increased (application-inteman user load, and higher ambient temperatures naturally result in elevated temperature of a wind turbine. It is however important to understand when an application deviates from such expected patterns and exhibits unexpected behaviour relative to its environment. We call such effects contextual anomalies.
network traffic expected with higher
In addition, a defining characteristic of such time series corpora is the sparsity and noisiness their associated labels. A label could indicate time spans when an application was in an a-typical state. This sparsity may be due to diverse reasons ranging from practical (e.g., data integration or cost of labelling) to fundamental (internal system failures may be exceedingly rare). Noisiness stems from the fact that failures are often subjective and human insight is needed or alarms come from rule- based systems that are themselves overly noisy (Bogatinovski et al. 2021 Wu & Keogh 2021).
Hence, unsupervised or self-supervised representations of time series are needed that take the char- acteristics of such modern time series corpora into account. However, while the field of representa- tion learning for sequential data has received considerable attention domains s.a. natural language
processing (NLP) (Lan et al. 2020 Mikolov al. 2013 Fang al.! 2020 Jaiswal et al. 202
'ach of these sensors may also measure multiple statistics of the signal (e.g., min, max, avg, std).

---

Under review as a conference paper at ICLR 2022
Server
Exugenous
Endegemous
0.8
Request
CPU
0.5
0.6
Net tratiic
RAM usage
0.4
0.2
pal
std
min
BBax
(a) DevOps signal hierarchy ex- ample. Each entity contains ex- ogenous and endogenous sensors and each sensor is comprised of different summary statistics.
The time se- (c) Contextual anomaly: The time ries is anomalous w.r. its typical
(b) Pattern anomaly:
series is anomalous w.r. context from another time series. The de- pendency structure between exoge- nous and endogenous variables is changed.
behaviour. The depency structure
between exogenous and endogenous
variables is preserved.
Figure 1: Structure of time series corpora (left) and different types of anomaly (middle and right).
similar work in the numerical time series domain remains rare. Specifically, rich, multi-purpose rep- resentations facilitating down-stream applications are common in NLP. Instead, feature extraction methods mostly dominate for time series (Lubba et al.) 2019 Christ et al., 2017) with [Fanceschi] et al. (2019) providing notable exception based on temporal convolution networks (TCN). The main contribution of our paper the extension of this TCN-based approach to cater for the afore- mentioned complications. We summarize our contributions as follows:
We propose context-invarian embeddings that allow to identify representations of time se- ries that are invariant to the exogenous variables. We achieve this by adapting domain adver- sity (Ganin et al. (016)) to the time series domain.
2. We extend the TCN (Franceschi al. 2019) model with (i) modern contrastive losses that we lift to the time series domain for the first time, (ii) data augmentation techniques, and (iii) considering time series simultaneously at multiple resolutions! [
3. We conduct an empirical study in which we show the effectiveness of our approach. We provide
semi-synthetic DevOps data set that contribute to the research community and consider an under-explored wind turbine dataset (apart from classical synthetic and physical datasets).
Our quantitative results show that context-invariant embeddings indeed represent time series data such that contextual anomalies can be identified in a label-effective way. The qualitative results show that the embeddings allow navigate complex data sets in an explorative manner (e.g., considering nearest/farthest neighbours of interesting time series snippets).
2 REPRESENTATION LEARNING WITH CONTEXT INVARIANCE
To motivate approach, consider a simplified system where under normal operation single endogenous variable y depends instantaneously on a single exogenous signal x via a function y = g(x) + where is noise term. The ideal signal to detect contextual anomalies (those that break this relation) is the residual ó := g(x). Under normal operation this signal carries information about the exogenous variable and thresholding the magnitude of this residual signal can detect anomalies. Our approach is motivated by this setup but extends it to more complex situ- ations where (i) exogenous & endogenous variables can be multivariate, (ii) the relation stochastic & highly non-linear, and (iii) may depend on the history of the system state. In this case we cannot simply compute "residual signal", but instead we can try learn unsupervised representations that are invariant to the exogenous variable. This means the embeddings should be independent of the driving signal as long as the endogenous variables respond in a typical manner, which captures some aspect of the residual signal of the toy example. In the following sections we formalize this intuition further and show that context invariance indeed helps detect such anomalies.
Let Z {z, E D be set of N equally spaced time series 2; of length at most 7 E N where D is a domain of numerical values. We do not assume time series to be of equal length. We assume
Indeed, time series corpora typically consist of equally-spaced time series (e.g., time series with measure- ments in 1-min, 5-min 10-min intervals). This allows reason multiple resolutions.
2

---

Under review as a conference paper at ICLR 2022
decomposition Z XUU such that time series in X allow to predict time series in Y. We call X the set of environmental/cxoeenuuu time series and Y the set of internl/endogeenuus time series. We assume that it is possible to predict Y from X, but we make no assumption causality.
The goal of this paper is to map sub-series of Z into a high- high-dimensional embedding space RM which preserves loosely defined properties such as: "normal" time series are close each other and far away from "abnormal" states. This facilitates down-stream tasks such as time series classification or anomaly detection in label sparse setting. In particular, our definition of "normal" should be context- context-invarint,, that is, only changes in the dependency structure between Y and X should result in large distances in the embedding space. For these tasks, a limited number of labels is available that allows to identify a time span of abnormal behaviour. Typically, the amount of labels is such that a supervised approach prohibitive and even evaluation may be challenge.
Our representation learning approach consists of two main components: predictor network g that ties the endogenous and exogenous time series together (either by predicting endogenous from ex- ogenous variables, or vice-versa), and, an embedding network f which learns embeddings using contrastive losses. We can combine both in multiple ways. One extreme is a two-step approach where we learn embeddings end-to-end approach, where we learn embeddings such that the distance between (multivariate) time series is adjusted based on the exogenous variables in main components of our approach. For the predictor network, we mainly resort to standard models, so we focus exposition the main (novel) components in the following.
the residuals of the predictor network. The other extreme is an
way. Figure deepicss the
2.1 CONTRASTIVE, SELF-SUPERVISED. LEARNING OF MULTI-RESOLUTION TCN NETWORK
The basic building block of our embedding network architecture (Franceschi et al. 2019) consists of stacked temporal dilated causal convolutions (Bai et al.) 2018). We have multiple such networks, one per time resolution. We illustrate in Figure 4 the effect of aggregation on the input time series. To obtain a consolidated representation, the concatenated representations are mapped through a neural network. These multi-resolution representations allow the network to encode patterns that are more pronounced in the higher resolutions of the time series in a way that is more effective than encoder which only operates on single resolution. We choose resolutions manually as the natural granularities corresponding to the base frequency of the time series we consider in our empirical studies (e.g., seconds, minutes, hours).
Similar to (Franceschi et al. 2019), we rely on a contrastive, self-supervised learning approach to train the embedding network. This crucially relies on loss function and a careful selection of positive (a, h)p E (X, Y), reference (c, d). E (X, Y) and negative (I, y), E (X, Y) time series snippets on which to compute the loss terms (depicted in Figure 2 Similar time series should be close to each other and dissimilar time series distant from each other in the embedding space. For
an embedding network fw with parameters W, the loss function takes the following general form:
min dist(fw ((c, dlc), fw ((a, b)p)) + max dist ((x, v)ns (a, b)p)) W
(1)
W
We choose (a, b)p. (c, d). such that (c, d), 2 (a, b), while (z, v)n is such that æ n No 0, dddx de 0 (c.g.. time snippets differrnt times and from different elements in the batch). Note further that y), is constructed to explicitly break the dependency structure in 2 by choosing to be the exogenous variables at a different time than y. During training, we further augment the examples randomly before feeding them to the TCN network. In particular we apply random jittering, scaling, flipping direction, 2d rotation around a center, permuting random segments, magnitude or time warping (Um et al. 2017) and window slicing or wrapping (Guennec et al. 2016).
Equation is designed to support a variety of contrastive loss functions. Apart from the loss discussed in Franceschi et al. (2019), we rely on other, more recent losses which we describe in the following briefly. These losses, in particular the latter two, aim to avoid collapse of the embeddings while taking practical consideration (e.g., the size of the batch) into account.
The SiR (Chen et al. 2020) takes two random windows A and 2g of a time series and encodes it to get two representations h and hg. It then maximizes the similarity between these two repre- sentations from the same time series and dissimilarity between others representations in the batch using the Normalized Temperature-Scaled Cross-Entropy loss (Sohn) 2016) the distance in (T.
3

---

Under review as a conference paper at ICLR 2022
Embedding
X
fw
fw
fw
Domain
gu
gu
Contrastive component
adversarial
component
@
b
b
Series 1 exog.
Series endog.
b d
Series 2 exog.
Series 2 endog.
Figure 2: Context-invariant embeddings with contrastive and domain adversarial learning. We select negative samples such that the correlation structure between exogenous and endogenous sig- nals explicitly broken, For the other samples, we augment them randomly, denoted by the squiggly arrows. In addition to the contrastive component, we further add domain adversarial component that, given a positive embedded sample, tries to reconstruct the context of the positive sample as badly possible while reconstructing the context of the negative sample well possible.
Formally, for temperature parameter and mini-batch size of N for each pair define the loss exp(dot(ha,,s)) hb)/r)
aA log E Ln= exp(dot(h hn)/T)
where dot is the dot-product between fg-normalized vectors. This contrastive loss benefits from larger N which might not be feasible in the time series setting and thus we also explore other losses.
Barlow Twins Zbootar et al. 2021) is a loss operating the same respective time series embeddings. ZA and ZB. It computes the cross-correlation matrix square matrix C. The final loss then encourages
batches of different windows from
along the batch dimension and stores the result the diagonal terms in this matrix to be close to and the off-diagona terms be close to 0. Formally,
i {= 2 - C1) ΣΣ
(3)
$
where trades off the contribution of the first and second term in the loss. Intuitively, this decorrelation reduces the redundancy between the output embeddings forcing them to contain non- redundant information about the time series.
Unlike SimCLR, MoCo (He et al. 2020) uses two encoders to obtain representations for the two random windows from the same time series. The representations through the 2nd momentum en- coder are preserved in a queue. During training, positive pairs in (2) are constructed from the current batch while negative pairs (denominator of ( are constructed from the queue of embeddings. The 2nd encoder is updated by linear interpolation of the two encoder with a momentum-basse moving average their weights during training. By using a queue with a slowly changing encoder, this loss attempts to construct large and consistent embeddings which better samples the continuous high dimensional space, independent of the batch size.
2.2 COMBINATIONS PREDICTOR AND EMBEDDING NETWORK: FROM TRIVIAL TO CONTEXT-INVARIANT
We can combine the components described above to obtain context-invare embeddings in a num- ber of ways. The most trivial way is to ignore the predictor network gu and learn representations of

---

Under review as a conference paper at ICLR 2022
the entire set Z (or similarly, only for the endogenous part Y). We denote this idea as Bas i cEmb. Next, a two-step approach for -nnttxx-invariance consists of training the predictor network gu' first, then learn embeddings gu (Y) Y top of the residual signals in second step. We refer this approach as Res EmbRegr. Finally, we descrihe how to combine both networks into an end-to-end model which we call I InvEm.. This approach is the most flexible in the sense that it allows to adjust the strength of the context-invariang depending on the need of the application.
Our goal construct embeddings that are invariant the corresponding (exogenous) context. Put differently, the embeddings should not contain information that allows prediction of the correspond- ing exogenous context. This in turn requires to regress the exogenous signals against the embedding but instead of minimizing the regression error, we attempt to maximize the regression error. More formally, we have
R(W)- min
Σ
L.W b,W, À max Σ (,,,WU,,,,b). (a.b),
(4)
(2..),,
where (W, U, a, b, x, v) is a regression loss, W are the parameters of the encoder network fw. and U are the free parameters of the prediction network gu-
The loss in Equation aims to reconstruct the exogenous variables from embeddings as badly as possible, thereby pushing embeddings to invariance wrt the exogenous context. The first loss term leads to good predictions the exogenous variables shifted with respect to the embedded signals. The second term leads to bad predictions of exogenous variables from embeddings of positive examples (a, p- Gradient reversal handles the adversarial learning aspect (Ganin et al. 2016).
Instead of using to regression loss in 4. we resort to multi-class classification problem by discretizing the input space. This has attractive properties related tasks (Rabanser et al., 2020), but importantly for this particular application avoids explicitly handling trivial predictions (like g()) =oo) and domains are naturally bounded in the practical applications we consider. Combining with contrastive loss, arrive the following overall loss:
(WW) AR(W)),
where I(W) is a contrastive loss as discussed in Section 2.1 and R(W) is weighted by À (a hyper- paramater) and acts regularization term. For À > 0, we obtain context-invariant embeddings and for À= = 0, we recover (Franceschi et al. 2019) (modulo our extensions).
3 RELATED WORK
Computer vision (CV) and NLP Chen et al. 2020 van den Oord al. 2019 He et al. 2020 Fang
jet al. 2020: Jaiswal et al. 2021 have embraced self-supervised representations. Most relevant to us, Sohn et al. (2021) learn representations with contrastive loss to enable anomaly detection in CV. In contrast, time series analysis has not seen similar adoption of self-supervised techniques
for learning general-purpose representations. Franceschi et al. (2019) provide a notable exception
by proposing a TCN based embedding (Bai et al. 2018) learnt with a contrastive loss.
The approach of Franceschi et al. (2019) departs from rich field of feature extraction from time series (Lubba et al. 2019: Christ et al. (017). While these approaches indeed classify time series well in practice, they mostly Tocus on the uni-variate case. Their extensions to the multi-variate case are out-performed by Franceschi et al. (2019); Bagnall et al. (2018). Furthermore, the versatility of the features learned by classical approaches limited by the fact that distances in the induced embeddings are not properly learnt. We extend Franceschi et al. (2099) in the following directions: (i) we adopt to be multi--rsoolttinn: (ii) we equip with more recent contrastive loss functions and (iii) we turn into context-invarian embedding via
learning.
For (i), we note that the de facto choice in a multi-resolution context would be to sub-sample the time series. This is done in classical Wavelet analysis (Mallat 1989). Instead, we draw inspiration from temporal hierarchical time series analysis (Athanasopoulos et al. 2017) and opt to aggregate time series along the time dimension leading to vectors with fixed dimension. For (ii), adopt loss functions (He et al., 2020; Zbootar et al.) 202T] Chen et al. 2020) recently proposed for con- trastive and self-supervised learning and transfer them the time series domain. For (iii), draw
5

---

Under review as a conference paper at ICLR 2022
representation learning, primarily Ganin et al. (016)) While Ganin et al. (2016) learns embeddings with respect to a specific label classification task, we instead adopt an unsupervised approach via contrastive learning. Further, replace the domain classifier with an exogenous context regressor whose loss we seek to maximize. Such deep prediction networks can be sophisticated, as is the case in particular in the forecasting literature (see e.g., Benidis et al. (2020) for an overview). Our approach readily extends to these, but we restrict ourselves to standard neural regression models as they suffice in the scenarios that we consider for our empirical studies. Other approaches such as hypemetworks (Ha et al. (2166) are conceivable. Yet, they suffer from a lack of computational efficiency and robustness which inhibits their practical applicability.
on
4 EMPIRICAL EVALUATION
Our empirical evaluation consists of two parts. We first focus on dissecting the improvements on embedding learning through multi-resolution handling, contrastive losses and data-augmentation. Second, we examine the context-invarian representations towards their anomaly detection potential.
4.1 IMPROVEMENTS TO EMBEDDING
For this base experiment, we aim to show the versatility of the embeddings through a down-stream forecasting and classification task each of which we evaluate on two datasets representing the easy and the hard spectrum of the task. We do not aim for comprehensiveness in these first set of ex- periments but rather for an assessment of the relative improvements through our extensions. As a reference point to gauge the absolute accuracy better, include a classical feature extraction base- line (Lubba et al. 2019), Cat ch22. In our experiments, we address the downstream classification and forecasting task via simple linear models.
In classification, we consider a synthetically generated data set for which we know the labels from the data generation process and the M5 forecasting competition dataset (Theodorou et al.) [02T) The latter data set is a retail demand forecasting data set that has product categories associated with the time series. Using a (multinomial) logistic classifier, we aim to predict the labels in both data sets. The synthetic data set is designed such that a high classification accuracy can be and the labels are "objective"" In contrast, for the M5 data set, high classification accuracy cannot be expected. Apart from the amount of product categories available (==3k), time series associated to different products may have similar characteristics and hence, from a time series classification point of view, labels do not represent a ground truth, a common scenario in practice.
In forecasting, we predict electricity and M5 Makridakis & Spiliotis, 2021 using a shared linear forecaster. The forecaster takes the context embeddings from large time series window together with a smaller context window of the actual time series to predict for the dataset's target horizon. This horizon is 24 time steps ahead for electricity and 28 days for M5 and compare the metrics with the corresponding test splits.
Discussion. Table summarizes our findings. The more recent proposals for contrastive losses, McCo and Barlow Low Twins, are superior, but there is no clear indication which both losses is small
superior overall. While present, we remark that the relative improvements in accuracy when compared to the performance wins reported in the original domains for which these losses were designed. We speculate that this may be more attributable to the datasets in the respective domains (and the objectivity of the associated labels) than the loss functions themselves.
Note that including multi-resolution leads such overwhelming improvements in the classification task that we show only multi-resolution embeddings (for the base resolution, factor 60 and 300) for the classification tasks as these are almost strictly superior (> 10%). Similarly, we report re- sults in the classification task with data augmentation, although improvements are not consistent for data augmentation (Appendix B.2 contains results for an ablation study). The effectiveness of multi-resolution may be surprising higher capacity models in general and convolutions with higher dilation in particular should in theory be able to model similar effects. However, adding multiple resolutions offers inductive bias akin to lagged values in RNN-based forecasting models (c.g., Salinas et al. (2019) which have been shown to lead to superior practical results.
https://archi iss uci www. htt.a//ww...oooooaaaaaa///aaaeaa

---

Under review as a conference paper at ICLR 2022
Loss
Classlicaaiion S:Sym
Classifiattion 1: M5
Forecasting J: Electtiityy
Forecasting J MS
SAnCLR
97.00
1.34
0.097
0.70
Rorlow Twine
97.25
5,93
0.091
0.609
MaCo
99.50
3.08
0.089
Catch22
64.50
0.03
Lineaz
0.092
Docpte
0.073
0.50
Table I: Classification and forecasting accuracy using embeddings learnt with different losses. For forecasting, the accuracy metric shown is P50 loss. Results averages over 5 runs. The last three lines are baselines using classic time series features, Cat ch22, and replacing embeddings with more historic values for the forecasting task, Linear and a pure forecasting method, DeepAR.
For forecasting, we note that past historic values instead of embeddings (Linear in Table I is a competitive approach, in particular for forecasting the M5 data set. Nevertheless, higher quality embeddings coincide with better forecasting accuracy (comparing Catch22 2 with our embeddings). Compared with the Electricity dataset, the M5 dataset offers less overall structure so historic values offer a strong signal. Consequently, forecasting accuracy wins are more pronounced in the Electric- ity dataset.
4.2 CONTEXT INVARIANT EMBEDDINGS
In our main experiments, we consider the effeet of domain-adveesarrian learnt context-invarian embeddings in both qualitative and quantitative experiments. The task consists of the identification of contextual anomalies. Perhaps surprisingly, qualitative evaluation is almost more meaningful for the anomaly detection task in general, and for contextual anomalies in particular, given the subjectivity and noisiness of the labels. Note that most publicly available anomaly detection data set are suitable for the task, we do not consider them. Instead, evaluate on different data sets, each of which allows for a separation into exogenous and endogenous signals. We discuss these data sets first, then the evaluation approach, the models under consideration, and finally discuss the results. Appendix B.3 contains further details.
The selection of the evaluation datasets aims to balance physical and virtual appliances well as synthetic and real-world data. Note that synthetic data come with perfect labels, while real-world data typically does not. While lamentable, we believe that this subjectivity and noisiness must be embraced as fundamental in the task. Appendix A contains further details the data sets.
Datasets
Synthetic data. We generate a total of x 360 time series of length 700, based on simple generative models. We generate two exogenous signals, well as two endogenous signals. We inject two types of anomalies into the data: (i) pattern anomalies, i.e., anomalies in the exogenous which are also instantly reflected in the endogenous variables and (ii) contextual anomalies only in the endogenous variables. We aim to detect contextual anomalies.
Pendulum. We consider the case of a swinging pendulum with added control signals, where we control the dampening of the acceleration from the outside as exogenous signal (towards which we want to be invariant) and consider as contextual anomamlies those where we inject an anomalies as change in the length of the chord which we capture as part of the endogenous signal. Our aim with this data set is to understand how well our models handle cases where the dependency structure between X and Y is more complex.
DevOps. This is a new, semi-synthetic data se that we generated for the purpose of this publication to resemble commonly observed data sets behind corporate firewalls. The object under considera- tion is a popular cloud-based microservice demo application 5 which is commonly used in an AIOPs context et 2020). As exogenous signal, we record user interaction approximated by the net-
Wu Keogh (2021) convincingly argue that many these datasets should be abandoned.
Jointly with this publication, open-source both the raw data well as the set-up produce the data. To the best of our knowledge, we the first extract data set from this set-up that allows the machine learning community interact with this area without deep engineering knowledge which is more present in the system's s community where the data generation framework is typically considered.
https: thub. com/m / wwwseroiiiiccccoo demo/mi croservi ces-cemo

---

Under review as a conference paper at ICLR 2022
work outbound traffic of application that induces synthetic load the application and which we fully control. The endogenous signals consists of metrics like CPU, memory and others of the actual microservice application. Each recorded metric has multiple statistics available. We inject anomalies both in the user behavior (leading to pattern anomalies) and in the internal state (contex- tual anomalies). We want ignore the former and find the latter. The appendix contains illustrative plots similar to Figure 6. Note that although we have almost total control of the application and its anomalies, the labelling of anomalies is still not perfect, thereby adding further to the complications of public anomaly detection benchmarks (Wu& & Keogh 2021).
Turbine. We consider wind turbine data set open-sourced by Energias de Portugal. The time series panel can be separated into exogenous and endogenous signals. The former consists of wind speed/direction, ambient temperature and, the pitch angle of the blades (this controlled from the outside and including it improves the quality of the predictive model). The latter consists of rotation speeds for the turbine and generators, internal temperature on different components, as well as the power output each. All series are available for 4 distinct turbines, are sampled at 10 minute frequency and are available for 2016. Note that this data set contains only few (43) labelled anomalies and visual inspections of the data reveals inconsistencies with these labels, e.g., some time series appear to be mislabeled (see Sec. [422 an example). The providence of these labels is from automated alarming systems which are often threshold based. Hence, quantitative evaluations cannot be taken at face value. However, given the rich structure of the data and the fuzziness of the task stemming from the labels, the versatility of our approach can be illustrated qualitatively.
Models & Evaluation We evaluate the following model configurations for their suitability for the contextual anomaly detection task: BasicEmb, the modified (Franceschi al.) 2019) (with multi-resolution) learnt ignoring the structure imposed by endogenous and exogenous signals; ResEmb embeddings on the residuals of a predictive models (a feed-forward neural network); and Cont InvEmb, context-invariant embeddings with simple linear model for the prediction task (the simplicity ensure that the embeddings are adjusted enough). Moreover, we also provide results for three baseline approaches: Restresh, which computes residuals in Res EmbRegr com- bined with a simple thresholding mechanism; and two instances of (Lubba et al. 2019): Catch22 and and ResCatch22 which compute feature vector per original and residual time series specitvely similar to ResEmbRegr. The only hyperparameter tuning that we perform is on the Synthetic data set we do not have enough labels available otherwise.
For all methods, we report standard AUROC scores on the contextual anomaly detection task. For the ResThrest sh method, we compute the AUROC score based on the maximum residual value over the full residual series, For the embedding- approaches, we use a k-nearest-neighbor classifier in the embedding space to determine discrete anomaly label.
Quantiative Results Tables 2 summarizes the quantiative results. First, we note that the diffi- culty of contextual anomaly detection differs widely with the data sets. For example, contextual anomaly detection seems relatively casier on the Turbine data set compared with the DevOps data set despite the latter being sytiisyythetically generated with controlled labels. Second, we note that Cont InvEmb leads overall superior results when comparing the embedding based approaches (the first three columns in Table 2 and is overall competitive results in many cases, but not always. For the DevOps and Turbine data sets ResTresh is overall best by a margin. One explanation is that ResTresh best resembles the label generation process by the automated alarming systems on the turbines. For Pendulum data set it is worthwhile to note that Cont InvEmb delivers an overall superior approach despite the complex non-linear interaction between the exogenous and endoge- nous signals and only a linear context-predictor component.
Qualitative Results: case study on wind turbines The embeddings allow to navigate the time series corpus via distances which can be helpful in exploring the data set and uncovering data or label issues. In Figure 3 we show this in qualitative results for context-invarian embeddings. The first two columns show sanity checks: as expected, reference time series (in the top row) labelled as normal or abnormal have as their nearest neighbors (in the same column as the reference time series below it) normal and abnormal time series respectively. Despite the multi-variate nature of the data, visual inspection confirms that the nearest neighbors are plausible. The last column in Figure
https::/ wwtpiaattaeddd..... wwt.://e..pppcccopapaeaeaaeaa http://aaaaaaaaaaae
8

---

Under review as a conference paper at ICLR 2022
20
40
60
80
100
120
140
0
20
40
60
80
100
120
140
20
40
60
80
100
120
140
(a) Clean reference series (correctly
(b)
Anomalous reference series
(c) Anomalous reference series (in-
labelled as clean).
(correctly lahelled as anomalous).
correctly labelled clean).
40
140
Figure 3: Nearest-neighbor analysis of context-invariant approach on the Turbine dataset. Each panel shows exogenous signals on the top and endogenous signals at the bottom. The first row shows (a) normal reference series; (b) an abnormal reference series (anomaly between time steps 20 and 70); and (c) an anomalous reference series that was mistakenly labeled as clean. Below each of these series show 3 nearest neighbors in the learned embedding space. For the left and center column, both the correct label and our predicted label match. For series (c), the reference time series is labelled clean, but its nearest neighbors abnormal. It is apparent that the erratic spiking patterns in either exogenouss/endogenous signals are usually not reflected in the other series.
3esicCmb
RestinoRegt
Cont Trvilibb
Res"Irest
Catich22
ResCatcch22
Synchetic
0,512 0.022)
1,000 (::: 0.000)
0.999 (1: 0.002)
1,000 (:L: 0.000)
0.494 (::: 0,008)
1,000 (::: 0.000)
Pendulum
0,969 (:1: 0.013)
0.951 (:2 0.015)
0,980 0.002)
0.510 0.060)
0.904 (2: 0.000)
0.891 0.000)
DevOps
0.036)
0.007)
0.573 0.000)
0.000)
Turbine
0.632 (::: 0.015)
0.725 (:: 0.018)
0.736 0.022)
0,845 (:): 0,000)
0.512 (::: 0.000)
0,680 0.000)
Table 2: AUROC results on the anomaly detection task with 5 seeds. Larger values are better.
shows an example where reference time series is labelled as normal, but its nearest neighbors consist time series that are labeled as abnormal. This could point to an issues with the labels. In this case, without further domain knowledge, it may make sense re-label the reference time series in column (c) as abnormal for the abnormal stretches 0-40 and 100-130.
CONCLUSION
In this work, we presented self- learning of time series embeddings that are invariant with respect known and fully-observed context. While the architectures that we presented here lean on techniques invented for computer vision, we make non-trivial contributions to adapt them to the time series domain. For example, we equip our embeddings to consider multiple resolutions of the original sensor signals simultancously. We observe that the learned embeddings sensitive to changes of the dependency structure between exogenous and endogenous variables. As confirmed in our evaluation, this allows our approach to learn embeddings that separate dependency bbraakinn anomalies in the state the appliance which is the object of interest.
Potential future works could explore techniques from causal discovery (Haufe al. 2009 Qiu et al. 2020 2012) to automatically derive an exogenouu/endogeeooss decomposition the multi-variate time series panel and extensions to causal representation learning (Schölkopf al. 2021)

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY
We use a combination of public datasets and synthetic data sets generated using publicy available code. Method implementations will be open sourced part of the review process. Finally, will share the data sets, the models to create the synthetic data and the code with notebooks that allows reproduce all our results with the final version of this paper.
REFERENCES
George Athanasopoulos, Rob J. Hyndman, Nikolaos Kourentzes, and Fotios Petropoulos. Fore- casting with temporal hierarchies. European Journal of Operational Research, 262(1):60- 74, 2017. ISSN 0377-2217. doi: http://dbb.ogg///000000....0200.
URL https:
//www www. www.sscciicccccccco rect. wtt.cc/ccccccce
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flyun, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The multivariate time series classification archive, 2018, 2018.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, 2018.
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bernie Wang. Danielle Mad- dix, Caner Turkmen, Jan Gasthaus, Michael Wohlke-Schnnice David Salinas, Lorenzo Stella, Laurent Callot, and Tim Januschowski. Neural forecasting: Introduction and literature overview, 2020.
Jasmin Bogatinovski, Sasho Nedelkoski, Alexander Acker, Florian Schmidt, Thorsten Wittkopp, Soeren Becker, Jorge Cardoso, and Odej Kao. Artificial intelligence for it operations (aiops) workshop white paper, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Maximilian Christ, Andreas W. Kempa-Liehr, and Michael Feindt. Distributed and parallel time series feature extraction for industrial big data applications, 2017.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self-supervised learning for language understanding. 2020.
Jean-Yve Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. arXiv preprint 191.,1090 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversend training of neural net- works. The journal of machine learning research, 17(1):2096-22) 2016.
Arthur Le Guennec, Simon Malinowski, and Romain Tavenard. Data augmentation for time series classification using convolutional neural networks, 2016. URL https://halshs:
David Ha, Andrew Dai, and Quoe V. Le. Hypemetworks, 2016.
Stefan Haufe, Guido Nolte, Klaus-Robes Mueller, and Nicole Kraemer. Sparse causal discovery in multivariate time series, 2009.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Dehapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning, 2021.
Christian Krupitzer, Tim Wagenhals, Marwin Züfle, Veronika Lesch, Dominik Schäfer, Amin Mozaffarin, Janick Edinger, Christian Becker, and Samuel Kounev. A survey on predictive main- tenance for industry 4.0. arXiv preprint 2rXiv:2002.08224 2020.
10

---

Under review as a conference paper at ICLR 2022
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for supervieee learning of language representations, 2020.
Yangguang Li, Zhen Ming (Jack) Jiang, Heng Li, Ahmed E. Hassan, Cheng He, Ruirui Huang, Zhengda Zeng, Mian Wang, and Pinan Chen. Predicting node failures in ultra-lagg--scale cloud computing platform: An aiops solution. ACM Trans. Softw. Eng. Methodol., 29(2), April 2020. ISSN 1049-331X. doi: 10.1145/338517. URL https: / org/ 10. 1145/3385181
Bjöm Lohrmann and Odej Kao. Processing smart meter data streams in the cloud. 2011 2nd IEEE PES International Conference and Exhibition on Innovative Smart Grid Technologies, pp.
1-8, 2011. doi: 10.1 20IISGTTEurrpe221111177777
Bin Lu, Yaoyu Li, Xin Wu, and Zhongzhou Yang. A review of recent advances in wind turbine condition monitoring and fault diagnosis. In 2009 IEEE Power Electronics and Machines in Wind Applications, pp. 1-7, 2009, doi: 10.1 S0.10099PMWWA22005522033
Carl H Lubba, Sarab S Sethi, Philip Knaute, Simon R Schultz, Ben D Fulcher, and Nick S Jones. catch22: Canonical time-series characteristics. Data Mining and Knowledge Discovery, 33(6): 1821-1852, 2019,
Spyros Makridakis and Evangelos Spiliotis. The m5 competition and the future of human expertise in forecasting. Foresight: The International Journal of Applied Forecasting, (60):33-37, 2021.
S.G. Mallat, A theory for multiresolution signal decomposition: the wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 11(7):674-693, 1989. doi: 10.1109/ 34.192463.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. C. Burges, I.. Bottou, M. Welling, Z. Ghahramani, and K. Q Weinberger (eds.), Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc., 2013. URL http::::/ www.:/innnnnnnnunnnneeee.e.ue.ee cccpappe///1// http:///2222aaeeeeeeeee3322222e.
Sasho Nedelkoski, Jorge Cardoso, and Odej Kao. Anomaly detection from system tracing data using multimodal deep learning. In 2019 IEEE 12th International Conference on Cloud Computing doi: 10. 10.109//LOUU..201.000000
(CLOUD), pp. 179-186, 2019.
Huida Qiu, Yan Liu, Niranjan A. Subrahmanya, and Weichang Li. Granger causality for time-series anomaly detection. In Mohammed Javeed Zaki, Arno Siebes, Jeffrey Xu Yu, Bart Goethals, Geoffrey I. Webb, and Xindong Wu (eds.), 12th IEEE International Conference on Data Mining, ICDM 2012, Brussels, Belgium, December 10-13, 2012, pp. 1074-1079. IEEE Computer Society, 1 0 ICDM. 2012. 73.
2012. doi: 10.1 1,/1109/.M.22212...... URL http:::::/doooorrgoo.o
Juan Qiu, Qingfeng Du, Kanglin Yin, Shuang-Li Zhang, and Chongshu Qian. A causality mining and knowledge graph based method of root cause diagnosis for performance anomaly in cloud doi: 10.3390/app10002166. URL
applications. Applied Sciences, 10(6), 2020. ISSN 2076-3417.
https::/
wwww http:/wwwpppp...m/ www 2076 20t0:m/2200333110/////10 3411111//11111163
Stephan Rabanser, Tim Januschowski, Valentin Flunkert, David Salinas, and Jan Gasthaus. The effectiveness of discretization in forecasting: An empirical study neural time series models, 2020.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 2019.
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning, 2021.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objec- tive. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https:: /ttps:://rroceddnss pdf

---

Under review as a conference paper at ICLR 2022
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluating representations for deep one-class classification, 2021.
Evangelos Theodorou, Shengjie Wang, Yanfei Kang, Evangelos Spiliotis, Spyros Makridakis, and Vassilios Assimakopoulos. Exploring the representativeness of the m5 competition data, 2021.
Terry Um, Franz M. Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang. Sandra Hirche, Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for parkinson's 's disease monitoring using convolutional neural networks. Proceedings of the 19th ACM International Conference on Multimodal Interaction, Nov 2017. doi: 10.1145/31365553133611 URLhttp:
//dx. doi. 11tg///001114/33755555555 3136817
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Li Wu, Jasmin Bogatinovski. Sasho Nedelkoski, Johan Tordsson, and Odej Kao. Performance Di- agnosis in Cloud Microservices using Deep Learning, In AIOPS 2020 International Workshop on Artificial Intelligence for IT Operations, Dubai, United Arab Emirates, December 2020. URL
https://mmt hhal.al..... inria. fr /haahha-- 0 02948735
Renjie Wu and Eamonn Keogh, Current time series anomaly detection benchmarks flawed and are creating the illusion of progress. IEEE Transactions on Knowledge and Data Engineering, URL httt::/ddt doi.
pp. 1-1, 2021. ISSN 2326-3865. doi: 20.1009/kde.2221.31111
org/ 10. 10.91119/TDEE 2021. 01122..11220 3112126
Jure Zbontar, Li Jing, Ishan Misra. Yann Lecun, and Stephane Deny. Barlow twins: Self- supervised learning via redundancy reduction. Marina Meila and Tong Zhang (eds.), Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 13100-2320. PMLR, 18 24 Jul 2021. URL https:
//proceedings mlr. wwwpppeesssll.. zbonta ar la. html
12


Reviews:
[{"review_id": "qkgOn0xibjk", "reviewer": "Reviewer_N1hN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper adapts self-supervised learning to the time series analysis domain for the anomaly detection problem. Authors propose a contrastive component to distinguish similar/dissimilar time sires and a domain adversarial component to ensure the embeddings are context invariant. \nGenerally, I think the studied problem is relatively new and the model design contains novel ideas. However, I also have the following concerns:\n1. Although the overall idea and the whole concepts of each component are very clear, some details of the proposed method are not easy to get. For example:\n* What is one training example feed into the network ($N \\times T$ matrix contains both internal and external series or just one series?)? \n* How to deal with the input with different resolutions? \n* What is the extract calculation of $L_r(W, U, a, b, x, y)$ in Eq. 4? What does it mean “The first loss term leads to good predictions of the exogenous variables shifted with respect to the embedded signals”?\n2. The justification for identifying representations of time series that are invariant to the exogenous variables is not enough. Especially, why this is important and necessary?\n3. While the experimental results seem good, they are not very strong and supportive to the motivation. First, the experimental results are not better than the baseline supervised methods. I know it’s reasonable since you do not use labels to train the embedding part, but the classifier is still trained with labels. For these small-scale datasets, I doubt the representative ability of embeddings since the classifier may influence the performance hugely. Second, the most attractive point of self-supervised learning is the transferability of learned features, which is not evaluated in this paper. Unsupervised learning features and then supervised learning the classifier can not support the method for anomaly detection applications. \n", "summary_of_the_paper": "This paper adapts self-supervised learning to the time series analysis domain for the anomaly detection problem. Authors propose a contrastive component to distinguish similar/dissimilar time sires and a domain adversarial component to ensure the embeddings are context invariant. ", "summary_of_the_review": "The paper studies a relatively new topic and the design contains new ideas in the specific area. However, some details of the model are not presented clearly and the experimental design could be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SmSPRQQDijk", "reviewer": "Reviewer_krgP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The article is well written and easy to follow.\n* The model is elegant, well described.\n* The approach and the motivation are clearly stated.\n\nCons:\n* The experimental part is very light and does not convince me of the interest of the proposed approach. No other deep learning models are used as baseline (even if not adapted to the context-invariant setting) when there is a rich literature about anomaly detection in time series. I understand that the goal is not to evaluate the performances in classification/anomaly detection but the learned representation, but I can not judge the difficulty of the used datasets with the provided baselines.\n* The novelty of the paper is not great wrt the previous work, it consists mainly of the use of adversarial learning. ", "summary_of_the_paper": "The paper presents a model for learning representations of time series. It addresses, in particular, the context of multivariate time series having exogenous and endogenous dimensions, and where the goal is to learn good representations independently from the context (exogenous dimensions). This work is an extension of existing work, the additions being the multivariate aspect and the context-invariant aspect. The context-invariant problem is treated thanks to an adversarial learning mechanism.  Experiments are conducted on synthetic and real datasets.", "summary_of_the_review": "The addressed problem is interesting and the proposed model is elegant but I think that the experimental part is too weak to assess the quality of the proposed approach and that the novelty is not sufficient wrt previous works.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QDohAB8z4QU", "reviewer": "Reviewer_PwXK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n+ interesting and relatively unexplored issue in the literature\n\nCons\n\n- article needs deep proofreading\n\n- experimental part not clear enough. Mix in the datasets between experiments is confusing.\n\n====\n\nThe article contains a number of English mistakes and deserves a thorough proofreading.\n\nFigure 2 is the weak point of the article: whereas it is suppose to give a general overview of the system, it not clear for me.\n\ndot should be removed from (2) and bolded font should be used for vectors to make the reading more convenient.\n\ntypo: in (4) there is a confusion between Lc and Lr\nmore generally, the formalization of section 2.2 could be improved.\nThe loss function R is not given in (5) whereas it is supposed to be an important contribution of the paper.\n\nThe classification tasks of section 4.1 are not clear for me: I don't understand the introduction of the synthetic dataset and I don't see the interest of classifying a global signal wrt the introduced architecture.\nSome additional content in appendix (& even in the body of the article) are required.\n\nRegarding the forcasting, we don't know the horizon of the prediction and we have no information about the number of scales taken into account. This information is critical both for understanding and reproduction.\n\nIn section 4.2, what is the discretization step? I don't know how to interpret results without this key information.\nThe proposed approaches should also be compared with a simple baseline to provide a better demonstration of the interest of the method.\n\nAt least, the authors should build a synthetic dataset that enables us to understand the strengths and weaknesses of the different formulations.\n", "summary_of_the_paper": "This article deals with representation of multivariate signals in the case where we can distinguish endogenous & exogenous channels. This article mainly relies on (Franceschi et al.,2019), whose proposal consists in representing signals using Dilated-TCNN and to apply triplet loss on the sequences as in word2vec. Then the authors introduce 3 cases: normal behavior, anomalies where both endogenous & exogenous are impacted and partial anomalies.\nThe encoded representation of a contextualized signal should be close to its temporal neigbors.\nThen, the authors tackle prediction & classification tasks on M5 and electricity datasets. The experimental part is a little bit confusing, as the datasets used in the different experiments are not the same.\nGenerally speaking, results are very difficult to interpret.", "summary_of_the_review": "This work seems very promising but the interest of the invariant representation is not clearly demonstrated according to me. Some experiments and critical information are missing so that I have to reject this paper. I strongly encourage the author to write & submit an improved version of this article in another conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a representation learning time series paper.

The reviewers appreciated aspects of the paper, but all agreed that primarily the experiments are lacking and to a lesser degree the presentation is unclear and needs further proofreading.

So definitely this work has merits. It is also much appreciated that the authors throughout the discussion have been engaged in adding results and further clarification. This can be used for an updated version for the next conference.
Decision: Reject