S$^3$ADNet: Sequential Anomaly Detection with Pessimistic Contrastive Learning
Authors: Quexuan Zhang
Yukio Ohsawa
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
anomaly detection
unsupervised learning
sequential data

Abstract: Anomalies are commonly found in sequential data generated by real-world applications, such as cyberattacks in network traffic, human activity changes in wearable sensors. Thanks to the development of computing technology, many impressive results have been obtained from deep learning-based anomaly detection approaches in recent years. This paper proposes a simple neural network framework for detecting anomalies on sequential data, called $S$elf-$S$upervised $S$equential $A$nomaly $D$etection $N$etwork (S$^3$ADNet). S$^3$ADNet first extracts the representations from each data point by performing feature augmentation for contrastive learning; then captures the contextual information from the sequential data points for estimating anomaly probabilities by optimizing the context-adaptive objective. Here, we design a novel loss function based on a pessimistic policy, considering that only anomalies can affect the contextual relationships in sequences. Our proposed method outperformed other state-of-the-art approaches on the benchmark datasets by F1-score with a more straightforward architecture.

Under review as a conference paper at ICLR 2022
ADEET: SEQUENTIAL ANOMALY DETECTION WITH PESSIMISTIC CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Anomalies are commonly found in sequential data generated by real-world appli- cations, such as cyberattacks in network traffic, human activity changes in wear- able sensors. Thanks to the development of computing technology, many impres- sive results have been obtained from deep learning-based anomaly detection ap- proaches in recent years. This paper proposes a simple neural network framework for detecting anomalies on sequential data, called Self-Supervieed Sequential Anomaly Detection Network (S ADNet). ADDD first extracts the represen- tations from each data point by performing feature augmentation for contrastive learning; then captures the contextual information from the sequential data points for estimating anomaly probabilities by optimizing the context-adaptivee objective. Here, we design a novel loss function based on pessimistic policy, considering that only anomalies can affect the contextual relationships in sequences. Our pro- posed method outperformed other ttaee-oftue offthhe-rr art approaches on the benchmark datasets by Fi-score with a more straightforward architecture.
INTRODUCTION
Sequential anomaly detection (SAD), subtask of anomaly detection (AD), focuses on detecting outliers or change points for sequential data. Sequential data are produced at a sub-milisecond rate by multifarious real-world applications, usually being of high dimensionality and high noise. Besides the complex nature of data, the sequential data points can have shifting distributions and
relationships in different contexts, resulting in the concept drift problem (Zliobaité 2010: Gama et al. 2014). Because anomalies can imply certain occurrences that endanger public safety, life, and property, such as intrusions in network connections (Jyothsna et al., 2011, [Samrin & Vasumathi]
2017 Moustafa et al., 2019), activity changes and health irregularities in human sensing (Nweke [018| Serhani et al.] 2020), abnormal events in video surveillance systems (Popoola & Wang 2012; Mabrouk & Zagroubal 2018). and risky signals in space technology (Hundman et al. Tariq et al., 2019; Yairi et al., [2017; [Shin et al., 2020), SAD is a challenging but crucial task (Pang al. 2022 Ruff al. (021). Learning meaningful representations from the data and capturing the contextual relationships is the key to solving SAD tasks.
cet al.
2018
Generally, AD methods are in the unsupervised scheme because anomalies considered to be much less than normal instances and hard to model. The approaches that train the models exclu- sively on the normal samples to identify the anomalies are one-class classification methods, such as one-class support vector machine (OC-SVM) (Schölkopf' et al.) 1999). Since deep neural net- works are expert extracting representative features from high--imensione and high-noise data, more and more methods leverage deep learning to handle AD tasks. Especially, autoencoders (AEs)
(Hinton & Zemel 1994, Kingma & Welling 2013) and generative adversarial networks (GANs) (Goodfellow et al. 2020) are frequently used as they are suitable for unsupervised learning, such as deep autoencoding Gaussian mixture model (DAGMM) (Zong et al. 2018), and AnoGAN (Schlegl et al. 2017). However, these methods could be weak in capturing the contextual information from sequences. so that some other approaches introduce recurrent neural networks (RNNss) [Rumelhart let al. 1986) to learn the sequential relationships. For example, the multi-scale convolutional current encoder-decoder (MSCRED) (Zhang et al. 2019), OmniAnomaly (Su et al.) 2019). and the temporal hierarchical one-class network (THON) (Shen et al. 2020) apply RNNs for time-series

---

Under review as a conference paper at ICLR 2022
data modeling. Nevertheless, most of these methods still require picking out the normal data from the mess to train the one-class classifiers or generative models in practice.
This paper proposes self-supervieed sequential anomaly detection network (S) ADNet) to learn the representations of scquential data points and extract the contextual relationships for anomaly detec- tion. Inspired by the simple framework for contrastive learning of visual representations (SimCLR) (Chen et al. 2020), we make the network generate two similar embeddings for input data points with feature augmentation, and the feature extraction layers capture the essential identities of the data points by minimizing the contrastive loss. By optimizing the contextual contrastive loss based on pessimistic assumption system, the multi-conccptual context (MCC) layer estimates anomaly probabilities with the sequential relationships modeled by the learned representations for each data point. We can use arbitrary parameter- shared neural networks for the representative feature extrac- tion. Additionally, we employ the MCC layer rather than RNN structures for a more intuitive context learning. The proposed model does not require any label information or even selecting the normal data for training one-class classifiers, yet makes the data points compare with each other to recog- nize anomalies. In the experiment, we compared our model's s estimation results to the state-of-the-art approaches using widely used network traffic dataset and human activity sensing dataset. Our proposed AAADNet obtained competitive Fi-score results to those methods with a simpler network architecture.
2 RELATED WORK
In this section, we review existing research endeavors related our work. We first overview the approaches for AD and SAD tasks. Then, we outline recent contrastive learning methods for repre- sentation learning.
2.1 ANOMALY DETECTION
In a narrow sense, AD indicates outlier detection (OD), where the target is the data points with extraordinary properties from most of the rest. Other than outliers, people are more concerned with change point detection (CPD) on sequential data since the abrupt characteristic shifts can intimate valuable and crucial information for the temporal trends.
OD methods can be grouped into three categories based on the main principle of modeling: 1) Noundary-basede 2) dbase, and 3) reconstruction-basee approaches. Boundary- meth- ods use certain distance indexes in the model optimization or the anomaly score calculation. OC- SVM is a typical boundary-basee data points on a hyperplane (Schölkopf et al.] [999).. Isolation Forest (IF), another widely used ap- proach, constructs trees by randomly splitting branches throughout randomly selected features and considers that the average feature path length of an anomaly to the root is larger than that of a normal instance (Liu et al., 2008). The main idea of classic density- based methods is performing probability density estimation to maximize the likelihood for normal samples, such as using Gaussian mixture model (GMM) for parametric estimation kernel density estimation (KDE) for nonparametric esti- mation (Laxhammar et al., 2009; Kim & Scott 2012). The online algorithm employed a discounting strategy update the density for sequences (Yamanishi et al.) 2004).. With the development of deep learning. deep structured energy-based models (DSEBMs) were proposed to model the distribu- tion of normal data points by using deep architectures Zhai et al. 2016). Another deep method, DAGMM, applies autoencoder to learning data's 's representations and predict the likelihood based on GMM (Zong et al.] 2018). Reconstruction- based methods aim to well restore data from compressed feature space for normal data points but not anomalies. Principal component analysis (PCA) and its variants are traditional ways for data reconstruction (Hawkins) 1974, Schölkopf et al. 1997 Thar- rault et al. 2008), while variational AE is of deep style without handcrafted features for the sake (An & Cho 2015). Methods using GANs can be considered to be hybrid approaches, for they model the distribution by regenerating data points in the pero-sum-gant
method that learns a kernel-based boundary surrounding normal
scheme, such as AnoGAN (Schlegl ett al. 2017). adversarially learned anomaly detection (ADLD) (Zenati et al. 2018). Mahalanobis based adversarial network (MDAN) (Hou et al. 2020).
For CPD, there is a similar taxonomy to OD. Fast low-cost online semantic segmentation (FLOSS) calculates the contextual boundaries based on the shapes of sequential patterns to eject change points
2

---

Under review as a conference paper at ICLR 2022
(Gharghabi et al., 2019). The autoregressive model updates the parametric probability density fune- tion across sequential data points (Yamanishi & Takeuchi 2002). Relative unconstrained least- squares importance fitting (RuLSIF) identifies whether two contiguous subsequences are from dif- ferent distributions using the nonparametric divergence estimation (Liu et al. 2013). KL,CPD gives the distribution discrepaney by using learnable deep kernel function and reconstructing samples from two sequential segments with RNNs (Chang et 2019). Entropy and shape aware time-series segmentation (ESPRESSO) hybrid approach that combines the temporal density and distance to detect changes and obtained better performance than employing one principle (Deldari et al.) 2020). Another sulf--uuer contrastive learning method TS-CP² takes the subsequences which are not contiguous with the current subsequence as the negative samples for the training and then predicts whether the two subsequences are from the same distribution (Deldari et al. 2021). TS-CP2 employs temporal convolutional networks (TCNs) (Bai et al., (018)) to learn contextual representa- tions, while ours introduces the MCC layer to capture the contextual information and allows using arbitrary feature extraction networks for each data point.
2.2 CONTRASTIVE LEARNING
Recently, contrastive learning (CL) has grabbed much attention due to the benefit of avoiding the
high cost of labeling large-scale datasets for deep representation learning in unsupervised/sell- supervised manner (Jaiswal et al.]
2021). CL. has been used for various data formats, including
images (Chen et al., 2020 He et al.
2020), time series (van den Oord et al.] [018 Franceschi al.,
2019) videos (Sermanet et al. 20118
Qian et al. 2021). texts (van den Oord et al. 2018 Gao et al.
2021
and graphs YYuu al. 2020 Zhu al. 2020).
The basic strategy of CL is to train a model that makes the feature distances between similar in- stances closer but the ones between diverse instances far possible. For this sake, CL methods design pretext tasks to optimize the distances and use contrastive loss or triplet loss as the learn- ing objective (Chen et al., 2020). Recently, data augmentation has been commonly introduced into the pretext tasks to help the feature extraction. While image data are relatively easy to augment by using conventional computer vision techniques (Shorten & Khoshgoftaar 2019),, there are sev- eral word-level (Wei & Zou| 2019) and sentence-level (Kobayashi) (0118) approaches for textual data augmentation. For other data formats, some domain-agnostic methods were proposed, such as adding Gaussian noise (GN) to embeddings Vrees & Taylor 2019) or dropout noise (DN) in rep- resentation layers (Gao et al. 2021 Liang et al. 2021). GN and DN are not only easy to implement but also have low computational costs for CL tasks.
In SimCLR, given a cluster of augmentation A and an input æ, two augmentation operator A₂ ~ A and A A generate two similar instances T. = >=( and æ, >x(x)). Then, the feature extraction network f encodes I, and D3 into two representations h, = f(x,) and hB = f(x)). Next, the head network g projects h. and hs into two embeddings 2, = gha) and 2 ghb)). whose similarity is maximized by minimizing a contrastive loss, the NT-Xent loss:
exp (sim (za,20) /T)
ENTX (a,b) log 2NN
(1)
exp (sim (Za; 2n) /T) [n + a)
21 To D
sim uu v)
(2)
| ||||
where T' temperature, and N denotes the minibatch size. Since the value range of Equation is
[-1, 1]. it is usual to set ( < T < 1 for an efficient learning. Here, we can regard the similarity as
state energy E. According to the Boltzmann distribution, the probability of state p satisfies p o
exp (-E/kT), where the right hand side is called the Boltzmann factor, and k is a constant (Kittel 2004). we use similarity to represent a sequential relationship, we can model the probability of the relationships. Also, can adapt the temperature to control the sensitivity of the relationship's strength to the probubility. This finding motivated the objective function design in our method, which we will describe in Section 3.2.2
3 METHODOLOGY
Before presenting the details of our proposed method, we depict the overview of S ADNet's archi- tecture, shown in Figure]] In the following sections, we first introduce the assumptions behind
3

---

Under review as a conference paper at ICLR 2022
pessimistic contrastive learning (PCL). Then, illustrate the pretext tasks and objective functions in our method.
m
Input
Enceder with sugmentation
Re-Senf
Probabilities of having exceptional relationships
a())
Relationships
771
CARE
Multi conceicl context (MCC) layer
Copy & transpose
Contextual pooling
Multi-conceptual pooling
mm
Anomaly probabilities
Waoc
The architecture of ADNet. Here, (.) indicates the sigmoid function, symbolizes the tensor dot operation, C' signifies the number of concepts, and WMCC stands for the weight of the MCC layer. The two yellowish tensor blocks are different augmented sequential embeddings generated from the batch input, while the yellow matrix is one sequential embedding for instance of the calculation in the MCC layer. The exact numbers of cells imply the identical dimension: one of the minibatch size N 3. the sequence length L 4, and the embedding size 5. The lookahead size for collecting relationships
Figure 1:
3.1 ASSUMPTIONS
Given a sequence elements [z1, z.. ,,, the anomalies based on the following pessimistic policy.
21,11 zrl (1 S < < L), we model
Assumption (existence of anomalies) Anomalies can occur at any element in the sequence.
We assume that there are anomalies in the training data. Picking out normal samples from a large- scale high-dimensional dataset is expensive, not mention that sometimes lack prior knowledge of the data. Therefore, the target of our method is the data mixed with anomalies.
Assumption (bias of the number of anomalies) The number of anomalies much less than the normal elements.
The higher bias can help the model capture the statistical features to recognize the anomalies. If the numbers of abnormal and normal instances are close, our method may fail. However, if the anoma- lies are much more than the normal samples, we can treat the virtual normal ones as "anomalies", such as the experiment on the KDDCup99 dataset.
Assumption (independence between anomalies) An anomaly that occurs at element is inde- pendent of the other anomaly occurrences in the sequence.
Though anomalies could be relevant in roal-world anomaly detection tasks, this assumption can simplify the modeling. Let p: and p5 be the probabilities of anomalies occurring elements 2;
4

---

Under review as a conference paper at ICLR 2022
and 2j. respectively, then the additive probability P.j = p, + Pj P.Pj, Pij = 1 Pij However, determining whether an anomaly has occurred still depends on contextual information.
Assumption (relationships among sequential elements with anomalies) For two elements in the sequence, if and only if at least one anomaly occurs, they have an exceptional relationship.
We assume that only anomalies can affect the quality of a relationship between two elements in the sequence. Here, relationship models the degree of correlation between two elements over the sequence. Then, we need to quantify the relationship for the modeling in the following definition.
Definition (quantification of two elements having a relationship in sequence) Given two el- ements 2; and 2j in a sequence, rij R quantifies the relationship between 2; and 2j. If the two elements have exceptional relationship, then Tij < 0. Otherwise, Tig 2 0. Irag measures the strength of the relationship.
According to Assumption and Definition] 1 we obtain the following corollary.
Corollary (relation between sequential anomaly and relationship) Pij r.j 2 Pij
1 + Tij < 0.
3.2 PESSIMISTIC CONTRASTIVE LEARNING
We need well representations for sequential data points to apply the above assumptions to caleu- late the quantified relationships. As result, we construct two pretext tasks with diverse objective functions for feature extraction and anomaly detection.
3.2.1 FEATURE EXTRACTION BY SEQUENTIALLY CONTRASTING
The contrastive feature extraction task mostly follows SimCLR. The encoder network in Figure consists of a feature network for learning representations and head network for projecting em- beddings. Nonetheless, for non-image and non-textual data, and modify the NT-Xent loss function for the training. The first augmentation method is adding GN A (0, o²I) into the embeddings (DeVries & Taylor 2019). The noise is sampled from N (0, I) and then dlement-wise multiplied by o², which is fit by using a full-connected layer attached the output of the feature network. The second approach is employing DN by operating dropout with a small probability parameter in every layer of the feature network (Liang et al. (202T), where the bidirectional Kullback-Leibler divergence (biKLD) used to regularize the augmentation. We revise Equation into the following formulation:
perform two types of domain-agnostie feature augmentation methods
with the regularization to the data augmentation
f. log
cxp
sim
2; 2z
(a) (b)
(T)
Eas(a, b)
-
Σ 1=1
2N n=1
exp
sim
Zi
(a) (n)
/T) [n a)
+ +DLLL (softmax
(2((0) ||softmax (z.") (b)
)=
where is hyperparameter control the regularization, and DKL denotes the Kullback-Leible divergence (KLD). The regularization term in Equation Blurns out to be the biKLD as described in
(Liang et 2021 in the loss function for minibatches:
N
LRS
Σ 1=1
(2n - 1m) (2n, 2n 1)].
(4)
2N
For convenience, we call this function Re-Sent2 (regularized sequential normalized temperature- scaled cross entropy loss). In the begimning of the training, we set a certain number of epochs to solely use Equation 4 as the warm-up for better representations for the next task.
3.2.2 ANOMALY ESTIMATION BY CONTEXTUALLY CONTRASTING
For the sake of detecting anomalies in sequences, we leverage the contextual information given by the representative features of each sequential data point. Here, concept represents bias due to
5

---

Under review as a conference paper at ICLR 2022
specific sequential contexts, such as locality and periodicity. Since sequence data may have multiple concepts, we propose the MCC layer to capture the contextual information of different concepts from the embeddings of the representations learned from sequential data, shown in Figure I The
estimation of an anomaly probability pe is formulated by
p: = o
9s
(5)
Vm
9: = sgn (Gi,e) max ( (G..))
(6)
L
Gi.e = L - 2 Mies j & arg fargmin, max] aM...))
(7)
X
M tanh (Z WMCC) _T,
where sgn indicates the sign function, |.| means the element-wise absolute value function, and c is an index of concept. Given a sequential embedding Z E RLxmm the MCC layer obtains multi- conceptual correlations M E IR Ixxx' by performing tensor contractions on Z and its transposed copy with a learnable weight WMcC E RRmxCxmm and the tanh activation. Next, the contextual pooling (CXP) aggregates the correlations for each data point in different concepts into G E R L After that, the multi-conceptuall pooling (MCP) selects the concept value, to which the data point is most possibly attributed, from G for each data point. Here, we adopt the robust average pooling to CXP, which calculates the mean by ignoring the largest and smallest values (Equation 7 To MCP, we use the absolute maximum pooling to select the one with the largest absolute value (Equation 6 The pooling result g E R is scaled by 1//m and activated with the sigmoid function. Finally, the activation result P E [o, 1)- is used to predict the anomaly probabilities. The tanh activation and the scaling can make the output values of MCC not too large and the initial values approximate to zero, i.c., the anomaly probabilities are estimated to be around 0.5 at the beginning of training.
We have modeled the anomaly probability so far. Next, we formulate the relationships in sequences by using the assumptions mentioned in Section 3.1 Due to Tij E R according to Definition | we calculate relationship by
sim (zi,
Tig
r(.,))
(9)
where r : + R+ is an adaptation function. Since the sigmoid function a R + [0, 1] keeps the monotonicity, we can model the probability of having an exceptional relationship as
4ú = o = (Ta)) = Gij-
(10)
Based on Corollary I the objective is to minimize the difference between the distribution of anoma- lies' occurrence P and the distribution of data points' relationships Q. We could minimize the biKLD to achieve this objective. However, observing (PII)) H())) gives the cross entropy and H(-) calculates Shannon entropy, the second term leads the dis- tributions into entropy maximization in the optimization. By Assumption [2] the anomaly probability considered be much low. Therefore, we use the following functions instead of biKLD:
Q) - (P)) where
1
LAKL (p.g) g) = [Pakz
DÆKτ (QI
(11)
DAKL (PIQ)= = 3(p Mogp + pog p log a- -
-:pgg-ppg plog
(12)
where @ is a penalty restrain the entropy maximization. We also clamp the probabilities with small value to avoid log O. For the n-th sequence in a minibatch, we formulate the loss function as
min(i+a.L)
(cR(n) = L L ---1 Σ min
Σ #=++1
(n) (n) LOKL (pi) 495
(pi)),
(13)
where s denotes the lookahead size to handle the number of forwarding relationships to calculate. For easy reference, we term this loss function CARE (context-adaptive relative entropy loss). CARE for minibatch of augmented embedding pairs is given by
N
1
Lor
Σ
[eRn(n - 1) + (cR(2n)]
(14)
2N
1=1
6

---

Under review as a conference paper at ICLR 2022
Moreover, hope that the contextual information reflect the representations of data points, and hence we combine Equation and [4 into a global loss function for multi-task learning (Ruder 2017). The global objective is formulated by
Σ = ARSCRS + ACRLCR.
(15)
where ÀRS and ÀçR are the weights for the two loss functions, respectively. When starting to optimize Equation 15 we also reduce the learning rate of the encoder network for finetuning.
EXPERIMENTS
This section reports the comparative evaluation of our proposed ADNet. We applied a simple fully connected neural network (FCNN) and simple convolutional neural network (CNN) as the encoder models for two datasets, respectively. We show and explain the evaluation results, and give an ablation study in the following. S ADDett was implemented by using Python 3.8 and PyTorch 1.8.
4.1 DATASETS
We evaluated S ADet on the widely used network traffic dataset KDDCup and a human activ- ity sensing dataset HASC? Table summarizes the properties of the datasets.
Table 1: The properties of the datasets.
Dataset
#instances
#features
#anomalies
KDDCup99 HASC
494021
121
97278
39397
65
Details of the two datasets and the experimental setup for them are described Appendix A
4.2 RESULTS
We applied three 3 ADNet variants based the data augmentation approach used for the compar- ative evaluation: 1) with only GN, 2) with only DN, and 3) with both GN and DN. As $ ADNet can predict the anomaly probability directly, the prediction less than 0.5 was supposed be negative (i.e., Pi = 0), otherwise positive (i.e., 1). Additionally, we use Fl-score as the primary metric for the comparison as it is better for imbalanced datasets.
Table 2 shows the result on KDDCup99, while Table reports the one on HASC. On the KDD- Cup99 dataset, our method was outperformed the other state-of-the-ee the methods. Since the number of learnable parameters in S ADNet (when = 8) was only 1/15 of the ones in MDAN, and 1/6 of those in TS-CP2, our proposed method simpler and more effective.
The CPD task on the HASC dataset was challenging due to the small size and the few targets, but our proposed ADNet still achieved the highest score the 100- 100-margin setting and the second-highest score in the 60-margin setting. A smaller batch size was better for this task. However, the training was unstable, and the Fl-scores oscillated both two tasks. We will continue the discussion on the problems in the next section.
4.3 ABLATION STUDY
We report the best scores for each hyperparameter settings in the following. The default values of the hyperparameter are shown in Table 4 The models were trained five times in each setting.
hhtp:///add.ccs.uc..edd hupp:/hasc.jp
7

---

Under review as a conference paper at ICLR 2022
The results on the KDDCup99 dataset. The bold score denotes the highest one in the column. We report the best scores for S ADNet.
Table 2:
Model
Precision
Recall
Fi-score
OC-SVM (Schölkopf et al.
1999)
0.7457
0,8523
0.7954
IF (Liu et al.
2008)
0,9216
0,9373
0,9294
DSEBM-r Zhai et al.
2016
0.8521
0.6472
0.7328
DSEBM-e Zhai et al.
2016
0.8619
0.6446
0.7399
DAGMM Zong et al.
2018
0.9297
0,9442
0,9369
AnoGAN (Schlegl et al. 201 7)
0.8786
0.8297
0.8865
ALAD Zenati al.
0.9427
0,9577
0,9501
MDAN Hou et al.
2020)
0.9472
0.9623
0.9547
S ADNet (GN)
0.9337
0,9633
0,9482
S ADNet (DN)
0.9389
0,9829
0,9604
S ADNet (GN+DN)
0.9453
0.9779
0.9613
Table The results on the HASC dataset. The margin is the maximum allowed detection error (the number of data points from ground truth an estimated change point) that within this range, the estimation is still considered a true positive. In the columns, the bold and underlined scores denote the highest and second- sooohd- scores, respectively. We report the best Fc-score of each method.
Margin
60
100
200
Model
FI-score
FLOSS (Gharghabi et al. 2019
0.3088
0.3913
0.543
aHSIC
Yamada al. 2013)
0.2308
0.3134
0.4167
RuLSIF Liu ct al. 2013)
0.3433
0.4999
0.4999
ESPRESSO (Deldar al.
2020
0.2879
0.4233
0.6933
KL-CPD Chang et al.
2019
0.4785
0.4726
0.4669
TS-CP² (Deldari al.
2021p
0.40
0.4375
0.6316
$ ADet (GN)
0.4783
0.5389
0.6225
$
ADNet (DN)
0.4471
0.5316
0.6225
S ADNet (GN+DN)
0.4457
0.5304
0.6214
Table 4: Default hyperparameters for the ablation study. The meanings of the symbols explained in Appendix A
Dataset
L
N
C
T
k
3
AGR
#warm-ups
KDDCup99
256
8
0.5
0.05
0.25
0.1
0.1
5
5
HASC
8
0.5
0.05
0.25
3
Table 5: Best Fl-scores obtained with different numbers of concepts on the two datasets.
Dataset
KDDCup99
HASC (w=100)
C
4
8
12
4
8
12 0.5056
F1-score
0.7576
0.9486
0.9242
0.9431
0.3649
0.4762
0.4792
4.3.1 IMPACTS OF THE NUMBER OF CONCEPTS
The results shown in Table 5 are obtained by selecting c E {1,4, 8, 12] to train S ADNet with DN on KDDCup99 and HASC. We found that more than concept could help improve the performance. Nevertheless, because the weight M dense, a large increases the number parameters and may affect the convergence of training.
8

---

Under review as a conference paper at ICLR 2022
Table 6: (w=100).
Best Fl-scores obtained with different temperatures and adaptation functions on HASC
T
k
constant
log
root
exp
0.1
0.41
0,4162
0.4471
0,4158
0.25
0.4513
0.4815
0.4340
0,4815
0.05
0.5
0.4815
0.3776
0.3983
0.4052
0.4324
0.5202
0.3882
0,4571
0.1
0.3902
0.4198
0.4590
0,4605
0.25
0.4444
0.3636
0.4123
0.4416
0.1
0.5
0.4444
0.3907
0.4342
0,4063
0.4835
0.4105
0.3908
0.4365
4.3.2 IMPACTS OF ADAPTATION FUNCTIONS
Since the adaptation function for the KDDCup99 dataset was constant, we used the results HASC to analyze the effect of adaptation functions. this case, we selected E {0.05, 0.1} and & e {0.1, 0.25, 0.5, 1} with different adaptation functions. We found that smaller temperature was better for variant adaptation functions.
DISCUSSION
Notwithstanding the outperformance of S ADNet in the evaluation, there are still some limitations in our method. Firstly, similar to the other deep learning methods, our method requires many hy- perparameters to set up the model. Although several hyperparameter optimization algorithms are available to (Yu & Zhu 2020), they need validation sets and specific metrics for the trials. While our method is sespeervsedd/sll-supeeeeeee
could not be swift enough to adapt to an entirely new dataset having no validation set. A potentially viable solution is using the ensemble learning technique to combine the results from several models with low prediction losses.
Nevertheless, there was the second problem. We observed that the training of S ADNet could be un- stable like the GAN-based methods. This problem results in the difficulty of selecting well enough models by the prediction losses. One possible reason is that the value instability from KLD. We have attempted to use the Jensen-Shannon divergence instead of KLD,, but the performance was even worse. the future work to seek better measure. Another cause could be in the model- ing of relationships. Before the context learning stage, the encoder has been trained to make the identical instances have the representations as similar as possible but not concerning the contextual relationships. After that, however, the identical relationship could be broken by optimizing Equation 14 seems reasonable to attempt some tricks for training stabilization, such as stop- noop-gradient (Chen) & He 202T). One more reason could be that the degrees of freedom of the MCC weight were too high due to the density. may be helpful employ sparsification or factorization to the weight.
Thirdly, there are still potential options to improve the performance, We proposed method based on the density assumption so far. Since the hybrid method ESPRESSO acquired the top score in the 200-margin setting of the CPD task on HASC, it could be wise to take advantage of other principles in our model. Furthermore, though we have applied a simple FCNN and a simple CNN as the encoder networks in the experiment, of encoders in depth. Also, we found that they were still hard to extract representative features from complex multivariate time-series data, such as SMAP and MSL. (Hundman et al.) 2018). SMD (Su et al. 2019). It needs more studies on selecting encoder networks for those data, such as leveraging RNN learn representations of short sequences.
have not investigated the selection of the hyperparameters
6 CONCLUSION
This paper proposed a flexible and straightforward framework S ADNet, for detecting anomalies sequential data by using contrastive learning under the pessimistic assumptions. The future work is to stabilize the training and attempt to leverage other aspects improve the performance.

---

Under review as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1):1-18, 2015.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv e-prints, art. MrXiv:1803.0122 2018.
Wei-Cheng Chang, Chun-Liang Li, Yiming Yang, and Barnabás Póczos. Kernel change-point de- tection with auxiliary deep generative models. In International Conference on Learning Repre- sentations, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Shohreh Deldari, Daniel Smith, Amin Sadri, and Flora Salim. Espresso: entropy and shape aware time-series segmentation for processing heterogeneous sensor data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(3):1-24, 2020.
Shohreh Deldari, Daniel V Smith, Hao Xue, and Flora D Salim. Time series change point detection with sule--upe contrastive predictive coding. In Proceedings of the Web Conference 2021, pp. 1124-3135. 2021.
Terrance DeVries and Graham W. Taylor. Dataset augmentation in feature space. 5th International Conference on Learning Representations, ICLR 2017 Workshop Track Proceedings, pp. 1-12, 2019.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In Thirty-third Conference on Neural Information Process- ing Systems, volume 32 of Advances in Neural Information Processing Systems, pp. 4650-4661, 2019.
João Gama, Indrè Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSB: simple contrastive learning of sentence embeddings. arXiv e-prints, art. arXiv:2104.08821, 2021.
Shaghayegh Gharghabi, Chin-Chia Michael Ych, Yifei Ding, Wei Ding, Paul Hibbing, Samuel LaMunion, Andrew Kaplan, Scott E Crouter, and Eamonn Keogh. Domain agnostic online se- mantic segmentation for multi-dimensional time series. Data mining and knowledge discovery, 33(1):96-130, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Alex Graves. Generating sequences with recurrent neural networks. arXiv e-prints, art. arXiv:1308.000 2013.
Douglas M Hawkins. The detection of errors in multivariate data using principal components. Jour- nal of the American Statistical Association, (9(346):300-344 1974.
Kaiming He. Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729- 2020.
10

---

Under review as a conference paper at ICLR 2022
Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length, and helmholtz free energy. Advances in neural information processing systems, 6:3-10, 1994.
Yubo Hou, Zhenghua Chen, Min Wu, Chuan -Shng Foo, Xiaoli Li, and Raed M Shubair. Maha- lanobis distance based adversarial network for anomaly detection. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3192-3196. IEEE, 2020.
Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. In Pro- ceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 387-395, 2018.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2021.
VVRPV Jyothsna, Rama Prasad, and K Munivara Prasad. review of anomaly based intrusion detection systems. International Journal of Computer Applications, 28(7):26 2011.
Nobuo Kawaguchi, Nobuhiro Ogawa, Yohei Iwasaki, Katsuhiko Kaji, Tsutomu Terada, Kazuya Murao, Sozo Inoue, Yoshihiro Kawahara, Yasuyuki Sumi, and Nobuhiko Nishio. Hasc challenge: gathering large scale human activity corpus for the real-world activity understandings. In Pro- ceedings of the 2nd augmented human international conference, pp. 1-5, 2011.
JooSeuk Kim and Clayton D Scott. Robust kernel density estimation. The Journal of Machine Learming Research, 13(1):2529-256) 2012.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv:1312.6114, 2013.
arXiv preprint
Charles Kittel. Elementary statistical physics. Courier Corporation, 2004.
Sosuke Kobayashi. Contextual augmentation: data augmentation by words with paradigmatic rela- tions. In NAACL-HLT (2), pp. 452-457, 2018.
Rikard Laxhammar, Goran Falkman, and Egils Sviestins. Anomaly detection in sea traffic-a compar- ison of the gaussian mixture model and the kernel density estimator. In 2009 12th International Conference on Information Fusion, pp. 756-763. IEEE, 2009.
Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie- Yan Liu. R-Drop: regularized dropout for neural networks. arXiv e-prints, art. arXiv:2106.14444 2021.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. 2008 eighth ieee international conference on data mining, pp. 413-422. IEEE, 2008.
Song Liu, Makoto Yamada, Nigel Collier, and Masashi Sugiyama. Change-point detection in time- series data by relative density-ratio estimation. Neural Networks, 43:72-83, 2013.
Andrew L. Maas, Awni Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
Amira Ben Mabrouk and Ezzeddine Zagrouba. Abnormal behavior recognition for intelligent video surveillance systems: a review. Expert Systems with Applications, 11:480-491, 2018
Nour Moustafa, Jiankun Hu, and Jill Slay. A holistic review of network anomaly detection systems: a comprehensive survey. Journal of Network and Computer Applications, 128:33-55, 2019.
Henry Friday Nweke, Ying Wah Teh, Mohammed Ali Al-Garadi, and Ungma Rita Alo. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Systems with Applications, 105:233-261. 2018.

---

Under review as a conference paper at ICLR 2022
Henry Friday Nweke, Ying Wah Teh, Ghulam Mujtaba, and Mohammed Ali Al-Garadi. Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions. Information Fusion, 46:147-170, 2019.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: a review. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
Oluwatoyin P Popoola and Kejun Wang. Video-based abnormal human behavior recognition- review. IEEE Transactions on Systems, Man, and Cybernetics. Part C (Appliccations and Reviews), (116)865--788, 2012.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6964-6974, 2021.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv e-prints, art. 1rXiv:170.005098, 2017.
Lukas Ruff, Jacob Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021.
David E Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by back- propagating errors. nalure, 323(6688)5333-566 1986.
Rafath Samrin and Vasumathi. Review on anomaly based network intrusion detection system. In 2017 International Conference on Electrical, Electronics, Communication, Computer, and Opti- mization Techniques (ICEECCOT), pp. 141-147. IEEE, 2017.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. International conference on information processing medical imaging, pp. 146- 157. Springer, 2017.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component anal- ysis. In International conference on artificial neural networks, pp. 583-588. Springer, 1997.
Bernhard Schölkopf, Robert C Williamson, Alexander J Smola, John et Support vector method for novelty detection. In NIPS, volume 12, pp. 582-588. Citeseer, 1999.
John C Platt,
Mohamed Adel Serhani, Hadeel T Kassabi, Heba Ismail, and Alramzana Nujum Navaz. Ecg monitoring systems: review, architecture, processes, and key challenges. Sensors, 20(6):1796, 2020.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierar- chical one-class network. Advances in Neural Information Processing Systems, 33:13016-1300 2020.
Youjin Shin, Sangyup Lee, Shahroz Tariq, Myeong Shin Lee, Okchul Jung, Daewon Chung, and Si- mon S Woo. Itad: Integrative tensor-based anomaly detection system for reducing false positives of satellite systems. Proceedings of the 29th ACM International Conference on Information & Knowledge Munagement, pp. 2733-2740) 2020.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning.
Journal of Big Data, 6(1):1-43 2019.
12

---

Under review as a conference paper at ICLR 2022
Salvatore J Stolfo, Wei Fan, Wenke Lee, Andreas Prodromidis, and Philip K Chan. Cost-based modeling for fraud and intrusion detection: results from the jam project. In Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00 volume 2, pp. 130- 144. IEEE, 2000.
Ya Su, Youjian Zhao, Chenhao Niu. Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2828- 2837, 2019.
Shahroz Tariq, Sangyup Lee, Youjin Shin, Myeong Shin Lee, Okchul Jung, Daewon Chung, and Simon S Woo. Detecting anomalies in space using multivariate convolutional Istm with mixtures of probabilistic pea. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2123-2133, 2019.
Yvon Tharrault, Gilles Mourot, and José Ragot. Fault detection and isolation with robust principal component analysis. In 2008 16th Mediterranean Conference on Control and Automation, pp- 59-64. IEEE, 2008.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with contrastive predic- tive coding. arXiv e-prints, art. arXiv: 1.4..18877.37488, 2018.
Jason Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNEL pp. 2382-6388. 2019.
Takehisa Yairi, Naoya Takeishi, Tetsuo Oda, Yuta Nakajima, Naoki Nishimura, and Noboru Takata. A data-driven health monitoring method for satellite housekeeping data based on probabilistic clustering and dimensionality reduction. IEEE Transactions Aerospace and Electronic Sys- tems, (3(3)) 2017
Makoto Yamada, Akisato Kimura, Futoshi Naya, and Hiroshi Sawada. Change-poine detection with feature selection in high-dimensional time-series data. In Twenty-Third International Joint Conference on Artificial Intelligence, 2013.
Kenji Yamanishi and Jun-ichi Takeuchi. A unifying framework for detecting outliers and change points from stationay time series data. In Proceedings of the eighth ACM SIGKDD interna- tional conference Knowledge discovery and data mining, pp. 676-681, 2002.
Kenji Yamanishi, Jun-Ichi Takeuchi, Graham Williams, and Peter Milne. On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. Data Mining and Knowledge Discovery, 8(3):255-300, 2004.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812 5823, 2020.
Tong Yu and Hong Zhu. Hyper-parameter optimization: review of algorithms and applications. arXiv e-prints, art 1iti:2000000688)) 2020.
Houssam Zenati, Manon Romain, Chuan-Sheng Foo. Bruno Lecouat, and Vijay Chandrasekhar. Adversarially learned anomaly detection. In 2018 IEEE International conference on data mining (ICDM). pp- 727-736. IEFE, 2018
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International Conference on Machine Learning, pp. 1100-1109, 10000110.PMMRR. 2016.
Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni, Bo Zong, Haifeng Chen, and Nitesh V Chawla. A deep neural network for un- supervised anomaly detection and diagnosis in multivariate time series data. Proceedings of the AAAI Conference Artificial Intelligence, (3(01):1409-13 2019.
13

---

Under review as a conference paper at ICLR 2022
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR, 1.,,20006.0411,,.0 2020.
Indrè Zliobaité. Learning under concept drift: an overview. arXiv preprint arXiv:1010.4784 2010.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national conference on learning representations, 2018
A DETAILS OF EXPERIMENT SETUP
A.1 DETAILS OF DATASETS
KDDCup99 (Stolfo et al. 2000). In the experiment, we used the KDDCCup99 10% dataset. As mentioned in Section 3.1 the number of records labeled as "normal" in the dataset is highly less than that of intrusions, so that we regarded the instances labeled as attacks positive samples for the SAD task. For each variable, the discrete ones were one-hot encoded, while the contiguous ones were standardized by removing the median and scaling unit variance according to the first and third quartile values. We split the dataset into two random halves, and employed the first half of data for training and the rest for testing in each epoch.
HASC (Kawaguchi et al., 2011). We utilized the same HASC subset used by KL-CPD and TS-CP², having only the a-axis accelerometer logs. The target of the task is to detect the human activity pattern changes. As following the setting in (Deldari et al. 2021), we set three chunk sizes w E (60, 100, 200} to vary the subset to three chunked versions and used the whole subset for both the training and testing in the evaluation. The values of each axis were standardized by removing the mean and scaling.
For each dataset, the samples in a minibatch were generated in sliding window fashion, where the window size was the sequence length L, and the stride was 1.
A.2 MODEL SETUP
According to the properties of the two datasets, we applied diverse but simple encoder networks for them. The details of the architectures and the hyperparameter settings are described as follows.
KDDCup99. Due to the sparsity of each data point æ E R 121 we employed a 2-layer FCNN for the feature network and 1 I-layer FCNN for the head network. In the feature network, the output size of each layer was 32. The embedding size the head network 8. For the hyperparameter optimization, the sequence length L E {4, 8, 16), the batch size N = 256, and the warm-up was up to (5, 10, 20) epochs. Since the data points were shuffled, we set the adaptation function to (i.,j) le (constant).
HASC. Depending on the chunk size w, each data point was a dense matrix X E R This time, we applied a 2-layer ID CNN for the feature network but 1 l-layer FCNN for the head network. For each convolution layer, the output channel size was 32, the kernel size was selected from (3,5), 5). and each input's head and tail were padded with a zero. A maximum pooling and an average pooling were adapted to each output channel from the second CNN layer to extract the concatenated representative features with the size 64. Then, the representations were projected to embeddings with the size m = 16. For the hyperparameter selection, the sequence length L. E (4, 8), the batch size N E (4, 8). and the number of warm-up epochs were chosen from {20, 30). The adaptation function was selected from constant, T(i,j) k In(j - i 1) (log), T(i,)) kvj - (roor), and T(i,j) 1.1 / -ikk ik (exp), making the relationships with further data points weaker.
The following settings were common for both datasets. The activation was LeakyReLU Maas et al. 2013) with negative slope of 0.2 in both of the feature networks. For the data augmentation using DN, the dropout probability was set to 0.1. The objective functions were optimized by applying RMSProp (Graves) 2013) with the learning rate 0.1, the momentum 0.9, the smoothing parameter
14

---

Under review as a conference paper at ICLR 2022
contrast
augment
relate
6
5.0
4.5
0.2
2
4r
0
250
500
1000
250
750
1000
250
500
750
1000
step
step
step
(a) good
augment
relate
7.0
6.5
6.0
5
5.3
0.5
4-
5.0
250
500
750
1000
250
500
750
1000
250
500
750
1000
step
stop
step
(b) bad
Figure 2: Two examples of loss patterns on KDDCup99. Losses of contrast and augment were obtained from the first and second terms in Equation respectively, and ones relate were obtained from Equation 13.
Figure 2a led to good Fl-score, while Figure dbd not.
0.99, and the weight decay 1.0 x 10-1 Since KLD could approach infinity, the gradients were clipped in the range [-10, 10). After the warm-up epochs, the learning rate for the encoder net- work was reduced to 1.0 x 10-4 for the finetuning. For the multi-task learning, we set ARS = and ACR 53.5}. Every training trial was up to 100 epochs. The lookahead size was given by [rL], where r {0.25, 0.5, 0.75). Others were, the number of concepts C' E {1, 12), the regularization weight a E f0.1, 0.5, 1}, the entropy penalty 8 E {0.1, 0.5, 1}, and the temper- ature T (0.05, 0.1}. Furthermore, the coefficients k in adaptation functions were selected from 1).
B EXAMPLES OF TRAINING LOSS PATTERNS
Figure shows two examples of the training loss patterns on KDDCup99. The changes of contrast losses and augment losses were intense in warm-up epochs. If the hyperparameter setting is selected appropriately, there could be significant and sustainable drops in relate losses, as shown in Figure 2a The instability in Figure 2b may be attributed to KLD.
15


Reviews:
[{"review_id": "kLkHXs-YHiQ", "reviewer": "Reviewer_6q4k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "·\tStrength:\n\no\tThe paper is easy to follow.\n\no\tThe empirical result on two datasets outperforms most of the baselines.\n\n·\tWeakness:\n\no\tThe acknowledgements in the paper implicitly reveal the identity of authors, which may against the submission policy.\n\no\tThe novelty of this work is limited. In this work, only multi-conceptual pooling layer is the newly developed components.\n\no\tThe two feature augmentation methods seems arbitrary as there are no justifications for the reason of adopting the two methods. \nProviding more justifications may alleviate the concern.\n\no\tThe key assumption underlying the model is Assumption 3, which arbitrarily assume that there is no causal relations between anomalies, which may not be applicable to various real-world applications such as fault detection. More justification on this assumption may alleviate the concern. \n\no\tThe anomalies in the adopted dataset seems not following the Assumption 3. For example, the human activity patterns in HASC dataset are not completely independent to each other, which makes it surprising that the model works on this case. More explanation on this part may be helpful to clarify the model capability.\n\no\tThe selection of adaptation function seems arbitrary in the experiment, maybe more explanations will help readers to understand the function selection and development.\n\no\tSince the main contribution of this work is multi-conceptual pooling layer, I would expect more empirical analysis on this components. For example, how does the “C” affecting the outlier detection and what is the selection criteria? More explorations on this will certainly help readers to understand about how to properly use the model.\n", "summary_of_the_paper": "·\tThe author propose a contrastive learning framework for sequential outlier detection.\n\n·\tThe author extract sequential feature representation for individual data points by applying dropout and Gaussian noise to SimCLR.\n\n·\tThe author propose a contextual contrastive loss via developing a multi-conceptual pooling layer to minimize the distance between the probability distribution of anomalies and whole datasets.\n", "summary_of_the_review": "·\tTo summarize, this paper proposed a pessimistic contrastive learning framework for sequential outlier detection. The proposed method is technically sound. However, the novelty is limited on multi-conceptual pooling and no further empirical exploration on this component is provided. Also, many selections for the model seems arbitrary, which needs more justifications to support the motivation; and the dataset selection seems conflict to the model assumption. The most important part is the author includes the acknowledgements in the draft, which implicitly reveal the author identity and is against the submission policy. Therefore, I recommend rejection to this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bsfCwH9-tx8", "reviewer": "Reviewer_Gpfn", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has several significant issues with the primary being imprecise and unclear technical description of their method:\n\n* __Lack of Clarity:__ Many parts of this paper are technically imprecise, incorrect, or too implicit. This issue is severe enough that I do not think one could implement the method from the description in the paper. This alone is enough for a paper to receive a strong reject. I will give some examples here:\n  * __S3.1 first paragraph:__ the definition of $\\mathbf{z}_i$ is inconsistent with the rest of the paper, here it represents data points, later it represents an embedding. On p. 6 it is introduced as an embedding however it is used in situations where it doesn't make sense before that, specifically (6).\n  * __Assumption 3:__ This assumption needs to be made more concrete, in particular independence between anomalies conditioned on what information? Total independence doesn't really make sense since this work is specifically working on _time series_ anomaly detection means there is some sort of dependence between different time points. Maybe we are assuming some model where the anomalous points are chosen independently a-priori and then theres a model that generates the time series conditioned on the location of the anomalies?\n  * __p.4__ \"Unlike one-class learning, we assume that there are anomalies in the training data.\" This is incorrect. For the one class support vector machine, for example, there are slack variables that explicitly allow for points in the training dataset to fall in the anomalous region\n  * __Assumption 4:__ \"For two data points in the sequence, if and only if no anomaly occurs, they have a normal or no relationship.\" This sentence is inscrutable. No anomaly anywhere? No anomalies at the points? At one of the points? What is a \"relationship\"? Is \"normal\" and \"no relationship the same thing\"? Does \"normal\" have something to do with the normal distribution?  This kind of language is unacceptable.\n  *__Assumption 5:__ Similar issues as the last point.\n  *__Definiton 1:__ See above.\n  *__(3):__ What is the motivation for the KL term?  It seems very ad hoc.\n  *__\"Tensor product\"__ This is being used incorrectly. For two tensors $S\\in \\mathbb{R}^{d_1\\times d_2\\times \\cdots \\times d_m}$ and $T \\in \\mathbb{R}^{k_1\\times k_2\\times \\cdots \\times k_n}$ the tensor product $S\\otimes T$ will lie in $\\mathbb{R}^{d_1\\times d_2\\times \\cdots \\times d_m \\times k_1\\times k_2\\times \\cdots \\times k_n}$ and be rank one in $\\mathbb{R}^{d_1\\times d_2\\times \\cdots \\times d_m} \\times \\mathbb{R}^{k_1\\times k_2\\times \\cdots \\times k_n}$. This is not how you are utilizing it in (8) for example where you are performing tensor multiplication along the first mode of $Z$ and $W$, for example. This needs to be introduced and explained. Also note that with this sort of notation transposes are not necessary.\n  *__\"tensor weight\"__ $\\mathbf{W}_{MCC}$ (4 lines below (8)): How is this selected? Is it learned? This is the only occurrence of this in the whole paper from what I can tell and how it is selected, learned, or what it means is never explained.\n  * __\"concept\":__ This needs some introduction and some intuition.\n  * There are many more instances of this issue, but being exhaustive would take far too long\n *__Insufficient Experimental Results:__ This paper needs a very comprehensive experimental evaluation since the theoretical and conceptual contributions are small. More than 2 datasets and more competitors, including better shallow baselines especially for the 3 dimensional dataset. Would be interesting to see _data depth_ based methods as well.", "summary_of_the_paper": "In this paper the authors present a method for deep anomaly detection. They being by stating a few assumptions their method is based on. Then then develop two loss terms. The first loss term is based on contrastive learning with an additional term. Their second term is used for determining the anomaly score and is based on how well each \"data point\" fits to concept. \n\nThey validate their method on a couple of datasets and their method works pretty well.", "summary_of_the_review": "I recommend the paper be rejected primarily due to its exposition not being clear and technical aspects of the paper being unclear or wrong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Vp6EGp6Jw0r", "reviewer": "Reviewer_KQ2g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n+ All of the assumptions and the subsequent corollary are clearly presented.\n+ The design of the novel loss function based on a pessimistic policy has been discussed in detail.\n+ Many state-of-the-art methods are used as benchmark studies and are compared with the proposed method.\nThorough discussions are provided on observations related to unstable training and possible ways to mitigate it in future work.\n\nWeak points:\n- I’m not sure how well Assumption 3 applies to the real-world sequential data. If an anomaly occurs at a point in time, it may typically take some time to recover and result in anomalies present in adjacent data points. If that’s the case, it will make Assumption 3 that assumes independence between anomalies at stake.\n- For the real-world dataset used in the experiment, such as KDDCup99, is the occurrence of an anomaly still independent of the other anomaly occurrences in the sequence? The authors have explicitly stated that the proposed method requires sequential information, so the dataset was not split up randomly. \n- It is mentioned in Section 4.3 that “As S$^3$ADNet can predict the anomaly probability directly, the prediction less than 0:5 was supposed to be negative, otherwise positive.” This seems to be an extension of Corollary 1 and needs more explanations or proof.\n- The last sentence of Section 1 claims that “Our proposed S$^3$ADNet obtained competitive F1-score results to those methods with a simpler network architecture.” However, no comparative results are provided that justify the proposed method is of simpler network architecture or has a fewer number of parameters than the state-of-the-art methods listed in Tables 2 and 3. To justify such a claim, the authors are advised to present a figure similar to Fig. 1 of the original SimCLR paper.\n- More details should be provided on the number of hyperparameters to set up the model, an ablation study is suggested to show the robustness of the proposed method using suboptimal hyperparameters.\n", "summary_of_the_paper": "Inspired by SimCLR, this paper proposes a simple neural network framework for detecting anomalies on sequential data. By first performing feature augmentation and then estimating anomaly probabilities from the sequential data points by optimizing the context-adaptive objective, it is shown that the proposed method - S$^3$ADNet, outperforms many state-of-the-art approaches on two benchmark datasets in terms of the F1-score. ", "summary_of_the_review": "It is indeed interesting and of practical value to distinguish any anomaly condition using sequential data with simpler network architecture. However, there are some places where this paper needs to clarify and provide more details to make it technically sound.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PEogiUrvIx7", "reviewer": "Reviewer_2onX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\nAs the title indicates \"Sequential anomaly detection\", the main idea strongly involves the sequential properties of data.\nThe loss design of minimizing KL-divergence between relationship probability and anomaly probability is interesting and intuitive.\nThe writing is easy to follow and clear.\nAssumptions are well addressed.\nThe discussion section is very helpful to readers addressing existing limitations and suggesting future directions.\nThe proposed method shows strong performance against competing methods.\n\n\n[weaknesses + feedback]\n- Although I like the idea of minimizing the distribution gap between anomaly probability and sequential relations, the model is complex with many hyperparameters (the authors already mentioned in the discussion section). Also, the choice of adaptation function, augmentation methods are adding more options for the model. In short, the design space is too broad, and hard to search for the optimal model. It would be great if the authors could suggest a way to narrow down this design space without using labels.  For example, by analyzing loss patterns of individual loss terms. Or the use of a small validation set can be an option.\n\n- Because of broad design space (and hyperparameters), the authors tested variants of the model and reported the mean performance.\nI suggest providing ablation studies of these hyperparameters to figure out which one is sensitive and which one is not. Without an ablation study, it is hard to measure which part of the model design plays a critical role in the performance. And I suggest providing a standard deviation of these models' performances to show whether the performance gap between variants is high or low.\n\n[feedback]\nAs suggested in the discussion section, I agree that the ensemble approach can be a good solution.\nThe paper would be stronger if the authors could reveal sensitive hyperparameters to reduce the critical hyperparameters for tuning. \nThen build up an ensemble based on the models with varying sensitive hyperparameters can be one good approach to solve design complexity.", "summary_of_the_paper": "The paper tackles the anomaly detection of time series data. The main idea is to exploit both the sequential relation between data and the normality of the data itself. The model is learned with two stages. First, the model is learned with the sequential contrastive loss to learn data representations. Then the model is learned by minimizing the distribution between normality and exceptional relationship probability.\nThe learning process is a self-supervised way without any supervision. The model is validated with KDDCup99 and HASC datasets.\n", "summary_of_the_review": "The idea is interesting and I like it.\nI highly acknowledge the discussion section.\nHowever, the model design choices are too broad (hyperparameters, augmentation, and adaptation functions) and necessary ablation studies are missing.\nIf analysis on important design factors can be provided, the paper would be more thorough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HsJ6TOY671l", "reviewer": "Reviewer_kP8v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n1) The paper is mostly well written.\n2) The idea is interesting.\n\n### Weakness\n1) The empirical analysis lacks depth.\n2) Many recent SOTA methods are missing from the analysis. \n3) Many recent anomaly detection dataset such as SMAP [a], MSL [a], SWAT, and SMD [b] are missing from the evaluation.\n4) The literature review include most of the recent methods but some recent papers on anomaly detection are missing. For example OmniAnomaly [b], THOC [c], and other [d],[e],[f] etc.\n5) Missing a proper ablation study of the proposed method.\n\n### References\n[a] Hundman, Kyle, et al. \"Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding.\" Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.\n\n[b] Su, Ya, et al. \"Robust anomaly detection for multivariate time series through stochastic recurrent neural network.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.\n\n[c] Shen, Lifeng, Zhuocong Li, and James Kwok. \"Timeseries anomaly detection using temporal hierarchical one-class network.\" Advances in Neural Information Processing Systems 33 (2020): 13016-13026.\n\n[d] Tariq, Shahroz, et al. \"Detecting anomalies in space using multivariate convolutional LSTM with mixtures of probabilistic PCA.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.\n\n[e] Yairi, Takehisa, et al. \"A data-driven health monitoring method for satellite housekeeping data based on probabilistic clustering and dimensionality reduction.\" IEEE Transactions on Aerospace and Electronic Systems 53.3 (2017): 1384-1401.\n\n[f] Shin, Youjin, et al. \"ITAD: Integrative Tensor-based Anomaly Detection System for Reducing False Positives of Satellite Systems.\" Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020.", "summary_of_the_paper": "The author proposes a self-supervised sequential anomaly detection network that optimizes a context-adaptive objective using feature argumentation and contextual information. The proposed method outperforms baselines.", "summary_of_the_review": "The idea is interesting but the paper lacks depth in empirical analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work proposes a simple neural network framework for detecting anomalies on sequential data. The manuscript is quite rough. The paper needs significant editing. The authors should take the reviewers' recommendations to heart and make deeper changes to the paper.
Decision: Reject