Finding Anomalies in Network System Logs
with Latent Variables
Kazuki Otomo

Satoru Kobayashi

The University of Tokyo
otomo@hongo.wide.ad.jp

NII
sat@hongo.wide.ad.jp

Kensuke Fukuda

Hiroshi Esaki

NII/Sokendai
kensuke@nii.ac.jp

The University of Tokyo
hiroshi@wide.ad.jp

ACM Reference Format:
Kazuki Otomo, Satoru Kobayashi, Kensuke Fukuda, and Hiroshi Esaki. 2018.
Finding Anomalies in Network System Logs with Latent Variables. In Proceedings of BIG-DAMA ’18: Big Data Analytics and Machine Learning for
Data Communication Networks (BIG-DAMA ’18). ACM, New York, NY, USA,
Article 4, 7 pages. https://doi.org/10.475/123_4

1

INTRODUCTION

System logs are one of the most useful sources to understand thestate of a network. In an operational network, syslog is widely used
for collecting network logs and allows one to gather logs from all
devices at one server. However, it is not easy for network operators
to investigate the details of network problems with the logs becauseof their diverse and massive nature. Many log analysis methods
for finding anomalies and their root causes have been proposed toovercome this problem [1, 9, 11, 12, 14, 18]. In many cases, one firstclassifies the logs by their message type (i.e., log template) thentreats them as statistical time series to be later processed throughstatistical analysis. System logs have an intrinsic nature that makes
automatic analysis very difficult; their time series show a variety
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
BIG-DAMA ’18, August 20, 2018, Budapest, Hungary© 2018 Copyright held by the owner/author(s).
ACM ISBN 123-4567-24-567/08/06.
https://doi.org/10.475/123_4

Sampling𝑧𝑞(𝑧|𝑥)
Latent var.
distribution⋯

Data: 𝑥/.

CVAE(Decoder)

Data: 𝑥/-

CVAE(Encoder)

Data: 𝑥/"𝑥. : “ssh login at dev1”, Apr 1

𝑥- : “ssh login at dev1”, Apr 2

𝑥" : “BGP stat at dev3”, Dec 31

One day long data(1,440 min)⋯

Latentvariables#data points(54,720)

Detection
Event ID(0-120)⋯

𝑥. : “ssh login at dev1”, Apr 1• Networks → Network management;

Minimize reconstruction error + KL-Divergence: 𝐾𝐿𝐷 (𝑞 𝑧 𝑥 ||𝑝(𝑧))Event ID(0-120)

𝑥- : “ssh login at dev1”, Apr 2

CCS CONCEPTS# of logs/minutetime series

𝑥0 : “BGP stat at dev3”, Dec 31

System logs are useful to understand the status of and detect faults
in large scale networks. However, due to their diversity and volumeof these logs, log analysis requires much time and effort. In thispaper, we propose a log event anomaly detection method for largescale networks without pre-processing and feature extraction. The
key idea is to embed a large amount of diverse data into hidden
states by using latent variables. We evaluate our method with 15
months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg’s univariate
burst detection and a traditional multivariate analysis (i.e., PCA),
we demonstrate that our proposed method detects anomalies andease troubleshooting of network system faults.

Training

One day long data(1,440 min)

ABSTRACT

Clustering/Anomaly Detection
for each log type

CVAE(Encoder)

𝐾𝐿𝐷 (𝑞 𝑧 𝑥 ||𝑝(𝑧))𝑞(𝑧|𝑥)
Latent Var.
Distribution#data points(54,720)

Figure 1: Analysis overview

of characteristics such as periodicity, burstiness, randomness, andsparseness, compared with other time series data. Most studies havebeen devoted to developing effective pre-processing methods (e.g.,filtering, smoothing, denoising) and appropriate discriminative log
time series features for analyzing their own data. However, we
require enough domain knowledge about the underlying networks
for optimizing the analysis methods.
We define an anomaly as time series behavior deviating fromthe normal or usual state in log time series (details are given in§3.3). To detect these anomalies, we propose a robust statisticalmethod that handles complicated system log time series, without
any specific (case-by-case) preprocessing (e.g., filtering, smoothing,auto-regression) for log anomaly detection. The key idea of ourapproach is to embed a large amount of diverse data into hidden
states by using latent variables, also known as topic modeling or
Latent Dirichlet Allocation (LDA) often used in natural language
processing. A topic model discovers topics as latent variables froma collection of documents. We borrow the idea of this approach:by assuming hidden states in each log time series, we represent
log time series features by latent variables. As a result, we cancharacterize each time series without any labels or preprocessing.

BIG-DAMA ’18, August 20, 2018, Budapest, Hungary
Our proposed method consists of two phases (see also Figure 1):(1) The training phase embeds log time series characteristics into latent variables by using Conditional Variational Autoencoder (CVAE).
To mitigate the difficulty in handling various time series from devices together, we provide CVAE with type of log messages as aconditional label. (2) The detection phase highlights anomalies deviating from the distribution of the latent variables with clustering
algorithms.
Using syslog data collected in a nation-wide academic network
in Japan (SINET4), we confirm through evaluation that CVAE has
higher discriminative power for analyzing complex time series than
Kleinberg’s univariate burst detection [8] and a traditional multivariate analysis (Principal Component Analysis; PCA). With casestudies, we demonstrate that our method is useful in troubleshooting network system faults.
The contribution of this paper is twofold. We first propose ourlatent variable-based method for highly diverse log time series
data without any data-specific preprocessing. Next, we discuss theeffectiveness of the method when we applied it to real network
syslog data.

2

RELATED WORK

Many studies have been conducted for finding anomalies and theirroot causes [1, 12, 14] in log data. Zhong et al. [18] proposed ananomaly detection method for both device and network errors withfine log time series feature creation. Kimura et al. [5] introduced an
online failure prediction method based on log time series features.
Lu et al. [11] focused on the task duration time and proposed root
cause analysis methods with distributed computing system logs. As
these methods are based on data specific feature creation, they perform well in each considered environment. However, to apply thesemethods to other network systems, we have to re-define features
to optimize these methods. This requires deep domain knowledgeof the underlying network systems to make efficient use of thesemethods. In addition, these methods miss unknown or new anomalies, which are not captured by the pre-defined features. Instead,
we focus on an approach that does not require pre-definition of
anomalies but learns the normal state of the system from log data.
There are methods of giving new insights for operators withknowledge mining from log data [4, 17]. Kobayashi et al. [9] proposed a time series causal inference method in network logs. Hacker
et al. [3] introduced a log classification method based on the severity of network operation. These methods do not require predefinedfeatures because the features characterizing the data can be obtained through learning. However, in contrast with our method,they do not aim at anomaly detection but knowledge mining.
We focus on anomaly detection without specific feature creation
for more general analysis of logs. PCA is a standard algorithm oflearning or obtaining time series features from data. Lakhina etal. [10] proposed a general traffic analysis method based on featurelearning with PCA. They successfully detected traffic anomalies
without specific definitions of those anomalies. However, as log
data are highly sparse and discrete, which differs with traffic data,
it is not enough to learn features with such a naive approach (i.e.,
PCA). Thus, we use CVAE, which is based on a higher assumptionthat observed data are subject to latent variables. We choose a PCA

K. Otomo et al.
based method as a baseline algorithm and aim to outperform it with
our proposed method.

3 METHODOLOGY
3.1 Overview
The key idea of our proposed method is to model log time series
characteristics based on the latent variables. Latent variable analysis
is conducted to attempts to explain observable data by unobservable (i.e., latent) variables. In our context, as the observable data
are log time series, we intend to represent the characteristics of observable log time series, such as periodicity, frequency, burstiness,by latent variables. By applying a clustering algorithm or anomaly detection method to the latent variables, we can distinguishanomalous behavior of log time series from the normal behavior.
Formally, as it is difficult to compute latent variables directly, we
have to estimate them by assuming certain statistical models. In
this study, we rely on Conditional Variational Autoencoder (CVAE)
for estimating latent variables.
An overview of the proposed method is illustrated in Figure 1.
First, we construct the time series of the number of log appearances
per device per template for one day. We call it data point. Next,
with CVAE, we reduce the dimensions of the original vectors byusing CVAE. We call such a reduced vector a latent variable. We
train CVAE model so that the latent variable effectively represents
the potential behavior of log time series. Then, each data point isembedded into a latent variable. After that, we apply a clustering
algorithm to the latent variables and finally detect outliers in log
time series as anomalies.

3.2

Training phase

We briefly explain CVAE, a variation of the Variational Autoencoder(VAE). The VAE [7] is a stochastic variational inference method
based on the variational Bayesian approach. Since this is an unsupervised method, annotations for logs are not necessary.
Let us consider one-day long log time series x generated by a random process based on invisible continuous random variable z. Thisz is also subject to a prior distribution pθ (z). Thus, x is subject to aconditional distribution pθ (x|z). The goal is to obtain latent variables z from input x. In order to get the latent variables, we estimate
the distribution pθ (z|x) with Bayes’ theorem and approximate distribution qϕ (z|x). Now, the objective is to maximize the marginal
likelihood (MLH) log pθ (x) = D K L (qϕ (z|x)∥pθ (z|x)) + L(θ, ϕ, x).
This equation is rearranged as follows (details are given in [6, 7]).
log pθ (x)≥L(θ, ϕ, x),L(θ, ϕ, x)=−D K L (qϕ (z|x)∥pθ (z))+Eqϕ (z|x) [log pθ (x|z)].

(1)
(2)

We train the model that generates qϕ (z|x) from input x by optimizing the lower bound L(θ, ϕ, x). The training process is asfollows. First, we consider the prior distribution as a Gaussianpθ (z) = N (z; 0, I) and let qϕ (z|x) be a multivariate Gaussian. Next,
we estimate the parameters (ϕ) of the posterior distribution (qϕ (z|x))with a fully connected neural network (encoder). Then, we acquireFinding Anomalies in Network System Logs
with Latent Variables
Kullback–Leibler divergence

BIG-DAMA ’18, August 20, 2018, Budapest, Hungary

sshd[21151]: Invalid user admin from 192.168.0.10
sshd[22569]: Invalid user virus from 192.168.0.15
sshd[ * ]: Invalid user * from *the distribution of the latent variable qϕ (z|x), and through the sampling process from it, we obtain the latent variable z. Finally the
decoder neural network reconstructs x̂ from the latent variable z,
and we feedback the loss values in Eq. (2). Reviewing Eq. (2), we
can consider the first term (Kullback-Leibler divergence, KLD) as
the distance between estimated distributions qϕ (z|x) and pθ (z), andthe second term (Marginal Likelihood, MLH) as the reconstruction
error between x and x̂. Through these processes, we can compute
Eq. (2) and proceed with the training of the encoder and decoder
neural networks.
The CVAE [6] is a variation of the VAE. CVAE uses label information when generating the latent variable z and reconstructingx̂. We define the conditional label as the event that is an index of aunique combination of devices and log templates.

3.3

Detection phase

After applying CVAE to log time series data, we obtain the latentvariables for each data. Now, as the latent variables well-describe
time series trends, the goal of this phase is to build upon the timeseries description by latent variables to find the deviation from
“normal” states.
Some latent analysis methods define the anomaly level basedon reconstruction errors [10, 16]. In our case, we can use MLH for
reconstruction errors. However, there are two problems with using
MLH. (1) The reconstruction process is stochastic and MLH values
follow a Gaussian distribution. (2) MLH is biased by the intensityof the original data.
To solve these problems, we rely on the distance between latentvariable distribution (qϕ (z|x)) and the assumed prior distribution(pθ (z)) by computing D K L (qϕ (z|x)∥pθ (z)) in Eq. (2). After training,
the learned model generates the distribution of the latent variable(qϕ (z|x)) from input data x as the output in a hidden layer (shownat the bottom of Figure 1). We use this distribution qθ (z|x) for detection, not the sampled value z. Then, we deterministically compute
the KLD (D K L (qϕ (z|x)∥pθ (z))). The KLD represents the trend in the
input time series x. Thus, input data with similar KLD value belong
to the same trend. Therefore, we can obtain reference behavior byclustering KLD values. Note that the KLD value is not an anomaly
level but a state of input time series. Thus, we consider the data
whose KLD value deviates from others as an anomaly regardless ofthe magnitude of numeric value of the KLD.
Figure 3 shows an example of the KLD and MLH values of thedataset. The horizontal axis shows data points in descending orderof the number of log appearances, and the vertical axis shows
the KLD (top) and MLH (bottom) values. In particular, the graymarks indicate days without any log appearances (i.e., 1,440-longzero vector), and the blue ones show more than equal to one log
appearances on that day. Even if we input no log appearances data

Marginal Likelihood

Figure 2: An example of log templates

10−1

KLD
KLD(no logs)

10−2
10−3
10−4
10−5
10−6

0

10000
20000
30000
40000
50000
Data points (descending order of #log appearance)
MLH
MLH(no logs)

101
10

0

10−1
10−2
10−3

0

10000
20000
30000
40000
50000
Data points (descending order of #log appearance)

Figure 3: Kullback Leibler divergence and Marginal Likelihood (whole dataset)(i.e., gray marks in the Figure) to CVAE, KLD values are different foreach event thanks to the weight of the conditional label (i.e., event
ID). It enables us to make clusters depending on the behavior of theevent. As shown in the figure, the MLH values are more spread andbiased by the number of log appearances. In addition, they increase
along with the number of log appearances. On the other hand, theKLD values are not biased by the number of log appearances andare more stable than MLH values. Therefore, we use the KLD as anindicator of time series trend.
Once latent variable distribution is obtained and the KLD is computed one can choose any clustering method for anomaly detection. In this study, we used a well-known density-based clustering
algorithm, Density-Based Algorithm for Discovering Clusters (DBSCAN) [2]. By applying DBSCAN to the KLD time series for each
event, we detect clusters of KLD (i.e., normal state of that type oflog) and anomalies that do not belong to any clusters.

4

DATASET AND COMPARISON
ALGORITHMS
4.1 Dataset
We use a set of network logs collected at SINET [15], a Japanese
research and education network. This network connects over 800
academic organizations in Japan and consists of eight core routers,
50 edge routers, and 100 layer-2 switches. We use 456 days datafrom 2012/1/1 to 2013/3/31, consisting of ≈ 35M log messages from
130 devices (see also Table 1). This dataset is composed of wellknown format syslog data gathered from commodity L2 switchesand L3 routers. Examples are shown in Figure 2 and the log categoryof the dataset are shown in Figure 4 (manually classified).
As log messages are string data, statistical techniques cannot beapplied directly. Thus, we generate log templates from raw log messages with the supervised learning approach proposed by Kobayashi
et al. [9]. Log templates are log messages without variables (e.g.,
IP address), as shown in Figure 2. We then classify logs into log
templates and extract time series data from their time stamps forBIG-DAMA ’18, August 20, 2018, Budapest, Hungary
Table 1: SINET4 Dataset

All
Selected

logs
34,722,578
943,257

devices130
24

templates
1,789
33

events9,226
120

Table 2: Comparison of methods
nameburst detection
PCA
CVAE

multivariate
no
yes
yes

determin.
yes
yes
no

linear trans.
yesnoeach template per device. We thus generate 1,789 unique log templates from the whole dataset. We define an event as 456 of 1-day
log time series, which have the same log template in one device.
As a preliminary analysis, we manually selected 120 events andapplied CVAE to them (shown in Table 1) to make sure that thedataset has several anomalies and the diversity of the time series.
We confirm that these selected events include a large variety oftime series trends such as periodicity, burstiness, and sparseness.
We construct log time series of the number of log appearancesfor each minute per event per day (i.e., data point). The processing
flow is as follows: After we classified raw log data into the typeof logs per device, one event (for example, “ssh login at device 1")
has 456 days long log entries. We first count the number of log
appearances for each minute. Then, we split them for each days.
We now have 456 data points of event “ssh login at device 1". In
other words, each data point consists of one time series; It has
1,440 minute-length as X-axis and the number of log appearances
as Y-axis (shown as x 1 and x 2 time series graphs in Figure 1). We
also use conditional labels built on event labels concatenating log
time series. To input event labels into CVAE, we first assign unique
IDs to all the events and then encode them to one-hot vectors. If
the “ssh login at device 1" event has ID i, the conditional label is a
120 long vector and values are zero except index i. The value ofindex i is one. Then, the conditional label has 120 dimensions. After
repeating the same process to all the 120 events, we finally obtain
120 × 456 = 54, 720 data points and conditional labels. In training
and detection phase, each data point is an input for CVAE.

4.2

Baseline algorithms

To understand the effectiveness of CVAE for log anomaly detection,
we compare it to two traditional baseline algorithms: Kleinberg’s
burst detection [8] and PCA. The three algorithms are summarizedin Table 2.
4.2.1 Burst detection. Otomo et al. [13] conducted log analysis
focusing on the burstiness and causality of log time series. They
first removed trivial logs (e.g., periodic and very frequent logs)
to prevent detecting trivial bursts then detected log bursts with
Kleinberg’s burst detection [8]. Applying the same algorithm to ourdataset, we confirm 448 log bursts out of all 54,720 data points. In
the following comparison, we use these bursts as a part of referenceof log anomalies.

K. Otomo et al.
4.2.2 PCA. To detect traffic anomalies, Principal Component
Analysis (PCA) is a well-studied algorithm [10, 16]. It translates
a set of input traffic time series data into main and residual subspaces with a linear transformation. This algorithm is similar to
CVAE mapping raw data to the latent space, though CVAE is a
non-linear transformation. We implement a PCA-based anomaly
detector. Applying PCA to a set of log time series, we obtain principal components of the input dataset and separate them into main
and residual principal components with a threshold based on theirvariance coverage. We then reconstruct each time series with themain components and compute the reconstruction errors with theoriginal data. Finally, we apply a clustering method (DBSCAN) tothese reconstruction errors and obtain anomalies.

5

EVALUATIONIn this section, we discuss the evaluation of our method using
456 day-long system log data obtained from SINET4. We compare
the performance of our method with PCA and burst detection
technique. Finally, we present two case studies that demonstrate
the effectiveness of our method for network troubleshooting.

5.1

Parameter tuning

We first briefly describe the parameter tunings of CVAE. Encoderand decoder networks in CVAE each have four hidden layers. The
input layer has 1,560 dimensions (1440 dimension of time series and
120 dimension of label data). The encoding neural network has four
layers, and each layer has 512, 256, 128 and 64 units in order fromthe input layer. We set the latent variables dimensions to 10. The
decoding neural network has the opposite structure of the encoder,but the output size is 1,440 so that the output is a reconstructed timeseries. To avoid over fitting, we dropout 30% of units for each layer
during training. During training, we empirically set the number of
epochs to 50 based on test trial results. We confirm that loss values
converge after training. Next, we compute KLD for each time series
using the learned encoder network and apply DBSCAN to KLD foreach event. We tune DBSCAN parameters for each event so thata size of cluster is longer than a week. As the training process isstochastic, the results are not the same even if the loss values areconverged. To mitigate this uncertainty, we train the model ten
times and adopt anomalies detected in majority votes.
We use a commodity computer (Xeon CPU and GTX-1080 GPU)for processing. One training takes 489.5s on average of 10 trials.
The detection phase requires 45.1s on average to calculate KLD and
conduct DBSCAN.

5.2

Result overview

The results are summarized in Table 3. CVAE and PCA detected asimilar number of anomalies; 1,759 anomalies with CVAE and 1,825
with PCA. The overlapping number of anomaly was 1,335. Notethat a detected anomaly means time series behavior deviating fromthe normal state as defined in §1.
We consider such anomalies as having a relationship to actual
network system anomalies. To evaluate the efficiency of detected
anomalies for network management, we manually inspected all
the detected anomalies and annotated them as true or false basedon the SINET4 trouble ticket data and our domain knowledge forFinding Anomalies in Network System Logs
with Latent Variables

BIG-DAMA ’18, August 20, 2018, Budapest, Hungary

CVAE
PCA
Burst

350

Anomaly count

300
250
200
150
100
50

er
)
k(
l2
or
)
k(
m
vp tu)
m
n(
on
l2
vc
ito
)
r(
sn
m
m
p)
ne gm
t(
tw
ss
or
h)
k(
st
in
or
te
m
rf
ac
)
e(
ne
ag
tw
g)
or
k(
la
cp
vp
)
n(
m
pl
s)
tw

th

or

ne

(o

tw

em

ne

)

p)

os

ar

(c

k(
or

st

em

tw

st

ne

sy

rt

)

cp
)

po

k(
t

sy

ne

tw

or

e(

tp
)

ac

(n

in

te

rf

p(
b
eg

se

rv

ic
e

gp

)

0

Log category

Figure 4: Number of anomalies per log category
Table 3: Summary of results

CVAE
PCA

Data
54,72054,720Anomalies
1,759
1,825

Precision
84.6%
69.5%

#Bursts
240 (53.6%)
323 (72.1%)

network management; For example, “ssh login" logs have periodic
appearance because they are generated by CRON process including
remote ssh access. In this case, we consider periodic appearance as“normal" and lack of periodicity as “anomaly".
As shown in the table, CVAE’s precision (T P/(T P + F P)) outperforms PCA’s. We also confirm that CVAE and PCA detected morethan half of the bursts. By inspecting the missing bursts, we findtwo parameter tuning issues: (1) when similar burst patterns last fora few days, DBSCAN extracts a cluster, not an outlier, the algorithm
thus does not detect this burst as an anomaly. (2) Intensive bursts
cause CVAE and PCA to detect a relatively small burst as a normal.
Figure 4 is the histogram of anomalies per log category. Orange bars indicate anomalies detected with CVAE, white ones with
PCA, and gray ones with burst detection. In most cases, we findthat CVAE correctly detects anomalies such as burstiness and lackof periodicity. Focusing on the case in which PCA detects moreanomalies than CVAE, we find anomalies not detected with CVAE
as false positives due to noise, which CVAE properly avoids (e.g.,interface(agg) and service(ntp)). On the other hand, when CVAE detects more anomalies than PCA (e.g., vpn(mpls)), we find that they
are mostly true positives thanks to its robustness against outliers(see § 5.3).

5.3

Detailed comparison

Anomalies hidden in periodicity: Figure 5 shows the detailed
results of periodic remote access logs for 456 days. The top graphshows KLD computed with CVAE and the bottom graph illustrates
reconstruction errors obtained with PCA. These logs appear once
per hour due to an automated remote monitoring script. Some
non-periodic bursts also occur in the logs. We confirm two significant outliers in the figure due to many log appearances (label (a))
and missing periodic logs (label (b)). As non-periodic anomalies
represent failures of automatic remote access login, we intend to
detect these non-periodic anomalies without detecting periodic
bursts. As shown in this figure, CVAE and PCA correctly separate

these anomalous data from others. Furthermore, the burst detection
finds only periodic bursts. CVAE and PCA correctly learn eventwise features (in this case, periodic appearance) thanks to theirdiscriminative power.
Sensitivity to phase changes: The periodic patterns of this log
appearances (i.e., its phase) changes at the 274th day (label (c)). This
phase change is due to the timing shift of a log appearing with thesame frequency; this log appears in five minutes periods. From day
0 to day 273, each log appears almost at the same time and period(beginning at 0:44 in an hour period). After day 274, the appearance
time shifts though the appearance period is the same (beginning at
0:10 in an hour period).
As shown in the bottom graph of Figure 5, it is difficult for PCA
to detect the state change at the orange line because the values ofreconstruction error seems stable.
On the other hand, CVAE easily detects the change in the behavior at the orange line and DBSCAN correctly separates the data at
this point. Therefore, CVAE is more discriminative to capture thecomplex change in time series, which is not detected with PCA.
Robustness against outliers: KLD is robust against the intensity (i.e., number of log appearances) of the original data, as shownin Figure 3. On the other hand, we confirm that PCA’s reconstruction errors (not shown in the figure) are also biased to the intensity,
similar to the case with MLH.
Figure 6 (MPLS path down event) shows an example of CVAE’s
sensitivity to a small number of log appearances. The top graphshows KLD with CVAE, and the bottom graph shows the reconstruction errors with PCA. The solid blue circles are normal points
and the crosses are anomalies based on DBSCAN clustering. Notethat anomalies in the top graph are clustering results after merging
ten trials, but the KLD values are a result of one trial. These data
have one large outlier (around the 17th day; label (a)) which has alarger number of log appearances than other days. As shown in the
figure, PCA is affected by this outlier, and many anomalous points
are not detected. With CVAE, however, the KLD values are relatively spread compared with reconstruction errors, and DBSCAN
detects other anomalies.
If there are no outliers but logs are noisy, PCA yields many
false anomalies due to its noise-sensitivity. However, as CVAE isstochastic and merges the multiple results, we can mitigate the effectof noises. Therefore, CVAE exhibits better robustness to outliers
and noises than PCA.
However, we also confirm that less than 1% of the data have high
KLD values (> 10−4 ), as shown in Figure 3. It is still possible thatthe clustering process may miss small KLD value changes due tothese high values. We will work to improve this issue by tuning theclustering process as future work.

5.4

Case Studies

We now discuss anomaly examples detected from our proposed
method.
5.4.1 BGP state change. Figure 7 shows the first example ofdetected anomalies. In this figure, each mark shows the KLD of
the BGP state change logs from a router per day. Almost all days
have a baseline (≈ 6.0 × 10−6 ). The detected anomalies in < 100days are due to changes in network configuration. Furthermore,BIG-DAMA ’18, August 20, 2018, Budapest, Hungary

K. Otomo et al.
2.00

×10−5

(c)

anomaly data (few logs)
anomaly data (much logs)
DBSCAN cluster 0
DBSCAN cluster 1
DBSCAN anomaly

5
4
3

(b)

CVAE

0

0

(a)
100

200

anomaly data (few logs)
anomaly data (much logs)
DBSCAN cluster 0
DBSCAN anomaly

1.25
1.00

300
(c)

CVAE
(a)

1.50

(a)

1.25
1.00
0.75

(b)

0.50

No logs
Multicast logs
Unicast logs
DBSCAN anomaly

0.25
0.00

400

0

Day

×10−5

Reconstruction Error

(b)

2
1

×10−5

1.75

KL-Divergence

KL-Divergence

6

100

200

300

400

Day

(a)

Figure 8: Case 2: MTU reduced logs in a router

0.75
0.50

(b)

0.25

PCA

0.00

0

(b)
100

200

300

400

Day

Figure 5: KLD (CVAE) and reconstruction errors (PCA)

KL-Divergence

×10−5

(a)

NormalAnomaly

CVAE

2.5
2.0
1.5

6

1.0
0.5
0

Reconstruction Errors

×10−4

6

100

200

300

400

Day

(a)

NormalAnomaly

PCA

5
4
3
2
1
0
0

100

200

300

400

Day

Figure 6: Robustness of CVAE

5
4
3

(a)

2
1

CVAE

DBSCAN normal
DBSCAN anomaly

0

0

100

200

300

400

Day

Figure 7: Case 1: BGP neighbor state change logs

checking the raw logs around the 300th day (label (a)), we find that aconnection with one particular AS suddenly became unstable. This
failure had been also reported in a trouble ticket. PCA fails to detect
most of these anomalies. CVAE, on the other hand, successfully
discovers them.
5.4.2 MTU. The next example is MTU reduced logs from arouter. This log template appears in a few days with two specific

CONCLUSION

We proposed a log analysis method based on the latent variable
model to detect network system anomalies. We used Conditional
Variational Autoencoder (CVAE) and applied a clustering algorithm
to log time series with KL-divergence of the latent variable distribution. We confirmed that this method correctly works for highly
sparse and diverse time series, which cannot be handled with a traditional PCA-based method. In addition, our case studies showed thatsome detected anomalies are helpful in finding network troubles.
Towards the deployment of our method to real network management, we will work on addressing two issues: (1) how to update
the trained model to use upcoming syslog data, and (2) how manylogs are sufficient for catching the normal state in a system.

7

×10−5
6

KL-Divergencevariables: multicast or unicast IP addresses. Figure 8 shows the KLD
results. The blue round circles consisting of a baseline indicate nologs on those days. The orange crosses represent multicast logs,
and green crosses mean some unicast logs appear on those days.
Compared with the baseline, the multicast logs show higher KLD
values (cf. label (a)) and the unicast ones lower (label (b)) ones
as anomalies. Thus, CVAE correctly separates these two types ofanomalies even in this unique log template. We find that PCA misses
most anomalies annotated by label (a).

ACKNOWLEDGEMENTS

The authors would like to thank the SINET operation team forproviding the syslog and trouble tickets data. The authors would
also like to thank Johan Mazel for his comments on this paper andRoberto Gonzalez for shepherding.

REFERENCES[1] E. Baseman, S. Blanchard, and E. Zongzelimyuntedu. Relational Synthesis of
Text and Numeric Data for Anomaly Detection on Computing System Logs. In
Proc. IEEE ICMLA’16, pages 2–5, 2016.
[2] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A Density-based Algorithm for
Discovering Clusters a Density-based Algorithm for Discovering Clusters in
Large Spatial Databases with Noise. In Proc. ACM KDD’96, pages 226–231, 1996.
[3] T. Hacker, R. Pais, and C. Rong. A markov random field based approach foranalyzing supercomputer system logs. IEEE Transactions on Cloud Computing,pages 1–1, 2017.
[4] T. Kimura, K. Ishibashi, T. Mori, H. Sawada, T. Toyono, K. Nishimatsu, A. Watanabe, A. Shimoda, and K. Shiomoto. Spatio-temporal factorization of log data forunderstanding network events. In Proc IEEE INFOCOM’14, pages 610–618, 2014.
[5] T. Kimura, A. Watanabe, T. Toyono, and K. Ishibashi. Proactive failure detection
learning generation patterns of large-scale network logs. In Proc CNSM’15, pages
8–14, 2015.
[6] D. P. Kingma, D. J. Rezende, S. Mohamed, and M. Welling. Semi-Supervised
Learning with Deep Generative Models. pages 1–9, 2014.
[7] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. (Ml):1–14, 2013.

Finding Anomalies in Network System Logs
with Latent Variables[8] J. Kleinberg. Bursty and Hierarchical Structure in Streams. In Proc. ACM KDD’02,
pages 91–101, 2002.
[9] S. Kobayashi, K. Otomo, K. Fukuda, and H. Esaki. Mining causality of network
events in log data. IEEE TNSM, 15(1):53–67, 2018.
[10] A. Lakhina, M. Crovella, and C. Diot. Diagnosing network-wide traffic anomalies.
In Proc. ACM SIGCOMM’04, 34(4):219, 2004.
[11] S. Lu, B. Rao, X. Wei, B. Tak, L. Wang, and L. Wang. Log-based Abnormal Task
Detection and Root Cause Analysis for Spark. In Proc. IEEE ICWS’17, 2017.
[12] M. Moh, S. Pininti, S. Doddapaneni, and T.-S. Moh. Detecting Web Attacks Using
Multi-stage Log Analysis. In Proc. IEEE IACC’16, pages 733–738, 2016.
[13] K. Otomo, S. Kobayashi, K. Fukuda, and H. Esaki. An Analysis of Burstiness andCausality of System Logs. In Proc. AINTEC’17, pages 16–23, 2017.
[14] M. Shatnawi and M. Hefeeda. Real-time failure prediction in online services. In
Proc. IEEE INFOCOM’15, pages 1391–1399, 2015.
[15] S. Urushidani, M. Aoki, K. Fukuda, S. Abe, M. Nakamura, M. Koibuchi, Y. Ji, andS. Yamada. Highly available network design and resource management of SINET4.
Telecommunication Systems, 56(1):33–47, 2014.
[16] W. Xu, L. Huang, A. Fox, D. Patterson, M. I. Jordan, L. Huang, A. Fox, D. Patterson,and M. I. Jordan. Detecting Large-Scale System Problems by Mining Console
Logs. In Proc. ACM SOSP’09, pages 117–131, 2009.
[17] Z. Zheng, Z. Lan, B. H. Park, and A. Geist. System log pre-processing to improve
failure prediction. In Prco IEEE DSN’09, pages 572–577, 2009.
[18] J. Zhong, W. Guo, and Z. Wang. Study on network failure prediction based on
alarm logs. In Proc. ICBDSC’16, pages 23–29, 2016.

BIG-DAMA ’18, August 20, 2018, Budapest, Hungary