I want to detect anomalies in logs and traces, based on this pipeline: 
3 APPROACH
The objective of DeepTraLog is to automatically and accurately
detect anomalous traces of microservice systems. It takes traces and
logs as input and trains a graph-based deep learning model. When
used for anomaly detection, it analyzes a trace and the associated
logs in a similar way and uses the model to generate a representation
for the trace to calculate its anomaly score.
An overview of DeepTraLog is presented in Figure 3, which includes six steps. Log Parsing parses the input logs and extracts log
events from the log messages. Trace Parsing parses the input traces
and converts their spans into span events, which will be analyzed
ICSE ’22, May 21–29, 2022, Pittsburgh, PA, USA Chenxi Zhang, Xin Peng, Chaofeng Sha, Ke Zhang, Zhenqing Fu, Xiya Wu, Qingwei Lin, and Dongmei Zhang
Figure 3: DeepTraLog Overview
together with log events. Event Embedding generates vector representations for span events and log events. Graph Construction
constructs a trace event graph (TEG) for each trace to represent various relationships between the span/log events of the trace. Model
Training trains a gated graph neural networks (GGNNs) based
deep SVDD (Support Vector Data Description) model, which learns
a latent representation for each TEG and a minimized hypersphere
that encloses the representations of the TEGs. Anomaly Detection determines whether a trace is anomalous based on its anomaly
score (i.e., the shortest distance from the latent representation of
its TEG to the hypersphere).
3.1 Log Parsing
Following existing researches on log anomaly detection [24, 40],
we adopt a state-of-the-art log parsing approach Drain [11]. It can
parse logs in a streaming and timely manner with high parsing
accuracy and efficiency. To support the combined analysis with
traces, we extract and record the trace ID and span ID of each log
message before log parsing. After log parsing, a trace ID and a span
ID are attached with each of the extracted log events for further
analysis.
3.2 Trace Parsing
The log events of a service instance constitute an event sequence. To
combine traces and logs for anomaly detection, we need to convert
the spans of a trace into span events. Span events are a special
kind of events that represent the request and response of service
invocations. When analyzed together with log events, span events
can indicate the starts and ends of service invocations.
For each span of a trace, we convert it into multiple span events
of related spans according to its type (Client/Server or Producer/Consumer). A Client/Server span represents a synchronous invocation,
while a Producer/Consumer span represents an asynchronous invocation. For a Client/Server span we generate a request event and
a response event for the current span (server) and its parent span
(client) respectively. The request event and the response event of
the client (server) represents the sending (receiving) of the service
invocation and the receiving (sending) of the invocation response
respectively. For a Producer/Consumer span we generate a consumer
event for the current span (consumer) and a producer event for the
parent span of the current span (producer). The producer event
and the consumer event represent the sending and receiving of the
message respectively.
The content of a span event includes two parts, an event type and
an operation name. For example, for a Client/Server span with the
operation name POST /api/v1/foodservice/orders, we generate two
span events “Server Request POST /api/v1/foodservice/orders” and
“Server Response POST /api/v1/foodservice/orders” for the current
span and two span events “Client Request POST /api/v1/foodservice/
orders” and “Client Response POST /api/v1/foodservice/orders” for
the parent span of the current span; for a Producer/Consumer span
with the operation name RabbitMQ/Topic/Queue/email/sendEmail,
we generate a span event “Consumer RabbitMQ/Topic/Queue/email/sendEmail” for the current span and a span event “Producer
RabbitMQ/Topic/Queue/email/sendEmail” for the parent span of
the current span. Each span event has a timestamp obtained from
the span record. For example, the timestamp of a client request
event is the time when the client service instance sends the service
invocation.
3.3 Event Embedding
Log event embedding is widely used in log anomaly detection. It
generates a vector representation for each log event. The vector
representation can identify semantically similar log events and also
distinguish different log events [40]. In DeepTraLog, span events are
analyzed together with log events for anomaly detection. Therefore,
our event embedding generates vector representations for both log
events and span events. Each log event or span event is a sequence
of English words, thus can be treated as a sentence. Following the
common practice of log event embedding, DeepTraLog implements
event embedding in three steps.
Step 1. Preprocessing. Log events and span events contain
non-character tokens (e.g., separators such as “/” and “,”, IP addresses), stop words (e.g., “a”, “the”, “is”), and compound words
(e.g., “verifycode”, “tripid”). Following previous works [24, 40], we
preprocess log events and span events by removing non-verbal
symbols and stop words, and splitting compound words into individual words. For example, a span event “Client Request POST
/api/v1/foodservice/orders” is preprocessed into “client request post
api v1 food service orders”.
Step 2. Word Embedding. We use the widely used pre-trained
GloVe model [29] to generate a vector representation for each word
in log events and span events. Particularly, we use the GloVe 300-
dimensional word vectors trained on Wikipedia and Gigaword 5
data [9]. Thus for each word we can obtain a 300-dimensional vector
as its representation.
Step 3. Sentence Embedding. Sentence embedding generates
a vector representation for each log event and span event based
on word embedding. The words in these events are not equally
important. Some words (e.g., “api”, “get”) are more common, thus
are less important than others (e.g., “food”, “submit”) in sentence
embedding. Therefore, we follow previous works [40] and use TFIDF [31] to measure the weight of each word in sentence embedding.
TF (Term Frequency) of a word 𝑤 in an event 𝑒, which measures
its importance in 𝑒, is calculated by 𝑇 𝐹 𝑤,𝑒 =
𝐿𝑤,𝑒
𝐿𝑒
, where 𝐿𝑤,𝑒 is
the occurrences of 𝑤 in 𝑒 and 𝐿𝑒 is the number of words in 𝑒. IDF
(Inverse Document Frequency) of a word 𝑤, which measures its
frequency in all events, is calculated by 𝐼𝐷𝐹 𝑤 = log 𝐿
𝐿𝑤
, where
𝐿 is the total number of events and 𝐿𝑤 is the number of events
containing 𝑤. Then the weight of a word 𝑤 in an event 𝑒 can be
measured by its TF-IDF score as 𝑊𝑤,𝑒 = 𝑇 𝐹 𝑤,𝑒 × 𝐼𝐷𝐹 𝑤. Thus, the
vector representation of an event 𝑒 can be calculated as the weighted
sum of the vector representations of all its words as following,
where 𝑁𝑒 is the number of different words in 𝑒 and 𝑉𝑤 is the vector
representation of a word 𝑤.
𝑉𝑒 =
1
𝑁𝑒
∑︁
𝑁𝑒
𝑤=1
𝑊𝑤,𝑒 · 𝑉𝑤 (1)
Note that log events and span events have quite different characteristics, so we calculate the TF-IDF scores and word weights for
log events and span events separately.
3.4 Graph Construction
A trace event graph (TEG) consists of the log events and span events
of a trace and their relationships. A relationship in a TEG can be
one of the following four types.
• Sequence: A sequence relationship represents the predecessor/successor relationship between two sequential span/log
events of the same span
• Synchronous Request: A synchronous request relationship
represents a synchronous request from a parent span to its
child span.
• Synchronous Response: A synchronous response relationship represents the response of a synchronous request from
a span to its parent span.
• Asynchronous Request: An asynchronous request relationship represents an asynchronous request from a parent
span to its child span.
Given a trace, we construct a TEG in three steps.
Step 1: Connection of Log Events. For each span of the trace,
obtain all the log events that belong to the span. Then order the log
events by their timestamps and add a sequence relationship from
each log event to the one next to it.
Step 2: Insertion of Span Events. For each span of the trace,
obtain all the span events that belong to the span. Then for each
obtained span event, insert it into the log event sequence of the
current span based on its timestamp and add a sequence relationship
between it and its predecessor/successor event.
Step 3: Connection of Spans. For each span connect it with its
parent span in the following way. If the span is a Client/Server span,
add a synchronous request relationship from the corresponding
client request event of its parent span to the corresponding server
request event of the current span, and a synchronous response
relationship from the corresponding server response event of the
current span to the corresponding client response event of its parent
span. If the span is a Producer/Consumer) span, add an asynchronous
request relationship from the corresponding producer event of its
parent span to the corresponding consumer event of the current
span.
Figure 4 shows an example of TEG, which corresponds to the
trace shown in Figure 1 and Figure 2. In the graph, colored rectangles and uncolored rectangles represent span events and log
events respectively. Span events of different colors are of different
types, including client/server request, client/server response, and
producer/consumer. Note that the dotted rounded rectangles which
represent spans in Figure 4 are not the nodes of the TEG but are
just used for illustration. The spans are implicitly represented in
the TEG by the event sequences starting from server request events.
3.5 Model Training
We frame the task of trace-log combined microservice anomaly
detection as an one-class classification problem. One-class classification problem aims to learn a model that can accurately describe
the train data which is considered belong to the same class. For
anomaly detection task, most of the training samples are normal
ones which can be regarded as the same class and the others can
identified as outliers (e.g., anomalous traces in our work).
DeepTraLog uses deep SVDD [30] to train a one-class classification model for detecting anomalous traces. Deep SVDD learns
useful feature representations of the training data together with
a minimized hypersphere that encloses the latent representations
of the data. Thus the data that is distant from the center of the
hypersphere can be considered anomalous. Existing deep learning
based anomaly detection approaches [20, 26] usually use deep autoencoder to detect anomalies based on the reconstruction error
of the data. Thus the anomaly detection relies on an empirically
determined threshold of the reconstruction error, which is often
challenging. In contrast, deep SVDD jointly learns the latent representations of the data and a classification boundary without a
threshold. Standard deep SVDD uses multi-layer perception (MLP)
or convolutional neural networks (CNN) to learns the latent representations of the data. However, in our work a trace is represented
by a graph (TEG) which cannot be well handled by MLP or CNN.
Therefore, we use gated graph neural networks (GGNNs) to learn
the trace representations and jointly train the GGNNs with deep
SVDD. GGNNs is implemented based on the neural message passing mechanism and can work well with a variety types of graphs,
such as directed graph, bipartite graph, and undirected graph.
In DeepTraLog, a trace is represented by a TEG, which is a
directed attributed graph 𝑔 = {𝑉 , 𝐴, 𝑋} where: 𝑉 is a set of nodes
(i.e., events); 𝐴 is adjacency matrix of the graph; and 𝑋 ∈ R
|𝑉 |×𝑑
is
the node attribute matrix, where each row 𝑥𝑣 of 𝑋 is the attribute
(i.e., event vector) of a node 𝑣 ∈ 𝑉 , 𝑑 is the dimension of the event
vector. GGNNs represents the nodes in a graph as units of a neural
network and the units are linked to each other according to the
adjacency matrix of the graph. GGNNs passes the node attributes as
the messages between the units in every iteration and uses GRU [4]

to determine which messages to remember or forget during the
message passing. The final representation of a node is determined
by a combination of its own state and the state of neighbouring
nodes. The representation of a node after the 𝑡-th iteration is defined
by the following equations:
ℎ
(0)
𝑣 = 𝑥𝑣
(2)
𝑚
(𝑡)
𝑣 = 𝐴
T
𝑣
[ℎ
(𝑡−1)T
1
...ℎ(𝑡−1)T
|𝑉 |
]
T + 𝑏 (3)
ℎ
(𝑡)
𝑣 = GRU(𝑚
(𝑡)
𝑣
, ℎ(𝑡−1)
𝑣
) (4)
where ℎ
(𝑖)
𝑣
is the representation of a node 𝑣 after the 𝑖-th iteration; 𝑥𝑣 is the initial event vector of node 𝑣; 𝐴𝑣 = [𝑎
T
𝑣,:
, 𝑎:,𝑣 ] is the
row and column in the adjacency matrix 𝐴, which represent the
incoming edges and outgoing edges of 𝑣; 𝐺𝑅𝑈 is the GRU function.
Finally after 𝑇 iterations each node in a TEG can have a vector as
the latent representation. GGNNs calculates the vector representation of the TEG based on the node vector representations using
a soft-attention mechanism. The soft-attention mechanism takes
the vector representations of the nodes as input and calculates the
weight (attention score) of each node through an attention function
to give higher weights to the nodes that contribute more to the
graph classification. The graph representation of TEG𝑔 is calculated
by the following equation:
ℎ𝑔 = tanh ∑︁
𝑣∈𝑉
𝜙

𝑓𝑖(ℎ
(𝑇 )
𝑣
, 𝑥𝑣 )

⊙ tanh 
𝑓𝑗 (ℎ
(𝑇 )
𝑣
, 𝑥𝑣 )

!
(5)
where ℎ𝑔 is the vector representation of a TEG 𝑔; 𝜙 (𝑓𝑖(ℎ𝑣 (𝑡), 𝑥𝑣 ))
is the soft-attention mechanism; 𝑓𝑖 and 𝑓𝑗 are neural networks; 𝑇 is
the number of layers of GGNNs; ⊙ is element-wise multiplication.
Readers can refer to [17] for more details about GGNNs.
DeepTraLog trains the GGNNs using the following loss function,
which expresses the objective of learning a minimized hypersphere
to enclose the vector representations of TEGs:
𝐿𝑜𝑠𝑠 = 𝑅
2 +
1
𝜇𝑁𝑔
∑︁
𝑁𝑔
𝑔=1
max {0, ||ℎ𝑔 − 𝑐||2−𝑅
2
} + 𝜆
2
∑︁
𝑁𝑙
𝑙=1
||𝜃
(𝑙)
||2
𝐹
(6)
where 𝑐 is the center of the hypersphere; 𝑅 is the radius of the
hypersphere; ||ℎ𝑔 − 𝑐||2
is the distance from the latent representation of a TEG 𝑔 to 𝑐; 𝑁𝑔 is the number of TEGs; 𝑁𝑙
is the number
of network layers in GGNNs; the hyperparameter 𝜇 controls the
trade-off between the hypersphere volume and the violations of
the boundary, which allows some training data mapped outside
the hypersphere (i.e., allowing some anomaly data in the training
set); 𝜆
2
Í𝑁𝑙
𝑙=1
||𝜃
(𝑙)
||2
𝐹
is a weight decay regularizer on the GGNNs
parameters 𝜃 with hyperparameter 𝜆.
In the training phase, we jointly optimize the GGNNs parameters
𝜃 and the hypersphere radius 𝑅 in Equation 6. We use Adam [14] to
optimize the GGNNs parameters 𝜃 in each epoch. As the radius 𝑅 is
not an inner parameter of GGNNs, we optimize it using a different
method as follows. We optimize 𝜃 with a fixed 𝑅 in the first few
epochs and after every 𝑘 epochs we calculate an optimized value
for 𝑅 by linear search. Each time when 𝑅 is updated its value is
calculated as the (1 − 𝜇) percentile of the distances of all the TEGs
in the current epoch. The hypersphere center𝑐 is set to the mean of
the vector representations of all the TEGs after an initial forward
pass.
3.6 Anomaly Detection
DeepTraLog trains a GGNNs based deep SVDD model for anomaly detection. Given a new trace and the corresponding log for
anomaly detection, DeepTraLog follows the same process as the
training phase (i.e., log parsing, trace parsing, event embedding,
and graph construction) to produce a TEG together with its event
embedding for the trace. Then DeepTraLog feeds the TEG into the
trained model to generate a latent representation of the TEG and
uses the latent representation to calculate an anomaly score. The
anomaly score is defined as the shortest distance from the latent
representation of the TEG to the learned hypersphere, which is
calculated by the following equation:
𝑎𝑛𝑠(ℎ𝑔) = ||ℎ𝑔 − 𝑐||2−𝑅ˆ2
(7)
where ℎ𝑔 is the vector representation of a TEG 𝑔; 𝑐 is the center
of the learned hypersphere; 𝑅ˆ is the final radius of the learned
hypersphere.
For a TEG𝑔 of a trace, if its anomaly score (i.e., 𝑎𝑛𝑠(ℎ𝑔)) is greater
than 0 it is treated as anomalous.
To help the users to understand the results of anomaly detection,
DeepTraLog provides a visualization of the TEGs of anomalous
traces. Figure 5 presents an example of the visualization of an
anomalous TEG. Specifically, DeepTraLog highlights the nodes in
the TEG that have high attention scores calculated using Equation 5,
which helps the users to quickly locate the anomalous parts of the
trace.