Log representation as an interface for log processing applications
Authors: Mohammad Amin Sadeghi
Shameem Parambath
Ji Lucas
Youssef Meguebli
Maguette Toure
Fawaz Al Qahtani
Ting Yu
Sanjay Chawla
Venue:  (2021)
Keywords: Vector embedding
Logs
Search
Causal Analysis
Anomaly Detection

Abstract: Log files are files that record events, messages, or transactions. Logs are rich containers of data because they can store a sequence of structured textual and numerical data. Many sequential forms of data including natural languages and temporal signals can be represented as logs.

We propose to represent logs at a few levels of abstraction including field level, log level, and log sequence level. The representation for each level can be computed from the previous level. These representations are in vector format and serve as interfaces to downstream applications. We use a version of transformer networks to encode numerical information as well as textual information that is suitable for log embedding. We show how a number of log processing applications can be readily solved with our representation.

Under review as a conference paper at ICLR 2021
LOG REPRESENTATION AS AN INTERFACE FOR LOG PROCESSING APPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Log files from computer systems are ubiquitous and record events, messages, or transactions. Logs are rich containers of data because they can store sequence of structured textual and numerical data. Many sequential forms of data including natural languages and temporal signals can be represented as logs.
We propose to represent logs at a few levels of abstraction including field level, log level, and log sequence level. The representation for each level can be puted from the previous level. These representations are in vector format and serve as interfaces to downstream applications. We use version of Transformer Networks (TNs) to encode numerical and textual information that is suitable for log embeddings. We show how number of log processing applications can be readily solved with our representation.
INTRODUCTION
wide range of computer systems record their events logs. Log-generating systems include telecommunication systems, data centers, software applications, operating systems. sensors, banks, markets, and block-chains (Barik et al. | (2016); Brandt et al. (2020): Busany Maoz (2016); Cu- curull & Puiggali () Sutton & Samavi 2017)). In these systems, transactions, events, actions, communications, and errors messages are documented as log entries. Log entries are stored as plain text, they can store any textual or numerical information. A variety of different data types could be viewed as logs, including: natural languages, temporal signals, and even DNA sequences.
All transactions in a Turing machine can be stored as logs. Therefore, logs theoretically strong enough that they can reproduce the state of any computer system. some systems log entries standard and complete. For example, in financial transactions and some database management systems, one can recreate the state of the system by applying a set of rules on the transactions [Mohan et al. (1922). In other systems like telecommunication networks log entries are ad hoc, unstructured, and diverse. Therefore, the state of the system cannot be recreated with a set of rules. We will use layered and learnt vector embeddings to represent the state of the system and use them for downstream diagnostic applications including: anomaly detection, classification, causal analysis and search.
Example: A snapshot of logs from a telecommunication product installed in cell tower is shown Figure [ The first entry of the log trigger for subsequent action of the unit be restarted. Each log entry will be embedded in vector space and the sequence of log entrys will have their own vector representation. Timestamp information provides crucial clues about the nature of the event(s).
We summarize our contributions as:
1. We propose levels of abstraction for log representation, in order to standardize and simplify log processing applications.
2. We present Transformer-bant based model to embed log sequences in vector space.
3. We show how log processing applications can be simplified using these log representations. We validate our approach on real data set obtained from a leading telccommunications vendor. The vocabulary of this data set is twenty times bigger than what is currently avail- able in open source and often used in other research papers
After suitable anonymization, plan release the data set the research community.

---

Under review as a conference paper at ICLR 2021
BCSU-1
2017-03-28
DISTUR BCSU-1 1A904-00
USAPRO
(1596) 1623 EXCESSIVE DISTURBANCES IN SUPERVISION
00A0 255d
<HIST> MSARSC4
OMU
SWITCH
2017-03-28
14:44:10.07
NOTICE BCSU-1 1A904-00
RCXPRO
(1597) 8691 AUTOMATIC RECOVERY ACTION
WO-EX SP-RE 0006 0062 7889 88001823 68800те0 86606000
<HIST> MSA8SC4
OMU RCXPRO
SWITCH
2017-03-28
14:44:22.09
DISTUR BCSU-1
1A804-80
(1599) 1061 UNIT RESTARTED
SP-RF 66
Figure 1: Sample telecommunication log file including three log entries. Telecommunication logs are complex and diverse because they involve various devices and software.
1.1 RELATED WORK
Much of the machine learning oriented work in log analysis has focused primarily on the outlier or anomaly detection problem Duu et al. (2017); Meng et al. (2019); Du et al. (2019a by: Yuan et al.)
2020 Nedelkoski al. 2020); Chalapathy & Chawla (2019h).
The prototypical work is DeepLog which, in a fashion analogous to natural language processing, models logs sequences from restricted vocabulary following certain patterns and rules [Du et al. (2017)) An LSTM model M log executions is inferred from a database of log sequences. To determine a given element W14-1 in log sequence is normal or anomalous, DeepLog outputs the probability distribution Par (- (-[W1:)) where W1:t (WI, W2...... w.) ) If the actual token W1+1 ranked high in PM (w W1:t) then it is deemed as a normal event otherwise is flagged as anomalous. Several variations on the above approach have been proposed. For example, (Yuan et al.
(2020) proposed ADA (Adaptive Deep Log Anomaly Detector) that exploit online deep learning Sahoo et al. a (018) methodology to build on the fly unsupervised adaptive deep log anomaly detector with LSTM. The new models are trained on new log samples.
More recently, Nedelkoski et al. 2020) proposed Logsy, classification-base method to learn ef- fective vector representations log entries for downstream tasks like anomaly detection. The core idea of the proposed approach is to make use of the easily accessible auxiliary data to supplement the positive (normal) only target training data samples. The auxiliary logs that constitute anomalous data samples can be obtained from the internet. The intuition behind such approach to anomaly detection that the auxiliary dataset is sufficiently informative to enhance the representation of the normal and abnormal data, yet diverse enough to regularize against over-fiting and improve generalization.
SYSTEM LOGS AND NATURAL LANGUAGE MODELS
Log processing models often adopt natural language processing techniques, because logs have sev- eral similarities natural language: (i) Both logs and natural language consist of sequence of tokens and (ii) Context matters in both data streams and both models need temporal memory, (ii) Both application scenarios have large datasets and (iv) Annotation is limiting factor both settings. This is why log processing literature often reuses natural language approaches.
However there several differences between system logs and natural languages that need to be brought to the fore: (i) There is temporal information associated with each log entry which can be exploited to give insights about the underlying processes that are generating the logs (ii) Each log entry itself is a composite record of different pieces of information unlike word in a sentence which nearly atomic, (iii) Log files often aggregate event logs from multiple threads, processes and devices. Therefore the inference model needs to identify the relevant context among all threads.
2.1 SEQ-TO-SEL MODELS
Transformer Networks (TNs) have become the de-facto choice for sequence-to-sequenee model- ing( Vaswann et al. 2017)). Based the concept of self-attention, TNs overcome some of the key limitations families of Recursive Neural Networks (RNN) including LSTMs and GRUs (Graves)
& Schmidhuber 2005)). Transformers Networks model long range dependencies with constant

---

Under review as a conference paper at ICLR 2021
(Q()) number of iterations overcoming the RNN limitation where modeling dependencies with dis- tance n takes ((n) iterations. Furthermore since TNs are intrinsically permutation invariant, they can process elements of log sequence in parallel. This last property is particularly useful in log analysis, perhaps more than NLP, due to the observation made above that log files aggregate event information from different processes. Before we formally describe our extension of Transformer Networks, we introduce the different levels of abstractions suitable for modeling log sequences.
3 LEVELS OF ABSTRACTION
In order to standardize and simplify machine learning applications logs, propose five levels of abstraction for log representation. Each level of abstraction can be computed from the previous levels.
1- Log sequence: A log sequence is a sequence of strings (1, 11,,,,,--,, ,, We refer to each 14 as a log entry. Each log entry itself is sequence of symbols (characters) from an alphabet
1 (cI, Cm.) where Vj,ej E
Log sequences often contain certain parameters including timestamps, IDs and numerical values. In order to access these parameters, we often need to parse logs.
2- Parsed log: Whether log entry has a simple or complex grammar, a log parser will convert it into a set of key-value pairs. Keys represent attributes and values represent the value for each attribute. A parsed log entry can be written as:
{ki vi),..., (kmm, Um mkg E 22 EV).
(2)
where K is the set of keys (attributes) and V is the set of possible values. Depending on the key, the value could be textual, numerical or categorical. wide range of log parsers are available and have been evaluated (He et al. (2016); Zhu et al. (2019): |: He et al. (2017))
3. Field Embedding: A field embedding vector representation for key-value pair. field embedding function p maps a key-value pair into a dddimensiona vector:
p: p:KVV - TR.
(3)
D, denotes the dimensionality of field representation. A field embedding function could either be learned or static. They choice of function could depend on key and data type. Since the set of categorical values is often small, categorical values could be stored in vector quantized format. Similar to many NLP techniques, a look-up table can store vector representations.
4- Log Embedding: A log embedding is vector representation for single log entry. A log embedding function & computes a log representation given its field embeddings:
R Dj
R
Ri
Di denotes the dimensionality the log embedding. Log embedding could be calculated from field embeddings using DeepSet (Zaheer al. (2017) other means. This log embedding be used for classification, regression, nearest neighbour search or other algebraic operations.
5- Sequence Embedding: A Sequence embedding is vector representation for log sequence. This sequence could either be a whole log file, a block of logs, or a sliding window. A sequence embedding function inputs a sequence of log embeddings and outputs single embedding for the whole sequence:
0 DI X
Di D
(5)
Dj denotes the dimensionality of log embedding. The function @ could be implemented using RNNs or Transformers or other means. We will show that this sequence representation can be used for several downstream operations on log sequences. Some sequence embedding models such as Trans- formers come with a language model. We show that this language model also useful in log processing applications.
3

---

Under review as a conference paper at ICLR 2021
fraction of abnormal logs cluster
Unsupervised anomaly detection
1.0
Abnormal Normal
0.0
Cluder stre
Proctsion
0.6
AP=0.678
0.0
2
9 10111213
0.00
0.25
0.50
0.75
1.00
DSNE visualization
number
Recall
Figure 2: Unsupervised anomaly detection Left: t-SNE embedding of our vector representations for log sequences in HDFS dataset. Yellow points mark normal log sequences and blue points mark abnormal log sequences (based on human expert annotation). The red plus sign marks the center of the distribution. Note that abnormal sequences are mostly clustered and are far from the center. Center: K-means clustering of our vector representations. Clusters are sorted decreasingly according their size. Note that abnormal examples are concentrated in the smallest clusters. Right: After we sort clusters, we consider cluster number as anomaly score. Here is the P-R curve for predicting anomaies. Note that every step this pipeline unsupervised.
4 TRANSFORMER NETWORK
We generalize TNs to incorporate the notion of time which is typically available in log data. We use the self-supervised task of masked language model (MLM) to train the network and obtain embeddings. Recall in the MLM task we are given set of sequences {s; (122... -, li 1,..., m) For every sequence s we randomly select a location ry in the sequence and then predict P(I, 1-, Devlin et al. (2088). While the MLM task by itself may not be particularly useful applications, the embeddings learnt can fine-tuned to other applications.
Recall transformer networks use the notion of attention to directly estimate the influence of the input context for generating the output token. Attention generalize the notion of the SELECT operation in database query languages. For example give a query q and table of key-value pairs (k,, u;), attention is defined Σ, sim(q, k;)u;, The sum is over the input sequence length and sim is the softmaxed similarity between the embedding of the query q and the keys kt
4.1
TIME ENCODINGS
Given a log sequence ((1, (1) .., (ly (n)) where 4 is the token and 4, is the associated timestamp. associate cumulative time CT()) = E)su(t) tz-1). The Time Encoding (TE) for token position i and dimension å 1,..., , d is given by
sin CT()) 10000
if 5 = 28'
TE(;,))
cos(
CT())
if = 28' +
1000026
Several deep learning architectures are developed to predict masked tokens. Since language models need have memory and need to handle sequences of unknown length, they have traditionally relied on recurrent models and their variants. TNs have shown several advantages over RNNs and their variants and are now the architecture of choice in many NLP tasks.
We performed an ablation study to evaluate the effect of time encoding (figure 3 left). We trained one log language model that ignores timestamps and one language model that uses timestamps. The language model that is trained with time, performs significantly better than the language model that does not use time. We evaluated this using masked language modeling. We mask a log and try to predict it using nearby words. Our experiments also show that when the timestamp of a masked log revealed, we can predict the masked word with a significantly smaller error. This confirm that timestamp is helpful in predicting masked logs.

---

Under review as a conference paper at ICLR 2021
effect masked language modeling
Performance different techniques HDFS benchmark
masked language modeling test loss
N-gram
Deeplog
Ours
0.0098
1.0
1.06
0.979
0:978
0.0086
0.92
0.95
0.95
0.360.96
0.91
0.94
0.9
0.88
0.0062
0.79
0.0080
0.7
0.67
0.0056
0.6
Time unavaliable
Time available
Time available during test
while training
Figure 3: Left Masked language modeling loss (cross entropy) during test. Left shows test loss using a model that is not trained to use time. Middle bar and right bar show test loss on model that trained using timestamp. In the middle bar, timestamp for the masked word is also masked. In the right bar, timestamp for the masked bar is revealed. Right: Comparison of different techniques oh HDFS dataset. Our model significantly outperform previous models.
5 APPLICATIONS AND EMPIRICAL RESULTS
As soon as log sequence is embedded into a vector space, many log processing tasks can be ex- pressed algebraic operations within the vector space. These tasks include: Anomaly detection, Predictive analysis, Causal analysis, Log search, Diagnosis Recommender system and Log genera- tion/synthesis. In this section we quantitatively evaluate the application of the obtained embeddings for these downstream tasks.
5.1 DATASETS
We have used two datasets for our experiments: HDFS and a proprietary data from a leading telecommunications vendor.
HDFS Log Data: This is an open-source dataset consisting of logs from Hadoop jobs on 200 node Amazon EC2 cluster. There over 11 million log entrys and around 2.9% anomalous as labeled by domain experts (Du et al. (2017)) Tokenization resulted in a vocabulary of size 29.
We obtained real but proprietary data set from leading telecommunication vendor. The data consists of logs from a product installed in a cell tower for 4G network. The data set consists of 4783 distinct log files consisting of 1464010 log entries and collected over period of two months. This dataset consists of about 860 different log templates and about 180 different software programs.
Radio Data:
5.2 SOFTWARE AND ARCHITECTURE
We extended the HuggingFace transformer library in pytorch to incorporate time embeddings. We used a simple two layer transformer network, train over ten epochs with batch size of sixteen. In terms of proposed levels of abstraction (Section 3), vector quantize log fields (Layer and feed them into the transformer network which constitute Layer 4 and of our proposed abstraction.
5.3 ANOMALY DETECTION
Identifying Log Anomalies is crucial for system administration because they could signal errors, system failures, or fraud. Log embedding simplifies the process of anomaly detection. Log embed- dings simplify anomaly detection both in unsupervised and supervised settings.
Self-supervised techniques are useful when manual supervision is unavailable. Since our embedding function is learned using self-supervised model, an application that uses our embedding, automatically benefits from seff-supervision (in transfer learning scheme).
Unsupervised:
We evaluate our unsupervised anomaly detection technique on HDFS dataset. Given a set of log sequences, We first compute vector representations for each sequence. Then we cluster the examples using k-means. sort clusters decreasingly according to their size. Since anomalous logs are clustered and rare, they tend to fall into the smallest clusters. We assign identical anomaly score

---

Under review as a conference paper at ICLR 2021
Average vs. time-frame
Normalized P-R time-frame
1.0
1.0
0.90
0.8
Precision
t=2 Sec
Precision
0.6
0.86
Precisioe
0.06
t-3 Sec
0.84
t=7 Sec
0.4
0.82
0..4
t=10 Sec
b
0.80
t=15 Sec
0.2
AP (normalized)
0.2
t=20
1-40 Sec
t-60 Sec
0.0
0.0
0.0
0.2
0.4
0.6 Recall
0.8
1.0
2
10
40
0.2
0.4
0.6
0.8
(seconds)
Recall
Figure 4: Predicting whether or not an event type will appear within the next seconds from now. Left: Precision-Recat curves for predicting whether event types will appear within the next t t= 10 seconds. Note that each of the six different event types has distinct P-R curve. For example, event types a,b.c are relatively common but harder to predict than d and Middle: Comparison of P-R curves for all event types across different prediction time-frames t. Please note that since longer time-frames are more likely to have an event. Thus, longer time-frames have higher chance baselines. We re-weight test examples to normalize positive-negative ratio and chance baseline. Note that by widening time-scale precision drops. Right: Average precision for predicting events at different time-frames Normalized AP generally decreases by widening time-frame. In Non- normalized AP higher time-frames have higher chance baselines. The increase at 2 40 seconds is because many processes go through repeatable shut-down process.
all points in the same cluster. Larger clusters receive smaller anomaly score and smaller clusters receive larger anomaly score. We evaluate this supervised method in figure[]
Supervised: If anomaly annotations are available, train supervised classifier to predict anoma- lies. Depending on the granularity of annotation (sequence level, time window level, or log entry level), we can extract log embeddings with the same level of granularity. Embeddings could be either context-free or context-dependen depending of the extent of log window. If several modes of anomaly are identified and annotated, classifiers can output scores/probabilities associated with each mode.
We evaluated our supervised anomaly detection model on the HDFS log dataset. Anomaly annota- tions are given log sequence level and techniques are compared based on sequence level predic- tion. In HDFS dataset, sequences are marked with block_id. Since anomalous logs are highly clustered,
SVM with RBF kernel for the classifier. Figure compares performance with standard baselines on these datasets including PCA, Invariant Mining Lou et al. (2010), N-grams, and Deeplog by (uu al. 2017)
5.4 PREDICTIVE ANALYSIS
Using log embedding also their timing and ordering. In figure we demonstrate an example where we predict whether a certain log message is going to appear within the next t seconds.
predict future events including event logs, alarms, transactions, and
We first extract embeddings from a sliding window of 64 log entries. Then we assign a target labels each window. In this example, target label is list of log entries that will appear within the next seconds after the end of the window. We then train classifier that given the embedding of log window, tries to predict the list of possible log entries that would appear within the next seconds. Our classifier predicts a probability score for the appearance of each possible log entry within the next seconds. A log entry could mark some error, the beginning or end process. Our experiments show that we can reliably predict log entries that will appear within the next few seconds.
This predictive scheme is generic. We can use a similar scheme to predict any event. For example, we can predict whether system request will fail, how long transaction will take to complete, what expect API to return, or whether a user is going to complete purchase.

---

Under review as a conference paper at ICLR 2021
ИУУУУууууу rlguazFaKk
Query:
mPDnqsBB] t
SVURQsbpxf
Exclude:
mPDnqs__..
-((tt))
log
Pitt' PP(T))
PPPqRRRERf
WtXIXDXagp
Include:
mPDnqsB_.. Pg(tt))
FB N1BBPZP
nQFWXDFRFM
Figure 5: For better readability we have visualized each log template with a letter. Left: Given a query log (highlighted blue), we use language model predict its probability twice: Once includ- ing and once excluding 4j. in both cases, log entries prior to I are included. Right: Visualization of few examples. Query log is highlighted blue. Previous logs are highlighted green according to their causal score C(1,, 4,). This visualization helps identify logs that help predict the query log the most.
5.5 CAUSAL ANALYSIS
Causal analysis helps track the cause of failures and events. True causality requires controlled experimentation which is often impractical for logs. We use proxy formulation for causality based on Granger causality (Granger (1969). We measures the ability to predict future log using
previous log. Given a log entry l at position j and a log entry 4 at position i where < we measure how well the knowledge of Ij helps us predict 4, We define C(l 1:)
C(G,4) = log P(L.4, 1.11,,,-1.11y22.1..
(6)
Given a log entry 4. we can measure the effect of each of the previous log entries 4,.j < on our prediction of P(4). We use three-layer deep neural network to predict the probability P(4). We found this model to be more accurate than our standard language model, because transformer language models are not explicitly trained to predict several locations in the future.
We did not quantitatively evaluate this causal analysis because no ground-truth available. How- ever, we illustrate results in figure 5) Depending on use case, variations of this causal analysis can be used. For example, the causal effect of one log field on another log field can be calculated. Or when two parallel log files are generated, the effect of one another can be calculated.
5.6
LOG SEARCH
Some investigativeddiagnsstt applications involve log search. For example, debugger could ob- serve an error pattern and needs to find similar patterns in history; An investigator may need to find whether certain user has followed certain user journey; Or financial analyst may need to identify the frequency of a certain pattern.
Nearest neighbor search in the log embedding space be used for search. Given a log history, extract log embedding from a sliding window of logs. Then we store the vector representations of each window in data structure. Given window of log entries query, we first compute the log embedding and then search for its nearest neighbors using the data structure.
The choice of data structure affects indexing/retrievall time complexity. One could KD-Tree, LSH-based KNN search or an inverted index. A plain array can handle thousands of log entries, KD-Trees can handle millions of log entries, and inverted index can handle billions of log entries.
plain array,
Evaluating log search performance not trivial because there is no labeled dataset. However, the closest evaluation criteria is based on edit distance (Levenshtein distance). Edit distance is not ideal because cannot handle time and continuous parameters. Furthermore, each practical application has its unique criteria that is not necessarily identical to edit distance.
Given all these limitations, we chose edit distance because it is concrete and has applications se- quence search in NLP and Bioinformatics. The best algorithms for Exact search based on edit dis- tance requires O(L²n) time complexity where L is the length of log sequences and n is the number of log windows in the database.
Computing euclidean distance to examples in log embedding space has complexity of (dm), where d is the dimensionality of vector embedding. This complexity is more favorable than that of

---

Under review as a conference paper at ICLR 2021
query log window
mean Average Precision for log search
KPRk) CTCClle Kok TRPLAPE.0B0Tn U Lo00 1000rio0000 Wotooooroooooooooooo
0.6
Top-5 matches based on our embedding
mean Average Precision
8.51
0.53
0.54
0.5
(CLK6PPS1KTCCEI ClINkKKTPPCPPL.C 0:t.::0000000000000000
0.41
0:36
0.39
JTCI IKTCIIKaKTCRPP (06.6//.0E0EE0006. BunnEE Eu00EE.0SEE 008.6.06SE.060000 D00.E/000000e000
RICIKoRP3 CnLKcKTCRRPPP L0000s000000 1080000000 0000:
0.25
STCT KcRTCRR
0.2
Top-5 matches based on edit distance
TCIRoRP] BETTCCTTTPPCPPPPE www.BennGGURRR.BBBEEE B uoRAnnnE ud
0.1
STCIKoRP RRTCIKOKTCRRPRR D wOORREEeen B.0r d066E-00
TTCC TTCINOKTRP BEU.KKKRRRRPBBUUUC... GRRGEE BeB.aa-EuBegguieeee EudBRguOARSS B-RRRE www.g.neegg................ www.egooRRRE .0
TCIKoRKP (TCIIOKTTTRRRRRTT Lu080niE00800 000
top
10
100
too
1000
top
top
10
top
100
1000
top
top 10
100
top
1000
top
STCIRORPJTTTT HRTCCTC WDB.RRRPEERRRUUUU....
window=16
window=32
window=64
Figure 6: Left: A sample query window, top-5 matches based on our embedding and top-5 matches based on edit distance. We have replaced each log template with unique letter and a unique color. Each line represents a sequence of length 64. Note that our top matches are more sensitive to displacement than matches from edit distance. Right: The performance of our embedding- search, assuming that edit distance ground truth. Here top-k identifies how many top edit distance matches are considered positive out of 100,000 sample log windows. We have experimented with three values of & and three log window sizes. Note that even though our embedding is not trained to mimic edit distance, it gives useful approximation to edit distance.
edit distance. Moreover, euclidean distance is embarrassingly parallelizable and can be sped up on GPUs. However, edit distance algorithms are sequential and do not speed up as well on GPUs. In evaluation experiment with accuracy, we retrieve five time the number of query points and pick the closest one fifth.
KD-tree to speed up nearest-neighhour search. To trade-off speed
We experiment on dataset of 100,000 log windows. We tried log windows of length 64, 32, and 16. Given a query sequence, we take the top 10, 100 and 1000 closest sequences (based on edit distance) as positive examples and the rest negative examples (in three different experiments). For each experiments, we we calculate average precision for 100 queries and report the mean Average Precision. Figure (oaneeollllnnee illustrates experiments.
5.7 DIAGNOSIS AND RESOLUTION RECOMMENDER SYSTEM
Many IT services providers, serve thousands of customers. These companies have a ticketing system where customers can raise their issues. These ticketing systems have history of diagnosis and solutions to each issue. Given a ticket, recommender system can search for the most similar issues in log history and present statistic of what actions solved similar tickets.
5.8 LOG SYNTHESIS AND TEST GENERATION
Auto-regressive transformer models such as GPT-3 have been very successful natural language generation (Brown al. (2 Since TNs can handle complex relationships between tokens, they are suitable for log synthesis and test generation as well as language generation. Given a sequence of logs, a log language model can predict probability distribution over the next log entry and sample from the distribution. This process can generate long and diverse log sequences. This process can be used to generate novel test cases for software testing.
CONCLUSION
We have presented a unified framework for carrying out several log diagnostic tasks using a pipeline of layered vector embeddings obtained by training the masked language task using the transformer architecture. Several unique aspects of log data, specifically time stamps, are incorporated into the transformer architecture. Armed with vector representations we have applied them to several down- stream tasks including anomaly detection, log search and root cause analysis. We have applied our results real data set obtained from leading telecommunications vendor. Our results strongly indicate that computer system diagnostics can be greatly enhanced using embeddings derived from from modern sequence to sequence models.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Titus Barik, Robert DeLine, Steven Drucker, and Danyel Fisher. The bones the system: A case study of logging and telemetry at microsoft. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 92-101. IEEE, 2016.
Carolin E Brandt, Annibale Panichella, Andy Zaidman, and Moritz Beller. Logchunks: A data set for build log analysis. In Proceedings of the 17th International Conference on Mining Software Repositories, pp- 583-587, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Nimrod Busany and Shahar Maoz. Behavioral log analysis with statistical guarantees. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 877-887. IEEE, 2016.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: survey. arXiv preprint arXiv:1901.000 003407, 2019.
Jordi Cucurull and Jordi Puiggalí. Distributed immutabilization of secure logs. In International Workshop on Security and Trust Management, pp. 122-137. Springer, 2016.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18 2018.
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1885-129,, 2017.
Min Du, Zhi Chen, Chang Liu, Rajvardhan Oak, and Dawn Song. Lifelong anomaly detection through unlearning. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1283-1297, 2019a.
Min Du, Ruoxi Jia, and Dawn Song. Robust anomaly detection and backdoor attack detection via differential privacy. In International Conference on Learning Representations, 2019b.
Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: journal of the Econometric Society, pp. 424-4344 1969.
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural networks, 18(5-6):602-611 2005.
Pinjia He, Jieming Zhu, Shilin He, Jian Li, and Michael Lyu. An evaluation study on log parsing and its use in log mining. In 2016 46th Annual IEEE/IFIP International Conference on Depend- able Systems and Networks (DSN), pp. 654-661. IEEE, 2016
Pinjia He, Jieming Zhu, Shilin He, Jian Li, and Michael R Lyu. Towards automated log parsing for large-scale log data analysis. IEEE Transactions on Dependable and Secure Computing, 15(6): 931-944, 2017.
Jian-Guang Lou, Qiang Fu, Shengqi Yang, Ye Xu, and Jiang Li. Mining invariants from console logs for system problem detection. In USENIX Annual Technical Conference, pp. 1-14, 2010.
Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et al. Loganomaly: Unsupervised detection of sequential and quan- titative anomalies unstructured logs. In IJCAI, pp. 4739-4745, 2019.
Chandrasekaran Mohan, Don Haderle, Bruce Lindsay, Hamid Pirahesh, and Peter Schwarz. Aries: transaction recovery method supporting ane-graaularrty locking and partial rollbacks using write- ahead logging, ACM Transactions Database Systems (TODS), 17(1)99-116, 1992.

---

Under review as a conference paper at ICLR 2021
Sasho Nedelkoski, Jasmin Bogatinovski, Alexander Acker, Jorge Cardoso, and Odej Kao. Self- attentive classification-base anomaly detection in unstructured logs. In ICDM, 2020.
Doyen Sahoo, Quang Pham, Jing Lu, and Steven CH Hoi. Online deep learning: learning deep neural networks on the fly. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2660-2666, 2018.
Andrew Sutton and Reza Samavi. Blockchain enabled privacy audit logs. In International Semantic Web Conference, pp. 645-660. Springer, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yali Yuan, Sripriya Srikant Adhatarao, Mingkai Lin, Yachao Yuan, Zheli Liu, and Xiaoming Fu. Ada: Adaptive deep log anomaly detector. In IEEE INFOCOM 2020-IEEE Conference Com- puter Communications, pp. 2449-24444 2458. IEEE, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391- 3401, 2017.
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu. Tools and benchmarks for automated log parsing. In 2019 IEEE/ACM 41 International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 121-130. IEEE, 2019.
10


Reviews:
[{"review_id": "acg4qdMwZTg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-level abstraction for representing logs that appear in a wide range of application domains and a Transformer-based model for embedding those representations in a vector space. The authors show that a variety of log processing tasks (anomaly detection, predictive analysis, search, etc.) can be implemented on top of this foundation along with empirical results for some of them based on two real-world log datasets.\n\nStrong points:\n- The paper is well written. It nicely educates the reader about the prior state of the art, motivates how the contributions of the work fit in, and then presents those contributions in a systematic manner with empirical evidence where applicable.\n- Log processing is a widespread application with increasingly large and diverse datasets and need for a range of automated data analysis tasks over them. The multi-level abstraction to provide a common representation for this domain as a whole is a neat idea that can help organize existing knowledge into a common framework as well as facilitating future innovation based on that framework.\n- The paper shows how transformer networks can be extended with the notion of time, which can be useful for time-oriented data in general, including logs.\n- The proposed models are implemented in practice as part of a transformer library in PyTorch and are applied to a variety of log processing tasks with good results.\n\nWeak points:\n- The levels of abstraction could have been connected more clearly with the transformer model and the log processing applications in the experimental part. The mapping between these parts is not as easy to understand as it could have been based on the current writing.\n- It would be good to add a figure to Section 3 to illustrate the various abstraction levels and how they are related (e.g., in the form of a pipeline together with an example log entry representation for each).\n\nAdditional comments:\n- Releasing the telecommunications dataset would be a good contribution to the research community.\n- Section 4.1, \"Given log sequence <(l_1, t_1), ..\": Please make sure to use consistent terminology. \"log sequence\" is defined slightly differently in Section 3.\n- Figure 4: In the caption, it looks like \"Middle\" and \"Right\" should be in the opposite order. Also, the colors in the third graph are not easily readable.\n- Typos:\nlearnt -> learned\nanomaies -> anomalies\nuseful applications -> useful (for?) applications\noh HDFS dataset -> on HDFS dataset\npytorch -> PyTorch\nfigure x -> Figure x\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_CMq-C2pNgH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with log data representation and analysis. Logs are important to understand the status of a system and do various root cause analysis. This paper proposes a transformer based approach to obtain vector representation of log data, in various levels such as for key-value pair with a log entry, a log entry within a sequence of logs and various blocks of log sequences. The usefulness of log embeddings are shown on multiple log downstream tasks.\n\nThe topic of the paper is very interesting and relatively less studied in machine learning domain, though important for various applications. However, it has several drawbacks, as follows. \n\n1. The technical contribution of the paper is very limited. The transformer based approach is not novel. The specific format of time encoding in Section 4.1 is not well motivated.\n\n2. In Section 3, is it reasonable to consider a log entry as a sequence of characters, instead of sequence of words? Most of the log entries consist of a set of key words (presented in a readable format) and some arguments (can be numeric).\n\n3. One key difficulty in handling log data is the presence of large number of arguments (or parameters) within each log entry. Thus, it becomes difficult to mine logs to different templates. The exact parameter values can be very different in different logs. So, applying text modeling approaches directly on log data may not be the best approach.\n\n4. In Section 3, What is the granularity used for sequence embedding in logs? Is it for a whole log file, or a block of logs? How to determine the appropriate granularity for some downstream application?\n\n5. Fir unsupervised anomaly detection, it is not clear why anomalous points would fall into a small cluster. Rather, they would probably be distributed over multiple clusters, but still will be far away from the respective cluster center. The paper could have also used some standard anomaly detection algorithm such as Isolation Forest once the vector embeddings are generated.\n\n6. For supervised anomaly detection, what is the % of anomalous logs used in the training?\n\n7. Being an application-oriented paper, more importance should be given on hyperparameter setup and tuning for all the experiments. The lack of availability of source code also makes the reproducibility of the results difficult.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TGFkZqmtEos", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to represent system logs at five levels of abstraction (including log sequence, parsed log, field embedding, log embedding, and sequence embedding). The representation at each level can be computed from the previous level. Transformer Network is utilized for time encoding. The paper also describes various log-related applications based on the proposed log representation. Some experiments were conducted to evaluate the proposed approach. \n\nLogs are useful for understanding and diagnosing software intensive systems. It is good to see that this paper proposes a new neural representation of log data. The authors also suggested various applications of the proposed log representation.\n\nIn section 4, the authors only described the proposed time encoding technique, while other parts of the log representation (such as encoding a log entry) were not described. Also, it is not clear if the proposed time encoding technique is better than the related methods (there are many related methods for encoding/representing time). \n\nThe evaluation of the proposed approach is very weak. In section 5, the authors mentioned the Radio datasets. However, the use of Radio dataset is not described. The authors only evaluated the anomaly detection model on the HDFS log dataset, which is not enough. Also, the obtained results on HDFS were not very different from the results of the related work (DeepLog). Furthermore, no experiments were conducted for the causal analysis task. Therefore, the effectiveness and generalizability of the proposed approach are not clear. \n\nThe paper only compared with a few related methods for log-related tasks. Actually, this area has been widely studied and there are a lot more research work (some also utilized deep learning and language models). The authors could discuss and compare with them. Just a few examples:\nZhang et al., Robust log-based anomaly detection on unstable log data. In Proc. ESEC/FSE 2019, 807-817.\nZhu et al., Learning to log: Helping developers make informed logging decisions, in Proc. ICSE 2015. pp. 415–425.\nP. He et al., “Characterizing the natural language descriptions in software logging statements,”  in Proc. ASE 2018, pp. 178–189.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hj3372pBirk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nLogs are widely used in computer systems to record their events. The recorded logs can be applied to a wide variety of diagnostic applications such as anomaly detection, root cause analysis, and causal analysis. This paper proposes levels of abstraction for log representation. There are in total 5 levels of abstraction, which are log sequences, parsed logs, field embeddings, log embeddings, and sequence embeddings. Each of the aforementioned levels can be derived from its proceeding levels. The paper uses the transformer model to generate a log representation. Moreover, the authors propose a time encoding method and add the encoding time to the transformer to improve the representation ability of the proposed log representation method. In the evaluation section, the authors apply the generated log representation to various downstream applications to test the effectiveness of the proposed method.\n\n**Strengths**:\n1. The idea of using different levels of abstraction to generate log embedding is interesting.\n2. The paper conducts a thorough experiment over various downstream tasks based on the learned log representation.\n\n**Weaknesses**:\n1. The motivation for adding time encoding to the transformer is not clear. It would be better if the intuition behind this variation could be discussed rather than simply using the result of the ablation study.\n2. It seems that the paper just applies the Transformer model to the log representation task and there is no comparison between the proposed method and other log representation methods in the evaluation part. In the related work section, the authors mentioned a recently proposed log embedding work (Logsy). It would be better if there is a comparison between the two different methods in the evaluation section.\n3. There is little discussion about the results of the experiments. It would be better if some discussion of why the use of the proposed representation can or cannot improve the performance could be mentioned at the end of each experiment.\n\n**Minor Weaknesses**:\nThe presentation of the paper could be improved. \n1. The figures for the experiment results are too far away from the section that describes and discusses the experiment.\n2. The “oh” in the caption of figure 3 should be “on”.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most of the reviewers had concerns on the model being considered and there are additional concerns in the paper's discussion on the experimental results. 
Decision: Reject