Log File Anomaly Detection

Analysis of log files pertaining to a failed run can be a tedious task, especially if the file runs into thousands of lines. Using the recent development in text analysis using deep neural networks, we present a method to reduce effort needed to analyze the log file by highlighting the most probably useful text in the failed log file, which can assist in debugging the causes of the failure. In essence we reduce the log file by removing the lines which are found to be of less importance to the debug. We measure the accuracy of our reduction by F1 score, as well as custom scoring method on manually labeled data, where important lines from a ‘failed’ log are extracted by the experts of the domain. 1 Introduction Anomaly detection in log file deals with finding text which can provide clues to the reasons and the anatomy of failure of a run. Most commonly, domain and tool specific regular-expression from previously seen error messages is used to dig the relevant text from the log file. However such regular expressions need persistent manual management to ensure the correctness and relevance. Also newer messages of failures are easily missed with this method. Conversely, a black-list regular-expression file, which contains regular expressions for all the un-harmful ‘text’ is also sometime maintained. However such list can grow huge fast, in our domain of vlsi-design, to thousands of lines. This makes further maintenance and execution practically prohibitive. Ahmed [1] used grammar inference to find anomalous text. Here a representative grammar is constructed out of training set, and compared against the test set to figure text which doesn’t conform to the learned grammar. Similar to above, our hypothesis on log file anomaly detection relies on the fact that any text found in a ‘failed’ log file, which looks very similar to the text found in ‘successful’ log file can be ignored for debugging of the failed run. Anomaly detection is trying to find ‘salient’ or ‘unique’ text previously unseen. In this work, we propose to use neural network language model learned from ‘successful’ log files, to predict the anomalies in a ‘failed’ run, there by attempting to reduce the size of the log file which needs manual review. With no domain knowledge or tokenization, we claim to get upto 85% log size reduction, while keeping more than 77% of useful information. The reduced log file contains most of the text relevant to the failure of the run, with additional text which is considered false positive. The text which is useful for debug, but missing from reduced log file is considered as false negative. We propose to use F1 score as measure of accuracy of the reduction. 2 Model description Language models for prediction of next word based on contextual statistical data has been in existence for many years with incremental improvements of n-gram models [2]. More recently, “Recurrent Neural Network [RNN]” based statistical model [4,5] have been able to provide better language model and more forgiveness for non-exact (n-1) phrase in an n-gram. Further, invent of character vector based RNN-language model [6,7] reduces the vocab size and allows better learning from text which has high variability tokens [such as date-stamp and runtime metrics]. We experimented with word vector based RNN language [word-RNNLM] model as well as char vector based language RNN model [char-RNNLM] as described in the following section. 2 . 1 Wo r d Ve c t o r b a s e d L a n g u a g e m o d e l [ w o r d R N N L M ] We used 1 hidden layer recurrent neural network for training. Figure 1: Structure of one hidden layer RNN used in word vector based language model Figure 1 shows the structure of word vector based language model. Here x(#) is the input word vector (of dimension ) at t timestamp, h(#) is the hidden layer (of dimension ), y(#) is the predicted probability of next word at t timestamp: h t = sigmoid h t H + x t I + b1 . (1) y(t) = softmax h t U + b2 . (2) Cross entropy loss function was used for training: J θ = CE y t , y t = − yi t log yi t V