LOG ANOMALY DETECTION

One or more computer processors classify each log line in a plurality of unlabeled log lines as an erroneous log line or a non-erroneous log line; templatize each classified erroneous log line and non-erroneous log line in the plurality of unlabeled log lines; cluster erroneous log templates into erroneous log template clusters and non-erroneous log templates into non-erroneous log template clusters; identify one or more log lines as anomalous utilizing a plurality of factors including a log maturity, a number of encountered log template clusters, and a ratio of classified erroneous log lines to classified non-erroneous log lines; responsive to one or more identified anomalous log lines, validate the identified anomalous log lines utilizing a site reliability engineer and human-in-the-loop validation; train a log anomaly model utilizing one or more validated log lines; and identify a subsequent log line as anomalous utilizing the trained log anomaly model.

The following disclosure(s) are submitted under 35 U.S.C. 102(b)(1)(A):

IBM Canada Software Announcement: IBM Cloud Pak for Watson AIOps 3.1 helps SREs and IT teams to maintain a high availability of applications and helps remediate and resolve incidents through automation; Sahil Bansal, Harshit Kumar, Lu An, Xiaotong Liu, and Anbang Xu; Mar. 18, 2021.

BACKGROUND

The present invention relates generally to the field of machine learning, and more particularly to anomaly detection.

Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. Machine learning is seen as a subset of artificial intelligence. Machine learning algorithms build a mathematical model based on sample data, known as training data, in order to make predictions or decisions without being explicitly programmed to perform the task. Machine learning algorithms are used in a wide variety of applications, such as email filtering and computer vision, where it is difficult or infeasible to develop a conventional algorithm for effectively performing the task.

Anomaly detection is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. Typically, the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are also referred to as outliers, novelties, noise, deviations, and exceptions.

SUMMARY

Embodiments of the present invention disclose a computer-implemented method, a computer program product, and a system. The computer-implemented method includes one or more computer processers classifying each log line in a plurality of unlabeled log lines as an erroneous log line or a non-erroneous log line utilizing a dictionary based classifier within a hybrid error classifier. The one or more computer processors templatize each classified erroneous log line and non-erroneous log line in the plurality of unlabeled log lines. The one or more computer processors cluster erroneous log templates into erroneous log template clusters and non-erroneous log templates into non-erroneous log template clusters. The one or more computer processors identify one or more log lines as anomalous utilizing a plurality of factors including a log maturity, a number of encountered log template clusters, and a ratio of classified erroneous log lines to classified non-erroneous log lines. The one or more computer processors, responsive to one or more identified anomalous log lines, validate the identified anomalous log lines utilizing a site reliability engineer and human-in-the-loop validation. The one or more computer processors train a log anomaly model within the hybrid error classifier utilizing one or more validated log lines. The one or more computer processors identify a subsequent log line as anomalous utilizing the trained log anomaly model.

DETAILED DESCRIPTION

Artificial intelligence in information technology operations (AIOps) has an essential role in modern, data driven organizations and operations. AIOps utilize big data to aggregate siloed information technology operations data allowing efficient digital transformation (e.g., multiple environments, virtualized resources, dynamic infrastructure), cloud migration, and developmental operations (e.g., provisioning and reconfiguring infrastructure). Anomaly pipelines play an important role within AIOps, where anomaly pipelines detect data anomalies within an AIOps data pipeline. Anomaly pipelines monitor integrated systems and components within an AIOps system, collect information associated with detected anomalies, and utilize the collected information to mitigate the anomaly, such as fault localization supporting root cause analysis. Anomaly pipelines underpin many AIOps systems, thus highly accurate anomaly detection is a requirement for efficient and effective AIOps. For example, false positives in anomaly detection streams errors to the rest of the pipeline causing a reduction in operational efficiency and increased computational wastage. Currently, modern anomaly pipelines require substantial amounts (e.g., thousands of labelled log lines) of log data (i.e., training data) corresponding to a healthy state of monitored system, data pipeline, or component. These modern anomaly pipelines are particularly inefficient and ineffective in “0-day” scenarios occurring as the pipeline is initially deployed and training data is scarce. Obtaining the required training data can be prohibitively expensive for many organizations due to the state of monitored system (e.g., mixture of healthy and unhealthy logs, log velocity, computational constraints, etc.). Anomaly pipelines trained with ineffective training data produce increased levels of false positives. There is an increasing need for an anomaly detection system that efficiently detects point-in-time anomalies in initial deployment situations (i.e., “0-day”).

Embodiments of the present invention provide a system to detect point-in-time anomalies in initial deployment situations with a reduction in anomalous false positives. Embodiments of the present invention reduce false positives by bootstrapping an internal dictionary from associated documentation while incrementally training a machine learning detector. Embodiments of the present invention improve newly deployed log anomaly detection systems through incremental system updates using clustering of templatized log lines to identify anomalous log lines. Embodiments of the present invention reduce the required amount of training data (i.e., logs) for effective anomaly detection. Embodiments of the present invention correlate anomalous events with other event data across environments to identify the cause of an outage or performance problem and suggest or implement remedies. Implementation of embodiments of the invention may take a variety of forms, and exemplary implementation details are discussed subsequently with reference to the Figures.

FIG.1is a functional block diagram illustrating a computational environment, generally designated100, in accordance with one embodiment of the present invention. The term “computational” as used in this specification describes a computer system that includes multiple, physically, distinct devices that operate together as a single computer system.FIG.1provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by those skilled in the art without departing from the scope of the invention as recited by the claims.

Computational environment100includes server computer120connected over network102. Network102can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN), such as the Internet, or a combination of the three, and can include wired, wireless, or fiber optic connections. Network102can include one or more wired and/or wireless networks that are capable of receiving and transmitting data, voice, and/or video signals, including multimedia signals that include voice, data, and video information. In general, network102can be any combination of connections and protocols that will support communications between server computer120, and other computing devices (not shown) within computational environment100. In various embodiments, network102operates locally via wired, wireless, or optical connections and can be any combination of connections and protocols (e.g., personal area network (PAN), near field communication (NFC), laser, infrared, ultrasonic, etc.).

Server computer120can be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, server computer120can represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment. In another embodiment, server computer120can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with other computing devices (not shown) within computational environment100via network102. In another embodiment, server computer120represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single pool of seamless resources when accessed within computational environment100. In the depicted embodiment, server computer120includes log122and program150. In other embodiments, server computer120may contain other applications, databases, programs, etc. which have not been depicted in computational environment100. Server computer120may include internal and external hardware components, as depicted and described in further detail with respect toFIG.3.

Log122is a repository for data used by program150. In the depicted embodiment, log122resides on server computer120. In another embodiment, log122may reside elsewhere within computational environment100provided program150has access to log122. A database is an organized collection of data. Log122can be implemented with any type of storage device capable of storing data and configuration files that can be accessed and utilized by program150, such as a database server, a hard disk drive, or a flash memory. In an embodiment, log122contains a plurality of log line124. Log line124is an unlabeled data point derived from historical performance and event data, streaming real-time operations events, system logs and metrics, network data (i.e., packet data), incident-related data and ticketing, and related document-based data. In an embodiment, log line (LL)124contains an event in a series of events that occurred in a monitored system or component, where each event is associated with a timestamp and a description describing the nature of the event. In a further embodiment, each event includes severity level, thread ID, ID of the originating component, host name, etc.

Program150is a program for evolutionary log anomaly detection. In various embodiments, program150may implement the following steps: classify each log line in a plurality of unlabeled log lines as an erroneous log line or a non-erroneous log line utilizing a dictionary based classifier within a hybrid error classifier; templatize each classified erroneous log line and non-erroneous log line in the plurality of unlabeled log lines; cluster erroneous log templates into erroneous log template clusters and non-erroneous log templates into non-erroneous log template clusters; identify one or more log lines as anomalous utilizing a plurality of factors including a log maturity, a number of encountered log template clusters, and a ratio of classified erroneous log lines to classified non-erroneous log lines; responsive to one or more identified anomalous log lines, validate the identified anomalous log lines utilizing a site reliability engineer and human-in-the-loop validation; train a log anomaly model within the hybrid error classifier utilizing one or more validated log lines; and identify a subsequent log line as anomalous utilizing the trained log anomaly model. In the depicted embodiment, program150is a standalone software program. In another embodiment, the functionality of program150, or any combination programs thereof, may be integrated into a single software program. In some embodiments, program150may be located on separate computing devices (not depicted) but can still communicate over network102. In various embodiments, client versions of program150resides on any other computing device (not depicted) within computational environment100. In the depicted embodiment, program150includes hybrid error classifier152. Program150is depicted and described in further detail with respect toFIG.2.

Hybrid error classifier152is representative of a dual classifier/model for determining whether a log line124is an erroneous log line (ELL) or a non-erroneous log line (NELL). In an embodiment, hybrid error classifier152initially utilizes a dictionary or list (i.e., classifier component) of symptom-words to identify the erroneous log lines. In this embodiment, program150bootstraps the dictionary utilizing a product and software documentation. In another embodiment, hybrid error classifier152is a NOI classifier that classifies each log line to one of the following categories: “Information”, “Unknown”, “Latency”, “Saturation”, “Exception”, “State Change”, etc. In another embodiment, as program150matures and continues to see novel logs, hybrid error classifier152, responsively, utilizes learning techniques to train (i.e., model component), calculate weights, ingest inputs, and output a classification. In an embodiment, hybrid error classifier152is comprised of any combination of deep learning model, technique, and algorithm (e.g., decision trees, Naive Bayes classification, support vector machines for classification problems, random forest for classification and regression, linear regression, least squares regression, logistic regression). In an embodiment, hybrid error classifier152utilizes transferrable neural networks algorithms and models (e.g., long short-term memory (LSTM), deep stacking network (DSN), deep belief network (DBN), convolutional neural networks (CNN), compound hierarchical deep models, etc.) that can be trained with supervised or unsupervised methods. The training of error classifier152is depicted and described in further detail with respect toFIG.2.

The present invention may contain various accessible data sources, such as log122, that may include personal storage devices, data, content, or information the user wishes not to be processed. Processing refers to any, automated or unautomated, operation or set of operations such as collection, recording, organization, structuring, storage, adaptation, alteration, retrieval, consultation, use, disclosure by transmission, dissemination, or otherwise making available, combination, restriction, erasure, or destruction performed on personal data. Program150provides informed consent, with notice of the collection of personal data, allowing the user to opt in or opt out of processing personal data. Consent can take several forms. Opt-in consent can impose on the user to take an affirmative action before the personal data is processed. Alternatively, opt-out consent can impose on the user to take an affirmative action to prevent the processing of personal data before the data is processed. Program150enables the authorized and secure processing of user information, such as tracking information, as well as personal data, such as personally identifying information or sensitive personal information. Program150provides information regarding the personal data and the nature (e.g., type, scope, purpose, duration, etc.) of the processing. Program150provides the user with copies of stored personal data. Program150allows the correction or completion of incorrect or incomplete personal data. Program150allows the immediate deletion of personal data.

FIG.2depicts flowchart200illustrating operational steps of program150for evolutionary log anomaly detection, in accordance with an embodiment of the present invention.

Program150creates a dictionary of invariants and parameters from a log (step202). In an embodiment, program150initiates responsive to a collection of unlabeled data (e.g., log data). In an embodiment, program150bootstraps the dictionary of invariants and parameters from product and software documentation, either provided by developers or available in the public domain. In this embodiment, documentation includes standard textual documents such as user manuals, incident descriptions or specialized lists of product offerings. In a further embodiment, program150extracts invariants and parameters from documentation utilizing entity extraction. For example, the incident descriptions provide error codes while, specialized lists of product offerings are utilized to populate the dictionary. In an embodiment, program150extracts a list of invariants and parameters based on the frequency distribution of an initial set of logs. In another embodiment, program150preserves identified invariants while replacing identified parameters with <P> tokens. In an embodiment, program150utilizes the bootstrapped dictionary for log line templatization as described in step206.

Program150templatizes the log from the created dictionary (step206). Program150respectively templatizes the identified erroneous and non-erroneous log lines through invariant and parameter log line identification and tokenization. In an embodiment, program150templatizes each identified erroneous and non-erroneous log line utilizing a dictionary of bootstrapped invariants and parameters over a temporal period converging onto template subsets representing the entirety of the identified log lines. In another embodiment, program150utilizes a template miner, such as AQL-rules, to create a plurality of log templates from the identified erroneous and non-erroneous log lines. In an embodiment, program150utilizes the template miner to extract patterns from log122with the following steps: cleaning log lines; checking for matching log templates using a fixed depth tree; responsive to one or more matched log templates, identifying a similar log within the one or more matched log templates; and responsive to no matched log templates, creating a new log template.

Program150clusters templatized log lines (step208). Program150clusters each log template or templatized log lines into one or more erroneous log template clusters (ELTCs) or non-erroneous log template clusters (non-ELTC). In an embodiment, program150utilizes fuzzy clustering techniques to cluster the log templates. In this embodiment, program150computes a lexical similarity between a log template and an existing log template cluster, where the lexical similarity is calculated from an aggregated value of all templates within the existing log template cluster. In another embodiment, program150utilizes curriculum clustering to cluster the templates. In yet another embodiment, program150utilizes hierarchical clustering without embedding. In this embodiment, program150first extracts entity, action and symptom words from log templates; then program150prepares a sequence of words for the log templates using the extracted words while preserving word order; lastly program150computes a jaro distance between sentences while using the jaro distance to perform the hierarchical clustering. In an embodiment, program150checks if a template is substantially similar (e.g., exceeds a jaro distance threshold or lexical similarity) to any of the existing template clusters. Responsive to a similar template, program150updates the frequency distribution and counts of the template and associated template clusters, thus keeping the existing template clusters updated.

Program150adjusts template clusters based on maturity frequency thresholding (step210). Responsive to one or more templates clusters, program150calculates a frequency distribution comprising the distributions for each template cluster. In an embodiment, program150removes highly frequent (e.g., template clusters containing more than 40% of the templates, exceed frequency threshold, etc.) template clusters while maintaining the remaining template clusters. Responsive to the cluster reduction, program150calculates another frequency distribution with the remaining template clusters. With this frequency distribution, program150computes a frequency threshold for the template clusters, where the frequency threshold is an exponentially increasing function that is updated based on the maturity (i.e., as the system ages, the threshold is increased) of a monitored system or component. In an embodiment, program150initially sets the frequency threshold to the median, but any point between the first and third quartile of the frequency distribution is suitable. In this embodiment, program150sets the rate of increase for the frequency threshold based on dataset considerations, precision or recall system requirements.

Program150identifies log anomalies (step212). In an embodiment, program150identifies any log template cluster (non-erroneous or erroneous), and all included log lines, with a frequency count less than the frequency threshold (e.g., median or third quartile) as anomalous. In another embodiment, responsive to a plurality of factors such as maturity level (e.g., log quantity, etc.), number of encountered template clusters by the system, and ratio of erroneous log lines compared to non-erroneous log lines, program150identifies said logs and clusters as anomalous. For example, at an early stage (e.g., a week of unlabeled collected logs, low ratio (e.g.,4to1) of non-erroneous to erroneous, and few (e.g., less than 4) encountered template clusters, etc.) and responsive to a plurality of identified non-erroneous logs or templates program150labels each log or template cluster as non-anomalous or healthy.

Responsive to a subsequent log line (e.g., novel log line or log line absent from initial logs (i.e., training data)) matching into an anomalous erroneous or non-erroneous template cluster, program150identifies the subsequent log line as anomalous. Responsive to the subsequent log line (i.e., erroneous log line) absent (i.e., failing to exceed a lexical similarity threshold) from the erroneous log template clusters, program150identifies the subsequent log line as an anomaly, templatizes the subsequent log line, creates a new erroneous log template cluster with the templatized subsequent log line, and sets the frequency count of the new erroneous log template cluster to one. Any future occurrence of similar log lines (i.e., within the new erroneous log template cluster) are classified as anomalous until a frequency threshold (e.g., third quartile) is reached or exceeded. Responsive to the subsequent log line (i.e., non-erroneous log line) absent from the non-erroneous log template clusters, program150identifies the subsequent log line as a non-anomaly, templatizes the subsequent log line, creates a new non-erroneous log template cluster with the templatized subsequent log line, and sets the frequency count of the new non-erroneous log template cluster to one. Any future occurrence of similar log lines (i.e., within the new non-erroneous log template cluster) are classified as non-anomalous until a frequency threshold (e.g., first quartile) and a timestamp threshold (i.e., abs[log line timestamp—threshold]) are reached or exceeded. In an embodiment, program150correlates anomalous events (i.e., log lines) with other events across environments (e.g., system, logs, versions, locations, etc.) to identify the cause of the anomalous event (e.g., outage, performance variation, or computational disparity) and suggest or implement remedies. For example, program150adjusts computational resources to a monitored system continuing to output anomalous log lines. In this embodiment, program150presents suggestions or remedies to a user.

Program150validates identified log anomalies (step214). Responsive to one or more identified anomalous log lines or anomalous template clusters, program150validates said anomalies. In an embodiment, program150involves human-in-the-loop techniques for validation and utilizes returned anomaly validation for semi-supervised training of hybrid error classifier152. For example, program150incorporates suggestions and observations (i.e., validation) from a site reliability engineer (SRE) as program150presents the SRE with the identified anomalies. This embodiment reduces the percentage of false anomalies in earlier stages of an anomaly detection pipeline. In an embodiment, SRE validation is triggered responsive to an incorrect prediction, identification, or cluster from one or more component classifiers and models within hybrid error classifier152. In a further embodiment, program150automatically corrects non-anomalous, anomalous log lines, and template clusters identified as incorrect by SRE validation. In this embodiment, program150responsively adjusts template clusters and reclassifies comprising log lines according to SRE recommendations. In another embodiment, program150incrementally updates hybrid error classifier152by training a component model (e.g., neural network as opposed to the initially utilized dictionary based component) with the frequency threshold reduced log template clusters to identify anomalous non-erroneous lines, anomalous non-erroneous clusters, anomalous erroneous log lines, and anomalous erroneous log clusters within subsequent log lines.

FIG.3depicts block diagram300illustrating components of server computer120in accordance with an illustrative embodiment of the present invention. It should be appreciated thatFIG.3provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

Server computer120each include communications fabric304, which provides communications between cache303, memory302, persistent storage305, communications unit307, and input/output (I/O) interface(s)306. Communications fabric304can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications, and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric304can be implemented with one or more buses or a crossbar switch.

Memory302and persistent storage305are computer readable storage media. In this embodiment, memory302includes random access memory (RAM). In general, memory302can include any suitable volatile or non-volatile computer readable storage media. Cache303is a fast memory that enhances the performance of computer processor(s)301by holding recently accessed data, and data near accessed data, from memory302.

Program150may be stored in persistent storage305and in memory302for execution by one or more of the respective computer processor(s)301via cache303. In an embodiment, persistent storage305includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage305can include a solid-state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

The media used by persistent storage305may also be removable. For example, a removable hard drive may be used for persistent storage305. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer readable storage medium that is also part of persistent storage305. Software and data312can be stored in persistent storage305for access and/or execution by one or more of the respective processors301via cache303.

Communications unit307, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit307includes one or more network interface cards. Communications unit307may provide communications through the use of either or both physical and wireless communications links. Program150may be downloaded to persistent storage305through communications unit307.

I/O interface(s)306allows for input and output of data with other devices that may be connected to server computer120. For example, I/O interface(s)306may provide a connection to external device(s)308, such as a keyboard, a keypad, a touch screen, and/or some other suitable input device. External devices308can also include portable computer readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention, e.g., program150, can be stored on such portable computer readable storage media and can be loaded onto persistent storage305via I/O interface(s)306. I/O interface(s)306also connect to a display309.