Interactive detection of system anomalies

Interactive detection of system anomalies is disclosed. One example is a system including a data processor, an anomaly processor, and an interaction processor. Input data related to a series of events and telemetry measurements is received by the data processor. The anomaly processor detects presence of a system anomaly in the input data, the system anomaly indicative of a rare situation that is distant from a norm of a distribution based on the series of events and telemetry measurements. The interaction processor is communicatively linked to the anomaly processor and to an interactive graphical user interface. The interaction processor displays, via the interactive graphical user interface, an output data stream based on the presence of the system anomaly, receives, from the interactive graphical user interface, feedback data associated with the output data stream, and provides the feedback data to the anomaly processor for operations analytics based on the feedback data.

BACKGROUND

Operations analytics are routinely performed on operations data. Operations analytics may include management of complex systems, infrastructure and devices. Complex and distributed data systems are monitored at regular intervals to maximize their performance, and detected anomalies are utilized to quickly resolve problems. In operations related to information technology, data analytics are used to understand log messages, and search for patterns and trends in telemetry signals that may have semantic operational meanings.

DETAILED DESCRIPTION

Operational analytics relates to analysis of operations data, related to, for example, events, logs, and so forth. Various performance metrics may be generated by the operational analytics, and operations management may be performed based on such performance metrics. Operations analytics is vastly important and spans management of complex systems, infrastructure and devices. It is also interesting because relevant analytics are generally limited to anomaly detection and pattern detection. The anomalies are generally related to operations insight, and patterns are indicative of underlying semantic processes that may serve as potential sources of significant semantic anomalies. Generally, analytics is used in IT operations (“ITO”) for understanding unstructured log messages and for detecting patterns and trends in telemetry signals that may have semantic operational meanings. Many ITO analytic platforms focus on data collection and transformation, and on analytic execution.

However, operational analytics are generally query-based. For example, a domain expert, such as a system engineer, may query input data to extract and analyze data related to an aspect of system operations. In many situations, relevant data may be normalized and readily available to be uploaded onto a flexible and powerful analytic execution engine. However, questions or problems may need to be translated into appropriate analytic formulations in order to generate the desired responses.

In a big data scenario, the size of the volume of data often negatively impacts processing of such query-based analytics. One of the biggest problems in big data analysis is that of formulating the right query. Although it may be important to extract features and execute data analytics, this may not be sufficient to address the issues related to big data. Once data is available in an appropriate format, it becomes important to know what analyses may be most productive in providing operational insights. When datasets are small and experts are readily available, platforms connecting analytic tools to automatically collected data are generally very effective. However, as the data grows larger and experts become scarce, operational data mining becomes difficult; there may be just too much data and the relationships are too complex to formulate queries that may provide much needed insights. Accordingly, there may be an overwhelming need for tools that help formulate analytic queries.

Therefore, in the context of operational data, it may be important to provide an interface that may be utilized by operational investigations to easily formulate and solve operational issues. As disclosed in various examples herein, such an interface may be based on concatenations of pattern and anomaly detectors. In particular, interesting analytics may be highlighted, and relevant analytics may be suggested, independent of a query. An interactive ecosystem may be disclosed where new combinations of anomalies and patterns may compete for selection by a domain expert.

Generally, it may be difficult to define a set of anomaly and pattern detectors that may encompass all the detection that may be necessary for operational analytics. Additionally every significant set of detectors may initially have an overwhelming set of anomalies and patterns for the domain expert to investigate, validate, and/or disqualify. As disclosed herein, such issues may be addressed by using a limited, but generic, set of anomaly detectors and pattern recognition schemes, which may combine automatically so that input data related to a series of events and telemetry measurements may be enriched whenever an anomaly or pattern may be detected. Such feedback enables deep semantic explorations that may eventually encompass a large set of complex analytics. Furthermore, such feedback-based interaction constitutes a competitive ecosystem for prioritized analytics, where analytics compete for the attention of the domain expert, highlighting the analyses that are most likely to be relevant to the domain expert. Moreover, changes in operational performance are driven by changes in the underlying input data and by continuous interactions with domain experts. New data may manifest new anomalies and patterns, whereas new interactions with domain experts may introduce new tagged patterns and system anomalies.

As described in various examples herein, interactive detection of system anomalies is disclosed. One example is a system including a data processor, an anomaly processor, and an interaction processor. Input data related to a series of events and telemetry measurements is received by the data processor. The anomaly processor detects presence of a system anomaly in the input data, the system anomaly indicative of a rare situation that is distant from a norm of a distribution based on the series of events and telemetry measurements. The interaction processor is communicatively linked to the anomaly processor and to an interactive graphical user interface. The interaction processor displays, via the interactive graphical user interface, an output data stream based on the presence of the system anomaly, receives, from the interactive graphical user interface, feedback data associated with the output data stream, and provides the feedback data to the anomaly processor for operations analytics based on the feedback data.

Generally, the term “system anomaly” as used herein may correspond to a time-slot where multiple events/signals show collectively anomalous behavior through their combined anomaly measures. Alternatively the term “rare situation” may be used to emphasize a co-location in time. Both these terms may indicate some collective anomaly situation/behavior. Generally, the system anomaly of interest may appear on the graphical user interface and analysis may proceed without the user needing to enter a query. The analysis may begin by selection of the system anomaly, where the major sources of the system anomaly are prioritized—so that the highest contribution appears more prominently, and similar system anomalies are identified, thereby allowing for fast analysis that usually does not require any further search or data queries. The interface also enables filtering of input data using keywords. This may be useful for instances where the problem the user may be set to investigate does not seem to appear on the initial interface. It allows for the interaction described herein from a filtered subset of data. The interface also highlights keywords related to system anomalies as potential filter words as another means of highlighting system anomalies for the benefit of a user, such as, for example, a domain expert reviewing the system anomalies for operations analytics.

Generally, the feedback data need not be based on the same type of received system anomalies, i.e., at each iteration, a certain anomaly type (rarity, flood, etc.) may be added and/or removed from the set of the events. As described herein, a weighting may be utilized (e.g., weight 0 for removal of a certain anomaly type). The techniques described herein enable automatic detection of system anomalies without a query. However, such automatic detection techniques may be combined with known system anomalies, and/or query-based detection of system anomalies to form a hybrid system.

FIG. 1is a functional block diagram illustrating an example of a system100for interactive detection of system anomalies. System100is shown to include a data processor102, an anomaly processor104, and an interaction processor106.

The term “system” may be used to refer to a single computing device or multiple computing devices that communicate with each other (e.g. via a network) and operate together to provide a unified service. In some examples, the components of system100may communicate with one another over a network. As described herein, the network may be any wired or wireless network, and may include any number of hubs, routers, switches, cell towers, and so forth, Such a network may be, for example, part of a cellular network, part of the internet, part of an intranet, and/or any other type of network.

The components of system100may be computing resources, each including a suitable combination of a physical computing device, a virtual computing device, a network, software, a cloud infrastructure, a hybrid cloud infrastructure that includes a first cloud infrastructure and a second cloud infrastructure that is different from the first cloud infrastructure, and so forth. The components of system100may be a combination of hardware and programming for performing a designated function. In some instances, each component may include a processor and a memory, while programming code is stored on that memory and executable by a processor to perform a designated function.

The computing device may be, for example, a web-based server, a local area network server, a cloud-based server, a notebook computer, a desktop computer, an all-in-one system, a tablet computing device, a mobile phone, an electronic book reader, or any other electronic device suitable for provisioning a computing resource to perform an interactive detection of system anomalies. Computing device may include a processor and a computer-readable storage medium.

The system100receives input data related to a series of events and telemetry measurements. The system100detects presence of a system anomaly in the input data, the system anomaly indicative of a rare situation that is distant from a norm of a distribution based on the series of events and telemetry measurements. In some examples, the system100detects presence of an event pattern in the input data. The system100displays, via an interactive graphical user interface, an output data stream based on the presence of the system anomaly. The system100receives, from the interactive graphical user interface, feedback data associated with the output data stream, and provides the feedback data to the anomaly processor for operations analytics based on the feedback data.

In some examples, the data processor102receives input data related to a series of events and telemetry measurements. The series of events may be customer transactions, Web navigation logs (e.g. click stream), security logs, and/or DNA sequences. In some examples, each event may be associated with an event identifier identifying a given event in the series of events, an event time identifier identifying a time when the given event occurred. In some examples, the series of events may be defined based on temporal constraints. For example, the series of events may be a collection of log messages for a specified period of time. In some examples, the series of events may be defined based on spatial constraints. For example, the series of events may be a collection of log messages for a specified geographic location. Combinations of spatial and temporal constraints may be used as well. Also, for example, the series of events may be based on additional system identifiers, such as, for example, usage or any other identifier of a system. Generally, such system identifiers may not be uniform. For example, system anomalies may appear over differing time intervals, and/or different usage values. As described herein, system anomalies from such non-uniform system identifiers may be appropriately modified and/or scaled to be uniform, additive, and so forth, to determine, for example, an anomaly intensity, an anomaly score, an anomaly fingerprint, and a fingerprint matching function.

The input data may be normalized in several ways. For example, a log analysis, and/or a signal analysis may be performed on the input data. In some examples, data processor102may receive a normalized input data. In some examples, data processor102may perform operations to normalize the input data. In some examples, the input data may be a stream of log messages. Log messages may be analyzed for latent structure and transformed into a concise set of structured log message types and parameters. In some examples, each source of log messages may be pre-tagged. The input data may be a corresponding stream of event types according to matching regular expression. Log messages that do not match may define new regular expressions. In some examples, telemetry signals may also be analyzed for periodicities and relevant features. Generally, the event type is a type of log message or a type of performance metric.

The input data may be fed into analysis processors, such as, for example, an anomaly processor104, In some examples, system100may include a pattern processor (not illustrated inFIG. 1). The anomaly processor104detects presence of a system anomaly104A in the input data. In some examples, such detection may be automatic. In some examples, such detection may be query-based. In some examples, query-less detection of system anomalies may be combined with query-based techniques.

As described herein, a system anomaly104A is an outlier in a statistical distribution of data elements of the input data. The term outlier, as used herein, may refer to a rare event, and/or an event that is distant from the norm of a distribution (e.g., an extreme, unexpected, and/or remarkable event). For example, the outlier may be identified as a data element that deviates from an expectation of a probability distribution by a threshold value. The distribution may be a probability distribution, such as, for example, uniform, quasi-uniform, normal, long-tailed, or heavy-tailed. Generally, the anomaly processor104may identify what may be “normal” (or non-extreme, expected, and/or unremarkable) in the distribution of clusters of events in the series of events, and may be able to select outliers that may be representative of rare situations that are distinctly different from the norm. Such situations are likely to be “interesting” system anomalies104A. In some examples, system anomalies may be identified based on an expectation of a probability distribution. For example, a mean of a normal distribution may be the expectation, and a threshold deviation from this mean may be utilized to determine an outlier for this distribution.

In some examples, a system anomaly may be based on the domain. For example, the distribution may be based on the domain, and an expectation or mean of the distribution may be indicative of an expected event. A deviation from this mean may be indicative of a system anomaly. Also, for example, a system anomaly in log messages related to security, may be different from a system anomaly in log messages related to healthcare data. In some examples, a domain expert may provide feedback data that may enable automatic identification of system anomalies. For example, repeated selection of an event by a domain expert may be indicative of a system anomaly.

A domain may be an environment associated with the input data, and domain relevance may be semantic and/or contextual knowledge relevant to aspects of the domain. For example, the input data may be data related to customer transactions, and the domain may be a physical store where the customer transactions take place, and domain relevance may be items purchased at the physical store and the customer shopping behavior. As another example, the input data may be representative of Web navigation logs (e.g. click stream), and the domain may be the domain name servers that are visited via the navigation logs, and domain relevance may be analysis of Internet traffic. Also, for example, the input data may be related to operational or security logs, and the domain may be a secure office space for which the security logs are being maintained and/or managed, and domain relevance may be tracking security logs based on preferences such as location, time, frequency, error logs, warnings, and so forth.

Generally, a domain expert may be an individual in possession of domain knowledge. For example, the domain may be a retail store, and the domain expert may be the store manager. Also, for example, the domain may be a hospital, and the domain expert may be a member of the hospital management staff. As another example, the domain may be a casino, and the domain expert may be the casino manager. Also, for example, the domain may be a secure office space, and the domain expert may be a member of the security staff.

In some examples, the anomaly processor104may operate on a series of classified structured log messages {ej}. Each log message or event may be associated with at least a time tj=t(ej), and an event type Tj=T(ej). In some examples, the event type may be a signal, and each event may be associated with, in addition to time and event type, numerical values Vm,j=vm(ej), where the numerical values associated with events of an event type Tn, vm(ej|T(ej)=Tn) may be attributed a signal type Tn,m. In some examples, the anomaly processor104may additionally operate on telemetry signals arriving in structured tabular form as a stream of discrete signal measurement events {el} where each signal measurement may be associated with a time tl=t(el), a signal type Tl=T(el) and a single numerical value vl=v(el).

In some examples, system100may include an evaluator (not shown in the figures) to determine various quantitative measurements related to the input data. Generally, the evaluator may determine measurements at different levels. For example, a first level measurement for anomaly intensity amounts may be determined for each event-type. Also, for example, a second level measurement may be a collective measurement based on anomaly types (e.g., Flood of Events, Rare Events, etc.). For example, the evaluator may determine an anomaly intensity, an anomaly intensity score, an anomaly fingerprint, and an anomaly fingerprint matching score for anomaly types. As another example, a third level measurement may be an aggregated measurement of an anomaly score for a system anomaly in a given time slot.

As described herein, a determination at each level may be based on a determination at a preceding level. For example, the anomaly intensity, the anomaly intensity score, the anomaly fingerprint, and the anomaly fingerprint matching score may be based on the anomaly intensity amounts. Likewise, the anomaly score may be based on the anomaly intensity, the anomaly intensity score, the anomaly fingerprint, the anomaly fingerprint matching score and the anomaly intensity amounts. As described herein, each measurement at each level may correspond to different distributions, different scales, different time-slots, and so forth. Accordingly, to meaningfully combine these measurements, they may need to be scaled and/or transformed to measurements that are comparable and additive, facilitating their respective combination, aggregation, comparison and/or matching. These and other aspects of detection of a system anomaly are described herein.

In some examples, the evaluator may determine anomaly intensity amounts Qk(tl,Tj) defined on discrete time-slotstl: tϵ[t0+iΔ,t0+(i+1)Δ] for each event type and signal type Tj. In some examples, an anomaly intensity amount for events may be the event-count n(Tj,tl)=∥Tj(tl)∥. In some examples an anomaly intensity amount for events may be a function of the event count, such as the event-indicator I(Tj,tl)=1 if n(Tj,tl)>0) else 0, or the event-count log-scale (Tj,tl)=1+log2n(Tj,tl) if n(Tj,tl)>0) else 0. In some examples, an anomaly intensity amount for signals may be the maximal signal value per signal type per time slot M(Tl,tl)=max(vl(tϵtl)). In some examples, an anomaly intensity for signals may be the range of signal values per signal type per time slot R(Tl,tl)=max(vl(tϵtl))−min(vl(tϵtl)).

In some examples, the evaluator determines, for a time interval, the anomaly intensity for each anomaly type. As described herein, the anomaly intensities for each of the different anomaly types may be determined before they are transformed into anomaly scores via a “distinctive residual rarity” transformation. In some examples, the evaluator determines, for each time interval for an anomaly type, incomparable anomaly intensity amounts, wherein each incomparable anomaly intensity amount may be transformed with respect to the distribution of associated incomparable anomaly intensity amounts in reference time intervals, based on a distinctive residual rarity extremity score, into comparable, additive, and distinctive anomaly intensity amounts. Accordingly, incomparable anomaly intensities associated with different event types may be transformed into comparable, additive and distinctive anomaly intensities to determine an anomaly score. For example, the anomaly processor104may comprise K components, each component k associated with a specific anomaly type for a specific group of events and/or signals Gk, and applying a transformation of one or more anomaly intensities into anomaly intensity amounts ck(Tj,tl). Each such transformation may be designed such that anomaly intensity amounts corresponding to different event types within the reference group Gkmay share a common scale so they may be combined into anomaly intensities representative of event type or signal-type groups, and they may also be compared to determine the event types that are the main contributors to the anomaly intensity, for example, to aid in root-cause identification. As used herein, an anomaly intensity amount measures a contribution of a certain event type to an anomaly intensity.

In some examples, the anomaly processor104may receive a time-based stream of events or signals, and the evaluator may determine an anomaly intensity and an anomaly score for each given interval of time. In some examples, for a given time slot, the anomaly processor104may identify events that contribute to a majority of the anomaly intensity, and such identified events may be used as a fingerprint to identify similar system anomalies. In some examples, the evaluator may determine three anomaly-related quantities from the anomaly intensity amounts per time slot. In some examples, such determinations may be performed by each component k of the anomaly processor104. The three anomaly-related quantities may be:1) Anomaly intensity: An anomaly intensity may be determined based on:
xk(tl)=Φk(ΣTϵGkckα(T,tl)),  (Eqn. 1)where each anomaly intensity amount may be raised to a power a associated with component k. In some cases α=1 (simple addition). In some cases α=2 (sum-of-squares) to emphasize the contribution of larger components relative to smaller components. In some examples, Φkmay be an optional non-linear monotonic mapping selected to equalize differences between anomaly intensity amounts. In some examples, Φkmay be chosen as the α-root function, i.e. Φk(γ)=α√{square root over (γ)}.2) Anomaly score to assess an extremity of an anomaly intensity xk(tl) relative to the probability distribution of corresponding values in reference group of time-slotstϵG(tl): αk(tl)=A(xk(tl), f(xk(t))). In some examples, a time-slot reference group may include a large contiguous time-span that may include relevant historical data relative to the particular time-slot (e.g. G(tl) may include all time-slotstup to a specified number of days beforetl). In some examples, a time-slot reference group may correspond to a periodic set of time-slots sharing same time-of-day and day-of-week. Use of such periodic reference groups may be a realization of base-lining for seasonality of work-loads and work-patterns of the system under analysis. In some examples, a time-slot reference group may be based on a geographical area. In some examples, a time-slot reference group may be based on the domain. The anomaly scoring function A may be designed to be additive with unified scales for anomaly scores of different components, so that by adding up the component anomaly scores: α(tl)=Σkαk(tl), the resulting total anomaly score may be meaningful. For example, anomaly intensities corresponding to different anomaly types may be transformed into normalized and comparable anomaly scores per anomaly type with respect to the time-slot reference group. As described herein, the transformation may be based on a distribution of related anomaly intensities, and may be based on base lining (e.g., periodic time-slot reference groups) and history spans (e.g., system anomalies with respect to last day, last week, etc.). In some examples, as described herein, the evaluator determines, for the time interval, anomaly intensities and the anomaly score, and each anomaly intensity may be transformed, with respect to the distribution of anomaly intensities of the same anomaly type in reference time-slots, based on a distinctive residual rarity extremity score, into comparable, additive, and distinctive anomaly intensity scores, that may in turn be combined to determine the anomaly score.3) Anomaly Fingerprint per time-slot specifying the identity and relative contributions of event types to the anomaly intensity at time-slottlmay be determined as:
Fk(tl)={Tj,ρjk(ti)}jϵJi,  (Eqn. 2)where the relative contribution may be defined as ρjk(ti)=ckα(Tj,tl)/ΣTϵGkckα(T,tl), and top contributing event types may be selected such that the sum of their relative contributions may be starting with larger contributions first, may be the minimum that exceeds a threshold close to 100%, for example 95%. Generally, each event type in each time-slot may be associated with an anomaly intensity amount for each anomaly type. The anomaly fingerprint may then be based on the anomaly intensity amounts for different event types. In some examples, this may be achieved via a vector of anomaly intensity amounts in time-slots different from the selected time-slot. As described herein, anomaly intensities may be determined by combining comparable anomaly intensity amounts for each event type, and such comparable anomaly intensity amounts for different event types may be combined to determine an anomaly intensity and an anomaly fingerprint.4) Fingerprint matching functions: determined for a sub-set of time slots t* that may be considered to be interesting (either by user selection or by their relatively higher anomaly scores). For each such time slott*the fingerprint matching function may be determined as:
αk|t*(tl)=Φk(ΣTjϵGkckα(Tj,tl)·ρjk(t*)),  (Eqn. 3)such that for each time-slottlthe fingerprint matching score may be high only if anomaly intensity amounts corresponding to top contributing event types in the fingerprint are high.

In some examples, as described herein, the anomaly type may include a Flood of Events, wherein the anomaly intensity amount is an event count, a Variety of Events, wherein the anomaly intensity amount is an event occurrence indicator, a Flood of Rare Events, wherein the anomaly intensity amount is a product of an event count extremity factor, and an event-type rarity factor, and a Flood of Extreme Signals, wherein the anomaly intensity amount is a maximal signal value per time interval transformed based on a distinctive residual rarity extremity score.

In some examples, the anomaly type may be a Partial Pattern. The Partial Pattern anomaly type may be characterized by multiple events appearing repeatedly in the same time slot. For example, a set of 30 events may be identified in the selected time slot, where each event corresponds to a service shutdown message and/or alert. Generally, the Partial Pattern anomaly type may be detected based on interactions with a domain expert via the interactive graphical user interface108.

In some examples, the anomaly processor104may include a component evaluating Flood of Events (“FoE”) anomaly type, where the anomaly intensity amount may be the occurrence-count of event type Tjin time slot ti, cFoE(Tj,tl)=n(Tj,tl), and power-law may be α=1 (regular sum), so that the anomaly intensity may be xFoE(tl)=nTϵG(tl). The anomaly-fingerprint components are the relative frequencies of the different events ρjk(ti)=n(Tj,tl)/nTϵG(tl) in each time-slot.

In some examples, the anomaly processor104may include a component evaluating Variety of Events (“VoE”) anomaly type, where the anomaly intensity amount may be the event-indicator cFoE(Tj,tl)=I(Tj,tl) equal to 1 for each event j that appeared at least once in at time slot tiso that the sum of anomaly intensity amount may be just the number of distinct event types that occurred in time slot ti. N(Tϵtl). The anomaly intensity may be xVoE(tl)=N(Tϵtl), and anomaly-fingerprint components are equal to 1/N(Tϵtl) for all event types that appeared in time slottland 0 otherwise.

In some examples, the anomaly processor104may include a component evaluating a Flood of Rare Events (“RE”) anomaly type. The RE anomaly intensity amount for each event type Tjthat appears in a certain time slottl, may be designed to be large if Tjis rare relative to other events in the time-slot reference grouptϵG(tl). In some examples, an event type rarity factor may be computed as the negative log of the occurrence-probability of event type Tjin the reference group of time-slots G(tl), in the reference group of event types G(Tj:r(Tj,tl)=−log2(P(Tj,tl)), where P(Tj,tl)=∥Tj∥lϵG(tl)/∥G(Tj)∥tϵG(tl). Furthermore, the RE anomaly intensity amount may be designed to be large if the count of an event Tjat time-slottlis high relative to the counts of that event type in other time-slots in reference grouptϵG(tl), so that e.g. a rare event that occurs several times in a single time-slot may contribute a larger RE anomaly intensity amount than the same event occurring once in a time slot. In some examples, an event-count extremity factor may be computed as the occurrence-probability of event type Tjin time-slot tirelative to the reference group of time-slots G(tl):h(Tj,tl)=n(Tj,tl)/ΣTϵG(tl)n(Tj,tl). Event-count extremity factors tend to be relatively high for rare events in the time-slots they appear in, and relatively low for frequent events in any time slot, which keeps the RE anomaly intensity amounts corresponding to frequent events low in all time-slots. In some cases, the RE anomaly intensity amount may be expressed as a product of the event type rarity factor and the event-count extremity factor cRE(Tj,tl)=r(Tj,tl)·h(Tj,tl). Since the RE anomaly intensity amount may be already given in the log-domain, and the event-count extremity factor may be typically much smaller than 1, the sum of RE anomaly intensity amounts over all events may tend to be small. In some case, an exponential non-linear mapping ΦRE(γ)=2γmay be applied to the sum, to emphasize relatively large score, and to transform the anomaly intensities to similar ranges as the other anomaly components.

In some examples, G(.) or cRE(.,.) maybe normalized compared to a baseline of a system, such as, for example, a value based on historical data and/or feedback data. Accordingly, G(.)=G(.)−hist (G), where hist (G) denotes the value based on historical data and/or feedback data. For example, the feedback data may be indicative of cropping of system anomalies below a threshold to zero. Also, for example, feedback data may be indicative of bucketing all system anomalies above another threshold to amplified values.

In some examples, the anomaly processor104may include components to evaluate signal related anomaly types. Unlike event-counts that have a common scale for all types of events, different signal types may have incomparable scales, so their anomaly intensities, like range or maximum within each time-slot, may not be used as anomaly intensity amounts, as there may be no meaning in adding quantities not defined on the same scale. Instead a generic transformation may be applied to transform an anomaly intensity into a value-extremity score, such that value-extremity scores corresponding to signals with significantly different types of distribution, and scale may be comparable and additive so they may be used as anomaly intensity amounts to compute a meaningful anomaly intensity, and an anomaly fingerprint. Furthermore, such additive value-extremity scores may be applied to the anomaly intensity to generate anomaly scores that are comparable and additive across anomaly types.

A value-extremity score may be expected to be high only for extreme values (outliers), which may be residually rare (very small percentage of the values are equal or above an extreme value), and well separated from the non-extreme majority of values (the inliers), One value-extremity score in the case of normally distributed values, may be the “Z-score” obtained by subtracting the distribution-mean from the value and dividing it by the distribution standard deviation a. However, each of the anomaly intensities may follow a different type of distribution including quasi-uniform, normal, long-tailed, or heavy-tailed. The Z-score may not work as well for non-normal distributions.

FIG. 2Aillustrates an example of hypothetical anomaly intensities distributed uniformly200a, normally202a, and with a long-tailed Weibull distribution204a. Dots whose Z-score may be above 3 (at least 3σ above the mean) are highlighted. This may capture correctly extreme-values in the uniform and normal distributions. For example three points with high extreme-value score206aare illustrated for the case of normal distribution, and no points with high extreme-value score are found in the case of uniform distribution. However, a relatively large number of values208aget high Z-score in the case of heavy tailed distribution, which defies the requirement for residual rarity of extreme values.

Another value-extremity score which may work well for long-tailed and heavy-tailed distribution may be the residual rarity of a value measured as the negative log of the probability of other values to be equal or higher—this probability may be associated with the complementary cumulative distribution function (CCDF)Fknown in the statistical literature:
R(Q(tl))=−log2P(Q(tϵG(tl))≥Q(tl))=−log2FtϵG(tl)(Q(tl))  (Eqn. 4).
The CCDF, like any function measuring probabilities, has an important property that when applied to joint value distributions (originating from multiple signals), the distribution function may be expressed as a product of the individual value distributions, provided the signals are statistically independent. Accordingly, the log of the joint probability for independent signals may be expressed by the sum of the logs of the individual signal distributions. In other words, the residual-rarity score of a multiple-signal set corresponds to the sum of individual residual-rarity scores for independent signals. Accordingly, CCDF-based value-extremity scores (referred to herein as residual rarity extremity scores) are comparable and additive as required.

In some examples, the residual rarity extremity scores may be equivalent to ‘top-p %’ detection and may have no regard to value separation criteria. Accordingly, it may attribute high scores to top values even if they are not well separated from lower values, like in uniform distributions. To avoid false detections of outliers for uniform distributions, an outlier-detection threshold to match the detection rate of Z-scores for normal distributions may be designed. However, such a technique may still leave several false outlier detections in uniform distributions, and too few true-outlier detections for long-tail distributions.

FIG. 2Billustrates an example of resultant unsatisfactory outliers obtained after a residual rarity transformation of the anomaly intensities illustrated inFIG. 2A. For example, resultant unsatisfactory outliers obtained via the residual rarity extremity score applied to the anomaly intensities illustrated inFIG. 2Aare illustrated. Hypothetical anomaly intensities distributed uniformly200b, normally202b, and with a long-tailed Weibull distribution204b. As illustrated, although the outliers206bare consistent with the outliers206afor the normal distribution202b, the outliers208bfor the long-tailed Weibull distribution204bare minimal, and there are two false outliers detected for the uniform distribution in200b.

To obtain a value-extremity score (an outlier criterion) that works well for a wide range of value distributions, and that may be comparable and additive and may address both the residual rarity and the separation criteria required from outliers, the residual rarity extremity scores may be modified to determine a “scaled CCDF”, referred to herein as a distinctive residual rarity extremity score. Assuming that for operations data, all anomaly intensities are non-negative (as is the case for event-counts and telemetry signal values), and that separation criteria should be relative to the value-scale of each signal, the distinctive residual rarity extremity score may be defined by a minimal ratio S between outlier and inlier values, where S may be larger than 1. The extremity score with separation factor S may be:

FIG. 2Cillustrates an example of resultant outliers based on a modified distinctive residual rarity anomaly transform. For example, an example of resultant outliers based on the distinctive residual rarity extremity score with separation factor S=1.2 (i.e. at least 20% separation) is illustrated. Hypothetical anomaly intensities may be distributed uniformly200c, normally202c, and with a long-tailed Weibull distribution204c. As expected, there are no outliers in the uniform distribution, and for the same three outliers206cin the normal distribution202c, a larger but limited set of outliers208cmay be realized for the Weibull samples204c.

In some examples, anomaly intensities may be transformed into anomaly scores that are comparable, additive and distinctive. The term “distinctive” as used herein refers to a requirement of a threshold separation between high values and lower values to be considered extreme. In some examples, the evaluator determines, for the time interval, anomaly intensities and the anomaly score, and where incomparable anomaly intensities are transformed, based on a distinctive residual rarity extremity score, into comparable, additive, and distinctive signals to determine the anomaly score. For example, the evaluator may include a component evaluating Extreme Signal (“ES”) anomaly type, where the anomaly intensity amount for signal-type Tlin time slot timay be a distinctive residual rarity extremity score:

corresponding to the maximal signal value per signal type Tlper time slot M(Tl,tl)=max(vl(tϵtl)). In some cases the separation factor S may be 1.2. In some examples the anomaly intensity may be computed according to Eq. 1, with power law α=2 (sum-of-squares), and a mapping function of ΦES(γ)=√{square root over (γ)}, such that signals with values intlthat have high value-extremity scores may be further emphasized relative to signals with lower value-extremity scores.

In some cases, the anomaly processor104, may attribute an anomaly score to each anomaly component k in each time-slottlby assessing the extremity of an anomaly intensity xk(tl) relative to the probability distribution of corresponding values in reference group of time-slotstϵG(tl), using the distinctive residual rarity extremity score:

Whereas event anomalies are generally related to insight into operational data, event patterns indicate underlying semantic processes that may serve as potential sources of significant semantic anomalies. As disclosed herein, an interaction processor106may be provided that allows operational analysis to be formulated as concatenations of pattern and anomaly detectors.

In some examples, system100may include a pattern processor to detect presence of an event pattern in the input data. Although the pattern processor may be described herein as a separate component, in some examples, the functions of the pattern processor may be performed by the anomaly processor104. Generally, the pattern processor identifies non-coincidental situations, usually events occurring simultaneously, Patterns may be characterized by their unlikely random reappearance. For example, a single co-occurrence in100may be somewhat likely, but 90 co-occurrences in100may be much less likely to occur randomly.

In some examples, interaction processor106may be communicatively linked to the anomaly processor104and to an interactive graphical user interface108. The interaction processor106displays, via the interactive graphical user interface108, an output data stream based on the presence of the system anomaly. In some examples, interaction processor106may generate an output data stream based on the presence of the system anomaly and the event pattern. In some examples, the interaction processor106receives feedback data associated with the output data stream from the interactive graphical user interface108, and provides the feedback data to the anomaly processor104and/or the pattern processor for operations analytics based on the feedback data. As described herein, feedback data may include feedback related to domain relevance, received via the interactive graphical user interface108and processed by the interaction processor106. The feedback data may be indicative of selection or non-selection of a portion of the interactive graphical user interface108. As used herein, selection may include copying a portion of text and/or images displayed by the interactive graphical user interface108, selection or non-selection of a selectable menu, hovering over, or clicking on a text and/or image displayed, or touching a touch-sensitive portion of the interactive graphical user interface108.

The interaction processor106processes the feedback data and supports interaction between the interactive graphical user interface108and a domain expert. Operations analytics, as used herein, may include any analytics associated with system performance. For example, operations analytics may include analysis of interesting patterns and incorporation of domain knowledge in the form of constraints into the detection of system anomalies. For example, the domain may be a retail store, and the domain knowledge may include knowledge about traffic patterns in the store, customer purchases, product placement, products sold, available inventory, clientele, store hours, and so forth. In some examples, the interaction processor106provides, via the interactive graphical user interface108, an interactive visual representation of the system anomalies and event patterns. For example, to enable the domain expert to better understand and discover patterns, interaction processor106may provide a context-augmented interface for visually guided exploration.

In some examples, operations analytics may include tagging of system anomalies and event patterns. In some examples, operations analytics may include identifying anomaly types and initiating system responses based on the identified anomaly types. In some examples, operations analytics may include adding and/or removing an anomaly type from the output data stream. In some examples, operations analytics may include an actionable response such as generating a system alert. For example, the anomaly processor104may identify an issue and trigger a system alert to act on the issue promptly. In some examples, such an alert may be based on a fingerprint of a past system anomaly that was identified, tagged, and associated with a preferred mitigation or remediation action. For example, a past anomaly may be associated with a service shutdown based on a Partial Pattern anomaly type, and the anomaly processor104may trigger a system alert for a service shutdown. Also, for example, the Partial Pattern anomaly type may be detected based on interactions with a domain expert via the interactive graphical user interface108, and a forced shutdown message may be generated by the anomaly processor104.

In some examples, the interaction processor106may display a detected system anomaly, and may identify selection of the system anomaly by a domain expert. In some examples, the anomaly processor104may identify an anomaly type associated with the system anomaly, and the interaction processor106may display the anomaly type via the interactive graphical user interface108. In some examples, the interaction processor106may identify interaction based on the system anomaly. For example, the domain expert may add or delete the system anomaly. Also, for example, the domain expert may select a word on a displayed word cloud to further investigate additional system anomalies similar to the selected system anomaly. In some examples, the anomaly processor104may determine an anomaly fingerprint for the selected pattern, determine a fingerprint matching function associated with the selected system anomaly, and detect additional system anomalies based on the fingerprint matching function.

As illustrated inFIG. 1, the interactive graphical user interface108may be communicatively linked to the anomaly processor104via the interaction processor106. Accordingly, the interactive graphical user interface108supports the anomaly processor104and/or a pattern processor. In some examples, the interactive graphical user interface108displays the output data stream, including a first selectable option associated with the system anomaly, and a second selectable option associated with the event pattern. Accordingly, the interactive graphical user interface108displays system anomalies and event patterns, and provides suitable interfaces, such as the first selectable option associated with the system anomaly, and the second selectable option associated with the event pattern for the domain expert to identify and tag significant system anomalies. The interactive graphical user interface108receives such feedback data associated with the first and second selectable options and provides the feedback data to the interaction processor106. In some examples, the interactive graphical user interface108provides the feedback data back into the anomaly processor104and/or the pattern processor via the interaction processor106.

In some examples, the interactive graphical user interface108further provides, in response to a selection of the first selectable option, a pop-up card with information related to the system anomaly. Generally, the feedback data need not be based on the same type of received system anomalies, i.e., at each iteration, a certain anomaly type (RE, FoE, etc.) may be added and/or removed from the set of the events. As described herein, a weighting may be utilized (e.g., weight 0 for removal of a certain anomaly type).

FIG. 3is an example display of an output data stream including a word cloud. A word cloud is a visual representation of a plurality of words highlighting words based on a relevance of the word in a given context. For example, a word cloud may comprise words that appear in log messages associated with the selected system anomaly. Words in the word cloud may be associated with term scores that may be determined based on, for example, relevance and/or position of a word in the log messages. In some examples, the word cloud may be interactive, and a system anomaly may be identified based on an interaction with the word cloud. For example, a term in the word cloud may be selected, and the interaction processor106may identify system anomalies that are associated with log messages that include the selected term.

In some examples, the example display of the output stream may be a snapshot of an application launcher interface provided via the interactive graphical user interface108. The output data illustrated relates to an input data of log messages received during an example time period including May 5 to July 31, represented by the x-axis of the graphical representation. System anomalies300are illustrated, along with a word cloud306and event patterns308. In some examples, a composite anomaly score may be displayed, where the composite anomaly score may be determined as a sum of several different anomaly scores. The first selectable option associated with the system anomaly may be, for example, a clickable node, such as node302. Every highlighted node on the graph, such as, for example, node302, may be clickable. Selection of the first selectable option, such as a node, may launch an analysis of the associated system anomaly. For example, clicking node302may launch an analysis of the system anomaly that occurred at or about July 1. As described herein, a selection may include a click, or may include hovering over node302in a touch-sensitive interactive display.

In some examples, the feedback data may include an indication of a selection of a system anomaly, and the graphical user interface108further provides, based on the feedback data, a pop-up card with information related to the selected system anomaly. For example, referring again toFIG. 3, in response to a selection of the first selectable option, a pop-up card with information related to the system anomaly may be displayed. For example, pop-up304may be displayed, with information related to the system anomaly. Pop-up340may include, for example, a date and time associated with the system anomaly, and a type of anomaly score for the system anomaly. For example, as indicated inFIG. 3, the selected system anomaly occurred on “2013-07-01” at “13:00:00”. Also, for example, the anomaly type may be indicated as “Variety of Events”.

In some examples, the anomaly processor104further generates a word cloud to be displayed via the interactive graphical user interface108, the word cloud highlighting words that appear in log messages associated with the selected system anomaly. Highlighting may be achieved via a distinctive font, font size, color, and so forth. In some examples, term scores may be determined for key terms, the term scores based on a modified inverse domain frequency. In some examples, the modified inverse domain frequency may be based on an information gain or a Kullback-Liebler Divergence.

For example, referring again toFIG. 3, word cloud306highlights words that appear in anomalous messages more than in the rest of the messages. In some examples, relevance of a word may be illustrated by its relative font size in the word cloud306. For example, “queuedtoc”, “version”, and “culture” are displayed in relatively larger font compared to the font for the other words. Accordingly, it may be readily perceived that the words “queuedtoc”, “version”, and “culture” appear in the messages related to the system anomaly more than in other messages. Event patterns308are displayed. In some examples, event pattern308may represent groups of events (or event groups) that appear almost exclusively together in the input data.

In some examples, the anomaly processor104may detect system anomalies based on at least one of the feedback data and a previously processed event pattern. In some examples, the evaluator may determine, for a time interval, the anomaly fingerprint based on a set of relative contributions of event types to the anomaly intensity; where a fingerprint matching score for the anomaly fingerprint may be computed in a second time interval to determine presence or absence of similar system anomalies in the second time interval, and where the fingerprint matching score may be computed based on a correlation between the anomaly fingerprint and anomaly intensity amounts in the second time interval.

For example, the anomaly processor104may identify an issue and trigger an alert to act on it promptly based on the fingerprint of a past system anomaly that was identified, tagged, and associated to a preferred mitigation or remediation action. The identification may be done by detecting other events that match the tagged fingerprint sufficiently well. Tagged system anomalies may increase the importance of their respective anomaly score, and deleted system anomalies may reduce the respective anomaly score.

Referring toFIG. 1, in some examples, the interactive graphical user interface108further provides, in response to a selection of the first selectable option, an analysis interface to analyze the system anomaly. In some examples, the analysis interface may be an interactive anomaly analysis interface. In some examples, the analysis interface may be generated by the anomaly processor104. For example, in response to a click on a system anomaly, the interaction processor106may prompt the interactive graphical user interface108to open an analysis interface. In some examples, interaction processor106may receive feedback data indicative of a domain expert's interaction with the interactive graphical user interface108and provide the feedback data to the anomaly processor104to generate and/or modify an analysis interface. For example, the domain expert may examine the system anomaly and perhaps tag it, indicating its underlying cause. Tagging the system anomaly may catalogue an anomaly fingerprint as a known event.

FIG. 4is an example of an analysis interface for system anomalies. A snapshot of the analysis interface triggered by selection of the system anomaly under the cursor inFIG. 3is illustrated. A threshold anomaly score may be utilized to filter system anomalies of interest. For example, a threshold402of 75% may be utilized. In some examples, an actionable menu408may be provided to receive input from a domain expert. The actionable menu408may provide data entry fields and/or drop-down menus, to “Save this anomaly”, “choose a severity”, “Delete this Pattern”, “Do Nothing”, and so forth. For example, an entry of “75%” may be entered as a threshold value. System anomaly list406may be a set of events constituting the fingerprint as in Eqn. 2 corresponding to the system anomaly associated with node302inFIG. 3. Based on selection of node302, the anomaly processor104may generate an anomaly fingerprint406. In some examples, the feedback data from a domain expert may indicate that the fingerprint406represents an instance of “Rogue Report”. That is, the underlying cause of the fingerprint406, and by association node302, may be that the transmission of a very complex report may be holding up data traffic and blocking efficient rendering of system resources. The top portion ofFIG. 4illustrates the fingerprint matching score from Eqn. 3 as a function of time. As illustrated, the system anomaly404matches the fingerprint perfectly, as expected. There may be other times where the fingerprint match may be high, but not sufficiently high for automatic recognition. Tagging the system anomaly404may indicate that the anomaly fingerprint may be stored and any future event that matches the anomaly fingerprint sufficiently well may be associated with the same tag and identified as a system anomaly.

In some examples, the anomaly processor104generates an interactive analysis interface to be provided via the interactive graphical user interface108, and the anomaly processor104modifies the output data stream based on interactions with the analysis interface. In some examples, the interaction processor106detects, based on the interactions with the interactive graphical user interface108, a Partial Pattern anomaly type. In some examples, the interaction processor106detects, based on the interactions with the analysis interface, a Partial Pattern anomaly type. In some examples, the interaction processor106displays, in the modified output data stream, a service shutdown message with the detected Partial Pattern anomaly type.

Referring toFIG. 1, in some examples, the interactive graphical user interface108further provides, in response to a selection of the second selectable option, an analysis interface to analyze the event pattern. For example, in response to an entry in the actionable menu408, the interaction processor106may prompt the interactive graphical user interface108to provide an analysis interface. In some examples, interaction processor106may receive feedback data indicative of a domain expert's interaction with the interactive graphical user interface108. For example, the domain expert may examine the event pattern, and perhaps tag it, indicating its underlying cause. Tagging the event pattern may catalogue them as a known event pattern.

FIG. 5is an example of an analysis interface for event patterns. In some examples, the anomaly processor104may generate the analysis interface to analyze system anomalies. In some examples, the pattern processor may generate the analysis interface to analyze event patterns. In some examples, a snapshot of the analysis interface may be displayed in response to a selection of the second selectable option, such as, for example, clicking a first pattern310illustrated inFIG. 3. Event patterns500are shown in the output data stream. Actionable menu504is shown including second selectable options. For example, a menu button may be provided to “Apply” changes, name and “Save this pattern”, select severity of a pattern, a clickable option to “Enable pattern anomaly”, “Delete this pattern”, and “Do nothing”. For example, the selected pattern represents an anomaly type, Partial Pattern, characterized by multiple events (e.g., set of 30) appearing repeatedly in the same time slot. The selected pattern may be detected as this coincidence may not be likely to be random. The bottom of the pattern investigation interface lists the pattern events502. The list of pattern events502may indicate, for example, that the 30 events likely correspond to a ‘service shutdown’ event. In some examples, tagging the pattern as “Service Shutdown” may automatically trigger an anomaly type, “Partial Pattern”.

FIG. 6is an example display of the example output data stream ofFIG. 3after anomaly and pattern interactions. For example, the output stream illustrated inFIG. 3may be modified based on the interactions described with reference toFIGS. 4 and 5. For example, interaction processor106may receive feedback data that the system anomaly associated with tag302and the event pattern310(inFIG. 3) have been tagged. Such feedback data may be provided to the anomaly processor104and the pattern processor. Based on the feedback data, two new partial pattern system anomalies may be detected, each corresponding to two instances where the “Service Shutdown” event patterns appeared partially. For example, system anomaly602may be identified, and pop-up604may be displayed, with information related to the system anomaly602. For example, as illustrated inFIG. 6, the selected system anomaly602occurred on “2013-06-07” at “01:15:00”. Also, for example, the anomaly type may be indicated as a “partial pattern”. Other changes inFIG. 6are a result of a preference given to the “Variety of Events” system anomaly due to tagging the “Rogue Report” system anomaly, and of deleting a flood of event system anomaly (not illustrated herein). Such interactions may re-evaluate anomaly scores for the system anomalies. For example, an original anomaly score606may be marked behind the modified anomaly score608. Also, for example, word cloud610indicates “queuedtoc”, “culture”, and “neutral” as relevant words. This is different from the word cloud306inFIG. 3. Patterns612are also displayed based on the interactions described with reference toFIGS. 4 and 5.

In some examples, the anomaly processor104may detect future system anomalies based on a previously detected event pattern. For example, by identifying and defining event patterns, the anomaly processor104may identify system anomalies when the previously detected event patterns are broken or modified. In some examples, the pattern processor may detect future event patterns based on a previously detected system anomaly. For example, system anomalies associated with a low priority may aggregate to event patterns and may be flagged as high priority event patterns. In some examples, a system anomaly associated with a low priority may be identified based on an absence of a selection of a first selectable option associated with the system anomaly.

As described herein, the interaction processor106processes interactions of a domain expert with the interactive graphical user interface108based on an explicit tagging of a system anomaly or an event pattern, and also based on a passing interest based on a selection of a particular system anomaly or event pattern. Such feedback data may enrich the input data, enable detection of more refined system anomalies and event patterns, and reprioritize the displayed information on the interactive graphical user interface108. The analytic tools, including pattern processor and anomaly processor104, may feed data to each other, and utilize each other to continuously enrich the information provided by the interaction processor106.

FIG. 7is a block diagram illustrating an example of a processing system700for implementing the system100for interactive detection of system anomalies. Processing system700may include a processor702, a memory704, input devices712, output devices714, and interactive graphical user interfaces716communicatively linked to the input devices712and the output devices714. Processor702, memory704, input devices712, output devices714, and interactive graphical user interfaces716are coupled to each other through communication link (e.g., a bus).

Processor702may include a Central Processing Unit (CPU) or another suitable processor. In some examples, memory704stores machine readable instructions executed by processor702for operating processing system700. Memory704may include any suitable combination of volatile and/or non-volatile memory, such as combinations of Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, and/or other suitable memory.

Memory704also stores instructions to be executed by processor702including instructions for a data processor706, an anomaly processor708, and an interaction processor710. In some examples, data processor706, anomaly processor708, and interaction processor710, include data processor102, anomaly processor104, and interaction processor106, respectively, as previously described and illustrated with reference toFIG. 1.

Processor702executes instructions of data processor706to receive input data718related to a series of events and telemetry measurements. The input data718may be data related to a series of events and telemetry measurements. In some examples, the input data718may be a stream of log messages. In some examples, raw input data718may comprise log messages, and may be received via the processing system700, and a data processor706may process the input data718to generate structured log data.

Processor702executes instructions of anomaly processor708to detect presence of a system anomaly in the input data718, the system anomaly indicative of a rare situation that is distant from a norm of a distribution based on the series of events and telemetry measurements. In some examples, processor702executes instructions of a pattern processor to detect presence of an event pattern in the input data. In some examples, processor702executes instructions of an anomaly processor708to generate an output data stream based on the presence of the system anomaly and/or the event pattern.

In some examples, each event in the series of events may be associated with a time, and processor702executes instructions of an evaluator (not shown in the figure) to determine, for a time interval, at least one of an anomaly intensity, an anomaly score, an anomaly fingerprint, and a fingerprint matching function. In some examples, processor702executes instructions of the anomaly processor708to detect a presence of a system anomaly based on the anomaly fingerprint, and the fingerprint matching function.

In some examples, processor702executes instructions of an evaluator to determine, for the time interval, anomaly intensities and the anomaly score, and where each anomaly intensity may be transformed, with respect to a distribution of anomaly intensities of the same anomaly type in reference time-slots, based on a distinctive residual rarity extremity score, into comparable, additive, and distinctive anomaly intensity scores that may be combined to determine the anomaly score.

In some examples, each event in the series of events is associated with an event type, a time, and zero or more measurement values, and processor702executes instructions of an evaluator to determine, for each event type, an anomaly intensity amount for an anomaly type from events in the time interval, where for each anomaly type, the anomaly intensity amounts for different event types may be combined to determine an anomaly intensity and an anomaly fingerprint.

In some examples, each event may be a signal, and the anomaly intensity in the time interval may be one of a maximal signal value per signal type, a range of signal values per signal type, and a value extremity score.

In some examples, processor702executes instructions of a pattern processor (not shown in the figure) to detect future event patterns based on at least one of the feedback data and detected system anomalies.

In some examples, processor702executes instructions of an evaluator to determine, for each time interval for an anomaly type, incomparable anomaly intensity amounts, where each incomparable anomaly intensity amount may be transformed with respect to the distribution of associated incomparable anomaly intensity amounts in reference time intervals, based on a distinctive residual rarity extremity score, into comparable, additive, and distinctive anomaly intensity amounts.

In some examples, processor702executes instructions of the anomaly processor708to generate an interactive analysis interface for system anomalies to be provided via the interactive graphical user interfaces716. In some examples, processor702executes instructions of the anomaly processor708to modify the output data stream based on interactions with the analysis interface. In some examples, processor702executes instructions of a pattern processor to generate an interactive analysis interface for event patterns to be provided via the interactive graphical user interfaces716.

In some examples, processor702executes instructions of an interaction processor710to display, via interactive graphical user interfaces716, an output data stream based on the presence of the system anomaly and/or the event pattern. In some examples, processor702executes instructions of an interaction processor710to receive, via interactive graphical user interfaces716, feedback data associated with the output data stream. In some examples, processor702executes instructions of an interaction processor710to provide the feedback data to the anomaly processor for operations analytics based on the feedback data.

In some examples, processor702executes instructions of an interaction processor710to identify selection of an anomaly fingerprint, and processor702executes instructions of an evaluator to compute a fingerprint matching score for the anomaly fingerprint in a second time interval, to determine presence or absence of similar system anomalies in the second time interval, the fingerprint matching score computed based on a correlation between the anomaly fingerprint and anomaly intensity amounts in the second time interval.

In some examples, processor702executes instructions of the anomaly processor708to detect, based on the interactions with the analysis interface, a system anomaly associated with a Partial Pattern anomaly type, and executes instructions of an interaction processor710to display, in the modified output data stream, a service shutdown message with the detected system anomaly.

In some examples, processor702executes instructions of an interaction processor710to display the output data stream, including a first selectable option associated with the system anomaly, and/or a second selectable option associated with the event pattern, receive feedback data associated with the first and/or second selectable options, and provide the feedback data to the anomaly processor708. In some examples, processor702executes instructions of an interaction processor710to further provide, in response to a selection of the first selectable option, a pop-up card with information related to the system anomaly. In some examples, processor702executes instructions of an interaction processor710to further provide, in response to a selection of the first selectable option, an analysis interface to analyze the system anomaly. In some examples, processor702executes instructions of an interaction processor710to further provide, in response to a selection of the second selectable option, an analysis interface to analyze the event pattern. In some examples, processor702executes instructions of an interaction processor710to display a word cloud, the word cloud highlighting words that appear in log messages associated with the system anomaly more than in the rest of the log messages.

Input devices712include a keyboard, mouse, data ports, and/or other suitable devices for inputting information into processing system200. In some examples, input devices712are used to by the interaction processor710to interact with the user. Output devices714include a monitor, speakers, data ports, and/or other suitable devices for outputting information from processing system700. In some examples, output devices714are used to provide interactive graphical user interfaces716.

FIG. 8is a block diagram illustrating an example of a computer readable medium for interactive detection of system anomalies. Processing system800may include a processor802, a computer readable medium812, a data processor804, an anomaly processor806, an interaction processor808, and an interactive graphical user interface810. Processor802, computer readable medium812, data processor804, anomaly processor806, interaction processor808, and interactive graphical user interface810, are coupled to each other through communication link (e.g., a bus).

Processor802executes instructions included in the computer readable medium812. Computer readable medium812may include receive instructions814of a data processor804to receive input data related to a series of events and telemetry measurements. Computer readable medium812may include detect instructions816of an anomaly processor806to detect system anomalies in the input data. In some examples, computer readable medium812may include detect instructions816of a pattern processor to detect event patterns.

Computer readable medium812may include generate instructions818of an interaction processor808to generate an output data stream based on detected system anomalies. Computer readable medium812may include display instructions820of an interaction processor808to display the output data stream via an interactive graphical user interface810. In some examples, computer readable medium812may include feedback data receipt instructions of an interaction processor808to receive feedback data associated with the output data stream.

In some examples, computer readable medium812may include aggregate instructions of an anomaly processor806to aggregate heterogeneous system anomalies detected from heterogeneous input data, where the input data may include event streams, performance metrics, log messages, and event patterns.

In some examples, computer readable medium812may include instructions of an interaction processor808to display the output data stream, including a first selectable option associated with the system anomaly, and/or a second selectable option associated with the event pattern, receive feedback data associated with the first and/or second selectable options, and provide the feedback data to the anomaly processor806.

In some examples, computer readable medium812may include instructions of an interaction processor808to further provide, in response to a selection of the first selectable option, a pop-up card with information related to the system anomaly. In some examples, computer readable medium812may include instructions of an interaction processor808to further provide, in response to a selection of the first selectable option, an analysis interface to analyze the system anomaly. In some examples, computer readable medium812may include instructions of an interaction processor808to further provide, in response to a selection of the second selectable option, an analysis interface to analyze the event pattern. In some examples, computer readable medium812may include display instructions820of an interaction processor808to display a word cloud, the word cloud highlighting words that appear in log messages associated with the system anomaly more than in the rest of the log messages.

As used herein, a “computer readable medium” may be any electronic, magnetic, optical, or other physical storage apparatus to contain or store information such as executable instructions, data, and the like. For example, any computer readable storage medium described herein may be any of Random Access Memory (RAM), volatile memory, non-volatile memory, flash memory, a storage drive (e.g., a hard drive), a solid state drive, and the like, or a combination thereof. For example, the computer readable medium812can include one of or multiple different forms of memory including semiconductor memory devices such as dynamic or static random access memories (DRAMs or SRAMs), erasable and programmable read-only memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs) and flash memories; magnetic disks such as fixed, floppy and removable disks; other magnetic media including tape; optical media such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices.

As described herein, various components of the processing system400are identified and refer to a combination of hardware and programming configured to perform a designated function. As illustrated inFIG. 8, the programming may be processor executable instructions stored on tangible computer readable medium812, and the hardware may include processor802for executing those instructions. Thus, computer readable medium812may store program instructions that, when executed by processor802, implement the various components of the processing system800.

Such computer readable storage medium or media is (are) considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any manufactured single component or multiple components. The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.

Computer readable medium812may be any of a number of memory components capable of storing instructions that can be executed by processor802. Computer readable medium812may be non-transitory in the sense that it does not encompass a transitory signal but instead is made up of one or more memory components configured to store the relevant instructions. Computer readable medium812may be implemented in a single device or distributed across devices. Likewise, processor802represents any number of processors capable of executing instructions stored by computer readable medium812. Processor802may be integrated in a single device or distributed across devices. Further, computer readable medium812may be fully or partially integrated in the same device as processor802(as illustrated), or it may be separate but accessible to that device and processor802. In some examples, computer readable medium812may be a machine-readable storage medium.

FIG. 9is a flow diagram illustrating an example of a method for interactive detection of system anomalies. At900, an output data stream may be generated based on system anomalies detected in input data, the system anomalies indicative of rare events and events distant from a norm of a distribution of the series of events. At902, the output data stream may be displayed via an interactive graphical user interface, the output data stream including an attribute associated with the output data stream. At904, feedback data indicative of selection of a system anomaly may be received from the interactive graphical user interface. At906, the feedback data may be processed to modify the output data stream. At908, an interactive analysis interface may be provided, via the interactive graphical user interface, for operations analytics based on the selected system anomaly.

In some examples, the attribute associated with the output data stream may include an anomaly intensity, an anomaly score, an anomaly Fingerprint, a fingerprint matching function, event patterns, a word cloud, an anomaly type, a service message associated with a selected system anomaly, an anomaly intensity for events in a time interval, an event count extremity factor, and an event type rarity factor.

In some examples, each event in the series of events may be associated with a time, and the method may include determining, for a time interval, at least one of an anomaly intensity, an anomaly score, an anomaly fingerprint, a fingerprint matching function, and event patterns. In some examples, the method may include detecting system anomalies based on the anomaly fingerprint, and the fingerprint matching function.

In some examples, each system anomaly may be associated with a time, and the method may include determining, for a time interval, at least one of an anomaly intensity, an anomaly score, an anomaly fingerprint, and a fingerprint matching function.

In some examples, the method may include detecting a presence of a system anomaly based on the anomaly fingerprint, and the fingerprint matching function.

In some examples, the method may include determining, for the time interval, anomaly intensities and the anomaly score, and where each anomaly intensity may be transformed, with respect to a distribution of anomaly intensities of the same anomaly type in reference time-slots, based on a distinctive residual rarity extremity score, into comparable, additive, and distinctive anomaly intensity scores that may be combined to determine the anomaly score.

In some examples, each event in the series of events is associated with an event type, a time, and zero or more measurement values, and the method may include determining, for each event type, an anomaly intensity amount for an anomaly type from events in the time interval, where for each anomaly type, the anomaly intensity amounts for different event types may be combined to determine an anomaly intensity and an anomaly fingerprint.

In some examples, the method may include determining, for each time interval for an anomaly type, incomparable anomaly intensity amounts, where each incomparable anomaly intensity amount may be transformed with respect to the distribution of associated incomparable anomaly intensity amounts in reference time intervals, based on a distinctive residual rarity extremity score, into comparable, additive, and distinctive anomaly intensity amounts.

In some examples, the anomaly type may include a Flood of Events, where the anomaly intensity amount is an event count; a Variety of Events, where the anomaly intensity amount is an event occurrence indicator; a Flood of Rare Events, where the anomaly intensity amount is a product of an event count extremity factor, and an event-type rarity factor; and a Flood of Extreme Signals, where the anomaly intensity amount is a maximal signal value per time interval transformed based on a distinctive residual rarity extremity score.

In some examples, the method may include identifying selection of an anomaly fingerprint, and where a fingerprint matching score for the anomaly fingerprint is computed in a second time interval to determine presence or absence of similar system anomalies in the second time interval, where the fingerprint matching score is computed based on a correlation between the anomaly fingerprint and anomaly intensity amounts in the second time interval.

In some examples, the method may include generating an interactive analysis interface to be provided via the interactive graphical user interface, and modifying the output data stream based on interactions with the analysis interface. In some examples, the method may include detecting, based on the interactions with the analysis interface, a system anomaly associated with a Partial Pattern anomaly type, and displaying, in the modified output data stream, a service shutdown message with the detected system anomaly. In some examples, the analysis interface may be an anomaly analysis interface to analyze the system anomaly. In some examples, the analysis interface may be a pattern analysis interface to analyze the event pattern.

In some examples, the feedback data may include indication of a selection of a system anomaly, and based on the feedback data the interaction processor further provides, via the graphical user interface, a pop-up card with information related to the selected system anomaly.

In some examples, the feedback data may include the anomaly score, a modified anomaly score, an anomaly fingerprint, and acceptance or rejection of an anomaly finger matching result.

In some examples, the method may include displaying a word cloud, the word cloud highlighting words that appear in log messages associated with the system anomaly. For example, key terms may appear in log messages associated with the system anomaly more frequently than in the rest of the log messages. Accordingly, such key terms may be highlighted in the word cloud. Highlighting may be achieved via a distinctive font, font size, color, and so forth. In some examples, term scores may be determined for key terms, the term scores based on a modified inverse domain frequency. In some examples, the modified inverse domain frequency may be based on an information gain or a Kullback-Liebler Divergence.

In some examples, the method may include aggregating heterogeneous system anomalies detected from heterogeneous input data, where the input data may include event streams, performance metrics, log messages, and event patterns.

Examples of the disclosure provide a generalized system for interactive detection of system anomalies. The generalized system provides for analyzing and managing operations data. The purpose of the system may be to facilitate managing operations of complex and distributed systems, making sure that they are continuously performing at their best, and whenever there may be a problem, to be able to resolve it quickly and save the problem fingerprint for future prevention and fast resolution. As described herein, data streams of various types streams into the system which analyses it automatically to provide an interface where data anomalies may be constantly prioritized so that the highest recent system anomalies may be visualized prominently.

Although the techniques described herein enable automatic detection of system anomalies (e.g., without a query), such automatic detection techniques may be combined with known system anomalies, and/or query-based detection of system anomalies to form a hybrid system.

Although specific examples have been illustrated and described herein, the examples illustrate applications to any input data. Accordingly, there may be a variety of alternate and/or equivalent implementations that may be substituted for the specific examples shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific examples discussed herein. Therefore, it is intended that this disclosure be limited only by the claims and the equivalents thereof.