Patent ID: 12259969

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerous examples and details are set forth in order to provide an understanding of various embodiments. It will be evident, however, to one skilled in the art that certain embodiments can be practiced without some of these details or can be practiced with modifications or equivalents thereof

1. Overview

Embodiments of the present disclosure are directed to techniques for detecting anomalies in a distributed application based on process data (i.e., information regarding the software processes executed by the distributed application). This process data can include the hierarchy (i.e., tree) of processes created and run, statistics of the processes' use of resources such as memory and compute, the file system operations performed by each process, the network access operations performed by each process, and more. In one set of embodiments, the techniques described herein can detect anomalous behavior in a particular part (e.g., workload) or sub-part (e.g., pod) of the distributed application by comparing current process data collected for that part/sub-part with a reference model that captures the part/sub-part's historical (i.e., normal) process behavior. In a further set of embodiments, these techniques can detect anomalous behavior in a sub-part based on a comparison of that sub-part's current process data with the current process data of other sub-parts in the same replica set.

2. Example Environment and Solution Architecture

FIG.1is a simplified block diagram of an example environment100in which embodiments of the present disclosure may be implemented. Environment100includes a computing cluster102comprising a plurality of nodes104, which may be physical or virtual machines. Each node104is configured to run one or more parts (e.g., workloads) or sub-parts (e.g., pods) of a distributed application106. In the embodiments described herein, computing cluster102is assumed to be a Kubernetes cluster and distributed application106is assumed to be a Kubernetes application. However, in alternative embodiments computing cluster102may be any type of cluster and distributed application106may be any type of distributed application known in the art.

As shown, distributed application106is composed of two workloads: a first workload108and a second workload110. These workloads can be understood as separate parts of distributed application106that are designed to carry out functionally distinct tasks. For example, if distributed application106is an e-commerce application, workload108may implement a storefront service for presenting the application's user interface and workload110may implement a checkout service for handling the checkout process. Although only two workloads are depicted for purposes of illustration, distributed application116may include any number of workloads.

Each workload108/110includes a set of pods (112and114respectively), and each pod includes one or more containers (not shown). These containers are executable software packages that run the various software processes of distributed application106. The terms “pod” and “container” are used interchangeably in the present disclosure because in many cases a pod will include exactly one container.

In some embodiments, the pods in one or both of workloads108and110may form a replica set. The idea behind a replica set is to replicate the processing of a workload across multiple identical pods running on different cluster nodes for load balancing and fault tolerance. Accordingly, in the case where, e.g., pods112of workload108form a replica set, all of these pods will be replicas of each other (which means they should exhibit the same or similar behavior on average) and will be deployed on separate nodes of computing cluster102.

As mentioned in the Background section, existing techniques for detecting anomalous behavior in monolithic applications are generally inadequate for detecting anomalous behavior in distributed applications like application106ofFIG.1for various reasons. These reasons largely stem from the inability of these existing techniques to analyze the behavior of individual parts (e.g., workloads) and sub-parts (e.g., pods) of a distributed application, both against historical normal behavior and against the behavior of other identical parts/sub-parts, such as other pods in the same replica set.

To address these issues,FIG.2depicts an enhanced version of environment100(i.e., environment200) that includes a novel process data collector202running on nodes104of computing cluster102and a novel anomaly detection system204coupled with cluster102. Anomaly detection system204may be deployed on one or more computer systems in any public or private cloud, and in some cases may be deployed in the same data center where computing cluster102is deployed. Further, in some embodiments anomaly detection system204may be operated by a third-party distinct from the operator of computing cluster102and may be connected to a plurality of computing clusters of different organizations/customers to whom the third-party provides anomaly detection services.

At a high level, one or more pods206of process data collector202can monitor the software processes running within pods112and114of workloads108and110respectively and can collect data regarding these processes. For example, the collected data may include identifiers (i.e., process IDs), names, and other metadata of the processes created and running in each pod, including relationships between parent processes and child processes spawned/forked by the parent processes. The collected data may also include information regarding operations performed by each process, such as file system operations, network access operations, and so on. One or more other pods206of process data collector202can transmit this information to anomaly detection system204, where it can be stored in a structured manner in a process database (DB)208. These steps of collecting process data regarding distributed application106and storing the collected data in process DB208can be performed on a continuous basis throughout the runtime of application106.

Concurrently with the above, anomaly detection system204can carry out a two-stage process for enabling anomaly detection with respect to distributed application106. During the first stage (referred to as model building) a model builder module210of anomaly detection system204can retrieve process data stored in process DB208that pertains to a reference time period over which distributed application106is known or expected to have exhibited normal behavior. This reference time period may be specified by an administrator of computing cluster102or anomaly detection system204or determined via an automated mechanism.

Model builder module210can then use the retrieved process data to build, for each workload of distributed application106, a reference model that captures (or in other words, models) the normal process behavior of that workload, and each pod of the workload, over the reference time period. For example, in the example ofFIGS.1and2, model builder module210may build a first reference model for workload108and a second reference model for workload110. In one set of embodiments, each reference model may take the form of a machine learning (ML) anomaly detection model, such as a neural network classifier. In these embodiments, model builder module210may build each reference model by training the reference model on the retrieved process data using an ML training technique, such as backpropagation in the case of a neural network classifier. In another set of embodiments, each reference model make take the form of a rules-based anomaly detection model that includes one or more static rules defining normal or anomalous process behavior. In these embodiments, model builder module210may build each reference model by creating the rules in accordance with the behavior observed in the retrieved process data. For example, model builder module210may compile statistics regarding the processes and process operations seen during the reference time period and create rules indicating that any process behavior outside of that observed behavior should be deemed anomalous. In some cases, such statistics may also be used to build the ML-based model mentioned above. Module builder module210can thereafter store the created reference models in a model DB212.

Once appropriate reference models have been created for distributed application116, during the second stage (referred to as inference), an anomaly detector module214of anomaly detection system204can retrieve process data stored in process DB208that pertains to an inference time period over which distributed application106may or may not be exhibiting anomalous behavior. This inference time period may correspond to the current operation of distributed application106, such as the most recent X minutes or hours, or some prior time period for which anomaly detection is desired.

Anomaly detector module214can then analyze the retrieved process data to identify any anomalies during the inference time period. In one set of embodiments, this analysis can include applying the retrieved process data for each pod and workload of distributed application106to its corresponding reference model (as stored in model DB212), resulting in a prediction of whether the pod or workload is anomalous or not based on its historical normal behavior. In this way, anomaly detection system204can evaluate and detect anomalous behavior with respect to individual parts and sub-parts of distributed application106, rather than solely with respect to the application as a whole. If the reference model generates a prediction of “anomalous” for any pod or workload, anomaly detector module214can store a structured record of this outcome, including the details of the affected pod/workload and the process data indicating anomalous behavior, in an alert DB216. An alert service218can subsequently issue alerts regarding the anomaly records stored in alert DB216to, e.g., an administrator of computing cluster102or anomaly detection system204according to preferences set by the administrator.

In a further set of embodiments, the analysis performed by anomaly detector module214can include comparing the retrieved process data for each pod in each replica set of distributed application106to the retrieved process data for other pods in the same replica set. In this way, anomaly detection system204can evaluate and detect anomalies in a particular pod based on how similar its process behavior is to its replica set peers over the same time period, even if the pod's process behavior appears normal from a historical (i.e., reference model) perspective. If any pod in a replica set exhibits process behavior that deviates from the process behavior of other pods in the same replica set beyond some predetermined threshold, anomaly detector module214can flag that pod as being anomalous and generate a record in alert DB216as mentioned above. This threshold can differ depending on the particular process behavior and can be configured according to a policy set by the administrator of computing cluster102or anomaly detection system204.

The remaining sections of this disclosure provide additional details regarding the model building performed by model builder module210and the inference performed by anomaly detector module214according to various embodiments. It should be appreciated thatFIG.2is illustrative and not intended to limit embodiments of the present disclosure. For example, althoughFIG.2depicts a particular arrangement of modules in anomaly detection system204, other arrangements are possible (e.g., the functionality attributed to a particular module may be split into multiple modules, modules may be combined, etc.). One of ordinary skill in the art will recognize other variations, modifications, and alternatives.

3. Model Building

FIG.3depicts a flowchart300that may be executed by model builder module210of anomaly detection system204for building a reference model for a workload of distributed application106according to certain embodiments. Flowchart300may be repeated for each workload of the application.

Starting with step302, model builder module210can retrieve, from process DB208, process data records received from process data collector202over a reference time period, where the process data records include information regarding software processes running within the pods of the workload. In one set of embodiments, these process data records can be classified into four types: process tree records, process resource usage records, process file system operation records, and process network operation records. With respect to the first type, each process tree record can include metadata regarding a process created/initiated within a workload pod, such as an ID of the pod, an ID of the process, a name of the process, a time of creation, an ID of its parent process (if spawned/forked from another process), etc. With respect to the second type, each process resource usage record can include metadata regarding the resources consumed by a running process, such as memory, compute (e.g., CPU cycles), etc. With respect to the third type, each process file system operation record can identify a file system operation performed by a process and metadata regarding that operation. Examples of such file system operations include opening a file, reading a file, writing a file, changing attributes of a file (e.g., access mode, etc.), and so on. And with respect to the fourth type, each process network operation record can identify a network operation performed by a process and metadata regarding that operation. Examples of such network operations include opening a network socket, transmitting data, receiving data, and so on. For network traffic that is received by the process, the metadata can comprise information regarding the origin of the traffic, such as a source address, source port, source process/pod/workload (if received from another pod or workload), and so on. For network traffic that is transmitted by the process, the metadata can comprise information regarding the destination of the traffic, such as a destination address, destination port, destination process, destination process/pod/workload (if sent to another pod or workload), and so on.

At step304, model builder module210can transform the process data records retrieved at step302into a format suitable for model generation. The specific manner in which this pre-processing step is implemented will differ depending on the type of each process data record. For example, for process tree records, model builder module210may build a process tree or graph for each pod of the workload and/or for the workload as a whole that captures all of the processes running with the pod/workload and the parent-child relationships between the processes. For process resource usage records, model builder module210may compile statistics regarding the resources consumed by each process. For process file system operation records, model builder module210may compile statistics regarding the file system operations performed by each process, such as the number of file writes to a given file or type of file, the number of file reads of a given file or type of file, etc. And for process network operation records, module builder module210may compile statistics regarding the network operations performed by each process, such as the number of network sockets opened, the amount of traffic sent to a particular destination address or domain, the amount of traffic received from a particular source address or domain, etc.

At step306, model builder module210can use the transformed process data to build a reference model that captures the behavior of the workload and its pods over the reference time period. In the case where the reference model is an ML anomaly detection model, this step can comprise training the reference model on the transformed process data using an appropriate ML training technique. In the case where the reference model is a rules-based model, this step can comprises creating a set of rules that define the boundaries of normal process behavior for the workload and its pods in accordance with the transformed process data (e.g., a normal process tree, a normal number of file reads by process P1to files of type A, a normal amount of outbound network traffic from process P2to domain D, etc.).

Finally, at step308, model builder module210can store the reference model in a structured manner in model DB212and the flowchart can end. It should be noted that over time, the configuration of distributed application106may change in a manner that modifies the composition of pods in a given workload, thereby invalidating a previously generated reference model for the workload. In these cases, the workload's reference model can be rebuilt by repeating flowchart300using process data collected over a reference time period occurring after the change.

4. Anomaly Detection

FIG.4depicts a flowchart400that may be executed by anomaly detector module214of anomaly detection system204for detecting anomalies in distributed application106according to certain embodiments. Flowchart400assumes that reference models have been built for the workloads of distributed application106via flowchart300ofFIG.3.

Starting with step402, anomaly detector module214can retrieve, from process DB208, process data records received from process data collector202over an inference time period, where the process data records include information regarding software processes running within distributed application116's workloads/pods. As mentioned previously, this inference time period may be the most recent X minutes/hours/etc., or a historical time period specified by an administrator.

At step404, anomaly detector module214can transform the process data records retrieved at step402into a format suitable for comparison (or being provided as input) to the reference models created for distributed application116. For example, if the retrieved process data records include records regarding specific processes running with a pod Y of workload108, anomaly detector module214can transform those records into a process tree for pod Y. As another example, if the retrieved process data records include records regarding file system or network operations performed by specific processes running with pod Y, anomaly detector module214can transform those records into file system operation statistics or network operation statistics for pod Y.

At step406, anomaly detector module214can compare the transformed process data for each pod and each workload to the workload's reference model. Any process behavior observed in the pod or workload via the transformed process data that deviates from the reference model can be flagged as an anomaly. For example, the following is a non-exhaustive list of such discrepancies:A difference in process trees for a given pod/workloadA difference, beyond a configurable threshold, in the usage of one or more types of resources for certain processes of a given pod/workloadA difference, beyond a configurable threshold, in one or more file system operation statistics for certain processes of a given pod/workloadA difference, beyond a configurable threshold, in one or more network operation statistics for certain processes of a given pod/workloadA difference in destinations to which network traffic is sent by certain processes of a given pod/workload (e.g., a new destination address, a new destination process/pod/workload, etc. not seen in the reference model)A difference in sources from which network traffic is received by certain processes of a given pod/workload (e.g., a new source address, a new source process/pod/workload, etc. not seen in the reference model)

In addition, at step408, anomaly detector module214can compare the transformed process data for each pod that is part of a replica set with the transformed process data of other pods in the same replica set. Any process behavior observed in the pod that deviates from the process behavior of its replica set peers can be flagged as an anomaly. As with step406, these discrepancies can include differences in process trees, differences in per-process resource usage statistics, differences in per-process file system operation statistics, and/or differences in per-process network operation statistics and network traffic sources/destinations.

At step410, anomaly detector module214can determine whether an anomaly was detected via the comparisons made at either step406or408. If the answer is yes, anomaly detector module214can store structured records of the detected anomalies in alert DB216(step412), which will cause alert service218to issue alerts regarding those anomalies to, e.g., an administrator or computing cluster102or anomaly detection system204per administrator-defined policy. Finally, flowchart400can end after step412or if the answer at decision step410is no (i.e., no anomalies detected).

Certain embodiments described herein can employ various computer-implemented operations involving data stored in computer systems. For example, these operations can require physical manipulation of physical quantities—usually, though not necessarily, these quantities take the form of electrical or magnetic signals, where they (or representations of them) are capable of being stored, transferred, combined, compared, or otherwise manipulated. Such manipulations are often referred to in terms such as producing, identifying, determining, comparing, etc. Any operations described herein that form part of one or more embodiments can be useful machine operations.

Further, one or more embodiments can relate to a device or an apparatus for performing the foregoing operations. The apparatus can be specially constructed for specific required purposes, or it can be a generic computer system comprising one or more general purpose processors (e.g., Intel or AMD x86 processors) selectively activated or configured by program code stored in the computer system. In particular, various generic computer systems may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations. The various embodiments described herein can be practiced with other computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.

Yet further, one or more embodiments can be implemented as one or more computer programs or as one or more computer program modules embodied in one or more non-transitory computer readable storage media. The term non-transitory computer readable storage medium refers to any storage device, based on any existing or subsequently developed technology, that can store data and/or computer programs in a non-transitory state for access by a computer system. Examples of non-transitory computer readable media include a hard drive, network attached storage (NAS), read-only memory, random-access memory, flash-based nonvolatile memory (e.g., a flash memory card or a solid state disk), persistent memory, NVMe device, a CD (Compact Disc) (e.g., CD-ROM, CD-R, CD-RW, etc.), a DVD (Digital Versatile Disc), a magnetic tape, and other optical and non-optical data storage devices. The non-transitory computer readable media can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Finally, boundaries between various components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention(s). In general, structures and functionality presented as separate components in exemplary configurations can be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component can be implemented as separate components.

As used in the description herein and throughout the claims that follow, “a,” “an,” and “the” includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.

The above description illustrates various embodiments along with examples of how aspects of particular embodiments may be implemented. These examples and embodiments should not be deemed to be the only embodiments and are presented to illustrate the flexibility and advantages of particular embodiments as defined by the following claims. Other arrangements, embodiments, implementations, and equivalents can be employed without departing from the scope hereof as defined by the claims.