Patent ID: 12198070

DETAILED DESCRIPTION

FIG.1depicts a diagram100of an example of a system for developing preventive operations procedures for response to potential outages. The diagram100includes a computer-readable medium (CRM)102, a data collection engine104coupled to the CRM102, a metrics datastore106coupled to the CRM102, a system behavior detection engine108coupled to the CRM102, an outage prediction engine118coupled to the CRM102, and a preventive operations engine120coupled to the CRM102. The system behavior detection engine108includes a univariate anomaly detection engine110, a multivariate anomaly detection engine112, and an anomaly severity scaling engine114.

The CRM102and other computer readable mediums discussed in this paper are intended to include all mediums that are statutory (e.g., in the United States, under 35 U.S.C. 101), and to specifically exclude all mediums that are non-statutory in nature to the extent that the exclusion is necessary for a claim that includes the computer-readable medium to be valid. Known statutory computer-readable mediums include hardware (e.g., registers, random access memory (RAM), non-volatile (NV) storage, to name a few), but may or may not be limited to hardware.

The CRM102and other computer readable mediums discussed in this paper are intended to represent a variety of potentially applicable technologies. For example, the CRM102can be used to form a network or part of a network. Where two components are co-located on a device, the CRM102can include a bus or other data conduit or plane. Where a first component is co-located on one device and a second component is located on a different device, the CRM102can include a wireless or wired back-end network or LAN. The CRM102can also encompass a relevant portion of a WAN or other network, if applicable.

The devices, systems, and computer-readable mediums described in this paper can be implemented as a computer system or parts of a computer system or a plurality of computer systems. In general, a computer system will include a processor, memory, non-volatile storage, and an interface. A typical computer system will usually include at least a processor, memory, and a device (e.g., a bus) coupling the memory to the processor. The processor can be, for example, a general-purpose central processing unit (CPU), such as a microprocessor, or a special-purpose processor, such as a microcontroller.

The memory can include, by way of example but not limitation, random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). The memory can be local, remote, or distributed. The bus can also couple the processor to non-volatile storage. The non-volatile storage is often a magnetic floppy or hard disk, a magnetic-optical disk, an optical disk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Some of this data is often written, by a direct memory access process, into memory during execution of software on the computer system. The non-volatile storage can be local, remote, or distributed. The non-volatile storage is optional because systems can be created with all applicable data available in memory.

Software is typically stored in the non-volatile storage. Indeed, for large programs, it may not even be possible to store the entire program in the memory. Nevertheless, it should be understood that for software to run, if necessary, it is moved to a computer-readable location appropriate for processing, and for illustrative purposes, that location is referred to as the memory in this paper. Even when software is moved to the memory for execution, the processor will typically make use of hardware registers to store values associated with the software, and local cache that, ideally, serves to speed up execution. As used herein, a software program is assumed to be stored at an applicable known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as “implemented in a computer-readable storage medium.” A processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.

In one example of operation, a computer system can be controlled by operating system software, which is a software program that includes a file management system, such as a disk operating system. One example of operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash., and their associated file management systems. Another example of operating system software with its associated file management system software is the Linux operating system and its associated file management system. The file management system is typically stored in the non-volatile storage and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile storage.

The bus can also couple the processor to the interface. The interface can include one or more input and/or output (I/O) devices. Depending upon implementation-specific or other considerations, the I/O devices can include, by way of example but not limitation, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other I/O devices, including a display device. The display device can include, by way of example but not limitation, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device. The interface can include one or more of a modem or network interface. It will be appreciated that a modem or network interface can be considered to be part of the computer system. The interface can include an analog modem, ISDN modem, cable modem, token ring interface, satellite transmission interface (e.g. “direct PC”), or other interfaces for coupling a computer system to other computer systems. Interfaces enable computer systems and other devices to be coupled together in a network.

The computer systems can be compatible with or implemented as part of or through a cloud-based computing system. As used in this paper, a cloud-based computing system is a system that provides virtualized computing resources, software and/or information to end user devices. The computing resources, software and/or information can be virtualized by maintaining centralized services and resources that the edge devices can access over a communication interface, such as a network. “Cloud” may be a marketing term and for the purposes of this paper can include any of the networks described herein. The cloud-based computing system can involve a subscription for services or use a utility pricing model. Users can access the protocols of the cloud-based computing system through a web browser or other container application located on their end user device.

A computer system can be implemented as an engine, as part of an engine or through multiple engines. As used in this paper, an engine includes one or more processors or a portion thereof. A portion of one or more processors can include some portion of hardware less than all of the hardware comprising any given one or more processors, such as a subset of registers, the portion of the processor dedicated to one or more threads of a multi-threaded processor, a time slice during which the processor is wholly or partially dedicated to carrying out part of the engine's functionality, or the like. As such, a first engine and a second engine can have one or more dedicated processors or a first engine and a second engine can share one or more processors with one another or other engines. Depending upon implementation-specific or other considerations, an engine can be centralized or its functionality distributed. An engine can include hardware, firmware, or software embodied in a computer-readable medium for execution by the processor that is a component of the engine. The processor transforms data into new data using implemented data structures and methods, such as is described with reference to the figures in this paper.

The engines described in this paper, or the engines through which the systems and devices described in this paper can be implemented, can be cloud-based engines. As used in this paper, a cloud-based engine is an engine that can run applications and/or functionalities using a cloud-based computing system. All or portions of the applications and/or functionalities can be distributed across multiple computing devices and need not be restricted to only one computing device. In some embodiments, the cloud-based engines can execute functionalities and/or modules that end users access through a web browser or container application without having the functionalities and/or modules installed locally on the end-users' computing devices.

As used in this paper, datastores are intended to include repositories having any applicable organization of data, including tables, comma-separated values (CSV) files, traditional databases (e.g., SQL), or other applicable known or convenient organizational formats. Datastores can be implemented, for example, as software embodied in a physical computer-readable medium on a specific-purpose machine, in firmware, in hardware, in a combination thereof, or in an applicable known or convenient device or system. Datastore-associated components, such as database interfaces, can be considered “part of” a datastore, part of some other system component, or a combination thereof, though the physical location and other characteristics of datastore-associated components is not critical for an understanding of the techniques described in this paper.

A database management system (DBMS) can be used to manage a datastore. In such a case, the DBMS may be thought of as part of the datastore, as part of a server, and/or as a separate system. A DBMS is typically implemented as an engine that controls organization, storage, management, and retrieval of data in a database. DBMSs frequently provide the ability to query, backup and replicate, enforce rules, provide security, do computation, perform change and access logging, and automate optimization. Examples of DBMSs include Alpha Five, DataEase, Oracle database, IBM DB2, Adaptive Server Enterprise, FileMaker, Firebird, Ingres, Informix, Mark Logic, Microsoft Access, InterSystems Cache, Microsoft SQL Server, Microsoft Visual FoxPro, MonetDB, MySQL, PostgreSQL, Progress, SQLite, Teradata, CSQL, OpenLink Virtuoso, Daffodil DB, and OpenOffice.org Base, to name several.

Database servers can store databases, as well as the DBMS and related engines. Any of the repositories described in this paper could presumably be implemented as database servers. It should be noted that there are two logical views of data in a database, the logical (external) view and the physical (internal) view. In this paper, the logical view is generally assumed to be data found in a report, while the physical view is the data stored in a physical storage medium and available to a specifically programmed processor. With most DBMS implementations, there is one physical view and an almost unlimited number of logical views for the same data.

A DBMS typically includes a modeling language, data structure, database query language, and transaction mechanism. The modeling language is used to define the schema of each database in the DBMS, according to the database model, which may include a hierarchical model, network model, relational model, object model, or some other applicable known or convenient organization. An optimal structure may vary depending upon application requirements (e.g., speed, reliability, maintainability, scalability, and cost). One of the more common models in use today is the ad hoc model embedded in SQL. Data structures can include fields, records, files, objects, and any other applicable known or convenient structures for storing data. A database query language can enable users to query databases and can include report writers and security mechanisms to prevent unauthorized access. A database transaction mechanism ideally ensures data integrity, even during concurrent user accesses, with fault tolerance. DBMSs can also include a metadata repository; metadata is data that describes other data.

As used in this paper, a data structure is associated with a particular way of storing and organizing data in a computer so that it can be used efficiently within a given context. Data structures are generally based on the ability of a computer to fetch and store data at any place in its memory, specified by an address, a bit string that can be itself stored in memory and manipulated by the program. Thus, some data structures are based on computing the addresses of data items with arithmetic operations; while other data structures are based on storing addresses of data items within the structure itself. Many data structures use both principles, sometimes combined in non-trivial ways. The implementation of a data structure usually entails writing a set of procedures that create and manipulate instances of that structure. The datastores, described in this paper, can be cloud-based datastores. A cloudbased datastore is a datastore that is compatible with cloud-based computing systems and engines.

Returning to the example ofFIG.1, the data collection engine104is intended to represent an engine for collecting metrics for full-stack performance monitoring. In a specific implementation, data collection includes episodes collected before previous outages, which are used for learning behavior with respect to attributes during a training (or update) phase of a pre-outage behavior model (and those collected before critical anomalies during an inference phase). Episodes include a subset of events observed within a contextual time frame determined based on metric collection intervals. Each episode is a collection of metrics of a duration ‘d’ for attributes that describe a current system state.

The metrics datastore106is intended to contain metrics collected by the data collection engine104and also but not limited to represent data structures associated with episodes. The metrics datastore106and other datastores described in this paper can have a corresponding engine to create, read, update, or delete (CRUD) data structures. While not shown inFIG.1, these engines may or may not be described in association with other figures to illustrate relevant functionality. In a specific implementation, an episode is composed of its attributes along with those of related data collection engines over a duration of ‘t(*)-d’ hours.

FIG.2is a diagram200of a conceptual visualization of episodes before outages as signals.

FIG.3is a diagram300of a conceptual illustration of an episode tree data structure with a set of linked nodes. Assume, upon mining through a wide range of episode patterns before recorded outages with domain insight, it has been found an outage can follow either a severe rise or fall. A tree with infrastructure levels as internal nodes and time series corresponding to attributes as leaf nodes serves as a reference topology for entity pattern matching. Advantageously, this design is compatible with dynamically changing infrastructure components.

The diagram300includes a root (account) node302, a monitor type node312coupled to the root node302, a monitor group node314coupled to the root node302, a monitor group/type node316coupled to the root node302, an attribute element node322-0to an attribute element node322-n(collectively, the attribute elements322) coupled to the monitor type node312, a monitor group node324coupled to the monitor group node314, a monitor type node326coupled to the monitor group node314, a monitor group/type node332-0to a monitor group/type node332-n(collectively, the monitor group/type nodes332), an attribute element node334-0to an attribute element node334-n(collectively, the attribute element nodes334) coupled to the monitor type node326, an attribute element node342-0to an attribute element node342-n(collectively, the attribute element nodes342) coupled to the monitor group/type332-0, an attribute element node344-0to an attribute element node344-n(collectively, the attribute element nodes344) coupled to the monitor group/type node332-n, and an attribute element node346-0to an attribute element node346-n(collectively, the attribute element nodes346) coupled to the monitor group/type node316. The various different nodes are provided for illustrative purposes but, in general, a root node is for an account or a monitor type or a monitor group, purely subject to application, and a leaf node is for an attribute element; non-leaf subtrees can have a monitor type node, a monitor group node, and/or a monitor group/monitor type node as a parent node. The value ‘n’ of each set of attribute elements can vary with monitor type.

In a specific implementation, the episodes tree data structure can be characterized as a model layout in which each level (represented by the 10's place of the reference numerals, such as ‘314’ representing a node in the first level after the root and ‘322’ representing a node in the second level after the root) is a hub of patterns its children exhibited in previous confirmed outages. In this specific implementation, a leaf represents a functional unit (engine and datastore) that computes, updates, and holds slope statistics of abnormal behaviors.

FIG.4is a diagram400illustrating a situation that would lead to an outage if a leaf node lights up with a match. The diagram400includes a level402with functional constituents wherein “E” denotes an entity (Monitor type/group), “n” is the number of entities in each unique pattern “P” and there can be “z” such patterns within a given level; and an attribute404wherein “a” denotes each attribute collected from “d” instants back in time from “t”. During Update phase, “t(*)” is the time at start point of downtime and is of a reported high risk anomaly during Inference phase. “τ” is the equivalent using which possible “k” distributions are maintained. For the sake of simplicity, capacity notations are not mentioned here.

In a specific implementation, a time series (actual metric of attribute) is modelled by linear regression without intercept using least squares approximation. Slopes estimated are thereafter used to create and maintain normal distribution(s). Continuous learning over time is effectuated with ease as Gaussian mixture models based on Expectation Maximization are employed. Incoming episodes are pre-processed in the same way and checked to determine if the current slope fits into any of the pre-existing slope distributions (within 2 standard deviations). This current situation would lead to an outage if any of the leaf nodes light up with a match. Given a probable fit, the cost of current episode curve to superimpose onto corresponding cluster equivalent in slope space translates to time units. Let regression function of the befitting cluster equivalent be f(t) and that of current episode be g(t).

FIG.5is a diagram500illustrating the magnitude of rejection of g(t) from f(t) that gives the time left for mitigation, “Ts” named as Survival potential at elementary level. Mathematically, Ts=oprojf(t)g(t). Thereafter, an overall deadline can also be deduced by consolidating time units estimated for individual entities (survival potentials of all predicted attributes) with severity and the effect spread (vulnerability). It is essentially the minimum of survival potentials across infrastructure entities/groups upon scrutinizing capacity.

Referring once again to the example ofFIG.1, the system behavior detection engine108is intended to represent an engine that classifies current system behavior (system state) relative to known system behaviors and assesses risk associated with such system states. In a specific implementation, analytics are used to ensure an operations environment can maintain performance. Complexity grows with customization and scale, which in turn generates a spectrum of user behavior profiles. Outages may be rare and are not comparable among users, but downtime associated with each type of outage typically hits revenue directly and can be damaging to reputation. Advantageously, an administrator of an enterprise can be provided an intuitive starting point for resolving issues before they lead to a potential outage. Predicting if current anomalous state would transform into an outage can help in making informed decisions or trigger remediation. Outage-related information is any data useful to a human or artificial agent of a system at risk of outages for the purpose of understanding a cause and effect of an outage and how to prevent an outage in the future.

The system behavior detection engine108includes a univariate anomaly detection engine110, a multivariate anomaly detection engine112, and an anomaly severity scaling engine114. In a specific implementation, univariate anomaly detection is based on robust principal component analysis (RPCA) improved to handle seasonal patterns and yield explainable results. For example, the univariate anomaly detection engine110generates an expected value (representing normal behavior) by tracing back internal estimates when an anomaly is observed, which also aids in grouping anomalies by severity. RPCA is a modification of principal component analysis (PCA) which works well with respect to grossly corrupted observations.

Multivariate anomaly architecture is designed using concepts from online matrix sketching to provide a streaming ready explanation generating lifetime model, wherein a live sketch is maintained as a snapshot that serves as a reduced representation of normal system behavior patterns (which include seasonal nature of system states as well). For example, for a given set of attributes as a vector at a given time, the multivariate anomaly detection engine112detects deviations from a behavior model for a system (state) along with identifying contributing factors using directional evaluation as a part of projection analysis.

The anomaly severity scaling engine114categorizes anomalies by severity. In a specific implementation, severity includes three thresholds (e.g., anomalous, critical, and catastrophic), but a different number of thresholds can be used as is deemed appropriate for a specific implementation, which may or may not be based upon a risk matrix. In operation, an episode (e.g., a current or recent episode) is extracted to further predict odds of a subsequent potential outage.

In a specific implementation, a pre-marking technique called a Deflection Ratio is employed to mark hidden risk-associated attributes while updating an outage prediction model with episodes before confirmed outages. A duration of time (e.g., a few hours) after the outage may also be taken into consideration for this purpose. Drastic change in any of the attributes (before and after an outage) is captured by thresholding on rate of change in variance. In a specific implementation, high risk anomalies (e.g., critical or catastrophic anomalies), when detected, can generate an alert for a human or artificial agent in order to enable preparation, countermeasures, or the like.

The outage prediction engine118is intended to represent an engine that provides capacity powered pattern-based infrastructure entity predictions. The outage prediction engine is for outage prediction and further analytics upon detecting aberrations. In a specific implementation, utilizing an outage prediction model comprises three phases: a train phase, an update phase, and an inference phase. A model is created with infrastructure topology using the first confirmed outage, i.e., with individual attribute episodes before and after downtime during the train phase. Every next confirmed outage is used to update this created model, which internally keeps the normal distribution on slopes updated. When any of the attributes are identified as associated with risk of outage, an episode is collected to infer from the pre-outage behavior model, as shown inFIG.7, for prediction. In a real-time environment, every consecutive high-risk episode is used to infer from the outage prediction model until the issue gets resolved or there is a confirmed prediction.

Advantageously, the outage prediction engine118generates numerical predictions for each attribute with occurrence probability, plus proofs. The prediction is explanation ready as it has all the attributes that could contribute to the current situation leading to a potential outage along with the chance of contributing. This information can serve as a starting point of root cause analysis and significantly reduces the mean time to detect (MTTD), investigate (MTTI) and resolve (MTTR) an outage when evaluated in production decoupling auto-remediation. Apart from having a model limited to an account, a global model per each monitor type and also but not limited to similar users or infrastructures, can be maintained to predict outages for new users. Minor tweaks include down-toning capacity influence.

The preventive operations engine120is intended to represent an engine for alerting human or artificial agents of a predicted outage risk. Providing the time left for mitigation along with possible solutions can prepare for a predicted outage or, ideally, act to prevent the outage, thereby making the environment proactive. For example, if an outage is predicted it may be possible to spin up new servers.

FIG.6is a diagram600of an example of a system for building a robust server behavior model. The diagram600includes a server monitor engine602, a metrics datastore624coupled to the server monitor engine602, a data stream626coupled to the server monitor engine602, an episodes data store604coupled to the metrics datastore624, a pre-outage behavior modelling engine606coupled to the episodes datastore604, a univariate and multivariate anomaly detection and severity scaling engine608coupled to the data stream datastore626and the metrics datastore624, an outage prediction analytics engine614coupled to the univariate and multivariate anomaly detection and severity scaling engine608, an attribute predictions datastore616coupled to the outage prediction analytics engine614, an entity predictions datastore618coupled to the outage prediction analytics engine614, a proofs datastore620coupled to the outage prediction analytics engine614, and a preventive operations engine622coupled to the attribute predictions datastore616, the entity predictions datastore618, and the proofs datastore620. In the example ofFIG.6, the univariate and multivariate anomaly detection and severity scaling engine608includes a robust principal component analysis (RPCA) engine610and an online matrix sketching engine612.

The server monitor engine602is intended to represent a full-stack performance monitor of events associated with a server. Depending upon the implementation, a server can include an application server, database server, file server, mail server, web server, or some other hardware or service. In a specific implementation, the server monitor engine602detects events that are later used to understand resource usage patterns.

The episodes datastore604includes a subset of events used for outage prediction. Other events may or may not be used for other purposes, such as aiding a systems administrator to understand system operations or aiding in other aspects of administration or management. Thus, the episodes datastore604could be considered part of a more general events datastore (not shown).

The pre-outage behavior modelling engine606is intended to contain clustered collections of behaviors before various outages that occurred in the past and subengines for generating and updating a pre-outage behavior model (which can be characterized as a datastore). There can be multiple different thresholds of increasing risk (e.g., nominal, moderate, critical, catastrophic, or the like) that may trigger different responses. What is considered a key metric for episode utilization, whether it be CPU utilization, disk utilization, server load, traffic characteristics, syslog errors, event logs, or the like, will depend upon results yielded by the system behavior detection engine, which includes the univariate anomaly detection engine, multivariate anomaly detection engine, and the severity scaling engine coupled with the metrics datastore for ingestion of relevant data.

Univariate and multivariate anomaly detection engines function independently from an outage prediction engine (see, e.g.,FIG.1, univariate anomaly detection engine110, multivariate anomaly detection engine112, outage prediction engine118). They learn normal behavior so as to spot aberrations while the latter does not learn any normal system behavior and instead gains knowledge from abnormal patterns. The univariate and multivariate anomaly detection and severity scaling engine608is intended to represent an engine that determines whether a set of key metrics deviates from system's normal behavior and the pre-outage behavior modelling engine606generates the odds of an outage. In operation, the univariate and multivariate anomaly detection and severity scaling engine determine metrics that behave abnormally, after which the outage prediction model speculates if such stage could lead to a potential outage, relative to already learnt episodes.

The RPCA engine610generates an expected value (representing normal behavior) by tracing back internal estimates when an anomaly is observed, which also aids in grouping anomalies by severity. In a specific implementation, an idealized version of RPCA is used to recover a low-rank matrix L0from highly corrupted measurements M=L0+S0; the decomposition in low-rank and sparse matrices can be achieved by techniques such as Principal Component Pursuit (PCP), Stable PCP, Quantized PCP, Block based PCP, and Local PCP. In a specific implementation, Iteratively Reweighted Least Squares (IRLS) optimization is used; alternatives include, but are not necessarily limited to, Augmented Lagrange Multiplier Method (ALM), Alternating Direction Method (ADM), and Fast Alternating Minimization (FAM). Seasonality is ironed out by framing the initial matrix for RPCA with seasonal frequency, detected using Fourier analysis and validated using Singular Spectrum Analysis (most predominant frequency from the seasonality profile generated), as a dimension. This is limited to scenarios wherein any predominant valid frequency exists, otherwise the dimensions of initial matrix shall be those prime numbers whose product remain closest to total number of instants being modeled at any point of time.

The online matrix sketching engine612is intended to represent an unsupervised anomaly detection framework that can detect anomalies in a data stream. The models have further reinforcement learning capabilities. Due to the massive amount of data that can be generated by a server, it may be desirable to limit storage utilization. In a specific implementation, the online matrix sketching engine612maintains a relatively small set of orthogonal vectors that form a good approximate basis for observed data.

The outage prediction analytics engine614is intended to represent an engine that provides information to a human or artificial agent relative to but not limited to pattern-capacity combinations and proofs. The information can be provided in a report that draws from various types of data including numerical predictions for each attribute with occurrence probability (represented by the attribute predictions datastore616), capacity powered pattern based infrastructure entity predictions (represented by the entity predictions datastore618), and proofs (represented by the proofs datastore620). Zero configuration workflow intends to discount decisions like thresholds and severity definitions that could vary with context and infrastructure. In other words, the user will be able to benefit just by enabling this detection-prediction system without any pre-settings.

The preventive operations engine622is intended to represent an engine that prompts human or artificial agents to act to prevent an outage when the outage is predicted. The preventive operations are aspirational in the sense that the operations may not always result in the prevention of an outage, though they will prevent an outage, or at least ameliorate the harm of an outage, in at least some instances.