Administering incident pools for event and alert analysis

Administering incident pools including assigning an incident received from one or more components of the distributed processing system to a pool of incidents; assigning to each incident a particular combined minimum time for inclusion of the incident in the pool; in response to the pool closing, determining for each incident in the pool whether the incident has met its combined minimum time for inclusion in the pool; if the incident has been in the pool for its combined minimum time, including the incident in the closed pool; if the incident has not been in the pool for its combined minimum time, moving the incident from the closed pool to a next pool; applying incident suppression rules using the incidents assigned to the next pool; and applying incident creation rules to the incidents that were assigned to the next pool, while omitting any duplicate incidents caused by the assignment.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention is data processing, or, more specifically, methods, apparatuses, and computer program products for administering incident pools for event and alert analysis in a distributed processing system.

2. Description of Related Art

Modern distributed processing systems for intensive computing may have millions of devices with many processes running on each device all of which are capable of error and status reporting for automated error recovery, reporting to a systems administrator, and for other reasons. In many cases, in the case of an error for example, the sheer number of such error reports and status reports are so overwhelming that they cannot be handled in a meaningful manner. For example, a systems administrator receiving a hundred thousand error reports may be overwhelmed by the sheer number of such reports and therefore in the aggregate those reports become more and more unhelpful and irrelevant.

SUMMARY OF THE INVENTION

Methods, apparatuses, and computer program products are provided for administering incident pools for event and alert analysis in a distributed processing system. Embodiments include assigning, by the incident analyzer, an incident received from one or more components of the distributed processing system to a pool of incidents; assigning, by the incident analyzer, to each incident a particular combined minimum time for inclusion of the incident in the pool; in response to the pool closing, determining, by the incident analyzer, for each incident in the pool whether the incident has met its combined minimum time for inclusion in the pool; if the incident has been in the pool for its combined minimum time, including, by the incident analyzer, the incident in the closed pool; if the incident has not been in the pool for its combined minimum time, moving the incident from the closed pool to a next pool; applying, by the incident analyzer, incident suppression rules using the incidents assigned to the next pool; and applying, by the incident analyzer, incident creation rules to the incidents assigned to the next pool, while omitting any duplicate incidents caused by the assignment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary methods, apparatuses, and computer program products for administering incident pools for event and alert analysis and relevant alert delivery in a distributed processing system and according to embodiments of the present invention are described with reference to the accompanying drawings, beginning withFIG. 1.FIG. 1illustrates an exemplary system for administering incident pools for event and alert analysis and relevant alert delivery in a distributed processing system according to embodiments of the present invention. A distributed processing system is typically implemented as multiple autonomous or semi-autonomous computers that communicate through a computer network. In such example distributed processing systems, the computers often interact with each other in order to achieve a common goal. A computer program that runs in such an example distributed system is typically called a distributed program, and distributed programming is often used to describe the process of writing such programs.

In the example ofFIG. 1, the distributed processing system (101) is implemented as a parallel computer (100), non-volatile memory for the computer in the form of data storage device (118), an output device for the computer in the form of printer (120), and an input/output device for the computer in the form of computer terminal (122). The parallel computer (100) in the example ofFIG. 1also includes a plurality of compute nodes (102). Each compute node is an automated computing device composed of one or more computer processors, its own computer memory, and its own input/output functionality. The compute nodes (102) are coupled for data communications by several independent data communications networks including a high speed Ethernet network (174), a Joint Test Action Group (‘JTAG’) network (104), a tree network (106) which is optimized for collective operations, and a torus network (108) which is optimized for point to point operations. Tree network (106) is a data communications network that includes data communications links connected to the compute nodes so as to organize the compute nodes as a tree. Each data communications network is implemented with data communications links among a number of compute nodes (312), identified collectively as the compute nodes (102). The data communications links provide data communications for parallel operations among the compute nodes of the parallel computer.

In addition to compute nodes, computer (100) includes input/output (‘I/O’) nodes (110,114) coupled to compute nodes (102) through one of the data communications networks (174). The I/O nodes (110,114) provide I/O services between compute nodes (102) and I/O devices (118,120,122). I/O nodes (110,114) are connected for data communications through local area network (‘LAN’) (130). Computer (100) also includes a service node (116) coupled to the compute nodes through one of the networks (104). Service node (116) provides service common to pluralities of compute nodes, loading programs into the compute nodes, starting program execution on the compute nodes, retrieving results of program operations on the computer nodes, and so on. Service node (116) runs an event and alert analysis module (124) and communicates with users (128) through a service application interface (126) that runs on computer terminal (122).

Many of the components of the distributed processing system ofFIG. 1, that is the devices of the distributed processing system or processes running on the devices of the distributed processing system ofFIG. 1are capable of some form of error or status reporting through events and many of such components are also capable of receiving alerts in response to one or more of such events. Often in distributed processing systems useful according to embodiments of the present invention hundreds of thousands or millions of components may provide incidents, often in the form of events or receive alerts.

An incident is a generic term used in this specification to mean an identification or notification of a particular occurrence on a component of a distributed processing system such as events described below, a refined identification of an occurrence often based on events such as an alert described below, or other notifications as will occur to those of skill in the art.

Incidents are administered in pools for event and alert analysis according to embodiments of the present invention. A pool of incidents is a collection of incidents organized by the time of either their occurrence, by the time they are logged in an incident queue, included in the pool, or other time as will occur to those of skill in the art.

The service node (116) ofFIG. 1has installed upon it an event and alert analysis module (124) that includes at least two incident analyzers implemented as events analyzers and alerts analyzers capable of administering incident pools for event and alert analysis in a distributed processing system according to embodiments of the present invention. The event and alert analysis module (124) is implemented as automated computing machinery capable of assigning, by the incident analyzer, an incident received from one or more components of the distributed processing system to a pool of incidents; assigning, by the incident analyzer, to each incident a particular combined minimum time for inclusion of the incident in the pool; in response to the pool closing, determining, by the incident analyzer, for each incident in the pool whether the incident has met its combined minimum time for inclusion in the pool; if the incident has been in the pool for its combined minimum time, including, by the incident analyzer, the incident in the closed pool; if the incident has not been in the pool for its combined minimum time, moving the incident from the closed pool to a next pool; applying, by the incident analyzer, incident suppression rules using the incidents assigned to the next pool; and applying incident creation rules to the incidents assigned to the next pool, while omitting any duplicate incidents caused by the assignment.

For example, when processing an event in an event pool, an event analyzer may for various reasons move the event forward to a next pool so that the moved event can influence analysis in the next pool. Suppressing alerts generated by the event in the prior pool may be suppressed according to embodiments of the present invention to keep duplicated analysis from being performed in the next pool. By only applying the suppression rules to the events that were moved to the next pool, the event in the next pool is able to combine with additional events to create alerts. That is, administering incident pools for event and alert analysis according to embodiments of the present invention may eliminate duplicate alerts and thus increase processing capabilities of incident analyzers.

In some embodiments the unsuppressed alerts are transmitted to one or more components of the distributed processing system. One such component may be a terminal (122) for display to a systems administrator. Other components may include a component that generated an event, a component for error reporting, a component for automated error recovery or any other component that will occur to those of skill in the art.

The event and alert module (124) ofFIG. 1allows the number of incidents occurring such as events received and alerts produced at any given time to be less overwhelming to a systems administrator (128) attempting to identify a problem or occurrence in the distributed processing system. Administering incident pools for event and alert analysis and administering incident pools for event and alert analysis in a distributed processing system according to embodiments of the present invention provide alerts that are more meaningful to a user in determining how to administer the functions and errors associated with a distributed processing system.

The arrangement of nodes, networks, and I/O devices making up the exemplary distributed processing system illustrated inFIG. 1are for explanation only, not for limitation of the present invention. Distributed data processing systems capable of administering incident pools for event and alert analysis and relevant alert delivery according to embodiments of the present invention may include additional nodes, networks, devices, and architectures, not shown inFIG. 1, as will occur to those of skill in the art. The parallel computer (100) in the example ofFIG. 1includes sixteen compute nodes (102); parallel computers capable of relevant alert delivery according to embodiments of the present invention sometimes include thousands of compute nodes. In addition to Ethernet and JTAG, networks in such data processing systems may support many data communications protocols including for example TCP (Transmission Control Protocol), IP (Internet Protocol), and others as will occur to those of skill in the art. Various embodiments of the present invention may be implemented on a variety of hardware platforms in addition to those illustrated inFIG. 1.

Administering incident pools for event and alert analysis and administering incident pools for event and alert analysis in a distributed processing system in accordance with the present invention is generally implemented with computers, that is, with automated computing machinery. In the system ofFIG. 1, for example, all the service nodes, I/O nodes, compute nodes, of the parallel computer are implemented to some extent at least as computers. For further explanation, therefore,FIG. 2sets forth a block diagram of automated computing machinery comprising an exemplary computer (152) useful in relevant alert delivery according to embodiments of the present invention. The computer (152) ofFIG. 2includes at least one computer processor (156) or ‘CPU’ as well as random access memory (168) (‘RAM’) which is connected through a high speed memory bus (166) and bus adapter (158) to processor (156) and to other components of the computer (152) and through an expansion bus to adapters for communications with other components of a distributed processing system (101).

Stored in RAM (168) is an event and alert analysis module (124), a module of automated computing machinery for relevant alert delivery according to embodiments of the present invention. The event and alert analysis module (124) includes two incident analyzers according to embodiments of the present invention. The incident analyzers include an event analyzer (208) and an alert analyzer (218). The event analyzer ofFIG. 2is a module of automated computing machinery capable of identifying alerts in dependence upon received events. That is, event analyzers typically receive events and produce alerts. In many embodiments, a plurality of event analyzers are implemented in parallel. Often such event analyzers are assigned to a particular pool of events and may be focused on events from a particular component or caused by a particular occurrence to produce a more concise set of alerts.

The alert analyzer ofFIG. 2is a module of automated computing machinery capable of identifying alerts for transmission from events and other alerts, identifying additional alerts for transmission, and suppressing unnecessary, irrelevant, or otherwise unwanted alerts identified by the event analyzer. That is, alert analyzers typically receive alerts and events and produce or forward alerts in dependence upon those alerts and events. In many embodiments, a plurality of alert analyzers are implemented in parallel. Often such alert analyzers are assigned to a particular pool of alerts and may be focused on alerts with particular attributes to produce a more concise set of alerts.

The event analyzer (208) and the alert analyzer are each capable of administering incident pools for event and alert analysis in a distributed processing system according to embodiments of the present invention. The event analyzer administers pools of events according to embodiments of the present invention by receiving, by the event analyzer from an event queue, a plurality of events from one or more components of the distributed processing system; creating, by the event analyzer, a pool of events, the pool having a predetermined initial period of time; assigning, by the event analyzer, each received event to the pool; assigning, by the event analyzer, to each event a predetermined minimum time for inclusion in a pool; extending, by the event analyzer, for one or more of the events the predetermined initial period of time of the pool by a particular period of time assigned to the event; determining, by the event analyzer, whether conditions have been met to close the pool; and if conditions have been met to close the pool, determining for each event in the pool whether the event has been in the pool for its predetermined minimum time for inclusion in a pool; and if the event has been in the pool for us predetermined minimum time, including the event in the closed pool; and if the event has not been in the pool for its predetermined minimum time, evicting the event from the closed pool and including the event in a next pool.

The alert analyzer administers pools of alerts according to embodiments of the present invention by receiving, by an alert analyzer from an alert queue, a plurality of alerts from one or more components of the distributed processing system; creating, by the alert analyzer, a pool of alerts, the pool having a predetermined initial period of time; assigning, by the alert analyzer, each received alert to the pool; assigning, by the alert analyzer, to each alert a predetermined minimum time for inclusion in a pool; extending, by the alert analyzer, for one or more of the alerts the predetermined initial period of time of the pool by a particular period of time assigned to the alert; determining, by the alert analyzer, whether conditions have been met to close the pool; and if conditions have been met to close the pool, determining for each alert in the pool whether the alert has been in the pool for its predetermined minimum time for inclusion in a pool; and if the alert has been in the pool for us predetermined minimum time, including the alert in the closed pool; and if the alert has not been in the pool for its predetermined minimum time, evicting the alert from the closed pool and including the alert in a next pool.

The event and alert analysis module (124) ofFIG. 2includes computer program instructions for assigning, by the incident analyzer, an incident received from one or more components of the distributed processing system to a pool of incidents; assigning, by the incident analyzer, to each incident a particular combined minimum time for inclusion of the incident in the pool; in response to the pool closing, determining, by the incident analyzer, for each incident in the pool whether the incident has met its combined minimum time for inclusion in the pool; if the incident has been in the pool for its combined minimum time, including, by the incident analyzer, the incident in the closed pool; if the incident has not been in the pool for its combined minimum time, moving the incident from the closed pool to a next pool; applying, by the incident analyzer, incident suppression rules using the incidents assigned to the next pool; and applying, by the incident analyzer, incident creation rules to the incidents that were assigned to the next pool, while omitting any duplicate incidents caused by the assignment.

Also stored in RAM (168) is an operating system (154). Operating systems useful for relevant alert delivery according to embodiments of the present invention include UNIX™ Linux™ Microsoft XP™ AIX™ IBM's i5/OS™ and others as will occur to those of skill in the art. The operating system (154), event and alert analysis module (124), the event analyzer (208), the alert analyzer (218) in the example ofFIG. 2are shown in RAM (168), but many components of such software typically are stored in non-volatile memory also, such as, for example, on a disk drive (170).

For further explanation,FIG. 3sets forth a block diagram of an exemplary system for administering incident pools for event and alert analysis and relevant alert delivery in a distributed processing system (102) according to embodiments of the present invention. The system ofFIG. 3includes an event and alert analysis module (124). The event and alert analysis module (124) ofFIG. 3receives in an event queue (206) a plurality of events (202) from one or more components of a distributed processing system (102). A component of a distributed processing system according to embodiments of the present invention may be a device of the distributed processing system or a process running on a device of the distributed processing. Such components are often capable of some form event transmission, often for error or status reporting.

An event according to embodiments of the present invention is a notification of a particular occurrence in or on a component of the distributed processing system. Such events are sent from the component upon which the occurrence occurred or another reporting component to an event and alert analysis module according to the present invention. Often events are notifications of errors occurring in a component of the data processing system. Events are often implemented as messages either sent through a data communications network or shared memory. Typical events for event and alert analysis according to embodiments of the present invention an occurred time, a logged time, an event type, an event ID, a reporting component, and a source component, and other attributes. An occurred time is the time at which the event occurred on the component. A logged time is the time the event was included in the event queue (206) and is typically inserted into the event by the monitor (204) in the example ofFIG. 3. An event type is a generic type of event such as for example, power error, link failure error, errors related to not receiving messages or dropping packets and so on as will occur to those of skill in the art. An event ID is a unique identification of the event. A reporting component is an identification of the component that reported the event. A source component is an identification of the component upon which the event occurred. In many cases, but not all, the reporting component and source component are the same component of the distributed processing system.

In the example ofFIG. 3, the event and alert analysis module (124) includes a monitor (204) that receives events from components of the distributed processing system and puts the received events (202) in the event queue (206). The monitor (204) ofFIG. 3may receive events from components of the distributed processing system on their motion, may periodically poll one or more of the components of the distributed processing system, or receive events from components in other ways as will occur to those of skill in the art.

They system ofFIG. 3includes an event analyzer (208). The event analyzer (208) ofFIG. 3is a module of automated computing machinery capable of identifying alerts in dependence upon received events. That is, event analyzers typically receive events and produce alerts. In many embodiments, a plurality of event analyzers are implemented in parallel. Often event analyzers are assigned to a particular pool of events and may be focused on events from a particular component or caused by a particular occurrence to produce a more concise set of alerts.

The event analyzer (208) ofFIG. 3assigns each received event (202) to an events pool (212). An events pool (212) is a collection of events organized by the time of either their occurrence, by the time they are logged in the event queue, included in the events pool, or other time as will occur to those of skill in the art. That is, event pools are a collection of events organized by time. Such events pools often provide the ability to analyze a group of time related events identify alerts in dependence upon them. Often such event pools are useful in identifying fewer and more relevant alerts in dependence upon multiple related events.

The events pool (212) is administered by the event analyzer (208) according to embodiments of the present invention. The event analyzer administers pools of events according to embodiments of the present invention by receiving, by the event analyzer from the event queue, a plurality of events from one or more components of the distributed processing system; creating, by the event analyzer, a pool of events, the pool having a predetermined initial period of time; assigning, by the event analyzer, each received event to the pool; assigning, by the event analyzer, to each event a predetermined minimum time for inclusion in a pool; extending, by the event analyzer, for one or more of the events the predetermined initial period of time of the pool by a particular period of time assigned to the event; determining, by the event analyzer, whether conditions have been met to close the pool; and if conditions have been met to close the pool, determining for each event in the pool whether the event has been in the pool for its predetermined minimum time for inclusion in a pool; and if the event has been in the pool for us predetermined minimum time, including the event in the closed pool; and if the event has not been in the pool for its predetermined minimum time, evicting the event from the closed pool and including the event in a next pool.

As mentioned an events pool according to the method ofFIG. 3has a predetermined initial period of time and in the example ofFIG. 3assigning by the event analyzer each received event to an events pool includes extending for each event assigned to the events pool the predetermined initial period of time by a particular period of time assigned to the event (432). In this manner, the pool is extended with each received event until a collection of events that may be usefully used to identify alerts is assigned to the events pool.

As mentioned above, in some embodiments of the present invention, more than one event analyzer may operate in parallel. As such, each event analyzer may maintain one or more event pools for relevant alert delivery according to embodiments of the present invention. Assigning by the event analyzer the events to an events pool may therefore include selecting only events from one or more particular components. In such embodiments, particular components may be selected for a particular events pool to provide events associated with a particular period of time from a particular set of one or more components.

Assigning by the event analyzer the events to an events pool may also be carried out by selecting only events of a particular event type. In such embodiments, particular events may be selected for a particular events pool to provide events associated with a particular period of time from a particular set of event types.

Event analyzer (208) in the example ofFIG. 3identifies in dependence upon the event analysis rules (210) and the events assigned to the events pool one or more alerts (214). Event analyses rules (210) are a collection of predetermined rules for meaningfully parsing received events to identify relevant alerts in dependence upon the events.

The event analysis rules (210) ofFIG. 3include event arrival rules (230), events pool operation rules (232), event suppression rules (234), and events pool closure rules (236). Event arrival rules (230) are configurable predetermined rules for identifying alerts in dependence upon events in real time as those events are assigned to the events pool. That is, event arrival rules (230) identify alerts in dependence upon events before closing the events pool. Such rules are typically predetermined to identify particular alerts in dependence upon attributes of those events. Event arrival rules may for example dictate identifying a particular predetermined alert for transmission to a systems administrator in dependence upon a particular event type or component type for the event or other attribute of that event. Such rules are flexible and may be tailored to a particular distributed computing system and its functions.

An alert according to embodiments of the present invention is refined identification of an occurrence—such and an error—based upon more than one event and therefore provides an identification of the occurrence in the context of its operation in the distributed processing system. Often an alert may be a notification of a particular error type of occurrence that is identified in dependence upon the plurality of events received from one or more components of the data processing system, such as, for example, a link failure among a plurality of devices each of which are producing many events based upon the single link failure, or a power failure provoking thousands of events, and so on.

Alerts are often implemented as messages to be sent through a data communications network or shared memory. Typical alerts according to embodiments of the present invention have attributes attached to them based upon the attributes of the events received from which they are identified.

Events pool operation rules (232) are configurable predetermined rules for controlling the operations of the events pool. Such rules includes rules identifying the initial predetermined period of time for each events pool, rules dictating the length of time extended to the pool upon the assignment of each new event to the pool, rules dictating the minimum time an event must be in a pool before that event is included in a collection of events when the pool is closed, rules governing the closing of an events pool, and others as will occur to those of skill in the art. Such rules are flexible and may be tailored to a particular distributed computing system and its functions.

Event suppression rules (234) are configurable predetermined rules for suppressing one or more events in a closed pool of events used in identifying alerts. That is, often events in the closed events pool may be duplicate events, redundant events, or otherwise unnecessary or unhelpful events in identifying alerts. Such suppression rules are typically predetermined to delete, drop, or otherwise ignore those suppressed events. Event suppression rules may for example dictate that more than a threshold number of events of a particular event type or component type are to be suppressed. Such rules are also flexible and may be tailored to a particular distributed computing system and its functions.

Events pool closure rules (236) are configurable predetermined rules for identifying alerts in dependence upon unsuppressed events in the closed events pool and alerts identified by the event arrival rules. That is, events pool closure rules identify new alerts in dependence upon one or more or even all of the unsuppressed events in the closed events pool. The events pool closure rules also identify alerts in dependence upon the alerts identified by the event arrival rules (230) or a combination of the alerts identified by the event arrival rules (230) and one or more of the unsuppressed events in the closed events pool.

Event analyzer (208) in the example ofFIG. 3sends all the alerts (214) identified by the event analyzer (208) to an alert analyzer (218). The alert analyzer ofFIG. 3is a module of automated computing machinery capable of identifying alerts for transmission from events and other alerts, identifying additional alerts for transmission, and suppressing unnecessary, irrelevant, or otherwise unwanted or unhelpful alerts identified by the event analyzer. That is, alert analyzers typically receive alerts and events and produce or forward alerts in dependence upon those alerts and events. In many embodiments, a plurality of alert analyzers are implemented in parallel. The alerts (214) in the example ofFIG. 3are sent from event analyzer (208) to an alert analyzer (218) through an alerts queue (216).

The alert analyzer (218) ofFIG. 3assigns each of the identified alerts (214) to an alerts pool (224). An alerts pool (224) is a collection of alerts organized by the time of one or more the events causing the alert to be identified, the time the alert is identified, or other time as will occur to those of skill in the art. That is, alerts pools are a collection of alerts organized by time. Such alerts pools often provide the ability to analyze a groups alerts identified and included in the alerts pool according to some time. Often such alerts pools are useful in identifying fewer and more relevant alerts in dependence upon multiple related events and multiple related alerts.

The alert analyzer administers pools of alerts according to embodiments of the present invention by receiving, by an alert analyzer from an alert queue, a plurality of alerts from one or more components of the distributed processing system; creating, by the alert analyzer, a pool of alerts, the pool having a predetermined initial period of time; assigning, by the alert analyzer, each received alert to the pool; assigning, by the alert analyzer, to each alert a predetermined minimum time for inclusion in a pool; extending, by the alert analyzer, for one or more of the alerts the predetermined initial period of time of the pool by a particular period of time assigned to the alert; determining, by the alert analyzer, whether conditions have been met to close the pool; and if conditions have been met to close the pool, determining for each alert in the pool whether the alert has been in the pool for its predetermined minimum time for inclusion in a pool; and if the alert has been in the pool for us predetermined minimum time, including the alert in the closed pool; and if the alert has not been in the pool for its predetermined minimum time, evicting the alert from the closed pool and including the alert in a next pool.

The alert analyzer may assign the identified alerts to an alerts pool (224) in dependence upon attributes of the alerts or attributes of the events from which those alerts were identified. For example, the alert analyzer ofFIG. 3may assign alerts to the alerts pool (224) by selecting alerts generated from events from one or more particular components, alerts associated with a particular alert type and so on as will occur to those of skill in the art.

The alert analyzer (218) ofFIG. 3determines in dependence upon alert analysis rules (222) and the alerts in the alert pool whether to suppress any alerts. Suppressing an alert is typically carried out by dropping the alert, deleting the alert or otherwise ignoring or not transmitting the suppressed alert to a component of the distributed processing system.

Alert analyses rules (222) are a collection of rules for suppressing one or more alerts to provide a more relevant set of alerts for transmission to a component of the distributed processing system, such as for example, for display to a systems administrator and to identify additional alerts for transmission to one or more components of the distributed processing system. Alert analysis rules for example may dictate that duplicate alerts are to be suppressed, alerts of a particular type for transmission to a particular component are to be suppressed, alerts of a particular type be transmitted to a particular component are to be suppressed and so on as will occur to those of skill in the art. Such alerts may be more meaningful to a component of the distributed processing system for automated error recovery or for a systems administrator who may otherwise be less informed by a number of raw unanalyzed alerts.

The alert analyzer (218) ofFIG. 3also has access to the events queue (206). The alert analyzer (218) ofFIG. 3in dependence upon the alert analysis rules may, in some embodiments select events from the events queue and determine whether to suppress any alerts in dependence upon the selected events. That is, alert analysis rules may also take into account events and their attributes for suppressing alerts and for identifying additional alerts for transmission to one or more components. Such events may be related to the alerts in the alerts pool or independent from such alerts.

The alert analyzer (218) ofFIG. 3transmits the unsuppressed alerts to one or more components of the distributed processing system. The alert analyzer may transmit the unsuppressed alerts to one or more components of the distributed processing system by sending the alert as a message across a data communications network, through shared memory, or in other ways as will occur to those of skill in the art. In the example ofFIG. 3, the unsuppressed alerts (220) are transmitted to a terminal (122) for display to a systems administrator (128).

The alert analyzer (218) ofFIG. 3is also capable of identifying in dependence upon alert analysis rules (222), the alerts in the alert pool (224), and selected events (206) one or more additional alerts and transmitting the one or more components of the distributed processing system. The additional alerts may include one or more alerts not identified by the event analyzer. Such additional alerts may provide additional information to a component of the distributed processing system of a systems administrator.

As mentioned above, relevant alert delivery according to the present invention includes assigning events to an event pool and those pools are administered according to embodiments of the present invention. For further explanation,FIG. 4sets forth a diagram illustrating assigning events to an event pool according to embodiments of the present invention. An events pool (212) is a collection of events organized by the time of either their occurrence, by the time they are logged in the event queue, included in the events pool, or other time as will occur to those of skill in the art.

That is, event pools are a collection of events organized by time. Such events pools often provide the ability to analyze a group of time related events and to identify alerts in dependence upon them. Often such event pools are useful in identifying fewer and more relevant alerts in dependence upon multiple related events.

Events pools according to embodiments of the present invention are typically operated according to events pool operation rules which are themselves often included in event analysis rules. Such events pool operation rules are configurable predetermined rules for controlling the operations of the events pool. Such rules includes rules identifying the initial predetermined period of time for each events pool, rules dictating the length of time extended to the pool upon the assignment of each new event to the pool, rules dictating the minimum time an event must be in a pool before that event is included in a collection of events when the pool is closed, rules governing the closing of an events pool, and others as will occur to those of skill in the art. Such rules are flexible and may be tailored to a particular distributed computing system and its functions.

Events are often assigned to an events pool according to their logged time. That is, events are typically inserted into the events pool in the order that they are received in the event queue. In the example ofFIG. 4, the timing of the events pool (212) is initiated when the first event ‘Event0’ (400) is assigned to the events pool (212) at time t0. The events pool ofFIG. 4is initiated for a predetermined initial period of time from t1to tf. That is, upon receiving the first event ‘Event0’ (400) the events pool ofFIG. 4has a predetermined initial period of time beginning at t1and ending at tf. The predetermined initial period of time may be configured in dependence upon a number of factors as will occur to those of skill in the art such as, the number of components in the distributed processing system, the frequency of receiving events, the types of events typically received and so on as will occur to those of skill in the art.

In the exampleFIG. 4, the initial period of time is extended for each new event assigned to the events pool during the predetermined initial period from t1to tfby a particular period of time assigned to the event. In the example ofFIG. 4upon assigning ‘Event1’ (402) to the events pool (212) the predetermined initial period of time t0-tfis extended by ‘Extension1’ (406) having a time of e1thereby creating a new time for closing the events pool (212) at tf+e1if no other events are assigned to the pool before tf+e1. Similarly, in the example ofFIG. 4upon assigning ‘Event2’ (404) to the events pool having a time of e2, the now extended period of time from t0-tf+e1is extended again by extension2(408) thereby establishing a new time for closing the pool at time tf+e1+e2if no other events are assigned to the pool before tf+e1+e2or before some maximum time for the events pool has expired. In this manner, the event pool is extended with each received event until a collection of events that may be usefully used to identify alerts is assigned to the events pool.

In typical embodiments of the present invention, event pools may have a maximum duration that can no longer be extended. In such cases, a requirement may exist that an event that has not resided in the event pool for a threshold period of time be moved to a next events pool. In some embodiments, the attributes of such an event that is moved to the next events pool are used for relevant alert delivery according to embodiments of the present invention with the initial events pool and in other embodiments, the attributes of such an event are used for relevant alert delivery with the next events pool to which that event is moved.

In the example ofFIG. 4, when conditions are met to close the pool an events analyzer determines for each event (400,402,404) in the pool (212) whether the event has been in the pool for its predetermined minimum time for inclusion in a pool. If the event has been in the pool for its predetermined minimum time, the event is included in the closed pool for event analysis for relevant alert delivery according to embodiments of the present invention. If the event has not been in the pool for its predetermined minimum time, the event is evicted from the closed pool and included a next pool for event analysis for relevant alert delivery according to embodiments of the present invention.

In many embodiments, a plurality of events pools may be used in parallel and one or more of such events pools are assigned to a particular events analyzer. In such embodiments, events analyzers may be directed to events in events pools having particular attributes.

As mentioned above, relevant alert delivery according to the present invention also includes assigning alerts to an alerts pool. For further explanation,FIG. 5sets forth a diagram illustrating assigning alerts to an alert pool according to embodiments of the present invention. The alerts pool (224) ofFIG. 5operates in a manner similar to the events pool ofFIG. 4. That is, the alerts pool according to the example ofFIG. 5includes alerts and the timing of the alerts pool begins with the first alert ‘Alert0’ (500) at time t0and is configured to have a predetermined initial period of time t0-tf. In the example ofFIG. 5, the initial period of time is extended for each new alert assigned to the alerts pool in the predetermined initial period from t1to tfby a particular period of time assigned to the alert. In the example ofFIG. 5, upon assigning ‘Alert1’ (502) to the alerts pool (224) the predetermined initial period of time t0-tfis extended by ‘Extension1’ (506) having a time of e1thereby creating a new time for closing the alerts pool (224) at tf+e1if no other alerts are assigned to the pool before tf+e1. Similarly, in the example ofFIG. 4upon assigning ‘Alert2’ (504) to the alerts pool having a time of e2, the now extended period of time from t0-tf+e1is extended again by ‘Extension2’ (508) thereby establishing a new time for closing the pool at time tf+e1+e2if no other alerts are assigned to the pool before tf+e1+e2or before some maximum time for the alerts pool has expired.

In typical embodiments of the present invention, alerts pools may have a maximum duration that can no longer be extended. In such cases, a requirement may exist that an alert that has not resided in the alert pool for a threshold period of time be moved to a next alerts pool. In some embodiments, the attributes of such an alert that is moved to the next alerts pool are used for relevant alert delivery according to embodiments of the present invention with the initial alerts pool and in other embodiments, the attributes of such an alert are used for relevant alert delivery with the next alerts pool to which that alert is moved.

In the example ofFIG. 5, when conditions are met to close the pool an alerts analyzer determines for each alert (500,502,504) in the pool (224) whether the alert has been in the pool for its predetermined minimum time for inclusion in a pool. If the alert has been in the pool for its predetermined minimum time, the alert is included in the closed pool for alert analysis for relevant alert delivery according to embodiments of the present invention. If the alert has not been in the pool for its predetermined minimum time, the alert is evicted from the closed pool and included a next pool for alert analysis for relevant alert delivery according to embodiments of the present invention.

In many embodiments, a plurality of alerts pools may be used in parallel and one or more of such alerts pools are assigned to a particular alerts analyzer. In such embodiments, alerts analyzers may be directed to alerts in alerts pools having particular attributes.

For further explanation,FIG. 6sets forth a flow chart illustrating an example method of administering incident pools for event and alert analysis in a distributed processing system in a distributed processing system according to embodiments of the present invention. The method ofFIG. 6includes receiving (402) in an event queue a plurality of events (202) from one or more components of a distributed processing system. Events useful in administering incident pools for event and alert analysis in a distributed processing system according to embodiments of the present invention may include an occurred time, a logged time, an event type, an event ID, a reporting component, and a source component.

Receiving (402) in an event queue a plurality of events (202) from one or more components of a distributed processing system may be carried out by receiving an event initiated by one or more components of the data processing system and storing the event in the event queue according to the time in which the event occurred or according to the time the event was received. Receiving (402) in an event queue a plurality of events (202) from one or more components of a distributed processing system also may be carried out by polling a component for status and receiving in response an event and storing the event in the event queue according to the time in which the event occurred or according to the time the event was received.

The method ofFIG. 6also includes assigning (404) by an event analyzer each received event to an events pool (212). In some embodiments of the present invention, assigning (404) by an event analyzer each received event (202) to an events pool (212) may be carried out by assigning events to the event pool according to the logged time. Assigning (404) by an event analyzer each received event (202) to an events pool (212) may also be carried out in dependence upon attributes of the event. Such attributes may include an identification or type of the component upon which an occurrence occurred to create the event, the reporting component of the event, the event ID, the event type, and so on as will occur to those of skill in the art.

An events pool according to the method ofFIG. 6includes events occurring during a predetermined initial period of time and in the example ofFIG. 6assigning (404) by the event analyzer each received event to an events pool includes extending (426) for each event assigned to the events pool the predetermined initial period of time by a particular period of time assigned to the event.

The event analyzer includes event analysis rules (210) including, event arrival rules, events pool operation rules, event suppression rules, and events pool closure rules. Event arrival rules are configurable predetermined rules for identifying alerts in dependence upon events in real time as those events are assigned to the events pool. That is, event arrival rules identify alerts in dependence upon events before closing the events pool. Such rules are flexible and may be tailored to a particular distributed computing system and its functions.

An alert according to embodiments of the present invention is refined identification of an occurrence—such and an error—based upon more than one event and therefore provides an identification of the occurrence in the context of its operation in the distributed processing system. Often an alert may be a notification of a particular error type of occurrence that is identified in dependence upon the plurality of events received from one or more components of the data processing system, such as, for example, a link failure among a plurality of devices each of which are producing many events based upon the single link failure, or a power failure provoking thousands of events, and so on.

Alerts are often implemented as messages to be sent through a data communications network or shared memory. Typical alerts according to embodiments of the present invention have attributes attached to them based upon the attributes of the events received from which they are identified.

Events pool operation rules are configurable predetermined rules for controlling the operations of the events pool. Such rules includes rules identifying the initial predetermined period of time for each events pool, rules dictating the length of time extended to the pool upon the assignment of each new event to the pool, rules dictating the minimum time an event must be in a pool before that event is included in a collection of events when the pool is closed, rules governing the closing of an events pool, and others as will occur to those of skill in the art. Such rules are flexible and may be tailored to a particular distributed computing system and its functions.

Event suppression rules are configurable predetermined rules for suppressing one or more events in a closed pool of events used in identifying alerts. That is, often events in the closed events pool may be duplicate events, redundant events, or otherwise unnecessary or unhelpful events in identifying alerts. Such suppression rules are typically predetermined to delete, drop, or otherwise ignore those suppressed events. Event suppression rules may for example dictate that more than a threshold number of events of a particular event type or component type are to be suppressed. Such rules are also flexible and may be tailored to a particular distributed computing system and its functions.

Events pool closure rules are configurable predetermined rules for identifying alerts in dependence upon unsuppressed events in the closed events pool and alerts identified by the event arrival rules. That is, events pool closure rules identify new alerts in dependence upon one or more or even all of the unsuppressed events in the closed events pool. The events pool closure rules also identify alerts in dependence upon the alerts identified by the event arrival rules or a combination of the alerts identified by the event arrival rules and one or more of the unsuppressed events in the closed events pool.

The method ofFIG. 6also includes identifying (410) by the event analyzer in dependence upon the event arrival rules and the events assigned to the events pool one or more alerts (214). Identifying (410) by the event analyzer in dependence upon the event arrival rules and the events assigned to the events pool one or more alerts (214) may be carried out by identifying alerts in dependence upon one or more attributes of the events as that event is assigned to the events pool. Identifying (410) by the event analyzer in dependence upon the event arrival rules and the events assigned to the events pool one or more alerts (214) may be carried by comparing the attributes of the events to the event arrival rules and identifying as a result of the comparison one or more alerts. Such attributes may include the type of component from which the event was received, the type of component creating the event, the identification of the component creating the event, the time the event was created or received, an error reported in the event, and many others as will occur to those of skill in the art.

The method ofFIG. 6also includes closing (412), by the event analyzer in dependence upon the events pool operation rules, the events pool (212). Closing (412), by the event analyzer in dependence upon the events pool operation rules, the events pool (212) may be carried out by determining that conditions dictated by the events pool operation rules have been met to stop assigning new events to the events pool and identifying in dependence upon those events pool operation rules the particular events that are included in the closed pool of events.

Closing the events pool may be carried out by determining that the initial period of time for the events pool and any particular periods of time for events received in the events pool extended to the initial period of time have expired. In such cases, if no new events are received prior to the expiration of the initial period of time for the events pool and any particular periods of time for events received in the events pool extended to the initial period of time the pool is closed.

Closing the events pool may also be carried out by determining that a maximum duration for the events pool has expired. In such cases, regardless of the number of new events being received after a maximum duration for the events pool has expired the pool is closed. In such embodiments, a maximum duration for the events pool prevents the events pool from including more events than are useful for relevant alert delivery according to embodiments of the present invention.

The method ofFIG. 6also includes determining (414), by the events analyzer in dependence upon the event suppression rules, whether to suppress one or more events in the closed events pool (212). Determining (414), by the events analyzer in dependence upon the event suppression rules, whether to suppress one or more events in the closed events pool (212) may be carried out by determining in dependence upon the attributes of one or more events in the closed pool whether to delete, drop, or otherwise ignore one or more of the events in the closed pool.

The method ofFIG. 6includes identifying (416) by the event analyzer in dependence upon the events pool closure rules and any unsuppressed events assigned to the events pool, one or more additional alerts (417). Identifying (416) by the event analyzer in dependence upon the events pool closure rules and any unsuppressed events assigned to the events pool, one or more additional alerts (417) may be carried out by identifying alerts in dependence upon one or more attributes of the events as that event is assigned to the events pool. Identifying (416) by the event analyzer in dependence upon the events pool closure rules and any unsuppressed events assigned to the events pool, one or more additional alerts (417) may be carried out by selecting the unsuppressed events for the events pool, comparing the attributes of the unsuppressed events of the events pool to the pool closure rules, and identifying as a result of the comparison one or more additional alerts. Such attributes may include the type of component from which one or more of the unsuppressed events are received, the type of components creating the unsuppressed events, the identification of the component creating the unsuppressed events, the time the events were created or received, one or more errors reported by the events event, the number of events in the pool, and many others as will occur to those of skill in the art.

The method ofFIG. 6includes sending (418) by the event analyzer to an alert analyzer all the alerts identified by the event analyzer. Sending (418) by the event analyzer to an alert analyzer all the alerts (214) identified by the event analyzer may be carried out by sending a message containing the alerts from the event analyzer to the alert analyzer. Such a message may be sent from the event analyzer to the alert analyzer across a network, through shared memory, or in other ways as will occur to those of skill in the art.

The method ofFIG. 6includes assigning (420) by the alert analyzer the identified alerts to an alerts pool (224). An alerts pool according to the method ofFIG. 6has a predetermined initial period of time and in the example ofFIG. 6assigning (420) by the alert analyzer the identified alerts to an alerts pool (224) includes extending for each alert assigned to the alerts pool the predetermined initial period of time by a particular period of time assigned to the alert. Assigning (420) by the alert analyzer the identified alerts to an alerts pool (224) also may be carried out in dependence upon attributes of the alerts. Such attributes may include an identification or type of the component upon which an occurrence occurred to create the event that was used to identify the alert, the alert ID, the alert type, and so on as will occur to those of skill in the art.

The method ofFIG. 6includes determining (422) by the alert analyzer in dependence upon alert analysis rules (222) and the alerts in the alert pool whether to suppress any alerts. Determining (422) by the alert analyzer in dependence upon alert analysis rules (222) and the alerts in the alert pool whether to suppress any alerts may be carried out in dependence upon one or more attributes of the alerts. Such attributes may include an identification or type of the component upon which an occurrence occurred to create the event that was used to identify the alert, the alert ID, the alert type, and so on as will occur to those of skill in the art. In such embodiments, determining (422) by the alert analyzer in dependence upon alert analysis rules (222) and the alerts in the alert pool whether to suppress any alerts may be carried out by comparing the attributes of the alerts in the alerts pool to the alert analysis rules and identifying as a result of the comparison one or more alerts for suppression according to the event analysis rules.

The method ofFIG. 6includes transmitting (428) the unsuppressed alerts to one or more components of the distributed processing system. Transmitting (428) the unsuppressed alerts to one or more components of the distributed processing system may be carried out by sending a message containing the alert to one or more components of the distributed processing system. In many cases, an alert may be sent as a message to a systems administrator advising the systems administrator of one or more occurrences within the distributed processing system.

As mentioned above, alert analysis rules may select additional alerts or suppress alerts in dependence upon events. In such embodiments, determining whether to suppress any alerts includes selecting events and determining whether to suppress any alerts in dependence upon the selected events. The method ofFIG. 6therefore also includes identifying (426) by the alert analyzer in dependence upon alert analysis rules (222), the alerts in the alert pool (224), and any selected events one or more additional alerts and in the method ofFIG. 6, transmitting (428) the unsuppressed alerts also includes transmitting (430) any additional alerts to one or more components of the distributed processing system.

As mentioned above, relevant alert delivery according to embodiments of the present invention includes the administration of one or more pools of incidents such as events, alerts or other incidents as will occur to those of skill in the art. For further explanation,FIG. 7sets forth a flow chart illustrating an exemplary method of administering incident pools for event and alert analysis in a distributed processing system according to embodiments of the present invention. The method ofFIG. 7includes assigning (710), by an incident analyzer (701), an incident (704) received from one or more components of a distributed processing system to a pool (708) of incidents. For example, components of a distributed processing system may produce events and the events may be stored in an event queue. The incident analyzer (701) may retrieve incidents from an event queue or an alert queue for assignment to an incident pool. Assigning (710) an incident (704) received from one or more components of a distributed processing system to a pool (708) of incidents may include receiving an event from an event queue and assigning the event to an event pool; and receiving an alert from an alert queue and assigning the alert to an alert pool. Assigning (710) an incident (704) received from one or more components of a distributed processing system to a pool (708) of incidents may also be carried out by creating a data structure for inclusion of events or alerts and establishing for that data structure an initial period of time to include within that data structure events, alerts or other incidents as will occur to those of skill in the art; and selecting events, alerts, or other incidents in dependence upon such attributes as the component creating an event, the event type, the alert type, the component reporting the event, the component that reported or created the event in whose dependence an alert was created, and so on as will occur to those of skill in the art.

The method ofFIG. 7also includes assigning (712), by the incident analyzer, to each incident a particular combined minimum time (714) for inclusion of the incident (704) in the pool (708). A combined minimum time is an amount of time that a particular incident must spend in one or more pools. That is, each incident has its own combined minimum time. The combined minimum time assigned to a particular incident may vary according to attributes of the incident as will occur to those of skill in the art. Assigning (712) to each incident a particular combined minimum time (714) for inclusion of the incident (704) in the pool (708) may be carried out by storing a received incident in an incident pool.

The method ofFIG. 7also includes in response to the pool closing, determining (718), by the incident analyzer, for each incident in the pool whether the incident has met its combined minimum time (714) for inclusion in one or more pools. A pool may close based on a determining, by the incident analyzer, whether conditions have been met to close the pool (718) and may be carried out by determining whether a maximum time for the pool has elapsed. The maximum time established for the pool is typically dependent upon factors of the distributed processing system such as the size of the distributed processing system, the components of the distributed processing system, the computational requirements of the distributed processing system and others as will occur to those of skill in the art. Typically, the maximum time is established to be long enough to allow enough incidents to accumulate into the pool for meaningful event or alert analysis but short enough such that that analysis is not overwhelmed by the sheer number of incidents and to allow for alerts to be reported in a timely manner. Determining (720) for each incident in the pool whether the incident has met its combined minimum time (714) for inclusion in one or more pools may be carried out by subtracting from the combined minimum time the amount of time that a particular incident spends in pools.

If the incident has been in the pool for its combined minimum time, the method ofFIG. 7includes including (722) the incident in the closed pool. Including (722) the incident in the closed pool is carried out by maintaining the incident in the closed pool for event analysis, alert analysis, or other incident analysis as will occur to those of skill in the art.

If the incident has not been in the pool for its combined minimum time, the method ofFIG. 7includes moving (724) the incident from the closed pool to a next pool of incidents. Moving (724) the incident in a next pool may be carried out by creating a next pool and moving the incident into the next pool.

The method ofFIG. 7also includes applying (730), by the incident analyzer (701), incident suppression rules (790) using the incidents assigned to the next pool (770). Incident suppression rules are rules indicating which incidents are suppressed and which are not suppressed. Applying (730) incident suppression rules (790) using the incidents assigned to the next pool (770) may be carried out by determining which incidents assigned to the next pool may be suppressed in accordance with the incident suppression rules; and suppressing the determined incidents.

The method ofFIG. 7also includes applying (732), by the incident analyzer (701), incident creation rules (792) to the incidents assigned to the next pool (770), while omitting any duplicate incidents caused by the assignment. Incident creation rules are rules indicating which incidents may be used to create new incidents, such as alerts. Applying (732) incident creation rules (792) to the incidents assigned to the next pool (770), while omitting any duplicate incidents caused by the assignment may be carried out by determining which incidents assigned to the closed pool may be used to create new alerts in accordance with the incident creation rules; and creating new incidents. For example, when processing an event in an event pool, an event analyzer may for various reasons move the event forward to a next pool so that the moved event can influence analysis in the next pool. Suppressing alerts generated by the event in the prior pool may be suppressed according to embodiments of the present invention to keep duplicated analysis from being performed in the next pool. By applying the suppression rules to the events that were moved to the next pool, the event in the next pool is able to combine with additional events to create alerts. That is, administering incident pools for event and alert analysis according to embodiments of the present invention may eliminate duplicate alerts and thus increase processing capabilities of incident analyzers.

For further explanation,FIG. 8sets forth a flow chart illustrating an additional method of administering incident pools for event and alert analysis in a distributed processing system according to embodiments of the present invention. The method ofFIG. 8is similar to the method ofFIG. 7in that the method ofFIG. 8includes: assigning (710), by an incident analyzer (701), an incident (704) received from one or more components of a distributed processing system to a pool (708) of incidents; assigning (712), by the incident analyzer, to each incident a particular combined minimum time (714) for inclusion of the incident (704) in the pool (708); determining whether conditions have been met to close the pool (718); in response to the pool closing, determining (720), by the incident analyzer, for each incident in the pool whether the incident has met its combined minimum time (714) for inclusion in one or more pools; if the incident has been in the pool for its combined minimum time, including (722) the incident in the closed pool; if the incident has not been in the pool for its combined minimum time, moving (724) the incident from the closed pool to a next pool of incidents; applying (730), by the incident analyzer (701), incident suppression rules (790) using the incidents assigned to the next pool (770); applying (732), by the incident analyzer (701), incident creation rules (792) to the incidents assigned to the next pool, while omitting any duplicate incidents caused by the assignment.

In the method ofFIG. 8, however, moving (724) the incident from the closed pool to a next pool of incidents includes forwarding (802) to the next pool, by the incident analyzer (701), any associations between the incident and any other incidents that are included in the pool (708). An association indicates a relationship between one incident and another incident. For example, an association may indicate whether an alert was generated with respect to the incident. In this case, future pools and incidents within those future pools may be aware of alerts created by the incident in a previous pool.

In the method ofFIG. 8, applying (730), by the incident analyzer (701), incident suppression rules (790) using the incidents assigned to the next pool (770) includes suppressing (804) one or more incidents created based on the incident's location with the closed pool. Event suppression rules are configurable predetermined rules for suppressing one or more events in a closed pool of events used in identifying alerts. That is, often events in the closed events pool may be duplicate events, redundant events, or otherwise unnecessary or unhelpful events in identifying alerts. Such suppression rules are typically predetermined to delete, drop, or otherwise ignore those suppressed events. Event suppression rules may for example dictate that more than a threshold number of events of a particular event type or component type are to be suppressed. Such rules are also flexible and may be tailored to a particular distributed computing system and its functions. Suppressing (804) one or more incidents created based on the incident's location with the closed pool may be carried out by identifying any additional incidents created based on the incident's prior location within the closed pool; deleting, dropping or otherwise ignoring the identified additional incidents.

In the method ofFIG. 8, applying (730), by the incident analyzer (701), incident suppression rules (790) using the incidents assigned to the next pool (770) includes preventing (805) within the next pool (770), alerts that are only generated based on incidents that generated previous alerts in the closed pool (708). Preventing (805) within the next pool (770), alerts that are only generated based on incidents that generated previous alerts in the closed pool (708) may be carried out by identifying incidents that generated alerts in the closed pool before being moved to the next pool; determining that a potential alert is generated in the next pool based entirely on one or more of the identified incidents; and preventing the potential alert generated based entirely on one or more of the identified incidents for generating an alert. That is, according to embodiments of the present invention, an alert created in the closed pool is not duplicated in subsequent pools (e.g., the next pool (770). However, if additional incidents are added to the next pool and combine with the identified incidents to produce new potential alerts, those new potential alerts would not be prevented from creating new actual alerts based on administering incident pools for event and alert analysis in a distributed processing system according to embodiments of the present invention.

In the method ofFIG. 8, applying (732) incident creation rules (792) to the incidents assigned to the next pool (770), while omitting any duplicate incidents caused by the assignment includes creating (806) one or more additional incidents based on the incidents that were moved from the closed pool (708) to the next pool (770). Creating (806) one or more additional incidents based on the incidents that were moved from the closed pool (708) to the next pool (770) may be carried out by creating an alert based on an event or alert within the next pool (770).