Fault detection and recovery as a service

The monitoring by a monitoring node of a process performed by a monitored node is often devised as a tightly coupled interaction, but such coupling may reduce the re-use of monitoring resources and processes and increase the administrative complexity of the monitoring scenario. Instead, fault detection and recovery may be designed as a non-proprietary service, wherein a set of monitored nodes, together performing a set of processes, may register for monitoring by a set of monitoring nodes. In the event of a failure of a process, or of an entire monitored node, the monitoring nodes may collaborate to initiate a restart of the processes on the same or a substitute monitored node (possibly in the state last reported by the respective processes). Additionally, failure of a monitoring node may be detected, and all monitored nodes assigned to the failed monitoring node may be reassigned to a substitute monitoring node.

BACKGROUND

Within the field of computing, many scenarios involve a detection of a fault in a computer system, such as an interference with a process; an unavailability of a resource utilized by the process, such as an exhaustion of free memory or a resource that is exclusively locked by another process; an inability of a process to complete a task; a logical fault in a process that leads to a consumption of resources, an unending loop, or an application crash; or a failure of the hardware of a device that interrupts the execution of processes. Such faults may range in severity from curiosities to inconveniences to severe problems (e.g., failures in realtime processes or processes upon which users depend for uptime). In these and other scenarios, an administrator may endeavor to monitor the process, such as utilizing a monitoring process operating on the same device or another device to monitor the instrumentation of a monitored process, verify that the monitored process continues to operate as anticipated, provides acceptable performance, and is accessible to users. If the monitored process shows indications of failure or becomes unreachable, the monitoring process may register the indications in a log, or may notify an administrator.

SUMMARY

The interface of monitoring processes and monitored processes is often highly specialized and proprietary. For example, a process to be monitored may report a specialized set of metrics indicating its status, and/or may report such metrics in specialized ways, such as a particular type of log or event model or a particular location. The monitoring process may also be specially designed to couple with the monitored process (e.g., to evaluate the log, subscribe to the events, and/or query the monitored process). The monitoring process and monitored process may together represent a tightly coupled pair of interoperating processes. However, the specialization of the monitoring process and the monitored process may be inefficient in some respects. As a first example, each monitored process may have a corresponding monitoring process, and the monitoring of many monitored processes may involve the deployment of many monitoring processes that each monitors one process. This coupling may reduce the re-use of monitoring equipment (e.g., it may be difficult to configure a group of machines to monitor a group of processes). As a second example, it may be uncommon to establish monitoring of a first process that is monitoring a second process (on the same machine or another machine). As a third example, each instance of monitoring may differ in insignificant or significant ways (e.g., the manner of notifying an administrator of failure indicators, the types of failure indicators used by different processes, and the location, format, and semantic use of a log file), leading to increased complexity in the administration of the systems.

Presented herein are techniques for implementing fault monitoring as a standardized service. In accordance with these techniques, a set of one or more monitoring nodes may be provided that are configured to perform the monitoring of various monitored nodes performing one or more processes to be monitored. One or more monitored nodes may register for monitoring with a monitoring node, and may initiate a periodic reporting of the statuses of the processes, possibly including the states of the processes (e.g., not just whether a process is running, but the internal state of the process). The monitoring nodes may record this information, and may take an action upon receiving an indication that a process or a monitored node is encountering difficulty or has stopped reporting. For example, the monitoring node may request a restart of a process that is no longer reporting, including restarting the process at the state last reported by the monitored node, and also may request a restart of the process on a different monitored node. If a monitored node entirely stops responding, the monitoring node may choose a substitute monitored node, and may request a restart of all such processes (possibly in the last reported states) on the substitute monitored node.

Moreover, in scenarios involving a set of two or more monitoring nodes, the monitoring nodes may also monitor each other. For example, monitored nodes may be assigned to report to a particular monitoring node. Moreover, the monitoring nodes may periodically synchronize the monitoring information thereamong (e.g., each monitoring node may inform the other monitoring nodes of the monitored nodes assigned thereto, the monitored processes executed thereupon, and the status and/possibly state of each such monitored process). If a monitoring node fails (e.g., stops reporting to the other monitoring nodes), the other monitoring nodes may choose a substitute monitoring node, to which all of the monitored node of the failed monitoring node may be reassigned.

The implementation of fault detection and fault recovery in accordance with these techniques may present some advantages. As a first example, these techniques may enable fault tolerance to be offered as a non-specialized, non-proprietary, subscription-based service, wherein a set of monitoring nodes may collaborate to monitor any participating process on a set of monitored nodes, irrespective of the type of process that is monitored or the role of the monitored node. As a second example, these techniques may promote efficient use of monitoring resources; for example, a single set of monitoring nodes, executing a small number of monitoring processes, may be configured monitor a large and diverse set of monitored nodes and monitored processes. As a third example, these techniques enable a monitoring node to assume the role of a failed monitoring node (e.g., by adopting the monitored nodes formerly assigned to the failed monitoring node), and a monitored node to assume the role of a failed monitored node (e.g., by restarting the processes performed by the monitored node at the moment of failure). This redundancy and fungibility of nodes may promote high availability by recovering from failures while reducing interruption of the monitored processes and/or monitoring service. As a fourth example, these techniques may promote the scalability of the monitoring service; e.g., monitored nodes and processes may easily register for monitoring, and monitoring capacity may be expanded simply by adding more monitoring nodes. These and other advantages may be achieved through the implementation of monitoring scenarios according to the techniques presented herein.

DETAILED DESCRIPTION

Within the field of computing, many scenarios involve a monitoring of one or more processes performed by one or more nodes (e.g., various types of devices and computers, and/or simulated operating environments operating thereupon) that are to be monitored in the event of a failure. A process may fail for various reasons (e.g., an interference with a process; an unavailability of a resource utilized by the process, such as an exhaustion of free memory or a resource that is exclusively locked by another process; a failure of a process to complete a task; a logical fault in a process that leads to a consumption of resources, an unending loop, or an application crash). Additionally, an entire node may experience software or hardware failures (e.g., an overheating of the processor, a cessation of power or network access, a hard disk failure, or a crashed operating system). The consequences of such failures may cover a range of severity, such as a curiosity, an inconvenience or frustration, and a severe problem (e.g., a realtime process may execute within a scenario having strict timing parameters, such as a process operating the motions of a robot on an assembly line or a process may be utilized by a large number of other processes, such as a database server that provides data stored in a database to a range of data-driven applications, or a webserver that concurrently receives and generates web pages in response to a large number of requests received from many users).

Within such scenarios, it may be desirable to configure a computer or device executing a process for monitoring by another computer or device. For example, a monitored node executing a particular process may be monitored by a monitoring node, which may periodically detect metrics or receive reports that indicate the status of the monitored node and/or process, such as whether the process and node remain operational and are performing in an acceptable manner. Upon detecting an emerging, imminent, or existing problem, such as a failure of the process or the monitored node, the monitoring node may perform various actions, such as logging the detected information or notifying an administrator for further action.

The monitoring of a process of a monitored node by a monitoring node may be implemented in various ways.FIG. 1presents an illustration of an exemplary scenario10featuring two exemplary techniques for configuring a monitored node14and a monitoring node18to interoperate to achieve the monitoring of a process16performed by the monitoring node14. In this exemplary scenario10, a first monitored node14performs a first process16that is monitored by a first monitoring node18, and a second monitored node14performs a second process16that is monitored by a second monitoring node18. However, the circumstances of the monitoring may significantly differ between these two implementations. For example, the first monitoring process14may utilize a first monitoring process20comprising a specialized application written to monitor the process16, and that implements a specialized interface22to communicate with a specialized interface24of the process16(e.g., the monitoring process20may receive metrics sent by the specialized interface24of the process16, and/or may query the process16through the specialized interface24to request status information). The first monitoring process20may also write significant information to a first log28that may be reviewed by an administrator12to determine the status of the monitored node14and process16. The second monitoring process14may utilize a second (different) monitoring process20comprising a specialized application written to monitor the process16. However, the process16may not be configured to communicate with the monitoring process20, but may provide a service30to a client32, and the monitoring process20may monitor34the service30of the process16with the client32(e.g., monitoring the provision of data from the process16to the client32). The second monitoring process20may also write significant information to a second (different) log28that may be reviewed by an administrator12to determine the status of the monitored node14and process16. In this manner, the first monitoring node18and the first monitored node14may interoperate to achieve the monitoring of the first process16, and the second monitoring node18and the second monitored node14may interoperate to achieve the monitoring of the second process16.

The exemplary scenario10ofFIG. 1therefore illustrates a specialized technique for monitoring each process16, involving a tight coupling of respective monitoring nodes18and monitored nodes16to achieve the monitoring and reporting of statuses and failures. However, this exemplary scenario10also illustrates some disadvantages that may arise with specialized techniques based on such tight coupling. As a first example, each monitoring node18performs a monitoring of the corresponding monitored node14and process16, but does not participate in the monitoring of the other monitored node14and/or process16. Therefore, the monitoring nodes18may not utilize the resources of one monitored node14in the event of a failure of the other monitored node14. As a second example, the monitoring nodes18do not intercommunicate, and a failure of either monitoring node18may not be detected or responded to by the other monitoring node18; indeed, a failure of a monitoring node18may not be reported to or noticed by an administrator12. As a third example, the monitoring nodes18report information about the monitoring to the administrator12in different ways (e.g., utilizing different logs28), and the administrator12may have to examine each log28and attend to the details of the monitoring of each process16in turn. As a fourth example, it may be difficult for the administrator12to introduce a new process16to be monitored, a new monitored node14performing a process16, or a new monitoring node20into the exemplary scenario10, due to the specialization and tight coupling of the resources already included therein. Moreover, the differences in the manner of recording information and reporting failures may reduce the consistency of the reporting process. Rather, the monitoring of a new process16and/or monitored node14may be achieved only by the introduction of a new monitoring node18, possibly including a specialized monitoring process20, thereby further increasing the complexity of the administration of the monitoring in the exemplary scenario10ofFIG. 1.

Presented herein are techniques for implementing one or more monitoring nodes18to monitor one or more processes16performed by one or more monitoring nodes14in a standardized manner that promotes the interoperation, fault detection and fault recovery capabilities, flexibility, extensibility, and consistency of such monitoring. In accordance with these techniques, a monitoring node18may be configured to accept the registration of one more monitored nodes14, and may performing the monitoring of processes performed thereupon. A monitored node14may register with a monitoring node18for monitoring, and may indicate one or more processes16executing on the monitored nodes14. The monitored node14may then notify the monitoring node18of the statuses of the respective processes16. In the event of a failure of a process16(e.g., metrics indicating the development of a problem, a reporting of a failure status, or a cessation of reported statuses), a monitoring node18may request that the monitored node14restart the process16. Alternatively, upon detecting the failure of an entire monitored node14(e.g., a cessation of reporting from the monitored node14or a lack of responsiveness), the monitoring node18may select a substitute monitored node14from the monitored node set, and may request the substitute monitored node14to restart each of the processes16that were executing on the monitored node14that has failed. Moreover, the monitoring nodes18may be configured to monitor each other; e.g., if a monitoring node18ceases reporting its status to the other monitoring nodes18, the other monitoring nodes18may collectively choose a substitute monitoring node18for the failed monitoring node18, and may reassign the monitored nodes14that had been assigned to the failed monitoring node18to the substitute monitoring mode18.

FIG. 2presents an exemplary scenario40featuring a monitoring of a set of processes16performed by a set of monitored nodes14and monitored by a set of monitoring nodes18. In this exemplary scenario40, a set of monitoring nodes18interoperates to perform the monitoring of the monitored nodes14and processes16. The monitored nodes14may be configured to, upon joining the monitoring scenario, register for monitoring, and the monitoring nodes18may confer to choose a monitoring node18to which the monitored node14is to be assigned. The monitored node14may receive a notification of the assignment, and may begin reporting a set of statuses42of respective processes16to a monitoring process20on the monitoring node18. The monitoring node18may therefore determine the status of the monitored node14and the processes16performed thereby. If a process16fails (as indicated by the statuses42reported thereto), the monitoring node18may request the monitored node14to restart the process16; but if an entire monitored node14fails, the monitoring node18may confer with the other monitoring nodes18choose a substitute monitored node14, and the processes16that had been executing on the failed monitored node14may be restarted upon the substitute monitored node14. Moreover, if a monitoring node18fails, the other monitoring nodes18may confer to choose a substitute monitoring node, and may reassign all of the monitored nodes14that had been assigned to the failed monitoring node18to the substitute monitoring node18. All of these actions may be reviewed by an administrator12, who may view a log shared by the monitoring nodes18and administrate the monitoring network (e.g., expanding the monitoring network by adding more monitored nodes14, processes16, and/or monitoring nodes18).

The techniques presented herein (including in the exemplary scenario40ofFIG. 2) may present some advantages over other monitoring techniques, such as those illustrated in the exemplary scenario10ofFIG. 1. As a first example, the techniques presented herein may represent a standardized monitoring framework, whereby any process16may participate in the monitoring through the reporting of information to a monitoring node18. As a second example, the techniques presented herein promote the extensibility of the monitoring scenarios; e.g., additional monitoring nodes18may be easily added to the monitoring node set18to share the computational load of the monitoring and improve the resiliency thereof, and a monitored node14comprising various processes16to be monitored may easily join the monitored node set to subscribe for monitoring. Additionally, this monitoring framework may enable monitoring to be offered, provided, and subscribed to as a standardized service; e.g., a monitoring host may offer a set of monitoring nodes18to which any set of monitored nodes14and processes16may be subscribed. As a third example, the fault detection and fault recovery among the processes16, monitored nodes14, and monitoring nodes18is significantly improved; e.g., a failure of any component may be detected and absorbed by the remaining resources of the monitoring scenario. Moreover, the ease and rapidity of fault recovery through these techniques may enable high availability of both the monitored processes and the monitoring service; e.g., device and process failures may be rapidly detected, and another monitoring node and/or monitored node may assume the role of a failed device. As a fourth example, the consistency of the monitoring scenario may be significantly improved, and the complexity thereof significantly reduced, by configuring the monitoring nodes18to record information and report to administrators12in a standardized manner. For example, an administrator12may view the status of all monitored resources by viewing one log that is shared by all of the monitoring nodes18. These and other advantages may be achieved through the implementation of monitoring according to the techniques presented herein.

FIG. 3presents a first embodiment of these techniques, illustrated as an exemplary method50of configuring a monitoring node18having a processor to monitor one or more monitored nodes14executing at least one process16. The exemplary method50may be implemented, e.g., as a set of processor-executable instructions stored in a memory component of the monitoring node18(e.g., a memory circuit, a platter of a hard disk drive, a solid-state storage device, or a magnetic or optical disc) and configured in a such a manner as to, when executed by the processor of the monitoring node18, cause the monitoring node18to perform the tasks of the exemplary method50. The exemplary method50begins at52and involves executing54the instructions on the processor. In particular, the instructions are configured to, upon receiving a request to monitor a monitored node14, register56at least one process16of the monitored node14for monitoring. The instructions are also configured to, upon receiving at least one status42from a process16of a monitored node14, record58the status42of the process16. The instructions are also configured to, upon detecting a failure of at least one process16of a monitored node14, restart60the process16on a monitored node14. In this manner, the exemplary method50causes the monitoring node18to perform fault detection and fault recovery of the processes16of one or more monitored nodes14, and so ends at62.

FIG. 4presents a first embodiment of these techniques, illustrated as an exemplary method70of configuring a monitored node14having a processor and executing at least one process16to be monitored by a monitoring node18. The exemplary method70may be implemented, e.g., as a set of processor-executable instructions stored in a memory component of the monitoring node18(e.g., a memory circuit, a platter of a hard disk drive, a solid-state storage device, or a magnetic or optical disc) and configured in a such a manner as to, when executed by the processor of the monitored node14, cause the monitored node14to perform the tasks of the exemplary method50. The exemplary method70begins at72and involves executing74the instructions on the processor. In particular, the instructions are configured to register76at least one process16with the monitoring node18. The instructions are also configured to report78(e.g., periodically) at least one status42of at least one process16to a monitoring node18. Additionally, the instructions are also configured to, upon receiving from a monitoring node18a request to restart a process16, restart80the process16. In this manner, the exemplary method70causes the monitored node14to enroll its processes16for monitoring by a monitoring node18and participate in the monitoring scenario according to the techniques presented herein, and so ends at82.

Still another embodiment involves a computer-readable medium comprising processor-executable instructions configured to apply the techniques presented herein. Such computer-readable media may include, e.g., computer-readable storage media involving a tangible device, such as a memory semiconductor (e.g., a semiconductor utilizing static random access memory (SRAM), dynamic random access memory (DRAM), and/or synchronous dynamic random access memory (SDRAM) technologies), a platter of a hard disk drive, a flash memory device, or a magnetic or optical disc (such as a CD-R, DVD-R, or floppy disc), encoding a set of computer-readable instructions that, when executed by a processor of a device, cause the device to implement the techniques presented herein. Such computer-readable media may also include (as a class of technologies that are distinct from computer-readable storage media) various types of communications media, such as a signal that may be propagated through various physical phenomena (e.g., an electromagnetic signal, a sound wave signal, or an optical signal) and in various wired scenarios (e.g., via an Ethernet or fiber optic cable) and/or wireless scenarios (e.g., a wireless local area network (WLAN) such as WiFi, a personal area network (PAN) such as Bluetooth, or a cellular or radio network), and which encodes a set of computer-readable instructions that, when executed by a processor of a device, cause the device to implement the techniques presented herein.

An exemplary computer-readable medium that may be devised in these ways is illustrated inFIG. 5, wherein the implementation90comprises a computer-readable medium92(e.g., a CD-R, DVD-R, or a platter of a hard disk drive), on which is encoded computer-readable data94. This computer-readable data94in turn comprises a set of computer instructions96configured to operate according to the principles set forth herein. In one such embodiment, the processor-executable instructions96may be configured to perform a method of configuring a monitoring node to monitor one or more processes on one or more monitored nodes, such as the exemplary method50ofFIG. 3. In another such embodiment, the processor-executable instructions96may be configured to implement a method of configuring a monitored node to participate in monitoring by one or more monitoring nodes, such as the exemplary method70ofFIG. 4. Some embodiments of this computer-readable medium may comprise a non-transitory computer-readable storage medium (e.g., a hard disk drive, an optical disc, or a flash memory device) that is configured to store processor-executable instructions configured in this manner. Many such computer-readable media may be devised by those of ordinary skill in the art that are configured to operate in accordance with the techniques presented herein.

The techniques discussed herein may be devised with variations in many aspects, and some variations may present additional advantages and/or reduce disadvantages with respect to other variations of these and other techniques. Moreover, some variations may be implemented in combination, and some combinations may feature additional advantages and/or reduced disadvantages through synergistic cooperation. The variations may be incorporated in various embodiments (e.g., the exemplary method50ofFIG. 3and the exemplary method70ofFIG. 4) to confer individual and/or synergistic advantages upon such embodiments.

A first aspect that may vary among embodiments relates to the scenarios wherein such embodiments may be utilized. As a first example, the monitoring techniques presented herein may be utilized to achieve the monitoring of many types of nodes, such as computers of various form factors (e.g., servers, workstations, consoles, notebooks, tablets, palmtop devices, and smartphones). As a second example, many types of processes16may be monitored according to the techniques presented herein, such as services (e.g., email servers, file servers, database servers, sensors, automation servers such as supervisory control and data acquisition (SCADA) components, and artificially intelligent processes. As a third example, many types of statuses42may be reported by such processes16, including an “OK” status, a “not OK” status, an “encountering difficulties” status, and a “locked up” status. Those of ordinary skill in the art may envision many scenarios wherein the techniques presented herein may be utilized.

A second aspect that may vary among embodiments of these techniques relates to the configuration of the monitoring of a monitored node14performing at least one process16. As a first example of this second aspect, a monitored node14and/or process16may be configured to send to the monitoring node18a series of persistence indicators that indicate the continued functionality of the monitored node14and/or process16. The monitored node14and/or process16may be configured to send a persistence indicator to the monitoring node18within a notification period (e.g., once per minute), and a monitoring node may be configured to detect a failure of the monitored node14and/or process16as an elapsing of a notification period without having received the persistence indicator.

As a second example of this second aspect, in addition to reporting a status42(e.g., as a persistence indicator), a process16may be configured to report its state to the monitoring node18, which may be configured to record the state of the process16. For example, the process16may comprise a state machine that may exist in various states and/or one or more data items upon which the process16is operating, and the process16may periodically report the state and the data items to the monitoring node18. This information may add detail to the reported status42of the process16, and may be useful in the event of a subsequent failure of the process16.

As a third example of this second aspect, a monitoring node18may be configured to take many types of actions upon detecting a failure of a monitored node14and/or process16. As a first example, the monitoring node18may simply record the failure in a log28, or may contact an administrator12with a notification of the failure. As a second example, the monitoring node18may request the monitored node14to restart a process16that has failed. Moreover, if the monitoring node18is monitoring a set of two or more monitored nodes14, the monitoring node18may select a substitute monitored node14, and may request the substitute monitored node14to restart the process16. This substitution may be advantageous, e.g., if the monitored node14is also performing other processes16that may be disrupted by an additional failure of the process16upon restarting on the same monitored node14, and/or for retrying the process16on a different monitored node14that may have different conditions that avoid a second failure of the process16. Moreover, this substitution may be advantageous, e.g., when an entire monitored node14fails, and when all of the processes16that had been performed by the monitored node14are to be restarted on one or more substitute monitored nodes14.

As a fourth example of this second aspect, if a process16is configured to report its state to a monitoring node16, then upon detecting a failure of the process16, the monitoring node18may request a restart of the process16on a monitored node14(including a substitute monitored node14) in the state last reported by the process16before failure. For example, a process16may comprise an algorithm performing a lengthy computation (e.g., the processing of a large array), and may periodically report to the monitoring process18its state within the algorithm (e.g., the index into the array that is currently being processed). If the process16fails, the monitoring process18may request a monitored node14to restart the process16at the last reported state, such as the last reported position within the algorithm (e.g., beginning with the last reported array index within the array). In this manner, the process16may be restarted without having to perform the previously performed states, thereby reducing a redundant performance of the completed portion of the algorithm and conserving the computing resources in the performance of the process16.

FIG. 6presents an illustration of an exemplary scenario100featuring a reporting to a monitoring node a set of states102of respective processes16performed by respective monitored nodes14. In this exemplary scenario100, two monitored nodes14are respectively performing two monitored processes16that are respectively configured to report the state102of the process16to the monitoring node18. For example, the processes16may report the states102to the monitoring node18periodically, or upon transitioning from a first state102to a second state102. The monitoring node18may record the states102of the processes16in a state log104. Accordingly, if a second process16executing on the first monitored node14experiences a failure106(e.g., if the second process16crashes, reports a problem, or fails to continue reporting states102and/or statuses42, such as performance indicators), the monitored node18may detect the failure106of the second process16, and may send to the monitored node14a request108to restart the second process16. Moreover, the monitoring node18may refer to the state log104, identify the last state102reported by the second process16(e.g., the second process16may have reported a second state102and then crashed), the monitoring node18may indicate in the request108that the monitored node14is to restart the second process16in the second state102. Similarly, a second monitored node14may be performing two processes16, but may experience a failure110(e.g., may overheat, lose power or network connectivity, or exhibit an operating system crash). The monitoring node18may detect the failure110of the second monitored node14, and may send to a third monitored node14a series of requests108to restart each of the processes16that the second monitored node14was performing at the time of the failure110. Moreover, the monitoring node18may indicate in the request108the state102in which the third monitored node14is to restart each process16, based on the states102of the processes16last reported before the failure110. In this manner, the monitoring node18may instruct the monitored nodes14to restart the processes16according to the states102last reported prior to a failure, thereby conserving the computing resources of the processes16within the monitoring scenario.

As a fifth example of this second aspect, a monitoring node18may be configured to, upon detecting a failure of a process16, take other actions in order to address the failure. For example, the monitoring node18may be configured to perform a particular logic (e.g., an invocation of a function or a set of instructions) when a process16enters a particular state102(e.g., when the process16raises a particular event). Moreover, the monitoring node18may receive the logic in advance from a monitored node14(e.g., during the registration of the monitored node14for monitoring) and/or process (e.g., when the monitored node14initiates a process16to be monitored by the monitoring node18). For example, a monitored node14or process16may comprise a dynamic link library (DLL) including one or more functions, and may provide the library to the monitoring node18with a request to perform one or more of the functions if the process16enters a particular state102. In this manner, a monitored node14may, while subscribing to a monitoring service, provide instructions to the monitoring node18to be performed in the event of a failure.

FIG. 7presents an illustration of an exemplary scenario120featuring monitoring node18configured to perform a logic124upon detecting a process16entering a particular state102(e.g., upon raising a particular event). In this exemplary scenario120, the monitoring node18receives from the monitored node14(e.g., while the monitored node14registers with the monitoring node18) a function library122, such as a dynamic link library (DLL) or a script, comprising a set of executable functions, as well as a specification of which functions are to be invoked upon a particular process16entering a particular state102. The monitoring process18may store this information, e.g., in a logic table124specifying a logic126to be performed upon a particular process16entering a particular state102, and may refer to the logic table124whenever a state102reports entering a new state102. Accordingly, in the exemplary scenario120ofFIG. 7, when the second process14reports to the monitoring node18that it has entered a second state, the monitoring process18may refer to the logic table102, determine that it is to perform a particular logic126in this event (e.g., invoking a third function of the function library122), and may perform the logic, resulting in a request128to the monitored node14to start a third process16in a particular state102(e.g., a repair process that addresses the particular type of failure indicated by the state102reported by the second process16). Those of ordinary skill in the art may devise many variations in the type of monitoring applied by a monitoring node18to a monitored node14and the processes16performed thereby in accordance with the techniques presented herein.

A third aspect that may vary among embodiments of these techniques relates to monitoring scenarios involving a set of monitoring nodes18that may provide additional features, such as improved fault tolerance and fault recovery, by interoperating to monitor the monitored nodes14and processes16. As a first example, each monitoring node18of the monitoring node set may store the statuses42received from the processes16of respective monitored nodes14, and may synchronize the statuses42with at least one other monitoring node18. This synchronization may enable the monitoring nodes18to share information about the statuses42of the processes16and to remain up to date about the status of the components in the monitoring scenario.

As a second example of this third aspect, when a failure of a monitored node14is detected, the monitoring nodes18may confer to choose a substitute monitored node14for the failed monitored node14. For example, a first monitoring node18may detect a failure of a monitored node14, but a second monitoring node18may be in communication with a second monitored node14that is performing few or no processes16, and the monitoring nodes14may confer to select the second monitored node14as the substitute node14for the failed monitored node14. This conferring may therefore enable a load-balancing effect in the choosing of substitute monitored nodes14in the event of a failure.

As a third example of this third aspect, respective monitored nodes14may be assigned for monitoring by a particular monitoring node18of the monitoring node set. For example, when a monitored node14registers for monitoring, the monitoring node set may confer to choose a monitoring node18to which the monitored node14is to be assigned (e.g., by choosing a monitoring node18that is currently monitoring few or no other monitored nodes14and/or processes16). When a process16of a monitored node14reports a status42or a state102, the monitored node14may send the status42or state102to the monitoring node18to which the monitored node14has been assigned (rather than sending the status42or state102to many monitoring nodes18, e.g., as a broadcast message). In this manner, the monitoring nodes18may perform a load-balancing among the monitoring nodes18, and may conserve the network resources of the monitoring scenario by reducing the broadcasting of reports of statuses42and/or states102.

As a fourth example of this third aspect, respective monitoring nodes18may be configured to monitor each other for failure, and to recover from such failure in a manner that does not disrupt the monitoring of the monitored nodes14and/or processes16. For example, respective monitoring nodes18may be configured to send persistence indicators to each other within a notification period (e.g., one persistence indicator from each monitoring node18per minute), and if a first monitoring node18detects that a notification period has elapsed without a second monitoring node18sending a persistence indicator, the first monitoring mode18may detect a failure of the second monitoring node18. Alternatively, the first monitoring node18may receive a failure indicator from a monitored node14regarding a second monitoring node to which the monitored node14is assigned, but that the monitored node14is unable to contact. A detected failure of a monitoring node18may also prompt the other monitoring nodes18to take various actions; e.g., the remaining monitoring nodes18may confer to choose a substitute monitoring node18, and may reassign to the substitute monitoring node18the monitored nodes14formerly assigned to the failed monitoring node18. This conferring may be performed via an election or consensus-building mechanism (e.g., a Paxos algorithm), where monitoring nodes18may nominate other monitoring nodes18as substitute monitoring nodes18for the failed monitoring node18, and a tallying of votes among the monitoring nodes18may lead to a consensus and an election of a substitute monitoring node18. The substitute monitoring node18may then contact the reassigned monitored nodes14to establish the reassignment. Moreover, if the failed monitoring node18has synchronized the statuses42and/or states102of the processes16of the reassigned monitored nodes14, then the substitute monitoring node18may quickly and easily assume the role of the failed monitoring node18. In this manner, the monitoring scenario may detect and recover from a failure of a monitoring node18without an interruption of monitoring service.

FIG. 8presents an illustration of an exemplary scenario130featuring the detection of and recovery from failures of monitoring nodes18of a monitoring node set. In this exemplary scenario130, respective monitoring nodes14of a monitored node set have assignments134for monitoring to a monitoring node set comprising four monitoring nodes18. The monitoring nodes18may communicate in various ways to identify a failure, and may respond to detected failures in various ways. As a first example, a first monitoring node18and a second monitoring node18may periodically exchange persistence indicators136to indicate continued performance. However, when the first monitoring node18fails to send a persistence indicator136to the second monitoring node18, the second monitoring node14may detect a failure138of the first monitoring node18, and may initiate a reassignment140of the monitored nodes14to the second monitoring node18. As a second example, a third monitoring node18may have assignments134to a set of monitored nodes14, but one such monitored node14may detect a failure142of the third monitoring node18(e.g., an inability to contact the monitoring node18while sending a status42or state102). The third monitoring node18may contact a fourth monitoring node18with a failure indicator144. The fourth monitoring node18may confer with the remaining monitoring nodes18of the monitoring node set (e.g., the second monitoring node18), and may negotiate a consensus146for a substitute monitoring node18to assume the role of the failed third monitoring node18; and upon being elected the substitute monitoring node18, the fourth monitoring node18may initiate a reassignment140of the monitored nodes14to the fourth monitoring node18. In this manner, the monitoring nodes18of the monitoring node set may collaborate to detect and recover from failures among the monitoring nodes18of the monitoring node set. Those of ordinary skill in the art may devise many ways of configuring the monitoring nodes18of a monitoring node set in accordance with the techniques presented herein.

FIG. 9illustrates an example of a system150comprising a computing device152configured to implement one or more embodiments provided herein. In one configuration, computing device152includes at least one processing unit156and memory158. Depending on the exact configuration and type of computing device, memory158may be volatile (such as RAM, for example), non-volatile (such as ROM, flash memory, etc., for example) or some combination of the two. This configuration is illustrated inFIG. 9by dashed line154.

In other embodiments, device152may include additional features and/or functionality. For example, device152may also include additional storage (e.g., removable and/or non-removable) including, but not limited to, magnetic storage, optical storage, and the like. Such additional storage is illustrated inFIG. 9by storage160. In one embodiment, computer readable instructions to implement one or more embodiments provided herein may be in storage160. Storage160may also store other computer readable instructions to implement an operating system, an application program, and the like. Computer readable instructions may be loaded in memory158for execution by processing unit156, for example.

Device152may also include communication connection(s)166that allows device152to communicate with other devices. Communication connection(s)166may include, but is not limited to, a modem, a Network Interface Card (NIC), an integrated network interface, a radio frequency transmitter/receiver, an infrared port, a USB connection, or other interfaces for connecting computing device152to other computing devices. Communication connection(s)166may include a wired connection or a wireless connection. Communication connection(s)166may transmit and/or receive communication media.

Device152may include input device(s)164such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, and/or any other input device. Output device(s)162such as one or more displays, speakers, printers, and/or any other output device may also be included in device152. Input device(s)164and output device(s)162may be connected to device152via a wired connection, wireless connection, or any combination thereof. In one embodiment, an input device or an output device from another computing device may be used as input device(s)164or output device(s)162for computing device152.

Those skilled in the art will realize that storage devices utilized to store computer readable instructions may be distributed across a network. For example, a computing device170accessible via network168may store computer readable instructions to implement one or more embodiments provided herein. Computing device152may access computing device170and download a part or all of the computer readable instructions for execution. Alternatively, computing device152may download pieces of the computer readable instructions, as needed, or some instructions may be executed at computing device152and some at computing device170.