Method and apparatus for adaptive declarative monitoring

A method of and apparatus for monitoring a computer system includes defining a monitoring policy for the computer system. At least one computer is employed to determine a status of a state of the computer system relative to the monitoring policy. At least one computer is employed to determine a condition of at least one monitored element to be monitored in the computer system based on the status of the state of the computer system. Furthermore, at least one computer is employed to monitor the condition of the at least one monitored element in the computer system, based on the monitoring policy. At least one computer is employed to perform an action in response to the condition assuming a predetermined status.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and derives the benefit of the filing date of U.S. Provisional Patent Application No. 60/989,757, filed Nov. 21, 2007. The entire content of this application is herein incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present disclosure relates to monitoring systems in general and, more particularly, to methods and apparatus for monitoring a system and automatically taking steps to remedy problems in the system without intervention from a user.

DETAILED DESCRIPTION OF SEVERAL EMBODIMENTS

FIG. 1illustrates an adaptive monitoring system10that enables autonomies. Autonomics is the ability to monitor a system, attend to issues discovered by the monitoring, and fix the discovered issues automatically, i.e., without user intervention. System10includes a user interface20, an autonomies engine30, a context40, monitoring agent50, and monitored elements60.

Generally, user interface20enables entry of policies to monitor the overall state of a system. Autonomics engine30receives these policies from user interface20, and transforms the policies into monitoring requirements. These monitoring requirements may identify the elements to be monitored to implement these policies, the possible states of these monitored elements, and the actions to be triggered given the current state of the monitored element and the requirement at hand. In order to perform these functions, autonomics engine30may interact with context40and monitoring agent50. Context40may store rules related to the monitoring requirements, configuration information related to the system, and/or data collected from monitored elements. Autonomics engine30may have bidirectional communication with context40to exchange this kind of information. Furthermore, autonomics engine30may communicate with monitoring agent50to ensure that appropriate elements of the system are being monitored as per the policies entered through user interface20.

Based on the information supplied by autonomics engine30, monitoring agent50monitors the condition of the monitored elements60and reports back on the monitored condition to autonomics engine30. If the state of the system is not normal, autonomics engine30may execute remedial actions. Thus, autonomics engine30may use policies entered through user interface20to turn system10into a self-managed, self healing system.

In an embodiment, the system to be monitored may be a computer system. The computer system may include hardware that is a combination of one or more servers, routers, switches, hubs, repeaters, and other such networking equipment. In addition, the computer system may also include a combination of application level and system level software running on the hardware. In an embodiment, user interface20may be used by a network administrator, network designer, or any other such person or group of persons involved in the maintenance of the system.

User interface20may be used to enter overall system states by making “statements.” The type of statements made by the user would depend on the type of system being monitored. For example, if a computer system is being monitored, user interface20, may be used by a network administrator to enter a “statement”, such as, for example, the following:Web tier of application X is saturated if the total number of concurrent connections serviced by web servers in some group exceeds some threshold for at least 5 minutes.Web server in some group is under used if current number of connections is less than 100 for 10 minutes.

In addition to the statements above, a user may also define actions that may be taken by system10in the event a condition in the statement is met. For example, a user may define the following actions that need to be taken based on the statements above:When web tier of application X is saturated, run a remediation workflow to provision a new virtual machine.When web tier is not saturated un-provision 20% of the web servers that are under used.When a web tier server is under used accept new connections to it.
A user may define the above-mentioned actions through user interface20. Furthermore, these actions, like the statements above, may be defined by the user in a natural language such as, for example, English or some sort of pseudo code.

Autonomics engine30may accept statements and actions such as those mentioned above, and translate each of these statements and actions into monitoring agents50which monitor monitored elements60on specific computers. For example, in an embodiment, monitored elements60would be the web tier of application X and a web server. In addition, autonomies engine30may define the conditions that need to be monitored. For example, based on the statements mentioned above, the conditions to be monitored are the number of concurrent connections serviced by web servers in a group for a set period of time.

Based on the statements and actions entered through user interface20, autonomies engine30may create autonomic rules also known as Action Rules that take action on the system. These Action Rules are the statements and actions defined by a user in a natural language that are converted into executable code by autonomics engine30.

Thus, a combination of states, rules and actions, termed a policy, can essentially turn an application into a self-managed, self healing system. Autonomics engine30is responsible for deriving and consolidating the monitoring requirements resulting from multiple active management policies and for distributing them to monitoring agents50. Monitoring agents50in turn monitor monitored elements60based on information obtained from autonomic engine30.

In an embodiment, autonomies engine30may be one or more computers running one or more software modules that perform the functions listed above. For example, in an embodiment, autonomics engine30may a single computer that runs one or more software modules that perform the various functions listed above. On other hand, autonomics engine30may include multiple computers linked to each other and running software modules that perform the functions above. The computers used as autonomics engine30may be generic or specialized computers including components such as, for example, a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory) and I/O devices.

Autonomics engine30may exchange information with context40. Context40is a set of data structures that may contain information regarding the application of the monitoring policy. The stored data may represent rules, data collected from monitored elements60, thresholds, computer names, etc. The context may be changed manually through the user interface20or in the actions of an Action Rule. In an embodiment, context40may be implemented on the same computer running autonomies engine30or alternatively on another computer separate from the one running autonomics engine30.

Autonomics engine30may also be in bidirectional communication with each monitoring agent50. In an embodiment, each monitoring agent50may be software processes that run on various computer systems. Furthermore, each monitoring agent50may be capable of monitoring for a given condition on one or more monitored element60and report when the condition is met by monitored element60. These conditions may be, for example, thresholds, Boolean conditions, and other such variables. In addition, each monitoring agent50may start and stop monitoring a monitored element60based on the Action Rule and the condition of monitored element60.

Each monitoring agent50may be configured to be in communication with one or more monitored elements60. Monitored Elements60are distinct elements whose status influence the behavior of a distributed system and can be collected by corresponding software monitoring agents50on various computer systems. In an embodiment, monitored elements60may be a hardware component, a software component, or any combination of the two in the monitored system. For example, in a computer system being monitored, monitored elements60may be a web server or an other application running on a server.

As described above, the status of monitored element60may cause the behavior of the overall system to change. For example, as indicated in an Action Rule above, if the web tier of application X in a computer system gets saturated, autonomies engine30may automatically run a remediation workflow to provision a new virtual machine. In another example, if a web tier server is under used, an Action Rule will allow it to accept new connections.

Each monitored element60is associated with one or more variables which help define the status of monitored element60. These variables may include, for example, a monitored condition of monitored element60and a state of monitored element60. Furthermore, the variables associated with monitored element60may be controlled automatically by autonomics engine30via Action Rules or may be controlled manually through user interface20.

A monitored condition is a first order logic expression based on the status of one or more monitored elements60, on context40, and on a date and time. For example, the relation of the total number of concurrent connections serviced by a web server within a specified time duration on a particular day to a threshold level which may be specified in context40may constitute a monitored condition for the web server. In addition, automatic states for monitored elements60may be defined by a monitored condition. The system may control the automatic state of monitored element60via one or more Action Rules.

Action Rules include a set of actions that need to be executed based on a Boolean condition on the states and on a monitoring condition. These actions include the ability to perform one or more of the following: set or reset the State; enable or disable Action Rules; run an external process or a remediation workflow; and change elements of the context. Thus, in an embodiment, when a web server services more than a threshold number of concurrent connections for a predefined period of time, an Action Rule may run a remediation workflow to provision a new virtual machine automatically without any intervention from a user. On the other hand, monitored element60may also have a user controlled state. A user controlled state may be a single Boolean value that is set/reset by a user, in an Action Rule as mentioned above, or by some external system.

By use of autonomies engine30that formulates and executes Action Rules to self-heal a system based on policies set forth through user interface20, system10may be considered to be an Adaptive Declarative Monitoring system. Adaptive Monitoring to be implemented through user interface20is the ability to monitor only the necessary monitored elements60given an overall state of system10composed of the status of every state (Automatic or User Controlled) of every monitored element60and the current context40. Declarative Monitoring allows Adaptive Monitoring to be implemented through user interface20without having to logically program when to start/stop monitoring elements on the components of the overall system10.

FIG. 2illustrates a block diagram indicating the interaction between components of an adaptive monitoring system according to an embodiment. Specifically,FIG. 2illustrates autonomies engine30in more detail. Autonomics engine30includes a policy builder110, a policy manager120, and a rules engine130. In addition, context40, monitoring agent50, and monitored elements60are also part of system10. Because context40, monitoring agent50, and monitored elements60have been discussed above, a repeat discussion of these components of system10will be omitted to avoid confusion.

In an embodiment, user interface20accesses policy builder110that is part of autonomics engine30to define monitoring policies. Policy builder110may be a GUI software running on a generic computer or specialized computer, and may be available online as a computer application. They syntax and semantics of the definitions in the monitoring policies may be saved in XML (Extensible Markup Language) files. XML is a flexible way to create common information formats and share both the format and the data on the World Wide Web, intranets, and elsewhere. XML is a formal recommendation from the World Wide Web Consortium. Using policy builder110a context40, the state of monitored element60and Action Rules may be specified through user interface20. The states and Action Rules may be translated to XML data using XSL transformations and may be saved persistently on system10. Furthermore, user interface20may be used to activate and deactivate a monitoring policy from the GUI of policy builder110.

Policy manager120is also a part of autonomies engine30. Policy manager120may be a separate software process that runs on the same computer that runs policy builder110. On the other hand, policy manager120may run on a separate computer that hosts policy builder110. Of course policy manager120and policy builder110can be integrated into a single software process. Policy manager120may convert the XML definitions of the policies entered through user interface20into execution rules using XSLT (XSL Transformations). XSLT is a standard way to describe how to transform (change) the structure of an XML (Extensible Markup language) document into a document with a different structure. XSLT is a recommendation of the World Wide Web Consortium. Policy manager120uses XSLT to produce a dynamic rule-based application suitable for execution by rules engine130. Based on the rule execution by rules engine130, policy manager120may communicate with monitoring agent50to start and stop monitoring individual monitored elements60. Monitoring agent50may report back to policy manager120with updated data once the monitoring condition on monitored element60is met.

In an embodiment, when user interface20is used to activate a monitoring policy, policy manager120may load the currently active XML files that contain the monitoring policy and use XSL transformations to generate rules in the syntax of rules engine130. For example, two sets of rules may be generated for each automatic state of monitored element60.(1) For every monitored element60referenced in the monitoring condition of its automatic state, generate the following rule in the Rules Engine Syntax, using an XSL Transformation:ifelement is not monitored on corresponding monitoring agent and policy is activatedthensend a request to the corresponding Monitoring Agent to start monitoring it(2) For every monitored element60referenced in the monitoring condition of its automatic state, generate the following rule in the Rules Engine Syntax, using an XSL Transformation:ifelement is monitored on corresponding Monitoring Agent and policy is de-activatedthensend a request to the corresponding Monitoring Agent to stop monitoring it.
Other such rules may be generated for each automatic state of monitored element60without departing from the scope of this disclosure.

Rules engine130may be a generally available computer software process capable of inferring logical consequences from a set of asserted facts and a set of rules. Several rule engines are available, including commercial and open source choices. Commercial rules engines usually let a user interface20to express rules in a proprietary English-like language. Rules engine130may receive XSL transformed rules from policy manager120and may create dynamic code that can be executed without run-time compilation.

Rules engine130evaluates the Action Rules when a monitoring policy is either activated or deactivated or a report is received from monitoring agent50. The evaluation may include: asserting facts; if the facts are true, then rules engine130may trigger actions; the actions may change information to be stored in context40, hence changing facts; and the context may be saved for recovery. For example, rules engine130may trigger the following three different actions based on the state of monitored element60, whether a monitoring policy is activated or deactivated, and whether monitored element60is actually being monitored or not monitored. The actions are indicated in the rules below in the “then” section.(1) For every monitored element60referenced in the monitoring condition of the Action Rule, generate the following rule in the Rules Engine Syntax, using an XSL Transformation:ifelement is not monitored on corresponding Monitoring Agent and policy is activated and Boolean expression of States is truethensend a request to the corresponding Monitoring Agent to start monitoring it.(2) For every monitored element60referenced in the Monitoring Condition of the Action Rule, generate the following rule in the Rules Engine Syntax, using an XSL Transformation:ifelement is monitored on corresponding Monitoring Agent and either policy is de-activated or Boolean expression of States is falsethensend a request to the corresponding Monitoring Agent to stop monitoring it.(3) Generate the following rule in the Rules Engine Syntax, using an XSL Transformation:ifMonitoring Condition is true and policy is activated and Boolean expression of States is truethenexecute the appropriate actions specified by the monitoring policy.

Monitoring agent50may be an independent computer software process that runs on various computers. Monitoring agent50may be able to start and stop monitoring for a specific condition that needs to be met by monitored element60. Once the condition has been met, monitoring agent50may return the information back to policy manager120with the updated context. Policy manager120and monitoring agent50may communicate with each other using standard data networking communication protocols such as TCP/IP, UDP/IP, etc.

Of course, two or more of policy manager120, policy builder110, rules engine130and/or monitoring agents50may be integrated into the same software process.

FIG. 3illustrates a flow chart300indicating the steps taken to implement an adaptive monitoring system according to an embodiment. At step310, a user defines a monitoring policy for a computer system that needs to be monitored. At step320, a status of a state of the computer system is determined relative to the monitoring policy. At step330, a condition of a monitored element to be monitored in the computer system is determined based on the status of the state of the computer system. At step340, the condition of the monitored element is monitored based on the monitoring policy. At step350, action is performed based on the results of the monitoring and on a predetermined status of the monitored element.

While the present disclosure has been described in connection with the illustrated embodiments, it will be appreciated and understood that modifications may be made without departing from the spirit and scope of the disclosure.