Abstract:
The invention provides method and apparatus for maintaining a networked computer system including first and second nodes and an event processing server, the method comprising the first and second nodes detecting changes in state, the event processing server receiving notification of the changes in state from the first and second nodes, the event processing server correlating changes in state detected in the first and second nodes, and the event processing server executing a maintenance decision which affects the first and second nodes. The detecting, transmitting, correlating, and executing occurs without human intervention.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
   This application claims priority to provisional U.S. application Ser. No. 60/298,592 filed Jun. 14, 2001 and entitled Stateful Distributed Event Processing and Adaptive Security, the disclosures of which are incorporated herein by reference. 
   This application is also related to co-pending U.S. application Ser. No. 10/071,328 filed Feb. 8, 2002 and entitled Stateful Reference Monitor, the disclosures of which are incorporated herein by reference. 

   BACKGROUND 
   This invention relates to computer network security, and more particularly to methods and apparatus for securing one or more nodes on a computer network. 
   Conventional network security systems can be said to provide either “active” or “passive” protection. Active security systems provide real-time barriers to intrusions, via software- or hardware-based pre-programmed intrusion detection measures. “Passive” systems provide the ability to detect and recover from previously observed security breaches, by examining data gathered about previous system access activity, so as to improve static access controls and policies over time. Active systems, then, function primarily to prevent intrusions, and passive systems function primarily to report on and examine data about previous intrusions to prevent future intrusions. 
   Examples of conventional active security systems include access control tools, content filtering tools, and system auditing tools. Access control tools, such as network firewalls, can be deployed on dedicated machines, usually at a network perimeter, to control inbound and outbound access using pre-configured permission levels. Content filtering tools, like computer virus scanners, typically execute on either an e-mail server or a workstation, and function by screening incoming content, like e-mail and attached files, for potentially threatening matter, based on known signatures of previously observed attacks. System auditing tools, like reference monitors, may provide either stateless or state-based monitoring (such as the state-based monitoring provided by the stateful reference monitor described in U.S. patent application Ser. No. 10/071,328 and incorporated by reference herein) of individual workstations or servers, by identifying variations from either pre-determined settings or a dynamic machine state. 
   Examples of conventional passive security systems include activity logging tools and auditing tools, which may be employed in conjunction with one another. Activity logging tools track the activity of one or more computers and transcribe observed system activity to a series of log files as individual entries. Auditing tools typically examine those log entries to discern breaches, attacks, or other potentially threatening activity, occurring either across machines or within individual machines. 
   Both types of security systems provide useful intrusion detection and prevention functions. However, both generally rely on pre-programmed network administration policy, business rules, or other parameters, and so neither (particularly passive systems) provides the adaptation capability sometimes necessary to counter novel types of attacks as they occur. Also, conventional active systems are unable to observe and correlate seemingly innocuous activity as it occurs across nodes to determine that an intrusion is in progress. Given the growing ubiquity of computer networks and the value of electronic assets, commensurate growth of network security threats is to be expected. Therefore, a security system which provides adaptive countermeasures in real time to localized (i.e., limited to one node) or non-localized intrusions would provide tremendous value to operators of computer networks. 
   SUMMARY OF THE INVENTION 
   A first embodiment of the invention provides a method of maintaining a networked computer system including first and second nodes and an event processing server, comprising the first and second nodes detecting changes in state, the event processing server receiving notification of the changes in state from the first and second nodes, the event processing server correlating changes in state detected in the first and second nodes, and the event processing server executing a maintenance decision which affects the first and second nodes, wherein the detecting, transmitting, correlating, and executing occurs without human intervention. 
   This embodiment may be practiced wherein the changes in state are a result of at least one of an event and the absence of an event, wherein the changes in state are recognized by a reference monitor, and/or wherein the event processing server receiving the report is the result of one of the first and second nodes reporting to the event processing server and the event processing server polling the first and second nodes. The embodiment may further include the event processing server updating an operating policy on the network, and updating the operating policy may include at least one of requesting security policy changes on at least one node, requesting changes to privileges to access system resources on at least one node, tuning system parameters on at least one node, and modifying network firewall parameters. At least one node may further enact the updated operating policy. Also, the embodiment may further include notifying an external entity of actions taken, and the external entity may be a network administrator or a software application executing on the network. 
   A second embodiment of the invention provides a method for maintaining a networked computer system including at least one node detecting a change in state, an event processing server on the network receiving notification of the at least one change in state from the at least one node, and the event processing server responding to the notification by executing a maintenance decision, wherein the detecting, receiving, and responding occurs without human intervention. 
   This embodiment may be practiced wherein the change in state is a result of at least one of an event and the absence of an event, wherein the change in state is recognized by a reference monitor, wherein the event processing server receiving the report is the result of one of the node reporting to the event processing server and the event processing server polling the node, and may be practiced wherein the maintenance decision affects the at least one node detecting the change in state, and/or wherein the maintenance decision affects at least one node other than the node detecting the change in state. The embodiment may further include the event processing server updating an operating policy on the network, wherein updating the operating policy may include at least one of requesting security policy changes on at least one node, requesting changes to privileges to access system resources on at least one node, tuning system parameters on at least one node, and modifying network firewall parameters. The embodiment may still further include at least one node enacting the updated operating policy, and/or notifying an external entity of actions taken, wherein the external entity is a network administrator or a software application executing on the network. 
   A third embodiment of the invention provides a method for maintaining a node on a networked computer system including at least one node detecting a change in state, and the at least one node reacting to the change in state, wherein the at least one node detecting and reacting occurs without human intervention. 
   The embodiment may be practiced wherein the change in state is a result of at least one of an event and the absence of an event, and/or wherein the change in state is recognized by a stateful reference monitor. The embodiment may further include at least one node notifying an event processing server on the network, the event processing server responding to the notification by updating an operating policy on the network, wherein updating the operating policy includes at least one of requesting updates to security policy on at least one node, requesting changes to privileges to access system resources on at least one node, tuning system parameters on at least one node, and modifying network firewall parameters. The embodiment may further include the at least one node enacting the updated operating policy, and/or notifying an external entity of actions taken, wherein the external entity is a network administrator and/or a software application executing on the network. 
   A fourth embodiment of the invention provides a computer-readable medium having instructions recorded thereon, which instructions, when executed, enable at least one processor in a networked computer system to detect a change in state of a node, and process instructions defining reacting to the detected change in state. 
   The embodiment may further include instructions defining communicating the change in state to an event processing server, instructions defining processing maintenance instructions received from the event processing server, and/or instructions defining transmitting notification to a network administrator of actions taken. 
   A fifth embodiment of the invention provides a computer-readable medium having instructions recorded thereon, which instructions, when executed, enable at least one processor in a networked computer system to maintain an operating policy for the network, receive notification of a change in state from at least one node, and update the operating policy based on the change in state. 
   The embodiment may further include instructions defining storing received notifications of changes in state in memory, instructions defining correlating notifications received from a plurality of nodes, instructions defining storing received notifications in electronic file storage, and/or instructions defining notifying an external entity of actions taken, wherein the external entity is a network administrator or a software application executing on the network. 
   A sixth embodiment of the invention provides a method for maintaining a networked computer system including at least one node detecting a change in state, an event processing server on the network receiving notification of the at least one change in state from the at least one node, and the event processing server responding to the notification by dispensing a maintenance decision. 
   The embodiment may further comprise executing, by a human operator, the maintenance decision on at least one node on the networked computer system, or executing, without human intervention, the maintenance decision on at least one node on the networked computer system. A human operator may be prompted and allotted a predetermined period to execute the maintenance decision before it is executed without human intervention. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is a functional block diagram depicting the interaction of system components which define aspects of at least one embodiment of the invention; 
       FIG. 2  is a functional block diagram depicting the interaction of system components which define aspects of at least one other embodiment of the invention; 
       FIG. 3  is a functional block diagram depicting the interaction of system components which define aspects of at least a third embodiment of the invention; 
       FIG. 4  is a block diagram of an exemplary computer system on which aspects of embodiments of the invention may be implemented; 
       FIG. 5  is a block diagram depicting exemplary computer system components with which aspects of embodiments of the invention may be implemented; and 
       FIG. 6  is a functional block diagram depicting the interaction of system components which define aspects of at least another embodiment of the invention. 
   

   DETAILED DESCRIPTION 
   Aspects of embodiments of the present invention provide methods and apparatus for securing a networked computer system through the coordinated execution of reference monitor and event agent software on individual nodes, and event processing server software on a network server, for achieving active security measures, administrative control, and the ability to correlate potentially threatening activity across multiple nodes in real time. 
   Computer system  400 , shown in  FIG. 4 , with which aspects of these embodiments, either individually or in combination, may be implemented, may include at least one main unit connected to both one or more output devices  401  which store information, transmit information or display information to one or more users or machines, and one or more input devices  402  which receives input from one or more users or machines. The main unit may include one or more processors  403  connected to a memory system  404  via one or more interconnection mechanisms  405 , such as a bus or switch. Any input device  402  and/or output device  401  are also connected to the processor  403  and memory system  404  via the interconnection mechanism  405 . The computer system  400  may further include a storage system  406  in which information is held on or in a non-volatile medium. The medium may be fixed in the system or may be removable. 
   Alternatively, computer system  400  may be distributed, and therefore may not include a main unit. In particular, input devices  402 , processors  403 , memory systems  404 , interconnection mechanisms  405 , and storage systems  406  may each comprise individual or multiple computer systems, and may be geographically disparate. For example, storage systems  406  may comprise a server farm residing in New York which communicates with a processor  403  residing in Pennsylvania, via the Internet, which serves as interconnection mechanism  405 . 
   Computer system  400  may be a general purpose computer system which is programmable using a computer programming language. Computer programming languages suitable for implementing such a system include procedural programming languages, object-oriented programming languages, combinations of the two, or other languages. The computer system may also be specially programmed, special purpose hardware, or an application specific integrated circuit (ASIC). 
   In a general purpose computer system, the processor is typically a commercially available processor which executes a program called an operating system, which controls the execution of other computer programs and provides scheduling, debugging, input/output control, accounting, compilation, storage assignment, data management, memory management, communication control and related services. The processor and operating system defines the platform for which application programs in other computer programming languages are written. The invention is not limited to any particular processor, operating system or programming language. 
   Storage system  406 , shown in greater detail in  FIG. 5 , typically includes a computer-readable and computer-writeable non-volatile recording medium  501 , in which data is stored that define a program to be executed by the processor, or information stored to be processed by the program. The medium may, for example, be a disk or flash memory. Typically, in operation, the processor causes data to be read from the nonvolatile recording medium  501  into another memory  502  that allows for faster access to the information by the processor than does the medium  501 . This memory  502  is typically a volatile, random access memory such as a dynamic random access memory (DRAM) or static memory (SRAM). It may be located in storage system  406 , as shown, or in memory system  404 , not shown. The processor  403  generally manipulates the data within the integrated circuit memory  404 ,  502  and then copies the data to the medium  501  after processing is completed. A variety of mechanisms are known for managing data movement between the medium  501  and the integrated circuit memory element  404 ,  502 , and the invention is not limited thereto. The invention is not limited to a particular memory system  404  or storage system  406 . 
   Aspects of embodiments of the invention may be implemented in software, hardware or firmware, or any combination thereof. The various elements of an embodiment, either individually or in combination, may be implemented as a computer program product including a computer-readable medium, e.g. storage  406 , on which instructions are stored for access and execution by a processor, e.g. processor  403 . When executed by the processor  403 , the instructions instruct the processor  403  to perform the various steps of the process. 
     FIG. 1  is a functional block diagram depicting the relationship between system components, such as those described above, adapted to enable aspects of embodiments of the invention. 
   Reference Monitor  25  executes on Node A, which may be a workstation, server, or other computer on the network. Reference Monitor  25  may be a software application, which may execute on Node A synchronously, asynchronously, or both, providing continuous monitoring capability. Reference Monitor  25  may be adapted to execute on nodes running any commercially prevalent operating system such as UNIX, LINUX, Windows NT, and others. Reference Monitor  25  acts to detect and intercept local node operations and/or network-originated requests through which the operating system and/or user applications attempt to access system resources on the node. In some embodiments, in order to monitor system events, Reference Monitor  25  comprises Interceptors  26 , which are inserted in the control or communication paths traversed by those events. For example, if an particular monitored event is a network access request, an Interceptor  26  may be inserted in the operating system at a point where the network access request is communicated from one portion of the operating system to another. Interceptor  26  may generate an event message for each event intercepted. Event messages may then be communicated to Reference Monitor  25 , which may return a policy message to Interceptor  26 . The policy message may be an action for Interceptor  26  to take, such as allowing an access request event to continue along the control or communication path so that it has its intended effect, or not allowing the event to pass along the path. Reference Monitor  25  may instead, or also, construe the absence of an anticipated request for system resources as a harmful activity. In these embodiments, Reference Monitor  25  may interpret an overall absence of event messages, of certain types of event messages, or of event messages received within a certain timeframe, for example, as cause to return a policy message to Interceptor  26 , such as an instruction to not to allow an event to continue along its intended path. 
   Event Agent  45  also executes on Node A, and in this embodiment is a software application which executes continuously in the background on Node A. Event Agent  45  executes in conjunction with, and may be integrated with, Reference Monitor  25  and coordinates communication between Reference Monitor  25  and Event Processing Server  100 . Event Agent  45  may be adapted to execute on nodes running any commercially prevalent operating system such as UNIX, LINUX, Windows NT, and others. Event Agent  45  may execute on the same physical machine as Reference Monitor  25 , or on a different physical machine (not shown) if the machines are networked. If Event Agent  45  and Reference Monitor  25  execute on different physical machines, they will preferably communicate via any secure network protocol that supports message integrity and authentication, such as HTTPS and others. Secure network protocols are desirable so that Event Agent  45  and Reference Monitor  25  can validate the origin and content of communication received. 
   Event Processing Server  100 , in this embodiment, is a software application in communication with one or more Event Agents  45 , and may execute on a workstation or server residing within the same sub-network as Event Agents  45 , or on a different sub-network connected via a router, gateway or other component. Data transport between Event Agent  45  and Event Processing Server  100  may be accomplished via such secure communication protocols as HTTPS and/or others. In the embodiment depicted, Event Processing Server  100  consists of integrated software components Transceiver  115 , Loader  125 , Instruction Engine  135 , Correlation Engine  145 , and Electronic File Storage  155 . Transceiver  115  receives event notifications from Event Agent  45  and prepares them for processing. Data is passed to Loader  125 , which prepares it for loading to Electronic File Storage  155 . In a preferred embodiment, Electronic File Storage  155  is a database organized to provide a constantly updated representation of the status of Event Agents  45  in quickly accessible form. Instruction Engine  135  and Correlation Engine  145  process data in Electronic File Storage  155 . Instruction Engine  135  determines whether individual notifications received from nodes warrant policy updates—for instance, determining whether a notification indicates an active attack (such as a buffer overrun attack) or a passive attack (such as a virus)—and determines steps to be taken, such as placing nodes in quarantine, defining system operations which may not be performed on any machine on which a Reference Monitor  25  executes, or tuning operating system, network, or firewall parameters. Instruction Engine  135  passes instructions to be issued to Transceiver  115 , which then transmits them to Event Agent  45 . Correlation Engine  145  determines whether event notifications, when considered in combination, warrant policy updates. By continuously tracking and analyzing the activity reported by various event agents across one or more networks, Event Processing Server  100  is able to correlate events that may seem unrelated across the distributed system to recognize potential attacks. Attacks may be defined by combinations of events, such as attempting to access a particular resource together with writing a particular file. Other relationships between time and machine resources accessed may also signify an attack. 
   Reference Monitor  25 , Event Agent  45 , and Event Processing Server  100  may interact in various ways to provide local protection of individual nodes, remote adaptive protection of one or more nodes, and correlative protection for one or more nodes, as described in detail below. 
   Local protection of individual nodes is provided by Reference Monitor  25 . In the embodiment depicted in  FIG. 1 , an instruction arrives in  10  from Network  5  to Application  15 , which may be an e-mail, browser, terminal server, or other software application running on Node A. This network-based instruction received in  10  causes Application  15  to issue a corresponding request in  20  for System Resources  35  (i.e., access to disk or CPU), which is detected and routed through Reference Monitor  25 . If the request does not violate pre-programmed administrative policies, which in the embodiment shown are stored as coded instructions within the Reference Monitor or in a database on hard disk  35 , the Reference Monitor allows Application  15  access to System Resources  35  in  30 . If the request violates these policies, Reference Monitor  25  may prevent Application  15  from accessing System Resources  35  in  30 . 
   Reference Monitor  25  may work in conjunction with Event Agent  45  and Event Processing Server  100  to provide remote adaptive protection in addition to, or instead of, local protection. Also depicted in  FIG. 1 , Reference Monitor  25  may perform basic analysis on the instruction arriving in  10  and communicate with Event Agent  45  in  40  as to its nature, as defined by current administrative policy. Event Agent  45  then sends a notification via Network  5  in  50  and  110  (which constitute transfer of the same notification) to Event Processing Server  100  as to the nature of the activity. Event Processing Server  100  receives the notification in  110  from Network  5 . Transceiver  115  receives the notification in  110  from Event Agent  45  and prepares it for processing. This data is then passed in  120  to Loader  125 , which prepares it for loading, and in  130  initiates the load of the data to Electronic File Storage  155 . Once loaded, Instruction Engine  135  and Correlation Engine  145  process the data in Electronic File Storage  155  in  140  and  150 , respectively. Instruction Engine  135  determines whether an update to administrative policies is warranted, and determines other steps to be taken, including placing Node A in quarantine, defining system operations which may not be performed on Node A, tuning Node A&#39;s operating system, or modifying network or firewall parameters. An update to administrative policies may be issued in response to a single request to access system resources, or a combination of requests to access system resources, reported by Reference Monitor  25 . Instruction Engine  135  issues instructions to update policies on Node A in  160  to Transceiver  115 , which transmits instructions in  180  and  190  via Network  5  to Event Agent  45 . 
   In certain embodiments, in addition to notifying Event Processing Server  100  software as to intercepted operations, Event Agent  45  application will periodically poll Event Processing Server  100  for administrative policy updates at a pre-programmed frequency. It is preferable for Event Agent  45  software to poll Event Processing Server  100 , and not the opposite, for at least one reason. Specifically, the node on which Event Agent  45  executes usually knows the address of Event Processing Server  100 , and the opposite may not be true, particularly if the node is a mobile laptop. However, in some cases, with sufficiently stringent security measures protecting Event Processing Server  100 , it might be safely programmed to poll or broadcast to Event Agents  45  to alert them to administrative policy updates. 
   A second embodiment of remote adaptive protection is depicted in  FIG. 2 , which illustrates an attack to a single node being diagnosed by Event Processing Server  100 , which then acts to prevent subsequent attacks on another node. In  10 A, data—which may be a request, message, or other content—arrives at Application  15 A running on Node A. Application  15 A may, for instance, be an e-mail application or other application designed to receive data or requests from other nodes on this or other networks. As described in the foregoing, when Application  15 A attempts to access system resources  35 A in  20 A, the attempt is intercepted by Reference Monitor  25 A. Reference Monitor  25 A decides whether the attempt to access resources is acceptable based on pre-programmed policies (in which case it allows resource access in  30 A), and alerts Event Agent  45 A in  40 A as to the attempt. Event Agent  45 A transmits a notification to Event Processing Server  100  in  50 A. 
   Event Processing Server  100  processes the notification, and determines after loading the data in  130  that the attempt to access resources on Node A should not have been allowed (for instance, by determining that the data arriving from the network was a virus not previously recognized by Reference Monitor  25 A). In  190 B, it transmits instructions to Event Agent  45 B to update administrative policies, so as to restrict access to system resources  35 B. System resources  35 A,  35 B may comprise multiple components, each of which may be accessed separately. For example, an e-mail virus may have attempted to access ten components comprising system resources  35 A in sequence, and Event Agent  45 B may attempt to restrict access to a subset of those ten components, all ten components, or those and other components within system resources  35 B. Event Agents  45 A,  45 B work in conjunction with Reference Monitor  25 A to determine the components accessed. If Reference Monitor  25 A is a stateful reference monitor, Reference Monitor  25 A recognizes accessed components because it maintains state. 
   Event Agent  45 B passes instructions in  41 B to Reference Monitor  25 B, which updates its administrative policies to incorporate these instructions. When data arrives from the network in  10 B to Application  16 B, which need not be the same application as application  15 A, a similar attempt to access system resources is initiated in  20 B. Reference Monitor  25 B, having updated its administrative policies, denies this attempt (denoted with an ‘X’ in  FIG. 2 ). In this manner, Event Processing Server  100  prevents potentially damaging activity from affecting more than one node on the network, by identifying possibly malicious instructions or requests and preventing them from being executed through updates to administrative policy, preferably in real time. 
   An example of correlative protection is depicted in  FIG. 3 , which depicts processing of notifications received from multiple nodes to determine that an attack is in progress, and preventing subsequent similar activity on other nodes. As in the foregoing embodiments, in  10 A and  10 B, data—which may be a request, message, or other content —arrives at Applications  15 A and  16 B, running on Nodes A and B, respectively. Applications  15 A and  16 B may, for instance, be e-mail applications or other application designed to receive data or requests from other nodes on this or other networks, but Applications  15 A and  16 B need not be the same application. As described in the foregoing, when Applications  15 A and  16 B attempt to access system resources  35 A and  35 B, in  20 A and  20 B respectively, the attempts are intercepted by respective Reference Monitors  25 A and  25 B. The Reference Monitors verify that attempts to access resources are acceptable based on pre-programmed policies, and allow access in  30 A and  30 B. Reference Monitors  25 A and  25 B also alert Event Agents  45 A and  45 B, in  40 A and  40 B respectively, as to this activity. Event Agents  45 A and  45 B transmit notification of the activity to Event Processing Server  100 , which loads it to Electronic File Storage  155  for processing. 
   Event Processing Server  100 , which in this embodiment includes a Correlation Engine  145  component as depicted in  FIG. 1 , processes data in Electronic File Storage  155  to determine, by comparing monitored activity to pre-programmed combinations and timeframes, that the actions on Nodes A and B represents possibly harmful activity. Event Agents  45 D and  45 E running on Nodes D and E receive these notifications, and pass instructions to respective Reference Monitors  25 D and  25 E to update administrative policies in  41 D and  41 E respectively. In some embodiments Event Agents  45 A and  45 B receive these notifications as well, and pass instructions to Reference Monitors  25 A and  25 B to update administrative policies in  41 A and  41 E, respectively. In alternative embodiments, instructions may include administrative policy changes specific to certain nodes, placing a particular node in quarantine, tuning network or firewall parameters, or other instructions. In the embodiment depicted, Reference Monitors  25 D and  25 E implement the updated administrative policy, so that when subsequent network requests arrive at Applications  17 D and  18 E in  10 D and  10 E respectively, requests for System Resources  35 D and  35 E are denied (denoted with an ‘X’ in  FIG. 3 ). In this manner, Event Processing Server  100  discerns that activity occurring on multiple workstations, when considered in combination, represents potentially malicious or harmful events, and transmits instructions to prevent the activity from subsequently affecting one or more other nodes on the network, in real time. 
   In certain embodiments, Event Processing Server  100  also disseminates instructions, such as policy updates or network parameter modifications, when polled by Event Agents  45 . However, in certain embodiments and/or emergency situations, Event Processing Server  100  may transmit unsolicited instructions to one or more Event Agents  45  to update administrative policies. 
   In certain embodiments, Event Processing Server  100  sends a notification to relevant Event Agents  45  enabling pre-programmed security measures, although in other embodiments Event Processing Server  100  may issue instructions to impose new measures based upon an analysis of activity observed by Event Agents  45 . Security measure updates are preferably performed in real time, by issuing instructions to some number of Event Agents  45 , so as to thwart an observed attack as quickly as possible. 
   In certain embodiments, Event Agent(s)  45 , in addition to notifying Event Processing Server  100  of activity detected by Reference Monitor(s)  25 , may also notify other entities. In one example, Event Agent(s)  45  may notify network administration staff about the activity by pager, e-mail, SNMP trap, message to any other server or console, SMS message, or other communication method. In another example, Event Agent(s)  45  may notify software applications or other programmed components residing on the network of detected activity. For instance, Event Agent  45  may notify a firewall application, so that the firewall&#39;s configuration can be updated. Event Processing Server  100  may also notify these entities using these or other communication methods, either before or after issuing resulting instructions to Event Agents  45 . 
   Embodiments of the invention may be configured such that certain event processing servers communicate with other event processing servers to provide protection for a larger community of nodes on the network. As depicted in  FIG. 6 , Nodes A, B are in communication  10 ,  20  with Server C, which they notify upon recognizing changes in state, and Nodes D, E are in communication  30 ,  40  with Server F, which they notify upon recognizing changes in state. Servers C, F are in communication  60 ,  70  with Server G. Server G may or may not be in communication with still other servers (not shown). According to one embodiment of the invention, Servers C or F, upon being notified of a change in state by a node with which they are in communication, communicate the change in state and/or resulting administrative policy modifications to Server G, so that instructions and/or updates may be disseminated to other servers and/or other nodes. 
   Various embodiments will have various communication structures in place. For instance, the “hierarchical” communication structure comprised of Servers C and F in exclusive communication  60 ,  70  with Server G may be replaced or supplemented by a “flat” structure comprised of Servers C and F in communication  50  with each other, so that notification need not travel through Server G to reach other servers on the network. Various embodiments may employ a hierarchical structure, a flat structure, or a combination of both. 
   While the invention has been particularly shown and described with reference to specific embodiments, and variations thereon have been indicated, it will be understood by those skilled in the art that various additional changes in form and detail may be made therein without departing from the spirit and scope of the invention, as defined by the following claims.