Abstract:
In a communications network, a system and method for fault notification and correlation. The system includes a notification collector that serves as an intermediary for processing fault notifications. If the fault notification volume exceeds a defined threshold, such that in the prior art could involve dropped notifications, the notification collector processes the critical fault notifications. The notification collector may also assign a higher priority to processing the fault notifications than the faulty components do for generating the fault notifications. In this manner, the need for a fault correlation subsystem is reduced.

Description:
CROSS-REFERENCES TO RELATED APPLICATIONS 
     NOT APPLICABLE 
     STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
     NOT APPLICABLE 
     REFERENCE TO A “SEQUENCE LISTING,” A TABLE, OR A COMPUTER PROGRAM LISTING APPENDIX SUBMITTED ON A COMPACT DISK 
     NOT APPLICABLE 
     BACKGROUND OF THE INVENTION 
     The present invention relates generally to networking systems and specifically to a system and method for reporting and correlating network faults for network fault management. 
     Telecommunications networks often consist of a large number of distinct components and sub-systems, referred to as managed entities, that require configuration and monitoring. The state of a particular managed entity often depends on, and is partially driven by, the states of other managed entities. 
     When each of the managed entities change state, they often have to notify a management system of the state change. The state change is communicated to the management system by a Simple Network Management Protocol (SNMP) notification, for example. Since managed entities often depend on the state of other managed entities to determine their current state, a state change in one managed entity could cause state changes in tens of other managed entities, and could therefore trigger a cascade of tens or hundreds of notifications to the management system. 
     As a result, such a large number of notifications can potentially create a broadcast storm and flood the management system. In the case of an external management system, the communication channel between a node in the telecommunication network and management system can become severely congested due to the high number of notifications, thus reducing the overall throughput of the telecommunication network. Furthermore, a flood of notifications may cause one or more of the notifications to be dropped. In such a case, the management system will be unaware of the problems at the nodes corresponding to the dropped notifications. 
     Therefore, it is an object of the present invention to obviate or mitigate the above-mentioned disadvantages. 
     BRIEF SUMMARY OF THE INVENTION 
     In accordance with an aspect of the present invention, there is provided a network fault management system for managing a plurality of managed entities. Ones of the managed entities have a predefined dependency relationship with others of the managed entities. The network fault management system includes the following elements. A management system manages the plurality of managed entities. A local notification collector receives fault notifications generated by corresponding ones of the managed entities in response to a fault, and reports the fault notifications to the management system, thereby reducing the likelihood of dropping fault notifications. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       An embodiment of the present invention will now be described by way of example only with reference to the following drawings in which: 
         FIG. 1  is a block diagram of a network of managed entities; 
         FIG. 2  is a block diagram of a notification collector according to an embodiment of the present invention; and 
         FIG. 3  is a flowchart of a method according to an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     For convenience, like numerals in the description refer to like structures in the drawings. As previously described, many telecommunications networks include distinct components and sub-systems that require management by a management system. These component and sub-systems are referred to as managed entities. The management system is a remote entity that observes and configures one or more telecommunication networks. Managed entities include hardware components, physical interfaces, logical interfaces and services. Managed entities typically have interfaces to include or exclude them from the telecommunication network, and to display state information, statistics, and configuration parameters. Managed entities have a cohesive data set that describes and configures them, and may have state behavior that is influenced by operating stimuli. 
     A managed entity is often related to other managed entities in a dependency relationship. The relationship includes referential integrity between a dependent entity and an entity upon which it depends, referred to as the “dependee”. The referential integrity is generally in the form of a foreign key, which is an attribute (or group of attributes) that is used to refer to a primary key of a dependee. The primary key is an attribute (or group of attributes) that uniquely identifies the dependee. Further, there is often a state observation relationship as described by the “observer” pattern detailed in the 1995 book “Design Patterns” by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 
     An example of the relationship between dependent and dependee managed entities is illustrated as follows. Referring to  FIG. 1 , a portion of a telecommunication network is illustrated generally by numeral  100 . The illustrated portion  100  includes a management system  108 , a pair of asymmetric digital subscriber loop (ADSL) channels  106 , an ADSL physical line  104 , and a line card  102 . The two ADSL channels  106  are multiplexed over the ADSL physical line  104 , which is coupled to the line card  102 . Thus, the ADSL channels  106  are dependent on the ADSL physical line  104 , which is dependent on the line card  102 . 
     The primary key for the line card  102  is &lt;shelf, slot&gt; and the primary key for the ADSL physical line  104  is &lt;shelf, slot, port&gt;. It can be seen that there is referential integrity between the &lt;shelf, slot&gt; portion of the primary key of the ADSL physical line  104  and the &lt;shelf, slot&gt; of the primary key of the line card  102 . Thus, the &lt;shelf, slot&gt; portion of the ADSL physical line primary key is referred to as the foreign key. If the state of the line card  102  is “inactive”, the state of the ADSL physical line  104  is also “inactive” because of their dependency. 
     Similarly, the primary key for the ADSL channel  106  is &lt;shelf, slot, port, type&gt;. The &lt;shelf, slot&gt; portion of the ADSL channel primary key is the same as the &lt;shelf, slot&gt; primary key of the line card  102  to which the ADSL channel  106  is coupled. The &lt;shelf, slot, port&gt; portion of the ADSL channel primary key is the same as the &lt;shelf, slot, port&gt; primary key of the ADSL physical line  104  on which the ADSL channel  106  is multiplexed. Thus, if the state of either the line card  102  or the ADSL physical line  104  is “inactive”, the state of the ADSL channel  106  is also “inactive” because of their dependency. 
     Thus it can be seen that if the line card  102  deactivates, the associated ADSL physical line  104  deactivates, as do the ADSL channels  106 , which can result in hundreds of fault notifications. For example, although only one line card  102  and two channels  106  are labeled in  FIG. 1 , it is not uncommon for a line card  102  to be coupled to 24 ADSL physical lines  104  and for each ADSL physical line  104  to have one or two ADSL channels  106 . Further, each ADSL channel  106  may be coupled to an Asynchronous Transfer Mode (ATM) interface (not shown). Thus, if a line card  102  goes down, it changes state. Each of the 24 ADSL physical lines  104  also has to change state. Each of the ADSL channels  106  also has to change state. Each ATM interface (not shown) also has to change state. As a result, between  73  and  121  notifications may be generated for one deactivated line card  102 , depending on the number of ADSL channels  106  per line. However, when this many notifications are sent, the management system can lose messages, thus giving it an incorrect picture of the system. 
     The number of notifications sent when a managed entity fails may be reduced as follows, with reference to  FIG. 2 , which is a block diagram showing a notification collector according to an embodiment of the present invention. Managed entities that depend on other managed entities register at initialization or provisioning time with the dependee entity for state change notifications. Typically, the management system  108  configures the managed entities and, thus, is aware of the system configuration. Managed entities that are a root cause of a state change send a state change notification  202  to a local notification collector  204  on the system. Since the notification collector  204  is a local network component, it can avoid the message flooding problem of the prior art by appropriately programming its task priority setting. That is, the notification collector  204  is assigned a higher task priority than all other tasks that generate notifications. Since the notification collector  204  is assigned a higher task priority than the managed entities, the managed entities will not be able to generate a notification while notification collector  204  is processing a notification. Therefore, the notification collector  204  will be able to process the notifications faster than they can be generated. The identity transmitted by the managed entity to the notification collector  204  includes its own identity as well as the identity of the managed entity that is the root cause of the state change. In this case, both identities are the same. 
     The managed entity also sends an internal state change message to all managed entities that depend on it, with its own identity as the root cause. Each dependent managed entity so notified sends a state change notification  202  to the notification collector  204 . The notification  202  includes the identity of the managed entity that is the root cause of the state change and the identity of the managed entity sending the notification  202 . In this case, the identities are different. 
     The dependent managed entities also send and internal state change notifications to their own dependent managed entities, along with the identity of the entity that was the root cause of the fault. Each dependent managed entity so notified sends a state change notification  202  to the notification collector  204 . The notification  202  includes the identity of the managed entity that is the root cause of the state change and the identity of the managed entity sending the notification  202 . In this case, the identities are different. This process continues until all dependent, managed entities have been notified and have notified the notification collector. 
     The notification collector  204  sends the received notifications  202  to the management system  108 . If there are too many notifications to send, the notification collector  204  discards any notifications where the root cause identifier is not the same as the entity that sent the notification. A threshold is defined for determining what constitutes too many notifications. The threshold is programmable and is determined by a system operator as will be appreciated by a person skilled in the art. Generally, the threshold depends on the bandwidth of the communication link, the capacity of the management system  108 , and the amount of notification buffers on the system. 
     The discarding of notifications is an effective notification filtering mechanism because, as previously described, the management system  108  is aware of the system configuration and dependencies between managed entities. Thus, the management system  108  is programmed to deduce the state changes of the dependent managed entities from the system configuration. In the present embodiment, the management system  108  includes codified fault correlation rules encapsulating associated managed entities in a directed acyclic state dependency graph, as illustrated in  FIG. 1 . In the diagram, managed entities are represented as nodes and dependency relationships as unidirectional edges. As previously described, dependents (represented by a tail of the graph edge) register with dependees (represented by an arrowhead of the graph edge) for state change notifications. 
     Continuing the example previously described, if the line card  102  is faulty, a state change notification  202  is sent to the notification collector  204  with the identity of the line card  102  as the root cause. The line card  102  also notifies each of the ADSL physical lines  104 , and each of them deactivates. Each of the ADSL physical lines  104  sends a notification  202  to the notification collector  204  with the identity of the line card  102  as the root cause. Each ADSL physical line  104  notifies its dependent ADSL channels  106 . Each of the ADSL channels  106  sends a notification  202  to the notification collector  204  with the identity of the line card  102  as the root cause. Each ADSL channel  106  notifies its dependent ATM interface (not shown), and each ATM interface (not shown) sends a state-change notification  202  to the notification collector  204  with the identity of the line card  102  as the root cause. 
     Therefore, the notification collector  204  has between  73  and  121  notifications to send to the management system  108 . If that number is above the threshold that defines “too many”, only the line card state change notification is sent to the management system  108  because that notification is the case where the root cause identity is the same as the notification sender identity. The management system  108  can deduce the corresponding state change in the ADSL physical lines, the ADSL channels, and the ATM interfaces accordingly. 
       FIG. 3  is a flowchart summarizing the steps performed by a system that includes a notification collector according to an embodiment of the present invention. (More detail on these steps is provided above with reference to  FIG. 2 .) When the network is configured, some of the managed entities have a predefined dependency relationship with others of the managed entities. In step  302 , each of the managed entities is registered with a dependee. In step  304 , when a fault is detected, a notification is transmitted from a faulty entity to receiving entities. The receiving entities are all those entities dependant on the faulty entity. In step  306 , upon receipt of the notification, a further notification is transmitted from each of the receiving entities to all managed entities dependant on those receiving entities. In step  308 , step  306  is repeated until all dependant entities have been notified, thereby removing the need for a fault correlation subsystem. 
     The system described above inhibits a single fault from causing a storm of fault notifications. This is achieved by tagging notification events with the identity of the root cause of the fault. As a result, congestion of the communication channel to the management system  108  is reduced, as the notification collector  204  decides whether or not to filter out notifications that are not sent by the managed entity that is the root cause of the problem. The management system  108  deduces the rest of the state changes in the system using its knowledge of the system configuration and the relationships between managed entities. Thus, the correct states of all managed entities managed by the management system  108  can be reflected with a minimum number of notifications. 
     A further advantage of the system described above is that it eliminates the need for a fault correlation subsystem. Typically, fault correlation is performed by a fault correlation subsystem, which operates as follows. The managed entity fault is sent to the fault correlation system. The fault correlation system runs rules based on a system configuration, and updates the visible state of the managed entities. 
     However, this approach is error-prone because the state correlation logic has to be coded twice, once in the managed entity control software and once in the fault-correlation subsystem. Having two components of software calculating the same state information is risky because it may not produce the same results. Thus, having each managed entity report to the dependee that it received status notifications, reduces the need for a fault correlation subsystem. 
     Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.