Embedded reactive and proactive intelligence

A network device may include logic configured to invoke proactive programs based on the expiration of a long time period within the network device, wherein the proactive programs collect data relating to operations of the network device, invoke reactive programs based on the detection of at least one of an event or expiration of a short time period, wherein the reactive programs collect, correlate and analyze data relating to the detected event or expiration of a short time period, determine a corrective action based on collected and analyzed data, and perform the determined corrective action within the network device.

BACKGROUND INFORMATION

Event detection and management systems are typically programmed with conditions, that when satisfied, generate errors within a device. Existing systems that detect errors can send a message to a remote location. These messages do not provide adequate information or procedures to fix errors within the device. Further, existing systems wait for error conditions to occur before taking actions relating to the error corrections.

SUMMARY

According to one aspect, a network device is provided. The network device may comprise logic configured to: invoke proactive programs based on the expiration of a long time period within the network device, wherein the proactive programs collect and transmit data relating to successful operations of the network device; invoke reactive programs based on the detection of at least one of an event or expiration of a short time period, wherein the reactive programs collect and analyze data relating to the detected event or expiration of the short time period; determine a corrective action based on collected and analyzed data; and perform the determined corrective action within the network device.

According to another aspect, a method is provided. The method may comprise detecting and/or correlating at least one of a plurality of events or a plurality of time periods within a network device; invoking a program in response to the detected event or time period; collecting data received from the invoked program; analyzing collected data to determine a corrective action; and performing the determined corrective action within the network device.

According to another aspect, a network device may comprise means for determining a plurality of time periods within the network device; means for detecting a plurality of events within a network device; means for invoking a program in response to one of the plurality of determined time periods or detected events; means for collecting data received from the invoked program; means for analyzing collected data; and means for performing corrective actions within the network device based on the analyzed data.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The following detailed description refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. In addition, while some of the following description is provided mainly in the context of routers or other network elements at layer2and/or layer3of the Open Systems Interconnection (OSI) Model, the principles and teachings may be applied to different types of network devices at different layers of communication (e.g., a Multi-protocol label switching (MPLS) routers, a Synchronous Optical Network (SONET) add-drop multiplexers, a Gigabit Passive Optical network (GPONs) switches, a Synchronous Digital Hierarchy (SDH) network elements, etc.).

FIG. 1shows an exemplary network in which concepts described herein may be implemented. As shown, network100may include network element110and a network120. In practice, network100may include additional elements than those illustrated inFIG. 1. Network element110may include devices for performing network-related functions, such as a router or a switch (e.g., a provider edge (PE) router in a MPLS network). Network120may include the Internet, an ad hoc network, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a cellular network, a public switched telephone network (PSTN), any other network, or a combination of networks. Network element110may communicate with other network elements in network120through a wired or wireless communication link.

FIG. 2shows an exemplary block diagram of network element110. As shown, network element110may include a processor210, memory220, interfaces230, an interconnect240, and a bus250. In other implementations, network element110may include fewer, additional, or different components than those illustrated inFIG. 2.

Processor210may include one or more processors, microprocessors, application specific integrated circuits (ASICs), field programming gate arrays (FPGAs), and/or processing logic optimized for networking and communications. Memory220may include static memory, such as read only memory (ROM), dynamic memory, such as random access memory (RAM), and/or onboard cache, for storing data and machine-readable instructions. Memory220may also include storage devices, such as a floppy disk, a CD ROM, a CD read/write (R/W) disc, and/or flash memory, as well as other types of storage devices. Interfaces230may include devices for receiving incoming data stream from networks and for transmitting data to networks (e.g., Ethernet card, optical carrier (OC) interfaces, asynchronous transfer mode (ATM) interfaces, etc.). Interconnect240may include one or more switches or switch fabrics for directing incoming network traffic from one or more of interfaces230to others of interfaces230. Bus250may include a path that permits communication among processor210, memory220, interfaces230, and/or interconnects240.

Depending on implementation, the components that are shown inFIG. 2may provide fewer or additional functionalities. For example, if network element110performs an Internet Protocol (IP) packet routing function as part of a MPLS router, processor210may perform tasks associated with obtaining routing information from other routers in a MPLS network. In such cases, conveying network traffic from one interface to another may involve label based routing, rather than IP address based routing.

FIG. 3is a functional block diagram of a network element110that includes a router. As shown, network element110may include event logic310, agents320, routing logic330, forwarding logic340, and packet buffering logic350. In different implementations, network element110may include fewer, additional, or different components than those illustrated inFIG. 3. For example, network element110may or may not provide certain network management functions, and in such instances, network element110may not include certain agents330.

Event logic310may include hardware and/or software for performing various event driven functions for management and operation of network element110and/or other network elements. For example, event logic310may provide embedded reactive and proactive scripts used to collect and analyze data to monitor performance within network element110. Event logic310may further provide correlation of events and time periods, and in response to the analysis of correlated events/time periods, make decisions to reconfigure operations of network element110. Event logic may also create new scripts to perform further collection and analysis of data within network element110. In another example, event logic310may provide a user interface via which a network administrator or a user can interact with network element110and receive problem report bundles that may include data relating to errors and/or corrective actions taken within network element110. A functional block diagram of the components of event logic310is shown inFIG. 4.

Agents320may include hardware and/or software for monitoring and/or controlling components on behalf of event logic310. The monitored components may include a physical device (e.g., a plug-in card, a multiplexer, a switch, etc.) or a logical device, such as a virtual connection or a logical interface. In monitoring the components, agent320may detect a fault or a recovery of an interface, an interconnect, or any other component of network element110and may report the fault or the recovery to event logic310. For example, agents320may detect a failure of one of interfaces230and may send associated alarm or error messages to event logic310. In another example, agents320may receive commands from a remote device and may make appropriate configuration changes to interfaces230. In some implementations, agents320may be attached or connected to other subcomponents of network element110.

Routing logic330may include hardware and/or software for communicating with other routers to gather and store routing information in a routing information base (RIB). Forwarding logic340may include hardware and/or software for directing a packet to a proper output port on one of interfaces230based on routing information in the RIB. Buffer manager350may provide a buffer for queuing incoming packets. If packets arrive simultaneously, one or more of the packets may be stored in the buffer until higher priority packets are processed and/or transmitted.

FIG. 4shows an exemplary functional block diagram of event logic310. As shown, event logic310may include an event trigger module420, a short timer based operation module430, a long timer based operation module440and an event policy module450. Event logic310may transmit and receive event data410and event problem report bundles460. In different implementations, event logic310may include fewer, additional, or different components than those illustrated inFIG. 4.

Event data410may include information related to events, time periods and/or the status of hardware and software contained within network element110. For example, the components shown inFIG. 2such as processor210, memory220, interfaces230and interconnect240may provide signals and information identifying certain events. For example, processor210may provide event data410including information relating to the status of internal registers becoming overloaded or unused. Similarly, the functional components shown inFIG. 3may also provide event data410that may include information related to status and events. For example, routing logic330may provide event data410that may include information indicating a processing failure regarding routing information base (RIB). In other examples, event data410may include information related to events and/or the status of hardware and/or software contained in another network element connected to network120.

Event trigger module420may include hardware and/or software that may receive a signal from event policy module450and invoke a script. For example, a script may include embedded event policies that may identify a hardware or software problem, collect data relating to the identified problem, perform an analysis of the collected data, perform a corrective action based on the analysis and generate an event problem report bundle460. The analysis performed by an invoked script within event trigger module420may include any appropriate analysis algorithm, such as using one of a decision tree or an artificial intelligence database. The specific type of analysis performed by each script may be determined by the specific event410that invoked the script. For example, if event data410contains information that some component of network element110is using90% of processor resources, a script contained in event trigger module420may be invoked by event policy module450in order to collect data and may determine what component is using the resources, what the nature of the problem is, what if any events may be correlated to this event, and/or what if any corrective actions may be taken. Event trigger module420may also invoke additional scripts based on the collected and analyzed data. For example, if a component is positively identified as using 90% of the resources of processor210, an additional script may be invoked to collect data from the identified component to further determine the nature of the problem. Event trigger module420may also create new scripts based on the analysis or results of additional scripts. For example, if a specific hardware device is identified as potentially faulty by a previous script, a new script may be created that contains new parameters to monitor regarding the identified hardware device. Scripts invoked by event trigger module420may be referred to as “reactive” scripts, as these scripts may be invoked to react to a received event.

Short timer based operation module430may include hardware and/or software that may receive a signal from event policy module450and invoke a script. For example, a detected event or a short timer value that may be contained in event policy module450may be detected. Upon detection of an event, for example, event policy module450may invoke a script in short timer based operation module430that may include event policies that may identify a hardware or software problem, collect data relating to the identified problem, perform an analysis of the collected data, perform a corrective action based on the analysis and generate an event problem report bundle460. The analysis performed by an invoked script within short timer based operation module430may include any appropriate analysis algorithm, such as using one of a decision tree or an artificial intelligence database. The specific type of analysis performed by each script may be determined by the specific event or short timer value that may have been detected. Short timer based operation module430may also create and send an event problem report bundle460based on an invoked script that may include event data410, collected data, and corrective actions taken. A script contained in short timer based operation module430may invoke processes to correlate events, perform trend analysis, detect increments of critical counter values and measure or detect memory leaks, and may also invoke additional scripts based on the collected and analyzed data. For example, if a trend is positively identified by a first script, an additional script may be invoked to reconfigure component(s) within network element110in order to adjust to the determined trend. Short timer based operation module430may also create new scripts based on the analysis or results of other scripts. For example, if a specific software process is identified as potentially faulty by a previous script, a new script may be created that contains new parameters to monitor and/or shortens a time period within short timer based operation module430that invokes the new scripts. Scripts invoked by short timer based operation module430may also be referred to as “reactive” scripts, as they may be invoked to react to a timer value.

Long timer based operation module440may include hardware and/or software that may receive a signal from event policy module450and invoke a script. For example, an event or long timer value that may be contained in event policy module450may be detected. Upon detection of an event, for example, event policy module450may invoke a script contained in long timer based operation module440that may collect data that may be sent to another network device via a problem report bundle460, for analysis. For example, data collected by a script invoked in long timer based operation module440may be analyzed to determine long term trends of network element110. Data collected by a script invoked by long timer based operation module440may also be strategically analyzed to determine potential risks and/or failures that may occur within network element110, for example. Scripts invoked in long timer based operation module440may be referred to as proactive scripts, as the data collected and analyzed, may be related to successful operations of network element110, and may be used to “proactively” reconfigure hardware and/or software within network element110before a problem or error occurs.

Event policy module450may include event policies and time periods that may be used to process received event data410and in response to the received event, time period, etc, invoke scripts contained in modules420-440. For example, event policy module450may store events and information relating to an appropriate script and or module420-440to invoke. After determining an appropriate script to invoke, event policy module450may send signals and information to the appropriate module (420-440) so that further processing of data may be performed.

Event problem report bundles460may include an XML document including information relating to received event data410and/or information relating to data collected and/or corrective actions taken by any of modules420-440in response to received event data410and/or a timing (short or long) operation. For example, after any one of modules420-440have collected, analyzed and processed event data410, an event problem report bundle460may be generated containing information.

The above paragraphs describe system elements that are related to intelligently monitoring and restoring the functionality of network element110. The system elements shown inFIG. 4may perform both proactive and reactive processes within network element110.FIGS. 5-6as described below, depict exemplary processes that are capable of being performed by network element110.

FIG. 5is a flowchart illustrating an exemplary proactive process500. Process500may begin when a long timer value expires (block510). For example, a timer value contained in event policy module450may expire. The timer values contained in event policy module450may be on the order of days, weeks, months, etc. Each individual timer value contained in event policy module450may invoke an appropriate script contained in long timer based operation module440upon timer expiration (block520).

Once invoked, a proactive script may collect data (block530). For example, a script may collect data at one of interfaces230or an invoked script may collect data relating to the operations of processor210. The invoked proactive script may generate an event problem report bundle460(block540). For example, an event problem report bundle460may be generated that contains the collected data that may relate to one of interfaces230. An event problem report bundle460may then be transferred (block550). For example, network element110may transmit an event problem report bundle460to a central network device that may further analyze the collected data. Upon analysis of the received data within a problem report bundle460, a central network device may determine that an interface230is currently operating. However, the central network device may determine or flag that interface230may potentially fail based on, for example, analyzed data contained in other received problem report bundles from other network devices. A central network device may transmit information back to network element110based on the analyzed data. For example, a central network device may transmit an updated software process to be used in place of a potentially faulty software process within network element110. A central network device may also transmit a script that may be contained and used within network element110based on analyzed data received from network element110.

FIG. 6is a flowchart illustrating an exemplary reactive process600. Process600may begin when a short timer expires or an event trigger occurs (block610). For example, a short timer value or event contained in event policy module450may be detected. Each short timer value and event in event policy module450may then invoke a script contained in short timer based operation module430based upon the expiration of a short timer value or detection of an event (block620).

Once invoked, a reactive script may initiate data collection (block630). For example, a received event410may invoke a reactive script contained in event trigger module420. The received event410may be an event such as50% of memory220is being used, that may indicate a potential memory leak. Data may be collected relating to hardware aspects of memory220and data may be collected relating to software processes or operations that may currently be accessing memory220. The data collected regarding the operation of memory220may then be analyzed (block640). For example, the analyzed data collected (in block630) may indicate that hardware aspects of memory220are functioning properly. Other data collected (in block630) by the invoked reactive script may relate to the amount of data stored in routing tables or forwarding tables contained in routing information base (RIB). This data may be analyzed to determine if 50% memory usage is normal based on the information in RIB.

Based on the analysis performed in block640, the invoked script may determine if additional data collection may be necessary or if corrective action may be taken (block650). Continuing with the above example, if the analysis of the data indicates that the network element is currently receiving large amounts of data, a 50% memory usage would be determined to be acceptable and no corrective action may be taken, (block650), thus completing the data analysis and corrective actions (block660).

If for example, an analysis of the data relating to the RIB indicates that a small amount of data is being received by network element110, this may indicate that a memory leak is occurring and another reactive script may be invoked or other event scripts may be correlated to collect data relating to other processes currently accessing memory220(block650). An additionally invoked reactive script may collect and analyze data and determine that a software process is not releasing memory space within memory220. In response to this analysis, the script may shut down the detected faulty software process and launch another process to replace the faulty software (block660).

The reactive script may then generate an event problem report bundle460(block670). For example, an event problem report bundle460may be generated that identified the detected faulty software process within network element110. The event problem report bundle460may also contain information relating to the corrective action that may have been taken in response to the detected faulty software process. Further, the reactive script may set an alarm or create another reactive script to monitor memory220based on the analyzed collected data obtained in blocks630-660. The event problem report bundle460may then be transferred to another network device (block680).

For example, an event problem report bundle460may be transferred from network element110to a central network device connected to network120. For example, a central server and/or computer may store and analyze data received from a number of network devices. Receiving event problem report bundles460from a plurality of network elements110and/or other network devices, allows a central network device to detect and compensate for trends or tendencies within network elements110that may be unnoticed by existing systems.

In another example of blocks610-660, an event may be received by event logic310indicating that a Border Gateway Protocol (BGP) has gone down. In this example, a reactive script may be invoked to collect further data relating to this event. At some time within the next thirty seconds for example, another event may be received by event logic310indicating that the BGP has gone back up. In this example, the two events may be correlated and the collected data may be analyzed to determine if the same device produced these events. If it is determined that the same device produced these events, it may indicate that there is a problem with the BGP on the identified device and another script may be created to further monitor the device. If, for example, 5 BGP up/down events continue to be received over the next 5 minutes by event logic310, it may be determined and/or confirmed that a BGP problem exists, in which case further scripts, commands and/or corrective actions may be taken in response to this determination.

Continuing with this example, once invoked by a BGP up/down event policy, an invoked reactive script may contain logic and commands such as, check the installed routes on the network element110and check the list of interfaces on the network element110(router). The data collected from these commands may be correlated to identify a faulty interface and the faulty interface may be disabled. After this corrective action has been taken, blocks670and680may be enacted to create and transmit an event problem report bundle460to an external network device. It should be understood that the above examples are exemplary only, and that each specifically identified event may invoke a specific script may include specific event polices and data collection and analysis of specific parameters used to determine and/or correct errors that generated the specific event.

FIG. 7shows an exemplary decision tree700. As shown, decision tree700includes a number of nodes701-709, where each node may represent a logical decision point within a process (such as a process contained in a script). The process of collecting and analyzing data, correlating events and performing corrective actions based on the analyzed data within decision tree700, may be equivalent to the process performed in blocks630-660, as described above.

The process of traversing exemplary decision tree700may begin when a short timer value has expired and an invoked script collects data related to memory utilization within network element110, for example. A decision may be made at node701to analyze the collected data and determine if memory utilization has changed (more than a threshold value) with respect to previously collected data. Based on this comparison made at node701, if the memory utilization data has changed more than a threshold amount, the process may continue at node702, and if there is no change in the memory utilization data, the process may continue at node703. If there is no change, a decision may be made at node703to reset the short time period and take no further action. If the process continues at node702, an analysis of the memory utilization data may be performed to determine if the change in memory utilization was an increase or a decrease. If the analyzed data indicates a decrease in memory utilization, the process may continue at node705. An analysis of the data made at node705may determine that memory utilization has steadily decreased and a trend may be detected. In response to this determination, a new script may be created that adjusts memory utilization threshold parameters, for example.

If the analyzed data at node702indicates an increase in memory utilization, the process may continue at node704. At node704data may be collected from the RIB to determine if the high memory utilization is due to a routing chum. If the data collected from RIB indicates that a routing chum is occurring, the process may end at node706, where no further action is taken. If the data collected from RIB indicates that a routing churn is not occurring, the process may continue at node707. Data may be collected by invoking additional scripts at node707to determine which tasks are using memory. Additionally, other detected events that have invoked scripts relating to memory usage may be correlated. In this example, a correlated event may also be associated with another decision tree (not shown), where data and results of this correlated decision tree may be employed into decision tree700, for example. Based on the analyzed and correlated data, it may be determined at node707that a specific software process is causing memory fragmentation problems within network element110. In this instance, the process may continue at node709. It may then be determined at node709to invoke a memory de- fragmentation algorithm in order to reduce memory utilization. If it is determined at node707that a memory fragmentation is not occurring, it may be determined that a potential memory leak is occurring, where the process may continue at node708. A corrective action may be taken at node708to reconfigure a hardware or software process in order to determine if the potential memory leak may be stopped. Additional scripts may be invoked in order to collect data to determine which (if any) software processes may be causing the potential memory leak. Other corrective actions taken at node708may include sending an alarm and/or generating and transmitting an event problem report bundle460. As described above the event problem report bundle460may include information indicating that a memory leak has been detected that may include information relating to corrective actions taken and/or information requesting a new software process to replace the faulty process. It should be understood that decision tree700is exemplary only, and that each specific script may include a specific decision tree based on analyzing parameters used to make decisions based on the detected event or time period.

As described above, the present embodiments allow for errors or events within a network element110to potentially be avoided, by collecting and analyzing data before errors occur using proactive scripts. Further, if an error or event is detected with network element110, the error or event may be intelligently corrected without operator intervention.

The foregoing description of implementations provides an illustration, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the teachings.

In addition, while series of blocks have been described with regard to the processes illustrated inFIGS. 5-6, the order of the blocks may be modified in other implementations. Further, non-dependent blocks may represent blocks that can be performed in parallel. For example, blocks510-550that are performed for one interface may be independent of blocks510-550for a second interface and, therefore, may be performed in parallel to blocks510-550for the second interface. Further, it may be possible to omit blocks within a process.

Further, certain portions of the implementations have been described as “logic” that performs one or more functions. This logic may include hardware, such as a processor, an application specific integrated circuit, or a field programmable gate array, software, or a combination of hardware and software.