Abstract:
A network device for use in a communication system having a technical support center operated by a technical support staff, the technical support center being in communication with the network device through a packet switching network. The network device includes one or more hardware subsystems, one or more software subsystems and means for monitoring the status of the hardware and software subsystems so that when a problem occurs with respect to one or more of the hardware and software subsystems of the network device, the network device for transmitting a first message to the technical support center to notify the technical support center of the problem, wherein the technical support staff is able to diagnose the problem without interruption to the operation of the network device.

Description:
RELATED APPLICATIONS 
     This application is a continuation of and claims priority to pending U.S. application Ser. No. 09/520,687, entitled DIAGNOSTIC/REMOTE MONITORING BY EMAIL, filed Mar. 7, 2000. 
    
    
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates generally to a method and apparatus for detecting a problem with a network device employed in a communication system and particularly to a method and apparatus for detecting a hardware or software-related problem within one or more network device among a large number of operational network devices within the communication system. 
     2. Description of the Prior Art 
     In modern communication systems, there may be a large number of network devices such as mail servers, routers and computers present within a system. Under such circumstances, it is common to have equipment failure, which would then require diagnostic evaluation and debugging. If the system includes hundreds of routers, such as in Cisco Systems Inc. laboratories, in order to identify the router that has failed, an engineer located at a technical support center must establish at least limited communication with every one of the routers (referred to as logging into the routers), of the large number of routers, in order to try to narrow the problem to one or more specific routers prior to diagnosis of the problem. This is commonly a considerably time-consuming and rigorous process. In fact, currently, among tens and hundreds of routers in operation, it is not unusual for engineers to spend one month in detecting a problem with a specific router. 
     Currently, when a component within a router fails, the router generates error messages for notification of the failure. 
     There are several ways in which a network communication system may fail. Among these are problems arising in the hardware and software components of various devices and communication lines and interfaces connecting the various devices of the communication system together. When there is a hardware problem, such as the failure of a board in one of the devices due to overheating, the driver in the device detects the problem by receiving an error message from the board thereby alerting the software that is being executed in the device of the device&#39;s failure. However, when the system fails, the valuable information regarding the reason for failure, which may be embedded in an error message in the software, may be lost, making the task of diagnosing the cause of failure more difficult and time-consuming by erasing any potential clues which might otherwise help an engineer in diagnosing the problem. 
     By way of execution of the software in a device, relevant information regarding the failure of the device exists but it is not necessarily communicated to the technical support staff after the device has failed. When the device, which might be a computer or an access server (router), is powered down and then powered back on, the original problem may disappear during rebooting or the conditions, which caused the problem, may no longer exist. Such is the case when a board malfunctions due to overheating and resumes functioning properly once it is cooled. Similarly, an existing problem may not recur immediately after the device is rebooted and may resurface at a later time making the task of troubleshooting (or debugging) more difficult. 
     Before the occurrence of the failure of the device, the operating system residing and being executed in the device or the software being executed on the device has the most current information regarding the status of various components in the device. Currently, such information is not communicated to the technical support center and remains isolated within the device. The engineers located at a technical support center, based on the status of the device immediately before its failure, could draw valuable insights into the mechanisms of failure and suggest ways of remedying the problem. 
     If the device is a computer, the operating system or the software within the computer has current information regarding the status of the modem, software updates, status of the hard drive and every other hardware and software subcomponent within the computer. If such information were available to the technical support center, troubleshooting the device could be performed much more efficiently and cost effectively. In addition, since the time duration in which the system is out of service is shortened, the customers making use of the system experience less delay, resulting in a higher degree of customer satisfaction. 
     Therefore it is desirable to devise a system and method for monitoring the status of a network device at all times and for reporting any problems that may arise in the hardware, software or the interface components of the device to a technical support center so as to rapidly detect a problem with one or more network devices within a large group of network devices. Additionally, the need arises for the monitoring system and method to include the capability to process instructions from the technical support center in order to execute diagnostic tests on the hardware components or request more detailed information from the software subsystems included within the device. 
     SUMMARY OF THE INVENTION 
     Briefly, in an embodiment of the present invention, a network device is disclosed for use in a communication system having a technical support center operated by a technical support staff, the technical support center being in communication with the network device through a packet switching network. The network device includes one or more hardware subsystems, one or more software subsystems and means for monitoring the status of the hardware and software subsystems so that when a problem occurs with respect to one or more of the hardware and software subsystems of the network device, the network device for transmitting a first message to the technical support center to notify the technical support center of the problem, wherein the technical support staff is able to diagnose the problem without interruption to the operation of the network device. 
     The foregoing and other objects, features and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments which make reference to several figures of the drawing. 
    
    
     
       DRAWINGS 
         FIG. 1  shows a high-level block diagram of a communications system with the network device assembly and technical support center. 
         FIG. 2  depicts a high-level block diagram of a network device including embedded software. 
         FIG. 3  illustrates the technical support center of  FIG. 1  including central process software. 
         FIG. 4  depicts an example of a computer register carrying an error message due to hardware or software failure. 
         FIG. 5  shows a high-level block diagram of the sequence of steps used in sending an email from a network device to an Internet Protocol network. 
         FIG. 6  shows a flow chart of the sequence of steps executed for receiving commands from the technical support center and responding thereto. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Referring now to  FIG. 1 , a communication system  10  is shown to include a network device assembly  12 , a packet switching network  18  and a technical support center  22  for use by technical support staff  23  in accordance with an embodiment of the present invention. The network device assembly  12  comprises numerous network devices  14 , which may be various types of network devices such as network access servers, routers, computer, etc. The network device assembly  12  may include a large number of network devices. A typical application includes hundreds of routers, each of which is an AS5800 model manufactured by Cisco Systems, Inc. of San Jose, Calif. 
     In one embodiment of the present invention, the packet switching network  18  may be an Internet Protocol (IP) network, such as the Internet. The technical support center  22  is a center equipped with computers and diagnostic equipment typically located at a laboratory site. 
     Each of the network devices  14  is coupled to the packet switching network  18  through an interface line  16 . The packet switching network  18  is coupled to the technical support center  22  through an interface line  20 . According to one embodiment of the present invention, there are two sets of software programs. The first set of software programs, which will be referred to as the embedded software throughout this document, resides in each of the network devices  14  which may be a router or a computer or any other network device used by a client (or user). The second set of software programs, referred to as the central process software throughout this document, resides in the technical support center  22 . 
     Each of the network devices  14  is in communication with the technical support center  22  through the packet switching network  18 . When there are a large number of network devices within the assembly  12 , it is likely that a network device may fail due to a hardware or software-related problem. 
     According to one embodiment of the present invention as shown in  FIG. 1 , the embedded software resides in every network device  14 . The embedded software monitors the status of every hardware component and every software subsystem of the network device  14  by collecting and analyzing data received from the hardware components and software subsystems thereof When a problem with a software or a hardware component of the network device  14  is detected by the embedded software, the embedded software notifies the technical support center  22  regarding the problem by transmitting an email message  20 , through the interface  16 , thereto. Each network device  14  includes the embedded software, accordingly, the technical support center  22  is notified of any problem that may arise within the network device assembly  12 . 
     Alternatively, the network device assembly  12  comprises a number of computers each of which is connected to the technical support center  22  through the network  18 . According to an embodiment of the present invention as shown in  FIG. 1 , the embedded software resides in each of the computers in the network device assembly  12  and monitors the status of various components and processes of the computer such as modems, software updates, hard drive, memory, and the like. When a problem within any hardware component or software subsystem of the computer develops, whether the problem leads to the failure of the computer or not, the embedded software detects the problem and notifies the technical support center  22  by sending an email message, such as the message  20 , to the technical support center  22 . 
     The technical support center  22  behaves essentially as an email server with a large database for storing email messages. In addition and as will be discussed in further detail relative to other figures of this patent document, the technical support center  22  carries the central process software, which facilitates communications between the network device  14  and the technical support center  22 . In particular, when a problem in the network device  14  is reported to the technical support center  22 , the technical support staff  23 , i.e. engineers, at the technical support center  22  may need more information to determine the cause of the problem than what is initially conveyed to them in the email message that was sent from one or more of the network devices  14  to the technical support center  22 . Thus, in one embodiment of the present invention, using the central process software, the technical support staff  23  send an email message  21  back to the network device  14  requesting more information regarding the problem. The email message  21  is optional in that no further information regarding the problem may be needed and/or solicited. 
     In the case where detail information regarding the problem is requested, as an example, the email message  20  transmitted to the technical support center  22  may only indicate that a board in a hardware component of the network device  14  is experiencing a problem. The technical support center  22  may require additional diagnostic tests to be performed on the problematic board prior to determining the cause of the problem Using the central process software, the technical staff  23  at the technical support center  22  perform diagnostic tests on the defective board through the embedded software of the network device  14 . To elaborate, if the network device  14  is a computer, the user continues to interact with the computer without any interruption while the diagnostic tests are being performed in the background by the computer. This clearly offers considerable advantage to the user. 
     Accordingly, the use of the embedded software and the central process software facilitate communications between the network assembly  12  and the technical support center  22  for diagnostics and remote monitoring of each of the plurality of network devices  14 . In accordance with another embodiment of the present invention, the technical support center  22  does not include the central process software and only the embedded software is used in each of the network devices  14  in order to monitor the status of the latter and report any problems associated therewith to the technical support center  22 . Under such circumstances, the email message  21  is not sent and consequently not as much information can be requested from the embedded software by the technical support center  22  as is the case when the latter includes the central process software. Accordingly, less detailed information is available to the technical support center  22  for diagnostic purposes; nevertheless it is possible to perform diagnostic and remote monitoring of each of the network device  14 . 
     Referring now to  FIG. 2 , one of the network devices  14  of  FIG. 1  is shown to be coupled to the network  18  through the interface  16  according to an embodiment of the present invention. The embedded software  25  is shown, in  FIG. 2 , to reside within the network device  14 . The embedded software comprises several software subsystems, i.e. a memory monitoring subsystem  26 , an email/page subsystem  24 , a remote diagnostic embedded process subsystem  28 , a software health status monitor subsystem  30  and a hardware health status monitor subsystem  32 . In addition, the network device  14  includes a plurality of other software subsystems  36  and a plurality of hardware devices  34 . Examples of the other software subsystems  36  are the Netscape browser application program, Microsoft Excel application program, and the like. Examples of hardware devices  34  are a graphics display board, a hard drive, a modem, and the like. 
     The remote diagnostic embedded process subsystem  28  is in communication with the other components of the embedded software subsystems. More specifically, the remote diagnostic embedded process subsystem  28  is in communication with the memory monitoring subsystem  26 , the email/page subsystem  24 , the hardware health status monitor subsystem  32  and the software health status monitor subsystem  30  through the software interfaces  40 ,  38 ,  42  and  44 , respectively. Moreover, the software health status monitor subsystem  30  is coupled to the other software subsystems  36  through the software interface  48  and the hardware health status monitor subsystem  32  is coupled to the hardware devices  34  through the software interface  46 . 
     The hardware health status monitor subsystem  32  monitors the status of the hardware devices  34  within the network device  14  and communicates such status information to the remote diagnostic embedded process subsystem  28 . The hardware health status monitor system  32  further performs background diagnostic tests on the hardware devices  34  as requested by the technical support center  22  (shown in  FIG. 1 ). The software health status monitor subsystem  30  monitors the status of the software subsystems  36  within the network device  14  and communicates such status information to the remote diagnostic embedded process subsystem  28 . 
     The remote diagnostic embedded process subsystem  28  is the main software subsystem of the embedded software. It is used to collect and analyze all of the information provided by the software health status monitor subsystem  30  and the hardware health status monitor subsystem  32 . During analysis of the status information, the remote diagnostic embedded process subsystem  28  detects problems encountered by the other software subsystems  36  or the hardware devices  34  resident within the network device  14 . In the event a problem develops within any of the software subsystems or hardware devices of the network device  14 , the remote diagnostic embedded process subsystem  28  alerts the technical support center  22  by sending information regarding the software or hardware problem thereto. 
     Transmittal of information from the remote diagnostic embedded process subsystem  28  to the technical support center  22  is accomplished through the email/page subsystem  24 . The latter constructs an email message (such as the email message  21  in  FIG. 1 ) incorporating the information received from the remote diagnostic embedded process subsystem  28  through the software interface  38  and transmits the email message, through the network  18 , to the technical support center  22  (shown in  FIG. 1 ). The email/page subsystem  24  can alternatively send an email or a facsimile message or alternatively page a user of the network device  14  in order to alert the user of the problem. 
     As an example, if the network device  14  is a computer whose Internet connection fails, the user has no way of knowing initially whether the Internet line or some component of the modern board has failed. In  FIG. 2 , the modem board would be one of the devices in the hardware devices  34 . The remote diagnostic embedded process subsystem  28  will know of the status of the modern board immediately before the failure of the Internet connection and thus will transmit this information to the technical support center  22 . The engineers at the technical support center  22  then detect the problem(s) associated with the modem board based upon status information regarding the modem board, which would have been received by the remote diagnostic embedded process subsystem  28  immediately before the connection failure. 
     If there are no problems with the modem board, a determination is made as to the failure of the Internet line to be properly connected. In this case, information regarding the failed Internet line is sent to memory, such as non-volatile random access memory (NVRAM) or flash memory for subsequent retrieval thereof by the network device when the latter is again operational. Alternatively, an alarm, in the form of a light indicator or otherwise, is set by the network device indicating a problem with the latter. On the other hand, if the problem emanates from the modem board, the technical support center  22  may decide to perform diagnostic tests on the board, in which case the center  22  will instruct the remote diagnostic embedded process  28  as to how to perform the diagnostic tests. 
     Memory monitoring subsystem  26  is another software subsystem of the embedded software for monitoring the memory of the network device  14 . In the case where the network device  14  is a computer, the memory monitoring subsystem  26  determines if the present memory offers adequate capacity for proper performance of the computer or if the memory is in need of upgrading. In addition, the memory monitoring subsystem  26  checks for memory leaks and memory corruption. Memory leaks occur when memory that is assigned for the performance of tasks becomes less and less over time resulting in at least the appearance of insufficient memory capacity and memory corruption is defective areas of the memory, which may result in insufficient memory capacity. If the user is working with a large program requiring more memory than is available in the computer, the memory monitoring subsystem  26  notifies the remote diagnostic embedded process subsystem  28  accordingly. The subsystem  28 , in turn, notifies the user, through the email/page subsystem  24 , regarding the inadequacy of the current memory capacity of the computer. 
     According to one embodiment of the present invention, the technical support center  22  instructs the remote diagnostic embedded process  28  as to how to detect a potential problem. For instance, referring to our previous example, the engineers at the technical support center  22 , i.e. technical support staff  23  in  FIG. 1 , may decide that interruption of the communication line  16  more than five times in an hour presents a potential problem and warrants special attention. Accordingly, the technical support center  22  configures the remote diagnostic embedded process subsystem  28  through email to detect a problem and to notify the center  22  of the same if and when the communication line is interrupted more than five times in an hour. 
     In  FIG. 2 , while not shown, the network device  14  includes a processor such as a central processor unit (CPU) (or computer medium) and a storage area, a computer readable medium, for storing software programs for carrying out the various functions discussed herein. The processor executes code from the computer readable medium for effectuating the functions discussed herein. 
     Referring now to  FIG. 3 , the technical support center  22  is shown to be coupled to the network  18  through the interface  20  according to an embodiment of the present invention. The technical support center  22  is shown to comprise the email server  50 , the command-formatter  54  and the user interface  58 . The email server  50  communicates with the command-formatter  54  through a software interface  52  and the command-formatter  54  communicates with the user interface  58  through a software interface  56 . 
     The email server  50  is a device for collecting the email messages originating from the network  18  and for transmitting the email messages originating from the user interface  58  to the network  18 . An example of an email server is a Personal Computer (PC). The command-formatter  54  is a software program for translating the email messages originating from the network device  14  into a format which is easily understandable by the technical staff and engineers at the user interface  58  and vice versa. While the command-formatter  54  in the embodiment of  FIG. 3  is a software program, the functions performed thereby may be implemented in a hardware structure without departing from the scope and spirit of the present invention. The user interface  58  provides a graphical representation for communicating information between the technical staff and engineers and the network device  14  through the command-formatter  54 , the email server  50  and the network  18 . 
     In addition, the command-formatter  54  has the capability to format commands when the engineers decide to request the remote diagnostic embedded process  28  to perform specific tasks. For instance if the remote diagnostic embedded process  28  is asked to report the modem board status every hour, perform diagnostic test on a modem when the modem experiences three consecutive connecting failures and report peak central processing unit (CPU) loading every three hours, an email message is sent from the user interface  58  to the remote diagnostic embedded process  28  as follows: 
     REPORT: modem_board_status INTERVAL: 60 
     RUN: modem_modem_diagnostic WHEN: 3_consecutive_fail 
     REPORT: cpu_load INTERVAL: 180 
     Consequently, the command format  54  allows the engineers to communicate with the network device  14  without the need to learn a special syntax. 
     In one embodiment of the present invention it is not necessary to include the user interface  58  and the command-formatter  54  within the technical support center  22 . In such a case, the technical support center  22  only includes the email server  50  for receiving and transmitting email messages to and from the technical support center  22 . However, engineers no longer can instruct the remote diagnostic embedded process  28  to perform specific tasks in order to facilitate the diagnostic testing and remote monitoring of the network device  14 . Nevertheless, it is still possible to carry out diagnostics and remote monitoring of the network device  14  using only the email server  50 . 
     Sometimes a problem in the hardware or software components of the network device  14  which has interrupted the normal operation of the network device  14  may disappear upon rebooting of the network device. To make matters worse, the problem may not occur for some time thus making recreating the problem for diagnostic purposes difficult. The problem may have its origins in a number of sources such as the hard drive, the memory, the power supply, etc. One of the advantages of the present invention, as shown in  FIG. 2 , is that the remote diagnostic embedded process  28  receives information concerning the status of every software and hardware component immediately before the failure of the network device  14  either from the software health status monitor subsystem  30  or from the hardware health status monitor subsystem  32 . The status information may then be used by the technical support center  22  to identify the source of the problem. 
     For the case when the network device  14  is a computer, an example of the status information is shown in  FIG. 4 .  FIG. 4  shows the status of a number of hardware and software components in the computer and is referred to as a computer register. The computer register indicates the status of various components immediately before the computer failed due to a problem in one of its hardware or software components. The computer register shown in  FIG. 4  includes error messages, which emanated from the faulty hardware/software component immediately before the failure of the faulty component. In  FIG. 4 , an error block  60  is shown to include three codes. As would be obvious to one of the engineers of a network device who would have designated such codes, the codes in the error block  60  indicate a problem within one of the subsystems of the computer. For example, a faulty board in the hard drive of the computer could have generated the codes in the error block  60 . A trained engineer could identify the subsystem that has failed by viewing the codes in the error block  60 . 
     The computer register in  FIG. 4  is compiled by the hardware health status monitor subsystem  32  and transmitted to the remote diagnostic embedded process subsystem  28 . After having recognized that the computer register includes an error message, the remote diagnostic embedded process  28  transmits the computer register to the technical support center  22  via the email/page subsystem  24 . The engineers in the technical support center  22  detect the problem by observing the codes in the error block  60  and transmit an email message back to the remote diagnostic embedded process subsystem  28  and the subsystem  28  implements diagnostic instructions accordingly. 
     There are several criteria used by the remote diagnostic embedded process subsystem  28  that need to be met in order for the latter to notify the technical support center  22  of potential problems within the network device  14 . One such criterion is met when an error message is detected by the remote diagnostic embedded process subsystem  28  as shown in  FIG. 4 . There are other criteria, besides detection of an error message, that are configurable, i.e., the technical support center  22  may alter these criteria by reconfiguring the remote diagnostic embedded process subsystem  28 . Listed below are some of the criteria that need to be met before a message is forwarded to the technical support center  22 . 
     1. Low memory (both shared or main memory) 
     2. High percentage of call failures on a modem or a trunk line 
     3. Detection of an error message 
     4. Detection of software reload due to software failure 
     3. Detection of a failed line or interface (i.e. a 11 line going down) 
     6. Detection of hardware problems (i.e. a board is shutdown due to a high temperature problem) 
     7. User defined interval (i.e. memory leak, CPU utilization, etc.) 
     8. Quality of an interface (i.e. high collision on Fast Ethernet) 
       FIG. 5  shows a block diagram outlining an example of the sequence of steps taken in sending an email message from the network device  14  to the network  18 . In the example of  FIG. 5 , information regarding the status of three software subsystems is gathered by the software health status monitor subsystem at step  76 . The three software subsystems are shown as software subsystem A, B through N. Examples of such software subsystems are an IP protocol, a user interface and a Netscape browser program. For instance, the software subsystem A in the example of  FIG. 5  is an IP protocol, the software subsystem B is a user interface and the software subsystem N is a Netscape browser program. While three software subsystems are shown in  FIG. 5 , there may be more or less than three software subsystems employed. Information regarding the status of the software subsystems A, B and N is gathered at steps  62 ,  64  and  66 , respectively, by the software health status monitor subsystem at step  76 . 
     In  FIG. 5 , information regarding the status of hardware subsystems is gathered by the hardware health status monitor subsystem at step  74 . The hardware subsystems are referred to as device driver A  68 , device driver B  70  and device driver N  72 . As in the case of software subsystems, more or less than three hardware subsystems, or device drivers, may be employed without departing from the scope and spirit of the present invention. The device drivers  68 - 72  may be different types of hardware structures. As an example, the device driver A  68  is shown to be an Ethernet device, the device driver B  70  is shown to be a hard disk controller device and the device driver N  72  is shown to be a modem device in  FIG. 5 . Next, information from the hardware health status monitor subsystem, the software health status monitor subsystem and the memory monitor subsystem, at step  78 , is forwarded to the remote diagnostic embedded process subsystem where it is collected, combined and updated at step  80 . 
     Based on the information compiled at the remote diagnostic embedded process subsystem  28 , the status of the network device, (referred to, as the “system” in  FIG. 5 ) is determined at step  82 . Subsequently, at step  84 , a determination is made as to whether or not the technical support center  22  should be notified regarding the status of the system. If there is no problem with the system there is no need to notify the technical support center  22  and the process of collecting information again continues from step  80 . However, if there is a problem in the system as indicated, for example, by an error message in the computer register shown in  FIG. 4 , then a notification is transmitted, as indicated at step  86 . The notification is formatted as either an email message or a fax or a page and is transmitted through the email/page subsystem at step  88  to the network  90 . 
       FIG. 6  shows a flow diagram of the sequence of steps taken in receiving commands from the technical support center  22  and responding thereto. Initially commands originating in the technical support center  22  are received through the packet switching network  92  by the email/page subsystem  94 . The commands are then parsed and interpreted at step  96  and stored in the remote diagnostic embedded process subsystem, as shown at step  98 . There are several types of actions that may be taken in response to the commands indicated at step  100 . The first type of action is to prepare a report of the status of the network device and send the report back to the network as shown at step  102 . Subsequently, the report is formatted as an email message, fax or a page at step  126  and sent to the email/page subsystem at step  128 . The email/page subsystem then transmits the information in the report through the packet switching network  130 . 
     The second kind of action is to send a command to a software subsystem at step  104  in order to perform diagnostic tests. The command is implemented on three software subsystems shown at steps  108 ,  110  and  112 . The result of the diagnostic testing is compiled in the software health status monitor subsystem at step  122 . The last kind of action taken by the remote diagnostic embedded software is to send a command to a hardware subsystem as indicated at step  106  for purposes of diagnostics. The command is implemented on three hardware subsystems  114 ,  116  and  118 . The result of the diagnostic tests is compiled in the hardware health status monitor subsystem at step  120 . Information regarding the status of the system as gathered by the software health status monitor subsystem and the hardware health status monitor subsystem is collected and combined together at step  124 . The status information is then formatted into an email message or a fax or a page at step  126  and transmitted via the email/page subsystem at step  128  to the network  130 . 
     Although the present invention has been described in terms of specific embodiments it is anticipated that alterations and modifications thereof will no doubt become apparent to those skilled in the art. It is therefore intended that the following claims be interpreted as covering all such alterations and modification as fall within the true spirit and scope of the invention.