Abstract:
An upgradable computer system has a first software component and a second software component, in which the first and second software components operate at a current version. The computer system upgrades the first software component to an upgraded version and validates the performance of the upgraded first software component. The validation includes translating messages originating at the first software component from an upgraded version format to a current version format.

Description:
FIELD OF THE INVENTION  
         [0001]    The present invention is directed to fault tolerant systems. More particularly, the present invention is directed to portable software for upgrades of fault tolerant systems.  
         BACKGROUND INFORMATION  
         [0002]    As computer systems, network systems and software systems become more complex and capital intensive, system failures become more and more unacceptable. This is true even if the system failures are minor. Generally, when systems fail, data is lost, applications become inaccessible, and computer downtime increases. Reducing system failures is often a major goal for companies that wish to provide quality performance and product reliability in the computer systems, network systems and/or software systems which they operate. As such, these systems must be highly dependable. Fault tolerance has been implemented as a way of achieving dependability.  
           [0003]    For a system to be fault tolerant, it must be able to detect, diagnose, confine, mask, compensate, and/or recover from faults. In general, there are three levels at which fault tolerance may be applied: hardware level, software level and system level. In the hardware level, fault tolerance is often achieved by managing extra hardware resources, through redundant communications, additional memory, duplicate processors, redundant power supply, etc. In the software level, computer software is structured to compensate for faults resulting from changes in data structures or applications because of transient errors, design inaccuracies, or outside attacks. In the system level, system fault tolerance provides functions that compensate for failures that are generally not computer-based. For example, application-specific software may detect and compensate for failures in sensors, actuators, or transducers.  
           [0004]    Even in the hardware level and the system level, application software is generally utilized to control, provide and/or assist in the detection and recovering of fault. As such, it is essential that to achieve system fault tolerance, application software itself must be fault tolerant. Hardware is generally a couple of orders of magnitude more reliable than software, and the majority of the failures in today&#39;s systems that incorporate software applications are in fact typically caused by software problems.  
           [0005]    Fault tolerance is typically achieved in application software by either the underlying operating system and hardware or by customizing the application to operate in an active/standby redundant configuration. However, when an application uses the underlying operating system and hardware to achieve fault tolerance, it becomes dependent upon, or “tied down” to that operating system and hardware platform.  
           [0006]    Application software in most systems are required to be upgraded from time to time to upgrade the software by incorporating new features or fix bugs. Most current mechanisms of upgrading software involve shutting down the system and reloading the system with the upgraded software. Known mechanisms to perform software upgrades without shutting down the system are also typically based on the characteristics and capabilities of the platform on which these mechanisms are implemented.  
           [0007]    Based on the foregoing, there is a need for an improved system and method that allows software on a fault tolerant system or distributed fault tolerant system to be upgraded without shutting down the system, or without being based on the characteristics and capabilities of the platform. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0008]    [0008]FIG. 1 is a block diagram of a computer system in accordance with one embodiment of the present invention.  
         [0009]    [0009]FIG. 2 is a block diagram of a non-upgraded software component and an upgraded software component in accordance with one embodiment of the present invention.  
         [0010]    [0010]FIG. 3 is a flow chart illustrating steps performed in accordance with one embodiment of the present invention for implementing rolling upgrades of a computer system.  
         [0011]    [0011]FIG. 4 is a block diagram illustrating the placement of translation functions in the software components. 
     
    
     DETAILED DESCRIPTION  
       [0012]    One embodiment of the present invention is a fault tolerant system or distributed fault tolerant system in which application software is upgraded using a rolling upgrade method. During the upgrading, upgraded and non-upgraded copies of the application software co-exist in the system while the performance of the upgraded version of the software is being validated. In one embodiment, a translation function on upgraded software components allow the upgraded components to communicate with the non-upgraded components.  
         [0013]    One embodiment of the present invention allows the computer system to be upgraded with a new version of software without loss of system availability and service availability and allows upgraded and non-upgraded versions of the software to co-exist in the system for the duration in which the functionality of the upgraded version of the software is being validated. One embodiment of the present invention further allows fallback to the non-upgraded version of the software if the upgraded version of the software does not function satisfactorily in the validation phase. One embodiment of the present invention further allows automatic enabling of new features in the upgraded software when all software components participating in the feature have been upgraded with the capability to support the new feature.  
         [0014]    [0014]FIG. 1 is a block diagram of a computer system  20  in accordance with one embodiment of the present invention. Computer system  20  includes multiple processors  21 - 24 . Processors  21 - 24  can be any type of general purpose processor. Processors  21 - 24  are coupled to a bus  28 . Also coupled to bus  28  is memory  25 . Memory  25  is any type of memory or computer readable medium capable of storing instructions that can be executed by processors  21 - 24 .  
         [0015]    One embodiment of computer system  20  is a fault tolerant system in which applications are made fault tolerant using a hot standby mechanism. In this embodiment, computer system  20  only includes two processors (e.g., processors  21 ,  22 ), one of which functions as an active processor, and one of which functions as a standby processor. An example of such a fault tolerant system is disclosed in U.S. patent application Ser. No. 09/967,623, entitled “System and Method for Creating Fault Tolerant Applications”, filed on Sep. 28, 2001 and assigned to Intel Corp. Another embodiment of computer system  20  is a distributed and fault tolerant system that includes more than two processors. An example of such a distributed fault tolerant system is disclosed in U.S. patent application Ser. No. 09/608,888, entitled “Apparatus and method for building distributed and fault tolerant/high-availability computer applications” filed on Jun. 30, 2000 and assigned to Intel Corp.  
         [0016]    [0016]FIG. 2 is a block diagram of a non-upgraded software component  30  and an upgraded software component  32  in accordance with one embodiment of the present invention. Each component  30 ,  32  includes a collection of interfaces  16 ,  18  and features  10 - 12 . Interfaces are means by which software components interact and connect with each other and are defined as a named collection of message and constant declarations. Each interface message has the following characteristics:  
         [0017]    Syntax—The structural information associated with the interface message (e.g., the type and number of parameters exchanged by the software components in communication over this interface, etc.); and  
         [0018]    Semantics—The behavior associated with the interface message (e.g., the actions taken by a software component when it receives a message over the interface, etc.).  
         [0019]    An upgraded version of the software component may contain new or modified interfaces, such as upgraded interface  18  (“I&#39;  18 ”) of software component  32 . Some of the software interfaces may even have been deleted. The upgraded software can contain new or modified features, such as feature  12  of software component  32 , and some of the software features may have been deleted.  
         [0020]    [0020]FIG. 3 is a flow chart illustrating steps performed in accordance with one embodiment of the present invention for implementing rolling upgrades of computer system  20 . In the embodiment described, the steps are stored as software in memory  25  and executed by processors  21 - 24 . In other embodiments, the steps are performed by any combination of hardware or software.  
         [0021]    A processor is selected for rolling upgrade (step  50 ) and is isolated from the rest of the system (step  52 ) by shutting down the software components residing on the processor. If the software component is fault-tolerant in nature, the other copy of the software component will take over and continue to provide service. If the software is distributed in nature, then the other copies of the software component take over the workload.  
         [0022]    The processor, which has been isolated, is reloaded with upgraded software (step  54 ) and is configured. The software is configured with a configuration equivalent to that existing in the system before attempting a rolling upgrade (i.e., new features in the upgraded software are kept disabled).  
         [0023]    The reloaded processor is re-integrated into the system (step  56 ) and starts providing service.  
         [0024]    The performance of the upgraded software is validated (step  58 ). If the validation fails (step  60 ) the process enters a fallback phase (step  62 ). In the fallback phase, all upgraded software components in the system are taken through an isolation, reload and integration cycle using the older version of the software. At the end of the fallback phase the system falls back to the old software version.  
         [0025]    If the validation of the upgraded software components has been performed and the performance is found to be acceptable, the process enters a closure phase. In this phase, the other processors in the system are taken through the isolation, reload and integration phases (steps  64 ,  66 , etc.).  
         [0026]    After completion of the closure phase all software components in the system are now upgraded and new features in the upgraded software are activated (step  68 ).  
         [0027]    If the software component is fault tolerant (i.e., has an active and a standby copy) or distributed (i.e., has multiple active and standby copies) the different processors hosting the active and standby copies of the application may be upgraded in an asynchronous manner at different times. Therefore, during the validation phase of the rolling upgrade process (step  58 ), upgraded and non-upgraded copies of the software components may have to co-exist. The software component, which is being upgraded, may have to communicate with other software components in the system. Different software components in the system may be upgraded at different times.  
         [0028]    As shown in FIG. 2, an upgraded version of a software component may contain new or modified interfaces and features. Therefore, an upgraded version of the software component should be able to adapt its interfaces and features to communicate with a non-upgraded software component. In addition, an upgraded copy of a software component in a distributed or fault tolerant environment may have to communicate with its peers or standby copies, which may not have been upgraded.  
         [0029]    To achieve this, in one embodiment of the present invention each interface of the software component is tagged with an interface version number. When the interface undergoes change (i.e., the syntax or semantics associated with the interface messages changes), the interface version number is incremented. Each interface version has is composed of a major number and a minor number. The major version number is incremented when the changes are made to the latest version of the interface and the minor version number is incremented when the changes are made to an older version of the interface.  
         [0030]    When a new processor has to be integrated into the system (step  56 ), the following operations are performed:  
         [0031]    1. Query the version numbers of the interfaces implemented by the software components on the processor to be integrated.  
         [0032]    2. Based on the capabilities of the interface version numbers supported by the other software components in the system, arrive at a compatible interface version to be used on each of the software component interfaces. This version number is the highest compatible interface version number implemented by all software components sharing that interface.  
         [0033]    3. Indicate to the software components the interface version number to be used on each of the software component interfaces. The software components use this interface version number to adapt the respective interfaces.  
         [0034]    In one embodiment of the present invention, the software component implementing the higher version of the interface adapts to communicate with a software component implementing a lower version of the interface. If a software component has to communicate over an interface to another software component implementing a lower version of the interface, the component implementing the higher interface version invokes translation functions to translate the interface message parameters from the order and type defined for the interface version implemented by the software component to the order and type defined by the lower version of the interface. The version number used to form the message is also sent to the destination of the message.  
         [0035]    At the destination of the message if the version number used to form the message (encoded into the message by the originator) is lower than the version number of the interface implemented by the destination, the message is passed through a translation function to translate the interface message parameters from the order and type defined for the interface version implemented by the destination.  
         [0036]    [0036]FIG. 4 is a block diagram illustrating the placement of translation functions in the software components. Software components  80 ,  90  communicate over an XYZ interface  82 ,  92 , where  82  is the XYZ interface implemented by software component  80  and  92  is the XYZ interface implemented by software component  90 . Software component  80  implements version 2 of XYZ interface  82 . Software component  90  implements version 1 of XYZ interface  92 . The rolling upgrade architecture in accordance to one embodiment of the present invention directs both the software components to use version 1 on the XYZ interface.  
         [0037]    The software component implementing the higher version of the interface has to adapt to the interface and therefore software component  80  passes all messages it originates over interface  82  towards software component  90  through a translation function  84  to convert from version 2 formats to version 1 formats. When software component  90  is upgraded to support version 2 of XYZ interface  92 , the rolling upgrade architecture directs both software components to start using version 2 for originating messages over this interface.  
         [0038]    The following pseudo-code provides the structure of translation function  
                                                                                                                                                                                                                                                         PROCEDURE TranslateAndSendMessageA                INPUT RemoteVersion           INPUT MessageAParams           START                SWITCH (RemoteVersion)                Case SELF_VERSION:                /* SELF_VERSION is version implemented by component sending the message */           i. Send message;                Case SELF_VERSION − 1:                i. Convert MessageAParams from SELF_VERSION to SELF_VERSION − 1                format                ii. Send Message                Case SELF_VERSION − 2:                ...                ENDSWITCH                FINISH           PROCEDURE ReceiveAndTranslateMessageA           INPUT IncomingVersion           INPUT MessageAParams           START                SWITCH (IncomingVersion)                Case SELF_VERSION:                /* SELF_VERSION is version implemented by component receiving the message */           i. Process Message;                Case SELF_VERSION − 1:                i.   Convert MessageAParams from SELF_VERSION−1 to SELF_VERSION               format           ii.   Process Message                Case SELF_VERSION − 2:                ...                ENDSWITCH                FINISH                      
 
         [0039]    In one embodiment, the higher version of the interface may introduce new parameters in the message. When a destination receives a message formed using a lower version of the message, it would have to derive the value of this new parameter from the other message parameters. In cases where such derivation is not possible, the translation functions can be instructed to substitute configurable default values for these parameters. Deleted and modified parameters are adapted in a similar manner.  
         [0040]    The message may also contain certain parameters under compile time flags. In one embodiment, the rolling upgradable system is reloaded with same version of software but compiled with a different set of compile time flags. Hence even though the version numbers of the interfaces implemented by all software components is same, the message packing order and format may change due to different set of compile time flags being enabled at the originator and the destination. In one embodiment, this issue is solved by the translation functions by introducing a bit vector into the message. Each bit of the bit vector indicates the state of a compile time flag at the originator of the message. The destination of the message then bases its message decoding decisions on the bits indicated in the message and the compile time flags enabled at the destination of the message.  
         [0041]    The following pseudo-code can provide the handling of the bit vector at the originator of the message:  
                                                                       FOR each compile time flag emabled at originator                Set corresponding bit in bit vector                ENDFOR           Encode bit vector in the message sent to destination                      
 
         [0042]    The following pseudo-code can provide the handling of the bit vector at the destination of the message:  
                                                                                                                                             FOR each message parameter under compile time flag                IF compile time flag enabled at destination           THEN                IF bit vector indicates compile time flag enabled at originator           THEN                Decode parameter from message for use                ELSE                Assume Default Value for parameter                ENDIF                ELSE                IF bit vector indicates compile time flag enabled at originator           THEN                Decode parameter from message and discard                ENDIF            ENDFOR                  
 
         [0043]    In one embodiment, the upgraded software components may have support for new features. The new features implemented by the upgraded software component should not be activated until all software components participating in the feature have been upgraded. Therefore, during the validation phase (step  58 ) of the rolling upgrade process, new features introduced in the upgraded software components should be disabled.  
         [0044]    After all copies of all the software components participating in a feature have been upgraded using the rolling upgrade process in accordance with the present invention, the new features may be activated. Software component features may be activated by one of the following mechanisms:  
         [0045]    Features activated by configuration—A new configuration has to be provided to activate the new feature.  
         [0046]    Features activated by control—A command has to be issued to the software component to active the new feature.  
         [0047]    Features activated by version synchronization—Certain features introduced in the software component may not require any new configuration for their activation. However they may be dependent on the interface capabilities for their proper function. When all software components participating in the feature have been upgraded, the upgraded software component is asked to use the latest version of the interface as described in the rolling upgrade process. When this event happens such features can become activated automatically.  
         [0048]    As described, one embodiment of present invention allows application software to be upgraded using a rolling upgrade method in a fault tolerant system or distributed fault tolerant system. During the upgrading, upgraded and non-upgraded copies of the application software co-exist in the system while the upgraded version of the software is being validated through the use of a translation function.  
         [0049]    Several embodiments of the present invention are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.