Patent Publication Number: US-6708283-B1

Title: System and method for operating a system with redundant peripheral bus controllers

Description:
FIELD OF THE INVENTION 
     The invention relates generally to fault tolerant computer systems and, more particularly, to mechanisms for fault tolerant access to system-critical devices on peripheral busses. 
     BACKGROUND OF THE INVENTION 
     Fault-tolerant computer systems are employed in situations and environments that demand high reliability and minimal downtime. Such computer systems may be employed in the tracking of financial markets, the control and routing of telecommunications and in other mission-critical functions such as air traffic control. 
     A common technique for incorporating fault-tolerance into a computer system is to provide a degree of redundancy to various components. In other words, important components are often paired with one or more backup components of the same type. As such, two or more components may operate in a so-called lockstep mode in which each component performs the same task at the same time, while only one is typically called upon for delivery of information. Where data collisions, race conditions and other complications may limit the use of lockstep architecture, redundant components may be employed in a failover mode. In failover mode, one component is selected as a primary component that operates under normal circumstances. If a failure in the primary component is detected, then the primary component is bypassed and the secondary (or tertiary) redundant component is brought on line. A variety of initialization and switchover techniques are employed to make a transition from one component to another during runtime of the computer system. A primary goal of these techniques is to minimize downtime and corresponding loss of function and/or data. 
     Fault-tolerant computer systems are often costly to implement since many commercially available components are not specifically designed for use in redundant systems. It is desirable to adapt conventional components and their built-in architecture whenever possible. 
     To reduce downtime, fault tolerant systems are designed to include redundancy for connections and operations that would otherwise be single points of failure for the system. Accordingly, the fault tolerant system may include redundant CPUs and storage devices. Certain devices on peripheral busses may also be single points of failure for the system. In a system that uses a Windows operating system, for example, the loss of a controller for peripheral busses and/or a video controller results in a system failure. 
     Devices such as a keyboard, mouse, monitor, floppy drives, CD ROM drives, and so forth typically communicate with a system I/O bus, such as a PCI bus, over a variety of peripheral busses such as a USB and an ISA/IDE bus. The various peripheral busses connect to the PCI bus through a peripheral bus controller, such as an Intel PCI to ISA/IDE Xcelarator. The windows operating systems require that the peripheral bus controller plug into location  0  on the system PCI bus, or what is commonly referred to as “PCI bus  0 .” 
     A PCI-to-PCI bridge may be used to provide additional slots on a PCI bus. A bridge for use with the PCI bus  0 , for example, provides slots for the system-critical peripheral bus controller and video controller, and various other devices. The PCI-to-PCI bridge is then a single point of failure, as is the peripheral bus controller and the video controller. While it is desirable to provide fault tolerance by including redundant paths to the peripheral devices, through redundant PCI-to-PCI bridges and associated peripheral bus controllers and video controllers, the operating system is not equipped to handle them. The operating system requires that all of the peripheral bus controllers connect to PCI bus  0 , and redundant controllers alone thus can not provide the desired, fully redundant paths to the peripheral devices. Accordingly, what is needed is a mechanism to achieve such redundancy within the confines of the commercially available operating systems. 
     SUMMARY OF THE INVENTION 
     The inventive system essentially hides redundant paths to the peripheral devices from the operating system, by reporting a single “virtual” path to the peripheral busses over PCI bus  0 . The virtual path includes at least a virtual peripheral bus controller and a virtual video controller. The system also tells the operating system that the real controllers are on another PCI bus on an opposite side of a PCI-to-PCI bridge connected also to PCI bus  0 . An I/O system manager selects one of the actual paths, which may, but need not, be connected to PCI bus  0 , to handle communications with the peripheral devices. 
     The I/O system manager maintains the controllers on the unselected path in an off-line or standby mode, in case of a failure of one or more of the controllers on the selected path. If a failure occurs, the I/O system manager performs a fail-over operation to change the selection of controllers, as discussed in more detail below. The operating system does not respond to the controller failure by declaring a system failure, however, because the operating system continues to look to the virtual path, with its virtual controllers, as a valid path to the peripheral devices. Accordingly, the fail-over operation does not adversely affect the overall operations of the system. 
     As discussed in more detail below, the system also allows hot swapping of PCI bridges, and associated devices on the PCI bus and the peripheral busses. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The invention description below refers to the accompanying drawings, of which: 
     FIG. 1 is a high-level functional block diagram of a system constructed in accordance with the invention; 
     FIG. 2 is a more detailed functional block diagram of the system; 
     FIG. 3 is a more detailed functional block diagram of a front panel included in the system of FIG. 2; 
     FIG. 4 is a high-level functional block diagram of system configuration software layers; 
     FIG. 5 is a flow chart of the operations of a hardware abstraction layer of FIG. 4; 
     FIG. 6 depicts a view of the I/O subsystem of FIG. 2 by a plug and play manager of FIG. 4; 
     FIG. 7 is a more detailed functional block diagram of an I/O board of FIG. 2; 
     FIG. 8 is a more detailed functional block diagram of an I/O board of FIG. 2; 
     FIG. 9 is a more detailed functional block diagram of a front panel of FIG.  3 . 
    
    
     DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT 
     Referring to FIG. 1 a fault tolerant computer system provides redundant communication paths between each central processing unit  12  and peripheral devices that are supported by or connected to a system front panel  36 . The communications paths include redundant system PCI busses  14 , redundant peripheral busses  34  and redundant PCI-to-peripheral bus connections, which are depicted in the drawing as blocks  25 . The CPUs  12  are redundant, and thus, the system is tolerant of faults in the CPUs  12 , the busses  14  and  34  and the bus-to-bus connections represented by the blocks  25 . We discuss the system hardware and system operations in more detail below. 
     Referring now to FIG. 2, each CPU  12  consists of one or more central processors  11  that reside on a CPU board  10  along with associated memory and registers  16  and a “north-side” PCI bridge interface processor  18 . The CPU  12  communicates over redundant PCI-to-PCI bridges  22  with an I/O subsystem  24 . The I/O subsystem includes redundant I/O boards  26  that provide connections to redundant peripheral busses  34 , which connect, in turn, to the various peripheral devices that are supported by or connected through the front panel  36 . 
     A given PCI-to-PCI bridge  22  includes the north-side PCI bridge interface processor  18  that connects to a north-side of the PCI bus  14  on an associated CPU board  10 , a bus  19  and a “south-side” PCI bridge interface processor  20  that connects to the south-side of the PCI bus  14  on an associated I/O board  26 . The north-side PCI bridge interface processor provides connections to two locations on the north side PCI bus  14 , namely, physical location  0  and location  1 , which are referred to hereinafter as PCI bus  0  and PCI bus  1 . The south-side PCI bridge interface processor  20  similarly connects to two locations on the south side PCI bus  14  and provides PCI slots  28  to connect at least a peripheral bus controller  30  and a video controller  32  to the PCI bus. In the exemplary system, there is also a slot  28  for an I/O system manager  31  that monitors and controls certain operations of I/O subsystem hardware. There may be additional PCI slots for SCSI devices (not shown), and so forth. 
     The peripheral bus controller  30  and video controller  32  provide the actual interconnection between the PCI bus  14  and one end of the peripheral busses  34 . The peripheral busses  34  connect at their other ends to storage devices such as a floppy drive  38  and a CD ROM drive  40  that are supported by the front panel  36 , and front-panel connectors  42 ,  46  and  50  for a keyboard  44 , a mouse  48  and a monitor  52 , respectively. The peripheral busses  34  include standard peripheral busses, such as a USB for the keyboard and so forth, and/or one or more ISA/IDE busses for the drives. The front panel  36  may support multiple floppy drives  38  or CD ROM drives  40  and/or include additional connectors for devices such as modems, and so forth. 
     As depicted in FIG. 3, the front panel also contains switches  60  that connect on one side to the redundant peripheral busses  34  and on the other side to the floppy drive  38 , CD ROM drive  40 , and the connectors  42 ,  46  and  50 . The switches  60  operate under the control of the I/O system manager  31 , to pass signals between the devices connected to or supported by the front panel  36  and a selected set of the peripheral busses  34 , as discussed in more detail below. 
     The system depicted in FIGS. 1-3 provides fully redundant communications paths from each CPU  12  to the front panel  36 . Accordingly, the failure of any devices on the redundant paths should not, from a hardware point of view, cause the system to go down. As is known to those skilled in the art, any path to the peripheral devices should be designed to ensure signal integrity. Accordingly, the lengths of each of the paths in the current system should be minimized and, in this system, the redundant paths should be essentially the same length. Further, the impedance of the traces in each path should be carefully controlled. 
     Failures of the non-redundant peripheral devices that plug into the front panel, such as the keyboard or monitor, while perhaps inconvenient to the user, are not system-critical failures. These devices can readily be replaced by plugging in replacements. As discussed in more detail below with reference to FIG. 9, the front panel  36  is designed for hot swapping, such that the floppy and CD ROM drives and the connectors can also be replaced without having to bring the system down. 
     We discuss immediately below how the system operates with redundant paths to the peripheral devices, when the operating system requires that the peripheral bus controller and the video controller connect only to PCI bus  0 . 
     When the system boots-up, it configures the system hardware using a predetermined, or default, pairing of a CPU and a set of I/O subsystem components that are resident on a selected I/O board  26 . The default pairing also includes a default selection of the peripheral bus and video controllers  30  and  32  that are resident on the selected I/O board. If the default selections should fail to boot the system, the system selects another CPU and/or set of I/O subsystem components to use for the boot-up operation. If another set of I/O subsystem components is selected for the boot-up operations, the system also changes its selection of the peripheral bus and the video controllers to those on the selected board. 
     Referring now also to FIGS. 4 and 5, as part of the boot-up operations, a plug and play manager  70  calls on routines, or functions,  74  in a hardware abstraction layer  72 , or HAL, which is essentially an interface between the operating system and the system hardware. The plug and play manager requests that the HAL, using these functions  74 , enumerate the PCI bus  14 , and thereafter, the devices attached to the PCI bus. 
     To enumerate the PCI bus  14  (step  500 ), the functions  74  send queries to the devices on the bus by location. The devices respond to the query by identifying themselves at least by type, that is, as peripheral bus controllers, video controllers and so forth. The functions  74  determine from the responses that there are redundant peripheral bus controllers  30  and video controllers  32  on PCI busses  0  and  1 . The functions  74  then essentially hide the redundant peripheral bus controllers and the redundant video controllers from the operating system by informing the plug and play manager that a “virtual” peripheral bus controller and a “virtual” video controller are on PCI bus  0  (step  502 ). The functions  74  report the virtual system-critical controllers on PCI bus  0  even if only a single set of controllers  30  and  32  are included in the system. In this way, the addition of redundant controllers to the system at a later time does not alter the path to the peripheral devices from the view point of the operating system. 
     Referring also to FIG. 6, which depicts the devices, both real and virtual, as seen from the point of view of the plug and play manager, the functions  74  also inform the plug and play manager that the two PCI-to-PCI bridges  22  are on PCI bus  0  and that various other devices such as, for example, ethernet controllers and so forth are on the south side of each of the bridges  22  (step  504 ). Further, the functions  74  list, as attached to the south side of each bridge  22 , the selected controllers  30  and  32  identified as “special” controllers, such as “stratus controllers,” so that the plug and play manager and/or the operating system will not interpret them to be the system-critical bus and video controllers (step  506 ). As a final enumeration step, the functions  74  query the devices on the peripheral busses  34  and tell the plug and play manager that the various peripheral devices, such as the keyboard, mouse, monitor and so forth are attached to the appropriate virtual controllers (step  508 ). 
     The plug and play manager  70  assigns appropriate device functions to the key board, the mouse, and so forth, and informs the operating system about the various peripheral devices that are attached to the virtual controllers. The operating system then, in a conventional manner, assigns appropriate device objects to the devices. 
     As part of the boot-up operations, the system also configures I/O address space for communications to and from the front-panel devices. The system sets up the I/O address space on the CPU side of the PCI-to-PCI bridges  22 , so that the space is accessible to all of the peripheral bus and video controllers. The I/O space is thus set up in the memory  16  on each CPU board. 
     At any given time, the system permits only one peripheral bus controller and one video controller to claim the I/O address space in the memory  16 . The system uses a peripheral bus configuration register  17  (FIG. 2) on the CPU board  10 , or at least one predetermined bit location in the register, to establish which controllers may claim the I/O address space in the memory  16 . The register bit is thus set to the appropriate value to select, as appropriate, the controllers connected to the PCI bus  0  at the CPUs or the controllers connected to the PCI bus  1  at the CPUs. The PCI bridge interface processor  18  responds to the state of the address space bit by passing information to and from the I/O address space and the selected controllers over the appropriate connections to PCI bus  0  or PCI bus  1 . 
     If one of the selected controllers  16  or  22  should later fail, the system performs a fail-over operation to change its controller selections. As part of the fail-over operation, the I/O system manager  31  disables the PCI bus interface processor  20  associated with the failed controller, and thus, the south sides of the associated PCI-to-PCI bridges  22 . The I/O system manager  31  on the south side of the other PCI-to-PCI bridges  22  is notified of the failure, and responds by changing the setting in the configuration registers  17  on each of the CPU boards  12 , to give the previously unselected controllers access to the I/O address space in the memory  16 . The I/O system manager  31  also changes the control setting of the switches  60 , such that the switches use the peripheral busses  34  associated with the newly selected set of controllers  30  and  32 . 
     The newly selected controllers  30  and  32  are either powered down or in standby mode at the time they are selected. Accordingly, the controllers must be configured to set the associated internal registers and memory to the appropriate states. The controller configuration may be performed in essentially the same manner as it is performed during a boot-up operation, and the internal registers and memory may be set to the default states. Alternatively, the I/O system manager  31  may periodically save the internal states of the selected controllers  30  and  31 , and use these states to configure the newly selected controllers. The I/O system manager must also handle the fail-over of the other controllers on the south side of the PCI bus in an appropriate manner. 
     The controller failure is reported to the plug and play manager  72  either by one of the I/O system managers  31  based on errors detected through system diagnostics, or by some other system component. The plug and play manager requests that the HAL functions  74  again enumerate the PCI bus  14 . In response, the functions provide to the plug and play manager the same information about the virtual controllers on PCI bus  0 , the attached PCI-to-PCI bridges, and so forth. The functions  74  also tell the plug and play manager that the special controllers  30  and  32  connected to the south side of one of the bridges  22  are no longer connected, and that other special controllers, that is, the previously non-selected controllers  30  and  32 , are now connected to the south side of the other bridge  22 . The plug and play manager determines that the reported change in the controller connections does not warrant the sending of an error message to the other system components, however, because the virtual controllers still provide the path to the peripheral devices. 
     The PCI-to-PCI bridge  22  associated with the failed controller is disabled in the exemplary system so that, as appropriate, the I/O board  26  and devices resident thereon can be replaced, or hot swapped, as discussed in more detail below. Alternatively, the faulty controller alone may be hot swapped while the associated bridge  22  is disabled. 
     As discussed above, the I/O system manager  31  may periodically save the internal states of the selected peripheral bus controller  30  and the video controller  32 . The system may also track the time between failures of the various controllers, or other statistics that allow the system essentially to predict controller failure. The system can then save the controller states, and change its selection of controllers at an appropriate time before a failure occurs. 
     We discuss below the hot plugging operations of the system. First, we discuss hot plugging an I/O board  26 . Then, we discuss hot plugging a device into a PCI slot  28 . Finally, we discuss hot plugging the front panel  36  and associated devices. 
     Referring now to FIG. 7, diagnostic routines performed by a system hardware manager  82  and/or error detection logic  83 , which are resident in whole or in part on a mother board  80  into which the I/O boards  26  plug, inform a system hot plug controller  84  and the I/O system manager  31  that a device resident on a particular I/O board, such as the peripheral bus controller  30 , has failed. The system hardware manager  82  and/or the error detection logic  83  may determine that the bus controller  30  has failed based on errors in the communications from the bus controller, such as the bus controller using an address with a parity error or the bus controller not starting a bus transaction within a predetermined time after being granted access to the bus, and so forth. 
     In response, the I/O system manager  31  asserts an I/O broken signal. The I/O manager then changes the controller selection for the peripheral busses, as discussed above, so that the board no longer receives signals from or directs signals to the peripheral devices. The hot plug controller  84  next electrically isolates the I/O board  26  and the various devices thereon from the PCI bus by disabling a switch, for example, FET  86 , through which power is supplied at least to the processor  20 . The faulty I/O board  26  can then be removed. 
     When the I/O board  26  is unplugged from the system an associated board present signal is de-asserted by a switch that is part of a back panel (not shown), which is opened to gain access to the board. The open back panel also opens a switch that cuts power to the board  26 . A new board is then plugged into the system and the back panel is closed. This again asserts the board present signal and provides power to the board. The I/O system manager  31  then asserts a reset signal as part of its boot-up operations. The I/O broken signal remains asserted, that is, it is still essentially held by the state of a bit in a status register on the CPUs. The I/O system manager then configures the new board  26 , that is, it sets the clock and logic levels on the new board in a conventional manner. 
     When the clock and logic levels are valid, the I/O system manager de-asserts the reset signal. A predetermined time thereafter the manager also de-asserts the I/O broken signal. By the time both signals are de-asserted, the new I/O board  26  is fully configured and operational, and the I/O system manager can then bring the board on-line by changing the board status to on-line at a time that allows the board to properly handle a next PCI bus transaction. Other register and memory information may be copied from the other I/O board  26 , as necessary. The insertion of the board in this manner does not disrupt the devices and/or transaction on the system PCI bus. 
     Referring now to FIG. 8, we discuss hot plugging a device into the PCI bus. As discussed above, the system hardware manager  82  or the error detection logic  83  detects the failure of a device  88  that is on the PCI bus. The I/O system manager  31  responds to the failure by asserting the I/O broken signal, also as discussed above. In the exemplary system, the system manager  31  may change the selection of controllers  30  and  32  if the failed device is, for example, the video controller  32 . Otherwise, the manager may leave the controller selection in place. 
     If the device is in a hot pluggable slot  28 , a hot plug controller  90  resident on the I/O board isolates the failed device  88  from the PCI bus  14  by turning off a signal FET  92  that resides in the slot  28  between the PCI bus  14  and the device  88 . The hot plug controller  80  then turns power off to the slot  28  through a power FET  94 , which powers down the device  88  that is plugged into the slot. At the same time the I/O system manager  31  changes the status of the slot  28  to off-line. The electrical isolation of the device  88  and later powering off of the PCI slot  28  does not adversely affect the operations of the other devices on the PCI bus  14 . 
     The isolated device  88  can now be removed from the powered-off slot  28 , and a new device can be hot plugged into the slot, without disrupting the other devices on the PCI bus. To remove the device, an ejector button  100  is activated. The button  100  controls a switch  102  that, in turn, controls a board present signal. The signal is de-asserted when the switch  102  is activated, to indicate that the slot  28  is empty. 
     When a new device  88  is plugged into the slot  28  the ejector button  100  is deactivated to close the switch  102  and the device present signal is again asserted. The hot plug controller  90  then applies power to the slot through the power FET  94 , and the device  88  boots, to establish its clock and logic levels. At the same time the I/O system manager  31  asserts a reset signal. After the clock and associated logic levels are established and guaranteed to be valid the I/O system manager  31  deasserts the reset signal. When the PCI bus is next idle, the hot plug controller  90  enables the signal FET  92 , to connect the device to the PCI bus. A predetermined time thereafter, the I/O system manager  30  de-asserts the I/O broken signal, and the device  88  goes on-line. 
     In the exemplary system, the connectors for the devices have power pins that are shorter than the signal pins and ground pins that are longer than the signal pins. Accordingly, as a device is removed from the slot, the power pins disconnect immediately and the ground pins are the last to disconnect, to prevent noise from disrupting the other devices on the board. Similarly, when a device is plugged into the system, the ground pins are the first to connect and the power pins the last to connect, to prevent damage to the device being inserted and to the other devices on the board. 
     Certain of the peripheral devices  18 , such as the CD ROM, floppy drive and so forth may be essentially part of the front panel  36 , along with the various connectors for the plug in devices such as, for example, the monitor and keyboard. Since these are noncritical devices, they are not duplicated in the system. When one of these front panel devices or connectors fails, the entire front panel in the exemplary system can be replaced without disrupting the system. 
     Referring now to FIG. 9, the front panel  36  electrically connects to system power through a power controller  62  and to the peripheral busses  34  through a front panel board  66  that includes the switches  60 . As discussed above, certain pins on the connectors between boards are shorter to provide last-to-make and first-to-break connections. For the connector  64 , the pins for two enable lines  65  and  63  are short. These pins connect to ground on a shield board (not shown), into which the front panel board  66  plugs to electrically connect the associated peripheral devices and/or connectors to the peripheral busses  34  and system power. When the front panel board is fully plugged into the shield board, the two enable lines are drawn to a desired low state to signal that the board is then present. 
     To ensure that the enable lines are not drawn to the desired state, which in the exemplary system is the low state, before the board  66  is fully inserted, pull-ups  68  are included on the lines in the switches  60  and power controller  62 . The pull-ups keep the lines  65  and  63  high until they are driven low by their connection to ground. 
     When the lines  65  and  63  are high, the switches  60  are disabled, and the IDE, USB and VGA busses are shut off. Accordingly, as soon as a user begins to remove the front panel board  66 , the busses are turned off by the switches  60 . In this way the bus signal pins and, in particular, the IDE signal pins, are protected from any power spikes that may occur as the board  66  powers down. 
     The enable lines  65  and  63  connect also to the power controller  62 . As the front panel board  66  is removed, the power controller turns off and powers down the board. The power ramps down, however, so that the switches  60  turn off before the power is fully off. This ensures that the switches  60  operate properly to protect the bus signal pins as the board is removed. 
     When the front panel is removed, the I/O system manager  31  notices the change and notifies the plug and play manager  72 . The plug and play manager requests a PCI bus enumeration, and the HAL functions  74  then tell the plug and play manager that certain peripheral devices are no longer connected, and the plug and play manager determines that the user should be notified. 
     Once the faulty devices and/or connectors are replaced, the front panel can be plugged back into the system. When the board is inserted, the two enable lines are drawn low and power is supplied to the front panel board  66  through the power controller  62 . The switches  60  are also enabled and provided power, such that they again connect the peripheral devices and connectors to the selected set of peripheral busses based on control signals supplied by the system manager. The plug and play manager is then told of the change to the I/O subsystem, and the change is reflected in adding the devices to the virtual controllers as part of PCI bus enumeration, and the system continues to operate.