Abstract:
A power control system comprises a plurality of power control groups, with each group comprising a plurality of individual point-of-load regulators each adapted to provide respective regulated voltage outputs. The point-of-load regulators may be selected for inclusion in a power control groups based on characteristics of loads supplied by the point-of-load regulators. An intermediate bus controller is coupled to each of said power control groups through a serial data bus interface common to each group and an OK status line for each respective group. A front end regulator provides an intermediate bus voltage to each of the plurality of power control groups and to the intermediate bus controller. The plurality of point-of-load regulators of each group each further comprises a respective fault manager adapted to detect fault conditions and selectively communicate notifications of the fault conditions to other ones of the plurality of point-of-load regulators of the group and to the intermediate bus controller. This way, a common response to the fault conditions is taken by the point-of-load regulators of the group and other groups. A method for managing faults in the power control system is also disclosed.

Description:
RELATED APPLICATION 
     This application is a divisional of U.S. patent application Ser. No. 10/890,573, filed Jul. 13, 2004 now U.S. Pat. No. 7,372,682. 
    
    
     BACKGROUND OF THE INVENTION 
     Perhaps more than ever, high-end computing and telecommunications applications, for example, are using highly optimized integrated circuits such as microprocessors, field programmable gate arrays (“FPGAs”), application-specific integrated circuits (“ASICs”), etc., wherein the silicon processes for such circuits are selected and/or adjusted to maximize performance and reduce costs. This very often yields differing power supply requirements for each circuit, i.e., different discrete voltage and current levels. Further, many of these circuits require a relatively low voltage (e.g., 1 v or less), but with relatively high current (e.g., 100 A). It is undesirable to deliver relatively high current at low voltages over a relatively long distance through an electronic device for a number of reasons. First, the relatively long physical run of low voltage, high current lines consumes significant circuit board area and congests the routing of signal lines on the circuit board. Second, the impedance of the lines carrying the high current tends to dissipate a lot of power and complicate load regulation. Third, it is difficult to tailor the voltage/current characteristics to accommodate changes in load requirements. 
     Decentralized power architectures have been developed to address the power supply requirements for such systems. In one such power architecture, an intermediate bus voltage is distributed throughout the electronic system, and an individual point-of-load (“POL”) regulators, i.e., DC/DC converters, are located at the point of power consumption within the electronic system. Each POL regulator would convert the intermediate bus voltage to the level required by the corresponding electronic circuit. Ideally, the POL regulator would be physically located adjacent to the corresponding electronic circuit so as to minimize the length of the low voltage, high current lines through the electronic system. The intermediate bus voltage can be delivered to the multiple POL regulators using low current lines that minimize loss. 
     This decentralization process can be pushed so far that almost all loads (microprocessors, FPGAs, etc.) in the application have their own power supply. In other words, while an application (as a whole) may be designed to perform a single main function, its power supply system (which provides power to the chips located therein) may be built from individual, stand alone POL regulators. A drawback of such power supply systems, however, is that they fail to reflect the inter-dependency of the loads the POL regulators are supplying. For example, if one POL regulator fails, then the circuit supplied by this converter will also fail without notifying or disabling the other chips that are dependant upon the first circuit. This can result in unpredictable malfunction of the load or further damage to the power supply system by overstressing the other chips and/or their related POL regulators. Conventional power supply systems provide only very simple fault management in the form of power-good signals, which an application can use to determine a faulty power supply condition of a particular POL regulator. This is generally insufficient to provide system-level protection for the loads in case of a fault. 
     Thus, it would be advantageous to have a system and method for managing faults in a distributed power system having a plurality of POL regulators. 
     SUMMARY OF THE INVENTION 
     The present invention provides a system and method for managing fault in a power supply system. Point-of-load regulators are assigned to virtual groups in accordance with the characteristics of the loads that they supply. In the event of a fault condition of one of the point-of-load regulators, corrective action to overcome the fault condition can be selectively applied to other point-of-load regulators of the same group n a like manner. If the fault is severe enough, the fault condition can be propagated to other groups for consistent corrective action. 
     In an embodiment of the invention, a power control system comprises a plurality of power control groups, with each group comprising a plurality of individual point-of-load regulators each adapted to provide respective regulated voltage outputs. The point-of-load regulators may be selected for inclusion in power control groups based on characteristics of loads supplied by the point-of-load regulators. An intermediate bus controller is coupled to each of the power control groups through a serial data bus interface common to each group and an OK status line for each respective group. A front end regulator provides an intermediate bus voltage to each of the plurality of power control groups and to the intermediate bus controller. The plurality of point-of-load regulators of each group each further comprises a respective fault manager adapted to detect fault conditions and selectively communicate notifications of the fault conditions to other ones of the plurality of point-of-load regulators of the group and to the intermediate bus controller. This way, a common response to the fault conditions is taken by the point-of-load regulators of the group and other groups. 
     More particularly, the intermediate bus controller further comprises a master fault manager in communication with each group through the OK status lines. The master fault manager receives the notifications of fault conditions and selectively communicates the notifications to the groups, which can then subsequently disable the point-of-load regulators in the respective groups. The point-of-load regulators each further comprise a status register in which is stored a data record of the detected fault conditions. The status register further comprises plural data fields corresponding to plural categories of detected fault conditions, such as reflecting differing levels of severity. Depending upon the type of fault detected, the fault manager of each of the point-of-load regulators may take any number of corrective action, including a) disabling a corresponding point-of-load regulator in response to one of the fault conditions and re-enabling the disabled point-of-load regulator after a pause period, b) disabling the point-of-load regulator and latching the disabled point-of-load regulator in that state, or c) disabling the point-of-load regulator and re-enabling the disabled point-of-load regulator after a pause, and if the fault persists, trying to re-enable the point-of-load regulator for a specific number of times, and if not successful, latching the point-of-load regulator in the disabled state. 
     The fault manager of each of the point-of-load regulators may also communicate notifications of the fault conditions to the intermediate bus controller via a corresponding one of the OK status lines. The fault manager of each of the point-of-load regulators of one of the groups receives the notifications of fault conditions from any one of the point-of-load regulators of the same group, and also receives notifications of fault conditions from the master fault manager in the intermediate bus controller assuring a synchronous enabling/disabling of several point-of-load regulators of one or more groups. In response to system wide faults, the intermediate bus controller may disable the front end regulator to thereby cut off the intermediate bus voltage from each of the plurality of power control groups. The power control system may further include a crowbar circuit responsive to the intermediate bus controller to drive the intermediate bus voltage to ground. The intermediate bus controller may also include a communication interface to a system controller to notify the user of any faults, their severity levels, and the corrective actions taken. The communication interface may also permit programming of the point-of-load regulators and/or the intermediate bus controller to define the manner in which fault conditions are managed and propagated to other point-of-load regulators and/or groups. 
     In another embodiment of the invention, a method for managing fault conditions in a power control system comprises the steps of detecting fault conditions locally within each point-of-load regulator of each group, selectively communicating notifications of fault conditions to other ones of the point-of-load regulators of the group and/or to other groups, and, taking a common corrective action in response to the notifications of the fault conditions by the plurality of point-of-load regulators of the group and/or to other groups. This may additionally include receiving notifications of the fault conditions and selectively communicating notifications to other groups. The corrective actions may include disabling a point-of-load regulator and re-enabling the point-of-load regulator after a pause period, disabling a point-of-load regulator and latching the point-of-load regulator in that state, disabling all point-of-load regulators of one of the groups, or disabling all point-of-load regulators of all of the groups. On a system wide basis, the corrective response may include cutting off an input voltage provided to each of the power control groups or driving to ground the input voltage provided to each of the power control groups. 
     A more complete understanding of the system and method of managing faults in a power system will be afforded to those skilled in the art, as well as a realization of additional advantages and objects thereof, by a consideration of the following detailed description of the preferred embodiment. Reference will be made to the appended sheets of drawings, which will first be described briefly. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of a point of load (POL) control system in accordance with an embodiment of the invention; 
         FIG. 2  is a block diagram of an exemplary POL regulator; 
         FIG. 3  is a block diagram of an exemplary intermediate bus controller; 
         FIG. 4  is a simplified block diagram of a POL control system exhibiting a fault condition and associated response; 
         FIG. 5  is a simplified block diagram of a POL control system exhibiting a fault condition propagated to other POLs within a group; 
         FIG. 6  is a simplified block diagram of a POL control system exhibiting a fault condition propagated to all groups within a system; 
         FIG. 7  is a flow diagram depicting a process for managing faults in a POL control system; 
         FIG. 8  is an exemplary screen shot depicting a graphical user interface (GUI) for programming the grouping of POL regulators within a POL control system; 
         FIG. 9  is another exemplary screen shot depicting a GUI for programming fault detection and handling for POL regulators within a POL control system; and 
         FIG. 10  is another exemplary screen shot depicting a GUI for programming of fault propagation for a POL control system. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
     The present invention provides a system and method for managing faults in a distributed power system having a plurality of POL regulators. In the detailed description that follows, like element numerals are used to describe like elements illustrated in one or more figures. 
     Referring first to  FIG. 1 , a POL control system is shown in accordance with an embodiment of the present invention. The POL control system includes an intermediate bus controller  102 , a front-end regulator  104 , and a plurality of groups  120 ,  130 ,  140 ,  150 . Each of the groups includes a plurality of individual POL regulators, such that Group A  120  includes exemplary POL regulators  122 ,  124 ,  126 , Group B  130  includes exemplary POL regulators  132 ,  134 ,  136 , Group C  140  includes exemplary POL regulators  142 ,  144 ,  146 , and Group D  150  includes exemplary POL regulators  152 ,  154 ,  156 . The POL regulators depicted herein include, but are not limited to, point-of-load regulators, power-on-load regulators, DC/DC converters, voltage regulators, and all other programmable voltage or current regulating devices generally known to those skilled in the art. 
     Each group of POL regulators produces a plurality of output voltages that are supplied to corresponding loads. The POL regulators may be grouped depending upon the characteristics of the loads that are supplied. For example, POL regulators supplying loads with high dependencies could be placed into one group, e.g., all POL regulators supplying CPU core voltages could be placed in one group (e.g., Group A) and POL regulators supplying auxiliary circuits could be placed in another group (e.g., Group B). By grouping plural POL regulators together, the POL regulators within a group can exhibit the same responsive behavior in the case of a fault condition. Moreover, each group of POL regulators represents a virtual, and not physical, grouping of POL regulators. The POL regulators of a particular group may actually be physically separated from each other within an electrical system. It should be appreciated that the number of groups and POL regulators depicted in each group in  FIG. 1  are presented solely for exemplary purposes, and that a greater or lesser number of groups and/or POL regulators within each group could be advantageously utilized. 
     The front-end regulator  104  provides an intermediate voltage (V IN ) to the plurality of groups over an intermediate voltage bus. The front-end regulator  14  may simply comprise another POL regulator. The intermediate bus controller  102  draws its power from the intermediate voltage bus. Although depicted as separate devices, the intermediate bus controller  102  and front-end regulator  104  may be integrated together in a single unit. Alternatively, the front-end regulator  104  may provide a plurality of intermediate voltages to the groups of POL regulators over a plurality of intermediate voltage buses. 
     The intermediate bus controller  102  communicates with the plurality of POL regulators by writing and/or reading digital data (either synchronously or asynchronous) via a unidirectional or bidirectional serial bus, illustrated in  FIG. 1  as the synch/data (SD) line. The SD line may comprise a two-wire serial bus (e.g., I 2 C) that allows data to be transmitted asynchronously or a single-wire serial bus that allows data to be transmitted synchronously (i.e., synchronized to a clock signal). In order to address any specific POL regulator in any group, each POL regulator is identified with a unique address, which may be hardwired into the POL regulator or set by other methods. The intermediate bus controller  102  also communicates with each one of the plurality of groups for fault management over respective unidirectional or bidirectional serial lines, illustrated in  FIG. 1  as the OKA, OKB, OKC and OKD lines (also referred to below as respective OK lines). 
     The intermediate bus controller  102  communicates with a user system via a serial data bus (e.g., I 2 C) for programming, setting, and monitoring the POL control system. A memory device  108  may optionally be coupled to the serial data bus for storing programming and initial condition data. The intermediate bus controller  102  may access this memory  108  via the serial data bus, such as to retrieve initial condition data during start-up of the POL control system. The intermediate bus controller  102  may additionally receive an input signal reflecting a failure of the AC main supply. Upon receipt of such an AC FAIL signal, the intermediate bus controller  102  may command the orderly shut down of the POL regulators. 
     Lastly, the intermediate bus controller  102  communicates with the front-end regulator  104  over a separate line (FE EN) to disable operation of the front-end regulator in the event of a system-wide fault. If there is a component failure in one of the POL regulators, the output of that POL regulator could experience an overvoltage condition that could damage its respective load. It is therefore very desirable to reduce as quickly as possible the intermediate bus voltage when such a fault is detected. Accordingly, the POL control system may further include an optional crowbar circuit  106  coupled to the intermediate voltage bus, which drives to ground the voltage remaining on the intermediate voltage bus and thereby cuts-off the intermediate voltage (V IN ) to the POL regulators and prevents any overvoltage conditions. 
     An exemplary POL regulator  122  of the POL control system is illustrated in greater detail in  FIG. 2 . The other POL regulators of  FIG. 1  have substantially identical configuration. The POL regulator  122  includes a power conversion circuit  162 , condition sensors  164 , a status register  166 , a fault manager  168 , a serial interface  172 , and a memory  174 . The power conversion circuit  162  transforms the intermediate voltage (V IN ) to the desired output voltage (V OUT ) according to settings received through the serial interface  172  or default settings stored in the memory  174 . The power conversion circuit  162  may comprise a conventional buck, boost, buck-boost, or other known DC-to-DC converter topology. 
     The condition sensors  164  monitor the output voltage and current, operating temperature, and other external parameters that are used for local control. The condition sensors  164  can detect fault conditions in the POL regulator. These detected fault conditions could further be classified into sub-categories depending on their severity, e.g., (i) low (i.e., warnings that parameters have exceeded certain tolerances, e.g., temperature is high, output voltage is outside of tight tolerance band, etc.), (ii) mid (i.e., fault conditions that require corrective action but are not yet urgent, e.g., temperature over limits, output voltage below limits, output current over limits, etc.), and/or (iii) high (i.e., fatal errors that require immediate corrective action to prevent harm to the POL, load or overall system, e.g., output voltage over limits, power switch of power conversion circuit in short circuit, etc.). Other classifications of detected fault conditions could also be advantageously utilized. The status register  166  is coupled to the condition sensors  164 , and maintains a status record of the detected fault conditions. When a fault condition is detected by the condition sensors  164 , a corresponding data record or flag is written to the status register  166 . 
     The fault manager  168  monitors the status register  166  and determines appropriate action for the detected fault conditions. The fault manager  168  can be configured, for example, to (i) poll the status register  166  and take action to protect the individual POL regulator  122 , the group to which it belongs, and/or the entire POL control system, (ii) disable the POL regulator and re-enable it after a pause period (i.e., auto recovery mode), (iii) disable the POL regulator and latch the fault, (iv) propagate the fault within the group, and/or (iv) disable the power conversion circuit  162  in case of external faults. The fault manager  168  can propagate any one of these fault conditions to other POL regulators and/or other groups by changing the status of the OK line and/or by communicating a message through the SD line. Each other POL regulator of the group would detect the fault condition by sensing the change in state of the OK line, and their respective fault managers would take corresponding action. The fault manager  168  could be programmed to latch the malfunction and prevent a restart of the POL regulator  122  when the fault trigger has disappeared, or enable the POL regulator to re-start automatically after the fault trigger has disappeared or after a predetermined period of time. It is important to note that, if the POL regulator attempts to re-start, then this can be made to occur synchronously with other POL regulators within the group that were disabled because of the fault detected by the first POL regulator. 
       FIG. 3  is a block diagram of an exemplary intermediate bus controller  102 . The intermediate bus controller  102  includes condition sensors  182 , a system status register  184 , a fault manager  186 , a plurality of group status registers  188   a - d , a serial interface  192 , and a memory  194 . As with the condition sensors  164  of  FIG. 2 , the condition sensors  182  monitor the system level fault conditions, such as operating temperature, AC line failure, intermediate bus voltage level, and other external interrupts that are used to control system power. These detected fault conditions could further be classified into sub-categories depending on their severity, e.g., (i) low (e.g., temperature is high, etc.), (ii) mid (e.g., temperature over limits, AC line failure, etc.), and/or (ii) high (e.g., system level interrupts, etc.). Other classifications of detected fault conditions could also be advantageously utilized. The system status register  184  is coupled to the condition sensors  182 , and maintains a status record of the detected fault conditions. When a fault condition is detected by the condition sensors  182 , a corresponding data record or flag is written to the system status register  184 . The group status registers  188   a - d  are each respectively coupled to a corresponding group via the respective OK line. The group status registers  188   a - d  reflect a change in status of one of the groups based on communication of information from one of the POL regulator fault managers. 
     The system fault manager  186  operates generally similar to the fault manager  168  of  FIG. 2 . The system fault manager  186  is coupled to the system status register  184  and the group status registers  188   a - d . The system fault manager  186  can also communicate with the groups via the serial data bus by use of the serial interface  192 . Depending on the severity of fault conditions reported by either the system status register  184  or one of the group status registers  188   a - d , the system fault manager can selectively propagate the fault conditions to other groups or to the entire system. In the event of significant fault conditions, the system fault manager  186  can also trigger the crowbar circuit  106  to short the intermediate voltage bus to ground and/or disable the front-end regulator  104 . The memory  194  stores the default configuration data for the intermediate bus controller  102 . The default configuration is selected such that the intermediate bus controller  102  will operate in a “safe” condition in the absence of programming signals. 
     More particularly, the system fault manager  186  monitors the system status register  184  and group status registers  188   a - c , and determines appropriate action for the detected fault conditions. If a system-wide fault is detected on the system status register  184 , the fault manager  186  may take system-wide corrective action, such as shutting down each of the groups and the front end regulator  104 . If a fault condition is detected that affects only one of the groups, the fault manager  186  can be configured, for example, to (i) poll the group status register  188  and take action to protect the individual group, (ii) disable each of the POL regulators of the group and re-enable them after a pause period (i.e., auto recovery mode), (iii) disable each of the POL regulators of the group and latch the disabled condition, (iv) propagate the fault condition to other groups, and/or (iv) disable the entire system by shutting off the front end regulator  104  and/or activating the crowbar circuit  106 . The system fault manager  186  can propagate any one of these fault conditions by changing the status of the corresponding group OK line and/or by communicating a message through the SD line. Each group would detect the fault condition by sensing the change in state of the OK line and/or receiving a message on the SD line, and their respective fault managers would take corresponding action. 
     Referring to  FIG. 7 , a flow diagram depicts an exemplary process  200  for managing faults by a fault manager  168  of a POL regulator. Steps  202  and  210  reflect an initial loop in which the fault manager  168  checks for the presence of external and internal faults (or changes in status), respectively. In the absence of such faults (or changes in status), the fault manager  168  will continuously loop through steps  202  and  210 . External faults (or changes in status) are handled by a portion of the process that includes steps  204 - 208 . Internal faults are handled by a separate portion of the process that includes steps  212 - 242 . 
     Starting at step  202 , the fault manager  168  checks whether there has been a change in status of the OK line propagated by the system fault manager  186  or by the fault manager of another POL regulator of the same group. A change in status of the OK line reflects the handling of external faults by the intermediate bus controller  102  or another POL regulator, i.e., outside of this particular POL regulator, as opposed to internal faults detected by this particular POL regulator. The OK line has two possible states: (1) a clear condition (i.e., no fault condition present); and (2) a set condition (i.e., fault condition present). Thus, a change in state of the OK line from set to clear means that a previous fault condition has been resolved or cleared, and a change in state from clear to set means that a fault condition has been detected by the system fault manager  186  or by a fault manager of another POL regulator of the same group, and is being propagated to the group. 
     If there has been an external fault (or change in status), the fault manager  168  passes to step  204 , in which the fault manager determines whether the status of the OK line has changed to set or to clear. If the status has changed to set, then the fault manager  168  shuts off the power conversion circuit  162  at step  206 , thereby propagating the system fault condition locally to the POL regulator. Conversely, if the status has changed to clear, then the fault manager  168  turns on the power conversion circuit  162  at step  208 , thereby propagating the clearing of the system fault condition. 
     Following either one of steps  206 ,  208 , or if there has been no change in status of the OK line as detected in step  202 , the fault manager passes to step  210  to determine whether an internal fault condition has been detected. If no internal fault condition has been detected, then the fault manager  168  returns to step  202  and repeats the initial check loop process. But, if an internal fault condition has been detected, the fault manager  168  decides at step  212  whether to initiate a recovery process to clear the fault condition or whether to set the fault condition. 
     Steps  230 - 242  illustrate the internal fault condition set process. The fault manager  168  selects the internal fault set process at step  212  when an internal fault event is detected for the first time. Then, at step  230 , the fault manager  168  determines the severity level of the detected fault event, such as by checking the status register  166 . If the severity level is low, then no corrective action is necessary other than to report the fault condition to the intermediate bus controller  102 . At step  242 , the fault manager  168  communicates a corresponding notification message to the intermediate bus controller  102  via the SD line. The power conversion circuit  162  of the POL regulator remains in an operational state, and the process returns to the beginning. Conversely, if the severity is not low, the fault manager  168  determines at step  232  if the severity level is mid or high. In either case, the fault manager  168  will turn off the power conversion circuit  162  of the POL regulator, albeit at a different rate depending upon the severity level. If the severity is high, the fault manager  168  immediately shuts of the power conversion circuit  162  at step  236 . Alternatively, if the severity is mid, the fault manager  168  shuts of the power conversion circuit  162  at step  240  in accordance with a more orderly or gradual process (e.g., ramping down the output voltage at a predetermined rate or sequence rather than abruptly shutting it off). 
     At intermediate steps  234 ,  238 , the fault manager  168  may additionally propagate the fault condition to the intermediate bus controller  102  and/or other POLs connected to the same OK line by changing the state of the OK line. The decision to propagate the fault to the intermediate bus controller  102  (i.e., steps  234 ,  238 ) may be based on initial programming of the POL regulator. 
     Steps  214 - 222  illustrate the internal fault clear (i.e., recovery) process. The fault manager  168  selects the internal fault clear process at step  212  on a subsequent pass through the process after an internal fault condition fault was previously detected. As in step  230 , the fault manager  168  determines in step  214  the severity level of the detected fault event, such as by checking the status register  166 . If the severity level is low, then no corrective action is necessary other than to report the fault condition to the intermediate bus controller  102 . At step  222 , the fault manager  168  communicates a corresponding notification message to the intermediate bus controller  102  via the SD line. The power conversion circuit  162  of the POL regulator remains in an operational state, and the process returns to the beginning. Conversely, if the severity level is not low, and the power conversion circuit  162  was turned off in a previous pass through steps  236  or  240 , the fault manager  168  determines at step  216  whether to restart the power conversion circuit  162 . This decision may be based on initial programming of the POL regulator. For example, the power conversion circuit  162  may be restarted if the fault condition was only a transient event lasting less than a predetermined period of time. If the decision is to not restart the power conversion circuit  122 , such as if the fault condition is particularly serious or long-lasting, or if the same fault condition has recurred repeatedly after successive restarts, then the power conversion circuit  162  is latched in the off state, i.e., precluding subsequent restart of the power conversion circuit  162  without first conducting further action such as a diagnostic evaluation or maintenance service on the POL regulator. Conversely, if the decision is to restart the power conversion circuit  162 , the fault manager  168  notifies the intermediate bus controller  102  of the change in operational status by changing the state of the OK line. The fault manager  168  then restarts the power conversion circuit  162 , and the process returns to the beginning. 
       FIGS. 4-6  illustrate the management of fault conditions within an embodiment of the present invention. These figures show an exemplary power management system with POLs organized into two groups with each group having two POLs. In  FIG. 4 , a fault condition is detected in POLL of Group A. The fault manager of POL 1  is programmed to shut down the power conversion circuit (as denoted by the X), either in accordance with a normal ramp down of output power or an immediate shut down, as dictated by the severity of the detected fault condition. In this embodiment, the fault manager has been programmed to not propagate the fault by changing the status of the OK line. Accordingly, POL 2  of the same group remains operating normally, as are POL 3  and POL 4  of Group B (denoted by the check mark). The fault manager of POL 1  decides whether to latch the fault or perform an auto-recovery of POL 1 . 
     In  FIG. 5 , a fault condition is again detected in POL 1  of Group A. As in  FIG. 4 , the fault manager of POL 1  is programmed to shut down the power conversion circuit (as denoted by the X), either in accordance with a normal ramp down of output power or an immediate shut down, as dictated by the severity of the detected fault. Unlike the previous embodiment, the fault manager has been programmed to propagate the fault within the group by changing the status of the OK line. Accordingly, POL 2  of the same group has also been shut down, but POL 3  and POL 4  of Group B remain operational (denoted by the check mark). If the fault manager of POL 1  decides to perform an auto-recovery, then both POL 1  and POL 2  would restart synchronously. 
     In  FIG. 6 , a fault condition is again detected in POL 1  of Group A. As in  FIG. 4 , the fault manager of POL 1  is programmed to shut down the power conversion circuit (as denoted by the X), either in accordance with a normal ramp down of output power or an immediate shut down, as dictated by the severity of the detected fault. Unlike the previous embodiments, the fault manager has been programmed to propagate the fault condition to the intermediate bus controller  102 , which in turn propagates the fault to Group B. POL 3  and POL 4  of Group B are now shut down. If the fault manager of POL 1  decides to perform an auto-recovery, then the POLs of each group would restart in an orderly manner in accordance with their programming. 
     As discussed above, the intermediate bus controller  102  has an interface for communicating with a user system for programming and monitoring performance of the POL control system. The user system would include a computer coupled to the interface, either directly or through a network, having suitable software adapted to communicate with the intermediate bus controller  102 . As known in the art, the computer would be equipped with a graphics-based user interface (GUI) that incorporates movable windows, icons and a mouse, such as based on the Microsoft Windows™ interface. The GUI may include standard preprogrammed formats for representing text and graphics, as generally understood in the art. Information received from the intermediate bus controller  102  is displayed on the computer screen by the GUI, and the user can program and monitor the operation of the POL control system by making changes on the particular screens of the GUI. 
       FIGS. 8-10  illustrate exemplary screen shots of a GUI used for programming fault and error detection for a POL control system.  FIG. 8  shows a screen used to define a configuration of POL regulators within a POL control system. The screen includes a matrix with the horizontal axis defining the POL number ( 00 - 31 ) and the vertical axis defining the group (A-D). A user can assign individual POL regulators to groups by activating (e.g., clicking on) the associated icon located at the intersection of a selected POL number and group. For example, POL numbers  00  and  02  are assigned to Group A, POL number  04  is assigned to Group C, and POL number  06  is assigned to Group D. The screen also enables the programming of interrupts, intermediate bus voltage over-voltage and under-voltage protection, and other performance characteristics. The user can create, save, and edit configuration files using the GUI screen. 
       FIG. 9  shows a screen used to program a configuration file for a POL regulator. The screen includes a table that identifies fault trigger type and severity. A discussed above, various fault trigger types are listed in order by severity, including temperature high and power good listed as warnings, tracking differential, over-temperature, over-current, and under-voltage listed as faults, and over-voltage and phase error listed as errors. Each fault type has corresponding user-selectable fields to designate desired handling by the fault manager, including propagate (i.e., the fault condition is propagated by changing the status of the OK line), latching (i.e., the fault condition will cause a shut-down of the power conversion circuit and latching in the shut-down state), enable (i.e., the fault condition is not reported), and report to IMC (intermediate bus controller). It should be appreciated that other fault trigger types and other choices of programmed handling can also be advantageously utilized. The user system can also determine whether the programming will be applied to one POL regulator, to all POL regulators of a group, or to all POL regulators (and groups) of the power control system. 
       FIG. 10  shows a screen used to program fault and error propagation for the POL control system. The screen graphically shows a first box reflecting group identity in which a fault condition originates and a second box reflecting group identity to which the fault condition is propagated. There is a series of lines connecting the two boxes, with intersections between the lines reflecting a communication from one group to another. The user can program the configuration by selectively activating icons at the intersections between the lines to define a desired propagation path. For example, the user can selectively activate icons to enable a fault condition of Group A to be propagated to Groups B and C, but not to Group D. For each group, the user can also program whether to turn off the front end  104  and activate the crowbar circuit  106  in the event of a fault condition. 
     Having thus described several embodiments of a system and method for managing fault in a power system, it should be apparent to those skilled in the art that certain advantages of the system have been achieved. It should also be appreciated that various modifications, adaptations, and alternative embodiments thereof may be made within the scope and spirit of the present invention. The scope of the invention is limited only by the following claims.