Abstract:
Presented is a modular uninterruptible power supply system utilizing common power modules, and providing redundant coordinated control thereof. The system and method of the instant invention provides identical control logic within each of the individual power modules, thus dispensing with the requirement for a separate control module to control and coordinate the operating modes and parameters of the UPS system. A system is presented whereby a virtual master is established through an arbitration scheme at initialization of the UPS system. The master then assigns a virtual vice master to provide the redundant back up control should the virtual master no longer be able to perform its functions. Nearly simultaneous control of operational mode and state change functions is accomplished through a coordinated communications system including a high-speed communications bus and digital control logic lines. Also provided is a mechanism for removal of the virtual master from its duties upon the detection of a problem that is not detected by the virtual master itself.

Description:
FIELD OF THE INVENTION 
     This invention relates generally to the control of uninterruptible power supplies, and more particularly to the control and coordination of multiple power modules in a modular uninterruptible power supply. 
     BACKGROUND OF THE INVENTION 
     As more and more segments of the business environment enter the information age, more and more computers and computing power are required. As business move from the old to the new economy their reliance on the processing, transference, and storage of digital information is becoming a more and more critical aspect of their overall business strategy. While in the past, computer crashes were seen as a mere nuisance, the loss of computing power and business data may well devastate a business&#39;s ability to survive in today&#39;s new economy. As such, the need for reliable, uninterruptible electric power to maintain the operational status of the computing equipment and the integrity of the digital data continues to rise. 
     To meet these requirements, uninterruptible power supplies (UPS) have been developed. These UPSs utilize a bank of electric storage batteries and solid state conversion equipment in association with the utility line voltage to provide continuous electric power to a business&#39;s computer system in the event of a loss or deviation of power quality from the utility. The number of batteries contained within an UPS is dependent upon the business&#39;s length of time that it needs to operate in the event of a utility power system failure. Likewise, the number of power modules included in a modular UPS is dependent on the overall total system load required to be supplied thereby. In the past, uninterruptible power supplies were only available in discreet sizes. However, with the recognition of the various requirements from the multitude of businesses that require UPSs, the development of modular uninterruptible power supplies has provided a flexibility heretofore unknown. Now, a business&#39;s UPS power output may grow as its business needs by simply purchasing additional power modules and adding them to their existing modular UPS. 
     Unfortunately, the advent of the modular UPS has also introduced new and different challenges to the UPS designer. Specifically, the modular UPS must function as a single system regardless of the number of power modules that are added to or taken away from the UPS chassis. Output power quality, system display, and transitions between operating modes must all be coordinated between the multiple power modules that make up the business&#39;s UPS system. 
     One such modular UPS that provides coordinated operation is described in U.S. Pat. No. 5,982,652 to Simonelli, et al. for a METHOD AND APPARATUS FOR PROVIDING UNINTERRUPTIBLE POWER USING A POWER CONTROLLER AND A REDUNDANT POWER CONTROLLER. As described in this patent, the coordination and control of the multiple power modules contained in the modular UPS is accomplished through the provision of a main intelligence module (MIM) and a redundant intelligence module (RIM). The MIM and RIM have connections to each other and to the other modules to allow monitoring of operational conditions of the UPS system, including characteristics of the input and output power, and to provide phase synchronization, frequency regulation, and voltage regulation of the output power. In this system, the MIM functions as the primary controller within the UPS system and the RIM is a redundant controller that can assume control of the UPS system upon failure of the MIM or removal of the MIM from the UPS system. All of the power modules are substantially identical and perform the functions of an uninterruptible power supply under the control of the MIM or the RIM. 
     In the system of the Simonelli, et al. &#39;652 reference, the main intelligence module (MIM) is the primary computer/controller in the UPS system. It acts as a central point in the system for collecting and communicating information about aspects of the power modules, battery modules, and the redundant intelligence module (RIM). The MIM contains a primary and a backup microprocessor for controlling and coordinating this operation. The redundant intelligence module (RIM) on the other hand is merely a backup version of the MIM, and provides redundancy in the event of a MIM failure, or while a MIM is being replaced. In this backup role, the RIM includes the same major systems as the MIM except for the main processor system, although some of the systems in the RIM are described as having less functionality than the corresponding systems in the MIM. 
     Unfortunately, such a control strategy greatly increase the overall cost of the modular UPS system by requiring that two separate additional modules be purchased and maintained in order to allow functionality of the UPS at all. In this way, in addition to the value providing power modules, battery modules, and battery charging modules, a user is forced to also purchase a separate main intelligence modules to control the other components in the system. Further, the user is forced to purchase a redundant intelligence module (RIM) whose only main purpose is to provide backup operation should the MIM fail. In other words, this module is totally superfluous to the actual operation and functionality of the uninterruptible power supply unless a failure of the MIM occurs. 
     In an economy that demands maximum efficiency from business resources, such a system that requires separate primary and redundant control modules simply cannot be justified. The unfeasibility of such a system may be clearly demonstrated in a situation where a business enterprise only needs a single power module to supply its uninterruptible power requirements. In such a situation, the Simonelli, et al., &#39;652 system would require the business enterprise to purchase two control units, the MIM and the RIM, to control the single power module in its UPS. In such a situation it becomes clear that the provision of two control units to control a single power module is a luxury not many businesses would desire to acquire. 
     Therefore, there exists a need in the art for a modular uninterruptible power supply that has coordinated control of its constituent components and an acceptable level of redundancy without requiring the purchase of additional multiple control unit. 
     BRIEF SUMMARY OF THE INVENTION 
     An embodiment of the invention provides a method and apparatus for coordinating the control of uninterruptible power supply modules to provide scalable, redundant power. Each of the independent UPS power modules are designed to contain identical control logic, both hardware and software, so that only a single power module need be purchased to scale the UPS system based on the business enterprises power requirements. Each of these identical UPS power modules are connected to common communications, power, and logic buses within the UPS system. Under such a system, the multiple power modules act as a single UPS with all of the controls and actions taken by the individual power modules being coordinated through a single power module acting as the system&#39;s virtual master. Preferably, the master role is virtual because any of the power modules in the system contain all of the necessary control logic and software to assume the master role. 
     For the system of the invention to be fault tolerant it must be capable of withstanding a failure or removal of the power module that has assumed the master role. Preferably, this is accomplished by requiring another module to be prepared to provide backup to the master. In this role, designated vice-master, the power module provides the controlling signals to itself and the other power modules if and when the designated master fails. Under this system, the vice-master then promotes itself to master and assigns a new vice-master from one of the operational power modules in the system. Under the system of the invention, no additional main intelligence modules or redundant intelligence modules are required to perform this functionality. Coordination and redundancy are provided within the control of the individual power modules themselves. 
     A power up arbitration scheme is used to assign one of the power modules to the master&#39;s role. Preferably, the master then assigns a vice-master and assigns all other modules to act as peers. Peer modules simply follow the control commands of the master or vice-master depending on the operational status of these two units. Modules in the peer role are also drawn upon when a vice-master needs to be appointed to the master role upon detection of a problem with that unit. This hierarchy of roles provides the means to coordinate identical UPS modules into a larger UPS system without the need for additional special main and backup controllers. In the event that only a single power module is connected in the UPS system, it is quite capable of controlling itself in the role of the master, without a requirement for a vice-master or any peer modules to be connected. Under the system of the invention, there is no need for two separate control modules to control the one power module connected in the system. 
     Power module interconnection and coordination of control is accomplished using a common set of signals, collectively known as the redundancy management bus (RMB). The RMB of the invention includes a high-speed communication bus, a set of bi-directional logic lines, and one analog reference for each power phase of the UPS system. Each power module may drive any of the signals. It is the role of the power module that determines which signals are driven and which signals are merely monitored. The high-speed communication bus may take any one of a number of forms, including CAN, Ethernet, Serial 485, etc. In a preferred embodiment, a high-speed communication bus takes the form of a CAN bus due to CAN&#39;s inherit fault tolerance and noise immunity. 
     In accordance with the invention, the master is responsible for driving most of the signals on the RMB. The other power modules take direction from the master by acting on these signals. When the master power module fails or is removed from the UPS system, a master ok signal typically supplied by the master power module is no longer sent. This provides an indication that the vice-master power module should take over driving the analog output voltage target signal used by the power modules to create their output voltage waveforms. Since the role of the vice-master has previously been assigned, the remaining peers do not see a discontinuity in the output voltage target signal and therefore their inverter output waveforms are clean during this transition. 
     In a preferred embodiment of the invention, the microprocessor inside each of the power modules is connected to the RMB through an interface computer programmable logic device (CPLD). Preferably two digital signals from the processor are used to indicate the power module&#39;s role to the CPLD. The role logic is used to enable bus drivers for each role&#39;s group of output signals. The power module&#39;s particular role in a given UPS system determines the bus input/output direction for each communicated signal. 
     In a preferred embodiment of the invention, the microprocessor inside each of the power modules is connected to the RMB through an interface computer programmable logic device (CPLD). Preferably two digital signals from the processor are used to indicate the power module&#39;s role to the CPLD. The role logic is used to enable bus drivers for each role&#39;s group of output signals. The power module&#39;s particular role in a given UPS system determines the bus input/output direction for each communicated signal. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The accompanying drawings incorporated in and forming a part of the specification illustrate several aspects of the present invention, and together with the description serve to explain the principles of the invention. In the drawings: 
     FIG. 1 is a system level single line diagram of a modular uninterruptible power supply system constructed in accordance with the teachings of the invention; 
     FIG. 2 is a detailed single line block diagram of the control structure of a power module constructed in accordance with the teachings of the invention; 
     FIG. 3 is a control flow diagram illustrating the operating role determination process of the invention; 
     FIG. 4 is a control flow diagram illustrating master control of a configuration state change of the individual power modules in the UPS of FIG. 1; 
     FIG. 5 is a control flow diagram illustrating a control methodology to allow power modules in the UPS system of FIG. 1 to vote out a master controller; 
     FIG. 6 is a control flow diagram illustrating the corresponding functional control logic within the virtual master in response to the vote out control flow of FIG. 5; and 
     FIG. 7 is a control flow diagram illustrating the assumption of the virtual master role by a virtual vice master in accordance with the teachings of the invention. 
     While the invention will be described in connection with certain preferred embodiments, there is no intent to limit it to those embodiments. On the contrary, the intent is to cover all alternatives, modifications and equivalents as included within the spirit and scope of the invention as defined by the appended claims. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     A preferred embodiment of the modular uninterruptible power supply (UPS)  10  constructed in accordance with the teachings of the invention is illustrated in single line block diagrammatic form in FIG.  1 . As with typical modular UPS&#39;s, the modular UPS  10  of the invention includes multiple slots  22   a-f  into which various modular components may be placed. As illustrated in FIG. 1, modules  22   a-c  have installed therein power modules  12   a-c  constructed in accordance with the teachings of the invention. Slots  22   d-f  are illustrated in FIG. 1 as being empty, thus providing the ability to expand or vary the power generation capability of the modular UPS  10 . 
     The AC input power from the utility is connected to the UPS  10  through lines  16 , and the output of the UPS  10  is supplied to the utilization equipment through output lines  18 . Preferably, each of the power modules  12   a-c  are identically constructed, and contain a power inverter  20   a-c  and a controller  24   a-c . As will be recognized by those skilled in the art, the particular topology utilized for the construction of the inverter  20  may vary substantially based on system and performance level requirements. Preferably, the inverter  20  utilizes a double conversion H-bridge with its own internal bypass. The control for this inverter  20  is provided by the controller  24  through the control lines  32   a-c . This control accomplishes both the proper operation of the power switches of the inverter to construct the output AC wave form, as well as the selection and designation of the proper operating mode for the inverter. These operating modes may include a bypass mode of operation for high efficiency or fault tolerance, and two modes of inverter operation. The first allowing the AC voltage from the utility to supply the inverter, while the second utilizing installed battery banks (not shown) to power the inverter. 
     As discussed above, the need for coordinated control between the individual power modules is paramount so that the UPS  10  may operate as an integrated unit. This unified and integrated control is accomplished by the separate control modules  24   a-c  in each of the individual, identical power modules  12   a-c . The controllers  24   a-c  each receive operating parameters of its own power module, such as input power parameters via line  26   c  and output power parameters via line  28   c . Additionally, the control modules communicate with one another via a high-speed communications bus  34 . This communications bus may be constructed in accordance with any of a number of high-speed communication standards, including CAN, Ethernet, serial 485, etc. Preferably, this bus is implemented as a CAN bus in the UPS system  10  due to CAN&#39;s inherent fault tolerance and noise immunity. 
     In addition to the high-speed communication, and for reasons which will be discussed more fully below, the controllers  24   a-c  also communicate via a set of bi-directional logic lines  36 . Further, communication between each of the controllers  24   a-c  to allow proper inverter coordination is accomplished via an analog reference  38  for each power phase of the UPS system  10 . Collectively, this common set of signals is known as the redundancy management bus (RMB). Advantageously, each of the power modules  12   a-c  are capable of driving any of the signals contained on the RMB based upon an established role assumed by the power module  12   a-c  as will be discussed below. This role of the power module determines which signals are driven and which signals are monitored. 
     FIG. 2 illustrates an individual power module  12  in greater detail. As may be seen from this detailed illustration, the controller  24  includes control logic  40  which is responsible for coordinating the integrated control of this particular power module  12  with the others included in the UPS system. The control logic  40  communicates via the bi-directional communication bus  34  to send and receive status information and nontime critical control signals. The control logic  40  may also receive a position signal  42  that designates in which slot of the UPS chassis the power module  12  is installed. This signal may also come from a user interface display controller that drives a user interface for the UPS, if one is included. The control logic  40  also communicates through a computer programmable logic device (CPLD)  44  over the set of bi-directional logic lines  36 . The CPLD is utilized to properly configure the control signals via lines  36  in input or output mode for communication to the control logic  40  via lines  46  based on the role that the control logic  40  has assumed in the overall system. This role is communicated to the CPLD  44  via the role selection lines  48 . 
     As will be discussed in greater detail below, the control logic  40  may determine that its power module  12  should be in the role of master, vice master, or peer. While various control signals may be utilized, a preferred embodiment of the invention contains control signals as indicated in Table 1. This table also indicates the direction of signal transfer depending on the role assumed by the control logic  40 . The role selection signals  48  for this embodiment are indicated in Table 2. Table 3 defines a preferred pin out arrangement for the CPLD  44  to allow proper coordination of the input and output information conveyance based upon the role assumed by the particular power module  12 . 
     
       
         
               
               
               
               
               
             
           
               
                 TABLE 1 
               
               
                   
               
               
                   
                 IO Direction 
                 IO Direction 
                 IO Direction 
                   
               
               
                 RMB Signal 
                 on Master 
                 on Vice-Master 
                 on Peer 
                 Description 
               
               
                   
               
             
             
               
                 MasterOK 
                 Output 
                 Input 
                 Input 
                 Indicates the master is functioning 
               
               
                   
                   
                   
                   
                 properly from its perspective. Its 
               
               
                   
                   
                   
                   
                 Watchdog and other checks are 
               
               
                   
                   
                   
                   
                 normal. When this signal is 
               
               
                   
                   
                   
                   
                 dropped, ViceMaster takes over 
               
               
                   
                   
                   
                   
                 driving the Output signals. 
               
               
                 ViceMaster 
                 Input 
                 Output 
                 Input 
                 Indicates the vise-master is 
               
               
                 OK 
                   
                   
                   
                 functioning properly from its 
               
               
                   
                   
                   
                   
                 perspective. 
               
               
                 VMVoteOut 
                 Input 
                 Output 
                 Input 
                 Vice-Master asserts this line and 
               
               
                   
                   
                   
                   
                 requests that the current master be 
               
               
                   
                   
                   
                   
                 voted out of office. This is done 
               
               
                   
                   
                   
                   
                 when errors are seen by the Vice- 
               
               
                   
                   
                   
                   
                 Master in the cross channel data. 
               
               
                 PeerVoteOut 
                 Input 
                 Input 
                 Output 
                 Any non-Master or non-Vice- 
               
               
                   
                   
                   
                   
                 Master asserts this line and 
               
               
                   
                   
                   
                   
                 requests that the current master is 
               
               
                   
                   
                   
                   
                 voted out of office. This is done 
               
               
                   
                   
                   
                   
                 when errors are seen by the Peer 
               
               
                   
                   
                   
                   
                 in the cross channel data. 
               
               
                 Start 
                 Output 
                 Input 
                 Input 
                 Controlled by the Master, this 
               
               
                   
                   
                   
                   
                 signal forces the previously 
               
               
                   
                   
                   
                   
                 requested action to begin on each 
               
               
                   
                   
                   
                   
                 of the functional peers. It is tied 
               
               
                   
                   
                   
                   
                 to an ISR on the power module. 
               
               
                 Line Loss 
                 Output 
                 Input 
                 Input 
                 Signals an input line has been lost 
               
               
                   
                   
                   
                   
                 and the ups should go on inverter. 
               
               
                 Verror 
                 Analog 
                 Analog input 
                 Analog 
                 This is the output voltage target, 
               
               
                   
                 output 
                 (output on 
                 input 
                 which is set by the master and 
               
               
                   
                   
                 master failure) 
                   
                 used by all power modules in the 
               
               
                   
                   
                   
                   
                 system. One for each line phase is 
               
               
                   
                   
                   
                   
                 needed. 
               
               
                 CAN 
                 Both 
                 Both 
                 Both 
                 All modules may communicate on 
               
               
                   
                   
                   
                   
                 the CAN bus. 
               
               
                   
               
             
          
         
       
     
     
       
         
               
               
               
             
           
               
                 TABLE 2 
               
               
                   
               
               
                 Role/Enable 
                   
                   
               
               
                 Group 
                 Drive Enable Equation 
                 Description 
               
               
                   
               
             
             
               
                 Master Output 
                 = Ready &amp; WDOK &amp; 
                 Acting as Master 
               
               
                 Enable (M_out- 
                 !VoteOut &amp; Master &amp; 
                 conditionally drives the 
               
               
                 en) 
                 !Vmaster 
                 Master&#39;s Output lines. All 
               
               
                   
                   
                 other lines are inputs to 
               
               
                   
                   
                 CPU. 
               
               
                 Vice-Master 
                 = Ready &amp; WDOK &amp; 
                 Acting as ViceMaster - 
               
               
                 Output Enable 
                 VoteOut &amp; !Master and  
                 conditionally drives the 
               
               
                 (VM_out_en) 
                 Vmaster 
                 ViceMaster&#39;s Output lines. 
               
               
                   
                   
                 All other lines are inputs to 
               
               
                   
                   
                 CPU. 
               
               
                 Peer Output 
                 = Ready &amp; WDOK &amp; 
                 Acting as Peer - may drive 
               
               
                 Enable 
                 !Master and !Vmaster 
                 the PeerVoteOut request. 
               
               
                 (P_out_en) 
                   
                 All other lines are input to 
               
               
                   
                   
                 CPU. 
               
               
                 Startup 
                 = Master &amp; Vmaster 
                 Startup condition - All 
               
               
                   
                   
                 outputs are high 
               
               
                   
                   
                 impedance. Ready latch 
               
               
                   
                   
                 state is set. On power-up 
               
               
                   
                   
                 this is the starting point. 
               
               
                   
                   
                 All lines are inputs to CPU. 
               
               
                 Verror Drive 
                 = M_out_en + 
                 Control&#39;s the driving of the 
               
               
                   
                 (VM_out_en &amp; 
                 analog Verror signal on the 
               
               
                   
                 !MasterOK) 
                 bus. 
               
               
                   
               
             
          
         
       
     
     
       
         
               
               
               
             
               
             
               
               
               
             
               
             
               
               
               
             
           
               
                 TABLE 3 
               
               
                   
               
               
                 Signal Name 
                 CPLD Assignment/Equations 
                 Description 
               
               
                   
               
             
             
               
                 LocalOK 
                 Pin on CPU side - I (input to CPLD) 
                 Local Controller software 
               
               
                   
                   
                 believes it is functioning 
               
               
                   
                   
                 properly. Output from 
               
               
                   
                   
                 CPU. 
               
               
                 Master 
                 Pin on CPU side - I 
                 Indicates if controller is in 
               
               
                   
                   
                 Master or Slave mode. 
               
               
                   
                   
                 Output from CPU. 
               
               
                 Vmaster 
                 Pin on CPU side - I 
                 Indicates if controller is in 
               
               
                   
                   
                 Vice-Master or Slave 
               
               
                   
                   
                 mode. Output from CPU. 
               
               
                 WDOK 
                 Pin on CPU side - I 
                 Local Watchdog monitor 
               
               
                   
                   
                 is OK and is being 
               
               
                   
                   
                 serviced. Output from 
               
               
                   
                   
                 CPU. 
               
               
                 RdyClock 
                 Input clock pin 
                 Input clock for Ready 
               
               
                   
                   
                 state. 
               
             
          
           
               
                 Bus Side 
               
             
          
           
               
                 MasterOK_bp 
                 = Master &amp;!Vmaster &amp; LocalOK &amp; 
                 Pin on Bus side - I/O 
               
               
                   
                 Ready OE = M_out_en 
                 (both Input/Output) Bus 
               
               
                   
                   
                 drive or input based on 
               
               
                   
                   
                 Out enable. 
               
               
                 ViceMasterOK_bp 
                 = Vmaster &amp; !Master &amp; LocalOK &amp; 
                 Pin on Bus side - I/O Bus 
               
               
                   
                 Ready OE = VM_out_en 
                 drive on input based on 
               
               
                   
                   
                 Out enable. 
               
               
                 VMVoteOut_bp 
                 = VMVoteOut_c &amp; vmaster OE = 
                 Pin on Bus side - I/O Bus 
               
               
                   
                 VM_out_en 
                 drive or input based on 
               
               
                   
                   
                 Out enable. 
               
               
                 PeerVoteOut_bp 
                 = PeerVoteOut_c &amp; peer OE = 
                 Pin on Bus side - I/O Bus 
               
               
                   
                 VM_out_en 
                 drive or input based on 
               
               
                   
                   
                 Out enable. 
               
               
                 Start_bp 
                 = Start_c OE = M_out_en 
                 Pin on Bus side - I/O Bus 
               
               
                   
                   
                 drive on input based on 
               
               
                   
                   
                 Out enable. 
               
               
                 LineLoss_bp 
                 = LineLoss_c OE = M_out_en 
                 Pin on Bus side - I/O Bus 
               
               
                   
                   
                 drive or input based on 
               
               
                   
                   
                 Out enable. 
               
               
                 Verr_Drive 
                 = M_out_en + (!MasterOK_b &amp; 
                 Pin on bus side - O 
               
               
                   
                 VM_out_en) OE = M_out_en 
                 Enables analog drive. 
               
             
          
           
               
                 CPU Side 
               
             
          
           
               
                 MasterOK_c 
                 = MasterOK_bp = M_out_en (if master) 
                 Pin on CPU side - O 
               
               
                   
                   
                 Indicates Master is ok or 
               
               
                   
                   
                 not. 
               
               
                 ViceMasterOK_c 
                 = ViceMasterOK_bp = !M_out_en (if 
                 Pin on CPU side - O 
               
               
                   
                 vice-master) 
                 Indicates ViceMaster is ok 
               
               
                   
                   
                 or not. 
               
               
                 VMVoteOut_c 
                 = VMVoteOut_bp (if master) 
                 Pin on CPU side - I/O I/O 
               
               
                   
                 OE=!VM_out_en 
                 for CPU based on Out 
               
               
                   
                   
                 enable. 
               
               
                 PeerVoteOut_c 
                 = PeerVoteOut_bp (if master) 
                 Pin on CPU side - I/O I/O 
               
               
                   
                 OE=!P_out_en 
                 for CPU based on Out 
               
               
                   
                   
                 enable. 
               
               
                 Start_c 
                 = Start_bp (if !master) OE = !M_out_en 
                 Pin on CPU side - I/O I/O 
               
               
                   
                   
                 for CPU based on Out 
               
               
                   
                   
                 enable. 
               
               
                 LineLoss_c 
                 = LineLoss_bp (if !master) OE = 
                 Pin on CPU side - I/O I/O 
               
               
                   
                 !M_out_en 
                 for CPU based on Out 
               
               
                   
                   
                 enable. 
               
               
                   
               
             
          
         
       
     
     As may further be seen in FIG. 2, the controller  24  also includes an inverter control section  50  which operates to control the inverter based upon inputs  52  from the control logic  40 , input and output power monitoring signals  26 ,  28 , as well as a voltage reference signal  54  from the analog reference line  38 . This analog voltage reference line  38  may be driven by this particular power module  12  depending on the role assumed by the control logic  40 . That is, if the control logic  40  has assumed the role of master, the CPLD  44  will enable  56  an analog voltage reference module  58  to generate and transmit the voltage reference signal on the analog voltage reference line  38 . In the case where this power module  12  has assumed the role of master, its own analog voltage reference circuitry  58  will be driving the inverter control  50 . For modules that have assumed roles other than the master, their analog voltage reference circuitry  58  will not be enabled via line  56  from the CPLD  44 , and therefore the voltage reference signal  54  will be that determined by another power module. 
     The determination of the proper role to be assumed by the control  40  of each individual power module  12  is determined initially at power up, as illustrated in flow diagrammatic form in FIG.  3 . Once the control logic powers up  60  it checks the master okay signal to determine if any other units have asserted the master role  62 . This is allowed at power up because the CPLD  44  (see FIG. 2) initially configures the control IO lines  46  in a receive mode until a role is established via the role selection lines  48 . If another unit has already asserted the role of master, as may be the case if a power module is inserted to the UPS during operation, the control logic  40  will request a role  64  from the master. This role request signal is preferably transmitted via the high-speed bi-directional communications bus  34 . Once the master has determined a role for this particular power module, it will receive this information  66  and thereafter properly configure its communications for the assigned role  68  before terminating this process  70 . As discussed above, the configuration of the communications based on the role assignment allows the bi-directional signal lines to be properly interpreted as inputs or outputs in accordance with the direction information contained above in Table 1. 
     If, however, upon power up  60  it appears that no other power module has asserted the master role  62 , this power module will begin to perform its master arbitration  72 . This master arbitration is designed to bias a race between identical power modules so that only one winner attempts to assert the master role. Known methods for this arbitration include the generation of a random number within the power module. This random number is then used as a multiplier for a fixed time delay which must time out prior to the attempted assertion of the master role. In a preferred embodiment, this master arbitration utilizes position information communicated to the control logic  40  either from physically sensing the particular slot in which it is installed, or via a location signal transmitted from a common user interface and display controller if such is provided in the UPS  10 . 
     Once the power module has completed its master arbitration  72 , it checks to determine if any other control unit has already asserted the master role  74 . If another unit has asserted the master role, this power module transmits a role request  64  to the master so that it may be assigned its proper role in the UPS configuration. Once it receives its role  66  it will then configure its communications for that role  68  before ending this process  70 . If, however, no other power module has yet asserted the master role, the control logic  40  will assert that it is the master  76 . Once this assertion has been made, the power module will then configure its communications for its role  68  before ending this process  70 . 
     While in the role of master, that power module is responsible for controlling the state configuration of the entire UPS. When a state change is required, it is the responsibility of the master to coordinate this state change so that all power modules will perform the state change transitions at nearly the identical time. This coordination through the master unit is illustrated in the simplified flow diagram of FIG.  4 . Once started  78  the master then determines that a configuration state change is needed  80 . This determination may be based on a number of different factors, including user input, operational status of the power generation and distribution, fault detection, etc. Once the new configuration is determined by the master, it then transmits the new configuration information to the vice master and the peer power modules in the UPS system  82 . This information is preferably transmitted via the high-speed communication bus  34  (see FIG. 1) to all of the other power modules. This information is used by the other power modules to prepare for the state change to minimize the system disturbance resulting therefrom. 
     Once the new configured information has been transmitted to all of the other power modules in the UPS system, the master then toggles  84  one of the control lines known as the synch line to initiate the state change. The sync line preferably initiates a high-level or non-maskable interrupt in controller  40  of each of the power modules in the UPS system, including itself. The high level interrupt is then processed and the units perform the state change  86  before ending the sequence  88 . This control structure is utilized in a preferred embodiment to allow for nearly simultaneous transition between operating modes on each of the individual power modules in the system. Controlling these state changes requires tight timing on the execution skew between these distributed power modules, and the synch line and the high-speed communication bus are used in combination to accurately synchronize actions and events between the collection of the uninterruptible power supply modules. 
     While the high-speed communication bus provides an excellent means of triggering actions, microsecond synchronization of actions on multiple different processors is difficult because of the asynchronous nature of the communications. Communication latencies and interrupt delays limit the accuracy to only a few milliseconds. While such delays may be acceptable in systems utilizing ferroresonant transformer coupled outputs, such delays may be catastrophic in a transformerless system such as is used in a preferred embodiment of the invention. By utilizing the synch line and tying it to a high level interrupt for each of the controllers, the communication variability is removed. By processing the state change as a high level interrupt, the actions across the multiple controllers for each of the multiple power modules are all started at approximately the same time within the limits of the processor&#39;s high level interrupt handling. In a preferred embodiment of the invention, this is typically a few tenths of microseconds, and is in any event, much tighter than the communications bus would allow if it were used independently. 
     In addition to providing redundancy through the role assumption scheme discussed above, the system of the invention also includes a method for removing the current master in the event that other units sense a problem that the master cannot sense for itself. As indicated in the flow diagram of FIG. 5, once this process is started  90  and the unit senses any type of problem with the master  92  that qualifies as a removable problem, the unit transmit a vote out signal  94  to the master before ending  96 . Qualifying problems are defined by the system requirements, and may include communication failures with the master, load sharing and balance problems due to a problem in the master&#39;s transmission of the V error  signal, etc. 
     Within the master, as illustrated in the flow diagram of FIG. 6, once this process is started  98 , the power module checks to determine that it is the master  100 . If the power module is not in the role of master, the process ends  102 . However, if the power module is in the role of master, it first checks to determine if it has received a vote out signal from its vice master  104 . If it has received the vice master vote out signal  104  it then checks to determine if it has received a vote out signal from at least one peer  106  in the UPS system. If the master has not received a vote out signal from two units, one being the vice master and one being at least one of the peers, the master does not relinquish its role as master. If, however, the master has received a vote out signal from both the vice master and one of the peers, it then drops its master role assertion  108  before ending this process  102 . 
     Once the master has dropped its assertion of the master role, it is then the duty of the vice master to assume those responsibilities as illustrated in the flow diagram of FIG.  7 . Upon initialization of this process  110  the power module checks to determine if its role has been assigned as the vice master  112 . If not, this process ends  114  and the power module continues as a peer in the UPS system. However, for the power module that has assumed the role of vice master, it continuously checks to make sure that the master is still okay  116 . As soon as it is determined that the master is no longer okay, the vice master then begins to transmit the V error  signal  118  so that there is no, or only a minimal disturbance in this signal to the other power inverters. The vice master then asserts the master role  120  and reconfigures its communications for the master role  122 . 
     Once this has been accomplished, the new master then chooses one of the peer modules as the vice master and transmits its new assignment to that power module  124  before ending this process  114 . The assignment of the vice master may be accomplished through any number of control strategies from the available peers in the UPS system. Preferably, the vice master simply picks one of the peers based on a location priority and transmits the new role assignment thereto. Alternatively, the new master could transmit a master arbitration signal to each of the peers in the UPS system and assign the vice master based upon the winner of that arbitration. 
     The foregoing description of various preferred embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obvious modifications or variations are possible in light of the above teachings. The embodiments discussed were chosen and described to provide the best illustration of the principles of the invention and its practical application to thereby enable one of ordinary skill in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated. All such modifications and variations are within the scope of the invention as determined by the appended claims when interpreted in accordance with the breadth to which they are fairly, legally, and equitably entitled.