Abstract:
In a multiprocessor environment, by executing cache-inhibited reads or writes to registers, a scan communication is used to rapidly access registers inside and outside a chip originating the command. Cumbersome locking of the memory location may be thus avoided. Setting of busy latches at the outset virtually eliminates the chance of collisions, and status bits are set to inform the requesting core processor that a command is done and free of error, if that is the case.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Technical Field 
         [0002]    The present application is related to co-pending application entitled “METHOD FOR INDIRECT ACCESS TO A SUPPORT INTERFACE FOR MEMORY-MAPPED RESOURCES TO REDUCE SYSTEM CONNECTIVITY FROM OUT-OF-BAND SUPPORT PROCESSOR”, Ser. No. ______, attorney docket number AUS920041003US1 and application entitled “METHOD FOR PROVIDING LOW-LEVEL HARDWARE ACCESS TO IN-BAND AND OUT-OF-BAND FIRMWARE”, Ser. No. ______, attorney docket number AUS920041004US1 all filed on even date herewith. The present invention generally relates to computer systems, and more specifically to an improved method of allowing firmware to access system status and configuration registers in a multiprocessor system. 
         [0003]    2. Description of Related Art 
         [0004]    The basic structure of a conventional symmetric multi-processor computer system  10  is shown in  FIG. 1 . Computer system  10  has one or more processing units arranged in one or more processor groups; in the depicted system, there are four processing units  12   a ,  12   b ,  12   c  and  12   d  in processor group  14 . The processing units communicate with other components of system  10  via a system or fabric bus  16 . Fabric bus  16  is connected to one or more service processors  18   a ,  18   b , a system memory device  20 , and various peripheral devices  22 . A processor bridge  24  can optionally be used to interconnect additional processor groups. System  10  may also include firmware (not shown) which stores the system&#39;s basic input/output logic, and seeks out and loads an operating system from one of the peripherals whenever the computer system is first turned on (booted). 
         [0005]    System memory device  20  (random access memory or RAM) stores program instructions and operand data used by the processing units, in a volatile (temporary) state. Peripherals  22  may be connected to fabric bus  16  via, e.g., a peripheral component interconnect (PCI) local bus using a PCI host bridge. A PCI bridge provides a low latency path through which processing units  12   a ,  12   b ,  12   c  and  12   d  may access PCI devices mapped anywhere within bus memory or I/O address spaces. PCI host bridge  22  also provides a high bandwidth path to allow the PCI devices to access RAM  20 . Such PCI devices may include a network adapter, a small computer system interface (SCSI) adapter providing interconnection to a permanent storage device (i.e., a hard disk), and an expansion bus bridge such as an industry standard architecture (ISA) expansion bus for connection to input/output (I/O) devices including a keyboard, a graphics adapter connected to a display device, and a graphical pointing device (mouse) for use with the display device. 
         [0006]    In a symmetric multi-processor (SMP) computer, all of the processing units  12   a ,  12   b ,  12   c  and  12   d  are generally identical, that is, they all use a common set or subset of instructions and protocols to operate, and generally have the same architecture. As shown with processing unit  12   a , each processing unit may include one or more processor cores  26   a ,  26   b  which carry out program instructions in order to operate the computer. An exemplary processor core includes the PowerPC™ processor marketed by International Business Machines Corp. which comprises a single integrated circuit superscalar microprocessor having various execution units, registers, buffers, memories, and other functional units, which are all formed by integrated circuitry. The processor cores may operate according to reduced instruction set computing (RISC) techniques, and may employ both pipelining and out-of-order execution of instructions to further improve the performance of the superscalar architecture. 
         [0007]    Each processor core  12   a ,  12   b  includes an on-board (L 1 ) cache (actually, separate instruction cache and data caches) implemented using high speed memory devices. Caches are commonly used to temporarily store values that might be repeatedly accessed by a processor, in order to speed up processing by avoiding the longer step of loading the values from system memory  20 . A processing unit can include another cache, such as a second level (L 2 ) cache  28  which, along with a memory controller  30 , supports both of the L 1  caches that are respectively part of cores  26   a  and  26   b . Additional cache levels may be provided, such as an L 3  cache  32  which is accessible via fabric bus  16 . Each cache level, from highest (L 1 ) to lowest (L 3 ) can successively store more information, but at a longer access penalty. For example, the on-board L 1  caches in the processor cores might have a storage capacity of 128 kilobytes of memory, L 2  cache  28  might have a storage capacity of 512 kilobytes, and L 3  cache  32  might have a storage capacity of 2 megabytes. To facilitate repair/replacement of defective processing unit components, each processing unit  12   a ,  12   b ,  12   c ,  12   d  may be constructed in the form of a replaceable circuit board, pluggable module, or similar field replaceable unit (FRU), which can be easily swapped installed in or swapped out of system  10  in a modular fashion. A command unit is a generic term that includes, among others, processor cores, and the service processors (which may also be called flexible service processor). 
         [0008]    A prior art architecture shown in published US patent application US 2004/0215929 “Cross-chip communication mechanism in distributed node topology”, discloses a predecessor external scan communications command XSCOM. Therein was shown an alternative method to provide supervisory commands to core processors through the use of a ring connected set of registers within a chip and extending between chips. The disclosed method had the advantage that commands circulated by way of the old XSCOM didn&#39;t have to interrupt fabric bus transported commands. 
         [0009]    However, the old XSCOM method, because it used a relatively low bandwidth medium, required significant overhead to place locks on the so called ‘mode ring’. Such locks could be placed by the service processor, or by individual threads operating on the cores of a chip. 
         [0010]    In addition, the old XSCOM method had the drawback that once a XSCOM operation was initiated, the thread that owned that operation had to poll for status in order to receive the result. As can be seen, the old XSCOM required several instruction cycles to complete, which further required that the processor core had to gate off interrupts to avoid being interrupted during these cycles. 
         [0011]    The architecture described above typically used a logical partition debugger and maintenance program known as the ‘hypervisor’, that allows, among other things, multiple computing environments on the same platform. The Hypervisor in a typical configuration would operate on one processor core. In order to coordinate functions on other chips, i.e. inter-chip commands, the Hypervisor needed to activate functions on a third command unit. This activity had the attendant drawback that in order to do this, many machine language instructions were needed, including a software lock on the resources of the target command unit. This meant that collisions of multiple processor cores needed to be avoided during the many machine instructions, as well as preventing the one or more service processors from trying to access the same facility at the same time. One such inter-chip command of the earlier design is the            
       SUMMARY OF THE INVENTION 
       [0012]    In this manner we provide, in a multiprocessor environment, improved communication between processors and from the processor to configuration registers on other chips in the system via external scan communications (XSCOM). Following prior art, a subset of control, mode, and status registers on each chip are normally made available to the service processor through a special scan communications (SCOM, PSCOM) path in the hardware. This invention also makes available that subset of registers to firmware running on the processors via the XSCOM mechanism. 
         [0013]    An embodiment also may provide a way to initiate an XSCOM command with an architected instruction that already exists in the IBM Power RISC architecture without adding new instructions to the ISA, preferably using a cache-inhibited (CI) load and a CI store. 
         [0014]    An embodiment is shown to address the XSCOM command to the appropriate chip in the system using a real memory address space assigned for use by XSCOM. This allows the firmware to use CI loads and CI stores to access these SCOM registers through a memory-mapped address space, greatly simplifying the programming model. Prior art used multiple commands to multiple registers to perform one operation; this invention allows one instruction to perform one XSCOM operation. 
         [0015]    We also provide a means to perform XSCOM operations without the need to poll for status information on the success or failure of the XSCOM operation itself. 
         [0016]    In addition we provide an error detection and reporting mechanism for XSCOM in lieu of polling for status. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS  
         [0017]    The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein: 
           [0018]      FIG. 1  is a prior art symmetric multiprocessor having an external scan communication macro according to an embodiment of the invention; 
           [0019]      FIG. 2  is a block diagram of an embodiment of the invention according to an embodiment of the invention; 
           [0020]      FIG. 3  is an address map of cache-inhibited read or write commands of an embodiment, wherein relative locations of bit-fields are shown according to an embodiment of the invention; and 
           [0021]      FIG. 4  is a flow diagram showing the steps that an embodiment may perform while handling a cache-inhibited command according to an embodiment of the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
       [0022]      FIG. 2  shows an embodiment capable of transporting an atomic XSCOM command along the fabric bus  207 . PSCOM (Parallel SCOM)  221  is directly connected to various “SCOM satellites” located in the clock-controlled components via serial address and data buses. These satellites are connected to control, mode, and status registers, referred to as SCOM registers. In the  FIG. 2  embodiment, there are six SCOM satellites shown, first core-a satellite  203   a , a second core-a satellite  203   b , first core-b satellite  206   a , and a second core-b satellite  206   b . Display/Alter satellite  213  is located in the circuits of Display/Alter device  211 , while PSCOM satellite  223  is located within the PSCOM unit  221 . 
         [0023]    Reads and writes from a SCOM register are possible where two or more SCOM register units or satellites are connected in a ring or hub topology, wherein processing-unit-to-processing-unit communication may be accomplished from a “next chip” circuit of one SCOM satellite to a “previous chip” circuit of another SCOM satellite. For simplicity, the lines interconnecting each of the SCOM satellites between each other have been omitted from  FIG. 2 . 
         [0024]    An external scan communications (XSCOM) command may be initiated by software running on either Core A or Core B by executing either a cache-inhibited load command or a cache-inhibited store command to a predefined address space. Each SCOM register in the satellite, e.g.  203   a , has a corresponding real-memory address. In the case of a cache-inhibited store, 8 bytes of data may be stored to any SCOM register in the system. Alternatively, a cache-inhibited load may retrieve 8 bytes of data from any SCOM register. There is at least one SCOM register for each SCOM satellite located on chip  200 . 
         [0025]    In the preferred embodiment, the real memory address outside of the core consists of 48 bits. An exemplary memory-map to address any SCOM register in the system with an XSCOM command is shown in  FIG. 3 . Base address  301  of the XSCOM address range is programmable and is stored in a SCOM accessible register in Display/Alter unit  211 , and may be referred to as a base address range (BAR). Processor chip identification number  303  that either holds the SCOM register being addressed or is connected to a support chip that holds the SCOM register being addressed, may appear in the “processor chip select” field  303  and use bits  26  through  31 . 
         [0026]      FIG. 3  further shows, “support chip fsi port select”  305 , bits  32  through  35  of the XSCOM address map, determines whether the SCOM register being addressed is on processor chip  200  or on support chip  250  connected to the processor chip. If the SCOM register being addressed is on support chip  250 , of  FIG. 2 , “support chip fsi port select” field  305  will identify which among several FSI ports  242  that the support chip with the SCOM register being addressed is attached. Such support chips may provide functions such as L 3  cache. If the SCOM register being addressed is on processor chip  300 , the hex F pattern may be used to indicate the chip to access is chip  300  itself. This field is used by the external scan communication or XSCOM unit  231  to determine which direction to route the XSCOM command, either to PSCOM unit  221  on processor chip  200  or to Master field replaceable unit (FRU) support interface (FSI) or MFSI unit  241  which connects the processor to the PSCOM unit on support chips  250 . Details of the MFSI or master FSI operation are in “METHOD FOR INDIRECT ACCESS TO A SUPPORT INTERFACE FOR MEMORY-MAPPED RESOURCES TO REDUCE SYSTEM CONNECTIVITY FROM OUT-OF-BAND SUPPORT PROCESSOR”, Ser. No. ______, attorney docket number AUS920041003US1, which sometimes calls the MFSI a FSI master, and is herein incorporated by reference. 
         [0027]    SCOM address  307  of the register to be accessed may be on bits ( 36 : 55 ). SCOM address portion of the memory-mapped XSCOM address is used by the PSCOM units either on processor chip  200  or on support chips  250  to address which one of the many SCOM registers attached to one of the many SCOM satellites  203   a ,  203   b ,  2061 ,  206   b , etc to access. 
         [0028]    Bit  56  signals the presence of SCOM command  309 , in the case of a ‘0’, and a non-SCOM command  309  using a ‘1’. SCOM command  309  provides the ability to perform more than just SCOM read and write operations using an XSCOM command. For the purpose of this embodiment, this bit is always ‘0’. 
         [0029]    The data portion of interconnect fabric bus  207  that resides on chip physically consists of 2 sets of 4 interleave buses that are each 8 bytes wide. “Interleave select”  311 , bits  57  to  58  of the XSCOM address map, are used to select which interleave bus contains the 8 bytes of XSCOM data. In this embodiment the Display/Alter unit is only connected to interleave 0 of each set of interleaves on the fabric data bus. Therefore the “interleave select” needs to be set to “00” for the Display/Alter data to see the XSCOM data. 
         [0030]    Multiple chips may be connected to a single FSI port in cascade. The position of a support chip attached to the FSI port selected in bits  32 : 35  is held in “cascaded support chip position” field  313  ( 59 : 60 ). Collectively, the “support chip fsi port select” field and the “cascaded support chip position” field may form a support chip address, i.e. an address that uniquely identifies a support chip connected to a processor selected with “processor chip select”  303  within the symmetric multiprocessor system. In cases where more than one chip is attached, these bits are used to select a specific chip. If only one chip is attached to the selected FSI port, then these two bits may be set to ‘00’ to signal that is the case. 
         [0031]    Lastly, a real memory address addresses 1 byte of data. Since an XSCOM command consists of 8 bytes of data, it is desired to have the 8 bytes of data aligned on a double word boundary. This double-word alignment may be accomplished by setting “double-word align”  315 , bits  61  to  63 , to ‘000’. 
         [0032]    The flowchart in  FIG. 4  shows the progression of an XSCOM read operation through the system. The software initiates the command by performing a cache-inhibited (CI) load from an XSCOM address as defined above and in  FIG. 3 . The core forwards the CI load request along with the address, having a CI base address register (CI BAR) and a CI identifier (CI ID) to the Non-Cacheable Unit (NCU)  209 , and the NCU then places the command on the fabric bus  207  of  FIG. 2 . 
         [0033]    Again, referring to  FIG. 4 , each Display Alter (DA) unit  211  of  FIG. 2  snoops the CI load request (step  401 ) and compares the address&#39; CI BAR to see if bits  16  to  25  match (step  403 ) the DA&#39;s the preprogrammed chip base address register (BAR) allowable range, thus indicating that the command is an XSCOM command. Detecting that the CI BAR is in the allowable range is a step of determining that an XSCOM command is on the fabric bus. If the BAR matches the allowable range then the DA unit compares bits  26  to  31  of the address&#39; processor chip select or CI ID to its own unique processor chip identification number or Chip ID. If these match then the DA unit will check to see if there is already a command in progress on the chip. This is done by checking a busy latch that is set when a command is accepted. If the busy latch is set, meaning that a command is already in progress then the DA unit will send a retry response back to the initiating NCU, and the initiating NCU will retry the command. This will continue until the DA unit completes the command that is in progress and clears the busy latch. If the busy latch is not set then the DA unit sends an acknowledgement response back to the initiating NCU  209  of  FIG. 2  to indicate that the command has been accepted. The DA unit will then set the busy latch (step  405 ) and forward (step  409 ) bits  32  to  56  and bits  59  to  60  of the address and an indication that it is a read command to the XSCOM unit  231  of  FIG. 2 . 
         [0034]    The XSCOM unit  231  will look at the “support chip FSI port select”  305 , bits  32  to  35  of the XSCOM address to determine if the request is addressing a SCOM register on the processor chip or a SCOM register on one of the support chips attached to the master FSI (MFSI) port. If the “support chip FSI port select” is a PSCOM directive, which may be represented by 0xF (so long as the bit pattern doesn&#39;t match an address of a support chip), then the read will be directed by XSCOM to the Parallel Scan Communication (PSCOM) unit for further processing and retrieval of data from the SCOM register addressed with the “SCOM address”. If the “support chip FSI port select” selects a valid MFSI port (step  413 ) then the “read” will be directed by XSCOM to the MFSI unit for further processing and retrieval of data from the support chip and SCOM register addressed (step  453 ) with the “support chip FSI port select”, “SCOM address”, and “cascaded support chip position”. The “command type” is also forwarded to the MFSI (step  451 ) to allow other types of read commands (non-SCOM) to be performed on a support chip connected to an MFSI port. If the “support chip FSI port select” is not the PSCOM directive (0xF) or a valid MFSI port, then an “address not acknowledged” error occurs, and the read attempt is aborted. Otherwise, read data is obtained (step  455 ). 
         [0035]    XSCOM may then wait for a done indication (step  417 ) to be received from PSCOM  221  or MFSI  241  of  FIG. 2  whichever was addressed. Once the done indication and data are received from either PSCOM  221  or MFSI  241 , XSCOM  231  may pulse a corresponding done line to the DA  211  along with transmitting the read data. 
         [0036]    The DA finishes by both clearing (step  421 ) the busy latch set earlier, and forwarding (step  419 ) the data back to the initiating NCU  209 . 
         [0037]    If an error occurs such as an “address not acknowledged” as described earlier, or any other detectable error, the XSCOM read command will be aborted. The initiating software is still be expecting data back from the CI load operation, therefore a fail signature of all 1&#39;s data is returned to the initiating software to indicate an error has possibly occurred. At the same time, a separate transaction is initiated on the fabric to send error and done status back to the processor chip that initiated the XSCOM read request. This data is propagated through the system back to the initiating core and will set an interrupt, or error status bit, in the core. When the software requesting the read sees the all 1&#39;s data returned, it is required to check for the interrupt or error status bit to see if the all 1&#39;s data was an error or valid data. The separate transaction to report status back to the initiating processor chip is sent every time that an XSCOM command is completed successfully or aborted. This status information may be provided in a separate software accessible register in each of the cores (not pictured) and is used to alert software used to monitor system integrity and may be known as the Hypervisor Maintenance Error Register (HMER). Bits set may be, in the case of an XSCOM command done, a done bit or bit  15 , and in the case of XSCOM command fail bit or bit  14 . 
         [0038]    The flowchart in  FIG. 4  shows the progression of an XSCOM write operation through the system. The software initiates the command by performing a cache-inhibited (CI) store to an XSCOM address as defined above and in  FIG. 3 . The core forwards the CI store request along with the address, having a CI base address register (CI BAR) and a CI identifier (CI ID), and data to the Non-Cacheable Unit (NCU)  209 , and the NCU then places the command request along with the address on the fabric bus  207 . 
         [0039]    The Display Alter (DA) unit  211  snoops (step  401 ) the CI store request and compares the address&#39; CI BAR to see if bits  16  to  25  of the address match (step  403 ) the DA&#39;s  211  preprogrammed chip base address register (BAR) which indicates that the command is an XSCOM command. If the BAR matches then the DA unit compares bits  26  to  31  of the address&#39; processor chip select or CI ID to its own unique processor chip identification number or Chip ID. If these match then DA unit  211  will check to see if there is already a command in progress on the chip by checking a busy latch. If the busy latch is set, meaning that a command is already in progress then the DA unit will send a retry response back to the initiating NCU, and the NCU will retry the command. This will continue until the DA unit completes the command that is in progress and clears the busy latch. If the busy latch is not set then the DA unit sends an acknowledgement response back to the initiating in accordance with the preferred embodiment of the present invention NCU  209  to indicate that the command has been accepted. The DA unit will then set the busy latch (step  405 ) and wait for the data to be sent from the initiating NCU. 
         [0040]    When the initiating NCU sees the “command acknowledged” response from the destination DA unit, it will place the write data on the fabric, along with a data tag to route the data to the correct DA unit on the correct chip. Once the DA unit has received the write data, it will forward (step  409 ) bits  32  to  56  ( 305 ,  307 ,  309 ) and bits  59  to  60  ( 313  of  FIG. 3 ) of the address, along with the write data and an indication (step  411 ) that it is a write command to XSCOM unit  231  of  FIG. 2 . 
         [0041]    XSCOM unit  231  will look at the “support chip FSI port select” bits  32  to  35  of XSCOM address to determine if the request is addressing a SCOM register on the processor chip or a SCOM register on one of the support chips attached to the master FSI (MFSI) port  241 . If the “support chip FSI port select” is a PSCOM directive, which may be represented by 0xF, (step  413 ) then the write will be directed by XSCOM to the Parallel Scan Communication (PSCOM) unit (step  415 ) for further processing and storage of the write data to the SCOM register addressed with the “SCOM address”  307 . If the “support chip FSI port select” selects a valid MFSI port then the write will be directed by XSCOM  231  to the MFSI unit  241  for further processing. Storage of the write data to the support chip and SCOM register includes sending the addressing information including support chip address (step  433  and “SCOM address”  307 . The “command type”  309  is also forwarded to the MFSI to allow other types of write commands (non-scom) to be performed on a support chip connected to an MFSI port. Write data, and when available, the “write data valid” signal is forwarded (step  435  to the MFSI. If the “support chip FSI port select” is not 0xF or a valid MFSI port, then an “address not acknowledged” error occurs, and the write attempt is aborted. 
         [0042]    XSCOM may then wait for a done indication (step  417 ) to be returned from PSCOM  221  or MFSI  241  whichever was addressed. Once the done indication is received from either PSCOM  221  or MFSI  241  of  FIG. 2 , or an error has occurred causing the write to be aborted, XSCOM  231  may pulse a corresponding done line to the DA  211 . 
         [0043]    DA  211  finishes by clearing (step  421 ) the busy latch set earlier. 
         [0044]    If an error occurs such as an “address not acknowledged” as described earlier, or any other detectable error, the XSCOM write command will be aborted. A separate transaction is initiated on the fabric to send error and done status back to the processor chip that initiated the XSCOM write request. This status data is propagated through the system back to the initiating core and will set an interrupt, or error status bit, in the core. The software can either check the status bit or wait to be interrupted to see if the write command complete successfully. The separate transaction to report status back to the initiating processor chip is sent every time that an XSCOM command is completed successfully or aborted. This status information may be provided in a separate software accessible register in each of the cores (not pictured) and is used to alert software used to monitor system integrity and may be known as the Hypervisor Maintenance Error Register (HMER). Bits set may be, in the case of an XSCOM command done, a done bit or bit  15 , and in the case of XSCOM command fail bit or bit  14 . 
         [0045]    It is important to note that while the present invention has been described in the context of a fully functioning data processing system, those of ordinary skill in the art will appreciate that the processes of the present invention are capable of being distributed in the form of a computer readable medium of instructions and a variety of forms and that the present invention applies equally regardless of the particular type of signal bearing media actually used to carry out the distribution. Examples of computer readable media include recordable-type media, such as a floppy disk, a hard disk drive, a RAM, CD-ROMS, DVD-ROMS, and transmission-type media, such as digital and analog communications links, wired or wireless communications links using transmission forms, such as, for example, radio frequency and light wave transmissions. The computer readable media may take the form of coded formats that are decoded for actual use in a particular data processing system. 
         [0046]    The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.