Broadcast of shared I/O fabric error messages in a multi-host environment to all affected root nodes

A method, mechanism and computer usable medium is provided for distributing I/O fabric errors to the appropriate root nodes in a multi-root environment. The case where the I/O fabric is attached to more than one root node and where each root can potentially share with the other roots the I/O adapter (IOA) resources which are attached to the I/O is addressed. Additionally, a method, mechanism and computer usable medium is provided by which errors detected in an I/O fabric may be routed to all root nodes which may be affected by the error, while not being reported to the root nodes that will not be affected by those errors. In particular, distributed computing system which uses the PCI Express protocol to communicate over the I/O fabric is addressed.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to communication between a host computer and an input/output (I/O) adapter through an I/O fabric. More specifically, the present invention addresses the case where the I/O fabric is attached to more than one root node and where each root can potentially share with the other roots the I/O adapter (IOA) resources which are attached to the I/O.

2. Description of the Related Art

Multi-root configurations which share I/O fabrics have not been addressed adequately in the past, and errors detected in an I/O fabric will generally bring down all the systems that may be using that fabric, due to the unknown of which I/O devices are affected and which root nodes are using that I/O.

Thus, it would be advantageous to provide a mechanism for defining to the I/O fabric which I/O devices are affected by which errors, and also to which root nodes those I/O devices are assigned.

SUMMARY OF THE INVENTION

The present invention provides a method, apparatus, and computer usable medium for distributing input/output fabric errors to the appropriate root nodes in a multi-root environment. The present invention addresses the case where the input/output fabric is attached to more than one root node and where each root can potentially share with the other roots the input/output adapter resources which are attached to the input/output fabric. Additionally, the present invention provides a mechanism and method by which errors detected in an input/output fabric can be routed to all root nodes which may be affected by the error, while not being reported to the root nodes that will not be affected by those errors. In particular, the present invention specifically addresses the distributed computing system which uses the PCI Express protocol to communicate over the input/output fabric.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The aspects of the present invention provide for distributing input/output fabric errors to the appropriate root nodes in a multi-root environment. Addressed is the case where the input/output fabric is attached to more than one root node and where each root can potentially share with the other roots the input/output adapter resources which are attached to the input/output. Additionally, a mechanism is provided by which errors detected in an input/output fabric can be routed to all root nodes which may be affected by the error, while not being reported to the root nodes that will not be affected by those errors. In particular, a distributed computing system is addressed which uses the PCI Express protocol to communicate over the input/output fabric.

With reference now to the figures and in particular with reference toFIG. 1, a diagram of a distributed computing system is illustrated in accordance with an illustrative embodiment of the present invention. Distributed computer system100represented inFIG. 1takes the form of one or more root complexes108,118,128,138, and139, attached to I/O fabric144through I/O links110,120,130,142, and143, and to memory controllers104,114,124, and134of root nodes (RNs)160,161,162, and163. I/O fabric144is attached to I/O adapters145,146,147,148,149, and150through links151,152,153,154,155,156,157, and158. I/O adapters145,146,147,148,149, and150may be single function I/O adapters such as in145,146, and149, or multiple function I/O adapters such as in147,148, and150. Further, I/O adapters145,146,147,148,149, and150may be connected to I/O fabric144via single links as in145,146,147, and148or with multiple links for redundancy as in149and150.

Root complexes108,118,128,138, and139are part of root nodes160,161,162, and163. More than one root complex per root node may be present as in root node163. In addition to the root complexes, each root node consists of one or more central processing units (CPUs)101,102,111,112,121,122,131, and132, memory103,113,123, and133, memory controller104,114,124, and134which connects CPUs101,102,111,112,121,122,131, and132, memory103,113,123, and133, and I/O root complexes108,118,128,138, and139and performs such functions as handling the coherency traffic for the memory.

Root nodes160and161may be connected together at connection159through their memory controllers104and114to form one coherency domain and which may act as a single symmetric multi-processing (SMP) system, or may be independent nodes with separate coherency domains as in root nodes162and163.

Configuration manager164may be attached separately to I/O fabric144or may be part of one or more of the root nodes160,161,162, and163. Configuration manager164configures the shared resources of I/O fabric144and assigns resources to root nodes160,161,162, and163.

Distributed computing system100may be implemented using various commercially available computer systems. For example, distributed computing system100may be implemented using an IBM eServer™ iSeries® Model 840 system available from International Business Machines Corporation. Such a system may support logical partitioning using an OS/400 operating system, which is also available from International Business Machines Corporation.

With reference now toFIG. 2, a block diagram of an exemplary logical partitioned platform is depicted in which the present invention may be implemented. The hardware in logical partitioned platform200may be implemented as, for example, distributed computing system100inFIG. 1. Logical partitioned platform200includes partitioned hardware230, operating systems202,204,206, and208, and partition management firmware210. Operating systems202,204,206, and208may be multiple copies of a single operating system or multiple heterogeneous operating systems simultaneously run on logical partitioned platform200. These operating systems may be implemented using OS/400, which are designed to interface with a partition management firmware, such as Hypervisor. OS/400 is used only as an example in these illustrative embodiments. Other types of operating systems, such as AIX® and Linux, may also be used depending on the particular implementation.

Operating systems202,204,206, and208are located in partitions203,205,207, and209. Hypervisor software is an example of software that may be used to implement partition management firmware210and is available from International Business Machines Corporation. Firmware is “software” stored in a memory chip that holds its content without electrical power, such as, for example, read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and nonvolatile random access memory (NVRAM).

Additionally, partitions203,205,207, and209also include partition firmware211,213,215, and217. Partition firmware211,213,215, and217may be implemented using initial boot strap code, IEEE-1275 Standard Open Firmware, and runtime abstraction software (RTAS), which is available from International Business Machines Corporation. When partitions203,205,207, and209are instantiated, a copy of boot strap code is loaded onto partitions203,205,207, and209by platform firmware210. Thereafter, control is transferred to the boot strap code with the boot strap code then loading the open firmware and runtime abstraction software. The processors associated or assigned to partitions203,205,207, and209are then dispatched to the partition's memory to execute partition firmware211,213,215, and217.

Partitioned hardware230includes a plurality of processors232,234,236, and238, a plurality of system memory units240,242,244, and246, a plurality of I/O adapters248,250,252,254,256,258,260, and262, storage unit270, and non-volatile random access memory storage298. Each of the processors232;234,236, and238, memory units240,242,244, and246, non-volatile random access memory storage298, and I/O adapters248,250,252,254,256,258,260, and262, or parts thereof, may be assigned to one of multiple partitions within logical partitioned platform200, each of which corresponds to one of operating systems202,204,206, and208.

Partition management firmware210performs a number of functions and services for partitions203,205,207, and209to create and enforce the partitioning of logical partitioned platform200. Partition management firmware210is a firmware implemented virtual machine identical to the underlying hardware. Thus, partition management firmware210allows the simultaneous execution of independent operating system images202,204,206, and208by virtualizing the hardware resources of logical partitioned platform200.

Service processor290may be used to provide various services, such as processing of platform errors in partitions203,205,207, and209. These services also may act as a service agent to report errors back to a vendor, such as International Business Machines Corporation. Operations of partitions203,205,207, and209may be controlled through a hardware management console, such as hardware management console280. Hardware management console280is a separate distributed computing system from which a system administrator may perform various functions including reallocation of resources to different partitions. Operations which may be controlled include things like the configuration of the partition relative to the components which are assigned to the partition, whether the partition is running or not.

In a logical partitioning (LPAR) environment, it is not permissible for resources or programs in one partition to affect operations in another partition. Furthermore, to be useful, the assignment of resources needs to be fine-grained. For example, it is often not acceptable to assign all I/O adapters under a particular PCI Host Bridge (PHB) to the same partition, as that will restrict configurability of the system, including the ability to dynamically move resources between partitions.

Accordingly, some functionality is needed in the bridges that connect I/O adapters to the I/O bus so as to be able to assign resources, such as individual I/O adapters or parts of I/O adapters to separate partitions; and, at the same time, prevent the assigned resources from affecting other partitions such as by obtaining access to resources of the other partitions.

Turning now toFIG. 3, a general layout of a message request packet is depicted in accordance with an illustrative embodiment of the present invention. In message request package300the key fields are the requestor ID301and the message code302. Of particular interest in message request package300is message code302which may represent an error that has been identified by an entity represented by requestor ID301. Requestor ID301provides an indication of the detector of the error, but not necessarily the entities that might be affected by the error. Accordingly, a way is needed to correlate missing information.

FIG. 4depicts a method where error correlation and routing may be performed in accordance with an illustrative embodiment of the present invention. I/O fabric401consists of root ports402,403,404,405, and406and secondary ports407,408,409,410,411,412,413, and414. Incoming transaction418contains an error that is detected by error detection logic415and the control logic for the error detection then generates a message request packet, such as message request package300ofFIG. 3, into which it puts requestor ID416. Requestor ID416is setup by the configuration code at fabric initialization time. As an exemplary aspect of the present invention, requestor ID416may be the bus number, device number, and function number of the device in the case of PCI Express. The message request packet is then passed through I/O fabric401at connection419until it reaches routing logic422, which is located in I/O fabric401at a place which has access to all root ports402,403,404,405, and406. At this point requestor ID416in the message request packet is used to access error routing table417and the information in routing table417is used to generate one error packet420and421per root port403and405that is affected. Routing table417may be any type of data structure where information is stored.

FIG. 5depicts a configuration that necessitates a split of a routing table into multiple cascaded routing tables in accordance with an illustrative embodiment of the present invention. In this case I/O fabric501has more than one switch or bridge515and517which interfaces to the root ports502,503,504,505, and506. If there is more than one switch or bridge, then a routing table, such as routing table417ofFIG. 4, needs to be split into routing tables516and518. Connection between routing tables516and518is via intermediate link526.

Additionally, there may be secondary bridge or switch519which may contain routing table520. In this configuration, error detector521generates an error packet522with the error detector's ID in the requestor ID. Routing table520uses this requestor ID in error packet522to look up the routing which then routes error packet522as shown in connection523. Likewise, routing table518determines that the correct routing of error packet522is to root ports505and506via connections524and525, and to switch or bridge515. When error packet522is received at switch or bridge515via intermediate link526, routing table516determines that error packet522should be routed to root port503via connection527.

FIG. 6shows an exemplary layout of a routing table entry which might be found in routing table, such as routing table417ofFIG. 4, in accordance with an illustrative embodiment of the present invention. Routing table entry600consists of requestor ID601, which may be one possible requester ID on the message request packet and corresponds to requestor ID416ofFIG. 4and is detected in error detector415ofFIG. 4or error detector521ofFIG. 5. Also in routing table entry600is root port bit array602of which each bit corresponds to a possible root port to which the error might need to be routed, and also intermediate port bit array603of which each bit corresponds to a possible intermediate port to which the error might need to be routed.

FIG. 7is a flowchart depicting the operation through one level of the routing mechanism in accordance with an illustrative embodiment of the present invention. As the operation begins, an error message is received by the routing mechanism (step702). The requestor ID in the error message is then searched for in the routing table (step704). Those knowledgeable in the art will recognize that the search of the correct entry in the routing table may be performed in any number of ways. Additionally, the routing table may be any type data structure where information is stored. For example, a content addressable memory, a scan of the table for a value of the requestor ID field in the table equal to the requestor ID in the error message, the use of the requestor ID in the error message as an index into the routing table, and so on. Upon finding the correct requestor ID entry, the associated root port bit array is checked for any bit that is set (step706). If any bit is set in the root port bit array, then for each bit set, an error message is generated with the requestor ID in the original message, a determination is made of which port or ports is associated with the error message by searching the routing table, and each error message is routed to the root port or ports corresponding to the position of the bit in the root port bit array (step708).

Next, the intermediate port bit array is checked for any bit that is set (step710). Step706also proceeds to step710if there are no bits set in the root port bit array. If any bit is set in the intermediate root port bit array, then for each bit set, an error message is generated with the requestor ID in the original message, a determination is made of which port or ports is associated with the error message by searching the routing table, and each error message is routed to the intermediate port or ports corresponding to the position of the bit in the intermediate port bit array (step712) with the operation ending thereafter. Step710also proceeds to the operation termination if there are no bits set in the intermediate port bit array.

FIG. 8depicts a high-level flowchart for the routing table build process in accordance with an illustrative embodiment of the present invention. As the operation begins the I/O fabric configuration code probes or “walks” the I/O fabric, remembering where the routing tables are in relationship to the I/O fabric requestor IDs and endpoint requestor IDs (step802). As an example, the requestor ID for PCI Express may be bus number, device number, and function number. That is, a requester ID “tree” is remembered by the configuration software. This tree determines which errors will potentially affect which other IDs. That is, in processing an error for a given ID the software needs to assume that this error could affect all other IDs in the tree below that ID, and thus, if an error occurs for an ID that affects a plurality of IDs, and those plurality of IDs are assigned to a plurality of root nodes, then the mechanism described in this invention needs to replicate those error messages and deliver them to all affected root nodes.

As the I/O fabric configuration code probes or “walks” the I/O fabric is completed, the I/O fabric is configured and a plurality of endpoints are assigned to a plurality of root nodes (step804). Assignment means that the root node is in control of the endpoint, including any error recovery. The method used to determine assignment is beyond the scope of this invention.

Next, the routing tables are built based on which errors will affect which endpoint requestor IDs and which endpoint requester IDs are assigned to which root nodes (step806). The routing tables are based on the information retained in steps802and804. The routing table will contain an entry for each of the I/O fabric requestor IDs below it in the tree and each of these entries will also specify which root nodes are affected by an error on that particular requestor ID. How the tables are accessed to place the information from this step into them is beyond the scope of this invention, but those skilled in the art will realize that the configuration mechanism used to setup the fabric could be extended to allow for such access.

Those skilled in the art will recognize that this mechanism and method replaces the PCI Express mechanism and method defined as routing the error message upward to a single root port. The mechanism and method disclosed in this invention, then, allows additional control of direction and in copying the message to route to multiple root ports that might be affected by the error.

The invention may take the form of an entirely hardware embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in hardware and software, which includes but is not limited to firmware, resident software, microcode, etc.

Furthermore, aspects of the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-readable medium can be any apparatus that can Contain or store that program for use by or in connection with the instruction execution System, apparatus, or device. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk—read only memory (CD-ROM), compact disk—read/write (CD-R/W) and DVD.