Multi root shared peripheral component interconnect express (PCIe) end point

A method of accessing a server address space of a shared PCIe end point system includes programming a primary address translation table with a server address of a server address space, setting up a direct memory access (DMA) to access a primary port memory map, the primary port memory map correlating with addresses in the primary address translation table, and re-directing the direct memory accesses to the primary port memory map to the server address space according to the primary address translation table.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to Peripheral Component Interconnect Express (PCIe) and particularly to sharing PCIe end points across servers.

The Peripheral Component Interconnect Express (PCIe) Specification allows only one host (one root) in the PCIe hierarchy. The PCI Multi Root IO Virtualization (MRIOV) Specification addresses how a PCIe end point, an example of which may be a High Bandwidth Input/Output (IO) resource, can be shared across multiple servers/hosts. Today, there is no support available for MRIOV in the entire eco system (eco system consisting of hosts, drivers, switch, and PCIe devices). There has been an attempt by the PCIe switch manufacturers to address this by adding a Non Transparent Bridge (NTB) ports to the PCIe switch. The existing solution however requires management software on all the servers and the “Shared PCIe subsystem” (PCIe subsystem consists of PCIe switch with NT ports, a local processor which acts as the root complex and the PCIe device). The management software is required because the server enumerates the PCIe end point as a Non Transparent Bridge port. The driver does not know what to do with the NTB. The device drivers for the end point behind the switch that exists on the server do not get loaded automatically. The Management Software on the server and on the shared PCIe subsystem have to communicate with each other and share the capabilities of the PCIe subsystem.

Another issue with using a PCIe switch with NT ports is that the entire memory space of the server cannot be exposed to the switch, rather, only a portion of the memory on the server is available for data transfer. In a PCI system, the end point can access the entire memory space of the server, which requires the server to move the data into the memory space mapped for the end point to access. This requires extensive changes to the existing device drivers and adds to the latency.

What is needed is a device and method for enabling sharing of a PCIe end point across multiple servers as a plug-n-play device.

SUMMARY OF THE INVENTION

Briefly, a method of accessing a server address space of a shared PCIe end point system includes programming a primary address translation table with a server address of a server address space, setting up a direct memory access (DMA) to access a primary port memory map, the primary port memory map correlating with addresses in the primary address translation table, and re-directing the direct memory accesses to the primary port memory map to the server address space according to the primary address translation table.

These and other objects and advantages of the invention will no doubt become apparent to those skilled in the art after having read the following detailed description of the various embodiments illustrated in the several figures of the drawing.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

As will be evident in the various methods and apparatus of the invention, a Peripheral Component Interconnect Express (PCIe) End Point subsystem enables sharing of PCIe end point across multiple servers as a plug-n-play device. In some embodiments of the invention, A PCIe switch's NT port has a secondary port that is configured as a PCIe end point, and further has a primary port that is configured is as a bridge. In an embodiment of the invention, an interrupt is generated to the root complex when there are any changes to the PCI configuration of the secondary port. In an embodiment of the invention, a primary address translation table is dynamically configured based on the address pointers inside the Direct Memory Access (DMA) descriptors submitted by the server.

FIG. 1shows a shared PCIe end point system100for sharing a PCIe end point116across multiple servers102_0to102—nwhere ‘n’ is an integer, using a PCIe switch108, in accordance with an embodiment of the invention. The system100is shown to include Shared PCIe End Point116, a PCIe switch108having n NT ports, and ‘n’ number of servers102_0to102—n. The secondary port112is mapped to a secondary port memory map106of a server address space104and micro-processor micro-processor address space122. The server address space104is the servers' address space and the micro-processor address space122is the micro-processor's117address space.

In the system100, the servers102are shown coupled to the PCIe switch108through non-transparent (NT) ports127and PCIe busses110. Further, the PCIe switch108is shown coupled to the shared PCIe end point116through a transparent bridge port129through a PCIe bus128. The PCIe switch108is shown to include a secondary port112and a primary port114and an exploded view of the memory map that is shown assigned to the secondary port112in the server's address space104. The server address space104is shown to include the secondary port memory map106. Similarly, an exploded view of the memory map assigned to the secondary port112—nis shown to include a secondary port memory map106—nand a server address space104—n. In fact, while not shown for the sake of brevity, each of the secondary ports112_0through112—nincludes a secondary port memory map106and a server address space104.

As will become evident shortly, in the embodiment ofFIG. 1, multiple servers share a PCIe Input/Output (IO) End Point through non-transparent (NT) ports of the switch. Further, memory mapping of the servers to the secondary configuration Base Address Resisters (BAR) is realized and the memory map of the PCIe end point subsystem to the primary configuration BAR is also realized.

The shared PCIe end point116of the shared PCIe end point system100is shown to include a micro-processor with PCIe root complex117, memory120, a multi-channel DMA118and a shared device119. The shared device119may be any type of device. The micro-processor117, memory120, multi-channel DMA118and the shared device119are coupled together using a high bandwidth system bus121, which is shown coupled to the PCIe bus128, in the embodiment ofFIG. 1. In alternate embodiments, the shared device119can also be coupled to the micro-processor117through a PCIe interface. The coupling between components in the shared PCIe end point116is merely an example.

The PCIe bus128couples the shared PCIe endpoint116to the switch transparent bridge port129.

The PCIe switch108is shown to include a secondary port112_0, and112_1to112—nand Primary Port114_0, and114_1to114—nin its non transparent bridge ports.

The server0102_0is shown coupled to the non-transparent bridge secondary port112_0of the PCIe switch108through the NT port127_0and the PCIe bus110_0, The server1is shown coupled to the non transparent bridge secondary port112_1of the PCIe switch108through the NT port127_1and the PCIe bus110_1and server n is shown coupled to the non transparent bridge secondary port112—nof the PCIe switch108through the NT port127—nand the PCIe bus110—n.

The shared PCIe end point116is shown coupled to the transparent bridge port of the PCIe switch108through the PCIe bus128.

The PCIe switch108is used as system interconnect switch for PCIe packet switching that supports simultaneous peer-to-peer traffic flows. The PCIe switch108has non-transparent bridging functionality that allows multiple hosts (servers) to be connected to the switch ports. Non-transparent bridge (NTB) ports are required when two or more PCIe domains need to communicate to each other. The main function of the NTB is to translate addresses and allow data exchange across PCIe domains, as is known to those in the industry. All of the foregoing interfaces are known in the art.

FIG. 2shows further details of the system100, in accordance with an embodiment of the invention. The system100ofFIG. 2, has an interrupt feature, in accordance with an embodiment and method of the invention.

InFIG. 2, the shared device119is shown to be shared across multiple servers over PCIe (one example is, without limitation, a shared storage device). The micro-processor117has PCIe root complex functionality and is used to enumerate the PCIe switch108. It also manages the primary and secondary port configurations of the NT ports of PCIe switch108. The micro-processor117also services the interrupt messages especially generated due to changes in the secondary configuration registers of the secondary ports112_0,112_1to112—n. The micro-processor117can be of any kind and further can be embedded. An exemplary microprocessor is the Intel-manufactured x86 type of processor. There are no limitations on the micro-processor used in the various embodiments of the invention. The memory120shown inFIG. 1is used as system memory, and it can be of any memory type. The multi-channel DMA118is used to enable transfer of data from one memory map to another memory map. All of the above is known to a person of ordinary skill in the art.

In operation, multiple servers are connected to the shared PCIe end point116through NT ports of the PCIe switch108. Each NTB port127of the PCIe switch108provides two PCI type0 configuration ports, a primary configuration port and a secondary configuration port. The NTB port which is connected to the server102_0through PCIe bus110_0has primary port114_0and secondary port112_0. Each of these ports has their PCI configuration space and registers.

The PCI configuration space corresponding to the secondary port112_0is modified to look like the configuration space of the shared PCIe end point116(such as a mass storage device) so that when server102_0, for example, enumerates, it will find a PCIe end point116(Mass Storage device as an example) instead of a NT bridge port.

Assuming that the server102_0has drivers for the PCIe end point116(as an example mass storage device—where in shared device119is a storage array), the drivers will be loaded automatically after the PCIe enumeration. The server102_0will not be aware that it is connected to a shared PCIe end point116through a switch108NTB port. The server102_0will not be aware of other servers102_1to102—nsharing the PCIe End Point116.

The following are PCI configuration registers which are part of PCIe configuration registers132_0as shown inFIG. 2are used by the PCI drivers on server102_0to identify the PCIe end point.

Vendor ID Register

Device ID Register

Revision ID Register

Class Code Register

Subsystem Vendor ID

Subsystem ID

As part of PCI device enumeration the server102_0will write to the secondary configuration registers132_0inFIG. 2. The writes to the secondary configuration registers will generate an interrupt message to the micro-processor117. The message will indicate the NT port with secondary port112_0as the source of the interrupt. The micro-processor117will read the configuration registers of secondary port112_0to detect the changes to the secondary configuration space. The server102_0can write to certain configuration registers as needed and such writes will trigger an interrupt message to the micro-processor117. This interrupt is used by the micro-processor117to take action based on the changes to PCIe configuration registers132_0.

The PCIe configuration space has Base Address Registers (BAR) which is used to map the PCIe end point resources to the systems memory map where in the root complex that enumerated the end point is part of. The system will access the PCIe end point using the memory map assigned to it. The PCIe end point will accept all the traffic with address range that fall within the programmed BAR registers. The PCIe end point can initiate a transfer to access any address space inside the system. All of the above would be known to a person of ordinary skill in the art.

The micro-processor117will initialize the secondary port112_0configuration BAR registers to request the memory segment to be mapped into server102_0address space104_0. The mechanism by which the end point requests a memory segment to be mapped to systems memory map (address space) would be known to a person of ordinary skill in the art. The micro-processor117as part of initialization will assign a end point memory map134_0at end point memory offset133_0(End point memory offset) as inFIG. 2to the PCIe end point associated with server102_0from micro-processor address space122. Further the micro-processor117programs the secondary address translation register130_0with an end point memory offset133_0, as shown inFIG. 2, to redirect all accesses from the server102_0that are directed to the secondary port memory map106_0to the end point memory map134_0. The secondary address translation register130_0is associated with BAR registers of configuration space of secondary port112_0. The size of the memory window will depend upon the functionality of the PCIe End Point. The server102_0during enumeration maps a memory segment to the secondary port112_0to access the shared PCIe end point116. InFIG. 1, only one memory segment is shown (BAR0/1) in 64-bit addressing. There could be additional, 2 more segments BAR2/3 and BAR4/5 in case of 64 bit addressing. If the addressing is 32 bit there could be 6 memory segments BAR0, BAR1, BAR2, BAR3, BAR4 and BAR5. After the configuration read of the secondary port112_0BAR registers the server102_0will assign a secondary port memory map106_0and does configuration write to the BAR registers of secondary port112_0with secondary port memory map offset105_0which is the starting address of the secondary port memory map106_0. The server102_0is now ready to use the hardware (PCIe end point) connected on its PCIe hierarchy.

After the enumeration of the end point, the end point is ready for operation. In general the end point will perform a function that would require DMA of data from/to system memory. In PCI hierarchy, the PCI device can access the entire memory space of the system of which it is part of. The device drivers for the PCI device (end point) are written with this assumption that the PCI end point can access the entire memory space.

In system100for sharing a PCIe end point116though a NT port of the PCIe switch the shared end point will not be able to access the entire memory space of the servers. The mechanism through which the shared PCIe end point can access the server memory space is through pre-configured memory window associated with the primary port (which is done by programming BAR registers inside the configuration space of primary port) and primary address translation table. In such systems primary address translation table are managed by the server. This scheme only opens up a small memory region on the server for access by the end point.

In the proposed method, the PCIe end point which is shared across multiple roots can access the entire memory map of each server.FIG. 3will be used to illustrate this method. The server page size is fixed; it is either 4 KB or 8 KB. In some systems it can be as high as 128 KB. In our illustration we assume the server page size as 4 KB. The server may create a data buffer of 64 KB using 16 4 KB pages or any other size using 4 KB pages. These pages are not physically contiguous. These pages can be anywhere in the system (server102_0) memory map. This layout of data will require a method to access the entire server address space104_0of the server102_0.

As part of initialization, the micro-processor117enumerates the primary port114of the PCIe switch's108NT ports127. As part of primary port114_0enumeration the micro-processor117assigns a primary port memory map124_0(primary port memory segment) to the primary port from micro-processor address space122to the Primary Port114_0. The size of memory segment is dependent upon the number of channels that are available on the DMA (multi-channel DMA118) and the size of the page in server102_0.

For further clarification, an example is now provided. Assuming that the DMA has 16 channels and the page size is 4 kilo bytes (KB) in the server102_0, the micro-processor117selects 64 KB primary port memory map124_0and assigns it to the primary port114_0by programming the BAR registers in configuration space of the primary port114_0with primary port memory segment0pointer123_0_0. The 64 KB memory segment can be realized as 16 4 KB primary port memory segments124_0_0,124_0_1to124_0_15, with primary port memory segment pointers123_0_0,123_0_1to123_0_15respectively. The difference between the address offsets123_0_0and123_0_1is 4 KB as the primary port memory segment124_0_0is 4 KB. There is a Primary Address Translation Table140_0associated with the primary port114_0. It is shown in theFIG. 3that the primary address translation Table has 16 primary address translation table entries140_0_0,140_0_1to140_0_15but it is just an example and it is not limited to 16. Any access by the shared PCIe end point to primary port memory segment124_0_0will be translated into the server0102_0address space104_0using the primary address translation table140, primary address translation table first entry140_0_0, similarly access by shared PCIe end point to primary port memory segment124_0_1is translated to server0102_0address space104_0using primary address translation table140_0, primary address translation table second entry140_0_1and so on.

In operation, the device drivers on the server102_0will operate as if the end point is attached to it directly. After the device driver configures the end point the end point is ready to perform the intended function. As part of its function, the end point has to perform DMA from/to server's102_0address space104_0. In the process the server102_0will create DMA descriptors for the end point to service. The server102_0will either DMA these descriptors or passes over the pointer to these descriptors to the PCIe end point116. The server102_0will use secondary port memory map106_0to DMA the descriptors or write the descriptor pointer to end point memory map134. The access by server102_0to secondary port memory map106_0will be translated by secondary address translation register130_0as inFIG. 2to end point memory map134_0. The descriptors will have source (destination) address pointers pointing to server102_0address space104_0(can be anywhere in its memory space). Source address pointer in case reading the data from server memory and destination address pointer in case writing to server memory. The device driver on server102_0will notify the end point to perform the DMA of the descriptors by handing over the pointer to the starting descriptor and number of descriptors. The device driver can access the end point memory using the secondary port memory map106_0. The shared PCIe end point116has the pointer to the descriptors inside the server102_0address space104_0and number of descriptors to be fetched. This pointer cannot be used by the shared PCIe end point to initiate the DMA because the address pointer belongs to server102_0address space104_0. The micro-processor117inside the shared end point116will program the primary address translation table140with the address pointer that points to the starting descriptor. Any access by the shared end point116to primary port memory segment124will hit the primary port114_0and will be address translated to server102_0address space104_0using primary address translation table140. The Shared PCIe end point116can read the descriptors from server102_0address space104_0. After fetching the descriptors the Shared PCIe end point116will start processing the descriptors. The source(destination) address pointers depending upon read(write) to server memory in each of these descriptors will be pointing to various locations inside the server102_0address space104_0. In this example we have assumed a 16 channel DMA118. The data buffers corresponding to these descriptors are dispersed across the server102_0address space104_0. The Shared PCIe end point116will program the primary address translation table140_0with the DMA source(destination) address. The shared PCIe end point116, will initiate the DMA with source(destination) address pointing to primary port memory segment124in micro-processor address space122. These accesses get translated to server102_0address space104_0. Any access that is address to primary port memory segment124_0_0will be translated using primary address translation table first entry140_0_0whose contents are server buffer pointer149_0_0which will point to new address in server102_0address space104_0, in this case server buffer pointer149_0_0. Resulting in, any access by the shared end point to primary port memory segment124_0_0will result in access to server buffer149_0_0in server102_0. Similarly access to primary port memory segment124_0_1will be translated to server buffer pointer149_0_1and so on. Once the DMA is complete, the shared end point116can re-use the primary address translation Table140entry to initiate another DMA.FIG. 3shows another exemplary embodiment and method of the invention using the system100. The example ofFIG. 3shows dynamically programming of the Primary Address Translation table and using Primary BAR to access any memory segment inside the server.

InFIG. 3, at146_0, there is shown a write to server memory, where the multi-channel DMA is programmed with a source as a segment of the micro-processor memory map144_0(mapped to micro-processor memory120) and a destination as a segment of the primary port memory map124_0. At148_0, there is shown a read from the server memory, where the multi-channel DMA is programmed with a source as a segment of the primary port memory map124_0and a destination as a segment of the micro-processor memory map144_0. Read or write to the primary port memory map124_0is translated using the primary address translation table140_0and the access is directed to the address space104_0of the server102_0.

By having shared end point116dynamically manage the programming of the primary address translation table140_0, the shared end point116can access the entire address space104_0of the server102_0. With multiple servers102_0,102_1to102—nconnected to the shared PCIe end point, this method can be used to access the entire address space of the servers using a small memory segment assigned to the primary port memory map124_0,124_1to124—nin the micro-processor address space122of the shared PCIe end point116.

FIG. 4shows a Shared PCIe end point116a, in accordance with yet another embodiment of invention. The Shared PCIe End Point116ais analogous to the Shared PCIe End Point116with the exception that the PCIe End Point210is the shared device119as shown inFIG. 1. The PCIe End Point210as shown inFIG. 4is coupled with the Micro-Processor117through a PCIe bus202. The memory120is coupled to the Micro-Processor using memory bus204. Multi-channel DMA118is coupled to the micro-processor using system bus206

FIG. 5shows a Shared PCIe end point116b, in accordance with yet another embodiment of invention. The Shared PCIe End Point116bis analogous to the Shared PCIe End Point116with the exception that the PCIe End Point210is the shared device119as shown inFIG. 1. The PCIe End Point210as shown inFIG. 4is coupled with the Micro-Processor117through a PCIe bus202. The memory120and multi-channel DMA118is coupled to the micro-processor using system bus206

FIG. 6shows a flow chart of the relevant steps performed for PCIe end point initialization and enumeration, in accordance with a method of the invention. InFIG. 6, at step702, the micro-processor117programs the secondary address translation registers130_0of the secondary port112_0to map any accesses to the secondary port memory map106_0to the endpoint memory map134_0. From then on, any access to the secondary port memory map106_0results in the secondary address translation register130_0to re-direct the access to the endpoint memory map134_0. The address of the location to be accessed is saved in the secondary address translation register130_0. Next, at step704, inFIG. 6, the micro-processor117programs secondary address translation registers of the remaining PCIe switch NT port's secondary port, as discussed above relative to step702. Next, at step706, the micro-processor117initializes the PCIe configuration space of the secondary port112_0, and112_1to112—nas a PCIe endpoint. Next, at set708, the micro-processor117initializes the PCIe configuration space of the primary port114_0, and114_1to114—nas a PCIe bridge port. Next, at step710, the micro-processor117completes the remaining initialization of the switch108and the shared device119. Next, at step712, the micro-processor initiates a secondary port112_0and112_1to112—nhot plug event generation to the server102_0and102_1to102—n, respectively.

Next, at714, a determination is made as to whether or not the server102—iis connected and powered (operational) and if so, the process continues to step718, otherwise, the process goes to716where another determination is made as to whether or not the server102—i(“i” being an integer value) hot plugs into the secondary port112—iand if not, the process waits until the server102—ihot plugs into the secondary port112—i, otherwise, the process continues to step718.

At step718, the server102—ienumerates the secondary port112—ias a PCIe device. Next, at step720, the server102—i, as a part of the enumeration, writes to the PCI configuration registers132—iof the endpoint (the secondary port112—i). Next, at step722, the secondary port112—isends an interrupt message to the micro-processor117to indicate the changes to the PCIe configuration registers132—iof the secondary port112—i. Next, at step724, the micro-processor117reads the PCIe configuration registers132—ito take action based on the changes to the PCI configuration registers132—i. Next, at step726, the PCIe device (secondary port112—i) is ready to perform its function during regular operation of the system.

It is understood that while specific ports/registers or other devices are indicated herein, such the secondary port112—i, any port/register or structure analogous may be used instead and the references to the specific structures merely serve as examples.

FIG. 7shows a flow chart of the relevant steps during DMA using the various embodiments of the invention and in accordance with a method of the invention. More specifically, the flow chart ofFIG. 7shows the process for dynamically programming the primary address translation tables. At802, inFIG. 7, a determination is made as to whether or not there are any pending DMA requests and if not, the process waits at802until there is and then, continues on to804where another determination is made as to whether or not, a DMA channel is available and if not, the process continues to806, otherwise, the process continues to the step808. At806, the process awaits completion of the DMA that was started at802and upon completion, the process continues to the step808. At804, when a DMA channel is determined to be available, the process continues to the step808.

At step808, an available DMA channel k is assigned by the micro-processor117to the server102—jwith the DMA request pending based on priority and arbitration. ‘j’ and ‘k’ each are integers. Next, at step810, the micro-processor117programs the primary address translation table140—jwith entry140—j—kwith the source/destination server buffer pointers149—j—k(pointers of server buffer segment150—j—k) inside of the server102—jaddress space104—j. Next, at811, a determination is made as to whether data has to be written to server memory or read from server memory if the latter, the process goes to step813, otherwise, the process goes to step812.

At step813, the micro-processor117programs the DMA with source address to the primary port memory segment k pointer123—j—kand destination address in the micro-processor address space122and the process continues to the step814. At step812, the micro-processor117programs the DMA with the source address in the micro-processor address space122and the destination address to primary port memory segment k pointer123—j—k.

After steps812and813, the step814is performed where the micro-processor117programs the DMA channel k of the multi-channel DMA118to perform the DMA operation, followed by the step816where DMA is initiated and the process continues to802.