Flexible I/O DMA address allocation in virtualized systems

An I/O DMA address may be translated for a flexible number of entries in a translation validation table (TVT) for a partitionable endpoint number, when a particular entry in the TVT is accessed based on the partitionable endpoint number. A presence of an extended mode bit can be detected in a particular TVT entry. Based on the presence of the extended mode bit, an entry in the extended TVT can be accessed and used to translate the I/O DMA address to a physical address.

BACKGROUND

The present disclosure relates to address translation, and more specifically, to direct memory access (DMA) translation in virtualized systems.

Modern computing systems, for example server class computers, can utilize DMA translation mechanisms to enable input/output (I/O) virtualization. DMA translation can also allow for the sharing of a physical I/O device across multiple operating system (OS) images. This can allow for access to large real or physical memories using smaller address sizes. Systems may also use DMA translation mechanisms to protect some areas of system memory. For example, memory containing an OS kernel or hypervisor may be protected from unauthorized DMA.

Direct memory access in virtualized environments may require software or firmware to generate a table of translations between the address presented to a DMA controller and the actual target address in the system memory.

SUMMARY

Embodiments of the present disclosure can be directed toward a method for translating an I/O DMA address where a number of entries in a translation validation table (TVT) for a partitionable endpoint number (PE#) is flexible. Based on the PE#, a system can access a particular entry in the TVT. The PE# can be determined by a DMA requester ID (RID). Based on one or more bits in the particular entry in the TVT, an extended mode bit can be detected in the particular entry in the TVT. An entry in the extended TVT can be accessed, based on a value in the extended mode bit. Based on the value, and from bits in the I/O DMA address, the I/O DMA address can be translated to a physical address. The physical address may be the physical address that was requested by an I/O device associated with the I/O DMA address.

Embodiments of the present disclosure may be directed toward a system for translating an I/O DMA address where a number of entries in a TVT for a PE# is flexible. The system may have a processor circuit that has been configured to access a particular entry in the TVT. This entry may be based on the PE# and the PE# may be determined by a DMA RID. A presence of an extended mode bit may be detected in the particular entry in the TVT and an entry in the extended TVT may be accessed from an extended TVT and based on a value in the extended mode bit. The I/O DMA address may be translated to a physical address requested by an I/O device associated with the I/O DMA address based on a value from the entry in the extended TVT and from bits in the I/O DMA address.

Embodiments of the present disclosure may be directed toward a computer program product for translating an I/O DMA address, where a number of entries in a TVT for a PE# is flexible. The computer program product may have a computer readable storage medium with program instructions embodied within. The computer readable storage medium is not a transitory signal per se. The program instructions may be executable by a computer processing circuit to cause the circuit to perform a method. The method can include accessing a particular entry in the TVT. The entry can be accessed based on the PE# and the PE# can be determined by a DMA RID. A presence of an extended mode bit in the particular TVT entry can be detected and based on the extended mode bit, an entry in the extended TVT can be accessed. Based on the value from the entry in the extended TVT and from bits in the I/O DMA address, the I/O DMA address can be translated to a physical address. The physical address may be the address requested by an I/O device associated with the I/O DMA address.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to data processing, more particular aspects relate to address translation in virtualized systems. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.

DMA in computer systems can allow certain hardware associated with the system to access the main system memory independent of the central processing unit (CPU). DMA channels can allow for the transfer of data to and from devices with less CPU overhead than systems that do not use DMA channels. Many hardware devices including disk drive controllers, graphics cards, network cards, and sound cards use DMA. To carry out an input/output (I/O) DMA operation, a host processor can initialize the DMA controller. The system can then send commands to a peripheral device to initiate the data transfer. The DMA controller can then provide addresses and read/write control lines to the system memory. Each time a unit of data is ready to be transferred between the peripheral (I/O) device and memory, the DMA controller can increment its internal address register once the I/O DMA operation is completed.

DMA translation mechanisms may be used by modern computing systems. In some DMA translation mechanisms, each 4 KB page of physical system memory may be associated with a corresponding region (e.g. page) of I/O bus DMA address space. This may allow for address translations from an I/O bus DMA address space to physical system memory locations.

System hardware that is a target location of a DMA operation from an I/O device can use the incoming DMA page address on the I/O bus to retrieve, from a translation table, a translation from the DMA page address to the physical system memory page address. For example, a Peripheral Component Interconnect Express (PCIe) Host Bridge (PHB) may use an incoming DMA page address to retrieve the translated physical address.

Continuing with the example, a PCIe PHB may attach a PCIe bus to the larger system. The PHB hardware can determine, for a given operation, the partitionable endpoint number(s) (PE#(s)) to which the operation belongs. The PHB can also keep track of the state of the PE#, for purposes of stopping the PE on an error and preventing further operations after the error. The PHB can do this on a per PE# basis. The current set up is relatively static, as a number of PEs that have been allocated and the number of entries allocated to each PE are predetermined. Thus, in order to allocate more memory to a particular adapter, all processing across the entire system may need to be stopped, so the entire system can be reconfigured.

The PHB can perform this function in DMA operations. In DMA operations, the PHB can use a requester identifier (RID) associated with the operation as an index into an RID Translation Table (RTT). In this instance, a PE# field of an RTT entry (RTE) can indicate the PE# associated with the RID, and thus the RID can be used as an index into the RTT to access the RTE.

Embodiments of the present disclosure are directed toward a more flexible address allocation for use in the translation of I/O DMA addresses. For example, a PE# may be obtained from an I/O device PCIe transaction RID (e.g., Bus, Device, and Function numbers), and using an RTT or RTT cache. The PE can then be used to access a particular entry in a translation validation table (TVT). The TVT may be a single, contiguous table organized by PE#. Thus, the PE# is used to locate a particular TVT entry (TVE). Once the TVE is validated, a check of a particular bit (e.g., bit58) may occur. If bit58is “0”, then it may indicate that TVT Extended mode is not used for that particular PE# (and thus the adapter associated with the PE#). In that case, the normal mode of operation as in the existing hardware is executed, in order to access the physical address. If, however, the bit (e.g., bit58) is “1”, this can indicate that the TVT extended mode is used, and an “extended mode bit” field is used to access the extended TVT.

FIG. 1depicts a high level block diagram of a data processing system100, according to embodiments. The data processing system100can be, for example, a symmetric multiprocessor (SMP) system including a plurality of processors102a-102n, each coupled for communication to a system fabric104, which may include one or more bused or switched communication links. For example, data processing system100may be implemented with an IBM ESERVER. A data processing system with a single processor102may also be used.

Consistent with embodiments, each processor102may be realized as a single integrated circuit chip having a substrate on which semiconductor circuitry is fabricated. As shown, processor102includes a plurality of processor cores110that process data through the execution and/or processing of program code, which may include, for example, software and/or firmware and associated data. Processor102further includes cache memory112providing one or more levels of relatively low latency temporary storage for instructions and data retrieved from lower levels of the data storage hierarchy. In addition, processor102includes an integrated memory controller114that controls access to an associated one of off-chip system memories116.

Each processor102further includes a fabric interface (FIF)118by which processor102communicates with system fabric104, as well as one or more host bridges supporting input/output communication with various input/output adapters (IOAs)130. In the figure as shown, all of the host bridges are implemented as Peripheral Component Interconnect (PCI) host bridges (PHBs)120, but in other embodiments the host bridges may implement one or more additional or alternative I/O bus standards.

PHBs120a,120k,120m, and120vprovide interfaces to PCI local busses122a,122k,122m, and122v, respectively, to which IOAs130, such as network adapters, storage device controllers, peripheral adapters, etc., may be directly connected or indirectly coupled. For example, PCI IOA130ais coupled to PCI local bus122aoptionally through an I/O fabric124a, which may comprise one or more switches and/or bridges. In a similar manner, PCI IOAs130kand130lare coupled to PCI local bus122koptionally through an I/O fabric124k, PCI IOA130mis coupled to PCI local bus122moptionally through I/O fabric124m, and PCI IOAs130vand130w, which may comprise, for example, a display adapter and hard disk adapter, are coupled to PCI local bus122voptionally through I/O fabric124v.

Data processing system100can further include a service processor140that manages the boot process of data processing system100and thereafter monitors and reports on the performance of and error conditions detected in data processing system100. Service processor140is coupled to system fabric104and is supported by a local memory142, which may include volatile (e.g., dynamic random access memory (DRAM) and non-volatile memory (e.g., non-volatile random access memory (NVRAM) or static random access memory (SRAM))). Service processor140is further coupled to a mailbox interface144through which service processor140communicates I/O operations with PCI bus122a.

The architecture and components of a data processing system can vary between embodiments. For example, other devices and interconnects may alternatively or additionally be used. Accordingly, the data processing system100inFIG. 1is not meant to imply architectural limitations with respect to the disclosed.

FIG. 2depicts a logical view of a data processing system200showing the hardware and software resources of the data processing system partitioned into multiple logical partitions (LPARs). Data processing system200may have, for example, the same components and/or architecture as data processing system100inFIG. 1and may accordingly identify common components with like reference numerals.

Data processing system200has a collection of partitioned hardware202, including processors102a-102n, system memories116a-116nand IOAs130a-130w. Partitioned hardware202may of course include additional unillustrated components, such as additional volatile or nonvolatile storage devices, ports, bridges, switches etc. The hardware components comprising partitioned hardware202(or portions thereof) can be assigned to various logical partitions (LPARs)210a-210pin data processing system200by system firmware204, also referred to as a virtual machine monitor (VMM) or hypervisor. System firmware204supports the simultaneous execution of multiple independent operating system instances by virtualizing the portioned hardware of data processing system200.

In addition to the hardware resources allocated by system firmware204, each of LPARs210a-210pincludes a respective one of multiple concurrently executable operating system instances212a-212p. In various embodiments, operating system instances212a-212p, which may include, for example, instances of LINUX, AIX, and/or WINDOWS, may be homogenous or heterogeneous. Each LPAR210may further include unillustrated application programs, as well as a respective instance of partition firmware214. When LPARs210a-210pare instantiated, boot strap code is loaded onto partitions210a-210pby system firmware204. Thereafter system firmware204may transfer control to the boot strap code, which can load firmware and software. The processor(s)102assigned to each LPAR210may then execute the partition firmware214of that LPAR to bring up the LPAR and initiate execution of an OS instance212.

In the logically partitioned environment shown inFIG. 2, service processor140can be used to provide various services, such as processing of errors in LPARs210a-210p. These services may also function as service agents to report errors back to a system administrator or a vendor of data processing system200. The operation of the different LPARs210may further be controlled through a hardware management console220. Hardware management console220can be implemented as a separate data processing system from which a system administrator may perform various functions within data processing system200including, but not limited to, creating and destroying LPARs210, as well as reallocating hardware and software resources among LPARs210.

Additionally, in a logically partitioned environment as shown, it may not be permissible for the hardware or software resources in one LPAR210to consume the resources of or affect the operations in another LPAR210. Furthermore, to be useful, the assignment of resources to LPARs210in certain embodiments needs to be fine-grained. For example, it is often not acceptable to assign all IOAs130under a particular PHB120(both fromFIG. 1) to the same partition, as that may restrict configurability of the system, including the ability to dynamically reallocate resources between partitions. Accordingly, PHBs120are able to assign resources, such as individual IOAs130(or portions thereof) to different LPARs210while preventing the assigned resources from accessing or affecting the resources of other LPARs210.

To support such isolation between the resources of different LPARs210, the I/O subsystem of a data processing system can be subdivided into multiple partitionable endpoints. A “partitionable endpoint” or “PE” is defined herein as any component or subcomponent of an I/O subsystem that can be allocated to an LPAR independently of any other component or subcomponent of the I/O subsystem. For example, some PEs may comprise a plurality of IOAs and/or I/O fabric components that function together and, thus, should be allocated as a unit to a single LPAR. Another PE, however, may comprise a portion of a single IOA, for example, a separately configurable and separately assignable port of a multi-port IOA. A PE may be identified by its function rather than its structure.

FIG. 3depicts an example TVT Entry300within a TVT, according to embodiments. The TVT entry (TVE)300depicted contains an extended mode bit312. Unlike a standard TVE, where a similar location within the TVE contains a TCE table address field (further discussion in later figures), the TVE300pictured points to an extended TVT, rather than a TCE table. The entry300can contain a field that indicates the number of entries in the extended TVT302. This field can indicate an encoded value of a number of TVT entries in the extended TVT table for the particular PE. Another field labeled “E”304, can be a one-bit field that indicates whether or not the TVE has an extended mode. For example, a “1” could indicate that the TVE is associated with an extended mode (e.g., points to the extended TVT rather than TCE table), where a “0” could indicate that the TVE is configured to operate in “normal” or “standard” mode (e.g., points to a TCE table not the extended TVT). Herein this one-bit field that can indicate whether or not the TVE has an extended mode can be referred to as an extended mode one-bit.

Fields306and310labeled “RSVD” are reserved fields and are not relevant to the current disclosure, other than to accurately depict a particular TVE layout. Field308labeled “total DMA window size” can, in some embodiments, be a 5-bit field that indicates the encoded value of the total DMA window size configured under the TVE300. This total DMA window may be equally distributed among all the entries in the extended TVT. The value in this field indicates a particular size of a DMA window. The number of bits allocated to this field (e.g., 5) and the encoding algorithm used may vary and be determined by the particular platform implementation. For example, Table 1 below lists relationships between the value in the field308and the size of the DMA window.

Field312is the extended mode bit, which may contain the physical real address of the extended TVT allocated in memory for the particular entry (and thus PE, as the TVEs in the TVT are organized by PE).

FIG. 4Adepicts a translation validation table (TVT)400, according to embodiments. The TVT400may be a contiguous table in system memory, and the TVT400may be comprised of a number of entries (TVEs). For example, entries402and404are depicted inFIG. 4A. Entries416-422are also depicted, and may represent any number of intervening entries in a TVT. The TVT400may be organized by PE#.

TVE404depicts a standard TVT entry. As indicated, TVE404does not have an address bit field that points to the extended TVT. Rather, it is comprised of, among potentially other fields, an I/O page size field408a, a TCE table size field408b, and a TCE table address field408c. The TCE table address field408cpoints to the start of the TCE table associated with the particular PE. A particular address within a TCE table address can be identified, which can then be used to access the physical address associated with the PE associated with the TVE404.

TVE402depicts a modified TVT entry, with an extended mode bit406c. The TVE402may include a number of entries field406a, a total DMA size field406band an extended mode bitfield406c, the latter of which points to a location in an extended TVT. TVEs depicted here are example TVEs only, and are not intended to limit configuration of TVEs or TVTs.

FIG. 4Bdepicts an extended translation validation table (TVT)410, according to embodiments. Extended TVT410may comprise extended TVT entries414and424-432. The extended TVT410may have more or fewer entries than those depicted. An example extended TVT entry (TVE)414depicted may contain an I/O page size field412a, a TCE table size field412b, and a TCE table address field412c. The TCE table address field412can point to the start of the TCE table, from which a particular location in the TCE table can be identified, the location of which can be used to complete the translation to a physical system memory address.

FIG. 5depicts a system flow diagram500for translating a DMA address. The system flow500may begin when a PHB receives a PCIE transaction RID502from an I/O device. The RID502may comprise bus, device, and function numbers. Using the PCIE transaction RID502and a PE# Translation Table (also known as a RID translation table (RTT))504, a PE# may be obtained. In some circumstances, a lookup in the RTT504may not be required, if the required data is stored in a local cache, for example an RID translation cache (RTC)506. If a lookup is required, the data obtained from the lookup in the RTT504may be stored in the RTC506, as indicated by an arrow between the two. The PE#510obtained from either the RTT504lookup or accessed from the cache506, as well as a particular address bit512obtained from a DMA address508can be used to retrieve the corresponding TVT entry (TVE)515from the TVT514. Data from the particular TVE515can be used, in conjunction with another particular bit or bits from the DMA Address508to retrieve a corresponding entry517in the extended TVT516. In this way, additional storage (e.g., entries) can be allocated to a particular PE#. Data from the extended TVT entry517can be used, along with additional data from the DMA address508to validate the address, per518and access the TCE based address, per520. From there, the physical address can be accessed from system memory in one of a number of ways.

FIG. 6depicts a flow diagram600for translating a DMA address, according to embodiments. The flow diagram600can begin when a system attempts to access an entry in a TVT (i.e., a TVE) based on a PE#, per602. If the TVE is valid, per604, a check for an extended mode bit in the TVE is conducted, per608. If at604, the TVE is not valid, then the operation may be ignored, per606. At608, if an extended mode bit is present in the particular TVE, then the system can detect an extended mode bit in the TVT entry, per610. Using the data in the extended mode bit of the TVE, an entry in the extended TVT may be accessed, per612. Using the data from the extended TVT entry and from the DMA address, the corresponding entry in the TCE table may be accessed, per614. The physical address can be accessed using this data, per616.

If at608, there is no extended mode bit present, then the address can be translated via standard processes which can include, for example, accessing the corresponding entry in the TCE table using data from the TVT entry, per618, and accessing the physical address in system memory using data from the accessed entry in the TCE table, per616.