Dynamic I/O translation table allocation for single root input output virtualization enabled I/O adapters

Dynamic I/O translation table allocation for SR-IOV enabled I/O adapters including receiving an instruction to transition the SR-IOV adapter to shared mode; determining, using a configuration file for the SR-IOV adapter, memory requirements for an I/O translation table; comparing the memory requirements for the I/O translation table to memory currently allocated for use by the SR-IOV adapter; based on the comparing the memory requirements for the I/O translation table to the memory currently allocated for use by the SR-IOV adapter, determining that the memory currently allocated for use by the SR-IOV adapter is insufficient to store the I/O translation table and determining an additional amount of memory required to store the I/O translation table; and allocating, at runtime, the additional amount of memory for storing the I/O translation table.

BACKGROUND

Field of the Invention

The field of the invention is data processing, or, more specifically, methods, apparatus, and products for dynamic I/O translation table allocation for Single Root Input Output Virtualization enabled I/O adapters.

Description of Related Art

SUMMARY

Methods, systems, and apparatus for dynamic I/O translation table allocation for Single Root Input Output Virtualization (SR-IOV) enabled I/O adapters are disclosed in this specification. Dynamic I/O translation table allocation for SR-IOV enabled I/O adapters includes receiving an instruction to transition the SR-IOV adapter to shared mode, wherein the SR-IOV in shared mode provides, to multiple logical partitions, access to the SR-IOV adapter; determining, using a configuration file for the SR-IOV adapter, memory requirements for an I/O translation table for use by the SR-IOV adapter in shared mode; comparing the memory requirements for the I/O translation table to memory currently allocated for use by the SR-IOV adapter; based on the comparing the memory requirements for the I/O translation table to the memory currently allocated for use by the SR-IOV adapter, determining that the memory currently allocated for use by the SR-IOV adapter is insufficient to store the I/O translation table and determining an additional amount of memory required to store the I/O translation table; and allocating, at runtime, the additional amount of memory for storing the I/O translation table.

DETAILED DESCRIPTION

Exemplary methods, apparatus, and products for dynamically bypassing a peripheral component interconnect (PCI) switch in accordance with the present invention are described with reference to the accompanying drawings, beginning withFIG. 1.FIG. 1sets forth a network diagram of a system configured for dynamically bypassing a PCI switch according to embodiments of the present invention. The example environment ofFIG. 1includes a computing system (102). Such a computing system (102) may provide clients on host devices (external to the computing system (102)) with virtualization services for enabling various cloud related product offerings.

The example computing system (102) ofFIG. 1includes at least one computer processor (156) or “CPU” as well as random access memory (168) or “RAM,” which is connected through a high speed memory bus (166) and bus adapter (158) to processor (156) and to other components of the computing system (102).

Stored in RAM (168) is a hypervisor (136) and a management console (138). The management console (138) may provide a user interface through which a user may direct the hypervisor (136) on instantiating and maintaining multiple logical partitions (logical partition A (116A), logical partition B (116B)), where each logical partition may provide virtualization services to one or more clients. The management console (138) may also provide a user interface through which a user may direct the hypervisor (136) to switch a Single Root Input Output Virtualization (SR-IOV) adapter from dedicated mode to shared mode.

Also stored in RAM (168) are two instances of an operating system (154), one for each logical partition (logical partition A (116A), logical partition B (116B)). Operating systems useful in computers configured for firmware management of SR-IOV adapters according to various embodiments include UNIX™, Linux™, Microsoft Windows™, AIX™, IBM's i™ operating system, and others as will occur to those of skill in the art. The operating systems (154), hypervisor (136), and management console (138) are shown in RAM (168), but many components of such software may typically be stored in non-volatile memory such as, for example, on a data storage (170) device or in firmware (132).

The computing system (102) may also include a storage device adapter (172) coupled through expansion bus (160) and bus adapter (158) to processor (156) and other components of the computing system (102). Storage device adapter (172) connects non-volatile data storage to the computing system (102) in the form of data storage (170). Storage device adapters useful in computers configured for dynamically bypassing a PCI switch according to various embodiments include Integrated Drive Electronics (“IDE”) adapters, Small Computing system Interface (“SCSI”) adapters, and others as will occur to those of skill in the art. Non-volatile computer memory also may be implemented for as an optical disk drive, electrically erasable programmable read-only memory (so-called “EEPROM” or “Flash” memory), RAM drives, and so on, as will occur to those of skill in the art.

The example computing system (102) may also include one or more input/output (“I/O”) adapters (178). I/O adapters implement user-oriented input/output through, for example, software drivers and computer hardware for controlling output to display devices such as computer display screens, as well as user input from user input devices (181) such as keyboards and mice. The example computing system (102) may also include a video adapter (114), which may be an example of an I/O adapter specially designed for graphic output to a display device (180) such as a display screen or computer monitor. Video adapter (114) may be connected to processor (156) through a high speed video bus (164), bus adapter (158), and the front side bus (162), which may also be a high speed bus.

The expansion bus (160) shown inFIG. 1also includes a SR-IOV adapter (126). SR-IOV is an extension to the PCIe specification. The SR-IOV adapter (126) allows a device, such as a network adapter, to separate access to its resources among various PCIe hardware functions. These functions consist of the following types: A PCIe physical function and a PCIe virtual function. The physical function advertises the device's SR-IOV capabilities. Each virtual function is associated with a device's physical function. A virtual function shares one or more physical resources of the device, such as a memory and a network port, with the physical function and other virtual functions on the device. From the perspective of a logical partition (logical partition A (116A), logical partition B (116B)) instantiated by a hypervisor (136), a virtual function appears as a fully functional physical PCIe adapter. In this way, a single physical adapter may be ‘shared’ amongst many logical partitions or multiple virtual functions may be instantiated for use by a single logical partition. Although referred to as a ‘virtual’ function, readers of skill in the art will recognize that a virtual function is in fact a physical channel that is not a resource virtualized entirely by the hypervisor.

The SR-IOV adapter (126) may operate in one of two modes. In a dedicated mode, the SR-IOV adapter (126) may be assigned for use by a single entity, such as a single logical partition. In shared mode, the SR-IOV adapter (126) may be assigned for use by multiple entities, such as multiple logical partitions (logical partition A (116A), logical partition B (116B)). Using the SR-IOV adapter in shared mode may require a greater number of resources relative to the SR-IOV adapter in dedicated mode. Such resources include memory for I/O translation tables, partitional endpoint (PE) numbers, bus/device/function (BDF) ranges, memory mapping input/output (MMIO) addresses, direct memory access (DMA) addresses, and message signal interrupt (MSI) addresses.

The computing system (102) ofFIG. 1also includes a service processor (not shown). A service processor is a separate, dedicated internal processor typically located on the motherboard of a server, a PCI card, or on the chassis of a blade server or telecommunications platform. The service processor operates independently from the server's CPU and operating system (OS), even if the CPU or OS is locked up or otherwise inaccessible. The service processor may monitor operation of various hardware, software, and firmware resources within the computing system (102) through an out-of-band bus. The service processor may report such monitoring to a system management server through an out-of-band network. The service processor may be coupled to a different power source than other components of the computing system (102) so that the service processor may operate even when the main components of the computing system (102) are powered down.

FIG. 2shows an exemplary system for dynamic I/O translation table allocation for Single Root Input Output Virtualization (SR-IOV) enabled I/O adapters according to embodiments of the present invention. As shown inFIG. 2, the exemplary computing system (102) includes an adjunct partition (202) and two logical partitions (logical partition A (116A), logical partition B (116B)) executing on a hypervisor (136). The adjunct partition (202) includes a physical function (204). Each logical partition (logical partition A (116A), logical partition B (116B)) includes a virtual function (virtual function A (206A), virtual function B (206B)). The computing system (102) also includes a management console (138), service processor (210), and a SR-IOV adapter (126). The hypervisor (136) includes an I/O translation table (208).

The computing system (102) ofFIG. 2shows the SR-IOV adapter (126) in shared mode. The SR-IOV adapter (126) is communicatively coupled, via the hypervisor (136), to the physical function (204) of the adjunct partition (202) and the virtual functions (virtual function A (206A), virtual function B (206B)) of the logical partitions (logical partition A (116A), logical partition B (116B)). In shared mode, the SR-IOV adapter (126) provides I/O services (e.g., such as access to a network) to both logical partitions (logical partition A (116A), logical partition B (116B)) via the virtual functions (virtual function A (206A), virtual function B (206B)).

The management console (138) may generate an instruction for the hypervisor (136) to transition the SR-IOV adapter (126) from dedicated mode to shared mode. The instruction may be generated in response to input from a user. The hypervisor (136) may communicate with the service processor (210) to prepare to transition the SR-IOV adapter (126) from dedicated mode to shared mode. Specifically, the hypervisor (136) may request information from the service processor (210) about the resource requirements of the SR-IOV adapter in shared mode. Such resource requirements may be stored in a configuration file for the SR-IOV adapter.

The I/O translation table (208) contains translation control entries used for I/O address protection. The translation control entries are used by each logical partition (logical partition A (116A), logical partition B (116B)) so that the hypervisor (136) may validate the DMA transactions between logical partitions (logical partition A (116A), logical partition B (116B)) and the SR-IOV adapter (126).

The I/O translation table (208) may be stored in memory assigned to the hypervisor (136). When the SR-IOV adapter is in dedicated mode, the amount of memory required for the I/O translation table (208) may be less, or significantly less, than the amount of memory required for the I/O translation table when the SR-IOV adapter is in shared mode. Memory reserved and allocated for the I/O translation table (208) at boot time may only be sufficient to store the I/O translation table (208) for the SR-IOV adapter (126) in dedicated mode. If the SR-IOV adapter (126) is to be transitioned from dedicated mode to shared mode during runtime, an additional amount of memory may need to be allocated for use by the hypervisor (136) to store the I/O translation table.

For further explanation,FIG. 3sets forth a flow chart illustrating an exemplary method for dynamic I/O translation table allocation for Single Root Input Output Virtualization (SR-IOV) enabled I/O adapters according to embodiments of the present invention that includes receiving (302) an instruction (320) to transition the SR-IOV adapter to shared mode, wherein the SR-IOV in shared mode provides, to multiple logical partitions, access to the SR-IOV adapter. Receiving (302) an instruction (320) to transition the SR-IOV adapter to shared mode, wherein the SR-IOV in shared mode provides, to multiple logical partitions, access to the SR-IOV adapter may be carried out by the hypervisor (136) receiving the instruction (320) from a user via a management console. The instruction (320) may be an instruction to transition the SR-IOV adapter from dedicated to shared mode.

The method ofFIG. 3further includes determining (304), using a configuration file (322) for the SR-IOV adapter, memory requirements for an I/O translation table for use by the SR-IOV adapter in shared mode. Determining (304), using a configuration file (322) for the SR-IOV adapter, memory requirements for an I/O translation table for use by the SR-IOV adapter in shared mode may be carried out by the hypervisor (136) retrieving the configuration file and extracting the memory requirements for the I/O translation table for use by the SR-IOV adapter in shared mode.

The configuration file is a data repository that include information about the SR-IOV adapter, including the memory requirements for the I/O translation table used in shared mode. The configuration file may include other information about the SR-IOV adapter, such as memory requirements for the I/O translation table used in dedicated mode, firmware information, and adapter characteristics. The configuration file may be retrieved via a service processor. Specifically, the hypervisor (136) may request the configuration file using an interface with the service processor. The request may include an identifier of the SR-IOV adapter. Alternatively, the configuration file may be retrieved from memory on the SR-IOV adapter itself.

The memory requirements for the I/O translation table used in shared mode may be a static amount of memory, such as 1028 megabytes of memory. Alternatively, the memory requirements for the I/O translation table used in shared mode may be a calculation based on a given configuration of the SR-IOV adapter. For example, the memory requirements for the I/O translation table used in shared mode may be in terms of an amount of memory per virtual function to be assigned.

The method ofFIG. 3further includes comparing (306) the memory requirements for the I/O translation table to memory currently allocated for use by the SR-IOV adapter. Comparing (306) the memory requirements for the I/O translation table to memory currently allocated for use by the SR-IOV adapter may be carried out by the hypervisor (136) determining an amount of available memory currently allocated for the I/O translation table in dedicated mode. The hypervisor may then determine the difference between the memory requirements for the I/O translation table in shared mode and the memory currently allocated for the I/O translation table in dedicated mode, and whether the memory requirements for the I/O translation table in shared mode exceed the memory currently allocated for the I/O translation table in dedicated mode.

The method ofFIG. 3further includes, based on the comparing the memory requirements for the I/O translation table to the memory currently allocated for use by the SR-IOV adapter, determining (308) that the memory currently allocated for use by the SR-IOV adapter is insufficient to store the I/O translation table and determining an additional amount of memory required to store the I/O translation table.

Determining (308) that the memory currently allocated for use by the SR-IOV adapter is insufficient to store the I/O translation table may be carried out by the hypervisor (136) determining that the memory requirements for the I/O translation table in shared mode exceed the memory currently allocated for the I/O translation table in dedicated mode. Determining an additional amount of memory required to store the I/O translation table may be carried out by the hypervisor (136) measuring the difference between the memory requirements for the I/O translation table in shared mode and the memory currently allocated for the I/O translation table in dedicated mode.

For example, the configuration file may indicate that the SR-IOV adapter requires one gigabyte of memory for the I/O translation table in shared mode. The hypervisor (136) may determine that there is only 32 megabytes allocated for the I/O translation table in dedicated mode. The hypervisor compares the memory requirements for the I/O translation table in shared mode and the memory currently allocated for the I/O translation table in dedicated mode and determines that an additional amount of 992 megabytes of memory are required to store the I/O translation table for operating the SR-IOV adapter in shared mode.

The method ofFIG. 3further includes, allocating (310), at runtime, the additional amount of memory for storing the I/O translation table. Allocating (310), at runtime, the additional amount of memory for storing the I/O translation table may be carried out by determining that the additional amount of memory is available from the hypervisor; and allocating, at runtime, memory from the hypervisor for storing the I/O translation table. Allocating (310), at runtime, the additional amount of memory for storing the I/O translation table may also be carried out by determining that the additional amount of memory is unavailable from the hypervisor; determining that the additional amount of memory is available from unallocated system memory; and allocating, at runtime, an amount of memory to the hypervisor to satisfy the additional amount of memory required to store the I/O translation table.

The above-described limitations improve the operation of the computer system by enabling an SR-IOV adapter to uniquely specify, via the configuration file, the memory requirements of the I/O translation table. The above-described limitations also improve the operation of the computer system by allowing for adjustments to the memory requirement after the SR-IOV adapter has been deployed by altering the configuration file to reflect updated memory requirements. Finally, the above-described limitations improve the operation of the computer system by enabling the allocation of memory for the I/O translation table in shared mode only after a transition from dedicated mode, preventing the memory from being underutilized during dedicated mode.

For further explanation,FIG. 4sets forth a flow chart illustrating a further exemplary method for dynamic I/O translation table allocation for Single Root Input Output Virtualization (SR-IOV) enabled I/O adapters according to embodiments of the present invention that includes receiving (302) an instruction (320) to transition the SR-IOV adapter to shared mode, wherein the SR-IOV in shared mode provides, to multiple logical partitions, access to the SR-IOV adapter; determining (304), using a configuration file (322) for the SR-IOV adapter, memory requirements for an I/O translation table for use by the SR-IOV adapter in shared mode; comparing (306) the memory requirements for the I/O translation table to memory currently allocated for use by the SR-IOV adapter; based on the comparing the memory requirements for the I/O translation table to the memory currently allocated for use by the SR-IOV adapter, determining (308) that the memory currently allocated for use by the SR-IOV adapter is insufficient to store the I/O translation table and determining an additional amount of memory required to store the I/O translation table; and allocating (310), at runtime, the additional amount of memory for storing the I/O translation table.

The method ofFIG. 4differs from the method ofFIG. 3, however, in that the method ofFIG. 4further includes determining (402) that the additional amount of memory is available from the hypervisor (136); and allocating (404), at runtime, memory from the hypervisor (136) for storing the I/O translation table. Determining that the additional amount of memory is available from the hypervisor may be carried out by the hypervisor (136) determining an amount of available memory currently assigned to the hypervisor and determining whether the amount of available memory currently assigned to the hypervisor is at least as large as the additional amount of memory required to store the I/O translation table.

Allocating (310), at runtime, memory from the hypervisor for storing the I/O translation table may be carried out by the hypervisor (136) reserving the additional amount of memory required to store the I/O translation table from memory assigned to the hypervisor. The allocation of memory for the I/O translation table is performed at runtime. Allocating the memory at runtime refers to allocating the memory without rebooting, restarting, or otherwise reinitializing the computing system. Allocating (310), at runtime, memory from the hypervisor for storing the I/O translation table may also be carried out by the hypervisor (136) reorganizing memory allocated to itself to ensure that a contiguous range of memory addresses are used for the I/O translation table.

For further explanation,FIG. 5sets forth a flow chart illustrating a further exemplary method for dynamic I/O translation table allocation for Single Root Input Output Virtualization (SR-IOV) enabled I/O adapters according to embodiments of the present invention that includes receiving (302) an instruction (320) to transition the SR-IOV adapter to shared mode, wherein the SR-IOV in shared mode provides, to multiple logical partitions, access to the SR-IOV adapter; determining (304), using a configuration file (322) for the SR-IOV adapter, memory requirements for an I/O translation table for use by the SR-IOV adapter in shared mode; comparing (306) the memory requirements for the I/O translation table to memory currently allocated for use by the SR-IOV adapter; based on the comparing the memory requirements for the I/O translation table to the memory currently allocated for use by the SR-IOV adapter, determining (308) that the memory currently allocated for use by the SR-IOV adapter is insufficient to store the I/O translation table and determining an additional amount of memory required to store the I/O translation table; and allocating (310), at runtime, the additional amount of memory for storing the I/O translation table.

The method ofFIG. 5differs from the method ofFIG. 3, however, in that the method ofFIG. 5further includes determining (502) that the additional amount of memory is unavailable from the hypervisor (136); determining (504) that the additional amount of memory is available from unallocated system memory; and allocating (506), at runtime, an amount of memory to the hypervisor (136) to satisfy the additional amount of memory required to store the I/O translation table.

Determining (502) that the additional amount of memory is unavailable from the hypervisor (136) may be carried out by the hypervisor (136) determining an amount of available memory currently assigned to the hypervisor and determining that the amount of available memory currently assigned to the hypervisor is insufficient to satisfy the additional amount of memory required to store the I/O translation table.

Determining (504) that the additional amount of memory is available from unallocated system memory may be carried out by the hypervisor (136) determining an amount of system memory that is unassigned to either the hypervisor or any logical partition and comparing the unassigned amount of memory to the additional amount of memory required to store the I/O translation table.

Allocating (404), at runtime, an amount of memory to the hypervisor (136) to satisfy the additional amount of memory required to store the I/O translation table may be carried out by the hypervisor (136) adding, to the memory already assigned to the hypervisor (136), an amount of memory sufficient to satisfy the additional amount of memory required to store the I/O translation table.

If the amount of memory necessary to satisfy the additional amount of memory required to store the I/O translation table is not available from either the hypervisor or from unallocated system memory, then the hypervisor may generate an error indication that there is insufficient memory available to transition the SR-IOV adapter to shared mode. Generating an error indicating that there is insufficient memory available to transition the SR-IOV adapter to shared mode may be carried out by the hypervisor (136) creating an error message and informing the user via the management console. The error message may indicate the amount of the memory insufficiency and ways in which the user may free memory for use by the SR-IOV adapter in shared mode.

For further explanation,FIG. 6sets forth a flow chart illustrating a further exemplary method for dynamic I/O translation table allocation for Single Root Input Output Virtualization (SR-IOV) enabled I/O adapters according to embodiments of the present invention that includes receiving (302) an instruction (320) to transition the SR-IOV adapter to shared mode, wherein the SR-IOV in shared mode provides, to multiple logical partitions, access to the SR-IOV adapter; determining (304), using a configuration file (322) for the SR-IOV adapter, memory requirements for an I/O translation table for use by the SR-IOV adapter in shared mode; comparing (306) the memory requirements for the I/O translation table to memory currently allocated for use by the SR-IOV adapter; based on the comparing the memory requirements for the I/O translation table to the memory currently allocated for use by the SR-IOV adapter, determining (308) that the memory currently allocated for use by the SR-IOV adapter is insufficient to store the I/O translation table and determining an additional amount of memory required to store the I/O translation table; and allocating (310), at runtime, the additional amount of memory for storing the I/O translation table.

The method ofFIG. 6differs from the method ofFIG. 3, however, in that the method ofFIG. 6further includes instantiating (602), at runtime, the I/O translation table using the allocated memory from the hypervisor; and providing (604), to the multiple logical partitions, access to the SR-IOV adapter using the I/O translation table stored in the allocated memory from the hypervisor.

Instantiating (602), at runtime, the I/O translation table using the allocated memory from the hypervisor may be carried out by the hypervisor (136) creating the I/O translation table using the allocated memory from the hypervisor and preparing the I/O translation table for use by the logical partitions and SR-IOV adapter in shared mode. Instantiating the I/O translation table at runtime refers to instantiating the I/O translation table without rebooting, restarting, or otherwise reinitializing the computing system.

Providing (604), to the multiple logical partitions, access to the SR-IOV adapter using the I/O translation table stored in the allocated memory from the hypervisor may be carried out by the hypervisor (136) transitioning, at runtime, the SR-IOV adapter from dedicated mode to shared mode. Once in shared mode, the multiple logical partitions may utilize the SR-IOV adapter using the virtual functions of each of the multiple logical partitions.

In view of the explanations set forth above, readers will recognize that the benefits of dynamic I/O translation table allocation for SR-IOV enabled I/O adapters according to embodiments of the present invention include:Improving the operation of a computing system by enabling an SR-IOV adapter to uniquely specify, via the configuration file, the memory requirements of the I/O translation table, increasing computing system utilizing and efficiency.Improving the operation of a computing system by allowing for adjustments to the requirement after the SR-IOV adapter has been deployed by altering the configuration file to reflect updated requirements, increasing computing system functionality.Improving the operation of a computing system by enabling the allocation of memory for the I/O translation table in shared mode only after a transition from dedicated mode, preventing the memory from being underutilized during dedicated mode, increasing computing system efficiency.