Patent Description:
Hardware resources, in particular I/O devices such as storage devices, accelerator hardware, and network interfaces, may be virtualized. The virtualization of the hardware resources allows multiple VMs to share the same hardware resources.

<CIT> discloses techniques for providing adaptive virtual function (VF) drivers capable of operating with physical devices having a plurality of different hardware configurations. In one embodiment, for example, an apparatus may include logic to implement a virtual machine (VM), the logic to initialize an adaptive virtual function (VF) driver to facilitate communication between the VM and a physical device to be virtualized, establish communication between the adaptive VF driver and a physical function (PF) driver of the hypervisor for the physical device, activate a standard feature set for the adaptive VF driver to execute on a PF of the physical device, and negotiate activation of an advanced feature set for the adaptive VF driver to execute on the PF, the adaptive VF driver to provide the advanced feature set to the PF, the PF activate each feature of the advanced feature set supported by the PF.

It is an object of the present disclosure to provide a computing device, a method, and a computer-readable storage medium for virtualized hardware resources of an input/output (I/O) device.

According to a first aspect, a computing device in communication with an input/output, I/O, device providing virtualized hardware resources for use by a plurality of virtual machines, VM, wherein the computing device is configured to: execute a virtual machine manager (VMM) that implements a physical function, PF, driver corresponding to the I/O device and to a plurality of virtual function, VF, drivers implemented by the plurality of VMs, receive at least one message of the first message type transmitted from a virtual function, VF, driver of a VM to the PF, driver, wherein the at least one message of the first message type is formatted in a dynamic message format that includes: a first dynamic indication denoting a number of a plurality of capabilities, and a plurality of second dynamic indications each associated with one respective capability, each respective dynamic second indication is associated with a dynamic capability data indication that stores data of the respective capability, wherein a size of the data stored in the dynamic capability data indication is denoted by the corresponding second dynamic indication, wherein the first dynamic indication does not lock the dynamic message format to a specific size and the size of the dynamic message format is adapted to change to any size needed to define any number of supported capabilities, wherein the at least one message of the first message types includes a set of requested capabilities supported by the VF driver for enablement on or by the I/O device, transmit at least one message of a second message type from the PF driver to the VF driver, wherein the at least one message of the second message types includes a set of supported capabilities selected from the requested capabilities that are supported by the I/O device and PF driver for implementation thereon, and enable the set of supported capabilities by the I/O device and the PF driver for use by the VF driver.

According to a second aspect, a method for providing virtualized hardware resources of an input/output, I/O, device for use by a plurality of virtual machines, VM, the method comprises: executing a virtual machine manager (VMM) that implements a physical function, PF, driver corresponding to the I/O device and to a plurality of virtual function, VF, drivers implemented by the plurality of VMs, receiving at least one message of a first message type transmitted from a virtual function, VF, driver of a VM to the PF driver, wherein the at least one message of the first message type is formatted in a dynamic message format that includes: a first dynamic indication denoting a number of a plurality of capabilities, and a plurality of second dynamic indications each associated with one respective capability, each respective dynamic second indication a size of the data stored in the dynamic capability data indication is denoted by the corresponding second dynamic indication, wherein the first dynamic indication does not lock the dynamic message format to a specific size and the size of the dynamic message format is adapted to change to any size needed to define any number of supported capabilities, wherein the at least one message of the first message type includes a set of requested capabilities supported by the VF driver for enablement on or by the I/O device, transmitting at least one message of a second message type from the PF to the VF, wherein the at least one message of the second message type includes a set of supported capabilities selected from the requested capabilities that are supported by the I/O device and PF driver for implementation thereon, and enabling the set of supported capabilities by the I/O device and the PF driver for use by the VF driver.

The method according to the second aspect may be performed by a computer-readable storage medium storing program code, the program code comprising instructions, which when performed on a computer cause the computer to perform the method according to the second aspect.

The dynamic message format scales for easy extension of capabilities. The first indication, which is dynamic, does not lock the dynamic message format to a specific size, allowing new/old drivers to communicate dynamically. The size of the dynamic message format may change to any size needed to define any number of supported capabilities. Additional capabilities may be added to the VF driver and/or PF driver, without necessarily requiring adjustment of the driver code. Each respective capability is associated with its own variable data structure (i.e., the second dynamic indication) that stores data regarding which details of the respective capability are supported and/or which details are not.

In a further implementation form of the first and second aspects, each one of the plurality of capabilities defined by the dynamic message format is independently processed sequentially until the number of the plurality of capabilities defined by the first dynamic indication is reached.

Each capability may be individually negotiated, providing increased flexibility. The first dynamic indication enables detecting when all capabilities in the received first message type(s) have been negotiated.

In a further implementation form of the first and second aspects, the computing device is further configured to poll for additional messages of the first message type when the capabilities, defined in the received at least one message of the first message type, have been processed and a number of the processed capabilities is less than the number of capabilities defined by the first dynamic indication.

When the capabilities do not fit into one messages, multiple messages may be sent. The total number of capabilities may be tracked for activation. When one or more of the total messages are missing, the missing messages with missing capabilities may be requested for activation thereof.

When all capabilities in the received first message type(s) have been received but the total number of capabilities defined in the first dynamic indication has not been reached, it is in an indication that some capabilities are missing.

In a further implementation form of the first and second aspects, the dynamic messages format further includes a plurality of third dynamic indications each associated with one respective capability, each respective third dynamic indication denoting a version of the corresponding respective capability.

The capability version of the third dynamic indication may assist with internal features and/or bug fixes, for example, to allow code bypass on the PF and/or VF. The capability version of the third dynamic indication may provide easier backwards compatibility, in particular where only one side is upgraded, i.e., when only one of the VF and the PF are upgraded. New capabilities may be defined by newer versions, which may have distinctive fields for setting. Versions may be defined for each package of capabilities, making it is easy to negotiate the supported version between a new side and an old side. Capabilities may be individually selected for activation, for example, where the respective capability is a newer version of an older existing capabilities. Capabilities for which the version is irrelevant and/or unchanged do not necessarily need to be activated (since they are already activated and/or are irrelevant).

In a further implementation form of the first and second aspects, the dynamic messages format is implemented based on a type-version-length-value, TVLV, format, the type comprises an opcode denoting that the respective message is a capabilities messages, the version denotes the third dynamic indication, the length denotes the first dynamic indication, and the value denotes the second dynamic indication and corresponding capabilities data.

In a further implementation form of the first and second aspects, the dynamic messages format is implemented based on a type- length-value, TLV, format, the type comprises an opcode denoting that the respective message is a capabilities messages, the length denotes the first dynamic indication, and the value denotes the second dynamic indication and corresponding capabilities data.

The TLV and/or TVLV format enables one or more of: searching sequences of the messages using generalized parsing functions, data of capabilities within each message, including the second and/or third dynamic implementations, may be placed in any order within the message. Data within the message may be in binary format, which makes parsing faster and/or the data smaller.

In a further implementation form of the first and second aspects, the at least one message of the first message type comprises a plurality of messages, wherein the first dynamic indication of the dynamic message format denotes a total number of capabilities included in the plurality of messages, and wherein the dynamic message format further includes a fourth dynamic indication denoting a number of capabilities of the total number of capabilities of the first dynamic indication that are included in the respective message.

The SW/HW may prepare to receive an amount of data according to the remaining messages formatted according to the dynamic message format. Missed messages formatted according to the dynamic message format and/or remaining messages formatted according to the dynamic message format may be polled, until all messages have been received.

In a further implementation form of the first and second aspects, a start location within the respective message of the data associated with each respective capabilities is set to a fixed alignment value.

In a further implementation form of the first and second aspects, in response to the data associated with a certain respective capability being less than the fixed alignment value, the data associated with the certain respective capability is padded to reach the fixed alignment value.

The fixed alignment value optimizes processing of the capabilities of the respective message formatted according to the dynamic message format. For example, capabilities may be split according to the fixed alignment value, for parallel negotiation of activation.

In a further implementation form of the first and second aspects, the dynamic message format further comprises an opcode denoting a capabilities message and an indication selected from the group of: a first message, a middle message, a last message, and an only message.

In a further implementation form of the first and second aspects, the opcode is located at a start region of the dynamic message format.

In a further implementation form of the first and second aspects, at least one capability of the dynamic message format includes an opcode for exchange of at least one user register of the I/O device, and one or more of: register names of the at least one user register, register offset of the at least one user register, masks of the at least one user register, field sizes of the at least one user register, and bit locations of the at least one user register. The dynamic message format may be used to change existing user registers.

In a further implementation form of the first and second aspects, the dynamic message format includes a fifth dynamic indication denoting an overall version of a package of the plurality of capabilities included in the at least one first message. The capability version of the fifth dynamic indication may assist with internal features and/or bug fixes, for example, to allow code bypass on the PF and/or VF. The capability version of the fifth dynamic indication may provide easier backwards compatibility, in particular where only one side is upgraded, i.e., when only one of the VF and the PF are upgraded. New capabilities may be defined by newer versions, which may have distinctive fields for setting. Versions may be defined for each package of capabilities, making it is easy to negotiate the supported version between a new side and an old side. According to a third aspect, a computer-readable storage medium storing program code is provided. The program code comprising instructions, which when performed on a computer cause the computer to perform the method according to the second aspect.

Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosure pertains.

Some embodiments of the disclosure are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the disclosure. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the disclosure may be practiced.

The present disclosure, in some embodiments thereof, relates to virtual machines (VM), and, more specifically, but not exclusively, to communication between VMs and virtualized resources of physical hardware devices.

An aspect of some implementations of the methods, apparatus, and/or code instructions described herein relates to a dynamic message format for messages transmitted between virtual function (VF) drivers implemented on VMs and a physical function (PF) function driver corresponding to a hardware device, optionally an input/output (I/O) device, for example, a network interface card (NIC), graphic adapter cards, and storage accelerator devices. The dynamic message format includes a first dynamic indication denoting a number of capabilities included in the respective message, and multiple second dynamic indications each associated with one respective capability. The number of second dynamic indications may correspond to the number of capabilities include in the respective message, i.e., one second dynamic indication per capability. Each respective dynamic second indication denotes a size of data stored in the respective message that is associated with the corresponding respective capability.

The messages are transmitted between the VF drivers and the PF driver during a phase of negotiating capabilities which are supported by the PF driver and the I/O device, and/or capabilities which are desired to be implemented on the I/O device by the VF drivers. In an exemplary negotiating session, one or more message of a first message type, which are formatted according to the dynamic message format, are transmitted from the VF driver to the PF driver. The first message type includes a set of requested capabilities supported by the VF driver for enablement on or by the I/O device. Second message(s) types are transmitted from the PF to the VF. The second message type(s) may be formatted according to the dynamic message format. The second message type(s) includes a set of supported capabilities selected from the requested capabilities that are supported by the I/O device and PF driver for implementation thereon. The set of supported capabilities are enabled by the I/O device and the PF driver for use by the VF driver.

Supported capabilities may include supported protocols, supported features, and/or supported technologies, for example, remote direct memory access (RDMA), receive side scaling (RSS).

At least some implementations of the methods, apparatus, and/or code instructions described herein address the technical problem of discovering capabilities of physical hardware devices for use by virtual machines, in particular, discovering capabilities provided by physical function drivers to virtual function drivers associated with VMs. In standard implementations, the structure of messages transferred between VF drivers and PF drivers is fixed or semi-fixed, limiting the number of capabilities which are negotiated, and limiting the parameters of the capabilities. The fixed structure of the messages lacks the flexibility and/or ability to scale. The problem is especially pronounced in datacenters, such as cloud services, where a large number of VMs are executed (e.g., thousands). Moreover, code executing on the VMs may be imported by external customers using the cloud service. Maximizing compatibility between the capabilities of the hardware devices available to the VM and capacities desired by the VM helps improve performance by maximizing the number of capabilities and/or parameters of the capabilities that the VMs are able to execute on the underlying hardware devices.

At least some implementations of the systems, methods, code instructions described herein provide a technical solution to the technical problem, by providing a dynamic message structure that accommodates any number of capabilities for negotiation and/or any size and/or any number of parameters of the capabilities. The dynamic messages structure is flexible and/or scales to increasing number of candidate capabilities and/or candidate parameters of the capabilities.

Before explaining at least one embodiment of the disclosure in detail, it is to be understood that the disclosure is not necessarily limited in its application to the details of construction and the arrangement of the components and/or methods set forth in the following description and/or illustrated in the drawings and/or the Examples. The disclosure is capable of other embodiments or of being practiced or carried out in various ways.

The present disclosure may be a system, a method, and/or a computer program product.

In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

Reference is now made to <FIG>, which is a block diagram of components of a computing device <NUM> that implements the dynamic message format for negotiating capabilities in a virtual environment, in accordance with some embodiments. Reference is also made to <FIG>, which depicts exemplary dataflow between components of computing device <NUM> described with reference to <FIG>, in accordance with some embodiments. Reference is also made to <FIG>, which is a schematic depicting an exemplary message <NUM> formatted in the dynamic message format, in accordance with some embodiments. The dynamic message format scales for easy extension of capabilities. Messages transmitted by components of computing device described with reference to <FIG> may be formatted according to the dynamic message format described with reference to <FIG>. Reference is also made to <FIG>, which is a flowchart of a method for negotiating capabilities between a VF driver and a PF driver using messages formatted in the dynamic message format, in accordance with some embodiments. Reference is also made to <FIG>, which depicts an exemplary virtualized environment <NUM> in which messages formatted according to the dynamic message format are implemented, in accordance with some embodiments. The features described with reference to <FIG> and/or <NUM> may be implemented by components of computing device <NUM> described with reference to <FIG> and/or <FIG>, using the dynamic message format described with reference to <FIG>, for example, by processor(s) <NUM> of a computing device <NUM> executing code instructions (e.g., code 106A) stored in a memory <NUM>.

Computing device <NUM> may be implemented as, for example, one or more of: a computing cloud, a single computing device (e.g., client terminal), a group of computing devices arranged in parallel, a network server, a local server, a remote server, a client terminal, a mobile device, a stationary device, a kiosk, a smartphone, a laptop, a tablet computer, a wearable computing device, a glasses computing device, a watch computing device, and a desktop computer.

Processor(s) <NUM>, implemented as for example, central processing unit(s) (CPU), graphics processing unit(s) (GPU), field programmable gate array(s) (FPGA), digital signal processor(s) (DSP), application specific integrated circuit(s) (ASIC), customized circuit(s), processors for interfacing with other units, and/or specialized hardware accelerators. Processor(s) <NUM> may be implemented as a single processor, a multi-core processor, and/or a cluster of processors arranged for parallel processing (which may include homogenous and/or heterogeneous processor architectures).

Memory <NUM> stores code instructions implementable by processor(s) <NUM>. Memory <NUM> is implemented as, for example, a random access memory (RAM), read-only memory (ROM), and/or a storage device, for example, non-volatile memory, magnetic media, semiconductor memory devices, hard drive, removable storage, and optical media (e.g., DVD, CD-ROM).

Memory <NUM> may store virtual machine manager (VMM) <NUM> that manages and/or runs one or more virtual machines (VM) <NUM>. VMM <NUM> may be implemented as a hypervisor. VMM <NUM> may be implemented in hardware, software, firmware, and/or combination of the aforementioned.

Each VM <NUM> executes one or more virtual function (VF) drivers <NUM>.

Computing device <NUM> includes and/or is in communication with one or more I/O devices <NUM>. Each I/O device <NUM> is associated with one or more physical function (PF) drivers <NUM>. The I/O devices <NUM> may be input/output (I/O) devices, for example, universal serial bus, network adapter (for wired and/or wireless networks), network interface card, accelerator, graphics devices, and storage device.

Computing device <NUM> may include and/or be in communication with one or more data storage devices <NUM>. Data storage devices <NUM> may store, for example, different versions of VMs, VFs, PF, and/or VMM, and/or other data. It is noted that code instructions may be selectively loaded from data storage device <NUM> into memory <NUM> for execution by processor(s) <NUM>. Data storage device(s) <NUM> may be implemented as, for example, a memory, a local hard-drive, a removable storage unit, an optical disk, a storage device, and/or as a remote server and/or computing cloud (e.g., accessed via a network connection).

Computing device <NUM> may be in communication with a network <NUM>, for example, the internet, a local area network, a virtual network, a wireless network, a cellular network, a local bus, a point to point link (e.g., wired), and/or combinations of the aforementioned.

Computing device <NUM> may include a network interface for communication with network <NUM>. The network interface may be implemented as one or more of I/O devices <NUM> with associated PF driver(s) <NUM>. The network interface may be virtualized, for use by multiple VMs <NUM> via corresponding executed VF drivers <NUM>. For example, different VMs <NUM> may access network <NUM> by VF driver(s) <NUM> used to access PF driver(s) <NUM> of the network interface.

Computing device <NUM> may include and/or be in communication with one or more physical user interfaces <NUM> that include a mechanism for user interaction, for example, to enter data (e.g., select a version of the VF to load) and/or to view data (e.g., view a message indicating which capabilities have been negotiated between the VF and PF).

Exemplary physical user interfaces <NUM> include, for example, one or more of, a touchscreen, a display, gesture activation devices, a keyboard, a mouse, voice activated software using speakers and microphone, and an orchestrator sending data over a network interface.

Referring now back to <FIG>, I/O device <NUM> is virtualized via PF driver <NUM>. The physical resources of I/O device <NUM> are partitioned, and made available for access and use by multiple VMs 110A-D via corresponding executed VF drivers 112A-D. Each VF driver 112A-D communicates with a PF <NUM> associated with I/O device <NUM>. The VF driver 112A-D may be agnostic to the actual underlying I/O device <NUM> that is being used.

As described herein, messages are transmitted between VF drivers 112A-D and PF driver <NUM> for negotiation of capabilities. Each VF driver 112A-D may negotiate its own set of desired capabilities with the capabilities supported by PF <NUM> and/or I/O device (e.g., HW) <NUM>.

VF drivers <NUM> and PF drivers <NUM> are defined, for example, by the PCI express (PCIe) format, in particular, a single root input/output (I/O) virtualization (SR-IOV) feature of the PCIe. SR-IOV enables the portioning of the resources of I/O device <NUM> (e.g., network adapter) for use by multiple VMs 110A-D. For example, memory and/or network ports of I/O device <NUM> such as a network adapter are partitioned and shared by VF drivers 112A-D. Network traffic streams flowing between each VF 112A-D and PF <NUM> may be differentiated and identified, allowing memory and/or interrupt translation between the respective VF 112A-D and PF <NUM> to be applied. The network traffic streams may flow between PF <NUM> and each respective VF 112A-D without affecting the VFs. The network traffic may achieve performance that is close to the bare metal (i.e., non-virtualized) performance.

Optionally, VF drivers 112A-D are implemented as Adaptive Virtual Function (AVF) drivers and/or assignable device interface (ADI). As used herein, the term AVF may sometimes refer to, and/or be interchangeable with, the term ADI. Multiple instances of the AVF and/or ADI are implemented on multiple VMs <NUM>. The instances of the AVF and/or ADI VF are mapped to a single common PF driver <NUM>. The instances of the AVF and/or ADI are concurrently and similarly adapted. The AVF adapts to changing capabilities of the physical function driver (PF) with which it is associated. Using the AVF enables updating the PF without necessarily updating all the associated VF drivers. The AVFs have a single common device ID and branding string. Adaptive Virtual Function (AVF) may be implemented as an SR-IOV Virtual Function with the same device ID on different vendor I/O devices <NUM>. In some implementations, the AVF driver is a VF driver which supports all future vendor devices without requiring a VM update. For the AVF driver, every new drop of the VF driver adds additional advanced capabilities that may be activated in the VM when the underlying HW device supports those advanced capabilities, based on a device agnostic way without compromising on the base functionality.

Referring now back to <FIG>, the dynamic message format includes a first dynamic indication <NUM> ("Total number of capabilities") indicating a number of capabilities included in the respective single or plurality of messages formatted in the dynamic message format. The total number of capabilities may represent a package of multiple capabilities.

The first indication, which is dynamic, does not lock the dynamic message format to a specific size, allowing new/old drivers to communicate dynamically. The size of the dynamic message format may change to any size needed to define any number of supported capabilities. Additional capabilities may be added to the VF driver and/or PF driver, without necessarily requiring adjustment of the driver code.

The dynamic message format includes one or multiple second dynamic indications <NUM> ("Capability #N Data Length") each associated with one respective capability. The number of second dynamic indications <NUM> corresponds to the number of capabilities <NUM>, optionally, the number of capabilities on the current message. Each respective dynamic second indication <NUM> indicates a size of data stored in the respective message that is associated with the corresponding respective capability. Each respective capability is associated with its own variable data structure (i.e., the second dynamic indication) that stores data regarding which details of the respective capability are supported and/or which details are not.

Each second dynamic indication <NUM> is associated with a dynamic capability data indication <NUM> ("Capability #N Data") that stores the data of the respective capability. The size of the data in dynamic capability data indication <NUM> is according to the corresponding second dynamic indication <NUM>.

Optionally, the dynamic message format includes one or more third dynamic indications <NUM> ("Capability #N Version") each associated with one respective capability. The number of third dynamic indications <NUM> corresponds to the number of capabilities <NUM>, optionally, the number of capabilities on the current message. Each respective third dynamic <NUM> indication indicates a version of the corresponding respective capability. The version may be, for example, one or more of: a major version, a minor version, and a minor sub version.

Optionally, the dynamic message format includes a fifth dynamic indication <NUM> indicative an overall version of the package of capabilities included in the respective message(s). Fifth dynamic indication <NUM> may be formatted the same or similarly to third dynamic indications <NUM>.

The capability version of the third and/or fifth dynamic indication may assist with internal features and/or bug fixes, for example, to allow code bypass on the PF and/or VF.

The capability version of the third and/or fifth dynamic indication may provide easier backwards compatibility, in particular where only one side is upgraded, i.e., when only one of the VF and the PF are upgraded. New capabilities may be defined by newer versions, which may have distinctive fields for setting. Versions may be defined for each package of capabilities, making it is easy to negotiate the supported version between a new side and an old side.

Capabilities may be individually selected for activation, for example, where the respective capability is a newer version of an older existing capabilities. Capabilities for which the version is irrelevant and/or unchanged do not necessarily need to be activated (since they are already activated and/or are irrelevant).

Optionally, the number of capabilities in first dynamic indication <NUM> are included in a single message. Alternatively, the total number of capabilities in first dynamic indication <NUM> are divided into multiple separate messages. In such implementation, first dynamic indication <NUM> indicates a total number of capabilities included in the multiple messages, and a fourth dynamic indication <NUM> indicates the number of capabilities (out of the total number of capabilities of first dynamic indication <NUM>) that are included in each respective message of the multiple messages. When the capabilities do not fit into one messages, multiple messages may be sent. The total number of capabilities may be tracked for activation. When one or more of the total messages are missing, the missing messages with missing capabilities may be requested for activation thereof.

Optionally, dynamic message format <NUM> includes an Opcode <NUM> ("Capabilities Message OPCODE") indicating that the respective message is a capabilities messages. Opcode <NUM> may be for the message as a whole, i.e., impacting the set of capabilities associated with the respective message. Opcode <NUM> may include an indication that the respective message is one of: a first of multiple messages, a middle of multiple message, a last of multiple, messages, or an only (i.e. single) message. Opcode <NUM> may be located at a start region of each respective message.

Optionally, dynamic message format <NUM> includes multiple capabilities opcodes <NUM> ("Capability #N Opcode), each associated with a respective capability. The value of each opcode <NUM> may be set for different features. For example, at least one capability opcode <NUM> includes a value indicating exchange of user register(s) of the I/O device, and one or more of: register names of the user register(s), register offset of the user register(s), masks of the user register(s), field sizes of the user register(s), and bit locations of the user register(s). The dynamic message format may be used to change existing user registers. An example for changing a user register is now provided:.

The above data may be stored in a message implemented using the dynamic message as follows: <NUM><NUM>. <NUM><NUM>,<NUM><NUM><NUM><NUM><NUM><NUM><NUM>.

Dynamic message format <NUM> may be implemented based on a type-length-value (TLV) format. The type may be implemented as an opcode <NUM> denoting that the respective message is a capabilities messages and/or type may be implemented as capabilities opcode <NUM> for each respective capability. The length is implemented as the first dynamic indication <NUM>. The value is implemented as the second dynamic indication <NUM> and corresponding capabilities data field <NUM>. Alternatively or additionally, dynamic message format <NUM> is implemented based on a type-version-length-value (TVLV) format. The version may be implemented as third dynamic indication <NUM> and/or fifth dynamic indication <NUM>.

The TLV and/or TVLV formats may be nested and/or sequentially implemented, for example, the TLV and/or TVLV format may be implemented for the message as a whole, and the TLV and/or TVLV format may be implemented for each capability.

Optionally, dynamic message format <NUM> is set to a fixed alignment, for example, fixed to a byte and/or word and/or double-word, and/or other alignment format, for example, <NUM> bytes. A start location within the respective message of the data stored in field <NUM> associated with each respective capabilities may be set to the fixed alignment value. In response to the data associated with a certain respective capability being less than the fixed alignment value, the data associated with the certain respective capability (i.e., stored in field <NUM>) may be padded to reach the fixed alignment value. The padding may be using irrelevant data, for example, zeros or other characters that do not represent actual data.

Referring now back to <FIG>, at <NUM>, the VF driver and/or ADI is initialized (the term VF driver and ADI may be interchanged). For example, the VM may initialize the VF driver corresponding to desired capabilities of the I/O device, for example, a new version of the VF driver is loaded to provide additional capabilities for enablement on the I/O device. Initialization may occur, for example, upon reset and/or boot-up of the VM, the VF driver, the PF driver, and/or the hardware device.

At <NUM>, communication between the VF driver and the PF driver corresponding to the I/O device is initiated. Optionally, the VF driver initiates communication with the PF driver.

At <NUM>, the VF driver sends a first message type formatted in the dynamic message format (e.g., as described herein, for example with reference to <FIG>) to the PF driver with requested capabilities and/or requested parameter of the capabilities supported by the VF driver for enablement on or by the I/O device. Alternatively, the message with requested capabilities and/or requested parameter of the capabilities is formatted according to a standard fixed and/or semi-fixed format. The VF may ask to enable the capabilities in the first message type on the hardware device for the VF and/or expose virtual interfaces of the hardware device for the VF.

The capabilities may include a set of base capabilities set and/or another set of advanced capabilities. Some implementations of the VF driver (e.g., AVF) define a minimum set of features referred to as "base mode," but may provide additional advanced capabilities depending on what capabilities are available in the PF with which the AVF is associated. The base mode includes a set of basic capabilities on the PF driver and the I/O device for use by the VF driver. The basic capabilities may be assumed to be available on hardware devices, as a standard. Examples of basic capabilities for a wired network adapter include basis receive and transmit queues for data transfer, signaling events to software, basic stateless operations such as checksum, segmentation offload, jumbo frames, RSS, and the like.

At <NUM>, the VF driver and the PF driver may enable the set of base capabilities, for operating in the base mode. In such implementation, advanced capabilities are enabled as described with reference to <NUM>. Alternatively, no base mode is initially set-up. In such implementation, there is no difference necessarily between the base capabilities and the advanced capabilities, with both types of capabilities being enabled as described with reference to <NUM>.

At <NUM>, the VF driver sends a first message type formatted in the dynamic message format to the PF driver with requested advanced capabilities and/or requested parameter of the advanced capabilities supported by the VF driver for enablement on or by the I/O device. When base mode is enabled, the capabilities in the first message type are advanced capabilities. When no base mode is enabled, <NUM> and <NUM> are not necessarily implemented. The capabilities in the first message type are the requested capabilities.

There may multiple first message types that include the requested capabilities, which may be set as a sequence. In such a case, the opcode (e.g., <NUM> as described with reference to <FIG>) may be set accordingly for each message. The opcode for the first message type of the sequence is set to the value indicating FIRST. The opcodes for the middle messages of the sequence are set to the value indicating MIDDLE. The opcode for the last message of the sequence is set to the value indicating LAST. When the requested capabilities are included in a single first message type, the opcode is set to the value indicating ONLY. The total number of capabilities (e.g., <NUM> field of <FIG>) and the number of capabilities on current message field (e.g., <NUM> field of <FIG>) are set according to the capabilities included in each respective message. The SW/HW may prepare to receive an amount of data according to the remaining messages formatted according to the dynamic message format. Missed messages formatted according to the dynamic message format and/or remaining messages formatted according to the dynamic message format may be polled, until all messages have been received.

At <NUM>, the sequence of first message types is received (e.g., by the PF driver) and processed. The values of opcode fields <NUM>, total number of capabilities <NUM> field, and the number of capabilities on current message field <NUM>, may be evaluated to determine if all messages of the sequence have been received, or if one or more messages have not been received. That one or more messages are missing may be detected, for example, when the FIRST message or the LAST message have not been received. In another example, that one or more messages are missing may be detected when a number of capabilities in the received messages (i.e., sum of capabilities in in field <NUM> as in <FIG>) is less than the total number of capabilities in the received messages (i.e., value in <NUM> field of <FIG>).

When all capabilities in the received first message type(s) have been received but the total number of capabilities defined in the first dynamic indication has not been reached, it is in an indication that some capabilities are missing. When one or more messages of the sequence have not been received, a poll for the additional first message types may be generated, to help ensure that the entire set of capabilities have been received. For example, after a timeout interval has expired (or other condition is met), and some messages are missing. Alternatively, an error is generated in response to not all messages being received and/or when the received sizes (e.g., according to the number of defined bytes) in the messages is invalid. A message indicating the error may be sent to the VF driver. The error message may indicate, for example, missing message(s), error in capabilities, and error in size.

At <NUM>, each one of the capabilities in the received first type of messages is independently processed. The capabilities may be sequentially processed until the total number of capabilities is reached.

The first dynamic indication enables detecting when all capabilities in the received first message type(s) have been negotiated.

One or more second message type are transmitted from the PF to the VF. The second message type includes a set of capabilities that are supported by the I/O device and/or PF driver for implementation thereon. Capabilities not supported by the I/O device and/or PF driver may be denied. The set of capabilities may be selected from the requested capabilities provided in the first message type by the VF.

The second message type(s) may be formatted in the dynamic messages format. Alternatively, another format is used for the second message type(s), for example, an array, where each element of the array corresponds to the capabilities of the first message type. The value in each element of the array is binary, indicating whether the PF and/or hardware supports or does not support the capability corresponding to the array element.

At <NUM>, each capability (e.g., advanced feature) is enabled (e.g., activated) by the I/O device and the PF driver for use by the VF drive. The capability may be according to the base (when the base version exists).

Optionally, the capabilities are enabled according to one or more third message types that are transmitted from the VF to the PF. The third message type may be formatted according to the dynamic messages format. The third message type includes a set of enabled capabilities selected from the supported capabilities (e.g., in the second message type) for enablement by the I/O device and/or the PF driver for implementation thereon.

At <NUM>, features described with reference to <NUM> and/or <NUM> are iterated. The iterations may be performed to enable each respective capability, or not to enable the respective capability.

Optionally, each capability is independently processed for enablement and/or non-enablement. The enablement or non-enablement of one capability does not necessarily depend on enablement or non-enablement of another capability.

At <NUM>, the enablement of the capabilities is complete. The VF driver may use the enablement capabilities.

Referring now back to <FIG>, a scalable IOV setup (S-IOV) may be implemented in virtualized environment <NUM>.

Virtualized environment <NUM> includes one or more of the following exemplary components:
A user space <NUM> that includes one or more VMs <NUM>, each implementing a virtual network interface controller (vNIC) driver <NUM> that communicates with ADI <NUM>. Dynamic messages formatted according to the dynamic message format described herein may be transmitted between vNIC <NUM> to PF <NUM> via ADI <NUM>, for example, as described with reference to the VF drivers and PF drivers.

A hypervisor/host kernel space <NUM> that includes a PF driver <NUM>, a virtual device (VDEV) <NUM> implementing one or more assignable device interfaces (ADI) <NUM>, and a virtual device composition module (VDCM) <NUM>. Virtual device composition module <NUM> may provide software managed resource mapping between VDEV <NUM> and ADIs <NUM>.

It is noted there are no VF implemented in scalable IOV virtualized environment <NUM>. ADIs are used instead.

An input/output memory management unit (IOMMU) <NUM> and direct memory access (DMA) with Process Address Space ID (PASID) remap code <NUM> implements one or more ADIs <NUM> and a PF <NUM>. Each ADI <NUM> is tagged with a unique PASID.

A I/O device is implemented as a network card (NETC) <NUM>. NETC <NUM> may be compliant with PCIe. The hardware resources of NETC <NUM> are virtualized via ADI <NUM>, enabling multiple VMs <NUM> to each access its own assigned ADI <NUM> to receive and/or transmit traffic via an attached network, for example, via one or more corresponding queues (Q) <NUM>.

NETC <NUM> includes a set of PF base address registers (BARs) <NUM> that map to ADIs <NUM>. NETC <NUM> may include additional registers, for example a PF configuration (Conf) register <NUM>, and an MSI-X register <NUM>.

Virtualizing the resources of the NETC <NUM> using software imposes a large overhead, reducing overall efficiency of the host.

Virtual device composition module <NUM> may provide a slow path enumeration (denoted by dashed lines <NUM>) and a fast path enumeration (denoted by solid lines <NUM>) pass-through. Slow path enumeration (denoted by dashed lines <NUM>) is by software. Fast path enumeration (denoted by solid lines <NUM>) is by hardware that is provisioned for direct access. DMA provides isolation between fast paths. Each vNIC driver <NUM> is provided with an isolated fast path via DMA to partitioned resources of NETC <NUM> via its assigned ADIs <NUM> using the corresponding unique PASID.

Messages (e.g., of the first, second, and/or type message types, as described herein) transmitted between vNIC drivers <NUM> and PF driver <NUM> for enablement of capabilities requested by vNIC driver <NUM> that are supported by the NETC <NUM> and the PF driver <NUM> for use by vNIC driver <NUM>.

The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

It is expected that during the life of a patent maturing from this application many relevant drivers will be developed and the scope of the term driver is intended to include all such new technologies a priori.

Any particular embodiment of the disclosure may include a plurality of "optional" features unless such features conflict.

Throughout this application, various embodiments of this disclosure may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the disclosure.

Claim 1:
A computing device (<NUM>) in communication with an input/output, I/O, device (<NUM>) providing virtualized hardware resources for use by a plurality of virtual machines, VM (<NUM>), wherein the computing device is configured to:
execute a virtual machine manager, VMM, (<NUM>) that implements a physical function, PF, driver (<NUM>) corresponding to the I/O device and to a plurality of virtual function, VF, drivers (<NUM>) implemented by the plurality of VMs;
receive at least one message of a first message type transmitted from a virtual function, VF, driver of a VM to the PF driver,
wherein the at least one message of the first message type is formatted in a dynamic message format (<NUM>) that includes: a first dynamic indication denoting a number of a plurality of capabilities (<NUM>), and a plurality of second dynamic indications (<NUM>) each associated with one respective capability, each respective dynamic second indication (<NUM>) is associated with a dynamic capability data indication (<NUM>) that stores data of the respective capability, wherein a size of the data stored in the dynamic capability data indication (<NUM>) is denoted by the corresponding second dynamic indication,
wherein the first dynamic indication does not lock the dynamic message format to a specific size and the size of the dynamic message format is adapted to change to any size needed to define any number of supported capabilities,
wherein the at least one message of the first message type includes a set of requested capabilities supported by the VF driver for enablement on or by the I/O device;
transmit at least one message of a second message type from the PF driver to the VF driver,
wherein the at least one message of the second message type includes a set of supported capabilities selected from the requested capabilities that are supported by the I/O device and PF driver for implementation thereon; and
enable the set of enabled capabilities by the I/O device and the PF driver for use by the VF driver.