Collecting quality of service statistics for in-use child physical functions of multiple physical function non-volatile memory devices

The disclosed technologies provide functionality for collecting quality of service (“QoS”) statistics for in-use child physical functions of multiple physical function (“PF”) non-volatile memory devices (“MFNDs”). A host computing device creates a child PF on a MFND and configures the child PF on the MFND to provide a specified QoS level to an associated VM executing on the host computing device. The MFND then collects child PF QoS statistics for the child PF that describe the utilization of resources provided by child PF to an assigned VM. The MFND provides the child PF QoS statistics from the MFND to the host computing device. The collected child PF QoS statistics can be utilized to inform decisions regarding reallocation of MFND-provided resources, provisioning of new MFND-provided resources, and for other purposes.

BACKGROUND

Non-Volatile Memory Express (“NVMe”) is an open host controller interface and storage protocol specification for accessing non-volatile storage devices attached via a Peripheral Component Interconnect Express (“PCIe”) bus. Certain NVMe devices can expose multiple PCIe physical functions (“PFs”), such as independent NVMe controllers. These types of devices might be referred to herein as multiple physical function non-volatile memory devices (“MFNDs”).

In a MFND, one PF, which might be referred to herein as a “parent PF,” can act as a parent controller to receive and execute administrative commands. Other physical functions on a MFND, which might be referred to herein as “child PFs” or “children PFs,” can act as child controllers that behave similarly to standard NVMe controllers. Through this mechanism, a MFND can enable the efficient sharing of input/output (“I/O”) resources between virtual machines (“VMs”) or bare metal instances. For example, child PFs can be directly assigned to and utilized by different VMs through various direct hardware access technologies, such as HYPER-V NVMe Direct or Discrete Device Assignment (“DDA”).

Through the mechanism described above, the child PFs exposed by a single MFND can appear as multiple, separate physical devices to individual VMs. This allows individual VMs to directly utilize a portion of the available non-volatile storage space provided by a MFND with reduced central processing unit (“CPU”) and hypervisor overhead.

Existing MFNDs, however, have limitations that restrict aspects of their functionality when used with VMs in the manner described above. As one specific example, it might not be possible to obtain detailed information regarding a VM's usage of the resources allocated to it by a MFND. Consequently, system administrators might not know when a VM is over or under-utilizing the resources provided by a MFND and, as a result, might not be able to make informed decisions regarding reallocating those MFND-provided resources or provisioning new MFND-provided resources. Current MFNDs can also suffer from other technical limitations, some of which are described in detail below.

It is with respect to these and other technical challenges that the disclosure made herein is presented.

SUMMARY

Technologies are disclosed herein for collecting Quality of Service (“QoS”) statistics for in-use child PFs on MFNDs. Through implementations of the disclosed technologies, MFNDs can be configured to collect QoS statistics, referred to herein as “child PF QoS statistics,” for in-use child physical functions that describe the utilization of resources provided by the child PFs to VMs. The collected child PF QoS statistics can then be utilized to inform decisions regarding reallocation of MFND-provided resources and provisioning of new MFND-provided resources, thereby making more efficient utilization of MFND hardware. The child PF QoS statistics can also be collected in a manner that reduces the performance impact of collecting the child PF QoS statistics on the MFND and minimizes the use of non-volatile memory. Other technical benefits not specifically mentioned herein can also be realized through implementations of the disclosed subject matter.

As discussed briefly above and in further detail below, the disclosed technologies include functionality for collecting QoS statistics for in-use child PFs of an MFND. In order to provide this functionality, a host computing device creates a child PF on a MFND and configures the child PF on the MFND to provide a specified QoS level to an associated VM executing on the host computing device. The host computing device also enables the MFND to collect child PF QoS statistics for the child PF.

The collected child PF QoS statistics describe the utilization of resources provided by child PFs to assigned VMs. The MFND provides the child PF QoS statistics from the MFND to the host computing device. As discussed above, the collected child PF QoS statistics can then be utilized to inform decisions regarding reallocation of MFND-provided resources, provisioning of new MFND-provided resources, and potentially other types of decisions.

In one embodiment, the specified QoS level defines maximum read input/output (“I/O”) operations per second (“IOPS”) and maximum write IOPS for the child PF. In this embodiment, the child PF QoS statistics for the child PF might specify a percentage of the maximum read IOPS and the maximum write IOPS provided by the child PF to the VM assigned to the child PF during a specified monitoring period.

The specified QoS level might also, or alternately, define a maximum read bandwidth and a maximum write bandwidth for the child PF. In this case, the child PF QoS statistics for the child PF might specify a percentage of the maximum read bandwidth and the maximum write bandwidth provided by the child PF to the VM assigned to the child PF during the specified monitoring period.

In other embodiments, the child PF QoS statistics for the child PF specify the percentage of read operations and write operations performed by the child PF during the specified monitoring period. The child PF QoS statistics for the child PF might also, or alternately, specify a size of I/O workloads performed by the child PF on behalf of an assigned VM during the monitoring period. The child PF QoS statistics might also, or alternately, specify an amount of the storage capacity of a non-volatile memory device on the MFND that is in use by a VM. Other types of child PF QoS statistics can be collected in the manner described herein in other embodiments.

In some embodiments, the host computing device specifies the duration of a QoS statistics monitor period and the duration of a QoS statistics swap bucket period to the MFND. In these embodiments, the MFND is further configured to store the child physical function QoS statistics in a log, which might be referred to herein as the “active log,” during the duration of the QoS statistics monitor period. When the QoS statistics swap bucket period elapses, the MFND swaps the active log with another log, which might be referred to herein as the “save log.” In these embodiments, the MFND provides the child PF QoS statistics from the MFND to the host computing device from the save log. In some embodiments, the MFND also generates a notification, such as an asynchronous event, to the host computing device when the QoS statistics swap bucket period elapses. In response thereto, the host computing device may request the child PF QoS statistics from the MFND.

It should be appreciated that the above-described subject matter can be implemented as a computer-controlled apparatus, a computer-implemented method, a computing device, or as an article of manufacture such as a computer-readable medium. These and various other features will be apparent from a reading of the following Detailed Description and a review of the associated drawings.

DETAILED DESCRIPTION

The following detailed description is directed to technologies for collecting QoS statistics for in-use child PFs on MFNDs. As discussed briefly above, MFNDs implementing the disclosed technologies can collect child PF QoS statistics for in-use child PFs that describe the utilization of resources provided by the child PFs to VMs. The child PF QoS statistics can be collected and stored in a manner that reduces the performance impact of collecting the child PF QoS statistics on the MFND and minimizes the use of volatile and non-volatile memory.

Through the use of the disclosed functionality, child PF QoS statistics can be utilized to inform decisions regarding reallocation of MFND-provided resources and provisioning of new MFND-provided resources, thereby making more efficient utilization of MFND hardware. Other technical benefits not specifically mentioned herein can also be realized through implementations of the disclosed subject matter.

While the subject matter described herein is presented in the general context of NVMe multiple physical function devices, those skilled in the art will recognize that the technologies disclosed herein can be used with other types of multiple physical function devices, including other types of multiple physical function non-volatile memory devices. Those skilled in the art will also appreciate that the subject matter described herein can be practiced with various computer system configurations, including host computers in a distributed computing environment, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, computing or processing systems embedded in devices (such as wearable computing devices, automobiles, home automation etc.), minicomputers, mainframe computers, and the like.

In the following detailed description, references are made to the accompanying drawings that form a part hereof, and which are shown by way of illustration specific configurations or examples. Referring now to the drawings, in which like numerals represent like elements throughout the several FIGS., aspects of various technologies for collecting QoS statistics for in-use child PFs on MFNDs will be described.

FIG.1is a computing architecture diagram that shows aspects of the configuration and operation of a MFND102that can implement the embodiments disclosed herein, according to one embodiment. As discussed briefly above, the MFND102is an NVMe Specification-compliant device in some embodiments. The MFND102can be hosted by a host computing device100(which might be referred to herein simply as a “host”), such as a server computer operating in a distributed computing network such as that described below with reference toFIG.8.

As also discussed briefly above, NVMe is an open logical device interface specification for accessing non-volatile storage media. In some embodiments, an NVMe device is accessible via a PCIe bus. In other embodiments, an NVMe device is accessible via a network or other packet-based interface. The NVMe Specification defines a register interface, command set, and collection of features for PCIe-based solid-state storage devices (“SSDs”) with the goals of high performance and interoperability across a broad range of non-volatile memory subsystems. The NVMe Specification does not stipulate the ultimate usage model, such as solid-state storage, main memory, cache memory or backup memory.

NVMe provides an alternative to the Small Computer System Interface (“SCSI”) standard and the Advanced Technology Attachment (“ATA”) standard for connecting and transmitting data between a host computing device100and a peripheral target storage device. The ATA command set in use with Serial ATA (“SATA”) SSDs and the SCSI command set for Serial Attached SCSI (“SAS”) SSDs were developed at a time when hard disk drives (“HDDs”) and tape were the primary storage media. NVMe was designed for use with faster media.

The main benefits of NVMe-based PCIe SSDs over SAS-based and SATA-based SSDs are reduced latency in the host software stack, higher IOPS and potentially lower power consumption, depending on the form factor and the number of PCIe lanes in use.

NVMe can support SSDs that use different types of non-volatile memory, including NAND flash and the3D XPOINT technology developed by INTEL and MICRON TECHNOLOGY. Supported form factors include add-in PCIe cards, M.2 SSDs and U.2 2.5-inch SSDs. NVMe reference drivers are available for a variety of operating systems, including the WINDOWS and LINUX operating systems. Accordingly, it is to be appreciated that the MFND102described herein is not limited to a particular type of non-volatile memory, form factor, or operating system.

As described briefly above, the MFND102described herein includes capabilities for exposing multiple PFs112A-112N to the host computing device100. Each of the PFs112A-112N is an independent NVMe controller in one embodiment. The PFs112A-112N are other types of controllers in other embodiments. At least a plurality of PF112A-112N are independent NVMe controllers and at least one distinct PF112A-112N is a non-NVME controller in other embodiments.

One PF112A in the MFND102, which might be referred to herein as the “parent PF112A” or the “parent controller112A,” acts as a parent controller. In one embodiment, for instance, the parent PF112A is the privileged PCIe function zero of the MFND102. In this regard, it is to be appreciated that the parent controller might be configured as another PCIe function number in other embodiments. The parent PF112A and child PFs described below might also be device types other than NVMe devices in some embodiments.

The parent PF112A can act as a parent controller to receive and execute administrative commands110generated by a root partition108. In particular, and as described in greater detail below, the parent PF112A can manage child PFs112B-112N such as, for example, by creating, deleting, modifying, and querying the child PFs112B-112N. The child PFs112B-112N might be referred to herein interchangeably as “child PFs112B-112N” or “child controllers112B-112N.”

The child PFs112B-112N are regular PCIe physical functions of the MFND102. The child PFs112B-112N can behave like regular and independent NVMe controllers. The child controllers112B-112N can also support the administrative and I/O commands defined by the NVMe Specification.

Through the use of the multiple PFs112A-112N exposed by the MFND102, I/O resources provided by the MFND102can be efficiently shared between VMs104A-104N. For instance, child PFs112B-112N can be directly assigned to different VMs104A-104N, respectively, through various direct hardware access technologies such as HYPER-V NVMe Direct or DDA. In this way, the child PFs112B-112N exposed by a single MFND102can appear as multiple, separate physical devices to individual VMs104A-104N, respectively. This allows individual VMs104A-104N to directly utilize a respective portion of the available storage space provided by a non-volatile memory device103on the MFND102with reduced CPU and hypervisor106overhead.

In some configurations, the host computing device100operates in a distributed computing network, such as that described below with regard toFIG.8. Additionally, the host computing device100executes a host agent116and a management application programming interface (“API”)118in order to enable access to aspects of the functionality disclosed herein in some embodiments.

The host agent116can receive commands from other components, such as other components in a distributed computing network such as that described below with regard toFIG.8, and make calls to the management API118to implement the commands. In particular, the management API118can issue administrative commands to the parent PF112A to perform the various functions described herein. Details regarding various methods exposed by the management API118to the host agent116for implementing the functionality disclosed herein are described below.

In some embodiments, the MFND102has two modes of operation: regular user mode and super administrator mode. In regular user mode, only read-only functions can be executed. The non-read-only management functions described herein (e.g. set the QoS level for a PF, etc.) must be executed in the super administrator mode. If an attempt is made to execute these functions in regular user mode, an error (which might be referred to herein as an “ERROR_ACCESS_DENIED” error) will be returned. The API118exposes methods for getting the device operation mode (which might be referred to herein as the “GetDeviceOperationMode” method) and switching the device operation mode (which might be referred to herein as the “SwitchDeviceOperationMode” method) in some embodiments.

As discussed briefly above, existing MFNDs102have limitations that restrict aspects of their functionality when used with VMs104in the manner described above. As one specific example, it might not be possible to obtain detailed information regarding a VM's104usage of the resources allocated to it by a MFND102. Consequently, system administrators might not know when a VM104is over or under-utilizing the resources provided by a MFND102and, as a result, might not be able to make informed decisions regarding reallocating those MFND-provided resources or provisioning new MFND-provided resources. The technologies presented herein address these and potentially other technical considerations by enabling collection of QoS statistics for in-use child PFs112B-112N of a MFND102. Additional details regarding these aspects will be provided below.

FIG.2Ais a computing architecture diagram showing aspects of one mechanism disclosed herein for creating child PFs112on a MFND102, according to one embodiment. As shown inFIG.2A, the host agent116can create a new child PF112B on the MFND102by calling an appropriate method exposed by the management API118. In response thereto, the management API118issues a command110A to the parent PF112A to create the desired child PF112B. The MFND102, in turn, creates the child PF112B. Thereafter, a VM104A may be assigned to the child PF112B. Additional details regarding the creation of child PFs112B on a MFND102and assignment of a VM104A to a child PF112B will be provided below with regard toFIG.5.

FIG.2Bis a computing architecture diagram showing aspects of one mechanism disclosed herein for setting the QoS level for a child PF112B on a MFND102, according to one embodiment. As discussed briefly above, once a child PF112B has been created in the manner described above with regard toFIG.2A, the MFND102can provide functionality for managing the QoS level provided by the child PFs112B. For example, and without limitation, implementations of the disclosed technologies can also enable a host agent116to query and modify the QoS level provided by a child PF112B of a MFND102.

In some embodiments, the MFND102supports multiple storage service level agreements (“SLAs”). Each SLA defines a different QoS level to be provided by a PF112A-112N. QoS levels that can be supported by child PFs112on the MFND102include, but are not limited to, a “reserve mode” wherein a child PF112is allocated at least a specified minimum amount of bandwidth and IOPS, a “limit mode” wherein a child PF112is allocated at most a specified maximum amount of bandwidth and IOPS, and a “mixed mode” wherein a child PF112is allocated at least a specified minimum amount of bandwidth and IOPS but at most a specified maximum amount of bandwidth and IOPS. Other QoS levels can be implemented in other embodiments.

The embodiments disclosed herein allow the parent PF112A to individually define the QoS level for each child PF112B-112N in a single MFND102. For instance, the parent PF112A might define the minimum and/or maximum bandwidth and/or IOPS to be supported by each child PF112B-112N. In order to provide this functionality, the host agent116can call a method exposed by the management API118. In response to such a call, the management API118issues a command110B to the parent physical function112A that includes QoS settings202for a child PF112B. The child PF112B then utilizes the QoS settings202when processing requests from an assigned VM104A.

One illustrative method for modifying the settings of child PFs112B-112N (which might be referred to herein as the “UpdateChildPhysicalFunctionSettings” method) takes an identifier (e.g. a handle) to a MFND102, an identifier (e.g. a serial number) of a child PF112, and a pointer to a data structure containing the QoS settings202for the child PF112as input. The data structure can include data specifying the resources (e.g. the amount of storage space, namespaces, and interrupt vectors that the identified PF112is to use) and the QoS level that are to be assigned to the identified child PF112. The UpdateChildPhysicalFunctionSettings method returns a success message if the supplied settings were successfully applied to the identified child PF112and otherwise returns an error code.

An illustrative method for querying the settings of child PFs112B-112N (which might be referred to herein as the “QueryChildPhysicalFunctionSettings” method) takes an identifier (e.g. a handle) for a MFND102and an identifier (e.g. a serial number) of a child PF112as input. The QueryChildPhysicalFunctionSettings method returns a pointer to a data structure containing the current settings of the identified child PF112. As discussed above, such a data structure can include data specifying the resources (e.g. the amount of storage space, namespaces, and interrupt vectors that the PF112can use) and QoS settings202that are currently assigned to the identified child PF112.

FIG.2Cis a computing architecture diagram showing aspects of one mechanism disclosed herein for enabling the collection of child PF QoS statistics210by a MFND102, according to one embodiment. As shown inFIG.2C, the host agent116can configure the MFND102to collect child PF QoS statistics210by calling an appropriate method on the management API118. In response thereto, the management API118issues a command110C to the parent physical function112A instructing the MFND102to enable the collection of the child PF QoS statistics210. The MFND102stores the child PF QoS statistics210in a child PF statistics log208in one embodiment. Details regarding the configuration and use of the child PF statistics log208will be provided below with respect toFIGS.3and4.

In one embodiment a single command110C can be utilized to enable collection of child PF QoS statistics210for all in-use child PFs112B-112N. Alternately, per child PF112commands110C can be issued to enable collection of child QoS statistics210by individual child PFs112B-112N.

As also shown inFIG.2C, the command110C specifies a QoS statistics monitor period204and a QoS statistics swap bucket period206in some embodiments. As will be described in greater detail below, the QoS statistics monitor period204specifies the duration of a monitoring period during which the MFND102is to collect the child PF QoS statistics210. In one embodiment, the QoS statistics monitor period204is specified in seconds with a minimum value of 60 seconds and increments of 30 seconds. The QoS statistics monitor period204might be specified in other ways in other embodiments.

The QoS statistics swap bucket period206defines a period of time after which the MFND102is to swap an “active log” with a “save log.” In these embodiments, the MFND102is further configured to store the child physical function QoS statistics210in the active log during the duration of the QoS statistics monitor period204. In one embodiment, the QoS statistics swap bucket period206is specified in minutes, with a minimum value of 30 minutes and a maximum value of 1440 minutes. The QoS statistics swap bucket period206might be specified in other ways in other embodiments. Additional details regard the contents and use of the active and save logs will be provided below with regard toFIGS.3and4.

FIG.2Dis a computing architecture diagram showing aspects of one mechanism disclosed herein for retrieving child PF QoS statistics210from a MFND102, according to one embodiment.FIG.2Dwill be described in conjunction withFIG.3, which is a computing architecture diagram showing aspects of one mechanism disclosed herein for swapping a child PF QoS statistics active log302, which might be referred to simply as the “active log302,” and a child PF QoS statistics save log304, which might be referred to simply as the “save log304,” on a MFND102, according to one embodiment.

As described briefly above, the child PF QoS statistics log208is implemented using two separate logs, the child PF QoS statistics active log302and the child PF QoS statistics save log304, in some embodiments. When the QoS statistics swap bucket period206described above elapses, the MFND102swaps the active log302with the save log304and clears the active log302. This can be performed as an atomic operation in order to avoid corruption of the logs302and304. In these embodiments, the MFND102provides the child PF QoS statistics210from the save log304in response to requests308received from the host computing device100. The MFND102also provides functionality for enabling the host computing device100to retrieve the contents of the active log302in some embodiments.

In some embodiments, the MFND102also generates a notification, such as an asynchronous event306, to the host computing device100when the QoS statistics swap bucket period elapses206. In response to receiving the notification, the host computing device100may issue a command110D to the MFND102to retrieve the child PF QoS statistics210from the MFND100. In response thereto, the MFND102retrieves the child PF QoS statistics210from the save log304and returns the child PF QoS statistics210to the host100in response to the command110D. In turn, the host agent116might provide the child PF QoS statistics210to a remote management system212or another component.

In one embodiment, the specified QoS level defines maximum read IOPS and maximum write IOPS for a child PF112B. In this embodiment, the child PF QoS statistics210for the child PF112B specify the maximum read IOPS and the maximum write IOPS provided by the child PF112B to the VM104A assigned to the child PF112B during the QoS statistics monitor period204.

The maximum read IOPS, and the maximum write IOPS are specified as a percentage of the maximum read IOPS and the maximum write IOPS specified by the QoS level for the child PF112B in some embodiments. By expressing the maximum read IOPS and the maximum write IOPS as a percentage of the maximum read IOPS and the maximum write IOPS specified by the QoS level, the maximum read IOPS and the maximum write IOPS can be expressed using only a single byte, thereby saving space on the non-volatile memory device103.

The specified QoS level might also, or alternately, define a maximum read bandwidth and a maximum write bandwidth for the child PF112B. In this case, the child PF QoS statistics210for the child PF112B specify the maximum read bandwidth and the maximum write bandwidth provided by the child PF112B to the VM104A assigned to the child PF112B during the specified QoS statistics monitor period204.

In some embodiments, the maximum read bandwidth and the maximum write bandwidth are specified as a percentage of the maximum read bandwidth and the maximum write bandwidth specified by the QoS level for the child PF112B. By expressing the maximum read bandwidth and a maximum write bandwidth as a percentage of the maximum read bandwidth and a maximum write bandwidth specified by the QoS level, the maximum read bandwidth and a maximum write bandwidth can be expressed using only a single byte, thereby saving space on the non-volatile memory device103.

In other embodiments, the child PF QoS statistics210for the child PF112B specify a percentage of read operations and write operations performed by the child PF112B during the specified QoS statistics monitor period204. The child PF QoS statistics210for the child PF112B might also, or alternately, specify a size of I/O workloads performed by the child PF112B on behalf of an assigned VM104A during the QoS statistics monitor period204.

The child PF QoS statistics210might also, or alternately, specify an amount of the storage capacity of a non-volatile memory device103on the MFND102that is in use by a VM104A. In these embodiments, the amount of the storage capacity of a non-volatile memory device103on the MFND102that is in use by a VM104A may be obtained from the MFND102by issuing an identified child controller command to a child PF112B to retrieve the Namespace Utilization field (“NUSE”) defined by the NVMe Specification. Other types of child PF QoS statistics210, such as but not limited to read/write I/O command latency and bytes written to media, can be collected in the manner described herein in other embodiments.

FIG.4is a data structure diagram showing an illustrative configuration for the child PF QoS statistics log208maintained by a MFND102, according to one embodiment. As shown inFIG.4, the child PF QoS statistics log208includes the fields402A-402P in the illustrated embodiment. In this regard, it is to be appreciated that the illustrated configuration is merely illustrative, and that other types and configurations of data might be utilized. It is to be further appreciated that a single child PF QoS statistics log208might store the child PF QoS statistics210for all of the in-use child PFS112B-112N on a MFND102or separate child PF QoS statistics logs208might be maintained for each of the in-use child PFS112B-112N.

The field402A stores data indicating a version number identified with the format of the child PF QoS statistics log208. The version number might be modified following changes to the format of the child PF QoS statistics log208.

The field402B stores a sequence number that is incremented whenever an active log302is generated (i.e., after each QoS statistics swap bucket period206elapses). When the value reaches 255 and a new active log302is generated, the value is reset to zero.

The field402C stores data identifying the number of log entries in the child PF QoS statistics log208. As described in greater detail below, the log entries are stored in the fields402G-402J.

The field402D stores data identifying the child PF QoS statistics monitor period204and the field402E stores data identifying the child PF QoS statistics swap bucket period206described above. The field402F stores a timestamp associated with the first log entry in the child PF QoS statistics log208. In one embodiment, the timestamp uses the data format for a timestamp as defined by the NVMe Specification. If the host computing device100does not set the timestamp, this field contains the time since the MFND102last powered up.

As discussed briefly above, the fields402G-402J contain log entries containing the child PF QoS statistics210. In the illustrated example, for instance, each log entry includes fields402M-402P specifying the maximum read IOPS percentage, the maximum write IOPS percentage, the maximum read bandwidth percentage, and the maximum write bandwidth percentage during the monitoring period, respectively. As discussed above, the log entries can include other types of child PF QoS statistics210, some of which were described above, in other embodiments. The field402K contains a version number for the log entries and the field402L stores a globally unique identifier (“GUID”) associated with the log entries.

FIG.5is a flow diagram showing a routine500that illustrates aspects of a method for configuring child PFs112on a MFND102, according to one embodiment disclosed herein. It should be appreciated that the logical operations described herein with regard toFIG.5, and the other FIGS., can be implemented (1) as a sequence of computer implemented acts or program modules running on a computing device and/or (2) as interconnected machine logic circuits or circuit modules within a computing device.

The particular implementation of the technologies disclosed herein is a matter of choice dependent on the performance and other requirements of the computing device. Accordingly, the logical operations described herein are referred to variously as states, operations, structural devices, acts, or modules. These states, operations, structural devices, acts and modules can be implemented in hardware, software, firmware, in special-purpose digital logic, and any combination thereof. It should be appreciated that more or fewer operations can be performed than shown in the FIGS. and described herein. These operations can also be performed in a different order than those described herein.

The routine500begins at operation502, where the host agent116can enumerate some or all of the MFND devices102that are present in a host100. One particular method (which might be referred to herein as the “GetMFNDList” method) for enumerating the MFND devices102connected to a host100returns device paths of all MFND devices102connected to a host100. If no MFND devices102are connected, or none are enumerated, the GetMFNDList method returns an error code.

From operation502, the routine500proceeds to operation504, where the host agent116can enumerate the child PFs112B-112N that are currently present on a MFND device102identified at operation502. One method (which might be referred to herein as the “GetChildPhysicalFunctionList” method) for enumerating the child PFS112A-112N on a MFND device102takes an identifier (e.g. a handle) for a particular MFND device102as input and returns adapter serial numbers of all child PFs112B-112N on the identified device.

From operation504, the routine500proceeds to operation506, where the host agent116can determine the capabilities of the MFND device102identified at operation502. For example, the host agent116can determine the maximum number of child PFs112B-112N supported by the MFND device102.

One method (which might be referred to herein as the “GetDeviceCapability” method) for getting the capabilities of a MFND device102takes an identifier (e.g. a handle) for a particular MFND device102as input and returns a device capability structure that specifies the capabilities of the identified device. In one embodiment, the device capability structure includes data identifying the maximum and available child PFs112B-112N, I/O queue pair count, interrupt count, namespace count, storage size, bandwidth, and IOPS of the identified device. The device capability structure might include additional or alternate data in other embodiments.

Once the capabilities of the MFND device102have been determined, the routine500can proceed from operation506to operation508, where child PFs112B-112N can be created or deleted on the MFND device102. By default, the MFND102has only one PF112, the parent PF112A, which is reserved for receiving administrative commands110from the root partition108.

In order to assign individual child PFs112B-112N to VMs104A-104N, the child PFs112B-112N are first created. The newly created child PFs112B-112N will appear to the host100following a reboot. One method for creating child PFs112B-112N (which might be referred to herein as the “CreateChildPhysicalFunction” method) takes an identifier (e.g. a handle) to a MFND102and a pointer to a data structure containing the settings for the new child PF112as input. The data structure can include data specifying the resources (e.g. the amount of storage space, namespaces, and interrupt vectors that the new PF112can use) and QoS level that are to be assigned to the new child PF112. The CreateChildPhysicalFunction method returns an identifier (e.g. a serial number) for the new child PF112as output if it completes successfully.

Child PFs112B-112N and their settings will persist across reboots of the host100, so the maximum number of child PFs112B-112N to be supported may be initially created to avoid rebooting the host100in the future. If a MFND102already has child PFs112B-112N, either as a result of a manufacturing configuration or previous user configuration, additional child PFs112B-112N can be created or deleted in order to configure the MFND102with the desired number of child PFs112B-112N to be supported.

One method for deleting child PFs112B-112N (which might be referred to herein as the “DeleteChildPhysicalFunction” method) takes an identifier for a MFND102(e.g. a handle) and the serial number for the child PF112to be deleted as input. The DeleteChildPhysicalFunction returns a success message if the identified child PF112was successfully deleted and otherwise returns an error code.

Once the host100has rebooted, the routine500proceeds from operation510to operation512, where the QoS level for the newly created child PFs112B-112N are set in the manner described above with regard toFIG.2B. Once the QoS levels have been set for the child PFs112B-112N, the routine500proceeds from operation512to operation514, where the MFND102enables the collection of child PF QoS statistics210for in-use child PFs112B-112N of the MFND102in the manner described above with regard toFIGS.2C and3and in further detail below with regard toFIG.6.

The routine500proceeds from operation514to operation516, where the child PFs112B-112N provided by a MFND102can be assigned to VMs104A-104N. As described briefly above, in some embodiments newly created child PFs112B-112N have zero storage size, minimal flexible resources, and no defined QoS level. In other embodiments, newly created child PFs112B-112N may have a default QoS level, a default amount of storage, and/or default configurations for other resources. Accordingly, the host100might need to provision the resources (NVM space, I/O queue pair count, QoS level, etc.) to a child PF112B-112N before it can be assigned to a VM104using DDA, HYPER-V NVMe Direct, or another direct storage assignment technology.

The child PFs112B-112N can also be securely erased before assignment to a VM104. There is no host reboot involved in this workflow. One method for securely erasing child PFs112B-112N (which might be referred to herein as the “SecureEraseChildPhysicalFunction” method) takes an identifier for a MFND102(e.g. a handle) and the serial number for the child PF112to be erased as input. The SecureEraseChildPhysicalFunction returns a success message if the identified child PF112was successfully erased and otherwise returns an error code. The routine500then proceeds from operation516to operation518, where it ends.

FIG.6is a flow diagram showing a routine600that illustrates aspects of a method for collecting child PF QoS statistics210for in-use child PFs112of a MFND102, according to one embodiment disclosed herein. The routine600begins at operation602, where the MFND102determines whether collection of child PF QoS statistics210has been enabled in the manner described above. If the collection of child PF QoS statistics210has been enabled, the routine600proceeds from operation602to operation604.

At operation604, the MFND102collects the child PF QoS statistics210in the manner described above. The routine600then proceeds from operation604to operation606, where the MFND102determines whether the QoS statistics monitor period204has elapsed. If the QoS statistics monitor period204has not elapsed, the routine600proceeds from operation606back to operation604, where the MFND102can continue to collect the child PF QoS statistics210in the manner described above. If the QoS statistics monitor period204has elapsed, the routine600proceeds from operation606to operation608.

At operation608, the MFND102stores the child PF QoS statistics210in a log entry in the child PF QoS statistics active log302in the manner described above. The routine600then proceeds from operation608to operation610, where the MFND102determines whether the QoS statistics swap bucket period206has elapsed. If the QoS statistics swap bucket period206has not elapsed, the routine600proceeds back to operation604, where the MFND102continues to collect child PF QoS statistics210and store the child PF QoS statistics210in entries in the active log302in the manner described above.

If the QoS statistics swap bucket period206has elapsed, the routine600proceeds from operation610to operation612, where the MFND102atomically swaps the active log302and the save log304and clears the active log302in the manner described above. The routine600then proceeds from operation612to operation614, where the MFND102generates a notification, such as an asynchronous event306, to the host computing device100to inform the host computing device100that child PF QoS statistics210are available from the MFND102. As discussed above, the host computing device100might subsequently transmit a command110D requesting the child PF QoS statistics210. The MFND102responds to the request with child PF QoS statistics210retrieved from the save log304. The routine600then proceeds from operation612to operation614, where it ends.

FIG.7is a computer architecture diagram showing an illustrative computer hardware and software architecture for a data processing system700that can act as a host100for a MFND102that implements aspects of the technologies presented herein. In particular, the architecture illustrated inFIG.7can be utilized to implement a server computer, mobile phone, an e-reader, a smartphone, a desktop computer, an AR/VR device, a tablet computer, a laptop computer, or another type of computing device that acts as a host100for the MFND102.

The data processing system700illustrated inFIG.7includes a central processing unit702(“CPU”), a system memory704, including a random-access memory706(“RAM”) and a read-only memory (“ROM”)708, and a system bus710that couples the memory704to the CPU702. A basic input/output system (“BIOS” or “firmware”) containing the basic routines that help to transfer information between elements within the data processing system700, such as during startup, can be stored in the ROM708. The data processing system700further includes a mass storage device712for storing an operating system722, application programs, and other types of programs. For example, the mass storage device712might store the host agent116and the management API118. The mass storage device712can also be configured to store other types of programs and data.

The mass storage device712is connected to the CPU702through a mass storage controller (not shown) connected to the bus710. The mass storage device712and its associated computer readable media provide non-volatile storage for the data processing system700. Although the description of computer readable media contained herein refers to a mass storage device, such as a hard disk, CD-ROM drive, DVD-ROM drive, or USB storage key, it should be appreciated by those skilled in the art that computer readable media can be any available computer storage media or communication media that can be accessed by the data processing system700.

According to various configurations, the data processing system700can operate in a networked environment using logical connections to remote computers through a network such as the network720. The data processing system700can connect to the network720through a network interface unit716connected to the bus710. It should be appreciated that the network interface unit716can also be utilized to connect to other types of networks and remote computer systems. The data processing system700can also include an input/output controller718for receiving and processing input from a number of other devices, including a keyboard, mouse, touch input, an electronic stylus (not shown inFIG.7), or a physical sensor such as a video camera. Similarly, the input/output controller718can provide output to a display screen or other type of output device (also not shown inFIG.7).

It should be appreciated that the software components described herein, when loaded into the CPU702and executed, can transform the CPU702and the overall data processing system700from a general-purpose computing device into a special-purpose computing device customized to facilitate the functionality presented herein. The CPU702can be constructed from any number of transistors or other discrete circuit elements, which can individually or collectively assume any number of states. More specifically, the CPU702can operate as a finite-state machine, in response to executable instructions contained within the software modules disclosed herein. These computer-executable instructions can transform the CPU702by specifying how the CPU702transitions between states, thereby transforming the transistors or other discrete hardware elements constituting the CPU702.

Encoding the software modules presented herein can also transform the physical structure of the computer readable media presented herein. The specific transformation of physical structure depends on various factors, in different implementations of this description. Examples of such factors include, but are not limited to, the technology used to implement the computer readable media, whether the computer readable media is characterized as primary or secondary storage, and the like. For example, if the computer readable media is implemented as semiconductor-based memory, the software disclosed herein can be encoded on the computer readable media by transforming the physical state of the semiconductor memory. For instance, the software can transform the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. The software can also transform the physical state of such components in order to store data thereupon.

In light of the above, it should be appreciated that many types of physical transformations take place in the data processing system700in order to store and execute the software components presented herein. It also should be appreciated that the architecture shown inFIG.7for the data processing system700, or a similar architecture, can be utilized to implement other types of computing devices, including hand-held computers, video game devices, embedded computer systems, mobile devices such as smartphones, tablets, and AR/VR devices, and other types of computing devices known to those skilled in the art. It is also contemplated that the data processing system700might not include all of the components shown inFIG.7, can include other components that are not explicitly shown inFIG.7, or can utilize an architecture completely different than that shown inFIG.7.

FIG.8is a computing network architecture diagram showing an illustrative configuration for a distributed computing environment800in which computing devices hosting MFNDs102implementing the disclosed technologies can be utilized. According to various implementations, the distributed computing environment800includes a computing environment802operating on, in communication with a network856. One or more client devices806A-806N (hereinafter referred to collectively and/or generically as “clients806”) can communicate with the computing environment802via the network804and/or other connections (not illustrated inFIG.8).

In one illustrated configuration, the clients806include a computing device806A such as a laptop computer, a desktop computer, or other computing device; a tablet computing device (“tablet computing device”)806B; a mobile computing device806C such as a smartphone, an on-board computer, or other mobile computing device; or a server computer806D. It should be understood that any number of devices806can communicate with the computing environment802. An example computing architecture for the devices806is illustrated and described above with reference toFIG.7. It should be understood that the illustrated devices806and computing architectures illustrated and described herein are illustrative only and should not be construed as being limited in any way.

In the illustrated configuration, the computing environment802includes application servers808, data storage810, and one or more network interfaces812. According to various implementations, the functionality of the application servers808can be provided by one or more server computers that are executing as part of, or in communication with, the network804. The application servers808can host various services, VMs, portals, and/or other resources. The application servers808can also be implemented using host computing devices100that includes MFNDs102configured in the manner described herein.

In the illustrated configuration, the application servers808host one or more virtual machines104for hosting applications, network services, or for providing other functionality. It should be understood that this configuration is illustrative only and should not be construed as being limiting in any way. The application servers808can also host or provide access to one or more portals, link pages, web sites, network services, and/or other information sites, such as web portals816.

According to various implementations, the application servers808also include one or more mailbox services818and one or more messaging services820. The mailbox services818can include electronic mail (“email”) services. The mailbox services818also can include various personal information management (“PIM”) services including, but not limited to, calendar services, contact management services, collaboration services, and/or other services. The messaging services820can include, but are not limited to, instant messaging services, chat services, forum services, and/or other communication services.

The application servers808also might include one or more social networking services822. The social networking services822can include various social networking services including, but not limited to, services for sharing or posting status updates, instant messages, links, photos, videos, and/or other information; services for commenting or displaying interest in articles, products, blogs, or other resources; and/or other services. Other services are possible and are contemplated.

The social networking services822also can include commenting, blogging, and/or micro blogging services. Other services are possible and are contemplated. As shown inFIG.8, the application servers808also can host other network services, applications, portals, and/or other resources (“other resources”)824. The other resources824can include, but are not limited to, document sharing, rendering, or any other functionality.

As mentioned above, the computing environment802can include data storage810. According to various implementations, the functionality of the data storage810is provided by one or more databases operating on, or in communication with, the network804. The functionality of the data storage810also can be provided by one or more server computers configured to host data for the computing environment802. The data storage810can include, host, or provide one or more real or virtual data stores826A-826N (hereinafter referred to collectively and/or generically as “datastores826”).

The datastores826are configured to host data used or created by the application servers808and/or other data. Although not illustrated inFIG.8, the datastores826also can host or store web page documents, word processing documents, presentation documents, data structures, and/or other data utilized by any application program or another module. Aspects of the datastores826might be associated with a service for storing files.

The computing environment802can communicate with, or be accessed by, the network interfaces812. The network interfaces812can include various types of network hardware and software for supporting communications between two or more computing devices including, but not limited to, the clients806and the application servers808. It should be appreciated that the network interfaces812also might be utilized to connect to other types of networks and/or computer systems.

It should be understood that the distributed computing environment800described herein can implement aspects of at least some of the software elements described herein with any number of virtual computing resources and/or other distributed computing functionality that can be configured to execute any aspects of the software components disclosed herein.

It should be further understood that the disclosure presented herein also encompasses the subject matter set forth in the following clauses:Clause 1. A computer-implemented method, comprising: creating a child physical function on a multiple physical function non-volatile memory device (MFND); configuring the child physical function on the MFND to provide a specified Quality of Service (QoS) level; collecting child physical function QoS statistics for the child physical function; and providing the child physical function QoS statistics from the MFND to a host computing device.Clause 2. The computer-implemented method of clause 1, wherein the specified QoS level defines maximum read input/output operations per second (IOPS) and maximum write IOPS for the child physical function, and wherein the child physical function QoS statistics for the child physical function specify a percentage of the maximum read IOPS and the maximum write IOPS provided by the child physical function during a monitoring period.Clause 3. The computer-implemented method of any of clauses 1 or 2, wherein the specified QoS level defines a maximum read bandwidth and a maximum write bandwidth for the child physical function, and wherein the child physical function QoS statistics for the child physical function specify a percentage of the maximum read bandwidth and the maximum write bandwidth provided by the child physical function during a monitoring period.Clause 4. The computer-implemented method of any of clauses 1-3, wherein the child physical function QoS statistics for the child physical function specify a percentage of read operations and write operations performed by the child physical function during a monitoring period.Clause 5. The computer-implemented method of any of clauses 1-4, wherein the child physical function QoS statistics for the child physical function specify a size of input/output (I/O) workloads performed by the child physical function during a monitoring period.Clause 6. The computer-implemented method of any of clauses 1-5, wherein the MFND comprises a non-volatile memory device, and wherein the QoS statistics for the child physical function specify an amount of the non-volatile memory device utilized.Clause 7. The computer-implemented method of any of clauses 1-6, wherein the host computing device specifies a QoS statistics monitor period and a QoS statistics swap bucket period to the MFND, and wherein the method further comprises: storing the child physical function QoS statistics in an active log during the QoS statistics monitor period; and swapping the active log with a save log when the QoS statistics swap bucket period elapses, wherein the child physical function QoS statistics are provided from the MFND to the host computing device from the save log.Clause 8. The computer-implemented method of any of clauses 1-7, further comprising generating an asynchronous event from the MFND to the host computing device when the QoS statistics swap bucket period elapses.Clause 9. A multiple physical function non-volatile memory device (MFND), comprising: a non-volatile memory device; a parent physical function; and a child physical function configured to provide a Quality of Service (QoS) level specified by a host computing device configured to perform read or write operations on the non-volatile memory device, wherein the MFND is configured to collect child physical function QoS statistics for the child physical function, and provide the child physical function QoS statistics to the host computing device.Clause 10. The MFND of clause 9, wherein the specified QoS level defines maximum read input/output operations per second (IOPS) and maximum write IOPS for the child physical function, and wherein the child physical function QoS statistics for the child physical function specify a percentage of the maximum read IOPS and the maximum write IOPS provided by the child physical function during a monitoring period.Clause 11. The MFND of any of clauses 9 or 10, wherein the specified QoS level defines a maximum read bandwidth and a maximum write bandwidth for the child physical function, and wherein the child physical function QoS statistics for the child physical function specify a percentage of the maximum read bandwidth and the maximum write bandwidth provided by the child physical function during a monitoring period.Clause 12. The MFND of any of clauses 9-11, wherein the child physical function QoS statistics for the child physical function specify a percentage of read operations and write operations performed by the child physical function during a monitoring period.Clause 13. The MFND of any of clauses 9-12, wherein the child physical function QoS statistics for the child physical function specify a size of input/output (I/O) workloads performed by the child physical function during a monitoring period.Clause 14. The MFND of any of clauses 9-13, wherein the QoS statistics for the child physical function specify an amount of the non-volatile memory device utilized.Clause 15. The MFND of any of clauses 9-14, wherein the host computing device specifies a QoS statistics monitor period and a QoS statistics swap bucket period to the MFND, and wherein the MFND is further configured to: store the child physical function QoS statistics in an active log during the QoS statistics monitor period; swap the active log with a save log when the QoS statistics swap bucket period elapses; and generate an asynchronous event from the MFND to the host computing device when the QoS statistics swap bucket period elapses, wherein the child physical function QoS statistics are provided from the MFND to the host computing device from the save log.Clause 16. A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors, cause the one or more processors to: create a child physical function on a multiple physical function non-volatile memory device (MFND); configure the child physical function on the MFND to provide a specified Quality of Service (QoS) level; collect child physical function QoS statistics for the child physical function; and provide the child physical function QoS statistics from the MFND to a host computing device.Clause 17. The non-transitory computer-readable storage medium of clause 16, wherein the specified QoS level defines maximum read input/output operations per second (IOPS) and maximum write IOPS for the child physical function, and wherein the child physical function QoS statistics for the child physical function specify a percentage of the maximum read IOPS and the maximum write IOPS provided by the child physical function during a monitoring period.Clause 18. The non-transitory computer-readable storage medium of any of clauses 16 or 17, wherein the specified QoS level defines a maximum read bandwidth and a maximum write bandwidth for the child physical function, and wherein the child physical function QoS statistics for the child physical function specify a percentage of the maximum read bandwidth and the maximum write bandwidth provided by the child physical function during a monitoring period.Clause 19. The non-transitory computer-readable storage medium of any of clauses 16-19, wherein the child physical function QoS statistics comprise statistics selected from the group consisting of a percentage of read operations and write operations performed by the child physical function during a monitoring period, a size of input/output (I/O) workloads performed by the child physical function during a monitoring period, and an amount of the non-volatile memory device utilized.Clause 20. The non-transitory computer-readable storage medium of any of clauses 16-20, wherein the host computing device specifies a QoS statistics monitor period and a QoS statistics swap bucket period to the MFND, and wherein the non-transitory computer-readable storage medium has further computer-executable instructions stored thereupon to: store the child physical function QoS statistics in an active log during the QoS statistics monitor period; swap the active log with a save log when the QoS statistics swap bucket period elapses; and generate an asynchronous event from the MFND to the host computing device when the QoS statistics swap bucket period elapses, wherein the child physical function QoS statistics are provided from the MFND to the host computing device from the save log.

Although the technologies presented herein have been described in language specific to structural features and/or methodological acts, it is to be understood that the appended claims are not necessarily limited to the features or acts described. Rather, the features and acts are described as example implementations of such technologies. Moreover, the above-described subject matter may be implemented as a computer-controlled apparatus, a computer process, a computing system, or as an article of manufacture such as a computer-readable storage medium.

The operations of the example methods presented herein are illustrated in individual blocks and summarized with reference to those blocks. The methods are illustrated as logical flows of blocks, each block of which can represent one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable the one or more processors to perform the recited operations.

Generally, computer-executable instructions include routines, programs, objects, modules, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be executed in any order, combined in any order, subdivided into multiple sub-operations, and/or executed in parallel to implement the described processes. The described processes can be performed by resources associated with one or more device(s) such as one or more internal or external CPUs or GPUs, and/or one or more instances of hardware logic such as FPGAs, DSPs, or other types of accelerators.

All of the methods and processes described above may be embodied in, and fully automated via, software code modules executed by one or more general purpose computers or processors. The code modules may be stored in any type of computer-readable storage medium or other computer storage device. Some or all of the methods may alternatively be embodied in specialized computer hardware.