Patent ID: 12222844

DETAILED DESCRIPTION

Reference will now be made in detail to several embodiments. While the subject matter will be described in conjunction with the alternative embodiments, it will be understood that they are not intended to limit the claimed subject matter to these embodiments. On the contrary, the claimed subject matter is intended to cover alternative, modifications, and equivalents, which may be included within the spirit and scope of the claimed subject matter as defined by the appended claims.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the claimed subject matter. However, it will be recognized by one skilled in the art that embodiments may be practiced without these specific details or with equivalents thereof. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects and features of the subject matter.

Portions of the detailed description that follows are presented and discussed in terms of a method. Although steps and sequencing thereof are disclosed in a figure herein (e.g.,FIG.6) describing the operations of this method, such steps and sequencing are exemplary. Embodiments are well suited to performing various other steps or variations of the steps recited in the flowchart of the figure herein, and in a sequence other than that depicted and described herein.

Some portions of the detailed description are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits that can be performed on computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer-executed step, logic block, process, etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, parameters, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout, discussions utilizing terms such as “accessing,” “writing,” “including,” “storing,” “transmitting,” “associating,” “identifying,” “encoding,” “labeling,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Some embodiments may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, algorithms, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

Parallel Testing of Virtual Functions Across Multiple DUTS

Embodiments of the present invention can provide an extended NVMe driver that supports exercising virtual functions (and related physical functions) of a DUT without using a VM or hypervisor. In this way, the amount of memory and processing resources used for testing NVMe SSDs can be significantly reduced, and a large number of DUTs (e.g., up to 16 DUTs) can be tested in parallel independently. In other words, each DUT is tested in isolation, as if is the only device being tested, and there are no race conditions or competition for resources between workloads during testing.

FIG.1depicts an exemplary host test system100(e.g., an ATE) coupled to DUTs105,110,115, and120. Fewer or additional DUTs (e.g., 16 DUTs) may be tested in parallel, according to embodiments. Host system100executes a test program that launches multiple test program processes125,130, and135, using a plurality of threads for exercising virtual functions of DUTs105,110,115, and120via NVMe drivers140,145,150, and155. NVMe drivers140,145,150, and155are extended drivers that can advantageously test virtual functions of multiple DUTs in parallel without executing a VM or hypervisor to reduce overhead and complexity of testing. Furthermore, testing virtual functions of DUTs in parallel using extended NVMe drivers140,145,150, and155can significantly improve performance during testing compared to traditional approaches.

The threads of the test programs are user level threads that generate workloads for testing the DUTs using a NVMe driver. For example, Thread1.1of test program process1(125) can generate a workload for exercising Virtual Function1of DUT1(105) using NVMe driver1(140), and Thread1.2can generate a workload for exercising Virtual Function2of DUT1(105) using NVMe driver2(145), etc. The workloads are executed independently and in isolation with no competition between devices for resources (e.g., memory, storage, etc.), and each driver can use an allocated data structure and other resources (e.g., interrupts and queues) for testing virtual functions of each DUT.

Testing the virtual functions may involve performing corresponding physical functions of the DUTs, and the test results may be returned to the corresponding test program process for evaluation. The physical functions of the DUTs are executed using dedicated local registers in each DUT that are accessible by host100. Typically, the virtual functions can be thought of as smaller light-weight versions of the physical functions tested by host100(e.g., fewer attributes).

The extended NVMe drivers typically include functions/routines and data structures instantiated for each virtual function to be tested, as well as physical resources allocated for testing (e.g., queues, interrupts, memory, etc.). According to some embodiments, the NVMe drivers include a controller handler for interfacing with an NVMe device, an interrupt handler, a queue for sending commands to a virtual function of a DUT, a queue for receiving responses back the DUT, and optionally other computer resources for implementing the queues, etc. Accordingly, the NVMe drivers can execute the functions/routines necessary to perform a workload based on the instantiated data structure. Moreover, the test program processes executed by the test program appear to be the sole user operating on said host computer system at any given time. In this way, the test system100can simulate a virtualization environment for said plurality of test program processes on said host computer system to test virtual functions of the DUTs separately, independently and concurrently, so that there is no competition for resources or need to execute a hypervisor or virtual machine.

The data structures can be used by the NVMe drivers to locate and enable virtual function the DUTs for testing. For example, each data structure can present access to a corresponding virtual function that is accessed by the NVMe drivers to perform workloads generated by the threads. Typically, a separate data structure is used to test virtual functions of each DUT. For example, to test 10 virtual functions of 16 DUTs in parallel,160data structures are instantiated. Moreover, data structures may be instantiated by the NVMe drivers for directly testing physical functions of DUTs (e.g., without testing a virtual function).

The threads are executed on the host100in user space and generate specific workloads that typically include combinations of NVMe commands (e.g., READ, WRITE, and administrative commands). The NVMe drivers service the request of the threads and issue commands to the corresponding device. According to some embodiments, multiple threads can be used to test a virtual function, using different workloads or commands, for example.

The NVMe drivers can also include one or more queues (e.g., NVMe queues, I/O queues), as well as a controller handler and namespace (advertised by the DUT) for accessing/modifying data stored on the DUTs during execution of the workload. The namespaces advertised by the DUT provide an access point for the threads to access the corresponding user space data using via an NVMe driver. In this way, 8 or even 16 DUTs can be tested in parallel independently without any competition between resources (e.g., race conditions) to test 64 or even 128 virtual functions of the DUTs.

FIG.2is a block diagram of an exemplary host test system200capable of testing multiple virtual functions in parallel across multiple DUTs without using a hypervisor or virtual machine according to embodiments of the present invention. In the example ofFIG.2, a single DUT205is depicted for clarity. It should be understood that typically several DUTs are coupled to host system200for testing in parallel.

In the example ofFIG.2, host200executes a test program that launches multiple test program processes210,215, and220. The test program processes210,215, and220execute multiple threads, each thread is typically associated with a different virtual function. Extended NVMe drivers are instantiated for testing of each namespace allocated to the virtual functions. The threads (e.g., threads1.1,2.1,8.1,8.n) generate workloads for a corresponding namespace to test virtual functions of the DUTs (e.g., virtual function1, virtual function2, virtual function3, virtual function4). Executing threads using test program processes210,215, and220to generate commands executed by extended NVMe drivers obviates the need for a hypervisor or virtual machines to significantly reduce testing complexity and overhead. According to some embodiments, multiple threads are used to test a virtual function, using different commands or workloads, for example.

As depicted inFIG.2, virtual function1is allocated NVMe namespace1, virtual function2is allocated NVMe namespace2, etc. Testing the virtual functions of DUTs (e.g., DUT205) using extended NVMe drivers225,230,235, and240typically involves executing corresponding NVMe physical functions265. Advantageously, the test program processes executed by the test program processes appear to be the sole user operating on said host computer system at any given time. In this way, the test system200can simulate a virtualization environment for said plurality of test program processes on said host computer system to test virtual functions of the DUTs separately, independently and concurrently, so that there is no competition for resources or need to execute a hypervisor or virtual machine.

FIG.3is a block diagram of an exemplary test system300in communication with multiple DUTs305,310,315, and320for parallel testing virtual functions using data structures instantiated by extended NVMe drivers according to embodiments of the present invention. As depicted inFIG.3, a number of NVMe drivers (1−n) are instantiated to test a corresponding number of virtual functions of the DUTs, and each extended NVMe driver includes different data structures for performing the tests.

Testing the virtual functions typically includes executing corresponding physical functions of the DUTs based on workloads generated by test program processes executed by the test program executed on the host test system300(e.g., test program process325and330). Importantly, each DUT instantiates a number of data structures corresponding to different virtual functions. For example, to test 10 virtual functions of 16 DUTs (n=16) in parallel,160data structures can be instantiated.

In the example ofFIG.3, NVMe drivers335and340each instantiate4data structures to test corresponding virtual functions of DUTs305,310,315, and320. Specifically, NVMe driver1(335) tests virtual function1of DUTs305,310,315, and320using data structures1,2,3, and4, respectively, and NVMe driver n (330) tests virtual function n of DUTs305,310,315, and320using data structures5,6,7, and8, respectively. By using the different data structures to test workloads generated by the test program processes, the host test system300can simulate a virtualization environment for said plurality of test program processes on said host computer system to test virtual functions of the DUTs separately, independently and concurrently, so that there is no competition for resources or need to execute a hypervisor or virtual machine.

FIG.4is a block diagram and data flow diagram400for testing virtual and physical functions of DUTs using an extended NVMe driver according to embodiments of the present invention. In step405of data flow diagram400, test program process1executes multiple threads that generate workloads for testing virtual functions of DUTs. At step410, NVMe driver1receives the workload generated by thread1.1of test program process1to test virtual function1of an NVMe device. As depicted inFIG.4, virtual function1is allocated namespace1in user space of the NVMe device.

The NVMe driver1instantiates a data structure for testing namespace1, and the driver can include a controller handler for interfacing with the NVMe device, one or more queues (e.g., I/O queues), an interrupt handler for controlling operation of the NVMe driver by the CPU, and optionally other computing resources used to carry out testing. At step415, the DUT executes virtual function1according to the NVMe commands received from NVMe driver1instantiated to test virtual function1. Executing virtual function1typically involves performing one or more physical functions that accesses memory registers of the NVMe device. In this way, test results can be generated by exercising various functions of the DUT, and the results can be returned to the host computer system executing test program process1without using a hypervisor or virtual machine, which significantly reduces overhead and complexity of testing. Moreover, the steps of dataflow diagram400can be performed by multiple test program processes and multiple threads concurrently to test multiple DUTs in parallel using a single host computer test system.

FIG.5is a block diagram of an exemplary NVMe driver510instantiated in memory of a host computer500for testing virtual and physical functions of DUT535without using a virtual machine or hypervisor according to embodiments of the present invention. DUT535can be disposed in a socket of a test interface board with other DUTs for testing in parallel, for example. Multiple NVMe drivers510can be executed by host computer500to test multiple virtual functions of the DUTs in parallel using threads executed by CPU505. CPU505executes a test program that includes multiple test program processes for generating the threads to produce different workloads performed by DUT535according to commands sent by controller handler530of NVMe driver510. As depicted inFIG.5, NVMe driver510includes an interrupt handler515for receiving interrupts from CPU505, one or more queues520, such as queues for sending and receiving data and NVMe commands to/from DUT535, and data structures525for executing the workloads within a virtualization environment.

FIG.6is a flow chart depicting an exemplary process600for testing virtual and physical functions of NVMe devices using an extended NVMe driver according to embodiments of the present invention. Process600can be performed without executing a hypervisor or virtual machine to test the virtual functions.

At step605, a host system executes a test program for testing multiple connected DUTs in parallel. The host system typically includes memory and a CPU. The test program executes multiple test program processes for testing the DUTs by launching threads for testing different virtual functions of the associated DUT. Each thread can be used to test a different virtual function (or physical function) of the DUT, for example.

At step610, the host system simulates a virtual environment (similar to a virtual machine that uses a hypervisor to create and manage virtual machines) for testing virtual functions of the DUTs using NVMe device drivers instantiated in the memory of the host system. The NVMe device drivers instantiated in the memory of the host system include different data structures generated for the different virtual functions. Typically there is one data structure for each virtual function (or namespace) of the DUTs to be tested, and the virtual functions are tested on the DUTs separately, independently and concurrently within the virtualization environment.

At step615, the host system exercises various functions of the DUTs in parallel according to the test program within the virtualization environment.

Exemplary Test System

Embodiments of the present invention are drawn to electronic systems for testing virtual functions of an NVMe device without using a hypervisor or virtual machine to reduce complexity and overhead during testing. A single test system can execute test programs that launch multiple threads for testing virtual functions of DUTs using extended NVMe drivers. Each DUT can be tested independently in isolation from the other DUTs without competing for time or resources according to workloads generated by the threads using data structures instantiated by the NVMe drivers for the corresponding virtual function. The virtual functions access user space data of a DUT corresponding to a namespace accessible to an NVMe driver.

In the example ofFIG.7, the exemplary test computer system712includes a central processing unit (CPU)701for running software applications and an operating system. Random access memory702and read-only memory703store applications and data for use by the CPU701. CPU701can instantiate NVMe drivers (FIG.5) and data structures for testing the virtual functions and physical functions of a DUT in memory702, and can send commands to the DUT (or test interface boards, etc.) using a controller handler allocated to an NVMe driver to communicate with the DUT.

Data storage device704provides non-volatile storage for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM or other optical storage devices. The data storage device704or the memory702/703can store historic and real-time testing data (e.g., test results, limits, computations, etc.).

The optional user inputs706and707comprise devices that communicate inputs from one or more users to the computer system712(e.g., mice, joysticks, cameras, touch screens, keyboards, and/or microphones). A communication or network interface708allows the computer system712to communicate with other computer systems, networks, or devices via an electronic communications network, including wired and/or wireless communication and including an Intranet or the Internet.

The optional display device710may be any device capable of displaying visual information, e.g., the final scan report, in response to a signal from the computer system712and may include a flat panel touch sensitive display, for example. The components of the computer system712, including the CPU701, memory702/703, data storage704, user input devices706, and graphics subsystem705may be coupled via one or more data buses700.

Embodiments of the present invention are thus described. While the present invention has been described in particular embodiments, it should be appreciated that the present invention should not be construed as limited by such embodiments, but rather construed according to the following claims.