Patent Description:
Additionally, trying to replace a running kernel device module (or module, for short) while a corresponding device is running can be challenging. The act of unloading and reloading the module can create a scenario of lost state. One issue is that the new module will not have any knowledge of the state that the old module left behind. During the period of time after the old module has unloaded and the new module starts loading there will be no driver managing the device. Any activity or change of state to the device will be lost. <CIT> and <CIT> relate to methods of updating drivers.

The present invention provides a system in accordance with claim <NUM>, a method in accordance with claim <NUM>, and non-transitory computer readable storage medium comprising program instructions executable by a processor in accordance with claim <NUM>.

In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

Various systems, apparatuses, and methods for performing a live update of a kernel device module are disclosed herein. In one implementation, a proxy module is loaded onto a system, with the proxy module providing a registration and communication point for the two different versions of a kernel device module. In one implementation, the proxy module and kernel device modules are loadable kernel modules. The proxy module performs the driver activity that cannot be duplicated or released while having two versions of the loadable kernel modules active at the same time when control is transferred from a first version of the kernel device module to a second version of the kernel device module. In one implementation, the proxy module performs activities which include, but are not limited to, device registration, input-output control (IOCTL) calls, sysfs calls, and other activities.

When a first version of a kernel device module loads it will use an application programming interface (API) exported from the proxy module to register with the proxy module. The proxy module will then register with the operating system (OS) pretending to be the first version of the kernel device module. The proxy module will maintain any required open sysfs attribute files and IOCTL file attributes. When a second version of the kernel device module loads, the second version of the kernel device module will also register with the proxy module. Since the proxy module has already registered with the OS, the proxy module does not need to repeat the registration step. Since each version of the kernel device module registers with the proxy module, the proxy module knows how to find and communicate with each version of the kernel device module. A new version of the kernel device module will query the proxy module to find the other version so that the two versions are able to communicate directly. In one implementation, once the two versions of the kernel device module are able to communicate directly with each other, the two versions manage the transfer of device control between themselves without needing further assistance from the proxy module.

Referring now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. In one implementation, computing system <NUM> includes at least processors 105A-N, input/output (I/O) interfaces <NUM>, bus <NUM>, memory controller(s) <NUM>, network interface controller (NIC) <NUM>, and memory device(s) <NUM>. In other implementations, computing system <NUM> includes other components and/or computing system <NUM> is arranged differently. Processors 105A-N are representative of any number of processors which are included in system <NUM>. In one implementation, processor 105A is a general purpose processor, such as a central processing unit (CPU). In one implementation, processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and so forth. In some implementations, processors 105A-N include multiple data parallel processors.

Memory controller(s) <NUM> are representative of any number and type of memory controllers accessible by processors 105A-N and I/O devices (not shown) coupled to I/O interfaces <NUM>. Memory controller(s) <NUM> are coupled to any number and type of memory devices(s) <NUM>. Memory device(s) <NUM> are representative of any number and type of memory devices. For example, the type of memory in memory device(s) <NUM> includes Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others.

I/O interfaces <NUM> are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to I/O interfaces <NUM>. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth. Network interface controller (NIC) <NUM> receives and sends network messages across network <NUM>.

Network <NUM> is representative of any type of network or combination of networks, including wireless connection, direct local area network (LAN), metropolitan area network (MAN), wide area network (WAN), an Intranet, the Internet, a cable network, a packet-switched network, a fiber-optic network, a router, storage area network, or other type of network. Examples of LANs include Ethernet networks, Fiber Distributed Data Interface (FDDI) networks, and token ring networks. In various implementations, network <NUM> further includes remote direct memory access (RDMA) hardware and/or software, transmission control protocol/internet protocol (TCP/IP) hardware and/or software, router, repeaters, switches, grids, and/or other components.

In various implementations, computing system <NUM> is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any of various other types of computing systems or devices. It is noted that the number of components of computing system <NUM> varies from implementation to implementation. For example, in other implementations, there are more or fewer of each component than the number shown in <FIG>. It is also noted that in other implementations, computing system <NUM> includes other components not shown in <FIG>. Additionally, in other implementations, computing system <NUM> is structured in other ways than shown in <FIG>.

Turning now to <FIG>, a block diagram of one implementation of a virtual environment of a server <NUM> is shown. In one implementation, server <NUM> includes clients 205AN, virtual machines (VMs) 215A-N, virtual machine monitor (VMM) <NUM>, host hardware <NUM>, memory <NUM>, bus/fabric <NUM>, and device <NUM>. In one implementation, server <NUM> is part of a cloud service provider environment. It should be understood that server <NUM> is merely one illustration of a server architecture that is used in one implementation. In other implementations, a server includes other numbers and/or types of components arranged in other suitable manners.

Client 205A-N are representative of any number of clients which are coupled to and/or using resources on server <NUM>. Clients 205A-N include any number and type of applications <NUM> and <NUM>, respectively. One of the applications within each of the clients 205A-N may be an operating system such as for example Linux® or Microsoft® Windows®. Each client 205A-N can also include a guest driver 210A-N, respectively, to interact with device <NUM>. In one implementation, a virtual machine (VM) 215A-N is instantiated on server <NUM> for each client 205A-N. In one implementation, VMM <NUM> schedules the VMs 215A-N and the operating system inside each VM manages the scheduling of applications <NUM>-<NUM> on device <NUM>. Host hardware <NUM> includes any number and type of processing resources. For example, in one implementation, host hardware <NUM> includes a plurality of CPUs (e.g., multi-core CPUs). Host hardware <NUM> also includes memory controller logic, cache subsystem(s), fabric and/or interconnect resources, network interface circuitry, power management circuitry, voltage regulators, and so on. Memory <NUM> is representative of any number and type of memory devices included within server <NUM>. Bus/fabric <NUM> includes any type of interconnect logic for connecting the various components of server <NUM>. While a single device <NUM> is shown connected to bus/fabric <NUM>, it should be understood that multiple other devices are able to be connected to bus/fabric <NUM>. In one implementation, device <NUM> is a GPU. In other implementations, device <NUM> is any of various other types of devices. In various implementations, clients 205A-N may include applications which are executed at least in part using the computing resources of device <NUM>. Guest driver 210A-N of clients 205A-N, respectively, which use device <NUM> are designed to manage the interactions between apps <NUM> and <NUM> respectively and the resources of device <NUM>.

To manage interactions between OS <NUM> and device <NUM>, memory <NUM> stores Nth version of kernel device module 260A, wherein kernel device module 260A is a device driver for device <NUM>. When Nth version of kernel device module 260A is loaded into memory <NUM> and installed on server <NUM>, Nth version of kernel device module 260A registers with proxy module <NUM> rather than registering with operating system (OS) <NUM>. In one implementation, OS <NUM> is a Linux®-based OS. In other implementations, OS <NUM> is any of various other types of OS's (e.g., Windows®).

When Nth version of kernel device module 260A is loaded into memory <NUM> and installed on server <NUM>, Nth version of kernel device module 260A registers with proxy module <NUM> rather than registering with operating system (OS) <NUM>. The Nth version of kernel device module 260A does not know if a previous version of the device driver is already installed and running on server <NUM> for device <NUM>. Proxy module <NUM> returns an indication during the registration process to notify Nth version of kernel device module 260A if it is the first device driver or if a previous version of the device driver is already present. During the lifetime of server <NUM>, it may be desired to update the device driver for device <NUM> to add new functionality, fix bugs, and/or make other types of changes. However, it is also desired that the updating of the device driver result in minimal interference with the operation of clients 205A-N. (N+<NUM>)th version of kernel device module 260B represents a newer version of the device driver for device <NUM> which is intended to replace Nth version of kernel device driver 260A. For example, in one implementation, (N+<NUM>)th version of kernel device module 260B uses one or more different data structures than Nth version of kernel device driver 260A.

When (N+<NUM>)th version of kernel device module 260B is loaded into memory <NUM> and installed on server <NUM>, (N+<NUM>)th version of kernel device module 260B registers with proxy module <NUM> rather than registering with operating system (OS) <NUM>. When (N+<NUM>)th version of kernel device module 260B registers with proxy module <NUM>, (N+<NUM>)th version of kernel device module 260B does not know if a previous version of the device driver is already installed and running on server <NUM> for device <NUM>. In the case where a previous version (Nth version) is already running, the proxy module <NUM> returns an indication during the registration process to notify (N+<NUM>)th version of kernel device module 260B that an existing device driver is already running. Proxy module <NUM> also provides (N+<NUM>)th version of kernel device module 260B with a pointer to Nth version of kernel device module 260A. This allows (N+<NUM>)th version of kernel device module 260B and Nth version of kernel device module 260A to communicate to manage the transfer of control to (N+<NUM>)th version of kernel device module 260B.

In some scenarios, this transfer of control from Nth version of kernel device module 260A to (N+<NUM>)th version of kernel device module 260B must occur without clients 205A-N being aware of the transfer. As part of the process of transferring control between modules 260A-B, static data associated with device <NUM> is copied from Nth version of kernel device module 260A to (N+<NUM>)th version of kernel device module 260B. While static data is being copied, Nth version of kernel device module 260A continues to manage device <NUM> and clients 205A-N continue to run and use device <NUM> without any interruptions. In one implementation, static data includes a number of VFs, a size of the frame buffer, data from configuration files, video basic input output system (BIOS) image, and/or other data. After the static data is copied, Nth version of kernel device module 260A is quiesced. In one implementation, quiescing Nth version of kernel device module 260A involves stopping all inputs coming into Nth version of kernel device module 260A. However, stopping all inputs coming into Nth version of kernel device module 260A does not necessarily disrupt clients 205A-N. For example, in one implementation, device <NUM> is a GPU, and a given client <NUM> might have a render operation running on device <NUM> when the quiesce operation is initiated. While Nth version of kernel device module 260A is being quiesced, the render operation is able to continue running on device <NUM> without interruption. Only if the given client <NUM> is waiting on a response from Nth version of kernel device module 260A will the given client <NUM> experience a small delay. If external stimuli to the Nth version of the kernel device module 260A occur during the time while the Nth version is quiesced and the (N+<NUM>)th version of kernel device module 260B has not yet started there may be a possible small delay to the system. After Nth version of kernel device module 260A is quiesced, transient data associated with device <NUM> is transferred from Nth version of kernel device module 260A to (N+<NUM>)th version of kernel device module 260B. Then (N+<NUM>)th version of kernel device module 260B takes over as the new device driver for device <NUM>. Clients 205A-N will be unaware of this change in the device driver for device <NUM>.

Referring now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. In one implementation, a kernel device module 305A registers with a proxy module 315A to operate as a device driver for a given device (not shown). In one implementation, the given device is a GPU. In other implementations, the given device is any of various other types of devices. It is noted that the terms "kernel device module" and "device driver" are used interchangeably herein.

VMs 320A, 330A, and 340A are representative of any number of VMs which are utilizing the processing resources of the given device. VMs 320A, 330A, and 340A correspond to virtual functions (VFs) 325A, 335A, and 345A, respectively, which take turns running on the given device. In another implementation, various clients which are not VFs could take turns running on the given device. After kernel device module 305A registers with proxy module 315A, proxy module 315A registers with the operating system (OS) (not shown) pretending to be kernel device module 305A. In other words, proxy module 315A identifies itself as kernel device module 305A to the OS so as to register kernel device module 305A as the device driver for the given device. Then, after proxy module 315A registers with the OS for kernel device module 305A, kernel device module 305A is able to start managing the given device. In one implementation, managing the given device involves processing IOCTL and sysfs calls, processing interrupts (e.g., Mailbox interrupts, World Switch interrupts), and processing debug requests. In other implementations, managing the given device involves other types of actions.

Turning now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. Computing system <NUM> is intended to represent computing system <NUM> (of <FIG>) at a later point in time. Accordingly, kernel device module 305B and proxy module 315B are intended to represent kernel device module 305A and proxy module 315A, respectively, at the later point in time. At the point in time illustrated in <FIG>, kernel device module 310B has been loaded onto system <NUM>. Being loaded onto system <NUM> means that kernel device module 310B has been stored in the memory of system <NUM> and that execution of kernel device module 310B has been initiated on one or more processors of system <NUM>.

For the purposes of this discussion, it is assumed that kernel device module 310B is a subsequent version of kernel device module 305B. For example, if kernel device module 305B is the first version of a device driver, then kernel device module 310B is the second or later (or earlier in the case of downgrading) version of the device driver. It is common for device drivers to be updated from time to time to incorporate new functionality, fix bugs, provide security updates, and/or implement other changes. Therefore, it is expected that the original device driver for a given device will not be the only device driver for the given device during the entire lifetime of system <NUM>. Accordingly, updates to kernel device module 305B will be loaded onto system <NUM> at various times. Kernel device module 310B is intended to represent a complete replacement of kernel device module 305B. When performing a complete replacement of kernel device module 305B, the handoff of control from kernel device module 305B to kernel device module 310B should not disrupt VMs 320B, 330B, and 340B. Or, put another way, it is desired to minimize the disruption to VMs 320B, 330B, and 340B during the transfer of control from kernel device module 305B to kernel device module 310B. As shown in <FIG>, VMs 320B, 330B, and 340B correspond to VFs 325B, 335B, and 345B, respectively, which take turns running on the given device. In another implementation, various clients which are not VFs could be running on the given device.

As used herein, the term "kernel device module" is defined as a device driver which handles the interface between a given device of system <NUM> and the OS or other software applications of system <NUM>. In one implementation, the given device is a GPU. In other implementations, the given device is any of various other types of devices. It is assumed for the purposes of this discussion that kernel device module 310B and kernel device module 310A are both designed for interfacing to the same device of system <NUM>. Therefore, if both kernel device module 310B and kernel device module 310A were to simultaneously attempt to handle the interface to the same given device, this would result in unpredictable behavior for system <NUM>. Accordingly, kernel device module 305B and kernel device module 310B will coordinate access to the given device to manage the hand-off of control.

When kernel device module 310B is loaded onto system <NUM>, kernel device module 310B registers with proxy module 315B rather than registering with the OS. The OS already has an existing driver registration with the given device, and so the OS will not allow kernel device module 310B to be registered with the same driver name for this given device. The existing registration was performed by proxy module 315B on behalf of kernel device module 305B. Accordingly, since kernel device module 310B is not able to register with OS to be the device driver for the given device, kernel device module 310B registers with proxy module 315B. In response to receiving this registration attempt, proxy module 315B will notify kernel device module 310B that another kernel device module (i.e., kernel device module 305B) is already registered to be the device driver for the given device. Proxy module 315B also provides kernel device module 310B with a pointer which allows kernel device module 310B to communication with kernel device module 305B.

In this implementation, kernel device module 310B uses the pointer to communicate with kernel device module 305B to facilitate the transfer of control of the given device. As a first step, kernel device module 310B requests any static data associated with the given device. In response to this request, kernel device module 305B sends the static data associated with the given device to kernel device module 310B. In one implementation, static data includes a number of VFs, a size of the frame buffer, data from configuration files, video basic input output system (BIOS) image, and/or other data. While these events are occurring, kernel device module 305B continues to act as the device driver for the given device. By continuing to act as the device driver for the given device, kernel device module 305B continues to process IOCTL and sysfs calls for the given device. Kernel device module 305B also continues to process mailbox interrupts, world switch interrupts, and debug signals for the given device.

Referring now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. <FIG> represents a continuation of the discussion of <FIG>. In other words, computing system <NUM> is intended to represent computing system <NUM> at a later point in time. Accordingly, kernel device module 305C, kernel device module 310C, and proxy module 315C are intended to represent kernel device module 305B, kernel device module 310B, and proxy module 315B (of <FIG>), respectively, at the later point in time. VMs 320C, 330C, and 340C and VFs 325C, 335C, and 345C also represent VMs 320B, 330B, and 340B and VFs 325B, 335B, and 345B (of <FIG>), respectively, at the later point in time.

At the point in time illustrated in <FIG>, the processing of mailbox interrupts, world switch interrupts, and debug signals by kernel device module 305C has been paused. Any interrupts or debug signals that are generated during this time for the given device that are destined for the kernel device module 305C will be queued. However, kernel device module 305C will continue to process IOCTL and sysfs calls at this point in time. In this implementation if one of the VMs 320B, 330B or 340B happens to be utilizing the given device they may continue to do so. Any interrupts or operations destined for the VMs may continue to be processed unhindered. Kernel device module 305C and kernel device module 310C will communicate with each other to coordinate the timing of the pausing of the processing of the interrupts and debug signals and the quiescing of kernel device module 305C. Once kernel device module 305C has been quiesced, kernel device module 305C sends transient data to kernel device module 310C. In one implementation, transient data includes the currently running VF, the state of the currently running VF, the order of VFs on the run list, event buckets, accumulated data, and/or other data. During the period of time where the kernel device module 305C has been quiesced any interrupts or operations destined for the VMs may continue to be processed unhindered.

Turning now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. Computing system <NUM> is intended to represent computing system <NUM> at a later point in time. Accordingly, kernel device module 305D, kernel device module 310D, and proxy module 315D are intended to represent kernel device module 305C, kernel device module 310C, and proxy module 315C, respectively, at the later point in time. Also, VMs 320D, 330D, and 340D as well as VFs 325D, 335D, and VF 345D represent VMs 320C, 330C, and 340C and VFs 325C, 335C, and VF 345C, respectively, at the later point in time.

At the point in time illustrated in <FIG>, the processing of mailbox interrupts, world switch interrupts, and debug signals has been restarted by kernel device module 310D. Any interrupts or debug signals that were generated during the time between when kernel device 305D stopped processing interrupts and debug signals and kernel device 310D started handling the processing of interrupts and debug signals will be processed out of the corresponding queues or status registers where they were recorded. Additionally, at the point in time represented by <FIG>, kernel device module 310D will start to process IOCTL and sysfs calls. Kernel device module 310D and kernel device module 305D communicate with each other to coordinate the transfer of control for handling the processing of IOCTL and sysfs calls.

Referring now to <FIG>, a sequence diagram of one implementation of the interaction between an OS, proxy module, and kernel device module is shown. In one implementation, when a modprobe call is made for kernel device module <NUM>, OS <NUM> resolves the dependencies of kernel device module <NUM>. Depending on the implementation, modprobe is executed manually or modprobe is automated as part of a startup script. When OS <NUM> determines that kernel device module <NUM> is dependent upon proxy module <NUM>, OS <NUM> will load and initialize proxy module <NUM> in step <NUM>. Proxy module <NUM> performs internal initialization in step <NUM> in preparation for kernel device module <NUM> to be loaded in step <NUM>. Then, after kernel device module <NUM> is loaded in step <NUM>, kernel device module <NUM> calls the function "register_driver" in step <NUM> to register with proxy module <NUM>. It is assumed for the purposes of this discussion that kernel device module <NUM> is the initial device driver to be loaded for a particular device of the system. Accordingly, since kernel device module <NUM> is the initial device driver to be loaded for the particular device, proxy module <NUM> marks its internal tables associated with this instance of kernel device module <NUM> as ACTIVE in step <NUM>, and a first time initialization of the hardware is performed for the particular device.

Rather than registering as a device driver with OS <NUM>, kernel device module <NUM> will register as a device driver with proxy module <NUM>. Proxy module <NUM> will act as a surrogate OS for kernel device module <NUM>. The bus (e.g., peripheral component interconnect (PCI)) device identifiers (IDs) are transparent to proxy module <NUM>. Proxy module <NUM> obtains the device IDs from kernel device module <NUM>. This adds an additional level of independence between versions of proxy module <NUM> and versions of kernel device module <NUM>. Proxy module <NUM> will register its own driver with OS <NUM> on behalf of kernel device module <NUM> using the device IDs provided by kernel device module <NUM>. Kernel device module <NUM> passes proxy module <NUM> a table of function pointers that proxy module <NUM> will use as callbacks into kernel device module <NUM>. For security purposes, proxy module <NUM> makes a copy of the function pointer table. This prevents kernel device module <NUM> from changing the table either maliciously on purpose or by accident (e.g., due to a bug).

In loop <NUM>, OS <NUM> searches for devices that match a given device ID and calls the. probe function of proxy module <NUM> for each matching device ID in step <NUM>. OS <NUM> will do this for all devices that match with the device IDs in the device ID table of kernel device module <NUM>. Upon receiving a call to the. probe function, proxy module <NUM> will not perform any hardware device initialization. Rather, the device initialization will be performed by kernel device module <NUM>. Proxy module <NUM> will call the. probe callback function for each matching device ID of kernel device module <NUM>. In one implementation, proxy module <NUM> calls the. probe function of kernel device module <NUM> with an extra parameter that is in addition to the OS parameters. This extra parameter indicates whether the load is an initial load of the driver or a replacement of a previous driver. For the implementation shown in sequence diagram <NUM>, proxy module <NUM> will use the extra parameter to notify kernel device module <NUM> that this is an initial load of the driver. In response to receiving the notification that this is an initial load of the driver, kernel device module <NUM> will perform a full hardware initialization.

Once all of the devices have been probed and initialized, kernel device module <NUM> will enter its normal runtime mode. During runtime, examples of stimuli that can affect the state of the device and kernel device module <NUM> may include a Mailbox interrupt, World Switch Timer interrupt, and notifications from the hardware emulator. A guest driver in the guest OS will use the Mailbox interrupt to generate a notification for kernel device module <NUM>. When kernel device module <NUM> receives a Mailbox interrupt, kernel device module <NUM> will schedule a work item on the Mailbox work queue for later processing. When OS <NUM> is available, OS <NUM> will execute the work on the work queue. In one implementation, kernel device module <NUM> processes world switch timer interrupts on a programmable interval. In one implementation, the programmable interval is in a range of <NUM> milliseconds (ms) to <NUM>. At every interval, kernel device module will schedule a world switch request in the world switch work queue.

In various implementations, the hardware emulator and kernel device module <NUM> communicate to each other through IOCTL calls (hardware emulator → kernel device module <NUM>) and sysfs calls (kernel device module <NUM> → hardware emulator). In one implementation, the hardware emulator is a quick emulator (QEMU). When the hardware emulator needs kernel device module <NUM> to process some work, the hardware emulator will open the IOCTL attribute, send the request, and then close the attribute. When kernel device module <NUM> returns from the IOCTL call the work has been completed. The work is done directly in kernel device module <NUM> in response to the call on the thread of the caller. There is no work queue involved as in the case of world switching and mailbox interrupts. When kernel device module <NUM> has work for the hardware emulator to do, kernel device module <NUM> will place a command in a buffer and signal an attribute that the hardware emulator instance is waiting on. The hardware emulator will read the buffer and respond with an IOCTL call when the hardware emulator has finished the work.

Kernel device module <NUM> will remain in loop <NUM> detecting and processing the stimuli until kernel device module <NUM> is either replaced or unloaded. In one implementation, during the normal runtime of kernel device module <NUM>, kernel device module <NUM> runs independently of proxy module <NUM>. In other words, in this implementation, proxy module <NUM> is not involved with the single-root input/output virtualization (SRIOV) operations of the device. In one implementation, proxy module <NUM> is responsible for the updating or replacing of kernel device module <NUM>. In this implementation, proxy module <NUM> is activated if either kernel device module <NUM> tries to unload from the system or if a newer kernel device module tries to replace kernel device module <NUM>.

Turning now to <FIG>, a sequence diagram of one implementation of the interactions when loading a new kernel device module <NUM> is shown. When a new kernel device module <NUM> is loaded onto the system, the new kernel device module <NUM> will register with proxy module <NUM> rather than registering with the OS. The new kernel device module version <NUM> has no knowledge that there is already an existing kernel device module <NUM> running. The new kernel device module version <NUM> will attempt to register with proxy module <NUM> the same way in which the existing kernel device module version <NUM> registered with proxy module <NUM> by calling a register_driver() given function to register with proxy module <NUM>. The new kernel device module version <NUM> passes in both a table of function pointers and a table of device IDs to proxy module <NUM> in step <NUM>.

In one implementation, loading of the new kernel device module <NUM> will trigger a live kernel device module update process, and no further user interaction is required. In one implementation, the existing kernel device module version <NUM> is removed on success or the new kernel device module version <NUM> is removed if an error is detected. One goal of the live kernel device module version update is to be able to update the kernel device module executable code without interrupting running client sessions. This includes both forward and backward updating of the code. In order to be able to do a complete update, it is desired that all of the live client sessions that are in existence before the update will also be active after the update. If the new kernel device module <NUM> supports the same devices that the existing kernel device module <NUM> supports, then new kernel device module <NUM> is registered as the new device driver for these devices. Alternatively, if the new kernel device module <NUM> cannot support the same devices that the existing kernel device module <NUM> supports, then the update is aborted with an error explanation logged.

Referring now to <FIG>, a sequence diagram of one implementation of the interactions when loading a new kernel device module <NUM> is shown. <FIG> is intended to be a continuation of the discussion of <FIG>. When transferring control of the device from existing kernel device module <NUM> to new kernel device module <NUM>, the static data is sent from existing kernel device module <NUM> to new kernel device module <NUM>. After the static data is transferred, existing kernel device module <NUM> is quiesced in order to transfer the transient data from existing kernel device module <NUM> to new kernel device module <NUM>.

In one implementation, in order to transfer sensitive transient data while transitioning to a new device driver, the state of the device should be stable. In this implementation, the device can still be running, but its state as known by the kernel device module cannot change. In one implementation, the stimuli that can change the state of the software and/or device are identified. In one implementation, these stimuli include Mailbox interrupts, World Switch interrupts, and notifications from the hardware emulator. In other implementations, other stimuli are identified and stopped to prevent the device's state from changing. Existing kernel device module <NUM> is quiesced in box <NUM> by stopping these identified stimuli to prevent any state change from occurring during the transfer of sensitive transient data.

Turning now to <FIG>, a sequence diagram of one implementation of quiescing an existing kernel device module version <NUM> is shown. It is assumed for the purposes of this discussion that new kernel device module version <NUM> is taking over as the device driver for a given device by replacing existing kernel device module version <NUM>. In one implementation, the given device is a GPU while in other implementations, the given device is any of various other devices (e.g., network controller, storage device). In order for existing kernel device module version <NUM> to be replaced, existing kernel device module version <NUM> will be quiesced. In one implementation, the quiescing of existing kernel device module version <NUM> is initiated by new kernel device module version <NUM>, but it is existing kernel device module version <NUM> that performs the various actions to reach a quiesced state. In one implementation, existing kernel device module version <NUM> is quiesced by stopping a plurality of threads which are executing on the host computing system. In one implementation, threads that are stopped include scheduler, mailbox, and IOCTL threads. In one implementation, each thread has an associated flag (per adapter) that will indicate whether or not the thread can accept new work.

In one implementation, the first thread to be stopped is the scheduler thread in step <NUM>. Setting the flag for the scheduler thread will indicate to the world switch scheduler in the kernel device module that the scheduler thread can no longer start new scheduler requests (i.e., prevent further world switches from occurring). The scheduler thread is stopped first since the mailbox thread may attempt to spawn requests to restart the scheduler. Note that the scheduler thread is still able to continue to process an inflight request, but the scheduler thread cannot start a new request once the scheduler thread has been stopped. If a call is made to restart the scheduler thread, this call will be ignored.

In one implementation, the scheduler thread is stopped in step <NUM> before stopping the mailbox thread in step <NUM>. When a guest driver is holding Exclusive Access Mode, the guest driver will also be holding the world switch mutex. The world switch mutex prevents world switching to any other virtual function (VF). The world switch mutex will be released when either the guest driver releases Exclusive Mode (via a Mailbox message) or times out. The Mailbox thread will give priority to requests to release Exclusive Mode over requests to grant Exclusive mode. If the Mailbox thread is actively running and there is a request to release Exclusive Mode, then this request will be inflight. The Mailbox thread is allowed to finish the release of Exclusive Mode before waiting for the scheduler thread to stop. Note that if Exclusive Mode is held and there is not a pending release of Exclusive Mode on the work queue, the scheduler thread can be stopped and the mutex state will be transferred to the kernel device module version <NUM> during the transfer of transient state. In other words, the live update does not need to wait until Exclusive Mode is completed before completing the live update.

The mailbox thread is stopped in step <NUM> to prevent the mailbox thread from accepting any new work. New work for the mailbox thread originates with the guest driver issuing requests to either enter or exit Exclusive Mode. In one implementation, rather than stopping the guest driver, mailbox interrupts are disabled. The guest driver can still try to trigger a Mailbox interrupt, but the Mailbox interrupt will not get processed. The interrupt request will be saved in the Mailbox status register and can be polled later to determine if there any work requests were generated while interrupts were disabled. The request to stop the Mailbox thread in step <NUM> can occur while the Mailbox thread is processing a work item. There can also be work items queued for the Mailbox thread. Setting the flag does not indicate that the Mailbox thread will be idle. The next step is to wait for the Mailbox thread to stop in step <NUM>.

After requesting to stop the Mailbox thread in step <NUM>, an inflight work item will be processed to completion, but a new work item will not be started. Any work items on the queue will be transferred to the new kernel device module version <NUM> for processing during the transfer of transient data. In one implementation, existing kernel device module version <NUM> waits for the Mailbox thread to stop at step <NUM> and the scheduler thread to stop before stopping the IOCTL thread in step <NUM>. Typically, a Mailbox work item will also require a response from the VMM. In order to receive the response and not deadlock, the IOCTL thread is permitted to continue running until the Mailbox thread has stopped in step <NUM>. After requesting for the IOCTL thread to stop in step <NUM>, existing kernel device module version <NUM> waits for the IOCTL thread to stop in step <NUM>. Existing kernel device module version <NUM> waits until any outstanding work has been completed before returning a successful response to new kernel device module version <NUM>. When any outstanding work has been completed, transient data is transferred from existing kernel device module version <NUM> to new kernel device module version <NUM> and then control of the device is passed to new kernel device module version <NUM>.

Turning now to <FIG>, one implementation of a method <NUM> for loading a kernel device module is shown. For purposes of discussion, the steps in this implementation and those of <FIG> are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method <NUM>.

A given version of a device driver registers with a proxy module, wherein the given version of the device driver is configured to manage interactions between software applications and a given device (block <NUM>). In one implementation, the device is a GPU. In other implementations, the device is any of various other types of processing or peripheral devices. Next, the proxy module determines if a previous version of the device driver is already loaded and executing on the system for the given device (conditional block <NUM>). If a previous version of the device driver is already loaded and executing on the system (conditional block <NUM>, "yes" leg), then the proxy module provides a first indication to the given version of the device driver, wherein the first indication specifies that a previous version of the device driver is already loaded (block <NUM>). The proxy module also provides, to the given version of the device driver, a pointer which points to the previous version of the device driver (block <NUM>). The given version of the device driver forgoes hardware initialization of the device responsive to receiving the first indication (block <NUM>). The given version of the device driver uses the pointer to communicate with the previous version of the device driver to handle the transfer of control of the device from the previous version of the device driver to the given version of the device driver (block <NUM>). After control of the device has been transferred from the previous version of the device driver to the given version of the device driver, the previous version of the device driver is deactivated (block <NUM>). After block <NUM>, method <NUM> ends.

If a previous version of the device driver is not already loaded and executing on the system (conditional block <NUM>, "no" leg), then the proxy module provides a second indication to the given version of the device driver, wherein the second indication is different from the first indication (block <NUM>). Also, the proxy module registers with the OS as the given version of the device driver (block <NUM>). In other words, the proxy module emulates the given version of the device driver to the OS. Also in response to receiving the second indication, the given version of the device driver causes the device to perform a hardware initialization routine (block <NUM>). Also in response to receiving the second indication, the given version of the device driver starts managing the device (block <NUM>). After block <NUM>, method <NUM> ends. It is noted that in various implementations, method <NUM> is performed for each new version of the device driver which is installed on the system.

Referring now to <FIG>, one implementation of a method <NUM> for quiescing a first version of a device driver in order to transfer control of the device from the first version of the device driver to a second version of the device driver is shown. A second version of a device driver receives an indication that a first version of a device driver is already managing a given device of a computing system (block <NUM>). In one implementation, the second version of the device driver receives the indication in response to registering with a proxy module. In response to receiving the indication, a request to stop a scheduler thread is generated (block <NUM>). In one implementation, the scheduler thread is stopped by setting a corresponding flag, which indicates to a guest driver that the scheduler thread can no longer start new scheduler requests. Next, a request to stop a mailbox thread is generated (block <NUM>). The request to stop the mailbox thread is generated in block <NUM> to prevent the mailbox thread from accepting any new work. Then, the first version of the device driver waits for the mailbox thread to stop (block <NUM>). Next, the first version of the device driver waits for the scheduler thread to stop (block <NUM>). Then, a request is generated to stop an IOCTL thread (block <NUM>). Next, the system waits for the IOCTL thread to stop (block <NUM>). Then, transient data is transferred from the first version of the device driver to the second version of the device driver (block <NUM>). Next, control of the device is transferred from the first version of the device driver to the second version of the device driver (block <NUM>). After block <NUM>, method <NUM> ends.

In various implementations, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions are represented by a high level programming language. In other implementations, the program instructions are compiled from a high level programming language to a binary, intermediate, or other form. Alternatively, program instructions are written that describe the behavior or design of hardware. Such program instructions are represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

Claim 1:
A system comprising:
a memory (<NUM>) storing program instructions of an operating system, OS (<NUM>), a proxy module (<NUM>), and a given version of a device driver (260A) configured to manage interactions between one or more software applications and a device; and
a processor (<NUM>) configured to:
register (<NUM>) the given version of the device driver with the proxy module rather than the operating system; and
execute program instructions of the proxy module to:
responsive to a previous version of the given version of the device driver already being on the system:
provide a first indication (<NUM>) from the proxy module to the given version of the device driver that indicates the previous version of the device driver is already loaded on the system and managing the device; and
transfer control of the device from the previous version of the device driver to the given version of the device driver, without hardware initialization of the device;
responsive to a previous version of the given version of the device driver not already being on the system:
provide a second indication (<NUM>), different from the first indication, to the given version of the device driver;
register, by the proxy module, with the OS as the given version of the device driver;
cause hardware initialization of the device, responsive to the second indication; and
manage, by the given device driver, the device.