Abstract:
A method for conveying a data packet received from a network to a virtual machine instantiated on a computer system coupled to the network, and a medium and system for carrying out the method, is described. In the method, a guest receive pointer queue of a component executing in the virtual machine is inspected in order to identify a location in a guest receive packet data buffer that is available to receive packet data. Data from the data packet received from the network is copied into the guest receive packet data buffer at the identified location. A standard receive interrupt is raised in the virtual machine. Thus, the kernel places the data packet received from the network into a memory space accessible to the virtual machine without any intervention by a virtual machine monitor component of the virtualization software.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application is a continuation of U.S. patent application Ser. No. 12/393,231 filed Feb. 26, 2009, which is a continuation of 10/383,702, filed Mar. 7, 2003, now U.S. Pat. No. 7,620,955, which is a continuation-in-part of U.S. patent application Ser. No. 09/877,378, filed Jun. 8, 2001, now U.S. Pat. No. 6,961,941. 
     
    
     BACKGROUND OF THE INVENTION 
       [0002]    1. Field of the Invention 
         [0003]    This invention relates to virtualized computer systems, in particular, to a system and method for improving the performance of network transfers to and from a virtual machine. 
         [0004]    2. Description of the Related Art 
         [0005]    The advantages of virtual machine technology have become widely recognized. Among these advantages is the ability to run multiple virtual machines on a single host platform. This makes better use of the capacity of the hardware, while still ensuring that each user enjoys the features of a “complete,” isolated computer. 
       General Virtualized Computer System 
       [0006]    As is well known in the field of computer science, a virtual machine (VM) is a software abstraction—a “virtualization”—of an actual physical computer system.  FIG. 1  illustrates, in part, the general configuration of a virtual machine  200 , which is installed as a “guest” on a “host” hardware platform  100 . 
         [0007]    As  FIG. 1  shows, the hardware platform  100  includes one or more processors (CPUs)  110 , system memory  130 , and a storage device, which will typically be a disk  140 . The system memory will typically be some form of high-speed RAM, whereas the disk (one or more) will typically be a non-volatile, mass storage device. The hardware  100  will also include other conventional mechanisms such as a memory management unit MMU  150 , various registers  160 , and any conventional network connection device  172  (such as a network adapter or network interface card—“NIC”) for transfer of data between the various components of the system and a bus or network  700 , which may be any known public or proprietary bus structure or local or wide-area network such as the Internet, an internal enterprise network, etc. 
         [0008]    Each VM  200  will typically include at least one virtual CPU  210 , a virtual disk  240 , a virtual system memory  230 , a guest operating system  220  (which may simply be a copy of a conventional operating system), and various virtual devices  270 , in which case the guest operating system (“guest OS”) will include corresponding drivers  224 . All of the components of the VM may be implemented in software using known techniques to emulate the corresponding components of an actual computer. 
         [0009]    If the VM is properly designed, then it will not be apparent to the user that any applications  260  running within the VM are running indirectly, that is, via the guest OS and virtual processor. Applications  260  running within the VM will act just as they would if run on a “real” computer, except for a decrease in running speed that will be noticeable only in exceptionally time-critical applications. Executable files will be accessed by the guest OS from the virtual disk or virtual memory, which will simply be portions of the actual physical disk or memory allocated to that VM. Once an application is installed within the VM, the guest OS retrieves files from the virtual disk just as if they had been pre-stored as the result of a conventional installation of the application. The design and operation of virtual machines is well known in the field of computer science. 
         [0010]    Some interface is usually required between a VM and the underlying host platform (in particular, the CPU), which is responsible for actually executing VM-issued instructions and transferring data to and from the actual memory and storage devices. A common term for this interface is a “virtual machine monitor” (VMM), shown as component  300 . A VMM is usually a thin piece of software that runs directly on top of a host, or directly on the hardware, and virtualizes all the resources of the physical host machine. Among other components, the VMM therefore usually includes device emulators  330 , which may constitute the virtual devices  270  that the VM  200  addresses. The interface exported to the VM is then the same as the hardware interface of the machine, so that the guest OS cannot determine the presence of the VMM. 
         [0011]    The VMM also usually tracks and either forwards (to some form of operating system) or itself schedules and handles all requests by its VM for machine resources, as well as various faults and interrupts. A mechanism known in the art as an exception or interrupt handler  355  is therefore included in the VMM. As is well known, such an interrupt/exception handler normally includes an interrupt descriptor table (IDT), or some similar table, which is typically a data structure that uses information in the interrupt signal to point to an entry address for a set of instructions that are to be executed when the interrupt /exception occurs. 
         [0012]    Although the VM (and thus the user of applications running in the VM) cannot usually detect the presence of the VMM, the VMM and the VM may be viewed as together forming a single virtual computer. They are shown in  FIG. 1  as separate components for the sake of clarity. 
         [0013]    Moreover, the various virtualized hardware components such as the virtual CPU(s)  210 , the virtual memory  230 , the virtual disk  240 , and the virtual device(s)  270  are shown as being part of the VM  200  for the sake of conceptual simplicity—in actual implementations these “components” are usually constructs or emulations exported to the VM by the VMM. For example,  FIG. 2  shows a virtual NIC  272  as being within the VM  200 . This virtual component, which may be one of the virtual devices  270 , may in fact be implemented as one of the device emulators  330  in the VMM. One advantage of such an arrangement is that the VMM may be set up to expose “generic” devices, which facilitate VM migration and hardware platform-independence. 
       Virtual and Physical Memory 
       [0014]    As in most modern computers, the address space of the memory  130  is partitioned into pages (for example, in the Intel x86 architecture), regions (for example, Intel IA-64 architecture) or other analogous units. Applications then address the memory  130  using virtual addresses (VAs), which include virtual page numbers (VPNs). The VAs are then mapped to physical addresses (PAs) that are used to address the physical memory  130 . (VAs and PAs have a common offset from a base address, so that only the VPN needs to be converted into a corresponding PPN.) The concepts of VPNs and PPNs, as well as the way in which the different page numbering schemes are implemented and used, are described in many standard texts, such as “Computer Organization and Design: The Hardware/Software Interface,” by David A. Patterson and John L. Hennessy, Morgan Kaufmann Publishers, Inc., San Francisco, Calif., 1994, pp. 579-603 (chapter 7.4 “Virtual Memory”). Similar mappings are used in region-based architectures or, indeed, in any architecture where relocatability is possible. 
         [0015]    An extra level of addressing indirection is typically implemented in virtualized systems in that a VPN issued by an application  260  in the VM  200  is remapped twice in order to determine which page of the hardware memory is intended. The first mapping is provided by a mapping module within the guest OS  202 , which translates the guest VPN (GVPN) into a corresponding guest PPN (GPPN) in the conventional manner; because the address offsets are the same, this is the same as translating guest physical addresses (GPAs) into actual physical (machine) addresses (PAs). The guest OS therefore “believes” that it is directly addressing the actual hardware memory, but in fact it is not. 
         [0016]    Of course, a valid address to the actual hardware memory must ultimately be generated. A memory management module  350 , located typically in the VMM  300 , therefore performs the second mapping by taking the GPPN issued by the guest OS  220  and mapping it to a hardware (or “machine”) page number PPN that can be used to address the hardware memory  130 . This GPPN-to-PPN mapping may instead be done in the main system-level software layer (such as in a mapping module  617  in the kernel  600 , as illustrated in  FIG. 2  and described further below), depending on the implementation: From the perspective of the guest OS, the GVPN and GPPN might be virtual and physical page numbers just as they would be if the guest OS were the only OS in the system. From the perspective of the system software, however, the GPPN is a page number that is then mapped into the physical memory space of the hardware memory as a PPN. 
       System Software Configurations in Virtualized Systems 
       [0017]    In some systems, such as the Workstation product of VMware, Inc., of Palo Alto, Calif., the VMM is co-resident at system level with a host operating system. Both the VMM and the host OS can independently modify the state of the host processor, but the VMM calls into the host OS via a driver and a dedicated user-level application to have the host OS perform certain I/O operations of behalf of the VM. The virtual computer in this configuration is thus fully hosted in that it runs on an existing host hardware platform and together with an existing host OS. 
         [0018]    In other implementations, a dedicated kernel takes the place of and performs the conventional functions of the host OS, and virtual computers run on the kernel.  FIG. 1  illustrates a kernel  600  that serves as the system software for several VM/VMM pairs  200 / 300 , . . . ,  200   n / 300   n . Compared with a system in which VMMs run directly on the hardware platform, use of a kernel offers greater modularity and facilitates provision of services that extend across multiple VMs (for example, for resource management). Compared with the hosted deployment, a kernel may offer greater performance because it can be co-developed with the VMM and be optimized for the characteristics of a workload consisting of VMMs. The ESX Server product of VMware, Inc., has such a configuration. The invention described below takes advantage of the ability to optimize a kernel as a platform for virtual computers. 
         [0019]    A kernel-based virtualization system of the type illustrated in  FIG. 1  is described in U.S. patent application Ser. No. 09/877,378 (“Computer Configuration for Resource Management in Systems Including a Virtual Machine”), which is incorporated here by reference. The main components of this system and aspects of their interaction are, however, outlined below. 
         [0020]    At boot-up time, an existing operating system  420  may be at system level and the kernel  600  may not yet even be operational within the system. In such case, one of the functions of the OS  420  may be to make it possible to load the kernel  600 , after which the kernel runs on the native hardware  100  and manages system resources. In effect, the kernel, once loaded, displaces the OS  420 . Thus, the kernel  600  may be viewed either as displacing the OS  420  from the system level and taking this place itself, or as residing at a “sub-system level.” When interposed between the OS  420  and the hardware  100 , the kernel  600  essentially turns the OS  420  into an “application,” which has access to system resources only when allowed by the kernel  600 . The kernel then schedules the OS  420  as if it were any other component that needs to use system resources. 
         [0021]    The OS  420  may also be included to allow applications unrelated to virtualization to run; for example, a system administrator may need such applications to monitor the hardware  100  or to perform other administrative routines. The OS  420  may thus be viewed as a “console” OS (COS). In such implementations, the kernel  600  preferably also includes a remote procedure call (RPC) mechanism to enable communication between, for example, the VMM  300  and any applications  800  installed to run on the COS  420 . 
       Actions 
       [0022]    In kernel-based systems such as the one illustrated in  FIG. 1 , there must be some way for the kernel  600  to communicate with the VMM  300 . In general, the VMM  300  can call into the kernel  600  but the kernel cannot call directly into the VMM. The conventional technique for overcoming this is for the kernel to post “actions” (requests for the VMM to do something) on an action queue  1360  (see  FIG. 2 ) stored in memory  130 . As part of the VMM code, the VMM looks at this queue  1360  periodically, and always after it returns from a kernel call and also before it resumes a VM. One typical action, used in this invention (described further below), is the “raise interrupt” action: If the VMM sees this action it will raise an interrupt to the VM  200  in the conventional manner. 
         [0023]    As is known, for example, from U.S. Pat. No. 6,397,242 (Devine, et al., 28 May 2002), some virtualization systems allow VM instructions to run directly (in “direct execution”) on the hardware CPU(s) when possible. When necessary, however, VM execution is switched to the technique known as “binary translation,” during which the VM is running in the VMM. In any systems where the VM is running in direct execution when it becomes necessary for the VMM to check actions, the kernel must interrupt the VMM so that it will stop executing VM instructions and check its action queue  1360 . This may be done using known programming techniques. 
       Worlds 
       [0024]    The kernel  600  handles not only the various VMM/VMs, but also any other applications running on the kernel, as well as the COS  420  and even the hardware CPU(s)  110 , as entities that can be separately scheduled. In this disclosure, each schedulable entity is referred to as a “world,” which contains a thread of control, an address space, machine memory, and handles to the various device objects that it is accessing. Worlds, represented in  FIG. 1  within the kernel  600  as module  612 , are stored in a portion of the memory space controlled by the kernel. Each world also has its own task structure, and usually also a data structure for storing the hardware state currently associated with the respective world. 
         [0025]    There will usually be different types of worlds: 1) system worlds, which are used for idle worlds, one per CPU, and a helper world that performs tasks that need to be done asynchronously; 2) a console world, which is a special world that runs in the kernel and is associated with the COS  420 ; and 3) virtual machine worlds. 
         [0026]    Worlds preferably run at the most-privileged level (for example, in a system with the Intel x86 architecture, this will be level CPL 0 ), that is, with full rights to invoke any privileged CPU operations. A VMM, which, along with its VM, constitutes a separate world, therefore may use these privileged instructions to allow it to run its associated VM so that it performs just like a corresponding “real” computer, even with respect to privileged operations. 
       Switching Worlds 
       [0027]    When the world that is running on a particular CPU (which may be the only one) is preempted by or yields to another world, then a world switch has to occur. A world switch involves saving the context of the current world and restoring the context of the new world such that the new world can begin executing where it left off the last time that it is was running. 
         [0028]    The first part of the world switch procedure that is carried out by the kernel is that the current world&#39;s state is saved in a data structure that is stored in the kernel&#39;s data area. Assuming the common case of an underlying Intel x86 architecture, the state that is saved will typically include: 1) the exception flags register; 2) general purpose registers; 3) segment registers; 4) the instruction pointer (EIP) register; 5) the local descriptor table register; 6) the task register; 7) debug registers; 8) control registers; 9) the interrupt descriptor table register; 10) the global descriptor table register; and 11) the floating point state. Similar state information will need to be saved in systems with other hardware architectures. 
         [0029]    After the state of the current world is saved, the state of the new world can be restored. During the process of restoring the new world&#39;s state, no exceptions are allowed to take place because, if they did, the state of the new world would be inconsistent upon restoration of the state. The same state that was saved is therefore restored. The last step in the world switch procedure is restoring the new world&#39;s code segment and instruction pointer (EIP) registers. 
         [0030]    When worlds are initially created, the saved state area for the world is initialized to contain the proper information such that when the system switches to that world, then enough of its state is restored to enable the world to start running. The EIP is therefore set to the address of a special world start function. Thus, when a running world switches to a new world that has never run before, the act of restoring the EIP register will cause the world to begin executing in the world start function. 
         [0031]    Switching from and to the COS world requires additional steps, which are described in U.S. patent application Ser. No. 09/877,378, mentioned above. Understanding of the details of this process is not necessary for understanding the present invention, however, so further discussion is omitted. 
       Memory Management in Kernel-Based System 
       [0032]    The kernel  600  includes a memory management module  616  that manages all machine memory that is not allocated exclusively to the COS  420 . When the kernel  600  is loaded, the information about the maximum amount of memory available on the machine is available to the kernel, as well as information about how much of it is being used by the COS. Part of the machine memory is used for the kernel  600  itself and the rest is used for the virtual machine worlds. 
         [0033]    Virtual machine worlds use machine memory for two purposes. First, memory is used to back portions of each world&#39;s memory region, that is, to store code, data, stacks, etc., in the VMM page table. For example, the code and data for the VMM  300  is backed by machine memory allocated by the kernel  600 . Second, memory is used for the guest memory of the virtual machine. The memory management module may include any algorithms for dynamically allocating memory among the different VM&#39;s  200 . 
       Interrupt and Exception Handling in Kernel-Based Systems 
       [0034]    Interrupt and exception handling is related to the concept of “worlds” described above. As mentioned above, one aspect of switching worlds is changing various descriptor tables. One of the descriptor tables that is loaded when a new world is to be run is the new world&#39;s IDT. The kernel  600  therefore preferably also includes an interrupt/exception handler  655 , that is able to intercept and handle (using a corresponding IDT in the conventional manner), interrupts and exceptions for all devices on the machine. When the VMM world is running, whichever IDT is currently loaded is replaced by the VMM&#39;s IDT, such that the VMM will handle all interrupts and exceptions. 
         [0035]    The VMM will handle some interrupts and exceptions completely on its own. For other interrupts/exceptions, it will be either necessary or at least more efficient for the VMM to call the kernel to have the kernel either handle the interrupts/exceptions itself, or to forward them to some other sub-system such as the COS. One example of an interrupt that the VMM can handle completely on its own, with no call to the kernel, is a check-action IPI (inter-processor interrupt), which is described below. One example of when the VMM preferably calls the kernel, which then forwards an interrupt to the COS, would be where the interrupt involves devices such as a mouse, which is typically controlled by the COS. The VMM may forward still other interrupts to the VM. 
       Device Access in Kernel-Based System 
       [0036]    In the preferred embodiment of the invention, the kernel  600  is responsible for providing access to all devices on the physical machine, in particular, to the NIC  172 . In addition to other modules that the designer may choose to load into the kernel, the kernel will therefore typically include conventional drivers as needed to control access to devices. Accordingly,  FIG. 1  shows within the kernel  600  a module  610  containing loadable kernel modules and drivers. 
       Conventional Networking and Packets 
       [0037]    In conventional non-virtualized systems, data transfer between an application and various devices  400 - 1 ,  400 - 2 , . . . ,  400 - m  often takes place over a shared or dedicated communication channel such as the bus or network  700 . It is assumed here that data transfer between the system hardware  100  and each device  400 - 1 ,  400 - 2 , . . . ,  400 - m  takes place in units such as “packets”; other types of devices may of course also be connected to the hardware  100 , both directly and via the network. 
         [0038]    Each device may be considered to be a separate “target” or “destination” when it comes to data transfer. A hardware device controller  175  is also typically included for each device, or for each group of devices that share the bus  700  and communicate using a common protocol. In  FIG. 1 , only one such device controller  175  is shown, merely for the sake of simplicity. A conventional driver is also loaded in the operating system in order to support the hardware controller  175 . 
         [0039]    Assume by way of a very common example that the devices  400 - 1 ,  400 - 2 ,  400 - m  are USB devices. Whenever some “source” sub-system or process, such as an application, initiates a request for transfer of a block of data D to a USB device, that is, an OUT operation, it establishes a buffer in memory  130  in which it stores the data D. The source sub-system then generates a corresponding transfer request to indicate to the controller&#39;s driver that it should begin the procedure (described below) for transferring the data set D. The buffer is also established for data that is to be input from the USB device that is, for an IN operation. Note that, in other systems, according to other protocols, the controller driver may be responsible for establishing the buffer. 
         [0040]    The driver then splits the source&#39;s data request into sub-blocks whose size is chosen to be consistent with bus bandwidth requirements and bus (for example, USB) protocol mechanisms. For the sake of illustration, assume that the source data set D is subdivided into three sub-sets or “sub-blocks”0 D1, D2, and D3. In most practical cases, the number of sub-blocks will be much greater, depending on the size of the original data set D. Each sub-block D1, D2, and D3 is used as the basis for a single “transaction,” which results in the data sub-block being transferred from the source&#39;s buffer to the USB device, or vice versa. The transfer procedure is typically the same regardless of the number of transactions. 
         [0041]    The “raw” data sub-sets D1, D2, D3, etc., alone are generally not enough to adequately define the parameters of a desired transfer. Rather, each sub-set is usually included in or referenced by another data structure that also specifies such information as the destination, the direction of transfer (IN or OUT), the size of the data sub-set to be transferred, etc. In the USB context, the data structures used for this purpose are known as “transfer descriptors” (TDs). Similar descriptors are usually also created for data transfer using other protocols. Continuing with the example of transfer according to the USB protocol, the driver then builds a list of pending transactions that are targeted for one or more USB devices attached to the bus  700 . Each TD defines one transaction. The TDs are also stored in memory, in particular, a TD buffer established for the purpose. 
         [0042]    In USB-based systems, at a predefined interval, the controller  175  begins to take TDs as inputs, usually (but not necessarily) one at a time, and from each TD and its respective data sub-block creates a data structure known as a “packet.” The controller then transfers the packets sequentially to the bus  700  via a hub (not shown). The concept of a “packet” has somewhat varying definitions in the literature, but is used here to refer to the data structure(s) used to transfer a single data sub-block D1, D2, and D3 to or from at least one destination (usually, a device) via the bus. 
         [0043]    In order to guarantee data delivery, during a “handshake” packet phase, the target device returns to the sender (here: controller  175 ) information in the form of a packet indicating whether the transaction was successful, whether it failed, or whether the intended target device was busy. If no signal is transmitted back to the controller within a predetermined time, then the controller assumes that the packet transfer failed. In the case of a failed packet transfer, assuming any information is returned at all, the returned information normally includes at least the number of bytes that transferred successfully before the failure, and also usually a flag indicating what the error was. In the case of a busy device, the controller typically attempts to resubmit the packet, and may continue to do so until the transfer is success or fails. 
         [0044]    Input of data from a device, that is, an IN operation, is also carried out in the form of packets, with the same protocol. As with OUT operations, TDs are generated that define the destination, buffer address, etc. of a data sub-set, but the result of transmission of a packet derived from such a TD is that the data sub-set is input from the destination and placed in the buffer. In short, input of a packet of data is handled in essentially the same manner as packet output, with the obvious difference that the direction in which the corresponding data sub-set is transferred is the opposite. Note that information (in particular, at least one TD) is transmitted from the initiating component to the network (and on to the target device) for both IN and OUT operations. 
       Conventional Networking in Virtualized Systems 
       [0045]    The description above relates to conventional computer systems, but applies also, with some extensions, to virtualized computer systems that run as “guests” on a underlying “host” hardware and software platform. According to the prior art, packet-based data transfer between a source (such as one of the applications  260 ) within the VM and a physical device (destination) is essentially the same as described above in the non-virtualized context, with the exception that the transfer is “duplicated”: The source data block D is first transferred (usually, copied) from the transfer-requesting source process into a buffer, which is normally established by the source process itself (the normal case) but could alternatively be established by a driver installed in the guest OS  220 . This “guest” driver, which is analogous to (and in many cases an identical copy of) the driver in the actual, “host” OS, then builds a list of TDs from the buffered data and stores the TDs in the VM&#39;s memory space. 
         [0046]    A virtual device controller (a software analog of the controller  175 ) then constructs packets from the TDs and corresponding data sub-blocks, and passes them sequentially to what it “believes” is a bus. In fact, however, the VM-issued packets are received (in particular, intercepted) by an emulated bus within the VMM. The VMM in turn passes each VM-issued packet to the system software and hardware, which places the (or a corresponding) packet on the “real” bus  700 . Note that the device to which (or from which) the packets are to be sent (or received) is typically one of the physical devices  400 - 1 ,  400 - 2 , . . . ,  400 - m , although these may also be emulations. 
         [0047]    As can be understood from the discussion above, with respect to packet-based transfer, the VM is designed and intended to act just like a conventional non-virtualized system, the major structural difference being that the various hardware components involved, including the controller and the bus, are implemented in software. Again, with respect to packet transfer, the VM/VMM interface is essentially a software “copy” of the hardware  100 /bus  700  interface. 
       Shortcomings of the Prior Art 
       [0048]    A well known goal of all networking is increased transfer speed. Unfortunately, the known method for VM networking described above has several structural and procedural features, each of which introduces delay. Delay is caused, for example, by each of the following: 
         [0049]    1) transitions within the host systems for both transmitting and receiving; 
         [0050]    2) transitions between the VM and the VMM; 
         [0051]    3) transitions between the VMM and the kernel; and 
         [0052]    4) the need to copy data. 
         [0053]    What is needed is a way to provide faster network I/O to and from a VM by eliminating some, and preferably all, of the causes of delay listed above. This invention provides a system configuration and method of operation that accomplishes this goal. 
       SUMMARY OF THE INVENTION 
       [0054]    The invention provides a method for transferring (transmitting and/or receiving) a data set between a virtual computer and a destination over a network as a sequence of packets. The virtual computer runs on a “real” (physical) host computer, which includes both conventional system hardware and system software. The system hardware thus includes a memory, which has a physical address space, and a physical network interface device. The virtual computer includes a virtual network interface device, to which it issues packets to be transferred according to a protocol in which the packets are transferred individually as units. 
         [0055]    According to the invention, a region in memory is mapped as a shared memory region, which lies within the addressable memory space of both the virtual computer and the system software. A plurality of pending packets issued by the virtual computer are simultaneously stored in the shared memory space. In the most common anticipated implementation of the invention each packet has a guest packet address and packet data, the packet data of at least one of the packets being a sub-set of the data set to be transferred. 
         [0056]    Upon occurrence of a transfer request, the plurality of pending packets is transferred between the shared memory space and the network via the physical network interface device. In other words, transmit and/or receive packets are clustered before being sent on the physical network interface device for actual transfer over the network. 
         [0057]    In the preferred embodiment of the invention, the virtual computer comprises a virtual machine (VM), which includes a guest operating system (OS), and a virtual machine monitor (VMM), which forms an interface between the VM and the system software. The VM has a guest address space, such that each guest address corresponds to a physical address. The transfer request is preferably generated from within the VM, preferably by a driver installed in the guest OS, and is detected in the VMM. 
         [0058]    One way to generate the transfer request is as a trap into the VMM, which may be deferred until the occurrence of a triggering event. One such triggering event is receipt by the virtual network interface device of an acknowledgement signal generated by the destination acknowledging transfer of at least one earlier transferred packet; in other words, a current transfer is initiated upon acknowledgement of the previous transfer. 
         [0059]    In order to avoid packets waiting too long for the acknowledgement signal to arrive, the rate of transmission of the pending packets by the physical network interface device may also be determined. As long as this rate is less than a predetermined minimum, pending packets may be transferred from the shared memory space to the network as soon as the pending packets are placed in the shared memory space. In other words, transmit and receive clustering may be turned off if actual transfer is found to be too slow and turned back on when the transfer rate is acceptably fast. 
         [0060]    Another possible triggering event is that the number of pending (unprocessed) packets exceeds a predetermined maximum; the number may be either counted or estimated. One way to estimate how many packets are pending is by determining the how long it has been since the most recent receipt by the virtual network interface device of an acknowledgement signal; the triggering event is then the time since the most recent receipt exceeds a predetermined maximum. 
         [0061]    The invention provides for clustering of both transmit and receive packets. In the context of transmits, the packet data of the pending packets are stored at the respective guest packet addresses in the shared memory region; guest address pointers to the pending packet&#39;s packet data are also stored in the shared memory region. When the transfer request is detected, the physical address(es) at which the packet data is stored are passed to control software of the physical network interface device; the physical network interface device can then transfer the pending packets between the VM and the destination using its conventional procedure. Depending on the system software, it is also possible according to the invention to combine all the pending packets into a single contiguous packet and then to pass the physical address of the single packet to the control software of the physical network interface device. 
         [0062]    When the transfer request is detected, the VM is preferably prevented from VM from modifying the guest address pointers and the packet data for the stored, pending packets. The guest address to which each guest address pointer points is then translated into the corresponding physical address. After completed transfer by the physical network interface device of the packets pointed to by the guest address pointers, the VM&#39;s ability to modify the guest address pointers and the packet data may then be restored. 
         [0063]    According to one refinement of the invention, a portion of the guest memory space sufficient to hold a maximum number of pending transmit packets is pre-allocated. The guest address pointers are then initialized to point into the pre-allocated portion of the guest memory space and a mapping is pre-computed of each address contained in the guest address pointers from the guest memory space to the physical memory space. 
         [0064]    In the context of receiving packets, the invention provides, in the shared memory region, a guest receive pointer queue that stores guest address pointers that point into a guest packet data buffer, which has storage locations for respective received packet data sub-sets. Data from a plurality of packets received from the physical network interface device is copied into respective available storage locations in the guest receive packet data buffer. An interrupt is then issued to the VM, which then dismisses the interrupt and processes the received packets. 
         [0065]    As with transmit clustering it is possible to delay issuing the interrupt to the VM until the number of unprocessed received packets exceeds a predetermined maximum. Alternatively, the longest time any currently pending, unprocessed packet has been in the guest receive packet data buffer can be measured and the interrupt may be delayed until this time exceeds a predetermined maximum; in other words, packets may wait for processing, but not for more than some maximum time. 
         [0066]    The invention also provides a mechanism for offloading interrupts in a system that includes multiple processors (CPUs); a programmable interrupt control sub-system; a system software kernel that has a kernel interrupt handler; and a destination entity that is running on a first one of the CPUs, which forms a destination CPU, and that has a destination interrupt handler. 
         [0067]    According to this aspect of the invention, the kernel selects a second one of the CPUs—preferably an idle or at least most idle CPU—as a local CPU. The kernel also sets the programmable interface control sub-system to forward interrupts from a source entity (such as a device, in particular, the physical network interface device) to the local CPU, such that the local CPU calls the kernel interrupt handler upon receipt of interrupts from the source entity. Any interrupt raised by the source entity is sensed by the programmable interface control sub-system, which then causes the kernel interrupt handler to be called by the local CPU. The kernel then sends an interrupt vector to the destination CPU—the interrupt vector contains information for directing the destination CPU to call the destination interrupt handler. 
         [0068]    As a further aspect of interrupt offloading according to the invention, execution of destination entity instructions is allowed to proceed uninterrupted on the destination CPU during the time from sensing the interrupt raised by the source entity at least until the time the interrupt vector is received by the destination CPU. 
         [0069]    In the preferred embodiment of the invention, the interrupt vector forms an index into an interrupt descriptor table (IDT) and identifies an entry in the IDT that points to the location of a predetermined interrupt-handling routine in the destination interrupt handler. Also in the preferred embodiment of the invention, the destination entity is a virtual computer that includes a virtual machine (VM) and a virtual machine monitor (VMM); the VMM forms an interface between the VM and the destination CPU and exports a hardware interface to the VM; and the destination interrupt handler is in the VMM. 
         [0070]    In a preferred implementation of the invention, the source entity is a network interface device; the programmable interrupt control sub-system comprises a CPU interrupt controller for each CPU and an I/O interrupt controller; and the I/O interrupt controller senses the occurrence of the interrupt raised by the source entity and the CPU interrupt controller of the destination CPU senses the interrupt vector. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0071]      FIG. 1  illustrates the main components of kernel-based, virtualized computer system. 
           [0072]      FIG. 2  illustrates the components and control flow in the VM networking system according to the invention. 
           [0073]      FIG. 3  illustrates various memory data structures used in the invention. 
           [0074]      FIG. 4  illustrates interrupt offloading according to the invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0075]    In broadest terms, two features of the invention enable it to overcome the delay-causing disadvantages of the prior art. First, to eliminate the need for host transitions, NIC drivers (one or more, depending on the number of NICs included) are installed and run within the kernel  600 . This allows all networking to be done by direct kernel interactions. Second, a novel virtual networking interface is provided that minimizes and, in some cases eliminates, the need for VM-to-VMM and VMM-to-kernel transitions, as well as the need to copy data. The interface according to the invention has the added advantage of simplicity and thus avoids the complexity of emulating a standard interface. These features of the invention are described below in greater detail. 
         [0076]    In the description of the invention below, it is assumed that the system hardware  100  has the Intel x86 architecture only because this is the platform on which the ESX Server product of VMware, Inc., currently runs and it is the ESX Server in which a prototype of the invention has been implemented and tested. The mechanisms provided by the invention are not specific to the x86 architecture, however. Those skilled in the art of operating systems-level programming will know how to adapt the embodiment of the invention described below for use with other architectures. 
         [0077]    The invention involves a system and method for transferring a data set over the network according to any protocol in which the data set is converted into a sequence of data sub-sets that are transferred as units. According to the common USB protocol, these sub-sets are referred to as “packets.” For the sake of simplicity, the term “packet” is used below to refer to any data sub-set transferred as a unit over the network, regardless of the protocol. 
         [0078]    The unique components of the VM networking system according to the preferred embodiment of the invention are illustrated in  FIG. 1 . Various hardware and software components shown in  FIG. 1  are omitted in  FIG. 2  merely for the sake of simplicity and clarity; those skilled in the art of networking software will realize which of these standard components will be used by the invention. Also for the sake of simplicity, the device(s) are labeled collectively with the number  400 . Moreover, the memory  130  is shown separated from the other system hardware  100  merely to make it easier to describe certain structures used by the invention. 
         [0079]    The unique features of the invention are the following:
       a driver (“vmxnet driver”)  225  that runs in the guest (VM  200 ) and an emulation component (“vmxnet emulation”)  360  that runs in the VMM  300 ; As will become clearer below, the driver  225  and the emulation component  360  form, in cooperation, a virtual networking interface that the VMM emulates for the guest (VM  200 );   an implementation component (“vmxnet implementation”)  611  that runs in the kernel  600 ;   a shared memory region  132  (both  FIG. 2  and  FIG. 3 ) that is mapped as shared between the vmxnet driver, the VMM  300  (and thus available to the emulation component  360 ), and the kernel  600  (and thus available to the implementation component  611 ); and   the physical NIC driver  614 , which, according to the invention, is loaded in and runs in the kernel  600 .       
 
         [0084]    Other than the memory  130 , all of these features are software and as such are instructions that are either stored on the disk  140  or are loaded into the system memory  130  for execution by the processor(s)  110 . The VM  200  also includes (or the VMM exports) a virtual NIC  272 , which any source within the VM “believes” is the device handling network transmission. In reality, of course, the physical NIC  172  performs this function. The design and operation of virtual components such as the virtual NIC  272  are well understood in the art. 
         [0085]      FIG. 3  illustrates various structures that are stored in the memory  130  for use by the invention. These structures include:
       a guest transmit pointer queue  1320  and a guest receive pointer queue  1322 ;   a guest transmit packet data buffer  1321  and a guest receive pointer queue  1322 ;   a physical transmit pointer queue  1310  and a physical receive pointer queue  1312 ;   a physical transmit packet data buffer  1311  and a physical receive pointer queue  1313 ;   an optional overflow queue  1370 ; and   optional parameters T max , T def , R max , and R def , whose purpose is explained below.       
 
         [0092]    The shared memory portion  132 , as well as the guest packet data buffers  1321 ,  1323 , lie within the guest physical address space, that is, in the space allocated for the virtual memory  230 . In  FIG. 3 , this space is shown within the dashed line  231 . As is discussed above, whenever an entity in the VM needs to access memory within this space, it uses either a guest PPN directly, or a guest VPN, which the guest OS maps to a guest PPN. Of course, the guest physical address space  231  resides in the actual machine memory  130  and, as such, is ultimately always accessed using actual (machine) physical addresses. 
         [0093]    Some map of guest virtual addresses (page numbers) to guest physical addresses (page numbers) will also be needed in most embodiments of the invention. 
         [0094]    Both Linux and Windows operating systems provide functions to accomplish this mapping within the guest VM. For efficiency when the guest OS is a version of Windows, however, the vmxnet driver  225  preferably precomputes the GVPN-to-GPPN mapping, which is illustrated in  FIG. 2  as map  217 . 
         [0095]    The vmxnet driver  225  then calls the VMM  600  with the GPPNs for the structure  132 . The driver  225  also preferably fills in receive buffer pointers inside of the shared data structure  132  and, in the case of non-zero-copy transmits in Windows (see below) also fills in the transmit buffer pointers. The kernel  600  then creates corresponding a GPPN-to-PPN map; this map will typically be a part of the general memory map  617  the kernel keeps for the VM, although this is not necessary. 
         [0096]    The entries in the guest pointer queues  1320 ,  1322  are GPPNs that point to respective entries in the buffers  1321  and  1323 . What this means is that the vmxnet driver  225  can access all of the structures within the guest memory space  231  using GPPNs alone. In order for the kernel  600  to access the shared memory structures or the guest transmit/receive packet data buffers, it must have the corresponding PPNs (machine). The kernel  600  gets these required PPNs by consulting the GPPN-to-PPN memory map it has earlier established for these structures. 
         [0097]    The remaining memory structures shown in FIG.  3 —the physical pointer queues  1310 ,  1312 , the physical packet data buffers  1311 ,  1313 , and the overflow queue  1370 , lie outside the guest address space  231  of the VM, but can be accessed by the VMM and kernel using actual physical (machine) addresses. 
         [0098]    The uses of these various memory structures are explained below. 
         [0099]    As with any other networking system, the invention must provide for two main operations, namely, transmits and receives between some source entity or component and some destination entity or component. In this invention, the source entity or component is anything within the VM  200  that needs to communicate over the network  700 . The designation “source” does not imply that it is this component that is sending data blocks to some remote device such as a printer (although this is of course the case for transmits), but rather that it is the source or initiator of the network transaction, regardless of whether this is an IN or an OUT. Similarly, “destination” does not necessarily mean that the entity or component referred to is the recipient of a data set sent by the source, but rather merely that it is the entity or component with which the source wants to communicate via the network; in other words, the destination is simply the source&#39;s counterpart in a network transaction. The two principle network operations—transmit and receive—will now be described separately. 
         [0100]    Consider now the way in which the kernel  600 —or any standard operating system—handles packet transmission and reception, even in systems with no virtual machine. For each packet to be transmitted, the packet data is stored beginning at an address in the physical transmit packet data buffer  1311 . The physical transmit pointer queue  1310  then contains an entry that points to (gives the address of) the packet data; in other words, the physical transmit pointer queue  1310  is a queue of address pointers that direct the NIC controller  175  to the data of the packets to be transmitted. For each packet to be received, an entry in the physical receive pointer queue  1312  contains the address of an available space in the physical receive packet data buffer  1313  in which incoming packet data can be stored. 
         [0101]    The pointer queues  1310  and  1312  are commonly implemented as first-in-first-out (FIFO) linked list, with the “last” element linked to the “first,” so that each queue forms a “ring” buffer. Any other known data structure may of course be used instead to perform the same function. 
       Trapping  
       [0102]    For both packet transmits and receives, the vmxnet driver  255  needs to cause the VMM to perform certain tasks. On the other hand, the VMM is preferably transparent to the VM. The question is then how the driver (a VM component) is to call into the VMM, that is, how to generate some form of transfer request that the VMM can sense and act on, but without the VM needing to “know” about the underlying VMM. In this invention, this is preferably done using the well-known technique of “trapping”: The driver  225  does something that causes an exception, which will be detected and handled by the VMM&#39;s interrupt/exception handler  355 ; here, handling the exception means executing the instructions that are needed to perform the transmit and receive operations explained below. 
         [0103]    In the preferred embodiment of the invention, the driver causes an exception by issuing protected instructions, that is, instructions that require a higher privilege level than the driver  225  (a user-level guest component) is at. IN or OUT operations are suitable for this purpose: an IN or OUT will cause a protection violation that will cause the CPU  110  to raise an exception, which will in turn be taken and handled by VMM&#39;s interrupt/exception handler  355 . Using IN and OUT operations to enable the VM to trap (in this case, to “call”) into the VMM is preferred because this is a common operation performed by conventional device drivers. Any other known mechanism may be used, however, to allow the vmxnet driver  225  to cause an exception that the VMM can trap and act on. 
       Transmits  
     Basic Transmit Path 
       [0104]    Assume that an entity within the VM  200  (either an application  260  or the guest OS  225  itself) wishes to send or receive information over the network  700  and that the information is transferred in units such as packets. The simple transmit path used in the preferred embodiment of the invention is the following: 
         [0105]    1. The guest OS  220  calls the vmxnet driver  225  with a network packet in the conventional manner. 
         [0106]    2. The vmxnet driver  225  puts the guest physical address(es) of the packet data into the next free entry(-ies) in the guest transmit pointer queue  1320 . If there is no room in the queue  1320 , then the driver  225  tells the guest OS  220  to stop sending data, using standard signals. 
         [0107]    3. The vmxnet driver  225  does an IN operation, which is trapped using known mechanisms by the vmxnet emulation module  360  in the VMM  300 . 
         [0108]    4. The VMM  300  calls into the vmxnet implementation module  611  in the kernel  600  to transmit the packet located at the address in the guest transmit data buffer  1321  to which the pointer queue  1320  entry points. The VMM returns the result of the kernel  600  call to the driver  225  as the result of the IN operation. Note that the result of the transmit returned to the virtual NIC  272  by the VMM will be consistent with the result returned to the physical NIC  172  by the destination device, except in rare cases such as hardware failure of the physical NIC. Even in such a case, however, the consequence would simply be that the guest OS would interpret any such packets as having been dropped; all standard networking code is written to deal with such packet drops. 
         [0109]    5. Using known mechanisms, the kernel then  600  takes ownership of the pointer queue  1320  entry that contains the packet address so that it cannot be reused until the physical NIC  172  has transmitted the packet. Here, “ownership” means the exclusive right to modify the entry. Note that this is an instance of both the guest (VM) and the host (kernel) accessing the shared memory region  132 . 
         [0110]    6. Recall that the guest transmit pointer queue  1320  entries are the GPPNs of the corresponding entries in the packet data buffer  1321 . For the kernel  600  to locate a packet in the machine address space, as it must, it takes the entry (GPPN) in the pointer queue  1320 , which the kernel gets from the VMM, and then consults its GPPN-to-PPN map to find the actual (machine) physical address (PA) of the packet data in the data buffer  1321 . The kernel  600  then gives the physical address of the packet data to the physical NIC  172 , that is, to its controller  175 . Note that it is not necessary for the kernel to copy the packet information into its own physical transmit packet data buffer  1311 —the controller  175  will have the physical (machine) address of the packet data in the guest transmit packet data buffer  1321  and can read the data directly from that guest buffer  1321 . 
         [0111]    7. When the physical NIC  172  is done with the packet, the kernel  600  gives ownership of the corresponding pointer queue  1320  entry back to the driver  225 . 
         [0112]    8. The kernel  600  tells the VMM  300  to interrupt the VM  200  if the VM has stopped sending packets because there is no free space in the pointer queue  1320  (see step 2 above) or if it has been too long since the VM has been given a chance to process transmitted packets. 
         [0113]    Note that this arrangement according to the invention needs only one trap to the VMM, via the IN operation, to send a packet. 
         [0114]    One advantage of virtualization systems such as those made by VMware, Inc., is that the guest OS  220  may be a conventional, commodity OS such as the different versions of Microsoft Windows and Linux. Of relevance to this invention, in particular, to step 2 above, is that both Windows and Linux provide drivers that support zero-copy transmits and that may be used as the guest driver  225 . 
         [0115]    In Linux, the driver is given a single virtual address for the packet data and a simple masking operation is used to translate from a virtual address to a physical address; this feature may be used for packet address translation. Note, however, that the GPPN-to-PPN translation is also a fast operation in the kernel  600  because only a single physical address is involved (offsets are invariant). Nonetheless, when the guest OS  220  is Linux, it will typically be slightly faster to allow the driver  225  to handle translation through mapping, thereby avoiding altogether the need to copy the untranslated (guest physical) packet address into the buffer  133 . 
         [0116]    In contrast, in systems where the guest OS  220  is a version of Windows, it will usually be more efficient to copy the whole packet: In Windows, packets are fragmented into several pieces. In order to do a zero-copy transmit, the vmxnet driver  225  must ask Windows to give it the guest physical addresses (GPPNs) for each piece of the packet and then pass this list of GPPNs to the kernel  600 . In order to copy the packet, the driver  225  must ask Windows to give it the virtual address of each piece of the packet. It can then take these virtual addresses and copy the packet into a single contiguous packet and then send down to the kernel  600  a single guest physical address for the packet. Throughput benchmarks show that, at least at present, copying is faster than not copying in Windows-based systems. 
         [0117]    Additionally, in Windows-based systems, copying can be made faster than not copying because of certain optimizations provided by this invention. When copying, the driver  225  preferably pre-allocates enough memory to hold the maximum number of outstanding, pending transmit packets. Each pointer queue  1320  entry is then a (guest physical address pointer) into this pre-allocated memory. For an Ethernet network, for example, the MTU (“Maximum Transmission Unit”—the limit on the size of data sent over a network; the MTU is typically a property of the physical network interface) is 1514 bytes, so that one page is allocated for every two packets; packets are thus guaranteed not to cross a page boundary. Each entry in the guest transmit pointer queue  1320  is then preferably initialized to point into the pre-allocated memory space. After the memory is allocated, the VMM  300  is called by the vmxnet driver  225  via an OUT operation to pin all of these pages, using conventional mechanisms. The kernel  600  is then called to pre-compute the GPPN-to-PPN mapping for each packet. The result is that the kernel  600  has to do very little work during a packet transmit since no further guest-to-physical (machine) memory translations will be required. 
         [0118]    Whether zero-copy transmit is provided by the operating system will also affect when the guest transmit and receive packet data buffers are created, and how: If zero-copy transmit is not provided, then the vmxnet driver  225  preferably creates the structures  132 ,  1321  and  1323 , for example, when the vmxnet driver  225  is loaded into the guest OS  220 . Where the guest OS does provide for zero-copy transmits, however, the transmit buffer  1321  will normally be given to the driver  225  by the guest OS  220 ; the driver  225  then gives the buffer back after the transmit completes to the physical NIC  172 . 
         [0119]    Where the guest OS is Linux, the receive buffer  1323  is created when needed by the driver  225  calling a Linux buffer allocator. After the driver receives the needed buffer, it passes it to the guest OS  220 . In systems where the guest OS is a version of Windows, the driver  225  preallocates any needed receive buffer, such as buffer  1323 . 
       Transmit Clustering 
       [0120]    Streaming is generally done using TCP/IP (Transmission Control Protocol/Internet Protocol). With this protocol, an acknowledgement packet is sent to the data sender after a number of packets is sent. The result is that there are receive interrupts that are processed very frequently by the kernel  600  while the VM is streaming data out at relatively high data rates. The invention takes advantage of these receive interrupts to implement “transmit clustering.” 
         [0121]    The idea behind transmit clustering is that the guest (VM  200 ), in particular, the vmxnet driver  225 , puts packet address information in the pointer queue  1320 , but does not immediately call the VMM to transmit the packets. Rather, packets are transmitted in groups—clusters—upon the occurrence of a triggering condition. A preferred triggering condition is such that queued packets are transmitted for the virtual NIC  270  when the next packet is received on the virtual NIC. Because packets are typically received very frequently, transmits can generally be done without any driver-to-VMM traps and without any VMM-to-kernel  600  calls. 
         [0122]    The invention provides alternative mechanisms for implementing transmit packet clustering. One other way, for example, is for the guest (in particular, the vmxnet driver  225 ) to determine that it has too many packets in its transmit pointer queue  1320  that the kernel  600  has not taken ownership of yet. Recall that packets are returned to the sender to acknowledge transmission of packets sent. One way to determine that too many packets are “waiting” or “queued” is therefore for the VMM to detect that it has been too long since receive interrupts have occurred. 
         [0123]    What is “too long” can be determined as a function of the number of queued packets, for example, when the number of transmitted packets that have not been sent exceeds a predetermined maximum. When this threshold is crossed, the VMM calls the kernel  600  to transmit all pending packets in the transmit pointer queue  1320 . In one prototype of the invention, for example, the threshold value for triggering the kernel to transmit pending packets was ten pending packets. 
         [0124]    In the preferred embodiment of the invention, the number of currently queued packets is tracked as follows, which also further clarifies that is meant by “too long”: 
         [0125]    When the kernel  600  turns clustering ON (see below), it sets a maximum number T max  of queued transmit packets in the shared memory data structure. The number T max  is a configuration parameter that can be determined using conventional design criteria, and may be made adjustable by a system administrator using known techniques. In a prototype of the invention, for example, the parameter had a default value of ten queued packets (T max =10). 
         [0126]    The vmxnet driver  225  inspects this value T max  to decide if it should send packets by trapping to the VMM  300  (see above) or if it should simply put the packet address (pointer) in the transmit pointer queue  1320  and continue with other tasks. Each time the vmxnet driver  225  puts a packet in the transmit pointer queue  1320  without trapping to the VMM to send the packet, it increments a count T def  of deferred transmits. The vmxnet driver  225  preferably includes a comparison routine such that, when this count exceeds the transmit cluster max count (T def &gt;T max ), the vmxnet driver  225  calls the VMM. Whenever the kernel  600  transmits packets out of the transmit pointer queue  1320  it resets T def =0. Note that, because T def  is in the shared memory region  132 , the vmxnet driver  225  can increment T def  with no need for any call to the kernel  600 . 
         [0127]    In the TCP/IP case, having too many pending packets should not happen very often. However, it will probably happen more often where the UDP (User Datagram Protocol) is used because there may then not be many packets that the VM receives. 
         [0128]    An alternative way to cluster packets to be transmitted is as a function of time, which may be measured using a known timing routine  615 , preferably in the kernel  600 . If more than a predetermined threshold time (for example, 10 milliseconds) has elapsed since the last packet was received on the virtual NIC  272  and there are pending packets to transmit, then the timer  615  will expire. Note that the kernel  600  will know when the virtual NIC  272  last got a packet because the kernel is the component that put it there. The kernel then interprets expiration of the timer as a signal to transmit the packets. This case should also happen only rarely. 
         [0129]    Transmit clustering is turned ON by the kernel  600  (preferably, as a routine  613  in the implementation module  612 ) when it determines that the rate of packets being transmitted during a predetermined period is high enough and is turned OFF when the rate of packets being transmitted is too low. Transmit clustering cannot be ON all of the time because it will have a negative impact on network performance. For example, if clustering is ON all of the time, and there is no network traffic (no incoming packets) then even a single ping (transmission of a single packet) by the VM will sit in the transmit queue  1320  until the maximum permitted (threshold or “time-out”) time has run out because there will be no received packet to trigger the transmit. In a prototype of the invention, this threshold was ten milliseconds, which is far too long to wait to transmit a single packet. 
         [0130]    Whether clustering should be activated may also be determined using different types of predetermined thresholds, either packet-based or time-based or both: In one prototype of the invention, for example, transmit clustering was activated (turned ON) when at least 4000 packets were transmitted on average per second, measured over a 30 millisecond interval (thus, 120 packets must be transmitted in the interval); clustering was turned OFF when fewer than 3000 packets were transmitted on average per second, measured over a 30 millisecond interval (90 packets transmitted in the interval). These numbers—representing “high enough” and “too low”—are of course examples, and may be adjusted and optimized using conventional techniques of performance analysis. 
       Hardware Transmit Overflow 
       [0131]    When the kernel  600  tries to transmit packets that are stored in the vmxnet driver&#39;s  225  guest transmit pointer queue  1320 , it tries to transfer the packets&#39; addresses to the physical transmit pointer queue  1310  of the physical NIC  172 . There is no guarantee, however, that there will be room in the physical transmit pointer queue  1310 . Whenever the kernel&#39;s  600  transmit code (in the implementation module  612 ) runs, it takes ownership of all of the packets in the vmxnet driver&#39;s  272  transmit pointer queue  1320  and then calls the physical NIC  172  in the conventional manner in order to send the packets. For each packet that the physical NIC  172  cannot handle, the kernel  600  puts the packet into a separate overflow queue  1370  for the device so that the packet can be sent when the physical NIC signals the kernel  600  in any conventional manner that space is available in its physical transmit pointer queue  1310 . These queued packets (in the overflow queue  1370 ) will be sent before any other packets are sent. 
       Receives 
       [0132]    The receive path also utilizes the data structures that are shared between the kernel  600  and the vmxnet driver  225  to minimize VMM-to-kernel transitions. Because the kernel  600  can access the guest receive pointer queue 1322 , received packets can be put into the memory space accessible to the VM  200  without any VMM intervention. 
       Basic Receive Path 
       [0133]    The basic path followed when a packet is received is the following: 
         [0134]    1) The kernel  600  determines the virtual NIC(s)  272  for which the packet is intended. (Only one virtual NIC  172  is shown, for the sake of simplicity, but any number may be included in a system that includes the invention, in particular, in a broadcast or multi-cast system.) 
         [0135]    2) For each virtual NIC, the kernel  600 :
       a) Inspects the guest receive pointer queue  1322  in the memory portion  132  shared with the vmxnet driver  225  to find an empty packet. If there is no empty packet, then the packet is dropped;   b) The kernel  600  copies the data from the received packet into the guest receive packet data buffer  1323  at the location pointed to by the corresponding entry on the guest receive pointer queue  1322 ; and   c) The kernel  600  posts an action to the VMM to tell it to raise a standard receive interrupt to the VM (in particular, to the vmxnet driver  225 ).       
 
         [0139]    3) On each receive interrupt the guest vmxnet driver  225 :
       a) Dismisses the interrupt and, in the case of Windows, blocks future interrupts;   b) Processes all incoming packets and gives them to the guest OS  220 ; the receive entries are then made ready again to receive more packets; and   c) In the case of Windows, enables future interrupts.       
 
       Receive Clustering 
       [0143]    In the simplest case, an interrupt is raised to the VM  200  for each packet received. The guest driver  225  then needs to do one or two conventional IN/OUT operations to trap to the VMM  300  to deal with the interrupt. The raising of the interrupt is a fairly expensive operation, however, as are the IN/OUT operations themselves. Receive clustering according to the invention reduces this overhead by a factor at least approximately equal to the receive cluster size—only one interrupt is raised per cluster of packets. For example, with a cluster size of ten packets, there will only be one interrupt raised for each group of ten packets. 
         [0144]    The idea behind receive clustering according to the invention is that if the VM is receiving enough packets, then they can be delivered in groups—again, clusters—instead of individually. Clustering is turned ON, for example using the routine  613 , when the kernel  600  determines that the VM is receiving a sufficient number of packets per second and it is turned OFF when the receive rate gets too low. 
         [0145]    Receive clustering according to the invention has a straightforward implementation: When a packet is inserted into the vmxnet driver&#39;s  225  receive pointer queue  1322 , a count R def  is incremented by any conventional routine in the emulation module  360  in the VMM  300 . If the count R def  exceeds a maximum predetermined number R max  of unprocessed received packets, then an interrupt is raised to the VM. 
         [0146]    The maximum number R max  of unprocessed received packets is a configuration parameter that can be determined using conventional design criteria, and may be made adjustable by a system administrator using known techniques. Note that if the threshold number R max  is made too big, then throughput will suffer because the VM will not be able to process the packets fast enough and the sender (in most cases, a remote device communicating via the network  700 ) will slow down. Lowering the threshold number, however, reduces the benefit of clustering by increasing interrupts to the guest. In a prototype of the invention, for example, the parameter had a default value of ten unprocessed packets (R max =10). 
         [0147]    Moreover, in the prototype, receive clustering was turned ON if at least 4000 packets were received on average per second, measured over a 30 millisecond interval (120 packets must be received in the interval) and turned OFF if fewer than 3000 packets were received on average per second, measured over a 30 millisecond interval (90 packets must be received in the interval). These numbers were tuned for a 100 Mbit network with relatively slow system hardware but should be adjusted using normal design techniques as new generations of hardware and networking become available. 
         [0148]    As in transmit clustering, a timer may be used to handle the case when a receive packet is pending too long: If a packet remains pending in the guest receive pointer queue  1322  for more than a predetermined threshold time, for example, ten milliseconds, without the guest VM being interrupted, then the timer forces the VMM to raise an interrupt to the VM. If the queue size is not too large, then the timer will not have to raise interrupts to the guest very often. 
       Interrupt Off-Loading 
       [0149]    In general, a VM suffers performance degradation relative to a native machine running benchmarks because of emulation overheads, that is, the CPU cycles required to run the emulations. Consequently, the availability of CPU cycles generally is a limiting factor when running benchmarks. In multi-processor systems, transmit and receive clustering according to the invention provides a way to off-load the networking burden to a different CPU. 
         [0150]    As mentioned above, in the preferred embodiment of the invention, with transmit clustering, most of the transmits are initiated as a result of a receive interrupt. Similarly, the work of handling a received packet, including copying the data into the guest&#39;s receive packet data buffer  1323  is done as a result of a receive interrupt. Using interrupt off-loading according to the invention, the interrupts from the (or one of the) physical NIC(s)  172  are directed to an idle CPU if one is available so that the idle CPU can do all other transmit and receive work. This relieves the CPU currently used to process the instructions relating to the execution of the VM (that is, the CPU on which the VM is running) from the burden of handling networking tasks. This off-loading arrangement will typically give the VM more CPU cycles in which to do other work besides handling these networking tasks. 
         [0151]    Both receive and transmit clustering can cause interrupts to be generated for the VM. Interrupts are generated by the kernel  600  for a guest VM by posting an action to the VMM, whose interrupt/exception handler  355  then takes over processing of the interrupt, including forwarding it, if appropriate, to the VM. If the VM is running, then it must be interrupted so that the VMM may check actions. If the interrupt for the physical NIC  172  happens on the same CPU where the VM is running, then processing may proceed as normal. However, if the interrupts are sent to a different CPU, then an IPI (inter-processor interrupt) must be sent to the CPU on which the VM is running. 
         [0152]    The general procedure for implementing an IPI is well known: In most existing multi-processor systems some device and/or mechanism is implemented specifically for delivering IPIs. In Intel-based systems, for example, each CPU has an APIC (Advanced Programmable Interrupt Controller), which has a unique ID; the APICs are all connected via a bus. In such systems, the following procedure is followed when a thread on one CPU (the “local” CPU) wants to send an IPI to another CPU (the “destination” CPU”); a similar procedure is followed in systems that have architectures not based on Intel processors. 
         [0153]    1) The thread programs the APIC on its local CPU with the APIC ID of the destination and the “vector” to deliver the IPI on. Note that the designation “destination” here does not mean the destination device of the network transfer, but rather the APIC/CPU on which the VMM  300  is executing. In this invention, the thread that programs the local APIC is the kernel  600  and the destination is the APIC/CPU on which the VMM  300  is executing. 
         [0154]    2) The local APIC puts a message on the shared APIC bus. This message typically takes the following form: &lt;type=IPI, destination=APIC id, vector=V&gt;. 
         [0155]    3) The destination APIC receives the message. It sees from the “type” parameter that the message involves an IPI, so it raises the interrupt line to the destination CPU. 
         [0156]    4) The destination CPU senses the interrupt and asks its APIC what the vector is. 
         [0157]    5) The destination APIC replies that the vector is V. 
         [0158]    6) The destination CPU then calls the interrupt handler stored in the IDT: IDT[V].handler() 
         [0159]    In the IPI context, in systems based on the Intel x86 architecture, the “vector” V is usually only a single byte and must have a value that lies in the range [32, 255]. Each vector V represents an index into the IDT. The IDT has 256 entries. Entries 0-31 are reserved for exceptions. Analogous structures are found in other architectures. 
         [0160]    Typically, all interrupts are fielded by the VMM&#39;s interrupt/exception handler  355 , whereupon the VMM calls the kernel  600  to actually deal with the device or the IPI. It would be preferable, however, to eliminate as many VMM-to-kernel  600  crossings as possible because they are expensive in terms of CPU cycles. To make this as efficient as possible, the invention preferably takes advantage of the IPI procedure outlined above and of the fact that when the kernel sends an IPI it can decide which CPU to send the IPI to and which IPI vector to use. According to this aspect of the invention, the kernel uses the IPI vector V as a “check-action” IPI. The procedure is also illustrated in  FIG. 4 . 
         [0161]    In  FIG. 4 , merely by way of example, four CPUs (CPU 0 -CPU 3 ) are shown, each having its own APIC (APIC 0 -APIC 3 , respectively). The APICs are connected to a bus  710 . 
         [0162]    Further in  FIG. 4 , the VMM  300  is shown separated from the kernel  600  and running directly on a hardware processor. This is done just for the sake of clarity in describing the different steps involved in interrupt offloading according to the invention. The actual relationship between the VMM and the kernel is shown in the preferred embodiment of the invention is shown in  FIG. 2 . In this example, the VMM  300  is running on CPU 1 , which forms the destination CPU. 
         [0163]    Typically, each CPU will have a separate IDT. It would also be possible, however, for CPUs to have separate pointers but share a single IDT. The invention may be used in either case. In  FIG. 4 , separate IDTs—IDT 1 , IDT 3 —are shown for CPU 1  and CPU 3 , respectively, other IDTs having been omitted for the sake of simplicity. 
         [0164]    As a preliminary step, the kernel  600  configures the hardware interrupt sub-system to send device interrupts (that is, interrupts from some source entity) to a currently idle CPU; in the illustrated example, CPUs CPU 0 , CPU 2 , and CPU 3  are currently idle (or at least not operating at full capacity), and the kernel selects CPU 3  as being the “most idle, using any known criterion. CPU 3  therefore becomes the “local” CPU. 
         [0165]    In Intel-based systems, the interrupt sub-system comprises at least one IOAPIC (I/O APIC)  1400 , which is on the same bus  710  as the APICs APIC 0 -APIC 3 . The interrupt lines of all devices are routed to pins of the IOAPIC. In  FIG. 4 , two devices are shown connected to the IOAPIC  1400 —the physical NIC  172 , and, by way of another example, a physical SCSI adapter  1410 ; in general, any number N of devices may be connected to the IOAPIC via a respective pin Pin  0 , Pin  1 , Pin  2 , . . . , Pin N. 
         [0166]    For each pin, or at least each pin in use, the kernel programs the IOAPIC using conventional commands to tell it which APIC to send each interrupt to and which vector to use. The IOAPIC therefore includes a table  1420 , which is indexed by pin number and whose entries indicate what to do when a given interrupt arises. When a device raises an interrupt, it is fielded by the IOAPIC. The IOAPIC, which knows which pin the interrupt came in on, looks up the information for the interrupt in the table  1420  using the pin number as an index. It then sends a message on the bus  710  to the appropriate APIC telling it that an interrupt has occurred. 
         [0167]    In  FIG. 4 , for example, an interrupt on pin  0  causes the IOAPIC to deliver the interrupt to CPU 3  (more specifically, to CPU 3 &#39;s APIC 3 ) at vector  85 . CPU 3  then takes the vector (shown as V 85 ) provided by the IOAPIC  1400  and with it as an index enters its IDT—IDT 3 —to get the address addr k , which is the location of the kernel&#39;s interrupt/exception handler  655  routine for handling Pin  0  interrupts. The result of this preliminary step is that an interrupt that arrives from the physical NIC  172 , for example to acknowledge packet transmission, leads to the kernel&#39;s interrupt/exception handler  655  being called at the routine whose entry point is addr k . 
         [0168]    As another preliminary step, the VMM  300  and the kernel  600  agree on which vector Vca (the designation “ca” indicating “check action”) to use for the check-action IPI. The VMM  300  then puts the address addr ca  of its check-action IPI routine  356  in its CPU&#39;s IDT—IDT 1 —indexed by Vca. 
         [0169]    Assume now that as part of executing the kernel&#39;s interrupt/exception handler  655 , the kernel determines that it needs the VMM, which is executing on CPU  1 , to check actions. This could happen, for example, if the VM receives a new packet that needs to be processed. The kernel then sends an IPI via the local APIC (here, APIC 3 ), which puts the IPI on the APIC bus  710 . This IPI will have the form &lt;type=IPI,
       destination=APIC 1 , vector=Vca&gt;. APIC 1  will then receive the IPI and pass to CPU 1  the vector Vca. CPU 1  will then enter IDT 1  at index Vca, which will direct it to the address addr ca  of the VMM&#39;s check action routine  356 .       
 
         [0171]    Two advantages of this aspect of the invention should now be clear: First, device interrupts, in particular from the physical NIC  172 , may be handled by an idle CPU, even though the interrupt may ultimately require VMM action; the CPU on which the VMM is running does not itself need to handle the device interrupts. Second, the interrupt is passed to the VMM solely using existing hardware structures, with no need for a VMM-to-kernel or kernel-to-VMM state transition. Moreover, if this interrupt off-loading aspect of the invention is included in the system, packets can be received and transmitted without a single VMM-to-kernel crossing. 
         [0172]    The interrupt offloading mechanism described above is useful regardless of the need for network operations on behalf of the VM: Regardless of the task that the VM needs done, this aspect of the invention reduces the need to interrupt the VM in order for the VMM to take and handle interrupts, and it also reduces and, in most cases, eliminates the need for a VMM-to-kernel or kernel-to-VMM crossing (world switch) in order to pass an interrupt to the VMM for handling. Note that interrupt offloading according to the invention will work even where the destination CPU is not supporting a VMM/VM, but rather some other software or even hardware entity—regardless of the nature of the entity that ultimately is to receive the interrupt, offloading may be used to relieve the CPU it is running on from the task of handling device interrupts and to enable the kernel to forward interrupts to the entity using hardware mechanisms, with no need for direct calls between the kernel and the entity.