Abstract:
A switch, a system and operational method for packet switching between virtual machines running in a server and a network. The server comprises a switch with swappable, virtual ports. The switch routes packets to and from the various virtual machines resident in the server memory.

Description:
BACKGROUND 
       [0001]    A virtual machine (VM) is an environment, usually a program or operating system, that does not physically exist but is created within another environment. In this context, the VM is called a “guest” while the environment it runs within is called a “host.” Virtual machines can be created to execute an instruction set different than that of the host environment. One host environment can run multiple virtual machines simultaneously. Since virtual machines are separated from the physical resources they use (i.e., memory, CPU cycles), the host environment is often able to dynamically assign those resources among them. 
         [0002]    The practice of running many VMs on the same physical server can improve the resource utilization and bring down the total cost of ownership. However, since each VM should be located to have adequate network bandwidth with other VMs, and with the external entities, increasing the number of VMs in a system can have the result of creating a packet throughput bottleneck and prohibitively high CPU utilization. 
         [0003]    For the reasons stated above, and for other reasons that will become apparent to those skilled in the art upon reading and understanding the present specification, there is a need in the art for efficient packet switching in a multiple core server system. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0004]      FIG. 1  depicts a block diagram of one embodiment of a switch. 
           [0005]      FIG. 2  depicts a block diagram of one embodiment of a virtualized server incorporating the switch of  FIG. 1 . 
           [0006]      FIG. 3  depicts a block diagram of one embodiment of a system incorporating the virtualized server of  FIG. 2 . 
           [0007]      FIG. 4  depicts a flow chart of one embodiment of a method for the operation of a switch in a virtualized server. 
       
    
    
     DETAILED DESCRIPTION 
       [0008]    In the following detailed description of the present embodiments, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments of the disclosure which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the subject matter of the disclosure. It is to be understood that other embodiments may be utilized and that process or mechanical changes may be made without departing from the scope of the present disclosure. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and equivalents thereof. 
         [0009]      FIG. 1  illustrates a block diagram of one embodiment of a switch  150  that can be used in a virtualized server environment. The switch  150  is comprised of a network side  100  and a server side  101  that couple the switch  150  to the network through a plurality of ports on both sides  100 ,  101  of the switch. The network side couples the switch  150  to a network through a subset of the ports  130 . For example, the network side  100  can be comprised of an Ethernet switch and a plurality of ports  130  coupled to a packet switch  105 . 
         [0010]    The packet switch  105  is further comprised of a link scheduler  107 . The link scheduler  107  enforces the overall network bandwidth allocations for each virtual machine. The link scheduler  107  also ensures adequate latency and other quality of service requirements. 
         [0011]    The switch  150  is further comprised of a switch control plane  109  that is coupled to and controls the packet switch  105 . The switch control plane  109  participates in network management protocols such as spanning tree, address learning, as well as other protocols. 
         [0012]    The server side  101  of the switch  150  is made up of a scalable mechanism that enables a large number of virtual machines to bypass the hypervisor (VMM) and access the switch  150  directly as if it were a traditional network interface card. The server side  101  is comprised of virtual ports (VPorts)  103  that couple the switch  150  to a server. 
         [0013]    Each VPort  103  is coupled to a single virtual machine and presents a virtual network interface card interface. These VPorts are carried over the system bus (PCI-E/Front Side Bus). They are referred to as virtual ports since each port is in essence a collection of control data items and the switch  150  minimizes the memory requirements by allowing each port to be swapped out to the system memory. 
         [0014]    In one embodiment, only the necessary control data is swapped based on the direction of the data packet and the quality of service class to which it belongs. 
         [0015]    The PCI-E/FSB attachment  104 , coupled to the plurality of VPorts  103 , forms the attachment to the server. The PCI-E/FSB attachment  104  implements the necessary electrical components (such as SERDES) and the protocol processing (such as DLP and TLP layer processing of PCI-E). This block  104  may also participate in coherence protocols such as cHT in the case of an FSB attachment. 
         [0016]    An I/O bus scheduler  121  is coupled to the PCI-E/FSB attachment  104 . The I/O bus scheduler  121  enforces the overall I/O bandwidth allocations for each of the virtual machines. This can be accomplished by prioritizing the I/O bus transactions and partitioning the I/O bus bandwidth across the virtual machines according the prioritization. 
         [0017]    A VPort swapper block  119  is coupled to the VPorts  103  through the I/O bus scheduler  121 . The swapper block  119  manages the VPort swaps to and from the system memory. This block  119  ensures adequate latency and bandwidth for network traffic by intelligently managing the on-board memory to hold the most useful VPorts. The VPort swapper block  119  exploits the packet bursts to maximize the locality (i.e., the probability a given VPort is found in the switch memory). Most applications are known to send bursts of packets (back-to-back sequences) interspersed with long silences. This knowledge can be used to improve accuracy of the guess as to which of the VPorts are not likely to be accessed in the near future. 
         [0018]    The VPort table  113 , coupled to the VPort swapper  119 , provides data on the most active or most useful VPorts to the VPort swapper  119 . These data can include the pointers to the corresponding receive and transmit queues, quality of service parameters, and accounting counters. 
         [0019]    Packet buffers  111  in the switch  150  are coupled to the packet switch  105  to provide temporary storage for holding data packets prior to being transferred to the server memory. The packet buffers  111  also temporarily store data packets prior to being transmitted to the network. 
         [0020]    A direct memory access (DMA) engine  117  is coupled between the packet buffers  111  and the I/O bus scheduler  121 . The DMA engine  117  is a means for controlling transfer of data packets to and from the main server memory in response to the instructions from the quality of service manager  115 . The DMA engine  117  collaborates with the I/O bus scheduler  121 . 
         [0021]    In operation, when a new virtual machine is created in the server, a network controller instructs the switch  150  to create a new virtual switch port (i.e., virtual network interface) through the control interface of the switch. The switch  150  first claims a free entry from the VPort table  113  and then allocates a per virtual machine structure in the system memory. The network controller then records a pointer to this structure in a global table of VPorts, and returns its index as the virtual function number to the controller. The network controller then uses this virtual function number to boot up the virtual machine. 
         [0022]    When a packet destined for a local virtual machine is received from the network, the network-facing side  100  recognizes the MAC address and forwards it to the server-facing side  101 . The server side  101  uses the destination MAC address and looks up the VPort cache for the corresponding per virtual machine context, if successful, the quality of service computation is done and the packet is queued up for the DMA engine  117 . If not successful, the packet descriptor is queued up in a pending queue and a cache replacement algorithm is run to select a victim VPort and swap it with the missing context. The DMA read completion for the missing context triggers the re-processing of the packet. 
         [0023]    When a virtual machine has a packet to send, it queues up the packet descriptor in its send queue and writes to a register in its virtual network port. When the switch  150  gets the write command, it uses a table pointer to look up the virtual network card cache. If successful, the send queue pointer is extracted, the packet header is read in, and after the quality of service computation, a future time for DMAing the packet payload is determined and the DMA request is queued up. When the DMA for the payload is complete, the packet descriptor is queued up for transmission by the network side  100  on one of the output ports  130 . 
         [0024]      FIG. 2  illustrates a block diagram of one embodiment of a typical server  200  that incorporates a switch  150  in accordance with the embodiment of  FIG. 1 . This figure shows two possible locations in the server  200  to which the switch  150  of  FIG. 1  can be coupled. Both locations are coupled to the front side bus (FSB) except the first location is coupled directly to the FSB while the second location goes through an I/O bus such as (PCI-E), and then through a chip set  201  that controls one or more multi-core processors and memory banks  222 . Both these couplings allow direct control of the switch  150  by the system software 
         [0025]    The server  200  is further comprised of a plurality of multi-core processors  211 ,  212  each with their own memory  220 ,  221 . Each processor  211 ,  212  can be comprised of a plurality of processor cores  218 . Each memory block  220 - 222  is comprised of an area  230 - 232  for back-up of the virtual port table  113  (see  FIG. 1 ) on the switch  150 . In one embodiment, the memory blocks  220 - 222  are coupled to their respective processors  211 - 213  through their own front side bus. 
         [0026]      FIG. 3  illustrates a block diagram of one embodiment of a network that incorporates a server  200  in accordance with the embodiment of  FIG. 2 . The illustrated network comprises a sub-network  310  that uses standard prior art network switches. This network is coupled to a plurality of I/O subsystems  301 - 303 . The I/O subsystems  301 - 303  can be servers such as the servers illustrated in  FIG. 2 . 
         [0027]      FIG. 3  shows three such servers  301 - 303  coupled to the central sub-network  310 . Alternate embodiments can use any number of servers. Each of the servers  301 - 303  is coupled to the sub-network  310  through a switch  150  as illustrated previously with reference to  FIGS. 1 and 2 . Each of the servers  301 - 303  can include multiple virtual machines  320  that are resident in the memory of the servers  301 - 303 . 
         [0028]      FIG. 4  illustrates a flowchart of one embodiment of a method for operation of an integrated server switch in a virtualized system. The method begins at the arrival of a new data packet  401 . 
         [0029]    It is determined if the data packet is outbound from the server switch  402 . If the packet is outbound from one of the local virtual machines to an external destination, the VPort identification based on the requesting virtual machine is determined  407 . Such an occurrence might result if a virtual machine has requested the server switch to send a packet. The requesting virtual machine is used to determine the VPort since there is a one-to-one correspondence between them. 
         [0030]    If the packet is not outbound from one of the local virtual machines, the VPort identification based on the destination media access control (MAC) address is determined  405 . Such an occurrence might result if the packet was just received from the network and should be sent to one of the virtual machines. Incoming packets do not have the virtual machine identification. They identify the destination virtual machine by its MAC address. Again, there is a one-to-one correspondence between the destination MAC address found in the packet and the virtual machine identification that can be translated to the VPort identification. The MAC address is a quasi-unique identifier for identification assigned to most network adapters or network interface cards by a manufacturer. 
         [0031]    It is then determined if a VPort is available in the server switch  409 . Since the server switch only holds a small number of VPorts (i.e., the control data that represents the VPort) in its local memory, a check is performed to determine if the necessary data is available locally (e.g., checking server memory). 
         [0032]    If the VPort is not available in the server switch, a swapping operation is performed. This operation is comprised of determining if there is memory available in the local memory (i.e., a free location in the VPort table)  421 . If no table slot is available, a slot is made available by selecting a VPort that is currently in the table  423 . A high priority DMA write with quality of service manager and DMA is performed  425  to write the selected VPort back into the back-up copies of the VPort table in the system memory. 
         [0033]    Once the write is complete, or a free slot in the VPort table is available, the necessary VPort is read in  427 . After a free slot is either determined to be available or made available, the necessary counters, allocations, quality of service parameters, and other data is extracted from the VPort that is available  411 . The quality of service computations are then performed  413  to determine the priority of the data packet. 
         [0034]    If the data packet was outbound from the local virtual machines  415 , the packet is sent for transmission to the link-scheduler for queuing  419 . If the data packet was not outbound, the data packet was an incoming packet that is handed over to the I/O scheduler for delivery to the proper virtual machine  417 . 
         [0035]    In an alternate embodiment, a hypervisor or other system software prioritizes packet direct memory accesses. For example, packets belonging to a guest virtual machine that is currently running should be prioritized. 
         [0036]    In summary, by integrating a switch with the server platform, a higher packet throughput can be achieved between communicating virtual machines regardless of their location. Thus, many virtual machines can be run on the same physical server to improve resource utilization and bring down total cost of ownership.