Abstract:
A method, system and computer program product are provided for implementing concurrent adapter driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter in a computer system. An adapter driver at start up time performs configuration of the adapter and each of a set of virtual functions (VFs). The adapter driver writes critical adapter and VF configuration data to a scratchpad buffer. When device driver maintenance is needed, such as to load updated adapter driver firmware, all VF drivers are held off temporarily, current adapter driver is detached, and then the adapter driver is reloaded with the updated driver firmware. Then the adapter driver is restarted with the updated adapter driver firmware, and performs a reinitialization process. The adapter driver performs adapter and VF configuration restoring existing configuration using values read from the scratchpad buffer.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates generally to the data processing field, and more particularly, relates to a method, system and computer program product for implementing concurrent adapter driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter in a virtualized system. 
       DESCRIPTION OF THE RELATED ART 
       [0002]    Single root input/output (IO) virtualization (SRIOV) is a PCI standard, providing an adapter technology building block for  110  virtualization within the PCI-Express (PCIe) industry. SRIOV capability is a feature of many new PCIe adapters for Fibre Channel, Ethernet, Infiniband, and Converged Network Adapters (CNA). 
         [0003]    The SRIOV adapter has an I/O adapter virtualization architecture that allows a single I/O adapter to be concurrently shared across many different logical partitions. The sharing is done at a physical level, so that each logical partition has access to a slice of the physical adapter. The sharing is accomplished via partitioning the adapter into many different PCI functions, and then distributing access to those functions. The adapter is presented as one or more physical functions (PFs) that control functions, for example used for both configuration and I/O, and a set of virtual functions (VFs), used for I/O and limited configuration, each VF represents a slice of the adapter capacity that can be assigned to a logical partition independently of other VFs. Each logical partition has a device driver for each of the VFs assigned to the logical partition. 
         [0004]    There is a requirement to periodically update the adapter driver, for example, either to add new function or to fix logic bugs. A VF device driver is limited in scope to a single VF, and can be more easily updated. The PF device or adapter driver is associated with the entire adapter, and updates are more difficult as a result. A significant part of the problem is the fact that the adapter driver configures the adapter itself, and will potentially need to reinitialize the adapter. 
         [0005]    One approach is to schedule a maintenance window and take the entire adapter temporarily off-line to perform the updates. This approach is highly disruptive, and can be difficult to achieve as there may be dozens of logical partitions associated with the adapter. 
         [0006]    Another approach is to have a second backup I/O channel to be used while the maintenance occurs. During the adapter driver update, I/O is switched to the backup channel. This is expensive, because it requires duplicates of the I/O resources. 
         [0007]    A need exists for an effective mechanism to enable concurrent device driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter in a virtualized system. It is desirable that such mechanism enables access to the adapter to be maintained during the update. 
       SUMMARY OF THE INVENTION 
       [0008]    Principal aspects of the present invention are to provide a method, system and computer program product for implementing concurrent adapter driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter. Other important aspects of the present invention are to provide such method, system and computer program product substantially without negative effects and that overcome many of the disadvantages of prior art arrangements. 
         [0009]    In brief, a method, system and computer program product are provided for implementing concurrent adapter driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter in a computer system. An adapter driver at start up time performs configuration of the adapter and each of a set of virtual functions (VFs). The adapter driver writes critical adapter and VF configuration data to a scratchpad buffer. When device driver maintenance is needed, such as to load updated adapter driver firmware or to fix logic bugs, all VF drivers are held off temporarily, the current adapter driver is detached, and then the adapter driver is reloaded with the updated driver firmware. Then the adapter driver is restarted with the updated adapter driver firmware, and performs a reinitialization process. The adapter driver performs adapter and VF configuration restoring existing configuration using values read from the scratchpad buffer. 
         [0010]    In accordance with features of the invention, it is not required to provide a scheduled maintenance window with the adapter off-line to perform the updates. The VFs remain configured throughout the concurrent device driver maintenance and recovery process. 
         [0011]    In accordance with features of the invention, the VF drivers need no special support with available error recovery processes used. All I/O remains intact, and there is only a brief pause during the reinitialization process. No backup I/O or failover needs to take place. 
         [0012]    In accordance with features of the invention, a system hypervisor manages physical functions (PFs) associated with the SRIOV adapter. The existing configuration is restored through the use of the scratchpad buffer held in the hypervisor that is read by the adapter driver during its reinitialization process. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]    The present invention together with the above and other objects and advantages may best be understood from the following detailed description of the preferred embodiments of the invention illustrated in the drawings, wherein: 
           [0014]      FIGS. 1 , and  2  illustrates a respective example computer system and example system for implementing concurrent adapter driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter in accordance with the preferred embodiment; 
           [0015]      FIGS. 3 , and  4  together provide a flow chart illustrating exemplary operations for implementing concurrent device driver maintenance and recovery for the SRIOV adapter in accordance with the preferred embodiment; and 
           [0016]      FIG. 5  is a block diagram illustrating a computer program product in accordance with the preferred embodiment. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0017]    In the following detailed description of embodiments of the invention, reference is made to the accompanying drawings, which illustrate example embodiments by which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the invention. 
         [0018]    The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. 
         [0019]    In accordance with features of the invention, a method, system and computer program product are provided for implementing concurrent adapter driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter in a computer system. 
         [0020]    Having reference now to the drawings, in  FIG. 1 , there is shown an example computer system generally designated by the reference character  100  for implementing concurrent adapter driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter  102  in accordance with the preferred embodiment. Computer system  100  includes one or more processors  104 , or central processor units (CPUs)  104  (one shown) coupled by an I/O hub or processor host bridge  106  to the Single Root Input/Output Virtualization (SRIOV) adapter or hardware I/O adapter  102 . 
         [0021]    Computer system  100  includes a memory  108  and one or more logical partitions (LPARs)  110  (one shown) coupled by a system bus  111  to the processor  104  and the processor host bridge  106 . Each operating system (OS)  112  resides in its own LPAR  110 , with each LPAR allocated a part of a physical processor  104 , an entire physical processor, or multiple physical processors from the computer  100 . A VF device driver  114  is provided with the logical partition (LPAR)  110 . A portion of the memory  108  is allocated to each LPAR  110 . Computer system  100  includes a hypervisor  116  including a configuration mechanism  118 . The hypervisor  116  is a part of the system firmware and manages the allocation of resources to each operating system  112  and LPAR  110 . 
         [0022]    As shown, a hardware management console (HMC)  120  used, for example, to manage system functions including logical partition configuration and hardware virtualization, is coupled to the hypervisor  116  via a service processor  122 . Computer system  100  includes a physical function (PF) manager or PF adjunct  124  provided with the hypervisor  116 . The PF adjunct  124  includes an adapter driver  128  to manage physical functions of the hardware I/O adapter  102 . The hypervisor  116  uses the PF adjunct  124 , for example, to configure physical functions (PFs) and virtual functions (VFs) of the hardware I/O adapter  102  based on configuration information provided by a system administrator via the hardware management console  120 . 
         [0023]    As shown, the hardware I/O adapter  102  includes, for example, a first physical function  130 , a second physical function  132 , a first port  134 , and a second port  136 . The hypervisor  116  using the PF adjunct  124  configures virtual functions based on the physical functions  130 ,  132  and associates virtual functions with one or more of the ports  134 ,  136  of the hardware I/O adapter  102 . 
         [0024]    For example, a first virtual function,  140 , instance 1, and the Mth instance of the first virtual function  142 , where M is greater than 1, are associated with the second port  136 . As shown, a second virtual function  144  , such as the first instance of the second virtual function  144  and the Pth instance of the second virtual function  146 , where P is greater than 1, are associated with the first port  134 . As shown, multiple instances of an Nth virtual function, where N is greater than 2, such as the first instance of the Nth virtual function  148  is associated with the first port  134  and the Qth instance of the Nth virtual function  150 , where Q is greater than 1, is associated with the second port  136 . 
         [0025]    Each instance of the first virtual function  140 ,  142 , the second virtual function  144 ,  146 , and Nth virtual function  148 ,  150  are hosted by a physical function, such as one of the first physical function  132 , the second physical function  132 , and another physical function (not shown). 
         [0026]    Each instance of the first virtual function  140 ,  142 , the second virtual function  144 ,  146 , and Nth virtual function  148 ,  150  includes a respective virtual function identifier (ID), shown as ID  152 , ID  154 , ID  156 , ID  158 , ID  160 , and ID  162 . Each virtual function identifier uniquely identifies a particular virtual function that is hosted by the hardware I/O adapter  102 . For example, when a message (not shown) is routed to a particular virtual function, the message includes the identifier associated with the particular virtual function. 
         [0027]    Computer system  100  is shown in simplified form sufficient for understanding the present invention. The illustrated computer system  100  is not intended to imply architectural or functional limitations. The present invention can be used with various hardware implementations and systems and various other internal hardware devices. 
         [0028]    Referring to  FIG. 2 , there is shown another example system generally designated by the reference character  200  for implementing concurrent device driver maintenance and recovery for a hardware I/O adapter or Single Root Input/Output Virtualization (SRIOV) adapter or hardware I/O adapter  202  in accordance with the preferred embodiment. 
         [0029]    System  200  includes a hypervisor  204  or other virtualization intermediary, used to enable multiple logical partitions to access virtual functions provided by hardware that includes the hardware I/O adapter  202 . For example, as shown in  FIG. 2 , the hypervisor  204  is used to enable a first logical partition  206 , a second logical partition  208 , and an Nth logical partition  210 , to access a plurality of virtual functions  212 ,  214 ,  216 ,  218  that are provided by the hardware I/O adapter  202 . For example, the hypervisor  204  used a first physical function  220  of the hardware I/O adapter  202  to provide a first instance of a first virtual function  212 , a second instance of a first virtual function  214 , and an Nth instance of a first virtual function  216  to the logical partitions  204 ,  206 ,  210 . As shown the hypervisor  204  uses a second physical function  222  of the hardware I/O adapter  202  to provide a second virtual function  218  to the logical partitions  206 ,  208 ,  210 . 
         [0030]    The physical functions  220 ,  222  advantageously include PCI functions, supporting single root I/O virtualization capabilities. Each of the virtual functions  212 ,  214 ,  216 ,  218  is associated with one of the physical functions  220 ,  222  and adapted to share one or more physical resources of the hardware I/O adapter  202 . 
         [0031]    Software functions or modules, such as a physical function (PF) adjunct  224  including an adapter driver  225 , is provided with the hypervisor  204  for managing the physical functions  220 ,  222  and the virtual functions  212 ,  214 ,  216 ,  218 . For example, a user may specify a particular configuration and the hypervisor  204  uses the PF adjunct  224  to configure the virtual functions  212 ,  214 ,  216 ,  218  from the physical functions  220 ,  222 . 
         [0032]    For example, in operation, the hypervisor  204  with the PF adjunct  224  enables the first virtual function instances  212 ,  214 ,  216  from the first physical function  220 . The hypervisor  204  with the PF adjunct  224  enables the second virtual function  218  from the second physical function  222 . The virtual functions  212 ,  214 ,  216 ,  218  are enabled, for example, based on a user provided configuration. Each of the logical partitions  206 ,  208 ,  210  may execute an operating system (not shown) and client applications (not shown). 
         [0033]    As shown, the client applications that execute at the logical partitions  206 ,  208 ,  210  perform virtual input/output operations and include a respective device driver to directly manage an associated virtual function. For example, a first client application executing at the first logical partition  206  may include a first client VF device driver  226 , and a second client application executing at the first logical partition  206  may include a second client VF device driver  228 . 
         [0034]    As shown, the first client VF device driver  226  accesses the first instance of the first virtual function  212 . The second client virtual VF device driver  228  accesses the second virtual function  218 . A third client VF device driver  230  executing at the second logical partition  208  accesses the second instance of the first virtual function  214 . An Nth client VF device driver  232  executing at the Nth logical partition  210  accesses the Nth instance of the first virtual function  216 . An access mechanism  234  and a configuration mechanism  236  are provided with the hypervisor  204  to associate a logical partition with an accessed virtual function. The hypervisor  304  uses the access mechanism  234  to enable logical partitions, such as LPAR  206  to access configuration space associated with one or more of the virtual functions  212 ,  214 ,  216 ,  218 . 
         [0035]    System  200  is shown in simplified form sufficient for understanding the present invention. The illustrated system  200  is not intended to imply architectural or functional limitations. The present invention can be used with various hardware implementations and systems and various other internal hardware devices. 
         [0036]    In accordance with features of the invention, critical configuration data in a scratchpad buffer kept in the hypervisor is read during the adapter driver restart. This critical hardware configuration data is defined as any configuration data in addition to adapter capability and protocol settings provided by the customer, which is generated when configuring the adapter and its VFs that are necessary to reconfigure the adapter and those VFs identically after resetting the adapter. This may include, but is not limited to the VF MMIO memory map, number of VFs configured per physical function, map of logical VF indexes to virtual functions on the adapter, and DMA window assignments for configured VFs. Note that these resources include both adapter resources and also platform resources. 
         [0037]    In accordance with features of the invention, this scratchpad buffer or scratchpad area is necessarily preserved during an adapter driver restart. However, it is necessary for the scratchpad buffer to be cleared at appropriate times. The scratchpad initial state is zeroed, indicating no configuration data is present for a clean or fresh adapter driver start. This is the scratchpad state at system power on, for example. However, actions where the physical adapter changes, such as a concurrent replacement of an adapter, result in the scratchpad area being cleared. This allows the adapter driver to have a clean start with the new hardware I/O adapter or adapter card. For example, this is necessary to handle cases where the physical adapter characteristics may have changed, such as from replacing an Ethernet adapter with a fiber channel adapter. Thus the data is preserved through adapter driver restarts, allowing maintenance of the adapter driver, while being cleared for a new adapter allowing a clean install to start fresh. This scratchpad area is completely managed within the hypervisor, requiring no external management, such as through the HMC or other channels. 
         [0038]    Referring to  FIGS. 3 and 4 , there are shown exemplary operations of the processing and logic provided by the hypervisor  130  for implementing concurrent device or adapter driver maintenance and recovery in accordance with the preferred embodiment. 
         [0039]    In  FIG. 3 , as indicated in a block  300 , the adapter driver starts and finds an empty scratchpad buffer. The adapter driver performs configuration of the adapter and configuration of the VFs as indicated in a block  302 . The adapter driver writes critical configuration data to the scratchpad buffer as indicated in a block  304 . Then the initial VF configuration is sent to the adapter driver, the adapter driver instantiates configuration and updates the configuration data in the scratchpad buffer as needed as indicated in a block  306 . The adapter driver updates the critical configuration data in the scratchpad buffer as configuration changes are made as indicated in a block  308 . 
         [0040]    In  FIG. 4 , as indicated in a block  400 , the hardware management console (HMC) notifies the hypervisor of new adapter driver firmware. The hypervisor places all VF Partitionable Endpoints (PEs) in a freeze state utilizing an enhanced error handling (EEH) functionality, as indicated in a block  402 . 
         [0041]    A Partitionable Endpoint (PE) is a separately assignable I/O unit. That is, any part of an I/O subsystem that can be assigned a logical partition independent of another PE. Each PE has independent domains (addressing, error, state, and the like) to provide PE level error isolation, detection, and recovery. 
         [0042]    As indicated in a block  404 , the VF device drivers or each VF device driver detects an error condition responsive to the freeze state of the PEs and commences a VF enhanced error handling (EEH) recovery. As indicated in a block  406 , the hypervisor shuts down the adapter driver, loads a new adapter driver; then the hypervisor restarts the adapter driver. The adapter driver starts, and finds existing critical hardware configuration data for the adapter and VFs in the scratchpad buffer as indicated in a block  408 . As indicated in a block  410 , the adapter driver uses the existing configuration data from scratchpad buffer to reconfigure adapter and VFs identically as in step  302  in  FIG. 3 . The adapter driver gives the hypervisor permission for the VF PEs to be unfrozen (as also performed in an EEH of a Shared Adapter process) as indicated in a block  412 . The VF drivers commence recovery independently (as also performed in an EEH of a Shared Adapter process) as indicated in a block  414 . 
         [0043]    Referring now to  FIG. 5 , an article of manufacture or a computer program product  500  of the invention is illustrated. The computer program product  500  is tangibly embodied on a non-transitory computer readable storage medium that includes a recording medium  502 , such as, a floppy disk, a high capacity read only memory in the form of an optically read compact disk or CD-ROM, a tape, or another similar computer program product. Recording medium  502  stores program means  504 ,  506 ,  508 , and  510  on the medium  502  for carrying out the methods for implementing concurrent device driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter of a preferred embodiment in the system  100  of  FIG. 1 . 
         [0044]    A sequence of program instructions or a logical assembly of one or more interrelated modules defined by the recorded program means  505 ,  506 ,  508 , and  510 , direct the computer system  500  for implementing concurrent device driver maintenance and recovery for a Single Root Input/Output Virtualization (SRIOV) adapter of a preferred embodiment. 
         [0045]    While the present invention has been described with reference to the details of the embodiments of the invention shown in the drawing, these details are not intended to limit the scope of the invention as claimed in the appended claims.