Abstract:
Exemplary embodiments provide subsidiary volume management. In one embodiment, a storage system comprises: a memory being operable to store information of a plurality of logical unit groups, each of the plurality of logical unit groups including information of an administrative logical unit (LU) and information of one or more subsidiary LUs to be accessed from a virtual machine on a computer; and a controller being operable to create or select an administrative LU and to inform the computer of the created or selected administrative LU according to evaluation of a subsidiary LU which relates to another administrative LU, when the controller receives a command from the computer to said another administrative LU.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    The present invention relates generally to computer systems, storage systems, server virtualization, and storage volume virtualization. More particularly, it relates to subsidiary volume management. 
         [0002]    According to the latest SCSI (Small computer system interface) specification, a LU (Logical Unit) Group is defined. The LU Group has one administrative LU and multiple subsidiary LUs. A conventional LU contains the LU Group which has many subsidiary LUs. The administrative LU of the LU Group is management LU to create, delete, migrate, or control subsidiary LUs in the LU Group. 
         [0003]    The current storage system does not have a functionality whereby one administrative LU is preferred over other administrative LUs, when the host hypervisor binds a subsidiary LU to prepare a VM (Virtual Machine) disk. Also, the current storage system does not have a functionality to migrate a rebinded subsidiary LU to some other administrative LU to rebalance performance of physical resources in the storage system, since the host hypervisor does not know how much usage of the storage internal resource for the subsidiary LU. 
       BRIEF SUMMARY OF THE INVENTION 
       [0004]    Exemplary embodiments of the invention provide subsidiary volume management. When a storage system receives an unbind command from a hypervisor, the storage system calculates the storage performance for each subsidiary LU, and then rebinds it to another administrative LU and migrates data from the source pool to another pool, if needed to rebalance the storage workflow. When the storage system receives a bind command from the host hypervisor, the storage system notifies that that specific subsidiary LU is rebinded to another administrative LU based on performance metric of each of the LUs, and then retries to send the bind command to the rebinded administrative LU. The storage system decides which administrative LU has sufficient resource. The server administrator does not know which administrative LU has sufficient storage resource without understanding how much usage there is of the storage internal resource. 
         [0005]    In accordance with an aspect of the present invention, a storage system comprises: a memory being operable to store information of a plurality of logical unit groups, each of the plurality of logical unit groups including information of an administrative logical unit (LU) and information of one or more subsidiary LUs to be accessed from a virtual machine on a computer; and a controller being operable to create or select an administrative LU and to inform the computer of the created or selected administrative LU according to evaluation of a subsidiary LU which relates to another administrative LU, when the controller receives a command from the computer to said another administrative LU. 
         [0006]    In some embodiments, evaluation of the subsidiary LU comprises at least one of evaluating throughput of the subsidiary LU against a throughput threshold or evaluating IOPS (Input/Output Operations per Second) of the subsidiary LU against an IOPS threshold or evaluating capacity usage of the subsidiary LU against a remaining storage capacity available for allocation. Selecting an administrative LU comprises one of selecting the administrative LU from a same capacity pool to which said another administrative LU belongs, selecting the administrative LU from a different capacity pool which is different from the capacity pool to which said another administrative LU belongs, and selecting the administrative LU from a different storage system which is different from a storage system in which said another administrative LU is disposed. 
         [0007]    In specific embodiments, the controller receives a bind command from the computer to bind the subsidiary LU to said another administrative LU, conducts evaluation of at least one of storage performance or capacity usage of the subsidiary LU for binding to said another administrative LU, and when the evaluation indicates at least one of insufficient capacity or inadequate performance in binding the subsidiary LU to said another administrative LU, creates or selects the administrative LU for binding the subsidiary LU to the created or selected administrative LU. The evaluation indicates at least one of insufficient capacity or inadequate performance in binding the subsidiary LU to said another administrative LU after a performance threshold for evaluating the storage performance has been updated. 
         [0008]    Another aspect of the invention is directed to a controller in a storage system which includes a memory to store information of a plurality of logical unit groups, each of the plurality of logical unit groups including information of an administrative logical unit (LU) and information of one or more subsidiary LUs to be accessed from a virtual machine on a computer. The controller is operable to create or select an administrative LU and to inform the computer of the created or selected administrative LU according to evaluation of a subsidiary LU which relates to another administrative LU, when the controller receives a command from the computer to said another administrative LU. 
         [0009]    Another aspect of this invention is directed to a non-transitory computer-readable storage medium storing a plurality of instructions for controlling a data processor to perform subsidiary logical unit (LU) management in a storage system which includes a memory to store information of a plurality of logical unit groups, each of the plurality of logical unit groups including information of an administrative LU and information of one or more subsidiary LUs to be accessed from a virtual machine on a computer. The plurality of instructions comprise instructions that cause the data processor to create or select an administrative LU and to inform the computer of the created or selected administrative LU according to evaluation of a subsidiary LU which relates to another administrative LU, when the controller receives a command from the computer to said another administrative LU. 
         [0010]    In some embodiments, the plurality of instructions further comprise, for the evaluation of the subsidiary LU, instructions that cause the data processor to perform at least one of evaluating throughput of the subsidiary LU against a throughput threshold or evaluating IOPS (Input/Output Operations per Second) of the subsidiary LU against an IOPS threshold or evaluating capacity usage of the subsidiary LU against a remaining storage capacity available for allocation. The instructions to select an administrative LU comprise instructions that cause the data processor to select one of the administrative LU from a same capacity pool to which said another administrative LU belongs, the administrative LU from a different capacity pool which is different from the capacity pool to which said another administrative LU belongs, and the administrative LU from a different storage system which is different from a storage system in which said another administrative LU is disposed. 
         [0011]    In specific embodiments, the instructions that cause the data processor to create or select an administrative LU comprise: instructions that cause the data processor to receive a bind command from the computer to bind the subsidiary LU to said another administrative LU; instructions that cause the data processor to conduct evaluation of at least one of storage performance or capacity usage of the subsidiary LU for binding to said another administrative LU; and instructions that cause the data processor, when the evaluation indicates at least one of insufficient capacity or inadequate performance in binding the subsidiary LU to said another administrative LU, to create or select the administrative LU for binding the subsidiary LU to the created or selected administrative LU. 
         [0012]    These and other features and advantages of the present invention will become apparent to those of ordinary skill in the art in view of the following detailed description of the specific embodiments. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]      FIG. 1  illustrates an example of a hardware configuration of a computer system in which the method and apparatus of the invention may be applied. 
           [0014]      FIG. 2  shows another example of the computer system having a plurality of physical servers and a plurality of storage systems. 
           [0015]      FIG. 3 a    shows an example of a storage system. 
           [0016]      FIG. 3 b    shows an example of a storage memory. 
           [0017]      FIG. 4 a    shows an example of a host server (physical server). 
           [0018]      FIG. 4 b    shows an example of the host memory. 
           [0019]      FIG. 5  shows an example of a Logical Unit table. 
           [0020]      FIG. 6  shows an example of a pool resource table. 
           [0021]      FIG. 7  shows an example of a performance monitoring table. 
           [0022]      FIG. 8  shows an example of a quality of service table. 
           [0023]      FIG. 9  shows an example of a flow diagram illustrating a process for creating administrative LU and binding subsidiary LU. 
           [0024]      FIG. 10  shows an example of a flow diagram illustrating a process for updating QoS configuration from the server administrator and operation of migrating (rebinding) a subsidiary LU. 
           [0025]      FIG. 11  shows an example of a flow diagram illustrating a process for subsidiary LU migration to another pool or another storage system. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0026]    In the following detailed description of the invention, reference is made to the accompanying drawings which form a part of the disclosure, and in which are shown by way of illustration, and not of limitation, exemplary embodiments by which the invention may be practiced. In the drawings, like numerals describe substantially similar components throughout the several views. Further, it should be noted that while the detailed description provides various exemplary embodiments, as described below and as illustrated in the drawings, the present invention is not limited to the embodiments described and illustrated herein, but can extend to other embodiments, as would be known or as would become known to those skilled in the art. Reference in the specification to “one embodiment,” “this embodiment,” or “these embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention, and the appearances of these phrases in various places in the specification are not necessarily all referring to the same embodiment. Additionally, in the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that these specific details may not all be needed to practice the present invention. In other circumstances, well-known structures, materials, circuits, processes and interfaces have not been described in detail, and/or may be illustrated in block diagram form, so as to not unnecessarily obscure the present invention. 
         [0027]    Furthermore, some portions of the detailed description that follow are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to most effectively convey the essence of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In the present invention, the steps carried out require physical manipulations of tangible quantities for achieving a tangible result. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals or instructions capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, instructions, or the like. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, can include the actions and processes of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system&#39;s registers and memories into other data similarly represented as physical quantities within the computer system&#39;s memories or registers or other information storage, transmission or display devices. 
         [0028]    The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer-readable storage medium including non-transitory medium, such as, but not limited to optical disks, magnetic disks, read-only memories, random access memories, solid state devices and drives, or any other types of media suitable for storing electronic information. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs and modules in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform desired method steps. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein. The instructions of the programming language(s) may be executed by one or more processing devices, e.g., central processing units (CPUs), processors, or controllers. 
         [0029]    Exemplary embodiments of the invention, as will be described in greater detail below, provide apparatuses, methods and computer programs for subsidiary volume management. 
         [0030]      FIG. 1  illustrates an example of a hardware configuration of a computer system in which the method and apparatus of the invention may be applied. The computer system includes storage system  2 , physical server  3 , and network  4 . The physical server  3  is coupled with a server management client  7 . The storage system is coupled with a storage management client  8 . The physical server  3  includes a plurality of virtual machines (VM)  5 . The storage system  2  includes a plurality of capacity pools  10  each of which has conventional Logical Units or Logical Unit Groups  12 . The Logical Unit Groups  12  each include an Administrative LU  13  and zero or a plurality of Subsidiary LUs  14  ( 14   a ,  14   b , etc.). The Administrative LU  13  controls the corresponding LU Group  12  to configure, create, delete, or migrate the plurality of subsidiary LU  14 . The Hypervisor issues a management command to the administrative LU to create, delete, or migrate (bind) subsidiary LU. Each Subsidiary LU  14  has a disk image of a VM  5 , respectively. 
         [0031]      FIG. 2  shows another example of the computer system having a plurality of physical servers and a plurality of storage systems. In this example, the storage system  2   a  has multiple capacity pools  10   a ,  10   c  and multiple LU Groups  12   a ,  12   b ,  12   c , and the storage system  2   b  has one capacity pool  10   d  and one LU group  12   d . LU groups  12   a  and  12   b  use capacity pool  10   a . LU group  12   c  uses capacity pol  10   c . LU group  12   d  uses capacity pool  10   d . A hypervisor of a physical server manages one or multiple LU Groups. Multiple hypervisors of multiple physical servers form a cluster group. Multiple hypervisors of the same cluster group manage one LU Group to create, delete, or migrate its subsidiary LUs. In this example, the Cluster Group  19  is the area of VM mobility from one hypervisor of a physical server to other hypervisor(s) of another physical server in the cluster group. 
         [0032]      FIG. 3 a    shows an example of a storage system. The physical storage system  2  includes a host I/F which connects to a host, CPU, Memory, Disk I/F which connects to HDDs forming logical volume  10 . In the storage controller, the host I/F, CPU, Memory, and Disk I/F are connected with each other by Bus I/F such as PCI, DDR, SCSI, or the like. 
         [0033]      FIG. 3 b    shows an example of a storage memory. The storage memory  33  includes storage program  34 , Logical Unit table  50  ( FIG. 5 ), pool resource table  60  ( FIG. 6 ), performance monitoring table  70  ( FIG. 7 ), and quality of service table  80  ( FIG. 8 ). 
         [0034]      FIG. 4 a    shows an example of a host server (physical server). The physical host  3  includes CPU, Memory, Disk I/F which connects to the storage system, and HDDs. The CPU, Memory, Disk I/F, and HDDs are connected with each other by Bus I/F such as PCI, DDR, SCSI, or the like. 
         [0035]      FIG. 4 b    shows an example of the host memory. The host memory  43  includes virtual machines (VMs)  5 , application software  45 , and virtual machine manager (VMM) or hypervisor  46 . 
         [0036]      FIG. 5  shows an example of a Logical Unit table. The LU table  50  has Logical Unit number field  51 , Pool ID field  52 , LU type field  53 , Administrative LU Number field  54 , Bind/unbind type field  55 , and Capacity field  56 . The Logical Unit number field  51  has identification number of Logical Unit  11 . The Pool ID field  52  identifies data pool for applying thin provisioning volume. The LU type field  53  shows the kind of LU type which may be Normal LU, administrative LU, or Subsidiary LU. The Administrative LUN field  54  has reference of LUN  51 . This field is valid when the LU type  53  is Subsidiary LU. The administrative LUN controls the subsidiary LU. The Bind/Unbind type field  55  shows current status of Bind or Unbind. This field is valid when the LU type  53  is Subsidiary LU. When the field  55  is Bind, the host VM can have access IO to the Subsidiary LU (which cannot be moved to another LU Group). When the field  55  is Unbind, the host VM does not have access IO to the Subsidiary LU (which can be moved to another LU Group). When the host hypervisor issues a bind or unbind command for a Subsidiary LU to an Administrative LU, the Subsidiary LU changes state and the host VM has access IO to the subsidiary LU. If the storage system fails to bind due to insufficient resource (e.g., insufficient remaining capacity for allocation) or poor/inadequate performance (e.g., performance below a preset performance threshold), the storage system returns the bind response with failure status. The Capacity field  56  shows allocation capacity from capacity pool  10  of the normal LU or the subsidiary LU. 
         [0037]      FIG. 6  shows an example of a pool resource table. The pool resource table  60  has pool ID field  61 , media type field  62  (e.g., Flash, SAS, NL-SAS/SATA), physical capacity field  63 , and remaining capacity field  64 . The media type field  62  contains media type in the capacity pool. The physical capacity field  63  shows physical media capacity for each media type. The remaining capacity field  64  shows remaining physical media capacity which can be allocated to new volume or new write data segment for thin provisioning. 
         [0038]      FIG. 7  shows an example of a performance monitoring table. The performance monitoring table  70  has pool ID field  71 , LUN field  72 , capacity usage field  73 , throughput field  74 , and IOPS (Input/Output Operations per Second) field  74 . The table describes capacity usage, transaction measurement value, and/or throughput performance measurement value, which are calculated by the storage program  34 . The administrative LU is control LU for the subsidiary LU and it does not have valid capacity, so that the table excludes the administrative LU. 
         [0039]      FIG. 8  shows an example of a quality of service table. The quality of service table  80  has pool ID field  81 , LUN field  82 , QoS group field  83 , throughput threshold field  84 , and IOPS threshold field  85 . This table describes host QoS group and performance threshold, which are used in monitoring storage performance. The administrative LU is control LU for the subsidiary LU and it does not have valid capacity, so that the table excludes the administrative LU. The QoS group field  83  is hypervisor cluster group of one or multiple physical host groups. The throughput threshold field  84  and lops threshold field  85  are threshold values of guaranteed LU performance. The storage system gets the QoS group  83  and performance threshold (throughput threshold  84  and/or IOPS threshold  85 ) from the server management client  7 . 
         [0040]      FIG. 9  shows an example of a flow diagram  900  illustrating a process for creating administrative LU and binding subsidiary LU. In step S 901 , the server administrator controls to power-on VM or deploy VM. The host hypervisor sends a bind command to the storage system. In step S 902 , the storage program  34  measures performance and updates the performance monitoring or measurement table  70 . The storage program checks whether the current performance is above the performance threshold (throughput threshold  84  and/or IOPS threshold  85 ) in the QoS table  80 , whether the high performance pool resource such as flash media tier which is described in the pool resource table  60  is insufficient, and/or whether the allocation capacity of thin provisioning will be higher than the physical capacity in the capacity pool which is described in the pool resource table  60 . If performance or capacity issue is detected (Yes), the next step is S 903 . If No, the next step is S 906 . 
         [0041]    In step S 903 , the storage program refers to the pool resource table  60 . The storage program chooses pool volume based on sufficient performance as indicated by the remaining capacity of the higher performance tier media or total remaining physical capacity in the pool. If the storage system has sufficient resource (YES), the next step is S 904 . If No, the next step is S 907 . In step S 904 , the storage program creates a new administrative LU, since the storage program detects that no administrative LU has sufficient resource in S 902 . In step S 905 , the storage program returns a failure response with the new administrative LUN created in S 904  as another candidate administrator LU to be rebinded. The host hypervisor will retry to bind to said another candidate administrative LU. 
         [0042]    In step S 906 , the storage program returns a success response of bind command from host hypervisor. 
         [0043]    In step S 907 , the storage program notifies alert of insufficient physical resource in the storage system to the storage administrator. In step S 908 , the storage program returns a failure response with a lack of administrative LU due to insufficient internal resources. 
         [0044]      FIG. 10  shows an example of a flow diagram  1000  illustrating a process for updating QoS configuration from the server administrator and operation of migrating (rebinding) a subsidiary LU. In step S 1001 , the server administrator updates the QoS configuration, and the host or server hypervisor sends a bind command to the storage system since the server hypervisor detects performance issue due to performance threshold and the hypervisor issues the bind command to migrate the subsidiary LU to another administrative LU. In step S 1002 , the storage program measures performance and updates the performance monitoring table  70 . The storage program checks whether the current performance is above the performance threshold (throughput threshold  84  and/or IOPS threshold  85 ) in the QoS table  80 , whether the high performance pool resource such as flash media tier which is described in the pool resource table is insufficient, and/or whether the allocation capacity of thin provisioning will be higher than the physical capacity in the capacity pool which is described in the pool resource table  60 . If performance or capacity issue is detected (Yes), next step is S 1003 . If No, the process ends. 
         [0045]    In step S 1003 , the storage program refers to the pool resource table  60 . The storage program chooses a pool volume based on sufficient performance as indicated by the remaining capacity of the higher performance tier media or total remaining physical capacity in the pool. If the storage system has sufficient resource (YES), the next step is S 1006 . If No, the next step is S 1005 . In step S 1004 , the storage program notifies alert of insufficient physical resource in the storage system to the storage administrator. In step S 1005 , the storage program returns a failure response with a lack of administrative LU due to insufficient internal resources. 
         [0046]    In step S 1006 , the storage program refers to the logical unit table  50 . If the capacity pool which is chosen in step S 1003  has one or multiple LU Groups (YES), the next step is S 1008 . If No, the next step is S 1007 . In step S 1007 , the storage program creates a new administrative LU, and the next step is S 1010 . In step S 1008 , the storage program chooses a candidate administrative LU for bind destination based on the pool resource table  60 . In step S 1009 , the storage system prepares to migrate data of the subsidiary LU to the destination administrative LU which was chosen in step S 1008 . In step S 1010 , the storage program returns a failure response with the new administrative LUN created in S 1007  or the candidate administrative LUN chosen in step S 1008  as another candidate administrator LU to be rebinded. The host hypervisor will retry to bind to said another candidate administrative LU. 
         [0047]      FIG. 11  shows an example of a flow diagram  1100  illustrating a process for subsidiary LU migration to another pool or another storage system. In step S 1101 , the server administrator controls to power-on VM or deploy VM. The host hypervisor sends a bind command to the storage system. In step S 1102 , the storage program refers to the performance monitoring or measurement table  70  to detect any performance or capacity issue in the administrative LU. If Yes, the next step is S 1103 . If No, the next step is S 1109 . 
         [0048]    In step S 1103 , the storage program chooses another administrative LU in the same pool. In step S 1104 , the storage program refers to the performance monitoring table  70  to detect any performance or capacity issue in the administrative LU. If Yes, the next step is S 1105 . If No, next step is S 1108 . In step S 1105 , the storage program chooses another administrative LU in another pool. In step S 1106 , the storage program refers to the performance monitoring table  70  to detect any performance or capacity issue in the administrative LU. If Yes, the next step is S 1107 . If No, the next step is S 1108 . In step S 1107 , the storage system chooses another administrative LU in another storage system. In step S 1108 , the storage program returns a failure response with a new administrative LUN chosen in step S 1103 , S 1105 , or S 1107 , as another candidate administrator LU to be rebinded. The host hypervisor will retry to bind to said another candidate administrative LU. In step S 1109 , the storage program returns a success response of the bind command from host hypervisor. 
         [0049]    Of course, the system configurations illustrated in  FIGS. 1 and 2  are purely exemplary of information systems in which the present invention may be implemented, and the invention is not limited to a particular hardware configuration. The computers and storage systems implementing the invention can also have known I/O devices (e.g., CD and DVD drives, floppy disk drives, hard drives, etc.) which can store and read the modules, programs and data structures used to implement the above-described invention. These modules, programs and data structures can be encoded on such computer-readable media. For example, the data structures of the invention can be stored on computer-readable media independently of one or more computer-readable media on which reside the programs used in the invention. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks, wide area networks, e.g., the Internet, wireless networks, storage area networks, and the like. 
         [0050]    In the description, numerous details are set forth for purposes of explanation in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that not all of these specific details are required in order to practice the present invention. It is also noted that the invention may be described as a process, which is usually depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. 
         [0051]    As is known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. Various aspects of embodiments of the invention may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out embodiments of the invention. Furthermore, some embodiments of the invention may be performed solely in hardware, whereas other embodiments may be performed solely in software. Moreover, the various functions described can be performed in a single unit, or can be spread across a number of components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions can be stored on the medium in a compressed and/or encrypted format. 
         [0052]    From the foregoing, it will be apparent that the invention provides methods, apparatuses and programs stored on computer readable media for subsidiary volume management. Additionally, while specific embodiments have been illustrated and described in this specification, those of ordinary skill in the art appreciate that any arrangement that is calculated to achieve the same purpose may be substituted for the specific embodiments disclosed. This disclosure is intended to cover any and all adaptations or variations of the present invention, and it is to be understood that the terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with the established doctrines of claim interpretation, along with the full range of equivalents to which such claims are entitled.