Abstract:
A system comprises a first storage system, a second storage system, a plurality of switches, and a server connected with the first storage system via a first group of switches and connected with the second storage system via a second group of switches. The first group and the second group have at least one switch which is not included in both the first and second groups. The first storage system receives I/O commands targeted to first logical units from the server via the first group of switches. The first storage system maintains first information regarding the ports of both the first and second storage systems. The first information is used to generate multipath communication between the server and the first storage system, including at least one path which passes through the second storage system and at least one other path which does not pass through the second storage system.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    The present invention relates generally to multipath storage systems and, more particularly, to nondisruptive data migration and I/O load balancing over multipath connections by running a virtual switch across storage systems and multipathing among storage and server. 
         [0002]    According to recent trends, a new multipath networking method known as TRILL (Transparent Interconnection of Lots of Links) is under standardization process for networking over Layer 2 (L2) Ethernet. TRILL allows L2 network to establish two parallel data transfer paths that have not been allowed in traditional Ethernet based on STP (Spanning Tree Protocol). When TRILL is set up and ready in a storage network, data transfer between server and storage will be performed through multiple paths. 
         [0003]      FIG. 1  is an example of a conventional storage area network topology. A server computer  300  is coupled to a switch  200 A, which is coupled to switches  200 B and  200 C in parallel, which are coupled to a switch  200 D, which is coupled to a data storage  200 . Data transfer from/to server to/from storage is executed through a path  300 - 200 A- 200 B- 200 D- 100  and  300 - 200 A- 200 C- 200 D- 100  in parallel. In this case, there may be a performance bottleneck at the network port  210 A of the switch  200 D or the network port  110  of the data storage  100  that cannot handle too much traffic received through both of the paths. 
         [0004]    In addition to L2 networking, another problem is that I/O service interruption happens through data migration beyond storage systems.  FIG. 2  is an example of a conventional logical configuration of a storage area network. A server computer  300  mounts one or more logical units  530  served by a storage system  100 A. An operating system running on the server  300  recognizes the logical units  530  by a network port  110 A identified by its network address (MAC Address, FCID) or WWN (World Wide Name). If an administrator tries to migrate a logical unit  530  from the network port  110 A to a port  110 B, a server operating system must stop I/O process to suspend the static data image stored in the logical unit, and to re-mount a new logical unit through the destination network port  110 B. However, a mission critical application or a business critical application running at an Enterprise Datacenter cannot be suspended while keeping its business stability. 
         [0005]    The same problem arises when a logical unit is to be migrated beyond the storage system boundary, for instance, from the port  110 A of one storage system  100 A to a port  110 C of another storage system  100 B. It requires data copy operation among systems, so that the I/O suspension time will be longer than internal LU migration. Furthermore, an additional problem of traditional single path network is that I/O service could be interrupted after the removal of the storage system  100 A because the network path must be reset onto the new data storage device. 
       BRIEF SUMMARY OF THE INVENTION 
       [0006]    Exemplary embodiments of the invention provide nondisruptive data migration and I/O load balancing over multipath connections by running a virtual switch across storage systems and multipathing among storage and server. The data storage is equipped with switching functionality and virtual port (network interface). Network routing and path configuration setting are shared among multiple data storage systems. Logically, a single virtual network switch runs across multiple data storage systems. Network devices have the capability to establish multipath communication between server and storage. This allows I/O service continuation after removal of old data storage device, because another data path keeps communication alive with the migration target device. In this way, the invention allows running a virtual switch across storage systems and multipathing among storage and server, so as to complete non-disruptive data migration and I/O load balancing over multipath connections. 
         [0007]    In accordance with an aspect of the present invention, a system comprises a first data storage system including at least one first interface port, a first CPU, a first memory, and a plurality of first logical units; a second data storage system including at least one second interface port, a second CPU, a second memory, and a plurality of second logical units, the second data storage system connected to the first data storage system; a plurality of switches; and a server which is connected with the first data storage system via a first group of the switches and is connected with the second data storage system via a second group of the switches, the first group and the second group having at least one switch which is not included in both the first group and the second group. The first data storage system receives I/O commands targeted to the plurality of first logical units from the server via the first group of switches. The first data storage system maintains a first information regarding the ports of both the first storage system and the second data storage system. The first information is used to generate multipath communication between the server and the first data storage system, including at least one path which passes through the second data storage system and at least one other path which does not pass through the second data storage system. 
         [0008]    In some embodiments, the first information includes information related to paths between ports of the first data storage system, the second data storage systems, the plurality of switches, and the server. The first information includes load information for transmitting data between ports of the first data storage system and the plurality of switches and the server and load information for transmitting data between ports of the second data storage system and the plurality of switches and the server. The ports of both the first and second data storage systems are identified by WWPN. 
         [0009]    In specific embodiments, one of the first and second data storage systems is a source system for migration of data to the other of the first and second data storage systems as a destination system. For data migration, the destination system creates a virtual port as a target port which has same identifier as a source port on the source system, and creates a logical unit on the target port, the source system runs data copy from a logical unit containing the data in the source system to the logical unit on the target port in the destination system, and deactivates the source port on the source system, and the destination system activates the target port on the destination system. In response to a detection of a device newly connected to one of the ports of the first data storage system, the first data storage system adds information related to a path between the newly connected device and the connected port of the first data storage system and notifies the added information to the plurality of switches, the server, and the second data storage system via connections to the first data storage system. A management computer is connected to one of the switches. In response to a request from the management computer, the switch updates path information between ports of the server and at least one of the first and second data storage systems. 
         [0010]    Another aspect of this invention is directed to a first data storage system in a system which includes a second data storage system having at least one second interface port, a plurality of switches, and a server which is connected with the first data storage system via a first group of the switches and is connected with the second data storage system via a second group of the switches, the first group and the second group having at least one switch which is not included in both the first group and the second group. The first data storage system comprises at least one first interface port; a first CPU; a first memory; and a plurality of first logical units. The first data storage system receives I/O commands targeted to the plurality of first logical units from the server via the first group of switches. The first data storage system maintains a first information regarding the ports of both the first storage system and the second data storage system. The first information is used to generate multipath communication between the server and the first data storage system, including at least one path which passes through the second data storage system and at least one other path which does not pass through the second data storage system. 
         [0011]    In some embodiments, the second data storage system is a source system for migration of data to the first data storage system as a destination system. For data migration, the first data storage system creates a virtual port as a target port which has same identifier as a source port on the second data storage system, creates a logical unit on the target port, and activates the target port on the first data storage system, after data copy is run from a logical unit containing the data in the second data storage system to the logical unit on the target port in the first data storage system, and after the source port on the second data storage system is deactivated. 
         [0012]    Another aspect of the invention is directed to a multipath communication method in a system which includes a first data storage system including at least one first interface port, a first CPU, a first memory, and a plurality of first logical units; a second data storage system including at least one second interface port, a second CPU, a second memory, and a plurality of second logical units, the second data storage system connected to the first data storage system; a plurality of switches; and a server which is connected with the first data storage system via a first group of the switches and is connected with the second data storage system via a second group of the switches, the first group and the second group having at least one switch which is not included in both the first group and the second group. The method comprises receiving an I/O command targeted to at least one of the plurality of first and second logical units from the server via the switches; maintaining a first information regarding the ports of both the first storage system and the second data storage system; and using the first information to generate multipath communication between the server and the first data storage system, including at least one path which passes through the second data storage system and at least one other path which does not pass through the second data storage system. 
         [0013]    In specific embodiments, the method further comprises a data migration process for migrating data from one of the first and second data storage systems as a source system to the other of the first and second data storage systems as a destination system. The data migration process comprises creating a virtual port as a target port which has same identifier as a source port on the source system; creating a logical unit on the target port; running data copy from a logical unit containing the data in the source system to the logical unit on the target port in the destination system; deactivating the source port on the source system; and activating the target port on the destination system. In response to a request from a management computer, the method further comprises updating path information between ports of the server and at least one of the first and second data storage systems. 
         [0014]    These and other features and advantages of the present invention will become apparent to those of ordinary skill in the art in view of the following detailed description of the specific embodiments. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0015]      FIG. 1  is an example of a conventional storage area network topology. 
           [0016]      FIG. 2  is an example of a conventional logical configuration of a storage area network. 
           [0017]      FIG. 3  shows an example of a storage network configuration according to an embodiment of the present invention. 
           [0018]      FIG. 4  illustrates a hardware configuration of the server computer. 
           [0019]      FIG. 5  illustrates a hardware configuration of the network switch. 
           [0020]      FIG. 6  illustrates a hardware configuration of the data storage. 
           [0021]      FIG. 7  illustrates a hardware configuration of the management computer. 
           [0022]      FIG. 8  illustrates an example of software that is stored on the memory and runs on the server computer. 
           [0023]      FIG. 9  illustrates an example of software that is stored on memory and runs on the switch. 
           [0024]      FIG. 10  illustrates an example of software that is stored on the memory and runs on the data storage. 
           [0025]      FIG. 11  illustrates an example of software that is stored on the memory and runs on the management computer. 
           [0026]      FIG. 12  illustrates an exemplary data structure of the volume configuration information in the memory of the server computer. 
           [0027]      FIG. 13  illustrates an exemplary data structure of the routing information in the memory of the switch. 
           [0028]      FIG. 14  illustrates an exemplary data structure of the transmission port information in the memory of the switch. 
           [0029]      FIG. 15  illustrates an exemplary data structure of the local storage network route information in the memory of the data storage. 
           [0030]      FIG. 16  illustrates an exemplary data structure of the shared storage network route information in the memory of the data storage. 
           [0031]      FIG. 17  illustrates an exemplary data structure of the storage transmission port information in the memory of the data storage. 
           [0032]      FIG. 18  illustrates an exemplary data structure of the LU configuration information in the memory of the data storage. 
           [0033]      FIG. 19  illustrates an example of the storage network topology according to the present embodiment. 
           [0034]      FIG. 20  is an example of a flow diagram to update the routing information and transmission port information on switch, or the shared local storage network route information and storage transmission port information on the data storage. 
           [0035]      FIG. 21  is an example of a flow diagram to select one or more paths from the server to the storage. 
           [0036]      FIG. 22  is an example of a flow diagram to combine the local storage network route information among two data storage systems. 
           [0037]      FIG. 23  is an example of a flow diagram of data migration beyond a data storage system. 
           [0038]      FIG. 24  is an example of a logical illustration of a virtual port over a virtual switch. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0039]    In the following detailed description of the invention, reference is made to the accompanying drawings which form a part of the disclosure, and in which are shown by way of illustration, and not of limitation, exemplary embodiments by which the invention may be practiced. In the drawings, like numerals describe substantially similar components throughout the several views. Further, it should be noted that while the detailed description provides various exemplary embodiments, as described below and as illustrated in the drawings, the present invention is not limited to the embodiments described and illustrated herein, but can extend to other embodiments, as would be known or as would become known to those skilled in the art. Reference in the specification to “one embodiment,” “this embodiment,” or “these embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention, and the appearances of these phrases in various places in the specification are not necessarily all referring to the same embodiment. Additionally, in the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that these specific details may not all be needed to practice the present invention. In other circumstances, well-known structures, materials, circuits, processes and interfaces have not been described in detail, and/or may be illustrated in block diagram form, so as to not unnecessarily obscure the present invention. 
         [0040]    Furthermore, some portions of the detailed description that follow are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to most effectively convey the essence of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In the present invention, the steps carried out require physical manipulations of tangible quantities for achieving a tangible result. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals or instructions capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, instructions, or the like. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, can include the actions and processes of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system&#39;s registers and memories into other data similarly represented as physical quantities within the computer system&#39;s memories or registers or other information storage, transmission or display devices. 
         [0041]    The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer-readable storage medium, such as, but not limited to optical disks, magnetic disks, read-only memories, random access memories, solid state devices and drives, or any other types of media suitable for storing electronic information. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs and modules in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform desired method steps. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein. The instructions of the programming language(s) may be executed by one or more processing devices, e.g., central processing units (CPUs), processors, or controllers. 
         [0042]    Exemplary embodiments of the invention, as will be described in greater detail below, provide apparatuses, methods and computer programs for nondisruptive data migration and I/O load balancing over multipath connections by running a virtual switch across storage systems and multipathing among storage and server. 
         [0043]      FIG. 3  shows an example of a storage network configuration according to an embodiment of the present invention. First and second server computers  300 A and  300 B are coupled to a switch  200 A, which is coupled to two switches  200 B and  200 D. The switch  200 B is connected to a switch  200 C which is connected to a first data storage  100 A. The switch  200  D is connected to a switch  200 E which is connected to a second data storage  100 B. A management computer  400  is connected to the switch  200 B. The server  300  runs business applications and generates I/O workload that targets a storage  100 . The switch  200  is a network switch, i.e., Layer 2 Ethernet switch that supports TRILL protocol. The data storage  100  is an external storage system that is installed with a bunch of disk drives (HDDs) or solid state drives (SSDs). The management computer  400  provides management of the entire storage network. In  FIG. 3 , communication between the first server computer  300 A and the first data storage  100 A can be established through both paths  300 A- 200 A- 200 B- 200 C- 100 A and  300 A- 200 A- 200 D- 200 E- 100 B- 100 A. 
         [0044]      FIG. 4  illustrates a hardware configuration of the server computer  300 . A CPU  330 , a memory  340 , an input device  360  (e.g., keyboard, mouse, etc.), and an output device  370  (e.g., video graphic card connected to external display monitor) are interconnected through a memory controller  350 . All I/Os handled by an I/O controller  320  are processed on an internal HDD device  380  or an external storage device through a network interface  310 . This configuration can be implemented by a multi-purpose PC. 
         [0045]      FIG. 5  illustrates a hardware configuration of the network switch  200 . A CPU  230  and a memory  240  are interconnected through a memory controller  250 . The I/Os handled by an I/O controller  220  are processed through a plurality of network interfaces  210 . 
         [0046]      FIG. 6  illustrates a hardware configuration of the data storage  100 . A CPU  130  and a memory  140  are interconnected through a memory controller  150 . The I/Os handled by an I/O controller  120  are processed on internal HDD devices  180  or external storage devices through network interfaces  310 . 
         [0047]      FIG. 7  illustrates a hardware configuration of the management computer  400 . A CPU  430 , a memory  440 , an input device  460 , and an output device  470  are interconnected through a memory controller  450 . The I/Os handled by an I/O controller  420  are processed on an internal HDD device  480  or an external storage device through a network interface  410 . 
         [0048]      FIG. 8  illustrates an example of software that is stored on the memory  340  and runs on the server computer  300 . An application program  3401  is a business application that generates I/O workload (e.g., database, SAP, E-Mail, exchange server, web application, etc.). An I/O transfer control program  3402  controls external data I/O transfer communication over SCSI protocol and also setup communication path between the server  300  and the storage  100 . Volume configuration information  3403  is a configuration definition of data volume handled by the server operating system. “/etc/fstab” is a simple example of the volume configuration information  3403 . Its data structure is illustrated in  FIG. 12 . 
         [0049]      FIG. 9  illustrates an example of software that is stored on memory  240  and runs on the switch  200 . Network route management program  2401  is a program to set and release communication route over the network. Traffic monitor program  2402  is a program to measure the traffic by the network interface  210 . It can be measured by metric such as bps (byte per sec) and IOPS. Route information  2403  is configuration data that expresses communication route set by the network route management program  2401 . Transmission port information  2404  is configuration data that expresses a target network interface  210  to transmit data. Routing information  2403  and transmission port information  2404  make it possible to determine communication paths over the network. 
         [0050]      FIG. 10  illustrates an example of software that is stored on the memory  140  and runs on the data storage  100 . I/O transfer control program  1401  operates external data I/O transfer communication over SCSI protocol and also sets up communication path between the data storage  100  and the server  300 . Storage network route management program  1402  is a unique program in this invention. This program generates and updates local storage network route information  1406  and shared storage network route information  1407 . It merges route information created by several data storage systems, so as to keep consistency among the storage systems. Configuration management program  1403  updates logical unit configuration as directed by the management computer  400 . Data copy program  1404  copies entire data stored in one logical unit  530  into another logical unit  530  so that the original logical unit  530  is duplicated. Traffic monitor program  1405  measures I/O traffic by network interface  110  and logical unit  530 . Its metric is acceptable in bps (byte per sec), IOPS, and the like. Shared storage network route information  1407  is information shared among multiple data storage systems  100 . It defines communication routes over the network. Storage transmission port information  1408  allows determining communication paths over the network. LU configuration information  1409  is a configuration setting of the logical units  530 . 
         [0051]      FIG. 11  illustrates an example of software that is stored on the memory  440  and runs on the management computer  400 . I/O path control program  4401  communicates with devices that comprise the storage network. It issues requests to set or update communication paths. LU configuration request program  4402  communicates with the data storage  100 . It issues requests to set or update the logical units  530 . LU configuration information  4403  is a collection of LU configuration information  1408  from multiple data storage systems  100 . Routing information  4404  is a collection of routing information  2403  and shared storage network route information  1407  from multiple switches  200  and data storage systems  300 . Transmission port information  4405  is a collection of transmission port information  2404  and storage transmission port information  1408 . Information collected from the switch  200  and information collected from the data storage  100  do not have to be distinguished, but can be handled in the same manner. The management computer  400  updates those pieces of information so that it always keeps the newest configuration. The memory  140  of the data storage  100  includes mapping information between the physical ports and virtual ports, so that virtual ports may be treated as physical ports. The relation between the virtual port and the physical port may not be limited to a one-to-one relationship. One physical port may be associated with multiple virtual ports and one virtual port may be associated with multiple physical ports. The mapping information should be controlled by the management computer; thus the mapping information of the data storages  100  may be integrated in the memory  440  of the management computer  400  and the updates would be communicated to each other. 
         [0052]      FIG. 12  illustrates an exemplary data structure of the volume configuration information  3403  in the memory  340  of the server computer  300 . Mount point  34031  is a logical directory defined on a file system. An external device such as a logical unit is mounted to this location. Target FCID  34032  is the identification of the network interface  110  that is dynamically assigned by the fibre channel network when it initialized a fabric login process. Target device  34033  is an identification of network interface  110 . World Wide Name is usually used as an identifier in the fibre channel network. LUN  34034  is a “logical unit number” that is assigned to each logical unit  530 . 
         [0053]      FIG. 13  illustrates an exemplary data structure of the routing information  2403  in the memory  240  of the switch  200 . Local port address  24031  is a network interface installed on the switch  200 . Remote port address  24032  is a port installed on the other devices. The remote port must be reachable from the local port address over one or more switches  200 . Transfer cost  24033  is a hop count of the devices to reach from local port to remote port. 
         [0054]      FIG. 14  illustrates an exemplary data structure of the transmission port information  2404  in the memory  240  of the switch  200 . Remote port address  24031  is a network interface installed on the other devices. Local delivery port address  24032  is a local network interface to communicate and transmit data with the remote port. 
         [0055]      FIG. 15  illustrates an exemplary data structure of the local storage network route information  1406  in the memory  140  of the data storage  100 . A first table  1406 A is an example of the route information generated on the first storage  100 A of  FIG. 19 . A second table  1406 B is an example of the route information generated on second storage  100 B of  FIG. 19 . Local port address  14061 , remote port address  14062 , and transfer cost  14063  represent the same entities serving the same functions, respectively, as those in  FIG. 13 . A unique feature in this table is that the virtual port  520  set on this data storage  100  can be recorded as same as the physical port  110 . 
         [0056]      FIG. 16  illustrates an exemplary data structure of the shared storage network route information  1407  in the memory  140  of the data storage  100 . Local port address  14071  is a network interface  110  installed on one of the data storage systems  100 . Remote port address  14072  is a network interface installed on an external device other than the data storage systems  100 . The remote port must be reachable from the local port. Transfer cost  14073  is a hop count of the devices to reach from local port to remote port. The contents of the table in  FIG. 16  are consistent with the storage network topology in  FIG. 19 . For example, the network interface  110 A on the data storage  100 A is directly connected to the network interface  210 E on the switch # 23 , so that the transfer cost is counted as “1.” On the other hand, a route from the network interface  110 A to the network interface  310 A on the server  300 A requires hops of four devices. A unique feature on third entry, network interface  110 C is logically considered as connected to virtual port  520 A. When the data storage  100  detects a configuration change on the other storage system(s)  100 , it updates its route information to keep the information current and consistent. 
         [0057]      FIG. 17  illustrates an exemplary data structure of the storage transmission port information  1408  in the memory  140  of the data storage  100 . Remote port address  14081  is a network interface  110  installed on the other devices. Local delivery port address  14082  is a local network interface  110  to communicate and transmit data with the remote port. 
         [0058]      FIG. 18  illustrates an exemplary data structure of the LU configuration information  1409  in the memory  140  of the data storage  100 . Local port address  14091  is a network interface  110  or virtual network interface  520  defined on the storage  100 . The virtual network interface  520  is not a physical network interface  110  but behaves as if it were installed on the data storage  100  against the server computer  300 . World Wide Name  14092  is the identification of the network interface  110  or virtual network interface  520 . LUN  14093  is a “logical unit number” to identify the logical unit  530  defined on the network interface  110  or virtual network interface  520 . Storage resource ID  14094  is a physical storage resource such as RAID group or a set of HDDs or SSDs. 
         [0059]      FIG. 19  illustrates an example of the storage network topology according to the present embodiment. The first server  300 A attaches a logical unit  530 A that is defined on a virtual network port  520 A in the first data storage  100 A at “/mount/data2” (see  FIG. 12 ). The switch  200 A is configured to use dual communication paths to the virtual network port  520 A, through the network interfaces  210 B and  210 C (see  FIGS. 13 &amp; 14 ). Originally this configuration does not happen because one path “ 210 C-# 24 -# 25 - 110 B- 110 D- 520 A” has a transfer cost of “5” to get to the virtual port  520 A, which is not equivalent to another path “ 210 B-# 22 -# 23 - 110 A- 520 A” having a transfer cost of only “4.” This is allowed by considering multiple virtual switches as a single device, as defined in  FIG. 16 . From the viewpoint of the server computer  300  and the switch  200 , physically multiple data storage systems  100  are recognized as a single data storage  100 . This aspect of the present embodiment solves the first problem of bottleneck mentioned above in the Background section. The bottleneck on the direct attached switch # 23  will not occur because another path is routed via the second data storage  100 B. 
         [0060]      FIG. 20  is an example of a flow diagram to update the routing information  2403  and transmission port information  2404  on switch  200 , or the shared local storage network route information  1406  and storage transmission port information  1408  on the data storage  100 . First of all, the switch  200  or data storage  100  detects a device newly connected to the network interface  210  or  110  (step S 101 ). Then it creates a new entry on the routing information  2403  or  1406 , then record “1” in its transfer cost field  24033  or  14063  (step S 102 ). The switch  200  or data storage  100  then notifies a new entry record to the other devices connected directly through its network interface  210  or  110  (step S 103 ). Next, the switch  200  or data storage  100  which has received a new device discovery notification updates its routing information  2403  or  1406  (step S 104 ). In this case, the transfer cost field  24033  or  14063  will be added “1.” This device repeats notification to the other network devices (step S 105 ). After that, it determines one or more paths to get to the newly detected network interface (step S 106 , step S 107 ). In the step S 106  and the step S 107 , the switch  200  or data storage  100  selects one or more network interfaces  210  or  110  that have minimum transfer cost to get to the new device and updates the transmission port information  2404  or  1407 . 
         [0061]      FIG. 21  is an example of a flow diagram to select one or more paths from the server  300  to the storage  100 . This is not a mandatory process but optional. In step S 201 , the management computer  400  chooses I/O paths that pass the target storage  100 . In step S 202 , the management computer  400  requests path update. In step  203 , the switch and virtual switch updates the path information. This is a conventional option, especially in a situation where an administrator wants to control its network traffic after monitoring and analyzing the data traffic. Also this is useful when three or more paths are available and the administrator wants to reduce them. 
         [0062]      FIG. 22  is an example of a flow diagram to combine the local storage network route information  1406  among two data storage systems  100  (e.g.,  100 A and  100 B). After detecting a newly connected device on the local network port, a data storage  100 A adds a new routing information entry on the local storage network information  1406 . Then the data storage  100 A transfers the new route information entry to another data storage  100 B (step S 301 ). The data storage  100 B receives the new route information and then searches its local storage network route information  1406  to confirm if there is a route information that is the same as that received from the original data storage  100  (step S 302 ). In the example of  FIG. 15 , after the data storage  100 B receives a new entry to express a path target to the network interface  210 F by a transfer cost of “2,” it searches and finds the same target route entry for the network interface  210 F by a transfer cost of “1.” If the result of step S 302  is “Yes,” it determines to adopt a route with a lower transfer cost (step S 303 ). In the case of  FIG. 15 , the data storage  100 B adopts its local entry that targets to the network interface  210 F. It updates the route information on the shared storage network routing information  1407  (step S 304 , step S 305 ). 
         [0063]      FIG. 23  is an example of a flow diagram of data migration beyond a data storage system  100 . A destination storage  100  (i.e., migration target device) creates a new virtual port  520  (step S 401 ). This virtual port  520  has the same identifier as the source port  520 . Then it creates a logical unit  530  on the port  520  (step S 402 ). It is clear that new entry is added to the LU configuration information  1409 . Then data copy program  1404  runs data copy from source LU to destination LU beyond the device (step S 403 ). After data copy is completed, the source storage  100  deactivates the source virtual port  520  (step S 404 ). Just after step S 404 , the target virtual port  520  is activated (step S 405 ). The data migration is typically performed in response to a request from the management server  400 . 
         [0064]      FIG. 24  is an example of a logical illustration of a virtual port  520  over a virtual switch  500 . In this embodiment, a hardware boundary across the data storage  100  can be ignored, so that the virtual port location is flexible over the virtual switch  500 . Also, the server  300  and the switch  200  would not get any impact caused by data migration. They do not have to reconfigure their configurations, and have a very short I/O service interruption period that happens at step S 404  and S 405 . 
         [0065]    Of course, the system configuration illustrated in  FIG. 19  is purely exemplary of information systems in which the present invention may be implemented, and the invention is not limited to a particular hardware configuration. The computers and storage systems implementing the invention can also have known I/O devices (e.g., CD and DVD drives, floppy disk drives, hard drives, etc.) which can store and read the modules, programs and data structures used to implement the above-described invention. These modules, programs and data structures can be encoded on such computer-readable media. For example, the data structures of the invention can be stored on computer-readable media independently of one or more computer-readable media on which reside the programs used in the invention. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks, wide area networks, e.g., the Internet, wireless networks, storage area networks, and the like. 
         [0066]    In the description, numerous details are set forth for purposes of explanation in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that not all of these specific details are required in order to practice the present invention. It is also noted that the invention may be described as a process, which is usually depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. 
         [0067]    As is known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. Various aspects of embodiments of the invention may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out embodiments of the invention. Furthermore, some embodiments of the invention may be performed solely in hardware, whereas other embodiments may be performed solely in software. Moreover, the various functions described can be performed in a single unit, or can be spread across a number of components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions can be stored on the medium in a compressed and/or encrypted format. 
         [0068]    From the foregoing, it will be apparent that the invention provides methods, apparatuses and programs stored on computer readable media for nondisruptive data migration and I/O load balancing over multipath connections. Additionally, while specific embodiments have been illustrated and described in this specification, those of ordinary skill in the art appreciate that any arrangement that is calculated to achieve the same purpose may be substituted for the specific embodiments disclosed. This disclosure is intended to cover any and all adaptations or variations of the present invention, and it is to be understood that the terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with the established doctrines of claim interpretation, along with the full range of equivalents to which such claims are entitled.