Source: https://patents.google.com/patent/US20040226021?oq=6%2C757%2C710
Timestamp: 2018-02-19 13:56:50
Document Index: 628240445

Matched Legal Cases: ['art 1350', 'art 1330', 'art 1350', 'art 1350', 'art 2050', 'art 2040', 'art 2040', 'art 2040', 'art 2020', 'art 1350', 'art 2050', 'art 2050', 'art 2020', 'art 2040', 'art 2040', 'art 2040', 'art 2040', 'art 4000', 'art 4000', 'art 4000', 'art 4030', 'art 4030', 'art 4030', 'art 4020', 'art 4000', 'art 4030', 'art 4000', 'art 4000', 'art 4020', 'art 2040', 'art 4000', 'art 4000']

US20040226021A1 - Storage device system having bi-directional copy control function - Google Patents
US20040226021A1
US20040226021A1 US10750859 US75085904A US2004226021A1 US 20040226021 A1 US20040226021 A1 US 20040226021A1 US 10750859 US10750859 US 10750859 US 75085904 A US75085904 A US 75085904A US 2004226021 A1 US2004226021 A1 US 2004226021A1
US10750859
US7047256B2 (en )
This invention, in order to enable bidirectional copying between storage device systems, provides a data consistency holding control device in the storage device systems which constitute the copy pair. This data consistency holding control device performs control so as to write write-in data which was received from the superior device and write-in data which was received from the other storage device system through the communication interface device into the physical storage device after such write-in data has been made to wait in a temporary storage device for more than predetermined time from the time of reception corresponding to the write-in of data to the logical volume which forms the copy pair, so that, when write-in data which was received from the superior device and write-in data which was received from the other storage device system through the communication interface device are written in an overlapped manner into the same storage location of the physical storage device, they are written in the order of the reception time when the write-in data was received from the superior device.
[0018]FIG. 1 is a block diagram of a computer system showing a mode for carrying out the invention;
[0019]FIG. 2 is a diagram illustrating a bidirectional remote copy function as a mode for carrying out the invention;
[0020]FIG. 3 is a hardware block diagram of a storage device system illustrating a mode for carrying out the invention;
[0021]FIG. 4 is a software block diagram of a storage device system representing a first embodiment of the present invention;
[0022]FIG. 5 is a data consistency holding table for use in the first embodiment;
[0023]FIG. 6 shows an example of a temporary bit map table for use in the first embodiment;
[0024]FIG. 7 is a flow chart showing process procedures for processing an I/O operation which was received from a host I/F in the first embodiment;
[0025]FIG. 8 is a flow chart showing process procedures for processing an I/O operation which was received from the DKCI/F of the first embodiment;
[0026]FIG. 9 is a flow chart showing the processing at the time of a write-in request of a data consistency holding control part of the first embodiment;
[0027]FIG. 10 is a flow chart of a doubled block deletion process of the data consistency holding control part of the first embodiment;
[0028]FIG. 11 is a flow chart of a simultaneous write-in process of the data consistency holding control part of the first embodiment;
[0029]FIG. 12 is a software block diagram of a storage device system according to a third embodiment;
[0030]FIG. 13 is a table of bit map values of the third embodiment;
[0031]FIG. 14 shows an example. of a locked state holding table of the third embodiment;
[0032]FIG. 15 is a flow chart showing the processing of an I/O request which was received from a host I/F of the third embodiment;
[0033]FIG. 16 is a flow chart showing the process procedures for processing an I/O request which was received from the DKCI/F of the third embodiment;
[0034]FIG. 17 is a flow chart showing the processing of a write-in request of a data consistency holding control part of the third embodiment;
[0035]FIG. 18 is a flow chart showing the processing of a write-in request of the data consistency holding control part of the third embodiment;
[0036]FIG. 19 is a flow chart of a superior device usable/unusable judging process of an exclusive control part of the third embodiment;
[0037]FIG. 20 is a flow chart of a lock forming process in response to a lock forming request which was received from a host I/F of the exclusive control part of the third embodiment;
[0038]FIG. 21 is a flow chart of a lock forming process in response to a lock forming request which was received from DKCI/F of the exclusive control part of the third embodiment;
[0039]FIG. 22 is a block diagram of a computer system to which the third embodiment is applied;
[0040]FIG. 23 is a process sequence chart representing one example of the processes of the system of FIG. 22;
[0041]FIG. 24 is a flow chart of a lock release process in response to a lock release request which was received from a host I/F of the exclusive control part of the third embodiment; and
[0042]FIG. 25 is a flow chart of a lock release process in response to a lock release request which was received from DKCI/F of the exclusive control part of the third embodiment.
Hereinafter, a first embodiment which relates to bidirectional copying will be described with reference to the drawings.
[0045]FIG. 1 is a block diagram of a computer system 1100 which employs the features for carrying out the invention. The computer system 1100 is configured by a site 1110 which comprises a plurality of superior devices 1000 and 1010 which are connected through a SAN (Storage Area Network) 1040 to a storage device system 1070, and a site 1120 which comprises a plurality of superior devices 1020 and 1030 which are connected through a SAN 1050 to a storage device system 1080. The storage device system 1070 and the storage device system 1080 are connected by a remote copy line 1060, which may be a dedicated line or a public line. The storage device system 1070 and the storage device system 1080 can communicate with each other through the remote copy line 1060, using SCSI protocol.
[0046]FIG. 2 is a view illustrating a situation in which bi-directional copying is carried out in the computer system 1100 shown in FIG. 1. In FIG. 2, the superior device 1010 carries out a write-in B 1200 of data to the storage device system 1070. The data of the write-in B 1200, after it was stored in the storage device system 1070, is transmitted (arrow 1240) through the remote copy line 1060, and it is copied by the storage device system 1080. In the same manner, as to write-in C 1210 from the superior device 1020 to the storage device system 1080, data is also transmitted (arrow 1230) through the remote copy line 1060, and it is copied by the storage device system 1070. In the same manner, as to write-in D 1220 from the superior device 1030 to the storage device system 1080, data is also transmitted (arrow 1250) through the remote copy line 1060, and it is copied by the storage device system 1070. In short, respective write-in operations B 1200, C 1210 and D 1220 of each superior device 1010, 1020 and 1030 are written into storage systems of the respective sites, and then, the data is copied by the storage device systems of other part sites without intervention by the superior devices.
[0047]FIG. 3 shows an example of the hardware structure of the storage device system 1070. As an example, the storage device system 1070 may consist of a disk array device, a semiconductor storage device and so on. The storage device system 1070 is configured by a host I/F 1300, DKCI/F 1320, a disk control part 1350, a shared memory 1360, a cache memory 1340, and a switching control part 1330, which is configured as a cross bus switch etc. for connecting these elements in such a manner that they can communicate with each other, as well as a timer 1310, a physical disk 1370 and a processor 1380, and so on.
The cache memory 1340 is mainly used for temporarily storing data which is given and received between the host I/F 1300, the DKCI/F 1320 and the disk control part 1350. For example, in case a data input/output command which the host I/F 1300 has received from a superior device, is a write-in command, the host I/F 1300 writes the write-in data which was received from the superior device into the cache memory 1340. Also, the disk control part 1350 reads out the write-in data from the cache memory 13-40 and writes it into the physical disk 1370.
[0053]FIG. 4 is a view showing software for carrying out a bi-directional remote copy of data according to the invention. In order to realize the bi-directional remote copy feature of the storage device system, each storage device system has a main control 2020, a cache part 2050 and a data consistency holding control part 2040 in the form of programs. These programs are executed by the processor 1380. Also, a bit map table 2030 is disposed in the memory.
The data consistency holding control part 2040 is activated when the input/output request is one which relates to data write-in, and it effects control in such a manner that the consistency of data is held between storage device systems in which the bidirectional copying is carried out, on the basis of the bit map table 2030.
[0061]FIG. 5 shows the a data format of a data consistency holding table 100 which is held by the data consistency holding control part 2040. The data consistency holding table 100 is configured of a table control number 101, reception time 102, superior device identification number 103, object block start address 104, object size 105, storage serial number 106 and cache data storage address 107.
[0065]FIG. 6 shows the data format of the bit map table 2030. Each column of the bit map table 2030 is arranged in the order of a block, corresponding to a block on the physical disk. The bit value 1 or 0 is set in each column, and, as indicated above, it shows whether or not data of the block is cached. The temporary bit map table 200 is a table for storing a bit map of a block, which is a target of an input/output operation of the input/output request from the superior device.
[0067]FIG. 7 is a flow chart showing an example of the process procedures of the main control part 2020, which processes an I/O request from the host I/F 1300. When the I/O request has come from the host I/F 1300 to the storage device system, at a step 3000, a reference series command (command such as read in SCSI) is discriminated from an update change series command (command such as write in SCSI), and the processes are bifurcated. In this embodiment, attention is paid only to the reference series command and the update change series command. In the case of the reference series command, the process goes to a step 3005. In the case of the update change series command, the process goes to a step 3050.
In the reference scope which is requested by the superior device, a storage location of the most recent data is changed between a block scope of, the bit value 0 and a block scope of the bit value 1 of the temporary bitmap table. 200. When the reference request in the block scope of the bit value is 0, a situation exists in which data on the physical disk 1370 is the most recent data, and the process goes to a step 3020. When the reference request in the block scope of the bit value is 1, a situation exists in which data-on the cache memory 1340 is the most recent data, and the process goes to a step 3040. The step 3020 is a process for reading reference scope data through the disk control part 1350. The step 3040 reads the reference scope data through the cache part 2050. On the occasion of referring to data through the cache part 2050, by use of the data consistency holding table 100, data in a reference object scope is read. More specifically, as to records of the data consistency holding table 100, in the order from the largest one toward a smaller one of the table control number 101, the reference scope and the object block address 204, the object size 205 are compared, and an address of the cache memory 1340, in which changed data is cached is searched, and the reference scope data is read from the cache memory 1340.
[0073]FIG. 8 is a flow chart of a process which is carried out when the main control part 2020 has received write-in data from the DKCI/F 1320. A step 3100 is a data consistency holding control part process.
[0074]FIG. 9 is a flow chart in the case where the data consistency holding control part 2040 has processed the write-in request from the superior device. The data consistency holding control part 2040 is a control part which carries out a process of write data (write-in content), so that a logical volume for forming the remote copy pair becomes a volume which was completely duplicated. The data consistency holding control part 2040 updates the data consistency holding table 100 and the bit map table 2030, and carries out a process for writing the write data on the cache memory 1340.
[0081]FIG. 10 is a flow chart showing an overlapped block deletion process. The overlapped block deletion process is a process which is carried out in case there is an overlap between a write-in scope of the write data which is requested to be written into the storage device system and a write-in scope of the cache data which exists on the cache memory 1340. The fact that there is an overlap means that the write data overwrites the cache data or the cache data overwrites the write data. The overlapped block deletion process is a process used for deleting a portion which is overwritten and updated in this overlapped scope.
Here, there is a case which requires an exceptional process. This exceptional process is carried out in case that, by a partial deletion of the cache data, an intermediate portion of the cache data is extracted, and the cache data is divided into two records. More specifically, it is supposed that the object size 105 is 20 blocks from the object block start address 104 of the cache data as a deletion object: 1000 address. In case that a write-in request of the object size 105: 100 blocks came there from the write data object block start address 104: address 1020, a record which was written in the data consistency holding table 100 of the cache data, due to deletion of the overlapped portion with the write data, (the object block start address 104, the object size 105) is divided into two records of (1000,19), (1121,79). A division process which is carried out here gets down a record after the record which is represented by current cache data by 1 row, and changes (+1) the control number to the record in which the table control number 101 was gotten down. By use of the record row which was prepared by this process and is 1 row below, the cache data is divided into two records.
[0093]FIG. 11 is a flow chart showing a process (simultaneous write-in process) of the data consistency holding control part 2040 for writing the cache data in the cache memory 1340 to the physical disk 1370. The simultaneous write-in process is activated simultaneously in all storage device systems once every 60 seconds by the timer 1310. The step 3400 replaces the variable current with 1 of the table control number 101 which is the oldest written record in the data consistency holding table 100. After the substitution, the process goes to the step 3410. In the step 3410, the table control number 101 refers to the reception time 102 of the current record. When the reception time 102 of the current. record has not passed 3 minutes since the simultaneous write-in process start time, a process goes to the step 3470, and when more than 3 minutes pass, the process goes to a step 3420.
The exclusive control part 4000 holds a lock state, holding table 400, and it manages a reserve state from a superior device. The exclusive control part 4000 effects control in such a manner that each of the storage device systems which become objects of the bidirectional remote copy have lock state holding tables 400 of the same content. By having lock state holding tables 400 of the same content, on the occasion that a superior device has locked a volume of a certain storage device system, a pair volume of the other site which constitutes the copy pair is turned in a locked state.
[0108]FIG. 14 shows the data format of a lock state holding table 400 which is held by an exclusive control part 4000. The lock state holding table 400 is configured by such items as a control number 401, a lock start time 402, a superior device identification number 403, a lock object start address 404 and a lock object size 405. The lock start time 402 indicates the time that a lock request was received from a superior device to the host I/F 1300 in the storage device system. The superior device identification number 403 has the same meaning as the superior device identification number 203 of the data consistency holding table 100. The lock start address 404 and the object size 405 set a block address and the number of blocks of a disk which becomes a lock object, respectively. The control number 401 is supposed to store integer control numbers from the head of a table in a sequence of 1, 2, 3, . . . The control number starts from 1 and increases with +1 at a time, and the control number next to a final record is set to −1.
[0109]FIGS. 15 and 16 are flow charts showing process procedures of the main control part 4030. FIG. 15 shows an example in which a modification was made to FIG. 7, and FIG. 16 shows an example in which a modification was made to FIG. 8. The modifications will be described hereinafter. FIG. 15 is a flow chart when the main control part 4030 received I/O from -the host I/F 1300.
[0114]FIG. 16 is a flow chart showing the process procedures in the case in which the main control part 4030 has received an I/O request from DKCI/F 1320. A step 5100 recognizes the update change series command or the lock series command, and the process is bifurcated. In the case of the update change series command, the process goes to a step 5110. In the case of the lock series command, the process goes to a step 5120. The step 5110 is a process of the data consistency holding control part 4020 which will be described later. The step 5120 is a step employed in the case where the lock series command was handed over from the DKCI/F 1320. The process of the step 5120 is carried out in the exclusive control part 4000, and the main control part 4030 receives its return value from the exclusive control part 4000. After reception of the return value has been completed, the process goes to a step 5130. The step 5130 hands over the return value, which was received from the exclusive control part 4000, to the DKCI/F 1320. After the return value has been transmitted to the DKCI/F 1320, the process is complete.
[0115]FIGS. 17 and 18 are flow charts showing the processing of the data consistency holding control part 4020. The main difference from the data consistency holding control part 2040 is a change due to an increase of the states which are possessed by the bit map table 4010. FIG. 17 shows a modification of the flow chart shown in FIG. 9. In FIG. 17, the step 3200 was changed to a step 5200, and the step 3210 was changed to a step 5210, and the step 3270 was changed to a step 5220. The step 5200 compares the write-in scope of the write data with corresponding bit values of the bit map table 4010. As a result of this comparison, when all of the bit values of the scope of the write data are the bit value 0 or 2 (when the most recent data exists in the physical disk 1370), the process goes to a step 5220. As to the scope of the write data, when corresponding bit values of the bit map table 4010 include the bit value 1 or 3 (when the most recent value exists on the cache memory 1340), the process goes to a step 5210.
[0118]FIG. 18 shows a process in which the step 3420 of writing cache data on the cache memory 1340 into the physical disk 1370 (simultaneous write-in process) was changed to a step 5300. The step 5300 is a process for changing bit values of the bit map table 4010. In case a bit value of the bit map table 4010 which is an object to be changed is the bit value 1, it operates to change it to the bit value 0; and, in case of the bit value 3, it operates to change it to the bit value 2. This is a process for changing the bit value of the bit map due to disappearance of the cache data in the cache memory 1340 by the simultaneous write-in process.
[0119]FIGS. 19, 20 and 21 are flow charts of the processes relating to the exclusive control part 4000. FIG. 19 shows a process for judging whether the write-in process in the write-in scope is possible, on the occasion that a write-in request of a superior device is received, i.e., whether the write-in scope has already been locked by one other than the superior device which issued the write-in request.
[0121]FIG. 20 is a flow chart of the processing by the exclusive control part 4000 in the case where the host I/F 1300 has received a lock request in connection with a disk from a superior device. A step 5500 judges whether the lock request scope (block scope of a disk which is requested to be locked) has been locked by another superior device, by referring to bit values of the bit map table 4010. When all of bit values which correspond to the lock request scope are 0 or 1 (a state in which no superior device effects a lock), the process goes to a step 5530. In case where 2 or 3 is included in the bit values which correspond to the lock request scope, it means that the lock request scope has been already locked by a certain superior device. On this occasion, in a step 5505, it is judged whether or not the reserve of a lock designation scope is a reserve state of a superior device which is requesting a lock this time. In the case of a lock request from the superior device which is locking, lock completion is set to the return value.
In the case of a lock state of another superior device, fundamentally, lock failure is set to the return value. However, under a remote copy environment, there may be a case in which a lock deletion process is being executed by the other site of the copy pair, and a lock request was received during a period in which its process result arrives at its own site. Thus, by the process of a step 5510,, the lock request is transmitted to the copy destination storage device system. This return value is judged in a step 5520, and in the case where lock failure was returned from a storage device system of the other party's site, lock failure is set to the return value. In the case of having received lock completion from the step 5520, the process goes again to the step 5500, and the lock process is carried out from the beginning.
Here, more specifically, an environment which is required for the process of the step 5520 will be described by reference to the computer system 6000 of FIGS. 22 and 23. FIG. 22 is a block diagram of this computer system. In this system, a host A 6010 and a host B 6020 of a site 1110 and a site 1120, respectively, establish a cluster environment. It is assumed that this cluster is managing a logical volume as a resource of the cluster. In the site 1110, there is a logical volume A 6040, and in the site 1120, there is a logical volume B 6050. The respective logical volumes are ones in which a copy pair is formed by way of bidirectional remote copy. Both hosts handle these plurality of logical volumes as the same logical volume. The hosts A and B communicate by use of an IP (Internet Protocol) network 6030.
As to the record which was locked later than the lock request time and overlapped with the lock request scope, the step 5650 deletes the record from the lock state holding table 400. After the deletion, records of the lock state holding table 400 are arranged so as to eliminate a vacant record from the top of the records of the lock state holding table 400, and control numbers are assigned again. As to bit values which represent the record scope to be deleted in the bit map table 4010, in case they were the bit value 3, they are changed to the bit value 1, and in case that they were the bit value 2, they are changed to the bit value 0. Next, the steps 5530 and 5540 are carried out in sequence, and lock completion is set to the return value.
[0135]FIGS. 24 and 25 shows processes in the case where the host I/F 1300 and the DKCI/F 1320 have received a lock release request (release command etc. of SCSI). FIG. 24 shows a process in the case where the host I/F 1300 received the lock release request, and FIG. 25 shows a process in the case where the DKCI/F 1320 received the lock release request from another site.
The embodiment 3 accomplishes propagation of reserve information in the case where the bidirectional remote copy of the embodiment 2 is used. The embodiment 4 shows a method of realizing propagation of the reserve information on the occasion of having used the bi-directional remote copy of the embodiment 1.
Here, a process for deleting the write-in content and for establishing such a situation that the write-in was not carried out will be described. Before lock completion is set to the return value, a search is made to determine whether or not there is write data which was written into, the lock object scope of the data consistency table after the lock start time, and when the write-in data exists, a situation is established such that this write-in data was not written, and the record in the data consistency holding table 100 and the cache data on the cache memory 1340 were deleted. By this change, the propagation of reserve information on the occasion of having used the bidirectional remote copy of the embodiment 1 can be realized.
According to this invention, when the copy pair is configured between a plurality of storage device systems, volumes which configure the copy pair can carry out copying in bi-directions. Each superior device can freely write in any volume which forms the copy pair. Also, it becomes possible to propagate the reserve state between the storage device systems under the bidirectional copying.
1. A storage device system in a computer system having a plurality of superior devices and a plurality of storage device systems for receiving write-in data from at least one of the superior devices, comprising:
a physical storage device, responsive to a logical volume which is controlled so that identical data is saved across the plurality of storage device systems, for storing data on the logical volume;
a device for saving the time of reception time that the on which write-in data was received from a superior device;
a communication interface device for transmitting write-in data addressed to the logical volume and a corresponding reception time to another storage device system and for receiving write-in data and corresponding reception time from the storage device system; and
a data consistency holding control device for effecting control to write write-in data which was received from the superior device and write-in data which was received through the communication interface device into the physical storage device after such write-in data has been made to wait in a temporary storage device for more than predetermined time from the reception time corresponding to the write-in data to the logical volume, so that, when write-in data which was received from the superior device and write-in data which was received through the communication interface device are written in an overlapped manner into the same storage location of the physical storage device, they are written in the order of the reception time thereof.
2. The storage device system as set forth claim 1, wherein the storage device system further has a table in which the reception tines corresponding to each write-in data which is waiting in the temporary storage device are arranged in the order from an older one, and a device for searching write-in data to determine for which data said more than predetermined time has passed from the reception time.
3. The storage device system as set forth in claim 1, Wherein the storage device system further has a bit map table for setting a bit value to indicate whether or not each block of the write-in data exists in the temporary storage device , and a device for judging whether or not new write-in data is written in an overlapped manner into the same storage location as the other write-in data with reference to the bit map table.
4. The storage device system as set forth in claim 1, wherein the storage device system further has a device for receiving a request for locking a partial region of the logical volume from the superior device and for locking the partial region, a device for transmitting the locking request which was received through the communication interface device to the other storage device system, a device for receiving the locking request through the communication interface device from the other storage device system and for locking a designated partial region , and a device for rejecting a request for write-in of data from the superior device and the other storage device system to the partial region, except for a case in which it is a request from the superior device in which the partial region was locked.
5. A storage device system in a computer system having a plurality of superior devices and a plurality of storage device-systems for receiving write-in data from at least one of the superior devices, comprising:
a device for saving the time of reception on which write-in data was received from superior device;
a communication interface device for transmitting write-in data addressed to the logical volume and a corresponding reception time to another storage device system and for receiving write-in data and corresponding reception time from the storage device system;
a table in which the reception times corresponding to each of write-in data from the superior device and write-in data from the other storage device system are arranged in order from an older one; and
a data consistency holding control device for effecting control with reference to the table to write write-in data to the logical volume for which more than a predetermined time has passed from the reception time into the physical storage device in the order of the older reception time.
6. The storage device system as set forth in claim 5, wherein the storage device system further has a device for receiving a request for locking a partial region of the logical volume from the superior device and for locking the partial region, a device for transmitting the locking request which was received through the communication interface device to the other storage device system, a device for receiving the locking request through the communication interface device from the other storage device system and for locking a designated partial region designated, and a device for rejecting a request for write-in of data from the superior device and the other storage device system to the partial region, except for a case in which it is a request from the superior device in which the partial region was locked.
US10750859 2003-05-06 2004-01-05 Storage device system having bi-directional copy control function Active 2024-06-10 US7047256B2 (en)
JP2003128163A JP2004334434A5 (en) 2003-05-06
JP2003-128163 2003-05-06
US20040226021A1 true true US20040226021A1 (en) 2004-11-11
US7047256B2 US7047256B2 (en) 2006-05-16
ID=33410453
US10750859 Active 2024-06-10 US7047256B2 (en) 2003-05-06 2004-01-05 Storage device system having bi-directional copy control function
US (1) US7047256B2 (en)
CN100527092C (en) 2005-07-21 2009-08-12 国际商业机器公司 System and method for creating an application-consistent remote copy of data using remote mirroring
US9262344B2 (en) 2013-12-17 2016-02-16 International Business Machines Corporation Local locking in a bi-directional synchronous mirroring environment
US7047256B2 (en) 2006-05-16 grant
JP2004334434A (en) 2004-11-25 application