Computer system with transparent data migration between storage volumes

A computer system includes a transparent data migration facility (TDNff) to accomplish automated movement of data (migration) from one location to another in the system. A data migration program includes a main module to control the start of a migration session when said application programs are using data accessed to and from the source volume, to migrate data from the source volume to the target volume, and to end the migration session whereby the application programs are using data accessed to and from the target volume. The data migration program includes a volume module to control the volumes during the migration session. The data migration program includes a copy module to control the copying of data from the source module to the target module during the migration session. The data migration program includes a monitor module for monitoring I/O transfers to the data volumes during the migration sessions. The computer system may have a plurality of operating systems associated with instances of the data migration program which allows for concurrent data migrations. The plurality of instances of the data migration program may also be controlled in a master slave relationship. A migration session may include a plurality of migration phases such as activation, copy, refresh, quiesce, synchronize, redirect, resume and termination phases.

BACKGROUND OF THE INVENTION 
The present invention relates to the management and maintenance of large 
computer systems and particularly to automated methods and apparatus for 
the movement of data (migration) from location in the system to another. 
In 1983, International Business Machines Corporation of Armonk, N.Y. (IBM) 
described the requirements and capabilities needed in order to efficiently 
manage and maintain the storage in the modern large data center and to 
evolve toward automated storage management IBM's Data Facility/System 
Management Storage (DF/SMS) capabilities. In discussing device support and 
exploitation, IBM identified the requirement for the timely support and 
exploitation of new device technologies and the need for tools to simplify 
the movement of data (migration) to new subsystems. 
A standard set of tools has been provided by IBM and other software 
developers that has allowed data to be automatically copied, archived, and 
restored. Evolutionary improvements have been made to these tools in the 
areas of performance, usability, and in some cases data availability; but 
a problem still exists in that the availability capabilities of these 
facilities have not kept pace with the availability requirements that 
exist in data centers. The storage administrator must be able to support 
the increasing demands of continuous 24 hour by 7day data availability. 
There is an explosive growth in the need to store and have on-demand access 
to greater and greater pools of data. As capacity requirements skyrocket, 
data availability demands increase. These factors coupled with the need to 
control costs dictate that new RAID (Redundant Array of Independent Disks) 
storage technology be implemented. The dilemma faced by data center 
management is that the implementation of new storage technology is 
extremely disruptive and therefore conflicts with the need to maximize 
availability of the data. Therefore, an additional tool is required that 
will allow data to be nondisruptively relocated or migrated within the 
data center. 
Essentially, a data migration facility provides the ability to "relocate" 
data from one device to another device. A logical relationship is defined 
between a device (the source) and another device (the target). The logical 
relationship between a source and target volume provides the framework for 
a migration. The data migration facility controls multiple concurrent 
migrations in a single group that is called a session. A migration is the 
process that causes the data on the source to be copied to the target. 
The characteristics that are critical components of a transparent data 
migration facility include the following: 
The facility is completely transparent to the end user and the application 
program. No application disruption is required in order to make program 
changes and the facility is dynamically and nondisruptively activated. 
The facility provides for full data access during the data migration. The 
data on the source volume is available to the end user for read and write 
access. 
The facility provides for a dynamic and nondisruptive takeover of the 
target volume, when the source and target volumes are synchronized. 
The migration facility must ensure complete data integrity. 
The migration facility should NOT be restricted to any control unit model 
type or device type. All devices in the data center should be able to 
participate in a migration and the facility should support a multiple 
vendor environment. 
The State of the Industry 
Migration facilities that exist today were primarily designed for disaster 
recovery or the facilities were meant to address single volume failures. 
However, these facilities can also be used as data migration tools. These 
facilities differ in implementation and use a combination of host software 
and/or control unit firmware and hardware in order to provide the 
foundation for the data migration capability. 
Local Mirroring 
The IBM 3990 host extended feature IBM Dual Copy and the EMC Symmetrix, 
from EMC Corporation (EMC), mirroring feature are two examples of local 
mirrors. A source volume and a target volume are identified as a mirrored 
paired and at the creation of the mirrored pair, data is transparently 
copied or migrated to the secondary volume. Continuous read and write 
access by applications is allowed during the data migration process and 
all data updates are reflected to the secondary volume. 
In the case of the IBM 3990 host, the mirror is under the complete control 
of the system operator. For example, through the use of system commands a 
mirrored pair can be created. At create time, the data will be copied to a 
secondary device. At the completion of this copy, the operator can then 
disconnect the pair and assign the secondary device to be the primary. 
This is called Transient Dual Copy and is an example of Dual Copy being 
used as a migration facility. 
The function of the EMC mirroring feature is to maximize data availability. 
The EMC subsystem will disconnect the mirror in the event of a failure of 
one of the paired devices. The mirror will be automatically reconstructed 
when the failing device is repaired by EMC field engineers. Unlike Dual 
Copy, the EMC subsystem does not provide an external control mechanism to 
enable the user to selectively initiate and disconnect mirrored pairs. 
Therefore, the EMC mirroring feature can not be used as a migration 
facility. 
Standard mirroring has a major restriction that prevents its universal 
utilization as a transparent data migration tool. The source and target 
volumes must be attached to the same logical control unit and therefore 
data can only be relocated within a single control unit. Although limited, 
this capability is an important tool to the storage administrator. 
Remote Mirroring 
IBM 3990-6 and EMC Symmetrix features support remote mirroring. A remote 
mirror function exists when paired devices can exist on different control 
units and subsystems. The primary objective of this function is to provide 
a disaster recovery capability. However, a remote mirroring function can 
also be used as a data migrator. 
DF/SMS eXtended Remote Copy (XRC) is a host-assisted remote mirror method 
that uses components of DF/SMS and DFP (Data Facility Product). The major 
component is the System Data Mover (SDM). This component is also used for 
Concurrent Copy. An IBM 3990-6 (or compatible) host is required as the 
primary or sending control unit. The secondary or receiving control unit 
can be an IBM 3990-3 or -6 or compatible host. 
Other characteristics of XRC include: 
Host application response time is not impacted because updates are 
reflected on the secondary volume asynchronously. The application does not 
have to wait for the new data to be copied to the secondary volume. The 
SDM reads data from the IBM 3990-6 "sidefile" and records the update on 
log files and then writes to the remote mirror on the recovery control 
unit. 
A common timer is required to insure updates are processed in the correct 
order and therefore target data integrity is guaranteed in a multiple 
system environment. 
No dynamic takeover is supported. Intervention is required in order to 
utilize the secondary copy. 
To invoke XRC as a data migration facility, the following steps are 
required. After identification of the source and target pair of devices, 
an image copy begins and the session is placed in a "duplex pending" 
state. All users of the source volume have total read and write access to 
the data. Source updates are reflected on to the target volume. When the 
copy is complete, the session enters the "duplex" state. The operator must 
query the pair in order to determine this state change. At this time, all 
applications using the source volume must be brought down in order to 
synchronize the source and target devices. Final synchronization is 
determined by operator command (XQUERY). This query displays a timestamp 
that represents the time of last update so that the operator can be 
assured that all updates have been reflected on the target. 
Although XRC does have some of the requirements for transparent migration, 
XRC does not have all of them. 
XRC is transparent to the end user and the application program. No 
application program changes are required and the facility is dynamically 
activated. 
The data on the source volume is available for read and write access during 
the XRC migration. 
XRC causes a disruption because XRC does NOT support a dynamic and 
nondisruptive takeover of the target volume when the source and target 
volumes are synchronized. The impact of this disruption can be expensive. 
All applications with data resident on the source volume must be disabled 
during the takeover process. 
XRC ensures complete data integrity through the use of the journaling data 
sets and a common timer. 
XRC is a relatively "open" facility and therefore supports a multiple 
vendor environment. Any vendor that supports the IBM 3990-6 XRC 
specification can participate as the sending or receiving control unit in 
an XRC session. Any vendor that supports the IBM 3990-3 or basic mode IBM 
3990-6 specification can participate as the receiving control unit in an 
XRC session. 
XRC is complex to use and therefore is operationally expensive and resource 
intensive. 
The IBM 3990-6 host also supports a feature that is called Peer-to-Peer 
Remote Copy (PPRC). PPRC is host independent and therefore differs from 
XRC in several ways. First, there is a direct ESCON (Enterprise Systems 
Connection) fiber link from one IBM 3990-6 host to another IBM 3990-6 
host. With this fiber link connection, the primary IBM 3990 host can 
directly initiate an I/O Input/Output operation to the secondary IBM 3990 
host. Secondly, PPRC operates as a synchronous process which means that 
the MVS (Multiple Virtual Systems) host is not informed of I/O completion 
of a write operation until both the primary and secondary IBM 3990 host 
control units have acknowledged that the write has been processed. 
Although this operation is a cache-to-cache transfer, there is a 
performance impact which represents a major differentiator of PPRC over 
XRC. The service time to the user on write operations for PPRC is 
elongated by the time required to send and acknowledge the I/O to the 
secondary IBM 3990 host. 
The link between the IBM 3990 host controllers utilize standard ESCON fiber 
but does require an IBM proprietary protocol for this cross controller 
communication. This proprietary link restricts the use of PPRC to real IBM 
3990-6 host controllers only and therefore does not support a multiple 
vendor environment. 
As suggested above, PPRC can also be used as a migration facility. PPRC 
requires a series of commands to be issued to initiate and control the 
migration and is therefore resource intensive. IBM has a marketing tool 
called the PPRC Migration Manager that is used to streamline a migration 
process with the use of ISPF (Interactive Structured Program Facility) 
panels and REXX (Restructured Extended Executor) execs. 
A migration using PPRC (Release 1) does not support an automatic takeover 
to the secondary device. In March of 1996, IBM announced an enhancement to 
PPRC called P/DAS, PPRC Dynamic Address Switch, which apparently when 
available eliminates the requirement to bring down the applications in 
order to perform the takeover of the target device. Therefore, P/DAS may 
allow I/O to be dynamically redirected to the target volume when all 
source data has been copied to that device. 
Use of P/DAS is restricted to IBM 3990-6 controllers and is supported only 
in an MVS/ESA (Multiple Virtual Systems/Enterprise Systems Architecture) 
5.1 and DFSMS/MVS 1.2 environment. Therefore the enhancement offered by 
P/DAS is achieved at the cost of prerequisite software. Furthermore, the 
dynamic switch capability is based on the PPRC platform and therefore 
supports only a IBM 3990-6 environment. 
Although PPRC does have some of the requirements for transparent migration, 
PPRC does not have all of them. 
PPRC is transparent to the end user and the application program. No 
application program changes are required and the facility is dynamically 
activated. 
The data on the source volume is available for read and write access during 
a PPRC migration. 
P/DAS apparently supports a dynamic and nondisruptive takeover of the 
target volume when the source and target volumes are synchronized. 
PPRC ensures complete data integrity because a write operation will not be 
signaled complete until the primary and the secondary IBM 3990 control 
units have acknowledged the update request. This methodology will elongate 
the time required to perform update operations. 
PPRC requires a proprietary link between two control units manufactured by 
the same vendor. For example, only IBM 3990-6 control units can 
participate in an IBM PPRC migration. Therefore PPRC does NOT support a 
multiple vendor environment. 
PPRC is complex to use and therefore is operationally expensive and 
resource intensive. 
EMC Corporation's remote mirroring facility is called Symmetrix Remote Data 
Facility (SRDF). The SRDF link is proprietary and therefore can only be 
used to connect dual Symmetrix 5000 subsystems. 
SRDF has two modes of operation. The first is a PPRC-like synchronous mode 
of operation and the second is a "semi-synchronous" mode of operation. The 
semi-synchronous mode is meant to address the performance impact of the 
synchronous process. In the synchronous mode, the host is signaled that 
the operation is complete only when both the primary and the secondary 
controllers have acknowledged a successful I/O operation. In the 
semi-synchronous mode, the host is signaled that the operation is complete 
when the primary controller has successfully completed the I/O operation. 
The secondary controller will be sent the update asynchronously by the 
primary controller. No additional requests will be accepted by the primary 
controller for this volume until the secondary controller has acknowledged 
a successful I/O operation. Therefore in the SRDF semi-synchronous mode, 
there may one outstanding request for each volume pair in the subsystem. 
EMC personnel must be involved in all SRDF activities unless the user has 
installed specialized host software that is supplied by EMC. The 
proprietary nature of SRDF restricts its use as a data migration facility. 
The primary function of SRDF is to provide data protection in the event of 
a disaster. 
Late in 1995, EMC announced a migration capability that is based on the 
SRDF platform. This facility allows a Symmetrix 5000 to directly connect 
to another vendor's subsystem. The objective of the Symmetrix Migration 
Data Service (SMDS) is to ease the implementation of an EMC subsystem and 
is not meant to be a general purpose facility. SMDS has been called the 
"data sucker" because it directly reads data off another control unit. The 
data migration must include all of the volumes on a source subsystem and 
the target is restricted to a Symmetrix 5000. 
An EMC Series 5000 subsystem is configured so that it can emulate the 
address and control unit type and device types of a existing subsystem 
(the source). This source subsystem is then disconnected from the host and 
attached directly to the 5000. The 5000 is then attached to the host 
processor. This setup is disruptive and therefore does cause an 
application outage. 
The migration begins when a background copy of the source data is initiated 
by the 5000 subsystem. Applications are enabled and users have read and 
write access to data. When the target subsystem (the 5000) receives a read 
request from the host, the data is directly returned if it has already 
been migrated. If the requested data has not been migrated, the 5000 will 
immediately retrieve the data from the source device. When the target 
subsystem receives a write request, the update is placed only on the 5000 
and is not reflected onto the source subsystem. This operation means that 
updates will cause the source and target volumes to be out of 
synchronization. This operation is a potential data integrity exposure 
because a catastrophic interruption in the migration process will cause a 
loss of data for any volume in the source subsystem that has been updated. 
Although the Symmetrix Migration Data Service does have some of the 
requirements for transparent migration, SMDS does not have all of them. 
a. SMDS is not transparent to the end user and the application program. 
Although no application program changes are required, the facility cannot 
be nondisruptively activated. All applications are deactivated so that the 
5000 can be installed and attached to the host in place of the source 
subsystem. Specialized software is loaded into the Symmetrix 5000 to allow 
it to emulate the source subsystem and initiate the data migration. This 
disruption can last as long as an hour. 
1. The data on the source volume is available for read and write access 
during a SMDS migration. 
2. SMDS may support a dynamic and nondisruptive takeover of the target 
volume when the source and target volumes are synchronized. At the end of 
the migration, the source subsystem must be disconnected and the migration 
software must be disabled and it is unknown whether this is disruptive and 
an outage is required. 
3. SMDS can link to control units manufactured by other vendors. However, 
the purpose of SMDS is to ease the disruption and simplify the 
installation of an EMC 5000 subsystem. Data can only be migrated to an EMC 
subsystem. Therefore SMDS does NOT support a multiple vendor environment. 
SMDS does NOT ensure complete data integrity. During the migration, data is 
updated on the target subsystem and is not reflected on the source 
subsystem. A catastrophic error during the migration can cause the loss of 
all application updates. 
The State of the Art in a Summary 
The products that are available on the market today do not meet all of the 
data migration requirements. 
The maintenance of continuous data availability is a fundamental mission of 
data centers. In order to support this goal, the migration facility must 
be initiated transparent to all applications and provide a means for the 
nondisruptive takeover of the target device. 
The value of data and information is critical to the operation and 
competitiveness of the enterprise and therefore any exposure to possible 
data loss is unacceptable. 
The control of the costs of the ever expanding storage assets is a large 
component in financing an information technology infrastructure. A 
competitive multiple vendor environment provides a mechanism to support 
effective cost management. Therefore, the migration facility should be 
vendor independent. Accordingly, there is a need for improved data 
migration methods and apparatus which have all the data migration 
requirements. 
SUMMARY OF THE INVENTION 
The present invention is a data migration facility for managing and 
maintaining large computer systems and particularly for automatic movement 
of large amounts of data (migration of data) from one data storage 
location to another data storage location in the computer system. 
The computer system has a plurality of storage volumes for storing data 
used in the computer system, one or more storage control units for 
controlling I/O transfers of data in the computer system from and to the 
storage volumes, one or more application programs for execution in the 
computer system using data accessed from and to the storage volumes, one 
or more operating system programs for execution in the computer system for 
controlling the storage volumes, the storage control units and the 
application programs, and a data migration program for migrating data from 
one of the data volumes designated as a source volume to one of said data 
volumes designated a target volume while the application programs are 
executing using data accessed from and to the storage volumes. 
The data migration program includes a main module to control the start of a 
migration session when the application programs are using data accessed to 
and from the source volume, to migrate data from the source volume to the 
target volume, and to end the migration session whereby the application 
programs are using data accessed to and from the target volume The data 
migration program includes a volume module to control the volumes during 
the migration session. The data migration program includes a copy module 
to control the copying of data from the source module to the target module 
during the migration session. The data migration program includes a 
monitor module for monitoring I/O transfers to the data volumes during the 
migration sessions. 
In one embodiment, the data migration program includes dynamic activation 
and termination, includes non-disruptive automatic swap processing that 
does not require operator intervention, is transparent to applications and 
end-users while providing complete read and write activity to storage 
volumes during the data migration, permits multiple migrations per 
session, permits multiple sessions. Additionally, the installation is 
non-disruptive (a computer program that can execute as a batch process 
rather than requiring the addition of a new hardware sub-system to the 
computer system), requires no IPL (Initial Program Load) of the operating 
system, is vendor independent with any-to-any device migration independent 
of DASD (Direct Access Storage Device) control unit model type or device 
type. The data migration program includes a communication data set 
(COMMDS) located outside the DASD control unit which helps ensure vendor 
independence. 
The data migration facility has complete data integrity at all times, 
provides the ability to introduce new storage subsystems with minimal 
disruption of service (install is disruptive), allows parallel or ESCON 
channel connections to ensure vendor independence and can be implemented 
as computer software only, without need for dependency on hardware 
microcode assist.

DETAILED DESCRIPTION 
Enterprise System--FIG. 1 
The Transparent Data Migration Facilities (TDMF) is resident as an 
application program under the MVS/ESA host operating system of the 
enterprise system 1 of FIG. 1. The Transparent Data Migration Facilities 
(TDMF) is an application program among other application programs 2. The 
TDMF migration provides model independence, operational flexibility, and a 
completely transparent migration. 
In FIG. 1, the enterprise system 1 is a conventional large-scale computer 
system which includes one or more host computer systems and one or more 
operating systems such as the MVS operating system. The operating systems 
control the operation of the host computers and the execution of a number 
of customer applications 2 and a TDMF application. The TDMS application is 
used for migrating data from one location in the enterprise system to 
another location in the enterprise system. The data locations in the 
enterprise system 1 are located on storage volumes 5. Data is transferred 
to and from the storage volumes 5 under control of the storage control 
units 4. The architecture of the FIG. 1 system is well-known as 
represented, for example, by Amdahl Corporation of Sunnyvale, Calif. 
(Amdahl) and IBM mainframe systems. 
For efficient operation of the enterprise system 1, it is necessary at 
times to migrate data from one of the storage volumes 5 to another one of 
the storage volumes 5. The storage volumes 5 between which the migration 
occurs may be located at the same place under control of a single 
operating system or may be located in volumes under the control of 
different operating systems. Also, some of the volumes may be located 
geographically remote such as in different cities, states or countries. 
When located remotely, the remote computers are connected by a high-speed 
data communication line such as a T3 line. 
The TDMF migration is intended for flexibility, for example, so that 
multiple MVS/ESA releases are supported (4.2, 4.3, 5.1, 5.2; OS390 V1.1, 
V1.2, V1.3, and V2.4), so that shared data system environments are 
supported, so that CKD/E (Count Key Data/Extended) compatible 388x and 
399x control units are supported (Read Track CCW (Channel Control Word) is 
required in the disclosed embodiment), so that 3380 and 3390 device 
geometries are supported, so that flexible device pairing options are 
possible (uses device pairs with equal track sizes and numbers of 
cylinders and requires the target volume to be equal to or greater than 
the source volume), so that a single TDMF session can support up to 640 
concurrent migrations, so that a single TDMF session can support 
concurrent migrations with differing control unit and device types, and so 
that optional point-in-time capability is available. 
The TDMF migration is intended to have an ease of use that includes, for 
example, easy installation, a simple parameter driven process, a 
centralized control and monitoring capability, and the ability to have 
integrated online help and documentation. 
The TDMF migration is intended to have a minimal impact on performance 
featuring, for example, a multiple tasking design, efficient user I/O 
scanning, asynchronous copy and refresh processes, minimization of device 
logical quiesce time and integrated performance measurement capability. 
The TDMF migration is intended to have application transparency including, 
for example, dynamic installation and termination, no requirement for 
application changes, continuous read and write access, dynamic enablement 
of MVS intercepts and dynamic and nondisruptive takeover of target 
devices. 
The TDMF migration is intended to have data integrity including, for 
example, continuous heartbeat monitoring, dynamic error detection and 
recovery capability, passive and nondestructive I/O monitoring design, 
optional data compare capability and maintenance of audit trails. 
Multiple Operating System Environment--FIG. 2 
In FIG. 2, the multiple operating systems 3 include the MVS operating 
systems 3-1, 3-2, . . . , 3-M. These operating systems 3 are typically 
running on a plurality of different host computers with one or more 
operating systems per host computer. Each of the operating systems 3-1, 
3-2, . . . , 3-M is associated with a plurality of application programs 
including the application programs 2-1, 2-2, . . . , 2-M, respectively. 
The application programs 2-1, 2-2, . . . , 2-M are conventional 
application programs as would be found in a customer enterprise system. In 
addition to the conventional application programs, the TDMF application 
program 2-T is also present on each of the operating systems 3-1, 3-2, . . 
. , 3-M that are to be involved in a data migration. 
In FIG. 2, each of the operating systems 3-1, 3-2, . . . , 3-M is 
associated with a storage control unit complex 4 including the storage 
control unit complexes 4-1, 4-2, . . . , 4-M, respectively. 
Each of the storage control units 4-1, 4-2, . . . , 4-M is associated with 
one or more volumes. In the example shown, the storage control unit 4-1 is 
associated with the data storage volumes 5-1.sub.1, . . . , 5-1.sub.V1. 
Similarly, the storage control unit 4-2 is associated with the volumes 
5-2.sub.1, 5-2.sub.V2. Finally, the storage control unit 4-M is associated 
with the volumes 5-M.sub.1, . . . , 5-M.sub.VM. 
The operation of the FIG. 2 data storage is conventional in the MVS 
environment. Specifically, any one of the applications 2-1, 2-2, . . . , 
2-M may store or retrieve data from any one of the volumes 5 under control 
of the operating systems 3-1, 3-2, . . . , 3-M through the storage control 
units 4-1, 4-2, . . . , 4-M. 
At any time, any one of the volumes 5 may be designated for replacement or 
otherwise become unavailable so that the data on the volume must be moved 
to one of the other volumes of FIG. 2. For example, the data of a source 
volume X may be migrated to a target volume Y. In FIG. 2, by way of 
example, volume 5-1.sub.1 has been designated as the source volume and 
volume 5-M.sub.1 has been designated as the target volume. The objective 
is to move all the data from the source volume X to the target volume Y 
transparently to the operation of all of the applications 2-1, 2-2, . . . 
, 2-M so that the enterprise system of FIG. 2 continues operating on a 
continuous 24-hour by seven day data availability without significant 
disruption. 
Single MVS Operating System Environment--FIG. 3 
In FIG. 3, the single master MVS operating system 3-1 controls the 
operation of the applications 2-1 on a source volume 5.sub.X. At some 
time, the enterprise system requires that the source volume 5.sub.X be 
taken out of service requiring the data on the source volume 5.sub.X to be 
migrated to a target volume 5.sub.Y. The operating system 3-1 controls 
transfers to and from the source volume 5.sub.X through the control unit 
4.sub.X. Similarly, the MVS operating system 3-1 controls transfers to and 
from the target volume 5.sub.Y through the storage control unit 4.sub.Y. 
The data migration from the source volume 5.sub.X to the target volume 
5.sub.Y is under control of the TDMF application 2-T.sub.1. In order to 
achieve the transparent data migration from the source volume 5.sub.X to 
the target volume 5.sub.Y, an additional data volume 5-1 which is not 
involved in the data migration is available to the MVS operating system 
3-1. During the entire migration from the source volume 5.sub.X to the 
target volume 5.sub.Y, the applications 2-1 continue their operation 
without disruption and while maintaining complete data integrity. 
Multiple Operating System Environment Example--FIG. 4 
In FIG. 4, the MVS operating systems 3-1, 3-2, . . . , 3-M are all part of 
the enterprise system 1 of FIG. 1. Each of the operating systems 3-1, 3-2, 
. . . , 3-M is associated with corresponding application programs 2-1, 
2-2, . . . , 2-M. Each of the applications 2-1, 2-2, . . . , 2-M is 
operative under the control of the MVS operating systems 3-1, 3-2, . . . , 
3-M to access the storage volumes 5 through the storage control units 4-1, 
4-2, . . . , 4-M. 
Specifically, the storage control units 4-1, 4-2, . . . , 4-M control 
accesses to and from the volumes 5-1, 5-2, . . . , 5-M, respectively. 
Specifically, the volumes 5-1 include the volumes 5-1.sub.1, . . . , 
5-1.sub.V1, the volumes 5-2 include the volumes 5-2.sub.1, . . . , 
5-2.sub.V2, . . . , and the volumes 5-M include the volumes 5-M.sub.1, . . 
. , 5-M.sub.VM, respectively. 
At some point in time it becomes desirable in the enterprise system of FIG. 
4 to migrate data from a source volume VOL.sub.X to a target volume 
VOL.sub.Y. In the example of FIG. 4, the source VOL.sub.X is designated as 
5-1.sub.1 controlled through the SCU.sub.X 4-1 and the target volume 
VOL.sub.Y is designated 5-M.sub.1 and is controlled through the SCU.sub.Y 
designated 4-M. The data migration occurs from the source VOL.sub.X to the 
target VOL.sub.Y without interruption of the customer applications 2-1, 
2-2, . . . , 2-M. 
In order to control the data migration from the source VOL.sub.X to the 
target VOL.sub.Y, the TDMF application 2-T operates in a distributive 
manner across each of the operating systems 3-1, 3-2, . . . , 3-M. 
Specifically, the operation of the TDMF migration application is in the 
form of a master/slave implementation. Each of the operating systems 3-1, 
3-2, . . . , 3-M includes a corresponding instance of the TDMF 
application, namely, the TDMF application 2-T. One of the applications is 
designated as a master TDMF application and the other of the applications 
is designated as a slave TDMF application. In the example of FIG. 4, the 
TDMF application associated with the MVS 3-1 is designated as the master 
TDMF application 2-T.sub.mas. Each of the other operating systems 3-2, . . 
. , 3-M is associated with a slave application 2-T.sub.s11, . . . , 
2-T.sub.s1M, respectively. 
The data migration in the FIG. 3 and FIG. 4 systems is carried out with a 
TDMF application 2-T (with a separate instance of that application for 
each MVS operating system) without necessity of any modified hardware 
connections. 
TDMF Migration Stages--FIG. 5 
In FIG. 5, the relationship between the migration stages for migrating data 
between a source volume and a target volume is shown. Specifically, the 
migration commences with an INITIALIZATION/ACTIVATION phase 10, followed 
by a COPY phase 11, followed by a REFRESH phase 12, followed by a QUIESCE 
phase 13, followed by a SYNCHRONIZE phase 14, followed by a REDIRECT phase 
15, followed by a RESUME phase 16, and ending in a TERMINATION phase 17. 
If during the INITIALIZATION/ACTIVATION phase 10, an error occurs, the 
error is detected by the ERROR module 20 which passes the flow to the 
TERMINATION stage 17. If an error is detected during the COPY phase 11, 
the ERROR module 21 passes the flow to the TERMINATION stage 17. If an 
error is detected during the REFRESH phase 12, the ERROR module 22 passes 
the flow to the TERMINATION phase 17. 
If an error occurs during the QUIESCE phase 13 or the SYNCHRONIZE phase 14, 
the ERROR modules 23 and 24, respectively, pass the flow to the RESUME 
phase 16. If an error occurs during the REDIRECT phase 15, the ERROR 
module 25 passes the flow to the BACKOUT module 26 which then returns the 
flow to the RESUME phase 16. 
The migration phases of FIG. 5 are active in each of the MVS operating 
systems 3 and the MASTER TDMF application and the MASTER MVS operating 
system insure that one phase is completed for all operating systems, both 
master and all slaves, before the next phase is entered. 
MASTER/SLAVE System--FIG. 6 
The MASTER/SLAVE system operation is under control of the TDMF application 
2-T. That application, and each instance thereof in an MVS operating 
system includes the modules of FIG. 6. Specifically, the TDMFMAIN module 
is the main task which calls other ones of the modules for controlling the 
phasewise execution of the migration as set forth in FIG. 5. The TDMFMAIN 
module starts the TDMF session of FIG. 5 on the master and slave systems, 
opens all necessary files, reads and validates control cards and validates 
all parameters and volumes to be involved in the migration session. 
The TDMFVOL module controls the migration process of any and all volumes 
being migrated within the TDMF session of FIG. 5. 
The TDMFICOM generates channel programs to implement I/O operations being 
requested to the COMMUNICATIONS DATA SET (COMMDS) via parameters passed by 
the caller. The caller may request that the system initialize the COMMDS 
or to selectively read or write caller specified TDMF control blocks as 
required by the controller. The TDMFICOM module is called by the TDMFMAIN 
module only. 
The TDMFIVOL module generates channel programs to implement I/O operations 
being requested to volumes being migrated to the IDMF session of FIG. 5 
via parameters passed by the caller. 
The TDMFSIO module issues a request to the MVS INPUT/OUTPUT SUPERVISOR 
(IOS) component to perform the I/O operation represented by the 
INPUT/OUTPUT SUPERVISOR control BLOCK (IOSB) in conjunction with its 
SERVICE REQUEST BLOCK (SRB) as requested by the caller. The IDMFSIO module 
can be called from the TDMFICOM module and the TDMFIVOL module. Upon 
completion of the I/O operation requested, control is returned to the 
calling module. 
COPY Sub-task--FIG. 7 
In FIG. 7, the COPY Sub-task functions with one per volume migration for 
the master only. The COPY Sub-task includes the TDMFCOPY module which 
provides the functions required by the TDMF COPY Sub-task. The TDMFCOPY 
Sub-task is attached as a Sub-task by the module TDMFVOL by means of an 
ATTACHX macro during its processing. The TDMFCOPY module implements three 
Sub-task phases, namely, the COPY sub-phase, the REFRESH sub-phase, and 
the SYNCHRONIZATION sub-phase. 
The TDMFIVOL module may also be called by the TDMFCOPY module to generate 
channel programs to implement I/O operation being requested to volumes 
being migrated to the TDMF session of FIG. 5 via parameters passed by the 
caller. 
The TDMFSIO module when called by the TDMFIVOL module issues a request to 
the MVS INPUT/OUTPUT SUPERVISOR (IOS) component to perform the I/O 
operation represented by the INPUT/OUTPUT SUPERVISOR CONTROL BLOCK (IOSB) 
in conjunction with its SERVICE REQUEST BLOCK (SRB) as requested by the 
caller. Upon completion of the I/O operation requested, control is 
returned to the TDMFIVOL module. 
I/O Monitoring--FIG. 8 
In FIG. 8, a block diagram of the monitoring of I/O operations during a 
migration session of FIG. 5 is shown. The FIG. 8 modules are conventional 
with the addition of the TDMFIMON module. The purpose of the TDMFIMON 
module is to monitor all customer I/O operations to the source and target 
volumes during the life of an active migration session of FIG. 5. The 
TDMFIMON module insures that the primary design objective of insuring data 
integrity of the target volume by insuring that any update activity by 
customer I/O operation that changes the data located on the source volume 
will be reflected on the target volume. The TDMFIMON module only monitors 
volumes involved in a migration session of FIG. 5 and any I/O operations 
to volumes not involved in the TDMF migration session are not impacted in 
any fashion. 
TDMF Module Overall Architecture--FIG. 9 
In FIG. 9, the overall architecture of the computer software for 
implementing the TDMF migration of FIG. 5 is shown. The TDMFMAIN module 
functions to control systems initialization and system determination and 
calls the TDMFVOL module, the TDMFICOM module, and the TDMFIVOL module. 
The TDMFVOL module is responsible for volume initialization with the 
TDMFIVOL module, volume activation with the TDMFAVOL module, volume 
quiesce with the TDMFQVOL module, volume resume with the TDMFRVOL module 
and volume termination with the TDMFTVOL module. The TDMFVOL module calls 
or returns to the TDMFMAIN module, the TDMFICOM module, the TDMFIVOL 
module, and the TDMFCOPY module. 
The TDMFCOPY module is called by and returns to the TDMFVOL module and the 
TDMFIVOL module. 
The TDMFICOM module calls and is returned to the TDMFSIO module. 
The TDMFIVOL module calls and is returned to the TDMFSIO module. 
Details of the various modules of FIG. 6, FIG. 7, FIG. 8 and FIG. 9 for 
performing the migration in the phases of FIG. 5 are described in detail 
in the following LISTING 1 near the end of the specification before the 
claims. 
Source and Target Volumes Before TDMF Migration--FIG. 10 
In FIG. 10, a data migration is to occur between the volume 5-1.sub.1 as 
the source through the storage control unit SCU.sub.X designated 4-1 to 
the target volume 5-M.sub.1 through the storage control unit SCU.sub.Y 
designated 4-M. The SOURCE volume has device address A00, serial number 
SRC001 and is in the ON-LINE status before the TDMF migration. The target 
volume has the device address FC0, serial number TGT001 and is in the 
ON-LINE status before volume migration. Before the volume migration 
begins, all I/O operations are being directed to the source volume 
5-1.sub.1. 
Source and Target Volumes After All TDMF Migration--FIG. 11 
In FIG. 11, the source volume 5-1.sub.1 has a device address of A00, a 
serial number of TGT001 and an OFF-LINE status. The target volume 
5-M.sub.1 has a device address of FC0, a serial number of SRC001, and an 
ON-LINE status. 
TDMF Migration Operation 
TDMF is initiated as an MVS batch job. TDMF's job control language 
identifies the system type, session control parameters, and the 
communications data set (COMMDS). The session control parameters define 
the paired source (VOL.sub.X) and target (VOL.sub.Y) volumes which make up 
the migrations under the control of this session. The control parameters 
also define the master system and all slave systems. The COMMDS set is the 
mechanism that is used to enable all systems to communicate and monitor 
the health of the migrations in progress. The COMMDS set is also used as 
an event log and message repository. 
All systems that have access to the source volume must have TDMF activated 
within them. This requirement is critical in order to guarantee data 
integrity. One system is designated as the master (M=MASTER) and all 
other systems are designated as slaves (M=SLAVE). 
The responsibilities of the master system include: 
Initialization of the master TDMF environment and the COMMDS. 
Validation of the initialization of all slave systems. 
The control of all migrations defined in this TDMF session. 
The monitoring of source volume user I/O activity in order to detect 
updates. 
The copying of data from the source to the target volume. 
Collection of update information from all systems. 
The refresh and synchronization of the target volume due to detected update 
activity. 
The monitoring of the internal health of the master environment and the 
health of all slave systems. 
The responsibilities of the slave systems include: 
Initialization of the slave TDMF environment. 
Establishment of communication to the master through the COMMDS. 
The processing and acknowledgment of all migration requests as received 
from the master. 
The monitoring of source volume user I/O activity in order to detect 
updates. 
Transmission of update information to the master through the COMMDS. 
The monitoring of the internal health of this slave environment and the 
health of all other systems. 
The master system initiates and controls all migrations. A migration is 
broken into the major phases as illustrated in FIG. 5. The master 
initiates each phase and all slave systems must acknowledge in order to 
proceed. If any system is unable to acknowledge or an error is detected, 
the systems will cleanup and terminate the migration. To ensure data 
integrity, TDMF is designed with a passive monitoring capability so that 
all user I/O operations to the source volume will complete as instructed 
by the application. This completion means that if any unexpected event 
occurs, the data on the source volume is whole. TDMF will merely cleanup 
and terminate the migration. 
Standard cleanup and recovery is active for all phases with the exception 
of the REDIRECT phase. During this REDIRECT phase, it may be necessary to 
take additional recovery actions. This specialized recovery is called the 
BACKOUT phase. Backout is required due to error detection in a multiple 
system environment. In a multiple system migration, the systems may be at 
differing stages within the REDIRECT phase at the point the error is 
detected. Some of the systems may have completed redirect and others may 
have not completed redirect processing. Backout recovery dynamically 
determines the state of each system in order to take the appropriate 
recovery action. However, even in the REDIRECT and BACKOUT phases, the 
data on the source volume is whole and user data will not be lost. 
Referring again to FIG. 5, a TDMF session begins with an INITIALIZATION 
phase, immediately followed by an ACTIVATION phase. During this phase, all 
systems confirm the validity of the source and target volumes and enable 
user I/O activity monitoring. When all systems acknowledge successful 
completion of this process, the master will begin the COPY phase and 
initiate a sub-task (COPY Sub-task) to copy data from the source to the 
target volume. This COPY Sub-task is a background activity and will copy 
data asynchronously from the source volume to the target volume. There is 
an independent COPY sub-task for each active migration. 
I/O monitoring provided by the TDMFIMON module of FIG. 8 provides the 
ability to detect updates to the source volume. A system that detects a 
source volume update will note this update in a refresh notification 
record and send this information to the master system. 
When a COPY sub-task has completed one pass of the source volume, the 
REFRESH phase is initiated by the master. During this phase, the updates 
that have been made to the source volume are reflected onto the target 
volume. The REFRESH phase is divided into multiple cycles. A refresh cycle 
is a pass through the entire source volume processing all updates. TDMF 
measures the time required to process the refresh operations in a single 
refresh cycle. The refresh cycles will continue until the time for a 
refresh cycle is reduced that allows the Synchronization Goal to be 
achieved to a threshold that is based on the collected performance data. 
When the threshold is reached, this event signals the master to be ready 
to enter the Quiesce and Synchronization phases. If there are multiple 
REFRESH phases during a migration, it is due to the inability of TDMF to 
meet the SYNCHRONIZATION goal. This is usually because of high write 
activity on the Source volume. If there are no issues with this, then 
there is no reason to change the SYNCHRONIZATION goal parameter. 
Prior to the Synchronization phase, a QUIESCE is issued to the Source 
volume. In the instance of a multi-system migration, the Master issues a 
request to the Slave(s) to Quiesce all I/O to the Source volume (from the 
slave side). At this time the final group of detected updates are 
collected and applied to the Target volume (SYNCHRONIZATION). At the end 
of Synchronization, the Master starts the volume REDIRECT (swap) phase. 
The Target volume has now become the new Source volume. When all systems 
have verified the Redirect, the Master initiates the RESUME phase so that 
user I/O can continue to the new source volume. The elapsed time between 
the last QUIESCE phase and the RESUME phase is approximately four (4) 
seconds plus the ACTUAL SYNCHRONIZATION time (which should always be less 
than the specified synchronization goal). 
The Synchronization Goal default is five (5) seconds. Synchronization will 
not occur unless the calculated synchronization time is less than the 
goal. If the synchronization goal is increased, then the time the user I/O 
is queued (quiesced) is greater. If the value 999 is used, this equates to 
synchronize as soon as possible; it does not matter how long it takes. 
This can be a significant amount of time depending on the write activity 
of the source volume. Therefore, use discretion when changing this value. 
Application I/O operations during the Copy and Refresh phases are impacted 
no more than if a backup utility is backing up the source volume at the 
same time as the application is executing. The Synchronization goal 
parameter may be specified for each volume migration, allowing the 
customer to specify the amount of time (in seconds) that he will allow the 
Synchronization phase to execute. This is the maximum amount of time. 
Performance Impact by Phase 
The Master and Slave(s) wake up for processing based upon a variable which 
is the minimum number of seconds based upon any migration volume's current 
phase or stage. The phases of a volume migration and their associated time 
intervals are: 
______________________________________ 
Phase Time Interval 
______________________________________ 
Copy 30 seconds 
Refresh pass #1 15 seconds 
Refresh pass #2 10 seconds 
Refresh pass #3 . . . n 
5 seconds 
Quiesce 1 second 
Synchronize 1 second 
(Compare) 1 second 
Swap/Point-in-Time 1 second 
Resume 1 second 
Termination/Completion 
15 seconds 
______________________________________ 
This allows TDMF to be responsive with a minimum of CPU overhead. 
The CPU overhead associated with running IDMF is less than 3 percent on 
average for the Master system. This is dependent upon the number of 
storage volumes within a session and write activity against those source 
volumes. A Slave system's CPU overhead will be almost non-measurable. 
For example, if the Master job takes 44 minutes, 22 seconds (2662 seconds 
total) to migrate 16 volumes, and the TCB time is 63.5 seconds, and the 
SRB time is 2.92 seconds, then the CPU overhead is equal to 2.49 percent 
((63.5+2.92)/2662) for that session. 
When a multi-volume migration is running in a TDMF session, not all volumes 
are going to be in the same phase at the same time. This is because 
different volumes may have different activity against them. The number of 
channels available to each Control Unit (CU) will also be a factor in 
this. Therefore, it is entirely possible to have a 4 volume migration 
running with volume 1 in the copy phase, volume 2 in a 4th refresh phase, 
volume 3 completed, and volume 4 is in the Synchronization phase. 
Placement of the Master 
The Master should be placed on the system that has the most updates or on 
the system where the owning application is executing. If multiple TDMF 
Master sessions are being run on multiple operating systems, then the MVS 
system(s) must have a global facility like GRS or MIM. This is to prevent 
inadvertent usage of the same volumes in a multi-session environment. If a 
GRS type facility is not available in the complex, then all Master 
sessions must run on the same operating system. 
Placement of the Communications Dataset 
The Communications Dataset (COMMDS) should be placed on a volume with low 
activity and the volume must not be involved in any migration pairing. 
Sample Performance Statistics 
In the following tables, time is designated in hours (H), minutes (MM), and 
seconds (SS.ss) and in the form H:MM:SS.ss; for example, 1:07:20.36 is 1 
hour, 7 minutes and 20.36 seconds. 
______________________________________ 
Description 
H:MM:SS.ss 
Description H:MM:SS.ss 
% 
______________________________________ 
3380-D No load situation with Compare option on: 
Single Volume 
Copy Time 11:31.02 TCB time 17.93 
Compare Time 
8:11.39 SRB time 0.33 
Total Time 
20:11.70 CPU overhead 1.5 
3380-K No load situation: 
Single Volume 
Copy Time TCB time 3.98 
Compare Time SRB time 0.26 
Total Time 
34:50.65 CPU overhead 0.2 
3390-3 with a heavy write load: 
Single volume 
Copy Time 24:11.05 TCB time 9.04 
Refresh Time 
16:35.23 SRB time 0.72 
Sync Time 6:29.29 
Total Time 
48:25.97 CPU overhead 0.3 
3390-3 No load situation: 
8 volume copy 
Copy Time TCB time 30.91 
Compare Time SRB time 1.67 
Total Time 
20:24.87 CPU overhead 2.6 
3390-3 No load situation: 
16 volume copy 
Copy Time TCB time 1:04.13 
Compare Time SRB time 3.20 
Total Time 
33:52.52 CPU overhead 3.3 
3390-3 No load situation: 
32 volume copy 
Copy Time TCB time 2:13.00 
Compare Time SRB time 5.89 
Total Time 
1:07:20.36 
CPU overhead 3.4 
3390-9 No load situation: 
Single volume 
Copy Time 1:02:36.67 
TCB time 12.54 
Compare Time SRB time 0.70 
Total Time 
1:03:23.55 
CPU overhead 0.3 
______________________________________ 
Master System Responsibilities 
Initializing the master TDMF environment and the COMMDS. 
Starting and controlling the migrations for all TDMF systems. 
Monitoring source volume user I/O activity to detect updates. 
Copying data from the source to the target volume. 
Processing detected updates from all systems. 
Performing refresh operations to the target volume to reflect update 
activity. 
Checking the internal health of the master environment and the health of 
all slave systems. 
Slave System Responsibilities 
Initializing the slave TDMF environment and establishing communications to 
the master, using the COMMDS. 
Acknowledging and processing migration requests from the master. 
Monitoring source volume user I/O activity and detecting updates. 
Notifying the master of update activity through the COMMDS. 
Checking the internal health of the slave environment and the health of all 
systems. 
At this time, the master issues the quiesce request to all systems so that 
activity to the source volume will be inhibited. When the quiesce request 
is received by a slave system, this is the indication for the slave system 
to enter the QUIESCE phase and to quiesce and send to the master the final 
group of detected updates so that the master may perform the 
synchronization process or optionally S restart an additional REFRESH 
phase. 
When all systems have successfully acknowledged the quiesce request, 
synchronization will begin in the SYNCHRONIZATION phase. The device 
quiesce is transparent to all applications. An application can issue I/O 
requests; however, they will not be executed and will be queued until the 
RESUME phase. The purpose of the collection of performance information 
during the REFRESH phase is to minimize the device quiesce time. At the 
conclusion of the SYNCHRONIZATION phase, the master will initiate the 
REDIRECT phase. 
The purpose of the REDIRECT phase is to cause I/O activity to be redirected 
to the target volume so that it can become the primary device. The master 
will request that all systems perform and confirm a successful redirect 
before the redirect operation is performed by the Master system. When all 
systems have acknowledged this activity, the master will then initiate the 
RESUME phase request so that any queued user I/O will begin execution. 
Subsequent I/O operations will then be directed to the target device. 
After all systems have processed the resume request, I/O monitoring will be 
disabled and the migration will terminate in the TERMINATE phase. 
The TDMF facility simplifies the relocation of user-specified data from 
existing hardware to new storage subsystems, for example, without loss of 
access to data, without the imposition of unfamiliar operator procedures, 
and without dependence upon vendor firmware or hardware release levels. 
TDMF is transparent to the end user and the application program. No 
application program changes are required and the facility is dynamically 
activated. 
TDMF provides for full data access during the data migration. The data on 
the source volume is available to the end user for read and write access. 
TDMF supports a dynamic and nondisruptive takeover of the target volume 
when the source and target volumes are synchronized. All applications with 
data resident on the source volume may remain enabled during the takeover 
process. 
TDMF ensures complete data in integrity through the use of passive I/O 
monitoring and background copy operations. 
TDMF is an "open" facility and therefore supports a multiple vendor 
environment. Any vendor that supports the IBM 3880 or IBM 3990 ECKD 
specification can participate as the sending or receiving control unit in 
a TDMF migration. (Read Track CCW support is required). 
TDMF operates in both a single CPU and/or multiple CPU environment with 
shared DASD. 
With TDMF, the implementation of new storage subsystem technologies can be 
accomplished in a nondisruptive manner. 
With TDMF, the value of existing storage equipment is protected because all 
devices can participate in a migration session. 
With TDMF, a competitive multiple vendor environment is enhanced because 
TDMF is designed to be independent of any specific storage subsystem. 
##SPC1##