DYNAMICALLY CHANGING THE ARCHITECTURE OF A DATASET WHILE ALLOWING CONCURRENT USER ACCESS TO DATA IN THE DATASET

An embodiment includes initiating a first migration of data rows in a source dataset in a source storage device to a target dataset in a target storage device, wherein a block size defined for the target dataset is different than a block size defined for the source dataset. The embodiment also includes, during the first migration, receiving a user request for access to a first data row in the source dataset, determining that the first data row was migrated to a first target block in the target dataset, loading the first target block from the target dataset into a first buffer in memory, and responding to the user request using the first data row in the first target block. In specific embodiments a device type that defines the source storage device is different than a device type that defines the target storage device.

BACKGROUND

The present disclosure relates in general to the field of data storage, and more specifically, to dynamically changing the architecture of a dataset while allowing concurrent user access to data in the dataset.

Mass storage devices (MSDs) are used to store large quantities of data. A wide variety of entities utilize MSDs to enable continuous or near-continuous access to the data. Retailers, government agencies and services, educational institutions, transportation services, and health care organizations are among a few entities that may provide ‘always on’ access to their data by customers, employees, students, or other authorized users.

A database is one example of a data structure used to store large quantities of data as an organized collection of information. Typically, databases have a logical structure such that a user accessing the data in the database sees logical data columns arranged in logical data rows. A Database Administrator (DBA) typically uses current technology to architect a database for a given entity. While the initial architecture may provide resources and expansion capabilities, technology advances may render the initial architecture comparatively inefficient and expensive. To exploit new data storage technology, however, a change in the architecture is often needed. For some entities, reconstructing the architecture and migrating old datasets to the newly constructed datasets requires significant downtime in which the database is ‘off-line’ and unavailable to users. In many scenarios, this downtime may not be acceptable.

BRIEF SUMMARY

According to one aspect of the present disclosure, a first migration of data rows in a source dataset in a source storage device to a target dataset in a target storage device is initiated. A block size defined for the target dataset can be different than a block size defined for the source dataset. Buffers in memory are available to handle both the source and target block size. During the first migration, a user request for access to a first data row in the source dataset can be received. A determination can be made that the first data row was migrated to a first target block in the target dataset. The first target block can be loaded from the target dataset into a first buffer in memory. A response to the user request can be made using the first data row in the first target block loaded into the first buffer.

DETAILED DESCRIPTION

Referring now toFIG. 1, a simplified block diagram is shown illustrating an example communication system100for dynamically changing the architecture of a dataset while allowing concurrent user access to data in the dataset according to at least one embodiment. In communication system100, a network110(e.g., a wide area network such as the Internet) facilitates communication between network user terminals120and a network server130. Network server130may be configured to communicate with and manage data storage devices140A,140B,140C, and150, such as direct-access storage devices (DASDs). Network user terminals120can enable users to interface with network server130and to consume data contained in storage devices (e.g.,140A-140C,150). A user terminal160may be used to enable an authorized user, such as a Database Administrator (DBA), to communicate with and issue commands to network server130to access the storage devices. In other embodiments, user terminal160could be directly connected to network server130or could be remotely connected to network server130over the Internet, for example. Also, although storage devices140A-140C are shown as separate storage devices communicating with network server130via local network115, it should be apparent that one or more of these storage devices may be combined in any suitable arrangement and that any of the storages devices140A-140B and150may be connected to network server130directly or via some other network (e.g., wide area network, etc.).

In at least one embodiment, network server130is configured to dynamically change the architecture of an existing dataset of a storage device (e.g.,140A-140C) while allowing concurrent user access (e.g., retrieving, reading, modifying, adding, deleting, etc.) of data in that dataset. The architecture of an existing (source) dataset can be changed by allocating a new (target) dataset on a separate storage device (e.g.,150) that offers a desired architecture configuration, and then migrating the data from the source dataset to the newly allocated target dataset.

For purposes of illustrating certain example techniques of communication system100for dynamically changing the architecture of a dataset while allowing concurrent user access to data of the dataset, it is important to understand the activities that may be occurring in a network environment that includes data storage devices configured with data structures capable of hosting large quantities of data and providing online user access to the data. The following foundational information may be viewed as a basis from which the present disclosure may be properly explained.

Data structures are used by storage devices (e.g., MSDs, DASDs) to store massive amounts of data across virtually every sector of society including, but not limited to, social media, business, retail, health, education, and government. A database is one type of data structure and generally refers to an organized collection of data. Although the concepts presented herein are applicable to any type of data structures used in storage devices, most of the world's data is stored in a data structure commonly referred to as a database. Therefore, although the discussion herein may reference databases for ease of illustration, it should be understood that the concepts are also applicable to other types of data structures.

Databases can have a logical structure that an end user can view online, such as logical data columns arranged in logical data rows. These logical data columns are stored in a logical data table. A database can contain any number of data tables, and a data table can be stored in a dataset of a storage device. A dataset is the physical storage of a storage device and is typically a long string of data representing data bytes. Data rows and logical data columns are configured in data tables to enable data to be retrieved and presented in a user-friendly format.

Generally, large database environments are created using the dataset architecture that exists at the time of creation. As time passes, new architectures may be developed that offer more efficiency, speed, and storage than the old architectures. In order to change a block size and/or a device type used for a database, a Database Administrator (DBA) (or other authorized individual) performs several actions. First, user processing to the database data tables is stopped. Second, the database is closed to all processing (e.g., user accesses, utility processes, etc.). Third, the database datasets are backed up to an external device (e.g., data is copied to a tape or other storage device). Fourth, old datasets may be deleted. Fifth, the datasets are reallocated with the new architecture (e.g., new block sizes, new device types, etc.). Finally, the reallocated datasets are initialized and loaded with data from the backup. This process can take hours, days, or even weeks depending on the size of the datasets. During that time, users, utility processes, and batch processes are all prevented from accessing the data.

In past decades, entities seeking to convert their old database architectures to new database architectures typically had certain windows of opportunity when their databases would go offline (e.g., for periodic maintenance, etc.) and would be inaccessible to users. As the interconnected world has evolved, however, many applications no longer have a scheduled offline period. Rather, many consumers and other users expect 24/7 access to online data needed to conduct business, purchase goods, manage finances, access services (e.g., transportation, etc.), etc. Although datasets architected for older direct access storage device (DASD) models may need to be updated to current DASD architectures to exploit improved features of the newer architecture, often the user data in the datasets of the old architecture cannot be taken offline.

In one example, consumers may expect 24-hour access to a retailer's online application so that goods (e.g., shoes, clothing, electronics, cosmetics, etc.) can be purchased whenever the consumer desires. In another example, some interconnected systems around the world require availability to certain types of data across time-zones. For example, a country's customs/border control branch may require an online vetting application to be available at all times to allow transportation services (e.g., airlines, railroads, water transport, etc.) to receive clearance for travelers into the country.

In one specific example, an entity may have datasets defined as an older DASD architecture (e.g., IBM 3380) that are in-use and being emulated to run on current DASD technology. Due to the emulation, the datasets provide limited capabilities and reduced performance. For example, an IBM 3380 DASD, which was first available in the1980s, is a device type characterized by a design specification of 47,476 bytes per track and 15 tracks per cylinder. The average seek time was 16 milliseconds. Newer DASD architecture such as the IBM 3390 is a device type characterized by a design specification of 56,664 bytes per track, 15 tracks per cylinder, and an average seek time of 9.5 milliseconds. Although many IBM 3380 devices have been replaced by modern DASD devices, during the conversion to the new hardware, the dataset definitions were often left unchanged to ensure compatibility with existing database processing. The new DASD devices may be capable of emulating the older mainframe architectures (e.g., IBM 3380), and so the amount and format of data is often defined using the specifications of the older architecture. Thus, capacity and capabilities of the new DASD hardware is limited due to the emulation of the older DASD architecture.

In another specific example, an entity may have datasets defined with older architectures that were defined for use in earlier processing complexes where data transfer rates of the DASD architecture were slower and users needed to limit block sizes to get the best input-output (I/O) throughput. Data, such as logical data rows, is stored in physical data blocks. These physical data blocks can range in size depending on the platform and the DASD hardware. For example, on the mainframe block sizes can be up to 32K bytes and are defined per user application. In older hardware devices the data transfer rate to retrieve a 16K block of data was typically more that the data transfer rate to move a 4K block of data. Consequently, smaller block sizes (e.g., 4K bytes) were often chosen when defining datasets for user applications using older architecture. Also, the actual transferred data block was stored in memory (also referred to herein as “data buffer,” “buffer” or “buffer memory”), and in older mainframe systems, the amount of memory was often limited. Database administrators (DBAs) needed to limit how much memory was used to store the retrieved data blocks. For most database applications, having four small blocks (4K) in buffer memory provided better performance than one large 16K block in a buffer memory.

Over time, significant changes have occurred both to the DASD storage devices as well as available memory in systems. Data transfer rates have grown exponentially, allowing much larger block sizes to perform at the same speed, while providing more data per I/O operation. These changes have greatly reduced the concern over data transfer rates. Moreover, the addition of (64-bit) memory has significantly increased the available memory for storing data in buffer memory. Thus, many database applications currently running on old architecture using a 4K block size, are likely to experience enhanced performance by increasing dataset block sizes to exploit more recent architecture (e.g., IBM 3390 DASD) implementations such as 16K or 28K block sizes.

In yet another example, there can be significant wasted DASD space and buffer memory when a poor block size is selected and implemented. This may occur when a DBA (or other individual who designs the database) does not have an adequate understanding of database buffering concepts. In a database, database blocks are stored in memory in data buffers. Data buffers are allocated in pools. A data buffer pool is chosen depending on the data block size that is retrieved. Accordingly, a buffer pool should be chosen that is as close to the data block size as possible without the data block size exceeding the buffer pool size. In some scenarios, however, a non-database practice of defining dataset block size as a multiple of data row size is sometimes used. By way of example, the following data block size selections are the result of defining the block sizes as 10- or 20-multiples of the data row size, which can yield significant wasted DASD space and buffer memory:

Thus, several scenarios can result in old dataset architectures being used even though newer technology offers more efficiency, space, and processing speed. Consequently, in some cases, databases rely on decades-old technology, resulting in higher costs, wasted resources, and unnecessary time spent waiting for processing to complete because they cannot afford for the data to be inaccessible to the end users.

A communication system, such as communication system100for dynamically changing the architecture of an existing dataset, as outlined inFIGS. 1 and 2, can resolve these issues and others. This system enables the underlying architecture of datasets to be re-architected to another (e.g., newer, improved) architecture without interruption to users who are accessing the database tables that reside on those datasets. For example, a database administrator (DBA) may determine that one or more database datasets are defined in a non-optimal or even unsupported dataset architecture. The DBA determines that processing for the data tables on this dataset could be improved by re-architecting the dataset to better fit current hardware capabilities. The DBA can define a new (target) dataset on a new storage device with the preferred dataset architecture. In other implementations, a target dataset can be automatically defined based on default or pre-defined architecture specifications. When ready, the DBA (or an automatic process), can trigger a background migration process where each data row is migrated from the old dataset to the new dataset without interrupting the end user access on the data tables.

More specifically, a DBA (or automatic process) can allocate a target dataset and define the preferred architecture, such as block size and device type. For example, an existing dataset defined on an IBM 3380 with a 4K block size may be re-architected to a target dataset defined on an IBM 3390 with a 28K block size. Once the target dataset is allocated, the architecture change process can be triggered when desired. In one embodiment, the architecture change process may first establish that the target dataset is sufficiently sized and suitably architected to hold the data tables being migrated from the source dataset. The architecture change process can establish an input-output (I/O) gateway around the source and target datasets to maintain consistency of reference for all data rows that are migrated from the source dataset to the target dataset. The I/O gateway begins migrating logical data rows from a data block in the source dataset to a data block in the target dataset. The data rows are migrated independently of data blocks, as the new architecture may change the number of data rows per data block. In at least one embodiment, the data rows are migrated in native sequence from the source dataset to the target dataset. Transactional logging may be provided for all data rows to enable a fully restartable and recoverable process in the event of an unintentional processing failure (e.g., power outage, processor failure, system failure, and other abnormal terminations, etc.).

One or more embodiments manage concurrent access to data in the datasets as data rows are migrated from the source dataset to the target dataset. End user processing is performed by logical data row and does not require a data row to be housed in a particular dataset. Thus, the I/O gateway manages access to the data rows by end users, where a particular data row may be accessed from either the source dataset or the target dataset depending upon whether it has been migrated at the time of the user request. The I/O gateway can also manage data row accesses by other database utility processes. This is achieved by ensuring that the data row migration is integrated with these other utility processes. For example, a utility process that attempts to run concurrently with I/O gateway may be blocked until a particular data row migration is complete. However, for at least some utility processes, the utility process is automatically integrated with the I/O gateway, which manages accesses to the source and target datasets by the utility process and allows for successful completion. In some cases, where the requested utility process is blocked because it conflicts with the migration process, an alternative utility process may be provided that performs the utility function integrated with the I/O gateway.

In one or more embodiments, the architecture change process can be completed by renaming the target dataset to the original name of the source dataset. The source dataset may be deleted or renamed. It should also be noted that multiple datasets can be re-architected at the same time. An I/O gateway can be created for each dataset being re-architected.

Embodiments of an architecture change process can offer several advantages. For example, embodiments described herein enable DBAs to quickly migrate data tables from one architecture to another, different architecture with enhanced capabilities and features. Moving to a different architecture can improve performance, remove restrictive requirements for older architectures (e.g., older DASD architectures), and reduce costs of maintaining the environment. The particular embodiments described herein for dynamically changing the architecture of a dataset enable a DBA to implement critical business required architecture changes without interrupting the business. Thus, users may continue to access needed data from a dataset being re-architected without any downtime.

Turning toFIG. 1, a brief description of the infrastructure of communication system100is now provided. Elements ofFIG. 1may be coupled to one another through one or more interfaces employing any suitable connections (wired or wireless), which provide viable pathways for network communications. Additionally, any one or more of these elements ofFIG. 1may be combined or removed from the architecture based on particular configuration needs.

Generally, communication system100can be implemented in any type or topology of networks. Within the context of the disclosure, networks such as networks110and115represents a series of points or nodes of interconnected communication paths for receiving and transmitting packets of information that propagate through communication system100. These networks offer communicative interfaces between sources, destinations, and intermediate nodes, and may include any local area network (LAN), virtual local area network (VLAN), wide area network (WAN) such as the Internet, wireless local area network (WLAN), metropolitan area network (MAN), Intranet, Extranet, virtual private network (VPN), and/or any other appropriate architecture or system that facilitates communications in a network environment or any suitable combination thereof. Additionally, radio signal communications over a cellular network may also be provided in communication system100. Suitable interfaces and infrastructure may be provided to enable communication with the cellular network.

In general, “servers,” “clients,” “computing devices,” “storage devices,” “network elements,” “database systems,” “network servers,” “user devices,” “user terminals,” “systems,” etc. (e.g.,120,130,140A-140C,150,160, etc.) in example communication system100, can include electronic computing devices operable to receive, transmit, process, store, or manage data and information associated with communication system100. As used in this document, the term “computer,” “processor,” “processor device,” “processing device,” or “I/O controller” is intended to encompass any suitable processing device. For example, elements shown as single devices within communication system100may be implemented using a plurality of computing devices and processors, such as server pools including multiple server computers. Further, any, all, or some of the computing devices may be adapted to execute any operating system, including IBM zOS, Linux, UNIX, Microsoft Windows, Apple OS, Apple iOS, Google Android, Windows Server, etc., as well as virtual machines adapted to virtualize execution of a particular operating system, including customized and proprietary operating systems.

Further, servers, clients, computing devices, storage devices, network elements, database systems, network servers, user devices, user terminals, systems, etc. (e.g.,120,130,140A-140C,150,160, etc.) can each include one or more processors, computer-readable memory, and one or more interfaces, among other features and hardware. Servers can include any suitable software component, manager, controller, or module, or computing device(s) capable of hosting and/or serving software applications and services, including distributed, enterprise, or cloud-based software applications, data, and services. For instance, in some implementations, a network server130, storage devices140A-140C and150, or other subsystem of communication system100can be at least partially (or wholly) cloud-implemented, web-based, or distributed to remotely host, serve, or otherwise manage data, software services and applications interfacing, coordinating with, dependent on, or used by other services, devices, and users (e.g., via network user terminal, other user terminals, etc.) in communication system100. In some instances, a server, system, subsystem, or computing device can be implemented as some combination of devices that can be hosted on a common computing system, server, server pool, or cloud computing environment and share computing resources, including shared memory, processors, and interfaces.

WhileFIG. 1is described as containing or being associated with a plurality of elements, not all elements illustrated within communication system100ofFIG. 1may be utilized in each alternative implementation of the present disclosure. Additionally, one or more of the elements described in connection with the examples ofFIG. 1may be located external to communication system100, while in other instances, certain elements may be included within or as a portion of one or more of the other described elements, as well as other elements not described in the illustrated implementation. Further, certain elements illustrated inFIG. 1may be combined with other components, as well as used for alternative or additional purposes in addition to those purposes described herein.

FIG. 2is a simplified block diagram that illustrates additional possible details that may be associated with certain components of communication system100. Specifically, a network server is one possible example of network server130, a source storage device240is one possible example of storage devices140A,140B, and/or140C, and target storage device250is one possible example of storage device150. The elements ofFIG. 2are representative of possible components during an architecture change process in which data rows in a source dataset242of source storage device240are being migrated to a target dataset252of target storage device250.

Network server230may include a database management system (DBMS)231, which creates and manages databases, including providing batch utilities, tools, and programs. A database manager232can create a database processing region (also referred to as a multi-user facility (MUF)) where user processing and most utility processes flow. During an architecture change process, database manager232can create an input/output (I/O) gateway234. In at least one embodiment, I/O gateway234may be created temporarily in software and removed from DBMS231once the architecture is changed. I/O gateway234, when executed, can create a background process236, which migrates data rows from a source dataset (e.g.,242) to a target dataset (e.g.,252), while I/O gateway234handles concurrent user processing to access the data rows being migrated. I/O gateway234can also create a log file233to store information related to each data row migration. Thus, log file233can provide information that enables restartability and recoverability if the architecture change process experiences a failure (e.g., power outage, system failure, etc.). Log file233may be implemented internal or external to DBMS231, based on particular implementations and needs. InFIG. 2, log file233is shown as internal to DBMS231in storage.

Network server230may also include hardware including, but not limited to, an I/O controller235, a processor237, and a memory element239. The I/O controller235may facilitate communication to both source storage devices (e.g.,240) and target storage devices (e.g.,250), or in other implementations, multiple I/O controllers may be used. In some implementations, a user interface270may also be coupled to network server230. User interface could be any suitable hardware (e.g., display screen, input devices such as a keyboard, mouse, trackball, touch, etc.) and corresponding software to enable an authorized user to communicate directly with network server230. For example, in some scenarios, a DBA may configure target datasets and initiate the architecture change process using user interface270.

At any given time, memory element239may contain data blocks238-1through238-X, which are loaded into memory based on user access requests received for data rows contained in those blocks. In at least one embodiment, memory element239may contain buffer memory and data blocks238-1through238-X may be loaded into buffers in the memory. Multiple users may access, via user terminals, data rows in data blocks that are loaded into memory element239. Database manager232can also be configured to manage concurrency control for users accessing data rows simultaneously, so that adverse effects are prevented if multiple users try to modify resources other users are actively using.

Source storage device240and target storage device250are representative of different types of physical storage devices capable of storing data in data structures (e.g., databases) that enable multiple users, processes, and utilities to access and, in some cases, modify the stored data. Each storage device240and250includes a respective dataset242and252, which is the physical storage of data in the storage device. Prior to an architecture change process, source dataset242may store data in data blocks245-1through245-N. In at least some embodiments, during the architecture change process, a control block247may be added to unused space in source dataset242to hold information related to the data migration. Target dataset252may be allocated with defined blocks, such as data blocks255-1through255-M, prior to an architecture change process being initiated for source dataset242. During the architecture change process, a control block257may be added to unused space in target dataset252to hold information related to the data migration. The background migration process can cause data blocks255-1through255-M to be filled with data rows from source dataset242.

In at least one scenario, source dataset242may be defined with a different architecture than target dataset252. For example, source dataset242may be defined on a less preferred architecture, such as an older data storage device using a small block size (e.g., IBM 3380 with a 4K block size). Target dataset252may be defined on a different architecture (e.g., a preferred architecture). In one example, target dataset252may be defined on newer technology that enables a larger block size to be utilized (e.g., IBM 3390 with 18K or 28K block size). Consequently, when the migration of source dataset242to target dataset252is complete, the number of data blocks (M) in target dataset252may be different than the number of data blocks (N) in source dataset242if their block sizes are different. For example, if source dataset242is defined on an IBM 3380 with a 4K block size and target dataset252is defined on an IBM 3390 with a 28K block size, then target dataset252will likely have fewer blocks than source dataset242(i.e., M<N).

Turning toFIGS. 3A-3K, block diagrams illustrate an example scenario of a database environment and an architecture change process applied to a dataset within the environment. A communication system300includes network user terminals320, a DBA user terminal360, a database manager332with a data processing region337, a memory339, and storage devices340A-340C of a database environment managed by database manager332. The database environment can include multiple database datasets (e.g.,342A,342B,342C). The datasets contain logical data tables and the datasets may be stored in multiple architectures (e.g., different device types, different block sizes). User data rows are stored as logical data table(s) in the datasets.

With reference toFIG. 3A, an example database environment scenario is shown. Data storage devices340A-340C contain respective datasets342A-342C. Data tables310are stored in dataset342A, and the other data tables (not shown) are stored in the other datasets342B and342C of the database. Each data table312A-312C stored in dataset342A may contain different information (e.g., customer information, order information, inventory information, etc.). For example, data table312A may be a customer data table, data table312B may be an order data table, and data table312C may be an inventory data table. Each dataset has a unique file name and, in this example scenario, dataset342A has a file name of “PROD.ACCOU NTS.ABC100.”

Also in this example scenario, datasets342A-342C are shown with different architectures. Dataset342A is defined on a first mass storage device type (MSD-1) with a block size of 4K bytes. Dataset342B is defined on a second mass storage device type (MSD-2) with a block size of 4K bytes. Dataset342C is defined on another MSD-1with a block size of 8K bytes.

Data processing region337receives flows of user requests from users via network user terminals320and from database administrator(s) via DBA user terminal360. Data processing region337can also receive database access requests from utility and other non-end user processes. In operation, multiple users (e.g., tens, hundreds, thousands, etc.) can access the database concurrently via network user terminals320.FIG. 3Ashows concurrent user requests (e.g., for data access or modification) for data contained in each of the datasets342A-342C. An example user request will now be described with reference to dataset342A. For illustrative purposes, the description is based on a user request for a data row in the customer data table312A, which is stored in dataset342A.

At302a, a user requests, via a network user terminal320, access to a customer data row in customer data table312A. Data processing region337receives the user request. At302b, data processing region337determines the location of a data block that contains the requested data row. In this example, data processing region337determines the location of the data block, which is in dataset342A of storage device340A.

At302c, data processing region337retrieves into memory339the identified data block from the appropriate dataset holding the customer data table. The data block is retrieved into memory as block338-1, with requested data row335. In one example, block338-1may be stored in buffer memory of memory339. At302d, the requested data row335is extracted and returned to the network user terminal that submitted the user request at302a.

User accesses to other data tables (e.g.,312B,312C) may occur at least partially concurrently to the user access of customer data table312A. In addition, other user accesses to customer data table312A may also occur at least partially concurrently with the user access shown and described inFIG. 3A. These other user requests may be directed to data rows in other data blocks or in the same data block338-1. Database manager332manages the concurrency of concurrent user requests for access and/or modifications to data contained in the same data table. In addition, as shown inFIG. 3A, user accesses to other data tables in different datasets (e.g.,342B,342C) may also occur at least partially concurrently (or not concurrently) to the user accesses of data tables310. It should be apparent that in at least some systems, continuous concurrent access by two or more users is possible.

FIGS. 3B-3Killustrate various stages during an architecture change process, which will now be explained. While normal database processing is occurring (e.g. multiple concurrent user requests), a determination can be made that the architecture of a dataset is to be changed (e.g., reconstructed, re-architected, redesigned, etc.). In this example scenario, dataset342A is the source dataset that is to be re-architected to a target dataset. Currently, source dataset342A is defined on a first mass storage device type (e.g., MSD-1) with a 4K byte block size.

InFIG. 3B, the DBA can access database manager332via DBA user terminal360to create a new target dataset with the desired architecture. In this example, the DBA allocates a target dataset352on a target storage device350and defines its architecture as a second mass storage device type (MSD-2) with a 27K byte block size. In addition, target dataset352is given a unique file name. In this example, the unique file name is the source dataset file name with an extra qualifier: “PROD.ACCOUNTS.ABC100.NEW.” In other embodiments, the target dataset may be allocated and defined dynamically based on default, pre-configured, or algorithmically configured architecture parameters. It should be apparent that while specific block sizes such as 18K, 28K and 27K are mentioned herein, the block size of the target dataset is selected to provide an optimal result for the desired architecture. Thus, any suitable block size may be selected for a target dataset based on particular implementations and/or needs.

A database pre-processing utility application may also be executed to prepare the target dataset for data migration from the source dataset. For example, pre-processing may include verifying the presence of source storage device340A, target storage device350, source dataset342A, target dataset352, the readiness of target dataset352for the data migration, etc. A utility application or the DBA may also ensure that enough buffer memory is available in memory339for the new target dataset352.

InFIG. 3C, the DBA may issue a command, via the DBA user terminal360, to cause database manager332to begin architecture change processing. Architecture change processing includes migrating data in the source dataset to the target dataset by data rows. Upon receiving the command, database manager332may begin periodically outputting status messages to a display (e.g., DBA user terminal360, display connected to a network server hosting database manager332, another remote or local display device, etc.) and/or to a log file of status messages indicating the status of the architecture change process. Initially, database manager332may output a start message to indicate the processing has started.

In response to the command to start processing, database manager332creates can create an input/output (I/O) gateway334in memory to isolate processing for source dataset342A while it is being re-architected. The I/O gateway334may be a dynamically generated, temporary process that runs in a separate processing region to handle the data migration of the source dataset to the target dataset and the concurrent user requests (and utility process requests) for access to data in source dataset342A during the data migration. Database manager332forwards user requests and utility process requests for access to source dataset342A to I/O gateway334. The location of a requested data row in dataset342A at any given time during the architecture change process depends on whether the data row has been migrated. I/O gateway334keeps track of where each data row is located during the migration and handles user requests (and utility process requests) accordingly.

Once the I/O gateway is created, as shown inFIG. 3D, the gateway can issue a command to open both the source dataset342A and the target dataset352and can establish connections (e.g.,353a,353b,343a,343b) to both datasets. I/O gateway334can access both source dataset342A and target dataset352and knows which data rows are on which dataset at any given time during the data migration performed during the architecture change process. Thus, I/O gateway334maintains exclusive control over the datasets during the data migration. Database manager332may also output a status message indicating the I/O gateway is built and the datasets are open.

Once the datasets are open and connections are established, as shown inFIG. 3E, I/O gateway334creates a background process336to migrate data rows from source dataset342A to target dataset352. The background process can be invisible to users who may continue to access data in the datasets. Data rows can be migrated by either copying each data row from source dataset342A to target dataset352, or by moving each data row from source dataset342A to target dataset352. If data rows are copied, a copy of the data rows remains in the source dataset upon completion of the migration. If data rows are moved, then they are deleted from the source dataset. During the migration, database manager332may output a status message periodically indicating the number of data rows that have been successfully migrated.

In at least one embodiment, background process336migrates data rows sequentially, rather than as a block. In one example, background process336migrates the data rows in native sequence. Native sequence is intended to mean a preferred order for the data rows. Often, the preferred order is selected (e.g., by a DBA or designer of the database) based on the most likely processing sequence of the data rows. For example, if requests are typically made in a particular order, then the performance of the database may be increased if data is stored in the dataset in the same order as the most common user requests and/or batch utility requests. It should be noted that, when migrating in native sequence, data rows may be selected across multiple blocks of storage in source dataset342A. For example, the first 4K block may contain the first data row to migrate, the second 4K block may contain the second data row to migrate, the fifth 4K block may contain the third data row to migrate, and so on. In other embodiments, background process336may simply migrate the data rows based on their current order in source dataset342A or in any other desired order based on particular implementations and needs.

As shown inFIG. 3F, I/O gateway334may also create a log file333during the migration. I/O gateway334can store information in the log file that is related to each successful data row migration. Log file333may be used to restart the architecture change process and the migration at the point of the last logged data row migration after a failure (e.g., power outage, system failure, etc.) that causes the architecture change process to cease running.

In many scenarios, it is desirable to perform the migration as quickly as possible. Therefore, in at least one embodiment, as background process336performs the data migration, any available processing power may be used to migrate the data. However, some processing power is also allocated to end user requests for data in source dataset342A. The user requests are directed through I/O gateway334so that the users can access any desired data row from source dataset342A during the architecture change process of source dataset342A.

FIG. 3Gillustrates a scenario that may occur during the architecture change process. In some cases, certain utility applications and other non-end user processes may be initiated during the migration. For example, a DBA may decide that a database backup utility process cannot wait until the architecture change process is finished. For example, a DBA may send a request, via DBA user terminal360, to database manager332to run database backup utility application380. Because a physical backup process of a dataset cannot be run during its data migration, the database backup utility application380is prevented from executing.

In at least one embodiment, upon receiving a request to run database backup utility application380, database manager332may send a response to DBA user terminal360denying the request and offering to run an alternative backup utility application within I/O gateway334during the data migration. If the DBA agrees to the alternative backup application, database manager332can instruct I/O gateway334to run the alternative database backup utility application. The alternative database backup utility application is integrated with the I/O gateway334such that data rows are provided to the integrated application from the I/O gateway, which has access to both datasets342A and352. Thus, the I/O gateway controls and coordinates the backup process with the data migration so that an accurate backup can be performed. The integrated application can store the data rows received from the I/O gateway in another data storage device, such as dataset backup383. Database manager332may provide status messages related to the alternative backup utility process.

FIG. 3Hillustrates the database environment once the data migration is complete. When every data row of source dataset342A has been migrated to target dataset352, then background process336ends. I/O gateway334may stop storing information in log file333. Once the background process ends, however, I/O gateway334remains connected to source dataset342A and target dataset352and continues to manage user requests for the data tables that are now stored entirely on target dataset352. Database manager332may output a status message stating the number of data rows that have been successfully migrated and indicating that the data migration is complete.

After the migration is complete, I/O gateway334can be disconnected from source dataset342A, as shown inFIG. 3I. In at least one embodiment, background process336may cause the I/O gateway334to disconnect from source dataset342A after the migration is complete, but before the background process ends. The original name associated with source dataset342A is released (either by deleting or renaming the source dataset) so that the target dataset can be renamed to the original name (i.e., PROD.ACCOUNTS.ABC100). In at least one embodiment, data processing region337can delete or rename source dataset342A. Database manager332can output a status message indicating that the old dataset (i.e., source dataset342A) has been deleted or renamed.

FIG. 3Jshows the additional cleanup that is performed once the data migration is finished. First, the background process may rename target dataset352to the original name of source dataset342A, which is now deleted or renamed. In this example scenario, target dataset352is renamed to PROD.ACCOUNTS.ABC100. Next, I/O gateway334can be stopped or removed and normal processing through data processing region337can resume. In at least one embodiment, database manager332may remove I/O gateway334. In one example, log file333can be deleted, either by I/O gateway before it is removed, or by database manager332. Database manager332can output a status indicating that the target dataset name has been changed to the original source dataset name, and the process is complete.

FIG. 3Killustrates the database environment after the architecture change process is complete. Target dataset352contains data tables310and has the original file name of the source dataset that was migrated to the target dataset. Target dataset352is accessed by data processing region337when a user request (or utility application request) is received for a data row contained in target dataset352. Data processing region337locates the requested data row355and retrieves a block358-1that contains requested data row355. The block is loaded into a buffer in memory339and data row355can be provided to the appropriate user terminal.

Turning toFIGS. 4A-11, various flowcharts illustrate example techniques related to one or more embodiments of a communication system, such as communication system100, for dynamically changing the architecture of a source dataset (e.g.,242) of a source storage device (e.g.,240) while allowing concurrent user access to the dataset. The preferred architecture (e.g., storage device type, block size, etc.) is defined for a target dataset (e.g.,252) in a target storage device (e.g.,250), and the data of a source dataset (e.g.,242) is migrated to the target dataset, without interrupting the user access (or utility application access) to data rows of data tables stored in the source dataset. In at least one embodiment, one or more sets of operations correspond to activities ofFIGS. 4A-11. A network server, such as network server230, or a portion thereof, may utilize the one or more sets of operations. In an embodiment, at least some operations of the flows ofFIGS. 4A-11may be performed by database manager232and at least some operations may be performed by I/O gateway234and background process236. Network server230may comprise means such as processor237, I/O controller235, and memory element239for performing the operations.

FIGS. 4A-4Bare simplified flowcharts400A and400B, respectively, illustrating some operations that may be performed by database manager232to prepare physical storage devices and processes to re-architect a source dataset, such as source dataset242. At402, database manager232receives a command to allocate a target dataset on a target storage device and to define the selected architecture for the target dataset. For example, the selected architecture could be a newer storage device type (e.g., IBM 3390) with a larger block size (e.g., 18K, 27K, etc.) than the block size currently defined for the source dataset.

At404, a target dataset is allocated on the target storage device, such as target dataset252on target storage device250, and the selected architecture is defined for the target dataset.

At406, pre-processing tasks may be performed before the architecture change process begins. For example, pre-processing tasks may include verifying the presence of the target storage device and target dataset, initializing the target dataset to the appropriate database internal format, verifying the presence of the source storage device and source dataset, and the overall readiness of the source and target datasets for the migration.

InFIG. 4B, at least some of the operations shown may be performed by database manager232. At410, a command may be received (e.g., from a DBA via a DBA user terminal) to begin the architecture change process to re-architect source dataset242to the preferred target dataset252.

At412, the database manager can output start messages to indicate the architecture change process has been initiated. Messages may be sent to a display and/or a log file of messages during the architecture change process. The display may be, for example, a display device of a DBA user terminal or any other display device configured to receive messages from database manager232.

At414, database manager232can build or create an input/output (I/O) gateway, such as I/O gateway234to run in a separate processing region. I/O gateway can open source dataset242and target dataset252and establish connections to the datasets.

I/O gateway234is created to re-architect the source dataset, but not other datasets. Thus, I/O gateway234handles only user requests and possibly utility application requests for data rows stored in the gateway's associated source dataset. In at least one embodiment, I/O gateway is temporary and is removed when the architecture change process completes. In other embodiments, I/O gateway234may be stopped, stored, and retrieved for later use as an I/O gateway for another source dataset.

When I/O gateway234establishes connections to source dataset242and target dataset252, database manager232can output a status message at416indicating that the I/O gateway is ready, and the architecture change process can begin.

At418, database manager232can provide user requests for data in source dataset242to I/O gateway234and can receive and appropriately forward responses to those requests from the I/O gateway234, until the architecture change process is complete. An example of this processing is discussed in further detail with reference toFIG. 6. Database manager232can also handle any database utility process requests, including batch process requests and other non-end user process requests. These scenarios are discussed in further detail with reference toFIG. 10.

At420, once the architecture change process is complete, the database manager232can remove the I/O gateway, establish a connection to the target dataset including opening the target dataset, and return to normal processing. Normal processing includes receiving and responding to user requests for data rows in the target dataset by accessing the target dataset, locating the appropriate data rows, and loading the appropriate blocks on the target dataset into memory. Normal processing also includes allowing utility processes that request access to the target dataset to run. At422, database manager232can output a status message indicating that the architecture change process is complete.

FIGS. 5A-5Bare simplified flowcharts500A and500B, respectively, illustrating at least some of the activities that may be performed by I/O gateway234during the re-architecture of source dataset242. In at least one embodiment, flowchart500A begins after database manager232has created the I/O gateway to re-architect source dataset242.

At501, I/O gateway234opens source dataset242and target dataset252. I/O gateway234also establishes connections to the source and target datasets.

At502, I/O gateway234can initiate a background process to migrate data rows from source dataset242to target dataset252.

At504, unused space is identified in both the source dataset242and the target dataset252. A control block can be built on both the identified unused space in the source dataset and the identified unused space in the target dataset. The control blocks can be used to store a last migrated key during the migration of data rows from the source dataset to the target dataset. In one embodiment, each row has a unique key value, and the migration of the data rows is performed sequentially based on the unique key values. In one example, the key values can correspond to the physical order in which the data rows are stored in the source dataset.

In another embodiment, the key values can correspond to the native sequence of the data rows. Over time, data rows in a dataset may become out-of-native-sequence due to modifications to the data rows (e.g., insertions, deletions). In order to migrate the data rows of source dataset242in native sequence, the rows may be selected for migration based on each row's native key value. Thus, the migration can effectively reorder the data rows into a native key sequence in target dataset252.

At506, the first block in which data rows are to be stored in target dataset252is identified. At508, the first data row to migrate from the source dataset is selected. The data row may be selected based on the last migrated key. Because no data rows have been migrated yet, the value of the last migrated key may be null or zero in some examples. Therefore, in this example, the first data row could be selected based on its associated key value being the lowest key value in a sequence of all the key values associated with data rows in source dataset242. As previously noted, the key values may be based on any desired order of the data rows depending on particular needs and implementations. For example, the key values may be based on a native sequence of the data rows or a stored sequence of the data rows.

At510, the selected data row is migrated from source dataset242to the identified block in target dataset252. At512, the key value associated with the migrated data row is stored in the control blocks in both the source dataset and the target dataset as the last migrated key value. The last migrated key value stored in the control blocks provides a reference to enable identification of which data rows have been migrated at any given time during the migration. For example, the last migrated key value stored in the control blocks can indicate that the data row associated with the last migrated key value, and any other data rows associated with key values that are less than the last migrated key value, have been successfully migrated.

At518, a message indicating the status of migration may be produced. Status messages may include the number of rows successfully migrated in one example. These messages may not be produced after every data row migration, but rather, may be produced periodically (e.g., 10,000 data rows migrated, 20,000 data rows migrated, etc.). In one embodiment, this message or information can be provided to database manager232, which can then output the message to an appropriate display or log file of status messages.

At520, inFIG. 5B, I/O gateway234can create a log file of data row migrations, if not already created. At522, information can be stored in the log file that is related to the migration of the selected data row. Relevant information is saved for each successful data row migration to enable restartability and recoverability if the network server (or components within the network server) should experience some failure that crashes or otherwise interrupts the architecture change process. Information may include, but is not necessarily limited to, the key value of the selected data row, the location of the selected data row in the target dataset, and/or the location of the selected data row in the source dataset.

At524, a determination is made as to whether there are more data rows in source dataset242to be migrated. If there are more data rows to be migrated, then at526, a determination is made as to whether the identified block in target dataset252is filled. If the identified target data block is filled, then at528, a next block in the target dataset is identified to store with data rows from the source dataset.

If the next block in the target dataset is identified at528, or if the currently-identified block in the target dataset is determined not to be filled at526, then the flow loops back to508, where the next data row is selected to migrate from source dataset242to target dataset252. The last migrated key value is retrieved from the control block of the source dataset or the target dataset. In this case, the last migrated key value from the control block is the key value associated with the first selected data row. The next data row to select is identified by determining the next sequential key value, after the last migrated key value, of a data row in the source dataset.

Flow then continues this loop as previously described until eventually, at524, it is determined that the source dataset contains no more rows to be migrated. I/O gateway234may disconnect from source dataset242but retain its connection with target dataset252. At530, a message is produced indicating the status of the migrated data rows. In at least one embodiment, information indicating the total amount of data rows that have been migrated may be provided to database manager232. Database manager232may then output the status message to the appropriate display and/or log file of status messages.

Operations at532-542are related to enabling database manager to resume normal operations with target dataset252replacing source dataset242in the database environment. In some cases, one or more operations at532-542may be performed by I/O gateway234, database manager232, background process236, and/or other background processes initiated for these activities.

At532, the original file name of source dataset242is released by either deleting or renaming the source dataset. At534, a message may be produced indicating the status of the source dataset (e.g., deleted or renamed). In at least one embodiment, information indicating the status of the source dataset may be provided to database manager232. Database manager232may then output the status message to the appropriate display and/or log file of status messages.

At536, target dataset252is renamed to the original file name of the source dataset. At538, a message may be produced indicating the status of the target dataset (e.g., renamed to original name of source dataset). In at least one embodiment, information indicating the status of the target dataset may be provided to database manager232. Database manager232may then output the status message to the appropriate display and/or log file of status messages.

At540, the log file of data row migrations may be deleted by I/O gateway234. In other embodiments, the log file of data row migrations may be deleted after the I/O gateway has stopped running (e.g., by database manager232), or may be saved for any desired length of time.

At542, I/O gateway234is disconnected from the target dataset and the I/O gateway stops handling user requests or utility process requests. As indicated inFIG. 4Bat420, processing returns to normal for accessing the data tables, which are now stored on target dataset252. The database manager can establish a connection to the target dataset and user requests to the target dataset can be handled by data processing region of the database manager.

FIG. 6is a simplified flowchart600illustrating at least some of the activities that may be performed by database manager232while the I/O gateway234is running. It should be noted that the I/O gateway234created to re-architect source dataset242may be one of multiple I/O gateways created for multiple datasets, respectively, of the database associated with network server230.

At602, a user request for access to a data row in a dataset is received. At604, a determination can be made as to whether the dataset is associated with an I/O gateway. A dataset is associated with an I/O gateway if the dataset is being re-architected by the I/O gateway.

If the requested dataset is not associated with an I/O gateway, then at606, the user request is processed normally. For example, the user request may be handled through a data processing region created by database manager232, as shown inFIGS. 3A-3K.

If the requested dataset is associated with an I/O gateway, then at608, the database manager identifies the I/O gateway that is associated with the dataset. At610, database manager232provides the user request to the identified I/O gateway. Thus, database manager232receives user requests and funnels them to the appropriate I/O gateway (if any) to allow the I/O gateway to manage user requests during the migration of data from source dataset242to target dataset252. This process may continue as long as at least one I/O gateway is still running in the database environment.

FIGS. 7A-7Bare simplified flowcharts700A and700B, respectively, illustrating some operations that may be performed by an I/O gateway (e.g.,234) during the architecture change process. Flowcharts700A and700B relate to handling user requests for access (read) to data rows in a source dataset (e.g.,242) that are being migrated from the source dataset to a target dataset (e.g.,252) concurrently with the user requests. Access (or read) of a data row typically makes up the majority of user requests.

At702, I/O gateway234receives a user request for access to a data row in source dataset242during the migration of its data rows to target dataset252. At704, a determination is made as to whether the requested data row is currently selected to be migrated. In some possibly rare scenarios, a user request for access to a data row may happen simultaneously with a background migration process (e.g.,236) selecting the same data row for migration. In this scenario, the user request may be briefly halted until the migration of the requested data row is complete. Accordingly, if the requested data row is currently selected for migrating, then at706, I/O gateway234temporarily blocks the user request. At708, a determination may be made that the data row migration is complete. At710, once the data row migration is complete, the user request is processed by the I/O gateway.

At712, a determination is made as to whether the requested data row has been migrated to the target dataset. In one example, the last migrated key value and the key value of the requested data row can be used to determine whether the requested data row has already been migrated. The last migrated key value can be obtained from a control block of either the source dataset or the target dataset. In one example implementation, if the key value of the requested data row is less than or equal to the last migrated key value, then the requested data row has already been migrated. Conversely, if the key value of the requested data row is greater than the last migrated key value, then the requested data row has not been migrated.

If the requested data row has not been migrated to the target dataset, then at714, a determination is made as to whether the requested data row is currently in a buffer in memory. The requested data row may be in a buffer in memory with its source block if the data row was previously requested by a user request. The source block is the block of data in the source dataset that contains the data row. For example, if the dataset architecture of the source dataset is defined as 4K byte blocks, then a 4K byte block of data containing the requested data row may be stored in buffer memory if access to the data row was previously requested by a user.

In at least one embodiment, a source block flag (or any other suitable indicator) may be set for each block of the source dataset that is loaded into memory. In this example, at714, the determination of whether the requested data row is already in memory can be made by determining whether a source block flag is set for the source block that contains the requested data row. If the source block flag is set, then the source block is in memory and therefore, the requested data row is in memory.

If the requested data row is not already loaded in buffer memory, then at716, a block of data that contains the requested data row is located in the source dataset, retrieved by I/O gateway234, and loaded into a particular area of memory used by I/O gateway. In addition, a source block flag associated with the source block may be set to indicate that the particular source block has been loaded into memory in response to a user request.

Once the source block containing the requested data row is loaded into memory, or if the source block containing the requested data row was already loaded in memory, at718, the requested data row from the source block in memory is provided to a user terminal associated with the user request for access to the data row.

With reference again to712, if the requested data row has already been migrated to target dataset252, then flow passes to720ofFIG. 7B. At720, a determination is made as to whether the requested data row is already in a buffer in memory with the target block that contains the requested data row. The requested data row may be in a buffer in memory with a target block if the requested data row (or any other data row in the target block) was previously requested by a user request after the requested data row was migrated. For example, if the dataset architecture of the target dataset is defined as 27K byte blocks, then a 27K byte block of data containing the requested data row may be stored in buffer memory if access to the requested data row (or any other data row in this 27K byte block) was previously requested by a user after the requested data row was migrated.

In at least one embodiment, a target block flag (or any other suitable indicator) may be set for each block of the target dataset that is loaded into memory. In this example, at720, the determination of whether the requested data row is already in memory can be made by determining whether a target block flag is set for the target block that contains the requested data row. If the target block flag is set, then the target block is in memory and therefore, the requested data row is in memory.

Even if the requested data row has not been previously requested, the requested data row may be loaded in memory if the target block containing the requested data row is “active.” A target block is “active” if the target block is currently receiving and storing data rows being migrated. If a target block containing a requested data row is active, then the target block may not be filled to capacity and may still have additional space to receive data rows migrating from the source dataset. For example, the active target block may be partially filled (e.g., 20 data rows of 40 possible data rows are stored in the target block). If the I/O gateway receives a user request for access to a data row that has already been migrated to the target dataset and stored in this active target block, which is still in memory, then the user request is processed using this active target block in buffer memory that is already in place.

If the target data block that contains the requested data row is not currently loaded in buffer memory, as determined at720, then at724, the target data block containing the requested data row can be located and retrieved from target dataset252and loaded into buffer memory. In addition, a target block flag may be set to indicate that the particular target data block has been loaded into memory in response to a user request.

Once the target block that contains the requested data row is loaded in buffer memory, then flow can proceed to718inFIG. 7A. At718, the requested data row from the target block in memory is provided to a user terminal associated with the user request for access to the data row.

FIG. 8is a simplified flowchart800illustrating some operations that may be performed by an I/O gateway (e.g.,234) during the architecture change process. Flowchart800relates to handling user requests to modify data rows in a source dataset (e.g.,242) that are being migrated from the source dataset to a target dataset (e.g.,252) concurrently with the user requests.

At802, I/O gateway234receives a user request to modify a data row in source dataset242. At804, a determination is made as to whether the requested data row is currently selected to be migrated. In some possibly rare scenarios, a user request to modify a data row may happen simultaneously with the background migration process (e.g.,236) selecting the same data row for migration. In this scenario, the user request may be temporarily blocked until the requested data row has been migrated. Accordingly, if the requested data row is currently selected for migrating, then at806, I/O gateway234temporarily blocks the user request. At808, a determination may be made that the data row migration is complete. At810, once the data row migration is complete, the user request is processed by the I/O gateway.

At812, a determination is made as to whether the requested data row has been migrated to the target dataset. In one example, the last migrated key value and the key value of the requested data row can be used to determine whether the requested data row has already been migrated. The last migrated key value can be obtained from a control block of either the source dataset or the target dataset. In one example implementation, if the key value of the requested data row is less than or equal to the last migrated key value, then the requested data row has already been migrated. Conversely, if the key value of the requested data row is greater than the last migrated key value, then the requested data row has not been migrated.

If the requested data row has not been migrated from the source dataset to the target dataset, then at814, the data row is modified in the source dataset based on user access to a source block in memory. In this scenario, the modification can be made based on the same block size in memory and in storage. This is because block size of the source block loaded in memory (e.g., old block size 4K) is the same as the block size defined for the source dataset in the source storage device (e.g., old block size 4K). Modifications of data can include changing the content of the data row, deleting the data row, compressing or decompressing the data row, encrypting the data row, etc.

If the requested data row has already been migrated to the target dataset, as determined at812, then at816the data row contained in a target block loaded in memory (e.g., new block size 27K) is updated based on the new block size. If the data row has been migrated, then the data row is modified in the target dataset even if the modification was requested by a user based on the user accessing the data row via a source block of the source dataset that is loaded in memory.

The internal processing of the user modification request for a data row in the source or target data block size is completely transparent to the user. The database manager in concert with the I/O gateway, manages all aspects of the data block size management and makes the process transparent to the end-user.

FIG. 9is a simplified flowchart900illustrating some of the data add operations (new rows) that may be performed by an I/O gateway (e.g.,234) during the architecture change process. Flowchart900relates to handling user requests to add data rows to a source dataset (e.g.,242) after the migration process to migrate the source dataset to a target dataset (e.g.,252) has been started. If a user request adds a data row once the migration process is started, the I/O gateway directs the addition of the new row to the target dataset. (e.g.,252). This insures that new rows can be added concurrently with the migration process. New rows may be added to the first available space in the active target block. The placement of the new data row may be out of “perfect sequence” with the other data rows, but the small number of adds that typically occur would not substantially affect the overall data row sequence. New data rows that are added during the migration process are tracked by the I/O gateway and the control blocks so that a subsequent request to read or modify the new row (while the migration process is still active) will automatically be directed to the target dataset.

At902, I/O gateway234receives a user request to add a new data row in source dataset242. At904, a determination is made as to whether the migration process has been started. If it has not started, then at906the data row is added to the source dataset following normal processing procedures.

If the migration process has begun, then at908, the I/O gateway234directs the addition of the new row to the target dataset252. The I/O gateway234finds space in the current active target block and memory for the new data row.

At910, the new data row is added to the located space in the current active target block and memory. The addition of the new data row by the I/O gateway234is synchronized with the migration activity. Synchronizing data row additions with migration activity allows concurrent migrations with data row additions.

At912, the migration control block is updated (e.g., by a key value associated with the newly added data row) so that any future access requests for this new data row will be directed to the target dataset252.

FIG. 10is a simplified flowchart1000illustrating some operations that may be performed by a database manager (e.g.,232) during an architecture change process. Flowchart1000relates to handling requests from utility processes (e.g., database utility application processes, DBA-initiated processes, other non-end user processes, etc.) to access data rows that are being migrated from a source dataset (e.g.,242) to a target dataset (e.g.,252). In some scenarios, if a utility process request is received for a particular source dataset during an architecture change process of that source dataset, then a database manager (e.g.,232) may block the request. In some other scenarios, if a requested utility process does not conflict with the architecture change process, then the I/O gateway associated with the architecture change process may provide full integration of the utility process with the data row migration. For at least some utility processes, however, the database manager may provide an alternative process that can be integrated by the I/O gateway associated with the architecture change process. In some embodiments, the database manager may send a request to an authorized user (e.g., DBA) to obtain permission to initiate the alternative process. In other embodiments, the database manager may initiate the alternative process automatically. In at least some embodiments, the database manager may issue the alternative process based on whether the alternative process has been pre-authorized to run automatically.

At1002, a database manager232may receive a utility process request that requires access to data of a source dataset (e.g.,242) being re-architected. At1004, database manager232determines whether the requested dataset is currently in an architecture change process. If the requested dataset is not being re-architected, then at1006, the utility process may be allowed to proceed.

If the requested dataset is currently in an architecture change process, then at1008, a determination is made as to whether the utility process conflicts with the migration. If the utility process is determined to not conflict with the migration, then at1010, the utility process is allowed to run and the utility processes that access data rows are handled by the I/O gateway providing full integration with the data row migration.

If the utility process is determined to conflict with the migration, then at1012, a determination is made as to whether an alternative utility process is available and authorized to run. Determining whether an alternative utility process is authorized to run can include, but is not limited to, requesting authorization from an authorized user (e.g., DBA) or determining whether running the utility has been pre-authorized.

If an alternative utility process is not available or is determined to not be authorized to run, then at1014, the database manager may block the utility process until the architecture change process is complete.

If an alternative utility process is available and authorized to run during an architecture change process, then at1016, the database manager can issue a command for I/O gateway234to run the alternative utility process.

An alternative utility process can be configured to allow the I/O gateway to integrate the alternative utility process with the background migration process. In one example, the alternative utility process issues requests to access data to the I/O gateway. The I/O gateway receives the utility process requests and, for each request, may use a process similar to flows previously described herein for data access requests (e.g.,FIGS. 7A-7B) and for data modification requests (e.g.,FIG. 8).

In another example, the I/O gateway may allow the alternative utility process to access data sequentially, as it is migrated to the target dataset. For example, if an alternative backup utility is run by the I/O gateway, then the I/O gateway may establish a connection to a backup storage device, and then provide the alternative backup utility with access to data rows after they are successfully migrated to the target dataset.

FIG. 11is a simplified flowchart1100illustrating some operations that may be performed by an I/O gateway (e.g.,234) during an architecture change process. Flowchart1100relates to handling requests from an authorized user (e.g., DBA) or authorized process to pause the architecture change process. In one example scenario, a very large dataset may take several hours to complete. If a decision is made that another system event (e.g., unscheduled maintenance) must take priority over the architecture change process, then the DBA or other authorized person may issue a command to pause the architecture change process. This may be preferable to allowing the process to crash and then initiating the restart/recovery process using the log file (e.g.,233) once system processing is resumed.

In flowchart1100, at1102, the I/O gateway receives a command to pause the architecture change process. In at least some embodiments, the database manager sends this command to the I/O gateway after receiving a command to pause the process from an authorized user or authorized process. In one example scenario, a command to pause the process may be received in order to allow an emergency action to proceed (e.g., stopping and restarting the system). In another example scenario, a DBA may pause the migration process to lessen the load on the database region while another critical process (e.g., billing) completes.

At1104, the I/O gateway pauses the architecture change process. For example, the I/O gateway stops migrating data rows. The I/O gateway may still process user data requests using the data rows in the source and target datasets. At this point, the DBA (or other system manager) may decide to take the system down and perform the action that triggered the need to pause the architecture change process.

At1106, once a determination is made to resume system processing (e.g., the unscheduled maintenance is complete), a command is received to restart the architecture change process. For example, the database manager may send the command to restart the architecture change process based on the completion of the system event (e.g., maintenance utility completes) or based on a command from the authorized user or process to restart the architecture change process.

At1108, the I/O gateway identifies a location in the source dataset where data migration is to resume. In one embodiment, the I/O gateway may retrieve the last migrated key value from the control block of the target dataset and/or the source dataset. The last migrated key value indicates the last data row in a sequence of all data rows in the source dataset that was successfully migrated. The I/O gateway may then select the next data row from the source dataset based on the next key value in the sequence after the last migrated key value. The I/O gateway may resume migration using this selected data row.

The flowcharts and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various aspects of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed sequentially, substantially concurrently, or in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.