In place migration when changing datatype of column

Techniques are described herein for altering the datatype of a column without having to immediately migrate the data items that currently reside in the column. In one embodiment, the alteration is performed without migration by creating a new column having the desired datatype while retaining the old column. Instead of migrating the data items from to the new column at the time the column is altered, the items remain in the original column until the items are updated.

FIELD OF THE INVENTION

The present invention relates to data containers and, more specifically, to commands that alter the datatype of a column of a table.

BACKGROUND

It is not uncommon to want to make changes to a data container, such as a relational table, after the container has been in use for a while. For example, a relational table may be created with a column X for storing a certain type of data (the “original datatype”). After the table has been in use for a while, a user may want to modify the table to allow column X to store a different type of data (the “new datatype”).

Unfortunately, simply modifying the definition of the table may be insufficient to make such changes, because the new datatype may have a different storage format than the original datatype. Consequently, the “target column” (in this case, column X) may not have the right amount of storage for the new datatype. Therefore, the old column needs to be replaced with a new column with the appropriate amount of storage.

However, creating a new column may also not be enough, since the original column may already contain stored data items that are formatted according to the original datatype. Therefore, in addition to changing the definition of the table and creating a new column, the data items in the original column have to be converted to the format of the new datatype, and then stored into the new column. Once all of the data from the target column has been migrated to the newly-created column, the original column may be dropped.

The bulk migration of target column data items may be accomplished, for example, by issuing a Recursive Procedure Invocation (a recursive SQL call). Unfortunately, the bulk migration of the data items that exist in the target column can take an unacceptably long time, especially for tables that contain millions of rows. Further, the conversion may cause row-chaining and non-locality of data storage.

DETAILED DESCRIPTION

Overview

Techniques are described herein for altering the datatype of a column without having to immediately migrate the data items that currently reside in the column. In one embodiment, the alteration is performed without migration by creating a new column having the desired datatype while retaining the old column. Instead of migrating the data items from to the new column at the time the column is altered, the items remain in the original column until the items are updated.

Data manipulation language (DML) operations performed against the column are handled in a special manner. For example, when the data items are updated, the data items are migrated to the new column. When new items are inserted, they are inserted into the new column. When items are selected, the new column is first checked. If the new column is NULL, then the old column is checked.

Alternative embodiments are described in which the old column is checked first. In addition, techniques are provided for performing scheduled migrations of items from the old column to the new column. Once all of the items have been migrated to the new column, the old column may be dropped, and the special handling of DML operations may stop.

Example Conversion Operation

For the purpose of explanation, assume that the target column is column target_col in table t. Further assume that the original datatype of target_col is datatype1. Thus, the following command may have been used to create table t:

After table t has been in use for a while, a user may decide to change the datatype of target_col to datatype2. The user may initiate this conversion by issuing the command:

ALTER TABLE t MODIFY (target_col <datatype2>);

In response to receiving this alter statement, the database server performs the metadata operations illustrated inFIG. 1. Significantly, none of the operations involve the movement or conversion of exiting data items.

Referring toFIG. 1, in step100, the existing column “target_col” is modified to be a hidden column, and its name is changed. For the purpose of explanation, it shall be assumed that the name is changed to SYS_NC. Control then proceeds to step102.

In step102, a new column is added to table T. The new column is defined as “target_col <datatype2>” with default value NULL. Thus, the new column has the original name of the old column, but is defined to store data items of the new datatype.

According to one embodiment, the storage for values for new column is added to the end of the storage for table T, so the database server does not actually have to access all of the existing rows to add the NULL value for the new column. Control proceeds form step102to step104.

At step104, the database server records, in the metadata associated with table t, the fact that the new column “target_col” depends on the old column “sys_nc”. For the purpose of discussion, the “sys_nc” column shall be referred to herein as the “sister column” of “target_col”. Control then proceeds to step106.

At step106, the database server moves all of the ancillary metadata previously associated with the original column to the new column. Such ancillary metadata may include, for example, triggers, constraints, default values, etc. that had previously been defined for the original column.

DML operations have to be performed differently after the column of a table has been converted using the technique described above. Specifically, after the conversion, the data item for the target column may reside in the old column (sys_nc) for some rows, and in the new column (target_col) for other rows.FIG. 2is a flowchart showing how the database server may process DML operations after such a conversion, according to an embodiment of the invention.

Control begins at step200, where the database server determines whether the DML operation is an INSERT operation, an UPDATE operation, a DELETE operation or a SELECT operation.

If the DML operation is an INSERT operation, then control proceeds to step202, where the database server inserts the specified data into the new column “target_col”.

If the DML operation is an UPDATE operation, then control proceeds to step204, where the database server stores the new value in the new column “target_col”. If the new value is NULL, then the database server also stores a NULL in the sister column sys_nc.

If the DML operation is a DELETE operation, then control proceeds to step206, where the database server deletes the row (including the values in both the target_col and sys_nc).

Finally, if the DML operation is a SELECT, then control proceeds to step208, where the database server modifies the select logic to do the following: If “target_col” is not NULL, then get that value, otherwise get the value from the sister column “sys_nc”. Data items retrieved from the sister column will reflect the format of the old datatype. Consequently, the database server may have to perform a format conversion operation on them to conform them to the new datatype.

By processing DML operations in this manner, all new data items for the target column are placed in the new column, and data items that existed in the target column before the conversion are gradually “migrated” from the old column to the new column in response to being updated.

Scheduled Migration

The overhead of migrating data from the old column in a single bulk migration may be unacceptably large, as explained above. However, there are benefits to achieving a state in which all of the data items from the original column have been migrated. For example, if it is known that all of the data items from the original column have been migrated, the original column may be dropped, and the special handling of DML operations may be discontinued.

To achieve a fully-migrated state without incurring the overhead of a single bulk migration, a phased migration plan may be used. For example, a user can issue an UPDATE command against a certain range of rows within the table. The UPDATE command may do nothing more than assign to the target column the values that are already in the target column (e.g. UPDATE target_col TO target_col). The data items in the target column of every row touched by this update will be migrated from the old column to the new column, if they have not already been migrated.

Such “scheduled migration queries” may be executed during convenient times, such as late at night when their execution will not have a significant impact on the operation of the system. After the data items from all rows have been migrated to the new column in this fashion, the old column and special DML handling may be dropped.

According to one embodiment, NULLs are stored in the old column in response to any UPDATE, not just UPDATES to NULL. In such an embodiment, the rows that need to be migrated to the new column are easily identified, since they will be the only rows that have non-null values in the old column. Therefore, the absence of non-null values in the old column may be used as a trigger for dropping the old column, and ceasing to perform the special handling of DML operations that involve the target column.

Variations

In the foregoing description, specific details are described relative to certain embodiments. However, the techniques described herein are not limited to those specific details. For example, step208describes an embodiment in which a SELECT operation is performed by first checking the new column and, if it contains a null, checking the old column. However, in an alternative embodiment, the database server (1) always stores a NULL in the old column when a data item is updated, and (2) performs a SELECT operation by first checking the old column. If the old column contains a NULL, then the database server checks the new column.

This alternative embodiment has the advantage that the data items from the old column are more likely to be located near (e.g. on the same disk block as) the rest of the data items in the row (assuming that the old column was allocated at the time the table was created). Consequently, the old column may be accessed more efficiently than the new column, increasing performance in the cases where the old column contains non-null values.

Over time, the number of null values in the old column will increase, since null values are stored in the old column whenever a data item is updated. Consequently, at some point the decreased likelihood of finding a non-null value in the old column may make it inefficient to check the old column before checking the new column. Upon detecting that the number of non-null values in the old column has fallen below a certain threshold, the database server may switch to checking the new column first.

Hardware Overview

The received code may be executed by processor304as it is received, and/or stored in storage device310, or other non-volatile storage for later execution. In this manner, computer system300may obtain application code in the form of a carrier wave.