Evolution of XML schemas involving partial data copy

XML schema evolutions can be performed on an XML-type database using partial data copy techniques. The partial data copy techniques provide mechanisms to identify a minimal set of data in the database that needs be copied out in order for schema evolution operations to occur. Identifying the minimal set of data involves comparing an existing XML schema to a new XML schema and determining the differences between the two schemas. Based on those differences, a minimal set of data can be determined. That data can then be copied to a temporary storage location, while the XML schema and its associated database structure are updated to conform to the new XML schema. Then, the minimal set of data is copied back into the modified database structure.

RELATED APPLICATION

The present application is related to U.S. patent application Ser. No. 11/540,341, entitled “IMPLEMENTATION OF BACKWARD COMPATIBLE XML SCHEMA EVOLUTION IN A RELATIONAL DATABASE SYSTEM,” filed by Agrawal, et al. on Sep. 28, 2006, the entire contents of which application is hereby incorporated by reference in its entirety for all purposes as if fully and originally disclosed herein.

The present application is related to U.S. patent application Ser. No. 11/519,720, entitled “MANAGEMENT OF COMPLEX XML SCHEMAS IN A DATABASE SYSTEM,” filed by Agrawal, et al. on Sep. 11, 2006, the entire contents of which application is hereby incorporated by reference in its entirety for all purposes as if fully and originally disclosed herein.

This application is related to U.S. patent application Ser. No. 10/259,278, entitled “MECHANISM FOR MAPPING XML SCHEMAS TO OBJECT-RELATIONAL DATABASE SYSTEMS”, filed by Ravi Murthy et al. on Sep. 27, 2002, the entire contents of which is hereby incorporated by reference for all purposes as if fully set forth herein.

BACKGROUND

An XML schema in an XML-type database defines the structure of instance XML documents inserted into a database. Instance XML documents are XML documents inserted into the XML-type database that conform to the XML schema. In an object-relational database system, the XML schema is used to determine the in-memory layout of the instance XML documents, as well as the database structures of the database representation in which XML documents are stored in the database system.

The database structures of a database representation of an XML schema are referred to herein as “base database structures.” Base database structures include, for example, table columns that each represent an element of an XML schema or object type tables whose object type attributes correspond to elements of the XML schema.

The in-memory layout of the instance XML documents is based in large part on the individual elements and attributes defined by the XML schema. However, an instance XML document does not need to be stored object-relationally on disk for it to have an XML schema-based in-memory layout. For example, an instance XML document may be stored on disk in a CLOB (Character Large Object), and yet its in-memory layout is still determined by the XML schema.

Once XML documents are stored in a database, a client application can perform queries on the database based on elements and attributes of the schema. Suppose, however, that the client application accessing the database changes. Often a change to a client application entails changes to an XML schema and its associated base database structures to maintain compatibility with the client application as changed. The procedure for making such changes to an XML schema and its associated base database structures is called schema evolution. One approach for performing schema evolution is copy-based evolution.

Copy-based evolution is a resource intensive technique that backs up all the data stored in tables associated with the evolving XML schema, deletes the old version of the XML schema (which deletes its associated base database structures), registers the new version of the XML schema (which creates new base database structures), and re-inserts the backed up data into the newly created structures. As part of re-inserting the backed up data into the new base database structures, the database system may apply a transformation on the data (e.g., in the form of an XSLT) to conform the data (e.g., XML documents) to the new version of the XML schema.

Although, copy-based evolution can update an XML schema, there are problems with its approach. For instance, copy-based evolution requires the database system to back up, transform, and restore the entire set of XML documents associated with an XML schema (e.g., both conforming and non-conforming XML documents) even though the change to the XML schema may be minimal or only affect a few documents. As a result, in large database systems, copy-based evolution may necessitate long downtimes.

In addition, there are some situations in which XML documents may be transformed during the copy-based evolution to conform the documents to the new version of the XML schema. Transforming XML documents is a time-consuming process. The reason the transformation process is time-consuming is that it involves copying each XML document in its entirety (instead of just those portions of the document that are affected by the XML schema changes) into memory before inserting new information into the document. Moreover, once the transformation is complete, the transformed document is copied as a whole back into the updated database.

DETAILED DESCRIPTION

Functional Overview

Mechanisms described herein perform XML schema evolutions using partial data copy techniques. The mechanisms perform the partial data copy techniques by comparing the existing version of the XML schema to the new version of the XML schema to determine the differences between the two schemas. The differences between the existing version of the XML schema and the old version of the schema are then used to determine a minimal set of data in a database that needs to be backed up before a schema evolution operation can take place. In an embodiment, mechanisms ensure that the minimal set of data is backed up to a temporary location. Once the minimal set of data has been backed up, additional mechanisms modify the base database structures in the database to conform to the new version of the XML schema. Then, according to an embodiment, the set of minimal data is moved back into the modified base database structures. According to an embodiment, the set of minimal data is transformed before it is moved back.

Partial Data Copy

Partial data copy schema evolution combines the benefits of both in-place evolution as well as copy based evolution. In-place evolution is described in “IMPLEMENTATION OF BACKWARD COMPATIBLE XML SCHEMA EVOLUTION IN A RELATIONAL DATABASE SYSTEM,” referenced above.

Using partial data copy techniques, users can perform a much more complex and varied set of schema evolution operations, without incurring the overhead of copy-based evolution techniques. Moreover, the partial data copy evolution does not have the same backward compatibility requirement as in-place evolution techniques. Note that a change to an XML schema is backward compatible if existing instance XML documents, without transformation, continue to conform to the new version of the XML schema. A database system implementing the partial data copy techniques and mechanisms described herein can execute the more complex and a varied schema evolution operations in a more reasonable execution time.

Procedure for Performing Partial Data Copy Evolution

FIG. 1is a flowchart illustrating the procedure100for performing partial data copy schema evolutions. The process begins when a user invokes a schema evolution operation on a database system. The schema evolution operation may modify an XML schema in a variety of ways, for example, adding elements, removing elements, or modifying existing elements in an XML schema.

According to an embodiment, when the database system determines that a schema evolution operation has been invoked, the database system performs the following steps: 1) the database system determines if there is a minimal set of data in the database that will be affected by the schema evolution operation110, 2) the database system copies the minimal set of data out of the database system into a temporary location120, 3) the database system modifies the base database structures based on the requested schema evolution operation130, and 4) the database system moves the minimal set of data back into the modified base database structures140. In some embodiments, the minimal set of data is transformed before it is moved back to the base database structures.

For example, suppose a user creates an XML schema that defines a Person and registers the schema with a database system.FIG. 2Aillustrates an example XML schema200defining a Person. InFIG. 2A, the Person XML schema200includes the elements Name, BirthDate, Address, and PhoneNumber. When the Person schema200is registered with a database system, the database system creates an object-relational table based on the Person schema. The resulting object-relational table has columns for each of Name, BirthDate, Address, and PhoneNumber. Moreover, each column has an associated type. In this example, BirthDate210initially holds simple string values. Now, assume that a number of instance XML documents are inserted into the table associated with Person.

After the instance XML documents have been inserted into the corresponding table, a database administrator updates the schema by creating a DateType that enforces a single date format. Accordingly, as illustrated inFIG. 2B, the Person schema200is updated to add the DateType220and to change BirthDate230from being of a string type to DateType. These changes call for schema evolution operations. Therefore, in an embodiment, the BirthDates stored in the current BirthDate column of the Person table are backed up to a temporary location. Then, the old BirthDate column is deleted from the table and a new BirthDate column of DateType is added. In some embodiments, the BirthDate column may simply be re-typed (e.g., changed from string type to DateType) without deleting the original column. In such cases, the amount of data backed up can be further limited to values in BirthDates that do not conform to DateType. Once the new BirthDate column has been created, the backed-up BirthDate information is transformed to the proper format and copied back into the new BirthDate column. In this way, a minimal amount of data (e.g., the BirthDate information) needs to be copied out.

Note that if the database system cannot transform some of the BirthDates, in an embodiment, the user is prompted to correct the problem.

Determine the Minimal Set of Data

At step110inFIG. 1, a database administrator performs a request to update a schema. Accordingly, the database system checks to see if the requested schema evolution operation is valid and then determines if there is a minimal set of data that is affected by the schema evolution request. The minimal set of data generally refers to an amount of data that has been determined to be the least amount needed to be backed up in order for a schema evolution operation to take place. To determine the minimal set of data, in one implementation, the database system compares the existing version of an XML schema to a new version of the XML schema in order to identify differences between the two. Comparing the two schemas can be performed using a lock-step traversal of both schemas. In alternative embodiments, the comparison is performed in some other way.

For example, suppose after performing a lock-step traversal of both schemas, the database system determines that the only difference between the two schemas is a single element corresponding to a column in the object-relational database. In such a case, that column comprises a minimal set of data to be backed up. Alternatively, if multiple columns are affected by the schema evolution operation, then multiple columns are marked to be backed up.

In an embodiment, additional filtering criteria may be used to identify single entries within a column or table that are affected by the schema evolution operation. For instance, in the BirthDate example above, suppose the vast majority of the BirthDates in the old BirthDate column are already in DateType format. Accordingly, in an embodiment, the minimal set of data may include only those BirthDate entries that do not conform to DateType. In such a case, instead of deleting the entire column, the database system may simply back up those entries that do not conform to the proper format and create a second BirthDate column for those entries.

In the end, the process of determining the data affected by a schema evolution operation involves identifying which elements in the schema are going to be updated by the schema evolution operations. In an embodiment, the database system evaluates the proposed changes to the schema to determine which elements' corresponding column (or entries) in the database structure will be changed by the schema evolution operation. Those elements' columns or entries are marked as data to be backed up.

Copy Out the Minimal Set of Data

After the database system has marked the minimal set of data that needs to be backed up, at step120, the database system performs a copy operation. The copy operation moves the identified minimal set of data to a temporary location until the underlying database structure can be modified. For example, the minimal set of data is copied to a temporary table or column until the schema evolution operations have completed. Note that the temporary location or table may be part of the same database system in which the XML schema is registered. In an embodiment, the temporary location may even be in the same database represesentation.

Since most schema evolution operations affect only a small portion of a schema at a time, the minimal set of data is often relatively small. Thus, by backing up the minimal set of data, the running time for the schema evolution operation in many cases is drastically reduced. This in turn drastically reduces database downtime.

In an embodiment, the ability to back-up minimal sets of data allows a database administrator to update a schema incrementally. This means that although an XML schema may need to be changed drastically, the database administrator can perform partial data copy schema evolution on just a few elements at a time. In this way, the schema evolution process can be rolled out over time.

In an embodiment, the partial data-copy techniques allow schema evolution operations to be performed while the database is live.

Modify the Database Structure

At step130, after the minimal set of data has been copied to a temporary location, in an embodiment, the database system performs the schema evolution operations to modify the base database structure that stores the instance XML documents. This step involves modifying the base database structures that store the instance XML documents to be consistent with the new version of the XML schema. The process of modifying the database structure typically involves invoking data definition language (“DDL”) or other low-level database commands to add, remove, and modify columns, tables, or other database structures from the base database structures.

For example, in the BirthDate example above, the schema evolution operations may invoke an “alter table” command once the minimal set of data has been copied to a temporary location. The “alter table” command instructs the database system to make changes to the underlying Person table. Accordingly, in an embodiment, the BirthDate column is deleted by an alter table command and a new BirthDate column added to the Person table.

Copy the Data Back into the Modified Database Structure

At step140, after all the changes have been made to the database structure associated with the instance XML documents, the database system copies the data from its temporary location into the modified structures.

Skipping the Copy Operation

According to an embodiment, steps120and140of procedure100can be combined to further reduce the amount of data being copied from location to another. For example, in the BirthDate example above, in an alternative implementation, instead of backing the minimal set of data to a temporary location, the database system creates a new BirthDate column in the table and copies the set of minimal data directly to the new column. In this way, the contents of the BirthDate column can be copied directly into a new BirthDate column while the old column is still active.

In an embodiment, the minimal set of data can also be transformed to conform to the DateType before being copied to the new BirthDate column. This allows the entire partial data copy evolution process to occur while the database is still up. Once the copying to the new BirthDate column has completed, the database system simply renames the columns.

In another embodiment, the XML schema evolution creates an entirely new BirthDate column called BirthDate2. In this embodiment, the database system can then not only copy the minimal set of data directly to the new column, but since the updated schema introduces a new column with a new name, the last rename step can also be avoided.

In yet other embodiments, the steps may be performed in a different order, other steps may be combined, some steps may be separated, or some steps may be skipped.

Example Operation of Partial Data Copy

To further illustrate procedure100, consider the following examples that are based on XML schema300illustrated inFIG. 3A. XML schema300is a new version of an XML schema that defines a PurchaseOrder table with a number of different element types (e.g., elements such as a user of type “user type”, a reference of type “reference type”, item number as type integer, etc.).

Suppose in old version of the XML schema, the Date element is defined as a string, and as a result errors, inconsistencies, and invalid date information are introduced to the database.FIG. 3Billustrates a segment301of the old version of XML schema300with Date element340defined as a string. Because in the old version of the schema the Date element is defined as a string, the administrator for the database decides to change the type of the Date310from string to DateType320. Accordingly, the database administrator inputs commands to evolve the old version of the XML schema to the new version300illustrated inFIG. 3A. As a result, the database system executes the following steps to evolve the Date element from a string type to DateType320.

As discussed in connection with procedure100illustrated inFIG. 1, the database system compares the old version of the XML schema to the new version of the XML schema300to identify changes between the two schemas. In this example, the only changes are those made in connection with the Date element310and DateType320. Accordingly, the database system copies out the set of data corresponding to Date from the PurchaseOrder table. In an embodiment, the information in the Date column is copied out to a temporary location using DDL commands. An example of a DDL command to copy the information stored in the Date column to a temporary location is illustrated below at Instruction 1:

Instruction 1 creates a table called Backup_Dates where all of the data stored in the PurchaseOrder table's Date column is copied. Then, the database system removes the old Date element from the old version of the XML schema by performing in-place evolution. In addition, the PurchaseOrder table is modified by the database system to remove the old Date column. In an embodiment, a command such as an “alter table” command can be used to remove the old Date column.

After the old Date element has been removed from the old version of the XML schema and the corresponding column removed from the Purchase Order table, the database system updates the schema by adding a new Date element310to the new version of the XML schema. The new Date element310is of type DateType320. In conjunction with this update, a new column is added to the PurchaseOrder table corresponding to the new Date element. The new column is also of type DateType.

After the new Date column has been added to the PurchaseOrder table, then the data stored in the Backup_Dates table is copied back into the new Date column. Instruction 2 below illustrates an example DDL command for copying the Date data back into the PurchaseOrder table.

It should be noted that, in this example, the partial data copy schema evolution affects one column in a large table. Accordingly, the execution time is relatively short compared to a full data copy operation.

In an embodiment, the new Date column of type DateType is added prior to the initial backing-up of the Date data. In this embodiment, the data in the original Date column is transformed and copied directly to the new Date column. Moreover, in this embodiment, the partial data copy schema evolution operations are performed while the PurchaseOrder table is active.

Consider another example of a partial data copy schema evolution. In this example, a scalar element is converted into a non-scalar element. This example is slightly more complex but has similar implications for the XML schema and its associated database structure. Referring again to the new version of XML schema300shown inFIG. 3A, suppose a database administrator wishes to change the sequence element listed under the complexType “ShippingInstructions Type”330from its current scalar value (e.g., only one shipping instruction can be listed) to an unbounded value (e.g., to store multiple instructions) as illustrated inFIG. 3C. The reason this example is a little bit more complex is that the partial data copy evolution schema operations are converting information that was stored as scalars to arrays (or collections) of information. Accordingly, the following steps outline what the database system does to accomplish a partial data copy schema evolution in this example.

Again, the database system begins by copying out all the data corresponding to the ShippingInstructions column. Instruction 3 below illustrates an example DDL command for creating the backup table for ShippingInstructions and copying the data associated with ShippingInstructions to the backup table.

Once the backup table has been created and all the ShippingInstruction information copied to it, the database system removes the ShippingInstructions element and the global ShippingInstructions type from XML schema300through in-place evolution operations. The reason for deleting both the ShippingInstructions element and the ShippingInstructions type is because ShippingInstructions type inFIG. 3Ais designed to store just one ShippingInstruction, not a collection of ShippingInstructions.

Once the ShippingInstructions element and the ShippingInstructions type have been removed, the database system creates a new global ShippingInstructions type with a new definition that can store collections of ShippingInstructions. Moreover, a new ShippingInstructions element is added to PurchaseOrderType with the desired ShippingInstructions type definition. As before, corresponding changes are made in the associated base database structures through in-place evolution commands.

Once a new ShippingInstruction column has been added to PurchaseOrder table, the data stored in backup is copied to the new ShippingInstructions column. Instruction 4 below illustrates an example command for copying the data back into the PurchaseOrder table.

By performing partial copy data evolution users have the benefits of both in-place and copy-based evolution. As a result, the schema evolution process runs more efficiently, reducing downtime and providing a wider variety of options on how to update XML schemas.

Hardware Overview