Efficient mirror data re-sync

Techniques are described herein for re-syncing a snapshot database with a base database. Data units of each database are associated with version numbers. When a data unit is modified, the data unit is associated with the current version number. When a snapshot database is re-synced with a base database, particular data units are copied from the base database to the snapshot database, based on the version number associated with the particular data units.

FIELD OF THE INVENTION

The present invention relates to database management. Specifically, the present invention relates to efficient data re-sync for mirrored databases.

BACKGROUND

For many database systems, it is desirable to have one or more physical copies of an original database. The database copies can be used to deploy and test applications or features prior to using them in production.

The database copy, referred to herein as a “snapshot database,” is a point-in-time copy of a base database. The snapshot database mirrors data contained in the base database up until the time the point-in-time copy is created. Once the snapshot database is created, changes to the snapshot database and the base database are made independently. However, in order to ensure testing continues to be accurate and relevant, the snapshot database periodically needs to be re-synced with the base database in order to ensure the snapshot database contains up-to-date data. When re-syncing the snapshot database with the base database, changes made to the snapshot database are reverted, while changes made to the base database are copied to or reproduced in the snapshot database.

One method for re-syncing the base database with the snapshot database is to create a new point-in-time copy of the base database, and replace the old snapshot database. However, for large databases, the process of generating a new physical copy of a base database may be time-consuming. Additionally, if the previous copy of the snapshot database is not (or cannot be) deleted prior to creating the new copy, twice as much storage space is required in order to store both copies.

A second method is to compare each data block in the snapshot database with the base database. If the blocks are different, then the data in either the snapshot or the base database has changed. The data block is copied from the base database to the snapshot database. However, for large databases, this results in a large number of data block comparisons. Not only are the data comparisons computationally expensive for large amounts of data, it is inefficient to compare all the data blocks if only a small portion of each database has changed. Additionally, if multiple snapshot databases have to be re-synced, then data block comparisons are performed for each copy, which significantly increases the time and resources required.

A third method is to track all changes to each database. For example, the database system could maintain one or more change logs with timestamps corresponding to changes made to each database. However, tracking changes adds additional processing costs to writing data. Additionally, if the database system is a clustered database system, it is difficult to maintain consistent time stamps across nodes in the cluster.

Based on the foregoing, there is a need for a method to provide an efficient and high performance re-sync of a snapshot database with a base database.

DETAILED DESCRIPTION

General Overview

Techniques are presented herein for efficiently re-syncing one or more snapshot databases with a base database. Changes made to a snapshot database and to a base database are tracked. Only portions of the snapshot database that were modified, or portions that correspond to changes made to the base database, need to be synced with the base database.

A base database comprises a plurality of data units. A “data unit” may be a data block, a set of data blocks, an allocation unit comprising data blocks from a disk or disk group, an extent (set of contiguous data blocks in a file), or any other unit of storage for database data.

A snapshot database is a copy of the base database. Each data unit of the snapshot database is a copy of a corresponding data unit of the base database. The snapshot database is re-synced with the base database by copying one or more data units from the base database to the snapshot database.

According to an embodiment, each data unit of a database is associated with a first version number. When a data unit is updated, the data unit is associated with a second version number. The database system does not have to update the associated version number if the data unit is modified many times before the snapshot database is re-synced with the base database. Thus, the database system does not have to track every change made to the base database or the snapshot database. The database system only needs to track whether data at a location was modified.

When the snapshot database is re-synced, if a data unit in the base database is associated with the second version number, then the data unit is copied to the snapshot database. If a data unit in the snapshot database is associated with the second version number, then a corresponding data unit is copied from the base database. Thus, only data units that were modified have to be copied. In addition, the database system only has to check the version number associated with each data unit, rather than comparing the data stored in each data unit, to determine whether the data unit should be copied to the snapshot database.

System Overview

A snapshot database is a point-in-time copy of a base database. In order to generate a snapshot database, a database system copies all data stored in the base database at the time the copy is generated. While the database system prepares the snapshot database, changes to the base database are made to the copied data. After the database system finishes creating the snapshot database, changes made to the snapshot database and the base database are made independent of the other database.

FIG. 1is a block diagram that illustrates an example database system with a base database and two snapshot databases. Database management system (DBMS)100includes four instances: instance102, instance104, instance106, and instance108. Instances102,104,106, and108may be database instances running on a database server, or may each be running on a respective node in a clustered database system.

Each of instances102,104,106, and108have access to storage120. Storage120may comprise one or more disks to which each of instances102,104,106, and108have access. Storage120is storing a base database130, snapshot database132, and snapshot database134. A snapshot database may be a copy of a base database or another snapshot database. For example, snapshot database132may be a copy of base database130at a first point in time, while snapshot database134may be a copy of snapshot database132at a second point in time.

In an embodiment, storage120comprises one or more disk groups managed by the DBMS. Each disk group may store data for databases130,132, and134as one or more respective database files. Each database file may comprise one or more data units. As referred to herein, a “data unit” is a unit of allocation for a disk group managed by the DBMS. Each disk group may be divided into a plurality of data units of a pre-determined size. For example, each data unit may store 1 MB of data. Instances102,104,106, and108may be database file system instances that allow a database instance to access data stored in storage120. In an embodiment, base database130comprises one or more database files. Creating a copy of database130includes copying the one or more database files.

For the purpose of illustrating a clear example, techniques are described with reference to data units, but may be used with data blocks, files, or other level of granularity and data storage structure.

Version Numbers

In order to re-sync data between snapshot databases and a base database, changes made to each database are tracked using version numbers. A version number indicates a point in time at which a copy of the base database was taken. The version number is the same across both the base database and all snapshot databases. For example, before any snapshot databases are created, the version number is set to 0. When the first snapshot database is created, the version number is increased to 1. If a second snapshot database is created, the version number is increased to 2.

In an embodiment, each data unit is associated with a version number.FIG. 2Aillustrates data units of a base database and snapshot database. For the purpose of illustration, assume snapshot database132has just been created by copying base database130. Base database130comprises data units200,202,204,206, and208. Snapshot database132comprises data units220,222,224,226, and228. Although the current version number is 1, the data units for base database130and snapshot database132are each associated with version number 0 because no data has been modified.

When data in a database is modified, the data unit that was modified is associated with the current version number.FIG. 2Billustrates data units of a base database and a snapshot database after data in each database have been modified. Assume data units202and208of base database130and data units224and226of snapshot database132have been modified. Data units202,208,224, and226are associated with version number 1, while data units200,204,206,220,222, and228are associated with version number 0.

In an embodiment, data units are associated with a version number in a unit-to-version mapping. The unit-to-version mapping may be stored in a change tracking file or a data structure in persistent disk or persistent memory, such as in storage120. Each database may have a respective unit-to-version mapping. In an embodiment, the unit-to-version mapping indicates a version number for each data unit of a database. In another embodiment, the unit-to-version mapping only indicates data units that have changed. For example, referring toFIG. 2B, the unit-to-version mapping may only indicate that data units202,208,224, and226are associated with version number 1, and not include the data units that were unchanged.

In an embodiment, the unit-to-version mapping is only updated when the version number associated with a data unit is increased. For example, the first time data unit202is changed, the unit-to-version mapping is updated to indicate that data unit202is associated with version number 1. However, if data unit202is changed again while the version number is 1, then the unit-to-version mapping does not need to be updated. The unit-to-version mapping will continue to indicate that data unit202is associated with version number 1, regardless of how many times data unit202is modified. If the version number increases to 2, then the next time data unit202is modified, the unit-to-version mapping will update to indicate that data unit202is now associated with version number 2.

In an embodiment, the version number increases when a snapshot database is created or when snapshot databases are re-synced with the base database.FIG. 3Aillustrates data units of a base database and two snapshot databases, after a third snapshot database is generated. For the purpose of illustration, assume snapshot database134is a copy of base database130after the data in base database130was modified as shown inFIG. 2B. Since another snapshot database was created, the version number is increased to 2. Snapshot database134contains the modified data from base database130while snapshot database132contains the changes previously shown inFIG. 2B. Data units302and308are associated with version number 1, like data units202and208in database130.

If data is modified in either base database130, snapshot database132, or snapshot database134, then the unit-to-version mapping will indicate that the modified data unit is associated with version number 2. Data units that were modified prior to the creation of snapshot database134will continue to be associated with version number 1, unless they are later modified. Data units that have not been modified will continue to be associated with version number 0.

FIG. 3Billustrates data units of base database130and snapshot database132and134after data has been modified. Data units204,220,302, and304have been modified. Each of data units204,220,302, and304are associated with version number 2.

In an embodiment, each snapshot database is associated with the version number from when it was created. In the present example, snapshot database132is associated with version number 1 while snapshot database134is associated with version number 2.

In an embodiment, re-syncing one or more snapshot databases with a base database increases the version number. The one or more snapshot database are associated with the new version number as if they were a newly created snapshot database. For example, if snapshot database132and snapshot database134were re-synced, then the version number increases to version number 3 and snapshot databases are associated with version number 3. If only snapshot database132was re-synced, then snapshot database132would be associated with version number 3 while snapshot database134remains associated with version number 2.

In a clustered database system, multiple database instances may update or modify the same database at the same time. In an embodiment, each database instance stores a respective unit-to-version mapping. Each database instance can update its own unit-to-version mapping without waiting for a different database instance to finish updating the mapping. The database system indicates, to each database instance, the current version number. When the version number changes, the database system may indicate the updated version number to each database instance.

When a snapshot database is re-synced with a base database, any changes that were made to the snapshot database are reverted while modified data in the base data are copied to the snapshot database. If a data unit in the base database was modified, then the data unit is copied to the snapshot database being re-synced. If a data unit in the snapshot database was modified, then the data unit is copied from the base database, regardless of whether that data unit was updated in the base database.

In an embodiment, the unit-to-version mapping is used to determine which data units in the snapshot database and the base database have changed. The database system checks the version number associated with a data unit in a snapshot database and a corresponding data unit in the base database. If a data unit has a version number equal to the current version number, then the data unit was modified. If either the data unit in the snapshot database or the corresponding data unit in the base database were modified, the data unit is copied from the base database to the snapshot database. The data unit in the snapshot database is overwritten with the data from the base database.

If both the data unit in the snapshot database and the corresponding data unit in the base database were unchanged, then that data unit does not need to be updated in the snapshot database. The only data units that need to be copied are data units that were modified in the base database and data units that were modified in the snapshot database.

FIG. 4illustrates a re-sync of database130and database132, as modified inFIG. 2B. Data units202and208of base database130and data units224and226of snapshot database132are associated with version number 1. Thus, to re-sync snapshot database132with base database130, data units202,204,206, and208are copied from base database130to snapshot database132. Data unit220does not need to be copied, since neither data unit220nor corresponding data unit200were changed.

In an embodiment, each snapshot database is associated with a particular version number. When a snapshot database is re-synced with a base database, only the data units that are associated with a version number greater than or equal to the particular version number associated with the snapshot database need to be updated. For example, assume a first snapshot database is associated with version number 1 and a second snapshot database is associated with version number 2. If any data units in the first snapshot database are associated with version number 1 or greater, then the corresponding data unit needs to be copied from the base database. If any data units in the base database are associated with version number 1 or greater, then it is copied to the snapshot database.

However, only data units in the second snapshot database that are associated with version number 2 or greater need to be copied from the base database. Similarly, only data units in the base database that are associated with version number 2 or greater need to be copied to the second snapshot database. Data units associated with version number 1 or lower do not need to be copied.

FIG. 5illustrates a re-sync of database130with database132and database134. Data units202and208of base database130, data units224and226of snapshot database132, and data unit308of snapshot database134are associated with version number 1. Data unit204of base database130, data unit220of snapshot database132, and data units302and304of snapshot database134are associated with version number 2.

Referring toFIG. 3B, snapshot database132is associated with version number 1. Any data units associated with version number 1 or higher need to be re-synced from base database130to snapshot database132. Thus, data units200,202,204,206, and208are each copied to snapshot database132.

Snapshot database134is associated with version number 2. Only data units associated with version number 2 or higher need to be re-synced from base database130to snapshot database134. Thus, only data units202and204are copied to snapshot database134.

DBMS Overview

Embodiments of the present invention are used in the context of DBMSs. Therefore, a description of a DBMS is useful.

A DBMS manages one or more databases. A DBMS may comprise one or more database servers. A database comprises database data and a database dictionary that are stored on a persistent memory mechanism, such as a set of hard disks. Database data may be stored in one or more data containers. Each container contains records. The data within each record is organized into one or more fields. In relational DBMSs, the data containers are referred to as tables, the records are referred to as rows, and the fields are referred to as columns. In object-oriented databases, the data containers are referred to as object classes, the records are referred to as objects, and the fields are referred to as attributes. Other database architectures may use other terminology.

A database block, also referred to as a data block, is a unit of persistent storage. A database block is used by a database server to store database records (e.g., to store rows of a table, to store column values of a column). When records are read from persistent storage, a database block containing the record is copied into a database block buffer in RAM memory of a database server. A database block usually contains multiple rows, and control and formatting information, (e.g., offsets to sequences of bytes representing rows or other data structures, list of transactions affecting a row). A database block may be referenced by a database block address (DBA).

A database block is referred to as being atomic because, at least in part, a database block is the smallest unit of database data a database server may request from a persistent storage device. For example, when a database server seeks a row that is stored in a database block, the database server may only read the row from persistent storage by reading in the entire database block.

A multi-node database management system is made up of interconnected nodes that share access to the same database or databases. Typically, the nodes are interconnected via a network and share access, in varying degrees, to shared storage, e.g. shared access to a set of disk drives and data blocks stored thereon. The varying degrees of shared access between the nodes may include shared nothing, shared everything, exclusive access to database partitions by node, or some combination thereof. The nodes in a multi-node database system may be in the form of a group of computers (e.g. work stations, personal computers) that are interconnected via a network. Alternately, the nodes may be the nodes of a grid, which is composed of nodes in the form of server blades interconnected with other server blades on a rack.

Resources from multiple nodes in a multi-node database system can be allocated to running a particular database server's software. Each combination of the software and allocation of resources from a node is a server that is referred to herein as a “server instance.” A database server may comprise multiple database instances, some or all of which are running on separate computers or separate server blades.

Hardware Overview

Computer system600further includes a read only memory (ROM)608or other static storage device coupled to bus602for storing static information and instructions for processor604. A storage device610, such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to bus602for storing information and instructions.