Data integrity procedure

Techniques are disclosed relating to ensuring data integrity between database objects. A computer system may receive a data integrity request for a first set of database tables. The computer system may generate at least two work items that correspond to respective data cell groups in the first set of database tables. The computer system may then cause a plurality of processes to perform the at least two work items to generate a first plurality of hash values that includes hash values for the respective data cell groups. The first plurality of hash values may be usable to compare with corresponding ones of a second plurality of hash values generated based on corresponding data cell groups in a second set of database tables replicated from the first set of database tables.

BACKGROUND

Technical Field

This disclosure relates generally to database systems and, more specifically, to ensuring data integrity between database objects.

Description of the Related Art

Companies routinely implement modern database systems that store information in an organized manner that can be accessed and manipulated. These database systems often include a collection of programs that together implement a database management system that interacts with a database that stores information. Such information is commonly represented in the form of tables that are composed of columns and rows in which each column defines a grouping of the information. During the operation of a database, information can be sent from that database to another database that serves as backup database in many cases.

The term “configured to” is not intended to mean “configurable to.” An unprogrammed FPGA, for example, would not be considered to be “configured to” perform some specific function, although it may be “configurable to” perform that function and may be “configured to” perform the function after programming.

As used herein, the terms “first,” “second,” etc. are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless specifically stated. For example, in a processor having eight processing cores, the terms “first” and “second” processing cores can be used to refer to any two of the eight processing cores. In other words, the first and second processing cores are not limited to processing cores 0 and 1, for example.

As used herein, a “module” refers to software and/or hardware that is operable to perform a specified set of operations. A module may refer to a set of software instructions that are executable by a computer system to perform the set of operations. A module may also refer to hardware that is configured to perform the set of operations. A hardware module may constitute general-purpose hardware as well as a non-transitory computer-readable medium that stores program instructions, or specialized hardware such as a customized ASIC. Accordingly, a module that is described as being “executable” to perform operations refers to a software module, while a module that is described as being “configured” to perform operations refers to a hardware module. A module that is described as operable to perform operations refers to both a software and a hardware module.

DETAILED DESCRIPTION

In many cases, information that is stored at a database is recreated at one or more other databases. For example, a company may implement a set of backup databases that store a copy of a portion or all of the information that is stored at a primary database. As another example, the information stored at a particular database might be split and copied into multiple databases in the event that the current amount of information reaches a threshold storage capacity of the particular database. It is usually desirable to ensure that information that is recreated at another database is consistent with the original information.

One approach for recreating information at another database is a block-level approach in which individual data blocks on disk are copied from the source database to the destination database—i.e., bytes of data are copied to create an exact copy. Accordingly, in order to ensure data integrity, the bytes of corresponding data blocks at the two databases can be compared to determine whether they match. Another upcoming approach for recreating information is an approach in which the database operations performed at the source database are replayed at the destination database—e.g., if an “INSERT” operation is performed at the source database, then that same INSERT operation is performed at the destination database. The same information is recreated at the destination database as the source database, but the underlying data blocks can be different. That is, the former approach performs a copy at the byte level while the latter approach performs a copy at a higher level of abstraction (at a logical level) and thus while the information that is recreated in both approaches is the same (e.g., the database tables include the same values for the same attributes), the underlying byte layout can be different. As a result, the data integrity approach of comparing data blocks (a byte-per-byte comparison) in order to ensure data integrity is not viable in the latter approach. The present disclosure addresses at least this technical problem of ensuring the integrity of data recreated under that latter approach of re-performing, at a destination database, database operations that are performed at a source database.

The present disclosure describes techniques for implementing a hashing scheme that is usable to ensure the integrity of data that has been recreated. This disclosure further describes techniques for providing concurrency support for the hashing scheme. In various embodiments that are described below, a computer system implements a data integrity procedure to generate hash values for database objects stored in a database. In some cases, a database object may be a database table comprising columns and rows whose intersections define data cells capable of storing data values. These data cells may be associated with respective hash values calculated based on the data values stored in those data cells. In various embodiments, the hash value for a database table is generated by first summing the hash values of the data cells of each column of the table to form column hash values. Those column hash values are then summed to form a table hash value that is representative of that table. The computer system may thus generate the hash values for the database objects in this manner, in some embodiments.

To determine whether a first set of database objects has been accurately recreated for a corresponding second set of database objects, in various embodiments, hash values for both of the sets of database objects are generated using the data integrity procedure. The hash values for the first set of database objects may be compared with the hash values for the second set of database objects. In some embodiments in which the first and second sets of database objects are stored at different databases managed by different computer systems, one of the computer systems may send the hash values for its set of database objects to the other computer system to perform a comparison. If the corresponding hash values match, then the first set of database objects has been accurately recreated; otherwise, the first and second sets are different.

When implementing the data integrity procedure, in some embodiments, the computer systems spawns multiple threads to concurrently process work items in order to generate hash values for corresponding database objects. A work item may designate a database object (or a portion thereof) to be processed in order to generate a hash value. In some embodiments, as a part of obtaining the data cell hashes mentioned above, processes issue database commands to extract those data cell hashes from a database. In various embodiments, however, the database is designed such that a maximum number of concurrent database operations can be performed with respect to the database—the maximum number may be set by a user as opposed to a limit that is imposed by the physical capabilities of the system. In various embodiment, the number of processes spawned is based on the maximum number of concurrent database operations and another number that specifies how many concurrent database operations to utilize per database object. Consider an example in which a database supports 64 concurrent database operations and a user requests that 16 concurrent database operations be used per database table. As such, the computer system may spawn 4 processes, each of which can issue 16 concurrent database operations to process their assigned database table.

These techniques may be advantageous as they provide a mechanism for ensuring that data that has been accurately recreated under a replication approach such as a statement-based replication approach. These techniques are further advantageous as they disclose a concurrent approach for implementing that mechanism so that the determination on whether data has been accurately recreated can be performed within a shorter interval of time by processing database tables in parallel. These techniques are further advantageous as they allow for that mechanism to be resumed without restarting from the beginning in the case of a failure associated with the data integrity procedure. An exemplary application of these techniques will now be discussed, starting with reference toFIG.1.

Turning now toFIG.1, a block diagram of a system100is shown. System100includes a set of components that may be implemented via hardware or a combination of hardware and software routines. In the illustrated embodiment, system100includes: a database110having database tables120, and a database node130. Also as shown, database tables120include data cells125, and database node130includes work items140, worker processes150, and hash values160. In some embodiments, system100may be implemented differently than shown—e.g., database110may include other database objects, such as indexes, that might be processed by database node130to generate hash values160.

System100, in various embodiments, implements a platform service (e.g., a customer relationship management (CRM) platform service) that allows users of that service to develop, run, and manage applications. System100may be a multi-tenant system that provides various functionality to multiple users/tenants hosted by the multi-tenant system. Accordingly, system100may execute software routines from various, different users (e.g., providers and tenants of system100) as well as provide code, web pages, and other data to users, databases, and other entities associated with system100. As shown for example, system100includes database node130that can store and access data from database tables120of database110on behalf of users associated with system100.

Database110, in various embodiments, is a collection of information that is organized in a manner that allows for access, storage, and manipulation of that information. Accordingly, database110may include supporting software that allows for database node130to carry out operations (e.g., accessing, storing, etc.) on information that is stored at database110. In some embodiments, database110is implemented by a single or multiple storage devices connected together on a network (e.g., a storage attached network (SAN)) and configured to redundantly store information to prevent data loss. Database110may be shared between multiple database nodes130that can read data from database110and/or write data to database110.

A database table120, in various embodiments, is a collection of information, including data elements that are organized into a structured format having rows and columns. In various embodiments, the intersection between a row and a column of a database table120corresponds to a data cell125that is capable of storing a data value. Consider an example in which a column corresponds to an “age” attribute and rows correspond to users. Accordingly, the data cell125at the intersection between the column and a particular row may store an age value, such as 56, that is representative of the corresponding user's age. In various embodiments, a database table120may store data for multiple tenants (e.g., users, companies, etc.) of system100. As a result, a subset of all rows of a database table120may correspond to one tenant while another subset of rows corresponds to another tenant. In various embodiments in which a database table120stores data for multiple tenants, that database table120includes a column defining tenant under which each row of database table120specifies a tenant corresponding to that row of data. An example database table120is discussed in greater detail with respect toFIG.2A.

Database node130, in various embodiments, is hardware, software, or a combination thereof capable of providing database services, such as data storage, data retrieval, and/or data manipulation. Such database services may be provided to other components within system100and/or to components external to system100. As an example, database node130may receive a database transaction request from an application server (not shown) that is requesting data to be written to or read from database110. Consequently, database node130may write database records to database110and read records from database110. In various embodiments, database node130maintains database110as a primary or secondary/standby database.

In various embodiments, database110is a standby database that is a database replica of a primary database and thus is capable of serving as a backup database in the event that there is a failure with the primary database. Database110may be maintained as a standby database through a statement-based data replication approach in which database operations performed on a primary database are replayed on database110. As such, database node130may receive database transaction requests having translation logs whose recorded database operations are replayed by database node130with respect to database110. In various embodiments, database110is a primary database against which database node130performs new database writes. These new writes may be recorded in translation logs that database node130sends to other database nodes130so that those database nodes130can perform, on standby databases created based on database110, the database writes that were performed with respect to database110—this can be referred to as “replaying” the transaction log as a database node130performs, at the destination database110, the transactions that were performed at the source database110corresponding to the transaction log. In various embodiments, a database110can become too large and can be split amongst multiple databases110. Accordingly, data that is stored at the original database110can be reproduced at the other databases.

When data is reproduced at one database110based on another database110, in various embodiments, a data integrity procedure can be performed to generate validation information, such as a set of hash values160, that is usable to determine whether the data has been accurately reproduced—that is, that the particular data in both databases110match. As shown, database node130receives a data integrity request135—this request may be received from a user (e.g., an administrator). Data integrity request135, in various embodiments, is a request to generate the validation information and may identify the particular data to be validated and the degree of granularity at which to parse the work in processing the particular data. For example, data integrity request135may request that validation be performed for a single database table120or multiple database tables120, such as all database tables120in database110. Database node130may generate work items140based on data integrity request135.

Work items140, in various embodiments, are designations of work to be performed on a collection of data. For example, a work item140may include information that specifies a set of data cells125for which to generate a hash value160. The amount of data cells125specified in a work item140may be based on the degree of granularity that is specified in data integrity request135. For example, if data integrity request135indicates that one hash value160should be generated per database table120, then database node130may create work items140such that each work item140indicates all data cells125of its corresponding database table120. In various cases where a database table120stores data for multiple tenants, a work item140may specify all data cells125of its corresponding database table120that belong to the tenant that is associated with data integrity request135. In some instances, data integrity request135may indicate other granularities than a hash-per-table granularity. Consider an example in which a database table120is split into multiple partitions. Data integrity request135may indicate that a hash value160should be generated for each partition and as a result, database node130may generate a work item140for each partition. In various embodiments, work items140are stored in one or more queues that are accessible to worker processes150for processing.

Worker processes150, in various embodiments, are sets of threads that are executable to process work items140to generate hash values160. A given worker process150may obtain a work item140from a queue, perform the work item140, and then retrieve another work item140if that queue is not empty. To process a work item140, in various embodiments, a worker process150obtains a set of hash values that corresponds to the set of data cells125associated with the work item140. The worker process150may then perform a set of operations on that set of hash values to generate a hash value160. An example process of generating a hash value160is discussed with respect toFIG.2B. After hash values160have been generated, they may be compared with corresponding hash values160generated based on corresponding database tables120that may be stored in another database110. If the hash values160match, then the particular data has been accurately reproduced; otherwise, there is at least one discrepancy.

Turning now toFIG.2A, a block diagram of an example database table120is shown. In the illustrated embodiment, database table120comprises columns210A-N and rows whose intersections define data cells125. Also as shown, data cells125collectively store data values220AA-220NN. In various embodiments, database table120may be implemented differently than shown. For example, database table120may include additional information, such as hash values derived from data values220of data cells125.

As mentioned, in various embodiments, database node130may store data at database110in the form of database tables120and/or other database objects, such as indexes. In some embodiments, database node130stores data for multiple tenants (e.g., companies, users, etc.) that are hosted by system100. Accordingly, a row of data stored in database table120may be associated with a particular tenant. For example, column210A might correspond to tenant IDs and thus data values220AA-AN may each specify a tenant ID for the tenant that is associated with the corresponding row of that data value220. As such, one or more rows of database table120may be associated with a first tenant while one or more other rows may be associated with a second tenant. When data is reproduced at another database110, in some cases, only data of a particular tenant is reproduced. As a result, the data integrity procedure discussed herein may be performed with respect to data belonging to a particular tenant. Accordingly, the tenant ID associated with each row of data may be used to select those rows that belong to the particular tenant for validation.

Turning now toFIG.2B, a block diagram of an example hashing scheme200that may be implemented by database node130is shown. Hashing scheme200, in various embodiments, involves performing a set of summations to derive a hash value160from a set of corresponding cell hash values230. A cell hash value230, in various embodiments, is a value that is derived by hashing (e.g., performing a MurmurHash function) the data value220of the corresponding data cell125. For example, data value220AA may be hashed in order to derive cell hash value230AA. In various embodiments, a given cell hash value230is calculated as part of database node130performing a database operation to update or add the corresponding data value220to database table120. Database node130may then store that cell hash value230with database table120. Consequently, when hashing scheme200is being implemented, database node130may extract stored cell hash values230associated with database table120by issuing database commands (e.g., SQL commands) for the cell hash values230. In some embodiments, a given cell hash value230is calculated when implementing hashing scheme200. Accordingly, when hashing scheme200is being implemented, database node130may extract data values220of database table120and then hash them to derive corresponding cell hash values230.

In the illustrated embodiment, hashing scheme200is performed in order to generate a hash value160that is representative of database table120. (In various cases, hashing scheme200may be performed using a subset of the data in database table120(e.g., data belonging to a particular tenant) to generate a hash value160that is representative of the subset of data.) As shown in the illustrated embodiment, cell hash values230AA-NN corresponding to data values220AA-NN are summed such that column hash values240A-N are derived. In particular, the cell hash values230associated with a particular column210may be summed together to derive a column hash value240corresponding to that particular column210. Also as shown, column hash values240are summed together to derive a hash value160. Accordingly, in various cases, database node130may sum together cell hash values230of the data being validated to derive a hash value160that is representative of that data. That hash value160may be compared with a hash value160that is derived from the corresponding data that is supposed to match.

Turning now toFIG.3, a block diagram of a data integrity procedure300is shown. In the illustrated embodiment, data integrity procedure300includes a splitter process310, a work item queue320that stores work items140, and worker processes150that produce hash values160and completed work indications330. In some embodiments, data integrity procedure300may be implemented differently than shown. As an example, worker processes150might not produce completed work item indications330.

Splitter process310, in various embodiments, is a set of threads executable to generate work items140. As mentioned, database node130may receive a data integrity request135. In response to receiving that data integrity request, database node130may spawn splitter process310to generate work items140. In various embodiments, splitter process310generates work items140such that the work of generating hash values160for the particular data that is being validated is split in accordance with a specified granularity. As an example, a user may request that all of a particular tenant's data be assessed and that a hash value160be generated per table120. Consequently, splitter process310may generate a work item140per database table120associated with the particular tenant. A work item140may identify a database table120(or a portion thereof) and the particular data of the database table120upon which to generate a hash value160(e.g., the work item140may specify a tenant identifier usable to select rows of that database table120). As an example, data integrity request135may identify a partitioning such splitter process310creates a work item140that corresponds to data cell group315A, a work item140that corresponds to data cell group315B, and a work item140that corresponds to data cell group315C. In some cases, a work item140may specify a database command (e.g., a SQL query) for pulling the data and/or cell hash values230for a database table120. In various embodiments, splitter process310enqueues work items140in a work item queue320that is accessible to work processes150.

As work items140are added to work item queue320, worker processes150may obtain them from work item queue320and process them to generate corresponding hash values160as discussed. In various embodiments, the number of worker processes150spawned to process work items140is based on a requested number of concurrent database operations to utilize in processing a given work item140. As mentioned, in various cases, to process a work item140, a worker process150may issue database commands to perform database operations to extract data or cell hash values230associated with a particular database table120. As an example, a worker process150may issue SQL commands to obtain the cell hash values230associated with data cell group315A. In some embodiments, however, database110is associated with a maximum number of concurrent database operations that is permitted to be performed with respect to database110. For example, 64 concurrent database operations may be permitted to be performed with respect to database110. Accordingly, the number of worker processes150that is spawned may be based a requested number of concurrent database operations to utilize in processing each work item140and the maximum number of concurrent database operations that is permitted to be performed with respect to database110. Consider an example in which data integrity request135specifies that 16 concurrent database operations be utilized per work item140and database110is associated with a maximum of 64 concurrent database operations that can be running against database110at a given point in time. Accordingly, database node130may spawn four worker processes150, each of which may retrieve a work item140and utilize 16 concurrent database operations, totaling 64 concurrent database operations.

After processing a work item140to generate a hash value160, the worker process150may store the hash value160and retrieve another work item140from work item queue320if work item queue320is not empty. Worker process150may continually grab and process work items140until work item queue320is empty. In some embodiments, after processing a work item140, the worker process150stores a completed work item indication330indicating that the work item140has been processed/completed. Accordingly, in the event that data integrity procedure300fails to complete (e.g., a system crash), database node130may use completed work items indications330to resume implementation of data integrity procedure300without having to restart from the beginning. As such, completed work item indications330may be stored at a location that is separate from database node130so that if database node130crashes those completed work item indications330are not lost.

Turning now toFIG.4, a block diagram of an example procedure to determine whether corresponding database tables120in separate databases110include the same information. In the illustrated embodiment, database110A is located on a network local to database node130A and stores database tables120A while database110B is located on a network that is remote to database node130A and stores database tables120B managed by database node130B. In some cases, database110A might be a database that is managed by a company using its own servers while database110B stores data for the company, but is managed by a cloud provider, such as AWS® or Azure®. The illustrated embodiment may be implemented differently than shown. For example, database tables120A and120B might be stored in the same database110or in separate databases110that are on the same local network.

As shown, database nodes130A and130B each receive a data integrity request135. In some cases, data integrity requests135may be received in response to an administrator logging into the corresponding database systems and submitting the requests135(e.g., via a command line tool). In some cases, the data integrity request135that is received by database node130A may cause database node130A to transmit the data integrity request135received by database node130B (or vice versa). In yet some cases, database nodes130may implement data integrity procedure300(without receiving a data integrity request135) in response to the occurrence of a particular event, such as a new version of the database.

In response to receiving the data integrity requests135, database nodes130A and130B may then generate hash values160A and160B, respectively, based on corresponding database tables120A and120B. In various embodiments, the two data integrity requests135specify the same configuration parameters for implementing data integrity procedure300. The parameters may specify, for example, the same partitioning of the data values220that are being validated (e.g., one work item140per table120, one work item140per shard of a table120, etc.), the same number of concurrent database operations to be utilized in processing a work item140(alternatively, the same number of processes150to spawn), and the version of the database to be used when performing data integrity procedure300.

In various embodiments, during the operation of a given database110, a database node130may store state information (e.g., database snapshots) that identifies the state of that given database110at different points in its lifecycle. For example, prior to performing a batch write to a database110, database node130may store state information that allows for database node130to access, after performing the batch write, a prior version of the database110that existed before the batch write. As a result, in various embodiments, a database node130can view the data that was stored at a database110at a particular point in time. Due to the constant changing nature of the data of a database110, data integrity request135may specify a particular version of database110to be used when performing data integrity procedure300. Accordingly, in the illustrated embodiment, database nodes130A and130B may generate hash values160based on the same version/checkpoint—this version may not correspond to the current version since database110A may be ahead of database110B as initial writes to database110A may be later replayed at database110B.

After generating hash values160A and160B, in various embodiments, database nodes130A and130B provide them to a user as results410A and410B. The user may compare hash values160A and160B to determine whether they indicate that particular data of database tables120A is the same as the corresponding data in database tables120B. Instead of providing hash values160to a user, in some embodiments, database node130B provides hash values160B to database node130A as result410B (or vice versa). Accordingly, database node130A may then compare hash values160A and160B to determine whether they match. Database node130A may indicate, as part of result410A, whether the particular data of database tables120A is the same as the corresponding data in database tables120B.

Turning now toFIG.5, a flow diagram of a method500is shown. Method500is one embodiment of a method performed by a computer system (e.g., database node130) to generate information (e.g., hash values160) that is usable to determine whether different database tables (e.g., database tables120) store the same data (e.g., data values220). In various cases, method500may be performed by executing a set of program instructions stored on a non-transitory computer-readable medium. In some embodiments, method500may include additional steps than shown. For example, method500may include a step in which the computer system sends the generated information to another computer system.

Method500begins in step510with the computer system receiving a data integrity request (e.g., a data integrity request135) for a first set of database tables. In some cases, the first set of database tables may be stored in a database (e.g., database110) capable of accessing different versions of the first set of database tables. The different versions of the first set of database tables may be replicated based on a second set of database tables. In some cases, the first set of database tables may be stored in a database that is located on a network local to the computer system and the second set of database tables may be stored in a database located on a network remote to the computer system.

In step520, the computer system generates at least two work items (e.g., work items140) that correspond to respective data cell groups (e.g., data cell groups315) in the first set of database tables. In some cases, the at least two work items may be generated such that each work item corresponds to a data cell group of a different respective database table of the first set of database tables—that is, one work item per database table. In some cases, the at least two work items may be generated such that a first work item corresponds to a first data cell group (e.g., data cell group315A) of a first database table and a second work item corresponds to a second data cell group (e.g., data cell group315B) of the same first database table. The computer system may store the at least two work items in a work item queue (e.g., work item queue320) that is accessible to a plurality of processes (e.g., worker processes150) that may retrieve and perform work items until the work item queue is empty.

In step530, the computer system causes the plurality of processes to perform the at least two work items to generate a first plurality of hash values that includes hash values for the respective data cell groups. A particular one of the first plurality of hash values may be derived by summing hash values (e.g., cell hash values230) mapped to data cells of a data cell group corresponding to the particular hash value. The particular process that performs the work item to generate the particular hash value may obtain the hash values mapped to the data cells by issuing a set of database queries for those hash values against the database that stores the first set of database tables.

The number of the plurality of processes spawned by the computer system may be based on a maximum number of concurrent database operations (e.g., 64 concurrent database operations) permitted by a database that stores the first set of database tables. In some cases, the computer system may receive, from a user via a user device, information that specifies a number of concurrent database operations to utilize for performance of a work item (e.g., 16 operations). The number of the plurality of processes spawned may be derived by dividing the maximum number of concurrent database operations by the number of concurrent database operations to utilize per work item (e.g., 64/16=4, thus four processes may be spawned).

The first plurality of hash values may be usable to compare with corresponding ones of a second plurality of hash values generated based on corresponding data cell groups in the second set of database tables replicated from the first set of database tables. The first and second pluralities of hash values may be generated based on the same non-current version of the first and second sets of database tables.

In some cases, the computer system may receive, from another computer system that manages a database that stores the second set of database tables, the second plurality of hash values generated based on the corresponding data cell groups in the second set of database tables. The computer system may perform a comparison between the first and second sets of hash values and, based on the comparison, return a response (e.g., a result410) to the data integrity request that indicates whether the data cell groups in the first set of database tables store the same information as the corresponding data cell groups in the second set of database tables. In yet some cases, the computer system may return, to an entity (e.g., an administrator) associated with the data integrity request, a response that includes the first plurality of hash values to enable the entity to perform a comparison between the first and second sets of hash values to determine whether the data cell groups in the first set of database tables store the same information as the corresponding data cell groups in the second set of database tables.

Turning now toFIG.6, a flow diagram of a method600is shown. Method600is one embodiment of a method performed by a computer system (e.g., database node130) to generate information (e.g., hash values160) that is usable to determine whether different database tables (e.g., database tables120) store the same data (e.g., data values220). In various cases, method600may be performed by executing a set of program instructions stored on a non-transitory computer-readable medium. In some embodiments, method600may include additional steps than shown. For example, method600may include a step in which the computer system sends the generated information to another computer system.

Method600begins in step610with the computer system receiving a request (e.g., a data integrity request135) to perform a data integrity procedure (e.g., data integrity procedure300) to generate validation information that is usable to determine whether particular data of a first set of database tables matches corresponding data in a second set of database tables. The data integrity procedure may be performed with respect to a non-current version of the first set of database tables that is specified by the request. In some cases, the particular data corresponds to a particular tenant of a plurality of tenants of the computer system that have data stored in the first set of database tables.

In step620, the computer system performs the data integrity procedure. As part of performing the data integrity procedure, in step622, the computer system generates a plurality of work items (e.g., work items140) based on a partitioning of the particular data (e.g., partitioned into data cell groups315). A particular work item may correspond to a set of data cells included in the first set of database tables. As part of performing the data integrity procedure, in step624, the computer system causes a plurality of processes (e.g., worker processes150) to concurrently perform ones of the plurality of work items to generate a first plurality of hash values. A particular hash value may be derived from a set of hash values (e.g., cell hash values230) that correspond to the set of data cells associated with the particular work item. The validation information may include the first plurality of hash values that are comparable with a second plurality of hash values generated based on the second set of database tables to determine whether the particular data matches the corresponding data in the second set of database tables. In some instances, the computer system sends the validation information to another computer system that is capable of performing a comparison between the first and second pluralities of hash values to determine whether the particular data matches the corresponding data in the second set of database tables.

In some embodiments, the plurality of processes store a set of indications of completed work items (e.g., completed work item indications330). In response to the data integrity procedure failing to be completed, the computer system may initiate the data integrity procedure such that only those ones of the plurality of work items that are not indicated by the set of indications of completed work items are performed, and wherein the plurality of work items correspond to sets of data cells associated with the particular tenant and not other ones of the plurality of tenants.

Exemplary Computer System

Turning now toFIG.7, a block diagram of an exemplary computer system700, which may implement system100, database110, or database node130, is depicted. Computer system700includes a processor subsystem780that is coupled to a system memory720and I/O interfaces(s)740via an interconnect760(e.g., a system bus). I/O interface(s)740is coupled to one or more I/O devices750. Computer system700may be any of various types of devices, including, but not limited to, a server system, personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, tablet computer, handheld computer, workstation, network computer, a consumer device such as a mobile phone, music player, or personal data assistant (PDA). Although a single computer system700is shown inFIG.7for convenience, system700may also be implemented as two or more computer systems operating together.

Processor subsystem780may include one or more processors or processing units. In various embodiments of computer system700, multiple instances of processor subsystem780may be coupled to interconnect760. In various embodiments, processor subsystem780(or each processor unit within780) may contain a cache or other form of on-board memory.

System memory720is usable store program instructions executable by processor subsystem780to cause system700perform various operations described herein. System memory720may be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM—SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read only memory (PROM, EEPROM, etc.), and so on. Memory in computer system700is not limited to primary storage such as memory720. Rather, computer system700may also include other forms of storage such as cache memory in processor subsystem780and secondary storage on I/O Devices750(e.g., a hard drive, storage array, etc.). In some embodiments, these other forms of storage may also store program instructions executable by processor subsystem780. In some embodiments, program instructions that when executed implement database110, database node130, worker processes150, data integrity procedure300, splitter process310may be included/stored within system memory720.

I/O interfaces740may be any of various types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I/O interface740is a bridge chip (e.g., Southbridge) from a front-side to one or more back-side buses. I/O interfaces740may be coupled to one or more I/O devices750via one or more corresponding buses or other interfaces. Examples of I/O devices750include storage devices (hard drive, optical drive, removable flash drive, storage array, SAN, or their associated controller), network interface devices (e.g., to a local or wide-area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system700is coupled to a network via a network interface device750(e.g., configured to communicate over WiFi, Bluetooth, Ethernet, etc.).