Patent Description:
Methods, systems, apparatuses, and computer-readable storage mediums described herein are for detecting data corruption in a distributed data set system. For example, a system in accordance with an embodiment comprises one or more compute nodes for processing queries with respect to a distributed data set (e.g., a database) comprising a plurality of storage segments (e.g., pages). A write transaction resulting from a query with respect to a particular storage segment is logged in a log record that describes a modification to the storage segment. Each log record is identified by a log sequence number associated therewith. A log service provides the log record to a data server managing a portion of the distributed data set in which the storage segment is included, which performs the write transaction with respect to the storage segment as specified by the log record. For redundancy purposes, the data server has one or more replicas that manage respective replicas of the portion of the distributed data set managed thereby. For backup purposes, snapshots of each of the replicas are periodically generated. To determine a data corruption, a snapshot of one replica is automatically cross-validated with a snapshot of another replica. For example, log sequence numbers of the storage segments of one replica are compared to log sequence numbers of corresponding storage segments of the other replica. If a log sequence mismatch is detected, this means that an inconsistency exists between the log records applied by one replica data server and the log records applied by another replica data server. Thus, a different set of write transactions were applied by both page server replicas, thereby resulting in inconsistent snapshots. In the event that such a data corruption is detected, an alert is automatically generated to notify an administrator of the existence of the data corruption.

Further features and advantages, as well as the structure and operation of various example embodiments, are described in detail below with reference to the accompanying drawings. It is noted that the example implementations are not limited to the specific embodiments described herein. Such example embodiments are presented herein for illustrative purposes only. Additional implementations will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.

The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate example embodiments of the present application and, together with the description, further serve to explain the principles of the example embodiments and to enable a person skilled in the pertinent art to make and use the example embodiments.

The features and advantages of the implementations described herein will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout.

The present specification and accompanying drawings disclose numerous example implementations. The scope of the present application is not limited to the disclosed implementations, but also encompasses combinations of the disclosed implementations, as well as modifications to the disclosed implementations. References in the specification to "one implementation," "an implementation," "an example embodiment," "example implementation," or the like, indicate that the implementation described may include a particular feature, structure, or characteristic, but every implementation may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an implementation, it is submitted that it is within the knowledge of persons skilled in the relevant art(s) to implement such feature, structure, or characteristic in connection with other implementations whether or not explicitly described.

In the discussion, unless otherwise stated, adjectives such as "substantially" and "about" modifying a condition or relationship characteristic of a feature or features of an implementation of the disclosure, should be understood to mean that the condition or characteristic is defined to within tolerances that are acceptable for operation of the implementation for an application for which it is intended.

Numerous example embodiments are described as follows. Implementations are described throughout this document, and any type of implementation may be included under any section/subsection. Furthermore, implementations disclosed in any section/subsection may be combined with any other implementations described in the same section/subsection and/or a different section/subsection in any manner.

Embodiments described herein are directed to detecting data corruption in a distributed data set system. For example, a system in accordance with an embodiment comprises one or more compute node for processing queries with respect to a distributed data set (e.g., a database) comprising a plurality of storage segments (e.g., pages). A write transaction resulting from a query with respect to a particular storage segment is logged in a log record that describes a modification to the storage segment. Each log record is identified by a log sequence number associated therewith. A log service provides the log record to a data server managing a portion of the distributed data set in which the storage segment is included, which performs the write transaction with respect to the storage segment as specified by the log record. For redundancy purposes, the data server has one or more replicas that manage respective replicas of the portion of the distributed data set managed thereby. For backup purposes, snapshots of each of the replicas are periodically generated. To determine a data corruption, a snapshot of one replica is cross-validated with a snapshot of another replica. For example, log sequence numbers of the storage segments of one replica are compared to log sequence numbers of corresponding storage segments of the other replica. If a log sequence mismatch is detected, this means that an inconsistency exists between the log records applied by one replica data server and the log records applied by another replica data server. Thus, a different set of write transactions were applied by both page server replicas, thereby resulting in inconsistent snapshots. In the event that such a data corruption is detected, an alert is automatically generated to notify an administrator of the existence of the data corruption.

Conventionally, a corruption is detected at the time the data server attempts to apply a log record to perform a write transaction with respect to the storage segment that it manages. At the time of performing the write transaction, the data server analyzes a field in the log record (also known as the "previous page log sequence number field" that indicates a log sequence number that should be included in the header of the storage segment to be modified. If that log sequence number is not included in the header, a determination is made that a data corruption exists. A problem with this approach is that it can be some time before a data server applies a log record, especially in instances when that storage segment is only being read. In certain instances, it can be weeks before the data server updates the storage segment. Thus, if the storage segment is not modified for a long time, the corruption remains undetected for a relatively long amount of time. During this time, applications will operate on the corrupt data, and therefore, return invalid results.

The embodiments described herein advantageously detect data corruption more quickly than conventional techniques by performing the cross-validation between replica snapshots, thereby enabling the corruptions to be acted on much sooner (as one no longer has to wait for the data to be modified to detect a data corruption). This limits the time window in which applications operate on the corrupt data. Thus, the integrity of the distributed data set system is advantageously improved, and the applications accessing the data set are more likely to operate on the correct data. Moreover, because such data corruptions are often a result of software bugs, the foregoing techniques may be implemented during the development and testing of new features of the distributed data set system. That way, data corruptions resulting from these new features may be detected, and the code causing these data corruptions may be fixed before the features are rolled out to the end users.

For example, <FIG> shows a block diagram of a system <NUM> for detecting a data corruption in a distributed data set system, according to an example embodiment. As shown in <FIG>, system <NUM> comprises one or more primary compute nodes <NUM>, secondary compute nodes 104A-104N, a persistent log <NUM>, a log service <NUM>, data servers 110A-110N, replica data servers 112A-112N, data sets 114A-114N, replica data sets 116A-116N, data set snapshots 118A-118N, replica data set snapshots 120A-120N, and snapshot service <NUM>. System <NUM> may be implemented in a cloud-based environment, although the embodiments described herein are not so limited. Each of primary compute node(s) and secondary compute nodes 104A-104N may comprise one or more physical computing devices or virtual machines. Examples of physical computing devices, include, but are not limited to, server computers, server systems, etc. Data sets 114A-114N, replica data sets 116A-116N, data set snapshots 118A-118N, and replica data set snapshots 120A-120N may be stored via storage nodes, each comprising a plurality of physical storage disks (e.g., hard disk drives, solid state drives, etc.) that are accessible by respective data servers of data servers 110A-110N and replica data servers 112A-112N, for example, via a network.

As shown in <FIG>, primary compute node(s) <NUM> are configured to execute one or more applications, such as database application <NUM>. Database application <NUM> may be any database server application, including, but not limited to Microsoft® Azure SQL Database™ published by Microsoft® Corporation of Redmond, Washington. Database application <NUM> is configured to execute statements to create, modify, and delete data file(s) based on an incoming query. Queries may be user-initiated or automatically generated by one or more background processes. Such queries may be configured to add data file(s), merge data file(s) into a larger data file, re-organize (or re-cluster) data file(s) (e.g., based on a commonality of data file(s)) within a particular set of data file, delete data file(s) (e.g., via a garbage collection process that periodically deletes unwanted or obsolete data), etc..

Each of secondary compute nodes 104A-104N may also be configured to execute an instance of database application <NUM>. One or more of secondary compute nodes 104A-104N may be utilized if one or more of primary compute node(s) <NUM> fails and recovery is not efficiently possible. In such an instance, one or more of secondary compute nodes 104A-104N is promoted to be a primary compute node and/or a new secondary compute node may be added to replace the promoted secondary compute node. It is noted that the secondary compute nodes may be added or removed regardless of whether a primary compute node failing.

Each of data sets 114A-114N include databases and/or the like, in embodiments, which may be very large data sets such as for "Big Data" analytics and/or data warehousing. It is contemplated herein that one or more of data sets 114A-114N are to the order of petabytes, or more, in embodiments. Data sets 114A-114N may be logically represented as structured, relational data, organized as rows of tables, having columns for the data. The data of data sets 114A-114N may be stored in one or more data files. Each of the data files may be logically divided into a storage segment, such as a page. The page may be <NUM> kilobytes in size, although the embodiments described herein are not so limited.

Data servers 110A-110N are configured to serve storage segments of data sets 114A-114N to primary compute node(s) <NUM> and update storage segments of data sets 114A-114N as transactions update data. Each of data servers 110A-110N is responsible for a subset of storage segments in the database. For example, data server 110A is responsible for storage segments of data sets 114A, data server 110B is responsible for storage segments of data sets 114B, data server 110C is responsible for storage segments of data sets 114C, and data server 110N is responsible for storage segments of data sets 114N. In accordance with an embodiment in which a storage segment corresponds to a page, each of data servers 110A-110N may be referred to as a page server.

System <NUM> also comprises one or more replicas for each of data servers 110A-110N, which are maintained for redundancy and availability. For example, as shown in <FIG>, system <NUM> comprises one or more replica data servers 112A (which are replicas of data server 110A), one or more replica data servers 112B (which are replicas of data server 110B), one or more replica data servers 112C (which are replicas of data server 110C), and one or more replica data servers 112N (which are replicas of data server 110N). Each of replica data servers 112A-112N is associated with a replica data set. For example, replica data server(s) 112A are associated with replica data set 116A (which is a replica of data set 114A), replica data server(s) 112B are associated with replica data set 116B (which is a replica of data set 114B), replica data server(s) 112C are associated with replica data set 116C (which is a replica of data set 114C), and replica data server(s) 112D are associated with replica data set 116D (which is a replica of data set 114D).

When performing a write transaction, such as creating or modifying a data file, primary compute node(s) <NUM> logs the operation (shown as <NUM>) into a persistent log <NUM>. Persistent log <NUM> is utilized to recover data in the event of a system failure. For example, database application <NUM> of primary compute node(s) <NUM> may first read a particular storage segment from a data server of data servers 110A-110N that is responsible for that storage segment. This ensures that a copy of the storage segment is brought into a cache of primary compute node(s) <NUM>. The storage segment may have been previously read by primary compute node(s) <NUM>, in which case that storage segment will already be in the cache of primary compute node(s) <NUM>. To obtain the storage segment, database application <NUM> of primary compute node(s) <NUM> may issue a read operation to the responsible data server (e.g., data server 110B), and the data server provides the data (shown as data <NUM>) to primary compute node(s) <NUM>.

Database application <NUM> of primary compute node(s) <NUM> may then write to the read data stored in the cache. Database application <NUM> also creates a log record of that write operation. The log record includes the incremental change made or to be made as part of the write operation. The log record also includes a log sequence number and a storage segment identifier. For instance, <FIG> depicts a log record <NUM> in accordance with an example embodiment. As shown in <FIG>, log record <NUM> includes a log sequence number <NUM>, a description <NUM> of an incremental change made via a write transaction, and a storage segment identifier <NUM>. Log sequence number <NUM> is a number that uniquely identifies a log record (e.g., log record <NUM>), and that represents a position of the log record within a log (e.g., persistent log <NUM>). Log sequence number <NUM> is typically a monotonically increasing number such that the higher the log sequence number, the more recent the log record was placed within the log. Storage segment identifier <NUM> identifies a location of the storage segment within one of data sets 114A-114N to which the write transaction is to be applied.

In accordance with an embodiment, log records are organized into blocks. A block is a unit of storage that may be written to and read from atomically (i.e., at one time-such that the write or read is either completed for the entire block, or abandoned with no write being performed at all). A typical block size will be constant in a computing system, but example block sizes may be any multiple of the size of the disk sector (e.g., multiples of <NUM> bytes or <NUM> bytes, depending on the disk type). Because a log record may be much smaller than a block, a block may include multiple log records. For example, <FIG> depicts a block <NUM> populated with multiple log records <NUM>, <NUM>, <NUM>, and <NUM> in accordance with an example embodiment. As an example, log record <NUM> of <FIG> may be log record <NUM> of <FIG>. Each of log records <NUM>, <NUM>, <NUM>, and <NUM> may occupy a slot within block <NUM>. When writing the log record to persistent log <NUM> (as shown in <FIG>), it may be a block of log records (such as block <NUM>) that is written to persistent log <NUM>. Primary compute node(s) <NUM> may wait for some number of log records to be generated before writing the log records as a block to the persistent log <NUM>.

Log service <NUM> may execute on one or more server computers or computing devices, or alternatively, a virtual machine. Log service <NUM> assists write operations that are performed and logged by the primary compute node(s) <NUM> to be propagated to the appropriate data server of data servers 110A-110N and its corresponding replica of replica data servers 112A-112N. For example, when determining that a particular log record is to be applied to one of data sets 114A-114N, log service <NUM> identifies a particular data server of data servers 110A-110N that is assigned to the data set of data sets 114A-114N that includes the particular storage segment to be written to via storage segment identifier <NUM>. Log service <NUM> may also identify a particular replica data server of replica data servers 112A-110N corresponding to the particular data server that is responsible for the particular storage segment in the corresponding replica data set of replica data sets 116A-116N. In the case in which the set of storage segments are contiguous in an address space, the assignment of the set of storage segments to each data server may be made to align with the most significant bits of the storage segment identifier. For instance, suppose that the storage segment identifiers range from binary <NUM><NUM> to binary <NUM><NUM>. In this case, there might be seven data servers, one for storage segments having identifiers with the four most significant bits being <NUM>, another for the storage segments having identifiers with the four most significant four bits being <NUM>, and so on, until the seventh storage segment for storage segments having identifiers with the four most significant bits being <NUM>. Thus, determining which data server is assigned to store a particular storage segment becomes an efficient computation. Log service <NUM> may thus determine the appropriate data server of data servers 110A-110N for the log record (e.g., log record <NUM>) based on the storage segment identifier (e.g., storage segment identifier <NUM>) included in the log record. Log service <NUM> may maintain a mapping between data servers 110A-110N and replica data servers 112A-112N. Thus, when log service <NUM> determines which data server to provide the log record, log service <NUM> easily determines which replica data server corresponds thereto.

Then, the data server, and corresponding replica data server, are caused to perform the particular write transaction (as specified in the log record) to the particular storage segment (designated in the particular log record) of a corresponding data set of data sets 114A-114N (and corresponding data set of replica data sets 116A-116N). For instance, suppose log service <NUM> determines that data server 110C is responsible for the storage segment identified by storage segment identifier <NUM>. In this example, log service <NUM> provides a request <NUM> comprising the log record to data server 110C and replica data server 112C. Data server 110C and replica data server <NUM> performs the write transaction specified in the received log record (referred to as re-doing the log record) with respect the storage segment identified by the received log record. This would represent the completion of the write transaction that began when the primary compute node(s) <NUM> read that storage segment (e.g., data <NUM>) from the data server 110B. In another example, rather than log service <NUM> by providing request <NUM>, data servers 110A-110N and data servers 112A-112N may query (e.g., periodically) log service <NUM> for the appropriate log records. Thus, the providing of the appropriate log record to a data server may be in response to the request from a data server.

As data servers 110A-110N (and replica data servers 112A-112N) apply changes to storage segments of their respective data sets, data servers 110A-110N (and replica data servers 112A-112N) also update a header of the storage segment. The header may comprise various fields describing various attributes of the storage segment. For example, the header may comprise a storage segment identifier (e.g., storage segment identifier <NUM>, as shown in <FIG>), and a log sequence number (e.g., log sequence number <NUM>, as shown in <FIG>) for that storage segment. In an embodiment in which a storage segment is a page, the storage segment identifier and the log sequence number may be referred to as a page identifier and a page log sequence number. In accordance with such an embodiment, each page identifier is a multiple of <NUM> kilobytes, where a first page is located at offset <NUM>, the next page is located at an offset of <NUM> kilobytes, a third page is located at an offset of <NUM> kilobytes, etc. The log sequence number stored in the header of a particular storage segment identifies a corresponding log record (e.g., log record <NUM>) that identifies the latest modification made to the storage segment. The log sequence number is updated each time a modification is made to the storage segment.

Log service <NUM> may provide log records one at a time, or alternatively, one block at a time. For instance, if log service <NUM> determines that there are log records (e.g., any of a block of log records that have a storage segment identifier within the set of storage segments assigned to a particular storage segment server_, log service <NUM> may send the entire block to the particular data server.

In addition, log service <NUM> may ensure that the log record is not communicated to any entity until log service <NUM> has been notified (e.g., by database application <NUM>) that the log record has been securely written into persistent log <NUM>. This helps the recovery process be consistent. During recovery, the recovery process uses persistent log <NUM> to determine what operations have been completed. If other components in the system (e.g., a secondary compute node of secondary compute nodes 104A-104N or any of data servers 110A-110N) have performed operations that persistent log <NUM> is not aware of, then the recovery will fail to achieve a reliable state. Data sets 114A-114N then become ambiguous, and thus corrupt.

Log service <NUM> may provide log records to secondary compute nodes 104A-104N, which update their respective caches in accordance with the incremental changes described in the log records. For example, log service <NUM> may provide the corresponding log record via requests 107A-107N to secondary compute nodes 104A-104N. While the log record may be pushed to secondary compute nodes 104A-104N, log service <NUM> may likewise handle requests for log records (e.g., from secondary compute nodes 104A-104N). As described above, log service <NUM> may ensure that log records are not communicated to the secondary compute systems 104A-104N unless the log record is already confirmed as written to persistent log <NUM>. In accordance with an embodiment, secondary compute nodes 104A-104N may be configured to ignore the log record if it is for writing to a storage segment that is not already cached (and thus would cause a read from a data server). In that case, if secondary compute nodes 104A-104N were to use that storage segment later, secondary compute nodes 104A-104N may read that storage segment from the data server (which already has the write of the skipped log record applied).

While secondary compute nodes 104A-104N primary act as hot standby nodes for failover purposes, in accordance with an embodiment, secondary compute nodes 104A-104N may act as read-only compute nodes for offloading read workloads. That is, secondary compute nodes 104A-104N may be utilized to read data sets 114A-114N via issuing read commands to data servers 110A-110N, which in turn, retrieve the data from data sets 114A-114N, respectively, and returns the retrieved data to secondary compute nodes 104A-104N.

Snapshot service <NUM> may execute on one or more server computers or computing devices, or alternatively, a virtual machine. Snapshot service <NUM> is configured to generate snapshots of data sets 114A-114N and replica data sets 116A-116N. For example, snapshot service <NUM> may generate data set snapshot 118A for data set 114A, data set snapshot 118B for data set 114B, data set snapshot 118C for data set 114C, data set snapshot 118D for data set 114D, replica data set snapshot(s) 120A for replica data set(s) 116A, replica data set snapshot(s) 120B for replica data set(s) 116B, replica data set snapshot(s) 120C for replica data set(s) 116C, and replica data set snapshot(s) 120D for replica data set(s) 116D. Data set snapshots 118A-118N and replica data set snapshots 120A-120N are utilized as backups.

Snapshot service <NUM> may generate any number of snapshots for a given data set of data sets 114A-114N and for a given replica data set of replica data sets 116A-116N. For instance, snapshot service <NUM> may periodically (every hour, every day, every week, etc.) generate snapshots, thereby enabling data to be restored at various points in time in accordance with a backup retention period implemented for data sets 114A-114N and/or replica data sets 116A-116N.

When generating a snapshot for a particular data set, snapshot service <NUM> logs the log sequence number of the first log record (in persistent log <NUM>) not yet redone at the time the snapshot is generated. That is, snapshot service <NUM> determines the first log record that has not yet been written to the data set by its corresponding data server. Snapshot service <NUM> associates the log sequence number of that log record with the snapshot at the time of generation. For instance, snapshot service <NUM> may store the log sequence number as metadata of the generated snapshot. Such a log sequence number is referred herein as the begin log sequence number of the snapshot. Snapshot service <NUM> may also store a timestamp representative of a time at which the snapshot was generated in the metadata of that snapshot.

In certain instances, a data server and/or one or more of its replicas may inadvertently skip the application of a log record (e.g., due to software bugs, hardware crashes, etc.). In such instances, a data corruption is created, as a data set and/or one or more of its replicas have inconsistent versions of the data. The detection of such data corruptions should be detected as soon as possible to avoid applications from utilizing incorrect data. The techniques described herein detect data corruption by cross-validating snapshots for a given data set.

To cross-validate snapshots, a snapshot pair is first determined. For example, <FIG> depicts a system <NUM> for determining a snapshot pair in accordance with an example embodiment. As shown in <FIG>, system <NUM> comprises a validation manager <NUM>, a first replica data server 404A, a second replica data server 404B, a first replica data set 406A, a second replica data set 406B, first replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>, second replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>, and a snapshot service <NUM>. Replica data server 404A and replica data server 404B are replicas of a page server. For example, replica data server 404A and replica data server 404B are examples of replica data server(s) 112A, which are replicas of data server 110A. Replica data set 406A and replica data set 406B are replicas of a data set managed by the page server associated with replica data servers 404A and 404B. For example, replica data set 406A and replica data set 406B are examples of replica data set(s) 116A, which are replicas of data set 114A. Snapshot service <NUM> is an example of snapshot service <NUM>, as described above with reference to <FIG>. As shown in <FIG>, snapshot service <NUM> has generated four snapshots of replica data set 406A (replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>) and has generated four snapshots of replica data set 406B 406A (replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>). Each of replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> represent snapshots of replica data set 406A that were generated at different times. Similarly, each of replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> represent snapshots of replica data set 406B that were generated at different times.

Validation manager <NUM> is configured to determine a snapshot pair between a first set of snapshots generated for replica data set 406A (i.e., snapshots <NUM>, <NUM>, <NUM>, and <NUM>) and a second set of snapshots generated replica data set 406B (i.e., snapshots <NUM>, <NUM>, <NUM>, and <NUM>). To determine the snapshot pair, validation manager <NUM> determines the most recent snapshot from one of the replica snapshots generated for replica data set 406A or from one of the replica snapshots generated for replica data set 406B. The most recent snapshot may be determined based on a timestamp associated with the replica snapshots. For instance, validation manager <NUM> may designate replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> as a primary list of replica snapshots and determine the most recent snapshot generated therefrom based on their respective timestamps. The replica snapshot of replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> having the most recent timestamp may be designated as the primary snapshot.

After determining the primary snapshot, validation manager <NUM> may determine replica snapshot from replica snapshots <NUM>, <NUM>, <NUM> and <NUM> being closest in time to the primary snapshot. For example, validation manager <NUM> may compare the timestamps of each of replica snapshots <NUM>, <NUM>, <NUM> and <NUM> to the timestamp of the primary snapshot and determine which of the timestamps is closest in time to the timestamp of the primary snapshot. The replica snapshot having the timestamp closest to the timestamp of the primary snapshot is designated as the secondary snapshot. Validation manager <NUM> attempts to select the nearest snapshot to the primary snapshot to minimize the amount of unvalidatable pages because a snapshot pair that is much newer will experience many more write transactions and have higher storage segment log sequence numbers for corresponding storage segments, thereby making them unvalidatable (as will be described below). The determined primary snapshot and the secondary snapshot are designated to be the snapshot pair to be cross-validated. It is noted that in other embodiments, replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> may designated as the primary list, that a primary snapshot may be determined therefrom, and that a secondary snapshot may be determined from among replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>.

Once the snapshot pair is determined, validation manager <NUM> transitions to the cross-validation phase. In this phase, validation manager <NUM> compares various characteristics of the primary snapshot to characteristics of the secondary snapshot to determine a data corruption. <FIG> depicts a system <NUM> for cross-validating a snapshot pair to detect a data corruption in accordance with an example embodiment. As shown in <FIG>, system <NUM> comprises a validation manager <NUM>, a primary replica snapshot <NUM>, and a second replica snapshot <NUM>. Validation manager <NUM> is an example of validation engine <NUM>, as described above with reference to <FIG>. Primary replica snapshot <NUM> represents a determined primary snapshot (e.g., from among replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>, as described above with reference to <FIG>). Secondary replica snapshot <NUM> represents a determined secondary snapshot (e.g., from among replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>, as described above with reference to <FIG>). Validation manager <NUM> comprises an input/output (IO) reader <NUM>, a storage segment checker <NUM>, a cross-validator <NUM>, and an alert generator <NUM>.

IO unit reader <NUM> is configured to read a predetermined amount (or "IO unit") of data from each of primary replica snapshot <NUM> and secondary replica snapshot <NUM> (shown as IO units <NUM> and <NUM>, respectively, where each IO unit comprises a plurality of storage segments (e.g., each being <NUM> kilobytes). In accordance with an embodiment, the IO unit size is <NUM> megabytes. IO unit reader <NUM> provides IO units <NUM> and <NUM> to storage segment checker <NUM>.

Storage segment checker <NUM> is configured to analyze each storage segment in each of IO units <NUM> and <NUM> and determine which storage segments therein should be cross-validated. For example, for each storage segments in IO units <NUM> and <NUM>, storage segment checker <NUM> may determine whether the storage segment comprises all zeroes. Such storage segments are designated by storage segment checker <NUM> as being corrupted.

In accordance with an embodiment, storage segments may be encrypted for security purposes. Storage segment checker <NUM> may be configured to decrypt each of the storage segments in IO units <NUM> and <NUM>. If the decryption fails for certain storage segments, storage segment checker <NUM> determines that such storage segments are corrupt and designates these storage segments as such.

Storage segment checker <NUM> may also be configured to perform one or more logical consistency checks on each storage segment of IO units <NUM> and <NUM>. If any of the logical consistency check(s) fail for a particular storage segment, storage segment checker <NUM> designates the storage segment as being corrupt. Examples of logical consistency checks include, but are not limited to, checksum-based checks, torn page-based checks, short transfer-based checks, stale read-based checks, page audit failure-based checks, etc..

Storage segment checker <NUM> may also be configured to determine whether a particular storage segment of IO unit <NUM> and/or IO unit <NUM> is designated as being free. If a particular storage segment of an IO unit is designated as being free, validation for that storage segment and its corresponding storage segment in the other IO unit is skipped. Storage segment checker <NUM> designates such storage segments as being unavailable for cross-validation.

Any storage segment that is not designated as being corrupted or unavailable may be left undesignated, or alternatively, may be designated as being validateable. After storage segment checker <NUM> completes its various checks, storage segment checker <NUM> provides IO units <NUM> and <NUM> to cross-validator <NUM>.

Cross-validator <NUM> is configured to cross-validate storage segments of IO unit <NUM> with corresponding storage segments of IO unit <NUM> that are located at the same logical position. Cross-validator <NUM> only validates storage segments that are not designated as being corrupted or unavailable. Any storage segment designated as being corrupted or unavailable are skipped, as these storage segments are not validateable with complete certainty. For instance, cross-validator <NUM> may initially compare the storage segment identifier of a first storage segment from IO unit <NUM> to the storage segment identifier of a corresponding first storage segment from IO unit <NUM> (i.e., a storage segment located at the same offset) (assuming both are not designated as being corrupted or unavailable). If the storage segment identifiers do not match, cross-validator <NUM> determines that a data corruption has been detected between the two storage segments. Cross-validator <NUM> performs this check for each storage segment of IO units <NUM> and <NUM> that are not designated as being corrupted or unavailable.

If the storage segment identifies match, cross-validator <NUM> then compares the storage segment log sequence number of storage segments of IO unit <NUM> (that are not designated as being corrupted or unavailable) to storage segment log sequence numbers of corresponding storage segments of IO unit <NUM> (that are not designated as being corrupted or unavailable) that are located at the same logical position. However, in order to compare with certainty, cross-validator <NUM> only compares storage segment log sequence numbers that are lower than the minimum between the begin log sequence number of primary replica snapshot <NUM> and the begin log sequence number of secondary replica snapshot <NUM> (referred herein as the minimum log sequence number). If both storage segment log sequence numbers are below the minimum log sequence number, both storage segment log sequence numbers should match. Otherwise, one of primary replica snapshot <NUM> or secondary replica snapshot <NUM> has missed a log record application.

Accordingly, cross-validator <NUM> compares the storage segment log sequence numbers to determine whether they match. If they match, cross-validator <NUM> determines that no data corruption exists between the two storage segments and performs the foregoing validation for the storage segments of IO unit <NUM> and <NUM> located at the next offset. If they do not match, cross-validator <NUM> determines that a data corruption exists between the two storage segments.

In the event that cross-validator <NUM> detects a data corruption, cross-validator <NUM> provides a notification <NUM> to alert generator <NUM>. Alert generator <NUM> may automatically issue an alert <NUM> to a computing device of an administrator indicating that a data corruption with respect to the two storage segments has been detected. This way, the administrator can quickly identify the cause of the data corruption and resolve the issue that caused the data corruption. Examples of alert <NUM> include a short messaging service (SMS) message, a telephone call, an e-mail, a notification that is presented via an incident management service, etc..

It is noted that validation manager <NUM> performs the foregoing operations for each IO unit of which primary replica snapshot <NUM> and secondary replica snapshot <NUM> comprises. For example, after IO units <NUM> and <NUM> are read and cross-validated, IO unit reader <NUM> may read the next IO units from primary replica snapshot <NUM> and secondary replica snapshot <NUM>, respectively, and cross-validate these IO units in a similar manner as described above. In another example, IO unit reader <NUM> may read the next IO units from primary replica snapshot <NUM> and secondary replica snapshot <NUM>, respectively, after providing IO units <NUM> and <NUM> to storage segment checker <NUM>.

Accordingly, data corruptions may be detected in many ways. For example, <FIG> shows a flowchart <NUM> of a method for detecting data corruptions based on cross-validating snapshots in accordance with an example embodiment. In an embodiment, flowchart <NUM> may be implemented by systems <NUM> and <NUM> shown in <FIG> and <FIG>, although the method is not limited to that implementation. Accordingly, flowchart <NUM> will be described with continued reference to <FIG> and <FIG>. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart <NUM> and systems <NUM> and <NUM> of <FIG> and <FIG>.

Flowchart <NUM> begins with step <NUM>. In step <NUM>, a snapshot pair comprising a first snapshot of a first replica of a data set and a second snapshot of a second replica of the data set is determined. The first replica comprises a plurality of first storage segments of the data set, and the second replica comprises a plurality of second storage segments of the data set. For example, with reference to <FIG>, validation manager <NUM> determines a snapshot pair comprising a first snapshot of a first replica of a data set (e.g., primary replica snapshot <NUM>, which is a snapshot of replica data set 406A) and a second snapshot of a second replica of the data (e.g., secondary replica snapshot <NUM>, which is a snapshot of replica data set 406B) is determined. Primary replica snapshot <NUM> comprises a plurality of first storage segments, and secondary replica snapshot <NUM> comprises a plurality of second storage segments.

In accordance with one or more embodiments, the plurality of first storage segments correspond to a plurality of first pages of the data set, and the plurality of second storage segments correspond to a plurality of second pages of the data set. For example, with reference to <FIG>, the storage segments of IO unit <NUM> correspond to pages of the data set, and the storage segments of IO unit <NUM> correspond to pages of the data set.

In accordance with one or more embodiments, the third log sequence number identifies a corresponding first log record that identifies the latest modification made to the first storage segment, and the fourth log sequence number identifies a corresponding second log record that identifies the latest modification made to the second storage segment. For example, with reference to <FIG>, the third log sequence number identifies a corresponding first log record that identifies the latest modification made to the first storage segment of IO unit <NUM>, and the fourth log sequence number identifies a corresponding second log record that identifies the latest modification made to the second storage segment of IO unit <NUM>.

In accordance with one or more embodiments, the snapshot pair is determined in accordance with flowchart <NUM>, which is described below with reference to <FIG>.

At step <NUM>, a minimum log sequence number from among a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot is determined. The first log sequence number identifies a corresponding first log record that identifies a change to the data set that is not incorporated in the first snapshot, and the second log sequence number identifies a corresponding second log record that identifies a change to the data set that is not incorporated in the second snapshot. For example, with reference to <FIG>, cross-validator <NUM> determines a minimum log sequence number from among a first log sequence number associated with primary replica snapshot <NUM> and a second log sequence number associated with secondary replica snapshot <NUM>.

At step <NUM>, for each first storage segment of the plurality of first storage segments, a determination is made as to whether a third log sequence number that is lower than the minimum log sequence number and that is associated with the first storage segment matches a fourth log sequence number that is lower than the minimum log sequence number and that is associated with a second storage segment of the plurality of second storage segments that corresponds to the first storage segment. If a determination is made that there is mismatch between the third log sequence number and the fourth log sequence number, flow continues to step <NUM>. Otherwise, flow continues to step <NUM>. For example, with reference to <FIG>, cross-validator <NUM> determines whether a third log sequence number that is lower than the minimum log sequence number and that is associated with the first storage segment of IO unit <NUM> matches a fourth log sequence number that is lower than the minimum log sequence number and that is associated with a second storage segment of the plurality of second storage segments of IO unit <NUM> that corresponds to the first storage segment.

At step <NUM>, in response to determining a mismatch between the third log sequence number and the fourth log sequence number, a determination is made that a data corruption exists with respect to the data set. For example, with reference to <FIG>, cross-validator <NUM> determines that a data corruption exists with respect to the data set (e.g., replica data sets 406A and/or 406B).

In accordance with one or more embodiments, in response to determining that a data corruption exists with respect to the data set, an alert that is automatically generated that indicates that the data corruption exists with respect to the data set. For example, with reference to <FIG>, in the event that cross-validator <NUM> detects a data corruption, cross-validator <NUM> provides a notification <NUM> to alert generator <NUM>. Alert generator <NUM> may automatically issue an alert <NUM> to a computing device of an administrator indicating that a data corruption with respect to the two storage segments has been detected. This way, the administrator can quickly identify the cause of the data corruption and resolve the issue that caused the data corruption. Examples of alert <NUM> include a short messaging service (SMS) message, a telephone call, an e-mail, a notification that is presented via an incident management service, etc..

At step <NUM>, in response to determining that the third log sequence number and the fourth log sequence number match, a determination is made that no data corruption exists with respect to the data set. For example, with reference to <FIG>, cross-validator <NUM> determines that a data corruption does not exist with respect to the data set (e.g., replica data sets 406A and/or 406B).

<FIG> shows a flowchart <NUM> of a method for determining a snapshot pair in accordance with an example embodiment. In an embodiment, flowchart <NUM> may be implemented by system <NUM> shown in <FIG>, although the method is not limited to that implementation. Accordingly, flowchart <NUM> will be described with continued reference to <FIG>. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart <NUM> and system <NUM> and <FIG>.

Flowchart <NUM> begins with step <NUM>. In step <NUM>, a most-recently generated snapshot from a plurality of snapshots generated for the first replica is determined. The most-recently generated snapshot comprises a first timestamp and is first snapshot of the snapshot pair. For example, with reference to <FIG>, validation manager <NUM> determines a most recently-generated snapshot from among replica snapshots <NUM>, <NUM>, <NUM>, and <NUM>. For example, validation manager <NUM> may determine the snapshot of replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> having the most recent timestamp.

In step <NUM>, a snapshot from a plurality of snapshots generated for the second replica that comprises a second timestamp that is closest to the first timestamp is determined, the determined snapshot being the second snapshot of the snapshot pair. For example, with reference, with reference to <FIG>, validation manager <NUM> determines a snapshot from among replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> that comprises a second timestamp that is closest to the first timestamp, the determined snapshot being the second snapshot.

In accordance with one or more embodiments, a data corruption may also be detected if a storage segment identifier of a first storage segment of a first replica snapshot does not match a storage segment identifier of a second storage segment of a second replica snapshot that corresponds to the first storage segment. <FIG> shows a flowchart <NUM> of a method for determining a data corruption based on a storage segment identifier mismatch in accordance with an example embodiment. In an embodiment, flowchart <NUM> may be implemented by system <NUM> shown in <FIG>, although the method is not limited to that implementation. Accordingly, flowchart <NUM> will be described with continued reference to <FIG>. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart <NUM> and system <NUM> and <FIG>.

Flowchart <NUM> begins with step <NUM>. In step <NUM> for each first storage segment of the plurality of first storage segments, a determination is made as to whether a first storage segment identifier associated with the first storage segment matches a second storage segment identifier associated with the second storage segment of the plurality of second storage segments corresponding to the first storage segment. In response to determining a mismatch between the first storage segment identifier and the second storage segment identifier, flow continues to step <NUM>. Otherwise, flow continues to step <NUM>. For example, with reference to <FIG>, cross-validator <NUM> determines whether a first storage segment identifier associated with a first storage segment of IO unit <NUM> matches a second storage segment identifier associated with the second storage segment of IO unit <NUM> that corresponds to the first storage segment (e.g., both the first storage segment and the second storage segment have the same logical offset).

In step <NUM>, a determination is made that a data corruption exists with respect to the data set. For example, with reference to <FIG>, cross-validator <NUM> determines that a data corruption exists with respect to the data set (e.g., replica data sets 406A and/or 406B).

In step <NUM>, in response to determining that the first storage segment identifier and the second storage segment identifier match, a determination is made that no data corruption exists with respect to the data set. For example, with reference to <FIG>, cross-validator <NUM> determines that a data corruption does not exist with respect to the data set (e.g., replica data sets 406A and/or 406B).

The systems and methods described above in reference to <FIG>, may be implemented in hardware, or hardware combined with one or both of software and/or firmware. For example, system <NUM> may be used to implement any of nodes <NUM> and 104A-104N, persistent log <NUM>, log service <NUM>, data servers 110A-110N, replica data servers 112A-112N, data sets 114A-114N, replica data sets 116A-116N, data set snapshots 118A-118N, replica data set snapshots 120A-120N, database application <NUM>, and/or snapshot service <NUM> of <FIG>, replica data servers 404A and 404B, replica data sets 406A and 406B, snapshot service <NUM>, validation manager <NUM>, replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> and/or replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>, validation manager <NUM> primary replica snapshots <NUM> and <NUM>, IO unit reader <NUM>, storage segment checker <NUM>, cross-validator <NUM> and/or alert generator <NUM> of <FIG> and/or any of the components respectively described therein, and flowcharts <NUM>, <NUM>, and/or <NUM> may be each implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer readable storage medium. Alternatively, any of nodes <NUM> and 104A-104N, persistent log <NUM>, log service <NUM>, data servers 110A-110N, replica data servers 112A-112N, data sets 114A-114N, replica data sets 116A-116N, data set snapshots 118A-118N, replica data set snapshots 120A-120N, database application <NUM>, and/or snapshot service <NUM> of <FIG>, replica data servers 404A and 404B, replica data sets 406A and 406B, snapshot service <NUM>, validation manager <NUM>, replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> and/or replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>, validation manager <NUM> primary replica snapshots <NUM> and <NUM>, IO unit reader <NUM>, storage segment checker <NUM>, cross-validator <NUM> and/or alert generator <NUM> of <FIG> and/or any of the components respectively described therein, and flowcharts <NUM>, <NUM>, and/or <NUM> may be implemented as hardware logic/electrical circuitry. In an embodiment, any of nodes <NUM> and 104A-104N, persistent log <NUM>, log service <NUM>, data servers 110A-110N, replica data servers 112A-112N, data sets 114A-114N, replica data sets 116A-116N, data set snapshots 118A-118N, replica data set snapshots 120A-120N, database application <NUM>, and/or snapshot service <NUM> of <FIG>, replica data servers 404A and 404B, replica data sets 406A and 406B, snapshot service <NUM>, validation manager <NUM>, replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> and/or replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>, validation manager <NUM> primary replica snapshots <NUM> and <NUM>, IO unit reader <NUM>, storage segment checker <NUM>, cross-validator <NUM> and/or alert generator <NUM> of <FIG> and/or any of the components respectively described therein, and flowcharts <NUM>, <NUM>, and/or <NUM> may be implemented in one or more SoCs (system on chip). An SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a central processing unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and/or further circuits, and may optionally execute received program code and/or include embedded firmware to perform functions.

<FIG> depicts an exemplary implementation of a computing device <NUM> in which embodiments may be implemented, including any of nodes <NUM> and 104A-104N, persistent log <NUM>, log service <NUM>, data servers 110A-110N, replica data servers 112A-112N, data sets 114A-114N, replica data sets 116A-116N, data set snapshots 118A-118N, replica data set snapshots 120A-120N, database application <NUM>, and/or snapshot service <NUM> of <FIG>, replica data servers 404A and 404B, replica data sets 406A and 406B, snapshot service <NUM>, validation manager <NUM>, replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> and/or replica snapshots <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>, validation manager <NUM> primary replica snapshots <NUM> and <NUM>, IO unit reader <NUM>, storage segment checker <NUM>, cross-validator <NUM> and/or alert generator <NUM> of <FIG> and/or any of the components respectively described therein, and flowcharts <NUM>, <NUM>, and/or <NUM>. The description of computing device <NUM> provided herein is provided for purposes of illustration, and is not intended to be limiting. Embodiments may be implemented in further types of computer systems, as would be known to persons skilled in the relevant art(s).

A number of program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include operating system <NUM>, one or more application programs <NUM>, other programs <NUM>, and program data <NUM>. Application programs <NUM> or other programs <NUM> may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the systems described above, including the embodiments described above with reference to <FIG>.

Display screen <NUM> may display information, as well as being a user interface for receiving user commands and/or other information (e.g., by touch, finger gestures, a virtual keyboard, by providing a tap input (where a user lightly presses and quickly releases display screen <NUM>), by providing a "touch-and-hold" input (where a user touches and holds his finger (or touch instrument) on display screen <NUM> for a predetermined period of time), by providing touch input that exceeds a predetermined pressure threshold, etc.).

As used herein, the terms "computer program medium," "computer-readable medium," and "computer-readable storage medium" are used to generally refer to physical hardware media such as the hard disk associated with hard disk drive <NUM>, removable magnetic disk <NUM>, removable optical disk <NUM>, other physical hardware media such as RAMs, ROMs, flash memory cards, digital video disks, zip disks, MEMs, nanotechnology-based storage devices, and further types of physical/tangible hardware storage media (including system memory <NUM> of <FIG>). Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave. Embodiments are also directed to such communication media.

A method is described herein. The method includes: determining a snapshot pair comprising a first snapshot of a first replica of a data set and a second snapshot of a second replica of the data set, the first replica comprising a plurality of first storage segments of the data set, and the second replica comprising a plurality of second storage segments of the data set; determining a minimum log sequence number from among a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot, the first log sequence number identifying a corresponding first log record that identifies a change to the data set that is not incorporated in the first snapshot, and the second log sequence number identifying a corresponding second log record that identifies a change to the data set that is not incorporated in the second snapshot; and for each first storage segment of the plurality of first storage segments: determining whether a third log sequence number that is lower than the minimum log sequence number and that is associated with the first storage segment matches a fourth log sequence number that is lower than the minimum log sequence number and that is associated with a second storage segment of the plurality of second storage segments that corresponds to the first storage segment; and in response to determining a mismatch between the third log sequence number and the fourth log sequence number, determining that a data corruption exists with respect to the data set.

In one implementation of the foregoing method, the method further comprises in response to determining that the third log sequence number and the fourth log sequence number match, determining that no data corruption exists with respect to the data set.

In one implementation of the foregoing method, determining the snapshot pair comprises determining a most-recently generated snapshot from a plurality of snapshots generated for the first replica, the most-recently generated snapshot comprising a first timestamp, the most-recently generated snapshot being the first snapshot of the snapshot pair; and determining a snapshot from a plurality of snapshots generated for the second replica that comprises a second timestamp that is closest to the first timestamp, the determined snapshot being the second snapshot of the snapshot pair.

In one implementation of the foregoing method, the method further comprises: for each first storage segment of the plurality of first storage segments: determining whether a first storage segment identifier associated with the first storage segment matches a second storage segment identifier associated with the second storage segment of the plurality of second storage segments corresponding to the first storage segment; in response to determining a mismatch between the first storage segment identifier and the second storage segment identifier, determining that a data corruption exists with respect to the data set; and in response to determining that the first storage segment identifier and the second storage segment identifier match, determining that no data corruption exists with respect to the data set.

In one implementation of the foregoing method, the third log sequence number identifies a corresponding first log record that identifies the latest modification made to the first storage segment, wherein the fourth log sequence number identifies a corresponding second log record that identifies the latest modification made to the second storage segment.

In one implementation of the foregoing method, the plurality of first storage segments correspond to a plurality of first pages of the data set, and wherein the plurality of second storage segments correspond to a plurality of second pages of the data set.

In one implementation of the foregoing method, the method further comprises in response to determining that a data corruption exists with respect to the data set, automatically generating an alert that indicates that the data corruption exists with respect to the data set.

A system in accordance with any of the embodiments described herein is also disclosed. The system includes: at least one processor circuit; and at least one memory that stores program code configured to be executed by the at least one processor circuit, the program code comprising: a validation manager configured to: determine a snapshot pair comprising a first snapshot of a first replica of a data set and a second snapshot of a second replica of the data set, the first replica comprising a plurality of first storage segments of the data set, and the second replica comprising a plurality of second storage segments of the data set; determine a minimum log sequence number from among a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot, the first log sequence number identifying a corresponding first log record that identifies a change to the data set that is not incorporated in the first snapshot, and the second log sequence number identifying a corresponding second log record that identifies a change to the data set that is not incorporated in the second snapshot; and for each first storage segment of the plurality of first storage segments: determine whether a third log sequence number that is lower than the minimum log sequence number and that is associated with the first storage segment matches a fourth log sequence number that is lower than the minimum log sequence number and that is associated with a second storage segment of the plurality of second storage segments that corresponds to the first storage segment; and in response to determining a mismatch between the third log sequence number and the fourth log sequence number, determine that a data corruption exists with respect to the data set.

In one implementation of the foregoing system, the validation manager further configured to: in response to determining that the third log sequence number and the fourth log sequence number match, determine that no data corruption exists with respect to the data set.

In one implementation of the foregoing system, the validation manager is configured to determine the snapshot pair by: determining a most-recently generated snapshot from a plurality of snapshots generated for the first replica, the most-recently generated snapshot comprising a first timestamp, the most-recently generated snapshot being the first snapshot of the snapshot pair; and determining a snapshot from a plurality of snapshots generated for the second replica that comprises a second timestamp that is closest to the first timestamp, the determined snapshot being the second snapshot of the snapshot pair.

In one implementation of the foregoing system, the validation manager further configured to: for each first storage segment of the plurality of first storage segments: determine whether a first storage segment identifier associated with the first storage segment matches a second storage segment identifier associated with the second storage segment of the plurality of second storage segments corresponding to the first storage segment; in response to determining a mismatch between the first storage segment identifier and the second storage segment identifier, determine that a data corruption exists with respect to the data set; and in response to determining that the first storage segment identifier and the second storage segment identifier match, determine that no data corruption exists with respect to the data set.

In one implementation of the foregoing system, the third log sequence number identifies a corresponding first log record that identifies the latest modification made to the first storage segment, wherein the fourth log sequence number identifies a corresponding second log record that identifies the latest modification made to the second storage segment.

In one implementation of the foregoing system, the plurality of first storage segments correspond to a plurality of first pages of the data set, and wherein the plurality of second storage segments correspond to a plurality of second pages of the data set.

In one implementation of the foregoing system, the validation manager further configured to: in response to determining that a data corruption exists with respect to the data set, automatically generate an alert that indicates that the data corruption exists with respect to the data set.

A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor, perform a method. The method includes: determining a snapshot pair comprising a first snapshot of a first replica of a data set and a second snapshot of a second replica of the data set, the first replica comprising a plurality of first storage segments of the data set, and the second replica comprising a plurality of second storage segments of the data set; determining a minimum log sequence number from among a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot, the first log sequence number identifying a corresponding first log record that identifies a change to the data set that is not incorporated in the first snapshot, and the second log sequence number identifying a corresponding second log record that identifies a change to the data set that is not incorporated in the second snapshot; and for each first storage segment of the plurality of first storage segments: determining whether a third log sequence number that is lower than the minimum log sequence number and that is associated with the first storage segment matches a fourth log sequence number that is lower than the minimum log sequence number and that is associated with a second storage segment of the plurality of second storage segments that corresponds to the first storage segment; and in response to determining a mismatch between the third log sequence number and the fourth log sequence number, determining that a data corruption exists with respect to the data set.

In one implementation of the foregoing computer-readable storage medium, the method further includes: in response to determining that the third log sequence number and the fourth log sequence number match, determining that no data corruption exists with respect to the data set.

In one implementation of the foregoing computer-readable storage medium, determining the snapshot pair comprises: determining a most-recently generated snapshot from a plurality of snapshots generated for the first replica, the most-recently generated snapshot comprising a first timestamp, the most-recently generated snapshot being the first snapshot of the snapshot pair; and determining a snapshot from a plurality of snapshots generated for the second replica that comprises a second timestamp that is closest to the first timestamp, the determined snapshot being the second snapshot of the snapshot pair.

In one implementation of the foregoing computer-readable storage medium, the method further comprises: for each first storage segment of the plurality of first storage segments: determining whether a first storage segment identifier associated with the first storage segment matches a second storage segment identifier associated with the second storage segment of the plurality of second storage segments corresponding to the first storage segment; in response to determining a mismatch between the first storage segment identifier and the second storage segment identifier, determining that a data corruption exists with respect to the data set; and in response to determining that the first storage segment identifier and the second storage segment identifier match, determining that no data corruption exists with respect to the data set.

In one implementation of the foregoing computer-readable storage medium, the third log sequence number identifies a corresponding first log record that identifies the latest modification made to the first storage segment, wherein the fourth log sequence number identifies a corresponding second log record that identifies the latest modification made to the second storage segment.

In one implementation of the foregoing computer-readable storage medium, the plurality of first storage segments correspond to a plurality of first pages of the data set, and wherein the plurality of second storage segments correspond to a plurality of second pages of the data set.

Claim 1:
A method, comprising:
determining (<NUM>) a snapshot pair comprising a first snapshot (118A-118N) of a first replica (120A-120N) of a data set (114A-114N) and a second snapshot (118A-118N) of a second replica (120A-120N) of the data set, the first replica comprising a plurality of first storage segments of the data set, and the second replica comprising a plurality of second storage segments of the data set;
determining (<NUM>) a minimum log sequence number (<NUM>) from among a first log sequence number associated with the first snapshot and a second log sequence number associated with the second snapshot, the first log sequence number identifying a corresponding first log record that identifies a change to the data set that is not incorporated in the first snapshot, and the second log sequence number identifying a corresponding second log record that identifies a change to the data set that is not incorporated in the second snapshot; and
for each first storage segment of the plurality of first storage segments:
determining (<NUM>) whether a third log sequence number that is lower than the minimum log sequence number and that is associated with the first storage segment matches a fourth log sequence number that is lower than the minimum log sequence number and that is associated with a second storage segment of the plurality of second storage segments that corresponds to the first storage segment; and
in response to determining a mismatch between the third log sequence number and the fourth log sequence number, determining (<NUM>) that a data corruption exists with respect to the data set.