Database node soft restart

Techniques are disclosed relating to restarting a database node. A database node may allocate memory segments that include a restart segment for storing data records. The database node may spawn processes to read a log and replay log records of the log to update the restart segment to store data records. The database node may determine to perform a restart operation to transition from a first mode to a second mode. Performing the restart operation may include ceasing reading the log at a stop position and storing, based on the stop position, database state information that enables the processes to resume reading the log from the stop position. The database node may further deallocate the memory segments except for the restart segment and terminate the processes. After performing the restart operation, the database node may spawn the processes, which may resume reading the log based on the database state information.

BACKGROUND

Technical Field

This disclosure relates generally to a database system and, more specifically, to various mechanisms for restarting a database node.

Description of the Related Art

Modern database systems routinely implement management systems that enable users to store a collection of information in an organized manner that can be efficiently accessed and manipulated. In some cases, these management systems maintain a log-structured merge-tree (LSM tree) comprising multiple levels that each store information in database records as key-value pairs. A database system can include a persistent storage that houses the LSM tree and a database node having an in-memory buffer. During operation, the database node initially writes records into the in-memory buffer before later flushing them to the persistent storage. As a part of flushing records, the database system writes the records into new files that are stored in one of the many levels of the LSM tree. Over time, those records are rewritten into new files stored in lower levels as the records are moved down the LSM tree.

DETAILED DESCRIPTION

In certain cases, a database system comprises multiple database nodes that facilitate the operations of a database. For example, a database system may include a primary database node that can process read and write transactions and one or more non-primary database nodes, such as secondary database nodes that process certain read transactions. In some cases, there can be multiple node clusters with one cluster having a primary database node and others having non-primary database nodes. In order to be able to process read transactions, non-primary database nodes read a database log that is produced by the primary database node that describes database operations performed by that primary database node. As the non-primary database nodes read that database log, they replay the database operations, which can include inserting new records into their in-memory cache. During the operation of the database system, the primary database node might become unavailable (e.g., because it crashed). As a result, one of the non-primary database nodes may be restarted to become a new primary database node. But when restarting, that database node clears the memory that underlies its in-memory cache, causing the records stored therein to be deleted. Before being able to fulfill the role of the primary database node, the restarted database node has to rebuild its in-memory cache based on the database log. But the process of rebuilding the in-memory caches takes a considerable amount of time and thus a service-level agreement regarding an allowable amount of downtime might not be met. This disclosure addresses, among other things, the problem of how to restart a database node in a more efficient manner, such that it can be brought back up to speed to fulfill its obligations in accordance with a service-level agreement.

The present disclosure describes various techniques for preserving certain information (e.g., records stored in an in-memory cache) of a database node in such a way that the database node can be restarted without losing much (if any) of its progress. In various embodiments that are described below, a system comprises multiple node clusters that each have a set of database nodes. One of the node clusters may be designated a primary node cluster and include a primary database node and the other node clusters are designated non-primary node clusters and include non-primary database nodes. Upon starting up, in various embodiments, a database node may allocate multiple memory segments for facilitating its operation within the system. One of the memory segments is a restart segment that can be used to house an in-memory cache in which records are stored as a part of processing database transactions. In various embodiments, non-primary database nodes spawn their own set of processes to read a database log of the primary database node and replay log records of the database log in order to update their restart segment to store a set of records of the database. During operation of the system, a database node may determine to perform a restart operation to transition from a first database mode (e.g., being a secondary database node) to a second database mode (e.g., being a primary database node). In some cases, a database node may restart to transition from being a secondary database node of a secondary cluster to a secondary database node of a primary cluster.

As part of performing the restart operation, in various embodiments, the database node ceases reading the database log at a stop position and then stores, based on that stop position, database state information that enables its set of processes to resume reading the database log from the stop position. Before terminating that set of processes, the database node may permit queues that store log records read from the transaction log to be drained. Thereafter, in various embodiments, the database node terminates the processes and deallocates the multiple memory segments (that were previously allocated for the operation of the database node) expect for the restart segment, which is preserved. After performing the restart operation, the database node reallocates the deallocated memory segments and spawns the set of processes. Those processes access the stored database state information and then resume reading the database log from the stop position and replaying log records. As a result of preserving the restart segment (and thus its in-memory cache and the database state information), the database node is able to be brought back up to speed more quickly in order to fulfill its obligations.

These techniques may be advantageous as they enable a quicker failover from a primary database node to a non-primary database node. In particular, certain agreements can be in place regarding the availability of a service. When the primary database node becomes unavailable (e.g., a network crash), functionality of the service becomes unavailable. In order to meet the agreements regarding availability, it is desirable to quickly failover from the primary database node to a non-primary database node such that the non-primary database node can become the primary database node. By preserving the non-primary database node's in-memory cache and maintaining database state information about the current state of the in-memory cache, the non-primary database node is able to be restarted more quickly to become a primary database node than it otherwise would be. These techniques can also be extended to cases in which there are multiple database node clusters, one of which may be a primary cluster and the others may be secondary clusters. The primary cluster might become unavailable and thus it may be desirable to failover from the primary cluster to a secondary cluster. Using the disclosed techniques, the database nodes of that secondary cluster may be restarted more quickly to take on the role of the prior primary cluster. An exemplary application of these techniques will now be discussed, starting with reference toFIG.1.

Turning now toFIG.1, a block diagram of a system100is shown. System100includes a set of components that may be implemented via hardware or a combination of hardware and software routines. In the illustrated embodiment, system100includes a database store110and a database node130that can access database store110. As shown, database store110includes data records115and a database log120that comprises log records125, and database node130includes an orchestration process135, memory segments140, and worker processes150. Also as illustrated inFIG.1, memory segments140include temporary memory segments142and a restart memory segment144having data records115and database state information155. In some embodiments, system100is implemented differently than shown. For example, system100may comprise multiple node clusters spread across multiple zones (as depicted inFIG.5), database state information155may be stored at database store110, database log120might be stored at separate store from data records115, etc.

System100, in various embodiments, implements a platform service (e.g., a customer relationship management (CRM) platform service) that allows users of that service to develop, run, and manage applications. System100may be a multi-tenant system that provides various functionality to users/tenants hosted by the multi-tenant system. Accordingly, system100may execute software routines from various, different users (e.g., providers and tenants of system100) as well as provide code, web pages, and other data to users, stores, and other entities that are associated with system100. In various embodiments, system100is implemented using a cloud infrastructure that is provided by a cloud provider. Accordingly, database store110and database node130may utilize the available cloud resources of that cloud infrastructure (e.g., computing resources, storage resources, etc.) in order to facilitate their operation. For example, database node130may execute within a virtual environment hosted on server-based hardware included in a datacenter. But in some embodiments, system100is implemented using a local or private infrastructure as opposed to a public cloud.

Database store110, in various embodiments, includes a collection of information that is organized in a manner that allows for access, storage, and manipulation of that information. Database store110may include supporting software (e.g., storage nodes) that enables database node130to carry out those operations (e.g., accessing, storing, etc.) on the information that is stored at database store110. In various embodiments, database store110is implemented using a single or multiple storage devices that are connected together on a network (e.g., a storage attached network (SAN)) and configured to redundantly store information in order to prevent data loss. The storage devices may store data persistently and therefore database store110may serve as a persistent storage for system100. In various embodiments, data written to database store110by database node130is accessible to other database nodes130within a multi-node configuration (e.g., a database node cluster or a system having multiple database node clusters spread across different zones provided by a cloud provider).

In various embodiments, database store110stores two main types of files (also herein referred to as “extents”): a data file and a log file. A data file may comprise the actual data and may be append-only such that new data records115are appended to the data file until its size reaches a threshold and another data file is created. A data record115, in various embodiments, comprises data and a database key that is usable to look up that data record115. For example, a data record115may correspond to a row in a database table where the record specifies values for attributes of the database table. A log file may comprise log records125describing database modifications (e.g., record insertions) made as a result of executing database transactions. As with data files, a log file may be append-only and continuously receive appends as transactions do work. In various embodiments, database log120is a set of log files having log records125that collectively identify a state of the database system implemented by system100. As such, by reading database log120, database node130may determine an ordering in which database operations were performed, including an ordering in which transactions were committed within system100. Data files and log files, in various embodiments, are associated with file identifiers that can be used to locate them. Consequently, database node130may access data records115and log records125from database store110by issuing access requests having file identifiers to the storage nodes that implement database store110.

Database node130, in various embodiments, facilitates database services, such as data retrieval, data manipulation, and data storage. In various embodiments, database node130is software that is executable on hardware, while in some embodiments, it encompasses both the hardware and the software. The database services may be provided to components internal and external to system100. For example, database node130may receive a transaction request from an application node to perform a database transaction. A database transaction is a logical unit of work (e.g., a specified set of database statements) to be performed in relation to a database of system100. As an example, processing a database transaction may include executing a SQL SELECT statement to select one or more rows of table. The contents of a row may be specified in a data record115and thus database node130may return one or more requested data records115that correspond to those one or more rows.

In various embodiments, database node130can operate in at least two different modes: a read mode and a write mode. When operating in the write mode, database node130can write data records115for the database of system100. In particular, write transactions may be routed to database node130that involve writing data records115. Consequently, database node130may write those data records115into an in-memory cache that stores its data in restart memory segment144. In various embodiments, database node130also generates log records125that identify the database operations (e.g., insert, delete, etc.) performed by database node130and appends them to database log120. After writing data records115to restart memory segment144, database node130may flush those data records115to database store110by writing them into data files that are stored at database store110. In addition to processing write transactions, database node130can also process read-only transactions (i.e., transactions that do not involve writing data records115) when operating in the write mode. A database node130that operates in the write mode can be referred to as a “primary” database node.

When operating in the read mode, in various embodiments, database node130cannot write data records115but can access and return them. Thus, certain read transactions may be routed to database node130to offload work from a primary database node130. In order to be able to return data records115that are stored at a primary database node130but not stored in database store110, in various embodiments, database node130reads database log120(when operating in the read mode) and replays log records125to recreate the state of the in-memory cache of the primary database node130. As a result, database node130may return data records115from its in-memory cache (which is implemented via restart memory segment144) as part of processing particular read transactions. A database node130that operates in the read mode can be referred to as a “secondary” database node. To facilitate the operations of database node130in either the read or write mode, database node130may execute orchestration process135and worker processes150.

Orchestration process135, in various embodiments, is a computer process that allocates memory segments140and spawns worker processes150that implement the database services provided by database node130. Orchestration process135may be instantiated when database node130is initially started up or as part of starting a database service. After being instantiated, orchestration process135may allocate memory segments140before spawning a set of worker processes150. A given memory segment140, in various embodiments, is a portion of memory that is allocated on a memory device (e.g., random access memory) and can be used by worker processes150to store data. As depicted, orchestration process135allocates a set of temporary memory segments142and restart memory segment144. One difference between the two types of memory segments is that restart memory segment144is not deallocated during a soft restart operation, discussed further below. Consequently, a temporary memory segment142may be used to store items (e.g., query plans, table locks, etc.) that facilitate the operation of a database service but are less desired (e.g., from the perspective of database operator) to preserve during a database node restart. In contrast, restart memory segment144may be used to store records115and other information that is more desired to preserve during a database node restart. After allocating memory segments140, orchestration service135may then spawn worker processes150—in some cases, orchestration service135spawns a single worker process150that spawns other worker processes150.

Worker processes150, in various embodiments, are computer processes that facilitate the operations of the database service. The actions performed by worker processes may depend on the database mode in which database node130is operating. As an example, when operating in the read mode, worker processes150may include a first set of processes that read database log120and provide log records125to a second set of processes that replay those log records125, which includes writing data records115to restart memory segment144as shown. When operating in the write mode, worker processes150may write log records125to database log120instead of reading log records125in order to replay them. In various embodiments, worker processes150include processes that purge data (e.g., data records115) from memory segments140, merge data records115from different levels of a log-structured merge tree (an LSM tree), and other database maintenance operations. Examples of worker processes150are discussed in more detail with respect toFIG.2.

In various embodiments, orchestration process135can perform a soft restart operation for database node130in which it terminates worker processes150and deallocates temporary memory segments142but preserves restart memory segment144. As a result, the data stored in those temporary memory segments142is lost, but the data stored in restart memory segment144is preserved. In various embodiments, a soft restart operation is performed in response to receiving a restart indication105that indicates that a database node130has been promoted to become a primary database node. By preserving the data stored in restart memory segment144and storing database state information155that details the state of that data, that database node130may become a primary database node more quickly. Examples of the information included in database state information155is discussed in more detail with respect toFIG.3. An example soft restart operation is discussed in more detail with respect toFIGS.4A-4B. In contrast to the soft restart operation, a hard restart operation involves orchestration process135deallocating all memory segments140, including restart memory segment144. In various embodiments, a hard restart operation is performed in response to determination to restart the database service.

Turning now toFIG.2, a block diagram of example worker processes150that can read and replay log records125is shown. In the illustrated embodiment, there is database log120, restart memory segment144, a set of worker processes150, a storage catalog240, and a set of redo queues250. As further shown, database log120comprises log files230and is associated with a recovery start position242and a last read position244. Also as shown, worker processes150include a reader process210and a set of applier processes220. The illustrated embodiment may be implemented differently than shown—e.g., there may be multiple reader processes210that read from database log120, worker processes150may include a purger process, etc.

As mentioned, orchestration service135may spawn one or more worker processes150to provide the database services of database node130. In various embodiments, orchestration service135spawns reader process210and then reader process210spawns one or more applier processes220—reader process210may also spawn other worker processes150(e.g., a purger process). In some embodiments, orchestration service135spawns both reader process210and one or more applier processes220. Reader process210and applier processes220might not be spawned when database node130starts up in the aforementioned write mode.

Reader process210, in various embodiments, is a computer process that reads database log120and enqueues log records125from database log120in redo queues250. When reader process210is spawned, in various embodiments, it accesses storage catalog240to learn about database log120. Storage catalog240includes information about database log120, which may include the locations of log files230—a log file230comprises a set of log records125. When the primary database node130creates a log file230, it may store information in storage catalog240that identifies the location of the log file230. Accordingly, reader process210may access that information and then use it to begin reading log records125from that log file230. In some cases, the primary database node130stores that information in storage catalog240after it has finished writing to that log file230. In various embodiments, storage catalog240also specifies a recovery start position242that identifies where to begin reading from database log120upon database node130starting up from a cold start or a hard restart. The results (e.g., data records115) associated with log records125occurring in database log120after recovery start position242may not be stored at database store110but instead still reside at the primary database node130and have not been flushed to database store110. That is, in various embodiments, the log records125between recovery start position242and the end of database log120correspond to data records115that have not been persisted to database store110.

After accessing the log information from storage catalog240about the locations of log files230and recovery start position242, reader process210may begin accessing log records125from database log120based on recovery start position242. As discussed in greater detail with respect toFIG.4, subsequent to a soft restart, reader process210may begin accessing log records125from a recorded stop position that is different than recovery start position242. As reader process210accesses log records125, it writes them to redo queues250, as shown. Redo queues250, in various embodiments, are queues (e.g., first in, first out (FIFO) structures) that are implemented using the memory of one or more temporary memory segments142allocated by orchestration process135.

Applier processes220, in various embodiments, are computer processes that replay log records125accessed from redo queues250. As mentioned, a log record125may identify one or more database operations (e.g., insert, update, etc.) that are performed by a primary database node130as part of processing database transactions. Accordingly, replaying a log record125includes performing the one or more database operations identified by that log record125. As a result, an applier process220may insert data records115(e.g., the data record115resulting from an insert operation) into restart memory segment144, as shown. In various embodiments, applier processes220replay log records in parallel, where a given applier process220replays the log records125from a particular database transaction. Consequently, multiple transactions can be replayed in parallel by using multiple applier processes220.

Turning now toFIG.3, a block diagram of example database store information155is shown. In the illustrated embodiment, database store information155includes a stop position310, a flush transaction commit number (flush XCN)320, a visible transaction commit number (visible XCN)330, and purger state information340. Database store information155may be implemented differently than shown. As an example, database store information155might not include purger state information340.

Stop position310, in various embodiments, specifies a position within database log120at which reader process210has stopped reading database log120. As a part of database node130performing a soft restart, in various embodiments, reader process210is instructed to cease reading database log120. Consequently, reader process210may record where it has stopped and then include that information in database state information155as stop position310. Stop position310may specify the last read log record125or the last read fragment (a collection of log records125) of database log120. Stop position310may alternatively specify a log file230and an offset within that log file230corresponding to the last read position.

Flush XCN320, in various embodiments, identifies the most recent transaction whose data records115have been flushed from database node130to database store110. In particular, after a storage threshold has been reached in regards to an in-memory cache implemented using restart memory segment144(e.g., the in-memory cache is 80% full), database node130may perform a flush operation in which a set of committed data records115are written into database store110and then later removed from the in-memory cache by a purger process. In some cases, database node130may not wait for a threshold to be satisfied, but instead continually performs the flush operation as transactions are committed. The set of committed data records115that are flushed may correspond to those data records in the in-memory cache whose XCN is equal to or less than a specified XCN. Once the flush operation has been completed, flush XCN320may be updated by database node130to reflect the specified XCN. Accordingly, flush XCN320, in various embodiments, is used by database node130to determine where (e.g., database store110or the in-memory cache) to access a data record115as part of processing a database transaction. As an example, if a query involves searching for a record with an XCN less than flush XCN320, then database node130may search database store110but not the in-memory cache. Because flush XCN320indicates what has been flushed, in various embodiments, flush XCN320and stop position310together define a transaction window that indicates, for which transactions, there are data records115stored in restart memory segment144.

Visible XCN330, in various embodiments, identifies the most recent transaction whose data records115have been committed and therefore are visible outside of that transaction. In particular, data records115may be written to the in-memory cache implemented using restart memory segment144. In various embodiments, before the corresponding transaction has been committed, those data records115are not accessible to other transactions or other components of system100. As part of committing the transaction, those data records115are stamped with an XCN, and visible XCN330is updated to reflect that XCN until another transaction has been committed. Consequently, when searching for the latest version of a data record115to return for a key, database node130may not return any data record115associated with a transaction occurring after visible XCN330—that is, visible XCN330indicates which data records115stored in restart memory segment144are accessible to queries executed at database node130. By maintaining this information, worker processes150can determine, upon their instantiation after a soft restart, what the most recent committed transaction is at database node130.

Purger state information340, in various embodiments, is information that allows for a purger process of worker processes150to resume a purge operation. As mentioned, after data records115have been flushed to database store110, they are purged from database node130in a purge operation. When database node130is preparing to perform a soft restart, the purge process may be in the middle of the purge operation. In a similar manner to reader process210, in various embodiments, the purge process stores purger state information340about the state of the purge operation that it is performing so that it may resume the purge operation after the soft restart. Purger state information340may specify a XCN range and the most recent XCN purged from database node130. The XCN range may identify a range of XCN for which data records115having an XCN in that range are purged in the purge operation. The purge process may start with data records115belonging to the lowest XCN of the range and work towards data records115belonging to the highest XCN. In various embodiments, the most recent XCN corresponds to an XCN in the XCN range whose records have been purged. Accordingly, the most recent XCN may serve as a purge stop position that can be used by the purge process to resume the purge operation.

Turning now toFIG.4A-B, block diagrams of a soft restart operation of a database node130is shown. InFIG.4A, there is orchestration process135, worker processes150, temporary memory segments142, restart memory segment144, and redo queues250. As shown, worker processes150include reader process210and a set of applier processes220. The illustrated embodiment may be implemented differently than shown. For example, worker processes150may include a purger process that removes data records115from restart memory segment144, a merger process that merges data records115of an LSM tree implemented at database store110, etc.

As shown, orchestration process135initially receives a restart indication105. A restart indication105may be a request from another component of system100, a request from a user, or orchestration process135may poll a metadata store and receive an indication that it should restart (e.g., it has been promoted). In response, orchestration process135may begin the restart operation by issuing a preparation request to reader process210to prepare itself for termination as part of the soft restart operation. Upon receiving the preparation request, reader process210then ceases reading database log120and enqueuing log records125. In various embodiments, reader process210ceases reading after finishing a portion of database log120(e.g., a block of log records125, a log file230, etc.). In some embodiments, reader process210ceases reading at or near the log record125that it was reading when it received the preparation request. After ceasing reading of database log120, reader process210records a stop position310and stores it in restart memory segment144as a part of database state information155.

Reader process210further waits for redo queues250to empty, which indicates that all log records125read from database log120have been replayed by applier processes220. Thus, the state of restart memory segment144may be consistent with the recorded stop position310. After reader process210has detected that redo queues250have been drained, reader process210may store, as a part of database state information155, a visible XCN330associated with an in-memory cache implemented using restart memory segment144. Reader process210may also wait for any active flush operations to complete and then store a flush XCN320associated with the last completed flush operation. In some embodiments, reader process210waits for a purger process to store purger state information340. Once reader process210has determined it is prepared to be terminated, reader process210issues a preparation response to orchestration process135.

In response to receiving the preparation response, orchestration process135terminates worker processes150(which can include database processes that are processing transactions) and deallocates temporary memory segments142. To terminate worker process150, in various embodiments, orchestration process135issues a terminate signal to worker processes150that causes them to terminate gracefully. Once those worker processes150have terminated and the temporary memory segments142have been deallocated, orchestration process135may begin a boot-up process, as shown inFIG.4B.

InFIG.4B, there is database log120, orchestration process135, worker processes150, temporary memory segments142, restart memory segment144, storage catalog240, and redo queues250. As depicted, restart memory segment144includes data records115and database state information155, worker processes150include reader process210and applier processes220, storage catalog240includes a recovery window410, and redo queues250include log records125. The illustrated embodiment may be implemented differently than shown. As an example, worker processes150may include a purger process, a merger process, etc.

As shown, orchestration process135initially spawns worker process150and allocates temporary memory segments142. Since a soft restart is being performed, orchestration process135does not reallocate restart memory segment144since it was preserved. But if a hard restart is performed, then orchestration process135may allocate all memory segments140. In some embodiments, orchestration process135spawns reader process210, which then spawns applier processes220. After being spawned, reader process210may access database state information155and information from storage catalog240that describes recovery window410. Recovery window410, in various embodiments, identifies information (e.g., recovery position242, log files230, etc.) that facilities the reading of database log120. In various embodiments, reader process210updates recovery window410using database state information155by replacing recovery position242with stop position310. Reader process210may further update recovery window410based on flush XCN320and visible XCN330. Reader process210may then start reading database log120from stop position310instead of recovery position242and inserting log records125into redo queues250.

Turning now toFIG.5, a block diagram of an example cloud environment500is shown. In the illustrated embodiments, cloud environment500includes cloud zones510A-C that each include a respective database cluster530(i.e., database cluster530A-C respectively). Also as shown, cloud environment500includes a metadata store520that is implemented across cloud zones510A-C. In some embodiments, cloud environment500is implemented differently than shown. As an example, cloud environment500may include a cloud zone510that has multiple database clusters530.

Cloud environment500, in various embodiments, is a cloud infrastructure that includes various components (e.g., hardware, virtualized resources, storage, and network resources) for providing cloud computing services to users. In some cases, cloud environment500may be a public cloud provided by a cloud provider to multiple customers that implements their systems using the various components/resources of the public cloud; in other cases, cloud environment500is a private cloud that is available to only select users instead of the general public. In some embodiments, cloud environment500is spread across various geographical locations and each location may define a “region” of cloud environment500. Within a given region, there may be one or more cloud zones510. As an example, cloud zones510A-C might be a part of the same region, although they can be in separate regions. A cloud zone510, in various embodiments, is a logical or physical grouping of components (e.g., computing resources, storage resources, etc.) within a region. In many cases, the components of a cloud zone510are isolated from the failures of components in other cloud zones510. For example, cloud zone510A may be a first data center in a particular region and cloud zone510B may be a second data center in that same region. Cloud zone510A may be isolated from cloud zone510B such that a failure at the data center of cloud zone510B does not affect the data center of cloud zone510A. In some cases, cloud zones510A and510B might be the same data center but correspond to components on separate networks such that one cloud zone510might not be affected by the other cloud zone510.

Metadata store520, in various embodiments, is a repository that stores metadata, which can pertain to the operation of a database service. In particular, database clusters530A-C may collectively implement a database service and use metadata store520as a repository of at least of portion of the metadata used to facilitate the operation of the database service. Accordingly, the metadata that is written to metadata store520by one database node130may be accessible to other database nodes130of database clusters530. To facilitate this, in various embodiments, each cloud zone510executes an instance of metadata store520that communicates with other instances of metadata store520. Accordingly, metadata written to one instance may be synced with the other instances in order to create a consistent view of metadata store520. For example, a database node130of database cluster530A may write metadata to its instance of metadata store520and then that metadata may be synced with an instance in cloud zone510C so that a database node130of cloud zone510C can access the metadata. In various embodiments, one piece of metadata stored by metadata store520is an indication of which database cluster530is the primary database cluster and which database clusters530are secondary database clusters of the database service.

As shown inFIG.5, metadata store520initially indicates that database cluster530C is the primary database cluster and thus is responsible for handling transactions that involve write operations, in various embodiments. During operation of the database service, cloud zone520C or database cluster530C may become unavailable. For example, power may be lost to the data center of cloud zone520C. In order to avoid long downtimes of the database service, in various embodiments, another service (not shown) selects one of the other cloud zones510to become the primary cloud zone with the primary database cluster. The selection may be based on a set of characteristics of the cloud zones510(e.g., the cloud zone510with the lowest unavailability in cloud environment500) or it might be random. After selecting a cloud zone510, the service updates metadata store520to reflect the new selection and that information is synced across the cloud zones510. As shown, metadata store520is updated to indicate that cloud zone510A has become the new primary cloud zone and thus database cluster530A should be the primary database cluster.

In various embodiments, a database node130periodically polls metadata store520for certain information, which includes the indication of which database cluster530is the primary database cluster—that indication can serve as restart indication105when there is a difference between what a given database node130expects and what is actually indicated. In particular, in response to observing that cloud zone510A has become the primary cloud zone, the database nodes130of database cluster530A may perform the soft restart operation disclosed herein. At least one of the database nodes130may become a primary node and the other database nodes130may become secondary nodes within the primary database cluster.

Turning now toFIG.6, a flow diagram of a method600is shown. Method600is one embodiment of a method performed by a computer system (e.g., system100) to perform a soft restart operation. Method600may be performed by executing program instructions stored on a non-transitory computer-readable medium. In some embodiments, method600includes more or less steps than shown. For example, method600may include a step in which the computer system ceases processing database transactions.

Method600begins in step610with the computer system allocating a plurality of memory segments (e.g., memory segments140) for a database. In various embodiments, the plurality of memory segments includes a restart segment (e.g., restart memory segment144) that can be used to store data records (e.g., data records115). In step620, the computer system spawns a set of processes (e.g., worker processes150) to read a database log (e.g., database log120) and replay log records (e.g., log records125) of the database log to update the restart segment to store a set of data records of the database. In various embodiments, the set of processes includes a process (e.g., reader process210) that is operable to read the database log and enqueue log records in a set of queues (e.g., redo queues250) accessible to other ones of the set of processes (e.g., applier processes220) that are operable to apply the enqueued log records.

In step630, the computer system determines to perform a restart operation to transition the computer system from a first database mode to a second database mode. The computer system may access node cluster metadata from a metadata store (e.g., metadata store520) that is separate from the database node. The node cluster metadata may identify a primary node cluster from a plurality of node clusters (e.g., database clusters530) that implement a database service. The determining to perform the restart operation may be in response to determining that a node cluster having the computer system has become the primary node cluster. In some cases, the determining to perform the restart operation is in response to the computer system receiving a request to perform the restart operation. In various embodiments, the first database mode corresponds to a read mode in which the computer system processes read transactions but not write transactions, and the second database mode corresponds to a write mode in which the computer system processes read and write transactions.

As a part of performing the restart operation, in step632, the computer system ceases reading the database log at a stop position (e.g., a stop position310). In step634, the computer system stores, based on the stop position, database state information (e.g., database state information155) that enables the set of processes to resume reading the database log from the stop position. The database state information may specify a flush commit number (e.g., a flush transaction commit number320) that indicates transactions whose data records have been written to a persistent storage (e.g., database store110) separate from the database node. The stop position and the flush commit number may define a transaction window that indicates, for which transactions, there are data records stored in the restart segment. The database state information may specify a read commit number (e.g., a visible transaction commit number330) that indicates which data records stored in the restart segment are accessible to queries of the database node. In various embodiments, the database state information is stored in the restart segment.

In step636, the computer system deallocates at least one (e.g., a temporary memory segment142) of the plurality of memory segments other than the restart segment. In various cases, all of the plurality of memory segments other than the restart segment are deallocated as a part of the deallocating. In step638, the computer system terminates the set of processes. The deallocating and the terminating may be performed after a determination that the set of queues have been drained of log records

In step640, after performing the restart operation, the computer system spawns the set of processes. The set of processes may be operable to resume reading of the database log based on the database state information. In some embodiments, the set of processes includes a process that is operable to, after being spawned subsequent to the restart operation access recovery metadata (e.g., recovery window410) from a catalog store (e.g., storage catalog240) that is separate from the database node. The recovery metadata may specify a recovery position (e.g., a recovery position242) from which to read the database log and merge the stop position with information of the recovery metadata such that the process reads the database log from the stop position instead of the recovery position. After performing the restart operation, the computer system may allocate the at least one memory segment but not the restart segment

In various embodiments, the computer system stores purger state information (e.g., purger state information340) that identifies a purge stop position in a purge process involving purging data records from the database node. The set of processes may be operable to, after being spawned subsequent to the restart operation, resume the purge process from the purge stop position. The purger state information may identify a purge commit number that indicates a most recent transaction whose records have been purged from the computer system.

Exemplary Computer System

Turning now toFIG.7, a block diagram of an exemplary computer system700, which may implement system100, database store110, or database node130, is depicted. Computer system700includes a processor subsystem780that is coupled to a system memory720and I/O interfaces(s)740via an interconnect760(e.g., a system bus). I/O interface(s)740is coupled to one or more I/O devices750. Although a single computer system700is shown inFIG.7for convenience, system700may also be implemented as two or more computer systems operating together.

Processor subsystem780may include one or more processors or processing units. In various embodiments of computer system700, multiple instances of processor subsystem780may be coupled to interconnect760. In various embodiments, processor subsystem780(or each processor unit within780) may contain a cache or other form of on-board memory.

System memory720is usable store program instructions executable by processor subsystem780to cause system700perform various operations described herein. System memory720may be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM—SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read only memory (PROM, EEPROM, etc.), and so on. Memory in computer system700is not limited to primary storage such as memory720. Rather, computer system700may also include other forms of storage such as cache memory in processor subsystem780and secondary storage on I/O Devices750(e.g., a hard drive, storage array, etc.). In some embodiments, these other forms of storage may also store program instructions executable by processor subsystem780. Program instructions that, when executed, implement orchestration process135and/or worker processes150may be included/stored within system memory720.

I/O interfaces740may be any of various types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I/O interface740is a bridge chip (e.g., Southbridge) from a front-side to one or more back-side buses. I/O interfaces740may be coupled to one or more I/O devices750via one or more corresponding buses or other interfaces. Examples of I/O devices750include storage devices (hard drive, optical drive, removable flash drive, storage array, SAN, or their associated controller), network interface devices (e.g., to a local or wide-area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, computer system700is coupled to a network via a network interface device750(e.g., configured to communicate over WiFi, Bluetooth, Ethernet, etc.).

The present disclosure includes references to “embodiments,” which are non-limiting implementations of the disclosed concepts. References to “an embodiment,” “one embodiment,” “a particular embodiment,” “some embodiments,” “various embodiments,” and the like do not necessarily refer to the same embodiment. A large number of possible embodiments are contemplated, including specific embodiments described in detail, as well as modifications or alternatives that fall within the spirit or scope of the disclosure. Not all embodiments will necessarily manifest any or all of the potential advantages described herein.