Patent Description:
A cloud platform (i.e., a computing platform for cloud computing) may be employed by many users to store, manage, and process data using a shared network of remote servers. Users may develop applications on the cloud platform to handle the storage, management, and processing of data. In some cases, the cloud platform may utilize a multi-tenant database system. Users may access the cloud platform using various user devices (e.g., desktop computers, laptops, smartphones, tablets, or other computing systems, etc.). Computer networks permit the transport of data between interconnected computers. Techniques for data storage using a distributed storage cluster may be improved.

<CIT> discloses a distributed caching system in which cache data is shared between redundant systems and when a write command to write data to a database is received by a primary leader cache cluster associated with a first database, it is broadcast to a secondary leader cache cluster associated with a second database that is synchronized with the first database.

A method for data storage is described. The method may include receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes, generating, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request, and transmitting, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes.

An apparatus for data storage is described. The apparatus may include a processor, memory coupled with the processor, and instructions stored in the memory. The instructions may be executable by the processor to cause the apparatus to receive, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes, generate, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request, and transmit, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes.

Another apparatus for data storage is described. The apparatus may include means for receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes, means for generating, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request, and means for transmitting, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes.

A computer-readable medium carrying code for data storage is described. The code may include instructions executable by a processor to receive, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes, generate, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request, and transmit, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes.

Some examples of the method, apparatuses, and computer-readable medium described herein may further include operations, features, means, or instructions for instructing the one or more access manager nodes to store the data in the distributed storage system in accordance with the instruction. In some examples of the method, apparatuses, and computer-readable medium described herein, generating the cache data may include operations, features, means, or instructions for creating, in the cache of the leader control plane node, a replication log entry comprising the instruction for storing the data indicated in the request by the one or more access manager nodes of the distributed storage system.

Some examples of the method, apparatuses, and computer-readable medium described herein may further include operations, features, means, or instructions for receiving, from the one or more access manager nodes, a confirmation message indicating that the data indicated in the request may have been stored data by the one or more access manager nodes and updating the replication log entry of the cache to indicate that the data indicated in the request may have been stored data by the one or more access manager nodes of the distributed storage system.

Some examples of the method, apparatuses, and computer-readable medium described herein may further include operations, features, means, or instructions for determining a failure of the leader control plane node and transmitting, to a follower control plane node of the one or more follower control plane nodes, an indication that the follower control plane node may be elected to be a new leader control plane node based at least in part on failure of the leader control plane node.

In some examples of the method, apparatuses, and computer-readable medium described herein, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples of the method, apparatuses, and computer-readable medium described herein, the request to store data in the distributed storage system comprises a write request.

A method for data storage is described. The method may include receiving, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node, receiving, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node, processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request, and instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

An apparatus for data storage is described. The apparatus may include a processor, memory coupled with the processor, and instructions stored in the memory. The instructions may be executable by the processor to cause the apparatus to receive, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node, receive, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node, process, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request, and instruct the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

Another apparatus for data storage is described. The apparatus may include means for receiving, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node, means for receiving, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node, means for processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request, and means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

A computer-readable medium carrying code for data storage is described. The code may include instructions executable by a processor to receive, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node, receive, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node, process, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request, and instruct the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

Some examples of the method, apparatuses, and computer-readable medium described herein may further include operations, features, means, or instructions for processing the cache to identify a replication log entry comprising the instruction for storing the data indicated in the request via the one or more access manager nodes of the distributed storage system and instructing the one or more access manager nodes of the distributed storage system to store the data indicated in the request based at least in part on the replication log entry.

Some examples of the method, apparatuses, and computer-readable medium described herein may further include operations, features, means, or instructions for receiving, from the one or more access manager nodes, a confirmation message indicating that the data indicated in the request may have been stored data by the one or more access manager nodes.

Some examples of the method, apparatuses, and computer-readable medium described herein may further include operations, features, means, or instructions for receiving, at the new leader control plane node, a second request to store data in the one or more access manager nodes, generating, by the new leader control plane node, a second cache data identifying a second instruction from the new leader control plane node to the one or more access manager nodes, the second instruction instructing the one or more access manager nodes to store the data indicated in the second request, and transmitting, by the new leader control plane node, a second replication instruction to one or more follower control plane nodes of the new leader control plane node to replicate the second cache data in a respective cache of the one or more follower control plane nodes.

Some examples of the method, apparatuses, and computer-readable medium described herein may further include operations, features, means, or instructions for instructing the one or more access manager nodes to store the data in the distributed storage system in accordance with the second instruction.

In some examples of the method, apparatuses, and computer-readable medium described herein, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples of the method, apparatuses, and computer-readable medium described herein, a second cache of the new leader control plane node is configured to achieve cluster fail over within a defined amount of time.

A platform of an online marketplace may use a distributed storage architecture to store data that scales as the amount of data increases. This storage architecture is designed with the goal of providing high throughput and low latency storage access. In some examples, the storage architecture may be implemented using a neuron cluster that includes a set of control plane nodes that store and replicate data for storage across data centers at different geographic locations. The set of control plane nodes may coordinate with a set of data plane nodes, referred to herein as access manager storage nodes. The access manager nodes may be used for storing data in underlying hard disks control by a respective access manager storage node. The neuron cluster may be designed continue to operate even if one of the control plane nodes fail. The neuron cluster may use a distributed consensus protocol to elect a leader control plane node. With a leader control plane node elected, the other control plane nodes of the cluster are referred to as follower control plane nodes.

In some examples, the leader control plane node may manage a subset of the storage nodes. Additionally or alternatively, the leader control plane node may instruct its storage nodes to perform data placement on underlying storage disks, and may coordinate with its storage nodes to retrieve data. The leader control plane node may coordinate with its subset of access manager storage nodes to store data at a first data center, and may coordinate with follower control plane nodes to replicate the data for storage of the data by a second subset of the access manager nodes at a second data center. If there is a failure of the control plane node, then the neuron cluster may elect one the follower control plane nodes to assume the leadership role. In some data storage systems when a leader fails, a cluster manager may read a local database, and reconstruct at least some or all cluster states of the failed leader control plane node. The cluster manager may send a leader change request to the storage nodes indicating that the leader has been changed from the old leader to a new control plane node. In addition, the cluster manager may provide information associated with the new leader to the storage nodes. The storage nodes may use this information for future communication with the new leader. However, this reconfiguration may take a long time (e.g., several minutes) and during this time, the storage nodes may not be able to communicate with the neuron cluster, thus contributing to latency.

One or more aspects of the present disclosure may provide for replication of cache data across multiple control plane nodes. The cache data may include a mapping between incoming data and an access manager node storing the data. In some examples, a neuron cluster may include multiple control plane nodes (referred to as "Neuron nodes"), where one control plane node may be a leader control plane node and the remaining control plane nodes may be follower control plane nodes. A leader control plane node may receive a request to store data in a distributed storage system. Upon receiving the request, the leader control plane node may generate and store cache data in its cache identifying instructions for storing or retrieving the data indicated in the request by one or more of access manager storage nodes of the distributed storage system at a first data center. For instance, the cache data stored in the cache may be a set of log entries indicating a set of one or more operations being performed by the leader control plane node and its access manager storage nodes, and may indicate whether a particular operation has been completed. The leader control plane node may also replicate the request to cause the follower nodes to perform the same operation with their access manager storage nodes of the distributed storage system at a second data center, such that the data is replicated and stored at different data centers.

According to some aspects, the leader control plane node may replicate its cache data for storage to a cache of one or more follower control plane nodes. By replicating the access instructions in each cache of each of the follower control plane nodes, the need to reconstruct at least one or all cluster states by a cluster manager when a leader fails prior to implementing a leader change is reduced or eliminated, and so is the corresponding latency. As described herein, each of the follower control plane nodes may include a replica of the cache of the leader control plane node (e.g., the caches may be coherent across the nodes of the cluster). In some examples, the new leader node may thus use its cache to continue from where the old leader node left off, resulting in reduced latency. As such, a failover response time by replicating a leader cache at each follower node provides a reduced latency during a change in leader, such as, for example, when the current leader node fails. Thus, maintaining the cache coherency between leader control plane node and follower control plane nodes may result in faster leader switching when a failure of a leader node occurs thereby enabling the distributed storage system to continue with nominal interruption to storage replication operations.

Aspects of the disclosure are initially described in the context of an environment supporting an on-demand database service. Aspects of the disclosure are further described in the context of a storage architecture, a recovery architecture, a distributed consensus protocol, and a process flow. Aspects of the disclosure are further illustrated by and described with reference to apparatus diagrams, system diagrams, and flowcharts that relate to techniques to achieve cache coherency across distributed storage clusters.

<FIG> illustrates an example of a system <NUM> for cloud computing that supports techniques to achieve cache coherency across distributed storage clusters in accordance with various aspects of the present disclosure. The system <NUM> includes cloud clients <NUM>, user devices <NUM>, cloud platform <NUM>, and data center <NUM>. Cloud platform <NUM> may be an example of a public or private cloud network. A cloud client <NUM> may access cloud platform <NUM> over network connection <NUM>. The network may implement transfer control protocol and internet protocol (TCP/IP), such as the Internet, or may implement other network protocols. A cloud client <NUM> may be an example of a computing device, such as a server (e.g., cloud client <NUM>-a), a smartphone (e.g., cloud client <NUM>-b), or a laptop (e.g., cloud client <NUM>-c). In other examples, a cloud client <NUM> may be a desktop computer, a tablet, a sensor, or another computing device or system capable of generating, analyzing, transmitting, or receiving communications. In some examples, a cloud client <NUM> may be part of a business, an enterprise, a non-profit, a startup, or another organization type.

A cloud client <NUM> may facilitate communication between the data center <NUM> and one or multiple user devices <NUM> to implement an online marketplace. The network connection <NUM> may include communications, opportunities, purchases, sales, or other interaction between a cloud client <NUM> and a user device <NUM>. A cloud client <NUM> may access cloud platform <NUM> to store, manage, and process the data communicated via one or more network connections <NUM>. In some cases, the cloud client <NUM> may have an associated security or permission level. A cloud client <NUM> may have access to certain applications, data, and database information within cloud platform <NUM> based on the associated security or permission level, and may not have access to others.

The user device <NUM> may interact with the cloud client <NUM> over network connection <NUM>. The network may implement transfer control protocol and internet protocol (TCP/IP), such as the Internet, or may implement other network protocols. The network connection <NUM> may facilitate transport of data via email, web, text messages, mail, or other appropriate form of electronic interaction (e.g., network connections <NUM>-a, <NUM>-b, <NUM>-c, and <NUM>-d) via a computer network. In an example, the user device <NUM> may be computing device such as a smartphone <NUM>-a, a laptop <NUM>-b, and also may be a server <NUM>-c or a sensor <NUM>-d. In other cases, the user device <NUM> may be another computing system. In some cases, the user device <NUM> may be operated by a user or group of users. The user or group of users may be a customer, associated with a business, a manufacturer, or other appropriate organization.

Cloud platform <NUM> may offer an on-demand database service to the cloud client <NUM>. In some cases, cloud platform <NUM> may be an example of a multi-tenant database system. In this case, cloud platform <NUM> may serve multiple cloud clients <NUM> with a single instance of software. However, other types of systems may be implemented, including-but not limited to-client-server systems, mobile device systems, and mobile network systems. In some cases, cloud platform <NUM> may support an online application. This may include support for sales between buyers and sellers operating user devices <NUM>, service, marketing of products posted by buyers, community interactions between buyers and sellers, analytics, such as user-interaction metrics, applications (e.g., computer vision and machine learning), and the Internet of Things. Cloud platform <NUM> may receive data associated with generation of an online marketplace from the cloud client <NUM> over network connection <NUM>, and may store and analyze the data. In some cases, cloud platform <NUM> may receive data directly from a user device <NUM> and the cloud client <NUM>. In some cases, the cloud client <NUM> may develop applications to run on cloud platform <NUM>. Cloud platform <NUM> may be implemented using remote servers. In some cases, the remote servers may be located at one or more data centers <NUM>.

Data center <NUM> may include multiple servers. The multiple servers may be used for data storage, management, and processing. Data center <NUM> may receive data from cloud platform <NUM> via connection <NUM>, or directly from the cloud client <NUM> or via network connection <NUM> between a user device <NUM> and the cloud client <NUM>. Data center <NUM> may utilize multiple redundancies for security purposes. In some cases, the data stored at data center <NUM> may be backed up by copies of the data at a different data center (not pictured).

Server system <NUM> may include cloud clients <NUM>, cloud platform <NUM>, cache replication component <NUM>, and data center <NUM> that may coordinate with cloud platform <NUM> and data center <NUM> to implement an online marketplace. In some cases, data processing may occur at one or more of the components of server system <NUM>, or at a combination of these components. In some cases, servers may perform the data processing. The servers may be a cloud client <NUM> or located at data center <NUM>.

The cache replication component <NUM> may communicate with cloud platform <NUM> via connection <NUM>, and may also communicate with data center <NUM> via connection <NUM>. The cache replication component <NUM> may receive signals and inputs from user device <NUM> via cloud clients <NUM> and via cloud platform <NUM> or data center <NUM>.

Some ecommerce systems may provide a marketplace where billions of products are listed from millions of sellers. The amount of new data creation may continuously grow and a large amount of new data may be created every day. Such data may include product listing information, media files, various logs generated by services in production, machine learning, and analytical workloads, and/or other secondary sources that monitor these transactions. In some instances, some of the data may be ephemeral while others may be durable for a long time to satisfy regulatory compliances. Such different use-cases may be associated with different service level objectives from the underlying storage in terms of latency and throughput. In some aspects, some storage systems may provide a block storage access.

Some conventional data storage solutions may provide for a data storage architecture including multiple storage nodes. The data storage architecture may be implemented by a neuron cluster including a set of control plane nodes. The set of control pane nodes may store data in one or more access manager storage nodes. A node, as described herein, may be implemented in hardware, software, or a combination of both. In addition, the neuron cluster may elect one of the control plane nodes as a leader control plane node and the remaining control plane nodes are designated as follower control plane nodes. In some examples, the leader control plane node may coordinate with its subset of access manager storage nodes to store data at a first data center, and may coordinate with follower control plane nodes to replicate the data for storage of the data by a second subset of the access manager nodes at a second data center. However, if a leader control plane node fails, it may be challenging to seamlessly continue data storage operation. That is, upon failure of a leader control plane node, the neuron cluster elects a new control plane node as the new leader. In such cases, the newly elected control plane node may be unaware of the cache of the prior leader control plane node. Thus, when a leader is changed from an old leader to a new leader, the reconfiguration may take a long time, thus contributing to latency.

In contrast, system <NUM> implements procedures and techniques to achieve cache coherency across distributed storage clusters. Specifically, server system <NUM> may implement operations similar to those as described herein. One or more components of server system <NUM>, including cache replication component <NUM>, as described herein, may operate to perform cache replication across multiple storage nodes. The cache replication component <NUM> within server system <NUM> may receive, at a leader control plane node of a set of control plane nodes of a node cluster, a request to store data in a distributed storage system including a set of access manager nodes. The cache replication component <NUM> within server system <NUM> may generate, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the set of access manager nodes. In some examples, the instruction may instruct the one or more access manager nodes to store the data indicated in the request. The server system <NUM> and cache replication component <NUM> may transmit, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the set of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes. The cache data may include a mapping between incoming data and an access manager node storing the data. In particular, the cache data may include an instruction for an access manager node to store data. The instruction may be in a form of a replication log entry. For high availability and fault tolerance, a cluster manager (implemented by the cache replication component <NUM>) may request replication of a storage node state machine transitions to a follower node. The leader node may serialize a replication log entry and store the replication log entry into the cache of the leader node. The leader node may then send the replication log entry to one or more follower nodes, where the log entry is deserialized and applied to the local key-value storage.

In some examples, the cache data may indicate one or more access manager nodes of a distributed storage system managed by the leader node. For example, the cache data may include an indication of the one or more access manager nodes used by a leader node to store data. Additionally or alternatively, the cache data may identify one or more storage volumes of the distributed storage system managed by the leader node, one or more storage pools of the distributed storage system managed by the leader node, or any combination thereof. A storage pool may include one or more storage nodes (and/or access manager nodes). In some examples, a storage pool may be an aggregation of storage from across different failure domains. A failure domain may represent an aggregation storage resources under a common top of a rack switch. Additionally or alternatively, storage pools may be used to store data from different tenants. For example, a storage pool may be uniquely identified by a combination of its name and an associated tenant identifier.

In some examples, the server system <NUM> and the cache replication component <NUM> receives, from a leader control plane node at a follower control plane node of a set of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node. The server system <NUM> and the cache replication component <NUM> may receive, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based on failure of the leader control plane node. The server system <NUM> and the cache replication component <NUM> may process, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system. In some cases, the instruction may instruct the one or more access manager nodes managed by the leader control plane node to store data indicated in a request. The server system <NUM> and the cache replication component <NUM> may then instruct the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

As one example, a decreased availability of control plane node may increase the failover response time by creating several local caches (one for each state machine such for storage node, volumes, and pools) within each node that is updated from leader to follower nodes at a time when there is a state machine transition happening on the leader. Aspects of the present disclosure may use replication techniques and may schedule a background thread to make the various caches coherent across the control plane nodes. Such cache coherency between control planes node and storage nodes during a leader change operation may effectively switch a few bits to enable the cluster manager to start accepting incoming data and metrics traffic. With this change, the system <NUM> may reactivate cluster management operations almost instantly after a leader failover. For example, when a leader control plane node fails, and another control plane node becomes leader, then the cache on that new leader node helps a cluster manager to achieve fail over within a defined amount of time (e.g., an instant cluster fail over or within a short amount of time).

It should be appreciated by a person skilled in the art that one or more aspects of the disclosure may be implemented in a system <NUM> to additionally or alternatively solve other problems than those described above. Furthermore, aspects of the disclosure may provide technical improvements to "conventional" systems or processes as described herein. However, the description and appended drawings only include example technical improvements resulting from implementing aspects of the disclosure, and accordingly do not represent all of the technical improvements provided within the scope of the claims.

<FIG> illustrates an example of a storage architecture <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. In the example of <FIG>, the storage architecture <NUM> is implemented using a neuron cluster <NUM>. The neuron cluster <NUM> includes a set of servers (e.g., one or more, and in some instances may include five or seven). Such servers may be known as replicas. Each server may include a storage node, a cache, a database, or any combination thereof. The neuron cluster <NUM> may provide for a storage architecture, where one or more replicas may continue to operate with quorum even under the failure of at least one of the replicas. If a replica fails, the storage architecture may provide for automatic spawning of a replacement replica. The replacement replica may then be added (as a follower control plane node) to the neuron cluster <NUM>.

In some aspects, the neuron cluster <NUM> may use a distributed consensus protocol to elect a leader control plane node, and the other control plane nodes of the cluster are referred to as follower control plane nodes. During the bootstrapping of the neuron cluster <NUM>, a workflow engine (bootstrapper service <NUM>) may create the replicas, and each replica may include an ability to boot up as a leader control plane node. In some examples, the workflow engine may randomly choose one of the replicas and designate it as a leader of the neuron cluster. The workflow engine may add the rest of the replicas to the leader control plane node as follower control plane nodes. In the example of <FIG>, the neuron cluster <NUM> may include Node <NUM>, Node <NUM> and Node <NUM> (although any number of nodes are contemplated). The workflow engine may elect Node <NUM> as a leader control plane node and Node <NUM> and Node <NUM> as follower control plane nodes. Once the neuron cluster <NUM> is formed, the storage architecture <NUM> may implement a protocol for management of the cluster operations. The storage architecture <NUM> may provide for replication of log entries to achieve faster failover in case of a change in the leader node. Each node may have a local key-value store (database <NUM>, database <NUM>, and database <NUM>) which the state machines are persisted. In some examples, a leader control plane node may initiate a state machine replication, and subsequent read operations and write operations may flow from the leader node to follower nodes.

A neuron architecture (e.g., neuron replicated control plane service), as depicted in <FIG>, may include five components: a cluster manager, a provisioner, a storage pool manager, an API server, and a raft manager. The cluster manager may configure cluster membership management of the storage nodes. The provisioner may manage volume related operations and may perform intelligent data placement. The storage pool manager may categorize the storage nodes based on tenants. The API server may provide an external API interface. The raft manager may implement a consensus protocol for leader election, may provide fault-tolerance by state machine replication of cluster meta-data and may manage a consistent cache across up to all replicas.

As depicted in the example of <FIG>, the neuron cluster <NUM> is facilitated by a bootstrapper service <NUM>, also known as a workflow engine. The bootstrapper service <NUM> may initiate the control plane nodes to form the neuron cluster <NUM>. The bootstrapper service <NUM> may then initiate one or more storage nodes also known as access manager nodes. The set of control plane nodes (Node <NUM>, Node <NUM> and Node <NUM>) may coordinate with a set of data plane nodes, referred to herein as access manager nodes, for storing data in underlying hard disks control by a respective access manager node. As depicted in the example of <FIG>, the bootstrapper service <NUM> initiates access manager nodes (access managers <NUM> through N) in storage pool <NUM>. The bootstrapper service <NUM> may also initiate multiple storage pools (storage pools <NUM> through N). In some examples, the storage nodes associated with each access manager may register with a cluster manager running on a leader control plane node. In the example where the leader control plane node is Node <NUM>, the storage nodes register with Node <NUM>. The cluster manager residing on Node <NUM> may assign a unique identifier to each of these nodes and may grant them a lease which the storage nodes renew during a time period (e.g., every few seconds). The storage or access manager nodes may include three main components: a storage engine (also known as Homestore), a control plane component, and a storage target. The storage engine may be a high-performance engine built using extensible b+tree. The control plane component may have a server running to create a volume automatically that includes replica and partitions which are on several different storage nodes. In some examples, a storage engine may be built with a fully asynchronous threading model. The storage engine may run in user-space and may be configured to bypass the kernel, and use stream based allocations - where the storage engine uses different types of block allocation schemes to balance the speed of allocation, write amplification, and fragmentation.

According to aspects depicted in the present disclosure, the leader control plane node (Node <NUM>) may receive a request to store data in a distributed storage system including the set of access manager nodes (access manager <NUM>, access manager <NUM> and access manager <NUM>). For example, the leader control plane node may receive a request from one or more compute nodes (compute nodes <NUM> through M) requesting storage of data. The leader control plane node (Node <NUM>) may generate cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the set of access manager nodes. Node <NUM> may generate cache data including the instruction and may store the cache data in Cache <NUM>. A cache, as described herein, may refer to a memory or other computer storage device capable of storing instructions, data, information, or the like. In some examples, the instruction may instruct the one or more access manager nodes to store the data indicated in the request. For instance, Node <NUM> may receive a request to store data portions P1, P2 and P3. The instructions may instruct the access manager <NUM> to store data portion P1, the access manager <NUM> to store data portion P2, and the access manager <NUM> to store data portion P3. The leader control plane node (Node <NUM>) may then transmit a replication instruction to one or more follower control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes. In the example of <FIG>, Node <NUM> may instruct Node <NUM> and Node <NUM> to replicate the cache data of Node <NUM> in a respective cache of Node <NUM> and Node <NUM>. Upon receiving the indication of replication, Cache <NUM> of Node <NUM> and Cache <NUM> of Node <NUM> may include the instruction instructing the access manager nodes (access manager <NUM>, access manager <NUM>, and access manager <NUM>) to store the data indicated in the request to Node <NUM>. By replicating the instructions across multiple caches, the techniques depicted herein provide for maintaining cache coherency across a set of control plane nodes to quickly establish a new leader node when a leader node fails, thus resulting in reduced latency.

As depicted herein, the storage architecture <NUM> may represent a central repository of a storage service. The storage architecture <NUM> may maintain a storage cluster metadata and may replicate the metadata across several nodes. The number of replicas may depend on the amount of fault tolerance desired in each deployed environment.

<FIG> illustrates an example of a recovery architecture <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. The recovery architecture <NUM> may be implemented using a neuron cluster <NUM>, and a state machine <NUM>. The state machine <NUM> may include a local database <NUM> and an in-memory cache <NUM>. As described with reference to <FIG> and <FIG>, the neuron cluster <NUM> maintains a distributed storage system including a set of control plane nodes. The neuron cluster <NUM> may designate a control plane node as a leader control plane node and other control plane nodes as follower control plane nodes. For instance, during bootstrapping, an administrator may choose a leader and add followers to the existing leader. The follower control plane nodes may each maintain a copy of cache data of the leader control plane node. If there is a change in leader, the new leader may broadcast information indicating the change to one or more registered storage nodes so that subsequent requests can be sent to the new leader. In some examples, techniques depicted herein may use a state machine replication technique to maintain consistency between different control plane nodes. Each committed operation in a log may be considered as an action to the state machine.

For the recovery architecture <NUM>, at <NUM>, the recovery process may read the local database <NUM>. At <NUM>, the recovery process may write the contents to an in-memory cache <NUM>. At <NUM>, the recovery process may check for incremental log entries from the neuron cluster <NUM>. Based on checking for incremental log entries, at <NUM>, the recovery process may capture the incremental logs from a last local snapshot point. At <NUM>, the recovery process may apply one or more missed committed log entries to the state machine <NUM>. Thus, the recovery architecture <NUM> may manage an in-memory cache <NUM> with a replacement policy to store both access manager nodes and volume related information (e.g., metadata) associated with up to each log entry to, for example, a newly elected leader to complete one or more unfinished tasks corresponding to a log entry that a prior leader did not complete due to, for example, failure of the prior leader. For a successful commit to the state machine <NUM>, the update is reflected in the in-memory cache <NUM> in addition to the local database <NUM>.

In some examples, the metadata size may be small enough to keep the metadata in-memory without a replacement policy. If one or more of the control plane nodes are rebooted in the neuron cluster <NUM>, it may first read the tables at the local database <NUM> (at <NUM>) and may then load the metadata information in the in-memory cache <NUM>. The neuron cluster <NUM> may then participate in communications (e.g., APIs, consensus, etc.). After that, the neuron cluster <NUM> may implement a protocol to check a commit sequence number with other replicas. The neuron cluster <NUM> may then synchronize with one or more other replicas by applying the uncommitted log entries to the state machine <NUM> (i.e., both local database <NUM> and in-memory cache <NUM>).

<FIG> illustrates an example of a distributed consensus protocol <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. According to the distributed consensus protocol <NUM>, a neuron cluster may select a control plane node as a leader node and may select one or more control plane nodes (distinct from the leader node) as follower control plane nodes. The distributed consensus protocol <NUM> may adopt a policy such that there is one leader at a time. A leader node may accept incoming write requests and may then replicate the write requests to the follower nodes. An incoming write request may correspond to a single log entry, which is stored in a separate log section. A log entry may be associated with a unique index number (e.g., a log number) in a monotonically increasing order.

As depicted in the example of <FIG>, the neuron cluster may select a leader node <NUM>, a first follower node <NUM> and a second follower node <NUM>. The leader node <NUM> may receive a write request. Once the leader node receives the write request, at <NUM>, the leader node <NUM> may append a log entry to its log section. For example, the leader node <NUM> may append log <NUM>. At <NUM>, the leader node <NUM> may replicate the request to the follower nodes (e.g., first follower node <NUM> and second follower node <NUM>). Each follower node may receive the replication request from the leader node <NUM>. In the example of <FIG>, the first follower node <NUM> and the second follower node <NUM> may receive the log entry (log <NUM>). At <NUM>, each follower node may append the log to its log section. For example, the first follower node <NUM> and the second follower node <NUM> may append log <NUM> and commit log <NUM>. At <NUM>, the follower nodes may return a response (e.g., acknowledgment) back to the leader node <NUM>. The distributed consensus protocol <NUM> may use a write policy where the leader node <NUM> commit the log entry and apply the log entry to the state machine once the leader node <NUM> gets responses from a majority of follower nodes including itself. At <NUM>, the leader node <NUM> may commit log <NUM>.

After commit, the leader node <NUM> may replicate the committed log number to the follower nodes, and the follower nodes may then commit the log corresponding to the given log number and apply it to their state machines. In some examples, a log replication request may include the committed log number of previous log replication, such that they are always pipelined. The leader node <NUM> may periodically send one or more updates to the follower nodes (e.g., follower node <NUM> and follower node <NUM>). In some examples, each follower node may maintain a randomized timer whose lower bound may be larger than the update interval. When a follower node receives the update, the follower node may reset its timer with a new random expiry. If the timer is expired due to no update within a threshold time period, the follower node may determine that the leader node has failed and may request a vote on the election of the next leader to up to all the other follower nodes. Once a majority of nodes vote for the node who initiated the election, the node becomes a leader and starts to serve write requests. In some examples, the node who initiated the election may become a leader based on a threshold number of nodes voting for the node. In other examples, a node may be elected leader based on the node having sufficient or the most processing bandwidth of the nodes, the node having sufficient processing capability to perform leader operations, the node being able to meet at least one latency criterion in performing leader operations, or the like.

In some examples, a conflict may occur during an absence of a leader node or a network disconnection. To resolve the conflict, techniques depicted herein may provide for a counter called term, which is increased when a new leader election is initiated. In some examples, up to follower nodes are configured to have the same term value as that of a current leader. Each replication request or update (from the leader node) may include the term value of a current leader node. Additionally or alternatively, up to each log entry may keep the term value of the time when the log entry was generated. If the current leader receives a message with higher term from a follower node, the message with the higher term may mean that a new leader has been elected by a majority of nodes. In such cases, the leader node may immediately give up the leader role and may become a follower. Conflict may happen when the leader node succeeds to append a new log entry to its log section, but fails to replicate the new log entry to followers due to various reasons such as server crash or network isolation. In such cases, a new leader is elected and the new leader may serve new logs. If the previous leader is recovered from the fault and re-joins as a follower, the previous leader and the current leader may see different log entries whose log number is the same. In such a case, the previous leader may find the last log entry whose term is the same as that of corresponding log entry in the current leader. Then, the previous leader may initiate overwriting log entries starting from that point using the current leader's log entries.

<FIG> illustrates an example of a process flow <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. Process flow <NUM> may include a user device <NUM> and a server system <NUM>. The server system <NUM> may be an example of server system <NUM> as described with reference to <FIG>. The user device <NUM> may be example of a user device <NUM> as described with reference to <FIG>. The server system <NUM> may receive data for storage from the user device <NUM>. The server <NUM> may include a leader control plane node (Node <NUM>), a follower control plane node (Node <NUM>) and an access manager node. Although a single follower node is shown, it is to be understood that the process flow <NUM> may support multiple follower nodes. Additionally or alternatively, although a single access manager node is shown, it is to be understood that the process flow <NUM> may support multiple access manager nodes.

The server may represent a set of computing components, data storage components, and the like, as described herein. In some examples, the operations illustrated in the process flow <NUM> may be performed by hardware (e.g., including circuitry, processing blocks, logic components, and other components), code (e.g., software or firmware) executed by a processor, or any combination thereof. Alternative examples of the following may be implemented, where some steps are performed in a different order than described or are not performed at all. In some cases, steps may include additional features not mentioned below, or further steps may be added.

At <NUM>, the leader control plane node (Node <NUM>) may receive a request to store data in a distributed storage system including a set of access manager nodes. As depicted herein, the leader control plane node may manage one or more access manager nodes. At <NUM>, the leader control plane node may generate cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node. In some examples, the instruction may instruct the one or more access manager nodes to store the data indicated in the request. In some examples, the leader control plane node may create, in the cache of the leader control plane node, a replication log entry including the instruction for storing the data indicated in the request by the one or more access manager nodes of the distributed storage system.

At <NUM>, the leader control plane node may transmit the instruction to the access manager node. The leader control plane node may instruct the one or more access manager nodes to store the data in the distributed storage system in accordance with the instruction. At <NUM>, the leader control plane node may receive, from the one or more access manager nodes, a confirmation message indicating that the data indicated in the request has been stored data by the one or more access manager nodes. In some examples, the leader control plane node may update the replication log entry of the cache to indicate that the data indicated in the request has been stored data by the one or more access manager nodes of the distributed storage system, or otherwise indicating that the access manager has completed the requested instruction.

At <NUM>, the leader control plane node may transmit a replication instruction to one or more follower control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes. The follower control plane node (Node <NUM>) may receive, from the leader control plane node, the replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node. At <NUM>, the follower control plane node may replicate the cache data.

At <NUM>, the follower control plane node may receive an indication that the follower control plane node is elected to be a new leader control plane node based on failure of the leader control plane node. In some cases, the follower control plane node may receive the election indication from elsewhere in the neuron cluster if, for example, the prior leader control plane node failed. At <NUM>, the follower control plane node may process its replicated cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system. In some examples, the instruction may instruct the one or more access manager nodes managed by the leader control plane node to store data indicated in a request. In some examples, the follower control plane node may identify a replication log entry associated with the prior leader node. The follower control plane node may identify incomplete log entries from the replication log entry associated with the prior leader node (e.g., due to failure of the prior leader node). For example, the follower control plane node may identify that at least one instruction has not been executed by the access manager nodes. At <NUM>, the follower control plane node may instruct the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache. For example, the follower control plane node may instruct the access manager nodes to complete an incomplete tasks at the prior leader node.

<FIG> shows a block diagram <NUM> of a device <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. The device <NUM> may include an input module <NUM>, an output module <NUM>, and a cache replication component <NUM>. The device <NUM> may also include a processor. Each of these components may be in communication with one another (e.g., via one or more buses).

The input module <NUM> may manage input signals for the device <NUM>. For example, the input module <NUM> may identify input signals based on an interaction with a modem, a keyboard, a mouse, a touchscreen, or a similar device. These input signals may be associated with user input or processing at other components or devices. In some cases, the input module <NUM> may utilize an operating system such as iOS®, ANDROID®, MS-DOS®, MS-WINDOWS®, OS/<NUM>®, UNIX®, LINUX®, or another known operating system to handle input signals. The input module <NUM> may send aspects of these input signals to other components of the device <NUM> for processing. For example, the input module <NUM> may transmit input signals to the cache replication component <NUM> to support techniques to achieve cache coherency across distributed storage clusters. In some cases, the input module <NUM> may be a component of an I/O controller <NUM> as described with reference to <FIG>.

The output module <NUM> may manage output signals for the device <NUM>. For example, the output module <NUM> may receive signals from other components of the device <NUM>, such as the cache replication component <NUM>, and may transmit these signals to other components or devices. In some examples, the output module <NUM> may transmit output signals for display in a user interface, for storage in a database or data store, for further processing at a server or server cluster, or for any other processes at any number of devices or systems. In some cases, the output module <NUM> may be a component of an I/O controller <NUM> as described with reference to <FIG>.

For example, the cache replication component <NUM> may include a request component <NUM>, an instruction component <NUM>, a replication instruction component <NUM>, a leader election component <NUM>, or any combination thereof. In some examples, the cache replication component <NUM>, or various components thereof, may be configured to perform various operations (e.g., receiving, monitoring, transmitting) using or otherwise in cooperation with the input module <NUM>, the output module <NUM>, or both. For example, the cache replication component <NUM> may receive information from the input module <NUM>, send information to the output module <NUM>, or be integrated in combination with the input module <NUM>, the output module <NUM>, or both to receive information, transmit information, or perform various other operations as described herein.

The cache replication component <NUM> may support data storage in accordance with examples as disclosed herein. The request component <NUM> may be configured as or otherwise support a means for receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes. The instruction component <NUM> may be configured as or otherwise support a means for generating, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request. The replication instruction component <NUM> may be configured as or otherwise support a means for transmitting, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes.

Additionally or alternatively, the cache replication component <NUM> may support data storage in accordance with examples as disclosed herein. The replication instruction component <NUM> may be configured as or otherwise support a means for receiving, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node. The leader election component <NUM> may be configured as or otherwise support a means for receiving, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node. The instruction component <NUM> may be configured as or otherwise support a means for processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request. The instruction component <NUM> may be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

<FIG> shows a block diagram <NUM> of a cache replication component <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. The cache replication component <NUM> may be an example of aspects of a cache replication component or a cache replication component <NUM>, or both, as described herein. The cache replication component <NUM>, or various components thereof, may be an example of means for performing various aspects of techniques to achieve cache coherency across distributed storage clusters as described herein. For example, the cache replication component <NUM> may include a request component <NUM>, an instruction component <NUM>, a replication instruction component <NUM>, a leader election component <NUM>, a log entry component <NUM>, a failure component <NUM>, a confirmation component <NUM>, or any combination thereof. Each of these components may communicate, directly or indirectly, with one another (e.g., via one or more buses).

In some examples, the instruction component <NUM> may be configured as or otherwise support a means for instructing the one or more access manager nodes to store the data in the distributed storage system in accordance with the instruction. In some examples, to support generating the cache data, the log entry component <NUM> may be configured as or otherwise support a means for creating, in the cache of the leader control plane node, a replication log entry comprising the instruction for storing the data indicated in the request by the one or more access manager nodes of the distributed storage system.

In some examples, the confirmation component <NUM> may be configured as or otherwise support a means for receiving, from the one or more access manager nodes, a confirmation message indicating that the data indicated in the request has been stored data by the one or more access manager nodes. In some examples, the log entry component <NUM> may be configured as or otherwise support a means for updating the replication log entry of the cache to indicate that the data indicated in the request has been stored data by the one or more access manager nodes of the distributed storage system.

In some examples, the failure component <NUM> may be configured as or otherwise support a means for determining a failure of the leader control plane node. In some examples, the leader election component <NUM> may be configured as or otherwise support a means for transmitting, to a follower control plane node of the one or more follower control plane nodes, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node.

In some examples, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples, the request to store data in the distributed storage system comprises a write request.

Additionally or alternatively, the cache replication component <NUM> may support data storage in accordance with examples as disclosed herein. In some examples, the replication instruction component <NUM> may be configured as or otherwise support a means for receiving, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node. The leader election component <NUM> may be configured as or otherwise support a means for receiving, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node. In some examples, the instruction component <NUM> may be configured as or otherwise support a means for processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request. In some examples, the instruction component <NUM> may be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

In some examples, the log entry component <NUM> may be configured as or otherwise support a means for processing the cache to identify a replication log entry comprising the instruction for storing the data indicated in the request via the one or more access manager nodes of the distributed storage system. In some examples, the instruction component <NUM> may be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data indicated in the request based at least in part on the replication log entry.

In some examples, the confirmation component <NUM> may be configured as or otherwise support a means for receiving, from the one or more access manager nodes, a confirmation message indicating that the data indicated in the request has been stored data by the one or more access manager nodes.

In some examples, the request component <NUM> may be configured as or otherwise support a means for receiving, at the new leader control plane node, a second request to store data in the one or more access manager nodes. In some examples, the instruction component <NUM> may be configured as or otherwise support a means for generating, by the new leader control plane node, a second cache data identifying a second instruction from the new leader control plane node to the one or more access manager nodes, the second instruction instructing the one or more access manager nodes to store the data indicated in the second request. In some examples, the instruction component <NUM> may be configured as or otherwise support a means for transmitting, by the new leader control plane node, a second replication instruction to one or more follower control plane nodes of the new leader control plane node to replicate the second cache data in a respective cache of the one or more follower control plane nodes.

In some examples, the instruction component <NUM> may be configured as or otherwise support a means for instructing the one or more access manager nodes to store the data in the distributed storage system in accordance with the second instruction. In some examples, the cache data indicates one or more access manager nodes of the distributed storage system managed by the leader control plane node, or one or more storage volumes of the distributed storage system managed by the leader control plane node, or one or more storage pools of the distributed storage system managed by the leader control plane node, or any combination thereof. In some examples, a second cache of the new leader control plane node may be configured to achieve cluster fail over within a defined amount of time.

<FIG> shows a diagram of a system <NUM> including a device <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. The device <NUM> may be an example of or include the components of a device <NUM> as described herein. The device <NUM> may include components for bi-directional data communications including components for transmitting and receiving communications, such as a cache replication component <NUM>, an I/O controller <NUM>, a database controller <NUM>, a memory <NUM>, a processor <NUM>, and a database <NUM>. These components may be in electronic communication or otherwise coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more buses (e.g., a bus <NUM>).

The I/O controller <NUM> may manage input signals <NUM> and output signals <NUM> for the device <NUM>. In some cases, the I/O controller <NUM> may be implemented as part of a processor <NUM>. In some examples, a user may interact with the device <NUM> via the I/O controller <NUM> or via hardware components controlled by the I/O controller <NUM>.

The database controller <NUM> may manage data storage and processing in a database <NUM>. In some cases, a user may interact with the database controller <NUM>. In other cases, the database controller <NUM> may operate automatically without user interaction. The database <NUM> may be an example of a single database, a distributed database, multiple distributed databases, a data store, a data lake, or an emergency backup database.

Memory <NUM> may include random-access memory (RAM) and ROM. The memory <NUM> may store computer-readable, computer-executable software including instructions that, when executed, cause the processor <NUM> to perform various functions described herein.

The processor <NUM> may include an intelligent hardware device, (e.g., a general-purpose processor, a DSP, a CPU, a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the processor <NUM> may be configured to operate a memory array using a memory controller. In other cases, a memory controller may be integrated into the processor <NUM>. The processor <NUM> may be configured to execute computer-readable instructions stored in a memory <NUM> to perform various functions (e.g., functions or tasks supporting techniques to achieve cache coherency across distributed storage clusters ).

The cache replication component <NUM> may support data storage in accordance with examples as disclosed herein. For example, the cache replication component <NUM> may be configured as or otherwise support a means for receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes. The cache replication component <NUM> may be configured as or otherwise support a means for generating, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request. The cache replication component <NUM> may be configured as or otherwise support a means for transmitting, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes.

Additionally or alternatively, the cache replication component <NUM> may support data storage in accordance with examples as disclosed herein. For example, the cache replication component <NUM> may be configured as or otherwise support a means for receiving, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node. The cache replication component <NUM> may be configured as or otherwise support a means for receiving, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node. The cache replication component <NUM> may be configured as or otherwise support a means for processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request. The cache replication component <NUM> may be configured as or otherwise support a means for instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache.

By including or configuring the cache replication component <NUM> in accordance with examples as described herein, the device <NUM> may support techniques for improved data storage functionality, reduced latency, improved user experience related to efficient data processing, and more efficient utilization of storage resources.

<FIG> shows a flowchart illustrating a method <NUM> that supports techniques to achieve cache coherency across distributed storage clusters in accordance with aspects of the present disclosure. The operations of the method <NUM> may be implemented by a storage architecture or its components as described herein. For example, the operations of the method <NUM> may be performed by a storage architecture as described with reference to <FIG>. In some examples, a storage architecture may execute a set of instructions to control the functional elements of the storage architecture to perform the described functions. Additionally or alternatively, the storage architecture may perform aspects of the described functions using special-purpose hardware.

At <NUM>, the method may include receiving, at a leader control plane node of a plurality of control plane nodes of a node cluster, a request to store data in a distributed storage system comprising a plurality of access manager nodes. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by a request component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include generating, by the leader control plane node, cache data identifying an instruction from the leader control plane node to one or more access manager nodes managed by the leader control plane node of the plurality of access manager nodes, the instruction instructing the one or more access manager nodes to store the data indicated in the request. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by an instruction component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include transmitting, by the leader control plane node, a replication instruction to one or more follower control plane nodes of the plurality of control plane nodes to replicate the cache data in a respective cache of the one or more follower control plane nodes. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by a replication instruction component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include instructing the one or more access manager nodes to store the data in the distributed storage system in accordance with the instruction. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by an instruction component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include determining a failure of the leader control plane node. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by a failure component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include transmitting, to a follower control plane node of the one or more follower control plane nodes, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by a leader election component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include receiving, from a leader control plane node at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by a replication instruction component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include receiving, at the follower control plane node, an indication that the follower control plane node is elected to be a new leader control plane node based at least in part on failure of the leader control plane node. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by a leader election component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include processing, by the new leader control plane node, the cache to retrieve an instruction from the leader control plane node to one or more access manager nodes of a distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in a request. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by an instruction component <NUM> as described with reference to <FIG>.

At <NUM>, the method may include instructing the one or more access manager nodes of the distributed storage system to store the data in accordance with the instruction retrieved from the cache. The operations of <NUM> may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of <NUM> may be performed by an instruction component <NUM> as described with reference to <FIG>.

By way of example, and not limitation, non-transitory computer-readable media can comprise RAM, ROM, electrically erasable programmable ROM (EEPROM), compact disk (CD) ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor.

Claim 1:
A computer-implemented method for data storage, comprising:
receiving (<NUM>),
from a leader control plane node, at a follower control plane node of a plurality of control plane nodes of a node cluster, a replication instruction to replicate cache data of the leader control plane node in a cache of the follower control plane node and to instruct one or more access manager nodes of a distributed storage system at a second data center to store data in the distributed storage system in accordance with a request received by the leader control plane node, to cause the data to be replicated and stored at a first data center and the second data centers, the cache data identifying an instruction from the leader control plane node to one or more access manager nodes of the distributed storage system at the first data center managed by the leader control plane node, the instruction instructing the one or more access manager nodes of the distributed storage system at the first data center to store the data indicated in the request;
processing, by the follower control plane node, the cache data to retrieve an instruction from the leader control plane node to one or more access manager nodes of the distributed storage system, the instruction instructing the one or more access manager nodes managed by the leader control plane node to store data indicated in the request; and
instructing, by the follower control plane node, the one or more access manager nodes of the distributed storage system at the second data center to store the data in accordance with the instruction retrieved from the cache.