Patent Description:
The present technology pertains to distributed storage systems, and more specifically to master data placement in distributed storage systems.

The ubiquity of Internet-enabled devices has created an enormous demand for Internet services and content. In many ways, we have become an inter-connected society where users are increasingly reliant on network services and content. This Internet and inter-connectivity revolution has created significant challenges for content and storage providers who struggle to service a high volume of client requests while often falling short of performance expectations. For example, data providers typically need large and complex datacenters to keep up with network and data demands from users. These datacenters are often equipped with server farms configured to host specific data and services, and include numerous network devices configured to route and process data requests. In many cases, a specific datacenter is expected to handle millions of traffic flows and data requests.

Not surprisingly, such large volumes of data can be difficult to manage and create significant performance degradations and challenges. In some cases, load balancing solutions may be implemented to improve performance and service reliability. However, current load balancing solutions are prone to node failures, often fail to adequately account for dynamic changes and fluctuations in the network and data requests, and may be susceptible to latency and bottlenecks. Additional resources can be purchased and implemented to increase the capacity of the network and thereby reduce latency and performance issues. Unfortunately, this approach is expensive, introduces added complexity to the network, and remains susceptible to network fluctuations and varying data access patterns, which can lead to latency from overload conditions, waste from underload conditions, and highly variable performance.

<CIT> is directed to systems and corresponding methods for storing and/or redistributing data within a network. In various aspects, data and/or sets of data stored in a database, data store, or other type of database storage system may be pulled, pushed, distributed, redistributed, or otherwise positioned at one or more data caches and/or servers strategically located across an enterprise network, a content delivery network ("CDN"), etc., and may be accessible over such networks, other networks, and/or the Internet.

<CIT> is directed to controlling the storage of data among multiple regional storage centers coupled through a network in a global storage system. A method includes steps of: defining at least one dataset comprising at least a subset of the data stored in the global storage system; defining at least one ruleset for determining where to store the dataset; obtaining information regarding a demand for the dataset through one or more data requesting entities operating in the global storage system; and determining, as a function of the ruleset, information regarding a location for storing the dataset among regional storage centers having available resources that reduces the total distance traversed by the dataset in serving at least a given one of the data requesting entities and/or reduces the latency of delivery of the dataset to the given one of the data requesting entities.

In order to describe the manner in which the various advantages and features of the disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings.

Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes. The invention of the present European patent is set out in the appended claims.

Disclosed herein are systems, methods, and computer-readable media for intelligent and dynamic management of data item placements in distributed, stateful storage systems. The data placement techniques herein can reduce data access latencies and increase data access performance by intelligently placing master copies of data items in certain locations on a distributed storage system based on access patterns, network statistics, and/or events or conditions.

According to at least one example, a method for managing a placement of data items in a distributed storage system is provided. The method is described in claim <NUM>.

According to at least one example, a system for managing a placement of data items in a distributed storage system is provided. The system is described in claim <NUM>.

According to at least one example, a non-transitory computer-readable storage medium for managing a placement of data items in a distributed storage system is provided. The non-transitory computer-readable storage medium is described in claim <NUM>.

In at least some aspects, the method, system, and non-transitory computer-readable storage medium described above can include collecting information associated with the distributed storage system, the information including statistics associated with one or more resources, one or more data access restrictions associated with one or more data items on the distributed storage system, one or more events, data access patterns, and/or network statistics associated with at least one of the distributed storage system and one or more networks associated with the distributed storage system, wherein the one or more resources include a storage node, a compute node, a virtual machine, a software container, a server, a network, and/or a networking device; based on the information associated with the distributed storage system, determining a data placement action estimated to improve a data access performance associated with one or more data items on the distributed storage system or improve a performance of the distributed storage system, the data placement action including moving at least one data item from at least one storage location to at least one different storage location, the at least one storage location and the at least one different storage location including different data stores from the plurality of data stores; and in response to determining the data placement action, moving the at least one data item from the at least one storage location to the at least one different storage location.

In some aspects, determining the different location on the distributed storage system for storing the master copy of the data item can include, based on the access pattern associated with the master copy of the data item, identifying, from the one or more originating locations, an originating location of a highest number of access requests associated with the master copy of the data item; determining which of the plurality of data stores is located closest to the originating location of the highest number of access requests associated with the master copy of the data item; and determining that the different data store associated with the different location is located closest to the originating location of the highest number of access requests associated with the master copy of the data item.

In some examples, the one or more originating locations can correspond to one or more client devices that generated the set of access requests received by the distributed storage system for the master copy of the data item. Moreover, in some examples, placing the master copy of the data item at the different location on the distributed storage system can include moving the master copy of the data item from the current data store to the different data store.

In some cases, determining that the different data store is located closest to the originating location of the highest number of access requests associated with the master copy of the data item can include determining that a number of hops between the different data store and the originating location is less than a respective number of hops between each of the plurality of data stores and each of one or more remaining locations from the one or more originating locations, and/or determining that a distance between the different data store and the originating location is less than a respective distance between each of the plurality of data stores and each of the one or more remaining locations from the one or more originating locations.

In some aspects, determining the different location on the distributed storage system for storing the master copy of the data item can include determining a second current location of a second master copy of a second data item stored on the distributed storage system, wherein the second current location of the second master copy of the second data item includes a second current data store from the plurality of data stores on the distributed storage system; selecting, based on the access pattern associated with the master copy of the data item and a second access pattern associated with the second master copy of the second data item, the different location on the distributed storage system for storing both the master copy of the second data item and the second master copy of the second data item; and placing both the master copy of the data item and the second master copy of the second data item at the different location on the distributed storage system.

In some examples, the second access pattern can include one or more respective originating locations of a second set of access requests associated with the second master copy of the second data item and a second respective number of access requests received from each of the one or more respective originating locations. In some cases, placing both the master copy of the data item and the second master copy of the second data item at the different location on the distributed storage system can include moving the master copy of the data item from the current location to the different location and moving the second master copy of the second data item from the second current location to the different location. Moreover, in some examples, the master copy of the data item can include a first partition of a partitioned data set and the second master copy of the second data item can include a second partition of the partitioned data set.

In some aspects, the method, system, and non-transitory computer-readable storage medium described above can include determining that the master copy of the data item and a second data item on the distributed storage system have been accessed together a threshold number of times; and after determining the different location on the distributed storage system for storing the master copy of the data item, moving the second data item from a current respective location of the second data item to the different location, the current respective location including one of the plurality of data stores on the distributed storage system.

the method, system, and non-transitory computer-readable storage medium described above can include determining that the master copy of the data item includes a reference to a particular copy of a second data item stored on the distributed storage system, wherein the particular copy of the second data item includes one of a respective master copy of the second data item or a replica of the respective master copy of the second data item; selecting, in response to determining the different location on the distributed storage system for storing the master copy of the data item and determining that the master copy of the data item comprises the reference to the particular copy of the second data item, the different location on the distributed storage system for storing the particular copy of the second data item; and moving the particular copy of the second data item from a respective location on the distributed storage system to the different location on the distributed storage system.

This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this application, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

Distributed, stateful storage systems manage state across a network environment to provide read and write access to data items. In some examples, state can be managed in a centralized manner, with one or more data stores located in a central location, such as a core network, configured to manage read and write access to data items. Here, clients can perform read and write operations by sending requests to the one or more data stores in the central location. In some cases, read-only state replicas can optionally be managed in a decentralized manner. For example, read-only state replicas can be managed by one or more data stores located "at the edge" (e.g., caches) of the network. The one or more data stores at the edge of the network can manage read-only access to data items, and clients can perform read operations by sending requests to a decentralized data store (e.g., a cache) from the one or more data stores at the edge of the network.

While implementing data stores or caches at the edge of the network can improve performance for read operations, write operations can experience latency and reduced performance as the requesting clients are often located far from the data stores in the central location. For example, the distance between the clients and the centralized data stores providing such clients write access to data items can increase the latency and reduce the data access performance for write operations from such clients, as the added distance increases the number of networks, devices, and potential bottlenecks traversed by the data access communications between such clients and the centralized data stores.

In some cases, to reduce the latency and increase the data access performance for write operations from clients, the approaches herein can intelligently place master copies of data items at strategic locations on the distributed storage environment based on data access patterns such as data access locations (e.g., the location of a client relative to the data item) and frequency. For example, a master copy of a data item frequently accessed by a particular client can be moved to a data store located closer to the client based on the access patterns calculated for the client and/or the master copy of the data item. This can reduce the number of hops traversed by data access communications to and from the client, decrease the number of potential bottlenecks traversed by such data access communications, and generally decrease latency and increase data access performance for that client.

In some cases, other factors can also be considered when selecting a location for storing or moving a master copy of a data item. For example, when determining a strategic location to place a master copy of a data item (and/or a replica), the system can take into account various factors such as cost considerations, network performance considerations, client access requirements, client subscription levels (e.g., premium access levels, standard access levels, etc.), access patterns associated with a group of data items and/or clients, a type of data items, a relationship of a data item to other data items (e.g., a partition of a data item, a referencing data item, etc.), historical and/or predicted access patterns, etc..

As further described below, the disclosed technology provides systems, methods, and computer-readable media for master data placement in distributed storage systems. The present technology will be described in the subsequent disclosure as follows. The discussion begins with a description of an example distributed storage system, as illustrated in <FIG>, and a description of various examples and techniques for master data placement in a distributed storage system, as illustrated in <FIG>. A description of an example method for master data placement in distributed storage system, as illustrated in <FIG>, will then follow. The discussion concludes with a description of an example network device, as illustrated in <FIG>, and an example computing device architecture including example hardware components suitable for performing storage and computing operations, as illustrated in <FIG>. The disclosure now turns to <FIG>.

<FIG> is a simplified block diagram of an example distributed storage system <NUM>, in accordance with some examples. In this example, the distributed storage system <NUM> includes a core <NUM> and edges <NUM>A-N (collectively "<NUM>" hereinafter). The core <NUM> can serve as the backbone, centralized network and/or central hub for network and storage services provided by the distributed storage system <NUM>. Moreover, the core <NUM> can include one or more networks such as, for example, a cloud network, a datacenter, etc., and/or one or more segments of a network environment of the distributed storage system <NUM>, such as a core segment or hub.

The edges <NUM> can be connected to the core <NUM> via one or more network devices, such as, for example, one or more switches or routers and/or via one or more networks such as, for example, one or more public networks (e.g., wide-area networks, public clouds, etc.), one or more private networks (e.g., private datacenters, private clouds, local area networks, virtual private networks, etc.), and/or one or more hybrid networks (e.g., hybrid clouds, etc.). In some examples, the edges <NUM> can be interconnected with each other through the core <NUM> and/or directly (or without going through the core <NUM>).

In some cases, the edges <NUM> can represent segments or sections of a network environment associated with the distributed storage system <NUM>. For example, the edges <NUM> can be network segments or sections located on an edge or periphery of a network environment associated with the core <NUM> and/or the distributed storage system <NUM>. In other cases, the edges <NUM> can represent separate networks such as, for example, fog networks, local area networks (LANs), on-premises datacenters, enterprise networks, etc. In some examples, such networks can be located on an edge or periphery of a network environment associated with the distributed storage system <NUM>. Thus, the edges <NUM> can be physically and/or logically situated closer to one or more clients <NUM>-<NUM> than the core <NUM>.

The core <NUM> can include one or more storage nodes <NUM> for storing or hosting one or more data stores <NUM>. Similarly, the edges <NUM> can include storage nodes <NUM> for storing or hosting data stores <NUM>-<NUM>. For example, edge 110A can include one or more storage nodes <NUM> for storing one or more data stores <NUM>, edge 110B can include one or more storage nodes <NUM> for storing one or more data stores <NUM>, edge 110C can include one or more storage nodes <NUM> for storing one or more data stores <NUM>, and edge 110N can include one or more storage nodes <NUM> for storing one or more data stores <NUM>.

The storage nodes <NUM> and <NUM> can represent hardware and/or virtual storage infrastructure on the distributed storage system <NUM>. Moreover, the storage nodes <NUM> and <NUM> can include one or more physical storage servers, one or more virtual storage servers (e.g., virtual machines (VMs), software containers, etc.), one or more physical and/or logical storage components (e.g., storage drives, logical volumes, storage partitions, storage arrays, etc.), and/or any other physical and/or virtual/logical storage element. Each of the storage nodes <NUM> and <NUM> can be implemented by an individual storage element or can span or be distributed across multiple storage elements and provide a distributed storage infrastructure. In some cases, a storage node can span multiple physical or virtual storage elements. For example, a storage node can represent a virtual storage device, container, or location created from two or more physical servers and/or storage devices.

In some cases, the storage nodes <NUM> and <NUM> can be grouped into storage node pools or clusters. For example, the storage nodes <NUM> on the core <NUM> can be grouped into one or more storage node pools or clusters, and the storage nodes <NUM> on each of the edges <NUM> can be grouped into one or more storage node pools or clusters. Storage nodes can be grouped into storage node pools or clusters based on one or more factors, such as one or more common characteristics. For example, storage nodes can be grouped into storage node pools or clusters by storage type, type of data (e.g., the type of data they store), underlying storage platform, physical or virtual location, capacity, configuration settings or architecture, storage role, priorities, network segments (e.g., IP prefixes or subnets), shared resources, operating conditions, etc. In some cases, a pool or cluster of storage nodes (e.g., <NUM> and/or <NUM>) can be configured to function as a single storage node or distributed storage. In other cases, a pool or cluster of storage nodes (e.g., <NUM> and/or <NUM>) can represent a collection of storage nodes which can operate separately and/or individually.

The data stores <NUM> on the core <NUM> and the data stores <NUM>-<NUM> on the edges <NUM> can include storage repositories, containers or structures for persistently storing and managing master data items 108A-N and replica data items 130A-N. The data stores <NUM> on the core <NUM> and the data stores <NUM>-<NUM> on the edges <NUM> can include, for example, databases, files, file systems, storage systems, and/or any other data repositories. In some implementations, the data stores <NUM> and <NUM>-<NUM> can include one or more databases.

Each of the master data items 108A-N and replica data items 130A-N on the data stores <NUM> and <NUM>-<NUM> can include, for example, a record (e.g., a database record), one or more database fields, a data object, a data structure containing data or data values, a collection of data elements, a data value(s), a data partition, a content item, etc. In some examples, a data item (e.g., 108A-N and/or 130A-N) on the data stores <NUM> and/or <NUM>-<NUM> can include any type of self-contained (or largely self-contained) data item, such as a profile, a record, a table row, a set of data that does not reference other data items (or has limited references to other data items), etc..

The master data items 108A-N in the data stores <NUM> on the core <NUM> can represent master copies of the replica data items 130A-N, while the replica data items 130A-N in the data stores <NUM>-<NUM> on the edges <NUM> can represent replicas or read-only copies of the master data items 108A-N in the data stores <NUM> on the core <NUM>. The master data items 108A-N can be read-write data items and can provide the source of truth (e.g., the current and/or authoritative data state and/or version) for the replica data items 130A-N. For example, master data item 108A can be a read-write data item and can provide the current and authoritative state or version of the data associated with the master data item 108A and the replica data item 130A, and master data item 108N can be a read-write data item and can provide the current and authoritative state or version of the data associated with the master data item 108N and replica data item <NUM>N.

Clients <NUM>-<NUM> can access the master data items 108A-N and replica data items 130A-N through the distributed storage system <NUM>. In particular, clients <NUM>-<NUM> can access the master data items 108A-N and replica data items 130A-N through the core <NUM> and/or the edges <NUM>. For example, for read operations, clients <NUM>-<NUM> can access master data items 108A-N and/or replica data items 130A-N from the core <NUM> and/or the edges <NUM>, and for write operations, clients <NUM>-<NUM> can access the master data items 108A-N (e.g., the master copies of the replica data items 130A-N) through the core <NUM>. However, as further described below, master data items 108A-N can be moved or placed elsewhere on the distributed storage system <NUM> based on one or more factors. Thus, in some cases, clients <NUM>-<NUM> can access master data items 108A-N from one or more of the edges <NUM>.

Clients <NUM>-<NUM> can represent any computing devices or networks. For example, in some cases, clients <NUM>-<NUM> can include one or more client endpoints (e.g., client computing devices such as personal computers, smartphones, tablet computers, smart televisions, gaming systems, set-top boxes, smart wearables, etc.), one or more servers, one or more Internet-of-Things (IoT) devices, one or more autonomous vehicles, one or more network devices (e.g., switches, routers, etc.), etc. In other cases, clients <NUM>-<NUM> can include one or more networks such as, for example, one or more LANs, one or more datacenters, one or more enterprise networks, one or more campus networks, one or more private networks, etc..

The distributed storage system <NUM> can include a coordinator system <NUM> that can collect and analyze information about the distributed storage system <NUM> and coordinate or orchestrate the placement or movement of master data items 108A-N and replica data items 130A-N on the distributed storage system <NUM>. Moreover, the coordinator system <NUM> can include, or can be implemented by, one or more computing devices (physical and/or virtual). For example, in some cases, the coordinator system <NUM> can be implemented by a server, a network controller, an orchestrator appliance, a router, or any other computing device. In other cases, the coordinator system <NUM> can be implemented by multiple devices, such as multiple servers, multiple network controllers, multiple orchestrator appliances, multiple routers, etc..

In some examples, the coordinator system <NUM> can track and monitor statistics and information associated with the distributed storage system <NUM>, the master data items 108A-N, and/or the replica data items 130A-N and move (or instruct the distributed storage system <NUM> to move) one or more of the master data items 108A-N and/or the replica data items 130A-N to a specific location(s) in the distributed storage system <NUM> based on the statistics and information tracked and monitored by the coordinator system <NUM>.

To illustrate, if the statistics and information monitored by the coordinator system <NUM> indicate that master data item 108A is frequently accessed by client <NUM> from a location that is closest to edge 110A, the coordinator system <NUM> can trigger a move of the master data item 108A from the data stores <NUM> on the core <NUM> to the data stores <NUM> on the edge 110A. Such a move would reduce the distance (as well as the number of hops and/or potential number of bottlenecks) between the master data item 108A and the client <NUM>, which frequently accesses the master data item 108A, and thus may decrease the latency and increase the access performance experienced by the client <NUM> when accessing the master data item 108A from the data stores <NUM> on the edge 110A - as opposed to accessing the master data item 108A from the data stores <NUM> on the core <NUM>.

In some cases, the coordinator system <NUM> can use the collected information to coordinate or orchestrate the move of replica data items 130A-N on the distributed storage system <NUM>. For example, when moving the master data item 108A from the data stores <NUM> on the core <NUM> to the data stores <NUM> on the edge 110A as described in the previous example, the coordinator system <NUM> can also move replica data item 130A on the data stores <NUM> of the edge 110A to the data stores <NUM> on the core <NUM>. As another example, if the information collected and monitored by the coordinator system <NUM> indicates that replica data item 130A is frequently accessed by client <NUM> from a location that is closest to the core <NUM> or receives a faster response time from core <NUM> than edges <NUM>, the coordinator system <NUM> can trigger a move of the replica data item 130A from one of the edges <NUM> to the core <NUM>. As yet another example, if the information collected and monitored by the coordinator system <NUM> indicates that replica data item 130N is infrequently accessed from edge 110N, the coordinator system <NUM> can trigger a move of the replica data item <NUM>N from edge 110N to the core <NUM> in order to reduce network and/or resource utilization at edge 110N (and thereby increase bandwidth, reduce congestion, and increase resource availability at edge 110N).

Moreover, in some cases, rather than moving a master data item from a current data store to a different, destination data store, the coordinator system <NUM> can instead designate a replica of the master data item (e.g., replica data item 130A or 130N) stored at the different, destination data store as the master data item, and designate the previous master data item at the current data store as a replica data item. In some examples, if the coordinator system <NUM> changes the designation of a replica data item to a master data item instead of moving the master data item to the location of the replica data item, the coordinator system <NUM> can perform a consistency check to ensure that the replica data item being designated as the master data item reflects the most current data or is not outdated and/or to check that designating the replica data item as a master data item does not create any conflicts or errors.

In some cases, if the coordinator system <NUM> determines that designating a replica data item as a master data item does create a conflict or error and/or that the replica data item does not reflect the most current data or is outdated, the coordinator system <NUM> can try to correct such conflict or error and/or update the replica data item to reflect the most current data prior to designating the replica data item as a master data item. In other cases, rather than trying to correct such conflict or error and/or update the replica data item to reflect the most current data, the coordinator system <NUM> can decide to not designate the replica data item as a master data item and instead decide to move the actual master data item to the location where the replica data item is stored. In such cases, the coordinator system <NUM> can leave the replica data item as a replica, and can either leave the replica data item at its current location or move the replica data item to a different location, such as the location where the master data item was moved from.

In some examples, the statistics and information tracked/monitored by the coordinator system <NUM> and used to trigger data placement actions can include access patterns and/or metrics associated with the master data items 108A-N, the replica data items 130A-N, the core <NUM>, one or more of the edges <NUM>, and/or one or more of the clients <NUM>-<NUM>. In some cases, the coordinator system <NUM> can use such information to determine whether an access latency and/or access performance associated with a particular master data item and/or a particular replica data item can be improved by moving the master or replica data item to a different location (e.g., a different data store, storage node, segment, and/or network) in the distributed storage system <NUM>. For example, the coordinator system <NUM> can use such information to determine whether moving a master or replica data item to a particular data store and/or location (e.g., core <NUM>, edge 110A, edge 110B, edge 110C, or edge 110N) located closest to a client that has the most interactions or the most frequent interactions (e.g., read and/or write access operations or interactions) with that master or replica data item would improve the access statistics for that master or replica data item, such as the access delay or latency, the round trip time (RTT) for access requests/operations, the error rate, etc..

Non-limiting examples of access pattern information that can be used by the coordinator system <NUM> to determine/perform a data item move can include the locations (e.g., network addresses, physical locations, networks, geographic regions, storage nodes, data stores, etc.) from where data items (e.g., 108A-N and/or 130A-N) on the distributed storage system <NUM> have been accessed over a period of time and/or in general (e.g., the location of the clients <NUM>-<NUM> that have accessed the data items), an access frequency of the data items (e.g., 108A-N and/or 130A-N) from such locations (e.g., how many times and/or how often each data item has been accessed from each location and/or by each client at each location, etc.), which clients (e.g., <NUM>-<NUM>) and/or how many clients have accessed the various data items (e.g., 108A-N and/or 130A-N), the type of access (e.g., read, write, etc.) from each access location and/or client, what other data items (if any) have been accessed in conjunction or association with a particular data item, what other operations have been performed in conjunction or association with access operations corresponding to the data items (e.g., 108A-N and/or 130A-N) and/or the access locations, the times and/or days that the data items (e.g., 108A-N and/or 130A-N) have been accessed from the access locations (and/or the relative frequency of access between the different times and/or days), the latency and/or other performance or access metrics observed for requests/operations from the access locations, routing information (e.g., number of hops, network path, routing cost, delay, performance, security, bandwidth, congestion, etc.) associated with access of the data items (e.g., 108A-N and/or 130A-N) and/or the access locations, and/or any other access pattern information.

In some cases, the information tracked/monitored by the coordinator system <NUM> and used to trigger data placement actions can include other types of information that can affect the state, security, stability, and/or performance of the distributed storage system <NUM> and/or the performance, cost and/or other metrics of access requests or operations received or processed by the distributed storage system <NUM>. For example, the information can include the locations of the master data items 108A-N and replica data items 130A-N, the status/condition of one or more elements of the distributed storage system <NUM> (e.g., the core <NUM>, the edges <NUM>, the storage nodes <NUM> and <NUM>, the data store <NUM>, the data stores <NUM>-<NUM>, etc.), the response times for access requests associated with the master data items 108A-N and/or replica data items 130A-N, the bandwidth available at the core <NUM> and/or one or more of the edges <NUM>, resource usage and/or workload conditions at the core <NUM> and/or one or more of the edges <NUM>, a resource availability and/or processing capacity at the core <NUM> and/or one or more of the edges <NUM>, the location of the core <NUM> and/or edges <NUM>, a topology of the core <NUM> and/or edges <NUM>, performance metrics/statistics (e.g., input/output operations per second (IOPS), latency, bandwidth, throughput, packet loss, jitter, connectivity, retransmission, error rate, RTT, congestion, availability, utilization, etc.) associated with the core <NUM> and/or edges <NUM>, client subscription levels (e.g., premium access, standard or basic access, guest access, etc.) associated with the clients <NUM>-<NUM>, etc..

In some cases, the coordinator system <NUM> can use the information collected from/for the distributed storage system <NUM> to determine a data item placement scheme or action for placing or distributing one or more of the master data items 108A-N and/or replica data items 130A-N at specific locations (e.g., data stores, storage nodes, segments, networks, etc.) on the distributed storage system <NUM>. For example, the coordinator system <NUM> can use such information to determine a data item placement scheme estimated to better balance or distribute the load or burden on the distributed storage system <NUM>, improve the access performance for the master data items 108A-N and/or replica data items 130A-N, reduce error rates, satisfy certain access criteria (e.g., access performance levels, quality-of-service (QoS) requirements, service level agreements (SLAs), security requirements, access restrictions, etc.), satisfy data sovereignty laws/policies, increase resource availability, decrease resource overutilization, reduce congestion, reduce costs, improve bandwidth, increase efficiency, reduce network traffic, etc..

To illustrate, the coordinator system <NUM> can use the information to determine the sources and locations of write requests associated with the master data items 108A-N and the respective frequency of the write requests from each of the locations to identify a target location for moving or placing one or more of the master data items 108A-N. The target location can be, for example, a location (e.g., an edge, the core, a data store, a storage node, etc.) on the distributed storage system <NUM> that is closest to the location where the most frequent write requests originate. Moving or placing one or more of the master data items 108A-N at such location can reduce the distance traversed by the write requests for such master data item(s) originating from that location (which is the most frequent access location and thus accounts for a significant portion of the write requests for such master data item(s)) to the location on the distributed storage system <NUM> where such master data item(s) are stored. This can help reduce the latency and improve the performance of write requests for such master data item(s) from that particular location.

The coordinator system <NUM> can receive the information (e.g., data item access histories and/or statistics, operating conditions at the various locations on the distributed storage system <NUM>, data item information, topology information associated with the distributed storage system <NUM>, performance metrics, storage or placement information, resource parameters, request parameters, configuration information, etc.) collected and monitored/tracked by the coordinator system <NUM> from the distributed storage system <NUM> (e.g., the core <NUM>, the edges <NUM>, one or more network devices on the distributed storage system <NUM>, etc.) on a push and/or pull basis. For example, the coordinator system <NUM> can pull, from the distributed storage system <NUM>, a respective stream of data, such as access patterns, statistics, metrics, status information, operating conditions, etc. As another example, the distributed storage system <NUM> can push to the coordinator system <NUM> such respective stream of data.

The elements and components shown in <FIG> are illustrative examples provided for explanation purposes. Thus, while <FIG> illustrates a certain type and number of networks or segments (e.g., core <NUM>, edges 110A through 110N) and components (e.g., storage nodes <NUM> and <NUM>, data stores <NUM> and <NUM>-<NUM>, master data items 108A-N, replica data items 130A-N, etc.), one of ordinary skill in the art will recognize that other examples may include a different number and/or type of networks/segments and/or components than those shown in <FIG>. For example, in some cases, the distributed storage system <NUM> can include more or less edges, cores or core segments, and/or components (e.g., storage nodes <NUM> and <NUM>, data stores <NUM> and <NUM>-<NUM>, master data items 108A-N, replica data items 130A-N, devices, etc.) than those shown in <FIG>.

<FIG> illustrates an example data item placement scenario <NUM> in the distributed storage system <NUM>. In this example, client <NUM> is located closer/closest to edge 110A, client <NUM> is located closer/closest to edge 110B, client <NUM> is located closer/closest to edge 110C, and client <NUM> is located closer/closest to edge 110N. Moreover, the coordinator system <NUM> can receive and monitor placement data <NUM> from the distributed storage system <NUM>.

The placement data <NUM> can include metrics, statistics, events data, status information, state information, notifications, configuration information, job/workload information, information about data items, access pattern information, parameters, preferences, client information, information about access and/or data restrictions, location information, resource information, network information, etc. In some examples, the placement data <NUM> can include a history of access requests and/or operations (e.g., access patterns) associated with master data items 108A-N and replica data items 130A-N. The history of access requests and/or operations can indicate one or more access patterns associated with the master data items 108A-N and the replica data items 130A-N, such as which clients <NUM>-<NUM> have accessed the master data items 108A-N and the replica data items 130A-N, the locations of the access requests and/or operations (e.g., the location of the clients <NUM>-<NUM> associated with the access requests and/or operations) associated with the master data items 108A-N and the replica data items 130A-N, the frequency of access requests and/or operations from the locations associated with the access requests and/or operations, the type of access requests and/or operations (e.g., read, write, etc.), the day/time of such access requests and/or operations, performance statistics associated with the access requests and/or operations, access anomalies, access trends, etc..

For example, the placement data <NUM> can indicate that the client <NUM> is closest to the edge 110A (relative to edges 110B-N and core <NUM>), the client <NUM> is closest to the edge 110B (relative to edge 110A, edges 110C-N and core <NUM>), the client <NUM> is closest to the edge 110C (relative to edges 110A-B, edge 110N and core <NUM>), and the client <NUM> is closest to the edge 110N (relative to edges 110A-C and core <NUM>). The placement data <NUM> can also indicate that the client <NUM> has accessed master data item 108A on the data stores <NUM> of the core <NUM> more frequently and/or a greater number of times than clients <NUM> and <NUM>-<NUM>, and that client <NUM> has accessed master data item 108N on the data stores <NUM> of the core <NUM> more frequently and/or a greater number of times than clients <NUM>-<NUM>. As further explained below, the coordinator system <NUM> can use this information in the placement data <NUM> to move the master data items 108A and 108N closer to clients <NUM> and <NUM> respectively, and thereby reduce the latency and improve the performance of access requests and operations for the master data items 108A and 108N from the clients <NUM> and <NUM>.

In some examples, the placement data <NUM> can include information (e.g., performance statistics, bandwidth, congestion, capacity, delays, resource availability, a state, a condition, metrics, location information, configuration information, access restrictions, access policies, etc.) about the core <NUM>, the edges <NUM>, and/or components of the core <NUM> and/or the edges <NUM> such as, for example, the storage nodes <NUM> and <NUM>, the data store <NUM> on the core <NUM>, the data stores <NUM>-<NUM> on the edges <NUM>, the master data items 108A-N and replica data items 130A-N, etc..

The coordinator system <NUM> can analyze the placement data <NUM> and determine whether any of the master data items 108A-N and/or replica data items 130A-N should be moved (e.g., whether moving such data items would yield one or more improvements/benefits) based on the access patterns of the clients <NUM>-<NUM> and/or any other information associated with the distributed storage system <NUM>. In other words, the coordinator system <NUM> can analyze the placement data <NUM> and determine whether moving any of the master data items 108A-N and/or the replica data items 130A-N to a different storage location would reduce a latency of future access requests/operations, improve a performance of future access requests/operations, reduce a load/burden on the distributed storage system <NUM>, improve resource utilization at the distributed storage system <NUM>, reduce congestion, optimize bandwidth usage/availability, improve resource availability at the distributed storage system <NUM>, and/or provide any other benefits for data access requests/operations and/or network/system conditions.

For example, as previously noted, the placement data <NUM> can indicate that the master data item 108A on the core <NUM> has been most frequently accessed by the client <NUM> (relative to the clients <NUM> and <NUM>-<NUM>). Accordingly, to improve the access performance and efficiency for future requests by the client <NUM> to access master data item 108A and by the client <NUM> to access master data item 108N, the coordinator system <NUM> can move the master data item 108A and the master data item 108N to locations that are closer to client <NUM> and client <NUM>, and thereby reduce the latency and increase the performance of future access requests from clients <NUM> and <NUM> for the master data items 108A and 108N.

Thus, in some examples, after determining (e.g., based on the placement data <NUM>) that the master data item 108A on the core <NUM> is most frequently accessed by the client <NUM> and that the client <NUM> is closer/closest to edge 110B than the other storage locations on the distributed storage system <NUM> (e.g., core <NUM> and edges 110A, 110C, and 110N), the coordinator system <NUM> can decide to move the master data item 108A from the core <NUM> (e.g., from the data stores <NUM>) to the edge 110B (e.g., to the data stores <NUM>) to improve the performance and reduce the latency of requests/operations from the client <NUM> for the master data item 108A.

Similarly, after determining (e.g., based on the placement data <NUM>) that the master data item 108N on the core <NUM> is most frequently accessed by the client <NUM> and that the client <NUM> is closer/closest to edge 110N than the other storage locations on the distributed storage system <NUM> (e.g., core <NUM> and edges 110A, 110B, and 110C), the coordinator system <NUM> can decide to move the master data item 108N from the core <NUM> (e.g., from the data stores <NUM>) to the edge 110N (e.g., to the data stores <NUM>) to improve the performance and reduce the latency of requests/operations from the client <NUM> for the master data item 108N.

To move data items or trigger placement actions, the coordinator system <NUM> can send a placement request <NUM> to the distributed storage system <NUM> (e.g., to the core <NUM>, the edge 110A, the edge 110B, the edge 110C, the edge 110N, and/or a network device in the distributed storage system <NUM>). For example, the coordinator system <NUM> can send the placement request <NUM> to the core <NUM>, which can trigger a move of the master data item 108A to the edge 110B and a move of the master data item 108N to the edge 110N. The placement request <NUM> can include, for example, a command, instruction, and/or request to move the master data item 108A to the edge 110B and the master data item 108N to the edge 110N.

The distributed storage system <NUM> can receive (e.g., at the core <NUM>) the placement request <NUM> from the coordinator system <NUM>, and perform data placement actions <NUM> and <NUM> as requested in, or triggered by, the placement request <NUM>. In some examples, the data placement action <NUM> can be a move of the master data item 108A from the core <NUM> to the edge 110B so the master data item 108A is stored closer to the client <NUM> that most frequently accesses the master data item 108A. Similarly, in some examples, the data placement action <NUM> can be a move of the master data item <NUM>N from the core <NUM> to the edge 110N so the master data item <NUM>N is stored closer to the client <NUM> that most frequently accesses the master data item <NUM>N.

After the data placement actions <NUM> and <NUM>, the master data item 108A will be hosted by (e.g., stored on) the edge 110B, as opposed to the core <NUM>, and the master data item <NUM>N will be hosted by (e.g., stored on) the edge <NUM>N, as opposed to the core <NUM>. Thus, on future instances, the client <NUM> will be able to access the master data item 108A from a corresponding one of the data stores <NUM> at the edge 110B, and the client <NUM> will be able to access the master data item 108N from a corresponding one of the data stores <NUM> at the edge <NUM>N. By placing the master data item 108A on the edge 110B, the coordinator system <NUM> can reduce the distance between the client <NUM> and the master data item 108A and thereby reduce the latency and increase the performance of future access requests and operations by client <NUM> for the master data item 108A. In addition, by placing the master data item <NUM>N on the edge <NUM> N, the coordinator system <NUM> can reduce the distance between the client <NUM> and the master data item <NUM> N and thereby reduce the latency and increase the performance of future access requests and operations by client <NUM> for the master data item <NUM> N.

The coordinator system <NUM> can continue to receive and monitor placement data <NUM> to determine whether additional data placement actions should be performed. For example, if new placement data indicates that client <NUM> is now the client that most frequently accesses (reads and/or writes) the master data item 108A previously moved to the edge 110B, the coordinator system <NUM> can send a new placement request to the distributed storage system <NUM> to trigger a move of the master data item 108A from the edge 110B to the edge 110A that is closer to the client <NUM>.

While the data placement actions <NUM> and <NUM> were described in the examples above as involving moving the master data items 108A and 108N to the locations (e.g., the edges 110B and 110N and the data stores <NUM> and <NUM>) closest to the clients <NUM> and <NUM> that most frequently access the master data items 108A and 108N, it should be noted that, in other examples, the data placement actions <NUM> and <NUM> can move the master data items 108A and <NUM>N to other locations (e.g., other data stores and/or edges, the core, etc.) and/or based on other factors and/or access patterns.

For example, in some cases, a master data item (e.g., 108A, 108B, or 108N) can be moved to or from a particular location (e.g., data store <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>; the core <NUM>; edge 110A, 110B, 110C, or edge <NUM>N; etc.) based on one or more conditions (e.g., bandwidth, resource availability, congestion, connectivity or downtime, error rates, state, performance, access restrictions, etc.) at a source and/or destination location (e.g., data store, core, edge, etc.), one or more characteristics (e.g., location, platform or infrastructure, configuration, performance statistics, relative rankings, network type, data type, type of resources, etc.) of the source and/or destination location, one or more events (e.g., traffic fluctuations, one or more network or resource failures, one or more errors, one or more network changes, etc.), one or more preferences or requirements (e.g., one or more QoS requirements, SLAs, client preferences, data or job requirements, restrictions, etc.), costs and/or client subscription levels, other access patterns (e.g., read and/or write access patterns associated with a group of data items, clients, and/or locatins), and/or any other factors or combination of factors.

<FIG> illustrates an example group data item placement scenario <NUM> in the distributed storage system <NUM>. The coordinator system <NUM> can receive and monitor placement data <NUM> from the distributed storage system <NUM>, as previously explained. In this example, the placement data <NUM> can include access patterns associated with partitioned data <NUM>. The partitioned data <NUM> can include the master data items 108A-N. For example, each of the master data items 108A-N can represent a partition of a partitioned data set (e.g., <NUM>), and together the master data items 108A-N can make up the partitioned data <NUM>. Thus, in this example, the placement data <NUM> can provide access patterns for different data partitions (e.g., 108A-N) in a partitioned data set (e.g., <NUM>).

In some cases, the placement data <NUM> can include statistics identifying the respective locations of clients accessing the partitioned data <NUM> and/or each individual partition (e.g., master data items 108A-N) in the partitioned data <NUM>, as well as the respective access frequency by such clients. In some examples, the placement data <NUM> can also include statistics identifying access patterns for other data items, such as replica data items 130A-N. The coordinator system <NUM> can thus analyze the placement data <NUM> and determine that the partitioned data <NUM> is most frequently accessed by the clients <NUM> and <NUM>. In some cases, the coordinator system <NUM> can also determine (e.g., based on the placement data <NUM>) that client <NUM> is located closest to edge 110A and client <NUM> is located closest to edge 110B. The coordinator system <NUM> can also determine that, even though client <NUM> is closest to edge 110A, edge 110B is closer to client <NUM> than core <NUM> (and edges 110C and 110N).

Based on this information, the coordinator system <NUM> can determine that moving the partitioned data <NUM>, including the master data items 108A-N, to the edge 110B may provide the biggest boost and/or advantages (and/or balance of advantages and disadvantages) in performance and/or cost for access requests/operations associated with the partitioned data <NUM> as a whole and/or one or more of its constituent parts (e.g., master data items 108A-N). Accordingly, after determining that moving the partitioned data <NUM> to the edge 110B may provide the biggest boost and/or advantages (and/or the optimal balance of advantages and disadvantages) in performance and/or cost for access requests/operations associated with the partitioned data <NUM>, the coordinator system <NUM> can send a group placement request <NUM> to the distributed storage system <NUM> (e.g., to the core <NUM>, the edge 110B, and/or a network device in the distributed storage system <NUM>).

The group placement request <NUM> can include an instruction to move the partitioned data <NUM>, including the master data items 108A-N, to the edge 110B. The instruction can then trigger the group placement action <NUM>, which can include moving the partitioned data <NUM>, including the master data items 108A-N, to the edge 110B (e.g., to one or more of the data stores <NUM> on the edge 110B). After the group placement action <NUM>, the partitioned data <NUM> will be hosted by (e.g., stored on) the edge 110B as opposed to the core <NUM>. Thus, in the future, the clients <NUM> and <NUM> (and any other clients) will be able to access the partitioned data <NUM> (and/or any of the master data items 108A-N in the partitioned data <NUM>) from the edge 110B, which is closer to the clients <NUM> and <NUM> than the core <NUM> and can therefore yield access performance improvements/benefits (e.g., reduced latency), cost improvements/benefits and/or other improvements/benefits. For example, by placing the partitioned data <NUM> on the edge 110B, the coordinator system <NUM> can reduce the distance between the clients <NUM> and <NUM> and the partitioned data <NUM> and thereby reduce the latency and increase the performance of future access requests and operations by clients <NUM> and <NUM> for the partitioned data <NUM>.

The coordinator system <NUM> can continue to receive and monitor placement data <NUM> to determine whether additional placement actions should be performed. For example, if new placement data indicates that clients <NUM> and/or <NUM> have accessed the partitioned data <NUM> with a greater frequency than clients <NUM> and/or <NUM> for a configurable amount of time, the coordinator system <NUM> can determine whether the partitioned data <NUM> should be moved to a different location. To illustrate, if the coordinator system <NUM> determines that the clients <NUM> and <NUM> are located closer to edge 110C than to edge 110B, the coordinator system <NUM> can estimate whether moving the partitioned data <NUM> from edge 110B to edge 110C would yield performance or other (e.g., cost, resource availability, congestion/bandwidth, etc.) improvements (overall and/or in balance).

The coordinator system <NUM> can estimate whether moving the partitioned data <NUM> from edge 110B to edge 110C would yield such improvements based on one or more factors. For example, in some cases, the coordinator system <NUM> can estimate whether moving the partitioned data <NUM> from edge 110B to edge 110C would yield such improvements based on the relative distances of clients <NUM>-<NUM> to edges 110B and 110C, the frequency (and/or differences in frequency) in which clients <NUM>-<NUM> access the partitioned data <NUM>, the days/times in which clients <NUM>-<NUM> access the partitioned data <NUM>, bandwidth and/or congestion metrics associated with the clients <NUM>-<NUM> and/or the edges 110B and 110C, QoS requirements and/or SLAs associated with the clients <NUM>-<NUM>, and/or any other relevant factors.

As previously mentioned, in the example shown in <FIG>, client <NUM> is closest to edge 110A and client <NUM> is closest to edge 110B, and the partitioned data <NUM> was moved to edge 110B based on the access frequencies of clients <NUM> and <NUM> despite client <NUM> being located closer to edge 110A. Here, the coordinator system <NUM> can determine that moving the partitioned data <NUM> to edge 110B would provide a bigger advantage/benefit (overall and/or in balance), such as a bigger performance boost, than moving the partitioned data <NUM> to edge 110A. For example, the coordinator system <NUM> can compare the advantages and disadvantages of moving the partitioned data <NUM> to edge 110A and edge 110B, and select edge 110B as the target location of the partitioned data <NUM> based on the comparison of advantages and disadvantages. The coordinator system <NUM> can determine the advantages and disadvantages based on, for example, the relative distances of clients <NUM> and <NUM> to edges 110A and 110B, the relative frequency in which clients <NUM> and <NUM> access the partitioned data <NUM>, the type of operations (e.g., read and/or write) performed by the clients <NUM> and <NUM> on the partitioned data <NUM>, bandwidth and/or congestion metrics associated with the clients <NUM> and <NUM> and/or the edges 110A and 110B, QoS requirements and/or SLAs associated with the clients <NUM> and <NUM>, and/or any other relevant factors.

<FIG> illustrates another example group data item placement scenario <NUM> in the distributed storage system <NUM>. As previously explained, the coordinator system <NUM> can receive and monitor placement data <NUM> from the distributed storage system <NUM>. In this example, the placement data <NUM> can include access patterns associated with groups of data items, such as groups of data items from the master data items 108A-N and replica data items 130A-N. For example, the placement data <NUM> can include statistics identifying one or more groups of data items that are frequently accessed together, as well as the respective locations of clients accessing the one or more groups of data items and the respective access frequency by such clients.

To illustrate, the placement data <NUM> can indicate that master data items 108A and 108N are frequently accessed together by client <NUM> for write operations, and that client <NUM> is located closer to the edge 110B than the core <NUM> and the edges 110A, 110C, and 110N. The placement data <NUM> can indicate that client <NUM> accesses the master data items 108A and 108N (e.g., together and/or individually) more frequently than clients <NUM>, <NUM>, and <NUM> (e.g., together and/or individually). Based on this information, the coordinator system <NUM> can determine that moving the master data items 108A and 108N to the edge 110B would provide a greater performance and/or other benefit/improvement (overall and/or in balance) than leaving the master data items 108A and 108N at their respective locations (e.g., core <NUM> and edge <NUM>N) or moving the master data items 108A and 108N to a different location in the distributed storage system <NUM>.

Accordingly, after determining that moving the master data items 108A and <NUM>N to the edge 110B may provide a bigger boost and/or benefit/advantage (and/or the optimal balance of advantages and disadvantages) in performance and/or aspects for access requests/operations associated with the master data items 108A and <NUM>N, the coordinator system <NUM> can send a group placement request <NUM> to the distributed storage system <NUM> (e.g., to the core <NUM>, the edge 110B, the edge <NUM>N, and/or a network device in the distributed storage system <NUM>).

The group placement request <NUM> can include an instruction to move the master data items 108A and <NUM>N to the edge 110B. The instruction can trigger the group placement action <NUM>, which can include moving the master data items 108A and 108N from the core <NUM> to the edge 110B (e.g., to one or more of the data stores <NUM> on the edge 110B). After the group placement action <NUM>, the master data item 108A will be hosted by (e.g., stored on) the edge 110B, as opposed to the core <NUM>, and the master data item <NUM>N will be hosted by (e.g., stored on) the edge 110B as opposed to the edge <NUM>N. Thus, in the future, the client <NUM> (and any other clients) will be able to access the master data items 108A and <NUM>N from the edge 110B, which is closest to the client <NUM> and can therefore yield access performance improvements/benefits (e.g., reduced latency), cost improvements/benefits and/or other improvements/benefits. For example, by placing the master data items 108A and <NUM>N on the edge 110B, the coordinator system <NUM> can reduce the distance between the client <NUM> and the master data items 108A and <NUM>N and thereby reduce the latency and increase the performance of future access requests and operations by client <NUM> for the master data items 108A and <NUM>N. Overall and/or on balance, such improvements/benefits can be estimated to be greater than any disadvantages of moving the master data items 108A and <NUM>N to the edge 110B, such as any decrease in performance for access requests/operations from clients <NUM>, <NUM>, and/or <NUM>.

The coordinator system <NUM> can continue to receive and monitor placement data <NUM> to determine whether additional placement actions should be performed. If new placement data indicates that other client(s) located closer to a different location of the distributed storage system <NUM> have accessed the master data items 108A and/or <NUM>N with a greater frequency than client <NUM> for a configurable amount of time, the coordinator system <NUM> can determine whether the master data items 108A and/or 108N should be moved to the different location, as previously described.

Having disclosed example systems, components and concepts, the disclosure now turns to the example method <NUM> for managing a placement of data items (e.g., 108A-N, 130A-N) on a distributed storage system (e.g., <NUM>), as shown in <FIG>. The steps outlined herein are non-limiting examples provided for illustration purposes, and can be implemented in any combination thereof, including combinations that exclude, add, or modify certain steps.

At step <NUM>, the method <NUM> can include determining a current location (e.g., core <NUM>; edge 110A, 110B, 110C, or <NUM>N; data stores <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>) of a master copy of a data item (e.g., master data item 108A, 108B, or <NUM>N) stored on a distributed storage system (e.g., <NUM>). In some examples, the current location of the master copy of the data item can include a current network or network segment (e.g., core <NUM>, edge 110A, edge 110B, edge 110C, or edge <NUM>N) on the distributed storage system and/or a current data store (e.g., data store <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>) from a plurality of data stores (e.g., data stores <NUM> and <NUM>-<NUM>) on the distributed storage system.

At step <NUM>, the method <NUM> can include determining an access pattern associated with the master copy of the data item. In some cases, the access pattern associated with the master copy of the data item can include one or more originating locations of a set of access requests (e.g., read and/or write requests) received by the distributed storage system for the master copy of the data item, a number of access requests received from each of the one or more originating locations, types of access requests (e.g., read, write, etc.), days/times of access requests, data access trends (e.g., time-based data access trends, client-based data access trends, location-based access trends, access trends associated with specific data items or types, access trends associated with specific access triggers, access trends associated with specific groups of data items, data access sequences, access trends associated with specific events, access trends associated with specific conditions, etc.), and/or any other access patterns characteristics.

In some examples, the access pattern associated with the master copy of the data item includes one or more originating locations of a set of access requests (e.g., read and/or write requests) received by the distributed storage system for the master copy of the data item and a number of access requests received from each of the one or more originating locations. The one or more originating locations can refer to the locations (e.g., networks, clients, addresses, geographic locations, regions, sources, etc.) from where the set of access requests originate (e.g., where the access requests are generated, transmitted, etc.). The number of access requests can refer to the quantity and/or frequency of access requests. For example, the number of access requests received from an originating location can refer to how many access requests originate from that location and/or a frequency in which access requests are received from that location.

In some cases, the access pattern associated with the master copy of the data item can be determined based on placement data (e.g., <NUM>) collected and/or monitored from the distributed storage system. For example, in some cases, a coordinator system (e.g., <NUM>) can collect, from the distributed storage system, information such as access pattern statistics (e.g., data access requests or operations, number and/or frequency of data access requests or operations, the location of clients associated with the data access requests or operations, the location of data items accessed by such clients from such locations, the distance between the location of such data items and the location of such clients, days/times of such data access requests or operations, etc.), network statistics, resource statistics, data access restrictions, data access subscription levels, network and/or distributed storage system characteristics, system events, errors, failures, metrics, data item characteristics, client and/or data preferences, data types, network or system configuration information, data access policies, etc..

At step <NUM>, the method <NUM> can include determining, based on the access pattern associated with the master copy of the data item, a different location (e.g., a different one of the core <NUM>; the edge 110A, 110B, 110C, or <NUM>N; the data stores <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>) on the distributed storage system for storing the master copy of the data item. In some examples, the different location can include a different network or network segment (e.g., core <NUM>, edge 110A, edge 110B, edge 110C, or edge <NUM>N) on the distributed storage system and/or a different data store from the plurality of data stores.

In some aspects, determining the different location on the distributed storage system for storing the master copy of the data item can include, based on the access pattern associated with the master copy of the data item, identifying, from the one or more originating locations, an originating location of a highest number of access requests associated with the master copy of the data item; determining which of the plurality of data stores is located closest (e.g., geographically closest, logically closest, etc.) to the originating location of the highest number of access requests associated with the master copy of the data item; and determining that the different data store associated with the different location is located closest to the originating location of the highest number of access requests associated with the master copy of the data item.

The originating location of the highest number of access requests can refer to the location that originated the highest amount and/or frequency of access requests for the master copy of the data item and/or the location from which the highest amount and/or frequency of access requests for the master copy of the data item were received by the distributed storage system. In some examples, the one or more originating locations can correspond to one or more client devices (e.g., <NUM>, <NUM>, <NUM>, <NUM>) that generated the set of access requests received by the distributed storage system for the master copy of the data item
In some examples, determining that the different data store is located closest to the originating location of the highest number of access requests associated with the master copy of the data item can include determining that a number of hops between the different data store (e.g., associated with the different location) and the originating location is less than a respective number of hops between each of the plurality of data stores and each of one or more remaining locations from the one or more originating locations (e.g., the one or more originating locations excluding the originating location of the highest number of access requests), and/or determining that a distance between the different data store and the originating location is less than a respective distance between each of the plurality of data stores and each of the one or more remaining locations from the one or more originating locations.

In other examples, determining that the different data store is located closest to the originating location of the highest number of access requests associated with the master copy of the data item can instead or additionally include determining that a number of hops between the different data store and the originating location is less than a respective number of hops between each of a plurality of networks or networks segments (e.g., core <NUM>, edges <NUM>) associated with the distributed storage system and each of one or more remaining locations from the one or more originating locations, and/or determining that a distance between the different data store and the originating location is less than a respective distance between each of the plurality of networks or networks segments (e.g., core <NUM>, edges <NUM>) associated with the distributed storage system and each of the one or more remaining locations from the one or more originating locations.

In some aspects, determining the different location on the distributed storage system for storing the master copy of the data item can include determining a second current location of a second master copy of a second data item (e.g., master data item 108A, 108B, or 108N) stored on the distributed storage system; selecting, based on the access pattern associated with the master copy of the data item and a second access pattern associated with the second master copy of the second data item, the different location on the distributed storage system for storing both the master copy of the second data item and the second master copy of the second data item; and placing both the master copy of the data item and the second master copy of the second data item at the different location on the distributed storage system. In some examples, the second current location of the second master copy of the second data item can include a second current data store from the plurality of data stores on the distributed storage system. Moreover, in some examples, the master copy of the data item can include a first partition of a partitioned data set (e.g., partitioned data <NUM>) and the second master copy of the second data item can include a second partition of the partitioned data set.

In some cases, the second access pattern can include one or more respective originating locations of a second set of access requests associated with the second master copy of the second data item (e.g., the locations from where the second set of access requests originated or where received) and a second respective number of access requests (e.g., an amount and/or frequency of access requests) received from each of the one or more respective originating locations. In some examples, placing both the master copy of the data item and the second master copy of the second data item at the different location on the distributed storage system can include moving or migrating the master copy of the data item from the current location to the different location and moving or migrating the second master copy of the second data item from the second current location to the different location.

At step <NUM>, the method <NUM> can include placing the master copy of the data item at the different location on the distributed storage system. In some examples, placing the master copy of the data item at the different location on the distributed storage system can include moving or migrating the master copy of the data item from the current data store to the different data store.

In some aspects, the method <NUM> can include determining that the master copy of the data item and a second data item on the distributed storage system have been accessed together a threshold number and/or frequency of times; and after determining the different location on the distributed storage system for storing the master copy of the data item, moving the second data item from a current respective location of the second data item to the different location. The current respective location can include, for example, a data store from the plurality of data stores on the distributed storage system.

In some aspects, the method <NUM> can include determining that the master copy of the data item includes a reference (e.g., a pointer, an association, a link, a relation, etc.) to a particular copy of a second data item stored on the distributed storage system; in response to determining the different location on the distributed storage system for storing the master copy of the data item and determining that the master copy of the data item includes the reference to the particular copy of the second data item, selecting the different location on the distributed storage system for storing the particular copy of the second data item; and moving the particular copy of the second data item from a respective location on the distributed storage system to the different location on the distributed storage system. In some examples, the particular copy of the second data item can include a master copy of the second data item or a replica of the master copy of the second data item.

In some implementations, the method <NUM> can determine the different location for placing (e.g., moving, migrating, storing, etc.) a data item based on access pattem information and/or one or more other factors. For example, in some aspects, the method <NUM> can include collecting information associated with the distributed storage system; based on the information associated with the distributed storage system, determining a data placement action estimated to improve a data access performance (e.g., latency, response time, error rate, etc.) associated with one or more data items on the distributed storage system and/or improve a performance of the distributed storage system (e.g., the performance of the distributed storage system as a whole and/or one or more components/resources of the distributed storage system); and in response to determining the data placement action, moving at least one data item (e.g., master data items 108A, 108B, and/or 108N; and/or replica data items 130A and/or 130N) from at least one storage location (e.g., a data store, core <NUM>, edge 110A, edge 110B, edge 110C, or edge <NUM>N) to at least one different storage location (e.g., a different data store, core <NUM>, edge 110A, edge 110B, edge 110C, or edge <NUM>N).

In some examples, the collected information associated with the distributed storage system can include statistics associated with one or more resources (e.g., storage nodes <NUM> and/or <NUM>, compute nodes, core <NUM>, edges <NUM>, network devices, bandwidth, etc.), one or more data access restrictions (e.g., location-based restrictions, policy-based restrictions, client-based restrictions, network-based restrictions, resource-based restrictions, source-based restrictions, data type-based restrictions, subscription-based restrictions, etc.) associated with one or more data items on the distributed storage system, data storage restrictions (e.g., restrictions based on data sovereignty laws, data types, resource types, network utilization, resource utilization, day/time, resource availability, etc.), one or more events (e.g., errors, failures, network changes, traffic changes, access events, system events, etc.), data access patterns, and/or network statistics associated with the distributed storage system and/or one or more networks associated with the distributed storage system. In some examples, the one or more resources can include a storage node, a compute node, a virtual machine, a software container, a server, a network, and/or a networking device (e.g., a switch, a router, a firewall, an appliance, etc.).

In some cases, the data placement action can include moving at least one data item from at least one storage location (e.g., a data store, core <NUM>, edge 110A, edge 110B, edge 110C, or edge <NUM>N) to at least one different storage location. In some examples, the at least one storage location and the at least one different storage location can include different data stores from the plurality of data stores, different networks associated with the distributed storage system, different network segments, etc..

In some examples, when determining a data placement action (e.g., moving or migrating a data item) based multiple more factors (e.g., access patterns, performance statistics, resource metrics, environment conditions, events, etc.), the factors used to determine the data placement action can be weighed. For example, assume that a data placement action for a data item is determined based on a frequency of access of the data item from different access locations, a status of different storage nodes on the distributed storage system, and an amount of bandwidth available at different networks or segments of the distributed storage system. Here, different weights can be determined for, and applied to, the frequency of access, the status of the different storage nodes, and the amount of bandwidth available used to determine the placement action. This will allow the various factors to be taken into account for the placement action with certain factors receiving higher weight, emphasis, or priority.

The disclosure now turns to <FIG> and <FIG>, which illustrate example network devices and computing devices, such as switches, routers, nodes, servers, client devices, orchestrators (e.g., coordinator system <NUM>), and so forth.

<FIG> illustrates an example network device <NUM> suitable for performing switching, routing, load balancing, and other networking operations. Network device <NUM> includes a central processing unit (CPU) <NUM>, interfaces <NUM>, and a bus <NUM> (e.g., a PCI bus). When acting under the control of appropriate software or firmware, the CPU <NUM> is responsible for executing packet management, error detection, and/or routing functions. The CPU <NUM> preferably accomplishes all these functions under the control of software including an operating system and any appropriate applications software. CPU <NUM> may include one or more processors <NUM>, such as a processor from the INTEL X86 family of microprocessors. In some cases, processor <NUM> can be specially designed hardware for controlling the operations of network device <NUM>. In some cases, a memory <NUM> (e.g., non-volatile RAM, ROM, etc.) also forms part of CPU <NUM>. However, there are many different ways in which memory could be coupled to the system.

The interfaces <NUM> are typically provided as modular interface cards (sometimes referred to as "line cards"). Generally, they control the sending and receiving of data packets over the network and sometimes support other peripherals used with the network device <NUM>. Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like. In addition, various very high-speed interfaces may be provided such as fast token ring interfaces, wireless interfaces, Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces, WIFI interfaces, <NUM>/<NUM>/<NUM> cellular interfaces, CAN BUS, LoRA, and the like. Generally, these interfaces may include ports appropriate for communication with the appropriate media. In some cases, they may also include an independent processor and, in some instances, volatile RAM. The independent processors may control such communications intensive tasks as packet switching, media control, signal processing, crypto processing, and management. By providing separate processors for the communications intensive tasks, these interfaces allow the master CPU (e.g., <NUM>) to efficiently perform routing computations, network diagnostics, security functions, etc..

Although the system shown in <FIG> is one specific network device of the present disclosure, it is by no means the only network device architecture on which the present disclosure can be implemented. For example, an architecture having a single processor that handles communications as well as routing computations, etc., is often used. Further, other types of interfaces and media could also be used with the network device <NUM>.

The network device <NUM> can also include an application-specific integrated circuit (ASIC), which can be configured to perform routing and/or switching operations. The ASIC can communicate with other components in the network device <NUM> via the bus <NUM>, to exchange data and signals and coordinate various types of operations by the network device <NUM>, such as routing, switching, and/or data storage operations, for example.

<FIG> illustrates an example computing system architecture of a system <NUM> which can be used to process data operations and requests, store and move data items, coordinate data placement actions, and perform other computing operations. In this example, the components of the system <NUM> are in electrical communication with each other using a connection <NUM>, such as a bus. The system <NUM> includes a processing unit (CPU or processor) <NUM> and a connection <NUM> that couples various system components including a memory <NUM>, such as read only memory (ROM) <NUM> and random access memory (RAM) <NUM>, to the processor <NUM>. The system <NUM> can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of the processor <NUM>. The system <NUM> can copy data from the memory <NUM> and/or the storage device <NUM> to cache <NUM> for quick access by the processor <NUM>. In this way, the cache can provide a performance boost that avoids processor <NUM> delays while waiting for data. These and other modules can control or be configured to control the processor <NUM> to perform various actions. Other memory <NUM> may be available for use as well. The memory <NUM> can include multiple different types of memory with different performance characteristics. The processor <NUM> can include any general purpose processor and a hardware or software service, such as service <NUM><NUM>, service <NUM><NUM>, and service <NUM><NUM> stored in storage device <NUM>, configured to control the processor <NUM> as well as a special-purpose processor where software instructions are incorporated into the actual processor design. The processor <NUM> may be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing system <NUM>, an input device <NUM> can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device <NUM> can also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input to communicate with the computing system <NUM>. The communications interface <NUM> can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

The storage device <NUM> can include services <NUM>, <NUM>, <NUM> for controlling the processor <NUM>. Other hardware or software modules are contemplated. The storage device <NUM> can be connected to the connection <NUM>. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as the processor <NUM>, connection <NUM>, output device <NUM>, and so forth, to carry out the function.

Although a variety of examples and other information was used to explain the invention, its scope is limited by appended claims.

Claim 1:
A method (<NUM>) comprising:
determining (<NUM>) a current location of a master copy of a data item stored on a distributed storage system, wherein the current location of the master copy of the data item comprises a current data store from a plurality of data stores on the distributed storage system;
determining (<NUM>) an access pattern associated with the master copy of the data item, the access pattern comprising one or more originating locations of a set of access requests received by the distributed storage system for the master copy of the data item and a respective number of access requests received from each of the one or more originating locations;
determining (<NUM>), based on the access pattern associated with the master copy of the data item, a different location on the distributed storage system for storing the master copy of the data item, the different location comprising a different data store from the plurality of data stores; and
placing (<NUM>) the master copy of the data item at the different location on the distributed storage system,
wherein the master copy of the data item represents an authoritative state or version of the data item and has been replicated to a replica data item.