INPUT-OUTPUT PROCESSING IN SOFTWARE-DEFINED STORAGE SYSTEMS

An apparatus comprises a processing device configured to receive, from one of two or more storage nodes of a software-defined storage system comprising a write cache and persistent storage, a request to read a given portion of data. One or more portions of data in the persistent storage are also stored as read-only data in local storage of virtual computing instances implementing the storage nodes. The processing device is configured to read the given portion of data from the local storage of the virtual computing instances responsive to determining that the given portion of data is part of the read-only data in the local storage of the virtual computing instances, and to read the given portion of data from the write cache or persistent storage responsive to determining that the given portion of data is not part of the read-only data in the local storage of the virtual computing instances.

FIELD

The field relates generally to information processing, and more particularly to storage in information processing systems.

BACKGROUND

Various types of storage systems, including storage systems implementing software-defined storage (SDS) solutions, may be configured to run workloads from multiple different end-users or applications. Different end-users or applications may have different performance and feature requirements for their associated workloads. In some workloads, performance may be most important. In other workloads, capacity utilization or other feature requirements may be most important. There is thus a need for techniques which enable a storage system to offer flexibility in storage offerings for workloads with different performance and feature requirements.

SUMMARY

Illustrative embodiments of the present disclosure provide techniques for input-output processing in software-defined storage systems.

In one embodiment, an apparatus comprises at least one processing device comprising a processor coupled to a memory. The at least one processing device is configured to perform the step of receiving, from one of two or more storage nodes of a software-defined storage system, a read input-output request to read a given portion of data, the software-defined storage system comprising a write cache and persistent storage, the two or more storage nodes being implemented as respective virtual computing instances having local storage, wherein one or more portions of data stored in the persistent storage of the software-defined storage system are also stored as read-only data in the local storage of the virtual computing instances. The at least one processing device is also configured to perform the step of determining whether the given portion of data is part of the one or more portions of the data stored in the persistent storage of the software-defined storage system that are also stored as read-only data in the local storage of the virtual computing instances. The at least one processing device is further configured to perform the step of, responsive to determining that the given portion of data is part of the one or more portions of the data stored in the persistent storage of the software-defined storage system that are also stored as read-only data in the local storage of the virtual computing instances, reading the given portion of data from the local storage of the virtual computing instances. The at least one processing device is further configured to perform the step of, responsive to determining that the given portion of data is not part of the one or more portions of the data stored in the persistent storage of the software-defined storage system that are also stored as read-only data in the local storage of the virtual computing instances, reading the given portion of data from at least one of the write cache and the persistent storage of the software-defined storage system.

DETAILED DESCRIPTION

FIGS.1A and1Bschematically illustrate an information processing system which is configured for optimizing or improving input-output (IO) processing utilizing software-defined storage (SDS) resources according to an exemplary embodiment of the disclosure. More specifically,FIG.1Aschematically illustrates an information processing system100which comprises a plurality of compute nodes110-1,110-2, . . . ,110-C (collectively referred to as compute nodes110, or each singularly referred to as a compute node110), one or more management nodes115(which support a management layer of the system100), a communications network120, and a data storage system130(which supports a data storage layer of the system100). The data storage system130comprises a plurality of storage nodes140-1,140-2, . . . ,140-N (collectively referred to as storage nodes140, or each singularly referred to as a storage node140). In the context of the exemplary embodiments described herein, the management nodes115and the data storage system130implement IO request processing logic117and data caching and synchronization logic119supporting optimization or improvement of IO processing in the data storage system130utilizing SDS resources.FIG.1Bschematically illustrates an exemplary framework of at least one or more of the storage nodes140.

In particular, as shown inFIG.1B, the storage node140comprises a storage controller142and a plurality of storage devices146. In general, the storage controller142implements data storage and management methods that are configured to divide the storage capacity of the storage devices146into storage pools and logical volumes. Storage controller142is further configured to implement IO request processing logic117and data caching and synchronization logic119in accordance with the disclosed embodiments, as will be described in further detail below. Various other examples are possible. It is to be noted that the storage controller142may include additional modules and other components typically found in conventional implementations of storage controllers and storage systems, although such additional modules and other components are omitted for clarity and simplicity of illustration.

In the embodiment ofFIGS.1A and1B, the IO request processing logic117and the data caching and synchronization logic119may be implemented at least in part within the one or more management nodes115as well as in one or more of the storage nodes140of the data storage system130. This may include implementing different portions of the IO request processing logic117and the data caching and synchronization logic119functionality described herein being implemented within the management nodes115and the storage nodes140. In other embodiments, however, the IO request processing logic117and the data caching and synchronization logic119may be implemented entirely within the management nodes115or entirely within the storage nodes140. In still other embodiments, at least a portion of the functionality of the IO request processing logic117and the data caching and synchronization logic119is implemented in one or more of the compute nodes110.

The compute nodes110illustratively comprise physical compute nodes and/or virtual compute nodes which process data and execute workloads. For example, the compute nodes110can include one or more server nodes (e.g., bare metal server nodes) and/or one or more virtual machines. In some embodiments, the compute nodes110comprise a cluster of physical server nodes or other types of computers of an enterprise computer system, cloud-based computing system or other arrangement of multiple compute nodes associated with respective users. In some embodiments, the compute nodes110include a cluster of virtual machines that execute on one or more physical server nodes.

The compute nodes110are configured to process data and execute tasks/workloads and perform computational work, either individually, or in a distributed manner, to thereby provide compute services such as execution of one or more applications on behalf of each of one or more users associated with respective ones of the compute nodes. Such applications illustratively issue IO requests that are processed by a corresponding one of the storage nodes140. The term “input-output” as used herein refers to at least one of input and output. For example, IO requests may comprise write requests and/or read requests directed to stored data of a given one of the storage nodes140of the data storage system130.

The compute nodes110are configured to write data to and read data from the storage nodes140in accordance with applications executing on those compute nodes for system users. The compute nodes110communicate with the storage nodes140over the communications network120. While the communications network120is generically depicted inFIG.1A, it is to be understood that the communications network120may comprise any known communication network such as, a global computer network (e.g., the Internet), a wide area network (WAN), a local area network (LAN), an intranet, a satellite network, a telephone or cable network, a cellular network, a wireless network such as Wi-Fi or WiMAX, a storage fabric (e.g., Ethernet storage network), or various portions or combinations of these and other types of networks.

In this regard, the term “network” as used herein is therefore intended to be broadly construed so as to encompass a wide variety of different network arrangements, including combinations of multiple networks possibly of different types, which enable communication using, e.g., Transfer Control/Internet Protocol (TCP/IP) or other communication protocols such as Fibre Channel (FC), FC over Ethernet (FCoE), Internet Small Computer System Interface (iSCSI), Peripheral Component Interconnect express (PCIe), InfiniBand, Gigabit Ethernet, etc., to implement IO channels and support storage network connectivity. Numerous alternative networking arrangements are possible in a given embodiment, as will be appreciated by those skilled in the art.

The data storage system130may comprise any type of data storage system, or a combination of data storage systems, including, but not limited to, a storage area network (SAN) system, a network attached storage (NAS) system, a direct-attached storage (DAS) system, etc., as well as other types of data storage systems comprising software-defined storage, clustered or distributed virtual and/or physical infrastructure. The term “data storage system” as used herein should be broadly construed and not viewed as being limited to storage systems of any particular type or types. In some embodiments, the storage nodes140comprise storage server nodes having one or more processing devices each having a processor and a memory, possibly implementing virtual machines and/or containers, although numerous other configurations are possible. In some embodiments, one or more of the storage nodes140can additionally implement functionality of a compute node, and vice-versa. The term “storage node” as used herein is therefore intended to be broadly construed, and a storage system in some embodiments can be implemented using a combination of storage nodes and compute nodes.

In some embodiments, as schematically illustrated inFIG.1B, the storage node140is a physical server node or storage appliance, wherein the storage devices146comprise DAS resources (internal and/or external storage resources) such as hard-disk drives (HDDs), solid-state drives (SSDs), Flash memory cards, or other types of non-volatile memory (NVM) devices such non-volatile random access memory (NVRAM), phase-change RAM (PC-RAM) and magnetic RAM (MRAM). These and various combinations of multiple different types of storage devices146may be implemented in the storage node140. In this regard, the term “storage device” as used herein is intended to be broadly construed, so as to encompass, for example, SSDs, HDDs, flash drives, hybrid drives or other types of storage media. The storage devices146are connected to the storage node140through any suitable host interface, e.g., a host bus adapter, using suitable protocols such as ATA, SATA, eSATA, NVMe, NVMeOF, SCSI, SAS, etc. In other embodiments, the storage node140can be network connected to one or more NAS nodes over a local area network.

The storage controller142is configured to manage the storage devices146and control IO access to the storage devices146and/or other storage resources (e.g., DAS or NAS resources) that are directly attached or network-connected to the storage node140. In some embodiments, the storage controller142is a component (e.g., storage data server) of a software-defined storage (SDS) system which supports the virtualization of the storage devices146by separating the control and management software from the hardware architecture. More specifically, in a software-defined storage environment, the storage controller142comprises an SDS storage data server that is configured to abstract storage access services from the underlying storage hardware to thereby control and manage IO requests issued by the compute nodes110, as well as to support networking and connectivity. In this instance, the storage controller142comprises a software layer that is hosted by the storage node140and deployed in the data path between the compute nodes110and the storage devices146of the storage node140, and is configured to respond to data IO requests from the compute nodes110by accessing the storage devices146to store/retrieve data to/from the storage devices146based on the IO requests.

In a software-defined storage environment, the storage controller142is configured to provision, orchestrate and manage the local storage resources (e.g., the storage devices146) of the storage node140. For example, the storage controller142implements methods that are configured to create and manage storage pools (e.g., virtual pools of block storage) by aggregating capacity from the storage devices146. The storage controller142can divide a storage pool into one or more volumes and expose the volumes to the compute nodes110as virtual block devices. For example, a virtual block device can correspond to a volume of a storage pool. Each virtual block device comprises any number of actual physical storage devices, wherein each block device is preferably homogenous in terms of the type of storage devices that make up the block device (e.g., a block device only includes either HDD devices or SSD devices, etc.).

In the software-defined storage environment, each of the storage nodes140inFIG.1Acan run an instance of the storage controller142to convert the respective local storage resources (e.g., DAS storage devices and/or NAS storage devices) of the storage nodes140into local block storage. Each instance of the storage controller142contributes some or all of its local block storage (HDDs, SSDs, PCIe, NVMe and flash cards) to an aggregated pool of storage of a storage server node cluster (e.g., cluster of storage nodes140) to implement a server-based storage area network (SAN) (e.g., virtual SAN). In this configuration, each storage node140is part of a loosely coupled server cluster which enables “scale-out” of the software-defined storage environment, wherein each instance of the storage controller142that runs on a respective one of the storage nodes140contributes its local storage space to an aggregated virtual pool of block storage with varying performance tiers (e.g., HDD, SSD, etc.) within a virtual SAN.

In some embodiments, in addition to the storage controllers142operating as SDS storage data servers to create and expose volumes of a storage layer, the software-defined storage environment comprises other components such as (i) SDS data clients that consume the storage layer and (ii) SDS metadata managers that coordinate the storage layer, which are not specifically shown inFIG.1A. More specifically, on the client-side (e.g., compute nodes110), an SDS data client (SDC) is a lightweight block device driver that is deployed on each server node that consumes the shared block storage volumes exposed by the storage controllers142. In particular, the SDCs run on the same servers as the compute nodes110which require access to the block devices that are exposed and managed by the storage controllers142of the storage nodes140. The SDC exposes block devices representing the virtual storage volumes that are currently mapped to that host. In particular, the SDC serves as a block driver for a client (server), wherein the SDC intercepts IO requests, and utilizes the intercepted IO request to access the block storage that is managed by the storage controllers142. The SDC provides the operating system or hypervisor (which runs the SDC) access to the logical block devices (e.g., volumes).

The SDCs have knowledge of which SDS control systems (e.g., storage controller142) hold its block data, so multipathing can be accomplished natively through the SDCs. In particular, each SDC knows how to direct an IO request to the relevant destination SDS storage data server (e.g., storage controller142). In this regard, there is no central point of routing, and each SDC performs its own routing independent from any other SDC. This implementation prevents unnecessary network traffic and redundant SDS resource usage. Each SDC maintains peer-to-peer connections to every storage controller142that manages the storage pool. A given SDC can communicate over multiple pathways to all of the storage nodes140which store data that is associated with a given IO request. This multi-point peer-to-peer fashion allows the SDS to read and write data to and from all points simultaneously, eliminating bottlenecks and quickly routing around failed paths.

The management nodes115inFIG.1Aimplement a management layer that is configured to manage and configure the storage environment of the system100. In some embodiments, the management nodes115comprise the SDS metadata manager components, wherein the management nodes115comprise a tightly-coupled cluster of nodes that are configured to supervise the operations of the storage cluster and manage storage cluster configurations. The SDS metadata managers operate outside of the data path and provide the relevant information to the SDS clients and storage servers to allow such components to control data path operations. The SDS metadata managers are configured to manage the mapping of SDC data clients to the SDS data storage servers. The SDS metadata managers manage various types of metadata that are required for system operation of the SDS environment such as configuration changes, managing the SDS data clients and data servers, device mapping, values, snapshots, system capacity including device allocations and/or release of capacity, RAID protection, recovery from errors and failures, and system rebuild tasks including rebalancing.

WhileFIG.1Ashows an exemplary embodiment of a two-layer deployment in which the compute nodes110are separate from the storage nodes140and connected by the communications network120, in other embodiments, a converged infrastructure (e.g., hyperconverged infrastructure) can be implemented to consolidate the compute nodes110, storage nodes140, and communications network120together in an engineered system. For example, in a hyperconverged deployment, a single-layer deployment is implemented in which the storage data clients and storage data servers run on the same nodes (e.g., each node deploys a storage data client and storage data servers) such that each node is a data storage consumer and a data storage supplier. In other embodiments, the system ofFIG.1Acan be implemented with a combination of a single-layer and two-layer deployment.

Regardless of the specific implementation of the storage environment, as noted above, various modules of the storage controller142ofFIG.1Bcollectively provide data storage and management methods that are configured to perform various function as follows. In particular, a storage virtualization and management services module may implement any suitable logical volume management (LVM) system which is configured to create and manage local storage volumes by aggregating the local storage devices146into one or more virtual storage pools that are thin-provisioned for maximum capacity, and logically dividing each storage pool into one or more storage volumes that are exposed as block devices (e.g., raw logical unit numbers (LUNs)) to the compute nodes110to store data. In some embodiments, the storage devices146are configured as block storage devices where raw volumes of storage are created and each block can be controlled as, e.g., an individual disk drive by the storage controller142. Each block can be individually formatted with a same or different file system as required for the given data storage system application.

In some embodiments, the storage pools are primarily utilized to group storage devices based on device types and performance. For example, SSDs are grouped into SSD pools, and HDDs are grouped into HDD pools. Furthermore, in some embodiments, the storage virtualization and management services module implements methods to support various data storage management services such as data protection, data migration, data deduplication, replication, thin provisioning, snapshots, data backups, etc.

Storage systems, such as the data storage system130of system100, may be required to provide both high performance and a rich set of advanced data service features for end-users thereof (e.g., users operating compute nodes110, applications running on compute nodes110). Performance may refer to latency, or other metrics such as IO operations per second (IOPS), bandwidth, etc. Advanced data service features may refer to data service features of storage systems including, but not limited to, services for data resiliency, thin provisioning, data reduction, space efficient snapshots, etc. Fulfilling both performance and advanced data service feature requirements can represent a significant design challenge for storage systems. This may be due to different advanced data service features consuming significant resources and processing time. Such challenges may be even greater in software-defined storage systems in which custom hardware is not available for boosting performance.

Device tiering may be used in some storage systems, such as in storage systems that contain some relatively “fast” and expensive storage devices and some relatively “slow” and less expensive storage devices. In device tiering, the “fast” devices may be used when performance is the primary requirement, where the “slow” and less expensive devices may be used when capacity is the primary requirement. Such device tiering may also use cloud storage as the “slow” device tier. Some storage systems may also or alternately separate devices offering the same performance level to gain performance isolation between different sets of storage volumes. For example, the storage systems may separate the “fast” devices into different groups to gain performance isolation between storage volumes on such different groups of the “fast” devices.

Illustrative embodiments provide functionality for optimizing or improving performance of IO request processing in SDS infrastructure. Data storage system130, as an example, may be configured to support different types of storage services (e.g., SDS services) having different characteristics (e.g., cost, performance, availability, etc.). By way of example, the above-described “device tiering” may be used to provide different storage tiers with different characteristics (e.g., “fast” devices which are relatively expensive and “slow” devices which are relatively inexpensive). In some embodiments, the compute nodes110and/or storage nodes140are assumed to comprise virtual computing resources such as VMs or containers, which are instantiated with associated local storage. The local storage of such virtual computing resources used to implement the compute nodes110and/or storage nodes140is ephemeral, in that the local storage is lost when the virtual computing resources are shut down. To optimize or improve the processing of IO requests, the IO request processing logic117is configured to intelligently utilize the local storage of the virtual computing resources implementing the compute nodes110and/or storage nodes140as an ephemeral read cache that is separate from other storage services providing a write cache and backend persistent storage. When data is available in the ephemeral read cache (e.g., the local storage of the virtual computing resource implementing the compute nodes110and/or storage nodes140), the IO request processing logic117directs read requests to the ephemeral read cache. When data is not available in the ephemeral read cache, the IO request processing logic117directs read requests to a write cache and/or backend persistent storage in the data storage system130. In some embodiments, the write cache is implemented using higher performance storage resources (e.g., the “fast” device tier) and the backend persistent storage is implemented using relatively lower performance storage resources (e.g., the “slow” device tier).

The data caching and synchronization logic119is configured to synchronize data to the ephemeral read cache (e.g., the local storage of the virtual computing resource implementing the compute nodes110and/or storage nodes140). Such synchronization may be performed between the backend persistent storage, or from the write cache. For example, when write requests are processed by the IO request processing logic117, data in the write cache may be changed. This triggers the data caching and synchronization logic119to synchronize or update the backend persistent storage, and to invalidate any contents of the ephemeral read cache that are affected by the write requests. The data caching and synchronization logic119may further synchronize the written data to the ephemeral read cache (e.g., from the write cache, or from the backend persistent storage).

An exemplary process for IO processing utilizing SDS resources will now be described in more detail with reference to the flow diagram ofFIG.2. It is to be understood that this particular process is only an example, and that additional or alternative processes for IO processing utilizing SDS resources may be used in other embodiments.

In this embodiment, the process includes steps200through206. These steps are assumed to be performed using the IO request processing logic117and the data caching and synchronization logic119, which as noted above may be implemented in the management nodes115of system100, in storage nodes140of the data storage system130of system100, in compute nodes110of system100, combinations thereof, etc. The process begins with step200, receiving from one of two or more storage nodes (e.g., compute nodes110and/or storage nodes140) of a SDS system, a read IO request to read a given portion of data. The SDS system comprises a write cache and persistent storage, and the two or more storage nodes are implemented as respective virtual computing instances having local storage. One or more portions of data stored in the persistent storage of the SDS system are also stored as read-only data in the local storage of the virtual computing instances. At least one of the virtual computing instances comprises at least one of a VM and a software container. The local storage of the virtual computing instances comprises ephemeral storage that does not persist when the virtual computing instances are shut down.

The write cache is implemented utilizing a first type of SDS resources providing a first performance level, and the persistent storage is implemented utilizing a second type of SDS resources providing a second performance level, wherein the second performance level is lower than the first performance level. The local storage of the virtual computing instances may be implemented utilizing a third type of SDS resources providing a third performance level, the third performance level being higher than the first and second performance levels. In some embodiments, the write cache is implemented utilizing cloud-based block storage, and the persistent storage is implemented utilizing cloud-based object storage. The local storage of the virtual computing instances may be implemented utilizing cloud-based block storage with a lower latency than the cloud-based block storage used for implementing the write cache.

In step202, a determination is made as to whether the given portion of data (e.g., that is the subject of the received read IO request) is part of the one or more portions of the data stored in the persistent storage of the SDS system that are also stored as read-only data in the local storage of the virtual computing instances. The given portion of data is read from the local storage of the virtual computing instances in step204responsive to determining that the given portion of data is part of the one or more portions of the data stored in the persistent storage of the SDS system that are also stored as read-only data in the local storage of the virtual computing instances. The given portion of data is read from at least one of the write cache and the persistent storage of the SDS system in step206responsive to determining that the given portion of data is not part of the one or more portions of the data stored in the persistent storage of the SDS system that are also stored as read-only data in the local storage of the virtual computing instances.

TheFIG.2process may also include synchronizing, in a background process, the one or more portions of the data stored in the persistent storage of the SDS system with the read-only data in the local storage of the virtual computing instances.

In some embodiments, theFIG.2process further includes receiving, from one of the two or more storage nodes of the SDS system, a write IO request to write new data. Responsive to writing the new data to the write cache of the SDS system, theFIG.2process may include at least one of: invalidating at least a portion of the read-only data in the local storage of the virtual computing instances; and synchronizing the new data as new read-only data in the local storage of the virtual computing instances.

TheFIG.2process may also or alternatively include, responsive to determining that a first storage capacity of the local storage of the virtual computing instances is less than a second storage capacity of the persistent storage of the SDS system, instantiating one or more additional storage nodes for the SDS system, the one or more additional storage nodes being implemented using additional virtual computing instances having additional local storage. TheFIG.2process may further or alternatively include, responsive to determining that IO activity of the SDS system exceeds a designated threshold, instantiating one or more additional virtual computing instances implementing additional storage nodes for the SDS system each comprising additional local storage.

In some embodiments, theFIG.2process includes analyzing IO activity of the SDS system to determine a given subset of the data stored in the storage system having at least a designated threshold likelihood of being the subject of one or more read requests from a given one of the two or more storage nodes of the SDS system, and storing the given subset of the data stored in the storage system as read-only data in a given portion of the local storage of a given one of the virtual computing instances implementing the given storage node. Storing the given subset of the data stored in the SDS system as read-only data in the given portion of the local storage associated with the given virtual computing instance implementing the given storage node may comprise relocating the given subset of the data from one or more other portions of the local storage of one or more other virtual computing instances implementing one or more other ones of the two or more storage nodes of the SDS system.

The particular processing operations and other system functionality described in conjunction with the flow diagram ofFIG.2are presented by way of illustrative example only, and should not be construed as limiting the scope of the disclosure in any way. Alternative embodiments can use other types of processing operations. For example, the ordering of the process steps may be varied in other embodiments, or certain steps may be performed at least in part concurrently with one another rather than serially. Also, one or more of the process steps may be repeated periodically, or multiple instances of the process can be performed in parallel with one another.

Illustrative embodiments provide technical solutions for optimizing or improving storage performance in information processing system utilizing SDS resources. In some embodiments, the information processing system is assumed to comprise SDS infrastructure including SDS cloud appliances.FIG.3shows an example of a SDS infrastructure300, which includes various storage appliances having different cost and performance characteristics. In theFIG.3example, the SDS cloud infrastructure300includes cloud-based virtual computing resources301(e.g., implementing compute nodes110and/or storage nodes140), a cloud-based block storage service303, and a cloud-based object storage service305. For ease of illustration, in the description below it is assumed that the cloud-based virtual computing resources301comprise cloud VMs, and are thus also referred to herein as cloud VMs301. It should be appreciated, however, that other types of virtual computing resources such as containers may be used in other embodiments.

The cloud VMs301implement ephemeral storage310(e.g., in the form of local drives that are provided by default in VMs offered by a cloud service provider on which the cloud VMs301execute). As a particular example, the ephemeral storage310may include local drives in Amazon Elastic Compute Cloud (EC2) in an Amazon Web Services (AWS) cloud infrastructure. The ephemeral storage310(e.g., the local drives of the cloud VMs301) can perform well, and has cost advantages in that such storage does not incur extra costs (e.g., as it is provided by default in the cloud VMs301). While the ephemeral storage310can have a large storage capacity, it has drawbacks in that it is ephemeral—the data stored in the local drives of the cloud VMs301is lost when the cloud VMs301are shut down.

The cloud-based block storage service303implements networked block storage in the SDS infrastructure300, and is an example of underlying storage which may be used to provide a write cache in the data storage system130. As a particular example, the cloud-based block storage service303may include networked block storage such as Amazon Elastic Block Store (EBS) in AWS cloud infrastructure. The cloud-based block storage service303(e.g., which may be general purpose or have provisioned IOPS options) can also perform well, but has some additional network latency relative to the ephemeral storage310of the cloud VMs301. The cloud-based block storage service303is persistent (e.g. it is not lost if the cloud VMs301are shut down). The cost of the cloud-based block storage service303, however, can be significant (e.g., the costs may be based on the amount of data stored, such as per gigabyte (GB)). Further, the cloud-based block storage service303may have IOPS and bandwidth limits imposed by a cloud service provider. Generally, the cloud-based block storage service303provides small, fast and expensive storage media relative to the cloud-based object storage service305.

The cloud-based object storage service305implements object storage in the SDS infrastructure300, and is an example of the underlying storage which may be used to provide persistent backend storage in the data storage system130. As a particular example, the cloud-based object storage service305may include object storage such as Amazon Simple Storage Service (S3) in AWS cloud infrastructure. The cloud-based object storage service305provides a relatively lower cost storage option, but its performance is also relatively slow and requires transformation of block storage into objects using object storage pools. Generally, the cloud-based object storage service305provides large, slow and inexpensive storage media relative to the cloud-based block storage service303.

It should be noted that while particular examples of the ephemeral storage310in the cloud VMs301(e.g., Amazon EC2), cloud-based block storage service303(e.g., Amazon EBS), the cloud-based object storage service305(e.g., Amazon S3) are given in the context of an AWS cloud infrastructure, various other types of cloud infrastructure (e.g., Microsoft Azure, Google Cloud, etc.) may offer similar options with equivalent functionality and performance characteristics. Further, it is not required that all portions of the SDS infrastructure300be hosted on clouds of a same cloud service provider. For example, the cloud VMs301may be hosted on clouds of a first cloud service provider, while the cloud-based block storage service303and/or the cloud-based object storage service305may be hosted on clouds of one or more other cloud service providers.

The technical solutions described herein provide an architecture for processing I/O requests which can leverage different types of storage services that are available in a SDS cloud infrastructure (e.g., such as the SDS infrastructure300) to provide an optimal price versus performance tradeoff while maintaining a highest level of availability and resilience.

In some embodiments, a first cloud storage service having a first performance level (e.g., the cloud-based block storage service303in the SDS infrastructure300) is used for a write cache, and a second cloud storage service having a second performance level lower than the first performance level (e.g., the cloud-based object storage service305in the SDS infrastructure300) is used for backend persistent storage where the full capacity of the address space of a storage appliance is stored. Local storage of virtual computing resources implementing compute and/or storage nodes of a storage system (e.g., the ephemeral storage310of the cloud VMs301) is used as a storage “side car” which has a third performance level, with the third performance level being greater than the first and second performance levels of the first and second cloud storage services providing the write cache and backend persistent storage. At least a portion of the storage capacity of the storage appliance (e.g., most or all of the storage capacity) is capable of being stored in the storage side car acting as a read cache that maximizes performance for read IO requests.

FIG.4shows a storage read request workflow400in a storage array, including a host adapter402, a write cache404, protected storage406, and unprotected storage408. Here, the protected storage406is the primary storage, which is relatively slow compared with the unprotected storage408. In some embodiments, the protected storage406includes Redundant Array of Independent Disks (RAID) protected HDDs (e.g., large, slow, expensive storage media) while the unprotected storage408includes SSDs (e.g., fast, expensive media). In the storage read request workflow400, the host adapter402submits read requests first to a write cache404. In step401, if read request data is present in the write cache404(e.g., a cache hit), such data is read from the write cache404. In step403, if there is a cache read miss and the read data is in the unprotected storage408, the data is read from the unprotected storage408. In step405, if there is a cache read miss and read data is not in the unprotected storage408, the data is read from the protected storage406. The protected storage406synchronizes stored data with the unprotected storage408as a background process in step407.

In some embodiments, a storage “side car” is used which leverages the local drive capacity that is already paid for or included as part of virtual computing resources providing compute and/or storage nodes of a storage system (e.g., the cloud VMs301of the SDS infrastructure300). This local drive capacity is advantageously very low latency, and can provide large capacity. If one of the virtual computing resources is shut down, contents in its associated local drive will be lost (e.g., the local drive provides ephemeral storage). Until a new instance of the virtual computing resource is spun up elsewhere and re-populated with data stored in backend persistent storage (e.g., the cloud-based object storage service305in the SDS infrastructure300), read IO requests may be served from the backend persistent storage or, potentially, a write cache (e.g., the cloud-based block storage service303in the SDS infrastructure300).

In some cases, it is assumed that the full storage capacity of the backend persistent storage will fit into the storage side car provided by local drives associated with virtual computing resources implementing compute and/or storage nodes of a storage system. This may be facilitated in that protection is not required for the data stored in the storage side car (e.g., the local drives associated with the virtual computing resources implementing compute and/or storage nodes of the storage system). When the full storage capacity is made available in the storage side car, almost all read requests may be served from the storage side car. In some cases, almost 100% of read requests may be served by the storage side car storage if invalidation of data in the storage side car (e.g., in response to write requests) is replaced with synchronous update of the storage side car (e.g., from a write cache to the storage side car).

Where the full storage capacity of the backend persistent storage does not fit into the storage side car storage, additional algorithms can be used to populate data which is most likely to be accessed in the storage side car. In this case, the storage side car performs like a large capacity read cache. As another option when the full storage capacity of the backend persistent storage does not fit into the storage side car, additional virtual computing resources (e.g., cloud VMs) can be spun up to provide the needed local drive capacity. This can be done when a storage array is set up, or in real time as capacity of the storage array is increased. The additional virtual computing resources advantageously provide not only additional local drive capacity for the storage side car, but also additional IOPS processing capability due to the addition of the new virtual computing resources. The real-time addition of additional virtual computing resources may, in some cases, be done temporarily depending on performance demands. For example, when performance demand increases (e.g., there is need for more compute and/or storage nodes to meet IOPS requirements) or when there is a need for lower latency (e.g., to meet latency-based service level objectives (SLOs)), additional virtual computing resources (e.g., cloud VMs) may be spun up. Conversely, when performance demand decreases, one or more virtual computing resources (e.g., cloud VMs) may be shut down. Additional performance optimizations and improvements may be provided by placing data in the local drive of a particular virtual computing resource (e.g., a particular cloud VM) which is processing a read request directed to that data. This localization will avoid network hops across compute and/or storage nodes, and will gain additional latency reductions.

FIG.5shows storage write and read request workflows500in the SDS infrastructure300. More particularly,FIG.5shows the cloud-based virtual computing resources301(e.g., cloud VMs), ephemeral storage310on the cloud-based virtual computing resources301, the cloud-based block storage service303implementing a write cache, and a cloud-based object storage service305implementing persistent storage. In step501, a write request is submitted by the cloud-based virtual computing resources301to the write cache of the cloud-based block storage service303. The new data will be written in the write cache, and is synchronized to the persistent storage of the cloud-based object storage service305in step502. In step503, the write cache of the cloud-based block storage service303invalidates or synchronizes the newly written data in the ephemeral storage310that is local to the cloud-based virtual computing resources301.

In step504, a read request is submitted from the cloud-based virtual computing resources301. In step505, a determination is made as to whether the data in the read request is available in the ephemeral storage310local to the cloud-based virtual computing resources301. If the result of the step505determination is yes, the data is read from the ephemeral storage310local to the cloud-based virtual computing resources301in step506. If the result of the step505determination is no, the data is read from the write cache implemented by the cloud-based block storage service303in step506-1, or from the persistent storage implemented by the cloud-based object storage service305in step506-2. In some embodiments, data is read from the write cache if available and is only read from the persistent storage if the data is not available in the write cache (e.g., as the write cache is assumed to provide a higher level of performance than the persistent storage). In other embodiments, step506-1may be skipped and the data may be read directly from the persistent storage if the result of the step505determination is no. In step507, data is synchronized in a background process between the persistent storage in the cloud-based object storage service305and the ephemeral storage310local to the cloud-based virtual computing resources301.

The use of a storage side car in the technical solutions described herein provides various technical advantages in meeting cost, performance and availability requirements. For example, an approach which uses only a cloud-based block storage service (e.g., EBS) is a very expensive solution. An approach which uses the cloud-based block storage service for a write cache and puts persistent data in a cloud-based object storage service (e.g., S3) would have long read IO latency. Another option is to develop a read cache, and to put the read cache in the cloud-based block storage service. This may be effective from a performance perspective, but the read cache size would be limited due to cost requirements. Still another option is to run an entire SDS instance in ephemeral storage (e.g., local drives of cloud VMs providing storage nodes of a storage system). This may achieve performance objectives, but will not meet availability requirements in the case where one or more of the cloud VM instances fails and loses its ephemeral data. A variation of this approach would be to snapshot the SDS instance in the ephemeral storage to cloud-based object storage in regular intervals. While this provides some data protection, there will always be a lag between the latest update and the last snapshot and thus the Recovery Point Objective (RPO) will never be zero.

FIG.6schematically illustrates a framework of a server node (or more generally, a computing node) for hosting logic for IO processing utilizing SDS resources according to an exemplary embodiment of the disclosure. The server node600comprises processors602, storage interface circuitry604, network interface circuitry606, virtualization resources608, system memory610, and storage resources616. The system memory610comprises volatile memory612and non-volatile memory614. The processors602comprise one or more types of hardware processors that are configured to process program instructions and data to execute a native operating system (OS) and applications that run on the server node600.

For example, the processors602may comprise one or more CPUs, microprocessors, microcontrollers, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and other types of processors, as well as portions or combinations of such processors. The term “processor” as used herein is intended to be broadly construed so as to include any type of processor that performs processing functions based on software, hardware, firmware, etc. For example, a “processor” is broadly construed so as to encompass all types of hardware processors including, for example, (i) general purpose processors which comprise “performance cores” (e.g., low latency cores), and (ii) workload-optimized processors, which comprise any possible combination of multiple “throughput cores” and/or multiple hardware-based accelerators. Examples of workload-optimized processors include, for example, graphics processing units (GPUs), digital signal processors (DSPs), system-on-chip (SoC), tensor processing units (TPUs), image processing units (IPUs), deep learning accelerators (DLAs), artificial intelligence (AI) accelerators, and other types of specialized processors or coprocessors that are configured to execute one or more fixed functions.

The storage interface circuitry604enables the processors602to interface and communicate with the system memory610, the storage resources616, and other local storage and off-infrastructure storage media, using one or more standard communication and/or storage control protocols to read data from or write data to volatile and non-volatile memory/storage devices. Such protocols include, but are not limited to, non-volatile memory express (NVMe), peripheral component interconnect express (PCIe), Parallel ATA (PATA), Serial ATA (SATA), Serial Attached SCSI (SAS), Fibre Channel, etc. The network interface circuitry606enables the server node600to interface and communicate with a network and other system components. The network interface circuitry606comprises network controllers such as network cards and resources (e.g., network interface controllers (NICs) (e.g., SmartNICs, RDMA-enabled NICs), Host Bus Adapter (HBA) cards, Host Channel Adapter (HCA) cards, I/O adaptors, converged Ethernet adaptors, etc.) to support communication protocols and interfaces including, but not limited to, PCIe, DMA and RDMA data transfer protocols, etc.

The virtualization resources608can be instantiated to execute one or more service or functions which are hosted by the server node600. For example, the virtualization resources608can be configured to implement the various modules and functionalities of the volume tiering logic as discussed herein. In one embodiment, the virtualization resources608comprise virtual machines that are implemented using a hypervisor platform which executes on the server node600, wherein one or more virtual machines can be instantiated to execute functions of the server node600. As is known in the art, virtual machines are logical processing elements that may be instantiated on one or more physical processing elements (e.g., servers, computers, or other processing devices). That is, a “virtual machine” generally refers to a software implementation of a machine (i.e., a computer) that executes programs in a manner similar to that of a physical machine. Thus, different virtual machines can run different operating systems and multiple applications on the same physical computer.

A hypervisor is an example of what is more generally referred to as “virtualization infrastructure.” The hypervisor runs on physical infrastructure, e.g., CPUs and/or storage devices, of the server node600, and emulates the CPUs, memory, hard disk, network and other hardware resources of the host system, enabling multiple virtual machines to share the resources. The hypervisor can emulate multiple virtual hardware platforms that are isolated from each other, allowing virtual machines to run, e.g., Linux and Windows Server operating systems on the same underlying physical host. The underlying physical infrastructure may comprise one or more commercially available distributed processing platforms which are suitable for the target application.

In another embodiment, the virtualization resources608comprise containers such as Docker containers or other types of Linux containers (LXCs). As is known in the art, in a container-based application framework, each application container comprises a separate application and associated dependencies and other components to provide a complete filesystem, but shares the kernel functions of a host operating system with the other application containers. Each application container executes as an isolated process in user space of a host operating system. In particular, a container system utilizes an underlying operating system that provides the basic services to all containerized applications using virtual-memory support for isolation. One or more containers can be instantiated to execute one or more applications or functions of the server node600as well execute one or more of the various modules and functionalities as discussed herein. In yet another embodiment, containers may be used in combination with other virtualization infrastructure such as virtual machines implemented using a hypervisor, wherein Docker containers or other types of LXCs are configured to run on virtual machines in a multi-tenant environment.

The various components of, e.g., the IO request processing logic117and the data caching and synchronization logic119, comprise program code that is loaded into the system memory610(e.g., volatile memory612), and executed by the processors602to perform respective functions as described herein. In this regard, the system memory610, the storage resources616, and other memory or storage resources as described herein, which have program code and data tangibly embodied thereon, are examples of what is more generally referred to herein as “processor-readable storage media” that store executable program code of one or more software programs. Articles of manufacture comprising such processor-readable storage media are considered embodiments of the disclosure. An article of manufacture may comprise, for example, a storage device such as a storage disk, a storage array or an integrated circuit containing memory. The term “article of manufacture” as used herein should be understood to exclude transitory, propagating signals.

The system memory610comprises various types of memory such as volatile RAM, NVRAM, or other types of memory, in any combination. The volatile memory612may be a dynamic random-access memory (DRAM) (e.g., DRAM DIMM (Dual In-line Memory Module), or other forms of volatile RAM. The non-volatile memory614may comprise one or more of NAND Flash storage devices, SSD devices, or other types of next generation non-volatile memory (NGNVM) devices. The system memory610can be implemented using a hierarchical memory tier structure wherein the volatile memory612is configured as the highest-level memory tier, and the non-volatile memory614(and other additional non-volatile memory devices which comprise storage-class memory) is configured as a lower level memory tier which is utilized as a high-speed load/store non-volatile memory device on a processor memory bus (i.e., data is accessed with loads and stores, instead of with I/O reads and writes). The term “memory” or “system memory” as used herein refers to volatile and/or non-volatile memory which is utilized to store application program instructions that are read and processed by the processors602to execute a native operating system and one or more applications or processes hosted by the server node600, and to temporarily store data that is utilized and/or generated by the native OS and application programs and processes running on the server node600. The storage resources616can include one or more HDDs, SSD storage devices, etc.