Patent Description:
The majority of enterprise datacenters today do not have the capacity to effectively manage and handle petabytes of data at scale and at performance. Data-intensive applications and tools such as Artificial Intelligence (AI) inferencing and analytics generate and consume an exploding amount of data and telemetry that needs to be moved, stored and processed in a more secure, faster, and scalable way. In a hyper-scaled datacenter, this is typically performed by adding additional servers to the datacenter. However, dependent on the workloads being run in the datacenter, one type of component in these servers may be over-subscribed, while another maybe underutilized, which means customers and service providers are not optimizing the use of their investment.

<CIT> relates to background information.

In the following description, any embodiment referred to and not falling within the scope of the claims is merely an example useful to the understanding of the invention.

Features of embodiments of the claimed subject matter will become apparent as the following detailed description proceeds, and upon reference to the drawings, in which like numerals depict like parts, and in which:.

Although the following Detailed Description will proceed with reference being made to illustrative embodiments of the claimed subject matter, many alternatives, modifications, and variations thereof will be apparent to those skilled in the art. Accordingly, it is intended that the claimed subject matter be viewed broadly, and be defined as set forth in the accompanying claims.

A data management platform includes accelerator servers and compute servers. Storage devices managed by accelerator servers are disaggregated from compute servers to enable storage capacity to scale independent of compute.

<FIG> is a conceptual view of an embodiment of a Data Management Platform (DMP) <NUM>. In the embodiment shown in <FIG>, the Data Management Platform <NUM> is a rack-centric, physical cluster with racks <NUM> interconnected via a routing interconnect <NUM>. The routing interconnect <NUM> can be an Ethernet fabric arrayed in a multi-stage Clos topology or any other Open Systems Interconnect (OS1) layer <NUM> routing interconnect.

A rack <NUM> in a datacenter is a type of physical steel and electronic framework that is designed to house servers, networking devices, cables and other data center computing equipment. Each rack <NUM> connects into the routing interconnect <NUM> and can include one or more compute servers <NUM>, accelerator servers <NUM>, utility servers <NUM> and infrastructure servers <NUM>. A server can also be referred to as a node.

The utility servers <NUM> are used to initialize the physical cluster. During initialization the utility servers <NUM> perform orchestration and scheduling functions. In an embodiment, Kubernetes (K8s) is used to perform functions for the orchestrator/scheduler <NUM>. Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. The Kubernetes Control Plane is hosted on the Infrastructure Servers <NUM>. The Kubernetes Host Agent runs on all Compute servers <NUM> and accelerator servers <NUM>.

Application deployment can also be automated through the use of a virtual machine. Other examples of an Orchestrator/scheduler <NUM> include OpenShift (a platform as a service (PaaS) from Red Hat that is built on Docker and Kubernetes) and Pivotal Container Service (PKS).

Control plane manager <NUM> can perform functions to create, manage, and update infrastructure resources such as Virtual Machines. The control plane manager <NUM> can also initialize physical machines and network switches. Examples of control plane managers <NUM> include Fleet, Red Hat Satellite, Teraform and Metal As A Service (MaaS).

Each of the compute servers <NUM>, accelerator servers <NUM>, utility servers <NUM> and infrastructure servers <NUM> includes a Baseboard Management Controller (BMC) <NUM>. The BMC <NUM> is a specialized service processor that monitors the physical state of the compute servers <NUM>, accelerator servers <NUM>, utility servers <NUM> and infrastructure servers <NUM> and provides services to monitor and control operations via Management APIs <NUM>. Examples of Management APIs <NUM> include the Intelligent Platform Management Interface (IPMI), Redfish® (a Distributed Management Task Force (DMTF) Standard) and Dell® Open Manage Enterprise (OME).

<FIG> is a block diagram of an embodiment of the Data Management Platform (DMP) <NUM> shown in <FIG> in a physical cluster <NUM>. The physical cluster <NUM> has N racks <NUM>, <NUM>-<NUM>,. In one embodiment, N is <NUM>. Each rack <NUM> includes a compute server <NUM> and at least one accelerator server <NUM>. Each compute server <NUM> and accelerator server <NUM> is communicatively coupled to a data switch <NUM> and a management switch <NUM>. The data switch <NUM> in each rack <NUM> provides a data plane <NUM> (also referred to as a data fabric) between compute servers <NUM> and accelerator servers <NUM> in a same rack <NUM>, in other racks <NUM> and infrastructure servers <NUM> shared by the plurality of racks <NUM>. The management switch <NUM> in each rack <NUM> provides a control plane <NUM> (also referred to as a management network) between the racks <NUM> and utility servers <NUM> shared by the plurality of racks <NUM>.

<FIG> is a block diagram of an embodiment of one of the compute servers <NUM> in the physical cluster <NUM> shown in <FIG>. In the embodiment shown, the compute server <NUM> includes a System-on-Chip <NUM>, a network interface controller <NUM> and compute server control logic <NUM>. The network interface controller <NUM> is communicatively coupled to the data plane <NUM> shown in <FIG>. An embedded network interface controller <NUM> in the System-on-Chip <NUM> is communicatively coupled to the control plane <NUM> shown in <FIG>.

<FIG> is a block diagram of an embodiment of one of the accelerator servers <NUM> in the physical cluster <NUM> shown in <FIG>. In the embodiment shown, the accelerator server <NUM> performs storage processing tasks, and can be referred to as a storage server <NUM>.

The storage server <NUM> includes storage server control logic <NUM> communicatively coupled to System-on-Chip <NUM>, network interface controller <NUM> and one or more solid-state drives <NUM>. In an embodiment, the storage server control logic <NUM> is communicatively coupled to the solid-state drives <NUM> and network interface controller <NUM> using the Peripheral Component Interconnect (PCI)-Express (PCIe) protocol. An embedded network interface controller <NUM> in the System-on-Chip <NUM> is communicatively coupled to the control plane <NUM> shown in <FIG>.

The storage server control logic <NUM> performs storage processing tasks offloaded by the System-on-Chip <NUM> to allow compute and storage to be disaggregated into independently scalable resources.

<FIG> is a logical view of access to solid-state drives <NUM> from the compute server <NUM> in the physical cluster <NUM> shown in <FIG>. Virtual Routing Functions <NUM> in operating system kernel space <NUM> provide access for a relational database management system <NUM> in user space <NUM> to data stored in solid-state drives <NUM> in the storage server <NUM> (<FIG>) via the data plane <NUM>. The virtual routing functions <NUM> include a Forwarding Information Base (FIB) <NUM> and a Flow Table <NUM> that stores routes and policy.

Router <NUM> provides secure network connectivity for virtual managers and containers. An example of the router <NUM> is Calico. Calico provides secure network connectivity for containers and virtual machine workloads. Calico uses Layer <NUM> (the network layer) of the Open System Interconnection (OSI) model and the Border Gateway Protocol (BGP) to build routing tables. Calico creates a flat Layer-<NUM> network and assigns a fully routable Internet Protocol (IP) address to every rack <NUM>. Workloads can communicate without IP encapsulation or network address translation for bare metal performance. Calico uses Felix (a per node domain daemon) to configure routes and enforce network policies.

Shared resources, such as last level cache (LLC) and main memory bandwidth have a significant effect on workload performance in the Data Management Platform (DMP). Monitoring and managing these resources more closely enables deployments to meet more stringent workload demands including increasingly strict performance service-level agreements (SLAs).

<FIG> is a block diagram of the storage server <NUM> shown in <FIG> that performs resource control of storage services.

The storage server <NUM> includes a system on chip (SOC or SoC) <NUM> that combines processor, memory, and Input/Output (I/O) control logic into one SoC package. The SoC <NUM> includes at least one Central Processing Unit (CPU) module <NUM> and a memory controller <NUM>. In other embodiments, the memory controller <NUM> can be external to the SoC <NUM>. The CPU module <NUM> includes at least one processor core <NUM> that includes a Level <NUM>(L1) and Level <NUM> (L2) cache <NUM>, and a level <NUM> (L3) cache <NUM> that is shared with other processor cores <NUM> in the CPU module <NUM>.

Although not shown, each of the processor cores <NUM> can internally include execution units, prefetch buffers, instruction queues, branch address calculation units, instruction decoders, floating point units, retirement units, etc. The CPU module <NUM> can correspond to a single core or a multi-core general purpose processor, such as those provided by Intel® Corporation, according to one embodiment.

Within the I/O subsystem <NUM>, one or more I/O interface(s) <NUM> are present to translate a host communication protocol utilized within the processor cores <NUM> to a protocol compatible with particular I/O devices. Some of the protocols that I/O interfaces can be utilized for translation include Peripheral Component Interconnect (PCI)-Express (PCIe); Universal Serial Bus (USB); Serial Advanced Technology Attachment (SATA) and Institute of Electrical and Electronics Engineers (IEEE) <NUM> "Firewire".

The I/O interface(s) <NUM> can communicate via memory <NUM> and/or L3 cache <NUM> with one or more solid-state drives <NUM> and network interface controller <NUM>. The solid-state drives <NUM> can be communicatively and/or physically coupled together through one or more buses using one or more of a variety of protocols including, but not limited to, SAS (Serial Attached SCSI (Small Computer System Interface)), PCIe (Peripheral Component Interconnect Express), NVMe (NVM Express) over PCIe (Peripheral Component Interconnect Express), and SATA (Serial ATA (Advanced Technology Attachment)). In other embodiments, other storage devices, for example, other storage devices such as Hard Disk Drives (HDD) can be used instead of solid-state drives <NUM> and the Hard Disk Drives and/or Solid-State drives can be configured as a Redundant Array of Independent Disks (RAID).

Non-Volatile Memory Express (NVMe) standards define a register level interface for host software to communicate with a non-volatile memory subsystem (for example, solid-state drive <NUM>) over Peripheral Component Interconnect Express (PCIe), a high-speed serial computer expansion bus. The NVM Express standards are available at www. nvmexpress. The PCIe standards are available at www.

In an embodiment, memory <NUM> is volatile memory and memory controller <NUM> is a volatile memory controller. Volatile memory is memory whose state (and therefore the data stored in it) is indeterminate if power is interrupted to the device. Dynamic volatile memory requires refreshing the data stored in the device to maintain state. One example of dynamic volatile memory includes DRAM (Dynamic Random Access Memory), or some variant such as Synchronous DRAM (SDRAM). A memory subsystem as described herein can be compatible with a number of memory technologies, such as DDR3 (Double Data Rate version <NUM>, original release by JEDEC (Joint Electronic Device Engineering Council) on June <NUM>, <NUM>). DDR4 (DDR version <NUM>, initial specification published in September <NUM> by JEDEC), DDR4E (DDR version <NUM>), LPDDR3 (Low Power DDR version3, JESD209-3B, August <NUM> by JEDEC), LPDDR4) LPDDR version <NUM>, JESD209-<NUM>, originally published by JEDEC in August <NUM>), WIO2 (Wide Input/Output version <NUM>, JESD229-<NUM> originally published by JEDEC in August <NUM>, HBM (High Bandwidth Memory, JESD325, originally published by JEDEC in October <NUM>, DDR5 (DDR version <NUM>, currently in discussion by JEDEC), LPDDR5, HBM2 (HBM version <NUM>), currently in discussion by JEDEC, or others or combinations of memory technologies, and technologies based on derivatives or extensions of such specifications. The JEDEC standards are available at www.

In another embodiment, memory <NUM> is a non-volatile memory (NVM) and memory controller <NUM> is a non-volatile memory controller. A non-volatile memory device is a memory whose state is determinate even if power is interrupted to the device. A non-volatile memory device can include a byte-addressable write-in-place three dimensional crosspoint memory device, or other byte addressable write-in-place NVM devices (also referred to as persistent memory), such as single or multi-level Phase Change Memory (PCM) or phase change memory with a switch (PCMS), NVM devices that use chalcogenide phase change material (for example, chalcogenide glass), resistive memory including metal oxide base, oxygen vacancy base and Conductive Bridge Random Access Memory (CB-RAM), nanowire memory, ferroelectric random access memory (FeRAM, FRAM), magneto resistive random access memory (MRAM) that incorporates memristor technology, spin transfer torque (STT)-MRAM, a spintronic magnetic junction memory based device, a magnetic tunneling junction (MTJ) based device, a DW (Domain Wall) and SOT (Spin Orbit Transfer) based device, a thyristor based memory device, or a combination of any of the above, or other memory.

In yet another embodiment, memory <NUM> includes both byte addressable write-in-place NVM devices and volatile memory devices that can be included on one or more memory modules.

Shared resources that have a significant effect on workload performance include processor cache and memory bandwidth resources, which can have a large impact on application performance and runtime determinism. Monitoring and managing these resources more closely enables deployments to meet more stringent workload demands including increasingly strict performance service-level agreements (SLAs) to support emerging workloads such as Network Function Virtualization (NFV).

<FIG> illustrates data transfer between the network interface controller <NUM> and the solid-state drive <NUM> in the storage server <NUM>. I/O adapter(s) <NUM> include a first PCIe adapter <NUM> to manage communications over a first PCIe interface <NUM> from the solid-state drive <NUM> and a second PCIe interface <NUM> to manage communications over a first PCIe interface <NUM> to the network interface controller <NUM>. The network interface controller <NUM> can exchange data using Remote Direct Memory Access (RDMA), for example, a direct memory access from L3 cache <NUM> and/or memory <NUM> to memory of the compute server <NUM> or another accelerator server <NUM> communicatively coupled to the data plane <NUM>.

The solid-state drive <NUM> and network interface controller <NUM> exchange data through the L3 cache <NUM> and/or memory <NUM>. L3 cache <NUM> can also be referred to as a last level cache (LLC). In addition to being shared with other processor cores <NUM> in the CPU module <NUM>, the level <NUM> (L3) cache <NUM> is also shared with the first PCIe interface <NUM> and the second PCIe interface <NUM>.

Multiple agents (processor cores <NUM>, the first PCIe interface <NUM> and the second PCIe interface <NUM>) all competitively accessing the same L3 cache <NUM> can result in cache misses in L3 cache <NUM>, cache evictions from L3 cache <NUM> to memory <NUM> and high latency variability in transactions for the agents. If the bandwidth of the network interface controller <NUM> is well matched to the bandwidth of the solid-state drive <NUM> and the L3 cache <NUM> is of sufficient size, the majority of the data transfer between the solid-state drive <NUM> and the network interface controller <NUM> occurs through the L3 cache <NUM> with no eviction ("spill") to the memory <NUM> via path <NUM>.

To minimize spill from L3 cache <NUM> to the memory <NUM>, a configurable portion (subset of cache ways) of the L3 cache <NUM> is dedicated to storing data to be transferred between the solid-state drive <NUM> and the network interface controller <NUM>. In the embodiment shown in <FIG>, there are N processor cores <NUM>-<NUM>,. A first subset of cache ways <NUM> of the L3 cache <NUM> is dedicated to both processor core <NUM>-<NUM> and processor core <NUM>-<NUM>. A second subset of cache ways <NUM> of the L3 cache <NUM> is dedicated to processor core <NUM>-<NUM>, solid-state drive <NUM> and the network interface controller <NUM>.

In an embodiment, Intel® Cache Allocation Technology (CAT) can be used to enable a subset of cache ways of the L3 cache <NUM> to be dedicated to specific processor cores <NUM>-<NUM>,. <NUM>-N and/or I/O memory spaces (PCIe), controlling which agents share/compete for a specific subset of cache ways (or portion) of the L3 cache <NUM>. All other agents are excluded from using the second set of cache ways <NUM> in the L3 cache <NUM> that is dedicated to storing data to be transferred between the solid-state drive <NUM> and the network interface controller <NUM>.

The use of the configurable second subset of cache ways <NUM> of the L3 cache <NUM> reduces workload variability providing a more precise and predictable resource allocation of storage services versus other co-located workloads enabling more accurate service level predictability of co-located storage services with other workloads. Three quality of service mechanisms (cache, core/Input/Output memory and logical volume bandwidth throttling) are combined to provide tunable resource sharing, isolation and reduction of variation.

Containers (for example, a Kubernetes container or a Virtual machine container) and threads related to storage services and networking are allocated a subset of cache ways or buffer space and an effective allocation to a sub-set of memory bandwidth (memory transaction credits in one embodiment - memory bandwidth enforcement) to constrain network/storage functions to a sub-set of cache/buffer ways and/or memory bandwidth. With suitable bandwidth matching of storage server network allocation and L3 cache <NUM> bandwidth/size allocation, the storage server <NUM> can support storage to/from network data flows wholly or nearly wholly through the second portion of cache ways <NUM> in the L3 cache <NUM> with little or no spill of data to memory <NUM>.

In addition, there are operating systems mechanisms (for example, a device mapper in the Linux operating system), to partition the access bandwidth of logical volumes (LVM) on a single storage device (such as but not limited to an NVMe solid-state drive) to a set bandwidth, for example, <NUM> Mega Bytes per second (MB/s). Combining logical volume rate Quality of Service controls with the configurable subset of cache ways described above provide an overall system solution to provide a storage service workload co-existing with other workloads in a more tunable and predicable manner.

<FIG> is a flowgraph illustrating a method to move data from the solid-state drive <NUM> to the data plane <NUM> via the L3 cache <NUM> and network interface controller <NUM>. Data can also be moved from in the opposite direction from the network interface controller <NUM> to the solid-state drive <NUM> to the data plane <NUM> via the L3 cache <NUM>.

At block <NUM>, a fixed number of cache ways (for example, second subset of cache ways <NUM>) are allocated in the L3 cache <NUM> to store data shared by the solid-state drive <NUM> and the network interface controller <NUM>. The fixed number of cache ways is tunable based on system performance requirements. The number of cache ways that are allocated in second subset of cache ways <NUM> for use only to store data to be transferred between the solid-state drive <NUM> and the network interface controller <NUM> reduces the number of cache ways in the L3 cache that are available for use by other cores and results in reduced performance for the other cores.

In an embodiment, the fixed number that is selected is not dynamically modified during operation. The N cache ways in the configurable portion of the L3 cache <NUM> are also shared by one or more processor cores <NUM> (for example, <NUM>-<NUM>). The second subset of cache ways <NUM> of the L3 cache <NUM> are isolated from other agents. The remaining cache ways in the L3 cache <NUM> (for example, first set of cache ways <NUM>) can be used/shared by other agents.

At block <NUM>, the solid-state drive <NUM> is configured to write data directly (via Direct Memory Access) to the second subset of cache ways <NUM> of the L3 cache <NUM> and the network interface controller <NUM> is configured to read data directly (via Direct Memory Access) from the second subset of cache ways <NUM> of the L3 cache <NUM>. The solid-state drive <NUM> writes data to the second subset of cache ways <NUM> of the L3 cache <NUM> while the network interface controller <NUM> is reading data from the second subset of cache ways <NUM> of the L3 cache <NUM>.

At block <NUM>, if the rate that the solid-state drive <NUM> writing to the second subset of cache ways <NUM> of the L3 cache <NUM>, and the rate that the network interface controller <NUM> is reading data from the second subset of cache ways <NUM> of the L3 cache <NUM> is not matched, processing continues with block <NUM>. If the rate that the solid-state drive <NUM> is writing to the second subset of cache ways <NUM> of the L3 cache <NUM>, and the network interface controller <NUM> is reading data from the second subset of cache ways <NUM> of the L3 cache <NUM> is matched, processing continues with block <NUM>.

At block <NUM>, data is evicted from the L3 cache <NUM> to the memory <NUM> to allow the solid-state drive <NUM> to continue to write data to the second subset of cache ways <NUM> of the L3 cache <NUM>. The eviction of data from the L3 cache can be referred to as cache spill.

<FIG> illustrates an embodiment to configure the last level cache to isolate N sets of cache ways <NUM> of the L3 cache <NUM> (also referred to as the last level cache) to be shared by the solid-state drive <NUM> and the network interface controller <NUM>. Intel® Cache Allocation Technology (CAT) includes a Class of Service (CLOS) that acts as a resource control tag into which a thread/app/Virtual Memory (VM)/container can be grouped. Each Class of Service has an associated resource capacity bitmask (CBM) indicating how much of the last level cache can be used by a given Class of Service. In the embodiment shown in <FIG>, a Class of Service table <NUM> has four classes of service labeled CLOS1-CLOS4. In other embodiments there can be more or less than four classes of service. Each CLOS register has a bit per processor core <NUM>, the state of the bit indicates if the core is part of the particular class of service. Enforce mask <NUM> is circuitry/logic that limits cache mapping to the cache-way bitmap encoded in respective CLOS register.

In the embodiment shown, each class of service CLOS1-CLOS4 has an associated capacity bit mask labeled mask1-mask4. The values of each bit in the capacity bit mask indicate the amount of the L3 cache <NUM> available for the class of service and if any of the cache ways in the sets of cache ways <NUM> are shared by the classes of service CLOS1-CLOS4.

In an embodiment, a capacity bit mask associated with a class of service is dedicated to storing data shared by the network interface controller a portion of the sets of cache ways <NUM> are shared by the solid-state drive <NUM> and the network interface controller <NUM>.

<FIG> is a flowgraph illustrating an embodiment of a method to configure and use a set of cache ways in the last level cache to be shared by the solid-state drive <NUM> and the network interface controller <NUM>.

Cache Allocation Technology enables resource allocation based on application priority or Class of Service (COS or CLOS). A processor exposes a set of Classes of Service into which applications (or individual threads) can be assigned. Cache allocation for the respective applications or threads is restricted based on the class with which they are associated. Each Class of Service can be configured using capacity bitmasks which represent capacity and indicate the degree of overlap and isolation between classes. For each logical processor there is a register exposed to allow the Operating System/Virtual Machine Manager to specify a class of service when an application, thread or Virtual Machine is scheduled. The usage of Classes of Service are consistent across resources and a class of service may have multiple resource control attributes attached, which reduces software overhead at context swap time. Rather than adding new types of class of service tags per resource, the class of service management overhead is constant. Cache allocation for the indicated application/thread/container/VM is controlled automatically by the hardware based on the class and the bitmask associated with that class. Bitmasks can be configured via mode status registers for L3 cache.

At block <NUM>, Cache Allocation Technology enables an Operating System (OS), Hypervisor /Virtual Machine Manager (VMM) or similar system service management agent to specify the amount of cache space into which an application can fill. Enumeration support is provided to query which levels of the cache hierarchy are supported and specific Cache Allocation Technology capabilities, such as the max allocation bitmask size.

At block <NUM>, the Operating System or Hypervisor configures the amount of a resource available to a particular Class of Service via a list of allocation bitmasks. The bit length of the capacity mask available is dependent on the configuration of the L3 cache.

At block <NUM>, if there is a context switch, processing continues with block <NUM>. If not, processing continues with block <NUM>.

At block <NUM>, a currently running application class of service is communicated to the execution environment (Operating System/Virtual ). A different class of service can be loaded if class of service for new thread is different from currently running application class of service. Processing continues with block <NUM>.

At block <NUM>, if there is a memory request, processing continues with block <NUM>. If not, processing continues with block <NUM>.

At block <NUM>, the class of service associated with the memory access is used to enforce the cache allocation. Processing continues with block <NUM>.

Returning to <FIG>, recovery of hardware failures in the Data Management Platform <NUM> in a physical cluster <NUM> can be performed by hardware or software load balancers together with health checks. Hardware failures can include hardware failures in compute servers <NUM>, accelerator servers <NUM>, the data switch <NUM>, the management switch <NUM>, infrastructure servers <NUM> and utility servers <NUM>. Each of the servers (compute servers <NUM>, accelerator servers <NUM>, infrastructure servers <NUM> and utility servers <NUM>) can also be referred to as a node. Typically, a logging system is used to flag events and an operator manually intervenes to remove or replace the failing hardware when a particular event is logged.

However, current load balancers do not consider degrading hardware components and/or degrading performance based on failing or degrading hardware in the Data Management Platform <NUM>. In addition, current load balancers are difficult to scale in enterprise datacenters.

<FIG> is a block diagram of an embodiment of a rack <NUM> in the Data Management Platform <NUM> in the physical cluster <NUM> shown in <FIG> for normal operation in a system with no failing or degrading hardware. A method and system for transparent system service healing of hardware failures and degrading hardware enables direct and efficient exposure of processor hardware events and measurements. Telemetry, in conjunction with an integration interface to a routing information base (RIB) <NUM>, a forwarding information base (FIB) <NUM> (<FIG>), filtering system (FS) <NUM> and Internet Protocol Anycast, selectively allows or supresses dynamic routes from a server (compute servers <NUM>, accelerator servers <NUM>, infrastructure servers <NUM> and utility servers <NUM>), based on a hardware event. A routing information base (RIB) <NUM>, is a data table that stores routes to particular network destinations.

Dynamic routing is a networking technique that provides optimal data routing. Dynamic routing enables routers to select paths according to real-time logical network layout changes. In dynamic routing, the routing protocol operating on a router is responsible for the creation, maintenance and updating of a dynamic route table. A dynamic route is a process in which network traffic to an endpoint can be forwarded via different routes, based on environment.

A failing component in a server (node) 1100a-e can impact the functionality and performance of one or more applications running on the server (node) 1100a-e. Examples of failing components in a server (node) 1100a-e include a solid-state drive, memory module or a power distribution unit. If a server (node) 1100a-e in the Data Management Platform <NUM> is degrading, a failing component event is detected by a node failure detector in the filtering system (FS) <NUM> and the route associated with the service is withdrawn from the route table <NUM>. In an embodiment, the route is withdrawn by the router <NUM> (<FIG>) for example, Calico in Kubernetes.

In an embodiment, the operating system (OS) <NUM> is the Linux operating system. A Border Gateway Protocol (BGP) client on a server (node) 1100a-e reads a routing state from the FIB <NUM> and distributes it to other BGP clients running on other servers (nodes) 1100a-e. The routes in the FIB <NUM> are set up by an agent in response to a request to provision connectivity for a particular workload. The BGP client in response to an update to the FIB <NUM>, distributes the updated route(s) to BGP clients running on other servers (nodes) 1100a-e.

In an embodiment, the agent that sets up the routes in the FIB <NUM> is Felix and the BGP clients are BIRD. BIRD is an open source implementation for routing Internet Protocol packets on Unix-like operating systems. Felix is a per node domain daemon to configure routes and enforce network policies.

The node failure detector in the filtering system <NUM> monitors hardware metrics in the node and generates alerts (for example, the failing component event). In an embodiment, the failure or degradation is detected via platform telemetry, the failure/degradation event is communicated to an open collector, for example, "collectd", and then to an event handler that takes corrective action. An example of a corrective action is to remove routes.

In an Internet Protocol Anycast implementation, the route associated with the failing server service is withdrawn, triggering removal of the route from connected peers. Connected peers are all network devices (both servers and switches) in the data-plane network in the Data Management Platform <NUM>. Flows are redirected to healthy or available server nodes 1100a-e transparently.

In the embodiment shown, a rack <NUM> that includes a plurality of nodes (N) <NUM>, <NUM> of N nodes 1110a-1110e are shown. Each node is a physical server that can be a compute server <NUM>, an accelerator server <NUM>, an infrastructure server <NUM> or a utility server <NUM>. The utility server <NUM> can also be referred to as a control plane server node that performs management tasks in the Data Management Platform.

In an embodiment, there are upto <NUM> racks <NUM> and upto <NUM> nodes <NUM> per rack in a physical cluster <NUM>. In other embodiments there can be more than <NUM> racks and <NUM> nodes per rack. There is one utility server <NUM> per node in the first three racks, one infrastructure server <NUM> per rack in the second and third rack, upto <NUM> compute servers <NUM> per rack <NUM> in the first three racks <NUM>, upto <NUM> compute servers in the next <NUM> racks <NUM>, and upto <NUM> accelerator servers <NUM> per rack <NUM>. The accelerator server <NUM> performs storage processing tasks, and can be referred to as a storage server <NUM> (<FIG>).

In an embodiment, each node 1110a-1110e includes a pod <NUM> and an operating system (OS) <NUM> (for example, a Red Hat Enterprise Linux (RHEL) operating system). A pod <NUM> is the basic execution unit of a Kubernetes application, the smallest and simplest unit in the Kubernetes object model that can be created or deployed. The pod <NUM> represents a unit of deployment: a single instance of an application in Kubernetes, which can include either a single container or a small number of containers that are tightly coupled and that share resources.

The pod <NUM> is a group of one or more containers with shared storage/network. Containers within a pod <NUM> share an Internet Protocol (IP) address and port space and can communicate with other pods <NUM> other using standard inter-process communications. Containers in different pods <NUM> have distinct Internet Protocol addresses and communicate with each other using IP addresses for pods <NUM>.

Anycast is a network addressing and routing methodology in which a single destination address has multiple routing paths to two or more endpoint destinations. A router <NUM> selects a path between nodes 1110a-e based on number of hops, distance, lowest cost, latency measurements or based on the least congested route. Under normal operation, each node 1110a-e in the rack <NUM> advertises the same Internet Protocol (IP) address (Anycast address) for a distributed common service.

Referring to the example shown in <FIG>, a service is advertised from each of the nodes 1110a-e and the anycast address (IP address) associated with that service is the same across all six nodes 1100a-e. In this example, the IP address "<NUM>. Each node 1110a-e has a unique Ethernet address that is stored in a route table <NUM> in the data switch <NUM>. The data switch <NUM> in the rack <NUM> can also be referred to as a Top of Rack (TOR) switch.

For example, the IP address for node 1100a is '<NUM>. <NUM>" and the Ethernet Address for node 1100a is <NUM>. When there are no hardware failures or degradation events, a route table <NUM> managed by the orchestrator/scheduler <NUM> (for example, Kubernetes) allows all routes (via all nodes 1100a-e in the rack <NUM>) to be advertised. The data switch <NUM> sees a single IP address (<NUM>. <NUM>) and six paths (via one of the nodes 1100a-e) to get to the destination. The destination is an application instance. In an embodiment, the application instance is a Kubernetes service. An application can be spawned as multiple application instances to load balance network traffic in the Data Management Platform <NUM> and provide access to the application and data.

The data switch <NUM> can use a built-in load balancing method, for example, Equal Cost Multipath routing (ECMP), to select one of the paths to nodes 1100a-e. Equal-cost multi-path routing (ECMP) is a routing strategy where next-hop packet forwarding to a single destination can occur over multiple "best paths" which tie for top place in routing metric calculations. Multi-path routing can be used in conjunction with most routing protocols, because it is a per-hop decision that is limited to a single router.

<FIG> is a block diagram of an embodiment of a rack <NUM> in the Data Management Platform <NUM> in the physical cluster <NUM> shown in <FIG> for degraded operation in a failing system.

During a failure or degradation event (for example, a failed Network Interface Controller <NUM> in a compute server <NUM>, failed solid-state drive <NUM> in a storage node <NUM> or an unstable operating system <NUM> in a node 1100a-e), the impacted node 1100a-e suppresses the advertisement of the route associated with the application (also referred to as a service) that is being impacted.

In an embodiment in which the orchestrator/scheduler <NUM> is Kubernetes, if the failure or degradation event is related to network connectivity, the event is handled by Kubernetes\OpenShift and Kubernetes network component. Kubernetes detects that the node 1100a-e is not available over the network. A network component updates the route table <NUM> across the physical cluster <NUM>.

If the failure or degradation event is not related to network connectivity, the event is handled by a Logging Monitoring Alerting (LMA) stack in the pod <NUM>. An exporter, that is spawned on each node 1100a-e, periodically provides metrics data to the LMA stack. Based on the metrics, the node 1100a-e is marked with additional labels and potentially additional actions can occur. For example, containers that are running on a failed or degraded node 1100a-e can be rescheduled on another node 1100a-e.

As shown in <FIG>, node 1100a has a degradation or failure event and either suppresses (never advertises) or withdraws the route associated with the impacted service via the route table <NUM>. In this example, the Anycast address is <NUM>. for each node 1100a-e in the rack <NUM>. The route to node 1100a is withdrawn, for example, node 1100a stops advertising that route and the data switch <NUM> removes that route from the route table <NUM>.

Of the six available paths to the Anycast address (<NUM>. <NUM>), the path to node 1100a (destination <NUM>. <NUM>, next-hop <NUM>. <NUM>) is removed from the route table <NUM> in the data switch <NUM>. The next-hop is the unique Internet Protocol (IP) address associated with the respective node 1100a-e. The AnyCast IP address is the IP address for the application instance or pod <NUM> on the respective node 1100a-e. Network traffic is forwarded to the pod <NUM> on the node 1100a-e and then to the application instance in the node 1100a-e. All remaining traffic flows are distributed over the remaining paths that are available via the route table <NUM> in the data switch <NUM>.

In an embodiment of a system that uses the Linux operating system, during a failure or degradation event, such as a value obtained from raw sensor data (for example, a "critical maximum power match" via the Linux "libsensors" library through the "sysfs" interface), an event action detector and manager in the pod <NUM> in the impacted node 1100a-e suppresses the advertisement of the route associated with the Kubernetes service being impacted.

A Kubernetes Service is an abstraction which defines a logical set of pods <NUM> running in a cluster <NUM>, that all provide the same functionality. When created, each Kubernetes Service is assigned a unique Internet Protocol (IP) address (also called a clusterIP), which is the route. The assigned IP address is not changed while the Kubernetes Service is alive.

A pod <NUM> can be configured to communicate with the Kubernetes Service in the orchestrator/scheduler <NUM>. The communication to the Kubernetes Service is automatically load-balanced to a pod <NUM> that is a member of the Kubernetes Service. Multiple nodes can advertise the same service IP, which is referred to as "Anycast". An example of a Kubernetes service is the ClusterIP backing a pod <NUM> or a set of pods <NUM> hosting applications, for example, NGINX (an open-source, high-performance HTTP server and reverse proxy and an IMAP/POP3 proxy server, Domain Name System (DNS) and Apache(an open-source web server).

After the event has been detected, the event action detector and manager in the pod <NUM> in the impacted node 1100a initiates a script to blackhole (suppress and not advertise another path) the route associated with the service on the impacted node 1100a.

When the failed node (in this case node 1100a) is functioning normally again, the failed node automatically advertises the Anycast IP address associated with the previously failed service (<NUM>. <NUM>) and is reinserted into the physical cluster <NUM> transparently. The data switch <NUM> detects another path for the node 1100a (destination (IP address for the pod) <NUM>. <NUM>, next-hop (IP address for node 1100a) <NUM>. <NUM>) and adds it to its existing route table <NUM> as a multi-path destination.

<FIG> is a block diagram that illustrates metrics exporters in containers in pod <NUM> (<FIG>) in a node 1100a-e that are used by the Data Management Platform <NUM> to detect node condition and failures. In the embodiment shown, there are four metrics exporters and other exporters <NUM>. Each metrics exporter is in a separate container in the pod <NUM> (<FIG>).

A device-mapper exporter <NUM> collects low-level metrics from device-mapper volumes. Examples of low-level metrics that are collected from device mapper volumes include average read/write time, average wait time, percentage utilization, queue size, number of writes/reads per second, read/write size per second, reads/writes merged per second.

A storage exporter <NUM> collects low-level metrics from solid-state drives. Examples of low-level metrics that are collected include a count of the number of program and erases to the non-volatile memory in the solid-state drive that have failed, and end-to-end error detection count, a cyclic redundancy check (CRC) error count, timed workload timer, thermal throttle status, retry buffer overflow count, wear leveling count, timed workload media wear, timed workload host read/write ratio, power loss imminent (pli)-lock loss count, bytes written to non-volatile memory in the solid-state drive, bytes written by the host to the solid-state drive and system area life remaining.

A memory bandwidth exporter <NUM> collects low-level metrics based on a memory bandwidth monitor. A Processor Counter Monitor (PCM) is an application programming interface (API) and a set of tools based on the API to monitor performance and energy metrics of Intel® processors. A memory bandwidth exporter1308 uses the Processor Counter Monitor to collect low-level metrics. Examples of low-level metrics related to memory bandwidth that are collected include channel read/write, memory read/write Mega Bytes per second, read/write, memory and Memory Mega Bytes per second.

A network interface controller <NUM> exporter collects low-level metrics from a Network Interface Controller. Examples of low-level metrics that are collected include transmit queue dropped, transmit queue stopped, receive out of buffer, transmit errors, receive buffer passed threshold, and receive/transmit signal integrity.

Other exporters included in the pod <NUM> include a server chassis exporter <NUM>, a node exporter <NUM>, and a blackbox exporter <NUM>. A server chassis exporter <NUM> collects low-level metrics from the server chassis. A node exporter <NUM> collects operating system level metrics. A blackbox exporter <NUM> collects metrics related to Hyper Text Transfer Protocol (HTTP)/ Transmission Control Protocol (TCP) endpoints.

Some exporters (device mapper <NUM> and storage <NUM>) are only used in a storage node <NUM>, because they are only monitoring metrics on the solid-state drives <NUM>. As shown in <FIG>, non-storage nodes <NUM> (for example, compute servers <NUM>, utility servers <NUM>, infrastructure servers <NUM> and non-storage accelerator servers <NUM>) do not include the device mapper <NUM> and the storage <NUM> exporters. Based on metrics from exporters (device mapper, solid-state drive, memory bandwidth and Network Interface Controller) the Data Management Platform <NUM> can detect and react on such events by redirecting traffic to application instances on healthy nodes.

When all of the nodes in the Data Management Platform cluster <NUM> are all working correctly, there is no limitation in spawning application(s) instances on multiple nodes. In that case, traffic from the network is working as described in conjunction with <FIG>. If one of exporters detects a metric that indicates that there is a hardware problem in a node, Logging Monitoring Alerting (LMA) in pods <NUM> performs an action to exclude the unhealthy node and block network traffic as described in conjunction with <FIG>.

<FIG> is a flowgraph illustrating a method for managing hardware failures in the Data Management Platform <NUM> in the physical cluster <NUM>.

At block <NUM>, the exporters (network interface exporter <NUM>, memory bandwidth exporter <NUM>, device mapper exporter <NUM>, storage exporter <NUM> and other exporters <NUM>) described in conjunction with <FIG> continuously monitor metrics in the node 1100a-e. The LMA in the pod <NUM> in the node 1100a-e gathers the metrics from the exporters.

At block <NUM>, if all metrics gathered from the exporters are good, processing continues with block <NUM>. If not, processing continues with block <NUM>.

At block <NUM>, if the node 1100a-e is operational, the node is marked operational, all metrics are good indicating that the node 1100a-e is operating without errors. If the node has recovered from a non-operational state and had previously been marked non-operational, the node is marked operational.

At block <NUM>, all nodes 1100a-e in the rack <NUM> are operational. The RIB <NUM> is updated to restore the route to the previously non-operational node 1100a-e and restore traffic to application instance(s) on the recovered node 1100a-e.

At block <NUM>, traffic is resumed to all application instances on operational nodes 1100a-e in the rack <NUM>. Processing continues with block <NUM>.

At block <NUM>, all of the metrics gathered from the exporters are not good indicating a failure or degradation event in the node 1100a-e. The node 1100a-e is marked non-operational. The anycast service advertisement and ECMP forwards accesses to an application via other application instances in other nodes 1100b-e).

At block <NUM>, the data network is not available to the non-operational node 1100a and access to the application instance on the non-operational node 1100a is not available. The RIB for all nodes in the cluster is updated.

At block <NUM>, traffic to application instances in operational nodes 1100b-e is resumed to all operational nodes 1100b-e in the rack <NUM>. Traffic to application instances is not sent to the non-operational node 1100a. Processing continues with block <NUM>.

Current load balancers (software or hardware), along with health checks, scripting or monitoring systems do not dynamically react to exceeded performance thresholds (for example, a Central Processor Unit (CPU) that is exceeding <NUM>% utilization). Monitoring and managing these performance thresholds more closely enables deployments to meet more stringent workload demands including increasingly strict performance service-level agreements (SLAs) to support emerging workloads such as Network Function Virtualization (NFV).

In an embodiment, dynamic and transparent scaling in response to pressure conditions and performance thresholds that provide an indication of performance degradation is on a per-Kubernetes service level based on defined performance thresholds. This allows for dynamic detection and transparent service scaling based on triggered performance thresholds, enabling a more optimized and scalable Kubernetes implementation.

As discussed earlier, the Kubernetes Control Plane is hosted on the Infrastructure Servers <NUM> and the Kubernetes Host Agent runs on all Compute servers <NUM> and accelerator servers <NUM>. Direct and efficient exposure of hardware events and measurements, in conjunction with an integration interface to the Routing and Information base (RIB) <NUM> is provided. Examples of hardware events and measurements include telemetry, such as raw sensor data that are exposed through the Linux "libsensors" library via a "sysfs" interface. A node is a worker machine in Kubernetes, previously known as a minion. A node may be a virtual machine or physical machine (server), depending on the cluster. Each node contains the services necessary to run pods <NUM>. The Kubernetes services on a node include the container runtime (software that executes containers and manages container images on a node), kubelet (that runs the pod <NUM>) and kube-proxy (a network proxy that runs on each node in the cluster, implementing part of the Kubernetes Service that maintains network rules on nodes).

<FIG> is a block diagram that illustrates hardware events and measurements in pod <NUM> (<FIG>) in a node <NUM> that are used by the Data Management Platform <NUM> to Monitor and manage performance thresholds <NUM> to detect node condition and failures.

Exposure of hardware events and measurements is provided though an event detector and monitor <NUM> in the pod <NUM> and IP Anycast. The exposure of hardware events and measurements allows for selectively allowing or suppressing dynamic routes from a server (a node in the Data Management Platform <NUM>), based on defined performance thresholds, such as CPU utilization percentage. These thresholds can be set before or during runtime.

Examples of performance thresholds include a percentage of CPU utilization, Input/Output Operations per second IOPS for a solid-state drive <NUM> or bandwidth utilization. The performance thresholds are associated with a specific Kubernetes service, providing per-Kubernetes-service granularity. After the performance threshold event is detected or a metric is matched, the event detector and monitor <NUM> blackholes the route associated with the service on the impacted node.

In networking, black holes refer to places in the network where incoming or outgoing traffic is silently discarded (or "dropped"), without informing the source that the data did not reach its intended recipient. When examining the topology of the network, the black holes themselves are invisible, and can only be detected by monitoring the lost traffic.

A utilization threshold specifies the percentage of the resources over a configured period of time. For example, if the resource is bandwidth of the NVMe interface to the solid-state drive <NUM>, the utilization threshold of the bandwidth can be <NUM>% of maximum bandwidth (for example, <NUM> Gigabits per second) of an NVMe interface on a solid-state drive <NUM>. If a utilization threshold is met by the filtering system <NUM>, the route associated with that given service is withdrawn. In an embodiment that uses IP Anycast, the route associated with the failing node service is withdrawn, triggering removal of the route from connected peers. Flows are redirected transparently to nodes that are within the "operating range" transparently.

A triggered utilization threshold specifies the percentage of resources that, when exceeded for a configured period of time, triggers a threshold notification. Each node has a set of routes that the node advertises. The set of routes is visible in the route table <NUM> of the node.

If there are no triggered performance thresholds, all active routes are advertised (the default mode of operation). One Anycast IP address is visible to the connected data switch <NUM>, for example, Anycast IP address (<NUM>. <NUM>) and five paths via one of the five nodes 1100a-e are available to get to the destination, which in this case is the Kubernetes service. A load balancing method, for example, ECMP can be used to select one of the paths.

During a triggered performance threshold event, the impacted node suppresses the advertisement of the route associated with the service being impacted. The node suppresses the advertisement of the route by blackholing the route associated with the service on the impacted node.

Referring to <FIG>, node 1100a has a matching performance threshold event and either suppresses (never advertises) or withdraws the route associated with the impacted service. In this example, the address is Anycast IP Address is <NUM>. The route (path) is withdrawn, that is, node 1100a stops advertising a path for Anycast IP Address <NUM>. <NUM> and the data switch <NUM> removes the path via node 1100a for Anycast IP Address <NUM>. <NUM> from the route table <NUM>. Of the five available paths to that address, the path to node 1100a (destination <NUM>. <NUM>, next-hop of <NUM>. <NUM>) is removed from the route table <NUM> in the data switch <NUM>. This removes connections to the impacted node. All remaining traffic flows are distributed over the remaining paths.

After the "performance impacted" node (in this case node 1100a) is functioning normally, node 1100a automatically advertises the next-hop (Ethernet address associated with the previously failed service (destination <NUM>. <NUM>, next-hop of <NUM>. Node 1100a is reinserted into the cluster transparently by removing the previously installed blackhole route associated with the service on node 1100a. The data switch <NUM> detects the route that was blackholed (that is, suppressed and not advertised another path for that address (destination <NUM>. <NUM>, next-hop of <NUM>. <NUM>) and adds it to the route table <NUM> as a multipath destination.

<FIG> is a flowgraph illustrating a method implemented in a storage node <NUM> in the rack <NUM> to monitor performance of the storage node <NUM>.

At block <NUM>, the Event Detector and Monitor <NUM> described in conjunction with <FIG> continuously monitors performance thresholds <NUM> in the node.

At block <NUM>, if performance thresholds do not match predefined threshold maximums, processing continues with block <NUM>. If they match, processing continues with block <NUM>.

At block <NUM>, the node meets Service Level Agreement parameters, the node is marked compliant. If the node had previously been marked non-compliant, the node is marked compliant.

At block <NUM>, all nodes in cluster are compliant and the Routing Information Base (RIB) is updated to restore the route to the previously non-compliant node to restore traffic to application instance(s) on such node.

At block <NUM>, traffic is resumed to all application instances on compliant nodes in the cluster. Processing continues with block <NUM>.

At block <NUM>, the node does not meet SLA parameters, the node is marked non-compliant.

At block <NUM>, the data network is not available to the non-compliant node and access to the application instance is not available. The RIB for all nodes in the cluster is updated.

At block <NUM>, traffic is resumed to all compliant nodes in the cluster. Traffic is not sent to the non-compliant node. Processing continues with block <NUM>.

Returning to <FIG>,as discussed earlier, each of the servers (compute servers <NUM>, accelerator servers <NUM>, infrastructure servers <NUM> and utility servers <NUM>) can also be referred to as a node. The Orchestrator/Scheduler <NUM> manages a fixed number of nodes. The number of nodes is selected to accommodate peaks in traffic in the data center and are typically overprovisioned. In current data centers, if a workload is under pressure, the orchestrator/scheduler <NUM> can either throttle workloads or prevent the scheduling of additional workloads on the nodes on which workloads are under pressure which reduces the performance of the datacenter.

Typically, when the load in a data center reaches capacity in terms of CPU, memory or storage, manual data center resizing is performed. Data center resizing involves adding new nodes, provisioning and configuration. Upon decrease in load, the data center is even more overprovisioned.

In an embodiment, the total cost of ownership (TCO) of a data center can be lowered by decreasing over-subscription of resources in data centers. Total Cost of Ownership (TCO) is lowered by monitoring various pressure conditions in an orchestrator managed data center and requesting resizing of existing nodes with additional logical resources.

<FIG> is a block diagram of an embodiment of a compute node <NUM>. The compute node <NUM> includes a system on chip (SOC or SoC) <NUM> that combines processor, memory, and Input/Output (I/O) control logic into one SoC package. The SoC <NUM> includes at least one Central Processing Unit (CPU) module <NUM> and a memory controller <NUM>.

In the embodiment shown, the SoC <NUM> also includes an Internal Graphics Processor Unit (GPU) <NUM>. The internal GPU <NUM> can include one or more GPU cores and a GPU cache which can store graphics related data for the GPU core. The GPU core can internally include one or more execution units and one or more instruction and data caches. Additionally, the Internal Graphics Processor Unit (GPU) <NUM> can contain other graphics logic units that are not shown in <FIG>, such as one or more vertex processing units, rasterization units, media processing units, and codecs.

In other embodiments, the memory controller <NUM> can be external to the SoC <NUM>. The CPU module <NUM> includes at least one processor core <NUM> that includes a Level <NUM>(L1) and Level <NUM> (L2) cache <NUM>, and a level <NUM> (L2) cache <NUM> that is shared with other processor cores <NUM> in the CPU module <NUM>.

In an embodiment, memory <NUM> is volatile memory. In yet another embodiment, memory <NUM> includes both byte addressable write-in-place NVM devices and volatile memory devices that can be included on one or more memory modules. A resource manager agent <NUM> and workloads <NUM> are stored in memory <NUM>. The compute node <NUM> also includes a persistent memory <NUM>. The persistent memory <NUM> can include a byte-addressable write-in-place three dimensional cross point memory device, or other byte addressable write-in-place non-volatile memory devices, or other memory. An example of byte-addressable write-in-place three dimensional cross point memory device is 3DXPoint (for example, Intel® Optane ® and Micron® QuantX®).

<FIG> is a block diagram of another embodiment of a compute node <NUM>. The compute node <NUM> includes a system on chip (SOC or SoC) <NUM> that combines processor, memory, and Input/Output (I/O) control logic into one SoC package. The SoC <NUM> includes at least one Central Processing Unit (CPU) module <NUM> and a memory controller <NUM>.

The compute node <NUM> also includes a Field Programmable Gate Array (FPGA) <NUM> and an accelerator <NUM> that are communicatively coupled to the Input/Output (I/O) subsystem <NUM> in the SoC <NUM>. In an embodiment, FPGA <NUM> is an Intel® Agilex® FPGA Device.

<FIG> is a block diagram of an embodiment of a rack <NUM> in the Data Management Platform <NUM> in the physical cluster shown in <FIG> that includes a resource manager <NUM> to automatically add and remove logical resources. The rack <NUM> includes a plurality of compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> and an accelerator node <NUM>. The compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> can include a compute node <NUM> an as discussed in conjunction with <FIG> or a compute node <NUM> as discussed in conjunction with <FIG>.

The resource manager <NUM> monitors metrics to determine when to automatically attach and configure logical resources. In one embodiment, the resource manager <NUM> is in the orchestrator/scheduler <NUM>. In other embodiments, the resource manager <NUM> is included in one of the compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> or in another component in the Data Management Platform <NUM>. The resource manager <NUM> has access to the accelerator node <NUM> and to all metrics for the Data Management Platform <NUM>. The resource manager <NUM> in the Data Management Platform <NUM> automatically attaches, detaches and configures logical resources (for example, memory, storage volumes, Graphics Processor Unit (GPU), and Field Programmable Gate Array (FPGA) logical resources) without user intervention.

In the particular non-limiting example depicted in <FIG>, there are three compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> and one accelerator node <NUM> in the rack <NUM>. The orchestrator/scheduler <NUM> monitors workloads and processes in each of the compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> in the rack <NUM>.

The data management platform <NUM> includes orchestrator metrics <NUM>, node metrics <NUM> and workload metrics <NUM>. The orchestration metrics <NUM> are managed by the orchestrator/scheduler <NUM>. The workload metrics <NUM> are accessible by the resource manager <NUM>. The workload metrics <NUM> are exposed by a workload and can be queried by the resource manager <NUM> or queried by a metrics aggregator <NUM> which in turn is queried by the resource manager <NUM>. The node metrics <NUM> are exposed by a node exporter (for example, github. com/prometheus/node_exporter). The node metrics <NUM> can be queried by the resource manager <NUM> or queried by a metrics aggregator <NUM> which in turn is queried by the resource manager <NUM>.

The orchestrator/scheduler <NUM> stores basic node metrics in node metrics <NUM>. The basic node metrics include: the number of workloads per compute node; the number of processes per compute node; pressure states; CPU utilization per compute node, and memory utilization per compute node. The pressure states indicate whether a compute node <NUM> is under pressure.

A compute node <NUM> is under pressure if the compute node <NUM> is experiencing high resource utilization that is impacting the performance of a workload <NUM> running on the compute node <NUM>. Additional node metrics are monitored and stored in node metrics <NUM> to determine if a compute node <NUM> is under pressure. The additional node metrics include CPU utilization per process; memory bandwidth utilization per process; memory utilization per process; storage latency per process; storage utilization per process; storage Input/Output per second per process; GPU and/or FPGA utilization per process and GPU and/or FPGA latency per process.

The orchestrator/scheduler <NUM> also monitors and stores workload metrics <NUM>. Workload metrics <NUM> include: number of clients; average response latency and percentile metrics. Examples of percentile metrics are a 99th percentile latency or a <NUM>. 9th percentile latency, that is the maximum latency for <NUM>% or <NUM>% of workloads.

The resource manager <NUM> aggregates metrics (node metrics <NUM>, workload metrics <NUM> and orchestrator metrics (<NUM>)) to detect a pressure condition when the pressure condition occurs. The resource manager <NUM> also aggregates the metrics to detect a pressure condition before the pressure condition occurs. A pressure condition can be detected prior to occurrence of the pressure condition through the use of Time Series Analysis algorithms. Time Series Analysis algorithms include Markov Sequence/Chain algorithms or Artificial Intelligence Algorithms (for example, Neural Networks or Genetic Algorithms). In addition, the resource manager <NUM> aggregates the metrics to detect which resources (memory/disk/GPU/FPGA) are under pressure and to request the addition of more resources to one or more of the compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>.

<FIG> is a flow graph of a method to automatically add or remove logical resources in response to detection of pressure in the rack <NUM> in the Data Management Platform <NUM> shown in <FIG>.

In general pressure detection is dependent on multiple input sources. Pressure detection can be based on an event that occurs after the fact (post-factum), for example, a <NUM>th percentile, 99th percentile or <NUM>. 9th percentile latency spike (a workload metric). Pressure detection can also be based before the fact (pre-factum), that is before a <NUM>th percentile, 99th percentile or <NUM>. 9th percentile latency spike is detected based on an increase in resource utilization in the compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> and workloads <NUM>-<NUM>,. , <NUM>-<NUM>.

If pressure detection is post factum, the orchestrator metrics <NUM> and node metrics <NUM> are used to detect the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> and the resource associated with the pressure detection. If pressure detection is pre-factum, a prediction is made that there will be pressure based on an increase in resource utilization in the nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> and workloads <NUM>-<NUM>,. , <NUM>-<NUM>.

At block <NUM>, the resource manager <NUM> monitors system metrics. The monitored system metrics include orchestrator metrics <NUM>, node metrics <NUM> and workload metrics <NUM>.

At block <NUM>, a pressure condition occurs if a compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> is under stress. While monitoring the system metrics, the resource manager <NUM> can detect whether the status of a pressure condition is active or non-active. The pressure condition is active if a pressure condition is about to happen, the pressure condition is about to end or the pressure condition is in process. If an active pressure condition is detected by the resource manager <NUM>, processing continues with block <NUM>. If not, processing continues with block <NUM> to continue to monitor metrics.

At block <NUM>, an active pressure condition has been detected. An example of a pressure condition is a 99th percentile latency or a <NUM>. 9th percentile latency spike (a workload metric). The applications running on the compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> that are impacted by the active pressure condition are determined. Processing continues with block <NUM>.

At block <NUM>, the compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> that are impacted by the active pressure condition are determined.

At block <NUM>, the detected active pressure condition can be if a pressure condition is about to happen, about to end or is in process. If the pressure condition is about to happen or is in process, processing continues with block <NUM>. If the pressure condition is about to end, processing continues with block <NUM>.

At block <NUM>, the pressure condition is about to end, logical resources are removed from the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>. Processing continues with block <NUM> to continue to monitor metrics.

At block <NUM>, the pressure condition is about to happen or is in process, logical resources are added to the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>. In an embodiment, more logical resources are added to the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> that is experiencing the pressure condition. Additional logical resources can be used by all workloads <NUM> on the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> or can be restricted for usage only by specific workloads <NUM> on the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>. Logical resources that can be added to relieve pressure include storage, memory, accelerator and Field Programmable Gate Array (FPGA) resources.

A pressure condition for a disk (for example, solid-state drive <NUM> (<FIG>)) can be due to a lack of space on the disk or an increased Input/Output latency to the disk. Upon detecting a pressure condition (post-factum or pre-factum) for the disk, the Resource Manager <NUM> requests that the accelerator node <NUM> create a new volume and logically attach the newly created volume to the respective compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM><NUM>. The Resource Manager Agent <NUM> in the respective compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> performs a file system extension on the newly created volume and mounts the newly created volume directly for the running workload <NUM> on one of the compute nodes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>.

A pressure condition for memory (for example, memory <NUM> or persistent memory <NUM> (<FIG>)) can be due to high memory bandwith usage, low free memory on the compute node <NUM>-<NUM>,<NUM>-<NUM>, <NUM>-<NUM> or a memory usage spike in a workload <NUM> on the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>. The resource manager <NUM> can use the persistent memory <NUM> (<FIG>), a Simple Storage Service (S3) endpoint or a remote solid-state drive <NUM> in a storage node <NUM> to allocate a new pool of memory for the compute node <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>. The Simple Storage Service can be accessed via the network interface controller <NUM> in the compute node <NUM>-<NUM>,<NUM>-<NUM>, <NUM>-<NUM>. The Simple Storage Service uses multiple remote drives accessible via the Network Interface Controller <NUM> to present one endpoint. Simple Storage Service is an Application Programming Interface (API) that provides object storage through a web service interface. Amazon® Simple Storage Service Simple Storage Service is the de facto standard in object storage solutions. Examples of interfaces that are compatible with Amazon Simple Storage Service include Ceph RADOS Gateway, OpenIO, Scality and MinIO. In an embodiment, the Simple Storage Service is provided by MinIO. The remote solid-state drive <NUM> is in the storage node <NUM> in the same rack <NUM> as the compute node <NUM>-<NUM>,<NUM>-<NUM>, <NUM>-<NUM> and workload <NUM>.

The new pool of memory is accesible for utilization by the compute node <NUM>-<NUM>,<NUM>-<NUM>, <NUM>-<NUM> via the Resource Manager Agent <NUM>. The Resource Manager Agent <NUM> maps the new pool of memory as an extension of the memory already allocated to the compute node <NUM>-<NUM>,<NUM>-<NUM>, <NUM>-<NUM>. The new pool of memory allocated in persistent memory <NUM> is accessible directly by the workload <NUM>. The new pool of memory allocated by Simple Storage Service is exposed to the workload <NUM> via a kernel function (for example, a 'userfaultfd') function that allows on-demand paging from user space <NUM>.

The newly allocated pool of memory in the persistent memory <NUM> or remote solid-state drive <NUM> is used as a warm tier of logical memory and memory <NUM> includes volatile memory and is a cache for the warm tier of logical memory. Local memory <NUM> is a hot tier of logical memory with low latency and high bandwidth. The persistent memory <NUM> has a greater capacity than memory <NUM> and has a greater latency and lower bandwidth. The solid-state drive <NUM> has greater capacity that the persistent memory <NUM> and has a greater latency and lower bandwidth.

A pressure condition for the accelerator <NUM> or FPGA <NUM> results in the workload <NUM> being impacted by lack of sufficient resources of the accelerator <NUM> or FPGA <NUM>. A job queue for resources of the accelerator <NUM> or FPGA <NUM> that is used by one or more workloads <NUM> can fill during a pressure condition.

In an embodiment, the resource manager <NUM> detects a percentile latency increase due to a stall in CPU processing. For example, if the requested data is not in the CPU cache, the requested data must be fetched from far memory or storage. This results in variability in the average response time (that is, deviations from the mean). In response to the detection of the latency increase, the resource manager <NUM> requests a new accelerator server or a FPGA resource from the storage node <NUM> using Remote Direct Memory Access (RDMA) based communication protocols.

Examples of RDMA based communication protocols include NVMeOF (NVM Express over Fabric) or to a FPGA resource over Fabric (for example, an FPGA that is accessible using RDMA over Fabric). NVM Express over Fabrics defines a common architecture that supports a range of storage networking fabrics for NVMe block storage protocol over a storage networking fabric. This includes enabling a front-side interface into storage systems, scaling out to large numbers of NVMe devices and extending the distance within a datacenter over which NVMe devices and NVMe subsystems can be accessed.

The new logical accelerator or FPGA is connected to one of the compute nodes <NUM>-<NUM>,<NUM>-<NUM>, <NUM>-<NUM>, and used by the workload <NUM>. Processing continues with block <NUM> to continue to monitor metrics.

A shared-nothing architecture (SN) is a distributed-computing architecture in which an update request is satisfied by a single node. The node can be a compute node, a memory node or a storage node. The intent is to eliminate contention among nodes. Each node independently accesses memory and storage. Nodes do not share memory or storage.

A shared-nothing architecture system can scale by adding nodes because there is no central resource that bottlenecks the system. Another term for a shared-nothing architecture is sharding. A database shard is a horizontal partition of data in a database or search engine. Each individual partition is referred to as a shard or database shard. Each shard is stored in a separate database server instance, to spread load.

Distributed applications that are used in a system with a shared-nothing architecture need their shard stored in a database server instance to be persistent. Examples of distributed applications with a shared-nothing architecture include Structured Query Language (SQL) databases, Simple Storage Service (S3) Object Store and Time Series Databases. Structured Query Language is a domain-specific language used in programming and designed for managing data stored in a relational database management system (RDBMS), or for stream processing in a relational data stream management system (RDSMS).

Failure of a database server instance, or group of database server instances impacts the users of the database server. The failure can result in an increase in the latency of a request for data stored in a database server instance or a failed request for data stored in the database server instance. Additionally, recovery from the failure is time and resource consuming because the data associated with the failed database server instance or group of database server instances has to be restored.

Manual intervention is required to perform the restoration of the database server instances with knowledge of application topology and failure domains in which the application is deployed. In a scale-out architecture the application is composed of several processes, each running in a Kubernetes pod. These pods are distributed across fault domains, that is, racks in a Data Management Platform <NUM>, such that a failure does not impact the application's availability or the durability of the data the application is managing. The distribution of these pods is the application topology.

In an embodiment, a storage self-healing mechanism that may also referred to as storage self-healing logic or circuitry monitors a storage sub-system and monitors workloads that use the storage sub-system (storage nodes and solid-state drives) to ensure that all the workloads are spread across available failure domains.

<FIG> is a block diagram of an embodiment of a physical cluster <NUM> in the Data Management Platform <NUM> that includes a storage self-healing mechanism <NUM>. The physical cluster <NUM> includes an orchestrator/scheduler <NUM> and a rack <NUM>. In an embodiment, the storage self-healing mechanism <NUM> is in the orchestrator/scheduler <NUM>. In other embodiments, the storage self-healing mechanism may be in another component of the Data Management Platform <NUM>.

In the particular non-limiting example depicted in <FIG>, there is one rack <NUM> with one data switch <NUM>, three compute nodes 2110a-c and two storage nodes 2102a-b. The compute nodes 2110a-b and storage nodes 2102a-b are communicatively coupled to the data switch <NUM>.

The physical cluster <NUM> has a plurality of failure domains for an application that uses the storage sub-system (storage nodes 2102a-b and solid-state drives 2106a-d). A first failure domain is the data switch <NUM>, a second failure domain is a compute node 21000a-b in which a workload 2104a-c runs, a third failure domain in a storage node 2102a-b and a fourth failure domain in a solid-state drive 2106a-b.

The storage self-healing mechanism <NUM> periodically performs a health check for each of the plurality of failure domains. One of the health checks performed by the storage self-healing mechanism for the data switch <NUM> is to determine if the orchestrator/scheduler <NUM> can access the data switch <NUM>. In an embodiment Internet Control Message Protocol (ICMP) can be used to determine if the orchestrator <NUM> can access the data switch. For example, ICMP echo request/reply or extended echo request/reply messages can be used to determine if the orchestrator/scheduler <NUM> can access the data switch <NUM>. Internet Control Message Protocol is an error reporting protocol and is an extension to the Internet Protocol (IP) defined by Request for Comments (RFC) <NUM>.

Another health check performed by the storage self-healing mechanism <NUM> for the data switch <NUM> is to determine if routes are available in the data switch. In an embodiment, an "ip r g" command (a Linux utility command) checks if the routes to the compute node 2100a-c or storage node 2102a-b that is bound to the specified IP address are visible and routes to the pod are visible.

One of the health checks performed for the compute node 2100a-c or storage node 2102a-b by the storage self-healing mechanism <NUM> is to determine if the compute node 2110a-c or storage node 2102a-b is reachable via the data switch <NUM>. In an embodiment Internet Control Message Protocol (ICMP) can be used to determine if the compute node 2110a-c or storage node 2102a-b can access the data switch.

Another health check for the compute node 2110a-c or storage node 2102a-b performed by the storage self-healing mechanism <NUM> is to check if the orchestrator <NUM> reports the compute node 2110a-c or storage node 2102a-b as ready. The orchestrator <NUM> checks the health of the compute node 2110a-c or storage node 2102a-b and reports if the respective the compute node 2110a-c or storage node n is ready to accept workloads. The health of the compute node 2110a-c or storage node 2102a-b can include memory and CPU checks from the operating system perspective and network connectivity between the orchestrator and the respective compute node 2110a-c or storage node 2102a-b.

One of the health checks performed for the solid-state drive 2106a-d performed by the storage self-healing mechanism <NUM> is write amplification. For example, write amplification factors such as free user space and overprovisioning can be used to predict a failure in the solid-state drive 2106a-d. Another health check for the solid-state drive 2106a-d performed by the storage self-healing mechanism <NUM> is to check the health of the solid-state drive 2106a-d.

The health of the solid-state drive 2106a-d can be monitored using S. (Self-Monitoring, Analysis and Reporting Technology. T is a monitoring system included in solid-state drives that monitors and reports indicators of reliability of the solid-state drive that can be used to take preventative action to prevent data loss. Examples of S. T metrics for a NAND based solid-state drive 2106a-d include Program Fail Count, Erase Fail Count, Wear Leveling Count, End-to-End Error Detection Count, Cyclic Redundancy Code (CRC) Error Count, Timed Workload - Media Wear, Timed Workload - Host, Read/Write Ratio, Timed Workload Timer, Thermal Throttle Status, Retry Buffer Overflow Count, PLI Lock Loss Count, NAND Bytes Written, Host Bytes Written and System Area Life Remaining.

In a scale-out, shared nothing architecture a workload has multiple instances. A minimum number of accepted failed instances is workload specific. Simple Storage Service protects data against hardware failures and silent data corruption using erasure code and checksums.

Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. Simple Storage Service shards objects into data and parity blocks. With <NUM> data blocks and <NUM> parity blocks allows data to be recovered if there are upto <NUM> instance failures. A database having <NUM> replicas (copies of the database) allows up to <NUM> instances to recover the data.

The self-healing mechanism <NUM> can detect incoming failures, and can trigger and schedule recreation of data stored on failed solid-state drives 2106a-d. The storage self-healing mechanism <NUM> can use the metrics obtained via the health checks described earlier to trigger automatic actions for the storage subsystem (storage nodes 2102a-b and solid-state drives 2106a-d.

<FIG> illustrates an embodiment of mapping of workloads in the cluster <NUM> shown in <FIG>. The storage self-healing mechanism <NUM> tracks the mapping of workloads 2104a-c to the storage subsystem. For example, the mapping tracks the solid-state drive 2106a-d to which a logical volume is mapped.

As shown in <FIG>, workload A (instance <NUM>) is mapped to compute <NUM>, accelerator <NUM>, solid-state drive <NUM>, volume a in rack <NUM>.

In a system that includes the storage self-healing mechanism <NUM>, there is no administrator/operator involvement required for recovery of the application. Also, latency and bandwidth impact of failing clustered application instances of overall system performance is reduced.

Upon detecting a failure in the data switch <NUM> or a failure related to all of the storage nodes 2102a-b and compute nodes 2110a-c in the rack <NUM>, multiple workloads are impacted. The storage self-healing mechanism <NUM> detects the workloads that are impacted by the failure. Another rack <NUM> is selected from available racks in the physical cluster <NUM>. The storage self-healing mechanism <NUM> in the orchestrator <NUM> via the NVMe over Fabric interface disconnects the volume on the solid-state drive and removes the volume from the failed compute nodes and storage nodes.

Resources for the workload that was running on the compute node in the failed rack are created in the other rack. The workload is rescheduled to run on a compute node in the other rack. After the rescheduled workload instances have been rescheduled, the storage self-healing mechanism <NUM> in the orchestrator <NUM> triggers a workload "repair/heal" mechanism in the workload for all of the data on the volume in the other rack after the storage has been recreated in the other rack and repairs blocks upon detecting an error in the respective block.

If the storage self-healing mechanism <NUM> detects a failure in a compute node or the compute node reports a failure, the storage self-healing mechanism <NUM> detects which workloads are impacted on the failed compute node. The impacted Workload(s) are rescheduled to run on another compute node within the same rack, that does not already host an instance of the workload. The storage self-healing mechanism <NUM> in the orchestrator <NUM> requests a disconnect of the volume from the failed compute node and a connect of the volume to the other compute node. In an embodiment in which the communications path between the solid-state drive and the storage node is via NVMeOF, the requests to disconnect and connect are sent via the NVMeOF interface. After the workload instances have been restarted on the other compute node, the orchestrator <NUM> triggers a workload "repair/heal" mechanism on all of the data on the volume and repairs any blocks upon error.

If a failure in a storage node 2102a-b that impacts multiple workloads and data on the solid-state drives cannot be recovered, the storage self-healing mechanism <NUM> in the orchestrator <NUM> determines the impacted workloads and the volumes used in the failed storage node. The storage self-healing mechanism <NUM> reschedules all impacted workloads onto different compute nodes 2110a-c within the rack <NUM>, creates new volumes on a solid-state drive in another storage node 2100a-b and connects the new volumes on the solid-state drive to the new compute nodes 2110a-c via the NVMeOF interface.

If the compute nodes 2110a-c in the in rack <NUM> already host workload(s) of the same type, the orchestrator <NUM> selects another rack <NUM> in the physical cluster <NUM> and the orchestrator <NUM> reschedules all impacted workloads to run on compute nodes 2100a-c in the other rack <NUM>. The orchestrator <NUM> creates new volumes on a solid-state drive in another storage node 2100a-b in the other rack <NUM> and connects to the new compute nodes 2110a-c in the other rack <NUM> via the NVMeOF interface. After the workload instances have been restarted on the other compute node, the orchestrator <NUM> triggers a workload "repair/heal" mechanism on all the data on the volume and repairs any blocks upon error.

Upon failure of one/or multiple solid-state drives in the storage nodes 2102a-d, multiple workloads are impacted. The storage self-healing mechanism <NUM> determines the workloads that are impacted, that is, the workloads that are using logical volumes on the failed solid-state drive. The storage self-healing mechanism <NUM> creates new volumes on other operational solid-state drives within the same storage node 2102a-b or in another storage node 2102a-b within the same rack <NUM>. New volumes are connected via the NVMeOF interface to the compute nodes, old volumes are disconnected. After the workload instances have been restarted on the other compute node, the orchestrator <NUM> triggers a workload "repair/heal" mechanism on all the data on the volume and repairs any blocks upon error.

Flow diagrams as illustrated herein provide examples of sequences of various process actions. The flow diagrams can indicate operations to be executed by a software or firmware routine, as well as physical operations. In one embodiment, a flow diagram can illustrate the state of a finite state machine (FSM), which can be implemented in hardware and/or software. Although shown in a particular sequence or order, unless otherwise specified, the order of the actions can be modified. Thus, the illustrated embodiments should be understood as an example, and the process can be performed in a different order, and some actions can be performed in parallel. Additionally, one or more actions can be omitted in various embodiments; thus, not all actions are required in every embodiment. Other process flows are possible.

To the extent various operations or functions are described herein, they can be described or defined as software code, instructions, configuration, and/or data. The content can be directly executable ("object" or "executable" form), source code, or difference code ("delta" or "patch" code). The software content of the embodiments described herein can be provided via an article of manufacture with the content stored thereon, or via a method of operating a communication interface to send data via the communication interface. A machine readable storage medium can cause a machine to perform the functions or operations described, and includes any mechanism that stores information in a form accessible by a machine (e.g., computing device, electronic system, etc.), such as recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). A communication interface includes any mechanism that interfaces to any of a hardwired, wireless, optical, etc., medium to communicate to another device, such as a memory bus interface, a processor bus interface, an Internet connection, a disk controller, etc. The communication interface can be configured by providing configuration parameters and/or sending signals to prepare the communication interface to provide a data signal describing the software content. The communication interface can be accessed via one or more commands or signals sent to the communication interface.

Various components described herein can be a means for performing the operations or functions described. Each component described herein includes software, hardware, or a combination of these. The components can be implemented as software modules, hardware modules, special-purpose hardware (e.g., application specific hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), etc.), embedded controllers, hardwired circuitry, etc..

Besides what is described herein, various modifications can be made to the disclosed embodiments and implementations of the invention without departing from their scope.

Claim 1:
An apparatus comprising:
a compute server (<NUM>); and
a storage server (<NUM>) to manage a plurality of storage devices communicatively coupled to the storage server, the compute server and the storage server communicatively coupled via a network, the plurality of storage devices managed by the storage server disaggregated from the compute server to enable storage capacity of the plurality of storage devices to scale independent of the compute server; characterized by the storage server further comprising:
a network interface controller (<NUM>) communicatively coupled to the network; and
a system-on-chip (<NUM>, <NUM>), the system-on-chip comprising a plurality of cores and a last level cache memory, the plurality of cores communicatively coupled to the last level cache memory, the last level cache memory comprising a plurality of cache ways, a portion of the plurality of cache ways allocated for exclusive use by a logical volume in the plurality of storage devices and the network interface controller being configured to transfer data between the logical volume and the plurality of cache ways.