Technologies for providing advanced management of power usage limits in a disaggregated architecture

Technologies for providing advanced management of power usage limits in a disaggregated architecture include a compute device. The compute device includes circuitry configured to execute operations associated with a workload in a disaggregated system. The circuitry is also configured to determine whether a present power usage of the compute device is within a predefined range of a power usage limit assigned to the compute device. Additionally, the circuitry is configured to send, to a device in the disaggregated system and in response to a determination that the present power usage of the present compute device is not within the predefined range of the power usage limit assigned to the present compute device, offer data indicative of an offer to reduce the power usage limit assigned to the present compute device to enable a second power utilization limit of another compute device in the disaggregated system to be increased.

BACKGROUND

In some data centers in which operations are performed on behalf of customers (e.g., tenants), resources are organized in a disaggregated architecture in which sets of resources (e.g., compute devices, accelerator devices, data storage devices, etc.) are physically separate from each other (e.g., a compute device may be in a separate circuit board than an accelerator device). Typically, a service level agreement (SLA) is established between an operator of the data center and each tenant. The SLA defines a set of quality of service (QoS) targets (e.g., latency, throughput, cost, etc.) to be satisfied in the execution of operations by the resources in the disaggregated architecture. The resources use electrical power to execute the operations and produce more heat when using more power. To control the wear on the resources and thermal conditions (e.g., temperature, air flow, etc.) in the data center, the resources are subjected to hard (e.g., fixed) limits on power usage. As such, situations may arise in which a QoS target is not met because a resource is unable to utilize additional power beyond its hard limit to provide the performance needed for the QoS target. Distributed data storage systems are particularly sensitive to such scenarios, as the overall performance of a cluster (e.g., group) of data storage resources can be affected when any single resource in the cluster is hampered by a hard limit.

DETAILED DESCRIPTION

Referring now toFIG.1, a data center100in which disaggregated resources may cooperatively execute one or more workloads (e.g., applications on behalf of customers) includes multiple pods110,120,130,140, each of which includes one or more rows of racks. Of course, although data center100is shown with multiple pods, in some embodiments, the data center100may be embodied as a single pod. As described in more detail herein, each rack houses multiple sleds, each of which may be primarily equipped with a particular type of resource (e.g., memory devices, data storage devices, accelerator devices, general purpose processors), i.e., resources that can be logically coupled to form a composed node, which can act as, for example, a server. In the illustrative embodiment, the sleds in each pod110,120,130,140are connected to multiple pod switches (e.g., switches that route data communications to and from sleds within the pod). The pod switches, in turn, connect with spine switches150that switch communications among pods (e.g., the pods110,120,130,140) in the data center100. In some embodiments, the sleds may be connected with a fabric using Intel Omni-Path technology. In other embodiments, the sleds may be connected with other fabrics, such as InfiniBand or Ethernet. As described in more detail herein, resources within sleds in the data center100may be allocated to a group (referred to herein as a “managed node”) containing resources from one or more sleds to be collectively utilized in the execution of a workload. The workload can execute as if the resources belonging to the managed node were located on the same sled. The resources in a managed node may belong to sleds belonging to different racks, and even to different pods110,120,130,140. As such, some resources of a single sled may be allocated to one managed node while other resources of the same sled are allocated to a different managed node (e.g., one processor assigned to one managed node and another processor of the same sled assigned to a different managed node).

A data center comprising disaggregated resources, such as data center100, can be used in a wide variety of contexts, such as enterprise, government, cloud service provider, and communications service provider (e.g., Telco's), as well in a wide variety of sizes, from cloud service provider mega-data centers that consume over 100,000 sq. ft. to single- or multi-rack installations for use in base stations.

The disaggregation of resources to sleds comprised predominantly of a single type of resource (e.g., compute sleds comprising primarily compute resources, memory sleds containing primarily memory resources), and the selective allocation and deallocation of the disaggregated resources to form a managed node assigned to execute a workload improves the operation and resource usage of the data center100relative to typical data centers comprised of hyperconverged servers containing compute, memory, storage and perhaps additional resources in a single chassis. For example, because sleds predominantly contain resources of a particular type, resources of a given type can be upgraded independently of other resources. Additionally, because different resources types (processors, storage, accelerators, etc.) typically have different refresh rates, greater resource utilization and reduced total cost of ownership may be achieved. For example, a data center operator can upgrade the processors throughout their facility by only swapping out the compute sleds. In such a case, accelerator and storage resources may not be contemporaneously upgraded and, rather, may be allowed to continue operating until those resources are scheduled for their own refresh. Resource utilization may also increase. For example, if managed nodes are composed based on requirements of the workloads that will be running on them, resources within a node are more likely to be fully utilized. Such utilization may allow for more managed nodes to run in a data center with a given set of resources, or for a data center expected to run a given set of workloads, to be built using fewer resources.

Referring now toFIG.2, the pod110, in the illustrative embodiment, includes a set of rows200,210,220,230of racks240. Each rack240may house multiple sleds (e.g., sixteen sleds) and provide power and data connections to the housed sleds, as described in more detail herein. In the illustrative embodiment, the racks in each row200,210,220,230are connected to multiple pod switches250,260. The pod switch250includes a set of ports252to which the sleds of the racks of the pod110are connected and another set of ports254that connect the pod110to the spine switches150to provide connectivity to other pods in the data center100. Similarly, the pod switch260includes a set of ports262to which the sleds of the racks of the pod110are connected and a set of ports264that connect the pod110to the spine switches150. As such, the use of the pair of switches250,260provides an amount of redundancy to the pod110. For example, if either of the switches250,260fails, the sleds in the pod110may still maintain data communication with the remainder of the data center100(e.g., sleds of other pods) through the other switch250,260. Furthermore, in the illustrative embodiment, the switches150,250,260may be embodied as dual-mode optical switches, capable of routing both Ethernet protocol communications carrying Internet Protocol (IP) packets and communications according to a second, high-performance link-layer protocol (e.g., PCI Express) via optical signaling media of an optical fabric.

It should be appreciated that each of the other pods120,130,140(as well as any additional pods of the data center100) may be similarly structured as, and have components similar to, the pod110shown in and described in regard toFIG.2(e.g., each pod may have rows of racks housing multiple sleds as described above). Additionally, while two pod switches250,260are shown, it should be understood that in other embodiments, each pod110,120,130,140may be connected to a different number of pod switches, providing even more failover capacity. Of course, in other embodiments, pods may be arranged differently than the rows-of-racks configuration shown inFIGS.1-2. For example, a pod may be embodied as multiple sets of racks in which each set of racks is arranged radially, i.e., the racks are equidistant from a center switch.

Referring now toFIGS.3-5, each illustrative rack240of the data center100includes two elongated support posts302,304, which are arranged vertically. For example, the elongated support posts302,304may extend upwardly from a floor of the data center100when deployed. The rack240also includes one or more horizontal pairs310of elongated support arms312(identified inFIG.3via a dashed ellipse) configured to support a sled of the data center100as discussed below. One elongated support arm312of the pair of elongated support arms312extends outwardly from the elongated support post302and the other elongated support arm312extends outwardly from the elongated support post304.

In the illustrative embodiments, each sled of the data center100is embodied as a chassis-less sled. That is, each sled has a chassis-less circuit board substrate on which physical resources (e.g., processors, memory, accelerators, storage, etc.) are mounted as discussed in more detail below. As such, the rack240is configured to receive the chassis-less sleds. For example, each pair310of elongated support arms312defines a sled slot320of the rack240, which is configured to receive a corresponding chassis-less sled. To do so, each illustrative elongated support arm312includes a circuit board guide330configured to receive the chassis-less circuit board substrate of the sled. Each circuit board guide330is secured to, or otherwise mounted to, a top side332of the corresponding elongated support arm312. For example, in the illustrative embodiment, each circuit board guide330is mounted at a distal end of the corresponding elongated support arm312relative to the corresponding elongated support post302,304. For clarity of the Figures, not every circuit board guide330may be referenced in each Figure.

Each circuit board guide330includes an inner wall that defines a circuit board slot380configured to receive the chassis-less circuit board substrate of a sled400when the sled400is received in the corresponding sled slot320of the rack240. To do so, as shown inFIG.4, a user (or robot) aligns the chassis-less circuit board substrate of an illustrative chassis-less sled400to a sled slot320. The user, or robot, may then slide the chassis-less circuit board substrate forward into the sled slot320such that each side edge414of the chassis-less circuit board substrate is received in a corresponding circuit board slot380of the circuit board guides330of the pair310of elongated support arms312that define the corresponding sled slot320as shown inFIG.4. By having robotically accessible and robotically manipulable sleds comprising disaggregated resources, each type of resource can be upgraded independently of each other and at their own optimized refresh rate. Furthermore, the sleds are configured to blindly mate with power and data communication cables in each rack240, enhancing their ability to be quickly removed, upgraded, reinstalled, and/or replaced. As such, in some embodiments, the data center100may operate (e.g., execute workloads, undergo maintenance and/or upgrades, etc.) without human involvement on the data center floor. In other embodiments, a human may facilitate one or more maintenance or upgrade operations in the data center100.

It should be appreciated that each circuit board guide330is dual sided. That is, each circuit board guide330includes an inner wall that defines a circuit board slot380on each side of the circuit board guide330. In this way, each circuit board guide330can support a chassis-less circuit board substrate on either side. As such, a single additional elongated support post may be added to the rack240to turn the rack240into a two-rack solution that can hold twice as many sled slots320as shown inFIG.3. The illustrative rack240includes seven pairs310of elongated support arms312that define a corresponding seven sled slots320, each configured to receive and support a corresponding sled400as discussed above. Of course, in other embodiments, the rack240may include additional or fewer pairs310of elongated support arms312(i.e., additional or fewer sled slots320). It should be appreciated that because the sled400is chassis-less, the sled400may have an overall height that is different than typical servers. As such, in some embodiments, the height of each sled slot320may be shorter than the height of a typical server (e.g., shorter than a single rank unit, “1 U”). That is, the vertical distance between each pair310of elongated support arms312may be less than a standard rack unit “1 U.” Additionally, due to the relative decrease in height of the sled slots320, the overall height of the rack240in some embodiments may be shorter than the height of traditional rack enclosures. For example, in some embodiments, each of the elongated support posts302,304may have a length of six feet or less. Again, in other embodiments, the rack240may have different dimensions. For example, in some embodiments, the vertical distance between each pair310of elongated support arms312may be greater than a standard rack until “1 U”. In such embodiments, the increased vertical distance between the sleds allows for larger heat sinks to be attached to the physical resources and for larger fans to be used (e.g., in the fan array370described below) for cooling each sled, which in turn can allow the physical resources to operate at increased power levels. Further, it should be appreciated that the rack240does not include any walls, enclosures, or the like. Rather, the rack240is an enclosure-less rack that is opened to the local environment. Of course, in some cases, an end plate may be attached to one of the elongated support posts302,304in those situations in which the rack240forms an end-of-row rack in the data center100.

In some embodiments, various interconnects may be routed upwardly or downwardly through the elongated support posts302,304. To facilitate such routing, each elongated support post302,304includes an inner wall that defines an inner chamber in which interconnects may be located. The interconnects routed through the elongated support posts302,304may be embodied as any type of interconnects including, but not limited to, data or communication interconnects to provide communication connections to each sled slot320, power interconnects to provide power to each sled slot320, and/or other types of interconnects.

The rack240, in the illustrative embodiment, includes a support platform on which a corresponding optical data connector (not shown) is mounted. Each optical data connector is associated with a corresponding sled slot320and is configured to mate with an optical data connector of a corresponding sled400when the sled400is received in the corresponding sled slot320. In some embodiments, optical connections between components (e.g., sleds, racks, and switches) in the data center100are made with a blind mate optical connection. For example, a door on each cable may prevent dust from contaminating the fiber inside the cable. In the process of connecting to a blind mate optical connector mechanism, the door is pushed open when the end of the cable approaches or enters the connector mechanism. Subsequently, the optical fiber inside the cable may enter a gel within the connector mechanism and the optical fiber of one cable comes into contact with the optical fiber of another cable within the gel inside the connector mechanism.

The illustrative rack240also includes a fan array370coupled to the cross-support arms of the rack240. The fan array370includes one or more rows of cooling fans372, which are aligned in a horizontal line between the elongated support posts302,304. In the illustrative embodiment, the fan array370includes a row of cooling fans372for each sled slot320of the rack240. As discussed above, each sled400does not include any on-board cooling system in the illustrative embodiment and, as such, the fan array370provides cooling for each sled400received in the rack240. Each rack240, in the illustrative embodiment, also includes a power supply associated with each sled slot320. Each power supply is secured to one of the elongated support arms312of the pair310of elongated support arms312that define the corresponding sled slot320. For example, the rack240may include a power supply coupled or secured to each elongated support arm312extending from the elongated support post302. Each power supply includes a power connector configured to mate with a power connector of the sled400when the sled400is received in the corresponding sled slot320. In the illustrative embodiment, the sled400does not include any on-board power supply and, as such, the power supplies provided in the rack240supply power to corresponding sleds400when mounted to the rack240. Each power supply is configured to satisfy the power requirements for its associated sled, which can vary from sled to sled. Additionally, the power supplies provided in the rack240can operate independent of each other. That is, within a single rack, a first power supply providing power to a compute sled can provide power levels that are different than power levels supplied by a second power supply providing power to an accelerator sled. The power supplies may be controllable at the sled level or rack level, and may be controlled locally by components on the associated sled or remotely, such as by another sled or an orchestrator.

Referring now toFIG.6, the sled400, in the illustrative embodiment, is configured to be mounted in a corresponding rack240of the data center100as discussed above. In some embodiments, each sled400may be optimized or otherwise configured for performing particular tasks, such as compute tasks, acceleration tasks, data storage tasks, etc. For example, the sled400may be embodied as a compute sled800as discussed below in regard toFIGS.8-9, an accelerator sled1000as discussed below in regard toFIGS.10-11, a storage sled1200as discussed below in regard toFIGS.12-13, or as a sled optimized or otherwise configured to perform other specialized tasks, such as a memory sled1400, discussed below in regard toFIG.14.

As discussed above, the illustrative sled400includes a chassis-less circuit board substrate602, which supports various physical resources (e.g., electrical components) mounted thereon. It should be appreciated that the circuit board substrate602is “chassis-less” in that the sled400does not include a housing or enclosure. Rather, the chassis-less circuit board substrate602is open to the local environment. The chassis-less circuit board substrate602may be formed from any material capable of supporting the various electrical components mounted thereon. For example, in an illustrative embodiment, the chassis-less circuit board substrate602is formed from an FR-4 glass-reinforced epoxy laminate material. Of course, other materials may be used to form the chassis-less circuit board substrate602in other embodiments.

As discussed in more detail below, the chassis-less circuit board substrate602includes multiple features that improve the thermal cooling characteristics of the various electrical components mounted on the chassis-less circuit board substrate602. As discussed, the chassis-less circuit board substrate602does not include a housing or enclosure, which may improve the airflow over the electrical components of the sled400by reducing those structures that may inhibit air flow. For example, because the chassis-less circuit board substrate602is not positioned in an individual housing or enclosure, there is no vertically-arranged backplane (e.g., a backplate of the chassis) attached to the chassis-less circuit board substrate602, which could inhibit air flow across the electrical components. Additionally, the chassis-less circuit board substrate602has a geometric shape configured to reduce the length of the airflow path across the electrical components mounted to the chassis-less circuit board substrate602. For example, the illustrative chassis-less circuit board substrate602has a width604that is greater than a depth606of the chassis-less circuit board substrate602. In one particular embodiment, for example, the chassis-less circuit board substrate602has a width of about 21 inches and a depth of about 9 inches, compared to a typical server that has a width of about 17 inches and a depth of about 39 inches. As such, an airflow path608that extends from a front edge610of the chassis-less circuit board substrate602toward a rear edge612has a shorter distance relative to typical servers, which may improve the thermal cooling characteristics of the sled400. Furthermore, although not illustrated inFIG.6, the various physical resources mounted to the chassis-less circuit board substrate602are mounted in corresponding locations such that no two substantively heat-producing electrical components shadow each other as discussed in more detail below. That is, no two electrical components, which produce appreciable heat during operation (i.e., greater than a nominal heat sufficient enough to adversely impact the cooling of another electrical component), are mounted to the chassis-less circuit board substrate602linearly in-line with each other along the direction of the airflow path608(i.e., along a direction extending from the front edge610toward the rear edge612of the chassis-less circuit board substrate602).

As discussed above, the illustrative sled400includes one or more physical resources620mounted to a top side650of the chassis-less circuit board substrate602. Although two physical resources620are shown inFIG.6, it should be appreciated that the sled400may include one, two, or more physical resources620in other embodiments. The physical resources620may be embodied as any type of processor, controller, or other compute circuit capable of performing various tasks such as compute functions and/or controlling the functions of the sled400depending on, for example, the type or intended functionality of the sled400. For example, as discussed in more detail below, the physical resources620may be embodied as high-performance processors in embodiments in which the sled400is embodied as a compute sled, as accelerator co-processors or circuits in embodiments in which the sled400is embodied as an accelerator sled, storage controllers in embodiments in which the sled400is embodied as a storage sled, or a set of memory devices in embodiments in which the sled400is embodied as a memory sled.

The sled400also includes one or more additional physical resources630mounted to the top side650of the chassis-less circuit board substrate602. In the illustrative embodiment, the additional physical resources include a network interface controller (NIC) as discussed in more detail below. Of course, depending on the type and functionality of the sled400, the physical resources630may include additional or other electrical components, circuits, and/or devices in other embodiments.

The physical resources620are communicatively coupled to the physical resources630via an input/output (I/O) subsystem622. The I/O subsystem622may be embodied as circuitry and/or components to facilitate input/output operations with the physical resources620, the physical resources630, and/or other components of the sled400. For example, the I/O subsystem622may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, integrated sensor hubs, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, waveguides, light guides, printed circuit board traces, etc.), and/or other components and subsystems to facilitate the input/output operations. In the illustrative embodiment, the I/O subsystem622is embodied as, or otherwise includes, a double data rate 4 (DDR4) data bus or a DDR5 data bus, as described further below.

In some embodiments, the sled400may also include a resource-to-resource interconnect624. The resource-to-resource interconnect624may be embodied as any type of communication interconnect capable of facilitating resource-to-resource communications. In the illustrative embodiment, the resource-to-resource interconnect624is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem622). For example, the resource-to-resource interconnect624may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to resource-to-resource communications.

The sled400also includes a power connector640configured to mate with a corresponding power connector of the rack240when the sled400is mounted in the corresponding rack240. The sled400receives power from a power supply of the rack240via the power connector640to supply power to the various electrical components of the sled400. That is, the sled400does not include any local power supply (i.e., an on-board power supply) to provide power to the electrical components of the sled400. The exclusion of a local or on-board power supply facilitates the reduction in the overall footprint of the chassis-less circuit board substrate602, which may increase the thermal cooling characteristics of the various electrical components mounted on the chassis-less circuit board substrate602as discussed above. In some embodiments, voltage regulators are placed on a bottom side750(seeFIG.7) of the chassis-less circuit board substrate602directly opposite of the processors820(seeFIG.8), and power is routed from the voltage regulators to the processors820by vias extending through the circuit board substrate602. Such a configuration provides an increased thermal budget, additional current and/or voltage, and better voltage control relative to typical printed circuit boards in which processor power is delivered from a voltage regulator, in part, by printed circuit traces.

In some embodiments, the sled400may also include mounting features642configured to mate with a mounting arm, or other structure, of a robot to facilitate the placement of the sled600in a rack240by the robot. The mounting features642may be embodied as any type of physical structures that allow the robot to grasp the sled400without damaging the chassis-less circuit board substrate602or the electrical components mounted thereto. For example, in some embodiments, the mounting features642may be embodied as non-conductive pads attached to the chassis-less circuit board substrate602. In other embodiments, the mounting features may be embodied as brackets, braces, or other similar structures attached to the chassis-less circuit board substrate602. The particular number, shape, size, and/or make-up of the mounting feature642may depend on the design of the robot configured to manage the sled400.

Referring now toFIG.7, in addition to the physical resources630mounted on the top side650of the chassis-less circuit board substrate602, the sled400also includes one or more memory devices720mounted to a bottom side750of the chassis-less circuit board substrate602. That is, the chassis-less circuit board substrate602is embodied as a double-sided circuit board. The physical resources620are communicatively coupled to the memory devices720via the I/O subsystem622. For example, the physical resources620and the memory devices720may be communicatively coupled by one or more vias extending through the chassis-less circuit board substrate602. Each physical resource620may be communicatively coupled to a different set of one or more memory devices720in some embodiments. Alternatively, in other embodiments, each physical resource620may be communicatively coupled to each memory device720.

The memory devices720may be embodied as any type of memory device capable of storing data for the physical resources620during operation of the sled400, such as any type of volatile (e.g., dynamic random access memory (DRAM), etc.) or non-volatile memory. Volatile memory may be a memory that requires power to maintain the state of data stored by the medium. Non-limiting examples of volatile memory may include various types of random access memory (RAM), such as dynamic random access memory (DRAM) or static random access memory (SRAM). One particular type of DRAM that may be used in a memory module is synchronous dynamic random access memory (SDRAM). In particular embodiments, DRAM of a memory component may comply with a standard promulgated by the Joint Electronic Device Engineering Council (JEDEC), such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209 for Low Power DDR (LPDDR), JESD209-2 for LPDDR2, JESD209-3 for LPDDR3, and JESD209-4 for LPDDR4. Such standards (and similar standards) may be referred to as DDR-based standards and communication interfaces of the memory devices that implement such standards may be referred to as DDR-based interfaces.

In one embodiment, the memory device is a block addressable memory device, such as those based on NAND or NOR technologies, such as multi-threshold level NAND flash memory or NOR flash memory. A memory device may also include byte-addressable write-in-place nonvolatile memory devices, such as Intel 3D XPoint™ memory, Micron QuantX™ memory, magnetoresistive random access memory (MRAM) memory that incorporates memristor technology, or other byte addressable write-in-place nonvolatile memory devices. In some embodiments, the memory device may comprise a transistor-less stackable cross point architecture in which memory cells sit at the intersection of word lines and bit lines and are individually addressable and in which bit storage is based on a change in bulk resistance. In one embodiment, the memory device may be or may include memory devices that use chalcogenide glass, single or multi-level Phase Change Memory (PCM), a resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), anti-ferroelectric memory, magnetoresistive random access memory (MRAM) memory that incorporates memristor technology, resistive memory including the metal oxide base, the oxygen vacancy base and the conductive bridge Random Access Memory (CB-RAM), or spin transfer torque (STT)-MRAM, a spintronic magnetic junction memory based device, a magnetic tunneling junction (MTJ) based device, a DW (Domain Wall) and SOT (Spin Orbit Transfer) based device, a thyristor based memory device, or a combination of any of the above, or other memory. The memory device may refer to the die itself and/or to a packaged memory product.

Referring now toFIG.8, in some embodiments, the sled400may be embodied as a compute sled800. The compute sled800is optimized, or otherwise configured, to perform compute tasks. Of course, as discussed above, the compute sled800may rely on other sleds, such as acceleration sleds and/or storage sleds, to perform such compute tasks. The compute sled800includes various physical resources (e.g., electrical components) similar to the physical resources of the sled400, which have been identified inFIG.8using the same reference numbers. The description of such components provided above in regard toFIGS.6and7applies to the corresponding components of the compute sled800and is not repeated herein for clarity of the description of the compute sled800.

In the illustrative compute sled800, the physical resources620are embodied as processors820. Although only two processors820are shown inFIG.8, it should be appreciated that the compute sled800may include additional processors820in other embodiments. Illustratively, the processors820are embodied as high-performance processors820and may be configured to operate at a relatively high power rating. Although the processors820generate additional heat operating at power ratings greater than typical processors (which operate at around 155-230 W), the enhanced thermal cooling characteristics of the chassis-less circuit board substrate602discussed above facilitate the higher power operation. For example, in the illustrative embodiment, the processors820are configured to operate at a power rating of at least 250 W. In some embodiments, the processors820may be configured to operate at a power rating of at least 350 W.

In some embodiments, the compute sled800may also include a processor-to-processor interconnect842. Similar to the resource-to-resource interconnect624of the sled400discussed above, the processor-to-processor interconnect842may be embodied as any type of communication interconnect capable of facilitating processor-to-processor interconnect842communications. In the illustrative embodiment, the processor-to-processor interconnect842is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem622). For example, the processor-to-processor interconnect842may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications.

The compute sled800also includes a communication circuit830. The illustrative communication circuit830includes a network interface controller (NIC)832, which may also be referred to as a host fabric interface (HFI). The NIC832may be embodied as, or otherwise include, any type of integrated circuit, discrete circuits, controller chips, chipsets, add-in-boards, daughtercards, network interface cards, or other devices that may be used by the compute sled800to connect with another compute device (e.g., with other sleds400). In some embodiments, the NIC832may be embodied as part of a system-on-a-chip (SoC) that includes one or more processors, or included on a multichip package that also contains one or more processors. In some embodiments, the NIC832may include a local processor (not shown) and/or a local memory (not shown) that are both local to the NIC832. In such embodiments, the local processor of the NIC832may be capable of performing one or more of the functions of the processors820. Additionally or alternatively, in such embodiments, the local memory of the NIC832may be integrated into one or more components of the compute sled at the board level, socket level, chip level, and/or other levels.

The communication circuit830is communicatively coupled to an optical data connector834. The optical data connector834is configured to mate with a corresponding optical data connector of the rack240when the compute sled800is mounted in the rack240. Illustratively, the optical data connector834includes a plurality of optical fibers which lead from a mating surface of the optical data connector834to an optical transceiver836. The optical transceiver836is configured to convert incoming optical signals from the rack-side optical data connector to electrical signals and to convert electrical signals to outgoing optical signals to the rack-side optical data connector. Although shown as forming part of the optical data connector834in the illustrative embodiment, the optical transceiver836may form a portion of the communication circuit830in other embodiments.

In some embodiments, the compute sled800may also include an expansion connector840. In such embodiments, the expansion connector840is configured to mate with a corresponding connector of an expansion chassis-less circuit board substrate to provide additional physical resources to the compute sled800. The additional physical resources may be used, for example, by the processors820during operation of the compute sled800. The expansion chassis-less circuit board substrate may be substantially similar to the chassis-less circuit board substrate602discussed above and may include various electrical components mounted thereto. The particular electrical components mounted to the expansion chassis-less circuit board substrate may depend on the intended functionality of the expansion chassis-less circuit board substrate. For example, the expansion chassis-less circuit board substrate may provide additional compute resources, memory resources, and/or storage resources. As such, the additional physical resources of the expansion chassis-less circuit board substrate may include, but is not limited to, processors, memory devices, storage devices, and/or accelerator circuits including, for example, field programmable gate arrays (FPGA), application-specific integrated circuits (ASICs), security co-processors, graphics processing units (GPUs), machine learning circuits, or other specialized processors, controllers, devices, and/or circuits.

Referring now toFIG.9, an illustrative embodiment of the compute sled800is shown. As shown, the processors820, communication circuit830, and optical data connector834are mounted to the top side650of the chassis-less circuit board substrate602. Any suitable attachment or mounting technology may be used to mount the physical resources of the compute sled800to the chassis-less circuit board substrate602. For example, the various physical resources may be mounted in corresponding sockets (e.g., a processor socket), holders, or brackets. In some cases, some of the electrical components may be directly mounted to the chassis-less circuit board substrate602via soldering or similar techniques.

As discussed above, the individual processors820and communication circuit830are mounted to the top side650of the chassis-less circuit board substrate602such that no two heat-producing, electrical components shadow each other. In the illustrative embodiment, the processors820and communication circuit830are mounted in corresponding locations on the top side650of the chassis-less circuit board substrate602such that no two of those physical resources are linearly in-line with others along the direction of the airflow path608. It should be appreciated that, although the optical data connector834is in-line with the communication circuit830, the optical data connector834produces no or nominal heat during operation.

The memory devices720of the compute sled800are mounted to the bottom side750of the of the chassis-less circuit board substrate602as discussed above in regard to the sled400. Although mounted to the bottom side750, the memory devices720are communicatively coupled to the processors820located on the top side650via the I/O subsystem622. Because the chassis-less circuit board substrate602is embodied as a double-sided circuit board, the memory devices720and the processors820may be communicatively coupled by one or more vias, connectors, or other mechanisms extending through the chassis-less circuit board substrate602. Of course, each processor820may be communicatively coupled to a different set of one or more memory devices720in some embodiments. Alternatively, in other embodiments, each processor820may be communicatively coupled to each memory device720. In some embodiments, the memory devices720may be mounted to one or more memory mezzanines on the bottom side of the chassis-less circuit board substrate602and may interconnect with a corresponding processor820through a ball-grid array.

Each of the processors820includes a heat sink850secured thereto. Due to the mounting of the memory devices720to the bottom side750of the chassis-less circuit board substrate602(as well as the vertical spacing of the sleds400in the corresponding rack240), the top side650of the chassis-less circuit board substrate602includes additional “free” area or space that facilitates the use of heat sinks850having a larger size relative to traditional heat sinks used in typical servers. Additionally, due to the improved thermal cooling characteristics of the chassis-less circuit board substrate602, none of the processor heat sinks850include cooling fans attached thereto. That is, each of the heat sinks850is embodied as a fan-less heat sink. In some embodiments, the heat sinks850mounted atop the processors820may overlap with the heat sink attached to the communication circuit830in the direction of the airflow path608due to their increased size, as illustratively suggested byFIG.9.

Referring now toFIG.10, in some embodiments, the sled400may be embodied as an accelerator sled1000. The accelerator sled1000is configured, to perform specialized compute tasks, such as machine learning, encryption, hashing, or other computational-intensive task. In some embodiments, for example, a compute sled800may offload tasks to the accelerator sled1000during operation. The accelerator sled1000includes various components similar to components of the sled400and/or compute sled800, which have been identified inFIG.10using the same reference numbers. The description of such components provided above in regard toFIGS.6,7, and8apply to the corresponding components of the accelerator sled1000and is not repeated herein for clarity of the description of the accelerator sled1000.

In the illustrative accelerator sled1000, the physical resources620are embodied as accelerator circuits1020. Although only two accelerator circuits1020are shown inFIG.10, it should be appreciated that the accelerator sled1000may include additional accelerator circuits1020in other embodiments. For example, as shown inFIG.11, the accelerator sled1000may include four accelerator circuits1020in some embodiments. The accelerator circuits1020may be embodied as any type of processor, co-processor, compute circuit, or other device capable of performing compute or processing operations. For example, the accelerator circuits1020may be embodied as, for example, field programmable gate arrays (FPGA), application-specific integrated circuits (ASICs), security co-processors, graphics processing units (GPUs), neuromorphic processor units, quantum computers, machine learning circuits, or other specialized processors, controllers, devices, and/or circuits.

In some embodiments, the accelerator sled1000may also include an accelerator-to-accelerator interconnect1042. Similar to the resource-to-resource interconnect624of the sled600discussed above, the accelerator-to-accelerator interconnect1042may be embodied as any type of communication interconnect capable of facilitating accelerator-to-accelerator communications. In the illustrative embodiment, the accelerator-to-accelerator interconnect1042is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem622). For example, the accelerator-to-accelerator interconnect1042may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications. In some embodiments, the accelerator circuits1020may be daisy-chained with a primary accelerator circuit1020connected to the NIC832and memory720through the I/O subsystem622and a secondary accelerator circuit1020connected to the NIC832and memory720through a primary accelerator circuit1020.

Referring now toFIG.11, an illustrative embodiment of the accelerator sled1000is shown. As discussed above, the accelerator circuits1020, communication circuit830, and optical data connector834are mounted to the top side650of the chassis-less circuit board substrate602. Again, the individual accelerator circuits1020and communication circuit830are mounted to the top side650of the chassis-less circuit board substrate602such that no two heat-producing, electrical components shadow each other as discussed above. The memory devices720of the accelerator sled1000are mounted to the bottom side750of the of the chassis-less circuit board substrate602as discussed above in regard to the sled600. Although mounted to the bottom side750, the memory devices720are communicatively coupled to the accelerator circuits1020located on the top side650via the I/O subsystem622(e.g., through vias). Further, each of the accelerator circuits1020may include a heat sink1070that is larger than a traditional heat sink used in a server. As discussed above with reference to the heat sinks870, the heat sinks1070may be larger than traditional heat sinks because of the “free” area provided by the memory resources720being located on the bottom side750of the chassis-less circuit board substrate602rather than on the top side650.

Referring now toFIG.12, in some embodiments, the sled400may be embodied as a storage sled1200. The storage sled1200is configured, to store data in a data storage1250local to the storage sled1200. For example, during operation, a compute sled800or an accelerator sled1000may store and retrieve data from the data storage1250of the storage sled1200. The storage sled1200includes various components similar to components of the sled400and/or the compute sled800, which have been identified inFIG.12using the same reference numbers. The description of such components provided above in regard toFIGS.6,7, and8apply to the corresponding components of the storage sled1200and is not repeated herein for clarity of the description of the storage sled1200.

In the illustrative storage sled1200, the physical resources620are embodied as storage controllers1220. Although only two storage controllers1220are shown inFIG.12, it should be appreciated that the storage sled1200may include additional storage controllers1220in other embodiments. The storage controllers1220may be embodied as any type of processor, controller, or control circuit capable of controlling the storage and retrieval of data into the data storage1250based on requests received via the communication circuit830. In the illustrative embodiment, the storage controllers1220are embodied as relatively low-power processors or controllers. For example, in some embodiments, the storage controllers1220may be configured to operate at a power rating of about 75 watts.

In some embodiments, the storage sled1200may also include a controller-to-controller interconnect1242. Similar to the resource-to-resource interconnect624of the sled400discussed above, the controller-to-controller interconnect1242may be embodied as any type of communication interconnect capable of facilitating controller-to-controller communications. In the illustrative embodiment, the controller-to-controller interconnect1242is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem622). For example, the controller-to-controller interconnect1242may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications.

Referring now toFIG.13, an illustrative embodiment of the storage sled1200is shown. In the illustrative embodiment, the data storage1250is embodied as, or otherwise includes, a storage cage1252configured to house one or more solid state drives (SSDs)1254. To do so, the storage cage1252includes a number of mounting slots1256, each of which is configured to receive a corresponding solid state drive1254. Each of the mounting slots1256includes a number of drive guides1258that cooperate to define an access opening1260of the corresponding mounting slot1256. The storage cage1252is secured to the chassis-less circuit board substrate602such that the access openings face away from (i.e., toward the front of) the chassis-less circuit board substrate602. As such, solid state drives1254are accessible while the storage sled1200is mounted in a corresponding rack204. For example, a solid state drive1254may be swapped out of a rack240(e.g., via a robot) while the storage sled1200remains mounted in the corresponding rack240.

The storage cage1252illustratively includes sixteen mounting slots1256and is capable of mounting and storing sixteen solid state drives1254. Of course, the storage cage1252may be configured to store additional or fewer solid state drives1254in other embodiments. Additionally, in the illustrative embodiment, the solid state drivers are mounted vertically in the storage cage1252, but may be mounted in the storage cage1252in a different orientation in other embodiments. Each solid state drive1254may be embodied as any type of data storage device capable of storing long term data. To do so, the solid state drives1254may include volatile and non-volatile memory devices discussed above.

As shown inFIG.13, the storage controllers1220, the communication circuit830, and the optical data connector834are illustratively mounted to the top side650of the chassis-less circuit board substrate602. Again, as discussed above, any suitable attachment or mounting technology may be used to mount the electrical components of the storage sled1200to the chassis-less circuit board substrate602including, for example, sockets (e.g., a processor socket), holders, brackets, soldered connections, and/or other mounting or securing techniques.

As discussed above, the individual storage controllers1220and the communication circuit830are mounted to the top side650of the chassis-less circuit board substrate602such that no two heat-producing, electrical components shadow each other. For example, the storage controllers1220and the communication circuit830are mounted in corresponding locations on the top side650of the chassis-less circuit board substrate602such that no two of those electrical components are linearly in-line with each other along the direction of the airflow path608.

The memory devices720of the storage sled1200are mounted to the bottom side750of the of the chassis-less circuit board substrate602as discussed above in regard to the sled400. Although mounted to the bottom side750, the memory devices720are communicatively coupled to the storage controllers1220located on the top side650via the I/O subsystem622. Again, because the chassis-less circuit board substrate602is embodied as a double-sided circuit board, the memory devices720and the storage controllers1220may be communicatively coupled by one or more vias, connectors, or other mechanisms extending through the chassis-less circuit board substrate602. Each of the storage controllers1220includes a heat sink1270secured thereto. As discussed above, due to the improved thermal cooling characteristics of the chassis-less circuit board substrate602of the storage sled1200, none of the heat sinks1270include cooling fans attached thereto. That is, each of the heat sinks1270is embodied as a fan-less heat sink.

Referring now toFIG.14, in some embodiments, the sled400may be embodied as a memory sled1400. The memory sled1400is optimized, or otherwise configured, to provide other sleds400(e.g., compute sleds800, accelerator sleds1000, etc.) with access to a pool of memory (e.g., in two or more sets1430,1432of memory devices720) local to the memory sled1400. For example, during operation, a compute sled800or an accelerator sled1000may remotely write to and/or read from one or more of the memory sets1430,1432of the memory sled1400using a logical address space that maps to physical addresses in the memory sets1430,1432. The memory sled1400includes various components similar to components of the sled400and/or the compute sled800, which have been identified inFIG.14using the same reference numbers. The description of such components provided above in regard toFIGS.6,7, and8apply to the corresponding components of the memory sled1400and is not repeated herein for clarity of the description of the memory sled1400.

In the illustrative memory sled1400, the physical resources620are embodied as memory controllers1420. Although only two memory controllers1420are shown inFIG.14, it should be appreciated that the memory sled1400may include additional memory controllers1420in other embodiments. The memory controllers1420may be embodied as any type of processor, controller, or control circuit capable of controlling the writing and reading of data into the memory sets1430,1432based on requests received via the communication circuit830. In the illustrative embodiment, each memory controller1420is connected to a corresponding memory set1430,1432to write to and read from memory devices720within the corresponding memory set1430,1432and enforce any permissions (e.g., read, write, etc.) associated with sled400that has sent a request to the memory sled1400to perform a memory access operation (e.g., read or write).

In some embodiments, the memory sled1400may also include a controller-to-controller interconnect1442. Similar to the resource-to-resource interconnect624of the sled400discussed above, the controller-to-controller interconnect1442may be embodied as any type of communication interconnect capable of facilitating controller-to-controller communications. In the illustrative embodiment, the controller-to-controller interconnect1442is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem622). For example, the controller-to-controller interconnect1442may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications. As such, in some embodiments, a memory controller1420may access, through the controller-to-controller interconnect1442, memory that is within the memory set1432associated with another memory controller1420. In some embodiments, a scalable memory controller is made of multiple smaller memory controllers, referred to herein as “chiplets”, on a memory sled (e.g., the memory sled1400). The chiplets may be interconnected (e.g., using EMIB (Embedded Multi-Die Interconnect Bridge)). The combined chiplet memory controller may scale up to a relatively large number of memory controllers and I/O ports, (e.g., up to 16 memory channels). In some embodiments, the memory controllers1420may implement a memory interleave (e.g., one memory address is mapped to the memory set1430, the next memory address is mapped to the memory set1432, and the third address is mapped to the memory set1430, etc.). The interleaving may be managed within the memory controllers1420, or from CPU sockets (e.g., of the compute sled800) across network links to the memory sets1430,1432, and may improve the latency associated with performing memory access operations as compared to accessing contiguous memory addresses from the same memory device.

Further, in some embodiments, the memory sled1400may be connected to one or more other sleds400(e.g., in the same rack240or an adjacent rack240) through a waveguide, using the waveguide connector1480. In the illustrative embodiment, the waveguides are 64 millimeter waveguides that provide 16 Rx (i.e., receive) lanes and 16 Tx (i.e., transmit) lanes. Each lane, in the illustrative embodiment, is either 16 GHz or 32 GHz. In other embodiments, the frequencies may be different. Using a waveguide may provide high throughput access to the memory pool (e.g., the memory sets1430,1432) to another sled (e.g., a sled400in the same rack240or an adjacent rack240as the memory sled1400) without adding to the load on the optical data connector834.

Referring now toFIG.15, a system for executing one or more workloads (e.g., applications) may be implemented in accordance with the data center100. In the illustrative embodiment, the system1510includes an orchestrator server1520, which may be embodied as a managed node comprising a compute device (e.g., a processor820on a compute sled800) executing management software (e.g., a cloud operating environment, such as OpenStack) that is communicatively coupled to multiple sleds400including a large number of compute sleds1530(e.g., each similar to the compute sled800), memory sleds1540(e.g., each similar to the memory sled1400), accelerator sleds1550(e.g., each similar to the accelerator sled1000), and storage sleds1560(e.g., each similar to the storage sled1200). One or more of the sleds1530,1540,1550,1560may be grouped into a managed node1570, such as by the orchestrator server1520, to collectively perform a workload (e.g., an application1532executed in a virtual machine or in a container). The managed node1570may be embodied as an assembly of physical resources620, such as processors820, memory resources720, accelerator circuits1020, or data storage1250, from the same or different sleds400. Further, the managed node may be established, defined, or “spun up” by the orchestrator server1520at the time a workload is to be assigned to the managed node or at any other time, and may exist regardless of whether any workloads are presently assigned to the managed node. In the illustrative embodiment, the orchestrator server1520may selectively allocate and/or deallocate physical resources620from the sleds400and/or add or remove one or more sleds400from the managed node1570as a function of quality of service (QoS) targets (e.g., a target throughput, a target latency, a target number instructions per second, etc.) associated with a service level agreement for the workload (e.g., the application1532). In doing so, the orchestrator server1520may receive telemetry data indicative of performance conditions (e.g., throughput, latency, instructions per second, etc.) in each sled400of the managed node1570and compare the telemetry data to the quality of service targets to determine whether the quality of service targets are being satisfied. The orchestrator server1520may additionally determine whether one or more physical resources may be deallocated from the managed node1570while still satisfying the QoS targets, thereby freeing up those physical resources for use in another managed node (e.g., to execute a different workload). Alternatively, if the QoS targets are not presently satisfied, the orchestrator server1520may determine to dynamically allocate additional physical resources to assist in the execution of the workload (e.g., the application1532) while the workload is executing. Similarly, the orchestrator server1520may determine to dynamically deallocate physical resources from a managed node if the orchestrator server1520determines that deallocating the physical resource would result in QoS targets still being met.

Additionally, in some embodiments, the orchestrator server1520may identify trends in the resource utilization of the workload (e.g., the application1532), such as by identifying phases of execution (e.g., time periods in which different operations, each having different resource utilizations characteristics, are performed) of the workload (e.g., the application1532) and pre-emptively identifying available resources in the data center100and allocating them to the managed node1570(e.g., within a predefined time period of the associated phase beginning). In some embodiments, the orchestrator server1520may model performance based on various latencies and a distribution scheme to place workloads among compute sleds and other resources (e.g., accelerator sleds, memory sleds, storage sleds) in the data center100. For example, the orchestrator server1520may utilize a model that accounts for the performance of resources on the sleds400(e.g., FPGA performance, memory access latency, etc.) and the performance (e.g., congestion, latency, bandwidth) of the path through the network to the resource (e.g., FPGA). As such, the orchestrator server1520may determine which resource(s) should be used with which workloads based on the total latency associated with each potential resource available in the data center100(e.g., the latency associated with the performance of the resource itself in addition to the latency associated with the path through the network between the compute sled executing the workload and the sled400on which the resource is located).

In some embodiments, the orchestrator server1520may generate a map of heat generation in the data center100using telemetry data (e.g., temperatures, fan speeds, etc.) reported from the sleds400and allocate resources to managed nodes as a function of the map of heat generation and predicted heat generation associated with different workloads, to maintain a target temperature and heat distribution in the data center100. Additionally or alternatively, in some embodiments, the orchestrator server1520may organize received telemetry data into a hierarchical model that is indicative of a relationship between the managed nodes (e.g., a spatial relationship such as the physical locations of the resources of the managed nodes within the data center100and/or a functional relationship, such as groupings of the managed nodes by the customers the managed nodes provide services for, the types of functions typically performed by the managed nodes, managed nodes that typically share or exchange workloads among each other, etc.). Based on differences in the physical locations and resources in the managed nodes, a given workload may exhibit different resource utilizations (e.g., cause a different internal temperature, use a different percentage of processor or memory capacity) across the resources of different managed nodes. The orchestrator server1520may determine the differences based on the telemetry data stored in the hierarchical model and factor the differences into a prediction of future resource utilization of a workload if the workload is reassigned from one managed node to another managed node, to accurately balance resource utilization in the data center100. In some embodiments, the orchestrator server1520may identify patterns in resource utilization phases of the workloads and use the patterns to predict future resource utilization of the workloads.

To reduce the computational load on the orchestrator server1520and the data transfer load on the network, in some embodiments, the orchestrator server1520may send self-test information to the sleds400to enable each sled400to locally (e.g., on the sled400) determine whether telemetry data generated by the sled400satisfies one or more conditions (e.g., an available capacity that satisfies a predefined threshold, a temperature that satisfies a predefined threshold, etc.). Each sled400may then report back a simplified result (e.g., yes or no) to the orchestrator server1520, which the orchestrator server1520may utilize in determining the allocation of resources to managed nodes.

Referring now toFIG.16, a disaggregated system1600for providing advanced management of power usage limits includes a pod manager1610, which may be embodied as any compute device (e.g., a compute sled) capable of managing the operations of multiple sleds across the system1600(e.g., across multiple racks) and configuring the allocation (e.g., selection) of resources (e.g., processors, data storage devices, etc.) to compose nodes (e.g., managed nodes) to execute workloads (e.g., sets of operations, processes, applications, etc.) in satisfaction of a defined set of quality of service (QoS) targets. The QoS targets may be defined by a service level agreement (SLA) between an owner/operator of the system1600(e.g., data center owner) and tenants of the system1600.

The pod manager1610, in the illustrative embodiment, is connected to multiple sleds1620,1630,1640,1650through a fabric (e.g., one or more switches or other networking components)1614. In the illustrative embodiment, the sled1620is a compute sled, similar to the compute sled800and includes among other components, processors1624(e.g., similar to the processors820) to execute one or more applications1626(e.g., sets of instructions, processes, etc. defining a workload). The sleds1630,1640,1650, in the illustrative embodiment, are data storage sleds, similar to the storage sled1200, and include data storage devices1636,1646,1656on which data shards1638,1648,1658are stored. The data shards1638,1648,1658are illustratively embodied as data sets (e.g., data objects) or portions of data sets that may be selectively written to or read from on an as needed basis (e.g., in response to data access requests from the compute sled1620executing the application1626on behalf of a tenant). In the illustrative embodiments, the data storage sleds1630,1640,1650define at least part of a cluster1616(e.g., a group) of storage nodes of a distributed data storage system (e.g., a Ceph distributed data storage system). Each data storage sled1630,1640,1650executes an object storage daemon1634,1644,1654, each of which may be embodied as any process (e.g., executable instructions) that locates data shards1638,1648,1658in response to a corresponding data access request (e.g., from the compute sled1620, specifying a data object to be accessed) on the data storage devices1636,1646,1656and enables access to them (e.g., reading from the data shards and/or writing to the data shards). As described in more detail herein, the data shards1638,1648,1658include redundant data. For example, a data object may be encoded with an erasure coding scheme across the data shards1638,1648,1658such that the data object can be reconstructed using only a subset (e.g., less than all of) the data shards associated with the data object in the distributed storage system. Additionally or alternatively, replicas (e.g., copies) of data shards may be stored on different data storage sleds1630,1640,1650such that, if one of the replicas is unavailable (e.g., because the corresponding data storage sled1630,1640,1650is inoperative or otherwise unavailable), another of the replicas may be accessed instead.

In the illustrative embodiment, the pod manager1610includes a power management logic unit1612which may be embodied as any device or circuitry (e.g., a processor, reconfigurable circuitry, an FPGA, an ASIC, etc.) or software configured to execute a management scheme in which the power usage limits of the sleds1620,1630,1640,1650may be temporarily adjusted by a particular amount, based on a brokering system in which one or more of the sleds1620,1630,1640,1650offers to temporarily reduce its power usage limit (e.g., because that sled does not need to utilize its entire power usage limit to satisfy a given QoS target) and one or more others of the sled1620,1630,1640,1650requests authorization to temporarily increase its power usage limit (e.g., to improve the latency, throughput, etc. provided by one or more of the resources on the sled to satisfy a given QoS target). Correspondingly, each sled1620,1630,1640,1650is illustratively equipped with a power management logic unit1622,1632,1642,1652, each of which may be embodied as any device or circuitry (e.g., a processor, reconfigurable circuitry, an FPGA, an ASIC, etc.) or software configured to determine whether the corresponding sled1630,1640,1650,1660is near (e.g., within a predefined range, such as 5% of) its power usage limit and, if so, request authorization (e.g., from the pod manager1610) to temporarily raise its power usage limit (e.g., to increase performance of one or more resources on the sled to satisfy a QoS target) and, conversely, to send an offer (e.g., to the pod manager1610) indicating that the corresponding sled is available to reduce its power usage limit temporarily (e.g., because the QoS targets for the operations performed by the sled can be met without utilizing the entire power usage limit for that sled). Similarly, the power management logic units1622,1632,1642,1652are configured to adjust (e.g., increase or decrease) the power usage limits in response to authorization (e.g., from the pod manager1610) to do so. The system1600may additionally adjust fan speeds associated with changes in the power usage limits of the sleds1620. By matching offers to reduce power usage limits with requests to increase power usage limits, the pod manager1610may maintain an overall amount of power usage across the sleds1620,1630,1640,1650and correspondingly manage thermal conditions across the racks of the pod (e.g., maintaining target temperatures and target air pressures in zones of the pod).

In some embodiments, the power management logic unit1612of the pod manager1610may only match offers with requests associated with the same power source (circuit breaker, phase, etc.) and/or cooling zone to ensure that power usage limits associated with a given power source or cooling zone are not exceeded. When an offer made by a sled1620,1630,1640,1650to reduce its power usage limit expires (e.g., after a predefined time period has elapsed) or if the offering sled has rebooted, the power usage limit of that sled may be restored to the power usage limit prior to that sled's voluntary power usage limit decrease. Similarly, any power usage limit increase on a sled (e.g., the sled1620) may be revoked (e.g., by the power management logic unit1612) when the corresponding time limit for a voluntary power usage limit decrease on another sled (e.g., the sled1630) elapses. Offers and requests may be continually renewed by the sleds1620,1630,1640,1650and, in some embodiments, the offers and/or requests may include data indicating that partial usage of the total amount of power offered or requested is prohibited (e.g., all or nothing). Additionally or alternatively, in some embodiments, the power management logic unit1612may combine multiple offers to satisfy a request to increase the power usage limit of a sled.

Furthermore, and as described in more detail herein, when access to a particular data set (e.g., data object) is requested, the system1600may avoid requesting a corresponding data shard from a sled1630,1640,1650if that sled is near its power usage limit, is unable to temporarily increase its power usage limit, and the requested data is available elsewhere (e.g., on another of the data storage sleds1630,1640,1650). As such, the system1600, and in particular, the cluster1616of the distributed data storage system, may avoid unnecessarily incurring additional latency from a data storage sled that is overloaded or otherwise unlikely to satisfy a QoS target associated with the data access request. While one compute sled1620and three data storage sleds1630,1640,1650are shown inFIG.16, it should be understood that the system1600may include any number of sleds and may include other types of sleds as well (e.g., memory sleds, accelerator sleds, etc.).

Referring now toFIG.17, a sled (e.g., any of the sleds1620,1630,1640,1650) may perform a method1700for providing advanced management of power usage limits. The method1700, in the illustrative embodiment, begins with block1702, in which the sled executes operations associated with one or more workload(s) assigned to the present sled. In doing so, the sled may execute one or more application(s) (e.g., the application1626) on behalf of a tenant, as indicated in block1704. As indicated in block1706, the sled may execute operations in response to requests (e.g., request(s) from the tenant, requests from the application1626, requests from other sleds, etc.). The sled, as indicated in block1708, may perform data access operations, such as accessing data in a distributed data storage system (e.g., the cluster1616), as indicated in block1710. In performing data access operations, the sled may read data (e.g., from one or more of the data shard(s)1638,1648,1658) as indicated in block1712and/or may write data (e.g., modify one or more existing data shards1638,1648,1658and/or write new data shards to one or more of the data storage device(s)1636,1646,1658), as indicated in block1714.

Subsequently, and as indicated in block1716, the sled may determine whether the present sled is within a predefined range (e.g., within 5% or predefined another range) of a power usage limit assigned to the present sled (e.g., by the pod manager1610). Subsequently, in block1718, the sled may determine a subsequent course of action based on whether the sled is within the predefined range of (e.g., is near) the power usage limit. If so, the present sled may request authorization to temporarily (e.g., for a defined time period, such as ten seconds) increase (e.g., by 20 Watts or another amount) the power usage limit assigned to the present sled, as indicated in block1720. In doing so, and as indicated in block1722, the present sled may request authorization from the pod manager1610. In block1724, the sled may obtain authorization to increase the power usage limit assigned to the present sled for a predefined time period (e.g., ten seconds). As indicated in block1726, the sled may obtain the authorization from the pod manager1610. In block1728, the sled may determine the subsequent course of action based on whether authorization to increase the power usage limit was obtained (e.g., from the pod manager1610) in block1724. If so, the method1700advances to block1730in which the sled increases the power usage limit for the predefined time period. In doing so, the sled may also increase a fan speed associated with the sled (e.g., to dissipate the additional heat caused by the increase in power usage), as indicated in block1732. Subsequently, the method1700loops back to block1702in which the sled continues to execute operations associated with one or more workloads.

Referring back to block1718, if the sled instead determines that it is not near the power usage limit assigned to the sled, the method1700may advance to block1734ofFIG.18, in which the sled sends, to other compute device(s), offer data indicative of an offer to reduce (e.g., by 20 Watts or another amount) the power usage limit assigned to the present sled for a predefined time period (e.g., ten seconds). In doing so, the sled may send the offer data to the pod manager1610, as indicated in block1736. Relatedly, and as indicated in block1738, the sled may send offer data that is also indicative of an offer to temporarily reduce a fan speed associated with the sled. As indicated in block1740, the sled may send notification data to one or more compute device(s) (e.g., to the pod manager1610and/or other sleds1620,1630,1640,1650) that the present sled is not near its power usage limit. In doing so, the sled may send the notification data to the compute device(s) to update an availability status of the present sled in a data storage map (e.g., a Ceph controlled replication under scalable hashing (CRUSH) map) indicative of locations of data sets (e.g., data shards1638,1648,1658) in a distributed data storage system (e.g., the cluster1616), as indicated in block1742. The sled, as indicated in block1744, may receive a request to reduce the power usage limit (e.g., by 20 Watts or another amount) associated with the sled for a predefined time period (e.g., ten seconds). In doing so, the sled may receive a request to reduce the power usage limit of the sled in order to balance thermal zones (e.g., hot zones and cold zones, such as relatively hot aisles and relatively cold aisles) in the data center, as indicated in block1746. As indicated in block1748, the request may be to reduce the power usage limit of the present sled in order to enable another sled to increase its power usage limit (e.g., the offer made by the present sled was matched with a request from another sled). Subsequently, and as indicated in block1750, the sled may reduce its power usage limit for the predefined time period. In doing so, the present sled may also reduce the fan speed associated with the sled, as indicated in block1752. Subsequently, the method1700loops back to block1702ofFIG.17, in which the sled continues to execute operations associated with one or more workload(s).

Referring briefly toFIG.17, if, in block1728, the sled determines that authorization to increase its power usage limit was not obtained, the method1700advances to block1754ofFIG.19, in which the sled sends notification data to one or more compute device(s) (e.g., the pod manager1610and/or one or more of the sleds1620,1630,1640,1650) that the present sled is near its power usage limit. In doing so, and as indicated in block1756, the sled may send the notification to update an availability status of the present sled in a data storage map (e.g., Ceph CRUSH map) indicative of locations of data sets in a distributed data storage system (e.g., the cluster1616), such as to indicate that a data set should not be requested from the present sled if the data set is available elsewhere in the distributed data storage system. Subsequently, the method1700loops back to block1702ofFIG.17, in which the sled continues to execute operations for one or more workloads. While the method1700is shown and described in a particular order, it should be understood that the operations of the method1700may be performed in a different order and/or concurrently (e.g., executing operations associated with workload(s) while also increasing and/or decreasing the power usage limit associated with the present sled). Additionally, while the sled is described as requesting permission and obtaining permission to adjust its power usage limit, in some embodiments the request may be embodied as the sending of telemetry data indicative of the present power usage of the sled and the authorization may be embodied as an instruction (e.g., from the pod manager1610) to temporarily adjust the power usage limit of the sled.

Referring now toFIG.20, a sled of the system1600, in operation, may execute a method2000for selectively utilizing data storage sleds (e.g., the sleds1630,1640,1650) of a distributed data storage system (e.g., the cluster1616) to perform data access operations. In the illustrative embodiment, the method2000begins with block2002, in which the sled obtains (e.g., reads from memory, receives from another sled1630,1640,1650, receives from the pod manager1610, etc.) storage map data (e.g., a Ceph CRUSH map) indicative of locations of data sets among sleds of a distributed data storage system (e.g., the cluster1616). In doing so, and as indicated in block2004, the sled may obtain data indicative of sleds of the distributed data storage system that are near their power usage limits.

In block2006, the sled may obtain a request (e.g., from the compute sled1620) to access data. For example, and as indicated in block2008, the request may be to access data in the distributed data storage system (e.g., the cluster1616). Subsequently, in block2010, the sled determines a selection of one or more sleds of the distributed data storage system (e.g., the cluster1616) from which to access the requested data. In doing so, and as indicated in block2012, the sled may exclude, from the selection, one or more sleds (e.g., the data storage sled1650) that have been identified as being near their power usage limit (e.g., as indicated in the data received in block2004). In excluding one or more sleds from the selection, the present sled, in the illustrative embodiment, excludes sled(s) (e.g., the data storage sled1650) that have redundant data (e.g., the data may be obtained from other sled(s), such as the data storage sleds1630,1640), as indicated in block2014. For example, and as indicated in block2016, the sled may determine whether the number of data storage sleds1630,1640,1650that have erasure coded portions of the requested data and that are not identified as being near their power usage limit satisfies a reference number needed to decode the data. More specifically, a data set may be erasure coded such that only n data storage sleds are needed to respond with their respective erasure coded portion of the data set in order for the data set to be decoded, but the data set is actually stored across n+k data storage sleds. As such, if n of the data storage sleds are not identified as being near their power usage limit, the sled may exclude the remaining k data storage sleds from the selection (e.g., if those data storage sleds are near their power usage limit), as indicated in block2018. Similarly, with regard to replicated data sets, the sled may exclude, from the selection, a data storage sled (e.g., the data storage sled1650) that has a primary replica of the requested data if a secondary replica of the data is available on a data storage sled (e.g., the data storage sled1640) that is not near its power usage limit, as indicated in block2020. Subsequently, the sled requests (and obtains) the data from the selection of data storage sled(s), as indicated in block2022. The sled may perform additional operations after requesting and obtaining the data, such as providing the data to an entity that requested it (e.g., another sled, the application, etc.). Afterwards, the method2000loops back to block2002, in which the sled may obtain updated data storage map data. While the method2000is shown and described in a particular order, it should be understood that the operations of the methods2000may be performed in a different order and/or concurrently.

Examples

Example 1 includes a compute device comprising circuitry to execute operations associated with a workload in a disaggregated system; determine whether a present power usage of the compute device is within a predefined range of a power usage limit assigned to the compute device; and send, to a device in the disaggregated system and in response to a determination that the present power usage of the present compute device is not within the predefined range of the power usage limit assigned to the present compute device, offer data indicative of an offer to reduce the power usage limit assigned to the present compute device to enable a second power utilization limit of another compute device in the disaggregated system to be increased.

Example 2 includes the subject matter of Example 1, and wherein to send the offer data comprises to send offer data to reduce the power usage limit assigned to the present compute device for a predefined period of time.

Example 3 includes the subject matter of any of Examples 1 and 2, and wherein to send the offer data comprises to send offer data to additionally reduce a fan speed associated with the present compute device.

Example 4 includes the subject matter of any of Examples 1-3, and wherein to send the offer data comprises to send the offer data to a pod manager.

Example 5 includes the subject matter of any of Examples 1-4, and wherein the circuitry is further to receive a request to reduce the power usage limit assigned to the present compute device.

Example 6 includes the subject matter of any of Examples 1-5, and wherein to receive the request to reduce the power usage limit comprises to receive the request after the present compute device has sent the offer data.

Example 7 includes the subject matter of any of Examples 1-6, and wherein the circuitry is further to reduce, in response to the received request, the power usage limit assigned to the present compute device.

Example 8 includes the subject matter of any of Examples 1-7, and wherein the circuitry is further to reduce a fan speed associated with the present compute device.

Example 9 includes the subject matter of any of Examples 1-8, and wherein to receive the request to reduce the power usage limit comprises to receive the request to balance thermal zones in a data center in which the disaggregated system is located.

Example 10 includes the subject matter of any of Examples 1-9, and wherein the circuitry is further to request, in response to a determination that the present power usage is within the predefined range of the power usage limit, authorization from another device in the disaggregated system to increase the power usage limit of the present compute device for a predefined time period.

Example 11 includes the subject matter of any of Examples 1-10, and wherein to request authorization comprises to request authorization from a pod manager of the disaggregated system.

Example 12 includes the subject matter of any of Examples 1-11, and wherein the circuitry is further to increase, in response to obtaining authorization, the power usage limit of the present compute device for the predefined time period.

Example 13 includes the subject matter of any of Examples 1-12, and wherein the compute device is further to increase, in response to obtaining authorization, a fan speed associated with the present compute device.

Example 14 includes the subject matter of any of Examples 1-13, and wherein the circuitry is further to send, to at least one other compute device of the disaggregated system and in response to a determination that authorization to increase the power usage limit was not obtained, notification data indicating that the present compute device is near the power usage limit.

Example 15 includes the subject matter of any of Examples 1-14, and wherein to send the notification data comprises to send notification data to the at least one other compute device to update an availability status of the present compute device in a data storage map indicative of locations of data sets in a distributed data storage system.

Example 16 includes the subject matter of any of Examples 1-15, and wherein the circuitry is further to obtain storage map data indicative of locations of data sets among compute devices of a distributed data storage system implemented in the disaggregated system; obtain a request to access data in the distributed data storage system; determine a selection of compute devices of the distributed data storage system from which to access the requested data; exclude, from the selection, one or more compute devices identified as being near their power usage limit; and access the data from the selection of compute devices.

Example 17 includes the subject matter of any of Examples 1-16, and wherein to exclude, from the selection, one or more compute devices identified as being near their power usage limit comprises to exclude one more compute devices that are identified as being near their power usage limit and that have data that is redundant to data that is available from other compute devices in the selection.

Example 18 includes one or more machine-readable storage media comprising a plurality of instructions stored thereon that, in response to being executed, cause a compute device to execute operations associated with a workload in a disaggregated system; determine whether a present power usage of the compute device is within a predefined range of a power usage limit assigned to the compute device; and send, to a device in the disaggregated system and in response to a determination that the present power usage of the present compute device is not within the predefined range of the power usage limit assigned to the present compute device, offer data indicative of an offer to reduce the power usage limit assigned to the present compute device to enable a second power utilization limit of another compute device in the disaggregated system to be increased.

Example 19 includes the subject matter of Example 18, and wherein the instructions further cause the compute device to obtain storage map data indicative of locations of data sets among compute devices of a distributed data storage system implemented in the disaggregated system; obtain a request to access data in the distributed data storage system; determine a selection of compute devices of the distributed data storage system from which to access the requested data; exclude, from the selection, one or more compute devices identified as being near their power usage limit; and access the data from the selection of compute devices.

Example 20 includes a method comprising executing, by a compute device, operations associated with a workload in a disaggregated system; determining, by the compute device, whether a present power usage of the compute device is within a predefined range of a power usage limit assigned to the compute device; and sending, by the compute device and to a device in the disaggregated system and in response to a determination that the present power usage of the present compute device is not within the predefined range of the power usage limit assigned to the present compute device, offer data indicative of an offer to reduce the power usage limit assigned to the present compute device to enable a second power utilization limit of another compute device in the disaggregated system to be increased.