Patent Description:
<CIT> discloses techniques for input/ output (I/O) access to physical memory or storage by a virtual machine (VM) or a container including use of a queue pair maintained at a controller for I/O access to the physical memory or storage. <CIT> discloses that read write access to host physical memory or storage <NUM> for VMs or containers may be facilitated by assignable queue pairs such as assignable NVMe queue pairs (ANQPs).

<CIT> relates to apparatuses for integrating RDMA-enabled network interface controllers (RNIC) and Non-Volatile Memory Express (NVMe) devices at the controller/device side. <CIT> discloses that the RDMA-enabled network interface controller (RNIC) will include a plurality of Queue Pairs (QP) for establishing communications with a client computing device.

While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed.

The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or nonvolatile memory, a media disc, or other media device).

Referring now to <FIG>, a data center <NUM> in which disaggregated resources may cooperatively execute one or more workloads (e.g., applications on behalf of customers) includes multiple pods <NUM>, <NUM>, <NUM>, <NUM>, each of which includes one or more rows of racks. Of course, although data center <NUM> is shown with multiple pods, in some embodiments, the data center <NUM> may be embodied as a single pod. As described in more detail herein, each rack houses multiple sleds, each of which may be primarily equipped with a particular type of resource (e.g., memory devices, data storage devices, accelerator devices, general purpose processors), i.e., resources that can be logically coupled to form a composed node, which can act as, for example, a server. In the illustrative embodiment, the sleds in each pod <NUM>, <NUM>, <NUM>, <NUM> are connected to multiple pod switches (e.g., switches that route data communications to and from sleds within the pod). The pod switches, in turn, connect with spine switches <NUM> that switch communications among pods (e.g., the pods <NUM>, <NUM>, <NUM>, <NUM>) in the data center <NUM>. In some embodiments, the sleds may be connected with a fabric using Intel Omni-Path technology. In other embodiments, the sleds may be connected with other fabrics, such as InfiniBand or Ethernet. As described in more detail herein, resources within sleds in the data center <NUM> may be allocated to a group (referred to herein as a "managed node") containing resources from one or more sleds to be collectively utilized in the execution of a workload. The workload can execute as if the resources belonging to the managed node were located on the same sled. The resources in a managed node may belong to sleds belonging to different racks, and even to different pods <NUM>, <NUM>, <NUM>, <NUM>. As such, some resources of a single sled may be allocated to one managed node while other resources of the same sled are allocated to a different managed node (e.g., one processor assigned to one managed node and another processor of the same sled assigned to a different managed node).

A data center comprising disaggregated resources, such as data center <NUM>, can be used in a wide variety of contexts, such as enterprise, government, cloud service provider, and communications service provider (e.g., Telco's), as well in a wide variety of sizes, from cloud service provider mega-data centers that consume over <NUM>,<NUM> sq. to single- or multi-rack installations for use in base stations.

The disaggregation of resources to sleds comprised predominantly of a single type of resource (e.g., compute sleds comprising primarily compute resources, memory sleds containing primarily memory resources), and the selective allocation and deallocation of the disaggregated resources to form a managed node assigned to execute a workload improves the operation and resource usage of the data center <NUM> relative to typical data centers comprised of hyperconverged servers containing compute, memory, storage and perhaps additional resources in a single chassis. For example, because sleds predominantly contain resources of a particular type, resources of a given type can be upgraded independently of other resources. Additionally, because different resources types (processors, storage, accelerators, etc.) typically have different refresh rates, greater resource utilization and reduced total cost of ownership may be achieved. For example, a data center operator can upgrade the processors throughout their facility by only swapping out the compute sleds. In such a case, accelerator and storage resources may not be contemporaneously upgraded and, rather, may be allowed to continue operating until those resources are scheduled for their own refresh. Resource utilization may also increase. For example, if managed nodes are composed based on requirements of the workloads that will be running on them, resources within a node are more likely to be fully utilized. Such utilization may allow for more managed nodes to run in a data center with a given set of resources, or for a data center expected to run a given set of workloads, to be built using fewer resources.

Referring now to <FIG>, the pod <NUM>, in the illustrative embodiment, includes a set of rows <NUM>, <NUM>, <NUM>, <NUM> of racks <NUM>. Each rack <NUM> may house multiple sleds (e.g., sixteen sleds) and provide power and data connections to the housed sleds, as described in more detail herein. In the illustrative embodiment, the racks in each row <NUM>, <NUM>, <NUM>, <NUM> are connected to multiple pod switches <NUM>, <NUM>. The pod switch <NUM> includes a set of ports <NUM> to which the sleds of the racks of the pod <NUM> are connected and another set of ports <NUM> that connect the pod <NUM> to the spine switches <NUM> to provide connectivity to other pods in the data center <NUM>. Similarly, the pod switch <NUM> includes a set of ports <NUM> to which the sleds of the racks of the pod <NUM> are connected and a set of ports <NUM> that connect the pod <NUM> to the spine switches <NUM>. As such, the use of the pair of switches <NUM>, <NUM> provides an amount of redundancy to the pod <NUM>. For example, if either of the switches <NUM>, <NUM> fails, the sleds in the pod <NUM> may still maintain data communication with the remainder of the data center <NUM> (e.g., sleds of other pods) through the other switch <NUM>, <NUM>. Furthermore, in the illustrative embodiment, the switches <NUM>, <NUM>, <NUM> may be embodied as dual-mode optical switches, capable of routing both Ethernet protocol communications carrying Internet Protocol (IP) packets and communications according to a second, high-performance link-layer protocol (e.g., Intel's Omni-Path Architecture's, InfiniBand, PCI Express) via optical signaling media of an optical fabric.

It should be appreciated that each of the other pods <NUM>, <NUM>, <NUM> (as well as any additional pods of the data center <NUM>) may be similarly structured as, and have components similar to, the pod <NUM> shown in and described in regard to <FIG> (e.g., each pod may have rows of racks housing multiple sleds as described above). Additionally, while two pod switches <NUM>, <NUM> are shown, it should be understood that in other embodiments, each pod <NUM>, <NUM>, <NUM>, <NUM> may be connected to a different number of pod switches, providing even more failover capacity. Of course, in other embodiments, pods may be arranged differently than the rows-of-racks configuration shown in <FIG>. For example, a pod may be embodied as multiple sets of racks in which each set of racks is arranged radially, i.e., the racks are equidistant from a center switch.

Referring now to <FIG>, each illustrative rack <NUM> of the data center <NUM> includes two elongated support posts <NUM>, <NUM>, which are arranged vertically. For example, the elongated support posts <NUM>, <NUM> may extend upwardly from a floor of the data center <NUM> when deployed. The rack <NUM> also includes one or more horizontal pairs <NUM> of elongated support arms <NUM> (identified in <FIG> via a dashed ellipse) configured to support a sled of the data center <NUM> as discussed below. One elongated support arm <NUM> of the pair of elongated support arms <NUM> extends outwardly from the elongated support post <NUM> and the other elongated support arm <NUM> extends outwardly from the elongated support post <NUM>.

In the illustrative embodiments, each sled of the data center <NUM> is embodied as a chassis-less sled. That is, each sled has a chassis-less circuit board substrate on which physical resources (e.g., processors, memory, accelerators, storage, etc.) are mounted as discussed in more detail below. As such, the rack <NUM> is configured to receive the chassis-less sleds. For example, each pair <NUM> of elongated support arms <NUM> defines a sled slot <NUM> of the rack <NUM>, which is configured to receive a corresponding chassis-less sled. To do so, each illustrative elongated support arm <NUM> includes a circuit board guide <NUM> configured to receive the chassis-less circuit board substrate of the sled. Each circuit board guide <NUM> is secured to, or otherwise mounted to, a top side <NUM> of the corresponding elongated support arm <NUM>. For example, in the illustrative embodiment, each circuit board guide <NUM> is mounted at a distal end of the corresponding elongated support arm <NUM> relative to the corresponding elongated support post <NUM>, <NUM>. For clarity of the Figures, not every circuit board guide <NUM> may be referenced in each Figure.

Each circuit board guide <NUM> includes an inner wall that defines a circuit board slot <NUM> configured to receive the chassis-less circuit board substrate of a sled <NUM> when the sled <NUM> is received in the corresponding sled slot <NUM> of the rack <NUM>. To do so, as shown in <FIG>, a user (or robot) aligns the chassis-less circuit board substrate of an illustrative chassis-less sled <NUM> to a sled slot <NUM>. The user, or robot, may then slide the chassis-less circuit board substrate forward into the sled slot <NUM> such that each side edge <NUM> of the chassis-less circuit board substrate is received in a corresponding circuit board slot <NUM> of the circuit board guides <NUM> of the pair <NUM> of elongated support arms <NUM> that define the corresponding sled slot <NUM> as shown in <FIG>. By having robotically accessible and robotically manipulable sleds comprising disaggregated resources, each type of resource can be upgraded independently of each other and at their own optimized refresh rate. Furthermore, the sleds are configured to blindly mate with power and data communication cables in each rack <NUM>, enhancing their ability to be quickly removed, upgraded, reinstalled, and/or replaced. As such, in some embodiments, the data center <NUM> may operate (e.g., execute workloads, undergo maintenance and/or upgrades, etc.) without human involvement on the data center floor. In other embodiments, a human may facilitate one or more maintenance or upgrade operations in the data center <NUM>.

It should be appreciated that each circuit board guide <NUM> is dual sided. That is, each circuit board guide <NUM> includes an inner wall that defines a circuit board slot <NUM> on each side of the circuit board guide <NUM>. In this way, each circuit board guide <NUM> can support a chassis-less circuit board substrate on either side. As such, a single additional elongated support post may be added to the rack <NUM> to turn the rack <NUM> into a two-rack solution that can hold twice as many sled slots <NUM> as shown in <FIG>. The illustrative rack <NUM> includes seven pairs <NUM> of elongated support arms <NUM> that define a corresponding seven sled slots <NUM>, each configured to receive and support a corresponding sled <NUM> as discussed above. Of course, in other embodiments, the rack <NUM> may include additional or fewer pairs <NUM> of elongated support arms <NUM> (i.e., additional or fewer sled slots <NUM>). It should be appreciated that because the sled <NUM> is chassis-less, the sled <NUM> may have an overall height that is different than typical servers. As such, in some embodiments, the height of each sled slot <NUM> may be shorter than the height of a typical server (e.g., shorter than a single rank unit, "1U"). That is, the vertical distance between each pair <NUM> of elongated support arms <NUM> may be less than a standard rack unit "1U. " Additionally, due to the relative decrease in height of the sled slots <NUM>, the overall height of the rack <NUM> in some embodiments may be shorter than the height of traditional rack enclosures. For example, in some embodiments, each of the elongated support posts <NUM>, <NUM> may have a length of six feet or less. Again, in other embodiments, the rack <NUM> may have different dimensions. For example, in some embodiments, the vertical distance between each pair <NUM> of elongated support arms <NUM> may be greater than a standard rack until "1U". In such embodiments, the increased vertical distance between the sleds allows for larger heat sinks to be attached to the physical resources and for larger fans to be used (e.g., in the fan array <NUM> described below) for cooling each sled, which in turn can allow the physical resources to operate at increased power levels. Further, it should be appreciated that the rack <NUM> does not include any walls, enclosures, or the like. Rather, the rack <NUM> is an enclosure-less rack that is opened to the local environment. Of course, in some cases, an end plate may be attached to one of the elongated support posts <NUM>, <NUM> in those situations in which the rack <NUM> forms an end-of-row rack in the data center <NUM>.

In some embodiments, various interconnects may be routed upwardly or downwardly through the elongated support posts <NUM>, <NUM>. To facilitate such routing, each elongated support post <NUM>, <NUM> includes an inner wall that defines an inner chamber in which interconnects may be located. The interconnects routed through the elongated support posts <NUM>, <NUM> may be embodied as any type of interconnects including, but not limited to, data or communication interconnects to provide communication connections to each sled slot <NUM>, power interconnects to provide power to each sled slot <NUM>, and/or other types of interconnects.

The rack <NUM>, in the illustrative embodiment, includes a support platform on which a corresponding optical data connector (not shown) is mounted. Each optical data connector is associated with a corresponding sled slot <NUM> and is configured to mate with an optical data connector of a corresponding sled <NUM> when the sled <NUM> is received in the corresponding sled slot <NUM>. In some embodiments, optical connections between components (e.g., sleds, racks, and switches) in the data center <NUM> are made with a blind mate optical connection. For example, a door on each cable may prevent dust from contaminating the fiber inside the cable. In the process of connecting to a blind mate optical connector mechanism, the door is pushed open when the end of the cable approaches or enters the connector mechanism. Subsequently, the optical fiber inside the cable may enter a gel within the connector mechanism and the optical fiber of one cable comes into contact with the optical fiber of another cable within the gel inside the connector mechanism.

The illustrative rack <NUM> also includes a fan array <NUM> coupled to the cross-support arms of the rack <NUM>. The fan array <NUM> includes one or more rows of cooling fans <NUM>, which are aligned in a horizontal line between the elongated support posts <NUM>, <NUM>. In the illustrative embodiment, the fan array <NUM> includes a row of cooling fans <NUM> for each sled slot <NUM> of the rack <NUM>. As discussed above, each sled <NUM> does not include any on-board cooling system in the illustrative embodiment and, as such, the fan array <NUM> provides cooling for each sled <NUM> received in the rack <NUM>. Each rack <NUM>, in the illustrative embodiment, also includes a power supply associated with each sled slot <NUM>. Each power supply is secured to one of the elongated support arms <NUM> of the pair <NUM> of elongated support arms <NUM> that define the corresponding sled slot <NUM>. For example, the rack <NUM> may include a power supply coupled or secured to each elongated support arm <NUM> extending from the elongated support post <NUM>. Each power supply includes a power connector configured to mate with a power connector of the sled <NUM> when the sled <NUM> is received in the corresponding sled slot <NUM>. In the illustrative embodiment, the sled <NUM> does not include any on-board power supply and, as such, the power supplies provided in the rack <NUM> supply power to corresponding sleds <NUM> when mounted to the rack <NUM>. Each power supply is configured to satisfy the power requirements for its associated sled, which can vary from sled to sled. Additionally, the power supplies provided in the rack <NUM> can operate independent of each other. That is, within a single rack, a first power supply providing power to a compute sled can provide power levels that are different than power levels supplied by a second power supply providing power to an accelerator sled. The power supplies may be controllable at the sled level or rack level, and may be controlled locally by components on the associated sled or remotely, such as by another sled or an orchestrator.

Referring now to <FIG>, the sled <NUM>, in the illustrative embodiment, is configured to be mounted in a corresponding rack <NUM> of the data center <NUM> as discussed above. In some embodiments, each sled <NUM> may be optimized or otherwise configured for performing particular tasks, such as compute tasks, acceleration tasks, data storage tasks, etc. For example, the sled <NUM> may be embodied as a compute sled <NUM> as discussed below in regard to <FIG>, an accelerator sled <NUM> as discussed below in regard to <FIG>, a storage sled <NUM> as discussed below in regard to <FIG>, or as a sled optimized or otherwise configured to perform other specialized tasks, such as a memory sled <NUM>, discussed below in regard to <FIG>.

As discussed above, the illustrative sled <NUM> includes a chassis-less circuit board substrate <NUM>, which supports various physical resources (e.g., electrical components) mounted thereon. It should be appreciated that the circuit board substrate <NUM> is "chassis-less" in that the sled <NUM> does not include a housing or enclosure. Rather, the chassis-less circuit board substrate <NUM> is open to the local environment. The chassis-less circuit board substrate <NUM> may be formed from any material capable of supporting the various electrical components mounted thereon. For example, in an illustrative embodiment, the chassis-less circuit board substrate <NUM> is formed from an FR-<NUM> glass-reinforced epoxy laminate material. Of course, other materials may be used to form the chassis-less circuit board substrate <NUM> in other embodiments.

As discussed in more detail below, the chassis-less circuit board substrate <NUM> includes multiple features that improve the thermal cooling characteristics of the various electrical components mounted on the chassis-less circuit board substrate <NUM>. As discussed, the chassis-less circuit board substrate <NUM> does not include a housing or enclosure, which may improve the airflow over the electrical components of the sled <NUM> by reducing those structures that may inhibit air flow. For example, because the chassis-less circuit board substrate <NUM> is not positioned in an individual housing or enclosure, there is no vertically-arranged backplane (e.g., a backplate of the chassis) attached to the chassis-less circuit board substrate <NUM>, which could inhibit air flow across the electrical components. Additionally, the chassis-less circuit board substrate <NUM> has a geometric shape configured to reduce the length of the airflow path across the electrical components mounted to the chassis-less circuit board substrate <NUM>. For example, the illustrative chassis-less circuit board substrate <NUM> has a width <NUM> that is greater than a depth <NUM> of the chassis-less circuit board substrate <NUM>. In one particular embodiment, for example, the chassis-less circuit board substrate <NUM> has a width of about <NUM> inches and a depth of about <NUM> inches, compared to a typical server that has a width of about <NUM> inches and a depth of about <NUM> inches. As such, an airflow path <NUM> that extends from a front edge <NUM> of the chassis-less circuit board substrate <NUM> toward a rear edge <NUM> has a shorter distance relative to typical servers, which may improve the thermal cooling characteristics of the sled <NUM>. Furthermore, although not illustrated in <FIG>, the various physical resources mounted to the chassis-less circuit board substrate <NUM> are mounted in corresponding locations such that no two substantively heat-producing electrical components shadow each other as discussed in more detail below. That is, no two electrical components, which produce appreciable heat during operation (i.e., greater than a nominal heat sufficient enough to adversely impact the cooling of another electrical component), are mounted to the chassis-less circuit board substrate <NUM> linearly in-line with each other along the direction of the airflow path <NUM> (i.e., along a direction extending from the front edge <NUM> toward the rear edge <NUM> of the chassis-less circuit board substrate <NUM>).

As discussed above, the illustrative sled <NUM> includes one or more physical resources <NUM> mounted to a top side <NUM> of the chassis-less circuit board substrate <NUM>. Although two physical resources <NUM> are shown in <FIG>, it should be appreciated that the sled <NUM> may include one, two, or more physical resources <NUM> in other embodiments. The physical resources <NUM> may be embodied as any type of processor, controller, or other compute circuit capable of performing various tasks such as compute functions and/or controlling the functions of the sled <NUM> depending on, for example, the type or intended functionality of the sled <NUM>. For example, as discussed in more detail below, the physical resources <NUM> may be embodied as high-performance processors in embodiments in which the sled <NUM> is embodied as a compute sled, as accelerator co-processors or circuits in embodiments in which the sled <NUM> is embodied as an accelerator sled, storage controllers in embodiments in which the sled <NUM> is embodied as a storage sled, or a set of memory devices in embodiments in which the sled <NUM> is embodied as a memory sled.

The sled <NUM> also includes one or more additional physical resources <NUM> mounted to the top side <NUM> of the chassis-less circuit board substrate <NUM>. In the illustrative embodiment, the additional physical resources include a network interface controller (NIC) as discussed in more detail below. Of course, depending on the type and functionality of the sled <NUM>, the physical resources <NUM> may include additional or other electrical components, circuits, and/or devices in other embodiments.

The physical resources <NUM> are communicatively coupled to the physical resources <NUM> via an input/output (I/O) subsystem <NUM>. The I/O subsystem <NUM> may be embodied as circuitry and/or components to facilitate input/output operations with the physical resources <NUM>, the physical resources <NUM>, and/or other components of the sled <NUM>. For example, the I/O subsystem <NUM> may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, integrated sensor hubs, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, waveguides, light guides, printed circuit board traces, etc.), and/or other components and subsystems to facilitate the input/output operations. In the illustrative embodiment, the I/O subsystem <NUM> is embodied as, or otherwise includes, a double data rate <NUM> (DDR4) data bus or a DDR5 data bus, as described further below.

In some embodiments, the sled <NUM> may also include a resource-to-resource interconnect <NUM>. The resource-to-resource interconnect <NUM> may be embodied as any type of communication interconnect capable of facilitating resource-to-resource communications. In the illustrative embodiment, the resource-to-resource interconnect <NUM> is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem <NUM>). For example, the resource-to-resource interconnect <NUM> may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to resource-to-resource communications.

The sled <NUM> also includes a power connector <NUM> configured to mate with a corresponding power connector of the rack <NUM> when the sled <NUM> is mounted in the corresponding rack <NUM>. The sled <NUM> receives power from a power supply of the rack <NUM> via the power connector <NUM> to supply power to the various electrical components of the sled <NUM>. That is, the sled <NUM> does not include any local power supply (i.e., an on-board power supply) to provide power to the electrical components of the sled <NUM>. The exclusion of a local or on-board power supply facilitates the reduction in the overall footprint of the chassis-less circuit board substrate <NUM>, which may increase the thermal cooling characteristics of the various electrical components mounted on the chassis-less circuit board substrate <NUM> as discussed above. In some embodiments, voltage regulators are placed on a bottom side <NUM> (see <FIG>) of the chassis-less circuit board substrate <NUM> directly opposite of the processors <NUM> (see <FIG>), and power is routed from the voltage regulators to the processors <NUM> by vias extending through the circuit board substrate <NUM>. Such a configuration provides an increased thermal budget, additional current and/or voltage, and better voltage control relative to typical printed circuit boards in which processor power is delivered from a voltage regulator, in part, by printed circuit traces.

In some embodiments, the sled <NUM> may also include mounting features <NUM> configured to mate with a mounting arm, or other structure, of a robot to facilitate the placement of the sled <NUM> in a rack <NUM> by the robot. The mounting features <NUM> may be embodied as any type of physical structures that allow the robot to grasp the sled <NUM> without damaging the chassis-less circuit board substrate <NUM> or the electrical components mounted thereto. For example, in some embodiments, the mounting features <NUM> may be embodied as non-conductive pads attached to the chassis-less circuit board substrate <NUM>. In other embodiments, the mounting features may be embodied as brackets, braces, or other similar structures attached to the chassis-less circuit board substrate <NUM>. The particular number, shape, size, and/or make-up of the mounting feature <NUM> may depend on the design of the robot configured to manage the sled <NUM>.

Referring now to <FIG>, in addition to the physical resources <NUM> mounted on the top side <NUM> of the chassis-less circuit board substrate <NUM>, the sled <NUM> also includes one or more memory devices <NUM> mounted to a bottom side <NUM> of the chassis-less circuit board substrate <NUM>. That is, the chassis-less circuit board substrate <NUM> is embodied as a double-sided circuit board. The physical resources <NUM> are communicatively coupled to the memory devices <NUM> via the I/O subsystem <NUM>. For example, the physical resources <NUM> and the memory devices <NUM> may be communicatively coupled by one or more vias extending through the chassis-less circuit board substrate <NUM>. Each physical resource <NUM> may be communicatively coupled to a different set of one or more memory devices <NUM> in some embodiments. Alternatively, in other embodiments, each physical resource <NUM> may be communicatively coupled to each memory device <NUM>.

The memory devices <NUM> may be embodied as any type of memory device capable of storing data for the physical resources <NUM> during operation of the sled <NUM>, such as any type of volatile (e.g., dynamic random access memory (DRAM), etc.) or nonvolatile memory. Volatile memory may be a memory that requires power to maintain the state of data stored by the medium. Non-limiting examples of volatile memory may include various types of random access memory (RAM), such as dynamic random access memory (DRAM) or static random access memory (SRAM). One particular type of DRAM that may be used in a memory module is synchronous dynamic random access memory (SDRAM). In particular embodiments, DRAM of a memory component may comply with a standard promulgated by the Joint Electronic Device Engineering Council (JEDEC), such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209 for Low Power DDR (LPDDR), JESD209-<NUM> for LPDDR2, JESD209-<NUM> for LPDDR3, and JESD209-<NUM> for LPDDR4. Such standards (and similar standards) may be referred to as DDR-based standards and communication interfaces of the memory devices that implement such standards may be referred to as DDR-based interfaces.

In one embodiment, the memory device is a block addressable memory device, such as those based on NAND or NOR technologies, such as multi-threshold level NAND flash memory or NOR flash memory. A memory device may also include byte addressable write-in-place nonvolatile memory devices, such as Intel 3D XPoint™ memory, Intel Optane™ memory, Micron QuantX™ memory, magnetoresistive random access memory (MRAM) memory that incorporates memristor technology, or other byte addressable write-in-place nonvolatile memory devices. In some embodiments, the memory device may comprise a transistor-less stackable cross point architecture in which memory cells sit at the intersection of word lines and bit lines and are individually addressable and in which bit storage is based on a change in bulk resistance. In one embodiment, the memory device may be or may include memory devices that use chalcogenide glass, single or multi-level Phase Change Memory (PCM), a resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), anti-ferroelectric memory, magnetoresistive random access memory (MRAM) memory that incorporates memristor technology, resistive memory including the metal oxide base, the oxygen vacancy base and the conductive bridge Random Access Memory (CB-RAM), or spin transfer torque (STT)-MRAM, a spintronic magnetic junction memory based device, a magnetic tunneling junction (MTJ) based device, a DW (Domain Wall) and SOT (Spin Orbit Transfer) based device, a thyristor based memory device, or a combination of any of the above, or other memory. The memory device may refer to the die itself and/or to a packaged memory product.

Referring now to <FIG>, in some embodiments, the sled <NUM> may be embodied as a compute sled <NUM>. The compute sled <NUM> is optimized, or otherwise configured, to perform compute tasks. Of course, as discussed above, the compute sled <NUM> may rely on other sleds, such as acceleration sleds and/or storage sleds, to perform such compute tasks. The compute sled <NUM> includes various physical resources (e.g., electrical components) similar to the physical resources of the sled <NUM>, which have been identified in <FIG> using the same reference numbers. The description of such components provided above in regard to <FIG> and <FIG> applies to the corresponding components of the compute sled <NUM> and is not repeated herein for clarity of the description of the compute sled <NUM>.

In the illustrative compute sled <NUM>, the physical resources <NUM> are embodied as processors <NUM>. Although only two processors <NUM> are shown in <FIG>, it should be appreciated that the compute sled <NUM> may include additional processors <NUM> in other embodiments. Illustratively, the processors <NUM> are embodied as high-performance processors <NUM> and may be configured to operate at a relatively high power rating. Although the processors <NUM> generate additional heat operating at power ratings greater than typical processors (which operate at around <NUM>-<NUM> W), the enhanced thermal cooling characteristics of the chassis-less circuit board substrate <NUM> discussed above facilitate the higher power operation. For example, in the illustrative embodiment, the processors <NUM> are configured to operate at a power rating of at least <NUM> W. In some embodiments, the processors <NUM> may be configured to operate at a power rating of at least <NUM> W.

In some embodiments, the compute sled <NUM> may also include a processor-to-processor interconnect <NUM>. Similar to the resource-to-resource interconnect <NUM> of the sled <NUM> discussed above, the processor-to-processor interconnect <NUM> may be embodied as any type of communication interconnect capable of facilitating processor-to-processor interconnect <NUM> communications. In the illustrative embodiment, the processor-to-processor interconnect <NUM> is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem <NUM>). For example, the processor-to-processor interconnect <NUM> may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications.

The compute sled <NUM> also includes a communication circuit <NUM>. The illustrative communication circuit <NUM> includes a network interface controller (NIC) <NUM>, which may also be referred to as a host fabric interface (HFI). The NIC <NUM> may be embodied as, or otherwise include, any type of integrated circuit, discrete circuits, controller chips, chipsets, add-in-boards, daughtercards, network interface cards, or other devices that may be used by the compute sled <NUM> to connect with another compute device (e.g., with other sleds <NUM>). In some embodiments, the NIC <NUM> may be embodied as part of a system-on-a-chip (SoC) that includes one or more processors, or included on a multichip package that also contains one or more processors. In some embodiments, the NIC <NUM> may include a local processor (not shown) and/or a local memory (not shown) that are both local to the NIC <NUM>. In such embodiments, the local processor of the NIC <NUM> may be capable of performing one or more of the functions of the processors <NUM>. Additionally or alternatively, in such embodiments, the local memory of the NIC <NUM> may be integrated into one or more components of the compute sled at the board level, socket level, chip level, and/or other levels.

The communication circuit <NUM> is communicatively coupled to an optical data connector <NUM>. The optical data connector <NUM> is configured to mate with a corresponding optical data connector of the rack <NUM> when the compute sled <NUM> is mounted in the rack <NUM>. Illustratively, the optical data connector <NUM> includes a plurality of optical fibers which lead from a mating surface of the optical data connector <NUM> to an optical transceiver <NUM>. The optical transceiver <NUM> is configured to convert incoming optical signals from the rack-side optical data connector to electrical signals and to convert electrical signals to outgoing optical signals to the rack-side optical data connector. Although shown as forming part of the optical data connector <NUM> in the illustrative embodiment, the optical transceiver <NUM> may form a portion of the communication circuit <NUM> in other embodiments.

In some embodiments, the compute sled <NUM> may also include an expansion connector <NUM>. In such embodiments, the expansion connector <NUM> is configured to mate with a corresponding connector of an expansion chassis-less circuit board substrate to provide additional physical resources to the compute sled <NUM>. The additional physical resources may be used, for example, by the processors <NUM> during operation of the compute sled <NUM>. The expansion chassis-less circuit board substrate may be substantially similar to the chassis-less circuit board substrate <NUM> discussed above and may include various electrical components mounted thereto. The particular electrical components mounted to the expansion chassis-less circuit board substrate may depend on the intended functionality of the expansion chassis-less circuit board substrate. For example, the expansion chassis-less circuit board substrate may provide additional compute resources, memory resources, and/or storage resources. As such, the additional physical resources of the expansion chassis-less circuit board substrate may include, but is not limited to, processors, memory devices, storage devices, and/or accelerator circuits including, for example, field programmable gate arrays (FPGA), application-specific integrated circuits (ASICs), security co-processors, graphics processing units (GPUs), machine learning circuits, or other specialized processors, controllers, devices, and/or circuits.

Referring now to <FIG>, an illustrative embodiment of the compute sled <NUM> is shown. As shown, the processors <NUM>, communication circuit <NUM>, and optical data connector <NUM> are mounted to the top side <NUM> of the chassis-less circuit board substrate <NUM>. Any suitable attachment or mounting technology may be used to mount the physical resources of the compute sled <NUM> to the chassis-less circuit board substrate <NUM>. For example, the various physical resources may be mounted in corresponding sockets (e.g., a processor socket), holders, or brackets. In some cases, some of the electrical components may be directly mounted to the chassis-less circuit board substrate <NUM> via soldering or similar techniques.

As discussed above, the individual processors <NUM> and communication circuit <NUM> are mounted to the top side <NUM> of the chassis-less circuit board substrate <NUM> such that no two heat-producing, electrical components shadow each other. In the illustrative embodiment, the processors <NUM> and communication circuit <NUM> are mounted in corresponding locations on the top side <NUM> of the chassis-less circuit board substrate <NUM> such that no two of those physical resources are linearly in-line with others along the direction of the airflow path <NUM>. It should be appreciated that, although the optical data connector <NUM> is in-line with the communication circuit <NUM>, the optical data connector <NUM> produces no or nominal heat during operation.

The memory devices <NUM> of the compute sled <NUM> are mounted to the bottom side <NUM> of the of the chassis-less circuit board substrate <NUM> as discussed above in regard to the sled <NUM>. Although mounted to the bottom side <NUM>, the memory devices <NUM> are communicatively coupled to the processors <NUM> located on the top side <NUM> via the I/O subsystem <NUM>. Because the chassis-less circuit board substrate <NUM> is embodied as a double-sided circuit board, the memory devices <NUM> and the processors <NUM> may be communicatively coupled by one or more vias, connectors, or other mechanisms extending through the chassis-less circuit board substrate <NUM>. Of course, each processor <NUM> may be communicatively coupled to a different set of one or more memory devices <NUM> in some embodiments. Alternatively, in other embodiments, each processor <NUM> may be communicatively coupled to each memory device <NUM>. In some embodiments, the memory devices <NUM> may be mounted to one or more memory mezzanines on the bottom side of the chassis-less circuit board substrate <NUM> and may interconnect with a corresponding processor <NUM> through a ball-grid array.

Each of the processors <NUM> includes a heat sink <NUM> secured thereto. Due to the mounting of the memory devices <NUM> to the bottom side <NUM> of the chassis-less circuit board substrate <NUM> (as well as the vertical spacing of the sleds <NUM> in the corresponding rack <NUM>), the top side <NUM> of the chassis-less circuit board substrate <NUM> includes additional "free" area or space that facilitates the use of heat sinks <NUM> having a larger size relative to traditional heat sinks used in typical servers. Additionally, due to the improved thermal cooling characteristics of the chassis-less circuit board substrate <NUM>, none of the processor heat sinks <NUM> include cooling fans attached thereto. That is, each of the heat sinks <NUM> is embodied as a fan-less heat sink. In some embodiments, the heat sinks <NUM> mounted atop the processors <NUM> may overlap with the heat sink attached to the communication circuit <NUM> in the direction of the airflow path <NUM> due to their increased size, as illustratively suggested by <FIG>.

Referring now to <FIG>, in some embodiments, the sled <NUM> may be embodied as an accelerator sled <NUM>. The accelerator sled <NUM> is configured, to perform specialized compute tasks, such as machine learning, encryption, hashing, or other computational-intensive task. In some embodiments, for example, a compute sled <NUM> may offload tasks to the accelerator sled <NUM> during operation. The accelerator sled <NUM> includes various components similar to components of the sled <NUM> and/or compute sled <NUM>, which have been identified in <FIG> using the same reference numbers. The description of such components provided above in regard to <FIG>, <FIG>, and <FIG> apply to the corresponding components of the accelerator sled <NUM> and is not repeated herein for clarity of the description of the accelerator sled <NUM>.

In the illustrative accelerator sled <NUM>, the physical resources <NUM> are embodied as accelerator circuits <NUM>. Although only two accelerator circuits <NUM> are shown in <FIG>, it should be appreciated that the accelerator sled <NUM> may include additional accelerator circuits <NUM> in other embodiments. For example, as shown in <FIG>, the accelerator sled <NUM> may include four accelerator circuits <NUM> in some embodiments. The accelerator circuits <NUM> may be embodied as any type of processor, co-processor, compute circuit, or other device capable of performing compute or processing operations. For example, the accelerator circuits <NUM> may be embodied as, for example, field programmable gate arrays (FPGA), application-specific integrated circuits (ASICs), security co-processors, graphics processing units (GPUs), neuromorphic processor units, quantum computers, machine learning circuits, or other specialized processors, controllers, devices, and/or circuits.

In some embodiments, the accelerator sled <NUM> may also include an accelerator-to-accelerator interconnect <NUM>. Similar to the resource-to-resource interconnect <NUM> of the sled <NUM> discussed above, the accelerator-to-accelerator interconnect <NUM> may be embodied as any type of communication interconnect capable of facilitating accelerator-to-accelerator communications. In the illustrative embodiment, the accelerator-to-accelerator interconnect <NUM> is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem <NUM>). For example, the accelerator-to-accelerator interconnect <NUM> may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications. In some embodiments, the accelerator circuits <NUM> may be daisy-chained with a primary accelerator circuit <NUM> connected to the NIC <NUM> and memory <NUM> through the I/O subsystem <NUM> and a secondary accelerator circuit <NUM> connected to the NIC <NUM> and memory <NUM> through a primary accelerator circuit <NUM>.

Referring now to <FIG>, an illustrative embodiment of the accelerator sled <NUM> is shown. As discussed above, the accelerator circuits <NUM>, communication circuit <NUM>, and optical data connector <NUM> are mounted to the top side <NUM> of the chassis-less circuit board substrate <NUM>. Again, the individual accelerator circuits <NUM> and communication circuit <NUM> are mounted to the top side <NUM> of the chassis-less circuit board substrate <NUM> such that no two heat-producing, electrical components shadow each other as discussed above. The memory devices <NUM> of the accelerator sled <NUM> are mounted to the bottom side <NUM> of the of the chassis-less circuit board substrate <NUM> as discussed above in regard to the sled <NUM>. Although mounted to the bottom side <NUM>, the memory devices <NUM> are communicatively coupled to the accelerator circuits <NUM> located on the top side <NUM> via the I/O subsystem <NUM> (e.g., through vias). Further, each of the accelerator circuits <NUM> may include a heat sink <NUM> that is larger than a traditional heat sink used in a server. As discussed above with reference to the heat sinks <NUM>, the heat sinks <NUM> may be larger than traditional heat sinks because of the "free" area provided by the memory resources <NUM> being located on the bottom side <NUM> of the chassis-less circuit board substrate <NUM> rather than on the top side <NUM>.

Referring now to <FIG>, in some embodiments, the sled <NUM> may be embodied as a storage sled <NUM>. The storage sled <NUM> is configured, to store data in a data storage <NUM> local to the storage sled <NUM>. For example, during operation, a compute sled <NUM> or an accelerator sled <NUM> may store and retrieve data from the data storage <NUM> of the storage sled <NUM>. The storage sled <NUM> includes various components similar to components of the sled <NUM> and/or the compute sled <NUM>, which have been identified in <FIG> using the same reference numbers. The description of such components provided above in regard to <FIG>, <FIG>, and <FIG> apply to the corresponding components of the storage sled <NUM> and is not repeated herein for clarity of the description of the storage sled <NUM>.

In the illustrative storage sled <NUM>, the physical resources <NUM> are embodied as storage controllers <NUM>. Although only two storage controllers <NUM> are shown in <FIG>, it should be appreciated that the storage sled <NUM> may include additional storage controllers <NUM> in other embodiments. The storage controllers <NUM> may be embodied as any type of processor, controller, or control circuit capable of controlling the storage and retrieval of data into the data storage <NUM> based on requests received via the communication circuit <NUM>. In the illustrative embodiment, the storage controllers <NUM> are embodied as relatively low-power processors or controllers. For example, in some embodiments, the storage controllers <NUM> may be configured to operate at a power rating of about <NUM> watts.

In some embodiments, the storage sled <NUM> may also include a controller-to-controller interconnect <NUM>. Similar to the resource-to-resource interconnect <NUM> of the sled <NUM> discussed above, the controller-to-controller interconnect <NUM> may be embodied as any type of communication interconnect capable of facilitating controller-to-controller communications. In the illustrative embodiment, the controller-to-controller interconnect <NUM> is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem <NUM>). For example, the controller-to-controller interconnect <NUM> may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications.

Referring now to <FIG>, an illustrative embodiment of the storage sled <NUM> is shown. In the illustrative embodiment, the data storage <NUM> is embodied as, or otherwise includes, a storage cage <NUM> configured to house one or more solid state drives (SSDs) <NUM>. To do so, the storage cage <NUM> includes a number of mounting slots <NUM>, each of which is configured to receive a corresponding solid state drive <NUM>. Each of the mounting slots <NUM> includes a number of drive guides <NUM> that cooperate to define an access opening <NUM> of the corresponding mounting slot <NUM>. The storage cage <NUM> is secured to the chassis-less circuit board substrate <NUM> such that the access openings face away from (i.e., toward the front of) the chassis-less circuit board substrate <NUM>. As such, solid state drives <NUM> are accessible while the storage sled <NUM> is mounted in a corresponding rack <NUM>. For example, a solid state drive <NUM> may be swapped out of a rack <NUM> (e.g., via a robot) while the storage sled <NUM> remains mounted in the corresponding rack <NUM>.

The storage cage <NUM> illustratively includes sixteen mounting slots <NUM> and is capable of mounting and storing sixteen solid state drives <NUM>. Of course, the storage cage <NUM> may be configured to store additional or fewer solid state drives <NUM> in other embodiments. Additionally, in the illustrative embodiment, the solid state drivers are mounted vertically in the storage cage <NUM>, but may be mounted in the storage cage <NUM> in a different orientation in other embodiments. Each solid state drive <NUM> may be embodied as any type of data storage device capable of storing long term data. To do so, the solid state drives <NUM> may include volatile and nonvolatile memory devices discussed above.

As shown in <FIG>, the storage controllers <NUM>, the communication circuit <NUM>, and the optical data connector <NUM> are illustratively mounted to the top side <NUM> of the chassis-less circuit board substrate <NUM>. Again, as discussed above, any suitable attachment or mounting technology may be used to mount the electrical components of the storage sled <NUM> to the chassis-less circuit board substrate <NUM> including, for example, sockets (e.g., a processor socket), holders, brackets, soldered connections, and/or other mounting or securing techniques.

As discussed above, the individual storage controllers <NUM> and the communication circuit <NUM> are mounted to the top side <NUM> of the chassis-less circuit board substrate <NUM> such that no two heat-producing, electrical components shadow each other. For example, the storage controllers <NUM> and the communication circuit <NUM> are mounted in corresponding locations on the top side <NUM> of the chassis-less circuit board substrate <NUM> such that no two of those electrical components are linearly in-line with each other along the direction of the airflow path <NUM>.

The memory devices <NUM> of the storage sled <NUM> are mounted to the bottom side <NUM> of the of the chassis-less circuit board substrate <NUM> as discussed above in regard to the sled <NUM>. Although mounted to the bottom side <NUM>, the memory devices <NUM> are communicatively coupled to the storage controllers <NUM> located on the top side <NUM> via the I/O subsystem <NUM>. Again, because the chassis-less circuit board substrate <NUM> is embodied as a double-sided circuit board, the memory devices <NUM> and the storage controllers <NUM> may be communicatively coupled by one or more vias, connectors, or other mechanisms extending through the chassis-less circuit board substrate <NUM>. Each of the storage controllers <NUM> includes a heat sink <NUM> secured thereto. As discussed above, due to the improved thermal cooling characteristics of the chassis-less circuit board substrate <NUM> of the storage sled <NUM>, none of the heat sinks <NUM> include cooling fans attached thereto. That is, each of the heat sinks <NUM> is embodied as a fan-less heat sink.

Referring now to <FIG>, in some embodiments, the sled <NUM> may be embodied as a memory sled <NUM>. The storage sled <NUM> is optimized, or otherwise configured, to provide other sleds <NUM> (e.g., compute sleds <NUM>, accelerator sleds <NUM>, etc.) with access to a pool of memory (e.g., in two or more sets <NUM>, <NUM> of memory devices <NUM>) local to the memory sled <NUM>. For example, during operation, a compute sled <NUM> or an accelerator sled <NUM> may remotely write to and/or read from one or more of the memory sets <NUM>, <NUM> of the memory sled <NUM> using a logical address space that maps to physical addresses in the memory sets <NUM>, <NUM>. The memory sled <NUM> includes various components similar to components of the sled <NUM> and/or the compute sled <NUM>, which have been identified in <FIG> using the same reference numbers. The description of such components provided above in regard to <FIG>, <FIG>, and <FIG> apply to the corresponding components of the memory sled <NUM> and is not repeated herein for clarity of the description of the memory sled <NUM>.

In the illustrative memory sled <NUM>, the physical resources <NUM> are embodied as memory controllers <NUM>. Although only two memory controllers <NUM> are shown in <FIG>, it should be appreciated that the memory sled <NUM> may include additional memory controllers <NUM> in other embodiments. The memory controllers <NUM> may be embodied as any type of processor, controller, or control circuit capable of controlling the writing and reading of data into the memory sets <NUM>, <NUM> based on requests received via the communication circuit <NUM>. In the illustrative embodiment, each memory controller <NUM> is connected to a corresponding memory set <NUM>, <NUM> to write to and read from memory devices <NUM> within the corresponding memory set <NUM>, <NUM> and enforce any permissions (e.g., read, write, etc.) associated with sled <NUM> that has sent a request to the memory sled <NUM> to perform a memory access operation (e.g., read or write).

In some embodiments, the memory sled <NUM> may also include a controller-to-controller interconnect <NUM>. Similar to the resource-to-resource interconnect <NUM> of the sled <NUM> discussed above, the controller-to-controller interconnect <NUM> may be embodied as any type of communication interconnect capable of facilitating controller-to-controller communications. In the illustrative embodiment, the controller-to-controller interconnect <NUM> is embodied as a high-speed point-to-point interconnect (e.g., faster than the I/O subsystem <NUM>). For example, the controller-to-controller interconnect <NUM> may be embodied as a QuickPath Interconnect (QPI), an UltraPath Interconnect (UPI), or other high-speed point-to-point interconnect dedicated to processor-to-processor communications. As such, in some embodiments, a memory controller <NUM> may access, through the controller-to-controller interconnect <NUM>, memory that is within the memory set <NUM> associated with another memory controller <NUM>. In some embodiments, a scalable memory controller is made of multiple smaller memory controllers, referred to herein as "chiplets", on a memory sled (e.g., the memory sled <NUM>). The chiplets may be interconnected (e.g., using EMIB (Embedded Multi-Die Interconnect Bridge)). The combined chiplet memory controller may scale up to a relatively large number of memory controllers and I/O ports, (e.g., up to <NUM> memory channels). In some embodiments, the memory controllers <NUM> may implement a memory interleave (e.g., one memory address is mapped to the memory set <NUM>, the next memory address is mapped to the memory set <NUM>, and the third address is mapped to the memory set <NUM>, etc.). The interleaving may be managed within the memory controllers <NUM>, or from CPU sockets (e.g., of the compute sled <NUM>) across network links to the memory sets <NUM>, <NUM>, and may improve the latency associated with performing memory access operations as compared to accessing contiguous memory addresses from the same memory device.

Further, in some embodiments, the memory sled <NUM> may be connected to one or more other sleds <NUM> (e.g., in the same rack <NUM> or an adjacent rack <NUM>) through a waveguide, using the waveguide connector <NUM>. In the illustrative embodiment, the waveguides are <NUM> millimeter waveguides that provide <NUM> Rx (i.e., receive) lanes and <NUM> Tx (i.e., transmit) lanes. Each lane, in the illustrative embodiment, is either <NUM> or <NUM>. In other embodiments, the frequencies may be different. Using a waveguide may provide high throughput access to the memory pool (e.g., the memory sets <NUM>, <NUM>) to another sled (e.g., a sled <NUM> in the same rack <NUM> or an adjacent rack <NUM> as the memory sled <NUM>) without adding to the load on the optical data connector <NUM>.

Referring now to <FIG>, a system for executing one or more workloads (e.g., applications) may be implemented in accordance with the data center <NUM>. In the illustrative embodiment, the system <NUM> includes an orchestrator server <NUM>, which may be embodied as a managed node comprising a compute device (e.g., a processor <NUM> on a compute sled <NUM>) executing management software (e.g., a cloud operating environment, such as OpenStack) that is communicatively coupled to multiple sleds <NUM> including a large number of compute sleds <NUM> (e.g., each similar to the compute sled <NUM>), memory sleds <NUM> (e.g., each similar to the memory sled <NUM>), accelerator sleds <NUM> (e.g., each similar to the memory sled <NUM>), and storage sleds <NUM> (e.g., each similar to the storage sled <NUM>). One or more of the sleds <NUM>, <NUM>, <NUM>, <NUM> may be grouped into a managed node <NUM>, such as by the orchestrator server <NUM>, to collectively perform a workload (e.g., an application <NUM> executed in a virtual machine or in a container). The managed node <NUM> may be embodied as an assembly of physical resources <NUM>, such as processors <NUM>, memory resources <NUM>, accelerator circuits <NUM>, or data storage <NUM>, from the same or different sleds <NUM>. Further, the managed node may be established, defined, or "spun up" by the orchestrator server <NUM> at the time a workload is to be assigned to the managed node or at any other time, and may exist regardless of whether any workloads are presently assigned to the managed node. In the illustrative embodiment, the orchestrator server <NUM> may selectively allocate and/or deallocate physical resources <NUM> from the sleds <NUM> and/or add or remove one or more sleds <NUM> from the managed node <NUM> as a function of quality of service (QoS) targets (e.g., performance targets associated with a throughput, latency, instructions per second, etc.) associated with a service level agreement for the workload (e.g., the application <NUM>). In doing so, the orchestrator server <NUM> may receive telemetry data indicative of performance conditions (e.g., throughput, latency, instructions per second, etc.) in each sled <NUM> of the managed node <NUM> and compare the telemetry data to the quality of service targets to determine whether the quality of service targets are being satisfied. The orchestrator server <NUM> may additionally determine whether one or more physical resources may be deallocated from the managed node <NUM> while still satisfying the QoS targets, thereby freeing up those physical resources for use in another managed node (e.g., to execute a different workload). Alternatively, if the QoS targets are not presently satisfied, the orchestrator server <NUM> may determine to dynamically allocate additional physical resources to assist in the execution of the workload (e.g., the application <NUM>) while the workload is executing. Similarly, the orchestrator server <NUM> may determine to dynamically deallocate physical resources from a managed node if the orchestrator server <NUM> determines that deallocating the physical resource would result in QoS targets still being met.

Additionally, in some embodiments, the orchestrator server <NUM> may identify trends in the resource utilization of the workload (e.g., the application <NUM>), such as by identifying phases of execution (e.g., time periods in which different operations, each having different resource utilizations characteristics, are performed) of the workload (e.g., the application <NUM>) and pre-emptively identifying available resources in the data center <NUM> and allocating them to the managed node <NUM> (e.g., within a predefined time period of the associated phase beginning). In some embodiments, the orchestrator server <NUM> may model performance based on various latencies and a distribution scheme to place workloads among compute sleds and other resources (e.g., accelerator sleds, memory sleds, storage sleds) in the data center <NUM>. For example, the orchestrator server <NUM> may utilize a model that accounts for the performance of resources on the sleds <NUM> (e.g., FPGA performance, memory access latency, etc.) and the performance (e.g., congestion, latency, bandwidth) of the path through the network to the resource (e.g., FPGA). As such, the orchestrator server <NUM> may determine which resource(s) should be used with which workloads based on the total latency associated with each potential resource available in the data center <NUM> (e.g., the latency associated with the performance of the resource itself in addition to the latency associated with the path through the network between the compute sled executing the workload and the sled <NUM> on which the resource is located).

In some embodiments, the orchestrator server <NUM> may generate a map of heat generation in the data center <NUM> using telemetry data (e.g., temperatures, fan speeds, etc.) reported from the sleds <NUM> and allocate resources to managed nodes as a function of the map of heat generation and predicted heat generation associated with different workloads, to maintain a target temperature and heat distribution in the data center <NUM>. Additionally or alternatively, in some embodiments, the orchestrator server <NUM> may organize received telemetry data into a hierarchical model that is indicative of a relationship between the managed nodes (e.g., a spatial relationship such as the physical locations of the resources of the managed nodes within the data center <NUM> and/or a functional relationship, such as groupings of the managed nodes by the customers the managed nodes provide services for, the types of functions typically performed by the managed nodes, managed nodes that typically share or exchange workloads among each other, etc.). Based on differences in the physical locations and resources in the managed nodes, a given workload may exhibit different resource utilizations (e.g., cause a different internal temperature, use a different percentage of processor or memory capacity) across the resources of different managed nodes. The orchestrator server <NUM> may determine the differences based on the telemetry data stored in the hierarchical model and factor the differences into a prediction of future resource utilization of a workload if the workload is reassigned from one managed node to another managed node, to accurately balance resource utilization in the data center <NUM>.

To reduce the computational load on the orchestrator server <NUM> and the data transfer load on the network, in some embodiments, the orchestrator server <NUM> may send self-test information to the sleds <NUM> to enable each sled <NUM> to locally (e.g., on the sled <NUM>) determine whether telemetry data generated by the sled <NUM> satisfies one or more conditions (e.g., an available capacity that satisfies a predefined threshold, a temperature that satisfies a predefined threshold, etc.). Each sled <NUM> may then report back a simplified result (e.g., yes or no) to the orchestrator server <NUM>, which the orchestrator server <NUM> may utilize in determining the allocation of resources to managed nodes.

Referring now to <FIG>, an illustrative system <NUM> for remote direct memory access (RDMA) queue pair quality-of-service (QoS) management include an orchestrator server <NUM>, one or more compute sleds <NUM>, one or more storage sleds <NUM>, one or more accelerator sleds <NUM>, and one or more network switches <NUM>. In use, RDMA queue pairs of a send queue and a receive queue may be established as part of an RDMA protocol between various nodes, such as between two compute sleds <NUM>, between a compute sled <NUM> and a storage sled <NUM>, between a compute sled <NUM> and an accelerator sled <NUM>, etc. The RDMA queue pairs on the compute sled <NUM> may be associated with a virtual machine. Each RDMA queue pair of the virtual machine may have a different QoS parameter, such as a different class of service parameter, a different weight, etc., such that each RDMA queue pair of the virtual machine may, e.g., be allocated a different amount of bandwidth. For example, a first RDMA queue pair of a virtual machine that is associated with an RDMA queue pair on a remote compute sled <NUM> may have a first QoS parameter, a second RDMA queue pair of a virtual machine that is associated with an RDMA queue pair on a remote storage sled <NUM> may have a second QoS parameter, and a third RDMA queue pair of a virtual machine that is associated with an RDMA queue pair on a remote accelerator sled <NUM> may have a third QoS parameter, wherein the first, second, and third QoS parameters can all be different from each other. In some embodiments, the compute sled <NUM> and/or orchestrator server <NUM> may analyze RDMA flows and predict future bandwidth usage of RDMA queue pairs. The compute sled <NUM> may adjust the bandwidth assigned to each RDMA queue pair based on the predicted future bandwidth usage.

Referring now to <FIG>, the orchestrator server <NUM> may be embodied as any type of compute device capable of performing the orchestration functions described herein. For example, the orchestrator server <NUM> may be embodied as or otherwise be included in, without limitation, a server computer, an embedded computing system, a System-on-a-Chip (SoC), a multiprocessor system, a processor-based system, a consumer electronic device, a smartphone, a cellular phone, a desktop computer, a tablet computer, a notebook computer, a laptop computer, a network device, a router, a switch, a networked computer, a wearable computer, a handset, a messaging device, a camera device, and/or any other computing device. In some embodiments, the orchestrator server <NUM> may be composed of or otherwise include two or more disaggregated components, such as one or more compute sleds <NUM>, one or more storage sleds <NUM>, and/or one or more network switches <NUM>. The illustrative orchestrator server <NUM> includes a processor <NUM>, a memory <NUM>, an input/output (I/O) subsystem <NUM>, one or more storage devices <NUM>, and a network interface controller <NUM>. In some embodiments, one or more of the illustrative components of the orchestrator server <NUM> may be incorporated in, or otherwise form a portion of, another component. For example, the memory <NUM>, or portions thereof, may be incorporated in the processor <NUM> in some embodiments.

The processor <NUM> may be embodied as any type of processor capable of performing the functions described herein. For example, the processor <NUM> may be embodied as a single or multi-core processor(s), a single or multi-socket processor, a digital signal processor, a graphics processor, a microcontroller, or other processor or processing/controlling circuit. Similarly, the memory <NUM> may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory <NUM> may store various data and software used during operation of the orchestrator server <NUM> such as operating systems, applications, programs, libraries, and drivers. The memory <NUM> is communicatively coupled to the processor <NUM> via the I/O subsystem <NUM>, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor <NUM>, the memory <NUM>, and other components of the orchestrator server <NUM>. For example, the I/O subsystem <NUM> may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem <NUM> may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor <NUM>, the memory <NUM>, and other components of the orchestrator server <NUM> on a single integrated circuit chip.

The one or more storage devices <NUM> may be embodied as any type of device or devices configured for the short-term or long-term storage of data. For example, the one or more storage devices <NUM> may include any one or more memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices.

The network interface controller <NUM> may be embodied as any type of interface capable of interfacing the orchestrator server <NUM> with other compute devices, such as through the one or more network switches <NUM>. In some embodiments, the network interface controller <NUM> may be referred to as a host fabric interface (HFI). The network interface controller <NUM> may be capable of interfacing with any appropriate cable type, such as an electrical cable or an optical cable, and/or may be capable of interfacing with a wireless signal, such as through one or more antennae. The network interface controller <NUM> may be configured to use any one or more communication technology and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, near field communication (NFC), etc.). The network interface controller <NUM> may be located on silicon separate from the processor <NUM>, or the network interface controller <NUM> may be included in a multi-chip package with the processor <NUM>, or even on the same die as the processor <NUM>. The network interface controller <NUM> may be embodied as one or more add-in-boards, daughtercards, network interface cards, controller chips, chipsets, specialized components such as a field programmable gate array (FPGA) or application specific integrated circuit (ASIC), or other devices that may be used by the orchestrator server <NUM> to connect with another compute device. In some embodiments, network interface controller <NUM> may be embodied as part of a system-on-a-chip (SoC) that includes one or more processors, or included on a multichip package that also contains one or more processors. In some embodiments, the network interface controller <NUM> may include a local processor (not shown) and/or a local memory (not shown) that are both local to the network interface controller <NUM>. In such embodiments, the local processor of the network interface controller <NUM> may be capable of performing one or more of the functions of the processor <NUM> described herein. Additionally or alternatively, in such embodiments, the local memory of the network interface controller <NUM> may be integrated into one or more components of the orchestrator server <NUM> at the board level, socket level, chip level, and/or other levels.

In some embodiments, the orchestrator server <NUM> may include other or additional components, such as those commonly found in a compute device. For example, the orchestrator server <NUM> may also have a display <NUM> and/or peripheral devices <NUM>. The peripheral devices <NUM> may include a keyboard, a mouse, etc. The display <NUM> may be embodied as any type of display on which information may be displayed to a user of the orchestrator server <NUM>, such as a touchscreen display, a liquid crystal display (LCD), a light emitting diode (LED) display, a cathode ray tube (CRT) display, a plasma display, an image projector (e.g., 2D or 3D), a laser projector, a heads-up display, and/or other display technology.

Referring now to <FIG>, the compute sled <NUM> may be embodied as any hardware capable of performing the function described herein. In the illustrative embodiment, the compute sled <NUM> is embodied as a sled in a rack of a data center with several high-performance multi-core processors <NUM>. Additionally or alternatively, the compute sled <NUM> may be embodied as or otherwise be included in, without limitation, a server computer, an embedded computing system, a System-on-a-Chip (SoC), a multiprocessor system, a processor-based system, a consumer electronic device, a smartphone, a cellular phone, a desktop computer, a tablet computer, a notebook computer, a laptop computer, a network device, a router, a switch, a networked computer, a wearable computer, a handset, a messaging device, a camera device, and/or any other computing device.

The storage sled <NUM> includes a processor <NUM>, a memory <NUM>, an I/O subsystem <NUM>, and a network interface controller <NUM>. In some embodiments, the processor <NUM>, the memory <NUM>, etc. may be similar to the processor <NUM>, the memory <NUM>, etc. of the orchestrator server <NUM>, the description of which will not be included in the interest of clarity. Of course, it should be appreciated that, in some embodiments, the components of the compute sled <NUM> may differ from the orchestrator server <NUM> quantitatively or qualitatively. For example, in one embodiment, the orchestrator server <NUM> may have processors <NUM> that do not have as high of performance as the processors <NUM> on the compute sled <NUM>.

In some embodiments, certain components of the compute sled <NUM> may have certain capabilities that were not explicitly discussed in regard to the corresponding component of the orchestrator server <NUM>. For example, the processor <NUM> may have the capability of controlling allocation of resources among various processes, threads, application, virtual machines, containers, etc. In one embodiment, processor <NUM> may implement some or all of the Intel® Resource Director Technology (RDT). The processor <NUM> may be able to monitor cache usage, allocate cache usage, monitor memory bandwidth, allocate memory bandwidth, etc. to the various processes, threads, applications, virtual machines, containers, etc. Each process, thread, application, virtual machine, container, etc. may be associated with a particular quality of service (QoS) parameter, such as by being assigned to a particular class of service (CoS).

Additionally, the illustrative network interface controller <NUM> may have capability that the network interface controller <NUM> does not necessarily ahve. In particular, the network interface controller includes one or more remote direct memory access (RDMA) queues <NUM>. In the illustrative embodiment, the compute sled <NUM> may establish one or more pairs of RDMA queues <NUM>, each of which may be associated with an RDMA queue pair on a remote device, such as another compute sled <NUM>, a storage sled <NUM>, or an accelerator sled <NUM>. Each pair of RDMA queues <NUM> includes a send RMDA queue <NUM> and a receive RDMA queue <NUM>. The network interface controller <NUM> is configured to send data or commands placed in the send RDMA queue <NUM> to the corresponding receive RMDA queue on the remote device. The network interface controller <NUM> is also configured to place packets sent from the corresponding send queue on the remote device in the receive RDMA queue <NUM>. The network interface controller <NUM> may process the receive RDMA queue <NUM> by, e.g., writing data to the memory <NUM> and/or reading data from the memory <NUM> to be placed in the send RDMA queue <NUM>. The network interface controller <NUM> may service the various RDMA queues <NUM> using any suitable approach, such as weighted round robin, weighted fair queuing, etc..

In the illustrative embodiment, the network interface controller <NUM> is capable of servicing the pairs of RDMA queues <NUM> differently based on an associated QoS parameter. For example, each pair of RDMA queue <NUM> may be assigned to a different CoS, and the network interface controller <NUM> may then service the RDMA queue <NUM> with a different bandwidth or weighting based on the assigned class. As a result, the pairs of RDMA queues <NUM> may have a different performance based on the associated QoS parameter. A QoS parameter may become associated with a particular pair of RDMA queues <NUM> in any suitable manner, such as by execution by the processor <NUM> of a suitable operation code or opcode. In one embodiment, the processor <NUM> may get and set a QoS parameter of a pair of RDMA queues <NUM> using an opcode OP_GET_VF_QOS and OP_SET_VF_QOS, respectively.

In the illustrative embodiment, the compute sled <NUM> is primarily used for compute services, and other services such as storage services and/or accelerator services may be provided by, e.g., a storage sled <NUM> and/or an accelerator sled <NUM>. However, in some embodiments, the compute sled <NUM> may include one or more storage devices <NUM>, one or more accelerator devices <NUM>, a display <NUM>, and/or one or more peripheral devices <NUM>. Each of the storage devices <NUM>, display <NUM>, and peripheral devices <NUM> may be similar to the storage devices <NUM>, display <NUM>, and peripheral devices <NUM> of the orchestrator server <NUM>, the description of which will not be repeated in the interest of clarity. Each of the accelerator devices <NUM> may be any suitable accelerator device, such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), etc..

Each of the storage sleds <NUM> and accelerator sleds <NUM> may include some or all of the same components as the compute sled <NUM>, which will not be repeated in the interest of clarity. Of course, it should be appreciated that, in some embodiments, some of the components of the storage sleds <NUM> and/or the accelerator sleds <NUM> may different quantitatively or qualitatively from those of the compute sled <NUM>. For example, in the illustrative embodiment, the storage sleds <NUM> may include a relatively low-performance processor(s) and a large amount of high-capacity storage devices. Similarly, the accelerator sleds <NUM> may include a relatively low-performance processor(s) and several high-performance accelerator devices, with or without some storage devices.

Referring now to <FIG>, in an illustrative embodiment, the compute sled <NUM> establishes an environment <NUM> during operation. The illustrative environment <NUM> includes a host agent <NUM>, a virtual machine manager <NUM>, a resource controller <NUM>, an RDMA queue pair bandwidth predictor <NUM>, and a network interface controller <NUM>. The various modules of the environment <NUM> may be embodied as hardware, software, firmware, or a combination thereof. For example, the various modules, logic, and other components of the environment <NUM> may form a portion of, or otherwise be established by, the processor <NUM> or other hardware components of the compute sled <NUM>. As such, in some embodiments, one or more of the modules of the environment <NUM> may be embodied as circuitry or collection of electrical devices (e.g., a host agent circuit <NUM>, a virtual machine manager circuit <NUM>, a resource controller circuit <NUM>, etc.). It should be appreciated that, in such embodiments, one or more of the circuits (e.g., the host agent circuit <NUM>, the virtual machine manager circuit <NUM>, the resource controller circuit <NUM>, etc.) may form a portion of one or more of the processor <NUM>, the memory <NUM>, the I/O subsystem <NUM>, the network interface controller <NUM>, the storage devices <NUM>, and/or other components of the compute sled <NUM>. Additionally, in some embodiments, one or more of the illustrative modules may form a portion of another module and/or one or more of the illustrative modules may be independent of one another. Further, in some embodiments, one or more of the modules of the environment <NUM> may be embodied as virtualized hardware components or emulated architecture, which may be established and maintained by the processor <NUM> or other components of the compute sled <NUM>. It should be appreciated that, in some embodiments, some of the functionality of one or more of the modules of the environment <NUM> may require a hardware implementation, in which case embodiments of modules which implement such functionality will be embodied at least partially as hardware.

The host agent <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to interface with the orchestrator server <NUM>. The host agent <NUM> may allow the orchestrator server <NUM> to control the compute sled <NUM>, such as by controlling what virtual machines are being operated on the compute sled <NUM>.

The virtual machine manager <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to manage the virtual machines that may operate on the compute sled <NUM>. The virtual machine manager may configure, create, manage, terminate, and perform any other suitable control of virtual machines on the compute sled <NUM>.

The virtual machine manager <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, includes a virtual machine RDMA queue pair manager <NUM>. Some or all of the virtual machines on the compute sled <NUM> may have one or more RDMA queue pairs established in the network interface controller <NUM>. In the illustrative embodiment, each RDMA queue pair is assigned to a different virtual extensible local area network (VXLAN). Additionally or alternatively, some or all of the RDMA queue pairs may be assigned to the same VXLAN. The virtual machine RDMA queue pair manager <NUM> includes a compute sled RDMA queue pair manager <NUM>, a storage sled RDMA queue pair manager <NUM>, and an accelerator sled RDMA queue pair manager <NUM>. The compute sled RDMA queue pair manager <NUM> may manage one or more RDMA queue pairs associated with a remote compute sled <NUM>. The storage sled RDMA queue pair manager <NUM> may manage one or more RDMA queue pairs associated with a storage sled <NUM>. The accelerator sled RDMA queue pair manager <NUM> may manage one or more RDMA queue pairs associated with an accelerator sled <NUM>.

The resource controller <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to allocate certain resources to processes, threads, applications, virtual machines, containers, etc. of the compute sled <NUM>. The resource controller includes a cache QoS manager <NUM>, a memory bandwidth QoS manager <NUM>, and a RDMA queue pair QoS manager <NUM>. The cache QoS manager <NUM> is configured to monitor cache usage and allocate cache usage on a virtual machine by virtual machine basis. The memory bandwidth QoS manage <NUM> is configured to monitor memory bandwidth and allocate memory bandwidth on a virtual machine by virtual machine basis. The RDMA queue pair QoS manager <NUM> is configured to allocate bandwidth to each RDMA queue pair of each virtual machine based on one or more QoS parameters, such as a class of service ID or a weighting. In the illustrative embodiment, the QoS parameters may include a minimum and maximum bandwidth to allocate to each RDMA queue pair, and the RDMA queue pair QoS manager <NUM> may configure a minimum and a maximum bandwidth for each RDMA queue pair accordingly. The QoS parameters may also include a send queue priority setting, allocation of traffic classes to the virtual machine, and strict priority traffic classes.

The RDMA queue pair bandwidth predictor <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to predict a bandwidth usage for some or all of the RDMA queue pairs configured in the network interface controller <NUM>. To do so, the RDMA queue pair bandwidth predictor <NUM> includes a telemetry module <NUM> to monitor various parameters of the virtual machines such as RDMA flow data. In the illustrative embodiment, the telemetry module <NUM> may send telemetry data of an RDMA flow to a remote device such as the orchestrator server <NUM>, where RDMA flows may be classified as elephant flows or mouse flows, as discussed in more detail below in regard to the environment <NUM> of the orchestrator server <NUM>. The orchestrator server <NUM> may then send the classification back to the compute sled <NUM>, where the RDMA queue pair bandwidth predictor <NUM> may use the classification as an input in prediction of future bandwidth usage.

The RDMA queue pair bandwidth predictor <NUM> may use any suitable parameter to predict future RDMA queue pair bandwidth, such as flow rate, number of flows, flow types, applications associated with flows, the type of device on the other end of the flow, etc. In one embodiment, if a RDMA flow is identified as an elephant flow, the corresponding RDMA queue pair bandwidth may be predicted to increase, and the bandwidth assigned to that RDMA queue pair may be increased while the elephant flow is ongoing.

The network interface manager <NUM> may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above. In the illustrative embodiment, the network interface manager <NUM> forms part of the network interface controller <NUM>. Additionally or alternatively, some or all of the network interface manager <NUM> may be part of the processor <NUM>, the memory <NUM>, etc. The network interface manager <NUM> is configured to manage the network interface controller <NUM>. The network interface manager <NUM> may send and receive data to and from other compute devices. The network interface manager <NUM> includes the RDMA queue pair servicer <NUM>, which processes the RDMA queue pairs. The RDMA queue pair servicer <NUM> is configured to send data or commands placed in each send RDMA queue to the corresponding receive RMDA queue on a remote device. The RDMA queue pair servicer <NUM> is also configured to place packets sent from a send queue on a remote device in the corresponding receive RDMA queue. The RDMA queue pair servicer <NUM> may process the receive RDMA queue by, e.g., writing data to the memory <NUM> and/or reading data from the memory <NUM> to be placed in the corresponding send RDMA queue. In the illustrative embodiment, the RDMA queue pair servicer <NUM> may service the various RDMA queues using a weighted round robin approach, where the weighting is determined by a QoS parameter for each RDMA queue pair. Additionally or alternatively, other approaches may be used, such as using weighted fair queuing or applying an amount of bandwidth determined by a QoS parameter for each RDMA queue pair.

Referring now to <FIG>, in an illustrative embodiment, the orchestrator server <NUM> establishes an environment <NUM> during operation. The illustrative environment <NUM> includes a sled controller <NUM> and a network traffic analyzer <NUM>. The various modules of the environment <NUM> may be embodied as hardware, software, firmware, or a combination thereof. For example, the various modules, logic, and other components of the environment <NUM> may form a portion of, or otherwise be established by, the processor <NUM> or other hardware components of the orchestrator server <NUM>. As such, in some embodiments, one or more of the modules of the environment <NUM> may be embodied as circuitry or collection of electrical devices (e.g., a sled controller circuit <NUM>, a network traffic analyzer circuit <NUM>, etc.). It should be appreciated that, in such embodiments, one or more of the circuits (e.g., the sled controller circuit <NUM>, the network traffic analyzer circuit <NUM>, etc.) may form a portion of one or more of the processor <NUM>, the memory <NUM>, the I/O subsystem <NUM>, the storage devices <NUM>, the network interface controller <NUM>, and/or other components of the orchestrator server <NUM>. Additionally, in some embodiments, one or more of the illustrative modules may form a portion of another module and/or one or more of the illustrative modules may be independent of one another. Further, in some embodiments, one or more of the modules of the environment <NUM> may be embodied as virtualized hardware components or emulated architecture, which may be established and maintained by the processor <NUM> or other components of the orchestrator server <NUM>. It should be appreciated that, in some embodiments, some of the functionality of one or more of the modules of the environment <NUM> may require a hardware implementation, in which case embodiments of modules which implement such functionality will be embodied at least partially as hardware.

The sled controller <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to control the various compute sleds <NUM>, storage sleds <NUM>, and accelerator sleds <NUM> of the system <NUM>. The sled controller <NUM> may compose nodes, configure virtual machines, etc. Of course, in some embodiments, the sled controller <NUM> may perform additional functionality that is not described in detail in the interest of clarity.

The network traffic analyzer <NUM>, which may be embodied as hardware, firmware, software, virtualized hardware, emulated architecture, and/or a combination thereof as discussed above, is configured to analyze network traffic of the system <NUM>. In particular, in the illustrative embodiment, the network traffic analyzer <NUM> will receive telemetry data of an RDMA flow from one or more sleds such as a compute sled <NUM> that indicates various information about an RDMA flow, such as overall size and type of traffic. The network traffic analyzer <NUM> may analyze the telemetry data of the RDMA flow and may send the result of analysis back to the sled associated with the RDMA flow.

The network traffic analyzer <NUM> includes an elephant/mouse network flow classifier <NUM>, which is configured to classify an RDMA flow as either an elephant flow or a mouse flow. In the illustrative embodiment the elephant/mouse network flow classifier <NUM> may analyze an amount of traffic in a certain amount of time, and, if a flow has an amount of traffic over a certain threshold, the RDMA flow may be classified as an elephant flow. The amount of time over which the traffic amount is evaluated may be any suitable time, such as <NUM>, <NUM>, or <NUM> milliseconds or <NUM>, <NUM>, <NUM>, or <NUM> seconds. The threshold amount of traffic that triggers an RDMA flow to be classified as an elephant flow may be any suitable value, such as <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>,<NUM> KB. When an RDMA flow is classified as an elephant or mouse flow, that classification may be sent back to the compute sled <NUM> associated with the RDMA flow.

Referring now to <FIG>, in use, the compute sled <NUM> may execute a method <NUM> for managing RDMA queue pair QoS. The method <NUM> begins in block <NUM>, in which the compute sled <NUM> determines that a virtual machine should be created. The compute sled <NUM> may determine that a virtual machine should be created by, e.g., receiving a command from a user or from an orchestrator sled <NUM>. The compute sled <NUM> may determine one or more quality of service parameters for the virtual machine. The compute sled <NUM> may determine QoS parameters for cache monitoring and usage in block <NUM>, may determine QoS parameters for memory bandwidth monitoring and usage in block <NUM>, and may determine QoS parameters for one or more RDMA queue pairs in block <NUM>. The QoS parameters for the RDMA queue pairs may be embodied as, e.g., a class of service ID, a weighting factor to associate with the RDMA queue pair, a bandwidth to allocate to the RDMA queue pair, etc. In some embodiments, the QoS parameters may indicate a minimum and a maximum bandwidth to be allocated to each RDMA queue pair. It should be appreciated that different QoS parameters may be associated with different RDMA queue pairs for the same virtual machine. The various QoS parameters may be determined in any suitable way, such as being given QoS parameters to use from the orchestrator server <NUM>, which may stores the QoS parameters to use for various virtual machines in a table.

In block <NUM>, the compute sled <NUM> creates the virtual machines on the compute sled <NUM>. It should be appreciated that, in come embodiments, the virtual machine may include various resources on other sleds, such as storage resources on a storage sled <NUM> and/or accelerator resources on an accelerator sled <NUM>.

In block <NUM>, the compute sled <NUM> creates one or more RDMA queue pairs for the virtual machine. The compute sled <NUM> may create RDMA queue pairs for communication with other compute sleds <NUM> in block <NUM>. In the illustrative embodiment, each RDMA queue pair is assigned to a different VXLAN. Additionally or alternatively, some or all of the RDMA queue pairs may be assigned to the same VXLAN. The compute sled <NUM> may create RDMA queue pairs for communication with storage sleds <NUM> in block <NUM>. The compute sled <NUM> may create RDMA queue pairs for communication with accelerator sleds <NUM> in block <NUM>. The compute sled <NUM> may communicate with remote devices such as storage sleds <NUM>, accelerator sleds <NUM>, or other compute sleds <NUM> to coordinate creation of corresponding RDMA queue pairs on the remote devices.

In block <NUM>, the compute sled <NUM> allocates resources to the virtual machine based on the QoS parameters. The compute sled <NUM> may allocate cache of the processor <NUM> in block <NUM>. The compute sled <NUM> may memory bandwidth in block <NUM>. The compute sled <NUM> may allocate RDMA queue pair bandwidth in block <NUM>. It should be appreciated that the various RDMA queue pairs of the virtual machine may have different bandwidth allocated to them. The RDMA queue pair bandwidth may be allocated in any suitable manner, such as by assigning the RDMA queue pairs to a class of service or by associating a weighting to be used in a weighted round robin. In the illustrative embodiment, the compute sled <NUM> may allocate RDMA queue pair bandwidth to an RDMA queue pair on the network interface controller <NUM> by executing a special opcode on the processor <NUM> that configures the network interface controller <NUM> accordingly.

Referring now to <FIG>, in block <NUM>, the compute sled <NUM> operates the virtual machine. In block <NUM>, the compute sled <NUM> services the various RDMA queue pairs. The compute sled <NUM> is configured to send data or commands placed in each send RDMA queue to the corresponding receive RDMA queue on a remote device. The compute sled <NUM> is also configured to place packets sent from a send queue on a remote device in the corresponding receive RDMA queue. The compute sled <NUM> may process the receive RDMA queue by, e.g., writing data to the memory <NUM> and/or reading data from the memory <NUM> to be placed in the corresponding send RDMA queue. In the illustrative embodiment, the compute sled <NUM> may service the various RDMA queues using a weighted round robin approach, where the weighting is determined by a QoS parameter for each RDMA queue pair. Additionally or alternatively, other approaches may be used, such as using weighted fair queuing or applying an amount of bandwidth determined by a QoS parameter for each RDMA queue pair.

In block <NUM>, the compute sled <NUM> monitors the RDMA flows and associated bandwidth usage to generate telemetry data of the RDMA flows. The compute sled <NUM> may monitor various parameters such as flow rate, number of flows, flow types, applications associated with flows, the type of device on the other end of the flow, etc. In block <NUM>, in the illustrative embodiment, the compute sled sends the telemetry data of the RDMA flow from the compute sled <NUM> to the orchestrator server <NUM>.

In block <NUM>, in the illustrative embodiment, the compute sled <NUM> and/or the orchestrator server <NUM> analyzes the telemetry data of the RDMA flows. In the illustrative embodiment, the orchestrator server <NUM> determines whether each RDMA flow is an elephant flow or a mouse flow. The orchestrator server <NUM> may analyze an amount of traffic in a certain amount of time, and, if a flow has an amount of traffic over a certain threshold, the RDMA flow may be classified as an elephant flow. The amount of time over which the traffic amount is evaluated may be any suitable time, such as <NUM>, <NUM>, or <NUM> milliseconds or <NUM>, <NUM>, <NUM>, or <NUM> seconds. The threshold amount of traffic that triggers an RDMA flow to be classified as an elephant flow may be any suitable value, such as <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>,<NUM> KB.

In block <NUM>, the orchestrator server <NUM> sends an indication of whether the RDMA flow is an elephant or mouse flow to the compute sled <NUM>. It should be appreciated that, in some embodiments, the analysis of whether the RDMA flow is an elephant or mouse flow may be done by the compute sled <NUM> without necessarily involving the orchestrator server <NUM>.

In block <NUM>, the compute sled <NUM> predicts the future RDMA queue pair bandwidth usage. The compute sled <NUM> may do any suitable analysis to predict future RDMA queue pair bandwidth usage. In block <NUM>, the compute sled <NUM> may predict future RDMA queue pair bandwidth usage based on the indication of whether an RDMA flow is an elephant flow or a mouse flow. If an RDMA flow is marked as an elephant flow, in some embodiments, the compute sled <NUM> may predict that future bandwidth for the corresponding RDMA queue pair will increase. In block <NUM>, the compute sled <NUM> may perform a machine-learning-based algorithm to predict future RDMA queue pair bandwidth usage. Any suitable parameter may be provided as an input to the machine-learning based algorithm, such as flow rate, number of flows, flow types, applications associated with flows, the type of device on the other end of the flow, etc..

Claim 1:
A compute sled (<NUM>) for queue pair management comprising:
virtual machine manager circuitry (<NUM>) configured to:
create a virtual machine to operate on the compute sled; and
create each of a plurality of remote direct memory access, RDMA, queue pairs for the virtual machine; and
resource controller circuitry (<NUM>) configured to:
assign a different quality of service, QoS, parameter to each of the plurality of RDMA queue pairs for the virtual machine;
allocate bandwidth to each of the plurality of RDMA queue pairs for the virtual machine based on the corresponding QoS parameter;
monitor telemetry data associated with at least one flow through the plurality of RDMA queue pairs; and
update bandwidth allocation to one or more of the plurality of RDMA queue pairs based on the monitored telemetry data indicating whether the at least one flow is an elephant flow or a mouse flow.