High-availability (HA) management networks for high performance computing platforms

Embodiments of systems and methods for high-availability (HA) management networks for High Performance Computing (HPC) platforms are described. In some embodiments, an HPC platform may include a first Baseboard Management Controller (BMC) having a first network port; and a hardware accelerator comprising a second BMC having a second network port, where at least one of: (a) the first BMC is configured to share the first network port with the second BMC in response to a determination that the second network port has failed or has insufficient bandwidth, or (b) the second BMC is configured to share the second network port with the first BMC in response to a determination that the first network port has failed or has insufficient bandwidth.

FIELD

The present disclosure relates generally to Information Handling Systems (IHSs), and more particularly, to systems and methods for high-availability (HA) management networks for High Performance Computing (HPC) platforms.

BACKGROUND

As the value and use of information continue to increase, individuals and businesses seek additional ways to process and store it. One option available to users is Information Handling Systems (IHSs). An IHS generally processes, compiles, stores, and/or communicates information or data for business, personal, or other purposes thereby allowing users to take advantage of the value of the information. Because technology and information handling needs and requirements vary between different users or applications, IHSs may also vary regarding what information is handled, how the information is handled, how much information is processed, stored, or communicated, and how quickly and efficiently the information may be processed, stored, or communicated.

Variations in IHSs allow for IHSs to be general or configured for a specific user or specific use such as financial transaction processing, airline reservations, enterprise data storage, or global communications. In addition, IHSs may include a variety of hardware and software components that may be configured to process, store, and communicate information and may include one or more computer systems, data storage systems, and networking systems.

SUMMARY

Embodiments of systems and methods for high-availability (HA) management networks for High Performance Computing (HPC) platforms are described. In an illustrative, non-limiting embodiment, an HPC platform may include a first Baseboard Management Controller (BMC) having a first network port; and a hardware accelerator comprising a second BMC having a second network port, where at least one of: (a) the first BMC is configured to share the first network port with the second BMC in response to a determination that the second network port has failed or has insufficient bandwidth, or (b) the second BMC is configured to share the second network port with the first BMC in response to a determination that the first network port has failed or has insufficient bandwidth.

At least one of: (a) the first BMC is configured to instantiate a first bridge coupled to a first external management connection established over the first network port, or (b) the second BMC is configured to instantiate a second bridge coupled to a second external management connection established over the second network port. The first and second bridges may be teamed.

The HPC platform may include another hardware accelerator comprising a third BMC having a third network port, where at least one of the first or second BMCs is configured to share at least one of the first or second network ports with the third BMC in response to a determination that the third network port has failed or has insufficient bandwidth, and where the third BMC is configured to share the third network port with at least one of the first or second BMCs in response to a determination that at least one of the first or second network ports has failed or has insufficient bandwidth. The first, second, and third bridges may be teamed.

The hardware accelerator comprises one or more managed subsystems coupled to the second BMC. The one or more managed subsystems may include at least one of: a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), an Intelligence Processing Unit (IPU), a Data Processing Unit (DPU), a Gaussian Neural Accelerator (GNA), an Audio and Contextual Engine (ACE), or a Vision Processing Unit (VPU).

At least one of the managed subsystems may include a third BMC having a third network port. At least one of the first or second BMCs may be configured to share at least one of the first or second network ports with the third BMC in response to a determination that the third network port has failed or has insufficient bandwidth. The third BMC may be configured to share the third network port with at least one of the first or second BMCs in response to a determination that at least one of the first or second network ports has failed or has insufficient bandwidth.

The first BMC may be configured to communicate with the second BMC over a Peripheral Component Interconnect (PCIe) link using Management Component Transport Protocol (MCTP) over PCIe Vendor-Defined Messages (VDM). The second BMC may be configured to allocate a first portion of bandwidth available via the second network port to the second BMC, and a second portion of the bandwidth available via the second network port to the first BMC. The first and second portions may be selected based upon a priority of the second BMC over the first BMC. Moreover, the second BMC may be configured to instruct the first BMC to pause communications in response to a determination of insufficient bandwidth for a total network traffic of the first and second BMCs.

In another illustrative, non-limiting embodiment, a system BMC of an HPC platform may have program instructions stored thereon that, upon execution, cause the system BMC to: determine that a network port has failed or has insufficient bandwidth; and in response to the determination, communicate with a remote manager using another network port of an accelerator tray BMC.

The system BMC may be coupled to a host processor of the HPC platform, wherein the accelerator tray BMC is decoupled from the host processor, and wherein the system BMC and tray BMCs are coupled to each other via a RMII-Based Transport (RBT), I2C, or Universal Serial Bus (USB) link. The system BMC may be configured to instantiate a first bridge coupled to a first external management connection established over the network port, where the accelerator tray BMC is configured to instantiate a second bridge coupled to a second external management connection established over the other network port, and wherein the first and second bridges are teamed.

In yet another illustrative, non-limiting embodiment, a method may include instantiating, by a first BMC of an HPC platform, a first bridge coupled to a first external management connection established over a first network port; instantiating, by a second BMC of the HPC platform, a second bridge coupled to a second external management connection established over a second network port; and in response to a determination that the first network port has failed or has insufficient bandwidth, establishing the first external management connection over the second network port.

One of the first or second BMCs may be coupled to a host processor of the HPC platform, another of the first or second BMCs may be decoupled from the host processor, and the first and second BMCs are coupled to each other via a PCIe, RBT, I2C, or USB link. The method may also include apportioning a bandwidth available through the second network port to the first BMC based, at least in part, upon a priority of first external management connection over the second external management connection.

DETAILED DESCRIPTION

As used herein, the term “High Performance Computing” or HPC generally refers to the practice of aggregating computing power in a way that delivers much higher performance than what is otherwise available in a conventional computer. Formerly the domain of specialists using proprietary supercomputers, recent advances in computing, networking, and storage technologies have made HPC systems available to a wider range of users and organizations.

To many organizations, computing workloads such as Artificial Intelligence (AI), Machine Learning (ML), data analytics, modeling, and simulation can be important sources of competitive advantage. As HPC systems become smaller, simpler, and less costly, enterprise Information Technology (IT) teams have begun adopting HPC platforms to provide the throughput and capacity needed to execute these types of workloads.

In contrast with conventional computers, an HPC platform typically includes, in addition to its main or host processor(s), multiple hardware accelerators. For example, an HPC platform may include a plurality of: Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), Intelligence Processing Units (IPUs), Data Processing Units (DPUs), Gaussian Neural Accelerators (GNAs), Audio and Contextual Engines (ACEs), Vision Processing Units (VPUs), etc. In some cases, one or more hardware accelerators may be virtualized (e.g., vGPU).

In some HPC platforms, additional GPUs may offload portions of a workload while the remainder of the workload runs on the host processor (also referred to as Central Processing Unit or “CPU”), improving overall application performance by at least an order of magnitude. In other HPC platforms, FPGAs may be used to execute certain types of algorithms up to 1,000× faster than traditional solutions with less CPU time consumed. In yet other HPC platforms, IPUs may provide massively parallel, low-precision, floating-point computing with more than 1,000 cores that communicate with each other to share the complex workloads required for machine learning.

Despite recent advances in HPC technology, the inventors hereof have recognized that the design, deployment, management, and use of HPC platforms still present unique technological challenges. For example, a hardware accelerator deployed within an HPC system may include its own Baseboard Management Controller (BMC). Yet, BMCs typically found in a conventional HPC platform are not generally made and/or intended to work together.

To address these, and other concerns, systems and methods described herein may integrate two or more BMCs, for example, to distribute the execution of management tasks among them. An Out-of-Band (OOB) management link may be provided for inter-BMC communications. In some cases, a network connection may be shared across different BMCs to create a High-Availability (HA) management network. Additionally, or alternatively, these systems and methods may provide for the intelligent management of hardware accelerator licenses, collection of telemetry data, and/or power throttling of various components of an HPC platform.

In this disclosure, an Information Handling System (IHS) may include any instrumentality or aggregate of instrumentalities operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, originate, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes. For example, an IHS may be a personal computer (e.g., desktop or laptop), tablet computer, mobile device (e.g., Personal Digital Assistant (PDA) or smart phone), server (e.g., blade server or rack server), a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price.

An IHS may include Random Access Memory (RAM), one or more processing resources such as a CPU or hardware or software control logic, Read-Only Memory (ROM), and/or other types of nonvolatile memory. Additional components of an IHS may include one or more disk drives, one or more network ports for communicating with external devices as well as various I/O devices, such as a keyboard, a mouse, touchscreen, and/or a video display. An IHS may also include one or more buses operable to transmit communications between the various hardware components.

FIG.1is a block diagram illustrating components of IHS100configured as an HPC platform according to some embodiments. As shown, IHS100includes one or more main or host processor(s)101, such as a CPU, that execute code retrieved from system memory105.

Although IHS100is illustrated with a single processor, other embodiments may include two or more processors, that may each be configured identically, or to provide specialized processing operations. Processor(s)101may include any processor capable of executing instructions, such as an Intel Pentium™ series processor or any general-purpose or embedded processors implementing any of a variety of Instruction Set Architectures (ISAs), such as the x86, POWERPC®, ARM®, SPARC®, or MIPS® ISAs, or any other suitable ISA.

In the embodiment ofFIG.1, processor(s)101includes integrated memory controller118that may be implemented directly within its circuitry. Alternatively, memory controller118may be a separate integrated circuit that is located on the same die as processor(s)101. Memory controller118may be configured to manage the transfer of data to and from system memory105of IHS100via high-speed memory interface104.

System memory105is coupled to processor(s)101and provides processor(s)101with a high-speed memory that may be used in the execution of computer program instructions. For example, system memory105may include memory components, such as static RAM (SRAM), dynamic RAM (DRAM), NAND Flash memory, suitable for supporting high-speed memory operations by the processor101. In certain embodiments, system memory105may combine both persistent, non-volatile, and volatile memor(ies). In certain embodiments, system memory105may include multiple removable memory modules.

IHS100utilizes chipset103that may include one or more integrated circuits coupled to processor(s)101. In this embodiment, processor(s)101is depicted as a component of chipset103. In other embodiments, all of chipset103, or portions of chipset103may be implemented directly within the integrated circuitry of processor(s)101. Chipset103provides processor(s)101with access to a variety of resources accessible via bus102.

In IHS100, bus102is illustrated as a single element. However, other embodiments may utilize any number of separate buses to provide the illustrated pathways served by bus102.

In various embodiments, IHS100may include one or more I/O ports116that may support removeable couplings with various types of external devices and systems, including removeable couplings with peripheral devices that may be configured for operation by a particular user of IHS100. For instance, I/O116ports may include USB (Universal Serial Bus) ports, by which a variety of external devices may be coupled to IHS100. In addition to, or instead of USB ports, I/O ports116may include various types of physical I/O ports that are accessible to a user via an enclosure or chassis of IHS100.

In certain embodiments, chipset103may additionally utilize one or more I/O controllers110that may each support the operation of hardware components such as user I/O devices111. User I/O devices111may include peripheral components that are physically coupled to I/O port116and/or peripheral components wirelessly coupled to IHS100via network interface109.

In various implementations, I/O controller110may support the operation of one or more user I/O devices110such as a keyboard, mouse, touchpad, touchscreen, microphone, speakers, camera and other input and output devices that may be coupled to IHS100. User I/O devices111may interface with an I/O controller110through wired or wireless couplings supported by IHS100. In some cases, I/O controllers110may support configurable operation of supported peripheral devices, such as user I/O devices111.

As illustrated, a variety of additional resources may be coupled to processor(s)101of IHS100through chipset103. For instance, chipset103may be coupled to network interface109to enable different types of network connectivity. IHS100may also include one or more Network Interface Controllers (NICs)122and123, each of which may implement the hardware required for communicating via a specific networking technology, such as Wi-Fi, BLUETOOTH, Ethernet and mobile cellular networks (e.g., CDMA, TDMA, LTE).

Network interface109may support network connections by wired network controller(s)122and wireless network controller(s)123. Each network controller122and123may be coupled via various buses to chipset103to support different types of network connectivity, such as the network connectivity utilized by IHS100.

Chipset103may also provide access to one or more display device(s)108and/or113via graphics processor(s)107. Graphics processor(s)107may be included within a video card, graphics card, and/or an embedded controller installed within IHS100. Additionally, or alternatively, graphics processor(s)107may be integrated within processor(s)101, such as a component of a system-on-chip (SoC). Graphics processor(s)107may generate display information and provide the generated information to display device(s)108and/or113.

One or more display devices108and/or113are coupled to IHS100and may utilize LCD, LED, OLED, or other display technologies (e.g., flexible displays, etc.). Each display device108and113may be capable of receiving touch inputs such as via a touch controller that may be an embedded component of the display device108and/or113or graphics processor(s)107, for example, or may be a separate component of IHS100accessed via bus102. In some cases, power to graphics processor(s)107, integrated display device108and/or external display133may be turned off or configured to operate at minimal power levels in response to IHS100entering a low-power state (e.g., standby).

As illustrated, IHS100may support integrated display device108, such as a display integrated into a laptop, tablet, 2-in-1 convertible device, or mobile device. IHS100may also support use of one or more external displays113, such as external monitors that may be coupled to IHS100via various types of couplings, such as by connecting a cable from the external display113to external I/O port116of the IHS100, via wireless docking station, etc. In certain scenarios, the operation of integrated displays108and external displays113may be configured for a particular user. For instance, a particular user may prefer specific brightness settings that may vary the display brightness based on time of day and ambient lighting conditions.

Chipset103also provides processor(s)101with access to one or more storage devices119. In various embodiments, storage device119may be integral to IHS100or may be external to IHS100. Moreover, storage device119may be accessed via a storage controller that may be an integrated component of the storage device.

Generally, storage device119may be implemented using any memory technology allowing IHS100to store and retrieve data. For instance, storage device119may be a magnetic hard disk storage drive or a solid-state storage drive. In certain embodiments, storage device119may be a system of storage devices, such as a cloud system or enterprise data management system that is accessible via network interface109.

As illustrated, IHS100also includes Basic Input/Output System (BIOS)117that may be stored in a non-volatile memory accessible by chipset103via bus102. Upon powering or restarting IHS100, processor(s)101may utilize BIOS117instructions to initialize and test hardware components coupled to the IHS100. Under execution, BIOS117instructions may facilitate the loading of an operating system (OS) (e.g., WINDOWS, MACOS, iOS, ANDROID, LINUX, etc.) for use by IHS100.

BIOS117provides an abstraction layer that allows the operating system to interface with the hardware components of the IHS100. The Unified Extensible Firmware Interface (UEFI) was designed as a successor to BIOS. As a result, many modern IHSs utilize UEFI in addition to or instead of a BIOS. As used herein, BIOS is intended to also encompass UEFI.

Certain IHS100embodiments may utilize sensor hub114(e.g., INTEL Sensor Hub or “ISH,” etc.) capable of sampling and/or collecting data from a variety of hardware sensors112. In certain embodiments, sensor hub114may be an independent microcontroller or other logic unit that is coupled to the motherboard of IHS100. Sensor hub114may be a component of an integrated SoC incorporated into processor(s)101, and it may communicate with chipset103via a bus connection such as an Inter-Integrated Circuit (I2C) bus or other suitable type of bus connection. Sensor hub114may also utilize an I2C bus for communicating with various sensors supported by IHS100.

As illustrated, IHS100includes BMC155to provide capabilities for remote monitoring and management of various aspects of IHS100. In support of these operations, BMC155may utilize both in-band and sideband/OOB communications with certain managed components of IHS100, such as, for example, processor(s)101, system memory105, network controller109, storage device(s)119, BIOS117, and/or sensors112.

BMC155may be installed on the motherboard of IHS100or may be coupled to IHS100via an expansion slot provided by the motherboard. As a non-limiting example of a BMC, the integrated Dell Remote Access Controller (iDRAC) from Dell® is embedded within Dell PowerEdge™ servers and provides functionality that helps information technology (IT) administrators deploy, update, monitor, and maintain servers remotely.

BMC155may operate from a different power plane from processor(s)101, storage devices119, network controller109and various other components of IHS100, thus allowing the BMC155to operate, and management tasks to proceed, while the processing cores of processor(s)101are powered off In some embodiments, BMC155may control what OS BIOS117launches, by setting BIOS boot options that tell BIOS117where to load and launch the OS from. In some embodiments, BMC155may also perform various operations to verify the integrity of IHS100and its hardware components prior to initialization of IHS100(i.e., in a bare-metal state).

BMC155may support monitoring and administration of managed components via a sideband bus interface. For instance, messages utilized in device management may be transmitted using I2C sideband bus connections that may be individually established with each of managed component through the operation of I2C multiplexer155a. Managed components may communicate with the OS of IHS100via in-band buses supported by chipset103, while the sideband buses are used exclusively for communications with BMC155.

In certain embodiments, service processor155dof BMC155may rely on I2C co-processor155cto implement sideband communications between the BMC155and the managed components of the IHS100. I2C co-processor155cmay be a specialized co-processor or microcontroller that is configured to interface via a I2C bus interface with the managed components. In some cases, I2C co-processor155cmay be an integrated component of service processor155d, such as a peripheral SoC feature that may be provided by service processor155d. However, each I2C bus may be comprised of a clock line and data line that couple BMC155an I2C Endpoint on each of the managed components.

As illustrated, I2C co-processor155cmay interface with the individual managed components via individual sideband I2C buses selected through the operation of I2C multiplexer155a. Switching operations by I2C multiplexer155amay establish a sideband bus connection through a direct coupling between I2C co-processor155cand each individual managed component of IHS100. In providing sideband management capabilities, I2C co-processor155cmay interoperate with corresponding Endpoint I2C controllers that implement the I2C communications of the respective IHS components. Endpoint I2C controllers may be implemented as dedicated microcontrollers for communicating sideband I2C messages with BMC155or may be integrated into a processor in its respective endpoints.

In various embodiments, chipset103may provide processor101with access to hardware accelerator(s)125. Instances of accelerator(s)125include, but are not limited to, GPUs, FPGAs, IPUs, GNAs, ACEs, VPUs, etc. For example, hardware accelerator(s)125may be configured to execute HPC and/or AI/ML workloads offloaded by processor101and may be disposed on an accelerator tray deployed within the chassis of IHS100. In cases where one or more hardware accelerator(s)125have their own BMC(s), BMC155may be referred to as “system BMC155.”

In various embodiments, IHS100may not include each of the components shown inFIG.1. Moreover, IHS100may include various other components in addition to those that are shown inFIG.1. Some components that are represented as separate components inFIG.1may be integrated with other components. For example, in some implementations, all or a portion of the features provided by the illustrated components may instead be provided by an SoC.

BMC Integration

In modern IHS architectures, system BMC155is the focal point of systems management activities. When one or more hardware accelerator(s)125(e.g., a complex adapter card, a multi-processor tray, etc.) are deployed in an HPC platform, however, each hardware accelerator may have its own, independent BMC. Typically, these independent BMCs are not accessible to processor(s)101and there is no integration with system BMC155.

To address these shortcomings, embodiments described herein provide systems and methods for integrating a second BMC (or any additional number of BMCs) with system BMC155, such that the second BMC(s) may be configured to act as an extension of system BMC155and/or to offload management tasks (e.g., firmware updates, real-time monitoring, telemetry, etc.) while maintaining a single point of control.

For example, in a particular HPC implementation where an accelerator tray having a plurality of hardware accelerator devices or components (e.g., GPUs, FPGAs, IPUs, GNAs, ACEs, VPUs, etc.) and a tray BMC is deployed along with a high-speed management link between system BMC155and the tray BMC, system BMC155may communicate with tray BMC rather than having to access each accelerator device individually. Additionally, or alternatively, the tray BMC may be assigned, by system BMC155, local control, monitoring, and/or pre-processing of selected types of data (e.g., telemetry data).

FIG.2is a block diagram illustrating an example of a BMC integration system in HPC platform200. In this non-limiting embodiment, system BMC155may own the device management of HPC platform200while tray BMC203may act as a bridge.

As shown, service processor155dof system BMC155includes Peripheral Component Interconnect Express (PCIe) Endpoint204, Management Component Transport Protocol (MCTP) over PCIe Vendor-Defined Messages (VDM) initiator engine205, Video Graphics Array (VGA) and shared memory206, BMC PCIe Root Complex207, and PCIe enumerator/MCTP initiator208, each of which may be implemented, for example, as an IP core within an SoC. PCIe Endpoint204is coupled to Root Complex209of chipset103for communication with processor(s)101for in-band communications.

Accelerator tray201is an example of hardware accelerator(s)125in the context of HPC platform100. Particularly, accelerator tray201may include a plurality of managed subsystems202A-N (e.g., GPUs, VPUs, GNAs, FPGAs, IPUs, etc.) and a single, independent tray BMC203. Each of managed subsystems202A-N may include its own in-band PCIe connection to chipset103via Root Complexes210.

In contrast with managed subsystems202A-N, tray BMC203lacks access to Root Complexes210within chipset103and therefore sits outside of the host's PCIe hierarchy. Tray BMC203may include PCIe Endpoint211, MCTP bridge213, BMC MCTP Endpoint213, and MCTP over I2C or VDM interfaces214to managed subsystems202A-N.

MCTP bridge213may enable system BMC155to communicate directly with managed subsystems202A-N on accelerator tray201. Particularly, PCIe VDM messages include routing information, and bridge knows the address of tray BMC MCTP Endpoint213, as well as the addresses of managed subsystems202A-N and/or interfaces214. Accordingly, MCTP bridge213may operate as a transparent bridge to allow MCTP messages to be properly routed between system BMC155and managed subsystems202A-N.

System BMC155may target BMC MCTP Endpoint213to communicate with tray BMC203over MCTP bridge213. In response, tray BMC203may expose its directly managed devices, such as fan(s), memor(ies), etc., to system BMC155, as well as managed subsystems202A-N, so that system BMC155may control one or more such devices. System BMC155is coupled to tray BMC203via a high-speed, OOB management link, as BMC PCIe Root Complex207is coupled to PCIe Endpoint211via a PCIe connection.

FIG.3is a flowchart illustrating an example of method of BMC integration300. In some embodiments, method300may be performed, at least in part, by HPC platform200. Particularly, method300starts at301. At302, system BMC155discovers accelerator tray201, for example, while in the S5 Advanced Configuration and Power Interface (ACPI) power state. At303, system BIOS117enumerates the PCIe hierarchy and assigns Bus, Device, Function numbers (e.g., each a 16-bit number) to all PCIe Endpoints.

At304, system BMC155acts as MCTP Bus Owner and assigns Endpoint IDs (EIDs) to devices local to platform200and on acceleratory tray201. At305, tray BMC203acts as MCTP bridge providing, to system BMC155, connectivity to managed subsystems202A-N. Then, at306, tray BMC203may perform endpoint-to-endpoint tasks with managed subsystems202A-N for telemetry and/or other purposes. Method300ends at307.

FIG.4is a block diagram illustrating an example of another BMC integration system in HPC platform400. In this non-limiting embodiment, system BMC155may own a portion of the device management of platform400while tray BMC203owns another portion, thus acting as peer (or near-peer) BMCs. In contrast with platform200ofFIG.2, here tray BMC203includes Local Management MCTP Bus Owner401coupled to BMC MCTP Endpoint213.

In some embodiments, selected aspects of HPC platforms300and400may be combined in a single implementation. For example, Local Management MCTP Bus Owner401may be coupled to a first subset of managed subsystems202A-N, and MCTP bridge212may be coupled to a second subset of managed subsystems202A-N via MCTP over I2C or VDM interfaces214.

This may be useful, for example, when the first subset of managed subsystems202A-N is intended to be kept private or not directly accessible to system BMC155, and the second subset of managed subsystems202A-N is intended to be kept public, or otherwise directly accessible to system BMC155. Additionally, or alternatively, this may be useful in situations where the availability of unique PCIe addresses is restricted, such that more of (or all) managed subsystems202A-N are made accessible by system BMC155only through tray BMC203. In some cases, the type of connection between PCIe Endpoint211and managed subsystems202A-N within tray BMC203may be configurable and/or user selectable.

FIG.5is a flowchart illustrating an example of another method of BMC integration500. In some embodiments, method500may be performed, at least in part, by HPC platform400. Specifically, method500starts at501. At502, system BMC155discovers accelerator tray201. At503, system BIOS117enumerates the PCIe hierarchy and assigns Bus, Device, Function numbers (e.g., 16-bit) to PCIe Endpoints.

At504, system BMC155acts as MCTP Bus Owner and assigns Endpoint IDs (EIDs) to its directly managed devices. Tray BMC203does the same with respect to its managed subsystems202A-N. At505, tray BMC203performs one or more management tasks standalone or in conjunction with system BMC155, leaving system BMC155in charge of allocating and/or scheduling tasks between them, for example, based on context (e.g., more or fewer delegated tasks depending upon the type of workload being executed by managed subsystems202A-N, resources being consumed by managed subsystems202A-N, amount of data being collected from managed subsystems202A-N, etc.). For example, system BMC155may assign tray BMC203the task of compressing telemetry data collected from managed subsystems202A-N, deploying firmware updates to managed subsystems202A-N, etc.

At506, tray BMC203may aggregate and report out telemetry data to system BMC155, for example, to be combined with other platform telemetry. At507, tray BMC503may report selected management data from its managed devices to system BMC155for overall HPC platform400control. Method500ends at508.

FIG.6is a block diagram illustrating an example of BMC OS agent601deployed as a management link between system BMC155and tray BMC203in single HPC platform600. In some embodiments, when a high-speed, OOB link between BMC PCIe Root Complex207and PCIe Endpoint211(as previously discussed inFIGS.2and/or4) is not working properly, BMC OS agent601may provide a fail-over, in-band communication link between system BMC155and tray BMC203to relay management data.

Particularly, BMC OS agent601may serve as a passthrough connection by reading MCTP packets (or other suitable protocol) from one BMC and sending it to the other BMC. In some cases, an Intelligent Platform Management Interface (IPMI) or Redfish Application Programming Interface (API) command may be defined to encapsulate and transport messages and responses between the two BMCs.

FIG.7is a block diagram illustrating an example of BMC OS agents601A-N deployed as management links between system BMCs155A-N and tray BMCs203A-N across a plurality of HPC platforms600A-N. In some embodiments, system BMC755of head node701(e.g., in a server rack) may enable communications between BMC OS agents601A-N to relay management data across different HPC platforms600A-N.

As such, systems and methods described herein may employ MCTP over PCIe VDM (or other high bandwidth interface) for inter-BMC communications. Additionally, or alternatively, these systems and methods may use an adapter or tray BMC PCIe as a management terminus and/or an MCTP bridge for directly managed subsystems on an accelerator adapter, card, or tray. Additionally, or alternatively, these systems and methods may use an accelerator adapter, card, or tray BMC as an aggregator to pre-process and/or compress large amounts of telemetry data generated locally.

Additionally, or alternatively, these systems and methods may use an accelerator adapter, card, or tray BMC as an MCTP bridge to translate the high-speed PCI VDM interface to another, potentially slower management interfaces of its managed subsystems. Additionally, or alternatively, these systems and methods may use an accelerator adapter, card, or tray BMC to offload sideband firmware updates from the system BMC/update agent for managed subsystems.

Additionally, or alternatively, these systems and methods may use an accelerator adapter, card, or tray BMC in peer-to-peer mode with the system BMC in a shared management architecture. Additionally, or alternatively, these systems and methods may use an OS agent for inter-BMC communications, for example, as a fallback when the direct link between two or more BMCs is not available or operational.

High-Speed, OOB Management Links for Inter-BMC Communications

In an HPC platform, consider that the telemetry of as few as eight GPUs202A-N may require more bandwidth than the total bandwidth available via conventional I2C or I3C links. In many implementations (e.g., HPC platforms200and400), however, tray BMC203is outside chipset103's PCIe hierarchy and therefore it does not have access to PCIe resources otherwise available to system BMC155. Accordingly, under a new paradigm where system BMC155and tray BMC203are integrated and configured to communicate data (e.g., telemetry data) directly to each other, a high-speed inter-BMC connection is needed.

To address these, and other issues, systems and methods described herein may provide a high-speed, OOB management link between system BMC155and tray BMC203. The high-speed, OOB management link may be used, for example, as a sideband interface to managed subsystems202A-N, for localized Root-of-Trust (RoT) domain attestation, distributed operation of tray sensors, etc.

FIG.8is a block diagram illustrating an example of a high-speed, OOB management link between BMCs155and203in HPC platform800. In some embodiments, system BMC155of HPC platform800may own the OOB hierarchy, including direct bridges.

As shown in HPC platform800, tray201includes BMC203with BMC PCIe Endpoint211. Tray FPGA801includes PCIe bridge802coupled to BMC PCIe Endpoint211, and PCIe Endpoint803coupled to BMC PCIe Root Complex207. PCIe Endpoint (mailbox for PCIe/MCTP to SPI/I2C bridges)804couples PCIe bridge802to managed subsystems202A-N, and it includes logic configured to convert MCTP messages into low level SPI/I2C messages, and vice versa. In other implementations, however, because BMC PCIe Root Complex207of system BMC155is coupled directly to PCIe Endpoint807in tray201, the use of MCTP is not required.

FIG.9is a flowchart illustrating an example of method900for establishing and maintaining a high-speed, OOB management link between BMCs. In some embodiments, method900may be performed, at least in part, by HPC platform800. Specifically, method900starts at901. At902, system BMC155discovers accelerator tray201. At903, system BIOS117enumerates the PCIe hierarchy and assigns Bus, Device, Function numbers to PCIe bridge802, SPI/I2C bridges804, and BMC PCIe Endpoint211.

At904, in this implementation, system BMC155sends MCTP Route to ID messages to tray BMC203. At905, tray BMC203sends MCTP responses to the Bus Owner on OOB the PCIe network. At906, system BMC155directly enumerates SPI/I2C bridge Endpoint804for direct OOB communications with managed subsystems202A-N.

At907, tray BMC203sends Route to ID MCTP messages to SPI/I2C bridge802. PCIe Endpoint804translates MCTP messages to direct SPI/I2C bridge accesses to managed subsystems202A-N and converts responses from managed subsystems202A-N to the appropriate Route to Bus Owner (tray BMC203) or PCI configuration/memory/IO reads/writes back to system BMC155. Method900ends at908.

FIG.10is a block diagram illustrating another example of a high-speed, OOB management link between BMCs155and203in HPC platform1000. In some embodiments, BMC integration in HPC platform1000may include dual OOB Root Complex messaging via non-transparent bridging and BMC proxy.

PCIe to SPI/I2C bridges1002couple PCIe bridge802to managed subsystems202A-N. To access or communicate with managed subsystems202A-N, system BMC155may send MCTP messages to tray BMC203, which may then act as a proxy with respect to managed subsystems202A-N and forwards those messages (and responses) to SPI/I2C devices1002.

FIG.11is a flowchart illustrating an example of method1100for establishing and maintaining a high-speed, OOB management link between BMCs. In some embodiments, method1100may be performed, at least in part, by HPC platform1000. Specifically, method1100starts at1101. At1102, system BMC155discovers accelerator tray201.

As such, systems and methods described herein may use a BMC Root Complex as a virtual MCTP Bus Owner. Additionally, or alternatively, these systems and methods may use an add-in subsystem BMC's PCIe Root Complex and an MCTP target via non-transparent bridging and cross-route-to-root message routing. Additionally, or alternatively, these systems and methods may use tray BMC203for local peripheral communication via MCTP, along with a PCIe Endpoint that translates MCTP messages to local peripheral status/controls, and to round peripheral responses appropriate to either MCTOP via Route to Bus Owner or to the true root (system BMC155) via PCI transactions (e.g., config/mem/IO R/W, etc.).

Additionally, or alternatively, these systems and methods may enable selectively making devices (e.g., managed subsystems202A-N) behind each BMC private or public (e.g., based on context, type of container executing an application, type of application, type of workload requested by an application, etc.), such that private devices are not discoverable by other BMCs while public devices may be. For example, tray BMC203may execute filtering or access control algorithms that determine which of managed subsystems202A-N can be managed or accessed by system BMC155based on a policy (e.g., current context, workloads being executed, etc.).

High-Availability Management Network

Each BMC has a single, discrete/dedicated physical network port for management connections (e.g., via an Ethernet connector). If the management connection fails because of a malfunction of the BMC's network interface controller or switch port, for example, the BMC cannot be remotely managed. This concern applies equally to system BMC155, tray BMC203, or any other peripheral device BMC in an HPC platform.

In many cases, however, internal connections may exist between discrete BMCs. These connections may include, for example, I2C, PCIe VDM, USB, RMII-Based Transport (RBT), etc. As noted above, two BMCs can use these interconnects as well as one or more high-speed OOB management links to communicate status information and other management operations between system BMC155and tray BMC203(or other peripheral BMCs).

Accordingly, in various embodiments, systems and methods described herein may enable a first BMC in an HPC platform to use a second BMC's management connection for external communications with a remote manager, for example, when the first BMC's management connection fails or has insufficient bandwidth. To achieve high availability, BMCs in an HPC platform may be configured to bridge their external management connections to a channel/tunnel on internal interconnects, and then team/bond this bridge with channels/tunnels that are bridged by other BMCs to their own external management connections.

When a BMC's network fails and it starts using another BMC's network, there may not be sufficient bandwidth to handle traffic between the two BMCs. In some cases, a traffic priority mechanism may be implemented such that a first BMC can instruct a second BMC to pause sending packets out using the first BMC's network while the first BMC is handling high-priority traffic. In those cases, telemetry and other streamed data from the second BMC may be paused until the first BMC gives the second BMC permission to transmit using the first BMC's network.

Alternatively, if the pause frame method is not implemented or not honored, the BMC sharing the uplink can shape the traffic using traffic control, and it may drop packets from the other BMC when they exceed an allowed rate (leaky bucket). For Transmission Control Protocol (TCP) communications, the BMC may automatically slow down its transmission.

In some implementations, priority of access to a given management network may be configurable. For example, priority may be given to the BMC that has not lost its network. Alternatively, a greater amount of bandwidth may be allocated to whichever BMC and/or external management connection is considered higher priority, regardless of whether its network connection has failed or not, for example, based upon context (e.g., an ongoing telemetry operation by either BMC, an ongoing live-monitoring operation by either BMC, an ongoing firmware update operation by either BMC, etc.).

FIG.12is a block diagram illustrating examples of internal and external connections of a pair of BMCs155and203in HPC platform1200. As shown, system BMC155is coupled to network switch1201via external management network port1202(e.g., Ethernet). Meanwhile, tray BMC203of accelerator tray201is coupled to network switch1201via its own external management network port1203. System BMC155and tray BMC203are directly coupled to each other via internal bus1204(e.g., RBT, I2C, USB, etc.).

In normal operation, system BMC155and tray BMC103have independent network connectivity and internal bus1204is used for inter-BMC communications only. In response to network port1202not working, however, system BMC155may use tray BMC103's network port1203for external communications and vice-versa. This may be achieved, for instance, by bridging and teaming if bus1204between system BMC155and tray BMC103is Ethernet-capable. Otherwise, system BMC155may encapsulate its network traffic into MCTP packets to tray BMC203, and tray BMC203may send those packets out over network port1203and vice-versa.

FIG.13is a flowchart illustrating an example of management network topology1300for establishing and maintaining a HA network among BMCs1301A-N. In some embodiments, BMC1301A may be implemented as system BMC155, BMC1301B may be implemented as tray BMC203, and BMC1301N may be implemented as another BMC (e.g., another tray BMC, a BMC coupled to one of GPUs202A-N, etc.). Each of BMCs1301A-N is coupled to management TOR (The Onion Router).

Each of BMCs1301A-N may share its uplink with other BMCs using bridges1302A-N, respectively. Moreover, BMCs1301A-N may be integrated and/or in communication with each other via a high-speed, OOB management link using techniques described above, and the bridging may take place over that link. In the previous example, when network port1202not working system BMC155may send a notification to switch1201, via BMC2, to update a forwarding table or database (e.g., of IP and/or MAC addresses, etc.) to start redirecting, to tray BMC203, packets intended to system BMC155.

In some implementations, internal connections from each BMC may be a separate Ethernet interface, or VLAN (or other network overlay) with isolated broadcast domains. For HA and loop prevention, each of BMCs1301A-N may create a respective active/standby bond-team1303A-N for its respective uplink bridge1302A-N and internal connections with uplink as preferred. Moreover, in some cases, L2 pause frames and/or traffic control mechanisms may be employed to prioritize and shape traffic, for example, to maintain a Quality-of-Service (QoS) metric, to allocate a portion of the bandwidth (e.g., 20%) to the BMC that owns the external connection, to throttle another BMC's external communications based on priority, etc.

As such, systems and methods described herein may provide redundancy to a BMC network without adding a second dedicated network port to any given BMC. Additionally, or alternatively, these systems and methods may provide a priority mechanism for sharing a BMC network port when another BMC's network port has failed. Additionally, or alternatively, these systems and methods may enable a BMC to use another one or more BMC network port(s), in the absence of failure, for example, for additional communication bandwidth. Additionally, or alternatively, these systems and methods may prevent loops while all operational links are used because they are primary in an associated BMC NIC teaming. Additionally, or alternatively, these systems and methods may use Address Resolution Protocol (ARP) polling to send traffic over internal tunnels if an associated external link is operational.

Accelerator License Management

Software tools and intellectual property (IP) cores often require a license to operate. A license entitlement may be set up, for example, after a customer's order has been processed, and it may be managed by a license manager configured to generate a license.dat file. This file enables a customer to use the software or product licensed under the terms of the purchase. In some cases, a license maybe certificate-based for security purposes.

As hardware accelerators expand the types of applications they support, it becomes important for each device to adjust and tune their capabilities depending upon the type of workload(s) they are executing. Some workloads may require advanced features and others may require basic features. Prior to executing a workload, a hardware accelerator may perform a license verification operation whereby it may determine, from a larger set of potentially available features, which subset of features it has been allowed to use during execution of the workload (e.g., number of cores that can be active, available software capabilities, etc.).

In an HPC platform, the large number of individual accelerators and accelerator cores can make the license management process difficult. For example, although application and/or workload requirements may change dynamically, at least insofar as different applications can be executed by the same HPC platform, there is presently no mechanism capable of matching workload requirements to licenses in real time. These shortcomings become even more apparent when an HPC platform is made available “as-a-service,” such that users/customers may lease the HPC platform from a service provider who remains in control of the usage and monetization of the HPC platform's accelerators with a chosen level of granularity.

To address these, and other concerns, systems and methods described herein may create a flexible licensing model that supports dynamic changes to licenses based upon workload requirements. In such a model, hardware accelerator licenses may be dynamic selected based upon the workload, type of workload, and/or workload requirements (e.g., as identified by the application requesting execution of the workload). For example, system BMC155, tray BMC203, and/or license manager1405may generate a list or queue of workloads, or types of workloads, and/or workload requirements. Each workload (or type of workload, etc.) may be associated with a best available license to optimize performance (e.g., speed of execution) and/or cost depending upon a customer's priorities.

For example, a configuration option may be provided, via system BMC155and/or tray BMC203, for a customer to set a limit on a particular license activation based on: time (e.g., a maximum of 2 hours a day; usable only during a certain time of day; etc.); cost (e.g., the user selects a per hour/daily/weekly/monthly/yearly costs and the BMC determines how long a license can be active to respect the user's limits, etc.); workload (e.g., a computer vision workload requires compression, filtering, decoding, and inferencing; based on these requirements, a license can be spun up to allocate an instance of CPU core, storage, and hardware accelerator slice—such as Multi-Instance GPU (MIG)); and/or power consumption (e.g., a license can be used to maintain power below the maximum power limit of the HPC platform, and/or to select a maximum number of accelerators or cores usable to execute workloads).

In that regard, Table I shows a non-limiting example of workload and cost mapping table usable by system BMC155and/or tray BMC203to effect licensing allocation determinations:

Using Table I, for example, each workload to be executed may be classified as belonging to a particular workload type, for example, using a best fit approach, and each particular workload type may be associated with a distinct license type, with its own cost per hour/day/etc.

FIG.14is a block diagram illustrating an example of BMC-based accelerator license management system1400deployed in HPC platform1406. In some embodiments, system1400may be used to push licenses with respect to GPUs1403A-N to system BMC155and/or GPU BMC1402within HPC platform1406.

Particularly, HPC platform1406includes system BMC155and GPU subsystem1401(e.g., accelerator tray201). GPU subsystem1401includes GPU BMC1402(e.g., tray BMC203) and GPUs1403A-N (e.g., managed subsystems202A-N). System BMC155and/or GPU BMC1402may each have independent access to license manager service1405via network1404(e.g., the Internet). In some cases, system BMC155and/or GPU BMC1402may be integrated and/or in communication with each other via a high-speed, OOB management link using one or more of the techniques described above.

FIG.15is a flowchart illustrating an example of method1500for BMC-based accelerator license management. In some embodiments, method1500may be performed, at least in part, by BMC-based accelerator license management system1400ofFIG.14. Particularly, method1500begins at1501. At1502, a customer or user orders the execution of one or more workloads as-a-service (e.g., on a local HPC platform, on the cloud, etc.).

At1503, the service provider deploys the workload(s) and identifies the best-fit GPU license(s) for each workload, which may be based upon the type of workload and/or the user's preferences or settings (e.g., cost vs. speed, cost vs. fidelity, daily cost/usage limits, etc.). Examples of types of workloads may include, but are not limited to, visualization workloads (related to graphics processing), AI/ML workloads (e.g., training, inferencing, etc.), data compression/decompression workloads, data encryption/decryption workloads, or more generally any type of high performance, computationally intensive workloads.

In an alternative embodiment, system BMC155may determine the best license based on the workload, and it may request that license from license manager1405. For example, system BMC155may use Table I that maps workload types (column3) to license types (column1).

At1504, license manager1405transmits the selected or desired license(s) (e.g., a certificate, file, encryption key, etc.) to GPU BMC1402over network1404via side-band communications. Alternatively, license manager1405may push the license(s) to system BMC155over network1404, and system BMC155may validate and send the received license(s) to GPU BMC1402for enforcement during execution of one or more workload(s). This may be implemented, for example, in response to GPU BMC1402not being connected to network1404. Moreover, in response to system BMC155not being connected to network1404, license manager1405may push the license(s) to GPU BMC1402over network1404, GPU BMC1402may send the license(s) to system BMC155, and system BMC155may validate the license(s) and send them back to GPU BMC1402for enforcement. In yet other implementations, GPU BMC1402may be capable of validating license(s) on its own.

At1505, GPU BMC1402configures the license(s) on a respective one or more of GPUs1403A-N by turning on/off one or more features (e.g., selected clock frequencies, turbo modes, allowing or forbidding utilization peaks for certain durations, enabling more or fewer cores, etc.), for example, based on a policy (e.g., to enforce or activate the license when certain contextual information is met, such as time-of-day, geographic location, user proximity, calendar event, etc.). At1506, after the workload(s) are executed, GPU BMC1402removes the license(s) and informs license manager1405that the license(s) are free or available. In some cases, still at1506, GPU BMC1402may de-energize or power down one or more idle cores until a new workload is deployed. Method1500ends at1507.

As such, systems and methods described herein may provide for GPU subsystem1401(or accelerator tray201, etc.) to operate as a license manager/enforcer with respect to local resources1403A-N. Additionally, or alternatively, these systems and methods may provide for system BMC155to serve as a proxy license provisioner to GPU BMC1402. Additionally, or alternatively, these systems and methods may: dynamically adjust custom-built licenses while workloads are composed or queued for execution (which generally enables an HPC platform to operate initially or by default with under-provisioned licenses), de-energize and/or throttle specific accelerators1403A-N or cores when not in use, free licenses no longer in use, cap license usage based on desired maximum costs, map license types/capabilities for workload types, and/or gather licensing statistics (e.g., usage for auditing purposes).

Additionally, or alternatively, with respect to license usage, GPU BMC1402may be configured to release a license if idle or not used for a pre-configured amount of time. For example, an ML workload may be launched that is designed to predict license usage, and the results of the ML workload may be used to change the pre-configured amount of time after which GPU BMC1402releases a particular license, for example, based on contextual information. Moreover, GPU BMC1402may select PCI power excursion sub-groups (e.g., among accelerators1403A-N) based upon peak utilization models, or the like.

Prioritized Power Management

In conventional implementations, when the power consumption of an HPC platform exceeds a power limit (e.g., set by the user as a power cap, dictated by a PSU rating, etc.), its CPU is deeply throttled. Deep throttling brings the whole HPC platform to a crawling state with up to a 95% reduction in performance. Types of events that can trigger throttling include the detection of: a high output power supply current, a power supply input undervoltage, and/or an over temperature condition. After the event, the HPC platform's control logic may progressively reduce the amount of throttling applied to the CPU as part of a recovery process.

With the development of PCIe accelerators and add-in cards that consume 600 W or more, and with the growing the number of PCIe lanes in each HPC platform, however, the amount of power consumed by accelerator components now far exceeds a CPU's. Particularly when accelerator components are the primary contributor(s) of power excursion/throttling events, the traditional approach of slowing down the entire HPC platform, including its CPU, is ineffective, and it becomes important to manage the power consumption at the individual device level.

To address these, and other concerns, embodiments of systems and methods described herein may enable system BMC155and/or tray BMC203to read dynamic power consumption telemetry from managed subsystems202A-N. For example, with the support of a high-speed, OOB management link between system BMC155and/or tray BMC203, system BMC155may receive real-time power consumption information from managed subsystems202A-N, and it may use that information to determine how much each individual subsystem and/or core is contributing to the overall HPC platform's power load. Then, system BMC155may instruct tray BMC203to throttle managed subsystems202A-N, at least in part, in a manner proportional to their respective contributions.

In some implementations, system BMC155and/or tray BMC203may individually throttle any offending subsystems (e.g., the n larger contributors to the power excursion/throttling event) without affecting the power allocated to the HPC platform's CPU and/or memory complex. Additionally, or alternatively, system BMC155and/or tray BMC203may trigger the migration of one or more workloads from an offending subsystem to a non-offending subsystem. In some cases, system BMC155and/or tray BMC203may manage the amount of power allocated to each subsystem, for example, based upon a license and/or policy.

FIG.16is a block diagram illustrating an example of system1600for BMC-based power throttling of accelerators. In some embodiments, system1600may include system BMC155, Complex Programmable Logic Device (CPLD)1601, CPU101, and tray BMC203. System BMC155is configured to establish CPU data path1604with CPU101, and MCTP over PCIe VDM/SMBUS or direct power monitoring channel or link1605with tray BMC203. System BMC155may receive throttle event interrupt1606from CPLD1601, for example, in response to a determination that a power budget has been exceed for HPC platform1600.

System BMC155may communicate with CPLD1601over Serial Peripheral Interface (SPI) bus1602, for example. Moreover, CPLD1601may be configured to issue processor hot (PROCHOT) command1607(e.g., in Linux) to throttle CPU101and/or one or more power reduction (PWRBRK) command(s)1608(e.g., via a PCIe pin) to tray BMC203to prioritize the amount of throttling of each device or managed subsystem (e.g.,202A-N) in response to power excursion events where the total power consumed by HPC platform1600exceeds a maximum limit. In some cases, system BMC155may use System Management Interrupt (SMI) bus1603to communicate with system BIOS117via CPU101to change set a PCIe slot's power limit.

FIG.17is a flowchart illustrating an example of method1700for BMC-based power throttling of accelerators. In some embodiments, method1700may be performed, at least in part, by system1600ofFIG.16. Particularly, method1700begins at1701.

At1702, system BMC155monitors HPC platform1600's system power consumption (total power consumed) and CPU/memory/storage/fan contributions. At1703, system BMC155monitors the live power consumption of each of GPUs1403A-N, for example, via link1605with tray BMC203. System BMC155may track the consumption % and/or ratio of each subsystem (1403A-N) with respect to the overall, total power. System BMC155may also store the expected throttled power consumption of each subsystem. In some cases, system BMC155may apply an upper bound to each of the subsystem (e.g., based on live consumption data and per-policy for each subsystem, to ensure each subsystem is allotted a “do not exceed” power limit at or below their theoretical maximum, depending upon policy).

At1704, when the total power of blocks1702and1703exceed a maximum allowed amount of power consumption for HPC platform1600, system BMC155applies PROCHOT command1807and PWRBRK command1808.

At1705, system BMC155applies a user policy to prioritize the delivery of power to CPU101and/or specific ones of GPU slots1403A-N. For example, when GPU slots1403A-N are prioritized over CPU101, system BMC155identifies the specific adapter which is consuming addition power and changes the power limit on that specific card. I/O card throttling methods may include, for example: (a) pulse-width modulating (PWM) PWRBRK commands1608(and varying the duty cycle, etc.) transmitted to tray BMC203and/or individually/directly to each of managed subsystems202A-N on a per-slot basis; and/or (b) decoding a programmable throttling level for each GPU slot1403A-N based upon messages encoded in PWRBRK commands1608. At1706, system BMC155de-asserts the throttling of HPC platform1600, and method1700ends at1707.

In various embodiments, system BMC155may enable setting a power limit at each card, in addition to the HPC platform1600's level power cap (e.g., automatically or through user action, such as via an API call). To set or modify an add-in card slot power limit, BMC155may communicate with BIOS117over SMI1603. In addition to, or as an alternative to continuous polling by system BMC155(which can miss peaks if not performed at a sufficiently high frequency), each of GPUs1403A-N may be configured to alert system BMC155, through tray BMC203, when a programmable power threshold has been reached or exceeded.

Moreover, a policy may allow certain subsystems not to throttle, to be the last to throttle, etc., for example, depending upon a type or priority of workloads assigned to those devices. Additionally, or alternatively, a subsystem priority indication may be identified by system BMC155, for example, based upon a policy, and system BMC155may send that information to a locally executed and/or remotely located workload manager configured to assign one or more high-priority workload(s) to one or more last-to-be-throttled subsystem(s), and/or one or more low-priority workload(s) to other subsystem(s), to reduce the likelihood of performance reduction.

FIG.18is a chart illustrating example use-case1800of an application of systems and methods for BMC-based power throttling of accelerators. Initial state1802shows the power consumption of storage system1803, memory1804, CPU1805, and GPU slots1806and1807(1403A-N), which add up to more than system power limit1801(e.g., 1,800 W). State1808(“before”) shows the power consumed by storage system1809, memory1810, CPU1811, and GPU slots1812and1813being reduced under power limit1801but also deeply throttling CPU1811(e.g., by issuing a PWRBRK command with respect to all slots or PCIe Endpoints1812and1813and PROCHOT with respect to CPU1811).

In contrast, state1814(“after”) shows the total power consumption of storage1815, memory1816, CPU1817, and GPU slots1818and1819after implementation of systems and method described herein, which maintain the performance/consumption of CPU1805and reduces the performance/consumption of I/O slot1819while keeping the total power consumption under power limit1801. The maximum allowable power consumption of I/O slot1819may be reduced over I/O slot1818based on a policy, license, etc.

For example, a user may configure system BMC155via a policy to prioritize the throttling of I/O slot1819(priority=1, first to throttle, no less than 50%) over CPU1817(priority=2, second to throttle, for example, if the first throttling is insufficient to reduce the total power consumption below the power limit), and of CPU1817over I/O slot1818(priority=3, third to throttle, for example, if the second throttling is still insufficient to reduce the total power consumption below the power limit). Additionally, or alternatively, the user may configure system BMC155to reduce the power consumption of any given device by at least a first selected amount (e.g., at least a 25% reduction from a nominal or measured value) and/or by no more than a second selected amount (e.g., no more than a 50% reduction from a nominal or measured value).

In some cases, system BMC155may prioritize the throttling of individual devices without direct user input, for example, based upon a priority of a workload executed, a type of workload executed, a device, a type of device (e.g., GPU vs. VPU), and/or a license. For instance, a license may indicate throttling parameters to be applied to a given managed subsystem202A-N (e.g., relative to a CPU and/or other subsystems) in the case of power excursion events.

As such, systems and methods described herein may enable controlling throttling effects to target specific I/O subsystems (e.g., slot/GPU on an accelerator tray), separate from the CPU complex. For example, system BMC155may be configured to track the relative contributions of each I/O element separate from the CPU, memory, storage, along with expected responses to throttling events. Additionally, or alternatively, these systems and methods may enable configuring a PCIe Endpoint (add-in card or accelerator on a tray) to have dynamically programmable responses to PWRBRK (beyond static emergency power level reductions), including changes to power exclusions, set limits (e.g., 50%), etc.

In some cases, systems and methods may support an API to dynamically redirect throttling levels to different PCIe Endpoints (e.g., slots, accelerators on a given tray, etc.) for example, based on workload priority, managed subsystem priority, etc. Such an API may also enable system BMC155to communicate, to each PCIe Endpoint, what custom action to take in response to receiving a PWRBRK command. Moreover, these systems and methods may enable endpoints to alert a BMC when slot power limits and/or programmable PWRBRK responses see threshold crossings.

Telemetry for AI/ML Workloads

In various embodiments, systems and methods described herein may be implemented in enterprise environments, where AI and HPC platforms are being deployed at a fast pace and organizations are attempting to monetize them. Currently, when an IT department deploys an AI platform in its datacenter, they lack insight into accelerator (e.g., GPU, etc.) utilization, efficiency, and/or allocation (e.g., based workload requirement and GPU scheduling). Moreover, IT departments also lack the ability to “charge back” their customers based on compute resource used.

As more organizations continue to adopt “as a service” business models, having a single plane of glass into all accelerator resources becomes more important. To address these, and other concerns, systems and methods described herein may enable the gathering and analysis of telemetry to enable a user and/or an IT department to determine, for example, when a workspace is launched, how long the workspace will run, what its resource utilization will be, and/or how to charge the customer back.

FIG.19is a block diagram illustrating an example of AI/HPC enterprise platform1900suitable for employing an AI/ML telemetry system. In some cases, head node100may be implemented as any of the IHSs or HPC platforms described herein. User terminal and/or portal1901(e.g., a Jupyter notebook, a user's IHS, a web portal, etc.) receives ML workloads from a user and sends those ML workloads to head node100, which are then managed by orchestrator or workload manager1902(e.g., a Bright Cluster Manager or the like).

To execute ML workloads, head node100is coupled to HPC resources1906A-N (e.g., other HPC platforms, hardware accelerators, acceleratory trays, etc.) via switch1903(e.g., an InfiniBand switch or the like). In some cases, head node100may be coupled to Network File System (NFS)1904directly and/or to Network-attached storage (NAS)1905via switch1603.

FIG.20is a block diagram illustrating an example of AI/ML telemetry system2000. In some embodiments, telemetry system2000may be implemented in HPC platform1900ofFIG.19. Particularly, telemetry collector agents2006A-N (e.g., implemented in one or more BMCs) may be in communication with managers2007A (e.g., NVIDIA Data Center GPU Manager or “DCGM,” etc.), which are in turn configured to receive telemetry and/or usage data from GPUs2003A-N in each respective one of HPC resources2002A-N (e.g.,1906A-N inFIG.19).

Allocator agents2004A-N of management nodes2001A-N (e.g., each node as an instance of1900/100inFIG.19) are in communication with telemetry collector agents2006A-N via telemetry monitors2005A-N (e.g., Prometheus as a monitoring and alerting toolkit that scrapes the metrics exported from every collector agent2006A-N and places the data in a time series database that can be read by any suitable tool for interactive visualization of data, for example, to produce data dashboards, etc.).

In operation, telemetry system2000acts as a shim layer that sits between orchestrator1902and HPC resources1906A-N. In various embodiments, telemetry system2000may be configured to collect telemetry data from HPC resources1906A-N, run analytics on the telemetry data, provide usage statistics (e.g., in the form of visual dashboards, etc.) to an IT administrator, automatically allocate resources based on telemetry data, and/or calculate resource allocation based on ML workload profiles.

For example, telemetry collector agents2006A-N may be configured to collect data, per individual HPC resource (e.g., trays, accelerators, GPUs, cores, etc.), including, but not limited to: power consumption, operating temperature, memory usage, core load/utilization, disk usage, and network usage, etc. Additionally, or alternatively telemetry collector agents2006A-N may be configured to collect information usable to characterize aspects of workload(s) queued for execution, workload(s) currently in execution, workload(s) completed, workload(s) successfully completed, and associated performance metrics (e.g., workload execution time, etc.).

Meanwhile, allocator agents2004A-N may be configured to aggregate the telemetry data from each of collector agents2006A-N to determine, for example, how long it will take a particular workload or type of workload to be executed by a given one of GPUs2003A-N for workload allocation purposes. Additionally, or alternatively, allocator agents2004A-N may determine that a particular HPC resource is underutilized despite having a been assigned a given workload and, in response, orchestrator1902may deny a user access to the HPC resource. Additionally, or alternatively, allocator agents2004A-N may be configured to determine a node-level resource usage based on ML workload ID and/or a charge back for each ML workload ID (e.g., based upon the amount or resources and/or time spent).

Based on system power usage, allocator agents2004A-N may also determine infrastructure cost for power and/or cooling, which may in turn be used to determine, for example, based on the daily energy cost, when to deploy certain types of workloads with certain usage statistics. Moreover, allocator agents2004A-N may also be configured to determine, based upon the telemetry data, whether a workload should be allocated on-premises and/or on the cloud; which may be particularly useful, for example, in situations where large AI/ML training workloads are deployed.

AlthoughFIG.20shows allocator agents2004A-N running on management nodes2001A-N, in other implementations agents2004A-N may reside on the cloud, or in any suitable location from where they can access telemetry collector agents2006A-N.

As such, systems and methods described herein may provide collector agents configured to run on a smart controller and collect data from various APIs on compute resources to enable the operation of, for example, full charge back and/or resource allocation models. Additionally, or alternatively, these systems and methods may provide allocator agents configured to provide infrastructure-level service such as power allocation, cooling allocation, ML workload allocation, and/or resource allocation (e.g., at the cluster level).

Additionally, or alternatively, these systems and methods may enable the deployment of GPUs and other hardware accelerators of HPC platforms using an “as a service” model. Additionally, or alternatively, these systems and methods may allow improved prediction of compute resources or instances for IT administrators. Additionally, or alternatively, these systems and methods may enable GPUs for AI/ML workloads to be dis-aggregated since a collector may be the only service running on each GPU node. Additionally, or alternatively, allocator and/or collector agents may be configured to deploy AI privacy techniques to prevent un-authorized use of models and/or data.