Patent Description:
However, as the densities of networked storage systems and remote computing systems increase, various physical limitations can be reached. These limitations include density limitations based on the underlying storage technology, such as in the example of large arrays of rotating magnetic media storage systems. These limitations can also include computing or data processing density limitations based on the various physical space requirements for data processing equipment and network interconnect, as well as the large space requirements for environmental climate control systems. In addition to physical space limitations, these data systems have been traditionally limited in the number of devices that can be included per host, which can be problematic in environments where higher capacity, redundancy, and reliability is desired. These shortcomings can be especially pronounced with the increasing data storage and processing needs in networked, cloud, and enterprise environments.

<CIT> discloses a method and apparatus for providing peer-to-peer data transfer through an interconnecting fabric. The method and apparatus enable a first device to read and/or write data to/from a local memory of a second device by communicating read and write requests across the interconnectivity fabric.

Disaggregated computing architectures, platforms, and systems are provided herein. A method of operating a data processing system is provided as disclosed in claim <NUM> and its dependent claims. A data processing system is provided as disclosed in claim <NUM> and its dependent claims. A computer readable storage media is provided as disclosed in claim <NUM>.

<FIG> is a system diagram illustrating computing platform <NUM>. Computing platform <NUM> includes one or more management processors, <NUM>, and a plurality of physical computing components. The physical computing components include processors <NUM>, storage elements <NUM>, network elements <NUM>, Peripheral Component Interconnect Express (PCIe) switch elements <NUM>, and graphics processing units (GPUs) <NUM>. These physical computing components are communicatively coupled over PCIe fabric <NUM> formed from PCIe switch elements <NUM> and various corresponding PCIe links. PCIe fabric <NUM> configured to communicatively couple a plurality of plurality of physical computing components and establish compute blocks using logical partitioning within the PCIe fabric. These compute blocks, referred to in <FIG> as machine(s) <NUM>, can each be comprised of any number of processors <NUM>, storage units <NUM>, network interfaces <NUM> modules, and GPUs <NUM>, including zero of any module.

The components of platform <NUM> can be included in one or more physical enclosures, such as rack-mountable units which can further be included in shelving or rack units. A predetermined number of components of platform <NUM> can be inserted or installed into a physical enclosure, such as a modular framework where modules can be inserted and removed according to the needs of a particular end user. An enclosed modular system, such as platform <NUM>, can include physical support structure and enclosure that includes circuitry, printed circuit boards, semiconductor systems, and structural elements. The modules that comprise the components of platform <NUM> are insertable and removable from a rackmount style of enclosure. In some examples, the elements of <FIG> are included in a chassis (e.g. 1U, 2U, or 3U) for mounting in a larger rackmount environment. It should be understood that the elements of <FIG> can be included in any physical mounting environment, and need not include any associated enclosures or rackmount elements.

In addition to the components described above, an external enclosure can be employed that comprises a plurality of graphics modules, graphics cards, or other graphics processing elements that comprise GPU portions. In <FIG>, a just a box of disks (JBOD) enclosure is shown that includes a PCIe switch circuit that couples any number of included devices, such as GPUs <NUM>, over one or more PCIe links to another enclosure comprising the computing, storage, and network elements discussed above. The enclosure might not comprise a JBOD enclosure, but typically comprises a modular assembly where individual graphics modules can be inserted and removed into associated slots or bays. In JBOD examples, disk drives or storage devices are typically inserted to create a storage system. However, in the examples herein, graphics modules are inserted instead of storage drives or storage modules, which advantageously provides for coupling of a large number of GPUs to handle data/graphics processing within a similar physical enclosure space. In one example, the JBOD enclosure might include <NUM> slots for storage/drive modules that are instead populated with one or more GPUs carried on graphics modules. The external PCIe link that couples enclosures can comprise any of the external PCIe link physical and logical examples discussed herein.

Once the components of platform <NUM> have been inserted into the enclosure or enclosures, the components can be coupled over the PCIe fabric and logically isolated into any number of separate "machines" or compute blocks. The PCIe fabric can be configured by management processor <NUM> to selectively route traffic among the components of a particular compute module and with external systems, while maintaining logical isolation between components not included in a particular compute module. In this way, a flexible "bare metal" configuration can be established among the components of platform <NUM>. The individual compute blocks can be associated with external users or client machines that can utilize the computing, storage, network, or graphics processing resources of the compute block. Moreover, any number of compute blocks can be grouped into a "cluster" of compute blocks for greater parallelism and capacity. Although not shown in <FIG> for clarity, various power supply modules and associated power and control distribution links can also be included.

Turning now to the components of platform <NUM>, management processor <NUM> can comprise one or more microprocessors and other processing circuitry that retrieves and executes software, such as user interface <NUM> and management operating system <NUM>, from an associated storage system. Processor <NUM> can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processor <NUM> include general purpose central processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof. In some examples, processor <NUM> comprises an Intel or AMD microprocessor, ARM microprocessor, FPGA, ASIC, application specific processor, or other microprocessor or processing elements.

In <FIG>, processor <NUM> provides interface <NUM>. Interface <NUM> comprises a communication link between processor <NUM> and any component coupled to PCIe fabric <NUM>. This interface employs Ethernet traffic transported over a PCIe link. Additionally, each processor <NUM> in <FIG> is configured with driver <NUM> which provides for Ethernet communication over PCIe links. Thus, any of processor <NUM> and processor <NUM> can communicate over Ethernet that is transported over the PCIe fabric. A further discussion of this Ethernet over PCIe configuration is discussed below.

A plurality of processors <NUM> are included in platform <NUM>. Each processor <NUM> includes one or more microprocessors and other processing circuitry that retrieves and executes software, such as driver <NUM> and any number of end user applications, from an associated storage system. Each processor <NUM> can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of each processor <NUM> include general purpose central processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof. In some examples, each processor <NUM> comprises an Intel or AMD microprocessor, ARM microprocessor, graphics processor, compute cores, graphics cores, application specific integrated circuit (ASIC), or other microprocessor or processing elements. Each processor <NUM> can also communicate with other compute units, such as those in a same storage assembly/enclosure or another storage assembly/enclosure over one or more PCIe interfaces and PCIe fabric <NUM>.

A plurality of storage units <NUM> are included in platform <NUM>. Each storage unit <NUM> includes one or more storage drives, such as solid state drives in some examples. Each storage unit <NUM> also includes PCIe interfaces, control processors, and power system elements. Each storage unit <NUM> also includes an on-sled processor or control system for traffic statistics and status monitoring, among other operations. Each storage unit <NUM> comprises one or more solid state memory devices with a PCIe interface. In yet other examples, each storage unit <NUM> comprises one or more separate solid state drives (SSDs) or magnetic hard disk drives (HDDs) along with associated enclosures and circuitry.

A plurality of graphics processing units (GPUs) <NUM> are included in platform <NUM>. Each GPU comprises a graphics processing resource that can be allocated to one or more compute units. The GPUs can comprise graphics processors, shaders, pixel render elements, frame buffers, texture mappers, graphics cores, graphics pipelines, graphics memory, or other graphics processing and handling elements. In some examples, each GPU <NUM> comprises a graphics 'card' comprising circuitry that supports a GPU chip. Example GPU cards include nVidia Jetson or Tesla cards that include graphics processing elements and compute elements, along with various support circuitry, connectors, and other elements. Some example GPU modules also include CPUs or other processors to aid in the function of the GPU elements, as well as PCIe interfaces and related circuitry. GPU elements <NUM> can also comprise elements discussed above for GPUs <NUM>, and further comprise physical modules or carriers that are insertable into slots of bays of the associated JBOD or other enclosure.

Network interfaces <NUM> include network interface cards for communicating over TCP/IP (Transmission Control Protocol (TCP)/Internet Protocol) networks or for carrying user traffic, such as iSCSI (Internet Small Computer System Interface) or NVMe (NVM Express) traffic for storage units <NUM> or other TCP/IP traffic for processors <NUM>. Network interfaces <NUM> can comprise Ethernet interface equipment, and can communicate over wired, optical, or wireless links. External access to components of platform <NUM> is provided over packet network links provided by network interfaces <NUM>. Network interfaces <NUM> communicate with other components of platform <NUM>, such as processors <NUM> and storage units <NUM> over associated PCIe links and PCIe fabric <NUM>. In some examples, network interfaces are provided for intra-system network communication among for communicating over Ethernet networks for exchanging communications between any of processors <NUM> and processors <NUM>.

Each PCIe switch <NUM> communicates over associated PCIe links. In the example in <FIG>, PCIe switches <NUM> can be used for carrying user data between network interfaces <NUM>, storage units <NUM>, and processing units <NUM>. Each PCIe switch <NUM> comprises a PCIe cross connect switch for establishing switched connections between any PCIe interfaces handled by each PCIe switch <NUM>. In some examples, ones of PCIe switches <NUM> comprise PLX/Broadcom/Avago PEX8796 <NUM>-port, <NUM> lane PCIe switch chips, PEX8725 <NUM>-port, <NUM> lane PCIe switch chips, PEX97xx chips, PEX9797 chips, or other PEX87xx/PEX97xx chips.

The PCIe switches discussed herein can comprise PCIe crosspoint switches, which logically interconnect various ones of the associated PCIe links based at least on the traffic carried by each PCIe link. In these examples, a domain-based PCIe signaling distribution can be included which allows segregation of PCIe ports of a PCIe switch according to user-defined groups. The user-defined groups can be managed by processor <NUM> which logically integrate components into associated compute units <NUM> of a particular cluster and logically isolate components and compute units among different clusters. In addition to, or alternatively from the domain-based segregation, each PCIe switch port can be a non-transparent (NT) or transparent port. An NT port can allow some logical isolation between endpoints, much like a bridge, while a transparent port does not allow logical isolation, and has the effect of connecting endpoints in a purely switched configuration. Access over an NT port or ports can include additional handshaking between the PCIe switch and the initiating endpoint to select a particular NT port or to allow visibility through the NT port.

PCIe can support multiple bus widths, such as x1, x4, x8, x16, and x32, with each multiple of bus width comprising an additional "lane" for data transfer. PCIe also supports transfer of sideband signaling, such as System Management Bus (SMBus) interfaces and Joint Test Action Group (JTAG) interfaces, as well as associated clocks, power, and bootstrapping, among other signaling. Although PCIe is used in <FIG>, it should be understood that different communication links or busses can instead be employed, such as NVMe, Ethernet, Serial Attached SCSI (SAS), FibreChannel, Thunderbolt, Serial Attached ATA Express (SATA Express), among other high-speed serial near-range interfaces, various networks, and link interfaces. Any of the links in <FIG> can each use various communication media, such as air, space, metal, optical fiber, or some other signal propagation path, including combinations thereof. Any of the links in <FIG> can include any number of PCIe links or lane configurations. Any of the links in <FIG> can each be a direct link or might include various equipment, intermediate components, systems, and networks. Any of the links in <FIG> can each be a common link, shared link, aggregated link, or may be comprised of discrete, separate links.

In <FIG>, any compute module <NUM> has configurable logical visibility to any/all storage units <NUM> or GPU <NUM>/<NUM>, as segregated logically by the PCIe fabric. Any compute module <NUM> can transfer data for storage on any storage unit <NUM> and retrieve data stored on any storage unit <NUM>. Thus, 'm' number of storage drives can be coupled with 'n' number of processors to allow for a large, scalable architecture with a high-level of redundancy and density. Furthermore, any compute module <NUM> can transfer data for processing by any GPU <NUM>/191or hand off control of any GPU to another compute module <NUM>.

To provide visibility of each compute module <NUM> to any storage unit <NUM> or GPU <NUM>/<NUM>, various techniques can be employed. In a first example, management processor <NUM> establishes a cluster that includes one or more compute units <NUM>. These compute units comprise one or more processor <NUM> elements, zero or more storage units <NUM>, zero or more network interface units <NUM>, and zero or more graphics processing units <NUM>/<NUM>. Elements of these compute units are communicatively coupled by portions of PCIe fabric <NUM> and any associated external PCIe interfaces to external enclosures, such as JBOD <NUM>. Once compute units <NUM> have been assigned to a particular cluster, further resources can be assigned to that cluster, such as storage resources, graphics processing resources, and network interface resources, among other resources. Management processor <NUM> can instantiate/bind a subset number of the total quantity of storage resources of platform <NUM> to a particular cluster and for use by one or more compute units <NUM> of that cluster. For example, <NUM> storage drives spanning <NUM> storage units might be assigned to a group of two compute units <NUM> in a cluster. The compute units <NUM> assigned to a cluster then handle transactions for that subset of storage units, such as read and write transactions.

Each compute unit <NUM>, specifically a processor of the compute unit, can have memory-mapped or routing-table based visibility to the storage units or graphics units within that cluster, while other units not associated with a cluster are generally not accessible to the compute units until logical visibility is granted. Moreover, each compute unit might only manage a subset of the storage or graphics units for an associated cluster. Storage operations or graphics processing operations might, however, be received over a network interface associated with a first compute unit that are managed by a second compute unit. When a storage operation or graphics processing operation is desired for a resource unit not managed by a first compute unit (i.e. managed by the second compute unit), the first compute unit uses the memory mapped access or routing-table based visibility to direct the operation to the proper resource unit for that transaction, by way of the second compute unit. The transaction can be transferred and transitioned to the appropriate compute unit that manages that resource unit associated with the data of the transaction. For storage operations, the PCIe fabric is used to transfer data between compute units/processors of a cluster so that a particular compute unit/processor can store the data in the storage unit or storage drive that is managed by that particular compute unit/ processor, even though the data might be received over a network interface associated with a different compute unit/ processor. For graphics processing operations, the PCIe fabric is used to transfer graphics data and graphics processing commands between compute units/processors of a cluster so that a particular compute unit/processor can control the GPU or GPUs that are managed by that particular compute unit/ processor, even though the data might be received over a network interface associated with a different compute unit/ processor. Thus, while each particular compute unit of a cluster actually manages a subset of the total resource units (such as storage drives in storage units or graphics processors in graphics units), all compute units of a cluster have visibility to, and can initiate transactions to, any of resource units of the cluster. A managing compute unit that manages a particular resource unit can receive re-transferred transactions and any associated data from an initiating compute unit by at least using a memory-mapped address space or routing table to establish which processing module handles storage operations for a particular set of storage units.

In graphics processing examples, NT partitioning or domain-based partitioning in the switched PCIe fabric can be provided by one or more of the PCIe switches with NT ports or domain-based features. This partitioning can ensure that GPUs can be interworked with a desired compute unit and that more than one GPU, such as more than eight (<NUM>) GPUs can be associated with a particular compute unit. Moreover, dynamic GPU-compute unit relationships can be adjusted on-the-fly using partitioning across the PCIe fabric. Shared network resources can also be applied across compute units for graphics processing elements. For example, when a first compute processor determines that the first compute processor does not physically manage the graphics unit associated with a received graphics operation, then the first compute processor transfers the graphics operation over the PCIe fabric to another compute processor of the cluster that does manage the graphics unit.

In further examples, memory mapped direct memory access (DMA) conduits can be formed between individual CPU/GPU pairs. This memory mapping can occur over the PCIe fabric address space, among other configurations. To provide these DMA conduits over a shared PCIe fabric comprising many CPUs and GPUs, the logical partitioning described herein can be employed. Specifically, NT ports or domain-based partitioning on PCIe switches can isolate individual DMA conduits among the associated CPUs/GPUs.

In storage operations, such as a write operation, data can be received over network interfaces <NUM> of a particular cluster by a particular processor of that cluster. Load balancing or other factors can allow any network interface of that cluster to receive storage operations for any of the processors of that cluster and for any of the storage units of that cluster. For example, the write operation can be a write operation received over a first network interface <NUM> of a first cluster from an end user employing an iSCSI protocol or NVMe protocol. A first processor of the cluster can receive the write operation and determine if the first processor manages the storage drive or drives associated with the write operation, and if the first processor does, then the first processor transfers the data for storage on the associated storage drives of a storage unit over the PCIe fabric. The individual PCIe switches <NUM> of the PCIe fabric can be configured to route PCIe traffic associated with the cluster among the various storage, processor, and network elements of the cluster, such as using domain-based routing or NT ports. If the first processor determines that the first processor does not physically manage the storage drive or drives associated with the write operation, then the first processor transfers the write operation to another processor of the cluster that does manage the storage drive or drives over the PCIe fabric. Data striping can be employed by any processor to stripe data for a particular write transaction over any number of storage drives or storage units, such as over one or more of the storage units of the cluster.

In this example, PCIe fabric <NUM> associated with platform <NUM> has <NUM>-bit address spaces, which allows an addressable space of <NUM><NUM> bytes, leading to at least <NUM> exbibytes of byte-addressable memory. The <NUM>-bit PCIe address space can shared by all compute units or segregated among various compute units forming clusters for appropriate memory mapping to resource units. The individual PCIe switches <NUM> of the PCIe fabric can be configured to segregate and route PCIe traffic associated with particular clusters among the various storage, compute, graphics processing, and network elements of the cluster. This segregation and routing can be establishing using domain-based routing or NT ports to establish cross-point connections among the various PCIe switches of the PCIe fabric. Redundancy and failover pathways can also be established so that traffic of the cluster can still be routed among the elements of the cluster when one or more of the PCIe switches fails or becomes unresponsive. In some examples, a mesh configuration is formed by the PCIe switches of the PCIe fabric to ensure redundant routing of PCIe traffic.

Management processor <NUM> controls the operations of PCIe switches <NUM> and PCIe fabric <NUM> over one or more interfaces, which can include inter-integrated circuit (I2C) interfaces that communicatively couple each PCIe switch of the PCIe fabric. Management processor <NUM> can establish NT-based or domain-based segregation among a PCIe address space using PCIe switches <NUM>. Each PCIe switch can be configured to segregate portions of the PCIe address space to establish cluster-specific partitioning. Various configuration settings of each PCle switch can be altered by management processor <NUM> to establish the domains and cluster segregation. In some examples management processor <NUM> can include a PCIe interface and communicate/configure the PCIe switches over the PCIe interface or sideband interfaces transported within the PCIe protocol signaling.

Management operating system (OS) <NUM> is executed by management processor <NUM> and provides for management of resources of platform <NUM>. The management includes creation, alteration, and monitoring of one or more clusters comprising one or more compute units. Management OS <NUM> provides for the functionality and operations described herein for management processor <NUM>. Management processor <NUM> also includes user interface <NUM>, which can present a graphical user interface (GUI) to one or more users. User interface <NUM> and the GUI can be employed by end users or administrators to establish clusters, assign assets (compute units/machines) to each cluster. User interface <NUM> can provide other user interfaces than a GUI, such as command line interfaces, application programming interfaces (APIs), or other interfaces. In some examples, a GUI is provided over a websockets-based interface.

More than one more than one management processor can be included in a system, such as when each management processor can manage resources for a predetermined number of clusters or compute units. User commands, such as those received over a GUI, can be received into any of the management processors of a system and forwarded by the receiving management processor to the handling management processor. Each management processor can have a unique or pre-assigned identifier which can aid in delivery of user commands to the proper management processor. Additionally, management processors can communicate with each other, such as using a mailbox process or other data exchange technique. This communication can occur over dedicated sideband interfaces, such as I2C interfaces, or can occur over PCIe or Ethernet interfaces that couple each management processor.

Management OS <NUM> also includes emulated network interface <NUM>. Emulated network interface <NUM> comprises a transport mechanism for transporting network traffic over one or more PCIe interfaces. Emulated network interface <NUM> can emulate a network device, such as an Ethernet device, to management processor <NUM> so that management processor <NUM> can interact/interface with any of processors <NUM> over a PCIe interface as if the processor was communicating over a network interface. Emulated network interface <NUM> can comprise a kernel-level element or module which allows management OS <NUM> to interface using Ethernet-style commands and drivers. Emulated network interface <NUM> allows applications or OS-level processes to communicate with the emulated network device without having associated latency and processing overhead associated with a network stack. Emulated network interface <NUM> comprises a driver or module, such as a kernel-level module, that appears as a network device to the application-level and system-level software executed by the processor device, but does not require network stack processing. Instead, emulated network interface <NUM> transfers associated traffic over a PCIe interface or PCIe fabric to another emulated network device. Advantageously, emulated network interface <NUM> does not employ network stack processing but still appears as network device, so that software of the associated processor can interact without modification with the emulated network device.

Emulated network interface <NUM> translates PCIe traffic into network device traffic and vice versa. Processing communications transferred to the network device over a network stack is omitted, where the network stack would typically be employed for the type of network device/interface presented. For example, the network device might be presented as an Ethernet device to the operating system or applications. Communications received from the operating system or applications are to be transferred by the network device to one or more destinations. However, emulated network interface <NUM> does not include a network stack to process the communications down from an application layer down to a link layer. Instead, emulated network interface <NUM> extracts the payload data and destination from the communications received from the operating system or applications and translates the payload data and destination into PCIe traffic, such as by encapsulating the payload data into PCIe frames using addressing associated with the destination.

Management driver <NUM> is included on each processor <NUM>. Management driver <NUM> can include emulated network interfaces, such as discussed for emulated network interface <NUM>. Additionally, management driver <NUM> monitors operation of the associated processor <NUM> and software executed by processor <NUM> and provides telemetry for this operation to management processor <NUM>. Thus, any user provided software can be executed by each processor <NUM>, such as user-provided operating systems (Windows, Linux, MacOS, Android, iOS, etc.. ) or user application software and drivers. Management driver <NUM> provides functionality to allow each processor <NUM> to participate in the associated compute unit and/or cluster, as well as provide telemetry data to an associated management processor. Each processor <NUM> can also communicate with each other over an emulated network device that transports the network traffic over the PCIe fabric. Driver <NUM> also provides an API for user software and operating systems to interact with driver <NUM> as well as exchange control/telemetry signaling with management processor <NUM>.

<FIG> is a system diagram that includes further details on elements from <FIG>. System <NUM> includes a detailed view of an implementation of processor <NUM> as well as management processor <NUM>.

In <FIG>, processor <NUM> can be an exemplary processor in any compute unit or machine of a cluster. Detailed view <NUM> shows several layers of processor <NUM>. A first layer <NUM> is the hardware layer or "metal" machine infrastructure of processor <NUM>. A second layer <NUM> provides the OS as well as management driver <NUM> and API <NUM>. Finally, a third layer <NUM> provides user-level applications. View <NUM> shows that user applications can access storage, compute, graphics processing, and communication resources of the cluster, such as when the user application comprises a clustered storage system or a clustered processing system.

As discussed above, driver <NUM> provides an emulated network device for communicating over a PCIe fabric with management processor <NUM> (or other processor <NUM> elements). This is shown in <FIG> as Ethernet traffic transported over PCIe. However, a network stack is not employed in driver <NUM> to transport the traffic over PCIe. Instead, driver <NUM> appears as a network device to an operating system or kernel to each processor <NUM>. User-level services/applications/software can interact with the emulated network device without modifications from a normal or physical network device. However, the traffic associated with the emulated network device is transported over a PCIe link or PCIe fabric, as shown. API <NUM> can provide a standardized interface for the management traffic, such as for control instructions, control responses, telemetry data, status information, or other data.

<FIG> is s block diagram illustrating management processor <NUM>. Management processor <NUM> illustrates an example of any of the management processors discussed herein, such as processor <NUM> of <FIG>. Management processor <NUM> includes communication interface <NUM>, user interface <NUM>, and processing system <NUM>. Processing system <NUM> includes processing circuitry <NUM>, random access memory (RAM) <NUM>, and storage <NUM>, although further elements can be included.

Processing circuitry <NUM> can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing circuitry <NUM> include general purpose central processing units, microprocessors, application specific processors, and logic devices, as well as any other type of processing device. In some examples, processing circuitry <NUM> includes physically distributed processing devices, such as cloud computing systems.

Communication interface <NUM> includes one or more communication and network interfaces for communicating over communication links, networks, such as packet networks, the Internet, and the like. The communication interfaces can include PCIe interfaces, Ethernet interfaces, serial interfaces, serial peripheral interface (SPI) links, inter-integrated circuit (I2C) interfaces, universal serial bus (USB) interfaces, UART interfaces, wireless interfaces, or one or more local or wide area network communication interfaces which can communicate over Ethernet or Internet protocol (IP) links. Communication interface <NUM> can include network interfaces configured to communicate using one or more network addresses, which can be associated with different network links. Examples of communication interface <NUM> include network interface card equipment, transceivers, modems, and other communication circuitry.

User interface <NUM> may include a touchscreen, keyboard, mouse, voice input device, audio input device, or other touch input device for receiving input from a user. Output devices such as a display, speakers, web interfaces, terminal interfaces, and other types of output devices may also be included in user interface <NUM>. User interface <NUM> can provide output and receive input over a network interface, such as communication interface <NUM>. In network examples, user interface <NUM> might packetize display or graphics data for remote display by a display system or computing system coupled over one or more network interfaces. Physical or logical elements of user interface <NUM> can provide alerts or visual outputs to users or other operators. User interface <NUM> may also include associated user interface software executable by processing system <NUM> in support of the various user input and output devices discussed above. Separately or in conjunction with each other and other hardware and software elements, the user interface software and user interface devices may support a graphical user interface, a natural user interface, or any other type of user interface.

RAM <NUM> and storage <NUM> together can comprise a non-transitory data storage system, although variations are possible. RAM <NUM> and storage <NUM> can each comprise any storage media readable by processing circuitry <NUM> and capable of storing software. RAM <NUM> can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Storage <NUM> can include non-volatile storage media, such as solid state storage media, flash memory, phase change memory, or magnetic memory, including combinations thereof. RAM <NUM> and storage <NUM> can each be implemented as a single storage device but can also be implemented across multiple storage devices or sub-systems. RAM <NUM> and storage <NUM> can each comprise additional elements, such as controllers, capable of communicating with processing circuitry <NUM>.

Software stored on or in RAM <NUM> or storage <NUM> can comprise computer program instructions, firmware, or some other form of machine-readable processing instructions having processes that when executed a processing system direct processor <NUM> to operate as described herein. For example, software <NUM> can drive processor <NUM> to receive user commands to establish clusters comprising compute blocks among a plurality of physical computing components that include compute modules, storage modules, and network modules. Software <NUM> can drive processor <NUM> to receive and monitor telemetry data, statistical information, operational data, and other data to provide telemetry to users and alter operation of clusters according to the telemetry data or other data. Software <NUM> can drive processor <NUM> to manage cluster and compute/graphics unit resources, establish domain partitioning or NT partitioning among PCIe fabric elements, and interface with individual PCIe switches, among other operations. The software can also include user software applications, application programming interfaces (APIs), or user interfaces. The software can be implemented as a single application or as multiple applications. In general, the software can, when loaded into a processing system and executed, transform the processing system from a general-purpose device into a special-purpose device customized as described herein.

System software <NUM> illustrates a detailed view of an example configuration of RAM <NUM>. It should be understood that different configurations are possible. System software <NUM> includes applications <NUM> and operating system (OS) <NUM>. Software applications <NUM>-<NUM> each comprise executable instructions which can be executed by processor <NUM> for operating a cluster controller or other circuitry according to the operations discussed herein.

Specifically, cluster management application <NUM> establishes and maintains clusters and compute units among various hardware elements of a computing platform, such as seen in <FIG>. Cluster management application <NUM> can also provision/deprovision PCIe devices from communication or logical connection over an associated PCIe fabric, establish isolation functions to allow dynamic allocation of PCIe devices, such as GPUs, from one or more host processors. User interface application <NUM> provides one or more graphical or other user interfaces for end users to administer associated clusters and compute units and monitor operations of the clusters and compute units. Inter-module communication application <NUM> provides communication among other processor <NUM> elements, such as over I2C, Ethernet, emulated network devices, or PCIe interfaces. User CPU interface <NUM> provides communication, APIs, and emulated network devices for communicating with processors of compute units, and specialized driver elements thereof. PCIe fabric interface <NUM> establishes various logical partitioning or domains among PCIe switch elements, controls operation of PCIe switch elements, and receives telemetry from PCIe switch elements.

Software <NUM> can reside in RAM <NUM> during execution and operation of processor <NUM>, and can reside in storage system <NUM> during a powered-off state, among other locations and states. Software <NUM> can be loaded into RAM <NUM> during a startup or boot procedure as described for computer operating systems and applications. Software <NUM> can receive user input through user interface <NUM>. This user input can include user commands, as well as other input, including combinations thereof.

Storage system <NUM> can comprise flash memory such as NAND flash or NOR flash memory, phase change memory, resistive memory, magnetic memory, among other solid state storage technologies. As shown in <FIG>, storage system <NUM> includes software <NUM>. As described above, software <NUM> can be in a non-volatile storage space for applications and OS during a powered-down state of processor <NUM>, among other operating software.

Processor <NUM> is generally intended to represent a computing system with which at least software <NUM> is deployed and executed in order to render or otherwise implement the operations described herein. However, processor <NUM> can also represent any computing system on which at least software <NUM> can be staged and from where software <NUM> can be distributed, transported, downloaded, or otherwise provided to yet another computing system for deployment and execution, or yet additional distribution.

<FIG> is a flow diagram that illustrates operational examples for any of the systems discussed herein, such as for platform <NUM> of <FIG>, system <NUM> of <FIG>, or processor <NUM> of <FIG>. In <FIG>, operations will be discussed in context of elements of <FIG> and <FIG>, although the operations can also apply to elements of other Figures herein.

Management processor <NUM> presents (<NUM>) a user interface to a cluster management service. This user interface can comprise a GUI or other user interfaces. The user interface allows users to create clusters (<NUM>) and assign resources thereto. The clusters can be represented graphically according to what resources have been assigned, and can have associated names or identifiers specified by the users, or predetermined by the system. The user can then establish compute blocks (<NUM>) and assign these compute blocks to clusters. The compute blocks can have resource elements/units such as processing elements, graphics processing elements, storage elements, and network interface elements, among other elements.

Once the user specifies these various clusters and compute blocks within the clusters, then management processor <NUM> can implement (<NUM>) the instructions. The implementation can include allocating resources to particular clusters and compute units within allocation tables or data structures maintained by processor <NUM>. The implementation can also include configuring PCIe switch elements of a PCIe fabric to logically partition the resources into a routing domain for the PCIe fabric. The implementation can also include initializing processors, storage drives, GPUs, memory devices, and network elements to bring these elements into an operational state and associated these elements with a particular cluster or compute unit. Moreover, the initialization can include deploying user software to processors, configuring network interfaces with associated adresses and network parameters, and establishing partitions or logical units (LUNs) among the various storage elements. Once these resources have been assigned to the cluster/compute unit and initialized, then they can be made available to users for executing user operating systems, user applications, and for user storage processes, among other user purposes.

Additionally, as will be discussed below in <FIG>, multiple GPUs can be allocated to a single host, and these allocations can be dynamically changed/altered. Management processor <NUM> can control the allocation of GPUs to various hosts, and configures properties and operations of the PCIe fabric to enable this dynamic allocation. Furthermore, peer-to-peer relationships can be established among GPUs so that traffic exchanged between GPUs need not be transferred through an associated host processor, greatly increasing throughputs and processing speeds.

<FIG> illustrates continued operation, such as for a user to monitor or modify operation of an existing cluster or compute units. An iterative process can occur where a user can monitor and modify elements and these elements can be re-assigned, aggregated into the cluster, or disaggregated from the cluster.

In operation <NUM>, the cluster is operated according to user specified configurations, such as those discussed in <FIG>. The operations can include executing user operating systems, user applications, user storage processes, graphics operations, among other user operations. During operation, telemetry is received (<NUM>) by processor <NUM> from the various cluster elements, such as PCIe switch elements, processing elements, storage elements, network interface elements, and other elements, including user software executed by the computing elements. The telemetry data can be provided (<NUM>) over the user interface to the users, stored in one or more data structures, and used to prompt further user instructions (operation <NUM>) or to modify operation of the cluster.

The systems and operations discussed herein provide for dynamic assignment of computing resources, graphics processing resources, network resources, or storage resources to a computing cluster. The computing units are disaggregated from any particular cluster or computing unit until allocated by users of the system. Management processors can control the operations of the cluster and provide user interfaces to the cluster management service provided by software executed by the management processors. A cluster includes at least one "machine" or computing unit, while a computing unit include at least a processor element. Computing units can also include network interface elements, graphics processing elements, and storage elements, but these elements are not required for a computing unit.

Processing resources and other elements (graphics processing, network, storage) can be swapped in and out of computing units and associated clusters on-the-fly, and these resources can be assigned to other computing units or clusters. In one example, graphics processing resources can be dispatched/orchestrated by a first computing resource/CPU and subsequently provide graphics processing status/results to another compute unit/CPU. In another example, when resources experience failures, hangs, overloaded conditions, then additional resources can be introduced into the computing units and clusters to supplement the resources.

Processing resources can have unique identifiers assigned thereto for use in identification by the management processor and for identification on the PCIe fabric. User supplied software such as operating systems and applications can be deployed to processing resources as-needed when the processing resources are initialized after adding into a compute unit, and the user supplied software can be removed from a processing resource when that resource is removed from a compute unit. The user software can be deployed from a storage system that the management processor can access for the deployment. Storage resources, such as storage drives, storage devices, and other storage resources, can be allocated and subdivided among compute units/clusters. These storage resources can span different or similar storage drives or devices, and can have any number of logical units (LUNs), logical targets, partitions, or other logical arrangements. These logical arrangements can include one or more LUNs, iSCSI LUNs, NVMe targets, or other logical partitioning. Arrays of the storage resources can be employed, such as mirrored, striped, redundant array of independent disk (RAID) arrays, or other array configurations can be employed across the storage resources. Network resources, such as network interface cards, can be shared among the compute units of a cluster using bridging or spanning techniques. Graphics resources, such as GPUs, can be shared among more than one compute unit of a cluster using NT partitioning or domain-based partitioning over the PCIe fabric and PCIe switches.

<FIG> is a block diagram illustrating resource elements of computing platform <NUM>, such as computing platform <NUM>. The resource elements are coupled over a PCIe fabric provided by fabric module <NUM>. PCIe fabric links <NUM>-<NUM> each provide PCIe links internal to an enclosure comprising computing platform <NUM>. Cluster PCIe fabric links <NUM> comprise external PCIe links for interconnecting individual enclosures comprising a cluster.

Multiple instances of resource units <NUM>, <NUM>, <NUM>, and <NUM> are typically provided, and can be logically coupled over the PCIe fabric established by fabric module <NUM>. More than one fabric module <NUM> might be included to achieve the PCIe fabric, depending in part on the number of resource units <NUM>, <NUM>, <NUM>, and <NUM>.

The modules of <FIG> each include one or more PCIe switches (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), one or more power control modules (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) with associated holdup circuits (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), power links (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), and internal PCIe links (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>). It should be understood that variations are possible, and one or more of the components of each module might be omitted.

Fabric module <NUM> provides at least a portion of a Peripheral Component Interconnect Express (PCIe) fabric comprising PCIe links <NUM>-<NUM>. PCIe links <NUM> provide external interconnect for devices of a computing/storage cluster, such as to interconnect various computing/storage rackmount modules. PCIe links <NUM>-<NUM> provide internal PCIe communication links and to interlink the one or more PCIe switches <NUM>. Fabric module <NUM> also provides one or more Ethernet network links <NUM> via network switch <NUM>. Various sideband or auxiliary links <NUM> can be employed as well in fabric module <NUM>, such as System Management Bus (SMBus) links, Joint Test Action Group (JTAG) links, Inter-Integrated Circuit (I2C) links, Serial Peripheral Interfaces (SPI), controller area network (CAN) interfaces, universal asynchronous receiver/transmitter (UART) interfaces, universal serial bus (USB) interfaces, or any other communication interfaces. Further communication links can be included that are not shown in <FIG> for clarity.

Each of links <NUM>-<NUM> can comprise various widths or lanes of PCIe signaling. PCIe can support multiple bus widths, such as x1, x4, x8, x16, and x32, with each multiple of bus width comprising an additional "lane" for data transfer. PCIe also supports transfer of sideband signaling, such as SMBus and JTAG, as well as associated clocks, power, and bootstrapping, among other signaling. For example, each of links <NUM>-<NUM> can comprise PCIe links with four lanes "x4" PCIe links, PCIe links with eight lanes "x8" PCIe links, or PCIe links with <NUM> lanes "x16" PCIe links, among other lane widths.

Power control modules (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) can be included in each module. Power control modules receive source input power over associated input power links (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) and converts/conditions the input power for use by the elements of the associated module. Power control modules distribute power to each element of the associated module over associated power links. Power control modules include circuitry to selectively and individually provide power to any of the elements of the associated module. Power control modules can receive control instructions from an optional control processor over an associated PCIe link or sideband link (not shown in <FIG> for clarity). In some examples, operations of power control modules are provided by processing elements discussed for control processor <NUM>. Power control modules can include various power supply electronics, such as power regulators, step up converters, step down converters, buck-boost converters, power factor correction circuits, among other power electronics. Various magnetic, solid state, and other electronic components are typically sized according to the maximum power draw for a particular application, and these components are affixed to an associated circuit board.

Holdup circuits (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) include energy storage devices for storing power received over power links for use during power interruption events, such as loss of input power. Holdup circuits can include capacitance storage devices, such as an array of capacitors, among other energy storage devices. Excess or remaining holdup power can be held for future use, bled off into dummy loads, or redistributed to other devices over PCIe power links or other power links.

Each PCIe switch (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) comprises one or more PCIe crosspoint switches, which logically interconnect various ones of the associated PCIe links based at least on the traffic carried by associated PCIe links. Each PCIe switch establishes switched connections between any PCIe interfaces handled by each PCIe switch. In some examples, ones of the PCIe switches comprise PLX/Broadcom/Avago PEX8796 <NUM>-port, <NUM> lane PCIe switch chips, PEX8725 <NUM>-port, <NUM> lane PCIe switch chips, PEX97xx chips, PEX9797 chips, or other PEX87xx/PEX97xx chips. In some examples, redundancy is established via one or more PCIe switches, such as having primary and secondary/backup ones among the PCIe switches. Failover from primary PCIe switches to secondary/backup PCIe switches can be handled by at least control processor <NUM>. In some examples, primary and secondary functionality can be provided in different PCIe switches using redundant PCIe links to the different PCIe switches. In other examples, primary and secondary functionality can be provided in the same PCIe switch using redundant links to the same PCIe switch.

PCIe switches <NUM> each include cluster interconnect interfaces <NUM> which are employed to interconnect further modules of storage systems in further enclosures. Cluster interconnect provides PCIe interconnect between external systems, such as other storage systems, over associated external connectors and external cabling. These connections can be PCIe links provided by any of the included PCIe switches, among other PCIe switches not shown, for interconnecting other modules of storage systems via PCIe links. The PCIe links used for cluster interconnect can terminate at external connectors, such as mini-Serial Attached SCSI (SAS) HD connectors, zSFP+ interconnect, or Quad Small Form Factor Pluggable (QSFFP) or QSFP/QSFP+ jacks, which are employed to carry PCIe signaling over associated cabling, such as mini-SAS or QSFFP cabling. In further examples, MiniSAS HD cables are employed that drive 12Gb/s versus 6Gb/s of standard SAS cables. 12Gb/s can support at least PCIe Generation <NUM>.

PCIe links <NUM>-<NUM> can also carry NVMe (NVM Express) traffic issued by a host processor or host system. NVMe (NVM Express) is an interface standard for mass storage devices, such as hard disk drives and solid state memory devices. NVMe can supplant serial ATA (SATA) interfaces for interfacing with mass storage devices in personal computers and server environments. However, these NVMe interfaces are limited to one-to-one host-drive relationship, similar to SATA devices. In the examples discussed herein, a PCIe interface can be employed to transport NVMe traffic and present a multi-drive system comprising many storage drives as one or more NVMe virtual logical unit numbers (VLUNs) over a PCIe interface.

Each resource unit of <FIG> also includes associated resource elements. Storage modules <NUM> include one or more storage drives <NUM>. Compute modules <NUM> include one or more central processing units (CPUs) <NUM>, storage systems <NUM>, and software <NUM>. Graphics modules <NUM> include one or more graphics processing units (GPUs) <NUM>. Network modules <NUM> include one or more network interface cards (NICs) <NUM>. It should be understood that other elements can be included in each resource unit, including memory devices, auxiliary processing devices, support circuitry, circuit boards, connectors, module enclosures/chassis, and other elements.

<FIG> illustrate example graphics processing configurations. Graphics modules <NUM> and <NUM> can comprise two different styles of graphics modules. A first style <NUM> includes GPU <NUM> with CPU <NUM> and PCIe root complex <NUM>, sometimes referred to as a PCIe host. A second style <NUM> includes GPU <NUM> that acts as a PCIe endpoint <NUM>, sometimes referred to as a PCIe device. Each of modules <NUM> and <NUM> can be included in carriers, such as rackmount assemblies. For example, modules <NUM> are included in assembly <NUM>, and modules <NUM> are included in assembly <NUM>. These rackmount assemblies can include JBOD carriers normally used to carry storage drives, hard disk drives, or solid state drives. Example rackmount physical configurations are shown in enclosure <NUM> of <FIG>, and <FIG> below.

<FIG> illustrates a first example graphics processing configuration. A plurality of graphics modules <NUM> that each include GPU <NUM>, CPU <NUM>, and PCIe root complex <NUM> can be coupled through PCIe switch <NUM> to a controller, such as to CPU <NUM> in compute module <NUM>. PCIe switch <NUM> can include isolation elements <NUM>, such as non-transparent ports, logical PCIe domains, port isolation, or Tunneled Window Connection (TWC) mechanisms that allow PCIe hosts to communicate over PCIe interfaces. Normally, only one "root complex" is allowed on a PCIe system bus. However, more than one root complex can be included on an enhanced PCIe fabric as discussed herein using some form of PCIe interface isolation among the various devices.

In <FIG>, each GPU <NUM> is accompanied by a CPU <NUM> with an associated PCIe root complex <NUM>. Each CPU <NUM> is accompanied by an associated PCIe root complex <NUM>. To advantageously allow these PCIe root complex entities to communicate with a controlling host CPU <NUM>, isolation elements <NUM> are included in PCIe switch circuitry <NUM>. Thus, compute module <NUM> as well as each graphics module <NUM> can include their own root complex structures. Moreover, when employed in a separate enclosure, graphics module <NUM> can be included on a carrier or modular chassis that can be inserted and removed from the enclosure. Compute module <NUM> can dynamically add, remove, and control a large number of graphics modules with root complex elements in this manner. DMA transfers can be used to transfer data between compute module <NUM> and each individual graphics module <NUM>. Thus, a cluster of GPUs can be created and controlled by a single compute module or main CPU. This main CPU can orchestrate tasks and graphics/data processing for each of the graphics modules and GPUs. Additional PCIe switch circuits can be added to scale up the quantity of GPUs, while maintaining isolation among the root complexes for DMA transfer of data/control between the main CPU and each individual GPU.

<FIG> illustrates a second example graphics processing configuration. A plurality of graphics modules <NUM> that include at least GPU <NUM> and PCIe endpoint elements <NUM> can be coupled through PCIe switch <NUM> to a controller, such as compute module <NUM>. In <FIG>, each GPU <NUM> is optionally accompanied by a CPU 652and the graphics modules <NUM> act as PCIe endpoints or devices without root complexes. Compute modules <NUM> can each include root complex structures <NUM>. When employed in a separate enclosure, graphics modules <NUM> can be included on a carrier or modular chassis that can be inserted and removed from the enclosure. Compute module <NUM> can dynamically add, remove, and control a large number of graphics modules as endpoint devices in this manner. Thus, a cluster of GPUs can be created and controlled by a single compute module or host CPU. This host CPU can orchestrate tasks and graphics/data processing for each of the graphics modules and GPUs. Additional PCIe switch circuits can be added to scale up the quantity of GPUs.

<FIG> is a block diagram illustrating an example physical configuration of storage system <NUM>. <FIG> includes graphics modules <NUM> in a similar enclosure as compute modules and other modules. <FIG> and <FIG> show graphics modules that might be included in separate enclosures than enclosure <NUM>, such as JBOD enclosures normally configured to hold disk drives. Enclosure <NUM> and the enclosures in <FIG> and <FIG> can be communicatively coupled over one or more external PCIe links, such as through links provided by fabric module <NUM>.

<FIG> is a block diagram illustrating the various modules of the previous figures as related to a midplane. The elements of <FIG> are shown as physically mated to a midplane assembly. Midplane assembly <NUM> includes circuit board elements and a plurality of physical connectors for mating with any associated interposer assemblies <NUM>, storage sub-enclosures <NUM>, fabric modules <NUM>, compute modules <NUM>, graphics modules <NUM>, network modules <NUM>, or power supply modules <NUM>. Midplane <NUM> comprises one or more printed circuit boards, connectors, physical support members, chassis elements, structural elements, and associated links as metallic traces or optical links for interconnecting the various elements of <FIG>. Midplane <NUM> can function as a backplane, but instead of having sleds or modules mate on only one side as in single-ended backplane examples, midplane <NUM> has sleds or modules that mate on at least two sides, namely a front and rear. Elements of <FIG> can correspond to similar elements of the Figures herein, such as computing platform <NUM>, although variations are possible.

<FIG> shows many elements included in a 1U enclosure <NUM>. The enclosure can instead be of any multiple of a standardized computer rack height, such as 1U, 2U, 3U, 4U, 5U, 6U, 7U, and the like, and can include associated chassis, physical supports, cooling systems, mounting features, cases, and other enclosure elements. Typically, each sled or module will fit into associated slot or groove features included in a chassis portion of enclosure <NUM> to slide into a predetermined slot and guide a connector or connectors associated with each module to mate with an associated connector or connectors on midplane <NUM>. System <NUM> enables hot-swapping of any of the modules or sleds and can include other features such as power lights, activity indicators, external administration interfaces, and the like.

Storage modules <NUM> each have an associated connector <NUM> which mates into a mating connector of an associated interposer assembly <NUM>. Each interposer assembly <NUM> has associated connectors <NUM> which mate with one or more connectors on midplane <NUM>. In this example, up to eight storage modules <NUM> can be inserted into a single interposer assembly <NUM> which subsequently mates to a plurality of connectors on midplane <NUM>. These connectors can be a common or shared style/type which is used by compute modules <NUM> and connector <NUM>. Additionally, each collection of storage modules <NUM> and interposer assembly <NUM> can be included in a sub-assembly or sub-enclosure <NUM> which is insertable into midplane <NUM> in a modular fashion. Compute modules <NUM> each have an associated connector <NUM>, which can be a similar type of connector as interposer assembly <NUM>. In some examples, such as in the examples above, compute modules <NUM> each plug into more than one mating connector on midplane <NUM>.

Fabric modules <NUM> couple to midplane <NUM> via connector <NUM> and provide cluster-wide access to the storage and processing components of system <NUM> over cluster interconnect links <NUM>. Fabric modules <NUM> provide control plane access between controller modules of other 1U systems over control plane links <NUM>. In operation, fabric modules <NUM> each are communicatively coupled over a PCIe mesh via link <NUM> and midplane <NUM> with compute modules <NUM>, graphics modules <NUM>, and storage modules <NUM>, such as pictured in <FIG>.

Graphics modules <NUM> comprises one or more graphics processing units (GPUs) along with any associated support circuitry, memory elements, and general processing elements. Graphics modules <NUM> couple to midplane <NUM> via connector <NUM>.

Network modules <NUM> comprise one or more network interface card (NIC) elements, which can further include transceivers, transformers, isolation circuitry, buffers, and the like. Network modules <NUM> might comprise Gigabit Ethernet interface circuitry that can carry Ethernet traffic, along with any associated Internet protocol (IP) and transmission control protocol (TCP) traffic, among other network communication formats and protocols. Network modules <NUM> couple to midplane <NUM> via connector <NUM>.

Cluster interconnect links <NUM> can comprise PCIe links or other links and connectors. The PCIe links used for external interconnect can terminate at external connectors, such as mini-SAS or mini-SAS HD jacks or connectors which are employed to carry PCIe signaling over mini-SAS cabling. In further examples, mini-SAS HD cables are employed that drive 12Gb/s versus 6Gb/s of standard SAS cables. 12Gb/s can support PCIe Gen <NUM>. Quad (<NUM>-channel) Small Form-factor Pluggable (QSFP or QSFP+) connectors or jacks can be employed as well for carrying PCIe signaling.

Control plane links <NUM> can comprise Ethernet links for carrying control plane communications. Associated Ethernet jacks can support <NUM> Gigabit Ethernet (10GbE), among other throughputs. Further external interfaces can include PCIe connections, FiberChannel connections, administrative console connections, sideband interfaces such as USB, RS-<NUM>, video interfaces such as video graphics array (VGA), high-density media interface (HDMI), digital video interface (DVI), among others, such as keyboard/mouse connections.

External links <NUM> can comprise network links which can comprise Ethernet, TCP/IP, Infiniband, iSCSI, or other external interfaces. External links <NUM> can comprise links for communicating with external systems, such as host systems, management systems, end user devices, Internet systems, packet networks, servers, or other computing systems, including other enclosures similar to system <NUM>. External links <NUM> can comprise Quad Small Form Factor Pluggable (QSFFP) or Quad (<NUM>-channel) Small Form-factor Pluggable (QSFP or QSFP+) jacks, or zSFP+ interconnect, carrying at least <NUM> GbE signaling.

In some examples, system <NUM> includes case or enclosure elements, chassis, and midplane assemblies that can accommodate a flexible configuration and arrangement of modules and associated circuit cards. Although <FIG> illustrates storage modules mating and controller modules on a first side of midplane assembly <NUM> and various modules mating on a second side of midplane assembly <NUM>, it should be understood that other configurations are possible. System <NUM> can include a chassis to accommodate any of the following configurations, either in front-loaded or rear-loaded configurations: storage modules that contain multiple SSDs each; modules containing HHHL cards (half-height half-length PCIe cards) or FHHL cards (full-height half-length PCIe cards), that can comprise graphics cards or graphics processing units (GPUs), PCIe storage cards, PCIe network adaptors, or host bus adaptors; modules with PCIe cards (full-height full-length PCIe cards) that comprise controller modules, which can comprise nVIDIA Tesla, nVIDIA Jetson, or Intel Phi processor cards, among other processing or graphics processors; modules containing <NUM>-inch PCIe SSDs; or cross-connect modules, interposer modules, and control elements.

Additionally, power and associated power control signaling for the various modules of system <NUM> is provided by one or more power supply modules <NUM> over associated links <NUM>, which can comprise one or more links of different voltage levels, such as +12VDC or +5VDC, among others. Although power supply modules <NUM> are shown as included in system <NUM> in <FIG>, it should be understood that power supply modules <NUM> can instead be included in separate enclosures, such as separate 1U enclosures. Each power supply node <NUM> also includes power link <NUM> for receiving power from power sources, such as AC or DC input power.

Additionally, power holdup circuitry can be included in holdup modules <NUM> which can deliver holdup power over links <NUM> responsive to power loss in link <NUM> or from a failure of power supply modules <NUM>. Power holdup circuitry can also be included on each sled or module. This power holdup circuitry can be used to provide interim power to the associated sled or module during power interruptions, such as when main input or system power is lost from a power source. Additionally, during use of holdup power, processing portions of each sled or module can be employed to selectively power down portions of each module according to usage statistics, among other considerations. This holdup circuitry can provide enough power to commit in-flight write data during power interruptions or power loss events. These power interruption and power loss events can include loss of power from a power source, or can include removal of a sled or module from an associated socket or connector on midplane <NUM>. The holdup circuitry can include capacitor arrays, super-capacitors, ultra-capacitors, batteries, fuel cells, or other energy storage components, along with any associated power control, conversion, regulation, and monitoring circuitry.

<FIG> is a block diagram illustrating an example physical configuration of a graphics module carrier enclosure. In this example, JBOD assembly <NUM> is employed, with a plurality of slots or bays provided by enclosure <NUM>, which comprises a chassis and other structure/encasing components. Bays in JBOD assembly <NUM> normally are configured to hold storage drives or disk drives, such as HDDs, SSDs, or other drives, which can still be inserted into the bays or slots of enclosure <NUM>. A mixture of disk drive modules, graphics modules, and network modules (<NUM>) might be included. JBOD assembly <NUM> can receive input power over power link <NUM>. Optional power supply <NUM>, fabric modules <NUM>, and holdup circuitry <NUM> are shown in <FIG>.

JBOD carriers <NUM> can be employed to hold graphics modules <NUM> or storage drives into individual bays of JBOD assembly <NUM>. In <FIG>, each graphics module takes up only one slot or bay. <FIG> shows <NUM> graphics modules <NUM> included in individual slots/bays. Graphics modules <NUM> can each comprise a carrier or sled that carries GPU, CPU, and PCIe circuitry assembled into a removable module. Graphics modules <NUM> can also include carrier circuit boards and connectors to ensure each GPU, CPU, and PCIe interface circuity can physically, electrically, and logically mate into the associated bays. In some examples, graphics modules <NUM> in <FIG> each comprise nVIDIA Jetson modules that are fitted into a carrier configured to be inserted into a single bay of JBOD enclosure <NUM>. Backplane assembly <NUM> is included that comprises connectors, interconnect, and PCIe switch circuitry to couple the slots/bays over external control plane links <NUM> and external PCIe links <NUM> to a PCIe fabric provided by another enclosure, such as enclosure <NUM>.

JBOD carriers <NUM> connect to backplane assembly <NUM> via one or more associated connectors for each carrier. Backplane assembly <NUM> can include associated mating connectors. These connectors on each of JBOD carriers <NUM> might comprise U. <NUM> drive connectors, also known as SFF-<NUM> connectors, which can carry PCIe or NVMe signaling. Backplane assembly <NUM> can then route this signaling to fabric module <NUM> or associated PCIe switch circuitry of JBOD assembly <NUM> for communicatively coupling modules to a PCIe fabric. Thus, when populated with one or more graphics processing modules, such as graphics modules <NUM> in <FIG>, the graphics processing modules are inserted into bays normally reserved for storage drives that couple over U. <NUM> drive connectors. <NUM> drive connectors can carry per-bay x4 PCIe interfaces.

In another example bay configuration, <FIG> is presented. <FIG> is a block diagram illustrating another example physical configuration of a graphics module carrier enclosure. In this example, JBOD assembly <NUM> is employed, with a plurality of slots or bays provided by enclosure <NUM>, which comprises a chassis and other structure/encasing components. Bays in JBOD assembly <NUM> normally are configured to hold storage drives or disk drives, such as HDDs, SSDs, or other drives, which can still be inserted into the bays or slots of enclosure <NUM>. A mixture of disk drive modules, graphics modules, and network modules (<NUM>) might be included. JBOD assembly <NUM> can receive input power over power link <NUM>. Optional power supply <NUM>, fabric modules <NUM>, and holdup circuitry <NUM> are shown in <FIG>.

JBOD carriers <NUM> can be employed to hold graphics modules <NUM> or storage drives into individual bays of JBOD assembly <NUM>. In <FIG>, each graphics module takes up four (<NUM>) slots or bays. <FIG> shows <NUM> graphics modules <NUM> included in associated spanned slots/bays. Graphics modules <NUM> can each comprise a carrier or sled that carries GPU, CPU, and PCIe circuitry assembled into a removable module. Graphics modules <NUM> can also include carrier circuit boards and connectors to ensure each GPU, CPU, and PCIe interface circuity can physically, electrically, and logically mate into the associated bays. In some examples, graphics module <NUM> comprises nVIDIA Tesla modules that are fitted into a carrier configured to be inserted into four-bay span of JBOD enclosure <NUM>. Backplane assembly <NUM> is included that comprises connectors, interconnect, and PCIe switch circuitry to couple the slots/bays over external control plane links <NUM> and external PCIe links <NUM> to a PCIe fabric provided by another enclosure, such as enclosure <NUM>.

JBOD carriers <NUM> connect to backplane assembly <NUM> via more than one associated connectors for each carrier. Backplane assembly <NUM> can include associated mating connectors. These individual connectors on each of JBOD carriers <NUM> might comprise individual U. <NUM> drive connectors, also known as SFF-<NUM> connectors, which can carry PCIe or NVMe signaling. Backplane assembly <NUM> can then route this signaling to fabric module <NUM> or associated PCIe switch circuitry of JBOD assembly <NUM> for communicatively coupling modules to a PCIe fabric. When populated with one or more graphics processing modules, such as graphics modules <NUM>, the graphics processing modules are each inserted to span more than one bay, which includes connecting to more than one bay connector and more than one bay PCIe interface. These individual bays are normally reserved for storage drives that couple over individual bay U. <NUM> drive connectors and per-bay x4 PCIe interfaces. A combination of graphics modules <NUM> that span more than one bay, and graphics modules <NUM> that use only one bay might be employed in some examples.

<FIG> is similar to that of <FIG> except a larger bay footprint is used by graphics modules <NUM>, to advantageously accommodate larger graphics module power or PCIe interface requirements. In <FIG>, the power supplied to a single bay/slot is sufficient to power an associated graphics module <NUM>. However, in <FIG>, larger power requirements of graphics modules <NUM> preclude use of a single slot/bay, and instead four (<NUM>) bays are spanned by a single module/carrier to provide the approximately <NUM> watts required for each graphics processing module <NUM>. Power can be drawn from both <NUM> volt and <NUM> volt supplies to establish the <NUM> watt power for each "spanned" bay. A single modular sled or carrier can physically span multiple slot/bay connectors to allow the power and signaling for those bays to be employed. Moreover, PCIe signaling can be spanned over multiple bays, and a wider PCIe interface can be employed for each graphics module <NUM>. In one example, each graphics module <NUM> has a x4 PCIe interface, while each graphics module <NUM> has a x16 PCIe interface. Other PCIe lane widths are possible. A different number of bays than four might be spanned in other examples.

In <FIG> and <FIG>, PCIe signaling, as well as other signaling and power, are connected on a 'back' side via backplane assemblies, such as assemblies <NUM> and <NUM>. This 'back' side comprises an inner portion of each carrier that is inserted into a corresponding bay or bays. However, further communicative coupling can be provided for each graphics processing module on a 'front' side of the modules. Graphics modules can be coupled via front-side point-to-point or mesh communication links <NUM> that span more than one graphics module. In some examples, NVLink interfaces, InfiniBand, point-to-point PCIe links, or other high-speed serial near-range interfaces are applied to couple two or more graphics modules together for further communication among graphics modules.

<FIG> illustrates components of computing platform <NUM> in an implementation. Computing platform <NUM> includes several elements communicatively coupled over a PCIe fabric formed from various PCIe links <NUM>-<NUM> and one or more PCIe switch circuits <NUM>. Host processors or central processing units (CPUs) can be coupled to this PCI fabric for communication with various elements, such as those discussed in the preceding Figures. However, in <FIG> host CPU <NUM> and GPUs <NUM>-<NUM> will be discussed. GPUs <NUM>-<NUM> each comprise graphics processing circuitry, PCIe interface circuitry, and are coupled to associated memory devices <NUM> over corresponding links 1058a-1058n and 1059a-1059n.

In <FIG>, management processor (CPU) <NUM> can establish a peer-to-peer arrangement between GPUs over the PCIe fabric by at least providing an isolation function <NUM> in the PCIe fabric configured to isolate a device PCIe address domain associated with the GPUs from a local PCIe address domain associated with host CPU <NUM> that initiates the peer-to-peer arrangement between the GPUs. Specifically, host CPU <NUM> might want to initiate a peer-to-peer arrangement, such as a peer-to-peer communication link, among two or more GPUs in platform <NUM>. This peer-to-peer arrangement enables the GPUs to communicate more directly with each other to bypass transferring communications through host CPU <NUM>.

Without a peer-to-peer arrangement, for example, traffic between GPUs is typically routed through a host processor. This can be seen in <FIG> as communication link <NUM> which shows communications between GPU <NUM> and GPU <NUM> being routed over PCIe links <NUM> and <NUM>, PCIe switch <NUM>, and host CPU <NUM>. Latency can be higher for this arrangement, as well as other bandwidth reductions by handling the traffic through many links, switch circuitry, and processing elements. Advantageously, isolation function <NUM> can be established in the PCIe fabric which allows for GPU <NUM> to communicate more directly with GPU <NUM>, bypassing links <NUM> and host CPU <NUM>. Less latency is encountered as well as higher bandwidth communications. This peer-to-peer arrangement is shown in <FIG> as peer-to-peer communication link <NUM>.

Management CPU <NUM> can comprise control circuitry, processing circuitry, and other processing elements. Management CPU <NUM> can comprise elements of management processor <NUM> in <FIG> or management processor <NUM> of <FIG>. In some examples, management CPU <NUM> can be coupled to a PCIe fabric or to management/control ports on various PCIe switch circuitry, or incorporate the PCIe switch circuitry or control portions thereof. In <FIG>, management CPU <NUM> establishes the isolation function and facilitates establishment of peer-to-peer link <NUM>. A further discussion of the elements of a peer-to-peer arrangement as well as operational examples of management CPU <NUM> and associated circuity is seen in <FIG>. Management CPU <NUM> can communicate with PCIe switches <NUM> over management links <NUM>-<NUM>. These management links comprise PCIe links, such as x1 or x4 PCIe links, and may comprise I2C links, network links, or other communication links.

<FIG> illustrates components of computing platform <NUM> in an implementation. Platform <NUM> shows a more detailed implementation example for elements of <FIG>, although variations are possible. Platform <NUM> includes host processor <NUM>, memory <NUM>, control processor <NUM>, PCIe switch <NUM>, and GPUs <NUM>-<NUM>. Host processor <NUM> and GPUs <NUM>-<NUM> are communicatively coupled by switch circuitry <NUM> in PCIe switch <NUM>, which forms a portion of a PCIe fabric along with PCIe links <NUM>-<NUM>. Control processor <NUM> also communicates with PCIe switch <NUM> over a PCIe link, namely link <NUM>, but this link typically comprises a control port, administration link, management link, or other link functionally dedicated to control of the operation of PCIe switch <NUM>. However, other examples have control processor <NUM> coupled via the PCIe fabric.

In <FIG>, two or more PCIe addressing domains are established. These address domains (<NUM>, <NUM>) are established as a part of an isolation function to logically isolate PCIe traffic of host processor <NUM> from GPUs <NUM>-<NUM>. Furthermore, synthetic PCIe devices are created by control processor <NUM> to further comprise the isolation function between PCIe address domains. This isolation function provides for isolation of host processor <NUM> from GPUs <NUM>-<NUM> as well as provides for enhanced peer-to-peer arrangements among GPUs.

To achieve this isolation function, various elements of <FIG> are employed, such as those indicated above. Isolation function <NUM> comprises address traps <NUM>-<NUM> and synthetic devices <NUM>. These address traps comprise an address monitoring portion an address translation portion. The address monitoring portion monitors PCIe destination addresses in PCIe frames or other PCIe traffic to determine if one or more affected addresses are encountered. If these addresses are encountered, then the address traps translate the original PCIe destination addresses into modified PCIe destination addresses, and transfers the PCIe traffic for delivery over the PCIe fabric to hosts or devices that correspond to the modified PCIe destination addresses. Address traps <NUM>-<NUM> can include one or more address translation tables or other data structures, such as example table <NUM>, that map translations between incoming destination addresses and outbound destination addresses that are used to modify PCIe addresses accordingly. Table <NUM> contains entries that translate addressing among the synthetic devices in the local address space and the physical/actual devices in the global/device address space.

Synthetic devices <NUM>-<NUM> comprise logical PCIe devices that represent corresponding ones of GPUs <NUM>-<NUM>. Synthetic device <NUM> represents GPU <NUM>, and synthetic device <NUM> represents GPU <NUM>. As will be discussed in further detail below, when host processor <NUM> issues PCIe traffic for delivery to GPUs <NUM>-<NUM>, this traffic is actually addressed for delivery to synthetic devices <NUM>-<NUM>. Specifically, device drivers of host processor <NUM> uses destination addressing that corresponds to associated synthetic devices <NUM>-<NUM> for any PCIe traffic issued by host processor <NUM> for GPUs <NUM>-<NUM>. This traffic is transferred over the PCIe fabric and switch circuitry <NUM>. Address traps <NUM>-<NUM> intercept this traffic that includes the addressing of synthetic devices <NUM>-<NUM>, and reroutes this traffic for delivery to addressing associated with GPUs <NUM>-<NUM>. Likewise, PCIe traffic issued by GPUs <NUM>-<NUM> is addressed by the GPUs for delivery to host processor <NUM>. In this manner, each of GPU <NUM> and GPU <NUM> can operate with regard to host processor <NUM> using PCIe addressing that corresponds to synthetic devices <NUM> and synthetic devices <NUM>.

Host processor <NUM> and synthetic devices <NUM>-<NUM> are included in a first PCIe address domain, namely a 'local' address space <NUM> of host processor <NUM>. control processor <NUM> and GPUs <NUM>-<NUM> are included in a second PCIe address domain, namely a 'global' address space <NUM>. The naming of the address spaces is merely exemplary, and other naming schemes can be employed. Global address space <NUM> can be used by control processor <NUM> to provision and deprovision devices, such as GPUs, for use by various host processors. Thus, any number of GPUs can be communicatively coupled to a host processor, and these GPUs can be dynamically added and removed for use by any given host processor.

It should be noted that synthetic devices <NUM>-<NUM> each have corresponding base address registers (BAR <NUM>-<NUM>) and corresponding device addresses <NUM>-<NUM> in the local addressing (LA) domain. Furthermore, GPUs <NUM>-<NUM> each have corresponding base address registers (BAR <NUM>-<NUM>) and corresponding device addresses <NUM>-<NUM> in the global addressing (GA) domain. The LA and GA addresses correspond to addressing that would be employed to reach the associated synthetic or actual device.

To further illustrate the operation of the various addressing domains, <FIG> is presented. <FIG> illustrates components of computing platform <NUM> in an implementation. Platform <NUM> includes host processor <NUM>, control processor <NUM>, and host processor <NUM>. Each host processor is communicatively coupled to a PCIe fabric, such as any of those discussed herein. Furthermore, control processor <NUM> can be coupled to the PCIe fabric or to management ports on various PCIe switch circuitry, or incorporate the PCIe switch circuitry or control portions thereof.

<FIG> is a schematic representation of PCIe addressing and associated domains formed among PCIe address spaces. Each host processor has a corresponding 'local' PCIe address space, such as that corresponding to an associated root complex. Each individual PCIe address space can comprise a full domain of the <NUM>-bit address space of the PCIe specification, or a portion thereof. Furthermore, an additional PCIe address space/domain is associated with control processor <NUM>, referred to herein as a 'global' or 'device' PCIe address space.

The isolation functions with associated address traps form links between synthetic devices and actual devices. The synthetic devices represent the actual devices in another PCIe space than that of the devices themselves. In <FIG>, the various devices, such as GPUs or any other PCIe devices, are configured to reside within the global address space that is controlled by control processor <NUM>. In <FIG>, the actual devices are represented by 'D' symbols. The various synthetic devices, represented by 'S' symbols in <FIG>, are configured to reside on associated local address spaces for corresponding host processors.

In <FIG>, four address traps are shown, namely address traps <NUM>-<NUM>. Address traps are formed to couple various synthetic devices to various physical/actual devices. These address traps, such as those discussed in <FIG>, are configured to intercept PCIe traffic directed to the synthetic devices and forward to the corresponding physical devices. Likewise, the address traps are configured to intercept PCIe traffic directed to the physical devices and forward to the corresponding synthetic devices. Address translation is performed to alter the PCIe address of PCIe traffic that corresponds to the various address traps.

Advantageously, any host processor with a corresponding local PCIe address space can be dynamically configured to communicate with any PCIe device that resides in the global PCIe address space, and vice versa. Devices can be added and removed during operation of the host processors, which can support scaling up or down available resources for each added/removed device. When GPUs are employed as the devices, then GPU resources can be added or removed on-the-fly to any host processor. Hot-plugging of PCIe devices are enhanced, and devices that are installed into rack-mounted assemblies comprises dozens of GPUs can be intelligently assigned and re-assigned to host processors as needed. Synthetic devices can be created/destroyed as needed, or a pool of synthetic devices might be provisioned for a particular host, and the synthetic devices can be configured with appropriate addressing to allow corresponding address trap functions to route traffic to desired GPUs/devices. Control processor <NUM> handles the setup of synthetic devices, address traps, synthetic devices, and the provisioning/deprovisioning of devices/GPUs.

Turing now to example operations of the elements of <FIG>, <FIG> is presented. <FIG> is a flow diagram illustrating example operations of a computing platform, such as computing platform <NUM>, <NUM>, or <NUM>. The operations of <FIG> are discussed in the context of elements of <FIG>. However, it should be understood that elements of any of the Figures herein can be employed. <FIG> also discusses operation of a peer-to-peer arrangement among GPUs or other PCIe devices, such as seen with peer-to-peer link <NUM> in <FIG> or peer-to-peer link <NUM> in <FIG>. Peer-to-peer linking allows for more direct transfer of data or other information between PCIe devices, such as GPUs for enhanced processing, increased data bandwidth, and lower latency.

In <FIG>, a PCIe fabric is provided (<NUM>) to couple GPUs and one or more host processors. In <FIG>, this PCIe fabric can be formed among PCI switch <NUM> and PCIe links <NUM>-<NUM>, among further PCIe switches coupled by PCIe links. However, the GPUs and host processors as this point are merely coupled electrically to the PCIe fabric, and are not yet configured to communicate. A host processor, such as host processor <NUM> might wish to communicate with one or more GPU devices, and furthermore allow those GPU devices to communicate over a peer-to-peer arrangement to enhance the processing performance of the GPUs. Control processor <NUM> can establish (<NUM>) a peer-to-peer arrangement between the GPUs over the PCIe fabric. Once established, control processor <NUM> can dynamically add (<NUM>) GPUs into the peer-to-peer arrangement, and dynamically remove (<NUM>) GPUs from the peer-to-peer arrangement.

To establish the peer-to-peer arrangement, control processor <NUM> provides (<NUM>) an isolation function to isolate a device PCIe address domain associated with the GPUs from a local PCIe address domain associated with a host processor. In <FIG>, host processor <NUM> includes or is coupled with a PCIe root complex which is associated with local PCIe address space <NUM>. Control processor <NUM> can provide the root complex for a 'global' or device PCIe address space <NUM>, or another element not shown in <FIG> might provide this root complex. A plurality of GPUs are included in the address space <NUM>, and global addresses <NUM>-<NUM> are employed as the device/endpoint addresses for the associated GPUs. The two distinct PCIe address spaces are logically isolated from one another, and PCIe traffic or communications are not transferred across the PCIe address spaces.

To interwork PCIe traffic or communications among the PCIe address spaces, control processor <NUM> establishes (<NUM>) synthetic PCIe devices representing the GPUs in the local PCIe address domain. The synthetic PCIe devices are formed in logic provided by PCIe switch <NUM> or control processor <NUM>, and each provide for a PCIe endpoint that represents the associated GPU in the local address space of the particular host processor. Furthermore, address traps are provided for each synthetic device that intercepts PCIe traffic destined for the corresponding synthetic device and re-routes the PCIe traffic for delivery to appropriate physical/actual GPUs. Thus, control processor <NUM> establishes address traps <NUM>-<NUM> that redirect (<NUM>) traffic transferred by host processor <NUM> for GPUs <NUM>-<NUM> in the local PCIe address domain for delivery to ones of the GPUs in the device PCIe address domain. In a first example, PCIe traffic issued by host processor <NUM> can be addressed for delivery to synthetic device <NUM>, namely local address (LA) <NUM>. Synthetic device <NUM> has been established as an endpoint for this traffic, and address trap <NUM> is established to redirect this traffic for delivery to GPU <NUM> at global address (GA) <NUM>. In a second example, PCIe traffic issued by host processor <NUM> can be addressed for delivery to synthetic device <NUM>, namely LA <NUM>. Synthetic device <NUM> has been established as an endpoint for this traffic, and address trap <NUM> is established to redirect this traffic for delivery to GPU <NUM> at GA <NUM>.

Handling of PCIe traffic issued by the GPUs can work in a similar manner. In a first example, GPU <NUM> issues traffic for delivery to host processor <NUM>, and this traffic might identify an address in the local address space of host processor <NUM>, and not a global address space address. Trap <NUM> identifies this traffic as destined for host processor <NUM> and redirects the traffic for delivery to host processor <NUM> in the address domain/space associated with host processor <NUM>. In a second example, GPU <NUM> issues traffic for delivery to host processor <NUM>, and this traffic might identify an address in the local address space of host processor <NUM>, and not a global address space address. Trap <NUM> identifies this traffic as destined for host processor <NUM> and redirects the traffic for delivery to host processor <NUM> in the address domain/space associated with host processor <NUM>.

In addition to host-to-device traffic discussed above, isolation function <NUM> can provide for peer-to-peer arrangements among GPUs. Control processor <NUM> establishes address trap <NUM> that redirects (<NUM>) peer-to-peer traffic transferred by a first GPU indicating a second GPU as a destination in the local PCIe address domain to the second GPU in the global/device PCIe address domain. Each GPU need not be aware of the different PCIe address spaces, such as in the host-device example above where the GPU uses an associated address in the local address space of the host processor for traffic issued to the host processor. Likewise, each GPU when engaging in peer-to-peer communications can issue PCIe traffic for delivery to another GPU using addressing native to the local address space of host processor <NUM> instead of the addressing native to the global/device address space. However, since each GPU is configured to respond to addressing in the global address space, then address trap <NUM> is configured to redirect traffic accordingly. GPUs use addressing of the local address space of host processor <NUM> due to host processor <NUM> typically communicating with the GPUs to initialize the peer-to-peer arrangement among the GPUs. Although the peer-to-peer arrangement is facilitated by control processor <NUM> managing the PCIe fabric and isolation function <NUM>, the host processor and GPUs are not typically aware of the isolation function and different PCIe address spaces. Instead, the host processor communicates with synthetic devices <NUM>-<NUM> as if those synthetic devices were the actual GPUs. Likewise, GPUs <NUM>-<NUM> communicate with the host processor and each other without knowledge of the synthetic devices or the address trap functions. Thus, traffic issued by GPU <NUM> for GPU <NUM> uses addressing in the local address space of the host processor to which those GPUs are assigned. Address trap <NUM> detects the traffic with the addressing in the local address space and redirects the traffic using addressing in the global address space.

In a specific example of peer-to-peer communications, the host processor will initially set up the arrangement between GPUs, and indicate peer-to-peer control instructions identifying addressing to the GPUs that is within the local PCIe address space of the host processor. Thus, the GPUs are under the control of the host processor, even though the host processor communicates with synthetic devices established within the PCIe fabric or PCIe switching circuitry. When GPU <NUM> has traffic for delivery to GPU <NUM>, GPU <NUM> will address the traffic as destined for GPU <NUM> in the local address space (i.e. LA <NUM> associated with synthetic device <NUM>), and address trap <NUM> will redirect this traffic to GA <NUM>. This redirection can include translating addressing among PCIe address spaces, such as by replacing or modifying addressing of the PCIe traffic to include the redirection destination address instead of the original destination address. When GPU <NUM> has traffic for delivery to GPU <NUM>, GPU <NUM> will address the traffic as destined for GPU <NUM> in the local address space (i.e. LA <NUM> associated with synthetic device <NUM>), and address trap <NUM> will redirect this traffic to GA <NUM>. This redirection can include replacing or modifying addressing of the PCIe traffic to include the redirection destination address instead of the original destination address. Peer-to-peer link <NUM> is thus logically created which allows for more direct flow of communications among GPUs.

<FIG> is presented to illustrate further details on address space isolation and selection of appropriate addressing when communicatively coupling host processors to PCIe devices, such as GPUs. In <FIG>, computing platform <NUM> is presented. Computing platform <NUM> includes several host CPUs <NUM>, a management CPU <NUM>, PCIe fabric <NUM>, as well as one or more assemblies <NUM>-<NUM> that house a plurality associated of GPUs <NUM>-<NUM> as well as a corresponding PCIe switch <NUM>. Assemblies <NUM>-<NUM> might comprise any of the chassis, rackmount or JBOD assemblies herein, such as found in <FIG> and <FIG>. A number of PCIe links interconnect the elements of <FIG>, namely PCIe links <NUM>-<NUM>. Typically, PCIe link <NUM> comprises a special control/management link that enables administrative or management-level access of control to PCIe fabric <NUM>. However, it should be understood that similar links to the other PCIe links can instead be employed.

According to the examples in <FIG>, isolation functions can be established to allow for dynamic provisioning/de-provisioning of PCIe devices, such as GPUs, from one or more host processors/CPUs. These isolation functions can provide for separate PCIe address spaces or domains, such as independent local PCIe address spaces for each host processor deployed and a global or device PCIe address space shared by all actual GPUs. However, when certain further downstream PCIe switching circuitry is employed, overlaps in addressing used within the local address spaces of the host processors and the global address spacing of the GPUs can lead to collisions or errors in the handling of the PCle traffic by PCIe switching circuitry.

Thus, <FIG> illustrates enhanced operation for selection of PCIe address allocation and address space configuration. Operations <NUM> illustrate example operations for management CPU <NUM> used in configuring isolation functions and address domains/spaces. Management CPU <NUM> identifies (<NUM>) when downstream PCIe switches are employed, such as when external assemblies are coupled over PCIe links to a PCIe fabric that couples host processors to further computing or storage elements. In <FIG>, these downstream PCIe switches are indicated by PCIe switches <NUM>-<NUM>. Management CPU <NUM> can identify when these downstream switches are employed using various discovery protocols over the PCIe fabric, over sideband signaling, such as I2C or Ethernet signaling, or using other processes. In some examples, downstream PCIe switches comprise a more primitive or less capable model/type of PCIe switches than those employed upstream, and management CPU <NUM> can either detect these configurations via model numbers or be programmed by an operator to compensate for this reduced functionality. The reduced functionality can include not being able to handle multiple PCIe addressing domains/spaces as effectively as other types of PCIe switches, which can lead to PCIe traffic collisions. Thus, an enhanced operation is provided in operations <NUM>-<NUM>.

In operation <NUM>, management CPU <NUM> establishes non-colliding addressing for each of the physical/actual PCIe devices in the device/global address spaces with regard to the local PCIe address spaces of the host processors. The non-colliding addressing typically comprise unique, non-overlapping addressing employed for downstream/endpoint PCIe devices. This is done to prevent collisions among PCIe addressing when the synthetic PCIe devices are employed herein. When address translation is performed by various address trap elements to redirect PCIe traffic from a local address space of a host processor to the global address space of the PCIe devices, collisions are prevented by intelligent selection of addressing for the PCIe devices in the global address space. Global address space addresses for devices are selected to be non-overlapping, uncommon, or unique so that more than one host processor does not use similar device addresses in an associated local address space. These addresses are indicated to the host processors during boot, initialization, enumeration, or instantiation of the associated PCIe devices and synthetic counterparts, so that any associated host drivers employ unique addressing across the entire PCIe fabric, even though each host processor might have a logically separate/independent local address space. Once the addressing has been selected and indicated to the appropriate host processors, computing platform <NUM> can operate (<NUM>) upstream PCle switch circuitry and host processors according to the non-colliding address spaces. Advantageously, many host processors are unlikely to have collisions in PCIe traffic with other host processors.

Claim 1:
A method of operating a data processing system (<NUM>) comprising a control processor (<NUM>), the method comprising:
communicatively coupling endpoint devices (<NUM>, <NUM>) over a communication fabric (<NUM>-<NUM>, <NUM>); and
establishing, by the control processor (<NUM>), a peer-to-peer arrangement (<NUM>) between the endpoint devices (<NUM>, <NUM>) over the communication fabric (<NUM>-<NUM>, <NUM>) by at least providing an isolation function (<NUM>) in the communication fabric (<NUM>-<NUM>, <NUM>) configured to isolate a device fabric address domain (<NUM>) associated with the endpoint devices (<NUM>, <NUM>) from at least a local fabric address domain (<NUM>) associated with a host processor (<NUM>) that initiates the peer-to-peer arrangement (<NUM>) between the endpoint devices (<NUM>, <NUM>)
isolating, by the isolation function (<NUM>), the device fabric address domain (<NUM>) from the local fabric address domain (<NUM>) by at least establishing synthetic endpoint devices (<NUM>, <NUM>), by the control processor (<NUM>), representing the endpoint devices (<NUM>, <NUM>) in the local fabric address domain (<NUM>);
redirecting traffic transferred by the host processor (<NUM>) for the endpoint devices (<NUM>, <NUM>) in the local fabric address domain (<NUM>) for delivery to corresponding ones of the endpoint devices (<NUM>, <NUM>) in the device fabric address domain (<NUM>); and
responsive to peer-to-peer traffic from a first of the endpoint devices (<NUM>, <NUM>) indicating the second of the endpoint devices (<NUM>, <NUM>) as a destination in the local fabric address domain (<NUM>), employing a peer-to-peer address trap (<NUM>, <NUM>) in the isolation function (<NUM>) to receive the peer-to-peer traffic and transfer the peer-to-peer traffic to the second of the endpoint devices in the device fabric address domain (<NUM>).