Self identifying interconnect topology

A system for automatically discovering fabric topology includes at least one or more processing units, one or more memory devices, a security processor, and a communication fabric with an unknown topology coupled to the processing unit(s), memory device(s), and security processor. The security processor queries each component of the fabric to retrieve various attributes associated with the component. The security processor utilizes the retrieved attributes to create a network graph of the topology of the components within the fabric. The security processor generates routing tables from the network graph and programs the routing tables into the fabric components. Then, the fabric components utilize the routing tables to determine how to route incoming packets.

BACKGROUND

Description of the Related Art

Computing systems are increasingly integrating large numbers of different types of components on a single integrated circuit (i.e., chip) or on a multi-chip module. The complexity and power consumption of a system increases with the number of different types of components. Often, these components are connected together via switches, routers, communication buses, bridges, buffers, controllers, coherent devices, and other links. The combination of these interconnecting components is referred to herein as a “communication fabric”, or “fabric” for short.

Generally speaking, the fabric facilitates communication by routing messages between a plurality of components on a chip or multi-chip module. Examples of messages communicated over a fabric include memory access requests, status updates, data transfers, coherency probes, coherency probe responses, and the like. As computing systems increase in complexity, the interconnecting fabric to couple and communicate between the system components is also increasing in complexity. Assigning labels and routes statically is typically not viable, nor can assumptions be made about the regularity of the fabric topology. Accordingly, improved techniques for managing an irregular fabric topology are desired.

DETAILED DESCRIPTION OF EMBODIMENTS

Various systems, apparatuses, methods, and computer-readable mediums for implementing a self-identifying discovery process for an interconnect fabric topology are disclosed herein. In one embodiment, a system includes at least one or more processing units, one or more input/output (I/O) interfaces, an interconnect fabric of unknown topology, and one or more memory devices. In one embodiment, a system implements an autonomous bootstrap discovery process that can operate on arbitrary fabric topologies. To simplify programming of different fabric systems, a general solution for discovery and programming of an arbitrary network on chip (NoC) is implemented when the fabric spans multiple nodes (die or sockets). In one embodiment, a set of information registers as well as algorithms are implemented to traverse the network blocks, discovering the block type and capabilities as well as the connectivity between network components.

In one embodiment, fabric discovery is performed by a security processor. In another embodiment, fabric discovery is performed by a system management processor. In other embodiments, fabric discovery can be performed by other types of components. In one embodiment, fabric discovery is performed as part of the design flow where information obtained during fabric discovery is used to create reset conditions in system hardware. In one embodiment, fabric discovery begins with the security processor reading a fabric block instance count register. The fabric block instance count register stores an indication of the total number of fabric blocks in the system and the fabric block instance count register is present in the system at a fixed address. The fabric block instance count register allows for the firmware to have a general starting point regardless of the size or topology of the system.

Once the total number of fabric blocks is retrieved from the fabric block instance count register, the security processor proceeds through each of the blocks to read the block instance information registers. These registers contain information such as the block type (e.g., coherent master, non-coherent master, crossbar, coherent slave, non-coherent slave), types and number of command and data ports, data bus width, clock speed, neighbor instance identifiers (IDs), fabric IDs, and other attributes. Once this information has been obtained for each block, a network graph is constructed as a data structure for use in further processing. Routing algorithms can be run based on the network graph to determine the desired routing for the NoC as well as the multi-node system. Once the routes, block types, and capabilities are discovered, the firmware proceeds to program the masters, slaves, and switches appropriately for fabric initialization.

Referring now toFIG. 1, a block diagram of one embodiment of a computing system100is shown. In one embodiment, computing system100includes at least processing units110A-B, fabric115, input/output (I/O) interfaces120, memory controller(s)130, display controller135, other device(s)140, and security processor145. In other embodiments, computing system100can include other components and/or computing system100can be arranged differently. Processing units110A-B are representative of any number and type of processing units. For example, in one embodiment, processing unit110A is a central processing unit (CPU) and processing unit110B is a graphics processing unit (GPU). In other embodiments, processing units110A-B can include other numbers and types of processing units (e.g., digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC)).

Fabric115is representative of any communication interconnect with any of various types of protocols utilized for communicating among the components of the system100. Fabric115provides the data paths, switches, routers, and other logic that connect the processing units110A-B, I/O interfaces120, memory controller(s)130, display controller135, and other device(s)140to each other. Fabric115handles the request, response, and data traffic, as well as probe traffic to facilitate coherency. Fabric115also handles interrupt request routing and configuration access paths to the various components of system100. Additionally, fabric115handles configuration requests, responses, and configuration data traffic. Fabric115can be bus-based, including shared bus configurations, crossbar configurations, and hierarchical buses with bridges. Fabric115can also be packet-based, and can be hierarchical with bridges, crossbar, point-to-point, or other interconnects. From the point of view of fabric115, the other components of system100can be referred to as “clients”. Fabric115is configured to process requests generated by various clients and pass the requests on to other clients.

In one embodiment, security processor145is configured to initiate an autonomous discovery of the topology of the components of fabric115. This can be useful in situations where system100does not have a regular configuration, and where various different types of capabilities can be included within different implementations of system100. For example, one implementation of system100can include 16 processor cores and two memory channels per core, while a second implementation of system100can include 8 processor cores and one memory channel per core. Other implementations of system100can include other numbers of cores, memory channels, memory controllers, memory devices, and so forth. As such, the ability to configure and initialize fabric115prior to system boot-up is limited. Accordingly, in these circumstances, security processor145is configured to discover the unique topology of a given implementation of system100during the initial boot-up of system100.

In one embodiment, security processor145is configured to query each component of fabric115to retrieve various attributes associated with the component. In one embodiment, each component of fabric115includes one or more registers to store values specifying the attributes of the respective component. Then, after querying the components of fabric115, security processor145is configured to create a network graph based on the attributes retrieved from the various components. Next, security processor145is configured to program masters, slaves, and switches for fabric initialization based on the determined routes, block types, and capabilities. For example, in one embodiment, after creating the network graph, security processor145is configured to generate routing tables from the network graph. In this embodiment, security processor145programs the routing tables into the various components of fabric115. Then, the components of fabric115utilize the programmed routing tables to determine how to route received packets.

In one embodiment, security processor145is also configured to manage the configuration and security of system100. Security processor145is configured to execute instructions for performing authentication and validation functions which provide security protection for system100. Also, security processor145stores one or more unique encryption/decryption keys inaccessible to the rest of system100. Accordingly, security processor145provides a hardware-based root of trust for system100, allowing system100to start up in a secure environment. In one embodiment, security processor145manages the boot-up process of system100to ensure that system100boots up with authenticated boot code. Security processor145also manages various other functions associated with the boot-up process of system100. Then, security processor145releases processing units110A-N to execute the boot code and to launch the operating system of system100.

Memory controller(s)130are representative of any number and type of memory controllers which can be coupled to any number and type of memory device(s). For example, the type of memory device(s) coupled to memory controller(s)130can include Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others. Memory controller(s)130are accessible by processing units110A-B, I/O interfaces120, display controller135, and other device(s)140via fabric115. I/O interfaces120are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices can be coupled to I/O interfaces120. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth. Other device(s)140are representative of any number and type of devices (e.g., multimedia device, video codec).

In various embodiments, computing system100can be a computer, laptop, mobile device, server or any of various other types of computing systems or devices. It is noted that the number of components of computing system100can vary from embodiment to embodiment. There can be more or fewer of each component than the number shown inFIG. 1. It is also noted that computing system100can include other components not shown inFIG. 1. Additionally, in other embodiments, computing system100can be structured in other ways than shown inFIG. 1.

Turning now toFIG. 2, a block diagram of one embodiment of a fabric200is shown. In one embodiment, fabric200can be implemented as fabric115of computing system100(ofFIG. 1). In one embodiment, fabric200includes a plurality of different components arranged into an irregular topology. In some embodiments, fabric200is partitioned into a plurality of regions, with any number of components per region and any type of topology for each region. In one embodiment, each region of fabric200is associated with a different power-gating domain.

On the left-side ofFIG. 2, fabric200includes master205and slaves215and220coupled to crossbar210. As used herein, a “master” is defined as a component that generates requests, and a “slave” is defined as a component that services requests. Crossbar210is coupled to crossbar225, with crossbar225coupled to masters230A-B. Crossbar225is coupled to crossbar245, with crossbar245coupled to crossbars250and260and to master255A. Crossbar250is coupled to masters255B-C and crossbar260is coupled to masters265A-B and slave265C. At the far-right ofFIG. 2, fabric200includes crossbar270coupled to slave275and master280. In one embodiment, slave275is coupled to a memory controller.

In one embodiment, each component of fabric200includes a register to store attributes specifying information about the respective component. For example, in one embodiment, the register stores information such as component type, types and number of command and data ports, data bus width, clock speed, neighbor instance IDs, fabric IDs, as well as other information. In one embodiment, a security processor (e.g., security processor145ofFIG. 1) is configured to query each component of fabric200to retrieve the information stored in a corresponding register. The security processor can then utilize this information to create a network graph of fabric200and routing tables based on the network graph.

It is noted that the layout, number, and type of components of fabric200is merely indicative of one embodiment. In other embodiments, fabric200can be arranged differently and/or include other numbers and types of components. It is also noted that the various crossbars210,225,245,250,260, and270which are part of fabric200can also be referred to as switches or routers. A crossbar can include switch connections and buffering to buffer requests which are received by the crossbar. A crossbar can also include credit control logic to implement a flow control mechanism for transmitting requests from source to destination. Each crossbar allows for requests to connect from any source to any of the destinations coupled to the crossbar. Each crossbar also includes one or more connections to other crossbars to enable requests to be sent to other destinations which are not directly coupled to the respective crossbar.

Referring now toFIG. 3, a block diagram of another embodiment of a computing system300is shown. System300includes at least security processor305, fabric block instance count register310, fabric315, processing units335A-N, I/O interfaces340, memory controller(s)345, and any number of other components which are not shown to avoid obscuring the figure. In one embodiment, when system300initially powers up, system300does not have any knowledge of the topology or structure of fabric315, and system300is configured to discover this topology as part of the boot-up process. As a result, fabric topology discovery provides flexibility to build different types of systems with a variety of network topologies and resources using the same basic components.

In one embodiment, security processor305is configured to query fabric block instance count register310to retrieve an indication of the total number of components320A-N of fabric315. Then, security processor305initiates the automatic discovery process of components320A-N of fabric315. Security processor305can be implemented using any suitable combination of software, hardware, and/or firmware. In one embodiment, security processor305is a dedicated microprocessor configured to perform various security functions for system300. For example, in this embodiment, security processor305is configured to provide a hardware-based root of trust for system300.

Components320A-N are representative of any number and type of components arranged in any type of topology within fabric315. For example, components320A-N can include crossbars, switches, routers, non-coherent masters, coherent masters, non-coherent slaves, coherent slaves, and so on. It is noted that components320A-N can also be referred to as “blocks” herein. In one embodiment, each component320A-N has a corresponding register325A-N which stores various metadata for the respective component. Each register325A-N is representative of any number of registers or other types of storage locations for storing any number of attributes. For example, each register325A-N can specify a component type, type and number of command and data ports, data bus width, clock speed, neighbor instance IDs, fabric IDs, and so on.

In one embodiment, security processor305is configured to traverse the components320A-N of fabric315one at a time. In one embodiment, security processor305is coupled to the components320A-N of fabric315via the main data paths. In another embodiment, security processor305is coupled to the components320A-N of fabric315via sideband connections which are different from the main data path connections which packets traverse when flowing through fabric315. To initiate the automatic discovery process, security processor305starts by querying register325A of the nearest component320A to security processor305. Then, security processor305continues by querying the neighbors of component320A. In one embodiment, each component320A-N is assigned a unique ID within fabric315.

Security processor305continues traversing fabric315through the neighbors of neighbors and so on until the edges of fabric315are reached. After security processor305has retrieved metadata from registers325A-N of all of the components320A-N, security processor305is configured to build a network graph of the discovered components. Then, security processor305utilizes the network graph to generate routing tables for each component320A-N. After generating the routing tables, security processor305conveys a routing table to each component320A-N for use in determining how to route received packets.

Turning now toFIG. 4, a block diagram of one embodiment of a security processor405coupled to a fabric415is shown. Security processor405is configured to retrieve component attributes408from registers425A-N of components420A-N of fabric415during a fabric discovery process as previously described in the discussion regardingFIG. 3. After retrieving component attributes408, security processor405is configured to create network graph410to represent the components and topology of component interconnections of fabric415.

In one embodiment, security processor405utilizes network graph410to generate routing tables430A-N. For example, security processor405traverses network graph410by starting at a given node of network graph410, with the given node representing a given component. Then, the security processor405traverses network graph410from the given node until reaching the leaf nodes while tracking the connections originating from the given node. Security processor405utilizes each traversed path from the given node to a leaf node to construct a routing table for the given node. After constructing the routing table, security processor405programs the given component with this routing table. Security processor405can then perform a similar process for the other nodes of network graph410.

For example, security processor405utilizes network graph410to generate a routing table430A for component420A, and then security processor405programs routing table430A into component420A. During operation of the host system which contains fabric415, component420A utilizes routing table430A to determine how to route incoming packets. Similarly, security processor405programs routing tables430B-N into components420B-N, respectively, and then components420B-N utilize their corresponding routing tables430B-N to determine how to route received packets during actual operation of the host system. It should be understood that fabric415does not come pre-programmed with routing tables, but rather, the routing tables430A-N are generated during the self-discovery process which is implemented during initialization of the host system.

Turning now toFIG. 5, a block diagram of one embodiment of a network graph500is shown. Network graph500includes nodes which are representative of the components of a fabric which have been detected during an automatic discovery process. In one embodiment, the components of the fabric are modeled as the interacting nodes of network graph500. A security processor (e.g., security processor305ofFIG. 3) is configured to detect and query the various components of the fabric to retrieve attributes associated with these components. In one embodiment, the security processor stores the attributes of the fabric components in entries of a table, with a separate entry for each component and a separate column for each attribute. Then, the security processor is configured to create network graph500based on the retrieved attributes in the entries of the table.

Network graph500is intended to represent the components of a given interconnect fabric. Each node of network graph500represents a component of the fabric which was discovered during the discovery process. For example, as shown inFIG. 5, network graph includes component505coupled to components510,515and520, component520coupled to components505,510,515and535, component530coupled to component515, component525coupled to component515, component535coupled to component520, and component540coupled to component510. After creating network graph500, the security processor utilizes network graph500to generate routing tables for each of the components of the fabric. Then, the security processor programs the routing tables into the components, allowing the components to utilize the routing tables when determining how to route received packets. It is noted that network graph500is merely one representation of a network graph. In other embodiments, other types of network graphs can be created by a security processor.

Turning now toFIG. 6, a block diagram of one embodiment of a fabric component600is shown. In one embodiment, fabric component600includes an input buffer605, a crossbar615, an output buffer620, a control unit630, and one or more routing tables635. The input buffer605is coupled to one or more input ports610A-N. Each input port610A-N is coupled to a corresponding component internal or external to the fabric, and input buffer605is configured to buffer message data received via input ports610A-N. Similarly, output buffer620is connected to one or more output ports625A-N. Each output port625A-N is coupled to a corresponding component internal or external to the fabric, and output buffer620is configured to buffer message data received from crossbar615for the corresponding link. In one embodiment, each of components320A-N (ofFIG. 3) include the logic of fabric component600. It should be understood that the arrangement in logic in fabric component600is representative of one particular embodiment. In other embodiments, other suitable arrangements of logic can be utilized and/or other logic can be included in fabric component600.

In one embodiment, crossbar615includes multiplexers that switch packets flowing from input ports610A-N to output ports625A-N based on control signaling provided by control unit630. In one embodiment, control unit630utilizes configuration parameters specified by software and routing information represented in the one or more routing tables635to control crossbar615to effect a particular routing of input message data from an input port610A-N to an output port625A-N. Control unit630inspects incoming message headers, performs lookups of routing tables635to determine the next hop, and controls crossbar615to forward the data to the proper output port625A-N. Control unit630also manages virtual channels, implements arbitration and filtering per the configuration data provided by software, and otherwise implements components of one or more routing protocols.

In various embodiments, routing tables635provide routing information for packets that pass though fabric component600. In one embodiment, routing tables635are implemented as a plurality of table entries, with each entry associated with corresponding destination, next hop, and port fields according to the specified routing path between a source component and a destination component. In other embodiments, the entries of routing tables635can also contain other fields used by control unit630such as alternate routes, path length, link error status, and so on. The next hop field can be the address of the next component for a multi-hop route or the next hop field can be the same as the destination if the next hop is the final destination. In various embodiments, the entries of tables635are determined at run-time during an autonomous discovery process.

Accordingly, tables635are writeable or otherwise programmable to implement varying routes or to enable reconfiguration for different numbers or arrangements of die or network topologies. In various embodiments, tables635can be implemented using any of a variety of configurable storage elements, such as a register file, in RAM or flash memory, and the like. These configurable elements and writeable table entries can be managed by a number of elements, including an operation system, hypervisor, security processor, a basic input/output system (BIOS), firmware, or a combination thereof. As an example, during system boot-up, a security processor (e.g., security processor305ofFIG. 3) can program routing tables635after discovering the topology of the overall fabric, component population, and system interconnections. In one embodiment, the topology of the fabric is unknown beforehand, with varying numbers of components and connections that vary between different versions and implementations of the physical system. In one embodiment, a security processor is configured to inspect the crossbar and interconnection topology to discover the implemented fabric topology. The security processor can also reconfigure the components to implement quality of service policies such as network and memory bandwidth guarantees.

Referring now toFIG. 7, one embodiment of a method700for performing an automatic discovery process of a fabric topology is shown. For purposes of discussion, the steps in this embodiment are shown in sequential order. However, it is noted that in various embodiments of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method700.

A processor (e.g., security processor305ofFIG. 3) queries a fabric block instance count register (e.g., fabric block instance count register310) to determine a number of components of an interconnect fabric (block705). Next, the processor queries each component of a plurality of components to retrieve various attributes associated with each component from a corresponding register (block710). In various embodiments, the attributes can include block type (e.g., coherent master, non-coherent master, crossbar, coherent slave, non-coherent slave), types and number of command and data ports, data bus width, clock speed, neighbor instance identifiers (IDs), fabric IDs, and other attributes.

Then, the processor utilizes the retrieved attributes to create a network graph of the fabric components (block715). Next, the processor generates routing tables from the network graph (block720). Then, the processor programs the routing tables into the fabric components (block725). After being programmed with the routing tables, the fabric components utilize the routing tables to determine how to route incoming packets (block730). After block730, method700ends.

In various embodiments, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various embodiments, such program instructions can be represented by a high level programming language. In other embodiments, the program instructions can be compiled from a high level programming language to a binary, intermediate, or other form. Alternatively, program instructions can be written that describe the behavior or design of hardware, Such program instructions can be represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog can be used. In various embodiments, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.