Modifying system routing information in link based systems

Methods and apparatus to improve modification of system routing information in link based systems are described. In one embodiment, entries in a first table (storing data corresponding to routing paths between a plurality of components prior to a hot-plug event) and a second table (storing data corresponding to routing paths between the plurality of components after a hot-plug event) may be compared to determine which corresponding routing registers are to be modified in response to the hot-plug event. Other embodiments are also disclosed.

REFERENCE TO RELATED APPLICATION

This application is related to and claims priority from International Application No. PCT/CN2007/003367 filed on Nov. 29, 2007 and entitled, “MODIFYING SYSTEM ROUTING INFORMATION IN LINK BASED SYSTEMS”; which is entirely incorporated herein by reference for all purposes.

BACKGROUND

The present disclosure generally relates to the field of electronics. More particularly, an embodiment of the invention relates to techniques for modifying system routing information in link based systems.

RAS (Reliability, Availability, and Serviceability) has become a critical feature for modern computer system, especially in the server platforms. In a link based system, such as CSI (Common System Interface), the successful implementation of RAS features such as socket (or link) hot-plug depends on reconfiguration of routing data during runtime. Generally, routing data regarding immediate neighbors of a member of a link based system may be stored in storage devices local to each member of the linked based system. Routing data reconfiguration operations may be handled transparently to the OS (Operation System) by utilizing processing time that would otherwise be used by the OS. Since the OS has its own latency requirement, minimizing the routing table reconfiguration time becomes a key criterion in RAS implementations.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth in order to provide a thorough understanding of various embodiments. However, some embodiments may be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the particular embodiments. Various aspects of embodiments of the invention may be performed using various means, such as integrated semiconductor circuits (“hardware”), computer-readable instructions organized into one or more programs (“software”) or some combination of hardware and software. For the purposes of this disclosure reference to “logic” shall mean either hardware, software, or some combination thereof.

Some of the embodiments discussed herein may allow for more efficient and/or faster change of system routing configuration in a link based system (such as a CSI system), e.g., to improve RAS. More particularly, some techniques enable relatively large computing systems (such as blade servers having many routes between various nodes which may also be referred to herein as agents such as discussed with reference toFIG. 1) to more efficiently and/or quickly change system routing configuration, e.g., by reducing the amount of RTA (Routing Table Array) registers that need to be computed and updated, for example, during hot-plug events such as during limited time windows available for SMI (System Management Interrupt), PMI (Platform Management Interrupt) or other interrupt servicing. Also, in an embodiment, system routing configuration may be modified without changing system topology. Additionally, some of the techniques discussed herein may be utilized in various types of computing environments, such as those discussed with reference toFIGS. 1-8.

More particularly,FIG. 1illustrates a block diagram of a computing system100, according to an embodiment of the invention. The system100may include one or more agents102-1through102-M (collectively referred to herein as “agents102” or more generally “agent102”). In an embodiment, the agents102may be components of a computing system, such as the computing systems discussed with reference toFIGS. 2-8.

As illustrated inFIG. 1, the agents102may communicate via a network fabric104. In one embodiment, the network fabric104may include a computer network that allows various agents (such as computing devices) to communicate data. In an embodiment, the network fabric104may include one or more interconnects (or interconnection networks) that communicate via a serial (e.g., point-to-point) link and/or a shared communication network. For example, some embodiments may facilitate component debug or validation on links that allow communication with fully buffered dual in-line memory modules (FBD), e.g., where the FBD link is a serial link for coupling memory modules to a host controller device (such as a processor or memory hub). Debug information may be transmitted from the FBD channel host such that the debug information may be observed along the channel by channel traffic trace capture tools (such as one or more logic analyzers).

In one embodiment, the system100may support a layered protocol scheme, which may include a physical layer, a link layer, a routing layer, a transport layer, and/or a protocol layer. The fabric104may further facilitate transmission of data (e.g., in form of packets) from one protocol (e.g., caching processor or caching aware memory controller) to another protocol for a point-to-point or shared network. Also, in some embodiments, the network fabric104may provide communication that adheres to one or more cache coherent protocols.

Furthermore, as shown by the direction of arrows inFIG. 1, the agents102may transmit and/or receive data via the network fabric104. Hence, some agents may utilize a unidirectional link while others may utilize a bidirectional link for communication. For instance, one or more agents (such as agent102-M) may transmit data (e.g., via a unidirectional link106), other agent(s) (such as agent102-2) may receive data (e.g., via a unidirectional link108), while some agent(s) (such as agent102-1) may both transmit and receive data (e.g., via a bidirectional link110).

FIG. 2illustrates a block diagram of a point-to-point (PtP) computing system200, according to one embodiment of the invention.FIG. 2shows an example of a CSI link based system with four processors or central processing units (CPUs) (CPU0through CPU3) and two interface devices202and204, which may be implemented as Input/Output Hubs (IOHs) in some embodiments.

InFIG. 2, the illustrated CPUs and interface devices may be CSI components communicating with each other through CSI links (illustrated as bidirectional arrows). Furthermore, the numbers inside each of the boxes or circles that are coupled to the links illustrate sample port numbers for a given device. As shown inFIG. 2, each of the CPUs may be coupled to a memory subsystem that it is accessible by the respective CPU. Alternatively, one or more of the CPUs may share a memory unit (not shown), in addition to or in place of a dedicated memory subsystem. In some embodiments, the interface devices202-204may provide connectivity to I/O devices such as a Peripheral Component Interconnect Express (PCIe) (e.g., complying with PCIe Specification, Revision 2.0, October 2006) and/or an I/O Controller Hub (ICH) such as discussed further with reference toFIG. 7. In one embodiment, the processor-to-processor communication and/or between the processor and the interface device(s) may be performed using CSI packets. Furthermore, each of the CSI components (e.g., including one or more of the CPUs and/or interface devices) may contain a RTA and a Source Address Decoder (SAD). The RTA may provide the CSI packet routing information to other sockets. The SAD may provide a mechanism to represent routing of the resources such as memory, I/O, etc.

For the purpose of explaining one embodiment, assume that CPU3inFIG. 2needs to be removed while the OS is running. In some embodiments, the CSI components support a Quiesce mode by which normal traffic could be paused during the RTA/SAD change operations. Generally, such events which involve adding, removing, and/or modifying components of a computing system may be referred to herein as “hot-plug” events. To handle the event associated with removing CPU3, the platform may generate an SMI to inform the Basic Input Output System (BIOS) that a component needs to be removed. In some implementations, the removal operation after the SMI may involve the following:

1. Select one CPU as the Monarch CPU, which is responsible for executing most of SMI events handling code. (Below, CPU0is taken as Monarch CPU for example);

2. Monarch CPU quiesces the whole system to pause all the traffic going through the CSI links;

3. Monarch CPU computes the new values for the system configuration RTA registers and updates them;

4. Monarch CPU performs other system configuration operations, e.g., computing the new values for SAD registers and updating them, disabling links to the hot-removed socket, etc.; and

5. Monarch CPU de-quiesces the system and releases the non-monarch processors and returns from the SMI. The system continues to run.

In step3above, to change system routings, the re-computing routing is performed by first obtaining the new topology of the routing fabric of the socket (or link) being removed. Second, the new values for all RTA registers may be computed. Finally, all RTA registers are updated. This approach is rather inefficient because it will have to go through all the RTA registers, computing their values and updating them, even though they preserve the old value after the hot-plug event, e.g., those RTA registers for routings between CPU0and CPU1in the example ofFIG. 2. Even worse, for a real system, the RTAs could be a big amount. For example, some CPUs may have 12 ports with 80 RTA entries on each port respectively. So, the process of changing system routings can introduce significant latency, which in turn increases the system pause time during a hot-plug event, even exceeding the OS tolerable deadline in a larger system.

FIGS. 3-4illustrate flow diagrams of methods300and400, respectively, that may be utilized to more efficiently and/or quickly update routing information in a computing system, according to an embodiment. In one embodiment, various components discussed herein, e.g., with reference toFIGS. 1-2and5-8may be utilized to perform one or more of the operations discussed with reference toFIG. 3or4.

Referring toFIGS. 1-3, at an operation302, after initiation of a hot-plug event (e.g., as indicated by an interrupt such as an SMI or PMI), new topology of the routing fabric of the socket (or link) being removed, added, and/or modified may be obtained, e.g., by reference to a system level routing table. In one embodiment, an intermediate data table (which may be referred to herein as a “Routing-Data-Table”) may be used to compute register values for the new routing fabric as well as filter the changed RTA registers. The cells of the Routing-Data-Table may contain the information to route the transactions from a source to a destination.

For example, Table 1 below illustrates an example table corresponding toFIG. 2before CPU3is removed. The routing information is computed through the routing algorithm, which could be specific or based on the system requirement. The example of the Table 1 is computed by the minimal distance algorithm; namely, each cell in the table indicates the minimal routing path from a given source (row) to a give destination (column). However, embodiments of the invention are not limited to the minimal distance algorithm and any routing algorithm may be used with its output adapted to be compliant with the format of the Routing-Data-Table.

TABLE 1Example of (Current) Routing-Data-Table for Full ConnectionToFromCPU0CPU1CPU2CPU3IOH0IOH1CPU0n/a31241.2CPU11n/a2342.3CPU232n/a12.34CPU3213n/a1.24IOH0010.10.1n/a0.1IOH10.10.1010.1n/a

Moreover, such routing information may be the intermediate result to compute the final RTA register value. For a link based system, the routing fabric on each component may be implemented with ports, entries, and virtual channels, etc. So, a number of successive computations may be performed. Also, depending on the implementation, some or all of the routing table information may be computed at system startup or otherwise before you before-hand. Other implementations may compute at least some of the routing table information during a hot-plug event. Furthermore, in some embodiments, the new topology may be obtained by various means at operation302, such as based on information provided by a third-part system management/assistant agent, being discovered dynamically, etc.

At an operation304, any necessary modifications may be determined. For example, since each cell in the Routing-Data-Table contains the routing information from the source to the destination, if a cell value does not change before and after the hot-plug event, the corresponding RTA register values will not need to change either; otherwise, RTA register values are changed to route the transitions to new ports or paths.

For example, Table 2 below illustrates an example of (New) Routing-Data-Table for CPU3to be removed fromFIG. 2. In an embodiment, at operation304, by comparing the cell values in the Routing-Data-Tables of the topology before (Table 1) and after the hot-plug event (Table 2), the RTA registers, which need to be computed further and updated finally, could be filtered out to increase speed and/or reduce latency associated with a hot-plug event.

TABLE 2Example of (New) Routing-Data-Table for CPU3 to be RemovedToFromCPU0CPU1CPU2CPU3IOH0IOH1CPU0n/a31n/a41CPU11n/a2n/a42CPU232n/an/a2.34CPU3n/an/an/an/an/an/aIOH0010.1n/an/a0.1IOH1000n/a0n/a

For example, by comparing Table 2 with Table 1, it becomes apparent that removal of CPU3results in changes to 6 cells, while 15 cells preserve their previous values. In some embodiments, the current table may be stored in a different memory device than the new table (e.g., the tables may be stored in different memory subsystems or caches discussed with reference toFIG. 2). Alternatively, the tables may be stored in the same memory device (such a shared memory device or a private memory device).

For example, assuming that for each cell time T is needed for the successive computation and hardware updating, then the time spent is reduced from (15+6)T=21T to 6T and it is 3.5 times faster. If one considers the hot-add case (e.g., adding CPU3), then the Table 1 corresponds to the Routing-Data-Table after the topology changes, while the Table 2 is the original configurations. This is because the cells from and to CPU3should be counted, so the time spent is reduced from (21+10)T=31T to (6+10)T=16T, which is about 2 times faster. Accordingly, based on the determination made at operation304, an operation306computes the new values for the filtered RTA registers. At an operation308, the routing information may be updated based on the computed values of operation306(e.g., only filtered RTA registers of operation304are updated).

FIG. 4illustrates a flow diagram of a method400that may be used to perform operation304ofFIG. 3, according to an embodiment. At an operation402, new Routing-Data-Table for the new topology may be computed. In one embodiment, operation402may be performed for neighboring components of the component being removed, added, and/or modified. At an operation404, the new table may be compared against the current table as discussed with reference toFIG. 3. Based on the comparison of operation404, the registers (e.g., RTA registers) that are to be modified (or filtered) can be determined at operation406. At an operation408, the current routing table may be updated with data from the new routing table computed at operation402.

Because some embodiments may filter only changed RTA registers, as systems increase the number of their components, it becomes apparent that such embodiments will also result in better returns. For example, inFIG. 5, a relatively large system500with 12 processors is shown, where CPU7(which is in the center of the system) is to be added. Table 3 below corresponds to system500. As can be seen, the number of cells changed and unchanged are 62 and 70 respectively, so it improves the performance for (62+70)T to 62T, which is more than 2 times faster.

Further, in system600ofFIG. 6, a corner CPU (CPU4) is to be removed. Table 4 below corresponds to system600. For this example, only 3 cells are change, so the time spent on routing reconfiguration is reduced from (3+107)T to 3T, which is more than 35 times faster.

Accordingly, in some embodiments, an intermediate routing data table which contains routing information from a source to a destination may be used. By comparing data tables of before and after a hot-plug event, a minimal set of RTA registers need be computed and update to increase performance. Furthermore, even though socket hot-plug events are discussed herein as examples, embodiments discussed herein also work well with link hot-plug. The above example uses the SMI to describe the invention; similar techniques may be used in systems with the PMI.

FIG. 7illustrates a block diagram of an embodiment of a computing system700. One or more of the components ofFIG. 1and/orFIG. 2may comprise one or more components discussed with reference to the computing system700. The computing system700may include one or more central processing unit(s) (CPUs)702(which may be collectively referred to herein as “processors702” or more generically “processor702”) coupled to an interconnection network (or bus)704. The processors702may be any type of processor such as a general purpose processor, a network processor (which may process data communicated over a computer network104), etc. (including a reduced instruction set computer (RISC) processor or a complex instruction set computer (CISC)). Moreover, the processors702may have a single or multiple core design. The processors702with a multiple core design may integrate different types of processor cores on the same integrated circuit (IC) die. Also, the processors702with a multiple core design may be implemented as symmetrical or asymmetrical multiprocessors.

The processor702may include one or more caches (not shown), which may be private and/or shared in various embodiments. Generally, a cache stores data corresponding to original data stored elsewhere or computed earlier. To reduce memory access latency, once data is stored in a cache, future use may be made by accessing a cached copy rather than refetching or recomputing the original data. The cache(s) may be any type of cache, such a level 1 (L1) cache, a level 2 (L2) cache, a level 3 (L-3), a mid-level cache, a last level cache (LLC), etc. to store electronic data (e.g., including instructions) that is utilized by one or more components of the system700.

A chipset706may additionally be coupled to the interconnection network704. Further, the chipset706may include a memory control hub (MCH)708. The MCH708may include a memory controller710that is coupled to a memory712. In an embodiment, the MCH may also include graphics logic and as a result may be referred to as a graphics MCH (GMCH). The memory712may store data, e.g., including sequences of instructions that are executed by the processor702, or any other device in communication with components of the computing system700. In an embodiment, the memory712may be the same or similar to the memory subsystems shown inFIG. 2. Also, in one embodiment of the invention, the memory712may include one or more volatile storage (or memory) devices such as random access memory (RAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), etc. Nonvolatile memory may also be utilized such as a hard disk. Additional devices may be coupled to the interconnection network704, such as multiple processors and/or multiple system memories.

The MCH708may further include a graphics interface714coupled to a display device716(e.g., via a graphics accelerator in an embodiment). In one embodiment, the graphics interface714may be coupled to the display device716via an accelerated graphics port (AGP). In an embodiment of the invention, the display device716(such as a flat panel display) may be coupled to the graphics interface714through, for example, a signal converter that translates a digital representation of an image stored in a storage device such as video memory or system memory (e.g., memory712) into display signals that are interpreted and displayed by the display716.

As shown inFIG. 7, a hub interface718may couple the MCH708to an input/output control hub (ICH)720. The ICH720may provide an interface to input/output (I/O) devices coupled to the computing system700. The ICH720may be coupled to a bus722through a peripheral bridge (or controller)724, such as a peripheral component interconnect (PCI) bridge that may be compliant with the PCIe specification, a universal serial bus (USB) controller, etc. The bridge724may provide a data path between the processor702and peripheral devices. Other types of topologies may be utilized. Also, multiple buses may be coupled to the ICH720, e.g., through multiple bridges or controllers. For example, the bus722may comply with the PCI Local Bus Specification, Revision 7.0, Mar. 9, 7004, available from the PCI Special Interest Group, Portland, Oreg., U.S.A. (hereinafter referred to as a “PCI bus”). Alternatively, the bus722may comprise a bus that complies with the PCI-X Specification Rev. 7.0a, Apr. 73, 7003, (hereinafter referred to as a “PCI-X bus”) and/or PCIe specification, available from the aforesaid PCI Special Interest Group, Portland, Oreg., U.S.A. Further, the bus722may comprise other types and configurations of bus systems. Moreover, other peripherals coupled to the ICH720may include, in various embodiments of the invention, integrated drive electronics (IDE) or small computer system interface (SCSI) hard drive(s), USB port(s), a keyboard, a mouse, parallel port(s), serial port(s), floppy disk drive(s), digital output support (e.g., digital video interface (DVI)), etc.

The bus722may be coupled to an audio device726, one or more disk drive(s)728, and a network adapter730(which may be a NIC in an embodiment). In one embodiment, the network adapter730or other devices coupled to the bus722may communicate with the chipset706. Other devices may be coupled to the bus722. Also, various components (such as the network adapter730) may be coupled to the MCH708in some embodiments of the invention. In addition, the processor702and the MCH708may be combined to form a single chip.

Additionally, the computing system700may include volatile and/or nonvolatile memory (or storage). For example, nonvolatile memory may include one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), a disk drive (e.g.,728), a floppy disk, a compact disk ROM (CD-ROM), a digital versatile disk (DVD), flash memory, a magneto-optical disk, or other types of nonvolatile machine-readable media capable of storing electronic data (e.g., including instructions).

FIG. 8illustrates a computing system800that is arranged in a point-to-point (PtP) configuration, according to an embodiment of the invention. In particular,FIG. 8shows a system where processors, memory, and input/output devices are interconnected by a number of point-to-point interfaces. The operations discussed with reference toFIGS. 1-7may be performed by one or more components of the system800.

As illustrated inFIG. 8, the system800may include several processors, of which only two, processors802and804are shown for clarity. The processors802and804may each include a local memory controller hub (MCH)806and808to enable communication with memories810and812. The memories810and/or812may store various data such as those discussed with reference to the memory712ofFIG. 7. As shown inFIG. 8, the processors802and804may also include one or more cache(s) discussed with reference toFIG. 7.

In an embodiment, the processors802and804may be one of the processors702discussed with reference toFIG. 7. The processors802and804may exchange data via a point-to-point (PtP) interface814using PtP interface circuits816and818, respectively. Also, the processors802and804may each exchange data with a chipset820via individual PtP interfaces822and824using point-to-point interface circuits826,828,830, and832. The chipset820may further exchange data with a high-performance graphics circuit834via a high-performance graphics interface836, e.g., using a PtP interface circuit837.

In at least one embodiment, one or more operations discussed with reference toFIGS. 1-7may be performed by the processors802or804and/or other components of the system800such as those communicating via a bus840. Other embodiments of the invention, however, may exist in other circuits, logic units, or devices within the system800ofFIG. 8. Furthermore, other embodiments of the invention may be distributed throughout several circuits, logic units, or devices illustrated inFIG. 8.

Chipset820may communicate with the bus840using a PtP interface circuit841. The bus840may have one or more devices that communicate with it, such as a bus bridge842and I/O devices843. Via a bus844, the bus bridge842may communicate with other devices such as a keyboard/mouse845, communication devices846(such as modems, network interface devices, or other communication devices that may communicate with the computer network104), audio I/O device, and/or a data storage device848. The data storage device848may store code849that may be executed by the processors802and/or804.

In various embodiments of the invention, the operations discussed herein, e.g., with reference toFIGS. 1-8, may be implemented as hardware (e.g., circuitry), software, firmware, microcode, or combinations thereof, which may be provided as a computer program product, e.g., including a machine-readable or computer-readable medium having stored thereon instructions (or software procedures) used to program a computer to perform a process discussed herein. Also, the term “logic” may include, by way of example, software, hardware, or combinations of software and hardware. The machine-readable medium may include a storage device such as those discussed herein. Additionally, such computer-readable media may be downloaded as a computer program product, wherein the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a bus, a modem, or a network connection).

Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least an implementation. The appearances of the phrase “in one embodiment” in various places in the specification may or may not be all referring to the same embodiment.