Chip multiprocessor with configurable fault isolation

One embodiment relates to a high-availability computation apparatus including a chip multiprocessor. Multiple fault zones are configurable in the chip multiprocessor, each fault zone being logically independent from other fault zones. Comparison circuitry is configured to compare outputs from redundant processes run in parallel on the multiple fault zones. Another embodiment relates to a method of operating a high-availability system using a chip multiprocessor. A redundant computation is performed in parallel on multiple fault zones of the chip multiprocessor and outputs from the multiple fault zones are compared. When a miscompare is detected, an error recovery process is performed. Other embodiments, aspects and features are also disclosed.

BACKGROUND

1. Technical Field

The present application relates generally to computer systems and microprocessors. More particularly, the present application relates to chip multiprocessors.

2. Description of the Background Art

Technology scaling and decreasing power efficiency of uniprocessors has led to the emergence of chip multiprocessors (CMP) as a hardware paradigm. In a CMP, multiple processor cores are integrated on a single chip and are available for general purpose computing.

Components on the die of a CMP (on-chip components) may be shared to improve resource utilization. For example, cores may be shared via hyperthreading, and last level caches and input/output (I/O) interfaces may be shared. In addition, typically off-chip components, such as memory controllers and I/O links, are being integrated onto CMPs.

While the above-mentioned sharing and integration may provide better resource utilization and improved performance, it also results in lower overall reliability. The lower overall reliability is because an error in any one component of the chip may lead to the non-availability of the entire CMP. For example, single processor failure typically results in the loss of availability of all processors on that CMP. Also, failure in a shared component like the cache or memory controller typically affects all the cores sharing that component. The failure in time (FIT) of the individual cores, caches, memory and I/O components may add up to a rather high FIT for the CMP as a whole.

SUMMARY

One embodiment relates to a high-availability computation apparatus including a chip multiprocessor. Multiple fault zones are configurable in the chip multiprocessor, each fault zone being logically independent from other fault zones. Comparison circuitry is configured to compare outputs from redundant processes run in parallel on the multiple fault zones.

Another embodiment relates to a method of operating a high-availability system using a chip multiprocessor. A redundant computation is performed in parallel on multiple fault zones of the chip multiprocessor and outputs from the multiple fault zones are compared. When a miscompare is detected, an error recovery process is performed.

Other embodiments, aspects, and features are also disclosed.

DETAILED DESCRIPTION

FIGS. 1A and 1Bare schematic diagrams depicting a conventional highly-available system architecture using chip multiprocessors. Shown inFIG. 1Ais a comparison (voting) circuit102which receives output from multiple redundant processing systems104. Each processing system104may utilize a CMP106, as shown, for example, inFIG. 1B.

In the particular system depicted inFIG. 1A, the comparison circuit102receives outputs from two redundant processing systems104-1and104-2(dual modular redundancy or DMR). In this case, if the comparison circuit102detects a mismatch in the two outputs, then an error is indicated. Recovery from the error may be accomplished by various conventional techniques, such as roll-back recovery for soft errors.

In other conventional systems, the comparison circuit102may receive outputs from three (or more) redundant processing systems104. With three redundant systems104(triple modular redundancy or TMR), the comparison circuit102may determine which output is in error by assuming that the other two (matching) outputs are correct (two-to-one vote). In that case, the erroneous output may be discarded.

The example CMP106depicted inFIG. 1Bis a generalization of conventional CMP designs from vendors such as Intel Corporation of Santa Clara, Calif., Advanced Micro Devices Inc. of Sunnyvale, Calif., and Sun Microsystems Inc. of Santa Clara, Calif. This generalized CMP design includes eight processing cores (P0, P1, P2, . . . , P7), eight private L1(level1) caches (L1for the instruction caches and D1for the data caches), a bidirectional ring interconnect108, a shared L2(level2) cache (organized into eight banks, B0, B1, B2, . . . , B7), four shared memory controllers (Mem Ctrl), and four shared input/output interface units (Link Adpt). As shown, the memory controllers are communicatively coupled to main memory (FBDIMM).

While a bidirectional ring interconnect108is shown inFIG. 1B, other types of communication systems may be used, such as meshes. Further, although the illustrated design has all cores on one side and the shared cache banks on the other side (a “dance hall” architecture), other designs may have banks and cores interleaved.

The shared input/output interface units (Link Adpt) of the CMP106are communicatively coupled to the comparison circuit102inFIG. 1A. In this conventional architecture, the system104acts as a single processing element to the comparison circuit102.

Applicant has identified a drawback to the conventional architecture discussed above. The conventional CMP106is designed for efficient resource utilization. However, the conventional CMP106lacks fault isolation properties. For example, fault isolation is lacking at the core level, at the shared L2cache, and at the shared memory controllers. As a result of the lack of fault isolation within each CMP106, redundancy is achieved only by using multiple CMPs106to provide the desired replication of processing elements.

Applicant has determined novel designs for achieving a highly-available processing system using one or more chip multiprocessors. These designs provide and utilize fault isolation within a chip multiprocessor.

FIG. 2is a schematic diagram showing a chip multiprocessor200designed with complete fault isolation. Here, the chip multiprocessor200is designed with multiple independent microprocessors fabricated on the same die. The illustrated example shown includes eight independent systems, each system including a processor core (P0, P1, P2, P3, P4, P5, P6, or P7), a private first level cache (L1for instructions and D1for data), a private second level cache (B0, B1, B2, B3, B4, B5, B6, or B7), an independent memory controller (Mem Ctrl), and an independent input/output interface (Link Adpt).

However, the architecture ofFIG. 2has several disadvantages. By not sharing cache resources and other elements, the overall performance of the system is significantly reduced. Similarly, by not sharing pins, the valuable pin resource would be inefficiently used.

FIG. 3Ais a schematic diagram of a chip multiprocessor300with configurable isolation in an unconfigured state in accordance with an embodiment of the invention. In accordance with one embodiment, the CMP300may be dynamically reconfigurable into a higher-availability configuration or a higher-performance configuration by setting a small number of control points in the design.

This capability enables the CMP300to support higher-availability on demand. Note that providing this capability may be accomplished with relatively small changes to the ring interconnect and bank addressing, leaving the rest of the CMP300unchanged.

Regarding the ring interconnect, cross links (see, for example, the ring configuration units or RCUs which are described further below) may be activated to partition a larger ring to create to two logically independent rings. The cross links may be less than a millimeter long, and the activation of the cross links may require insertion of a multiplexer at the input of a ring interface incoming data port. These cross links and input multiplexers are a small additional fixed cost in terms of area and power which would not significantly increase the cost of design in cases where higher availability is not desired. Regarding the bank addressing, in higher-availability mode, the interleave among level2cache banks would use one fewer bit, interleaving references among half the banks, so as to keep references within the same color.

FIG. 3Bis a circuit diagram of a ring configuration unit (RCU) in accordance with an embodiment of the invention. As shown, an RCU may be implemented using various multiplexers (MUXes). The MUXes may be controlled to “pass through” the signals to create a larger ring, or partition to divide the larger ring into separate segments.

For example, in the “pass through” configuration, the MUXes would be configured as follows. MUX352would be configured to “pass through” signal362to output364. MUX354would be configured to pass through signal366to output368. MUX356would be configured to pass through signal370to output372. Finally, MUX358would be configured to pass through signal374to output376.

On the other hand, in the partition configuration, the MUXes would be configured as follows. MUX352would be configured to “redirect” or “circulate” signal374to output364. MUX354would be configured to redirect or circulate signal370to output368. MUX356would be configured to redirect or circulate signal366to output372. Finally, MUX358would be configured to redirect or circulate signal362to output376.

While a configurable ring interconnect is described in detail herein, other configurable communication systems and designs may be utilized in alternate embodiments of the invention. For example, configurable busses, configurable switching meshes, or configurable crossbar switching systems may be utilized. Alternatively, the isolation of the interconnection into logically-isolated zones may be performed through software control on routing network traffic.

As discussed below, the chip multiprocessor300ofFIG. 3Amay be configurable into one or more “color” or fault zones. As used herein, a color or fault zone is a logically-isolated zone. The assignment of color zones (fault zones) to specific components of the chip multiprocessor may be based on pre-determined policies to optimize performance or maximize fault coverage.

Various components may require modification to support multiple color zones. For example, additional support for the shared cache system to enable reconfiguration into multiple color zones may include extra tag bits and added control circuitry.

FIG. 4Ais a schematic diagram of a chip multiprocessor400with configurable isolation as configured for a single color (fault) domain402in accordance with an embodiment of the invention. In this case, the CMP400is configured so as to operate similar to the CMP106discussed above in relation toFIG. 1B. To provide the ring interconnect for the single color domain402, each of the three RCUs shown may be configured to be in the “pass through” configuration. This effectively results in a single bi-directional ring interconnect for the single color domain402.

Like the conventional CMP106, when the configurable CMP is in this single color domain configuration400, it provides for efficient resource utilization. However, this single color domain configuration400lacks fault isolation properties.

FIG. 4Bis a schematic diagram of a chip multiprocessor420with configurable isolation as configured for two color (fault) domains in accordance with an embodiment of the invention. To provide the ring interconnect for the two color domains422-1and422-2, the “outer” two of the three RCUs shown may be configured to be in the “pass through” configuration, while the “center” RCU is configured to be in the partition configuration. This effectively results in two bi-directional ring interconnects for the two color domains422-1and422-2.

A first color zone includes a first group of four processors (P0, P1, P2, and P3), while a second color zone includes a second group of four processors (P4, P5, P6, and P7). A first bi-directional ring interconnect422-1interconnects the processors of the first color zone with a shared L2cache having four banks (B0, B1, B2, and B3) and also with shared input/output interface devices (Link Adpt). A second bi-directional ring interconnect422-2interconnects the processors of the second color zone with a shared L2cache having four banks (B4, B5, B6, and B7) and also with shared input/output interface devices (Link Adpt). In this particular implementation, each pair of L2cache banks (B0and B1, B2and B3, B4and B5, and B6and B7) shares a memory controller (Mem Ctrl) such that each color zone has two memory controllers. The memory controllers of each color zone are configured to access main memory (FBDIMMs) for that color zone.

The CMP420ofFIG. 4Bis configured so that the colored domains may be units of fault containment or isolation. Any failure in a color shared component affects computation only on the cores mapped to that color. In accordance with an embodiment of the invention, to ensure that a failure in one color domain does not impact any of the other colored domains on the CMP420, logical isolation may be applied for interconnect, caches, and memory controllers. The logical isolation allows components to be isolated at the hardware level.

In one application of the CMP420ofFIG. 4B, resources from the two color zones may be used to run one or more dual modular redundant (DMR) process pairs. In that case, when higher availability is required, computations in the first color zone would be replicated by computations in the second color zone. For example, for a higher-availability higher-cost solution, the voters may be implemented using hardware circuitry in I/O hubs connected to a first color link adapter and a second color link adapter. For a lower-availability lower-cost solution, the voters may be implemented in hypervisors that communicate between the colored partitions through input/output.

Physical memory may be partitioned between the two logical processors using unique virtual-to-physical memory mapping. To provide complete memory fault isolation, the operating system may be configured to support statically-partitioned TLB (translation lookaside buffer) entries. However, redundant TLBs may also be used for fault tolerance.

Once a fault is detected in the CMP420, a reconfiguration process may be performed. For example, a core processor that fails may be deleted (put out of commission), but the remaining cores are then still usable.

As another example, if a failure in a cache bank is detected that cannot be corrected by line sparing (i.e. the failure is a logic failure in the bank controller), then the other cache bank sharing the same memory controller may be reconfigured to cache all lines serviced by that memory controller. This may be enabled by the provision of an extra bit in the bank cache tags and a mode bit in the cache bank.

As yet another example, if a memory controller in a color domain fails, that color domain may be reconfigured to use a single memory controller. This may be performed by caching all lines in the cache banks associated with the failed memory controller in the remaining controller's banks and may be enabled using one more bit in the cache tags and a second mode bit.

Given the large number of bits in a typical cache line (over 600 bits for a 64 B cache line with ECC plus previously required tag bits), providing two more bits to enable such reconfigurations is very modest area overhead.

While the two color zones are shown inFIG. 4B, the number of color zones may be larger than two to provide further redundancy or to provide smaller granularity of fault containment. For example, three colors may be used either in a triple modular redundant (TMR) configuration or to have fewer cores in a fault domain.

FIG. 4Cis a schematic diagram of a chip multiprocessor440with configurable isolation as configured for four color (fault) domains in accordance with an embodiment of the invention. To provide the ring interconnect for the four color domains442-1,442-2,442-3, and442-4, each of the three RCUs shown may be configured to be in the partition configuration. This effectively results in four bi-directional ring interconnects for the four color domains442-1,442-2,442-3, and442-4.

A first color zone includes a first group of two processors (P0and P1). A second color zone includes a second group of two processors (P2and P3). A third color zone includes a third group of two processors (P4and P5). Finally, a fourth color zone includes a fourth group of two processors (P6and P7).

A first bi-directional ring interconnect442-1interconnects the processors of the first color zone with a shared L2cache having two banks (B0and B1) and also with shared input/output interface devices (Link Adpt). A second bi-directional ring interconnect442-2interconnects the processors of the second color zone with a shared L2cache having two banks (B2and B3) and also with shared input/output interface devices (Link Adpt). A third bi-directional ring interconnect442-3interconnects the processors of the third color zone with a shared L2cache having two banks (B4and B5) and also with shared input/output interface devices (Link Adpt). Finally, a fourth bi-directional ring interconnect442-4interconnects the processors of the fourth color zone with a shared L2cache having two banks (B6and B7) and also with shared input/output interface devices (Link Adpt).

In this particular implementation, each pair of L2cache banks (B0and B1, B2and B3, B4and B5, and B6and B7) shares a memory controller (Mem Ctrl) such that each color zone has a memory controller. The memory controller of each color zone is configured to access main memory (FBDIMMs) for that color zone.

Like the CMP420ofFIG. 4B, the CMP440ofFIG. 4Cis configured so that the colored domains may be units of fault containment or isolation. Any failure in a color shared component affects computation only on the cores mapped to that color. In accordance with an embodiment of the invention, to ensure that a failure in one color domain does not impact any of the other colored domains on the CMP440, logical isolation may be applied for interconnect, caches, and memory controllers. The logical isolation allows components to be isolated at the hardware level.

In one example application of the CMP440inFIG. 4C, the CMP440may be used to run different processes in TMR using triplets of three different colors each. For example, one TMR process may run simultaneously on the first, second and third colored domains, while a second process may run simultaneously on the second, third, and fourth colored domains. In this example, DMR processing may simultaneously be supported on the first and fourth colored domains to balance load among the colored domains. Various other applications of the CMP440are also possible that also utilize the reconfiguration and fault isolation properties of the architecture.

FIG. 5is a flow chart showing a method of operating a high-availability system using a chip multiprocessor with multiple color zones in accordance with an embodiment of the invention. The CMP is configured502into multiple color zones, for example, by activating cross links and so forth as discussed above.

Redundant computations are then run504in the different color zones. These redundant computations may be utilized, for example, for DMR or TMR operation. Comparison is made506of the redundant outputs.

If the comparison506indicates that the redundant outputs match (i.e. that the redundant operation is going okay, without any detected error), then the method continues to run504the redundant computation in the different colors.

On the other hand, if the comparison506indicates that the redundant outputs do not match (i.e. that an error is detected due to a miscompare), then the method goes on with a procedure to deal with a detected error. As shown in the flow chart ofFIG. 5, the procedure may depend508on whether the redundant computation is being run in TMR or DMR.

If the redundant computation is in a TMR mode, then the method may, for example, perform a roll-forward recovery procedure. In particular, perFIG. 5, the method may determine510the faulty process (by way of a 2-to-1 vote count). Per block512, the faulty process may then be isolated and computation may then be continued in DMR mode. In addition, roll-forward error recovery may be performed514. Such roll-forward error recovery takes the faulty process in its erroneous state and corrects it so that the process may then rejoin in the redundant computation.

On the other hand, if the redundant computation is in a DMR mode, then the method may, for example, perform a roll-back recovery procedure. In particular, perFIG. 5, the method may restore516a recent checkpoint at which there was no detected error in the states of the redundant processes. The redundant computation may then be restarted518from that checkpoint.

FIG. 6is a plot of results from a Monte-Carlo simulation showing benefits of an embodiment of the invention. The Monte-Carlo simulations were performed assuming a heavy workload on a single 8-core system under three configurations: (a) full-resource sharing602; (b) full isolation (private resources)604; and (c) configurable isolation606.

All three configurations were assumed to be running in a DMR mode. Overheads of using colored domains was included in the performance evaluation of the configurable isolation design606. In the DMR configuration, each color is assumed to have access to only half the cache, and so we model the performance assuming two L2caches, each half the size of the cache in the shared configuration.

The fault model used was based on data of failure-in-time rates and distributions of errors per component. For reasons of simplification, the fault model used was limited to performance impact from hard faults. Reconfiguration was assumed at each fault instance for the configurable isolation design606. It was assumed that reconfiguration is performed only at full component granularity.

The vertical (Y) axis shows the mean cumulative performance normalized to the baseline performance of the fully-shared configuration with no faults and averaged across the 10,000 simulations. The horizontal (X) axis shows time, measured in years.

As shown inFIG. 6, for all three configurations, the average performance begins high and then degrades with the occurrence of hard errors over time. The shared configuration602begins with the highest average performance at time zero, but as time progresses, it performs the worst with a degradation of about 30% to 35% after the first couple of years and a degradation of close to 50% by the end of five years.

The fully isolated configuration604, by virtue of its lack of resource sharing, begins with the lowest average performance at time zero. However, the fully isolated configuration604becomes performance competitive with the shared configuration602at around two years (at the crossover shown inFIG. 6).

The configurable isolation design606achieves the most attractive average performance. The performance of the configurable isolation design606starts off at time zero in between the performance of the other two configurations. Shortly thereafter, sometime during the first year, the average performance of the configurable isolation design606surpasses that of the shared configuration602and thereafter continues to have an average performance advantage.

Note that reconfiguration provides performance benefits for the configurable isolation design606as compared to the fully isolated configuration604. Consider the case where one component fails, for example, a bank controller. In the fully isolated configuration604, the entire core associated with the failed bank controller is taken out of commission. In contrast, in the configurable isolation design606, the core associated with the failed bank controller may be reconfigured to use other banks of the same color. As discussed above, this benefit may be provided with relatively little area overhead.

Note further that a minimum number of cores in a fault zone may depend on a number of factors. These factors may include, for example, the number of color zones, the number of RCUs, and the number of cores in the system. The detailed embodiment discussed above in relation to the figures only illustrates an example case for the number of cores in a fault zone, but other configurations are possible depending on such factors.