HAT: Heterogeneous Adaptive Throttling for On-Chip Networks
Kevin Kai-Wei Chang, Rachata Ausavarungnirun, Chris Fallin, Onur Mutlu
Carnegie Mellon University{kevincha,rachata,cfallin,onur}@cmu.edu

Abstract—The network-on-chip (NoC) is a primary sharedresource in a chip multiprocessor (CMP) system. As core counts
continue to increase and applications become increasingly
data-intensive, the network load will also increase, leading
to more congestion in the network. This network congestion
can degrade system performance if the network load is not
appropriately controlled. Prior works have proposed sourcethrottling congestion control, which limits the rate at which
new network traffic (packets) enters the NoC in order to reduce
congestion and improve performance. These prior congestion
control mechanisms have shortcomings that significantly limit
their performance: either 1) they are not application-aware,
but rather throttle all applications equally regardless of applications’ sensitivity to latency, or 2) they are not network-loadaware, throttling according to application characteristics but
sometimes under- or over-throttling the cores.
In this work, we propose Heterogeneous Adaptive Throttling,
or HAT, a new source-throttling congestion control mechanism based on two key principles: application-aware throttlingand network-load-aware throttling rate adjustment. First, weobserve that only network-bandwidth-intensive applications(those which use the network most heavily) should be throttled,allowing the other latency-sensitive applications to make fasterprogress without as much interference. Second, we observethat the throttling rate which yields the best performancevaries between workloads; a single, static, throttling rate underthrottles some workloads while over-throttling others. Hence,
the throttling mechanism should observe network load dynamically and adjust its throttling rate accordingly. While some pastworks have also used a closed-loop control approach, none havebeen application-aware. HAT is the first mechanism to combine
application-awareness and network-load-aware throttling rateadjustment to address congestion in a NoC.
We evaluate HAT using a wide variety of multiprogrammed
workloads on several NoC-based CMP systems with 16-, 64-,
and 144-cores and compare its performance to two state-ofthe-art congestion control mechanisms. Our evaluations showthat HAT consistently provides higher system performance andfairness than prior congestion control mechanisms.

I. I NTRODUCTION
In chip multiprocessors (CMP), the interconnect serves as
the primary communication substrate for all cores, caches,and memory controllers. Since the interconnect implements
cache coherence between fast L1 caches and lies on the
datapath for all memory accesses, its latency and bandwidth
are important. Hence, a high-performance interconnect isnecessary to provide high system performance. At high core
counts, simpler interconnects such as busses and crossbars
no longer scale adequately. The most commonly proposedsolution is a Network-on-Chip (NoC), which allows cores
to communicate with a packet-switched substrate [6]. A
two-dimensional mesh [5] is frequently implemented forlarge CMPs [18], [20], [39].

NoC designers also encounter on-chip hardware implementation constraints, such as total power (due to thermal limits) and chip area consumed by the network. Because these resources are often tight in modern CMPs,
NoC designs with a large amount of buffers and wires
are sometimes not practical. To maintain overall design
efficiency, the system must be provisioned reasonably, andthus will sometimes experience network congestion when
the network load (the fraction of occupied links or buffers)
becomes too high. When this congestion occurs, packets
contend for shared network resources frequently, and thiscan reduce overall system throughput. Congestion in onand off-chip interconnects is a well-known problem that has
been described in many prior works (e.g., [2], [9], [11], [15],[30], [31], [32], [38]). Congestion can significantly degrade
system performance if it is not properly managed.
A common congestion-control approach is source throttling in which network nodes (sources) that are injecting
significant amounts of traffic and interfering with the restof the network are throttled, or temporarily prevented frominjecting new traffic. A number of previous works haveshown source throttling to be effective in interconnects [2],[15], [30], [31], [32], [38]. By reducing overall network
load, source throttling reduces congestion and thus improves
network performance.
Unfortunately, state-of-the-art source-throttling mechanisms have shortcomings that limit their performance, which
we address in this work. First, many of the past works
are not application-aware (e.g., [2], [32], [38]), i.e., do nottake application characteristics into account when choosing which applications to throttle. As we show in thiswork, application-aware throttling decisions are essential
to achieving good performance. Second, in the works thatare application-aware (e.g., [30], [31]), the throttling rate
itself (how aggressively the throttled nodes are prevented
from injecting traffic) is adjusted in a manner that does
not take into account the load or utilization of the network,
which can lead to performance loss as it causes either overor under-throttling of the sources. Adjusting the throttlingrate by observing network load and using a closed-loop
feedback mechanism that takes into account network loadyields higher performance, as we show in this work.
To overcome these shortcomings, this paper proposes
Heterogeneous Adaptive Throttling (HAT), a new sourcethrottling mechanism. Unlike past works, HAT is the firstto combine two key principles: application-aware throttlingand network-load-aware throttling rate adjustment. HAT
selectively throttles only some nodes (applications) in thenetwork, chosen according to application characteristics,
such that overall congestion is reduced without significant
impact on any throttled node. This application-awarenessallows applications that have little impact on network load tomake fast forward progress. HAT then dynamically adjusts
throttling rate of throttled nodes in a closed-loop fashion
based on the total load (or utilization) of the network. Adapting the throttling rate to the load on the network minimizes
the over- or under-throttling that occurs in a mechanism that
does not adapt the throttling rate to network conditions. By
combining the aforementioned two key principles together,
HAT provides better performance than past source-throttlingmechanisms [30], [31], [38], as we show in our quantitativeevaluations.
In summary, our contributions are:
• We observe that two key principles are necessary toachieve high performance using source throttling in acongested on-chip network: application-aware throttling(choosing which applications to throttle based on application characteristics), and network-load-aware throttlingrate adjustment (adjusting the aggressiveness of source
throttling dynamically to avoid under- or over-throttling
the applications).
• We introduce Heterogeneous Adaptive Throttling (HAT),
the first source-throttling mechanism that uses these two
key principles together to control congestion in a network. HAT 1) throttles bandwidth-intensive applications,thus allowing latency-sensitive applications to make fast
progress, and 2) throttles these selected applications at a
rate that is dynamically adjusted based on the network
load, thus maximizing efficient utilization of the network.
• We qualitatively and quantitatively compare our new congestion control mechanism to two state-of-the-art sourcethrottling congestion-control mechanisms [30], [31], [38]on a variety of interconnect designs with different routerdesigns and network sizes using a wide variety of workloads. Our results show that HAT provides the best system
performance and fairness.
II. BACKGROUND
A. Networks-on-Chip for Multi-Core Processors
In CMP systems (e.g., [18], [43]), NoCs (networks-onchip) serve as communication fabrics to connect individual
processing nodes. Each node has a core, a private cache, anda slice of the shared cache. One NoC router is placed at eachnode, and the NoC routers are interconnected by links. The
routers convey packets between nodes. In the CMP systemthat we evaluate, NoCs primarily service private cache missrequests, carrying request and data packets between private
caches, shared caches, and memory controllers. Therefore,
NoCs are on the critical path of every cache miss, making
NoC performance important for providing good CMP system
performance. Many NoC topologies have been proposed;
however, a 2D mesh is one of the most commonly used
network topologies in commercial NoC-based manycore
CMPs [43] and research prototypes [16], [18], [20], [37]because of the low layout complexity.

B. Network-on-Chip Operation and Router Design
NoCs carry packets between nodes (cores, caches, andmemory controllers) in a CMP. Each packet consists of one
or multiple flits, which are the unit of flow control. One flitis typically the unit of data conveyed by one network linkin one cycle. When a router receives a packet, it decides
which link to forward the packet to. When many packets
are in the network, they can contend for shared resources,
such as links. Conventional router designs often have input
buffers at each router input [3], [5]. A packet that cannot
obtain a link to travel to its next hop can wait in an
input buffer. Alternatively, some other router designs [13],[28] eliminate these input buffers and instead misroute acontending packet to a different output link. The routing
algorithm in such a bufferless NoC design ensures thateach packet nevertheless eventually arrives at its destination.
(We will evaluate our mechanisms on both such network
designs.) In both input-buffered and bufferless NoC routers,
performance degrades when contention for links or buffers
occurs frequently. Source throttling reduces such congestion
by limiting the rate at which traffic can enter the network.
III. M OTIVATIONIn this section, we first demonstrate that source-throttling
can improve system performance. Then, we show thatapplication-aware source-throttling is necessary when different applications share the same NoC. These two observations have motivated past work on source throttling [30],[31]. Finally, we show that the throttling rate which provides
the best performance is not constant across workloads, motivating the need for network-load-aware throttling rate adjustment. This final observation leads to our application-awareand network-load-aware Heterogeneous Adaptive Throttling(HAT).
Source Throttling Congestion Control: Source throttling [2], [15], [30], [31], [32], [38] is a commonly usedtechnique to address congestion in NoCs and general packetswitched networks. The idea is to reduce network loadwhen the network is congested by throttling packets entering
the network in order to enhance system performance. In
a source-throttling system, sources (network nodes which
inject packets) are notified when they are causing network
congestion, either by other network nodes or by a central
controller. Upon receiving such a notification, the source
temporarily throttles (delays) its packets queued for network
injection. By reducing the injection rate (rate of packets
entering the network), throttling reduces network load andhence reduces congestion.
To show the benefits of source throttling, we evaluate
the system performance of three heterogeneous application
workloads on a NoC-based CMP system (details in §VI)
when different static throttling rates are applied to all applications. As we will describe in detail later, a throttlingrate is simply the probability that any given injection of anew packet into the network is blocked (e.g., 80% throttlingallows new packets to be injected only 20% of the time).
Figure 1 shows that as the network is throttled (increasing
throttling rate), the system performance also increases toPerformancea point, because the reduced rate of injecting new packets
reduces contention for the shared resources in the network, hence reducing congestion. However, past a point,
the source-throttling is too aggressive: it not only reduces
congestion but also limits forward progress unnecessarily,
degrading performance. We conclude from this experimentthat throttling can improve system performance when applied appropriately.
16
14
12
10
8
6
4

92%
90%

workload 1
workload 2
workload 3

80

82

94%

84

86 88 90 92 94 96 98
Throttling Rate (%)
Figure 1. Performance of randomly mixed workloads with varied homogeneous throttling rate on 16-node (4x4) CMP systems.

Application-Aware Throttling: We motivate the importance of differentiating applications based on their intensity
by showing that the overall and per-application performancecan be significantly impacted when different applications arethrottled. To illustrate this point, we reproduce an experiment shown in Nychis et al. [30], [31] to motivate theirapplication-aware technique. Source throttling is appliedin two different ways to a 16-application workload on
a 4x4-mesh NoC-based CMP. This workload consists ofeight instances of mcf, and eight of gromacs, which are
network-intensive (use significant bandwidth) and networknon-intensive (use little bandwidth), respectively. Note that
although we are showing one particular workload here, its
behavior is representative of workloads with mixed application intensities in general. Table I shows the performance of
this workload when (i) only mcf is throttled, and (ii) only
gromacs is throttled (in both cases, by a fixed throttling rateof 95%), relative to the baseline where no applications arethrottled.
Table I shows three application (instruction) throughput
measurements for each throttling case normalized to nothrottling: overall system throughput, average across mcf
instances, and average across gromacs instances. We reiterate the following key observations demonstrated in priorwork [30], [31]. Throttling different applications affects
overall system performance differently. Throttling mcf (second column) increases system performance by 9%, whereas
throttling gromacs (third column) decreases system performance by 2%. The main reason is that applications responddifferently to network latency variations. Throttling mcf
significantly reduces network interference and this allows
gromacs to make faster progress (by 14%) with less network latency. Also, mcf’s performance increases due toreduced network congestion and because it is not sensitive
to increased network latency. In contrast, throttling gromacs
negatively impacts gromacs by 4% without any benefit tomcf. The reason for this behavior is that gromacs injects
fewer network packets than mcf, thus each packet represents
a greater fraction of forward progress for gromacs. As aresult, network-non-intensive applications (e.g., gromacs)
are more sensitive to network latency and network-intensive
applications (e.g., mcf) are more tolerant to network latency.
Based on these observations, we conclude that throttling
network-intensive applications is more beneficial to system
performance than throttling network-non-intensive applications. We thus use application-aware throttling in HAT
based on network intensity.
MetricNo Throttle Throttle mcf Throttle gromacs
System Throughput
1.0
1.09
0.98
mcf Throughput
1.0
1.05
1.0
gromacs Throughput
1.0
1.14
0.96
Table I. Normalized instruction throughput of selectively throttling in an
application-aware manner.

Network-Load-Aware Throttling Rate Adjustment: To
understand the benefits of adjusting throttling rate dynamically, let us re-examine Figure 1. As we have already argued,
throttling can be applied to all the workloads evaluated inthis figure to improve system performance. However, eachworkload requires a different throttling rate to achieve peak
performance (in the example given, 94% for workload 1,
92% for workload 2, and 90% for workload 3). Prior sourcethrottling techniques use either a static throttling rate [38] oran open-loop rate adjustment based on measured application
characteristics [30], [31]. As we will demonstrate in our results (§VII), adjusting the throttling rate dynamically with a
network-load-aware feedback mechanism can tune throttlingrate more precisely for higher performance. Thus, the second
key principle of Heterogeneous Adaptive Throttling (HAT)
is network-load-aware throttling rate adjustment.
IV. HAT: H ETEROGENEOUS A DAPTIVE T HROTTLING
A. Overview
Our proposed congestion-control mechanism, HAT, isdesigned to improve system performance through source
throttling. HAT applies throttling to reduce network loadwhen the network is congested, reducing interference andthus improving performance. In order to be most effectiveat achieving high system throughput, HAT applies two key
ideas, as motivated above: application-aware throttling andnetwork-load-aware throttling rate adjustment.
The first component is application-aware throttling. HAT
throttles network-intensive applications that severely interfere with network-non-intensive applications, and does not
throttle the network-non-intensive applications since they
do not significantly impact network load. As a result,the unthrottled network-non-intensive applications can makefaster progress with less interference. To determine whichapplications to throttle, HAT first measures applications’network intensity. Then HAT divides applications into twogroups based on their intensities and performs throttling on
the network-intensive group. Since it applies throttling tosome applications, HAT is heterogeneous.
The second component, network-load-aware throttlingrate adjustment, aims to find a throttling rate that reduces
congestion sufficiently (i.e., does not throttle too lightly)
while not severely degrading performance (by throttling too
heavily). In order to dynamically adapt to different workloads, HAT measures network load (in a simple way that

we define below), and controls throttling rate in a closedloop manner to keep network load near an empiricallydetermined optimal value (which is constant for a given
network topology and size). This ensures that the available
network bandwidth is efficiently used without undue congestion.
B. Source Throttling
We define throttling as the blocking of new traffic injection into the network from a particular application inorder to reduce load. We use a throttling technique thatis soft: throttled nodes can still inject with some non-zeroprobability, explained below. We also assume that each node
has separate injection queues for request packets and data
response packets to other nodes, and that only a node’s own
request packets are throttled. For a throttling rate r, every
time a node attempts to inject a request packet, it chooses toblock the injection with probability r. Throttling rates range
from 0% (unthrottled) to 100% (fully blocked). In this work,
the maximum throttling rate is set to 95% (mostly blocked),
explained further in §IV-D. When a packet is blocked frominjection, it will retry its injection in subsequent cycles until
it is allowed to inject.
C. Application-Aware Throttling
To perform application-aware throttling, HAT first observes each application’s network intensity, classifies applications as network-intensive or network non-intensive
based on their intensities, and then applies throttling only
to network-intensive applications.
Measuring Application Intensity: HAT must measureapplications’ network intensity in order to classify them.
Because the network in our baseline CMP design primarily
serves L1 cache misses (due to the cache hierarchy design
which places the NoC between private L1 caches and ashared, globally distributed L2 cache), HAT uses L1 MPKI
(misses per thousand instructions) as a metric that closely
correlates to network intensity (note that this measurement
also correlates with network injection rate). L1 MPKI iseasy to measure at each core with two counters (completed
instruction count and cache miss count).1 Other works [7],[8], [24], [29] also use L1 MPKI to measure application
intensity.
Application Classification: Once application intensity ismeasured, HAT must split applications into intensive
and non-intensive categories. Some past application-aware
source throttling techniques have used a threshold to split
applications: all applications with a network intensity above
a certain threshold are considered network-intensive. However, we observe that the NoC fundamentally has sometotal network capacity, and that throttling an individual
application that is above some threshold may over- or underthrottle it depending on the application’s network intensity.
If all applications have intensity just below the threshold,none would be throttled. In contrast, if total network load
1 Note that we do not use L1 MPKC (misses per thousand cycles) because
it is dependent on the throttling rate, hence dynamic throttling changes
could alter application classifications and induce control instability.

is low with only one application’s intensity just over thethreshold, that application could be unnecessarily throttled.
Thus, HAT instead sets a threshold on the total network
intensity of all unthrottled applications. This ensures thatunthrottled applications do not provide sufficient load tocause network congestion. All other applications are thenthrottled to limit their impact on total network load whilestill allowing them to make some forward progress.
Algorithm 1 shows HAT’s per-epoch application classification procedure which implements this key idea. Everyepoch, each application’s network intensity (measured as L1
MPKI) is collected. At the end of the epoch, applications
are sorted by measured network intensity. Starting withthe least network-intensive application, the applications areplaced into the network-non-intensive unthrottled application category until the total network intensity (sum of
L1 MPKIs) for this category exceeds a pre-set threshold,which we call N onIntensiveCap (we will examine theimpact of this parameter in §VII-G; in brief, it should bescaled with network capacity and tuned for the particular
network design). All remaining applications (which are more
network-intensive than these unthrottled applications) arethen considered network-intensive and are throttled withthe same global throttling rate for the following epoch.
Note that these throttled applications can still make someforward progress, since throttling does not prevent all packet
injections, but only delays some of them. However, thenetwork-intensive applications’ impact on network congestion is reduced significantly, and the network non-intensive
applications’ performance is therefore likely to increase.
Algorithm 1 HAT: Application Classification Algorithm
at the beginning of each epoch:
empty the groups
sort N applications by MPKI measurements M P KIi
for sorted application i in N do
if total MPKI of network-non-intensive group +M P KIi
N onIntensiveCap thenPlace application i into the network-non-intensive group
else
Place application i into the network-intensive groupend ifend for≤D. Network-Load-Aware Throttling Rate Adjustment
Recall that we observed in §III that workloads achieve
their peak performance at different global throttling rates.
We have observed that despite the differences in throttlingrate, the network utilization, or measure of the fraction oflinks and buffers in the network occupied on average, is agood indicator of where peak performance will occur. HAT
thus adjusts the global throttling rate that is applied to everyclassified network-intensive application within a workload
dynamically using a feedback-based mechanism that targets
a network utilization target. The network utilization target isfixed for a given network design (a sensitivity study on its
effect is presented §VII-G). This minimizes over-throttling(which could hinder forward progress unnecessarily) or
under-throttling (which could fail to alleviate congestion) of
the workload. Controlling Throttling Rate: At each epoch,
HAT compares actual network utilization with a target uti-

lization set statically for the network design. HAT adjusts theglobal throttling rate upward when the network utilization
is higher than the target and downward when the network
utilization is lower than the target by a throttling rate step.
Per-epoch rate steps are given in Table II: the rate moves
with larger steps when it is lower, and smaller steps when
it is higher, because most applications are more sensitive tothrottling at higher throttling rates.2 The throttling rate has
a maximum value at 95%, based on empirical sensitivity
sweeps that show a good performance improvement at thatpoint without unduly impacting the throttled nodes. Notethat when the baseline network utilization is below thetarget (indicating a lightly-used network that will likely havevery little congestion), HAT will converge to a rate of 0%,
effectively deactivating the throttling mechanism.
Current Throttling Rate Throttling Rate Step
0% – 70%10%
70% – 90%2%
90% – 94%
1%
Table II. Throttling rate adjustment used in each epoch.

E. Implementation Cost
Our mechanism consists of two major parts in each epoch:(i) measurement of L1 MPKI and network utilization, and(ii) application classification and throttling rate computation.
First, L1 MPKI and network utilization are measured
in hardware with integer counter structures resetting every
epoch. Each core requires two hardware counters to measure
L1 MPKI: one L1 miss counter and one instruction counter.
Measuring network utilization requires one hardware counterin each router to monitor the number of flits that are routed
to direct neighboring nodes. Second, throttling rates must
be enforced at each node. Throttling can be implemented
as a fixed duty cycle in which injection is allowed. Finally,
the application classification computation must be done insome central node. The algorithm consists of (i) a sort by
MPKI and (ii) one pass over the application list. Althougha detailed model is outside the scope of this work, such a
computation requires at most several thousand cycles out of
a 100K-cycle epoch, running on one central CPU node. This
computation and communication overhead is small given a
100K-cycle epoch. Assuming the computation takes 1000
cycles, then the overhead is only 1% on a single core. Similar
coordination mechanisms have been proposed in previous
work (e.g., [7], [8], [30], [31]) and are shown to have low
overhead.
V. R ELATED W ORK
To our knowledge, HAT is the first work to combine
application-aware source-throttling and dynamic network
load-aware throttling adjustment to reduce network congestion. In this section, we discuss prior work that addresses
network congestion in other ways.
Congestion Control in NoCs: The congestion problem in
NoCs has been widely explored in many previous works.
Various approaches use available buffers [22], available
2 We observe that non-linear steps help reduce the settling time of thethrottling rate. In all of our experiments, we choose an epoch length thatallows the throttling rate to settle before the end of an epoch.

virtual channels [4], output queue size [33] or a combination of these [17] to detect congestion, and then perform
adaptive routing to avoid congested regions. However, thesetechniques do not actually reduce high network load because
they do not throttle sources injecting intensive traffic into the
network. As congestion increases with higher network load,
adaptive routing can no longer find productive paths to routearound congested regions, leading to system performancedegradation. Furthermore, adaptive routing is less effective
in bufferless networks because deflection routing provides
adaptive routing anyway when network congestion is high.
In contrast to adaptive-routing approaches, our sourcethrottling proposal (i) reduces network congestion by source
throttling, effectively improving performance when network
load is too high, (ii) uses application-aware throttling rather
than network-level measurement to provide application-level
performance gains, and (iii) works on both buffered andbufferless networks.
A few prior works use throttling to manage congestionin NoCs. In particular, Nychis et al. [30], [31] propose anapplication-aware throttling mechanism to mitigate the effectof congestion in bufferless NoCs. The proposed techniquedetects congestion based on each application’s progress andtriggers throttling when any application’s starvation rate
(fraction of cycles a node attempts to inject a flit but
cannot) is above a threshold. Then it throttles applications
that have network intensity above average with a fixed
throttling rate proportional to their intensities. However, theirmechanism is not adaptive to varying network conditions,hence can under- or over-throttle the applications (which
reduces performance) as we will show in §VII.
Congestion Control in Off-chip Networks: Congestion
control in off-chip interconnects (e.g., between computenodes in a supercomputer) has been studied extensively.
While these networks resemble NoCs in some regards, anduse some of the same design techniques, many of thecongestion control mechanisms for off-chip networks are notdirectly applicable to NoCs. Off-chip networks differ fromon-chip networks in terms of both architecture and scale.
For example, the popular Infiniband [15], [19] interconnect
which is used in many large-scale supercomputers uses amultistage switch topology and can scale to thousands ofnodes. In this topology, switch nodes are separate from compute nodes, and switches are large devices with many input
and output ports. Thus, there are relatively fewer switches(since each switch serves more compute nodes). Latencies
are also typically orders of magnitude longer because link
distances are greater (meters rather than millimeters). For
both these reasons, each switch in an off-chip interconnect
can employ more complex routing and congestion control
algorithms than what is possible in on-chip networks, where
each router must be relatively small and very fast. Furthermore, central coordination (as we propose) is much moredifficult in such an environment. More complex distributed
schemes are typically used instead.
Despite these differences, source throttling is a common
technique used by many works [2], [15], [32], [38] to remove

Parameter
System topology
Core model
Private L1 cache
Shared L2 cache
Cache coherence
Interconnect Links
Bufferless router
Buffered router

Setting
8x8 or 12x12 2D-mesh; core and shared cache slice at every nodeOut-of-order, 128-entry instruction window, 16 miss buffers (i.e., MSHRs), stall when buffers are full
64 KB, 4-way associative, 32-byte block size
perfect (always hits), cache-block-interleaved mapping
directory-based with cache-to-cache transfers, perfect striped directory
1-cycle latency, 32-bit width (1-flit request packets, 8-flit data packets)
2-cycle latency, FLIT-BLESS [28] or CHIPPER [13]2-cycle latency, 4 VCs [3], 4 flits/VC, buffer bypassing [42]Table III. Simulated system parameters.

congestion in an off-chip network. Thottethodi et al. [38]propose a mechanism to completely block the injection of
every source when the globally measured virtual-channel
buffer occupancy in a buffered interconnect exceeds a threshold. This threshold is dynamically tuned to maximize systemthroughput. This mechanism thus is triggered by networklevel conditions rather than application-level performancemeasurements, and treats all applications the same (hence
reducing fairness when applications have differing network
requirements). In addition, to determine when to throttle
and to adjust its threshold, this throttling mechanism measures and exchanges congestion and throughput information
globally at fine-grained intervals (every 32 and 96 cycles,
respectively), which requires a dedicated side-band network
to avoid making congestion worse. Despite its high hardware
overhead, we compare, in §VII, HAT to a variant of thismechanism adapted to the on-chip network design. We
assume that global communication incurs no overhead for[38], which benefits the performance of this mechanism.
Another common off-chip congestion control approach isbased on avoiding head-of-line (HoL) blocking, in whichpackets at the head of a queue cannot allocate downstream
buffers and make forward progress. These blocked packets inturn block other packets further upstream. Many works [1],[9], [11], [36] have proposed allocating separate queues fordifferent packet flows within a switch to avoid HoL blocking.
For example, separate queues could be allocated by output
port [1] or dynamically for congested packet flows [9].
However, such techniques rely on large hardware buffers,
and require more complex packet processing, both of which
are poorly adapted to on-chip routers that have limited power
and on-die area, and only a short time to process each packet.
VI. E XPERIMENTAL M ETHODOLOGY
Simulator Model: We use an in-house cycle-level CMP
simulator that consumes instruction traces of x86 applications for our evaluation of application-level performance. We
model 64- and 144-node CMP systems with 8x8 and 12x12
2D-mesh cycle-level networks, respectively. Each node stalls
when its request buffers, instruction window, etc are full.
We faithfully model private L1 data caches and a shared L2
cache with a directory-based cache coherence protocol [26].
All private L1 caches use Miss Status Holding Registers(MSHRs) [25] to track requests to the shared L2 cache
until they are serviced. Unless stated otherwise, our detailed
system configuration is as shown in Table III.
Note that we model a perfect shared L2 cache to stress the
network, as also evaluated for CHIPPER [13], BLESS [28]and previous work on throttling [30], [31]. In this model, all
L1 misses hit in their destination L2 cache slice. This setup
potentially increases the network load compared to a systemwith realistic off-chip memory that often becomes the systembottleneck. Note that we still use realistic configurations(e.g., access latency and MSHRs) for the cache hierarchy.
As a result, using a perfect shared L2 cache allows us to
conduct studies on the fundamental congestion problem due
to capacity of the evaluated networks.
Because only the L1 caches require warming with aperfect shared cache, 5M cycles of warmup is found to be
sufficient. We find that 25M cycles of execution gives stable
results for the applications simulated.
Evaluated Router Designs: We evaluate HAT and priorthrottling mechanisms on NoCs with the following routerdesigns: i) BLESS [28], ii) CHIPPER [13], and iii) VCbuffered router with buffer bypassing [42].
Workloads: We focus on network-intensive workloads forour main evaluations since the problem we are solving occurs only when the network is under high load. In a sensitivity study in §VII-G, we will separately show behavior in network non-intensive workloads. To form suitable workloads
for our main evaluations, we split the SPEC CPU2006 [35]benchmarks based on L1 MPKI into three categories: High,
Medium and Low. In particular, High-intensity benchmarks
have MPKI greater than 50, Medium-intensity benchmarks
fall between 5 and 50 MPKI, and Low-intensity benchmarks
form the remainder. Based on these three categories, we
randomly pick a number of applications from each category and form seven intensity mixes with each containing
15 workloads: L (All Low), ML (Medium/Low), M (AllMedium), HL (High/Low), HML (High/Medium/Low), HM(High/Medium), and H (All High).
We also evaluate multithreaded workloads in §VII-F using
the SPLASH-2 [44] applications. For these workloads, an
8x8 network runs 64 threads of a single application. Simulations are run for a fixed number of barriers (i.e., main
loop iterations) and total execution times are compared.
Performance Metrics: We measure application-level system
performance with the commonly-used Weighted Speedup
PN IP Cisharedalone
metric: W S =i=1 IP C alone [12], [34]. All IP Ci
ivalues are measured on the baseline network without throttling. We also briefly report our results in System IPC [12]and Harmonic Speedup [12], [27], but we do not present full
results for brevity.
Additionally, to show that throttling does not unfairly
penalize any single application to benefit the others, wereport unfairness using maximum application slowdown:

IP C alonei[7], [23], [24], [29],U nf airness = maxi IP C sharedi[40]. For this metric, lower is better (more fair). Because
slowdown is the inverse metric of speedup, we report theharmonic mean rather than arithmetic mean of unfairness.
HAT Parameters: Unless stated otherwise, we set the theepoch length to 100K cycles. For the NonIntensiveCap(network utilization target), we use 150 (60%), 150 (55%),
and 350 (5%) for BLESS, CHIPPER, and VC-buffered,respectively (all determined empirically).
Comparisons to Other Mechanisms: We compare ourmechanism to two state-of-the-art throttling techniques inour evaluation. The first comparison is to an implementationof prior work on throttling in bufferless networks proposedby Nychis et al. [30], [31], which we discussed in §V (called
Heterogeneous Throttling in §VII). We adapt this work to
buffered NoCs for comparisons as well. Second, we compare
to the mechanism proposed by Thottethodi et al. [38] that is
also described in §V (called Self-Tuned in §VII). Note that
Self-Tuned was proposed explicitly for buffered networks,
and its adaptation to bufferless networks is non-trivial because the algorithm measures buffer utilization; hence wecompare against Self-Tuned only in VC-buffered networks.

VII. E XPERIMENTAL E VALUATIONIn this section, we evaluate the performance of HAT on
NoCs with different router designs: BLESS [28], CHIPPER [13], and VC-buffered [3]. We demonstrate that HAT
provides better system performance than prior throttlingmechanisms on all designs. In addition, we report results
on fairness, and show that HAT does not impair fairness
when throttling to improve overall performance.
A. Results on Buffered NoCs
Figure 2 shows the application-level system performance(left panel) and fairness (right panel) of the VC-bufferednetwork in a 64-node (8x8) CMP system. In fairness plots,
a higher number indicates higher maximum slowdown andthus worse fairness. For each bar group, results are presented
in the order of baseline, Heterogeneous Throttling [30], [31],Self-Tuned [38], and HAT. Results are split by workload categories as defined in §VI, with network intensities increasingfrom left to right. Based on the results, we find that HAT
consistently provides better system performance and fairness
than the two previously proposed mechanisms. We make thefollowing observations:
1. HAT improves system performance by 3.9% and fairness by 4.9% on average across all workloads on the VCbuffered network. The main reason for this performance gain
over the baseline is that HAT performs throttling on applications according to their network intensities. By throttling
network-intensive applications that contribute the majority
of congestion at all times, HAT reduces network load andallows better network access for the other applications.
2. Compared to Heterogeneous Throttling, which is thebest previous throttling mechanism that we evaluate, HAT
attains 3.5% (4.4%) better system performance (fairness) on
average across all workloads. This is because Heterogeneous
Throttling does not adjust its throttling rate according to

network load when it throttles network-intensive applications. Instead, it triggers throttling when at least one nodeis starved and uses a fixed throttling rate that is independentof actual network load. This can result in network under- orover-utilization.
3. As the workload’s network intensity increases, HAT’s
performance improvement becomes more significant because congestion increases. This trend changes in thehighest-load category (“H”) because these workloads consist
only of network-intensive applications, which are nearly
always network-bound, frequently stalling on outstanding
requests. In such a uniformly high-network-intensity workload, applications are not sensitive to network latency, thus
throttling cannot help as much. Throttling is most effective
when some applications in the “L” and “M” categories are in
the network, because these applications are more sensitive to
network latency and can benefit from reduced interference.
To confirm our qualitative explanation, we observe that HAT
reduces packet latency by 28.3% and 18.2% on average
across applications in the “L” and “M” categories, respectively. On the other hand, HAT increases packet latency by
3.0% on average across applications in the “H” category
with little system performance degradation. This is because
these applications have better network latency tolerancecompared to latency-sensitive applications. As a result, weconclude that HAT provides the highest improvement when
the workload mix is heterogeneous.
4. In our evaluations, Self-Tuned provides little performance gain over the baseline (and HAT attains 5.0%
better system performance and 5.3% better fairness thanSelf-Tuned). This is because the throttling algorithm in
Self-Tuned performs hard throttling: when the network isthrottled, no new packet injections are allowed until thecongestion (measured as the fraction of full packet buffers)
reduces below a threshold. Thus, Self-Tuned will temporarily
reduce congestion by halting all network injections, but thecongestion will return as soon as injections are allowed
again. Self-Tuned [38] was previously evaluated only withsynthetic network traffic, rather than with a realistic, closedloop CMP model as we do. This previous evaluation also
found that network throughput improvements only occur
near network saturation. In contrast to Self-Tuned, HAT isable to obtain performance and fairness improvements overa wide range of network load by (i) adaptively adjusting
its throttling rate according to network load and (ii) selectively throttling certain applications based to their network
intensity.
We conclude that, by using application-aware throttlingwith adaptive rate adjustment, HAT reduces inter-applicationinterference to achieve better system performance and fairness than previously proposed mechanisms.
B. Results on Bufferless NoCs
In this section, we present results on 64-node (8x8) systems with two different bufferless network designs. Figure 3
shows the application-level system performance and fairness
for CHIPPER and BLESS with the same set of workloads
shown in the previous section. For each network’s bar group,3.3%

BASELINE
HETERO.

4.6%

6

SELFTUNED
HAT

2.1%

3.9%

5.4%

3.6%

11.1%

3.1%

4
4.9%

2

1.4%

HL

HML

HM

H

AVG

HL

HML

HM

H

AVG

Unfairness(Lower is Better)

Weighted Speedup(Higher is Better)

50
46
42
38
34
30
26
22
18
14
10

0

48
44
40
36
32
28
24
20
16
12
8
4
0

9.4%
5.4%

20

11.5%
5.8%
10.4%
11.3%

CHIPPERCHIPPER-HETERO.
CHIPPER-HATBLESS
BLESS-HETERO.
BLESS-HAT

5.9%
6.7%
8.6%

25.6%

5.9%

HL

HML

HM

H

20.7%14.2% 19.2%16.9%

AVG

HL

HML

12

12.3%
15.4%
20.0%
17.2%

24.4%

HM

16

H

AVG

8
4

Unfairness(Lower is Better)

Weighted Speedup(Higher is Better)

Figure 2. System performance and fairness of 64-node (8x8) CMP systems with VC-buffered NoCs. % values are gains of HAT over VC-buffered.

0

Figure 3. System performance and fairness of 64-node CMP systems with CHIPPER and BLESS. % values are gains of HAT over CHIPPER or BLESS.

we show results in the order of baseline, Heterogeneous
Throttling [30], [31], and HAT. Self-Tuned [38] is notevaluated because the mechanism detects congestion basedon the fraction of full VC-buffers, which are not available
in these bufferless router designs. Several observations are
made based on the results:
1. Similar to our results on VC-buffered networks presented above, HAT provides the best performance andfairness among all evaluated mechanisms. Compared to
Heterogeneous Throttling (the best previous mechanism) ina CHIPPER (BLESS) network, HAT improves system performance by 5.1% (7.4%) and fairness by 4.5% (23.7%) on
average. Relative to a baseline network with no congestion
control, HAT improves system performance by 5.9% (CHIPPER) and 10.4% (BLESS) on average across all workloads.
Furthermore, HAT attains 20.0% (CHIPPER) and 17.2%(BLESS) better fairness than the baseline network. The
reason for the performance improvement is that in a CHIPPER (BLESS) network, HAT reduces average packet latency
by 22.4% (37.4%) and 9.1% (18.6%) for latency-sensitive
applications in the “L” and “M” categories, respectively.
Although the average packet latency increases by 14.5%(16.1%) for “H” applications, HAT still improves both
system performance and fairness since these applications aretolerant to network latency. Note that congestion is more
severe in bufferless networks than in VC-buffered networks
because the lack of buffers reduces total network throughput.
Because of this increased congestion, HAT attains larger improvements over baseline in both performance and fairness
in these bufferless networks than in VC-buffered networks.
2. In the “H” workload category that consists of only
high-intensity applications, in the CHIPPER network, Heterogeneous Throttling provides better fairness than HAT(contrary to the trend in other workload categories). Thisbehavior is due to the way Heterogeneous Throttling picks
nodes to throttle: the throttling mechanism happens to favor
the single most-slowed down application. This decisioncauses improvement in the maximum-slowdown definitionof unfairness that we use. However, if we examine Harmonic

Speedup (another measure of fairness [12], [27]), HAT
actually provides 4.3% higher fairness than Heterogeneous
Throttling for this workload category.
We conclude that HAT is effective for a variety of NoC
designs, and in fact it provides higher performance gains
when the network capacity is limited, as is the case inbufferless deflection networks.
C. Scalability with Network Size
Figure 4 presents system performance (left panel) and
fairness (right panel) of HAT on VC-buffered, CHIPPER,
and BLESS NoCs with the network size varied from 4x4(16 nodes) to 12x12 (144 nodes). The percentage on topof each bar indicates the performance improvement of HAT
over the baseline network. HAT always improves system
performance and fairness over the baselines. The best performance gain is observed on 8x8 networks because wereasonably optimized HAT’s parameters for these network
configurations. For other networks (4x4 and 12x12), we
did not optimize the parameters. Even with non-optimized
parameters, HAT still improves both system performanceand fairness. We expect that further performance gain can beachieved when parameters are tuned for these other network
sizes.
D. Sensitivity to Performance Metrics
To ensure no bias in selecting metrics to report our system
performance, we briefly present our results in system IPC
and harmonic speedup (described in §VI). HAT improves
average system IPC by 4.8%/6.1%/11.0% and average harmonic speedup by 2.2%/7.1%/10.5% over the best previous
mechanism on VC-buffered/CHIPPER/BLESS 8x8 NoCs.
We conclude that HAT provides consistent performance
improvement over all evaluated mechanisms.
E. Energy Efficiency
In addition to system performance, we report energy efficiency as performance-per-Watt, which is weighted speedup
divided by average network power. We use a powermodel developed by Fallin et al. [14], which is basedon ORION 2.0 [41], to evaluate static and dynamic power

50

1.9%

40

8

VC-BUFFERED
HAT

3.9%

3.7%

threads (e.g., as determined in [21], [10]), can provide better
performance in multithreaded workloads compared to HAT,
and the development of such mechanisms is a promisingavenue of future work.

6

30

2

1.3%

0
50
40

3.3%

30

10.4%

CHIPPER
HAT

5.9%
20.0%

20
10

6.0%

5.8%

0
50
6.2%
40

9.7%

BLESS
HAT

6

10.4%
30

17.2%
10.8%

20
10

0
14
12
10
8
6
4
2
0
8

Unfairness(Lower is Better)

Weighted Speedup(Higher is Better)

10

4

4.9%

11.1%

20

4
2

7.2%

0

0

16

64

144

Network Size16
64 144
Network Size

Figure 4. System performance and fairness of CMP systems with varied
network sizes and VC-buffered, CHIPPER, and BLESS routers.

consumed by the on-chip network. Table IV shows theenergy efficiency improvement of HAT compared to different networks without source throttling. HAT improvesenergy efficiency by up to 14.7% because it allows packets
to traverse through the network more quickly with lesscongestion, thus reducing dynamic energy consumption.
HAT shows the greatest improvement on CHIPPER because
dynamic energy dominates the total energy due to higher
deflection rate in CHIPPER.
Router design
VC-Buffered BLESS CHIPPER∆ Perf/Watt
5.0%
8.5%
14.7%
Table IV. Energy efficiency improvement of HAT.

F. Effect on Multithreaded and Low-Intensity Workloads
Multithreaded Workloads: Table V shows execution time
reduction of HAT relative to baselines without throttlingon 64-node systems with different network designs running four 64-threaded workloads from SPLASH-2 [44].
Although HAT is not explicitly designed for multithreaded
workloads, it can still provide benefit by reducing network contention. HAT improves performance of lun,
which has roughly 20 MPKI, by 7.5%/4.2%/1.0% on VCbuffered/BLESS/CHIPPER NoCs. This is because HAT reduces the average network latency by 15.2% (VC-buffered),
9.0% (BLESS), and 12.8% (CHIPPER). On the other hand,
Heterogeneous Throttling improves performance of lun by
1.6%/0.8%/2.0% on VC-buffered/BLESS/CHIPPER NoCs.
The reason that Heterogeneous Throttling shows less performance gain than HAT is that starvation rate, the metricmonitored by Heterogeneous Throttling to make the throttling decision, remains low through the execution.
In addition, neither HAT or Heterogeneous Throttlingshows performance impact on fft, luc, and cholesky
because these workloads have very low network intensitywith 4.3 MPKI, 2.0 MPKI, and 4.9 MPKI, respectively.
We do not present results on these applications for Heterogeneous Throttling because they show similar trend (noperformance improvement) as HAT. New throttling mechanisms that selectively throttle non-bottleneck or non-limiter

Benchmark
fft
luc
lun
cholesky
VC-Buffered
0.1%
0.0%
7.5%
-0.1%
BLESS
0.1%
0.0%
4.2%
0.0%
CHIPPER
0.1%
-0.1%
1.0%
-0.1%
Table V. Execution time reduction of HAT on multithreaded workloads.

Low-Intensity Workloads: We observe that HAT does not
impact system performance of low-intensity workloads (i.e.,
L, ML, and M mixes). Since these workloads have minimal
contention, there is no need for throttling. All techniquesthat we evaluate perform essentially the same.
G. Sensitivity Analysis
For brevity, we only present sensitivity results for 4x4
BLESS network and we do not present extensive results forsome studies.
Capacity of Network-Non-Intensive Applications: The
total network intensity of applications which are not throttled (N onIntensiveCap) determines what portion of the
network-non-intensive nodes are unthrottled. It is importantthat this parameter is set appropriately for a given network
design: if it is too large, then the network will be overloaded
because too many network-intensive applications will beallowed unthrottled access to the network. On the other hand,
if it is too small, then too few applications will be allowed
unthrottled network access, and some network non-intensive
applications may be throttled despite their low impact on
network load.
Table VI shows the performance of HAT compared tothe baseline with no throttling as N onIntensiveCap varies.
When every node is throttled (NonIntensiveCap= 0), HAT
provides the best fairness because the most slowed down
application can make faster progress with reduced congestion. As N onIntensiveCap increases, system performanceincreases since the low-intensity applications are given more
opportunities to freely inject, but fairness slightly degrades
because some nodes are throttled more heavily to reduce
the network load. When the parameter is increased past acertain point, HAT will be unable to bring the network load
down to the target even if the throttling rate is near 100%
because too few nodes are throttled. Such a situation resultsin unfair treatment of the few applications that are throttled
and small performance gains overall because network load
is lowered only slightly.
0
50
100
150
200
250
∆ WS
8.5% 10.1% 10.6% 10.8% 10.3% 9.6%
∆ Unfairness -11% -9.5% -7.1% -5.0% -2.5% -2.1%
Table VI. Sensitivity of HAT improvements to N onIntensiveCap.

Epoch Length: We use an epoch length of 100K cycles
which shows a good trade-off between responsiveness andoverhead. The performance delta is less than 1% if we vary
the epoch length from 20K to 1M cycles. A shorter epoch
captures more fine-grained changes in application behavior,
but with higher overhead. A longer epoch amortizes HAT’s
computation over more time, but does not promptly capturefine-grained changes in applications’ behavior.

Target Network Utilization: This parameter controls thethrottling aggressiveness of HAT and thus its effectiveness.
We find that weighted speedup peaks at between 50% and
65% network utilization with less than 1% variation at points
within this range. In addition, we observe that fairness is bestin this range. Beyond 65%, performance drops significantly(by as much as 10%) because throttling is not aggressive
enough, and the network remains congested.
VIII. C ONCLUSION
In this paper, we present Heterogeneous Adaptive Throttling (HAT), a new application-aware throttling mechanismthat reduces congestion to improve performance in NoCbased multi-core systems. HAT achieves this improvement
by using two key principles. First, to improve system
performance, HAT observes applications’ network intensity
and selectively throttles network-intensive applications, allowing latency-sensitive applications to make fast progresswith reduced network congestion. Second, to minimize
the over- and under-throttling of applications, which limits
system performance, HAT observes the network load anddynamically adjusts the throttling rate in a closed-loop
fashion. To our knowledge, HAT is the first source-throttlingmechanism that combines application aware throttling andnetwork-load-aware throttling rate adjustment. We show that
HAT outperforms two different state-of-the-art congestioncontrol mechanisms, providing the best system performanceand fairness. We conclude that HAT is an effective highperformance congestion-control substrate for network-onchip-based multi-core systems.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their feedback.
We gratefully acknowledge members of the SAFARI groupat CMU for feedback. Kevin Chang is partially supported by
Prabhu and Poonam Goel Fellowship. Rachata Ausavarungnirun is partially supported by Royal Thai Government
Scholarship. Chris Fallin is partially supported by an NSF
Graduate Research Fellowship. We acknowledge the generous support of AMD, Intel, Oracle, and Samsung. This
research was partially supported by an NSF CAREER AwardCCF-0953246.
R EFERENCES[1] T. E. Anderson et al. High-speed switch scheduling for local-area
networks. 1993.
[2] E. Baydal et al. A congestion control mechanism for wormhole
networks. PDP-9, 2001.
[3] W. Dally. Virtual-channel flow control. IEEE TPDS, 1992.
[4] W. Dally and H. Aoki. Deadlock-free adaptive routing in multicomputer networks using virtual channels. IEEE TPDS, 1993.
[5] W. Dally and B. Towles. Principles and Practices of Interconnection
Networks. Morgan Kaufmann, 2004.
[6] W. J. Dally and B. Towles. Route packets, not wires: On-chip
interconnection networks. DAC-38, 2001.
[7] R. Das et al. Application-aware prioritization mechanisms for on-chip
networks. MICRO-42, 2009.
[8] R. Das et al. Aérgia: exploiting packet latency slack in on-chip
networks. ISCA-37, 2010.
[9] J. Duato et al. A new scalable and cost-effective congestion management strategy for lossless multistage interconnection networks.
HPCA-11, 2005.
[10] E. Ebrahimi et al. Parallel application memory scheduling. In MICRO44, 2011.

[11] Escudero-Sahuquillo et al. FBICM: efficient congestion management
for high-performance networks using distributed deterministic routing.
HiPC-15, 2008.
[12] S. Eyerman and L. Eeckhout. System-level performance metrics for
multiprogram workloads. IEEE Micro, 28:42–53, May 2008.
[13] C. Fallin et al. CHIPPER: A low-complexity bufferless deflection
router. HPCA-17, 2011.
[14] C. Fallin et al. MinBD: Minimally-buffered deflection routing forenergy-efficient interconnect. NOCS, 2012.
[15] E. Gran et al. First experiences with congestion control in InfiniBand
hardware. IPDPS, 2010.
[16] P. Gratz et al. Implementation and evaluation of on-chip networkarchitectures. ICCD, 2006.
[17] P. Gratz et al. Regional congestion awareness for load balance innetworks-on-chip. HPCA-14, 2008.
[18] Y. Hoskote et al. A 5-GHz mesh interconnect for a Teraflops
processor. IEEE Micro, 27, 2007.
[19] InfiniBand Trade Association. InfiniBand architecture specification,
release 1.2.1, 2007.
[20] Intel Corporation. Single-chip cloud computer. http://techresearch.
intel.com/ProjectDetails.aspx?Id=1.
[21] J. A. Joao et al. Bottleneck identification and scheduling in multithreaded applications. In ASPLOS-21, 2012.
[22] J. Kim et al. A low latency router supporting adaptivity for on-chip
interconnects. DAC-42, 2005.
[23] Y. Kim et al. ATLAS: a scalable and high-performance schedulingalgorithm for multiple memory controllers. HPCA-16, 2010.
[24] Y. Kim et al. Thread cluster memory scheduling: Exploiting differences in memory access behavior. In MICRO-43, 2010.
[25] D. Kroft. Lockup-free instruction fetch/prefetch cache organization.
ISCA-8, 1981.
[26] J. Laudon and D. Lenoski. The SGI Origin: a ccNUMA highly
scalable server. ISCA-24, 1997.
[27] K. Luo et al. Balancing thoughput and fairness in SMT processors.
ISPASS, 2001.
[28] T. Moscibroda and O. Mutlu. A case for bufferless routing in on-chip
networks. ISCA-36, 2009.
[29] S. Muralidhara et al. Reducing memory interference in multicore systems via application-aware memory channel partitioning. In
MICRO-44, 2011.
[30] G. Nychis et al. Next generation on-chip networks: What kind ofcongestion control do we need? Hotnets-IX, 2010.
[31] G. Nychis et al. On-chip networks from a networking perspective:
Congestion and scalability in many-core interconnects. SIGCOMM,
2012.
[32] S. Scott and G. Sohi. The use of feedback in multiprocessors and its
application to tree saturation control. IEEE TPDS, 1990.
[33] A. Singh et al. GOAL: a load-balanced adaptive routing algorithm
for torus networks. ISCA-30, 2003.
[34] A. Snavely and D. M. Tullsen. Symbiotic jobscheduling for asimultaneous multithreaded processor. ASPLOS-9, 2000.
[35] Standard Performance Evaluation Corporation. SPEC CPU2006. http://www.spec.org/cpu2006.
[36] Y. Tamir and G. Frazier. Dynamically-allocated multi-queue buffers
for VLSI communication switches. IEEE TC, 1992.
[37] M. Taylor et al. The raw microprocessor: A computational fabric forsoftware circuits and general-purpose programs. MICRO-35, 2002.
[38] M. Thottethodi et al. Self-tuned congestion control for multiprocessor
networks. HPCA-7, 2001.
[39] Tilera Corporation. Tilera announces the world’s first 100-coreprocessor with the new TILE-Gx family. http://www.tilera.com/news& events/press release 091026.php.
[40] H. Vandierendonck and A. Seznec. Fairness metrics for multi-threaded
processors. IEEE Computer Architecture Letters, 2011.
[41] H. Wang et al. Orion: a power-performance simulator for interconnection networks. MICRO-35, 2002.
[42] H. Wang et al. Power-driven design of router microarchitectures inon-chip networks. MICRO-36, 2003.
[43] D. Wentzlaff et al. On-chip interconnection architecture of the Tile
processor. MICRO-40, 2007.
[44] S. Woo et al. The SPLASH-2 programs: characterization andmethodological considerations. ISCA-22, 1995.