Managing GPU Concurrency in Heterogeneous Architectures
Onur Kayıran∗Nachiappan Chidambaram Nachiappan∗
Adwait Jog∗
Rachata Ausavarungnirun†∗‡†
Mahmut T. Kandemir
Gabriel H. Loh
Onur Mutlu
Chita R. Das∗
∗ The Pennsylvania State University† Carnegie Mellon University‡ Advanced Micro Devices, Inc.
Email: ∗ {onur, nachi, adwait, kandemir, das}@cse.psu.edu, † {rachata, onur}@cmu.edu, ‡ gabriel.loh@amd.com

and HSA (Heterogeneous System Architecture) [2], where
there is a unified architecture and programming model forCPUs and GPUs, are being explored, these concepts arestill evolving and require effective resource management
frameworks to make GPU-based computing more efficient.
A recent trend in GPU-based computing has been thedesign of heterogeneous multicore architectures consistingof CPUs and GPUs on the same die. AMD’s accelerated
processing units (APUs) [3], NVIDIA’s Echelon project [11],and Intel’s Sandybridge/Ivybridge [18] architectures indicate
the commercial interest and future directions in designing
such heterogeneous multicores. However, the design spacefor such a cohesive but heterogeneous platform is much
more complex than that of a homogeneous multicore system.
For example, the number and placement of the CPU andGPU cores on a chip, the design of the underlying on-chip
network fabric, the design of the cache/memory hierarchy,
including resource management policies, and concurrency1
management strategies for effectively utilizing the on-chip
shared resources are some of the open issues that have notbeen sufficiently explored. In this paper, we focus on understanding and alleviating the shared-resource contention in aheterogeneous architecture because the contention problem
is likely to be more aggravated in a heterogeneous system
due to resource sharing between CPUs and GPUs, whichhave strikingly different resource demands.
CPU applications tend to be latency sensitive and havelower TLP than GPU applications, while GPU applications
are more bandwidth sensitive. These disparities in TLP
and sensitivity to latency/bandwidth may lead to low andunpredictable performance when CPU and GPU applications
share the on-chip network, last-level cache (LLC), andmemory controllers (MCs). To illustrate this, we ran ninedifferent mixes of CPU and GPU applications on an integrated 14-core CPU and 28-core GPU platform (describedin Section 5). We also ran these applications in isolation
to estimate the performance impact of resource sharing
on applications. Figure 1a shows GPU performance whenrunning each GPU application (KM, MM and PVR) with andwithout one of the three different CPU applications (fromthe most memory-intensive to the least: mcf, omnetpp,
perlbench). Figure 1b shows CPU performance whenrunning each CPU application with and without one of thethree GPU applications. We observe, from these figures,that the interference between CPU and GPU applications

Abstract—Heterogeneous architectures consisting of generalpurpose CPUs and throughput-optimized GPUs are projected
to be the dominant computing platforms for many classesof applications. The design of such systems is more complex
than that of homogeneous architectures because maximizing
resource utilization while minimizing shared resource interference between CPU and GPU applications is difficult. We showthat GPU applications tend to monopolize the shared hardware
resources, such as memory and network, because of their highthread-level parallelism (TLP), and discuss the limitations ofexisting GPU-based concurrency management techniques whenemployed in heterogeneous systems. To solve this problem, we
propose an integrated concurrency management strategy thatmodulates the TLP in GPUs to control the performance of both
CPU and GPU applications. This mechanism considers both
GPU core state and system-wide memory and network congestion information to dynamically decide on the level of GPU
concurrency to maximize system performance. We propose
and evaluate two schemes: one (CM-CPU) for boosting CPU
performance in the presence of GPU interference, the other(CM-BAL) for improving both CPU and GPU performancein a balanced manner and thus overall system performance.
Our evaluations show that the first scheme improves average
CPU performance by 24%, while reducing average GPU
performance by 11%. The second scheme provides 7% average
performance improvement for both CPU and GPU applications.
We also show that our solution allows the user to control
performance trade-offs between CPUs and GPUs.

1 I NTRODUCTION
GPUs have made headway as the computational
workhorses for many throughput-oriented applications compared to general purpose CPUs [32]. Traditionally, a GPU
operates in conjunction with a CPU in a master-slave mode,
where the CPU offloads parallel parts of a computation
to the GPU. Each kernel can spawn thousands of threads
to utilize the full potential of the thread-level parallelism(TLP) available in a GPU. Current programming models like
OpenCL [25] and CUDA [37] facilitate such programmingon GPUs. While this master-slave mode of computation has
become more powerful with innovations in both GPU hardware and software, it has some limitations that may impede
pushing its performance envelope further. These limitations
include the high overheads due to separate address spacesthat require explicit data movement between the CPU andthe GPU, and the lack of tight coordination between the
CPU and the GPU for dynamically sharing the computation.
Although concepts such as dynamic parallelism [39], wherea GPU can generate new work for itself, Hyper-Q [39],where multiple CPUs can launch computations on a GPU,

1 We use “concurrency”, “TLP”, and “parallelism” interchangeably.

1

noCPU

mcf

omnetppperlbench

1

0.75
0.5
0.25
0
KM

MM

PVR

(a) Effect of CPU Applications
on GPU performance.

Normalized CPU IPC

Normalized GPU IPC

1.25

1.25

noGPU

KM

MM

PVR

metrics as well as the latency tolerance of the GPU cores tomodulate the GPU TLP.
While one may argue that the alternative approach of partitioning network and memory resources between the CPU
and the GPU might solve the contention problem, we showthat such resource isolation actually leads to severe underutilization of resources, which in turn hurts either CPU
or GPU performance, or both, significantly (Section 2.2).
For this reason, we use a system with shared resources as
our baseline in most of our evaluations, but also show (inSection 6.4) that our proposed schemes work effectively ina system with partitioned resources as well.
Our main contributions are as follows:• We show that existing GPU concurrency management
solutions [24, 43] are suboptimal for maximizing overall system performance in a heterogeneous CPU-GPU system dueto the large differences in latency/bandwidth requirementsof CPU and GPU applications.
• We introduce two GPU concurrency management schemes,
CM-CPU and CM-BAL, which can be implemented withany warp scheduling technique by monitoring memory system congestion of the heterogeneous architecture and latency
tolerance of GPU cores. The key idea is to dynamicallyadjust the number of active warps on the GPU to either
unilaterally boost the CPU performance or enhance the entire
system performance based on the importance of CPU or
GPU performance as determined by the user.
• We show that both of our schemes reduce the monopolization of the shared resources by GPU traffic. CM-BAL also
allows the user to control the performance trade-off between
CPU and GPU applications.
• We extensively evaluate our proposals with 36 workloads
on an integrated 28-core GPU and 14-core CPU simulation
platform and show that CM-CPU improves CPU performance by 24%, but reduces GPU performance by 11%. CMBAL provides 7% performance improvement for both CPU
and GPU applications, without hurting GPU applicationsin any workload. We analyze the performance benefits
and provide sensitivity studies, showing that improvements
provided by the proposed techniques are robust.
To our knowledge, this is the first work that introduces
new GPU concurrency management mechanisms to improve
both CPU and GPU performance in heterogeneous systems.

1

0.75
0.5
0.25
0
mcf

omnetppperlbench(b) Effect of GPU Applications
on CPU performance.

Figure 1: Effects of heterogeneous execution on performance.

causes performance losses for both classes. CPU applications, however, are affected much more compared to GPU
applications. For example, when mcf (a CPU application)
is run with PVR (a GPU application), mcf’s performancedrops by 84%, whereas PVR’s performance drops by only
20%. The performance losses observed on both classes ofapplications are primarily due to contention in shared hardware resources. In fact, the high TLP of GPU applications
causes GPU packets to become the dominant consumers ofshared resources, which in turn degrades CPU performance.
Hence, concurrency management for GPUs is essential forcontrolling CPU, GPU, and overall system performance ina heterogeneous CPU-GPU system.
Previous works have explored concurrency control mechanisms in the context of GPUs. These works either target
performance improvements only for cache-sensitive applications (CCWS [43]) or propose solutions based on thelatency tolerance of GPU cores (DYNCTA [24]). Thesemechanisms are oblivious to the CPU cores and do nottake into account system-wide information (such as memoryand network congestion). Therefore, they lead to suboptimal
CPU performance when applied to heterogeneous CPU-GPU
systems due to the substantial difference between the latency
tolerance of GPU and CPU cores.
In this paper, we introduce a new GPU TLP management
mechanism that improves performance of both CPU andGPU applications in a heterogeneous system. The key idea
is to regulate the number of active GPU warps (i.e., GPU
TLP) to control CPU as well as GPU performance basedon information obtained by monitoring system-wide congestion and latency tolerance of GPU cores. We propose two
schemes targeted for different scenarios. Our first scheme,
CPU-Centric Concurrency Management (CM-CPU), aims
to improve the performance of CPU applications because
they are observed to be the worst sufferers in an integrated
platform. CM-CPU limits the number of active warps inGPUs by monitoring two global metrics that reflect memoryand network congestion. While this scheme is very effective
in boosting CPU performance, it causes most of the GPU applications to experience performance degradation. To recover
the performance loss of such applications, we propose oursecond scheme, Balanced Concurrency Management (CMBAL), which improves overall system performance based on
the user’s2 preference for higher CPU or GPU performance.
To achieve this, CM-BAL observes system-wide congestion

2 A NALYSIS AND BACKGROUND
Before discussing the necessity of TLP management ina heterogeneous platform and proposing our solution, we
describe our baseline architecture consisting of cores, a
network-on-chip (NoC), and memory controllers (MCs).
2.1 Baseline Configuration
Our baseline architecture configuration places throughputoptimized GPU cores and latency-optimized CPU cores on
the same chip, and connects these cores to the shared LLC
and MCs via an interconnect, as shown in Figure 2a. In orderto provide a scalable design for heterogeneous architectures,
we use a “tile-based” architecture, as shown in Figure 2b.

2 “User” can be an actual user, an application, or the system software.

2

LLC/
GPU
GPU LLC/
CPU
CPU
MC
GPU
GPU MCGPU
LLC/ GPU
LLC/
CPU
CPU
GPU
MC GPUMCCPU CPU CPU CPU
L1

L1

L1

L1

L2

L2

L2

L2

GPU GPU GPU GPU
L1

L1

L1

L1

mesh based design in Section 2.1 and examine four possible
scenarios: 1) shared NoC and shared memory channels forboth CPUs and GPUs; 2) partitioned NoC (one for CPUs
and one for GPUs) and shared memory channels; 3) shared
NoC and dedicated memory channels; and 4) partitioned
NoC and dedicated memory channels. We conduct an evaluation for these scenarios, keeping the network bisection
bandwidth and the amount of resources constant.
Shared Network and Shared MCs. This all-shared configuration, although resulting in interference of CPU andGPU requests, maximizes resource utilization and provides
the best baseline in terms of performance. Thus, we use
it as our baseline and evaluate other configurations withrespect to this. We run representative GPU applications withomnetpp, a high-MPKI CPU application that is sensitive
to GPU traffic.4
Partitioned Network and Shared MCs. This design has
similarities to some AMD APU implementations [3]. Two
separate networks for the CPU and GPU traffic reduce CPUGPU interference and probably helps latency-sensitive CPU
applications, but would lead to resource under-utilization.
The shared network uses 4 VCs for each router output port
and has 32B links. The partitioned network uses the sameamount of VC and link resources in total, but divides theseresources among the CPU and GPU traffic. For example,
3G1C configuration allocates 3 VCs and 24B links to the
GPU network, and 1 VC and 8B links to the CPU network.
For all configurations, we employ separate networks forrequest and reply traffic to prevent protocol deadlocks [12].
Figure 3a shows that the shared network is more suitable
when GPU performance is preferred over CPU performance,
because GPU traffic requires more network resources. Also,in the partitioned network, allocating more resources to the
GPU traffic results in better GPU performance. On the other
hand, Figure 3b demonstrates that the partitioned network ismore suitable when CPU performance is preferred over GPU
performance, as CPU applications are usually latency sensitive, and a dedicated network eliminates the interference
caused by GPU packets. A counter-intuitive observation hereis that increasing CPU network resources sometimes hurts
CPU performance, because most CPU packets stall at theMCs, blocked by GPU packets that are waiting in MCs to be
injected into the reply network (also observed by others [6]).
In this scenario, increasing CPU network resources causes
the system bottleneck to shift from memory to the GPU
reply network.
Shared Network and Dedicated MCs. In this design, both
CPUs and GPUs share the NoC, but each MC is dedicatedto serve either CPU or GPU traffic. Figure 4 shows theeffects of partitioning MCs. 1G7C denotes that 1 and 7
MCs are allocated for GPU and CPU traffic, respectively.
We observe that GPU performance tends to degrade withfewer MCs, whereas CPUs always prefer having 4 dedicatedMCs. Allocating more than 4 MCs to CPUs causes GPU

GPU
GPU
GPU
CPU
CPU
GPU
GPU
GPU
GPU
GPU
GPU
GPU CPU GPU CPU GPU CPU

CPU

On Chip Network
LLC

LLC

LLC

LLC

MC

MC

MC

MC

(a) Baseline system configuration

LLC/
GPU
GPU LLC/
CPU
CPU
MC
GPU
GPU MCGPU
LLC/ GPULLC/
CPU GPU CPU MC
MC GPU(b) 2D mesh layout

Figure 2: Baseline architecture.

Our baseline configuration consists of 7 processing tiles,
where each tile has 4 GPU and 2 CPU cores. We choose a
GPU to CPU core ratio of 2:1 because a single GPU core(i.e., streaming multiprocessor, SM) in Fermi GF110 (45nm
technology) occupies roughly half the area of one Nehalem
CPU core (45nm technology). We form a tile consisting of
4 GPU and 2 CPU cores, and replicate these tiles to provide
scalability. Section 6.4 analyzes sensitivity to the number ofGPU cores per tile. A total of 14 CPU and 28 GPU cores areconnected to 8 LLC slices through a 6x6 2D mesh NoC.3
The LLC, shared by both CPUs and GPUs, is distributedand each slice is directly attached to an MC. Cache blocks
are mapped to LLC tiles and MCs in chunks of 256B.
We use two warp schedulers [38] per GPU core, with theGreedy-then-oldest (GTO) [43] policy. We use GTO because
we found that it performs better than the round-robin andtwo-level schedulers [36] (also confirmed by Rogers etal. [43]). GTO already has a notion of TLP management
and slightly reduces L1 contention among warps. We use
the Static Wavefront Limiting (SWL) warp scheduler [43]to control the number of warps that can be scheduled tothe pipeline. This scheduler also uses the GTO policy tochoose between the available warps that can be issued tothe pipeline. Due to the available hardware resources, themaximum number of warps in a core is 48.
2.2 Network and Memory Controller Configuration
Design of a scalable network and memory system forheterogeneous architectures is more challenging than for
CMPs or GPUs due to the complex interactions and resource
sharing between the two classes of applications. Previous
work on NoCs for heterogeneous architectures either investigated the effects of a ring-based design on the system witha low number of cores [31] or partitioning of the virtualchannels (VCs) [30]. Similarly, previous work on CPU
memory channel/bank partitioning shows that partitioning
the available memory channels/banks across applications can
reduce interference between them [13, 20, 35, 49], while
Ausavarungnirun et al. assume a shared memory controller
between the CPU and GPU to maximize bandwidth utilization [4]. However, coordinated design of the NoC and thememory controllers for heterogeneous architectures is not
sufficiently explored.
To choose a good baseline for studying the impact ofconcurrency management in detail, we start with the 2D

4 Using a low-MPKI application (e.g., perlbench) leads to a lower
impact on CPU performance, but our conclusions do not change.

3 Each node in the 6x6 mesh contains a router.

3

1G3C

2G2C

3G1C

1
0.75
0.5
0.25
0
KM

IIX

Normalized CPU IPC

Normalized GPU IPC

1.25

1.25

2G2C

3G1C

and shared MCs as it performs the best. Section 6.4 revisits
the use of a partitioned network.

0.75
0.5

3 M OTIVATION
We make a case for a better TLP control technique by
discussing the limitations of existing techniques and theimpact of TLP in a CPU-GPU environment.

0.25
0
KM

MST

(a) Effect of network design on
GPU performance.

1G3C

1

IIX

MST

(b) Effect of network design on
CPU performance.

Figure 3: Effect of network design on GPU and CPU performance.
Results are normalized to shared network/shared MCs configuration. All CPU cores run omnetpp.

3.1 Limitations of Existing Techniques
We first discuss the limitations of two existing concurrency management techniques proposed for GPUs whenapplied to heterogeneous CPU-GPU platforms.
Cache-Conscious Wavefront Scheduling (CCWS). Rogers
et al. [43] proposed a throttling technique for GPUs thatprevents warps from issuing loads if high TLP causes
thrashing in the L1 data caches. This technique detects
L1D cache thrashing by employing a victim cache. Eachhit in the victim cache increases a score signifying thelevel of thrashing. A hit in a victim cache only happens
when an evicted cache line is accessed again. Thus, thismechanism works mainly for cache-sensitive applications
that benefit significantly from higher cache capacity, and isagnostic of the memory system beyond the L1D cache. Thismeans that CCWS would likely not provide improvements
for cache-insensitive applications that could still benefit fromthrottling. In fact, Kayiran et al. [24] demonstrate that there
are applications that do not benefit from an L1D cache,but benefit from lower concurrency. Importantly, CCWS
is not aware of the interactions of CPU and GPU in thememory system, and does not target managing concurrency
to improve CPU and overall system performance.
DYNCTA. Kayiran et al. [24] proposed a CTA schedulerthat limits the number of CTAs executing on a GPU core.
This mechanism modulates GPU TLP based on only thelatency tolerance of GPU cores. Because the latency tolerance of GPU and CPU cores are different, the decision basedon GPU latency tolerance might not be optimal for CPUs.
Figure 6 shows an example demonstrating this problem.
In this example, GPU performance is mostly insensitive
to the number of concurrently executing warps, except forthe 1 warp-per-core case. Because the latency tolerance ofGPUs does not change drastically with limited concurrency(except when TLP is greatly reduced to 1 warp), and because
DYNCTA takes into account only the latency tolerance of
the GPUs to modulate GPU concurrency, DYNCTA rarely
reduces the concurrency level below 8 warps. Changing the
concurrency level between 4 and 48 warps greatly affects
memory congestion caused by the GPUs, but has little
impact on GPU performance due to the latency tolerance
provided by ample TLP. However, this memory congestion
causes significant performance losses for CPU applications,which is not taken into account by DYNCTA. Thus, since
DYNCTA modulates GPU TLP based solely on GPU latency
tolerance without taking into account any effect GPU TLP
has on the CPUs, CPU applications perform poorly with
DYNCTA. In this example, executing only 4 warps per GPU
core would improve CPU performance by more than 20%

1.25

1G7C

2G6C

4G4C

6G2C

7G1C

1

0.75
0.5
0.25
0
KM

IIX

Normalized CPU IPC

Normalized GPU IPC

packets to congest their own dedicated MCs, eventuallycongesting the shared request network, and thus leading tolower CPU performance. Sharing MCs does not greatly hurt
CPU performance compared to equal partitioning of MCs(except when the GPU runs MST) and always provides better
performance for the GPU. As expected, overall memory
bandwidth utilization drops with dedicated MCs.
2

2G6C

4G4C

6G2C

7G1C

1.5
1
0.5
0

MST

(a) Effect of MC design on
GPU performance.

1G7C

KM

IIX

MST

(b) Effect of MC design on CPU
performance.

Figure 4: Effect of partitioned MC design on GPU and CPU
performance. Results are normalized to shared network/shared MCs
configuration. All CPU cores run omnetpp.

1.25

3G1C-7G1C

3G1C-4G4C

1

0.75
0.5
0.25
0
KM

IIX

MST

(a) Effect of resource partitioning on GPU performance.

Normalized CPU IPC

Normalized GPU IPC

Partitioned Network and Dedicated MCs. In this analysis, based on the above discussion, we use the 3G1C
configuration for the network, as it is the best-performing
network partitioning; and we evaluate 4G4C and 7G1C
configurations for MCs as they are the best for CPUs andGPUs, respectively. Figure 5 shows that partitioning boththe network and the MCs is not preferable compared tosharing both the network and the MCs (except for MST whenmemory partitioning is done equally) as such partitioning
greatly degrades both CPU and GPU performance.
1.5

3G1C-7G1C

3G1C-4G4C

1
0.5
0
KM

IIX

MST

(b) Effect of resource partitioning on CPU performance.

Figure 5: Effect of resource partitioning on GPU and CPU performance. Results are normalized to shared network/shared MCs
configuration. All CPU cores run omnetpp.

Based on the above analyses, we conclude that partitioning shared resources lead to underutilization of theseresources, and thus we use a baseline with a shared network41) stallM C = The number of stalled requests (per cycle)
due to an MC being full.
2) stallnet = The number of stalled requests (per cycle)
due to reply network being full.
Both metrics can take values between 0 and the number ofMCs. Figure 8 shows the value of these congestion metrics
when concurrency is varied between 4 and 48 warps. We run
representative GPU applications alongside omnetpp on the
CPU side,6 and normalize the results to the 48-warp concurrency level. In MUM, lower concurrency reduces memory
congestion, which improves GPU performance by almost
20%. In SSSP, lower concurrency reduces both memory andinterconnect congestion, improving performance. However,reducing the number of warps from 6 to 4 leads to insufficient latency tolerance and thus a drop in performance.
In PVR, very low concurrency reduces congestion as wellas latency tolerance of the application, causing a drop in
performance. In MST, memory pressure does not reducewith reduced concurrency, thus not affecting its performance.
Performance of MM and BP drops significantly at low concurrency levels because these two applications require high
TLP to effectively hide the memory access latencies. Notethat these metrics also reflect the effects of heterogeneous
execution on the shared LLC.
Figure 8 shows that the main performance bottleneck inthe system is likely memory bandwidth as many workloads
have high values for stallM C . The reply network is thebottleneck for SSSP. Applications that do not suffer greatly
from MC stalls (BP and MM) strongly prefer higher concurrency levels for effective latency tolerance (see Figure 7).

GPU IPC2

CPU IPC

1.5
1
0.5
0

Figure 6: GPU and CPU IPC with different GPU concurrency
levels, normalized to DYNCTA. GPU cores run BFS, CPU cores
run a High-MPKI application mix (H2 in Table IV).

3.2 CPU-GPU Interaction in Heterogeneous Execution

Normalized
GPU IPCWe analyze the impact of the TLP of GPU applications
on system performance in a CPU-GPU system, to motivate
the necessity of TLP control in GPUs.
3.2.1 Effects of GPU Concurrency on GPU Performance: The effects of GPU TLP on GPU performance havebeen studied by prior research [24, 43]. Figure 7 shows theeffect of six different concurrency levels (between 4 to 48
warps) on GPU performance for representative applications.5
GPU IPC is normalized to the baseline scheduler, where
there can be at most 48 warps running concurrently. The
results demonstrate that while some applications, such as
BP and MM, benefit from high concurrency (due to improved
parallelism and latency tolerance), other applications such
as MUM, suffer from it. There are two primary reasons
for reduced GPU performance at high concurrency levels.
First, cache-sensitive applications might thrash the caches by
issuing many memory requests [43] at higher concurrency
levels, resulting in reduced cache effectiveness, and in turnlower performance. Second, the high number of requests
generated by more threads causes congestion in the memory
subsystem [24], degrading overall GPU performance.
1.25
1
0.75
0.5
0.25
0

4 warpsMUM

6 warpsBP

MM

8 warps

PVR

16 warps

MST

24 warps

SSSP

stallMC

84 warps
16 warps

6 warps
24 warps

8 warps
48 warps

6

8

stallnet

Normalized IPC

without affecting GPU performance significantly, comparedto DYNCTA. This demonstrates that, although DYNCTA
is successful in managing GPU TLP to optimize GPU
performance, it fails to do so for CPU or overall system
performance, motivating the necessity for an integrated concurrency management solution for heterogeneous systems.

42
0

4 warps
16 warps

6 warps
24 warps

MUM

MM

8 warps
48 warps

6
4
2
0

MUM

BP

MM

PVR

MST SSSP(a) Effect of TLP on stallM C .

BP

PVR

MST SSSP(b) Effect of TLP on stallnet .

Figure 8: Effect of GPU TLP on memory subsystem for representative GPU applications. All CPU cores run omnetpp.

3.2.2 Effects of GPU Concurrency on CPU Performance: As high GPU TLP leads to congestion in thememory subsystem, it causes longer latencies for memory
requests. However, because CPUs cannot provide the level of
thread- and memory-level parallelism that GPUs can, to tolerate long latencies, the increase in memory latencies likely
hurt CPU performance. We observe that GPU applications
pressurize the memory more than CPU applications. Thus,
CPUs are highly affected by the GPU applications, whose
memory pressure usually depends on their TLP.
Figure 9 shows the effect of GPU TLP on CPU performance when the CPU cores are running omnetpp (a highMPKI application) alongside representative GPU applications. CPUs benefit from reduced congestion (see Figure 8)
when run with GPU applications MUM, PVR, and SSSP.

48 warps

HMEAN

Figure 7: Effect of GPU TLP on GPU performance for representative GPU applications. All CPU cores run omnetpp.

Memory subsystem congestion is observed at two points:
1) the MC cannot accept new packets coming from therequest network, as the memory queue is full, 2) the MCcannot inject packets into the reply network because thereply network is congested. To analyze the sources ofcongestion, we examine two metrics:
5 This figure shows the effect of GPU TLP on GPU performance. The
chosen CPU workload has little impact on our observations.

6 We observe the same trends when all CPU cores run perlbench.

5performance of both classes of applications, and also toachieve balanced improvements for both CPUs and GPUs.
Before describing each scheme in detail, we first discuss
why we employ concurrency management, why we do it
specifically in GPU cores, and how we manage concurrency.
Employing Concurrency Management. Memory bandwidth is the critical bottleneck in most workloads, as shownin Section 3. To mitigate the ill effects of bandwidth saturation, solutions at different layers are possible. Lee andKim [29] propose a new shared cache partitioning mechanism, and Ausavarungnirun et al. [4] propose new memory scheduling techniques. Both approaches aim to solve
this problem at the sink, where the problem is observed.
However, the source of the problem is the high number ofrequests generated by the GPU, leading to severe contention
at the shared resources. Thus, we propose concurrency
management strategies that attack the problem at the source,
instead of the sink. Such an approach, by controlling the rate
at which memory requests are issued, can reduce contention
at the shared cache, network and memory.
Employing Only GPU-based Concurrency Management.
We employ TLP management mechanisms only on GPUs
(as opposed to also throttling the CPUs as done in [15, 47,50]) due to two reasons. First, throttling CPUs does not havea significant effect on GPU performance but throttling GPUs
can have a great effect on CPU performance, since GPUs
tend to have many more memory requests than the CPUs [4].
Our experimental results show that reducing the number ofCPU cores in our baseline from 14 to 8 has only 2-3%
performance impact on GPU applications when both CPU
and GPU cores are running memory-intensive applications.
We showed in Section 1 that the effect of the GPUs on CPU
performance is much higher than the effect of the CPUs on
GPU performance. Second, unlike in GPUs, throttling CPUs
to reduce memory contention [15, 47, 50] is likely to resultin substantial performance loss for the throttled CPUs as a
CPU is inherently much less latency tolerant and throttling
reduces its latency tolerance even more [15]. Therefore, ourstrategy is to tune the impact of GPU execution on CPU
performance by controlling the TLP of GPUs.
Reducing GPU TLP. GPU TLP can be reduced by modulating the number of executing warps [43] (Section 2.1).
Reducing the number of concurrently executing warps would
result in fewer memory requests sent by the GPU cores tothe memory system. As long as there is enough parallelism
in the cores, GPU performance should not drop.

Normalized
CPU IPC

MM does not put any pressure on the memory subsystem,
thus does not affect CPU execution. Although reducing TLP
does not affect the congestion metrics for BP and MST(see Figure 8), the significant reduction in LLC miss rates
with reduced TLP leads to CPU performance improvement.
Therefore, the scope for CPU performance improvement
by reducing GPU TLP depends on the GPU application’smemory intensity.
1.75
1.5
1.25
1
0.75

4 warpsMUM

6 warps

8 warps

16 warps

24 warps

48 warps

BP

MM

PVR

MST

SSSP

Figure 9: Effect of GPU TLP on CPU performance when all CPUs
run omnetpp with each GPU application.

Normalized
CPU IPC

Figure 10 shows the results for the same experiment,
except now the CPU cores run a very-low-MPKI application(perlbench). perlbench generates very few memory
requests due to its very low MPKI, and, therefore, its
performance is not bottlenecked by the memory service it
receives.7 Thus, even when it is run with GPU applications
that saturate memory bandwidth, its performance does notdegrade greatly. We conclude that very low-MPKI CPU
applications have a limited scope for performance improvement by lowering GPU concurrency.
1.75
1.5
1.25
1
0.75

4 warpsMUM

6 warps

8 warps

16 warps

24 warps

48 warps

BP

MM

PVR

MST

SSSP

Figure 10: Effect of GPU TLP on CPU performance when all CPUs
run perlbench with each GPU application.

Summary. Reducing GPU TLP might (positively or negatively) or might not affect GPU performance, but does not
hurt CPU performance and can potentially greatly improveit. Thus, a prudent TLP management scheme can improve
performance in a heterogeneous CPU-GPU system.
4 M ANAGING GPU C ONCURRENCY
We describe our new approach for managing concurrency
in heterogeneous systems.
Overview. Our proposal aims to 1) reduce the negative
impact of GPU applications on CPUs, and 2) control theperformance trade-off between CPU and GPU applications
based on the user’s preference. To achieve these goals, we
propose two schemes. The first scheme, CM-CPU, achieves
the first goal by reducing GPU concurrency to unilaterally
boost CPU performance (Section 4.1). The second scheme,
CM-BAL, achieves the second goal by giving the user
multiple options to control the level of GPU concurrency(Section 4.2). This flexibility allows the user to control the

4.1 CPU-Centric Concurrency Management (CM-CPU)
The main goal of CM-CPU is to reduce GPU concurrency
to boost CPU performance. Recall from Section 3.2.2 that,
if lowering GPU TLP reduces stallM C or stallnet , CPUapplications have good scope for performance improvement.
The key idea of CM-CPU is to 1) dynamically monitor
the two congestion metrics, stallM C and stallnet , and 2)
modulate GPU TLP based on these metrics. If at least oneof these metrics is high, the memory subsystem is assumed

7 As previous works have shown [13, 14, 26–28, 34, 35], last-level cache
MPKI (or miss rate) is highly correlated with an application’s sensitivity
to memory latency or bandwidth.

6

modifying tH and tL to provide improvements for CPUs as
well as latency tolerance for GPUs is not trivial.

to be congested either at the MCs or the reply network. Then,
we reduce the level of concurrency by reducing the numberof active warps running on GPUs (called, the active warp
limit). If both metrics are low, we increase TLP, which might
improve GPU performance. Otherwise, we keep the active
warp limit unchanged. We use two thresholds, tH and tL
to determine if these metrics are low, medium, or high. We
use a medium range to avoid ping-ponging between different
concurrency levels. If the active warp limit is between 8 and48, increasing or decreasing concurrency is done in stepsof 2 warps. If the active warp limit is between 1 and 8,
the modulation step is 1, because the effect of changing
concurrency is more significant in that range.
Figure 11 shows the architectural diagram of our proposal.
To calculate stallM C ( 1 ) and stallnet ( 2 ), we collectthe congestion metrics ( 3 , 4 ) at each memory partition
separately. A centralized unit, called the CM-CPU logic ( 5 ),periodically (with an interval of 1024 cycles)8 aggregatesthese congestion metrics to calculate stallM C and stallnet ,which it compares against tH and tL . Based on the result,it sends the modulation decision to the GPU cores. The
CM-CPU logic is placed next to the global CTA scheduler.
The scheduler has a dedicated connection to all the GPU
cores [38, 39], and communicates with them frequently. The
CM-CPU logic also needs to communicate with the memory
partitions, where 3 and 4 are calculated, requiring dedicatedconnections with the memory partitions.
CTA
scheduler

Fetch/Decode
Fetch/Decode1

Warp Issue Arbiter
12
Ready
From CM-CPU
warps [1:N]
logic

I-Buffer/Scoreboard
I-Buffer/Scoreboard
I-Buffer/Scoreboard

stallMC CM-CPU
logic
stallnet

5

Fetch/Decode

Registers/Execute
Registers/Execute

Registers/Execute2

MemoryMemoryUnit
Unit

Memory Unit

Interconnect3
congestion
congestion
MC
congestionMC
MC
MC
MC
MC

GTO
priority
logic

True

Prioritized
warps
[1:N]

1 warp - stallGPU

48 warps - stallGPU

10

4congestion
netcongestion
netcongestionnet

9stallGPU table

2 warps - stallGPU

current
warp_limit

LLC
LLC
LLC

k
13

6
N==0

Warp
WarpIssue
IssueArbiter
Arbiter

Warp Issue Arbiter

CM-BAL tries to recover the performance of GPU applications that suffer due to insufficient latency tolerance
imposed by CM-CPU, up to the extent specified by the user.
The key idea of CM-BAL is to: 1) detect whether or not
GPU cores have enough latency tolerance, and 2) maintainor increase the number of active warps if GPU cores
are not able to hide memory latencies due to insufficientconcurrency. To quantify the latency tolerance of GPU cores,
for each core and each monitoring interval, we calculate thenumber of cycles during which the core is not able to issue a
warp: stallGP U . We use two warp schedulers per core in ourbaseline, so stallGP U for a core can be between 0 and twice
the number of cycles in an interval. stallGP U is incremented
if there are no ready warps that passed the scoreboard check( 6 ) and the pipeline register for the next stage is full9( 7 ). CM-BAL logic ( 8 ) calculates the moving average ofstallGP U for each concurrency level in each interval. Thesemoving averages are stored in a table ( 9 ). Depending on
the difference between moving averages of stallGP U of aparticular concurrency level and its neighboring levels, CMBAL logic either increases or maintains the concurrency
level ( 10 ). Based on this level, the SWL scheduler ( 11 ) limits
the number of warps that can be issued to the pipeline.
We observe that the performance difference between
higher TLP levels (e.g., between 48 warps and 24 warps,
which is negligible in most workloads) is much smaller thanthe performance difference between lower TLP levels. To
take this into account, our algorithm changes the TLP level
not with fixed steps, but using predetermined levels ( 9 ): 1,
2, 3, 4, 6, 8, 16, 24, and 48 warps. We maintain a moving
average of stallGP U for each of these levels. At the endof an interval, we update the one that has been used duringthat interval. Because during different phases of execution
stallGP U for a particular level might have significantly
different values, we opted to use moving averages in order
not to use information coming from only a single interval
and move to a suboptimal level. All moving average entries
are invalidated at the beginning and end of a kernel. A new
entry is created for a level with the value of stallGP U of theinterval in which the concurrency level was used for the firsttime. In subsequent intervals during which the same levelis used, CM-BAL logic updates the moving average using
mov avgnew = mov avgold × 0.25 + stallGP U × 0.75.
Figure 12 shows how CM-CPU and CM-BAL work. The
x-axis shows the level of GPU concurrency (TLP), and they-axis denotes stallGP U . This figure illustrates a typicalscenario10 where stallGP U increases with low concurrency,…

GPU cores

CPU cores

4.2 Balanced Concurrency Management (CM-BAL)

CM-BAL
logic

8

Warp-limitingscheduler
11(SWL)

Warps to bescheduled[1:warp_limit]

7
Pipeline register fullFigure 11: Proposed hardware organization. Additional hardware
shown as white components and dashed arrows.

The downside of CM-CPU is that throttling the GPUs
solely based on memory/network congestion metrics might
hurt GPU applications that require high TLP for sufficient latency tolerance. For such applications, CM-CPU canfail to provide enough latency tolerance due to aggressive
throttling. Because the memory congestion metrics are notcorrelated well with the latency tolerance of the GPU cores,

9 This might happen due to instruction latencies and write-back contention
for ALU instructions, and shared memory bank-conflicts and coalescingstalls for memory instructions.
10 This curve does not necessarily have to be parabolic. Its exact shapedepends on the application’s reaction to the variation in TLP.

8 We chose an interval of 1024 cycles empirically (but we did not optimize
the value). Such a small interval enables quick adaptation to fast fluctuations
in memory and network congestion.

7

CM-CPU’s working domain

stallGPUPart 2 of CM-BAL

2

R1

1

CM-CPUPart 1 of CM-BAL

might become unrepresentative of the current phase of theapplication if not updated.
The higher the k value is, the more difficult it is to improve the latency tolerance of GPU cores. Thus, we use thisparameter as a user-controlled knob ( 13 ) to specify the GPU
priority (or, the importance of the GPU relative to the CPU).
We provide the user with four different levels of control
for this knob: CM-BAL1 , CM-BAL2 , CM-BAL3 , and CMBAL4 . CM-BAL1 and CM-BAL4 provide the highest and thelowest GPU performance, respectively. The corresponding kfor each of these four levels is linearly proportional to thenumber of core cycles in an interval. k can take any value
between 0 and stallGP U . When k is set to stallGP U , CMBAL converges to CM-CPU.

CM-BAL

R2

Lowest TLP that provides enoughlatency tolerance

GPU TLP

Figure 12: Operation of our schemes. CM-CPU increases/decreases
TLP for all TLP ranges. CM-BAL might use Part 2 to improvelatency tolerance by increasing TLP if TLP is low.

mainly due to low latency tolerance, and increases with highconcurrency, mainly due to high memory contention. R1 is
the region where a GPU application does not have enoughlatency tolerance. R2 is the region where the application
has enough latency tolerance, but it might be sufferingfrom memory contention. CM-CPU works on the entire
TLP spectrum and increases/decreases concurrency based on
two congestion metrics, but it does not consider stallGP U .
CM-BAL, on the other hand, operates in two parts. Part1 ( 12 in Figure 11) operates the same way as CM-CPU(with the exception of using concurrency levels, instead of
fixed steps), and works in R2, as shown in Figure 12. If
the application is in R1, Part-2 of CM-BAL overrides any
decision made by CM-CPU, and might increase TLP. There
are multiple reasons as to why an application moves into R1.
First, due to a change in the application’s behavior (the curve
shifts on the x-axis), the current TLP point might move into
R1. Second, based on memory contention, CM-CPU mightpush the current TLP point of the application into R1.
We explain how Part-2 of CM-BAL works in R1. This partmakes three decisions. First, as shown in À in Figure 12, if
decreasing the number of active warps, i.e., moving to theleft on the x-axis, would potentially increase stallGP U by
more than a parameter k ( 13 in Figure 11), i.e., reducing
concurrency might increase stallGP U by more than k, CMBAL does not allow concurrency to be reduced. Second,as shown in Á in Figure 12, if increasing the number ofactive warps, i.e., moving to the right on the x-axis, would
potentially reduce stallGP U by more than k, i.e., increasing
concurrency might decrease stallGP U by more than k, CMBAL increases concurrency to the next level. In order topredict if changing concurrency would increase/decrease
the value of stallGP U by more than k in either case,
CM-BAL logic compares the moving average of stallGP U
of the current level with that of the lower/upper level.11
Third, if the algorithm stays in the same concurrency levelfor 4 consecutive intervals, we increase or decrease TLP
if the current concurrency level is less than 6 or greaterthan or equal to 6, respectively (all thresholds empiricallydetermined). We do this to solely update the moving average of stallGP U for neighboring levels as those values

5 E XPERIMENTAL M ETHODOLOGY
Simulated System. Table I provides the configuration details
of GPU and CPU cores. A GPU core contains 32-wide
SIMD lanes and is equipped with an instruction cache,private L1 data cache, constant and texture caches, andshared memory. Each CPU core is a trace-driven, cycleaccurate, 3-issue x86 core; and has a private write-back L1
instruction/data and L2 caches. We use a detailed GDDR5DRAM timing model [38].
Table I: Baseline configuration.
GPU Core Config.
GPU Resources / CoreGPU Caches / Core

CPU Core Config.

CPU L1 Cache / Core
CPU L2 Cache / Core
Shared LLC Cache
Default Warp Scheduler
FeaturesInterconnect

Memory ModelGDDR5 Timing

28 Shader Cores, 1400MHz, SIMT Width = 16 × 2
Max.1536 Threads (48 warps, 32 threads/warp),
48KB Shared Memory, 32684 Registers
16KB 4-way L1 Data Cache, 12KB 24-way Texture,
8KB 2-way Constant Cache, 2KB 4-way I-cache,
128B Line Size
14 x86 Cores, 2000MHz,
128-entry instruction window, OoO Fetch and Execute
3 instructions/cycle, max. 1 memory instruction/cycle
32KB 4-way, 2-cycle lookup, 128B Line Size
256KB 8-way, 8-cycle lookup, 128B Line Size
1 MB/Memory Partition, 128B Line, 16-way, 700MHz
Greedy-then-oldest [43]Memory Coalescing, Inter-warp Merging,
Post Dominator
6 × 6 Shared 2D Mesh, 1400MHz, XY Routing,
2 GPU cores per node, 1 CPU core per node,
32B Channel Width, 4VCs, Buffers/VC = 4
8 Shared GDDR5 MCs, 800 MHz,FR-FCFS [42, 51], 8 DRAM-banks/MC
tCL =12, tRP =12, tRC =40, tRAS =28, tCCD =2,
tRCD =12, tRRD =6, tCDLR =5, tWR =12

Evaluation Methodology. To evaluate our proposal, we
integrated GPGPU-Sim v3.2.0 [5] with an in-house cyclelevel x86 CMP simulator. The simulation starts with CPU
execution. After the slowest CPU core warms up with
500K instructions, GPU execution starts. To measure CPU
performance, we run until the slowest core reaches 5 million
instructions. To measure GPU performance, we run theapplication until completion or 100 million instructions,whichever comes first.
Application Suites for GPUs and Benchmarks for CPUs.
We evaluate a wide range of applications for GPUs andCPUs. Our repertoire of 13 GPU applications, shown in
Table II, come from Rodinia [9], Parboil [46], MapReduce [17], the CUDA SDK [5] and LonestarGPU [7]. Weclassify these applications into three categories based on

11 If the moving average of stall
GP U in the lower or upper level isinvalid for this comparison, Part-2 does not override the decision made byPart-1 of CM-BAL.

8and 3 GPU application types form 9 workload types (e.g.,
Type-H/L-MPKI), and 4 workloads for each workload type.

their optimal TLP. The applications that perform the bestwith higher TLP values belong to Type-H, and they havehigh latency tolerance. The applications that perform thebest with lower TLP values due to lower congestion andbetter cache performance belong to Type-L. The remaining
applications that are mostly insensitive to TLP belong toType-M. The CPU applications we use include scientific,
commercial, and desktop applications drawn from the SPECCPU 2000/2006 INT and FP suites and commercial server
workloads. We choose representative execution phases fromthese applications using PinPoints [33]. We classify the CPU
applications into three types based on their LLC misses-perkilo-instruction (MPKI) rates, as shown in Table III.

Table IV: List of CPU workloads (application mixes).
CPU Mix
L1
L2
M1
M2
H1
H2

Table II: GPU applications: Type-H: Applications that benefit fromhigher TLP. Type-M: Applications that are mostly insensitive to
TLP. Type-L: Applications that benefit from lower TLP. Optimal
TLP (Opt. TLP) is given in number of warps.
#
1
2
3
4
5
6
7
8
9
10
11
12

Suite
Rodinia
Rodinia
Rodinia
Parboil
CUDA SDK
MapReduce
MapReduce
LonestarGPU
LonestarGPU
CUDA SDK
MapReduce
LonestarGPU

13

LonestarGPU

Application
Backpropagation
Hotspot
Pathfinder
Matrix Mult.
BlackScholes
Page View Count
Page View Rank
Breadth-First Src.
Min. Span. Tree
MUMerGPU
InvertedIndex
Single-Source
Shortest Paths
Survey Propagation

Abbr.
BP
HOT
PATH
MM
BLK
PVC
PVR
BFS
MST
MUM
IIX
SP

Type
Type-H
Type-H
Type-H
Type-H
Type-M
Type-M
Type-M
Type-M
Type-M
Type-L
Type-L
Type-L

Opt. TLP
48
16-48
48
16-48
16
8
8
6
8
4
1
4

SSSP

Type-L

6

Applications in the mix
povray, tonto, applu, calculix, gcc, namd, barnes, gromacs,
sjeng, dealII, wrf, h264ref, cactusADM, gobmk
sjeng, dealII, wrf, h264ref, cactusADM, gobmk, sjeng,dealII, wrf, h264ref, cactusADM, gobmk, sjeng, dealIIsjas, tpcw, Gems, hmmer, astar, sjbb, swim, ocean, sphinx3,
libquantum, art, lbm, leslie3d, xalan
ocean, sphinx3, libquantum, art, lbm, leslie3d, xalan, ocean,sphinx3, libquantum, art, lbm, leslie3d, xalan
milc, soplex, milc, omnetpp, mcf, soplex, milc, omnetpp,
mcf, soplex, mcf, soplex, omnetpp, mcf
omnetpp, omnetpp, omnetpp, omnetpp, omnetpp, milc, milc,
milc, milc, milc, soplex, soplex, mcf, mcf

Performance Metrics.. To capture GPU performance, we
use GPU speedup (SUGP U ), which is the ratio of its
instructions-per-cycle (IP C) when it runs along with CPU
to its IP C when it runs alone. We use weighted speedup
to capture CPU performance (W SCP U ) [44]. W SCP U isnP
defined as(IP Ci,multiprogram /IP Ci,alone ), where n is
i=1

Workloads. From the three CPU application types, we form
two CPU mixes per type,12 listed in Table IV. Each of thesemixes are then coupled with a GPU application to form aworkload. The GPU application is chosen in such a waythat each CPU mix gets paired with two randomly chosen
GPU applications from each of the three GPU application
types. Thus, we have six workloads for each CPU mix, fora total of 36 workloads. These workloads cover all possiblecombinations of CPU and GPU application types; 3 CPU

the number of CPU applications in the workload. All average
speedup results in the paper use harmonic mean. Based onuser preferences, one might want to change the importancegiven to CPU or GPU when calculating speedup [4]. For thisreason, we use the overall system speedup (OSS) metric [4]:OSS = (1 − α) × W SCP U + α × SUGP U , where α isa number between 0 and 1. A higher α indicates higher
GPU importance. Note that α is only an evaluation metricparameter, not an input to our schemes.
Mechanism Parameters. k is a user-dependent parameterto specify the relative importance of GPU vs. CPU. A
high value of k places less importance on the GPU. In ourevaluations, we use four values for k: 32 (CM-BAL1 ), 64(CM-BAL2 ), 96 (CM-BAL3 ), 128 (CM-BAL4 ). To achievebalanced improvements for both the CPU and the GPU, theuser can set k to 32 (called the CM-BAL1 configuration).
Mechanism Thresholds. We use 1 and 0.25 for tH and tL ,respectively. These thresholds are determined empirically,
and they correlate well with their effect on CPU applications’ performance, as given in Figure 8. These thresholds
also correlate well with the memory bandwidth saturation
and reply network saturation, as memory access latencies
start to increase non-linearly beyond the high thresholdcongestion range identified by these thresholds. Monitoring
interval is set to 1024 cycles. The thresholds require calibration for different system designs.
Hardware Overhead. As shown in Figure 11, in each core,
the warp scheduler requires a table ( 9 in Figure 11) with 99bits13 , a 6-bit register ( 10 ), and a comparator ( 6 ). CM-BAL
logic ( 8 ) requires two 10-bit adders and two shift registers.
The CTA scheduler ( 5 ) requires two 13-bit counters and twoadders to calculate/store the congestion metrics, two 13-bit

12 We limited the number of CPU mixes because we found that the
behavior of the mixes that belong to the same category are similar.

13 There are nine entries in the table. Each entry requires 11 bits because
it is incremented by two warp schedulers during a 1024-cycle interval.

Table III: CPU applications: L2 MPKI and classification#
App. L2 MPKI Type1 perlbench
0.2
L-MPKI
2 povray
0.2
L-MPKI3
tonto
0.2
L-MPKI
4 applu
0.3
L-MPKI
5 calculix
0.3
L-MPKI6
gcc
0.3
L-MPKI
7 namd0.4
L-MPKI
8 barnes
0.7
L-MPKI
9 gromacs
0.7
L-MPKI
10 sjeng
0.8
L-MPKI
11 dealII
1
L-MPKI12
wrf
1.3
L-MPKI
13 h264ref
1.5
L-MPKI
14 cactus
1.6
L-MPKI
15 gobmk
1.6
L-MPKI
16 bzip2
3.1
M-MPKI
17
sjas
4
M-MPKI#App.
L2 MPKI Type
18
tpcw
4.8
M-MPKI19 Gems
5.2
M-MPKI
20 hmmer
7
M-MPKI
21
astar
7.1
M-MPKI22
sjbb
7.5
M-MPKI
23
swim
10
M-MPKI
24 ocean
10.6
M-MPKI
25 sphinx3
12M-MPKI
26 libquantum
12.5
M-MPKI
27
art
13.6
M-MPKI
28
lbm
14
M-MPKI
29 leslie3d
14.2
M-MPKI
30
xalan
14.6
M-MPKI
31
milc
25
H-MPKI
32 omnetpp
25.8
H-MPKI33
mcf
49.8
H-MPKI
34 soplex
103.2 H-MPKI

9registers to hold the congestion thresholds, and four comparators for the CM-CPU decision. Each memory partition( 3 , 4 ) requires two 10-bit counters to sample the congestion
metrics. As such, the total storage cost of our proposal is
16 bytes per GPU core, 3 bytes per memory partition, and8 bytes for the CTA scheduler.

scenario at C leads to lower TLP in the next interval,
because the concurrency level has remained unchanged forthe past 4 consecutive intervals. At D , Part-1 of CM-BAL
reduces TLP because dif fup < k, and dif flow < k. This
example shows that CM-BAL effectively adapts concurrency
to dynamic workload behavior at fine granularity.

48
40
32
24
16
8
0

L-MPKI (L1)Time

H-MPKI (H1)

Average # of active
warps

Average # of activewarps

6.1 Dynamism of Concurrency
We first analyze the effect of our techniques on GPU
concurrency with two studies. First, Figure 13 shows theaverage number of active warps on GPU cores during thecourse of execution. Figure 13a shows the concurrency
behavior of a Type-H GPU application (PATH) when it isexecuted alongside two different CPU mixes (L1 and H1),
with CM-CPU. Because higher MPKI CPU applications
exert more pressure on the memory system than lower MPKI
applications, and because CM-CPU decisions are based
solely on congestion metrics, the number of active warps islower when a GPU application is executed alongside higher
MPKI applications. This causes CM-CPU to hurt the performance of Type-H GPU applications when boosting CPU
performance. Figure 13b shows that the TLP in CM-BAL1
is higher than in CM-CPU for the same workloads (except
for sudden drops from 48 to 24 warps with the L-MPKI
workload because there are no concurrency levels between
24 and 48 warps). As expected, L-MPKI CPU workloads
lead to higher TLP compared to H-MPKI workloads. We
observed similar behavior for Type-M and Type-L GPU
applications when run with L- and H-MPKI CPU mixes.
48
40
32
24
16
8
0

L-MPKI (L1)16
12

# of warpsA

diff_low

diff_up

C

B

D

4000
2000

8

0

4
0

k

A

-2000

B
Time

C

D

CyclesThe number ofactive warps

6 E XPERIMENTAL R ESULTS-4000

Figure 14: Number of active warps over time with CM-BAL.
dif flow , dif fup , and k are shown in the secondary y-axis.

Average # ofactive warps

Figure 15 shows the average number of warps in a GPU
core across nine workload types, with CM-CPU and CMBAL1 . As expected, CM-CPU reduces concurrency very
aggressively. CM-BAL1 maintains high TLP for Type-H
applications, low TLP for Type-L applications, and reduces
TLP of Type-M applications as much as possible without
hurting them. Hence, our techniques effectively modulate
GPU concurrency in an application-aware manner.

H-MPKI (H1)

48
40
32
24
16
8
0

CM-CPU

CM-BAL1

Type-H Type-H Type-H Type-M Type-M Type-M Type-L Type-L Type-L/ L/ M/ H/ L/ M/ H/ L/ M/ HMPKI MPKI MPKI MPKI MPKI MPKI MPKI MPKI MPKI

Figure 15: Average number of warps with CM-BAL1 .

6.2 Application Performance Results
Figure 16 shows the average performance of GPU applications obtained by our schemes for all workload types.
The results are normalized to the performance of the same
workloads when they are executed using the baseline warp
scheduler, GTO. We also evaluate DYNCTA after tuning its
parameters. We make several observations. First, DYNCTA
works only for Type-L applications, and improves theirperformance by 7%, while not hurting any workload bymore than 2%. Second, CM-CPU causes significant performance losses in Type-H and Type-M applications, as
those applications require high latency tolerance and CMCPU is aware only of memory congestion, not the latency
tolerance of GPU cores. However, CM-CPU improves theperformance of Type-L applications due to successful congestion management. We observe high benefits for TypeL/L-MPKI workloads due to two workloads that run IIX.
IIX prefers 1-2 active warps, but its CTAs have more warps.
Because DYNCTA manages the TLP at the CTA level, itcannot reduce concurrency as much CM-CPU can. Third,
CM-BAL1 , recovers the losses caused by CM-CPU for
Type-H and Type-M applications by providing more latency
tolerance. As a result, the worst performing GPU application

Time(a) PATH with CM-CPU.
(b) PATH with CM-BAL1 .
Figure 13: Average number of active warps over time.

Second, Figure 14 provides insight into the working of
CM-BAL by showing the relationship between the metrics
monitored by CM-BAL and the number of active warps ona core, over a short period of time, in one of our workloads (Type-L/H-MPKI – MUM/H1). In this plot, dif flow
= (stallGP U of the lower concurrency level − stallGP U
of the current concurrency level); and dif fup = (stallGP U
of the current concurrency level − stallGP U of the upperconcurrency level). If dif flow > k, CM-BAL does not
reduce the GPU TLP. If dif fup > k, CM-BAL increases
the GPU TLP (see Section 4.2). At A , since dif fup > k,
the number of warps is increased for the next interval. This
workload stresses the memory, leading to high stallM C . As
a result, at B , Part-1 of CM-BAL wants to reduce TLP.
However, because dif flow > k (i.e., reducing TLP would
likely lead to increased GPU stall time), Part-2 of CM-BAL
kicks in, and TLP is not reduced. In contrast, a similar10

as much as CM-CPU. Yet, it also does not cause lower
latency tolerance (i.e., increased stalls) for GPU cores; infact, it reduces GPU stalls in most Type-L applications.

Normalızed GPU IPC

1.5

Type-H / M-MPKI
Type-M / M-MPKI
Type-L / M-MPKI

Type-H / H-MPKI
Type-M / H-MPKI
Type-L / H-MPKI

1.4
1.2
1
0.8
0.6
0.4
0.2
0

CM-CPU

CM-BAL1(b) stallGP U /GPU instruction.

Figure 18: stallM C and stallGP U normalized to 48 warps, foreach workload type. H-L denotes Type-H/L-MPKI workload.

0.5Overall System Speedup. Figure 19 compares the baseline, DYNCTA, and our schemes in terms of Overall System
Speedup (OSS) [4], defined in Section 5. Recall that ahigher α value indicates that GPU performance is more
important for overall system performance. Several observations are in order. First, DYNCTA is slightly better thanthe baseline for all α values, indicating balanced improvements for both CPUs and GPUs. Second, CM-BAL1 also
provides balanced improvement, but is significantly better
than DYNCTA. Third, for small α, where preference isgiven to CPU performance, CM-CPU is the best scheme.
As α approaches 0.5, where the OSS metric equally weights
CPU and GPU performance, both schemes provide almost
equal performance. Because CM-CPU deteriorates GPU
performance, its OSS drops below that of the baseline
for large α. Fourth, with large k, CM-BAL becomes more
preferable for small α, but performs worse than the baseline
for large α, showing the trade-off between CPU and GPU
performance in CM-BAL. These results show that the user
can pick a different CM-BAL level (based on preference of
how important the GPU is relative to the CPU) that would
lead to the highest OSS. Hence, CM-BAL can be configured
to consistently provide better overall system performance
than state-of-the-art designs regardless of how important the
CPU or the GPU is to system performance.

0
DYNCTA CM-CPU CM-BAL1 CM-BAL2 CM-BAL3 CM-BAL4

Figure 16: GPU speedup over baseline with 48 warps.

Figure 17 shows the performance of CPU workloads
with our schemes, normalized to that of our baseline.
First, as discussed in Section 3.1, DYNCTA is not suitablefor improving CPU performance, and we observe that itprovides only 2% performance improvement for CPUs.
Second, CM-CPU reduces the number of active warps very
aggressively, providing high benefits for CPUs. Because the
GPU performance loss of Type-M applications in CM-CPU
is substantial due to aggressive throttling, the corresponding
CPU benefits are high. Third, CM-BAL1 throttles Type-H
applications conservatively. Thus, CPU benefits for those
applications are low. As k increases, CPU benefits of CMBAL increase, and for the largest possible k (describedin Section 4.2), these benefits converge to those of CMCPU. Fourth, CM-BAL provides a knob for controllingtrade-offs between CPU and GPU performance, with CMBAL1 specifically helping GPU applications and CM-BAL4
helping CPU applications. Overall, CM-CPU and CM-BAL1
provide 24% and 7% average CPU performance improvement, respectively, across all workloads.
Type-H / L-MPKI
Type-M / L-MPKI
Type-L / L-MPKI

Type-H / M-MPKI
Type-M / M-MPKI
Type-L / M-MPKI

Type-H / H-MPKI
Type-M / H-MPKI
Type-L / H-MPKI

Normalized OSS (α)

Normalızed CPU WS

CM-BAL1(a) stallM C per cycle.

11.5

CM-CPU

1.2
1
0.8
0.6
0.4
0.2
0

Normalized
stallGPU/inst.

Type-H / L-MPKI
Type-M / L-MPKI
Type-L / L-MPKI

Normalized
stallMC/cycle

suffers only 4% performance loss. No workload has higher
GPU performance with CM-CPU than with CM-BAL1 . As
k increases, GPU benefits of CM-BAL reduce. Overall, CMCPU causes 11% performance loss, and CM-BAL1 provides
7% performance improvement for GPU applications.

1
0.5
0
DYNCTA CM-CPU CM-BAL1 CM-BAL2 CM-BAL3 CM-BAL4

DYNCTA

CM-CPU

CM-BAL2

CM-BAL3

CM-BAL4

CM-BAL1

1.25
1.125
1

0.875
0.0

Figure 17: CPU weighted speedup over baseline with 48 warps.

48 warps

0.5
α

1.0

Figure 19: Normalized Overall System Speedup of our schemes
over baseline with 48 warps for all α.

Figure 18 shows the effects of our schemes on the metrics
used by them, normalized to their values in our baseline.
We do not show stallnet because it is negligible in mostworkloads. stallM C per cycle goes down significantly with
CM-CPU, due to aggressive throttling. However, this leads
to an increase in stallGP U per executed GPU instruction
in most workloads, i.e., lower latency tolerance for GPU
cores. CM-BAL1 reduces memory congestion, although not

6.3 Comparison to Static Warp Limiting
Figure 20a compares the GPU performance of ourschemes with static limiting (SWL) schemes, where the
GPU cores execute a constant number of warps concurrently
throughout the whole execution. Throttling down from 48
11Type-H

Type-M

Type-L1

0.5
0Normalized CPU IPC

Normalized GPU IPC

1.5

2

L-MPKIM-MPKIH-MPKI

Effect of GPU Aggressiveness. Although we evaluate our
proposal with a Fermi-like GPU model (Section 5), we also
expect to observe benefits in newer architectures, due toincreased TLP and increased contention for shared memory
resources. For example, in the newer NVIDIA Kepler [39]:1) GPU cores have more processing elements, leading tomore GPU TLP, 2) L1 cache capacity is the same as
Fermi [38], leading to higher cache contention, 3) GDDR5
memory bandwidth [1] has not increased proportionallywith core count, leading to higher memory contention. We
envision that these three factors, which indicate trends in
GPU platform design, will make concurrency management
even more important in newer and future architectures.

1.5
1
0.5
0

(a) GPU performance.
(b) CPU performance.
Figure 20: Our schemes vs. static warp limiting. Performancecomparisons are clustered based on GPU/CPU application types.

warps to 4 causes performance loss for Type-H applications,improves Type-L applications, and does not affect TypeM applications. Further throttling hurts GPU applications
significantly. CM-BAL1 provides better performance forGPUs compared to all static warp limits. Figure 20b showsthe effect of static warp throttling on CPU performance.
Although limiting the number of warps to 4 or 6 benefits
CPU performance, this comes at the expense of significant
GPU performance degradation.14 We conclude that static
warp throttling cannot enable flexible performance trade-off
between the CPU and the GPU and optimization of overallsystem performance as CM-BAL does (Section 6.2).

7 R ELATED W ORKWhen CPU and GPU applications are co-scheduled on thesame hardware, they interfere with each other in the sharedresources. In this context, we discuss the prior works that
address such interference at various points in the hierarchy.
Managing Interference at NoC, Caches, and Memory.
Lee et al. characterize network contention [31] and propose
virtual channel partitioning [30] to mitigate it. Ausavarungnirun et al. [4] propose a memory scheduling design that reduces the interference of CPU and GPU applications. Jeonget al. [19] provide a mechanism that provides QoS for GPU
applications in a CPU-GPU system by aptly prioritizing
requests at the MCs. Lee and Kim [29] observe interference
between CPU and GPU applications at the LLC, and propose
cache management techniques that also consider TLP of
GPU applications. Jog et al. [21] propose a simple roundrobin memory scheduler for reducing contention caused bymultiple GPU applications. The focus of all these works is tohandle the application interference at only one of the levelsof the memory hierarchy. In contrast, we show that GPUs
create interference to CPUs at all parts of the shared memory
hierarchy. To address this interference, we modulate the TLPof GPU applications by considering system-wide congestion
metrics and GPU latency tolerance.
Managing Interference from the Core. Kayiran et al. [24]and Rogers et al. [43] modulate TLP in GPUs to reduce
contention in the memory system and L1 caches, respectively. Jog et al. [22, 23] propose warp scheduling techniques
to address contention in GPU caches and memory. Many
source throttling schemes [8, 10, 15, 40, 41, 48] are proposedin the context of CPUs to address congestion in NoCs andmemory. None of these works are designed or evaluated for
CPU-GPU architectures. To our knowledge, this is the firstwork that uses source throttling to 1) to control interference
between both CPU and GPU applications, and 2) to reduce
interference both at the NoCs and main memory in aheterogeneous CPU-GPU system.

6.4 Sensitivity Experiments
Scalability. Our scalability analysis shows that varying thenumber of GPU cores in a tile between 2, 4 (default), and6 shows minimal change in performance benefits of ourtechniques. We also evaluated a smaller and less powerhungry system with 4 GPU cores, 3 CPU cores, and 2 MCsconnected via a 3 × 3 mesh. This configuration is closer tocurrently available systems in terms of the number of cores.
CM-BAL1 provides 13% and 2% average CPU and GPU
performance improvement, respectively, on this system, evenwith unoptimized thresholds.
Partitioned vs. Shared Resources. We evaluated CMBAL1 using a partitioned network and shared MCs. On thisbaseline, which performs significantly worse than the shared
network-shared MC baseline (Section 2.2), CM-BAL1 provides 15% GPU and 8% CPU performance improvement.
6.5 Other Analyses and Discussion
LLC Contention. Our mechanisms reduce LLC contention
mainly due to two reasons. First, TLP management reduces
the rate at which memory requests are issued from GPU
cores. Second, TLP management reduces L1 miss rates,
leading to fewer LLC accesses. Overall, we observe that
CM-BAL1 reduces the number of LLC accesses by 13%.
Power Consumption. Recent works [16, 45] show that
concurrency management in GPUs not only improves performance but also increases energy-efficiency for memoryintensive applications. Thus, we expect our schemes toimprove energy-efficiency for memory-intensive workloads,
and not reduce it for compute-intensive workloads.

8 C ONCLUSIONSTLP management in GPUs is essential in hybrid CPUGPU environments with shared resources to maximize overall system throughput. In this paper, we introduce a new
GPU concurrency throttling strategy that tunes the impact

14 At the extreme case, completely disabling GPU execution leads to 69%
higher CPU performance over the baseline.

12

of GPU applications on both CPU applications and system
performance. Our strategy takes into account both systemwide memory/network congestion and GPU latency tolerance to dynamically decide the level of GPU concurrency.
We propose two different implementations of our strategy: 1)
to boost the performance of CPUs, and 2) to flexibly improveoverall system performance based on the user’s preference
for higher CPU or GPU performance. Experimental evaluations show that the first scheme significantly improves
average CPU performance, but causes some GPU performance loss. The tunable second scheme, when configured to
improve CPU and GPU performance in a balanced manner,provides 7% performance benefit for both CPU and GPU
applications, without significantly hurting any workload’s
performance. We conclude that our GPU TLP management
framework provides a flexible and efficient substrate tomaximize system performance and control CPU-GPU performance trade-offs in modern heterogeneous architectures.

[19] M. K. Jeong et al., “A QoS-aware Memory Controller for Dynamically Balancing GPU and CPU Bandwidth Use in an MPSoC,” in
DAC, 2012.
[20] M. K. Jeong et al., “Balancing DRAM Locality and Parallelism inShared Memory CMP Systems,” in HPCA, 2012.
[21] A. Jog et al., “Application-aware Memory System for Fair andEfficient Execution of Concurrent GPGPU Applications,” in GPGPU,
2014.
[22] A. Jog et al., “Orchestrated Scheduling and Prefetching for GPGPUs,”
in ISCA, 2013.
[23] A. Jog et al., “OWL: Cooperative Thread Array Aware SchedulingTechniques for Improving GPGPU Performance,” in ASPLOS, 2013.
[24] O. Kayiran et al., “Neither More Nor Less: Optimizing Thread-level
Parallelism for GPGPUs,” in PACT, 2013.
[25] Khronos OpenCL Working Group, “The OpenCL Specification,”
2008.
[26] Y. Kim et al., “ATLAS: A Scalable and High-Performance Scheduling
Algorithm for Multiple Memory Controllers,” in HPCA, 2010.
[27] Y. Kim et al., “Thread Cluster Memory Scheduling: Exploiting
Differences in Memory Access Behavior,” in MICRO, 2010.
[28] C. J. Lee et al., “Improving Memory Bank-Level Parallelism in the
Presence of Prefetching” ,” in MICRO, 2009.
[29] J. Lee and H. Kim, “TAP: A TLP-aware Cache Management Policy
for a CPU-GPU Heterogeneous Architecture,” in HPCA, 2012.
[30] J. Lee et al., “Adaptive Virtual Channel Partitioning for Network-onchip in Heterogeneous Architectures,” ACM TODAES, 2013.
[31] J. Lee et al., “Design Space Exploration of On-chip Ring Interconnection for a CPU-GPU Heterogeneous Architecture,” JPDC, 2013.
[32] E. Lindholm et al., “NVIDIA Tesla: A Unified Graphics and Computing Architecture,” Micro, IEEE, vol. 28, no. 2, 2008.
[33] C.-K. Luk et al., “Pin: Building Customized Program Analysis Tools
with Dynamic Instrumentation,” in PLDI, 2005.
[34] A. K. Mishra et al., “A Heterogeneous Multiple Network-on-chip
Design: An Application-aware Approach,” in DAC, 2013.
[35] S. P. Muralidhara et al., “Reducing Memory Interference in Multicore
Systems via Application-aware Memory Channel Partitioning,” in
MICRO, 2011.
[36] V. Narasiman et al., “Improving GPU Performance via Large Warps
and Two-level Warp Scheduling,” in MICRO, 2011.
[37] NVIDIA. (2011) CUDA C/C++ SDK Code Samples. Available:
http://developer.nvidia.com/cuda-cc-sdk-code-samples[38] NVIDIA.
(2011)
Fermi:NVIDIA’s
Next
Generation
CUDA
ComputeArchitecture.
Available:
http://www.nvidia.com/content/PDF/fermi white papers/NVIDIA
Fermi Compute Architecture Whitepaper.pdf[39] NVIDIA. (2012) NVIDIA’s Next Generation CUDA Compute Architecture: Kepler GK110. Available: http://www.nvidia.com/content/
PDF/kepler/NVIDIA-Kepler-GK110-Architecture-Whitepaper.pdf[40] G. Nychis et al., “Next Generation On-Chip Networks: What Kindof Congestion Control Do We Need?” in Hotnets, 2010.
[41] G. Nychis et al., “On-chip Networks from a Networking Perspective:
Congestion and Scalability in Many-core Interconnects,” in SIGCOMM, 2012.
[42] S. Rixner et al., “Memory Access Scheduling,” in ISCA, 2000.
[43] T. G. Rogers et al., “Cache-Conscious Wavefront Scheduling,” in
MICRO, 2012.
[44] A. Snavely and D. M. Tullsen, “Symbiotic Jobscheduling for a
Simultaneous Multithreaded Processor,” in ASPLOS, 2000.
[45] S. Song et al., “Energy-efficient Scheduling for Memory-intensive
GPGPU Workloads,” in DATE, 2014.
[46] J. A. Stratton et al., “Parboil: A Revised Benchmark Suite for
Scientific and Commercial Throughput Computing,” Univ. of Illinois,
Tech. Rep. IMPACT-12-01, March 2012.
[47] M. A. Suleman et al., “Feedback-driven Threading: Power-efficientand High-performance Execution of Multi-threaded Workloads on
CMPs,” in ASPLOS, 2008.
[48] M. Thottethodi et al., “Self-Tuned Congestion Control for Multiprocessor Networks,” in HPCA, 2001.
[49] M. Xie et al., “Improving System Throughput and Fairness Simultaneously in Shared Memory CMP Systems via Dynamic Bank
Partitioning,” in HPCA, 2014.
[50] X. Zhang et al., “Hardware Execution Throttling for Multi-core
Resource Management,” in USENIX, 2009.
[51] W. K. Zuravleff and T. Robinson, “Controller for a Synchronous
DRAM that Maximizes Throughput by Allowing Memory Requests
and Commands to be Issued Out of Order,” Patent U.S. 5,630,096,
1997.

ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable
feedback. This research is supported in part by NSF grants#1212962, #1205618, #1213052, #1302225, #1302557,#1317560, #1320478, #1409095, #1439021, #1439057,#1409723, the Intel Science and Technology Center on
Cloud Computing, SRC and Intel.
R EFERENCES[1] G.
Abbas.
(2012)NVIDIA
CUDA:
Keplervs.
Fermi Architecture. Available: http://blog.cuvilib.com/2012/03/28/
nvidia-cuda-kepler-vs-fermi-architecture/[2] Advanced Micro Devices, Inc. (2013) What is Heterogeneous
System Architecture (HSA)? Available: http://www.amd.com/en-gb/
innovations/software-technologies/hsa[3] AMD. (2014) Compute Cores. Available: http://www.amd.com/
Documents/Compute Cores Whitepaper.pdf[4] R. Ausavarungnirun et al., “Staged Memory Scheduling: Achieving
High Prformance and Scalability in Heterogeneous Systems,” in ISCA,
2012.
[5] A. Bakhoda et al., “Analyzing CUDA Workloads Using a Detailed
GPU Simulator,” in ISPASS, 2009.
[6] A. Bakhoda et al., “Throughput-effective On-chip Networks for
Manycore Accelerators,” in MICRO, 2010.
[7] M. Burtscher et al., “A Quantitative Study of Irregular Programs on
GPUs,” in IISWC, 2012.
[8] K. K.-W. Chang et al., “HAT: Heterogeneous Adaptive Throttling for
On-Chip Networks,” in SBAC-PAD, 2012.
[9] S. Che et al., “Rodinia: A Benchmark Suite for Heterogeneous
Computing,” in IISWC, 2009.
[10] H. Cheng et al., “Memory Latency Reduction via Thread Throttling,”
in MICRO, 2010.
[11] W. Dally, “GPU Computing to Exascale and Beyond,” in SC, 2010.
[12] W. Dally and B. Towles, Principles and Practices of Interconnection
Networks. Morgan Kaufmann, 2003.
[13] R. Das et al., “Application-to-core Mapping Policies to Reduce
Memory System Interference in Multi-core Systems,” in HPCA, 2013.
[14] R. Das et al., “Application-aware prioritization mechanisms for onchip networks,” in MICRO, 2009.
[15] E. Ebrahimi et al., “Fairness via Source Throttling: A Configurable
and High-performance Fairness Substrate for Multi-core Memory
Systems,” in ASPLOS, 2010.
[16] M. Gebhart et al., “Energy-efficient Mechanisms for Managing
Thread Context in Throughput Processors,” in ISCA, 2011.
[17] B. He et al., “Mars: A MapReduce Framework on Graphics Processors,” in PACT, 2008.
[18] Intel. (2012) Products (Formerly Ivy Bridge). Available: http://ark.intel.com/products/codename/29902/

13