Improving Inter-kernel Data Reuse
With CTA-Page Coordination in GPGPU
Xuanyi Li∗ , Chen Li∗‡ , Yang Guo∗ , Rachata Ausavarungnirun†‡∗ National University of Defense Technology, Changsha, China† King Mongkut’s University of Technology North Bangkok, Bangkok, Thailand∗ {lixuanyi, lichen, guoyang}@nudt.edu.cn, † r.ausavarungnirun@gmail.com

Abstract—Although modern GPUs are equipped with expanding memory, accommodating the entire working set of largescale workloads can still be a challenge. With the support ofunified virtual memory and demand paging, programmers cantransparently oversubscribe the main memory. However, thistransparent management still comes at a severe performancecost, especially for applications with inter-kernel data sharing.
While there have been many efforts to reduce additional data
migrations caused by the memory oversubscription, few consider
the reuse of shared data during the boundary of adjacent kernels.
Due to limited memory capacity, we observe that adjacentkernel often demands shared pages that were evicted by theprevious kernel, resulting in a significant number of costly data
migrations. In this paper, we propose a CTA-Page collaborative
framework, called CPC, that transparently reduces the impact ofmemory oversubscription using CTA dispatch switching and page
replacement switching coordinately to reuse inter-kernel shareddata.
We evaluate CPC with a variety of GPGPU benchmark
suites. Experimental results show that the system performance isimproved by 65% compared with the state-of-the-art techniquefor applications with inter-kernel data sharing.
Index Terms—memory oversubscription, CTA-Page Coordination, unified memory

I. I NTRODUCTION
Due to high computing throughput and increasing programmability, the Graphics Processing Units (GPUs) havebeen widely used in various fields, including machine learning [1], object detection [2], and image denoising [3] toaccelerate large scale parallel computation. However, limited
GPU memory capacity cannot satisfy the increasing GPU application working set size, which leads to serious performanceslowdown as data needs to be moved back and forth between
the CPU memory and the GPU memory.
To manage data between CPU’s memory and GPU’s memory, programmers have to manually manage the data movement between the CPU memory and the GPU memory, whichsignificantly sacrifices the productivity of programmers. While
Unified Virtual Memory and Demand Paging [4] are introduced to provide transparent memory oversubscription, thesetechniques still bring high-performance penalties as additional
data pages can migrate between the CPU memory and the GPU
memory. Therefore, reducing these additional data movements
becomes more and more important.
‡ refers to corresponding author.

We observe that there are kernels with producer-consumer
dependencies and common inputs in many applications according to the analysis across multiple benchmark [5]–[9].
Among these applications, shared data produced or utilized by
the CTA of the previous kernel is usually consumed or utilized
by the corresponding CTA of the current kernel. For suchcases, kernels with data sharing access the same data area ata similar sequence. When the GPU memory is oversubscribedand cannot accommodate the whole working set size of thekernel, old pages will be swapped out to make room for new
pages. At the end of each kernel, only the latest accessed
pages remain in the GPU memory. When the following kernel
launches, the same pages which have been evicted by theprevious kernel are accessed again. All these page faults
appear at the beginning of the new kernel, causing severecongestion in paging. Although kernels share lots of data withneighbored kernels in these applications, such data sharing
can be broken by the memory oversubscription. As shown in
Fig.1, data is migrated from the CPU memory to the GPU
memory contiguously throughout the whole executing processof the application with inter-kernel data sharing under theoversubscribed GPU memory. Therefore, original data reuse
between neighbored kernels no longer exists, leading to a large
number of page faults.
Footprint

GPU Memory

CPU Memory...
Kernel1

Kernel2

Time

Fig. 1. Paging process of the application with inter-kernel data sharing under
the oversubscribed GPU memory.

There are several previous works alleviating the performance loss caused by memory oversubscription, including
ETC [10], prefetching pages [11], [12] and batched faulting
pages [13], but they can hardly improve the performancefor the applications with inter-kernel data sharing. Thesemethods either hide the eviction latency or accelerate thepage migration, but they cannot reduce the number of page
movements. Therefore, our goal is to explore the opportunities
in reusing shared data between the adjacent kernels.
In this paper, we propose a CTA-Page Collaborative framework, called CPC, which improves the performance by utilizing the inter-kernel data sharing in an application-transparent

way. According to the execution of shared-data kernels,
CPC coordinates (1) cooperative thread array (CTA) dispatch
switching (denoted as DispatchSwitch) that switches the direction of CTA dispatching sequence and (2) page replacement
policy switching (denoted as ReplacementSwitch) that switches
the page replacement policy to effectively reuse the shared data
between kernels. We evaluate CPC with a variety of GPGPU
benchmark suites and show that CPC outperforms the stateof-the-art technique by 65%.
This work makes the following contributions:This paper provides an in-depth analysis of page accesspatterns for applications with inter-kernel data sharing.
We find that such data reuse is destroyed completely
when the memory is oversubscribed since data has to beevicted and paged back and forth among multiple kernels.
• We propose an application-transparent framework, CPC,
to reduce the impact caused by the memory oversubscription. Our solution reuses the inter-kernel shareddata, which is effective in reducing the data movements
between the CPU memory and the GPU memory.
• We find that both DispatchSwitch and ReplacementSwitch
are important to coordinate related computation and data
together for reducing the page fault rate at the boundaryof neighbored kernels. Overall, CPC outperforms thestate-of-the-art technique by 65% with an average 56.7%
data reuse rate.
•

II. BACKGROUNDThis section first introduces the GPU execution model (Section II-A), the architecture of modern GPUs (Section II-B), andthe CTA scheduling policy (Section II-C).
HOST
CPU

CPU
MemoryBridgeGPU DriverGPU
Command Processor
SM
CTA Scheduler
SM

SM

...

SM

Interconnect Network
L2 Cache

...

MC

...

MC

...

MemoryPartition

MemoryPartition

L2 CacheSP

Warp Scheduler/
Scoreboard
SP

SP

...

SP

Register File
Memory Subsystem
SharedMem

L1 Cache

Const
Cache

TextureCache

NVLink/PCI-e

Fig. 2. Baseline GPU Architecture

A. GPU Execution model
GPU applications usually consist of multiple kernels, wherekernels without data dependency can run simultaneously. EachGPU kernel consists of groups of threads called Thread Block(TB) or Cooperative Thread Array (CTA) 1 . A CTA is split
into multiple warps, where all threads execute in lockstep.
Each CTA shares a scratchpad memory called shared memory.
Though threads within the CTA execute coordinately throughvarious synchronization methods, CTAs can be executed in any
order while ensuring the correctness. Therefore, modifying the
CTA scheduling policy has no impact on application. Suchproperty is critical for our proposed designs.
With both Unified Virtual Memory and Demand Paging [4],modern GPUs can transparently manage data, which improves
programmability significantly. In this case, data can be accessed with the same virtual address by both CPU and GPU
on demand instead of all being copied before execution. When
the GPU intends to access a physical page absent in thedevice memory, it generates a page fault and the GPU driversends data at the page or multi-page granularity from theCPU memory to the GPU memory. This functionality provides
the opportunity to oversubscribe the GPU memory. However,this suffers significant performance overhead because the pagefaults block the execution until the requested data is loaded
and such process normally lasts for tens of microseconds [11].
B. GPU Architecture
Fig.2 shows the baseline GPU architecture based on
NVIDIA’s design. A command processor is used to receive thecommands from the CPU and controls the GPU accordingly[14]. The CTA scheduler selects CTAs and dispatches theminto the proper SMs based on the status of SMs. A single
GPU consists of multiple streaming multiprocessors (SMs) andseveral shared memory partitions connected via interconnect
network. Data transfer between CPU and GPU is through offchip links such as NVLink and PCI-e. Each SM is composedof multiple streaming processors (SPs) supporting multiple
threads executing concurrently. These threads form a warp,
which is the basic execution unit. Warps are selected andissued by the warp scheduler from a warp pending pooltracked by the scoreboard, which delivers the high utilizationof SMs. Local GPU memory system including various typesof L1 caches used in different data spaces for the computecores. A data access can fetch its data from the caches, theGPU memory via the interconnection network, or from CPU
memory through the external link, which can cost hundreds
or thousands of cycles [11].
C. CTA SchedulingMany prior works propose various types of CTA schedulingpolicy [15]–[19]. The default CTA scheduling policy has been
assumed as a simple round-robin (RR) scheme [20], [21].
CTAs are selected from the CTA pending pool in ascending
1 We use NVIDIA’s terminology here as our validation platforms are
NVIDIA products and the proposed methods are also applicable to other typesof GPUs.

0.9096

2

80%

85%

90%

Fig. 3. Impact of memory oversubscription to the performance of applications
with inter-kernel data sharing. The performance drop is only 0.9096 when theoversubscription ratio decreases from 95% to 50%.)

Fig.4 shows both the page access pattern and page fault
rate of FFT when the GPU memory can only fit 75% of
its footprint. We find that each kernel has a similar accesspattern and sequence. As shown in the Fig.4 with blue circles,
there are amounts of page faults at the boundary of kernels.
When a new kernel is launched, all new threads are dispatched
to SMs in the GPU and start accessing those pages at thesame time. However, all these pages have been evicted due tomemory oversubscription at the end of the last kernel shownin Fig.4. Moreover, since the kernel is newly launched, alldispatched threads are seeking new pages together. It causes
these large amounts of page faults. In order to reduce thenumber of page migrations under the oversubscribed GPU
memory, previous works [10]–[13] have been proposed toreduce the migration overhead by prefetching [11], [12], overlapping eviction latency with paging [10] and batched faulting
pages together [13]. However, these techniques provide little
performance benefit for inter-kernel data sharing applications.
Prefetching would cause severe thrashing for such applications
because it may evict used pages proactively. Proactive eviction

K8

100%

60%
40%

1500000

20%

0

0

200

400

600

0%
1000

800

Cycle Count (×10k)

Fig. 4. Page access pattern and page fault rate of FFT when 75% of the
footprint can be fit into the memory (kernels within FFT access the same
pages in the same sequence as shown by the black dots; dashed line represents
the end of each kernel; page fault rate is high at the boundary of neighboredkernels as shown in the circled regions).

In summary, we make two observations for applications
with inter-kernel data sharing. First, once memory is oversubscribed, shared data reusing is broken at the boundary of
kernels and the performance drops sharply due to thrashing.
Second, previous works have not considered data reuse issues
for inter-kernel data sharing under memory oversubscription.
Therefore, the goal of this paper is to alleviate the overheadof memory oversubscription through reusing the shared data.

K1

6000000

95%

Oversubscription Ratio

K7

3000000

Page ID

75%

K6

80%

Page ID

50%

K5Page Fault Rate

K44500000

1
0

Page Fault Rate

K3

K2
47.8

Page Fault Rate
K3
93.3

K4

K5

K6

K7

K8

100%
80%

4500000

60%

3000000

40%

1500000
0

20%
0

115

230

345

460

575

0%

Page Fault Rate

3

K2

Cycle Count (×10k)(a) DispatchSwitch is applied (page fault rate is low at the boundary ofneighbored kernels but the odd-round kernels cost more time than the evenround kernels)
K1

6000000

K2
44.8

K3 K4
41.8

K5

K6

K7

K8

80%

4500000

60%

3000000

40%

1500000
0

100%

20%
0

50

100

150

200

250

300

350

400

Page Fault Rate

4

Page ID
K1

6000000

Page ID

Runtime Normalized to
No Oversubscription

III. M OTIVATION
Efficient GPU memory utilization is critical to avoid thelong latency of page faults, especially when the GPU memory is oversubscribed. Fig.3 shows the average performancedegradation due to memory oversubscription for applications
with inter-kernel data sharing. We make two observations.
First, such applications suffer from an average of 198.4%
and 183.4% performance loss when 75% of the total memory
footprint fit in the GPU memory and when the oversubscription ratio varies from 50% to 95%, respectively. Second,
the performance impact of memory oversubscription to suchapplications is insensitive to the degree of oversubscription,
indicating that the performance of such applications degrades
drastically once the memory is oversubscribed.

can hide some eviction latency, but it cannot help with
reducing such page faults. Batched faulting pages is effectivein amortizing the page fault overhead, but it is useless forregular applications with streaming access pattern.

Page ID

order and assigned to each available SMs in multiple rounds
by the CTA scheduler (i.e., GigaThread Engine). This process
stops until all of the CTAs within the pool have been dispatched or no SMs are available, limited by both resources(e.g., shared memory, registers, etc.) or hardware (e.g., warp
slots, etc.). Whenever SMs finish CTAs, new CTAs can bescheduled. In this paper, we also leverage RR as the baseline
scheduling policy.

0%

Cycle Count (×10k)(b) CPC is applied (execution time of the odd-round kernels is reduced alot)
Fig. 5. Page access pattern and page fault rate of FFT using DispatchSwitch
and CPC when 75% of the footprint can be fit into the GPU memory (kernels
within FFT access the same pages at the same sequence as shown by theblack dots; dashed line represents the end of each kernel).

IV. CPC D ESIGN
Based on the analysis in Section III, we observe thatcoordinating CTAs and the remaining data in the memory isa key solution to reduce the page fault rate at the boundary of
kernels. This observation guides the design of the CTA-Pagecollaborative framework, CPC. The key principle of CPC is to1) dispatch related CTAs first to reuse the data remained in thememory through CTA dispatch switching (Section IV-A) and2) replace useless pages to cooperate with the corresponding
CTA dispatching strategy through page replacement switching(Section IV-B).
HOST
CPU

GPU Driver

Allocation

Eviction
Head

GPU

SM

...

SM

EvictionInterconnect Network

Mux

Page Replacement Policy❷

L2 CacheMCGPU Command Queue

MemoryPartition

...
...
...

0

CTA Pending Pool

CTA Scheduler

Dispatch

CTA Pending Pool

Page List

Page List

SWITCH

SWITCH

Command Processor

SM

Tail
Allocation

CPU
MemoryBridgen-1
Dispatch

L2 CacheMC
MemoryPartition

SWITCHMux

CTA Dispatching Strategy❶

NVLink/PCI-e

Normalized IPC toBaseline

Fig. 6. Architecture of CPC (¶: CTA Dispatch Switching; ·: Page Replacement Switching)

250%

58.5%

200%
20.2%

150%
100%
50%
0%

Baseline

ETC

DispatchSwitch

OracleFig. 7. Performance of DispatchSwitch, ETC, Oracle normalized to thebaseline.

A. CTA Dispatch Switching
However, how to find the CTAs related to the pages in thememory becomes our first design challenge. Observed fromapplications with inter-kernel sharing, we find that most ofthese applications have similar access patterns among different
kernels, which is also shown in the Fig.4. Within each kernel,
such page access behavior is strongly related to the CTA
scheduling scheme, since each thread typically uses thread ID
and associated CTA ID to identify the data that it operates on.
To this end, we propose a CTA dispatch switching mechanism, called DispatchSwitch. It simply switches the dispatching order of CTAs whenever a new kernel is launched to utilize

the remaining data in the GPU memory. As shown in ¶ ofFig.6, the switch flag denoted as SWITCH is reset when a newcontext is created and is reversed by the command processor
whenever a new kernel is launched. For each kernel, theCTA scheduler selects corresponding CTA dispatching strategy
based on the SWITCH. When SWITCH equals 0, CTAs of thecurrent kernel are dispatched in ascending order otherwise indescending order. Thereby remaining data in the GPU memory
is utilized effectively by the following kernel.
To show the effectiveness of this mechanism, we compare itagainst the state-of-the-art framework, called ETC, which focuses on reducing the impact of memory oversubscription [10].
We also introduce an oracle design in which all shared data
between neighbored kernels within the GPU memory is reused.
Fig.7 presents the average performance of DispatchSwitch,
ETC and Oracle normalized to the baseline using the default
configurations as shown in Table I. Page access pattern andpage fault rate of FFT using DispatchSwitch are shown in
Fig.5(a). We make three observations from these figures.
First, compared to ETC [10], DispatchSwitch improves theperformance by 20.2% on average in line with the pagefault reduction at the boundary of kernels. It means that
DispatchSwitch is effective in reusing shared data. Second,
DispatchSwitch performs 58.5% worse than Oracle, indicating
there is an opportunity to further reuse the shared data. Third,
an interesting finding is that the execution time of the evenround kernels is shorter than that of the odd-round kernels
as shown in the Fig.5(a). It leads us to further dig into thedifference between the odd and even rounds of kernels.
We create and analyze the execution process of a simple
workload with multiple kernels which access the same data as
shown in Fig.8. In order to simplify our analysis, we makeseveral assumptions as shown in Fig.8. We first analyze theexecution process of the baseline as shown in Fig.8(a). Kernels
are launched sequentially and CTAs are also dispatched inascending order. At first, requested pages are loaded into theempty GPU memory. Due to the limited memory capacity
and age-based LRU page replacement policy (first loaded,first evicted) [22], the earlier loaded pages are evicted first to
make room for new pages. Therefore, throughout the whole
execution process, no data requests hit in the GPU memory,leading to performance degradation consistent with Fig.3.
When DispatchSwitch is applied, the execution process
is shown in Fig.8(b). Different from the baseline, the CTA
dispatching sequences are switched every kernel. We maketwo observations. First, there are fewer page faults at theboundary of kernels compared to the baseline consistent withthe page faults reduction at the boundary (Fig.5(a)). Second,
for odd-round kernels, the remaining data is not fully reused
accounting for the performance gap with the oracle case. Insummary, switching the CTA dispatching sequence can provide a significant performance benefit, but it still cannot fully
utilize the remaining data in the GPU memory. Coordinating
the page replacement policy with DispatchSwitch is key tobridge such a performance gap.

CPU Memory
Kernel A

GPU Memory

Kernel B

Kernel C

Hit PageMiss Page

Kernel D

Kernel E

C1 C2 C3 C4 C5 C1 C2 C3 C4 C5 C1 C2 C3 C4 C5 C1 C2 C3 C4 C5 C1 C2 C3 C4 C5
P5
P4

...

P3
P2
P1

Time

(a) Execution process of the baseline
C1 C2 C3 C4 C5 C5 C4 C3 C2 C1 C1 C2 C3 C4 C5 C5 C4 C3 C2 C1 C1 C2 C3 C4 C5
P5
P4

...

P3
P2
P1

Time

(b) Execution process of DispatchSwitch
C1 C2 C3 C4 C5 C5 C4 C3 C2 C1 C1 C2 C3 C4 C5 C5 C4 C3 C2 C1 C1 C2 C3 C4 C5introduced in the previous section when both DispatchSwitch
and ReplacementSwitch (this combination is denoted as CPC),
are applied. As expected, Page 5 (P5) is evicted at the C2 of
Kernel B according to the reverse age-based LRU. Therefore,
all the remaining pages within the GPU memory are utilized
by Kernel C with no page faults at the boundary of kernels.
Fig.5(b) shows the page access pattern and page fault rateof FFT using CPC. We make two observations. First, theperformance of odd-round kernels is improved significantly,
leading to better overall performance than the case only using
DispatchSwitch shown in Fig.5(a). Second, more remaining
pages are reused resulting in fewer page faults at the boundaryof kernels. We conclude that CPC is effective at reducing pagefaults at the boundary of kernels via reusing the shared data.

P5
P4

V. M ETHODOLOGY

...

P3
P2
P1

Time

(c) Execution process of CPC

We extend the GPGPU-Sim v4.0.0 [24] in our experiments.
The configurations of the GPU system including both coresand memory are shown in TABLE I.

Fig. 8. Execution process of the microbenchmark (To simplify the analysis,
we make the following assumptions. 1) Only one CTA can run at the sametime. 2) Five kernels (A, B, C, D, E) access the same data and eachkernel consists of five CTAs (C1-C5) accessing only one data page (P1-P5,
respectively). 3) the GPU memory capacity is 3 pages.).

B. Page Replacement Policy Switching
As discussed in Section IV-A, odd-round kernels underutilize the shared data, as DispatchSwitch may mismatch withthe age-based LRU policy. The right part of · in Fig.6 shows
how age-based LRU policy works. To provide the support ofdemand paging, the GPU driver within the host maintains asingle list on recording migration order of pages from theCPU memory to the GPU memory. Although it cannot show
the access order of pages, it maintains the migration order withlow overhead when compared with ideal LRU [23]. Since our
CTA scheduler switches the execution order for each kernel,
traditional age-based LRU policy cannot cooperate with CTA
scheduling mechanism anymore. As shown in Fig.8(b), Page
3 (P3) is evicted instead of Page 5 (P5) at the C2 of Kernel
B according to the age-based LRU policy. However, P5 is theold page in Kernel B, which is preferred to be evicted.
To address this issue, we propose a page replacement
policy switching mechanism, called ReplacementSwitch, asshown in · of Fig.6. ReplacementSwitch introduces a newpage replacement policy, called reverse age-based LRU policy
shown in the left part of · in Fig.6. Different from age-based
LRU, the direction of both page eviction and page allocationis switched. Similar to DispatchSwitch, the GPU driver selects
the corresponding page replacement policy based on SWITCH.
When SWITCH equals 0, the GPU driver applies age-based
LRU as page replacement policy, otherwise reverse age-based
LRU is applied. At this time, CTA dispatching strategy andpage replacement policy change simultaneously to cooperate
for effective data reuse whenever a new kernel is launched.
Fig.8(c) shows the execution process of the microbenchmark

TABLE IC ONFIGURATIONS OF THE SIMULATED SYSTEMGPU Core Configurations
GPU Arch
GPU Cores
Private L1 Caches
Shared L2 Caches

NVIDIA RTX-2060 Turing-like
30 CUs @ 1.4GHz, 32 threads per warp,
4 GTO scheduler
64KB/CU, fully associative, LRU
Memory Side 256KB/DRAM Channel,
16-way associative, LRU

Memory System Configurations
DRAM
Unified Memory

GDDR6, 12-channel, FR-FCFS scheuler,
RCD=RP=20, RC=62, CL=WR=20
4KB page size, 20µs page fault latency,
16 GB/s PCIe bandwidth

Demand Paging and Oversubscription. We faithfully model
demand paging between the CPU memory and the GPU memory. Whenever data absent in the GPU memory is requested,
page faults are generated and the GPU driver resident in thehost serves this exception by sending the required pages to theGPU. If the GPU memory is full, pages in the memory shouldbe evicted based on the page replacement policy via the GPU
driver. To experiment under different memory oversubscription
degrees, the capacity of the GPU memory is configured to
fractions (from 75% to 95%) of each workload’s footprint.
Workloads. We select 12 applications from CUDA SDK [9],Rodinia [5], Parboil [6], Ispass [7] and Polybench [8] benchmark suites. The footprint of these workloads varies from
1MB to 96MB with an average of 18.5MB. Limited simulating
speed hinders us from emulating a larger footprint.
Evaluation Metrics. We use normalized IPC to evaluate theperformance of proposed mechanisms. To report the data reusesharedrate, we introduce a formula defined as Access
AccessT otal , where
Accessshared is the number of accesses requesting the interkernel shared pages and AccessT otal is the number of total

accesses. Page fault rate is used to illustrate the stability of thememory system when the GPU memory is oversubscribed.
VI. E VALUATION
We evaluate both DispatchSwitch (denoted as CPC-Disp)
and CPC via comparing them against four designs: 1) baseline2) ETC [10] 3) Oracle 4) an ideal baseline with unlimited
GPU memory. Since above designs are all orthogonal to ETC,
we also evaluate these designs when equipped with ETC. The
configurations of these designs are shown in TABLE II.
A. Performance

Normalized IPC to Baseline

Fig.9 shows the average IPC normalized to the baseline
with our proposed mechanisms. From the figure, we makefour observations. First, the state-of-the-art mechanism ETC
shows modest benefit with an average of 15% performance
improvement. As Section III states, the performance of applications with inter-kernel data sharing degrades significantly
once the GPU memory is oversubscribed. Therefore memory
compression hardly works. Second, CPC outperforms thebaseline and ETC by 90% and 65% respectively and obtains
a similar performance benefit to Oracle, indicating that CPC
can reuse the inter-kernel shared data effectively. Third, CPC
outperforms DispatchSwitch by 37.3%, illustrating that both
DispatchSwitch and ReplacementSwitch play important roles.
Last, all methods equipped with ETC improve the performancefurther, showing that CPC and ETC are orthogonal with eachother which can work together.
3.5
3
2.5
2
1.5
1
0.5
0

B. Page Fault Rate
We next investigate where such significant performance
improvement originates from through quantifying the pagefault rate as shown in Fig.11, from which we make threeobservations. First, in line with the performance results described in the previous section, we observe that CPC reduces
the page fault rate across all the workloads and there is anaverage reduction of 87.3% and up to 266.3% (SRADV2).
Second, for most of the applications, there is little page fault
rate reduction when ETC is applied since such applications
are insensitive to the memory size under the oversubscribed
memory as stated in Section III. Third, the page fault rateof BFS even under the baseline is low, indicating that there
remains little performance improvement space, consistent withthe results just described.
C. Data Reuse Rate
We further investigate where the page fault reduction iscoming from. We quantify the data reuse rate when 75% of the
footprint fits in the GPU memory (used in ETC [10]) as shownin Fig.12. From the figure, We make three observations. First,
little shared data is reused for the baseline for most of theapplications as shown in Fig.8(a). Second, CPC provides anaverage of 56.7% data reuse rate and up to 74.9% (SRADV2),
which is consistent with the page fault rate reduction stated in
the last section because more data to be reused leads to fewer
page faults. Third, when equipped with ETC, CPC can deliver
an average of 75.4% data reuse rate and up to 90% (SRADV2),
indicating that almost all the shared data is reused.
D. Sensitivity Analysis

Fig. 9. Overall performance normalized to the baseline.

Individual Workload Performance. Fig.10 shows the performance of CPC and the other techniques normalized to thebaseline across multiple workloads. We make three observations. First, performance across all of the workloads has been
improved with CPC compared to the baseline, especially for
SRADV2 (330%) and SPMV (292%). Second, for most of
the workloads, CPC outperforms ETC and DispatchSwitch
and obtains a similar performance to the oracle case. Third,
there is little performance improvement for some applications
like BFS, 3DCONV and FWT when CPC is applied because
little shared data between neighbored kernels leaves little
room for improvement. We conclude that CPC is effective at
reducing the performance impact of memory oversubscription
for applications with inter-kernel data reuse.

In this section, we measure the sensitivity of CPC’s effectiveness to the degree of memory oversubscription, the page
size and the page fault handling latency.
Memory Oversubscription Ratio. From Fig.3, we find thatdifferent memory oversubscription ratios have similar impacts
on the performance. Fig.13(a) shows the performance improvement of CPC when the memory oversubscription ratio
varies from 75% (i.e., only 75% of each workload’s working
set can fit in the GPU memory) to 100% (i.e., there is no
memory oversubscription). We observe that the performance
benefit CPC brings increases as the GPU memory becomeslarger because more data remained in the GPU memory canbe reused. We conclude that CPC can provide amounts ofperformance benefit under the oversubscribed GPU memory.
Page Size. Page size plays a critical role in the virtual memory
traffic observed on both CPUs and GPUs. Fig.13(b) shows
the average performance benefit of CPC when page sizechanges from 1KB to 32KB normalized to the baseline. We
observe that the performance improvement decreases as thepage size becomes larger. Applications access less pages as
the page becomes larger leading to less page faults and lesspage migrations when the GPU memory is oversubscribed.
Though there remains little improvement space to reduce thepage faults for applications with larger pages, results shows

TABLE IIC ONFIGURATIONS OF DIFFERENT DESIGNSDesignMemory Oversubscription

CTA Scheduling

Page Replacement

Memory Compression

Baseline

3

RR in ascending orderAge-based LRU

7

7

ETC

3

RR in ascending orderAge-based LRU

33

CPC-Disp3

DispatchSwitch

Age-based LRU

7

7

CPC

3

DispatchSwitch

ReplacementSwitch

77

Oracle3

RR in ascending order

Data accessed before hit

77

CPC-Disp + ETC

3

DispatchSwitch

Age-based LRU

33

CPC + ETC

3

DispatchSwitch

ReplacementSwitch

33

Oracle + ETC

3

RR in ascending order

Data accessed before hit

3

3

Ideal

7

RR in ascending orderAge-based LRU

7

7

Baseline
Normalized IPC toBaseline

Proactive Eviction

ETC

CPC-Disp

SRADV23DS

CPC

OracleCPC-Disp+ETC

CPC+ETC

Oracle+ETC

4
3
2
1
0

HOTSPOT

FFT

BFS

BLK

RED

SPMV

STENCILLPS

3DCONV

FWT

AverageFig. 10. Overall performance normalized to the baseline for different applications.

Page Fault Rate

Baseline

ETC

CPC-Disp

SRADV23DS

CPC

OracleCPC-Disp+ETC

CPC+ETC

Oracle+ETC

100%
80%
60%
40%
20%
0%

HOTSPOT

FFT

BFS

BLK

RED

SPMV

STENCILLPS

3DCONV

FWT

AverageFig. 11. Page fault rate for different applications.

Data Reuse Rate

Baseline

ETC

CPC-Disp

CPC

CPC-Disp+ETC

CPC+ETC

100%
80%
60%
40%
20%
0%

HOTSPOTFFTSRADV2

3DS

BFS

BLK

RED

SPMV

STENCILLPS

3DCONV

FWT

AverageFig. 12. Data reuse rate for different applications.

our mechanism is still successful in alleviating the impact ofmemory oversubscription.
Page Fault Latency. The page fault latency is critical to theperformance especially when the memory is oversubscribed.
To understand the impact of such latency, we perform sensitiveness studies. Fig.13(c) presents the average performance
improvement with CPC when the page fault latency is varied
from 20µs to 50 µs normalized to the configuration with 20µs
fault latency. We observe that CPC can always brings significant performance improvement with any page fault latency
because it focuses on reducing page faults and is insensitive

to the page fault latency, which shows the effectiveness of ourapproach in reducing the impact of memory oversubscription.
E. Hardware Overhead
We analyze the hardware overhead to support each component of CPC. To implement DispatchSwitch, the CTA scheduler needs extending with corresponding control logic withalmost no hardware overhead to support two CTA dispatchingstrategies. ReplacementSwitch is implemented in the GPU
driver with no hardware overhead. Besides, a single bit isadded to the command processor to indicate which strategy to

1
0

75%

85%

95%

100%

Oversubscription Ratio(a) Sensitivity of performance to memory oversubscription ratio

2.2
1.8
1.4
1

1K

2K

4K

8K

Page Size (Byte)

16K

32K

(b) Sensitivity of performance to page size

Normalized IPC to
20μs Fault Latency

2

Normalized IPC toBaseline

Normalized IPC toBaseline

3

1.1
1
0.9
0.8

20

30

40

50

Page Fault Latency (μs)(c) Sensitivity of performance to page fault latency

Fig. 13. Sensitivity studies for CPC

select for both DispatchSwitch and emphReplacementSwitch.
Overall, hardware overheads for our design are negligible.
VII. R ELATED WORKTo the best of our knowledge, this paper is the first to
reuse the inter-kernel shared data to improve the performance
under the oversubscribed memory. We survey related works
in GPU memory oversubscription, CTA scheduling, exploiting
data locality and replacement policies.
GPU Memory Oversubscription. Kim et al. [13] show thehigh cost of page faults and propose a scheme to increase thenumber of page faults handled together to amortize the fault
handling latency across multiple pages. Ganguly et al. [25]propose a programmer-agnostic scheme to decide whether tocopy the data from the CPU memory to the GPU memory
based on the classification of applications. Garcı́a-Flores [26]et al. propose a fine-grained data transferring scheme to makethe most use of limited bandwidth, reduce the transmission
delay and alleviate memory contention. Zheng [11] et al.
propose a scheme to hide the long latency by viewing the pagefault as the store operation. The above methods all focus on
how to reduce the impact of the page fault and none consider
the data reuse among kernels. These approaches are orthogonal
to our work.
CTA Scheduling. Several works explore inter-CTA locality
via scheduling the CTAs properly [15]–[19]. Li et al. [15]propose a software scheme that relocates CTAs which sharedata to the same SM to utilize the inter-CTA locality based on
the inherent scheduling policy of real GPU hardware. Wanget al. [16] deliver consecutive CTAs to the same cluster tocoalesce requests to the same data and reduce the redundant
traffic. Lee et al. [18] dispatch a group of two adjacent CTAs tothe same SM to improve the L1 performance. Chen et al. [19]modify the CTA scheduling policy to improve the inter-CTA
locality. None of these works explore the inter-kernel locality.
Huzaifa et al. [17] deploy several CTA scheduling policies toutilize inter-kernel locality within L1 cache, but they do notconsider how to utilize such locality within the GPU memory
to reduce the performance impact of memory oversubscription.
Exploiting Data Locality. Ang et al. [15] and Wang etal. [16] schedule a group of CTAs each time to utilize interCTA locality. Tabbakh et al. [27] propose a share-awarecache scheme that maintains private data in the L1 cache andshared data in the L2 cache. Vijaykumar et al. [28] proposea hardware and software cooperative scheme to utilize thedata locality marked explicitly. These techniques targets intrakernel locality but are orthogonal to our designs since wefocus on inter-kernel locality. Huzaifa et al. [17] utilize interkernel data locality within the L1 cache through several simple
schemes. Due to the small L1 cache size, it performs badly
for kernels with a large working set. Our research coordinates
both the CTA scheduling policy and the replacement policy to
exploit inter-kernel locality.
Replacement Policies. Lots of works research in replacement
policy for cache and memory in both CPU and GPU [29], [30].
LRU (Least Recent Used) [23] is widely used and performs
well in the cache system. While for global memory, it isunrealistic to implement LRU due to its expensive cost. Somevariants like NRU (Not Recently Used) [31] and CLOCK(Second Chance Replacement) [32] are commonly used. These
policies perform well for applications with strong temporal
locality while performing badly for those with thrashing accesspatterns. LFU (Least Frequently Used) [33] and DIP (Dynamic
Insertion Policy) [34] target workloads with large working set
and thrashing access patterns. Q. Yu et al. [35] propose ahierarchical page eviction scheme which selects proper pageeviction policy for each application detected at runtime. To
protect intra-warp locality, Koo el al. [36] propose accesspattern-aware cache management scheme, which bypasses
streaming accesses and protects frequently used data frombeing evicted. These policies except LRU do not match with
our design but can be employed to the cache system.
VIII. C ONCLUSION
Applications with inter-kernel data sharing suffer fromgreat performance loss under memory oversubscription. To
alleviate such performance degradation, we propose CPC,
an application-transparent framework that coordinates CTA
dispatching strategy and page replacement policy to effectively
reuse the shared data for reducing the performance impact
of memory oversubscription. CPC reduces the page fault rateby an average of 87.3% compared to the baseline, leading to
an average 90% and 65% performance improvement than thebaseline and the state-of-the-art memory management framework ETC, respectively. We conclude that CPC is an effective
low-cost framework to exploit inter-kernel data sharing.
ACKNOWLEDGMENT
We thank the anonymous reviewers from ICCAD 2021. This
work is supported by Research Project of NUDT ZK20-04.

R EFERENCES[1] K. Hazelwood, S. Bird, D. Brooks, S. Chintala, U. Diril, D. Dzhulgakov,
M. Fawzy, B. Jia, Y. Jia, A. Kalro, J. Law, K. Lee, J. Lu, P. Noordhuis,
M. Smelyanskiy, L. Xiong, X. Wang, and Ieee, Applied Machine
Learning at Facebook: A Datacenter Infrastructure Perspective, ser.
International Symposium on High-Performance Computer ArchitectureProceedings. Institute of Electrical and Electronics Engineers, 2018,
pp. 620–629.
[2] B. Tekin, S. N. Sinha, P. Fua, and Ieee, Real-Time Seamless Single
Shot 6D Object Pose Prediction, ser. IEEE Conference on Computer
Vision and Pattern Recognition. Institute of Electrical and Electronics
Engineers, 2018, pp. 292–301.
[3] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussiandenoiser: Residual learning of deep cnn for image denoising,” Ieee
Transactions on Image Processing, vol. 26, no. 7, pp. 3142–3155, 2017.
[4] T. C. Schroeder, “Peer-to-peer & unified virtual addressing,” in GPU
Technology Conference, NVIDIA, 2011.
[5] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S. Lee, andK. Skadron, “Rodinia: A benchmark suite for heterogeneous computing,”
in 2009 IEEE International Symposium on Workload Characterization(IISWC). Institute of Electrical and Electronics Engineers, 2009, pp.
44–54.
[6] J. A. Stratton, C. Rodrigues, I.-J. Sung, N. Obeid, L.-W. Chang,
N. Anssari, G. D. Liu, and W.-m. W. Hwu, “Parboil: A revised
benchmark suite for scientific and commercial throughput computing,”
Center for Reliable and High-Performance Computing, vol. 127, 2012.
[7] M. Giles, “Jacobi iteration for a laplace discretisation on a 3d structured
grid,” 2008.
[8] S. Grauer-Gray, L. Xu, R. Searles, S. Ayalasomayajula, and J. Cavazos,“Auto-tuning a high-level language targeted to gpu codes,” in 2012
innovative parallel computing (InPar). Ieee, 2012, pp. 1–10.
[9] N. Corp., “Nvidia cuda sdk code samples,” % urlhttp://developer.nvidia.com/.
[10] C. Li, R. Ausavarungnirun, C. J. Rossbach, Y. Zhang, O. Mutlu, Y. Guo,
J. Yang, and Acm, A Framework for Memory Oversubscription Management in Graphics Processing Units, ser. Twenty-Fourth International
Conference on Architectural Support for Programming Languages andOperating Systems. Association for Computing Machinery, 2019.
[11] T. Zheng, D. Nellans, A. Zulfiqar, M. Stephenson, and S. W. Keckler,“Towards high performance paged memory for gpus,” in 2016 IEEE
International Symposium on High Performance Computer Architecture(HPCA).
Institute of Electrical and Electronics Engineers, 2016,
Conference Proceedings, pp. 345–357.
[12] Q. Yu, B. Childers, L. Huang, C. Qian, H. Guo, and Z. Wang,“Coordinated page prefetch and eviction for memory oversubscription
management in gpus,” in 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Institute of Electrical andElectronics Engineers, 2020, pp. 472–482.
[13] H. Kim, J. Sim, P. Gera, R. Hadidi, H. Kim, and Acm, Batch-Aware
Unified Memory Management in GPUs for Irregular Workloads, ser.
Twenty-Fifth International Conference on Architectural Support for
Programming Languages and Operating Systems.
Association for
Computing Machinery, 2020.
[14] J. J. K. Park, Y. Park, and S. Mahlke, “Dynamic resource management
for efficient utilization of multitasking gpus,” in Proceedings of theTwenty-Second International Conference on Architectural Support for
Programming Languages and Operating Systems, ser. ASPLOS ’17.
New York, NY, USA: Association for Computing Machinery, 2017, p.
527–540.
[15] A. Li, S. L. Song, W. Liu, X. Liu, A. Kumar, and H. Corporaal,“Locality-aware cta clustering for modern gpus,” Operating Systems
Review, vol. 51, no. 2, pp. 297–311, 2017.
[16] L. Wang, X. Zhao, D. Kaeli, Z. Wang, and L. Eeckhout, “Intra-cluster
coalescing and distributed-block scheduling to reduce gpu noc pressure,”
Ieee Transactions on Computers, vol. 68, no. 7, pp. 1064–1076, 2019.
[17] M. Huzaifa, J. Alsop, A. Mahmoud, G. Salvador, M. D. Sinclair, andS. V. Adve, “Inter-kernel reuse-aware thread block scheduling,” ACM
Trans. Archit. Code Optim., vol. 17, no. 3, Aug. 2020.
[18] M. Lee, S. Song, J. Moon, J. Kim, W. Seo, Y. Cho, and S. Ryu,
“Improving gpgpu resource utilization through alternative thread block
scheduling,” in 2014 IEEE 20th International Symposium on High
Performance Computer Architecture (HPCA). Institute of Electricaland Electronics Engineers, 2014, pp. 260–271.

[19] L. Chen, H. Cheng, P. Wang, and C. Yang, “Improving gpgpu performance via cache locality aware thread block scheduling,” IEEE
Computer Architecture Letters, vol. 16, no. 2, pp. 127–131, 2017.
[20] A. Jog, O. Kayiran, N. Chidambaram Nachiappan, A. K. Mishra, M. T.
Kandemir, O. Mutlu, R. Iyer, and C. R. Das, “Owl: cooperative threadarray aware scheduling techniques for improving gpgpu performance,”
ACM SIGPLAN Notices, vol. 48, no. 4, pp. 395–406, 2013.
[21] J. Wang, N. Rubin, A. Sidelnik, and S. Yalamanchili, “Laperm: Locality
aware scheduler for dynamic parallelism on gpus,” in 2016 ACM/IEEE
43rd Annual International Symposium on Computer Architecture (ISCA).
IEEE, 2016, pp. 583–595.
[22] M. Knap and P. Czarnul, “Performance evaluation of unified memory
with prefetching and oversubscription for selected parallel cuda applications on nvidia pascal and volta gpus,” The Journal of Supercomputing,vol. 75, no. 11, pp. 7625–7645, 2019.
[23] L. A. Belady, “A study of replacement algorithms for a virtual-storage
computer,” IBM Systems journal, vol. 5, no. 2, pp. 78–101, 1966.
[24] M. Khairy, Z. Shen, T. M. Aamodt, and T. G. Rogers, “Accel-sim:
An extensible simulation framework for validated gpu modeling,” in2020 ACM/IEEE 47th Annual International Symposium on Computer
Architecture (ISCA). IEEE, 2020, pp. 473–486.
[25] D. Ganguly, Z. Zhang, J. Yang, and R. Melhem, “Adaptive pagemigration for irregular data-intensive applications under gpu memory
oversubscription,” in 2020 IEEE International Parallel and Distributed
Processing Symposium (IPDPS). IEEE, 2020, pp. 451–461.
[26] V. Garcia-Flores, E. Ayguadé, and A. J. Peña, “Efficient data sharing
on heterogeneous systems,” in 2017 46th International Conference on
Parallel Processing (ICPP). IEEE, 2017, pp. 121–130.
[27] A. Tabbakh, M. Annavaram, X. Qian, and Ieee, Power Efficient SharingAware GPU Data Management, ser. International Parallel and Distributed Processing Symposium IPDPS.
Institute of Electrical andElectronics Engineers, 2017, pp. 698–707.
[28] N. Vijaykumar, E. Ebrahimi, K. Hsieh, P. B. Gibbons, O. Mutlu, andIeee, The Locality Descriptor: A Holistic Cross-Layer Abstraction to
Express Data Locality in GPUs, ser. Conference Proceedings Annual
International Symposium on Computer Architecture.
Institute ofElectrical and Electronics Engineers, 2018, pp. 829–842.
[29] X. Zhao, M. Jahre, and L. Eeckhout, “Selective replication in memoryside gpu caches,” in 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2020, pp. 967–980.
[30] X. Zhao, A. Adileh, Z. Yu, Z. Wang, A. Jaleel, and L. Eeckhout,“Adaptive memory-side last-level gpu caching,” in 2019 ACM/IEEE
46th Annual International Symposium on Computer Architecture (ISCA),
2019, pp. 411–423.
[31] I. Sun Microsystems, “Ultrasparc t2 supplement to the ultrasparc architecture 2007.”[32] S. Bansal and D. S. Modha, “Car: Clock with adaptive replacement.” in
FAST, vol. 4, 2004, pp. 187–200.
[33] D. Lee, J. Choi, J.-H. Kim, S. H. Noh, S. L. Min, Y. Cho, and C. S.
Kim, “Lrfu: A spectrum of policies that subsumes the least recently
used and least frequently used policies,” IEEE Computer Architecture
Letters, vol. 50, no. 12, pp. 1352–1361, 2001.
[34] M. K. Qureshi, A. Jaleel, Y. N. Patt, S. C. Steely, and J. Emer, “Adaptive insertion policies for high performance caching,” ACM SIGARCH
Computer Architecture News, vol. 35, no. 2, pp. 381–391, 2007.
[35] Q. Yu, B. Childers, L. Huang, C. Qian, Z. Wang, and Ieee, HierarchicalPage Eviction Policy for Unified Memory in GPUs, ser. 2019 Ieee
International Symposium on Performance Analysis of Systems andSoftware. Institute of Electrical and Electronics Engineers, 2019.
[36] G. Koo, Y. Oh, W. W. Ro, and M. Annavaram, “Access patternaware cache management for improving data utilization in gpu,” in2017 ACM/IEEE 44th Annual International Symposium on Computer
Architecture (ISCA), 2017, Conference Proceedings, pp. 307–319.