日本ソフトウェア科学会第 34 回大会 (2017 年度) 講演論文集

Performance Characteristics of Linux for Java
Workloads Oversubscribing Memory
Takuya Nakaike

Yohei Ueda

Takanori Ueda

Moriyoshi Ohara

Cloud computing can reduce the cost of IT services by maximizing resource sharing. Memory oversubscription is a technology to increase the degree of resource sharing by allowing applications to request a memory
space beyond a given physical memory size. OS-based virtualization technologies such as Docker can increase it further compared to hypervisor-based virtualization technologies such as KVM, because the former
runs each application in a container to share the whole operating system with other applications while thelatter runs each application in a separate operating system. While the memory oversubscription of virtual
machines on a hypervisor has been studied extensively, that on an operating system hasn’t in a context ofmodern cloud services. This paper describes an analysis on how the current Linux virtual memory systemaﬀects the performance of Java applications that oversubscribe memory. Our major finding is that a major cause of performance degradation is page-cache reclamation operations which evict frequently-accessed
memory pages mapped to a file rather than swapping out rarely-accessed anonymous pages. This paper
also confirms that a Linux virtual memory system can be enhanced to improve the performance of Java
workloads that oversubscribe memory through an experiment which forces several file pages to persist in aphysical memory.

attractive for cloud service providers because mem-

1 Introduction

ory is one of the most expensive resources. It could,

Cloud computing can reduce the cost of IT ser-however, degrade the application performance dras-

vices by letting a massive number of applications

tically due to slow disk accesses to swap memory

share a given hardware resources such as CPU andpages.

memory in a data center. Cloud service providers

While there have been many studies on the mem-want to maximize the resource sharing to minimize

ory oversubscription for a hypervisor [23] [17] [24]the operational and capital cost for their data cen-[13] [15] [5] [21] [1] [16] [20] [27], there are few studies

ters without degrading the performance of hosted

on it for a modern operating system (OS), espe-

applications significantly.

cially in a context of interactions between an OS

Oversubscription is a concept to increase the de-and a language runtime. As container technolo-

gree of resource sharing. It allows an application

gies such as Docker [11] are becoming popular as

to request more resources than those available froma lightweight virtualization mechanism, the OS-

the actual hardware. Memory oversubscription islevel memory oversubscription is becoming impor-

Linux 上でメモリ超過した Java ワークロードの性能特性
This is an unrefereed paper. Copyrights belong to
the Author(s).
Takuya Nakaike, Yohei Ueda, Takanori Ueda,
Moriyoshi Ohara, IBM 東 京 基 礎 研 究 所, IBM Research - Tokyo.

tant since they rely on OS-level memory management.
Our work is to identify opportunities to improve
the performance of a Linux virtual memory system for Java workloads that oversubscribe mem-ory.

Many researchers have already shown thatcause less performance impacts.

swapping out memory pages belonging to a heap

Here are our contributions:area of a Java virtual machine (JVM) degrades the• Detailed behavior of a Linux virtual

performance significantly [2] [26] [4] [3] [25] [22]. Theirmemory system that runs Java work-common knowledge is that activities to touch ob-loads oversubscribing memory: We identi-

jects placed on swapped-out memory pages, espe-

fied how a Linux virtual memory system man-cially garbage collections (GCs), cause the perfor-

ages memory pages when Java workloads over-mance degradation, and thus they focus on resizing

subscribe memory.

the Java heap area to fit the total working set in• Root cause for performance degrada-a physical memory size. On the other hand, wetion in a memory-oversubscribed Linuxfocus on figuring out the detailed behavior of OS-

OS: We identified that page-cache reclama-level memory management to identify optimization

tion is too aggressive compared to anonymous-

opportunities for Linux.

page reclamation, resulting in the eviction of

We choose a Linux because it is one of the mostfrequently-accessed file pages instead of swap-

widely used OS for cloud services. We ran two rep-ping out rarely-accessed anonymous pages.

resentative Java workloads directly on Linux with-• Experiment to identify an optimizationout using containers to study the intrinsic charac-

opportunity for a Linux memory man-teristics of the Java workloads and the Linux virtual

agement system: We identified how a Linux

memory system.

virtual memory system can be enhanced to

We evaluated the performance characteristics of
two benchmarks when multiple Java virtual ma-

improve the performance of applications whenmemory is oversubscribed.

chines (JVMs) running on the same node oversub-

Sections 2 and 3 overview the memory manage-scribing the memory. While the performance ofment systems of a Linux OS and a JVM, respec-

both benchmarks degrades when memory pages aretively. Section 4 describes our experimental envi-

swapped out as expected, the root cause was notronment. Section 5 shows our experimental results.

obvious. In fact, we identified that the performance

Section 6 reviews related work. Section 7 concludes

degradation was due to page-cache (aka file cache)the paper.

reclamation to read files from a disk because theOS evicts those pages when the memory pressure

2 Linux Memory Management

is high. While disk accesses to a swap area oc-

In this section, we describe the memory manage-

curred, they were not the primary cause for thement system of a Linux OS [7] for the complete-degradation.

ness of the discussion on our analysis results. Sim-We conducted an experiment to identify whether

ilar to other modern operating systems, the Linux

there is a room to enhance the Linux memory man-

OS integrates the management of memory pages

agement system. We ran the benchmarks with pin-for the page cache (aka file cache) and those for thening several file pages to a physical memory. In

user process memory in the virtual memory sys-

this experiment, the file-page pinning improved thetem. This integration has multiple benefits for thethroughputs of two benchmarks significantly. Thisoverall system performance. For example, underresult indicates that we can enhance the Linux vir-certain circumstances when a user process exten-

tual memory system to select victim pages thatsively accesses a large file but does not require alarge amount of user process memory, the Linux OSquently accessed. When the OS needs to reclaim autilizes the free physical memory for page caches,

file page, it first checks whether or not the page con-which substantially reduces file-access latencies. In

tent has been changed since the content was copied

contrast, when a user process starts accessing afrom the file, and it writes back the content to thelarge amount of memory and stops accessing thefile if necessary. When the OS reclaims an anony-file, the OS reclaims the page caches for the file tomous page, it evicts the page content to a swap

increase its process memory. In this way, the OS

space on disk.

can avoid time-consuming page-swap operations.

Since the algorithm to select a page to be re-

A Linux OS manages the physical and virtual

claimed can aﬀect the system performance, the OS

memory through pages of two types: file and anony-needs to choose victim pages carefully. An impor-mous pages.

The file page is a memory pagetant factor is the balance of victim selection be-

mapped to a file and is kept on memory as a page

tween file and anonymous pages. Swappiness is acache for subsequent accesses to the file. A file canLinux kernel parameter that controls the balance.

be mapped to file pages implicitly when it is readA low swappiness value discourages swapping offrom a disk by using a read system call. In thisanonymous pages, while a high value encourages

case, the content of the file is first copied into file

it. When the swappiness value is 0, its minimum

pages and then copied to a buﬀer specified by thevalue, the OS never reclaims anonymous pages un-read system call. A file can be also mapped toless the system almost runs out of memory. When

file pages explicitly by using a mmap system call.

the swappiness value is 100, its maximum value,

Such a file is called memory-mapped file. In thison the other hand, the OS equally selects a victim

case, when a file is mapped to file pages, they arefrom files and anonymous pages.

not present a page cache. When the content of the

The victim selection algorithm in Linux is a vari-file is accessed, a page fault occurs to let a page-

ant of an approximate least recently used (LRU)

fault handler allocate one or more physical pages toalgorithm, called split LRU. The OS manages four

fill them with the content of the file, which is read

LRU lists of physical pages: active-file, inactive-file,from a disk.

active-anonymous, and inactive-anonymous pages.

The anonymous page is, on the other hand, a

When it allocates a new file or anonymous page, itmemory page, not mapped to a specific file, whichadds it at the head of the file-active or anonymous-

is typically allocated by a malloc function or an-

active list, respectively. It also moves pages be-other memory allocator in a language runtime. A

tween active and inactive lists based on their accessmalloc function internally uses a brk system call or

frequency, and selects a victim page from the taila mmap system call with a MAP ANONYMOUS

of either inactive list.

option. Both calls allocate one or more zero-filled

The OS identifies inactive pages in an active list

memory pages that are not associated to a specificby checking the reference bit of each page. Modern

file and can be used for any purposes.

processors have a hardware support to set the ref-

A Linux OS keeps track of free physical pages

erence bit of a page automatically when the page

to allocate one when a new page (file or anony-is accessed. The kernel also sets the reference bitmous) is requested. To maintain a certain num-when it handles the read and write system calls.

ber of free pages, the OS periodically checks the

The OS periodically scans the two active lists toallocated pages and reclaims those that are not fre-find the pages to be moved to an inactive list. Ifthe reference bit of a page is set, the OS considers

tional. One of the major diﬀerences between them

the page is recently accessed and moves it to theis whether or not the heap area is divided. In thehead of the list. If the reference bit is unset at thegenerational algorithm, the heap area is divided

tail of an active list, the OS moves the page to an

into nursery and tenure spaces. The nursery space

inactive list.

is used to allocate new objects, while the tenure

The OS also scans the two inactive lists. Whenspace is used to store long-lived objects, which are

the OS finds a page with the reference bit set in an

copied from the nursery space when their lifetime

inactive list, it moves the page back to an active

reaches a threshold.

list. When the OS finds a page with a reference

types of objects reduces the cost to detect live ob-

bit unset, it finally decides to evict the page. Eachjects, while it increases the frequency of GCs be-time after scanning a page, the OS clears its refer-cause it cannot use the entire heap area for object

ence bit.

allocation.

The swappiness value controls the scan frequencies of the file and anonymous lists.

The separation of the two

The generational algorithm involves two kinds ofWhen theGC operations: local and global. A local GC isswappiness value is 0, the OS never scans thetriggered to collect objects in a nursery space whenanonymous inactive list unless the system almost

it becomes full, while a global GC is triggered toruns out of memory. When the swappiness value iscollect objects both in nursery and tenure spaces

100, the OS scans the same number of pages in filewhen the tenure space becomes full. The IBM J9

and anonymous lists.

JVM assigns one fourth of a given heap area to thenursery space unless a command-line option speci-

3 Java Memory Management

fies its size explicitly.

In this section, we present an overview of the

The mark-and-sweep algorithm marks live ob-memory management system of the IBM J9 JVM,

jects by traversing objects from root references such

which we used for our experiments. The memory

as stack and static variables, and then discards (i.e.

space of a JVM consists of four subspaces [19] assweeps) unmarked objects. It traverses the entire

described below.

heap area per GC. It is also used for global GCs in

Heap Area: This area stores Java objects. Whenthe generational algorithm.

a JVM starts up, the number of physical pages al-

Internal Area: This area is used mainly as a work

located for this area is minimal. As a JVM cre-area for the JIT compiler. This area expands whenates Java objects, it requests physical pages to thethe compiler is active, and it does not disappear

OS, hence the physical memory size increases. The

even when the compiler becomes inactive because

size of this area can be specified explicitly by ait stores some persistent data such as profiling data.

command-line option (e.g. -Xmx<heap_size>).
When this area becomes full, a garbage collector discards dead (i.e., unreferenced) objects. Aftera GC, physical memory pages in this area can be
freed and returned to an OS when there is no ob-

Class Area: This area stores the metadata ofeach Java class such as the bytecode, the constantpool, and the method table, which is loaded froma Java class file. This area expands as the numberof loaded classes increases.

ject in them. Two major GC algorithms are usedin the IBM J9 JVM: mark-and-sweep and genera-

JIT (Code Cache) Area: This area stores thebinary code generated by the JIT compiler.

It

expands as the number of compiled methods increases. Because the IBM Testarossa JIT compiler
uses an adaptive compilation strategy, where thecompiler can recompile frequently-executed methods to apply high-level optimizations, the code that

Total

Nursery

Tenure1024 MB

256 MB768 MB

512 MB

128 MB384 MB

256 MB

64 MB

192 MB

Table 1: Java heap sizes for performance evaluation

has been optimized at low-optimization levels canbecome useless. The space for the useless code is

the disk-access activities for swap from the rest.

reclaimed for newly-generated code.

We used DB2 [12] and MongoDB [11] as the backend database for DayTrader and AcmeAir, respec-

4 Experimental Environment

tively. DayTrader sends an SQL query to the DB2

Figure 1 shows our experimental environ-server to access a database record, while AcmeAir

ment.

Our benchmarks are DayTrader [6] andsends a JSON message to the MongoDB server.

AcmeAir [9], which are Java-based Web applica-

The application server and the rest of the systemtions to emulate an online stock trading system andrun on a separate system - a separate OS on a sepa-

an online air ticket booking system, respectively.

rate logical partition (LPAR) running on a separate

We chose Web applications as benchmarks because

set of physical processors.

they are one of the most popular applications in

We measured the performance by increasing thecloud. Each benchmark processes a sequence of

number of JVMs (one JVM per application server)

requests from multiple clients by accessing a back-with a fixed heap size for each JVM. We tuned

end database system and then returns a response

the number of clients to maximize the throughput

to each client. The two applications have diﬀerent

when one JVM is running an application with aobject lifetime characteristics, which allow us to in-given heap size. When we increased the number ofvestigate the interplay between the GC behavior

JVMs, we divided the tuned number of clients toand the performance. DayTrader has more long-each JVM equally. Each measurement consists oflived objects than AcmeAir; this is because Day-

the following three steps.

Trader stores transaction states, such as a client

1. Warm up a given number of JVMs forstate, on the Java heap through an EJB frame-

five minutes to complete most of the JIT-

work, while AcmeAir stores them on the backend

compilation activities

database through a REST API that supports state-

2. Collect dead Java objects created in theless transactions.

warming-up phase by executing a System.gc()

We ran the two benchmarks on an IBM Web-

methodSphere Liberty application server [8] that runs on

3. Continue to run the JVMs concurrently for

an IBM Java runtime environment. Our operating

one hoursystem is Red Hat Enterprise Linux, and its kernel

Table 1 shows a set of heap size parameters thatversion is 3.10.0. Our software stack is built on a

we used. We used the default nursery space, which

4-core 5.5 GHz zEC12 processor [14]. We limited

is one fourth of the heap size as discussed in Section

the size of the physical memory to 2 GB intention-3.

ally to cause memory swapping. We used two hard

We used 100 as our default swappiness value,disks, one exclusively for a swap space, to separate

the maximum value, because it maximized theĂǇdƌĂĚĞƌ
ĐŵĞŝƌ
/DtĞď^ƉŚĞƌĞ>ŝďĞƌƚǇϴ͘ϱtĞďƌĞƋƵĞƐƚƐ͕ƌĞƐƉŽŶƐĞƐ

/D:ĂǀĂ^<sĞƌƐŝŽŶϴ>ŝŶƵǆϯ͘ϭϬ͘ϬϰͲĐŽƌĞϱ͘ϱ',ǌǌϭϮƉƌŽĐĞƐƐŽƌ͕Ϯ'ƉŚǇƐŝĐĂůŵĞŵŽƌǇ

WƌŝŵĂƌǇ
ĚŝƐŬ

^ǁĂƉ
ĚŝƐŬ

ůŝĞŶƚƐ

Ϯ
sϭϬ͘ϭ

DŽŶŐŽsϮ͘ϲ͘ϲ

>ŝŶƵǆϯ͘ϭϬ͘ϬtĞďƌĞƐƉŽŶƐĞƐ͕ ϰͲĐŽƌĞϱ͘ϱ',ǌǌϭϮƉƌŽĐĞƐƐŽƌ͕ƌĞƋƵĞƐƚƐ
ϭϲ'ƉŚǇƐŝĐĂůŵĞŵŽƌǇ

Figure 1: Experimental environment
throughput performance when memory swapping

each configuration four times to compute their av-

occurred. We used a generational GC algorithm

erage. We excluded configurations with low mem-as default except for the experiment in Section

ory pressure (i.e. the I/O-wait ratio is not higher

5.3. Section 5.4 shows the performance eﬀect forthan 1.0) because we want to focus on the workloada smaller swappiness value. We obtained the fol-characteristics when the memory pressure is high.

lowing statistics by using vmstat and iostat com-

We also drew a dotted line as an exponential trend

mands.

for each benchmark.

• I/O-wait ratio: We obtained this statistics
from the vmstat command directly.

A common characteristics of DayTrader andThis

AcmeAir is that the throughput degrades as thestatistics indicates the ratio of the CPU idle

I/O-wait ratio increases. For example, when we in-time when there is at least one outstanding I/O

creased the number of JVMs (256 MB heap) fromoperation.

10 to 12 for DayTrader, the I/O-wait ratio in-• Footprint: We calculated this statistics by using the following formula.
Footprint=

creased from 16% to 72% and the throughput degraded by 84%. Similarly, when we increased the

Physical memory size

number of JVMs (512 MB heap) from 8 to 10 for−

Free memory size

AcmeAir, the I/O-wait ratio increased from 1% to+

Swapped-out memory size

81% and the throughput degraded by 88%. Figure

A free memory size and a swapped-out memory

3 shows the relationship between the I/O-wait ratio

size are obtained from the vmstat command.

and the average response time. The response time

• Amount of disk accesses: We obtained thisincreases similarly as the I/O-wait ratio increases.

statistics from the iostat command.

We analyze the root cause of the increase in the
I/O-wait ratio in the next section.

5 Experimental Results
5. 1 Overall Performance of DayTraderand AcmeAir

Figure 4 shows the relationship between the footprint and the response time. Note that memory
swapping occurs when the footprint exceeds the

In this section, we look at the basic performance

2 GB physical memory size. It is interesting tocharacteristics of DayTrader and AcmeAir. Figure

see that the two benchmarks exhibit very diﬀer-

2 shows the relationship between the I/O-wait ratio

ent characteristics when the footprint exceeds theand the aggregated throughput. Each dot corre-

physical size. As shown in the square dots, thesponds to an experimental result with a configura-

response time of DayTrader is lower than 10 mil-tion defined by the number of JVMs and the heap

liseconds even when the footprint is around 4000

size for each JVM. We iterated the measurement for

MB. For example, the response time of DayTrader

$QH[SRQHQWLDOWUHQG OLQH




0%[FH 
VVT
HUWX SK
JX
RU
K 
7

0%[0%[0%[0%[0%[0%[0%[FH 
VVT
HUWX SK
JX
RU
K7 

0%[0%[0%[0%[








        

0%[        ,2ZDLWUDWLR ,2ZDLWUDWLR (b) AcmeAir(a) DayTrader

Figure 2: Relationship between the I/O-wait ratio and the aggregated throughput. Each dot corresponds
to an experimental configuration defined by the number of JVMs and the heap size. We excluded configurations with low memory pressure for ease of viewing. We drew a dotted line as an exponential trend.

$QH[SRQHQWLDOWUHQG OLQH
0%[FH 
VP
H 
P
LW 
HV
QR 
SV
H5 

0%[0%[0%[0%[0%ಹ

0%[        ,2ZDLWUDWLR (a) DayTrader




0%[FH 
VP
H P
WL 
HV
QR 
SV
H 
5

0%[0%[0%[0%[        ,2ZDLWUDWLR (b) AcmeAir

Figure 3: Relationship between the I/O-wait ratio and the response time. Each dot corresponds to an
experimental configuration defined by the number of JVMs and the heap size. We excluded configurations
with low memory pressure for ease of viewing. We drew a dotted line as an exponential trend.
running on four JVMs (1024MB heap) is 4.9 mil-on 10 JVMs (512 MB heap) is 130.9 milliseconds

liseconds when the footprint is 3992 MB. As shown

when the footprint is 3170 MB. These results in-in the circle dots, on the other hand, the response

dicate that DayTrader can have more swapped-out

time of AcmeAir is much longer than that of Day-

memory pages than AcmeAir without causing de-Trader even when the footprint is around 3000 MB.

structive performance degradation.

For example, the response time of AcmeAir running'D\7UDGHU

0%[

FH
V 
P
H
P
LW
HV
QR
SV 
H5

0%[$FPH$LU
0%[0%[0%[0[0%[

0%[0%[





0%[0%[0%[0%[)RRWSULQW 0%







Figure 4: Relationship between the footprint and the response time. Each dot corresponds to an experimental configuration defined by the number of JVMs and the heap size. All of the configuration in thisfigure causes memory swapping because the footprints exceed the 2 GB physical memory size.
5. 2 Deep Analysis of DayTraderthat of the swap-disk traﬃc. For example, whenIn this section, we discuss the Linux virtual-

DayTrader runs on 12 JVMs (256MB heap each),memory system algorithm evicts inappropriate

while the size of the swapped-out memory is 2.8GB

pages to degrade the throughput performance of- much larger than the size of the whole physi-

DayTrader when the memory is oversubscribed.

cal memory, the amount of the primary-disk traf-

More specifically, we identified that the algorithmfic (83KB per request) is much larger than thattends to evict memory pages for the page cacheof the swap-disk (17KB per request). Most of themore aggressively than anonymous memory pages,

primary-disk accesses (more than 90%) were read.

and that such evictions lead to a significant increase

To reduce the traﬃc of the primary disk, wein the amount of the disk traﬃc to read files. In

pinned file pages accessed during DayTrader runs.

fact, by pinning all pages for the page cache to dis-A smaps file under the Linux proc file system indi-able such evictions, we were able to improve thecated the following four types of memory-mapped

throughput performance significantly.

files are accessed during DayTrader runs.

To simplify our analysis on the disk traﬃc, we• Shared libraries in /usr/lib64

configured two disks: one for a primary disk, and• Shared libraries in a Java runtime system

the other for a swap disk.

• Class-cache files of a Java runtime system

The traﬃc for the

primary disk consists only of regular file accesses,
which fill a page cache with the file content, while• Files that WebSphere Liberty uses to manage
messagesthat for the swap disk consists only of swap oper-

As described in Section 2, a memory-mapped fileations for the process memory. Figure 5 (a) showsis accessed through file pages which are kept in a

the amount of disk accesses per request for the pri-

page cache. By pinning such pages, we can prevent

mary and swap disks when DayTrader runs on 10,

the virtual memory system from evicting them, and12, 14 JVMs. For all of the three experiments, thehence can avoid primary-disk accesses to read theamount of the primary-disk traﬃc is larger thanfile when the file is accessed later. To this end, wecreated a custom version of JVM, which parses atual page is present on the physical memory or issmaps file to identify the address of those file pages

swapped out. We invoked the correlation tool afterand pins them on the physical memory by using a

stopping all of the clients for DayTrader and be-

mlock system call.

fore terminating JVM processes which are needed

Figure 5 (b) shows the file-page pinning can maketo keep their page-map entries.

the traﬃc of the primary disk almost negligible

As shown in Figure 6, the Linux memory man-without significantly aﬀecting that of the swap disk.

agement system swaps out memory pages in the

When DayTrader runs on 12 JVMs, as a result, theother category (rightmost bars) to keep the pinned

total amount of disk accesses were reduced by morefiles pages in a physical memory when the number

than a factor of five, from 100KB per request toof JVMs is twelve. This other category includes

18KB per request. This result indicates that thevarious types of memory pages such as the memory

traﬃc for the primary disk mostly occurred to re-pages that are allocated outside of a Java heap areaclaim file pages, which had been evicted when thethrough Java native interface (JNI) calls. This in-memory pressure is high. This means that there isdicates that the file-page pinning helped the Linux

a significant room to enhance the page eviction al-

memory management system swap out such pages

gorithm in the Linux virtual-memory system to se-

as rarely-accessed pages because the destructive

lect victim pages that impact the performance less,performance degradation did not occur as shownsuch as anonymous pages rather than file pages forin Figure 5.

memory-mapped files in our experiments. Figure

When the number of JVMs is fourteen, the Linux5 (c) further shows the file-page pinning improved

memory management systems swaps out memory

the throughput significantly for all of the three con-pages in the tenure and nursery spaces to keep thefigurations we examined.

pinned file pages in a physical memory. Swapping

Finally, we analyze which anonymous memoryout the memory pages for the tenure space is a

pages are swapped out by the Linux memory man-good decision because they are rarely-accessed. In

agement system to keep files pages in a page cachefact, the destructive performance degradation did

when the file-page pinning is used. Figure 6 shows

not occur even when a large portion of the tenure

the breakdown of memory pages for DayTrader.

space is swapped from 12 JVMs. However, swap-

We obtained the sizes of file-pages in a page cache,ping out the memory pages in the nursery space,free pages, and swapped-out pages from a vmstat

which is accessed more frequently than the tenurecommand. We further broke down the swapped-

space, degraded the performance significantly be-out and physical memory pages into five parts de-cause of the increase in the amount of swap-disk

scribed in Section 3 respectively. To do it, we letaccesses.

each JVM generate a javacore file by sending a signal (kill -3) to the JVM process. The javacore

5. 3 Deep Analysis of AcmeAir

file includes the address range of each memory area

In this section, we discuss whether the enhance-

we discussed in Section 3. We implemented a tool

ment for the page-cache reclamation algorithm,

to correlate this information with page-map entries

which we mentioned in the previous section, is ef-

that are provided through the Linux proc file sys-

fective for AcmeAir.

tem. Each page-map entry corresponds to a virtual

Figure 7 (a) shows the amount of disk accesses

address and has a flag to indicate whether the vir-

per request for a primary disk and a swap disk for

3ULPDU\

6ZDS



%.
T
HU
UH 
SV
HV 
VH
FD 
N
,V'


3ULPDU\

6ZDS



%.
T
HU
UH 
SV
HV 
HVF
DN 
,V'
-90V-90V-90V-90V -90V -90V(b) Disk accesses per request with filepage pinning(c) Throughput improvement with filepage pinning-90V

0%KHDS

-90V

0%KHDS

0%KHDS(a) Disk accesses per request)LOHSDJHSLQQLQJFH 
VV
TH UW
SXK JX
URK
7 



-90V

2ULJLQDO

Figure 5: Eﬀect of file-page pinning for DayTrader. (a) the amount of disk traﬃc per request without
file-page pinning. (b) the amount of disk traﬃc per request with file-page pinning. (c) the throughput
with and without file-page pinning.


6ZDSSHGRXW% 
0V 
JHD 
S\ 
UR

P
H 
P
IR 
H 
L]6 

3K\VLFDOSDJHVDVVLJQHG

                                               ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ
7HQXUH

1XUVHU\-,7,QWHUQDO&ODVV

3DJHFDFKH)UHHSDJHV

2WKHUV

Figure 6: Breakdown of memory pages for DayTrader. We changed the number of JVMs with fixing theheap size to 256 MB.
AcmeAir. We choose the 512 MB heap for AcmeAir

that the file-page pinning reduces the traﬃc of the

because using the 256 MB heap kept the amountswap disk in addition to the traﬃc of the primaryof disk accesses small even when we ran fourteen

disk. We suppose that the reduction in swapped-

JVMs.

out memory pages for the nursery space brings thisWhen we ran 10 JVMs, the traﬃc of the primaryeﬀect. Figure 8 shows the breakdown of memory

disk was much larger than that of the swap disk.

pages for AcmeAir. As shown in this figure, theWhen we pinned file pages for the memory-mapped

file-page pinning reduces the swapped-out memory

files mentioned in the previous section, most of thepages for the nursery space when the number ofprimary-disk accesses disappeared and the perfor-

JVMs is ten. Since the nursery space is accessed

mance improved significantly as shown in Figure 7

more frequently than the tenure space, the reduc-(b) and (c). This indicates that the enhancementtion in the swapped-out memory pages for the nurs-

for the page-cache reclamation algorithm can be ef-

ery space can reduce the traﬃc of the swap disk.

fective for AcmeAir.
A diﬀerence between DayTrader and AcmeAir is

As shown in Figure 8, the Linux memory management system swaps out memory pages for theother category (rightmost bars) to keep the memory pages for the nursery space and file pages ina physical memory. This result is similar with theresult in DayTrader.
A counter intuitive result is that the file-page pinning reduces the traﬃc of the swap disk without reducing the swapped-out memory pages for the nurs-


6ZDSLQHVV6ZDSLQHVV

F 
HV
VTH UWX
SK
JX 
RU
K7


ery space when 12 JVMs run. We guess that this isa side eﬀect of our tool which correlates the information about the JVM memory areas with page-
1XPEHURI -90V

map entries given from the Linux proc file system.









'D\7UDGHU







$FPH$LU

As mentioned in the previous section, our correla-

Figure 9: Throughputs of two benchmarks when wetion tool works with keeping JVM processes alive

changed the number of JVMs with fixing the heap

because the page-map entries are removed if we ter-size to 256 MB. Two swappiness values are used.

minate the JVM processes. Therefore, our tool can

Reducing the swappiness value suppresses memory

swap out the memory pages that are resident in aswapping.

physical memory during a benchmark is running.
We consider that this side eﬀect causes swapping

eﬀect, such as excessive memory swapping, with theout the memory pages for the nursery space.

maximum swappiness value, while it lost opportunities to swap out rarely accessed memory pages

5. 4 Eﬀects of Swappinesswith the default swappiness value.

In this section, we show the impact of the swappiness value on the performance. Figure 9 shows

6 Related Workthe throughput of the two benchmarks when we

Memory ballooning is a technique that a hy-changed the number of JVMs with a fixed 256MB

pervisor uses to reduce the amount of accesses toheap size. We used two swappiness values: 30 (de-swapped-out memory pages for its managing vir-fault) and 100 (maximum). Thirty is the default

tual machines (VMs) [23] [15] [27]. When a hypervi-

value on our Linux OS.

sor lacks physical memory pages, it forces a guest

As shown in Figure 9, the default swappiness

OS to pin some physical memory pages in its VM

value causes a lower performance than the maxi-

and to pass the pages to other VMs. A benefit of

mum one. As described in Section 2, the lower thethis approach is to leave the memory reclamation

swappiness is, the more likely the OS selects a vic-

to guest OSes by assuming that each guest OS re-tim from file pages than anonymous pages. As aclaims memory more eﬃciently than the hypervisor

result, when we reduce the swappiness from 100 to

by using its own page access information. Because

30, the OS increases the amount of primary-disk ac-

our work focused on improving the OS-level mem-cesses by reclaiming memory from page caches evenory management, it is orthogonal to the ballooning

though the JVM has many rarely accessed pages inapproach.

the tenure space, which can be swapped out with aT. Salomie et al.

and M. R. Hines et al.

limited impact on the performance. For our bench-

have applied a memory ballooning technique for amarks, the Linux OS did not cause a negative side

JVM [5] [20]. This approach avoids swapping out

3ULPDU\

6ZDS



%.
T
HU
HUS 
V
HV 
VH
FD 
NV,
' 

3ULPDU\

6ZDS



%.
TH
UU 
HS
VH 
VV
HF
D 
NV,
' 


-90V-90V-90V0%KHDS

-90V-90V-90V-90V -90V
0%KHDS

0%KHDS(b) Disk accesses per request with filepage pinning(a) Disk accesses per request)LOHSDJHSLQQLQJ


HFV
VT
UH WX
SK JX
URK
7 



-90V

2ULJLQDO(c) Throughput improvement with filepage pinning

Figure 7: Eﬀects of pinning frequently-accessed file pages to a physical memory in AcmeAir. (a) shows
the original amount of disk accesses per request without file-page pinning. (b) shows the memory breakdown of JVMs without file-page pinning. (c) shows the amount of disk accesses per request with file-page
pinning.
% 
0V 
JHD 
S\ 
UR

P
H 
P
IR 
H
L]6 

6ZDSSHGRXW

3K\VLFDOSDJHVDVVLJQHG

                                               ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ ZRSLQ ZSLQ
7HQXUH

1XUVHU\-,7,QWHUQDO&ODVV

3DJHFDFKH)UHHSDJHV

2WKHUV

Figure 8: Breakdown of memory pages for AcmeAir. We changed the number of JVMs with fixing theheap size to 512 MB.
memory pages in a Java heap area by forcing aand VSwapper [1] keep track of I/O activities

JVM to pin some of them and to reclaim the pinnedof guest OSes to identify rarely-accessed memory

pages for a hypervisor. However, our experimental

pages for swapping-out targets.

results show that swapping out memory pages in a

reduces the disk-access overhead on swapped-out

Java heap area does not always cause substantial

memory pages by allowing a host machine to use

performance degradation. Swapping out memory

the memory areas of other host machines as its

pages of a tenure space could be another eﬀective

swap area through a network. Hypervisor exclu-

option to reclaim memory without much impact onsive cache [16], Tmem [17], and Mortar [13] are

the performance.

the hypervisor-level caches to store the memory

Overdriver [24]Many techniques have been used to reduce thepages that are swapped out by guest OSes. We

overhead to access the swapped-out memory pages

consider that the current Linux memory manage-

for guest OSes [17] [24] [13] [21] [1] [16]. CMM2 [21]ment system needs to incorporate these kinds oftechniques because our experimental results showa memory-oversubscribed environment. Our ma-that the page-cache reclamation algorithm is too

jor finding is that the page-cache reclamation on

aggressive.

a Linux virtual-memory system is a major cause

The eﬃciency of the memory management policyof performance degradation when Java workloads

in Linux is more important for workloads runningoversubscribe memory.

in Docker containers [10] since Docker is an OS-

pages are swapped out, accesses to the swapped-out

based virtualization technology that relies on thememory pages are not the primary reason for thehost operating system to manage memory pages.

performance degradation. The Linux virtual mem-

Nakazawa et al. already showed that swappinessory system tends to reclaim memory pages from thehas major performance impacts on container work-

page cache (i.e. file pages) more aggressively thanloads that oversubscribe memory especially whenthose from the user memory space (i.e. anonymous

the load is unbalanced [18]. Our future work is

pages). As a result, it frees a part of the page cache

to enhance the Linux memory management systemthat stores files to generate excessive disk accesses

to achieve a better trade-oﬀ between swapping andto re-read such files when the memory pressure is

page-cache reclamation for Docker containers.

high. We expect that a virtual memory system

Although many memory

Many researchers have evaluated the perfor-

can reduce this type of performance degradation

mance of Java applications when memory swap-

by using disk access information to select a victim

ping occurs [2] [26] [4] [3] [25] [22]. They have shown

page to be reclaimed. The thorough understandingthat memory swapping degrades the performanceof the interaction between the application frame-

substantially, and they have proposed techniques

work and the memory oversubscription resiliency

to resize the heap sizes of JVMs to fit the totalrequires studies on a larger number of applications

working set in a physical memory size. However,and is beyond the scope of this paper.

they have not analyzed which memory areas areswapped out and why the performance degrada-

Referencestion occurs. Although Grzegorczyk et al. [2] have[ 1 ] Amit, N., Tsafrir, D., and Schuster, A.: VSwapper: A Memory Swapper for Virtualized Environments, Proceedings of the 19th international conference on Architectural support for programming
languages and operating systems, 2014.
[ 2 ] Grzegorczyk, C., Soman, S., Krintz, C., andWolski, R.: Isla Vista heap sizing: Using feedback
to avoid paging, Proceedings of the International
Symposium on Code Generation and Optimization,
2007.
[ 3 ] Hertz, M., Feng, Y., and D, E.: Garbage collection without paging, Proceedings of the 2005
ACM SIGPLAN conference on Programming language design and implementation, 2005.
[ 4 ] Hertz, M., Kane, S., Keudel, E., Bai, T., Ding,
C., Gu, X., and Bard, J. E.: Waste not, want not:resource-based garbage collection in a shared environment, Proceedings of the ACM SIGPLAN International Symposium on Memory Management,
2011.
[ 5 ] Hines, M. R., Gordon, A., Silva, M., Silva,
D. D., Ryu, K., and Ben-Yehuda, M.: Applications
Know Best: Performance-Driven Memory Overcom-

pointed out that the tenure space is swapped out
first under a high memory pressure environment,
they have not shown any data as evidence. We are
the first to clarify the swapped-out memory areasof Java applications and the root cause of the performance degradation.

7 Conclusion
In this paper, we characterized the performanceof two benchmarks that represent two diﬀerent
Java-based Web application frameworks on a Linux
OS: DayTrader based on a stateful EJB framework and AcmeAir based on a stateless REST API.
We investigated how we should enhance a virtualmemory system to improve the performance inContainers in the Case of Extreme Memory Overmit with Ginkgo, Proceedings of the 2011 IEEE
commitment, Proceedings of the 10th IEEE InterThird International Conference on Cloud Computnational Conference on Cloud Computing, 2017.
ing Technology and Science, 2011.
[ 6 ] http://geronimo.apache.org/GMOxDOC22/daytrader-[19] Ogata, K., Mikurube, D., Kawachiya, K., Trent,
S., and Onodera, T.: A Study of Java’s Nona-more-complex application.html: Daytrader - a
Java Memory, Proceedings of the ACM internamore complex application.
tional conference on Object oriented programming[ 7 ] http://linux mm.org/PageReplacementDesign:
systems languages and applications, 2010.
Page Replacement Design.
[20] Salomie, T., Alonso, G., Roscoe, T., and Elphin[ 8 ] https://developer.ibm.com/wasdev/websphere libstone, K.: Application level ballooning for eﬃcient
erty/: About WAS Liberty.
server consolidation, Proceedings of the 8th ACM[ 9 ] https://github.com/acmeair/acmeair: A Java
European Conference on Computer Systems, 2013.
implementation of the Acme Air Sample Applica[21] Schwidefsky, M., Franke, H., Mansell, R., Raj,
tion.
H., Osisek, D., and Choi, J.: Collaborative memory[10] https://www.docker.com/: docker.
management in hosted Linux environments, Ottawa[11] https://www.mongodb.org/: MongoDB for GILinux Symposium, 2006.
ANT Ideas.
[12] http://www 01.ibm.com/software/data/db2/: IBM [22] Tay, Y. C., Zong, X., and He, X.: An equationbased heap sizing rule, Performance Evaluation,
DB2 database software.
2013.
[13] Hwang, J., Uppal, A., Wood, T., and Huang, H.:[23] Waldspurger, C. A.: Memory resource manMortar: Filling the Gaps in Data Center Memory,
agement in VMware ESX server, Proceedings ofProceedings of the 10th ACM SIGPLAN/SIGOPSUSENIX OSDI, 2002.
international conference on Virtual execution envi[24] Williams, D., Jamjoom, H., Liu, Y., and Weathronments, 2014.
erspoon, H.: Overdriver: Handling Memory Over[14] Jacobi, C., Slegel, T., and Greiner, D.: Transload in an Oversubscribed Cloud, Proceedings of theactional Memory Architecture and Implementation
7th ACM SIGPLAN/SIGOPS international conferfor IBM System Z, Proceedings of the 2012 45thence on Virtual execution environments, 2011.
Annual IEEE/ACM Interna-tional Symposium on[25] Yang, T., Berger, E. D., Kaplan, S. F., andMicroarchitecture, 2012.
Moss, J. E. B.: Cramm: Virtual memory support[15] Kim, J., Fedorov, V., Gratz, P. V., and Reddy,
for garbage-collected applications, Proceedings of
A. L. N.: Dynamic Memory Pressure Aware Balthe 7th Symposium on Operating Systems Design
looning, Proceedings of the 2015 International Symand Implementation, 2006.
posium on Memory Systems, 2015.
[26] Zhang, C., Kelsey, K., X. Shen, C. D., Hertz, M.,[16] Lu, P. and Shen, K.: Virtual machine memory
and Ogihara, M.: Program-level adaptive memory
access tracing with hypervisor exclusive cache, Promanagement, Proceedings of the 5th International
ceedings of the USENIX Annual Technical ConferSymposium on Memory Management, 2006.
ence, 2007.
[27] Zhao, W. and Wang, Z.: Dynamic memory bal[17] Magenheimer, D., Mason, C., McCracken, D.,ancing for virtual machines, Proceedings of the 2009
and Hackel, K.: Transcendent Memory and Linux.
ACM SIGPLAN/SIGOPS international conference[18] Nakazawa, R., Ogata, K., Seelam, S., and Onon Virtual execution environments, 2009.
odera, T.: Taming Performance Degradation of