Internet
Infrastructure
Review
Feb.2020

Vol.

45

Periodic Observation Report

Internet Trends as Seenfrom IIJ Infrastructure—2019
Focused Research

Acquiring Forensic Memory Images
on LinuxInternet Inf rast ruc ture Review
February 2020 Vol.45

Executive Summary ........................ ................................................ 3
1. Periodic Observation Report .......... ................................................ 4
Topic 1 BGP / Number of Routes . ................ ................................................ 4
Topic 2 DNS Query Analysis . .................................................................... 5
Topic 3 IPv6 . . . . ................................. ................................................ 7

Topic 4 Mobile FLET’S and Natural Disasters . ................................................... 11
Topic 5 The Transition of the Bandwidth of IIJ Backbone Circuits . ............................... 14

2. Focused Research ...................................................................... 16
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9

2

Linux Memory Dump Toolsn .............................................................. 16
What is LiME? ............................................................................ 16
Compiling LiME ............................ ............................................... 16
Dumping Memory to an External Drive .................................................... 17
Dumping Memory via the Network ......... ............................................... 18
What is Crash? ............................ ............................................... 19
Analyzing the Memory Image ............................................................. 20
Tips for Acquiring Memory Images ........................................................ 21
Volatility 3 . ............................... ............................................... 22© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

Executive Summary

Executive Summary

Here is the final IIR for 2019. Japan suffered many natural disasters in 2019, not least those associated withheavy rainfall events. The heavy downpours in northern Kyushu in August, Typhoon Faxai, which caused
widespread damage centering on Kanto in September, and Typhoon Hagibis remain fresh in memory. The
term global warming tends to bring to mind rising atmospheric temperatures, but sexperts tell us that risinglevels of greenhouse gases in the atmosphere will alter the overall state of the atmosphere, causing rainfall
patterns to gradually change.
Companies in the information and communications industry are also taking action by switching to energy-saving equipment and pursuing energy efficiency in the datacenter, for instance. Against this backdrop, it wasannounced that the prototype of RIKEN’s Fugaku supercomputer had taken the top global spot in the Green500 ranking of the world’s most energy-efficient computers.
Information and communications plays a diverse role in building a sustainable world, including through theuse of energy-saving equipment and the use of information and communications technology to reduce energy
consumption. IIJ will also continue to develop technology with that in mind.
The IIR introduces the wide range of technology that IIJ researches and develops, comprising periodic observation reports that provide an outline of various data IIJ obtains through the daily operation of services, aswell as focused research examining specific areas of technology.
The periodic observation report in Chapter 1 presents the 2019 edition of our rundown of Internet trends
as viewed from IIJ infrastructure. The report covers data on the number of IPv4 routes on the Internet, ananalysis of DNS queries from the full resolver IIJ provides to users, IPv6 usage on the IIJ backbone, traffic
on mobile networks and the FLET’S network around the time of natural disasters, and the history of the IIJ
backbone.
For the first time, we observed a decline in the number of unique IPv4 addresses in advertised routes, andthis will bear watching ahead. Our observations also confirm that the use of IPv6 continues to rise steadily,
as evidenced by increasing traffic volumes and increasing use among many service operators. And our analysis of traffic around October 12, when Typhoon Hagibis passed through Japan, showed a clear difference
from the usual usage patterns.
The focused research report in Chapter 2 explains Linux memory imaging tools that can be used with theVolatility framework, which we use to analyze memory images when performing incident response and forensics. The report looks at LiME and crash, two tools that can be used to acquire Linux memory images, anddescribes how to acquire memory images in a way that has minimal impact on disk forensics.
Through activities such as these, IIJ strives to improve and develop its services on a daily basis while maintaining the stability of the Internet. We will continue to provide a variety of services and solutions that ourcustomers can take full advantage of as infrastructure for their corporate activities.

Junichi Shimagami
Mr. Shimagami is a Senior Executive Officer and the CTO of IIJ. His interest in the Internet led to him joining IIJ in
September 1996. After engaging in the design and construction of the A-Bone Asia region network spearheaded by IIJ,
as well as IIJ’s backbone network, he was put in charge of IIJ network services. Since 2015, he has been responsible
for network, cloud, and security technology across the board as CTO. In April 2017, he became chairman of theTelecom Services Association of Japan MVNO Council.

© Internet Initiative Japan Inc.

3

1. Periodic Observation Report

Internet Trends as Seen from IIJ Infrastructure—2019
To provide Internet services, IIJ operates some of the largest

Topic 1

network and server infrastructure in Japan. Here, we exam-

BGP / Number of Routes

ine and discuss current Internet trends based on information
We start by looking at IPv4 full-route information advertised byobtained through the operation of this infrastructure.

our network to other organizations (Table 1). This time around,We cover the topics of network routing information, DNS

we also show the number of unique IPv4 addresses contained

query information, and IPv6 usage, as well as the impact ofin the IPv4 full-route information (Table 2). During the pastnatural disasters on mobile and FLET’S connection services.

year, the maximum size of IPv4 addresses allocated by APNICWe also report on the current state of the backbone network(and JPNIC) fell to /23 (512 addresses).

that supports the bulk of IIJ’s traffic.
The total number of routes, while seeing a slightly smaller
increase than in the previous year, now exceeds 760,000.
Together, the /22, /23, and /24 prefixes account for 80.1% of
all routes. Meanwhile, the number of unique IPv4 addresses,
although accounting for less than 1% of the total, fell for the
first time in the past nine years. Whether this is a temporary effect, perhaps due to the removal of unauthorized routes using
RPKI, or the first sign that the IPv4 Internet is shrinking issomething that will bear close watching ahead.

Table 1: Number of Routes by Prefix Length for Full IPv4 Routes
/8

/9

/10

/11

/12

/13

/14

/15

/16

/17

/18

/19

/20

/21

/22

/23

/24

total

Sep. 2010 20

10

25

67

198

409

718

1308

11225

5389

9225

18532

23267

23380

30451

29811

170701324736

Sep. 2011 19

12

27

81

233

457

794

1407

11909

5907

9885

19515

26476

26588

35515

34061

190276363162

Sep. 2012 19

14

29

84

236

471

838

1526

12334

6349

10710

20927

30049

31793

42007

39517

219343416246

Sep. 2013 16

11

30

93

250

480

903

1613

12748

6652

10971

22588

32202

34900

48915

42440

244822459634

Sep. 2014 16

12

30

90

261

500

983

1702

13009

7013

11659

24527

35175

37560

54065

47372

268660502634

Sep. 2015 18

13

36

96

261

500

999

1731

12863

7190

12317

25485

35904

38572

60900

52904

301381551170Sep. 2016 16

13

36

101

267

515

1050

1767

13106

7782

12917

25229

38459

40066

67270

58965

335884603443

Sep. 2017 15

13

36

104

284

552

1047

1861

13391

7619

13385

24672

38704

41630

78779

64549

367474654115

Sep. 2018 14

11

36

99

292

567

1094

1891

13325

7906

13771

25307

39408

45578

88476

72030

400488710293

Sep. 2019 10

11

37

98

288

573

1142

1914

13243

7999

13730

25531

40128

47248

95983

77581

438926764442

Date

Table 2: Total Number of Unique IPv4 Addresses in Full IPv4 Routes

4

Date

No. of IPv4 addresses

Sep. 2010

2,277,265,152

Sep. 2011

2,470,856,448

Sep. 2012

2,588,775,936

Sep. 2013

2,638,256,384

Sep. 2014

2,705,751,040

Sep. 2015

2,791,345,920

Sep. 2016

2,824,538,880

Sep. 2017

2,852,547,328

Sep. 2018

2,855,087,616

Sep. 2019

2,834,175,488© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

1. Periodic Observation Report

Next we take a look at IPv6 full-route data (Table 3). The

Topic 2

total number of routes increased by more than it did in the

DNS Query Analysis

previous year and now exceeds 70,000. However, route
advertisements for blocks that have been split into smaller

IIJ provides a full resolver to enable DNS name resolu-

fragments still constitute the majority, with the top threetion for its users. In this section, we discuss the state ofyear-on-year growth rates coming in the /30–/31, /41–/43,

name resolution, and analyze and reflect upon data fromand /45–/47 prefix ranges, sizes that are not all that com-

servers provided mainly for consumer services, based on a

monly allocated/assigned.

day’s worth of full resolver observational data obtained on
October 25, 2019.

Lastly, let’s also take a look at IPv4/IPv6 full-route Origin
AS figures (Table 4). Both the decrease in 16-bit Origin

The full resolver starts by looking at the IP address of an

Autonomous System Numbers (ASNs) and the increaseauthoritative name server for the root zone (the highest

in 32-bit-only Origin ASNs were the largest seen in thelevel zone), and based on the information available frompast nine years. And for the first time, IPv6-enabled ASNs,

that server, it then goes through other authoritative namewhich advertise IPv6 routes, accounted for over a quarter

serves to find the records it needs. Queries repeatedly sentof the total. It was predicted that RIPE NCC’s IPv4 address

to the full resolver can result in increased load and delays,pool would run out before 2020 arrived, so it will be inter-so the information obtained is cached, and when the sameesting to see what has happened when we next report onquery is received again, the response is sent from the cache.

the data.

Recently, DNS-related functions are also implemented on

Table 3: Number of Routes by Prefix Length for Full IPv6 Routes
Date

/29

/30-/31/32/33-/39/40/41-/43/44/45-/47

/48

total

Sep. 2010 38/16-/28

3

10

2023

33

2

9

4

17

436

2575

Sep. 2011 68

13

22

3530

406

248

45

87

95

2356

6870

Sep. 2012 102

45

34

4448

757

445

103

246

168

3706

10054

Sep. 2013 117

256

92

5249

1067

660

119

474

266

5442

13742

Sep. 2014 134

481

133

6025

1447

825

248

709

592

7949

18543

Sep. 2015 142

771

168

6846

1808

1150

386

990

648

10570

23479

Sep. 2016 153

1294

216

8110

3092

1445

371

1492

1006

14291

31470

Sep. 2017 158

1757

256

9089

3588

2117

580

1999

1983

18347

39874

Sep. 2018 168

2279

328

10897

4828

2940

906

4015

2270

24616

53247

Sep. 2019 192

2671

606

12664

6914

3870

1566

4590

4165

34224

71462

Table 4: IPv4/IPv6 Full-Route Origin AS Numbers
16-bit（1-64495）

ASN

32-bit only（131072-4199999999）
total

（IPv6
-enabled）

IPv4+IPv6

IPv4 only

IPv6 only

total

（IPv6
-enabled）

67

34549

（ 6.2%）

17

478

3

498

（ 4.0%）

115

37129

（11.8%）

90

1278

13

1381

（ 7.5%）

33434

125

39026

（14.3%）

264

2565

17

2846

（ 9.9%）

6579

34108

131

40818

（16.4%）

496

3390

28

3914

（13.4%）

Sep. 2014

7405

34555

128

42088

（17.9%）

868

4749

55

5672

（16.3%）

Sep. 2015

8228

34544

137

42909

（19.5%）

1424

6801

78

8303

（18.1%）

Sep. 2016

9116

33555

158

42829

（21.7%）

2406

9391

146

11943

（21.4%）

Sep. 2017

9603

32731

181

42515

（23.0%）

3214

12379

207

15800

（21.7%）

Sep. 2018

10199

31960

176

42335

（24.5%）

4379

14874

308

19561

（24.0%）

Sep. 2019

10642

31164

206

42012

（25.8%）

5790

17409

432

23631

（26.3%）

Advertised
IPv4+IPv6
route

IPv4 only

IPv6 only

Sep. 2010

2083

32399

Sep. 2011

4258

32756

Sep. 2012

5467

Sep. 2013© Internet Initiative Japan Inc.

5

devices that lie on route paths, such as broadband routers

the same (no major differences) in the middle of the night,

and firewalls, and these devices are sometimes involved inwhereas IPv6 queries per IP address show a tendency to rise

relaying DNS queries and applying control policies.

when people are active during the daytime and particularly
after 8:00 p.m. This suggests that the computing environ-

ISPs notify users of the IP address of full resolvers via various

ment needed to allow use of IPv6 in the home is coming into

protocols, including PPP, DHCP, RA, and PCO, depending onplace. And looking at total query count, both the number ofthe connection type, and they enable users to automaticallysource IPs and the number of actual queries are higher for

configure which full resolver to use for name resolution on

IPv6 than for IPv4. The number of IPv6-based queries is on

their devices. ISPs can notify users of multiple full resolvers,

the rise, accounting for around 60% of the total, up by more

and users can specify which full resolver to use, and add

than 4 points from 55% in the previous year.

full resolvers, by altering settings in their OS, browser, orelsewhere. When more than one full resolver is configured

Recent years have seen a tendency for queries to rise briefly

on a device, which one ends up being used depends on theat certain round-number times, such as on the hour marks

device’s implementation or the application, so any given fullin the morning. The number of query sources also increases,

resolver is not aware of how many queries a user is sending

which tells us that this is possibly due to tasks scheduled

in total. When running full resolvers, therefore, this meanson user devices and increases in automated network accessthat you need to keep track of query trends and always keepthat occur when devices are activated by, for example, ansome processing power in reserve.

alarm clock function. In the previous year, we noted an increase in queries 14 seconds before every hour mark, andObservational data on the full resolver provided by IIJ show

the 2019 results also show another increase 10 seconds be-

fluctuations in user query volume throughout the day, withfore every hour. The increase in queries that occurs on thevolume hitting a daily trough of about 0.05 queries/sec perhour tapers off gradually, but with the spikes that occurs

source IP address at around 4:30 a.m., and a peak of about

14 and 10 seconds before the hour, query volume immedi-

0.23 queries/sec per source IP address at around 12:30 p.m.

ately returns to about where it had been. Hence, because a

These values are almost the same as last year, with a slightlarge number of devices are sending queries in almost per-increase in the peak of 0.01 points. Broken down by proto-

fect sync, it seems like some sort of lightweight, quicklycol (IPv4 and IPv6), the trends in query volume are virtuallycompleted tasks are being executed.

OTHER

OTHER

0.13%

SRV

0.05%

SRV

0.4%
0.18%

PTR

1.26%

PTR

0.21%

AAAA

14.15%

AAAA

44.88%

A

84.01%

Figure 1: IPv4-based Queries from Clients

6

A

Figure 2: IPv6-based Queries from Clients© Internet Initiative Japan Inc.

54.73%

Feb.2020

Vol.

45

1. Periodic Observation Report

For example, there are mechanisms for completing basic

Topic 3tasks, such as connectivity tests or time synchronization, be-IPv6fore bringing a device fully out of sleep mode, and we posit
In this section, we report on the volume of IPv6 traffic onthat the queries used for these tasks are behind the spikes.

the IIJ backbone, the sources of that traffic, and the main
Looking at the query record types, most are A records that

protocols used. And for a new perspective on IPv6 in thequery the IPv4 address corresponding to the host name andmobile space, we look at the state of IPv6 connections ac-

AAAA records that query IPv6 addresses. The trends in A

cording to differences in device OS (Apple iOS / Android).

and AAAA queries differ by IP protocol, with more AAAA
record queries being seen for IPv6-based queries. Of IPv4-■ Traffic

based queries, around 84% are A record queries and 14%

As before, we again present IPv4 and IPv6 traffic measured

AAAA record queries (Figure 1). With IPv6-based queries,

using IIJ backbone routers at core POPs (points of presence—

meanwhile, AAAA record queries account for a higher share

Tokyo, Osaka, Nagoya), shown in Figure 3. The data spanof the total, with around 54% being A record and 44%

the year from October 1, 2018 to September 30, 2019.

being AAAA record queries (Figure 2). Compared with theprevious year, the levels are virtually the same for IPv6,

Over the year, IPv4 traffic increased by around 8% while

while for IPv4, we observe a drop of 3 percentage points or

IPv6 traffic rose by around 85%. IPv6 accounts for aroundso in A record queries and a rise of 3 points or so in AAAA

10% of overall traffic (Figure 4), a 4-point increase fromrecord queries.

around 6% last year. Further, IPv6 traffic hit a peak of

Linear total (IPv6)
Linear total (IPv4)
IPv6

9
30
Se

g.
Au

p.

19

,2

01

9
,2

01
,2
Ju

l.

31

30
n.
Ju

01

9

01
9
,2

01
9
31
,2
M

M

Ap

ar

r.

ay

30

.3
1,

,2

20
1

01
9

9

01
9
8,
2
b.
2
Fe

D

Ja

ec

n.

.3

31

1,

,2

20

01
9

18

01
8
,2
30
ov
.
N

O

Se

ct

p.

.3

30

1,
2

,2

01

01
8

8

IPv4
（Date）

Figure 3: IPv4/IPv6 Traffic Measured via IIJ Backbone Routers at Core Points of Presence (Tokyo, Osaka, and Nagoya)
（%）

14
12
10
8
6
4
2

9

（Date）

Se

p.

30

,2

01

9
01
Au

g.

19

,2

01
,2
31
l.
Ju

0,
.3
Ju
n

M
ay

9

20
19

9
31
,2

20
1
Ap

r.

30
,

20
.3
1,
M
ar

01

9

19

9
20
1
8,
b.
2
Fe

1,
.3
Ja
n

D

ec

.3

1,

20

20
19

18

01
8
30
,2
N

ov
.

31
ct
.
O

Se

p.

30

,2

,2
01

01

8

8

0

Figure 4: IPv6 Traffic as a Proportion of Total© Internet Initiative Japan Inc.

7

around 12% of total, and it continues to increase steadily as

time. This reflects the increasing use of IPv6 among manya proportion of overall traffic.

operators as well as an increase in IIJ’s IPv6 traffic due to
IPv6 being enabled on video streaming services that use

Figure 5 plots the data for the same period on a log scale. It

JOCDN’s platform (JOCDN is an IIJ affiliate that provides

is evident that IPv6 traffic is growing steadily while growtha video streaming platform).

in IPv4 traffic is slowing.
■ Protocols Used

■ Traffic Source Organization (BGP AS)

Figure 8 plots IPv6 traffic according to protocol number

Next, Figures 6 and 7 show the top annual average IPv6 and(Next Header) and source port number, and Figure 9 plots

IPv4 traffic source organizations (BGP AS Number) for the

IPv4 traffic according to protocol number and source port

year from October 2018 through September 2019.

number (for the week starting September 30, 2019).

Company A retains the top spot, but the traffic volume gap

In the IPv6 space, TCP 80 (HTTP) moved from No. 3 last

between it and No. 2 downward has narrowed further, andtime to No. 2, and UUD 443 (QUIC) came in at No. 3. This

its share of the pie is only about 60% of what it was last

mirrors the IPv4 ranking, so we can now say that IPv6

Linear total (IPv6)
Linear total (IPv4)
IPv6

9
Se

p.

30

,2

01

9
,2
19
g.
Au

31

,2

01

01

9

9

l.
Ju

30
,

20
1

01
9
M

Ju
n.

,2
ay

31

30
,2
Ap
r.

ar
.
M

01
9

31
,2
01
9

01
9
,2
28
b.
Fe

Ja
n.

1,
.3
ec
D

31
,

20

20
1

9

18

18
20
0,
.3
ov
N

O

Se

ct

p.

.3

30

1,

,2

20

01

18

8

IPv4
（Date）

Figure 5: IPv4/IPv6 Traffic Measured via IIJ Backbone Routers at Core Points of Presence (Tokyo, Osaka, and Nagoya)—Log ScaleIIJ（IPv6）

Company A

Company A

IIJ

Company F
Company Q
Company B

Company R

Company C
Other（IPv6）

Company D

Company P

Company E

Company J
Company T

Company O

Company F

Company N

Company G

Company Z

Company V

Company M

Company H

Company Y

Company W

Company L

Company I

Company X

Company O

Company K

Company J

Company C

Company D

Figure 6: Top Annual Average IPv6 Traffic Source Organizations(BGP AS Number) from October 2018 to September 20198

Company S
OtherCompany U

Figure 7: Top Annual Average IPv4 Traffic Source Organizations(BGP AS Number) from October 2018 to September 2019

© Internet Initiative Japan Inc.

© Internet Initiative Japan Inc.
UDP53
IP-ENCAP
TCP993
L2TP
TCP110

TCP51413
TCP54321
TCP995TCP8080
ICMPv6Total

TCP443

TCP80

UDP443ESP

IP-ENCAP

TCP8080

UDP4500

UDP12222

TCP1935

TCP8000

TCP110GRE

TCP2080

TCP22

TCP995

TCP8999

TCP44445

UDP8999

TCP182Total

Oct. 07, 2019, 00:00:00

UDP4500

UDP51388

Oct. 06, 2019, 20:30:00

ESP

TCP22

Oct. 06, 2019, 17:00:00

UDP443

TCP873Oct. 06, 2019, 13:30:00

TCP80

TCP8999
Oct. 07, 2019, 00:00:00

Oct. 06, 2019, 20:30:00

Oct. 06, 2019, 17:00:00

Oct. 06, 2019, 13:30:00

Oct. 06, 2019, 10:00:00

Oct. 06, 2019, 06:30:00

Oct. 06, 2019, 03:00:00

Oct. 05, 2019, 23:30:00

Oct. 05, 2019, 20:00:00

Oct. 05, 2019, 16:30:00

Oct. 05, 2019, 13:00:00

Oct. 05, 2019, 09:30:00

Oct. 05, 2019, 06:00:00

Oct. 05, 2019, 02:30:00

Oct. 04, 2019, 23:00:00

Oct. 04, 2019, 19:30:00

Oct. 04, 2019, 16:00:00

Oct. 04, 2019, 12:30:00

Oct. 04, 2019, 09:00:00

Oct. 04, 2019, 05:30:00

Oct. 04, 2019, 02:00:00

Oct. 03, 2019, 22:30:00

Oct. 03, 2019, 19:00:00

Vol.

Oct. 06, 2019, 10:00:00

TCP443Oct. 06, 2019, 06:30:00

Oct. 06, 2019, 03:00:00

Oct. 05, 2019, 23:30:00

Oct. 05, 2019, 20:00:00

Oct. 05, 2019, 16:30:00

Oct. 05, 2019, 13:00:00

Oct. 05, 2019, 09:30:00

Oct. 05, 2019, 06:00:00

Oct. 05, 2019, 02:30:00

Oct. 04, 2019, 23:00:00

Oct. 04, 2019, 19:30:00

Oct. 04, 2019, 16:00:00

Oct. 04, 2019, 12:30:00

Oct. 04, 2019, 09:00:00

Oct. 04, 2019, 05:30:00

Oct. 04, 2019, 02:00:00

Oct. 03, 2019, 22:30:00

Oct. 03, 2019, 15:30:00

This time around, we look at mobile device IMEI (International

Oct. 03, 2019, 19:00:00

Oct. 03, 2019, 12:00:00

Oct. 03, 2019, 08:30:00

Oct. 03, 2019, 05:00:00

Oct. 03, 2019, 01:30:00

Oct. 02, 2019, 22:00:00

Oct. 02, 2019, 18:30:00

Oct. 02, 2019, 15:00:00

Oct. 02, 2019, 11:30:00

Oct. 02, 2019, 08:00:00

Oct. 02, 2019, 04:30:00

Oct. 02, 2019, 01:00:00

Oct. 01, 2019, 21:30:00

Oct. 01, 2019, 18:00:00

Oct. 01, 2019, 14:30:00

Oct. 01, 2019, 11:00:00

Oct. 01, 2019, 07:30:00

Oct. 01, 2019, 04:00:00

Oct. 01, 2019, 00:30:00

Sep. 30, 2019, 21:00:00

Sep. 30, 2019, 17:30:00

Sep. 30, 2019, 14:00:00

Sep. 30, 2019, 10:30:00

Sep. 30, 2019, 07:00:00

Sep. 30, 2019, 03:30:00

Sep. 30, 2019, 00:00:00■ IPv6 Across Different Device OSs

Oct. 03, 2019, 15:30:00

Oct. 03, 2019, 12:00:00

Oct. 03, 2019, 08:30:00

Oct. 03, 2019, 05:00:00

Oct. 03, 2019, 01:30:00

Oct. 02, 2019, 22:00:00

Oct. 02, 2019, 18:30:00

Oct. 02, 2019, 15:00:00

Oct. 02, 2019, 11:30:00

Oct. 02, 2019, 08:00:00

Oct. 02, 2019, 04:30:00

Oct. 02, 2019, 01:00:00

Oct. 01, 2019, 21:30:00

Oct. 01, 2019, 18:00:00

Oct. 01, 2019, 14:30:00

Oct. 01, 2019, 11:00:00

Oct. 01, 2019, 07:30:00

Oct. 01, 2019, 04:00:00

Oct. 01, 2019, 00:30:00

Sep. 30, 2019, 21:00:00

Sep. 30, 2019, 17:30:00

Sep. 30, 2019, 14:00:00

Sep. 30, 2019, 10:30:00

Sep. 30, 2019, 07:00:00

Sep. 30, 2019, 03:30:00

Sep. 30, 2019, 00:00:00

Feb.2020

1. Periodic Observation Report

45

usage is similar to IPv4 usage or, in other words, that it hasof OS type (iOS or Android) and whether the device isbecome mainstream.
connecting via IPv6 or not (whether an IPv6 address is

UDP 4500, which was outside the rankings last time, cameassigned).

in at No. 5. NAT is basically not used with IPv6, and it is
The analysis covers around 1.07 million mobile phones lines

curious to note that UDP 4500, which is generally for IPSec
on a personal mobile service (IIJmio Mobile Service, num-

NAT traversal, ranks toward the top.
ber of lines subscribed to as of end-June 2019) that wereconnected on a particular weekday in October 2019 (MVNE

Last time, we noted that IPv6 is enabled by default inlines and business lines are not included).

Apple iOS from version 11, and that mobile IPv6 traffic
First, IPv6 was enabled on 48% and disabled on 52% of

had thus increased.
connections, a fairly even split, as shown in FIgure 10.

While the connections are split fairly evenly, the traffic is

Mobile Equipment Identity) numbers to facilitate an analysis
about 80% IPv4 and 20% IPv6.

UDP40003

Figure 8: Breakdown of IPv6 Traffic by Protocol Number (Next Header) and Source Port Number

TCP993Figure 9: Breakdown of IPv4 Traffic by Protocol Number and Source Port Number

9

We first look at which OS the IPv6-enabled devices are using.

and it appears that many of the recent SIM-lock-free devices

As Figure 11 shows, over 80% are on Apple iOS while 14%

have IPv6 enabled.

are on Android. And next we look at the OS on devices where
IPv6 is disabled. As Figure 12 shows, the proportions here

■ Summaryare inverted, with Android on 82% and iOS on 8%.

In this issue, we examined IPv6 traffic volume and protocols
used, as well as IPv6 connection rates based on mobile de-Android has supported IPv6 since the release of Android 5

vice OS. While IPv4 traffic growth is slowing, IPv6 traffic

in 2014, much earlier than Apple enabled IPv6 support withexhibited similar levels of growth to last time, indicating that

iOS 11, but IPv6 is often disabled by default on Android peruse of IPv6 continues to advance. Many recently available

device manufacturer and MNO policies. So from an IPv6

home Wi-Fi routers offer support for IPv6 IPoE, and there are

viewpoint, a stark difference has arisen between Android

moves to enable IPv6 on video streaming services as well,and iOS, which is exclusively controlled by Apple.

so IPv6 traffic looks set to rise even further.

Nonetheless, IPv6 is disabled on some Apple iOS devices;

IPv6 is available in the mobile space more than we had imag-

9.21% of all iOS devices to be precise. We would guess

ined, being enabled on almost half of connections. It still onlythat these are older devices that do not support iOS 11 or

accounts for around 20% of all traffic, so we hope that ser-higher. IPv6 is enabled on 14.08% of all Android devices,

vices continue to provide even more IPv6 support ahead.

IPv6 disabled 52%

IPv6 enabled 48%

Figure 10: Proportion of Connections with IPv6 Enabled
Other

3%

Android, Not Known 0%

Other

1%

Windows Phone1%

Not Known0%

Android, Not Known 1%

Not Known, iOS

2%

Not Known

5%

Android

14%

iOS

8%

iOS 83%Figure 11: OS Breakdown for IPv6-enabled Devices

10

Android 82%

Figure 12: OS Breakdown for IPv6-disabled Devices© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

1. Periodic Observation Report

Topic 4the recovery took weeks. Fortunately, IIJ’s backbone net-

Mobile FLET’S and Natural Disasters

work is designed to handle faults caused by events such as
natural disasters like this. The Tokyo–Osaka leg is distrib-

Usually, we would report on changes in the trends inuted among Pacific Ocean, Sea of Japan, and inland routes,

FLET’S and mobile traffic based on comparable observa-

so there was no real impact on user communications.

tions, but virtually the same content was presented in ourperiodic observation report in Vol. 44 (https://www.iij.

Meanwhile, Typhoon Hagibis had a clear impact on accessad.jp/en/dev/iir/044.html), so here we look at the effects

services linked directly users’ use of mobile and FLET’S ser-of natural disasters.

vices. Let’s examine this with reference to the data. Notethat FLET’S here refers only to PPPoE.

Many natural disasters occurred in 2019. Typhoon Hagibis,
in particular, wrought serious damage across many areas.

Figures 13 to 24 graph number of connections and traffic

On the IIJ backbone, several lines connecting Tokyo andvolume around October 12, 2019, when Hagibis hit Japan’s

Osaka experienced lengthy disconnections. A road wasmain island, along with the corresponding data for a week

washed away in Nagano Prefecture (between Tokyo andearlier around October 5, 2019 for comparison (upload andOsaka), and buried optical fiber cables were physically dam-

download). All of the series have been indexed to a value

aged. The damage spanned a long section of the route, andof 1 at 12 a.m. on the Fridays (October 4 and 11, 2019).

Figure 13: Number of FLET’S Connections in the Osaka Area

Figure 15: Number of FLET’S Connections in the Tokyo Area

Figure 14: FLET’S Traffic Volume in the Osaka Area

Figure 16: FLET’S Traffic Volume in the Tokyo Area

© Internet Initiative Japan Inc.

11

First, we focus on the number of connections. Currently, most

October 12 and 13, 2019. It dropped sharply in the FLET’S

FLET’S users have always-on connections via broadband

Chiba area in particular. These areas experienced manyrouters and home gateways, so connection volume remains

power outages, so the drops were likely due to household

fairly constant throughout the day. Indeed, connection vol-broadband routers and home gateways going offline whenume changed only slightly during the period in the Osaka

the power dropped out. For mobile, meanwhile, the naturearea, which was largely unaffected by Typhoon Hagibis.

of MVNO equipment precludes the ability to determine device location, so we use nationwide totals. The data show

12

However, the number of FLET’S connections in the heavily

that the number of connections fell over October 12 andimpacted Chiba, Fukushima, and Miyagi areas declined over13, 2019. Possible explanations are that MNO equipment

Figure 17: Number of FLET’S Connections in the Chiba Area

Figure 19: Number of FLET’S Connections in the Fukushima Area

Figure 18: FLET’S Traffic Volume in the Chiba Area

Figure 20: FLET’S Traffic Volume in the Fukushima Area

© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

1. Periodic Observation Report

was also damaged or that people refrained from going out

The Internet is now an important part of our infrastruc-and thus did not use their mobile connection.

ture alongside electricity, gas, and water, and one that is
all the more valuable during natural disasters. IIJ will con-Next we look at traffic volume. In Tokyo and areas to thetinue to design its backbone with due consideration to thenorth, FLET’S traffic volume increased during the daytime

potential impact of natural disasters and provide Internet

on October 12, 2019, while mobile traffic fell heavily. It

infrastructure that users can rest assured will be available

was a weekend, but many people probably stayed indoors

at all times.

to avoid the storm, using their FLET’S broadband at homeinstead of mobile data.

Figure 21: Number of FLET’S Connections in the Miyagi Area

Figure 23: Number of Mobile Connections

Figure 22: FLET’S Traffic Volume in the Miyagi Area

Figure 24: Mobile Traffic Volume© Internet Initiative Japan Inc.

13

Topic 5with some being 155Mbps (STM-1). The map shows how

The Transition of the Bandwidthof IIJ Backbone Circuits

the backbone circuit has changed out from 1998 gradually.
For a time, it was the case that circuits and router interfaces

In the 2018 edition (Vol. 41) of this report, we noted thatwith quadruple the bandwidth were released roughly every

total traffic on IIJ’s backbone had grown more than tenfold

two years. Bandwidth continued to expand to 600Mbps

in the past 10 years. Here, we look back on the transition(STM-4) in 2000, 2.4Gbps (STM-16) in 2002, and 9.6Gbps

of IIJ’s backbone, particularly the bandwidth of the back-(STM-64) in 2004. It was a happy era in which circuit ca-bone circuits. IIJ began its ISP operations as a Type 2

pacity expanded as traffic grew. While the circumstances

Telecommunication Carrier; License for the carrier provid-

may have been different for larger ISPs, IIJ was mostly able

ing telecommunication services using facilities or circuits

to keep up with the growing traffic by upgrading circuits

leased from Type 1 Telecommunication Carriers. So fromwithout significantly changing its network topology.

the outset, we have leased circuits installed by Type 1
Telecommunication Carriers to build a network and provide

The following graph, based on the backbone map, plots

our services. And this situation is unchanged even after the

total backbone bandwidth between Japan and the US along

repeal of the distinction between Type 1 and Type 2 carriers.

with the highest-capacity circuits that were available in the
backbone. The circuit bandwidth steadily increased fromWe still have the backbone map from 1998. Most of the1998 up until when STM-64 rolled out in 2004. But after

circuits’ bandwidth used at that time were 45Mbps (DS-3),

2005 through 2014, over 9 years, circuit bandwidth above（Mbps）

1000000

100000

10000

1000

100

Highest capacity line
Japan–US route14© Internet Initiative Japan Inc.

Ju

n.

18

,2

02

0

01
7
p.
Se

7,
ec
.2
D

Figure 25: Line Capacity and Japan–US Bandwidth

22
,2

4
20
1

2
1,
r.
Ap

6,
2
l.
Ju

20
1

00
9

06
O

ct

.1

0,

4,
2
Ja

n.
1

,2
19
r.
Ap

20

00

00
1

8
99
,1
24
l.
Ju

4

10

（Date）

Feb.2020

Vol.

45

1. Periodic Observation Report

STM-64 did not become standard. Various factors were be-Things settled somewhat once 100G ethernet became stan-hind this. STM-256 was not all that widely adopted, anddard in 2014. IIJ also finished converting core backbone

technical difficulties associated with 100G, the next step

circuits to 100G in 2018 and is in the process of upgradingup, caused development and standardization delays. Yet

the other network to 100G as traffic grows.

traffic continued to grow over this period, eventually reaching the point that we needed bandwidth of 200Gbps or 20

Meanwhile, a sixth 100G circuit between Japan and the US

STM-64 circuits between Japan and the US. (Note that theis in the schedule to go up in December 2019. With fivegraph’s y-axis is logarithmic!)

years having passed since 100G went mainstream, we’restarting to search for the next step up in capacity. The foot-Hence, the design of the IIJ backbone changed considerably

steps of 400G are drawing gradually closer.

over this period. To change our backbone topology freely,
we adopted the MPLS abstract layer, and to boost circuit

IIJ will continue to follow the latest technological develop-

utilization efficiency, we altered the network topology toments as it expands our backbone to ensure it remains fast

make it more amenable to ECMP (Equal Cost Multi-Path).

and efficient with high availability.

1.BGP / Number of RouteTomohiko Kurahashi
Technology Development Section, Operation Technology Department, Infrastructure Engineering Division, IIJ.

2.DNS Query Analysis
Yoshinobu Matsuzaki
Technology Development Section, Operation Technology Department, Infrastructure Engineering Division, IIJ

3.IPv6
Taisuke Sasaki
Deputy General Manager, Network Technology Department, Infrastructure Engineering Division, IIJ

4.Mobile FLET’S and Natural Disasters
Takafusa HoriManager, Network Technology Section, Network Technology Department, Infrastructure Engineering Division, IIJ

5.The Transition of the Bandwidth of IIJ Backbone Circuits
Takanori Sasai
Manager, Backbone Technology Section, Network Technology Department, Infrastructure Engineering Division, IIJ© Internet Initiative Japan Inc.

15

2. Focused Research

Acquiring Forensic Memory Images on Linux

2.1 Linux Memory Dump Toolsn

avoid changing data on the target host as much as possible,

We use the Volatility Framework when performing memorycompilation of the analysis tools, etc., should be done on aanalysis for incident response and forensics. In IIR Vol. 32,

separate host.

“1.4.1 Creating Profiles for the Volatility Framework,” we explained the procedure for creating a Volatility profile for Linux*1.

2.2 What is LiME?
Here, we explain how to acquire Linux memory images for

LiME is short for Linux Memory Extractor, and is the tool

analysis in Volatility. Several tools for acquiring Linux mem-that Volatility recommends*5 for acquiring memory images.

*3

ory images exist, but here we will look at LiME and crash .

Since LiME operates as a Linux kernel module, it must be

We will also describe a memory image acquisition method

compiled against the same kernel version as that running onthat minimizes the impact on disk forensics. Note that thisthe target host.

*2

article assumes your system is running CentOS 7.7-1908.
Another tool for acquiring memory images is Linpmem*4, but

2.3 Compiling LiME

it did not work well in our test environment, so we decided

To compile LiME, you first obtain the LiME source code by

to leave it out.

running the git command in Figure 1 or by downloading thezip file from LiME’s GitHub page and extracting it into a

Note that in this article we refer to the machine being ex-directory. The src subdirectory inside the source code di-amined as the “target host” and the machine on which therectory contains a Makefile, so running the make command

examination is performed as the “examination host”. To

from within that subdirectory will generate the LiME module$ git clone https://github.com/504ensicsLabs/LiME.git

Figure 1: Cloning the LiME Git Repository$ cd LiME/src$ make$ ls
disk.c lime-3.10.0-1062.el7.x86_64.ko
disk.o lime.hhash.c lime.mod.chash.o lime.mod.olime.omain.cmain.oMakefile

Makefile.sample tcp.o
modules.order
Module.symvers
tcp.c$ sudo yum install kernel-devel gcc

Figure 2: Compiling the LiME Module

16

Figure 3: Installing the kernel-devel Package*1Internet Infrastructure Review (IIR) Vol. 32, 1.4.1 Creating Profiles for the Volatility Framework (https://www.iij.ad.jp/en/dev/iir/032.html).

*2

LiME (https://github.com/504ensicsLabs/LiME).

*3

crash (https://people.redhat.com/anderson/).

*4

Velocidex/c-aff4 (https://github.com/Velocidex/c-aff4/releases).

*5

Linux · volatilityfoundation/volatility Wiki (https://github.com/volatilityfoundation/volatility/wiki/Linux#acquiring-memory).

© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research(see the red outline in Figure 2). Certain packages are re-kernel-devel package was extracted. You also need to installquired to compile LiME (kernel-devel, gcc, etc.), so if the

any other required packages (on our machine, we needed tomake command produces an error, try installing the corre-

install elfutils-libelf-devel). We confirmed that the LiME mod-sponding packages as shown in Figure 3.

ule generated in this manner works on CentOS 8 (1905).

Note that unless you specify a version at install time, the lat-

2.4 Dumping Memory to an External Drive

est version of the kernel-devel package will be installed. If a

When acquiring memory images, you need to minimize disk

different version of the kernel is running on the target host,write operations on the target host. In particular, the mem-run a search using the yum command, install the appropriate

ory image is almost always several GB or more is size, sokernel-devel package version, and then compile the LiME

writing it to the target host’s disk would cause many unused

module, as shown in Figure 4. You need to pass the KVER

areas to be overwritten, which could greatly affect forensic

option to the make command when doing this.

analysis of the disk.

If the OS version differs, you can generate a LiME module

So if you have physical access to the target host, you canfor that version by downloading the corresponding ker-connect a USB stick or mobile SSD containing the LiME mod-nel-devel package separately, extracting it using the cpio

ule to it and load the LiME module into the kernel from therecommand, and then running make with the KVER and KDIR

via the insmod command, as shown in Figure 6. This will

options specified, as shown in Figure 5. KVER specifies thelet you acquire a memory image while barely writing any-kernel version, and KDIR specifies the directory where thething to the target host’s disk. The double quotes in Figure$ yum --showduplicates search kernel-devel$ sudo yum install kernel-devel-3.10.0-1062.1.2.el7.x86_64$ make KVER=3.10.0-1062.1.2.el7.x86_64

Figure 4: Compiling LiME Against a Specific Kernel Version (1)$ curl -OL http://ftp.iij.ad.jp/pub/linux/centos/8.0.1905/BaseOS/x86_64/os/Packages/kernel-devel-4.18.0-80.11.2.el8_0.x86_64.rpm$ rpm2cpio ./kernel-devel-4.18.0-80.11.2.el8_0.x86_64.rpm | cpio -id$ sudo yum install elfutils-libelf-devel$ make KVER=4.18.0-80.11.2.el8_0.x86_64 KDIR=~/src/usr/src/kernels/4.18.0-80.11.2.el8_0.x86_64/

Figure 5: Compiling LiME Against a Specific Kernel Version (2)$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=/media/centos77.mem format=lime"

Figure 6: Loading the LiME Module© Internet Initiative Japan Inc.

17

6 contain the LiME module options. In this example, a lime

other tools to acquire a memory image via the network. Here,format dump file will be saved to the USB drive mounted at

we show how to combine it with Netcat, NFS, and SSH.

/media under the filename /media/centos77.mem.

The examination host’s IP address is 192.168.232.131,
and target host’s is 192.168.232.132.

The memory dump starts when the LiME module is loaded,
but the command prompt does not return until the mem-

■ Netcat (1)ory dump is completed. So don’t worry if you’re unable to

The LiME module has functionality that enables memory

enter commands after loading the module; just wait for thedumps over the network, and we combine this functional-

dump to complete. More recent servers, in particular, can be

ity with the Netcat command to acquire a memory image

expected to take a while because of their large memory ca-

via the network. Once the Netcat command is installed on

pacity. Note that the LiME module remains loaded even after

the examination host, the commands in Figures 8 and 9

the memory dump finishes, so unload it using the rmmod

will connect the examination host to the port on which thecommand, as shown in Figure 7.

LiME module is listening (4444/tcp) to acquire the memory
image data.

2.5 Dumping Memory via the Network
If you do not have physical access to the target host as■ Netcat (2)covered above (for example, if the target host is in a remote

The above procedure will result in data equivalent to thelocation that makes physical access difficult), or if you don’t

capacity of memory being received. If the target host is ahave a large-capacity USB stick, you can combine LiME with

server, that data may take up several dozen GB or more, so$ lsmod | grep lime
$ sudo rmmod lime

$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=tcp:4444 format=lime"
Switch to another login session to execute the following command$ nc localhost 4444 | gzip -c | nc 192.168.232.131 5555

Figure 7: Unloading the LiME Module

Figure 11: Commands Executed on the Target Host
$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=tcp:4444 format=lime"

Figure 8: Listening on 4444/tcp with LiME (Target Host)$ nc 192.168.232.132 4444 > centos77.mem

Figure 9: Acquiring a Memory Image via the Network Using Netcat(Examination Host)

Figure 12: Setting the NFS Server (Examination Host)/mnt/nfsserv/ 192.168.232.132(rw,all_squash)$ nc -l 5555 > memorydump.lime.gz

Figure 10: Command Executed on the Examination Host18

$ sudo yum install nfs-utils$ sudo mkdir /mnt/nfsserv/$ chown -R nfsnobody:nfsnobody /mnt/nfsserv/$ sudo vi /etc/exports$ sudo systemctl start nfs.service$ sudo systemctl status nfs.service

Figure 13: NFS Export Settings on the Examination Host (/etc/exports)© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research

we want to compress it as much as possible. If Netcat ishost with read/write capability. By copying LiME to this NFS

also installed on the target host, executing the commands

volume and mounting it as an NFS from the target host, you

in Figures 10 and 11 will result in the memory image being

can acquire a memory image without creating any files oncompressed using gzip when it is transferred to the exam-

the target host (Figures 12, 13, 14, and 15).

ination host. As mentioned, the command prompt does not
return when the LiME module is loaded, so the command■ SSH

following the nc command needs to be run from a separate

If you cannot use Netcat or NFS, you can use SSH instead.

login session.

By executing the commands in Figure 16, you can transfer amemory image to the examination host via SSH. However,In this example, we first use Netcat on the examination hostthis method can only be used in bash. As with the Netcat (2)to listen on 5555/tcp. Next, we use Netcat on the target

instructions, the exec command onward must be run from ahost to connect to 4444/tcp, where the LiME module is lis-

separate login session.

tening, acquire a memory image, compress it using gzip, andthen transfer it to the examination host via Netcat.

2.6 What is Crash?
The crash command is a tool for analyzing Linux mem-

■ NFS

ory images. The primary purpose of the tool is to perform

If the target host can mount an NFS volume, set up theanalysis, but a module can also be used to perform mem-examination host on a network that the target host has ac-ory dumps. But as there is no RPM package of the crashcess to. Then export the NFS volume on the examination

memory dump module, we have to compile it from source$ sudo firewall-cmd --permanent --add-service=nfs$ sudo firewall-cmd --reload$ sudo exportfs -v

Figure 14: Configuring the Examination Host’s Firewall
and Checking the NFS Export$ sudo mount -t nfs 192.168.232.131:/mnt/nfsserv/ /mnt/$ sudo insmod /mnt/lime-3.10.0-1062.el7.x86_64.ko "path=/mnt/centos77.mem format=lime"

Figure 15: Mounting the NFS and Acquiring a Memory Image (Target Host)$ sudo insmod /media/lime-3.10.0-1062.el7.x86_64.ko "path=tcp:4444 format=lime"
Switch to another login session to execute the following command$ exec 5<>/dev/tcp/127.0.0.1/4444; cat <&5 | ssh -c user@192.168.232.131 'cat > centos77.mem'

Figure 16: Commands Executed on the Target Host

© Internet Initiative Japan Inc.

19

(Figure 17). The crash command also requires a kernel with

2.7 Analyzing the Memory Image

debug symbols, so we install the kernel debugging package

To analyze a Linux memory image using Volatility*5, you(Figure 18). Next, we copy the three files shown in Figureneed a profile corresponding to the Linux kernel version.

19 to the same directory on a USB stick. On the target

As noted at the beginning, the procedure for creating ahost, we execute crash as shown in Figure 20 to acquire a

Volatility profile is described in IIR Vol. 32*1. Please refermemory image.

there if you are unsure.

If the kernel version of the target and examination hosts

While we were writing this article, Lorenzo Martínez

differ, then search for the appropriate version of the ker-

opened a Bitbucket repository that automatically gener-

nel-debuginfo package as demonstrated in Figure 4 (yumates and publishes LiME modules and Linux profiles for--showduplicates search). Download and extract the pack-

Volatility*7. This repository is updated whenever new Linuxage as shown in FIgure 5 (curl, rpm2cpio, cpio commands),

kernel packages are released. Note, though, that the OSs

and copy the vmlinux file. We also recommend using a crashthat it automatically generates for are CentOS 5, 6, 7,

version that corresponds to your OS version (for example,

8 and Ubuntu 14.04 LTS, 16.04 LTS, 18.04 LTS. You

crash-7.2.3-18 is provided in CentOS 8.0).

can find and download the appropriate LiME module and$ sudo yum install crash crash-devel$ yumdownloader --source crash$ rpm -ivh crash-7.2.3-10.el7.src.rpm$ cd rpmbuild/SPECS$ rpmbuild -bp crash.spec$ cd ../BUILD/crash-7.2.3$ make extensions

・/usr/bin/crash・rpmbuild/BUILD/crash-7.2.3/extensions/snap.so
・/usr/lib/debug/usr/lib/modules/3.10.0-1062.el7.x86_64/vmlinux

Figure 17: Installing the crash Source Package and Compiling the Module

Figure 19: Files Required for a Memory Dump

$ sudo yum install --enablerepo=base-debuginfo kernel-debuginfo-3.10.0-1062.el7.x86_64

Figure 18: Installing a Kernel with Debug Symbols

20*6

The Volatility Foundation - Open Source Memory Forensics (https://www.volatilityfoundation.org/).

*7

Lorenzo Martínez’ tweet (https://twitter.com/lawwait/status/1181469996821700609).

© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research

Volatility profile by filtering the repository webpage by ker-files one is not concerned about losing. It has been observed,

nel version.

however, that attackers take advantage of these properties
by using tmpfs as an anti-disk forensics haven for files.

Also, it does not support architectures other than x86_64,
so if you are using a Linux distribution or architecture otherFor this reason, Volatility provides the Linux-specific linux_

than those covered by the autogenerated files, you will need

tmpfs command. This command is used to restore files held

to prepare the files yourself. The same applies if you arein a tmpfs. Figure 21 illustrates tmpfs files mounted on theusing a customized Linux kernel. If using crash, you will also

/home/user/tmp directory being restored. The command re-need your own customized kernel with debug symbols.

sults in the file called “tmpfs_example.txt” being restored,
and evidently it contains the string “hello!!”

2.8 Tips for Acquiring Memory Images
In Linux kernel version 2.4 upward, the tmpfs file system is

If available memory is running low, however, the contents

available. Data on a tmpfs file system resides only in memoryof the tmpfs are swapped out, making it impossible to re-and is lost if the host is shut down or restarted, so tmpfs isstore the data in the tmpfs via a memory dump (Figure 22).

usually only used for temporary directories and the like with

A potential countermeasure in this case is to temporarily

Move to the directory where the files were copied and then execute thesecommands.
$ sudo ./crash ./vmlinux(Then from within the crash command prompt)
extend ./snap.so
snap centos77.mem$ hexdump -C ~/vol_output/swapout/tmpfs_example.txt
00000000 00 00 00 00 00 00 00 00
00000008

Figure 20: Acquiring a Memory Image|........|

Figure 22: Example of a Failure to Restore tmpfs Data due to a Swap-Out$ python2 ./vol.py --profile=LinuxCentOS77x64 -f ~/tmpfs_swapoff.mem linux_tmpfs -L
Volatility Foundation Volatility Framework 2.6.1
1 -> /sys/fs/cgroup2 -> /run3 -> /home/user/tmp
4 -> /dev/shm$ python2 ./vol.py --profile=LinuxCentOS77x64 -f ~/tmpfs_swapoff.mem linux_tmpfs -S 3 -D ~/vol_output/$ hexdump -C ~/vol_output/tmpfs_example.txt
00000000 68 65 6c 6c 6f 21 21 0a|hello!!.|00000008

Figure 21: Restoring and Examining the Contents of Files Stored in a tmpfs© Internet Initiative Japan Inc.

21

disable the swap. This means forcing data that has been

it with all types of memory images, but in our setup, weswapped out to be swapped in. Whether or not this can bewere able to analyze Windows RAW memory images anddone, however, depends on memory usage on the target

CentOS memory images acquired using LiME. A big change

host. If data has been swapped out because of a temporaryis that the option to specify a profile, which was required inincrease in memory usage, you may be able to subsequently

Volatility 2, is no longer present. Instead, Volatility 3 infers

disable the swap if memory has since been freed up. ThisOS type and version from the memory image being analyzed

means a situation like that illustrated in Figure 23, for ex-

and refers to the corresponding symbol table. For exam-

ample, where the value of Swap used is lower than freeple, when a Windows memory image is read in, Volatility

Mem. Analysis of the memory image acquired after disabling

3 automatically downloads the PDB file from Microsoft andthe swap confirms that the tmpfs data can be restored, as

analyzes it to construct a symbol table that it can reference.

shown in Figure 24.

However, you still need to prepare symbol tables for macOSand Linux in advance, as in Volatility 2. You can use theThis method, however, does overwrite data in unused areas

symbol tables provided by the Volatility developers, but theof memory. Because unused memory areas may still containsymbol tables for Linux lack information relative to those

useful data, a better way to perform Linux memory foren-

for Windows and macOS, so in most cases, you’ll need tosics is to dump memory once before disabling the swap andprovide the table yourself.

once again after.
Symbol tables are created using a tool called dwarf2json*9.

2.9 Volatility 3

As dwarf2json is written in Go, we first install the golang

Version 2 of Volatility has been in use for a long time, but

package and then build dwarf2json. We also install the ker-

during the writing of this article, a public beta version ofnel-debuginfo package because we need a Linux kernel with

Volatility 3 was released . Of course we did not check

symbols. Executing dwarf2json with a Linux kernel with*8

$ free
total
used
free
Mem:
186324875920
307192
Swap:
209714856776
2040372$ sudo swapoff -a$ free
total
used
free
Mem:
1863248118804247652Swap:000(After dumping memory, reenable the swap)$ sudo swapon -ashared buff/cache768
1480136

available
1591860

shared buff/cache
9800
1496792

available
1539936

Figure 23: Checking Memory Usage, and Disabling and Enabling the Swap$ hexdump -C ~/vol_output/swapoff/tmpfs_example.txt
00000000 68 65 6c 6c 6f 21 21 0a00000008|hello!!.|Figure 24: tmpfs Data Restored from a Memory Image
After Disabling the Swap

$ sudo yum install epel-release$ sudo yum install golang$ git clone https://github.com/volatilityfoundation/dwarf2json.git$ cd dwarf2json$ go build$ sudo yum install --enablerepo=base-debuginfo kernel-debuginfo-3.10.0-1062.1.2.el7.x86_64$ ./dwarf2json linux --elf /usr/lib/debug/usr/lib/modules/3.10.0-1062.1.2.el7.x86_64/vmlinux > centos77-3.10.0-1062.1.2.el7.x86_64.json$ xz -z centos77-3.10.0-1062.1.2.el7.x86_64.json$ wget https://downloads.volatilityfoundation.org/volatility3/symbols/linux.zip$ zip ./linux.zip ./centos77-3.10.0-1062.1.2.el7.x86_64.json.xz

Figure 25: Building dwarf2json and Generating a Symbol Table

22*8

Volatility Labs: Announcing the Volatility 3 Public Beta! (https://volatility-labs.blogspot.com/2019/10/announcing-volatility-3-public-beta.html).

*9dwarf2json (https://github.com/volatilityfoundation/dwarf2json).

© Internet Initiative Japan Inc.

Feb.2020

Vol.

45

2. Focused Research

symbols specified will generate a symbol table, so we add

analysis to fail. And with Windows memory images, some-

this to the file (linux.zip) that contains the symbol table dis-times analysis of the downloaded PDB fails, preventing you

tributed by the developers. See Figure 25 for the specificfrom advancing to the memory image analysis stage of thecommands. Copy the generated symbol table to the speci-

process.

fied Volatility 3 subdirectory (Figure 26).
The Volatility development team has announced an official
Volatility 3 is executed using this command line format: “py-Volatility 3 version will be released in August 2020. Support

thon3 vol.py -f <memory image file> <plugin>”. Figure 27

for Volatility 2 will continue for one year after that throughshows the results of analyzing a CentOS 7.7 memory image

August 2021, but with Volatility 3 set to become main-using the pstree plugin. The format has changed slightly fromstream ahead, it’s probably a good idea to get accustomed

that of Volatility 2, with the * character now used to denoteto the new usage and configuration methods before the of-process nesting. The method for specifying the plugin to beficial release hits.

executed has also changed. For the pstree plugin, you specify“linux_pstree” in Volatility 2, but in Volatility 3, you specify“linux.pstree.PsTree”. A list of available plugins can be foundby running “python3 vol.py -h”.
While we did get it to work properly, we also identified
some bugs. As noted above, OS type and version are inferred from the contents of the memory image specified on
the command line, but it can fail to recognize the correctLinux kernel version with some memory images, causing the$ cd ..
$ sudo yum install python3$ git clone https://github.com/volatilityfoundation/volatility3.git$ pip3 install --user pefile yara-python capstone$ cp ./dwarf2json/linux.zip ./volatility3/volatility/symbols/

Figure 26: Installing Volatility 3 and Copying the Symbol Tables$ cd volatility3$ python3 vol.py -f ~/centos77.mem linux.pstree.PsTree
Volatility 3 Framework 1.0.0-beta.1
Progress:
23.13
Scanning LimeLayer using RegExScanner
PID
PPID
COMM
1
0
systemd
* 833
1
login
** 1695 833
bash
* 841
1
firewalld* 843
1
NetworkManager** 992 843
dhclient* 558
1
systemd-journal* 593
1
systemd-udevd* 818
1
dbus-daemon* 1171 1
sshd** 1720 1171
sshd*** 1724
1720
sshd**** 1725
1724
bash***** 1751
1725
sudo
****** 1753
1751
insmod* 1172 1
tuned
* 821
1
systemd-logind* 822
1
irqbalance* 823
1
polkitd* 1174 1
rsyslogd* 1397 1
master** 1402 1397
pickup** 1405 1397
qmgr
(Subsequent output omitted)

Figure 27: Running the pstree Plugin on a Linux Memory Image

Minoru Kobayashi
Forensic Investigator, Office of Emergency Response and Clearinghouse for Security Information, Advanced Security Division, IIJ
Mr. Kobayashi is a member of IIJ-SECT, mainly dealing with digital forensics. He works to improve incident response capabilities and inhouse technical capabilities.
He gives lectures and training sessions at security events both in Japan and abroad, including Black Hat, FIRST TC, JSAC, and Security
Camp events.

© Internet Initiative Japan Inc.

23

45
February 2020 Vol.
About Internet Initiative Japan Inc. (IIJ)
IIJ was established in 1992, mainly by a group of engineers who
had been involved in research and development activities related
to the Internet, under the concept of promoting the widespreaduse of the Internet in Japan.
IIJ currently operates one of the largest Internet backbones
in Japan, manages Internet infrastructures, and provides
comprehensive high-quality system environments (includingInternet access, systems integration, and outsourcing services,etc.) to high-end business users including the government andother public offices and financial institutions.
In addition, IIJ actively shares knowledge accumulated throughservice development and Internet backbone operation, andis making efforts to expand the Internet used as a social
infrastructure.­­The copyright of this document remains in Internet Initiative Japan Inc.
(“IIJ”) and the document is protected under the Copyright Law of Japanand treaty provisions. You are prohibited to reproduce, modify, or makethe public transmission of or otherwise whole or a part of this document
without IIJ’s prior written permission. Although the content of thisdocument is paid careful attention to, IIJ does not warrant the accuracy andusefulness of the information in this document.
Initiative Japan Inc. All rights reserved.
© Internet
IIJ-MKTG020-0043

Internet Initiative Japan Inc.
Address: Iidabashi Grand Bloom, 2-10-2 Fujimi, Chiyoda-ku,
Tokyo 102-0071, Japan
Email: info@iij.ad.jp URL: https://www.iij.ad.jp/en/