Internet
Infrastructure
Review
Feb.2022

Vol.

53

Periodic Observation Report

Internet Trends as Seenfrom IIJ Infrastructure—2021
Focused Research (1)Running an Overlay Network
in a Multi-tenant Setup—Challenges of IIJ
GIO Infrastructure P2 Gen.2
Focused Research (2)IIJ’s Road to BCR Approval—Complying with EU GDPR and Beyond
Focused Research (3)In Pursuit of Carbon Neutrality
in the Data Center
Focused Research (4)IIJ’s Road to BCR Approval—Complying with EU GDPR and Beyond

Internet Inf rast ruc ture Review
February 2022 Vol.53

Executive Summary ........................ ................................................ 3
1. Periodic Observation Report .......... ................................................ 4
Topic 1 BGP and Routes .......................... ................................................ 4
Topic 2 DNS Query Analysis . .................................................................... 6
Topic 3 IPv6 . . . . ................................. ................................................ 8

Topic 4 State of the Mobile Industry and Traffic Trends .......................................... 11

Topic 5 IIJ Backbone ............................. ............................................... 13

2. Focused Research (1) .................. ............................................... 16
2.1
2.2
2.3
2.4
2.5

P2 Gen.2—IIJ’s new-generation IaaS ...................................................... 16
Overlay Networks Using SDN Technology .. ............................................... 18
Benefits of Overlay Networks with VMware NSX-T ........................................ 19
Operational Issues and Solutions .......................................................... 20

Looking Ahead . ........................... ............................................... 22

3. Focused Research (2) .................. ............................................... 24
3.1
3.2
3.3
3.4
3.5

Introduction ............................... ............................................... 24
Outcomes at Matsue Data Center Park .................................................... 25
Initiatives at Shiroi Data Center Campus . .. ............................................... 28
Carbon Neutral Data Center Model ......... ............................................... 33

Conclusion ................................ ............................................... 37

4. Focused Research (3) .................. ............................................... 38
4.1
4.2
4.3
4.4
4.5

2

Decision to Seek BCR Approval ........................................................... 38

What are BCRs? ........................... ............................................... 40
Personal Data Protection Initiatives Around the World ..................................... 42
IIJ’s Road to BCR Approval ................ ............................................... 46

Looking Ahead . ........................... ............................................... 46© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

Executive Summary

Executive Summary

To say that modern society runs on information and communications technology (ICT) is no exaggeration.
Computers and communication networks are no longer absent from any of our social activities, and we reap
the benefits of ICT in terms of increased sophistication and efficiency. What constitutes the infrastructure that
supports the workings of society? Energy, transport, government services, finance—the list goes on. Yet noneof these would function without ICT. Against this backdrop, Japan is no different from the rest of the worldin experiencing the impact on financial and communications services when faults cause system outages atmajor financial institutions and communications carriers. In our highly information-oriented society, there is anincreasing need for reliability with respect to information and communications, and the demands of governancewith respect to the businesses that ensure that reliability are also growing. There is no room for doubt about
the Internet’s role as crucial social infrastructure, and we are well aware that, as part of this, IIJ’s networks
also play a role underpinning society. We hope to continue meeting society’s expectations, and to that end we
strive to develop technologies to ensure that we can continue to provide highly reliable services.
The IIR introduces the wide range of technology that IIJ researches and develops, comprising periodic observation
reports that provide an outline of various data IIJ obtains through the daily operation of services, as well as focusedresearch examining specific areas of technology.
Our periodic observation report in Chapter 1 presents the 2021 edition of our look at Internet trends as seenfrom IIJ’s infrastructure. The report covers IPv4 and IPv6 routes on the Internet, an analysis of DNS queries obtained
from the full resolver provided by IIJ, IPv6 and mobile traffic, and an analysis of traffic during the Tokyo Olympics.
As the Internet expands, all sorts of resources and traffic continue to grow, and we are also observing a steadyshift in protocols—a rise in AAAA record and HTTPS record DNS queries, for instance, and a rise in absolute
IPv6 traffic levels.
The focused research report in Chapter 2 discusses the challenges encountered with the network newly developed
for IIJ GIO Infrastructure P2 Gen.2, IIJ’s new cloud service released in October 2021, as well as the outlook
ahead. While using VMware NSX-T, the engineers also worked in-house to create mechanisms to efficiently operate
the infrastructure, which are used for monitoring and capacity planning. The technology is currently used tobuild networks within data centers and between data centers, but development is ongoing with the aim ofextending its applications to connections with distributed edge-computing resources.
Our second focused research report in Chapter 3, titled “In Pursuit of Carbon Neutrality in the Data Center,”
describes our initiatives at IIJ’s Matsue Data Center Park and Shiroi Data Center Campus, offering insights
about technologies IIJ has actually deployed at the facilities. Matsue is host to Japan’s first commerciallyoperating outside-air cooled modular data center, which runs on a three-phase four-wire power supply, while
Shiroi features direct outside-air cooling, system modules, AI control, lithium-ion storage batteries, and more.
With carbon neutrality now a major topic, I think you will find the report highly interesting.
Chapter 4 presents our third focused research report on the IIJ’s Group’s BCRs (Binding Corporate Rules). The
IIJ Group created its BCRs in an effort to comply with the EU’s GDPR (General Data Protection Regulation),and on August 5, 2021, these BCRs were approved by the competent data protection authority in Germany. In
addition to BCRs and the GDPR, the report looks at global efforts around personal data protection, and walks
the reader through the IIJ’s BCR approval process. The story is a valuable record of events. After IIJ submitted
the BCRs for approval in the UK in 2016, the UK withdrew from the EU, and the approval eventually ended up
coming from Germany.
Through activities such as these, IIJ strives to improve and develop its services on a daily basis while maintaining
the stability of the Internet. We will continue to provide a variety of services and solutions that our customers cantake full advantage of as infrastructure for their corporate activities.

Junichi Shimagami
Mr. Shimagami is a Senior Executive Officer and the CTO of IIJ. His interest in the Internet led to him joining IIJ in
September 1996. After engaging in the design and construction of the A-Bone Asia region network spearheaded by IIJ,
as well as IIJ’s backbone network, he was put in charge of IIJ network services. Since 2015, he has been responsible
for network, cloud, and security technology across the board as CTO. In April 2017, he became chairman of theTelecom Services Association of Japan MVNO Council.

© Internet Initiative Japan Inc.

3

1. Periodic Observation Report

Internet Trends as Seen from IIJ Infrastructure—2021
Internet services provider IIJ operates some of the largest

the start of 2021 that APNIC would completely exhaust its

network and server infrastructure in Japan. Here, we report

IPv4 address pool, but that has not happened over the year

on Internet trends over the past year based on informationthat followed.

obtained through the operation of this infrastructure. We
analyze changes in trends from the perspective of BGP

The annual increase in the number of routes fell short ofroutes, DNS query analysis, IPv6, and mobile. We also

40,000 for the first time in 10 years. The increase fordiscuss conditions observed after the deployment of BGP

prefixes /21 through /24 was also below the previous year’s

ROV on the IIJ backbone.

level, and the total number of routes, now over 850,000,
may be nearing its peak. The large increase in /8 routes

Topic 1and unique addresses is noticeable. All of the additional /8

BGP and Routes

routes are advertised by AS8003. Information indicates thatthe routes advertised by AS8003 (subsequently changed to

We start by looking at IPv4 full-route information advertised

AS749) are for a special purpose, so excluding the effectby our network to other organizations (Table 1) and theof these (765 routes in total), the number of /8 routes wasnumber of unique IPv4 addresses contained in the IPv4 full-

down (-2), and the number of unique addresses was only up

route information (Table 3). Incidentally, it was projected atslightly (around 2.86 billion).

Table 1: Number of Routes by Prefix Length for Full IPv4 Routes
Date

/8

/9

/10

/11

/12

/13

/14

/15

/16

/17

/18

/19

/20

/21

/22

/23

/24

total

Sep. 2012 19

14

29

84

236

471

838

1526

12334

6349

10710

20927

30049

31793

42007

39517

219343416246

Sep. 2013 16

11

30

93

250

480

903

1613

12748

6652

10971

22588

32202

34900

48915

42440

244822459634

Sep. 2014 16

12

30

90

261

500

983

1702

13009

7013

11659

24527

35175

37560

54065

47372

268660502634

Sep. 2015 18

13

36

96

261

500

999

1731

12863

7190

12317

25485

35904

38572

60900

52904

301381551170Sep. 2016 16

13

36

101

267

515

1050

1767

13106

7782

12917

25229

38459

40066

67270

58965

335884603443

Sep. 2017 15

13

36

104

284

552

1047

1861

13391

7619

13385

24672

38704

41630

78779

64549

367474654115

Sep. 2018 14

11

36

99

292

567

1094

1891

13325

7906

13771

25307

39408

45578

88476

72030

400488710293

Sep. 2019 10

11

37

98

288

573

1142

1914

13243

7999

13730

25531

40128

47248

95983

77581

438926764442

Sep. 2020 9

11

39

100

286

576

1172

1932

13438

8251

14003

25800

40821

49108

10179984773

473899816017

Sep. 2021 16

13

41

101

303

589

1191

2007

13408

8231

13934

25276

41915

50664

10676391436

497703853591

Table 2: Number of Routes by Prefix Length for Full IPv6 Routes
Date

4

/29

/30-/31/32/33-/39/40/41-/43/44/45-/47

/48

total

Sep. 2012 102/16-/28

45

34

4448

757

445

103

246

168

3706

10054

Sep. 2013 117

256

92

5249

1067

660

119

474

266

5442

13742

Sep. 2014 134

481

133

6025

1447

825

248

709

592

7949

18543

Sep. 2015 142

771

168

6846

1808

1150

386

990

648

10570

23479

Sep. 2016 153

1294

216

8110

3092

1445

371

1492

1006

14291

31470

Sep. 2017 158

1757

256

9089

3588

2117

580

1999

1983

18347

39874

Sep. 2018 168

2279

328

10897

4828

2940

906

4015

2270

24616

53247

Sep. 2019 192

2671

606

12664

6914

3870

1566

4590

4165

34224

71462

Sep. 2020 205

3164

641

14520

9063

4815

2663

5501

4562

45160

90294

Sep. 2021 223

3628

705

20650

13050

10233

4170

11545

5204

61024

130432© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

1. Periodic Observation Report

Next, we look at IPv6 full-route information (Table 2) and

Lastly, let’s also look at IPv4/IPv6 full-route Origin AS

the number of unique IPv6 /64 blocks in the IPv6 full-routefigures (Table 4). In the past year, an additional 6144

information (Table 3).

32-bit-only AS numbers were allocated to APNIC and2048 to ARIN.

The total number of routes increased around 1.5-fold toover 130,000, well in excess of the forecast we madeBoth the decrease in 16-bit Origin Autonomous System

last year. The combined total with IPv4 included is now

Numbers (ASNs) and the increase in 32-bit-only Origin ASNs

over a million, so keeping network equipment runningwere around double those in the previous year. 32-bit-only

may have been a challenge for some. The number of

ASNs now account for 46.7% of all Origin ASNs, and we

unique /64 blocks, meanwhile, only increased by a bit

expect this figure to exceed 50% in 2022. IPv6-enabled

under 5%. The main reason for this seems to be that

ASNs, which advertise IPv6 routes, also rose substantially

58.6% of the total number and 79.1% of the increaseto account for 34.5% of the total. Within this, there was awere routes for which there were other shorter-prefix

notable increase in 32-bit-only ASNs only advertising IPv6

routes, meaning that they do not contribute to the increaseroutes, with 90% of those being APNIC-region ASNs. In

in the number of unique blocks. We can infer that the2022, we will be watching to see if this increase in ASNs

IPv6 rollout on end sites has progressed even further,is a temporary phenomenon amid the IPv6 rollout or if it isbut if unaggregated route advertisements come to dominate

an inevitable trend resulting from the recent surge in IPv4

the increase in the number of routes going forward, thisaddress prices that will continue ahead.

would be a little disappointing.

Table 3: Total Number of Unique IPv4 Addresses in Full IPv4 Routes
and Total Number of Unique IPv6 /64 Blocks in Full IPv6 Routes
Date

No. of IPv4 addresses

No. of IPv6 /64 blocks

Sep. 2012

2,588,775,936

41,097,754,610

Sep. 2013

2,638,256,384

20,653,282,947

Sep. 2014

2,705,751,040

62,266,023,358

Sep. 2015

2,791,345,920

31,850,122,325

Sep. 2016

2,824,538,880

26,432,856,889

Sep. 2017

2,852,547,328

64,637,990,711

Sep. 2018

2,855,087,616

258,467,083,995

Sep. 2019

2,834,175,488

343,997,218,383

Sep. 2020

2,850,284,544

439,850,692,844

Sep. 2021

3,036,707,072

461,117,856,035

Table 4: IPv4/IPv6 Full-Route Origin AS Numbers
16-bit (1–64495)

ASN

32-bit only (131072–4199999999)
Total

(IPv6
-enabled)

IPv4+IPv6

IPv4 only

IPv6 only

Total

(IPv6
-enabled)

125

39026

（14.3％）

264

2565

17

2846

（ 9.9％）

131

40818

（16.4％）

496

3390

28

3914

（13.4％）

34555

128

42088

（17.9％）

868

4749

55

5672

（16.3％）

8228

34544

137

42909

（19.5％）

1424

6801

78

8303

（18.1％）

Sep. 2016

9116

33555

158

42829

（21.7％）

2406

9391

146

11943

（21.4％）

Sep. 2017

9603

32731

181

42515

（23.0％）

3214

12379

207

15800

（21.7％）

Sep. 2018

10199

31960

176

42335

（24.5％）

4379

14874

308

19561

（24.0％）

Sep. 2019

10642

31164

206

42012

（25.8％）

5790

17409

432

23631

（26.3％）

Sep. 2020

11107

30374

229

41710

（27.2％）

7653

19668

574

27895

（29.5％）

Sep. 2021

11465

29219

302

40986

（28.7％）

9514

21108

5242

35864

（41.1%）

Advertised
routeIPv4+IPv6

IPv4 only

IPv6 only

Sep. 2012

5467

33434

Sep. 2013

6579

34108

Sep. 2014

7405

Sep. 2015© Internet Initiative Japan Inc.

5

Topic 2

name resolution on user devices. ISPs can notify users of

DNS Query Analysis

multiple full resolvers, and users can specify which full
resolver to use, and add full resolvers, by altering settings

IIJ provides a full resolver to enable DNS name resolution

in their OS, browser, or elsewhere. When more than one

for its users. Here, we discuss the state of name resolution,

full resolver is configured on a device, which one ends up

and analyze and reflect upon data from servers provided

being used depends on the device’s implementation or the

mainly for consumer services, based on a day’s worth of full

application, so any given full resolver is not aware of how

resolver observational data obtained on October 6, 2021.

many queries a user is sending in total. When running full
resolvers, therefore, this means that you need to keep track

The full resolver provides a name resolution function thatof query trends and always try to keep some processing

serves user queries. Specifically, to resolve a name, it starts

power in reserve.

by looking at the IP address of an authoritative name server
for the root zone (the highest level zone), which it queries,

Observational data on the full resolver provided by IIJ show

and then goes through other authoritative nameservers tofluctuations in user query volume throughout the day, withfind the records it needs. Queries repeatedly sent to thevolume hitting a daily trough of about 0.12 queries/sec

full resolver can result in increased load and delays, so theper source IP address at around 4:20 a.m., and a peak of

information obtained is cached, and when the same query

about 0.30 queries/sec per source IP address at around 9:00

is received again, the response is sent from the cache.

p.m. These are 0.06pt increases vs. 2020 in both cases.

Recently, DNS-related functions are implemented on devices

Comparing the data with the previous year’s, the trends

that lie on route paths, such as consumer-level routers andare not all that different at times during the day, which is

firewalls, and these devices are sometimes also involved inwhen traffic is higher, but the nighttime trend looks to haverelaying DNS queries and applying control policies. Some

shifted, with the number of queries being around 1.8-foldapplications, such as Web browsers, also have their own

higher. This growth is observed across almost all times of

implementations of name resolver functionality and in somethe night, so it may be that automated mechanisms of somecases resolve names without relying on OS settings.

kind (e.g., device control, checks of device active status,
scheduled tasks) are pushing the figures up.

ISPs notify users of the IP address of full resolvers via

6various protocols, including PPP, DHCP, RA, and PCO,

Broken down by protocol (IPv4 and IPv6), IPv4 queries

depending on the connection type, and they enable

per IP address rose vs. the previous year. As noted above,

automatic configuration of which full resolver to use for

automated mechanisms of some kind may be pushing the© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

1. Periodic Observation Report

number of queries up. Changes in implementation and soalmost perfect sync, we surmise that lightweight, quickly

forth may also be factors. Also, the figures on query sourcecompleted tasks of some sort are being executed. For

IPs show there to be more IPv6 query source IPs than IPv4

example, there are mechanisms for completing basic tasks,

ones during the day, while IPv4 query source IPs are moresuch as connectivity tests or time synchronization, before

numerous than IPv6 late at night. The counts are roughly

bringing a device fully out of sleep mode, and we posit thatthe same around 6:50 a.m. and 10:10 p.m. The increase

the queries used for these tasks are behind the spikes.

in IPv4-based queries also affected the overall trend in thenumber of queries. In a departure from the trend up until

Looking at the query record types, A records that query the2020, IPv4-based queries accounted for around 59% of the

IPv4 address corresponding to the host name and AAAA

total, while IPv6 accounted for around 41%.

records that query IPv6 addresses account for around 80%of the total. The trends in A and AAAA queries differ by IP

Recent years have seen a tendency for queries to rise briefly

protocol, with more AAAA record queries being seen forat certain round-number times, such as on the hour marks

IPv6-based queries. Of IPv4- based queries, around 64% arein the morning. The number of query sources also increases,

A record queries and 21% AAAA record queries (Figure 1).

with a particularly noticeable pattern around 7 a.m., which

With IPv6-based queries, meanwhile, AAAA record queries

is possibly due to tasks scheduled on user devices andaccount for a higher share of the total, with around 44%

increases in automated network access that occur when de-being A record and 36% being AAAA record queries (Figure

vices are activated by, for example, an alarm clock function.

2). Compared with the previous year, we observe drops

In the previous year, we noted increases in queries 20, 14,

in A record queries of 15 percentage points for IPv4 andand 10 seconds before every hour mark, but the increase

7 percentage points for IPv6. HTTPS-type records, whichat the 20-second mark was not very discernible in 2021.

we started to see in 2020, accounted for some 11% of

Similar to 2019, the 2021 figures show increases 14 andIPv4 and 18% of IPv6 queries, marking an increase of 9

10 seconds before every hour. At the hour mark, query vol-percentage points for IPv4 and 12 percentage points forume rises sharply and then tapers off gradually, but with

IPv6. The trend in HTTPS record queries appears to be

the sudden spikes that occur ahead of the hour mark, querycorrelated with AAAA records, and HTTPS record queries

volume quickly returns to roughly where it had been. Hence,

tend to come in at roughly half the volume of AAAA record

because a large number of devices are sending queries inqueries across all times of the day.

OTHER

0.34%

OTHER

0.07%

SRV

0.30%

SRV

0.11%

PTR

1.92%

PTR

0.76%

HTTPS

11.15%

HTTPS

18.24%

AAAA

21.57%

AAAA

36.31%

A

64.72%

Figure 1: IPv4-based Queries from ClientsA

44.51%

Figure 2: IPv6-based Queries from Clients© Internet Initiative Japan Inc.

7

■ Traffic

Topic 3

IPv6

Figure 3 shows traffic measured using IIJ backbone routersat core POPs (points of presence—3 in Tokyo, 2 in Osaka, 2

In this section, we again report on the volume of IPv6

in Nagoya). For reasons to do with our system, this edition

traffic on the IIJ backbone, source ASNs, and the main

looks at data for the nine months from the start of 2021 toprotocols used.

September 30, 2021, rather than a year’s worth of data.

Linear total (IPv6)
Linear total (IPv4)
IPv6
IPv4

21
Se

p

1,

20

21
20
1,
g
Au

Ju

Ju

n

l1

1,

,2

20

02

1

21

21
M

Ap

ay

r1

1,

,2

20

02

1

21
20
1,
ar
M

Ja

Fe

n

b

1,

1,

20

20

21

21

(Date)

Figure 3: Traffic Measured on Backbone Routes at IIJ’s Core POPs

2.500
2.000
1.500
1.000
0.500

NormTotal

Norm Total（IPv6）

0

21
Se

p

1,

20

21
20
1,
g
Au

Ju

Ju

n

l1

1,

,2

20

02

1

21

21
M

Ap

ay

r1

1,

,2

20

02

1

21
20
1,
ar
M

Ja

Fe

n

b

1,

1,

20

20

21

21

(Date)

Figure 4: Traffic Indexed to 1 as of January 4

20

(%)

18
16
14
12
10
8
6
4
2
0

Figure 5: IPv6 as a Proportion of Total Traffic

8© Internet Initiative Japan Inc.

21
Se

p

1,

20

21
20
1,
g
Au

20
1,
Ju
l

20
2
1,
Ju
n

21

1

1
ay
M

Ap

r1

,2

1,
2

02

02

1

21
20
1,
M
ar

1,
b
Fe

Ja
n

1,

20

20

21

21

(Date)

Feb.2022

Vol.

53

1. Periodic Observation Report

Possibly on a rebound from 2020’s muted growth, both

■ Traffic Source Organization (BGP AS)

IPv6 and IPv4 traffic volume rose continuously throughout

Next, Figures 6 and 7 show the top annual average IPv6

2021, with particularly strong increases in the latter half ofand IPv4 traffic source organizations (BGP AS Number) for

the year. As the normalized series in Figure 4 (indexed to 1

January 1, 2021 through September 30, 2021.

as of January 4, the first business day of the year) show,
IPv6 traffic was up 1.7x and IPv4 up 1.2x vs. the start ofCompany A retains the top IPv6 spot, with its share of

the year. The graph also shows a lump spanning mid-August

traffic down 3 percentage points vs. 2020 to 11%. The

through early September. During this period, IPv6 briefly

ranking saw a big reshuffling from the No. 2 spot downrose to 2.2x and IPv4 to 1.5x vs. the start of the year.

this time around. Company B in second place with 8% isa major Japanese content company, Company C in third

Figure 5 shows IPv6 as a proportion of total traffic. IPv6

place with 4% is a major US CDN operator, and Company

traffic continues to increase year after year, but it remains

D in fourth place with a 3% share is a major US digital

far below IPv4 in absolute terms. But considering that it

devices maker.

was essentially crawling along the bottom of the graph backwhen we started compiling this report in 2017, we can say

Company B in second place saw traffic rise sharply fromit has grown considerably over the subsequent four years.

around mid-July, rocketing into the No. 2 spot in the spaceWith this being the fifth edition of this report, Table 5 recapsof about two months. One imagines that it embarked on a

the IPv6/IPv4 ratios going back to the first edition.

fairly large campaign to drive the use of IPv6. So with thismajor Japanese content company finally taking real steps

We observed a slowing of growth in 2020 likely attributableto promote IPv6, similar moves by other companies will no

to COVID-19, but the observations this time around confirm

doubt bear close watching ahead.

that use of IPv6 is rising steadily by the year.

Table 5: IPv6 as a Proportion of Total Traffic

IPv6 ratio

IIR Vol. 37, 2018

IIR Vol. 41, 2019

IIR Vol. 45, 2020

IIR Vol. 490, 2021

IIR Vol. 53, 2022

4%

6%

10%

10%

16%

Company A

IIJ

Company C

IIJ

Company B

Company A

Company C

Company B

Company D

Company H

Company E

Company O

Company F

Company P

Company G

Company F

Company H

Company Q

Company I

Company R

Company J

Company M

Company K

Company S

Company L
Other

Company T

Company M

OtherCompany U

Company N

Company W

Company V

Figure 6: Annual Average IPv6 Traffic by Source Organization(BGP AS Number)

Figure 7: Annual Average IPv4 Traffic by Source Organization(BGP AS Number)© Internet Initiative Japan Inc.

9

■ Protocols Usedon source ASNs, major content company and CDN operator

Figure 8 plots IPv6 traffic according to protocol number

support for IPv6 is progressing, so it’s reasonable to think(Next Header) and source port number, and Figure 9 plotsthat individual use (games and entertainment) is growing.

IPv4 traffic according to protocol number and source port
number (for the week of Monday, October 4 – Sunday,■ Summary

October 10, 2021).

Rebounding from the 2020 doldrums, both IPv6 and IPv4
traffic grew substantially in 2021. IPv6 is also growing

The composition of protocols used is largely the same assteadily as a proportion of total traffic, and breaking through2020 for both IPv6 and IPv4. Not much has changed, but

the 20% barrier is certainly within the realm of possibility for

TCP 80 continues to decline, and it looks like the transition

2022. A major Japanese content company appears to havefrom HTTP to HTTPS or QUIC is progressing. QUIC wasstarted using IPv6 in earnest, and the level of IPv6 support

officially enshrined in RFC 9000 in May 2021, so use of theamong CDN operator traffic also seems to be rising.

protocol will no doubt grow even more ahead.
As a Japanese ISP, we will continue to keep an eye on
industry trends in the hopes of seeing a second and thendoubled vs. 2020, with particularly high growth froma third major Japanese content company start supporting

evening through into night. As mentioned in the discussion

IPv6 in earnest.

02
,2

21

10
ct
O

ct

O

O

ct

9,

20
8,

,2
ct
7
O

20

21

1
02

1
02
,2
ct
6
O

O

O

ct

ct
5

4,

,2

20

02

1

21

1

We cannot show absolute traffic levels, but IPv6 essentially

TCP443

UDP443

ESP

TCP80

UDP4500

UDP53

IPv4

TCP873

L2TP

TCP110

TCP54321

UDP6881

TCP17363

TCP995

UDP51413

TCP51413

TCP8080

TCP19595

UDP19305

Total

TCP993

02
10
O
ct

9,
ct
O

TCP443

TCP80

UDP443

ESP

UDP4500

IPv4

UDP8801

TCP8000

TCP8080

GRE

UDP5008

UDP3480

UDP4172

TCP183

UDP123

UDP23233

TCP44445

UDP9000

TCP110

Total

Figure 9: Breakdown of IPv4 Traffic by Source Port Number

10

,2

21

20
O

O

ct

ct

8,

7,

20

21

21
20

21
20
6,
ct
O

O

O

ct

ct

5,

4,

20

20

21

21

1

Figure 8: Breakdown of IPv6 Traffic by Source Port Number© Internet Initiative Japan Inc.

UDP12222

Feb.2022

Vol.

53

1. Periodic Observation Report

Topic 4the user volume forecasts, we look at what bandwidth weState of the Mobile Industry and Traffic Trends

need on the interconnection with the MNO and take steps
to deal with the speed with which users are set to increase

The Suga administration’s call for cuts to mobile phoneor decrease. To get more precise predictions when releasing

charges has driven the mobile communications industry to

GigaPlans, however, we monitored the number of GigaPlans

review its service offerings in various ways over the pastuser applications closely to determine a figure for the speed

year. IIJ itself released IIJmio Mobile Service GigaPlans

of user growth. As a result, we experienced no major quality(“GigaPlans”) on April 1, 2021. As a service provider, weissues when GigaPlans was released.

pursue a range of strategies oriented around attracting as
many subscribers as we can, but in this section I discuss

Once GigaPlans was released, our focus turned to building

how we responded in terms of equipment.

the sort of environment that would not end up being
a bottleneck as the services are expanded and enhanced.

From an equipment perspective, our goal was to hold down

What we did was review what the PGWs (packet data

costs as much as possible while maintaining consistent

network gateways), which interconnect with the MNO,

quality. Two key considerations when it comes to quality

were accommodating. IIJ itself has several PGWs runningare whether and by how much user numbers will rise or fall

and providing redundancy. We also had plans to add

and how soon we need to react to that forecast. Normally,

additional PGWs even before plans for the GigaPlans

we also need to take traffic trends (e.g., what time does theservice appeared, so preparations were already underway,

peak come each day?) into account, but here we are talking

but the release of GigaPlans prompted us to work quickly

only about IIJmio subscribers, so traffic trends don’t enterto accommodate IIJmio.

into it.
User volume forecasts are provided by the department
responsible for setting up the service. Usually, based on

Interconnection point

IP network,
eNB

SGW

backbone, etc.

PGW, PCEF

MNO’s equipment

Mobile network
Authenticationequipment

PCRF

OCS

MVNO’s equipment
Figure 10: Connection Sequence from Mobile Phone to MVNO Network© Internet Initiative Japan Inc.

11

Around June 2021 (near the middle of Figure 11), we

We also added a new PGW to accommodate IIJmio

added a PGW to accommodate IIJmio. As a result, trafficaround October 2021 (near the right end of the graph).

volume increased about 1.4x. We believe this was not

This expansion was equivalent in scale to the one in June,

simply due to the addition of a PGW but also due to theand traffic rose about 1.3 fold vs. immediately before thefollowing factors.

addition. Here, we believe this reflects an increase in the
PGWs’ service accommodation efficiency rather than the• The addition of accommodating PGWs increased capacity

impact of the rise in user volume immediately following

to handle IIJmio processing

the GigaPlans release.

• The number of users increased due to the GigaPlans
releaseWith mobile equipment, the extent to which you can keep

• The data limits set on GigaPlans resulted in an increase incosts down while still providing consistent quality of

traffic per user

service is key. Looking ahead, we will consider a range
of approaches as needed as we work to enhance mobile
service quality.

Bandwidth

That said, we were surprised to see traffic rise so starkly.

Jan 1

Feb 1

Mar 1

Apr 1

May 1

Jun 1

Jul 1

Aug 1

Figure 11:IIJmio Internet-bound Traffic Volume (Jan–Oct 2021)

12© Internet Initiative Japan Inc.

Sep 1

Oct 1

(Date)

Feb.2022

Vol.

53

1. Periodic Observation Report

Topic 5The Tokyo Olympics took place in 2021, so I would also

IIJ Backbone

like to look at IIJ’s traffic figures during the event. The
Games were postponed from 2020 because of COVID-19,

In the previous edition, we took a look at the situationand it was an unusual affair in that most of the events,

before IIJ fully deployed ROV using RPKI. We subsequently

including the opening ceremony, were held withoutcompleted the ROV rollout on IIJ’s external connection

spectators in attendance. These sorts of huge events arepoints as scheduled in December 2021, so roughly a year

major considerations even for communications carriers

has now passed.

not directly involved in them. Naturally, we have to beprepared for all sorts of incidents, but the availability ofComparing what we observed via IIJ’s ROA cache server

easy access to rich content in recent times means thatin October 2020 and October 2021, the number of unique

we also need to watch out for trends in traffic that differ

prefixes registered was up 1.5x for IPv4 and 1.75x for IPv6.

from what we see normally as well as sudden fluctuations

The number of unique Origin ASNs registered was also up

in traffic. There is no clear indication as to how much of

1.49x for IPv4 and 1.4x for IPv6. So we are seeing a steadyour infrastructure we will need to have ready, so we haverise. Comparing the number of unique ASNs registered in

to make capacity available based on estimates derived

ROAs as a proportion of Origin ASNs observable on IIJ’s

from the usual traffic trends. On top of that, with these

BGP routes, IPv4 was up around 11% and IPv6 up around

Games, there was a string of announcements right before

1.8% vs. 2020. Meanwhile, looking at the AS list managed

the opening date about the decision to hold events without

by JPNIC (https://www.nic.ad.jp/ja/ip/as-numbers.txt), we

spectators and to cancel public viewings. So we anticipated

see that although the number of ASNs registered as ROAs

an increase in the number of people viewing the Games

as a proportion of the JPNIC-managed ASNs that can beon TV at home, or via live feeds on the Internet, and soseen on IIJ’s BGP routes did increase vs. 2020, it still only

forth, so we set about preparing for that. In IIJ’s case, thissits at around 15.5%. IIJ has rolled out ROV, and it isinvolved increasing the capacity of some of the equipment

being progressively deployed across the globe as well, sothat we already had at the ready ahead of the Games andoperators that have Internet addresses and other resources

coordinating with other (non-IIJ) ISPs.

can see the effect simply by registering ROAs, and wethus look forward to seeing more and more registrations inJapan going forward as well.

© Internet Initiative Japan Inc.

13

Here, we present graphs overlaying the traffic data forand 13). The data for the period of the Games (July 21 –each of the five weeks from July 11 to August 14, 2021,

August 8) are plotted as solid lines; the remainder of thespecifically looking at Tokyo/Osaka traffic as well as traffic

data appear as dotted lines.

on external connections with IIJ within Japan (Figures 12

0
0：00

12：00

0：00

12：00

0：00

12：00

0：00

12：00

0：00

12：00

0：00

12：00

0：00

12：00

0：00

Aug 9–14, 2021
Aug 8, 2021
Aug 1–7, 2021
Jul 25–31, 2021
Jul 22–24, 2021
Jul 18–21, 2021
Jul 11–17, 2021

Figure 12: Tokyo/Osaka Traffic

0
0：00

12：00

0：00

12：00

0：00

12：00

0：00

12：00

0：00

12：00

0：00

12：00

Figure 13: Domestic Interconnection Traffic

14© Internet Initiative Japan Inc.

0：00

12：00

0：00

Aug 9–14, 2021
Aug 8, 2021
Aug 1–7, 2021
Jul 25–31, 2021
Jul 22–24, 2021
Jul 18–21, 2021
Jul 11–17, 2021

Feb.2022

Vol.

53

1. Periodic Observation Report

As the graphs show, we saw no major increases or decreases

We made all sorts of preparations heading up to thein traffic during the Games. The patterns are not all the

Olympics, including the ROV rollout mentioned above, butsame, but there was no major deviation from normal traffic

looking back now that it’s all over, it was a fairly calm

levels in the daytime, while traffic during the usually busy

period with no huge changes in traffic from the norm and6:00 – 11:00 p.m. period was a few percent below its usual

no major incidents or faults during the Games. We plan tolevel. Tokyo/Osaka traffic during the 8:00 p.m. timeslot

continue expanding our infrastructure to ensure a stable

when the opening ceremony took place on July 23 wasuser experience, not just during events like this, but at all

around 17% lower than during that time on the same day oftimes.

the week before. It may be that normal Internet usage fellduring this timeslot with a few more people watching the
opening ceremony on TV.

1.BGP and Routes
Tomohiko Kurahashi
Technology Development Section, Operation Technology Department, Infrastructure Engineering Division, IIJ

2.DNS Query Analysis
Yoshinobu Matsuzaki
Technology Development Section, Operation Technology Department, Infrastructure Engineering Division, IIJ

3.IPv6
Taisuke Sasaki
Deputy General Manager, Network Technology Department, Infrastructure Engineering Division, IIJ

4.State of the Mobile Industry and Traffic Trends
Tsuyoshi Saito
Mobile Technology Manager, Network Technology Department, Infrastructure Engineering Division, IIJ

5.IIJ Backbone
Fumiaki Tsutsuji
Network Planning Manager, Network Technology Department, Infrastructure Engineering Division, IIJ© Internet Initiative Japan Inc.

15

2. Focused Research (1)Running an Overlay Network in a Multi-tenant Setup—Challenges of IIJ GIO Infrastructure P2 Gen.2
2.1 P2 Gen.2—IIJ’s new-generation IaaS

long as customers want to use them. The development of

On October 1, 2021, IIJ launched IIJ GIO Infrastructure P2

new systems based on cloud technology under the banner

Gen.2 (P2 Gen.2 for short), a new IaaS that fully combinesof DX (digital transformation) is on the rise. But maintaining

and takes advantage of the characteristics of the public

and modifying existing systems to support DX is also a

IaaS and private IaaS developed and provided under the IIJ

key mission for enterprise IT teams. We developed P2

GIO brand. P2 Gen.2 is a new-generation IaaS designed toGen.2 to address this challenge by providing a third op-

confer the full benefits of public clouds while also taking

tion for cloud computing that can not only accommodate

advantage of VMware vSphere-based private clouds.

on-premise systems and private clouds that remain in
operation but also make it easy to migrate from public

To ensure customers can continue using P2 Gen.2 into theclouds as well as.

future, we adopted an architecture geared toward the next
generation of services. This report looks at the development

P2 Gen.2 is an IaaS offering that allows customers to useof the P2 Gen.2 service and the ideals and reality of its

resources in the form of flexible server resources. The

operation in the field with a focus on the network.

resource pool is virtualized, customers focus on system
operations within the resource pool, and IIJ runs all of the16

Our aim in developing and running P2 Gen.2 is to provide afunctions in the layers below it. With IaaS run in individual

service platform that will continue to be useful a decade or

VM instances, the size of the VM is made to conform tomore from now. Today’s rapidly changing world makes it

the instance model specified by the cloud service provider,

impossible to predict what will happen 10 years ahead. Yet

but with P2 Gen.2, customers are free to create VMs in

we believe that services should remain available for as

the resource pool. This means that customers can transfer© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

2. Focused Research (1)the specification of the machines running in their own

layer from users. This made it possible to offer resource

environments as is. Naturally, this design makes it possible

control permissions that are as flexible as with vSphereto bring in images or migrate VMs from the existing

while allowing IIJ, as service provider, to take on the hy-

environment as is via V2V and P2V migrations.

pervisor and hardware lifecycle management role. Defining
a new joint-responsibility model like this made it possible

P2 Gen.2 provides an image provision mechanism forfor IIJ to manage and operate the hypervisor network.

deploying VMs; backup, network, and migration functions; andWe adopted VMware NSX-T Data Center (NSX-T), which

file servers for storing files. The hypervisor and hardware

enables integration with vSphere, to run the hypervisor

such as servers, storage, and networks are abstracted inlayer network efficiently, and this significantly improved

a form that customers are able to visualize, so they do not

the IaaS network. With P2 Gen.2, in Layer 3 an NSX-T

need to worry about these elements. This eliminates theoverlay network sits on top of an IP fabric underlay net-

workload encountered when renewing the hypervisor andwork, and each tenant’s network is completely separated

hardware that results from handing ESXi directly over tofrom the others and provided as a VPC (Virtual Private

the customer, something that had been a major issue with

Cloud). Combining this with the operational knowledge inthe Virtualization Platform VW Series, a vSphere-basedlarge-scale server pools we amassed through IIJ GIO, we

service provided on IIJ GIO.

have made it possible to allocate resources to users in amanner independent of the allocation of physical computing

We adopted VMware Cloud Director (vCD), a VMware prod-

resources (CPU, memory, storage) (Figure 1).

uct for service providers, and hid the hypervisor (vSphere)

Customersite
IIJ Private
Backbone Service

Internet

Gateway
Customer

VPC

LB
VM
Virtual server

VM
VM

VM
VM

Virtual server resource pool
Virtual CPU / memory
Virtual storage

IIJ GIO

Hypervisor VMware
Physical server resource pool
Physical storage resource pool
Physical network resource pool

Figure 1: Flexible Server Resources Overview© Internet Initiative Japan Inc.

17

2.2 Overlay Networks Using SDN Technology

it possible to manage the configuration centrally. This

Overlay networks are nothing new. Along with VLAN

creates an environment in which administrators can auto-(Virtual LAN), they are a long-standing network virtualization

matically configure devices according to how the network

method. In the past, they were often used for point-to-point

is defined, instead of having to change the configuration ofconnections due to protocol specifications/constraints andeach individual device, so configurations can be changed

for operational considerations, but with the evolution ofquickly to meet the requirements of large-scale networks.

SDN (software-defined network) technology, they can now

This makes it possible to have more than 4,096 network

be used in large-scale networks. Encapsulation protocols

segments, the limit imposed by IEEE 802.1Q VLAN,

like L2TPv3, VXLAN, and Geneve are used to build virtual

something that has posed issues in large-scale networks

L2 networks. SDN technology uses software to control thethat use many segments. It also enables the efficient

networks, and is made up of the control plane, which

control of north-south traffic (traffic between devices andcontrols what happens between devices, and the data

the network center) and east-west traffic (traffic among

plane, which forwards the actual frames and packets. Using

devices) within virtualized networks (Figure 2).

software to control what happens between devices makes

External

Overlay Network

Edge Leaf

North-South

Spine

Leaf

East-West

Figure 2: Overlay Network in a Large-scale Environment

18© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

2. Focused Research (1)2.3 Benefits of Overlay Networks with VMware
NSX-T

IaaS solutions need to be multi-tenant (accommodate

Along with server virtualization technology, network

via software is also needed in order to provide functions

virtualization with overlay networks is one of the core

separately to the tenants and to maintain security andtechnologies underpinning the P2 Gen.2 infrastructure.

the specified service level. Network virtualization makes

Service networks can be configured separately from theit easier to link together the other components of theunderlay network that lives on the hardware layer. Using

IaaS solution as well as monitoring systems, facilitating

NSX-T for network virtualization on the IaaS layer of P2

proper control of a multi-tenant environment in which

Gen.2 provides a flexible network configuration and loose

each tenant operates their system in accord with theircoupling with the physical layer. When setting up networks

system requirements. NSX-T also implements VRF (virtualwith a view to expansion from minimum scale up to

routing and forwarding), and VRF can be used to enhance

hyperscale, they tend to become complicated in terms

equipment efficiency. In multi-tenant environments, theof physical device specs because design and sizing need

need to divide up routing domains among the tenants posed

to be chosen in anticipation of that expansion. But beinga resource constraint. VRF makes it possible to increase

able to control expansion via overlays confers considerable

infrastructure consolidation.

multiple customers). Centralized control and automation

benefits on the operational front.
Interface control is centralized into NSX Manager, NSX-T’s
An IaaS also needs to handle multi-site configurations

management component, making API-based operations andacross multiple data centers. Using infrastructure acrossautomation easy. This also helps to reduce development

multiple, physically separated sites facilitates disasterhours spent on making the service’s backend applications

preparedness and makes it possible to obtain the necessary

work together.

computing resources without needing a large-scale
facility. Virtualizing the network between sites and making

When an overlay network is set up as an IaaS solution,it an overlay network enables you to configure networks

network exits are needed. NSX-T also cannot communicate

independent of their location. Physically separated

outside the NSX-T domain (under its control) with theenvironments involve more considerations to take into

NSX-T encapsulation protocol Geneve, so to communicate

account than with single-site configurations, such aswith the external world, you need to convert it to a VLAN.

network latency and bandwidth between sites, and thisWe had not fully automated external connections withincreases design and operational difficulty, but ensuring

IIJ GIO, and a major issue was that users were unable toproper resource monitoring and control of systems basedchange network configuration in a complete form whenon the monitoring data makes it possible to set up flexible

migrating systems. P2 Gen.2 improves this situation by

networks.

automating route configuration for connections with the
IIJ Private Backbone Service. NSX-T can also be run in a

Network virtualization remains uncommon despite thesebare metal configuration with NSX Edge installed directly

benefits. With traditional networks where the configuration

on a server with no virtualization, and while this is finewas expected to remain static, administrators were ablein performance terms, it is not really up to the task of

to continue maintaining and running systems without themulti-tenant configurations. Many challenges also present

benefits of network virtualization. In recent years, however,themselves particularly with external connections, suchthere is a need for IT infrastructure as a whole to adapt

as downtime during software upgrades. So many necessary

flexibly and swiftly to rapidly changing and unpredictableareas of improvement remain, and we continue to workrequirements. Network virtualization will likely become more

on resolving these issues.

and more important in this landscape.

© Internet Initiative Japan Inc.

19

2.4 Operational Issues and Solutions

conventional network engineers and server engineers—

Ahead of the October 1, 2021 release, in 2020 we conducted

and instead, the engineers need to be familiar with thea small-scale rollout of the NSX-T SDN-based overlay

entire technology stack used in building the infrastructure.

network, along with NSX-T’s NFV (network function

When we started out, our team was small but included

virtualization) capabilities, for some data center functions.

people with a strong knowledge of VMware technologies

This revealed a whole host of issues, which we continue

and people with experience running large-scale service

working to address as part of our operations. Below, we

infrastructure, so as a team, we had the capabilities togo over the issues we discovered and their solutions.

handle ongoing development and operations. The mostimportant part of getting engineers with differing areas of20

Creating an overlay network using SDN requires a moreexpertise to work both autonomously and in collaboration

complicated configuration than with conventional networks.

with each other as a team is to have a vision for yourThe configuration has two layers: the physical layer (under-

products and services and strive continuously to makelay) and the IaaS layer (overlay, logical layer).

them better based on the DevOps philosophy.

Another point is that to get the most out of the NSX-T

With P2 Gen.2, we revisited infrastructure control fromarchitecture, you need to be well-versed in vSpherescratch to ensure it would work the way we needed it

operations, so in addition to networking, the engineers

to. We put as much effort as possible into streamlining

also need to be able to work with server, hypervisor,

operations and automation. Under DevOps principles, weand virtualization technologies. This is a departure frombuilt a Git-based CI/CD pipeline, and the engineers

the past practice of specializing in specific areas—à lacontrol the majority of elements that make up the IaaS,© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

2. Focused Research (1)not just the VMs, under a policy of making changes to thephysical equipment involves delivery lead times, andoperations and configuration of the infrastructure without

setting up physical equipment in the data center is notdirectly manipulating the environment (Figure 3).

something that you can do via software control alone.
Configuring facilities involves some level of physicalWhile there is a lot of knowledge and software out there

human interaction, so making sure the associated leadfor controlling VMware products, the de facto standard

times do not become a bottleneck is key. Having an

in server virtualization, it is not quite adequate for totalabundant inventory of resources can solve this issue, but

control of multi-tenant infrastructure in the form of athis then leads to inefficiencies because the overall

SDDC (Software-Defined Datacenter), so some of theresource utilization rate falls. Also, out-of-resources

tools and libraries we use were developed from scratch.

conditions are unacceptable in cloud services. So resource
management, monitoring, and demand forecasting are

We are not only simply carrying out operations tasks

crucial. Monitoring and demand forecasting should bethrough these efforts. We have also laid the groundwork

done not just on the overlay, it should also aggregate

for implementing SRE (site reliability engineering).

and facilitate visualization of the various underlay data
sources, including servers, network equipment, and storage.

Taking on board agile values and​​methods like this canSince the October 1, 2021 release, we have been

facilitate more flexible and agile infrastructure operations,collecting and analyzing data on actual usage conditions

which can otherwise tend to be fixed and inflexible, but

to enable just-in-time resource inventory deployments

you also need to be keenly aware of the opposing relationshipbased on demand forecasting.

between agile principles and facility operations. Procuring

Figure 3: The IaaS Components are Managed via Git© Internet Initiative Japan Inc.

21

2.5 Looking Ahead

will allow IIJ to deliver high-quality services on edge nodes

P2 Gen.2 still only has the minimum functionality, and wecloser to where the customer is actually located. In addition

plan to continue developing it to serve customer needs. The

to conventional networks, 5G mobile networks and the likeuse of computing resources in the form of cloud-native IT

can also be used to connect the resources. Linking the coresystems has become commonplace, and a more agile digitaland the edges requires not only VMware-based workflow

infrastructure is now being demanded. Our first priority is

management but also application container-based resource

to implement the features needed to ensure that customers

management, so also providing resources in the form of

will continue to use the service, but we also want to be able

Containers as a Service (CaaS), allowing applications to be

to address future needs beyond that.

run inside containers, might be the optimal solution forcustomers (Figure 4).

In addition to migrations into the cloud, the need for edge
computing and distributed computing that deploys computing

Software control that brings the network closer to theresources close to where users are is also rising. IIJ’s role isstatus of an application is needed here. This goes beyond

to connect the cloud at the core, the distributed edge nodes,

just resource services that provide VMs, and I think it canand the network in between. These interconnected elements

be achieved by evolving the network from the current

Autonomous driving
and connected cars

5G
network
APP

loT gateway

APP

APP

APP

East Japan region
data center cluster

Shiroi DCC

APP

Factory loT

5G
network

Industrial devices

Smart cityloT gateway

5G
network

APP

APP

Existing
IP network

Existing GIO sites
Figure 4: Looking Ahead

22

DR services

IIJ Backbone Network(Wide area / secure)© Internet Initiative Japan Inc.

West Japan region
data center cluster

Matsue DCP

Feb.2022

Vol.

53

2. Focused Research (1)architecture into something that is even less dependent

To make this sort of world a reality, we are also working

on physical resources. This will be difficult to accomplish

on a project to use the technology stack we have developed

simply by deploying resources in a compact form that can

not just for data center purposes but also as an edge

also be deployed on the edges. You also need to be able

computing platform. We have set up a micro data centerto configure the network flexibly so that it works together

outdoors at IIJ’s Shiroi Data Center Campus, and we arewith the other resources.

conducting tests in the area of cloud and edge computing
as well as multi-access edge computing (MEC), with our

Personally, I am interested in SmartNICs as a technology

objective being to develop the technology further along

suitable for this sort of edge computing and distributedwith local 5G, IoT, and other use cases. My hope is thatcomputing architecture. VMware is working on Project

evolving network services in this way will make it possible

Monterey, which runs ESXi on Arm-processor based

to provide customers with unique infrastructure of theSmartNICs. The hope is that managing SmartNICs as NSX-T

sort that only IIJ can deliver.

domains will make it possible to control and deploy networks
in a manner that makes them equivalent to applications.

Takehiro Yamamoto
Infrastructure Service Design Manager, Cloud Services Division 1, IIJ System Cloud Division.
Mr. Yamamoto is engaged in the planning, development, and operation of cloud-related services.

© Internet Initiative Japan Inc.

23

3. Focused Research (2)In Pursuit of Carbon Neutrality in the Data Center

3.1 Introduction

are effectively impacting climate change rather than simply

Over 120 countries have declared their intention to achievemechanically disclosing data, and these developments alsocarbon neutrality—achieve zero greenhouse gas (CO2,no doubt create an incentive for carbon neutrality.

methane, N2O, chlorofluorocarbons [CFCs]) emissions—by 2050, a goal of the international climate change treaty

Data center carbon neutrality will be achieved via two

known as the Paris Agreement. The Japanese government

avenues: saving energy by using it more efficiently and using

also declared its 2050 carbon neutrality goal in Octoberrenewable energy not tied to CO2 emissions. Matsue Data

2020. And in the Green Growth Strategy it unveiled inCenter Park (Matsue DCP) and Shiroi Data Center Campus

December 2020, it set goals for a wide range of industries,(Shiroi DCC), which IIJ built and operates, feature highlywith the following goals being laid out for data centers.

energy-efficient equipment and outside-air cooling systems
the technology for which was developed and tested with

• By 2030, all newly constructed data centers should
consumer 30% less energy and be partly powered byhigh energy-saving goals in mind, and the resulting energy
savings are reducing IIJ’s greenhouse gas emissions.

renewable energy• Aim for carbon-neutral data centers by 2040

In addition to such energy savings, we also plan to makeincreasing use of renewable energy ahead by setting up

Data center carbon neutrality is to be achieved primarily bycarbon-neutral data centers so that renewable energy

reducing CO2 emissions from power consumption to zero.

generation systems and the data center can work in concert

This will entail costs including for the purchase of renewablewith each other.

electricity, technology development, and capital investment,24

but achieving carbon neutrality early will help companies

Below, we go over our energy-saving initiatives at Matsue

differentiate from peers and enhance service value. Carbon

DCP and Shiroi DCC, where we have achieved strong

neutrality is also starting to appear among procurement

energy-saving performance, and we round out the discussion

criteria for products and services, and equity markets areby presenting a reference model for carbon neutrality in thecalling on companies to disclose information on how they

data center.

© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

3. Focused Research (2)3.2 Outcomes at Matsue Data Center Parkand introduce new systems that consume less power.

On April 26, 2011, we opened Japan’s first commercial

Based on past demonstration testing, IIJ determined thatmodular data center with outside-air cooling systems,

outside-air cooling systems that use outside air to reduce

Matsue Data Center Park in Matsue-shi, Shimane Prefecture.

power consumption and do not require cooling towers andMatsue DCP features IZmo units, IT modules developed by

the like would be suited to its next-generation data centers.

IIJ and imbued with its extensive data center operational

Outside-air cooling, however, involves the exchange ofknowhow. To expand into more enterprises in Japana large volume of air, so the intakes and outlets need toand abroad and tap into new markets, we also operate

be installed directly on the server rooms. This raises a

co-IZmo/I modular data centers, which use indirect

number of difficult problems to do with building structure

outside-air cooling systems. Matsue DCP has been in stable

when seeking to install such systems on existing buildings.

operation since being opened as the core facility for IIJ’s

So we developed the IZmo IT module, in which the air

GIO cloud services, and with 90% of the server-hous-

ducts are integrated with the server room.

ing containers now installed, the total number of servers
running is in the tens of thousands. Site 2, added next

PUE (Power Usage Effectiveness), a metric developed by theto Site 1 in 2013, introduced co-IZmo/I units in addition

Green Grid, is often used as a measure of how effectively ato IZmo and uses a three-phase four-wire configuration

data center uses power. Figure 2 shows the PUE equation.

to power its servers and other IT equipment, reducing
power distribution losses and thus saving energy (Figure

Theoretically, the smallest (best) PUE score is 1.0, which

1). Data centers generally house large-capacity electricaloccurs when zero power is consumed by air conditioning

equipment and air conditioning systems to create anequipment etc. Figure 3 shows five years of measurements

environment in which large numbers of servers and otherfor the IT-module-only metric pPUE (partial PUE, ignores

IT equipment can be installed efficiently. IT equipment

energy losses from shared resources etc.).

is the biggest consumer of data center power, but air
conditioning systems are right behind it. So we needed to

The summer pPUE readings are higher than for otherrethink the use of conventional air conditioning systems

seasons because the air conditioning modules run in a

Total facility energy (energy used by IT equipment + energy used by air conditioners etc.)

PUE =Energy used by IT equipment

Figure 2: The PUE Equation

Site 1
1.5
1.4
1.3■Average
■2020
■2019
■2018
■2017
■2016

1.2

Site 2

1.1
1.0

Figure 1: Matsue Data Center Park Site 1 and Site 2(Photo of Matsue Data Center Park taken around 2019)

Apr.

May

Jun.

Jul.

Aug.

Sep.

Oct.

Nov. Dec.

Jan.

Feb.

Mar.

Figure 3: Annual pPUE Scores for the Past Five Years© Internet Initiative Japan Inc.

25

high-power-consumption mode that does not use outside air.

centers, the servers had to be unpacked one by one,

In comparison, spring and autumn pPUE readings are aroundinstalled in a rack, and wired up. But with containerized

1.1 because the modules run in a low-power mode that uses

data centers, the servers are installed into IT modules at

outside air. In winter, the modules mix IT equipment exhaustthe server factory, and the IT modules are transported byand outside air to achieve the right temperature, and sincetruck for installation at the site. This eliminates box andthis mode also consumes less power, pPUE is around 1.1.

packaging waste, makes better use of resources and is

The average over the past five years is 1.18.

better for the environment, and helps reduce CO2 emissions
from equipment transportation. Recently, when replacing■ IZmo and co-IZmo/I

IT equipment, instead of removing the entire container,

The IZmo IT module is a non-building container conforming

operators can opt to remove all of the server racks

to technical advice issued by Japan’s Ministry of Land,

from the container and take them to the server factory,

Infrastructure, Transport and Tourism in March 2011.

where the new equipment is installed in the racks so

The modules are easy to set up because there is no need

they can then be reinstalled in the containers.

to apply for building permission. With conventional data

PUE score
2.2
Conventional data center PUE

2.0
1.8
1.6

1.483

1.4

PUE of data centerwith high-efficiency equipment

1.426

1.2

1.240

1.209

1.210

1.237

2017

2018

2019

2020

1.0
0.8
0.6
0.4
0.2
0
2015

2016

Figure 4: Annual Average PUE for Matsue DCP as a Whole

Figure 5: IZmo and co-IZmo/I

26© Internet Initiative Japan Inc.

(year)

Feb.2022

Vol.

53

3. Focused Research (2)IZmo uses direct outside-air cooling and automatically

voltage (low current). It is widely used in Japanese factories.

switches between three modes depending on outside air

One of the three 400V wires from the UPS can be isolated

temperature and humidity. It controls the temperature atalong with the neutral wire to pull out a 230V supply

the intake port within the IT module to meet thewithout a transformer. This is common in data centers

recommended temperature and humidity conditions set byoverseas where 100V is not required, but the common

Ashrae (American Society of Heating, Refrigerating andneed for 100V in Japan means that, until now, transformers

Air Conditioning Engineers) in 2008.

have commonly been installed in three-phase three-wire
UPSs to transform power from 400V to 100V. Since servers

co-IZmo/I uses indirect outside-air cooling. In contrast

can operate at any voltage from 100V to 230V, andwith direct outside-air cooling, outside air is not channeledbecause IIJ uses a lot of servers for its cloud infrastructure,

directly into the container, and instead, internal heat is

we chose a three-phase four-wire system, eliminating theexpelled indirectly via a heat exchanger, so this method

need for transformers.

can be used even when outside air conditions are poor.
We have also exported some of these modules internationally.

Given that power (W) = current (A) x voltage (V), a

See Figure 5.

reduction in wire size means that, for any given load
capacity (W), the higher the voltage (V), the more you canTheoretically, IZmo, which uses direct outside-air cooling,

reduce the current (A). Specified currents for electricalis more efficient (lower pPUE) than co-IZmo/I with its indirect

wire (the maximum current it can carry) are set according

outside-air cooling (pPUE value is low), but as Figure 6

to the type of insulating sheath and the size of the wireshows, the fine-tuning of control mechanisms and the like(mm2). So at lower specified currents, wires with a lower

make it possible to run co-IZmo/I modules at pPUE scorespermissible current can be used, and the lower a wire’s

similar to those of IZmo.

permissible current, the smaller its diameter, which canhelp reduce the investment cost. Further, voltage drop

■ Three-Phase Four-Wire System

can be reduced because three-phase four-wire systems

At Site 2, we introduced a three-phase four-wire unin-are even less susceptible to voltage drop than three-phaseterruptible power supply (UPS) system, a first among

three-wire systems, in which the voltage drop decreases

Japanese data centers (Figure 7). Three-phase four-wireas the current decreases. At Matsue DCP, this allows for

power transmission involves three wires for three-phasea lower transmission current between the UPS output

AC plus a grounded neutral wire. It is more efficient thanand the server input, and it reduces losses because no

single-phase AC, uses a lower current, and conductortransformers are used. This results in a theoretical(wire) size can be reduced by transmitting power at highreduction in losses of around 25%.

*Assuming a Y configuration is used
1.3

400V

400V÷√3 230V

400V

230V

1.195

1.2
1.163

1.161

1.159

1.168 1.161

1.179

230V

1.168

1.1

UPS
■co-IZmo/I
■IZmo

1.0
2016

2017

2018

2019

2020

(year)

Figure 6: Comparison of IZmo and co-IZmo/I pPUE Scores

400V equipment(UPS etc.)

Ground230V equipment(Servers etc.)

Figure 7: Y-connection Overview of Three-phase Four-wire System© Internet Initiative Japan Inc.

27

■ Environmental Initiatives

Factory. We have appointed an Energy Manager and formu-

In terms of environmental initiatives, we have ISO 14001

lated a medium-term plan, and we implement and evaluate(EMS, Environmental Management Systems) certificationenergy-reducing initiatives based on the facility’s energy

and work to comply with energy conservation laws andusage every year. As the facility has many air conditioning

regulations.

systems, we also comply with Japan’s CFC emissions law
by measuring the amount of CFC gas leaked and identifying

Matsue DCP operates in accord with its ISO 14001

leaks and their causes through inspections and such.

certification, acquired in 2013. We set annual targets forenvironmental initiatives. Our past efforts have included

As Matsue DCP’s electricity usage increases by the year,

going paperless, reviewing the air conditioner temperature

efforts to save more energy and further reduce CO2

settings within buildings and operations rooms, and reducing

emissions are becoming increasingly important. We will

pPUE by 0.01 per year. Using power more efficiently is

continue to focus on environmental initiatives including

one thing, but the EMS certification also requires us tothose to reduce energy consumption, use renewableaddress the risk of the environmental impact on lives andenergy, and install renewable-energy equipment.

ecosystems from atmospheric pollution caused by fires
and the impact of soil pollution caused by the leakage of

3.3 Initiatives at Shiroi Data Center Campus

fuel from emergency power generation facilities. We have

We opened Shiroi Data Center Campus on May 1, 2019

therefore prepared response manuals for emergencies

in Shiroi-shi, Chiba Prefecture (Figure 8). Shiroi DCC issuch as fires, earthquakes, and equipment accidents, anda large-scale data center designed to cope with the explosive

we regularly conduct response training sessions, environ-

growth in data center demand for 5G, IoT, AI, cloud

mental management education and training sessions, andservices, and the like. It brings together the data center

environmental compliance assessments. Matsue DCP is intechnologies and knowhow that IIJ has developed or

an area that experiences relatively few natural disasters

evaluated to address issues with data center operations

and earthquakes, but we strive to raise environmental

in the past. It is also a system module-type data center

awareness through regular education and training.

where we can actively roll out new technologies.

On the topic of environment, energy, and CO2 emissions,

Shiroi DCC draws on the successes of Matsue DCP and

we have compiled data on our energy usage into periodic

is designed to overcome the obstacles and challenges

reports in accord with energy conservation laws andencountered. On the energy front, it inherits the concept of

regulations since Site 1 was opened in 2011. Because

modular server rooms, direct outside-air cooling systems,

Matsue DCP’s energy usage exceeded 1,500kL (crude oil

three-phase four-wire UPS and bus duct power transmission

equivalent) in fiscal 2013, we have also been working

technology. The entire Shiroi DCC facility runs at an annual

on energy saving measures in accord with the facility’s

average PUE of under 1.2, and we are endeavoring to expandclassification as a Type 2 Designated Energy Management

the usage scenarios for its power systems.

Administrationbuilding

Site area~40,000m2

Modular structureServerbuilding

Maximum power input capacity

Equipment capacity

50MW

6,000 racks

Figure 8: Artist’s Impression of
Shiroi Data Center Campus Once Completed

28© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

3. Focused Research (2)■ Direct Outside-Air Cooling Systems and System Modules■ Chimney Effect

As a new modular data center, Shiroi DCC takes advantage

The direct outdoor-air air conditioning systems in use atof operational experience from Matsue DCP and knowhow

Matsue DCP and on IZmo containerized data center units

gleaned from containerized data center development,

have greatly reduced the annual power consumption attrib-

applying this on a floor scale (several hundred racks or

utable to outdoor units, resulting in an annual average PUEso) and using system modules capable of serving needsof 1.237 for Matsue DCP and annual average pPUE of 1.18

on a larger scale. The buildings follow a systematizedfor containers. The systems need ventilation fans to blow

construction scheme based on standard shapes and ar-cool air into the cold areas year round, however, and therangements for the steel building frames and outer walls,

power used by these fans was the next energy-saving issue

facilitating rapid construction at low cost and high quality

we faced with the air conditioning systems. To tackle this(Shiroi DCC phase-1 building: from start of construction,issue, in 2012 we conducted tests (Figure 9) to see if we

it took four months to have the roof frame in place, eight

could take advantage of the chimney effect (phenomenon

months to fully complete). All of the internal elements

whereby the difference in temperature on the inside andof the system modules—electrical system, air conditioning

outside of a chimney causes an upward current) to cool thesystem, and server racks—are modularized, making it

servers. If this was possible, we thought, it could greatly

possible to add or replace (upgrade) equipment for eachreduce the power consumed by the ventilation fans and takeserver room individually.

us closer to a PUE of 1.0, and if it also made it possible toreduce IT equipment internal fans, this could also help to

These system modules are also integral to the reduced energy

further reduce overall data center power consumption.

usage of the air conditioning systems at Shiroi DCC,
which has a huge number of racks. Using the system

Although the conditions under which the chimneys were in-modules’ characteristically large space, large apertures,stalled (weather, air temperature and humidity, wind speed,

and wide span, we created a side-flow direct outside-airetc.) did affect the results, our tests showed that airflow

cooling system that utilizes the chimney effect.

volume increased roughly proportionally with increases in
the height and load capacity of the chimneys (Figure 10). As
a result, we determined that we could assist the air conditioner ventilation fans by grouping several racks into a single
module and positioning several modules at the center of theroom so that outside air could be drawn in from the sides of
the room, and exhaust air from the modules could be expelled from the top of the building. Shiroi DCC is designedwith these test results in mind in terms of the system

Load capacity (kWh)

Airflow volume (m3/h)

module layout and the direct outside-air cooling systems.

Chimney height (m)Figure 9: Test of the Chimney Effect (2012)*Patent no.: 6153772 (granted June 9, 2017)

Figure 10: Chimney Effect Test Results(excerpt: change in airflow volume according to chimney height)© Internet Initiative Japan Inc.

29

■ Side-Flow Air Conditioning

and the heat generated by the servers can be dissipated by

Shiroi DCC uses a side-flow system (IDC-SFLOW, a trademark

blowing air via the inlets into the server room at a slow

of Takasago Thermal Engineering Co., Ltd.) that supplies

airflow rate of 2m/s. This made it possible to greatly reduce

cool air directly into the server room via the partition wall

(to around a third) the amount of power consumed by the

between the server room and the air conditioner room

air conditioning system’s ventilation fans. The low airflow(Figure 11).

rate also makes for a better working environment in theserver rooms and greatly reduces the stress on workers.

Data centers housed in buildings often employ floor ventilation
with air supplied via the space under a raised, free-access■ Direct Outside-Air Coolingfloor. Space is tight, however, and the speed of airflow

Shiroi DCC uses direct outside-air cooling systems that arethrough the server room inlet (free-access grille) is high.

integrated with the system modules. Air is drawn into the air

This sort of design also has communications and power

conditioner room via intake ports on the eaves on the side ofcables routed through the same space, which increases air

the system module, passing through a medium-performance

supply resistance and results in a greater loss of pressure.

filter, and then mixed with server exhaust heat and supplied

A large pressure loss means more energy is required forinto the server room at a set temperature and humidity. And

air conditioning, so the ventilation fans use more power.

server exhaust heat, in the same amount as the outside air

Shiroi DCC’s wall-based side-flow air conditioning uses thedrawn in via the eaves, is expelled via exhaust ports at thesystem modules’ characteristically large space, large aper-

top of the building.

tures, and wide span as the air supply air route. The inlet
openings are designed to be as large as possible so that

Figure 12 shows the basic plan for a large-scale data center

when air is blown through the server room, air supply re-created in 2012. At the time, there was skepticism within IIJ

sistance is lower and the pressure loss is thus minimized.

about whether we could actually build something like this.

Shiroi DCC is designed to use an average of 6kW/rack,

But the direct outside-air cooling systems designed on top

Wall-mounted side-flow outlets

Figure 11: Side-flow Air Conditioner Outlets

ExhaustHot area A/C

Chimney

Exhaust port

Outside air

Inlet port

Hot aisle capping

Above ceiling(hot area)

Outside air

Cold area A/C

Below ceiling(cold area)

Server racks

Server room

Figure 12: Modular Data Center Design Proposal
at Time of Chimney Effect Testing (2012)30

A/C plant
room

Cross-section of A-A’

Figure 13: Cross-sectional View of Shiroi DCC System Module (2019)© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

3. Focused Research (2)of knowhow gleaned from Matsue DCP and technical tests

etc. When Matsue Site 2 went into service in 2013, we de-on the chimney effect and the like culminated, after almost

veloped co-IZmo/D, a new type of containerized data center

10 years of effort, in Shiroi DCC as depicted in Figure 13.

module that uses year-round outside-air cooling, and started
conducting a whole series of tests (Figure 14). co-IZmo/D■ Using AI to Optimize Air Conditioning Systems

integrates air conditioning features and IT equipment modules

At Shiroi DCC, we use AI to find the optimal combination

into an ISO-standard 20-foot container without the use ofof parameters for not only the air conditioners but the IT

external air conditioners or humidifiers. In one of our tests,

equipment as well. We endeavor to optimize air conditioning

dubbed the 2013 IT Equipment Adaptation Test, we worked

system operating conditions based on metrics of energy

with major IT equipment vendors to evaluate the character-savings, cost savings, and CO2 reductions.

istics of IT equipment in summer and winter in a completely
outside-air air conditioned environment and assessed the■ 2012 Chillerless Tests and 2013 IT Equipment Adaption

health of IT equipment under year-round outside-air cooling.

Tests
The air conditioning systems on the IZmo units in use at

We ran the tests on IT equipment supplied with a maximum

Matsue act as direct outside-air systems during autumn,

air temperature of 35°C/40°C, and the IT equipment

winter, and spring, but during summer, the outside air isfunctioned normally in a high-temperature environment of

shut off and indoor units are used to cool and dehumidify

45°C without any drop in CPU processing performance etc.

(circulation mode), so the energy savings in summer are

In a high-humidity environment, however, we did observe aworse than in the other seasons (autumn/winter/spring

noticeable deterioration in the IT equipment boards. FrompPUE = 1.095, summer pPUE = 1.331). In an effort tothese tests, we concluded that high humidity would reduce

resolve these issues, we did an assessment of data center

IT equipment operating life and increase fault rates. The

health under a scenario of using outside-air cooling year

tests also revealed that the rotation speed characteristics

round and not running the external air conditioners, chillers,of IT equipment internal fans vary a bit depending on the

Figure 14: co-IZmo/D© Internet Initiative Japan Inc.

31

IT equipment manufacturer’s design (Figure 15). From atemperature, actual IT equipment current draw, amount offacilities perspective, letting server room temperature riseheat processed, and so forth are taken and analyzed while

by 1°C will save on air conditioner power consumption.

the facility is in operation to provide (rule-based) estimates

But with tens of thousands of IT equipment units all runningof the characteristics of each individual rack. Similarly, air

in a unique way as the temperature rises, the power

conditioning system characteristics are estimated based

consumption of those units also changes. To maintain dataon certain operating conditions, parameters, and so forth,center quality and achieve true energy savings, therefore,

and a rule engine infers the operational settings that will

we determined that, instead of simply raising server room

minimize the combined power consumption of the IT equip-

temperature, we should understand the individual

ment and air conditioning systems, with the result being

characteristics of the IT equipment and control operations

to optimize air conditioning system operating conditions.

to be jointly optimal for both the IT equipment and thefacility itself.

So we have deployed the direct outside-air cooling systems
developed at Matsue on system modules that minimize■ AI Control

pressure loss by virtue of a large space, large apertures,

Shirai DCC uses AI-based air conditioning control thatand wide span, and combined this with AI control, and we

minimizes the combined power consumption of the IT

intend to proactively adopt new technologies going forward,

equipment and air conditioning systems, the main data

our ultimate goal being to achieve an annual average PUE

center elements, based on what we learned from ourof under 1.2.

testing efforts, including chillerless operation tests and IT
We have chased energy savings at Matsue DCP throughtures (as measured in Shiroi DCC’s test environment) are

the use of outside-air cooling and air conditioning units

recorded in a database, and measurements of server roomwith high COP scores, and by selecting high-efficiency IT

RPM■ Lithium-ion Batteries

such as rated current and current draw at different tempera-

RPM

equipment adaptation tests. IT equipment characteristics

Cold area temperature (°C)

RPM

RPM

Cold area temperature (°C)

Cold area temperature (°C)

Cold area temperature (°C)

Figure 15: IT Equipment Internal Fan Rotation Speed Characteristics by Manufacturer (Chillerless Tests, 2012)32© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

3. Focused Research (2)equipment. Air conditioning systems and IT equipment are

peak shaving at a similar cost level to the lead-acid battery

always the main factor in data center energy savings. The

UPS. Powerpack enables peak shaving and load shifting

key element in a data center, however, is the power system.

with respect to the power received by the data center,

We see the power system as the next key avenue for saving

which peaks during summer daytimes. It makes it possibleenergy. Accordingly, we adopted a three-phase four-wire

to reduce basic charges by reducing the demanded power

UPS and bus ducts at Matsue DCP Site 2, and we also con-and to reduce electricity charges by using power at off-tinue to evaluate software for predicting electricity usage

peak times and thus purchasing electricity at lower prices

and for peak-shaving control, and continue to devise and(Figure 17).

evaluate mechanisms for the selective supply of power fromfuel cells, PV systems, and DC UPS systems.

In the past, UPS systems, emergency generators, and othersuch power systems played a behind-the-scenes role in

Before designing Shiroi DCC, we also conducted a tech-ensuring high quality. At Shiroi DCC, we are working tonology survey, mainly looking at lithium-ion batteries, with

develop a new business model by actively looking for morea focus on two particular themes: (i) highly economicallyways to use such systems as data center energy resources.

viable power generation systems based on summer usage,
peak usage, etc. and (ii) electricity/power systems capable

3.4 Carbon Neutral Data Center Modelof delivering high efficiency in view of future technology■ IIJ’s Carbon Status and Road to Carbon Neutrality

trends. Based on this, we selected Powerpack, a lithium-ion

Manufacturing industries are being called on to reduce

battery storage solution from Tesla that offers operating

the CO2 emissions that result from making products incontrol features, installing it in November 2019 (Figure

factories, and the transport industry to reduce the CO216). Powerpack replaced the lead-acid battery UPS that weemissions from trucks. Given that over 90% of IIJ’s CO2

initially intended to install as a backup power supply for thecan be traced back to the power consumed by its data

air conditioning systems, and it offers the added feature ofcenters, the use of energy-saving equipment and renewables

Electricity (kW)Load shifting means electricity
can be bought at cheaper
off-peak prices
Load

Fluctuation
in A/C power
through day/night

Large

Discharging
Charging

Fluctuationin IT load
through day/night

Air conditioning power

Charging

Power for lighting etc.

Small

IT power

00:00

Figure 16: Powerpack installed at Shiroi DCC

Cutting peak usage
reduces demand rates

12:00

24:00

Figure 17: Data Center Power Demand and Peak Shaving/ Peak Load Shifting© Internet Initiative Japan Inc.

33

directly impacts on the carbon neutrality status of IIJ as

Although renewable generation costs are falling, power

a whole.

generated from non-solar sources including wind andbiomass remains more expensive than ordinary power

We plan to continuously improve the energy-saving tech-purchased from electric utilities. And building a power

nologies implemented at Matsue DCP and Shiroi DCC

plant will take time, including the time needed to obtainas described above to make the facilities even morea suitable site, so it is more realistic as a medium- toenergy efficient. Meanwhile, we have only just started

long-term proposition. Near term, we need to implement

working on the use of renewable energy, and one of themechanisms for the supply of power with a focus onearly possibilities for tackling this will be to switch tosolar, or photovoltaic (PV), generation.

an electricity retailer supply contract for the supply of
renewable energy certified as non-fossil fuel electricity.

■ Carbon Neutral Data Center Model

There are disadvantages, however. Renewable energy is

In light of all this, with a view to achieving carbon

more expensive than ordinary electricity, and there is noneutrality, we have defined a carbon-neutral data center

guarantee that it will be supplied at a stable price overreference model with the features listed in Table 1, andthe long term. So with the cost of generating renewable

we plan to continue modifying our data centers andenergy falling year by year, possible next steps may be tobuilding new ones in this light.

buy electricity directly from renewable energy plants or tohave our own generation plants.

No.

Feature1

Use of energy-saving technologies, such as outside-air cooling systems that substantially reduce air conditioning power consumption and an efficient three-phase four-wire
UPS, to reduce absolute power consumption levels.

2

Absence of other equipment and fixtures on building roofs to make large roof areas available for the installation of on-site PV systems.

3

In the short term, procure power from a combination of extra-high-voltage, high-voltage, and low-voltage off-site PV systems. Long term, allow for the possibility of procuring
power from other sources such as wind and hydrogen.

4

When daytime power supply exceeds consumption, store the excess in batteries for use at night.

5

Use of network measurement and control functions between generation equipment and the data center to balance the amount of power generated and the amount consumed.

6

Use of IT load control to channel loads toward times of surplus power.

Table 1: Features of a Carbon-Neutral Data Center

Type of facility

Office building

Warehouse

Conventional data center

Carbon-neutral data
centerPower consumption

Low

Low

Medium to highMedium to highPower density

Low

Low

High

High

On-site PV installation space

Small
Typically, external A/C units etc.
are installed on rooftops

MediumLarge-scale rooftop installations
possible

Small to medium
Similar to office buildings, but
dedicated equipment can be used
to create space

MediumRoof structure allows forlarge-scale rooftop installations

Generated
Consumed

Electric power

Typical daily power profile

Morning

Noon

Night

Morning

Noon

Night

Morning

Noon

Table 2: Differences in Electric Power by Type of Facility

34© Internet Initiative Japan Inc.

Night

Morning

Noon

Night

Feb.2022

Vol.

53

3. Focused Research (2)The first feature is the use of energy-saving technology

3 shows, off-site PV is more expensive than on-site PV,

to the greatest possible extent to reduce absolute power

but increasing off-site PV capacity is a viable option for

consumption.

securing the required amount of power.

The second feature is the installation of as much on-site PV

Off-site PV systems can be classified into three types according

generation capacity as possible, as the cost of generating

to the voltage at which they connect to the power grid,electricity this way is now lower than ordinary electricity

as shown in Table 4. To adequately supply a several-doz-charges. As Table 2 shows, data centers bring together

en-MW-scale data center, a large, extra-high-voltage PV

vast amounts of IT equipment and thus consume several

system is efficient, but these tend to take a long time todozen times more power per unit floor area than officebuild because you have to find a suitably large site, nego-buildings and the like (said to be around 50–100W/m ). As

tiate connection terms and conditions with the electric

they consume so much in absolute terms, PV generation

utility, and so forth. The high-voltage and low-voltage

alone cannot keep up with demand for power across theoptions are quicker to build, but if you want to generate

entire data center. But by adopting a warehouse-like build-the same amount of power as a 50MW extra-high-volt-ing structure and making the roof, where PV systems are

age PV installation, for example, you will need 1,000

installed, as large as possible, you can increase low-cost

50KW low-voltage generation plants. Given the amounton-site PV generation capacity.

of power required and time required, we think off-site

2

PV power needs to be sourced from a combination
The third feature is the installation of off-site powerof extra-high-voltage, high-voltage, and low-voltage

generation equipment (off-site equipment that supplies

systems.

power to your site via the power utility grid). As Table

System typeOn-site PV

Off-site PV

Characteristics

・No need for a dedicated site if installed on your building rooftops etc.

・Initial costs, including those for the land, are higher than for on-site if you take・In contrast with off-site PV, involves no wheeling charges or renewableenergy levies, and use of subsidies can make it even more cost effective.

on the capital investment yourself.
・Need to enter into long-term contracts if having power supplied from another

・But scale is small (can only supply a few percent of total data center usage).

company’s equipment.
・Involves wheeling costs as power is supplied via the power utility grid.
・Systems/procedures can change.
・Can adjust the amount of power generated by increasing scale or adding
plants.

Table 3: Differences Between On-site and Off-site PV

Power grid connection

Extra-high-voltage

High-voltage

Low-voltage

voltage

2,000kW or more

500–2,000kW

up to 50kW

Required site area (at 10m2/KW)

20,000m2 (2ha) or more

500m2 – 20,000m2 (2ha)up to 500m2

Initial investment

22.2 yen/kW

22.2–25.5 yen/kW

25.5 yen/kW

440mn yen or more

13–440mn yen

up to 13mn yen

Construction period

1 year or more

up to 1 year

up to 6 months

No. plants needed for 50MW output

5MW/plant

500kW/plant

50kW/plant

10 plants

100 plants

1,000 plants

Table 4: PV System Categories© Internet Initiative Japan Inc.

35

Storage is the fourth feature required for carbon-neutral data

The sixth feature is the use of IT load control, which involves,centers. As Figure 18 illustrates, as off-site PV increases,

for example, halting servers on nights when there is not

the power supplied by PV capacity in the daytime canenough power and only running them in the day, so thatovershoot data center power consumption, in which casethe load is channeled toward times when there is surplus

it is stored and used at night. This makes it possible topower. Information systems generally cannot be stopped

use renewable energy at night as well, but the fact thatfor 24 hours at a time, but we can expect more and more

the cost of storage batteries is still high is an obstacle to

new use cases amenable to load control to appear—data

the full-fledged rollout of such systems.

analysis tasks that can be given a manageable timeframe tocomplete or mining tasks that require low-cost processing.

The fifth feature is the use of network functions thatenable measurement and control of what happens between

Figure 19 illustrates how a carbon-neutral data center withthe power generation equipment and the data center. This

these features could be set up. To realize carbon neutrality

makes it possible to balance supply and demand and ensure

while maintaining the reliability and other quality attributes

the stable operation of the grid.

required of a data center, we need to create a new model

Electric
power
Load balancing
using storage batteries

50MW
Data center
power consumption

Non-PV
renewable energy

Powerfrom PV plants

Night

Day

Night

Figure 18: Data Center Power Supply/Demand Overview

Extra-high-voltage (2000kW or more)
5MW × 10 = 50MW

High-voltage (up to 2000kW)
500kW × 100 = 50MW

Low-voltage (up to 50kW)
50kW × 1000 = 50MW

Off-site PV plants

Measurement/control network
Power gridWind, hydrogen, biomass power, etc.

Load balancing battery(includes imbalance correction)

On-site PV

Figure 19: Carbon-Neutral Data Center Model

36© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

3. Focused Research (2)that organically combines the generation equipment that

JCM (Joint Crediting Mechanism) to reduce greenhousesupplies the power and the data center that consumes thatgas emissions, as part of which IIJ provided support for

power. While we only have a conceptual model to illuminate

the construction and operation of a highly energy-efficient

the way forward at this stage, we will be conducting technicalcontainerized data center in the Laotian capital of Vientiane.

tests and working with external partners going forward, onboth the business and technological fronts, to sort out the

The road to carbon neutrality will no doubt be long anddetails and apply what we have learned when modifying ourchallenging, but we will continue to take advantage ofown data centers or building new ones.

Internet technologies, including IoT and mobile, and test
and implement systems, and we will seek to apply the

3.5 Conclusion

knowledge and insight we gain not just internally at IIJ

IIJ has also been involved in external initiatives using the

but also in the world at large, our aim being to contributetechnologies it has developed in-house, one example being a

toward carbon neutrality in the data center and, ultimately,

NEDO (New Energy and Industrial Technology Development

across society as a whole.

Organization) demonstration project aimed at using the

Isao Kubo
General Manager, Infrastructure Services Department, Infrastructure Engineering Division, IIJ.
Mr. Kubo joined IIJ in 2008. He oversees the data center business and the construction of Matsue DCP and Shiroi DCC. His aim is toachieve carbon neutrality as soon as possible.

3.2 Outcomes at Matsue Data Center Park
Yoshimasa Kano
Manager, Matsue Data Center Park, Infrastructure Services Department, Infrastructure Engineering Division, IIJ.
Mr. Kano joined IIJ in 2016. He is engaged in the operation of containerized data centers and the testing of next-generation modular data centers.

3.3 Initiatives at Shiroi Data Center Campus
Akio Hashimoto
Deputy General Manager, Infrastructure Services Department, Infrastructure Engineering Division, IIJ.
Director, Shiroi Data Center Campus, Infrastructure Services Department, Infrastructure Engineering Division, IIJ.
Mr. Hashimoto joined IIJ in 2009. He works on the study and design, construction, and operation of next-generation data centers andactively pursues data center automation and efficiency gains.

© Internet Initiative Japan Inc.

37

4. Focused Research (3)IIJ’s Road to BCR Approval—Complying with EU GDPR and Beyond
In August 2021, we received notice of the long-awaited

With the rise of major platform operators, exemplified byapproval for IIJ’s BCRs (Binding Corporate Rules) from LDI-GAFA, that use personal data to the hilt and Snowden’s

NRW, the supervisory authority in North Rhine-Westphalia

exposure of the US government’s excessive surveillance(NRW), Germany. Since the EU’s GDPR (General Data

practices, there was strong distrust within Europe about

Protection Regulation) came into force, 18 companies world-the US’s use of personal data, and I think this is the real

wide have received BCR approval (as of August 2021), IIJ

backdrop for what prompted the creation of the GDPR.

being one of them. The five years since we started looking

I think, at its heart, this traces back to a marked difference

at BCRs as a means for complying with the GDPR have beenin cultural values about personal data. This is something

eventful, and with the approval, we can now proudly say

I will discuss later.

that our efforts so far have finally been recognized.
It became evident that the EU was pretty serious, but
Here, I discuss why we chose the IIJ BCR approval route,

IIJ doesn’t provide personal services on a global basis,

what the road to acquiring that approval was like, and where

and there was talk about the Japanese government’s

we are headed from here.

Personal Information Protection Commission working
toward some sort of national certification from the EU

4.1 Decision to Seek BCR Approval

anyway, so truth be told, we hadn’t intended to actively

The EU drafted the GDPR for the purpose of protecting

pursue GDPR compliance. But as we worked through all

personal data within the EEA (European Economic Area) in

the factors to consider, and with a proposal from Shinpei

2016, and it came into effect in 2018. The GDPR was the

Ogawa, a director of IIJ Europe at the time, we came to

subject of a lot of speculation back when it was announced,

the conclusion that seeking IIJ’s own BCR approval would

and I remember some extreme arguments saying, for example.

be advantageous.

that it would harm the free distribution of information on
We had four main reasons for embarking on our journey for

the Internet.

IIJ’s BCR approval.
The EU’s objective was of course not to restrict the dis-38

tribution of personal data on the Internet. Rather, it was■ We saw the need for global security governancecalling for the appropriate use of data on people in the EU

IIJ has been deploying group companies all over the world

at a time when such information was already being used infor some time, and doing business naturally requires us tocountries everywhere. Where practices had been vague, theaddress personal data protection and other factors in each

EU wanted to lay out clearer protections on personal data

country. In the past, however, we did not fully comprehend

that had real effect. It was a truly advanced initiative tothe situation in every country, so it was left up to each group

institutionalize such protections. I think that, even now, thecompany to mount its own response, and to be honest, we

policies and practices for personal data protection laid out in

weren’t able to address the issue of control on a group level.

the GDPR have a lot to say about personal data protection

Creating IIJ’s BCRs to comply with the GDPR and puttingin the information age.

this into effect at group companies was crucial in enabling© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

4. Focused Research (3)us to implement control at a time when the situation around

implementing responses to a range of issues that potentiallypersonal data was becoming more and more nuanced.

lay down the track. We were honestly surprised at how far
they were taking it. Japan was also naturally making an■ Alternatives to BCR were too complicatedeffort in that regard, and the EU did adopt an adequacy

An alternative to obtaining BCR approval as a means

decision on Japan, but it is telling that supplementary rulesof meeting the GDPR requirements is to enter into SCC

were applied to transfers of personal data from within the(Standard Contractual Clauses), a form of template contract,

EEA to Japan. This leads into the discussion that follows,

as necessary and perform data transfers on that basis.

but the approach of giving careful thought not just to the

These contracts, however, must be meshed in the sense

controller but also to the processors, and developing athat there needs to be contracts between the controller

framework with respect to the processors, is a fairly ad-(the party responsible for holding the personal data) andvanced one. The EU obviously implemented this because it

the processors (the parties who receive the personal data

felt it was necessary to protect personal data, but I think it

for processing), as well as between processors themselves.

also shows that they were somewhat prescient about how

And if changes are made to how the data are processed,

personal data would be handled in the Internet’s cloud era.

the contracts must be entered into again, so managing thecontracts is cumbersome. It was easy to see that we would
end up with quite a lot of contracts just with IIJ’s own group■ It would enable protection of personal data on cloudservices in addition to personal data held by IIJ

companies, and given the nature of our business, in many

The EU’s motivation for creating the GDPR was clearly tocases information from our customers is deployed across eachregulate the handling of personal data on the Internet byof our group companies’ platforms. For these reasons, we

platform operators and the like within the EEA, to whichwere concerned about this approach eventually collapsing.

end it sought to impose rules on business operators andIIJ’s own BCR approval was really the only way to solve thethe like that handle personal data so as to ensure that they

problem.

do so properly in accord with EU standards. As such, theGDPR naturally applies not only to controllers responsible■ GDPR represented a more advanced approach thanfor managing personal data but also to processors who

Japan’s own personal information protection practices

receive such data from controllers. I think the EU made such

IIJ naturally implements internal controls for the protection

provisions explicit because it was seeking to confront whatof personal information and has acquired Japan’s P Markit saw as a less than appropriate state of affairs particularly(Privacy Mark). Yet something that became eminently

with respect to the platform operators who process personal

apparent as we started to understand more about the GDPR

data. This potentially opened the way for outsourcing, suchis that the EU’s commitment is on another level. Japan’s

as cloud services, to comply with the GDPR by meeting

personal information protections certainly do comprise

the BCR conditions under the processor category. That is,a range of carefully thought out measures, but the EU’s

it provided a way for controllers, the customers of suchapproach was to tackle these difficult problems throughservices, to prove that they use processors who have imple-

sound reasoning and ideas, and to look ahead and startmented proper protections.

© Internet Initiative Japan Inc.

39

IIJ Europe began looking seriously at GDPR compliance inakin to the efforts developed in Japan over the past many

January 2016, and IIJ’s board of directors officially gave

years around internal control for information security andapproval to start working toward BCR approval in May

personal information protection. That said, the EU’s GDPR

2016. The approval was actually received on August 5,

is the most stringent personal information protection regulation

2021, partly because of how long the review takes, but

anywhere in the world, so obtaining BCR approval is byalso due to the impact of Brexit and other subsequentno means easy.

events. While not all that many people were involved
over that period, a number of internal units, led by the

At the IIJ Group level, the IIJ Binding Corporate Rules

Risk Management Office, had a role to play, including

appear at the very bottom of the privacy policy on IIJ

the Business Risk Consulting Headquarters, the Global

Europe’s website, and IIJ Group companies also link to this.

Business Division, and the Compliance Department, andwe actually used a law firm when negotiating with theOnce a corporate group’s BCRs have been approved by theEU’s supervisory bodies as well. Our efforts did finallycompetent data protection authority in the EU, this certifies

pay off in the form of BCR approval. IIJ is the first global

that, in terms of personal data protection under EU law,cloud vendor, not just in Japan but worldwide, to receive

the group has appropriate safeguards in place, and underapproval since the GDPR took effect.

GDPR Article 46, Item 2(b), this allows it to legally transfer
personal data fro within the EEA to locations outside of

But as with other aspects of data governance, obtaining

the EU.

approval is not the goal. Instead, it provides an opportunity
to embark on new efforts to establish internal control forTransfers of personal data from within the EEA to theglobal data governance.

outside of the EEA are in principle prohibited. There are
a number of recognized ways of making it possible to do

4.2 What are BCRs?

this, though. Among them, BCRs represent the strictest

Below is a brief explanation of BCRs.

standards at the corporate level. As long as they comply
with the BCRs, transfers of personal data outside the region

BCRs are data protection policies that are adhered to by an

to corporate groups that have BCR approval, or between

entire corporate group (these policies are also made known

companies within a group that has received approval, areto the subjects of personal data and are thus widely dis-

recognized as being subject to the appropriate safeguards

closed) and embody rules that are binding on the grouprequired by the GDPR.

companies and their employees. The objective is data

40

protection, but this is not so much about implementing

Approval is obtained by submitting BCRs as stipulated intechnical security measures as it is about sharing information

GDPR Article 47 to the competent data protection authority

and educating employees on the basic principles and rules

for approval. A rigorous review process that goes beyond

around the handling of personal data, creating an operating

the competent authority ensues. First, the competentenvironment that allows employees to raise queries orauthority reviews the BCRs with the assistance of super-complaints, and so forth. In that sense, the approach isvisory authorities, and corrections are repeatedly made© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

4. Focused Research (3)in conjunction with the company under review. Once it

a federation, all 16 of its states have their own personalis satisfied, the competent authority communicates withdata protection supervisory authority.

the EDPB (European Data Protection Board), part of theEuropean Commission, the EU’s executive branch, andThe EU adopted an adequacy decision on Japan on January

submits a pre-review request to the ITES (International23, 2019. The decision means the EU recognizes that Japan

Transfer Expert Subgroup) meeting. ITES is made up ofhas personal information protection guarantees that areexperts from the supervisory authorities of all EU member

in line with those that apply in the EEA. This recognition

states. They may think that something should be done

from the European Commission was the result of work bydifferently or that a particular rule is a little loose, and theyJapan’s Personal Information Protection Commission alongaccordingly send revision requests to the competentwith other stakeholders and no doubt came as a boon forauthority, based on which the corporate group revises

many Japanese companies.

the BCR draft. This process is repeated until it is apparent
there are no further opinions on the pre-review, at which

It does not mean, however, that all Japanese companies arepoint the competent authority asks for an Opinion from a

now GDPR compliant.

plenary meeting of the EDPB, the highest decision-making
authority in this case, composed of representatives of allFirst of all, the GDPR does not in principle allow the

EU national data protection authorities. The Opinion is antransfer of personal data out of the EEA, so the cross-borderofficial document of the EDPB, in accord with which thetransfer rules must be observed when transferring data.

competent authority provides instructions to the corporate

The cross-border transfer rules, per GDPR Articles 45 andgroup to finalize the BCRs, and then grants final approval.

46, allow such transfers under conditions including theThis rigorous process means that obtaining BCR approvalfollowing.

from your competent data protection authority in the EU
takes quite a lot of effort and time.

• The European Commission has adopted an adequacy
decision on the country

In the IIJ Group’s case, we submitted our BCRs to the UK’s

• BCR approval has been acquired

ICO (Information Commissioner’s Office), our competent• SCCs have been entered into

authority at the time, in October 2016. The impact of Brexit,• In compliance with a (an industry) code of conduct

however, means that the competent authority for us is now

approved by the EDPB

the supervisory authority in North Rhine-Westphalia (NRW),
Germany. IIJ Europe (based in London) had been the IIJ

Put differently, only transfers of personal data out of theGroup’s headquarters in the EU, but the UK’s withdrawal

EEA pursuant to the GDPR are exempted. In addition, thefrom the EU meant that this role passed to IIJ Deutschland

Personal Information Protection Commission has published(based in Dusseldorf), and thus the supervisory authority

official notice that supplementary rules will apply toin NRW, in which Dusseldorf is located, became the IIJ

personal data transferred from within the EEA wherever theGroup’s competent authority. The norm in other countries islocal rules are deemed inadequate. The EU’s adequacy

to have a single national authority, but with Germany being

decision means that transfers of personal data from the EU© Internet Initiative Japan Inc.

41

to Japan are permitted, but such data may not be further

I will now briefly explain the background to the creation oftransferred from Japan to a third country. So the adequacy

the GDPR. An essential element is that the EU authorities

decision does not fully cover cases in which, for instance,

are in opposition to the US in terms of privacy protection.

an employee register is shared globally.
■ Historical background
Japan’s pursuit of an adequacy decision was the rightThe 9/11 terrorist attacks in the US were more than enoughapproach for the nation to take, and an understandable

to shake the world’s collective consciousness. After 9/11,

one, in view of the circumstances of many Japanese

the US embarked on a secret mass surveillance program andcompanies’ businesses, but it is unfortunate that it led to

implemented mechanisms for installing backdoors in social

a sense that companies now no longer need to take any

media and other services in order to expose terrorists. These

particular steps of their own with respect to the GDPR.

operations should only have been carried out under a court

Personal data protection is an extremely important issueorder, but in reality, as exposed by Edward Snowden in Junefor the Internet and other types of new information2013, the US had made it possible for intelligence personnel

infrastructure, and we should not forget that the situation

to freely snoop into people’s privacy in the course of their

these days is such that even companies that do businessintelligence activities. I think it was difficult for the American

mainly within Japan cannot ignore its impact.

people to oppose this because the atmosphere was suchthat they felt compelled to allow the government to do what

4.3 Personal Data Protection Initiatives Around
the World

it needed to do for national security. I think Americans have

As my mind became increasingly wound up in the various

the sense that they see the government as the people’s

personal data protection initiatives out there, I was maderepresentative and spokesperson, an attitude that dates

acutely aware of differences in perspectives on personal

back to the nation’s founding. But then Snowden revealed

data and astonished by just how different the culturalthat the US had been eavesdropping indiscriminately on

backgrounds can be. Norms around personal data in Japanembassies of the country’s allies including Germany andmerely represented Japan’s own local perspective. My

France. This enraged the EU’s member states and hadrealization that there are rights and responsibilities withmajor ramifications, the German government’s axing of its

respect to personal data in other parts of the world that

Verizon contract, for instance.

a very strong sense of ownership about their country in

differ completely from those found in Japan was a very

42

important insight that I gained from IIJ’s BCR approval

The idea that the government is absolutely right does

process. To be honest, I realized that IIJ’s mindset as a

not exist in the EU. For more than a millennium, thetraditional Japanese company simply would not cut it on

European continent was again and again the scene of

the world stage. There exist multiple ideas about what iswars and conflicts that ultimately failed to settle nationalright and just when it comes to personal data around theborders or territory. The rise of tyrants repeatedly led toworld, and they will all no doubt have a major influence on

tragic events—people oppressing those from different

the information society we live in going forward.

denominations within their own religion, for example, or© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

4. Focused Research (3)massacring other ethnic groups. The European Coal and

Ireland. The details are complex, but roughly speaking, under

Steel Community was created with the aim of establishing

Facebook’s Safe Harbor agreement, Facebook users’ per-final national boundaries and securing a lasting peace after

sonal data was, for example, being transferred from Ireland

World War II. Gradually building its cooperation with the(location of Facebook’s EU headquarters) to the US. But as

European Atomic Energy Community and the European

the Snowden expose revealed, the US government was able

Commission, it eventually grew to become the EU we know

to freely view that data. This means, said the complaint,

today. The EU operates on the premise that governmentsthat Facebook is breaking the Safe Harbor agreement, and if

can also overstep and run wild, so when EU laws are

the US government can legally spy on the services of private

passed, it is stipulated that independent, third-party

companies under US law, then the Safe Harbor agreement

supervisory authorities that have the power to enforce theitself is pretty much meaningless in the first place. The Irish

law with respect to governments also be established. So

High Court, unable to make a decision, referred the matterthe EU’s privacy protection supervisory authorities strictly

to the Court of Justice of the EU (CJEU). In a shocking

enforce the law with respect to privacy breaches not only

ruling, in October 2015 the CJEU declared the Safe Harborover private-sector companies but also over governments

Agreement invalid. This is known as the Schrems I decision.

and public agencies.

The inability to transfer personal data from the EU to the
US represented an extreme impediment to trade, however,This difference in thinking means the US and the EU areso economic proponents within the European Commission

basically incompatible in the world of privacy. But because

and the US Department of Commerce adopted a new spe-

they are each other’s largest trading partner, people in thecial framework called Privacy Shield in August 2016. The

EU involved in the business of trade desire a good relation-

Patriot Act, created after the 2001 terrorist attacks, expired

ship with the US. Personal data is generally distributed

in 2015, but the mechanisms for exposing terrorists arguably

as part of commercial activities and is thus inseparably

remained intact, albeit with increased transparency, underlinked with trade negotiations. In 2000, therefore, theits successor, the Freedom Act. Privacy protection advocates

EU created a framework called the EU-US Safe Harborwithin the European Commission claimed, therefore, that

Principles, under which private-sector organizations werenothing had really changed, and debate about the validity

permitted to freely transfer personal data from the EU toof Privacy Shield thus continued to smolder on during andthe US provided they submitted a notification to the US

after 2017. Once the GDPR came into effect on May 25,

Department of Commerce attesting that they had adequate

2018, Schrems immediately filed another suit claiming that

security measures in place and will not pass personal data

it was illegal for US companies to transfer EU personal datafrom the EU to third parties.

to the US based on Privacy Shield. The case was settled
in July 2020 with the ruling that Privacy Shield was alsoThis collapsed in the wake of the Snowden expose.

illegal. This is the Schrems II decision. Following Schrems

Snowden’s revelations prompted doubt about the effective-

II, the SCCs were also reviewed, and modernized SCCs

ness of the Safe Harbor agreements in the mind of Austrian

were issued in June 2021.

lawyer Max Schrems, who then filed a legal complaint in© Internet Initiative Japan Inc.

43

The new SCCs provide extra protections that require

Although the process is gradual, the revised Personal

companies and other bodies that transfer data to disclose

Information Protection Act, which was passed in June 2020

information about access by public authorities in theand will come into effect from April 1, 2022, also looks

destination country (whether the country has laws that

set to tighten Japan’s protections and bring them closer inallow the government requisition data and whether theyline with the GDPR. For instance, penalties will be raised

are actually enforced).

from 300,000 yen and 500,000 yen to 100 million yen.
Stronger information disclosure requirements will apply whenOnce personal data is transferred from one country to

personal data is provided to a third party in a foreign country

another, it is no longer protected by the origin country’s(from the EU’s perspective, this means when personal data

laws, so various restrictions are thus imposed. Meanwhile,is transferred out of its domain). Cookies will be subject

the EU does recognize a number of countries as offering

to restrictions, although not quite to the extent as in theadequately secure personal data protections. It refers to

EU. And it will be mandatory to report personal data leaks

these as “adequate” countries (recognized by adequacythat match certain patterns to the Personal Information

decisions), and they include Switzerland, New Zealand,Protection Commission and other bodies (preliminary reportand Argentina. Personal data can be transferred from thewithin 3–5 days and a final report within 30 or 60 days;

EU to these countries. Japan joined these ranks in January

incidentally, the GDPR mandates reporting within 72 hours).

2019. Japan’s Personal Information Protection Act is notquite up to the GDPR standards, however, so personal data

■ Cultural backgroundmay only be transferred if certain supplementary rules are

In light of the above historical background, I see three major

applied.

trends in personal data protection across the globe today.

The Personal Information Protection Act is to be reviewed

One is a sort of public welfare idea that the sharing of big

every three years to bring it up to speed with technological

data will benefit people all over the world, which seems todevelopments and other countries’ laws and regulations,be the mainstream view in the US. Another is the humanand thus it may approach parity with the GDPR going forward.

rights assertion that says that the ability to manage yourOn the other hand, Japanese personal data can also be

own personal data is a basic human right, which is the main-transferred to the EU. As announced in January 2019,

stream view in Europe. And then there is the security-oriented

Japan and the EU reached a mutual agreement on the flow

idea that the management of data within a country’s ownof personal data between each other’s domains, and thisdomain is its own national security issue, which is probably

means that Japan also recognizes the EU as having

exemplified by China.

adequate protections. The only other place Japan recognizes

44

in this manner is the UK. Given these developments, it

In the US, personal data is seen not only as holding

does seem like the Japanese government’s approach whenconvenience for the user but also as conveying benefits toit comes to protecting privacy is to strengthen individuals’

a wide swath of other users, as exemplified by the rise ofrights and interests using the EU’s GDPR as a reference.

GAFA. This is evident from Google’s mission, for instance,© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

4. Focused Research (3)which reads: “Our mission is to organise the world’s infor-

than emphasizing any single factor alone. But I think it’smation and make it universally accessible and useful.” The

important to note that, broadly speaking, there are threehistory of American society is one of pioneering on the

major perspectives around personal data, each emerging

frontier, and this is possibly what embedded the idea of

from its own historical and cultural backdrop, and there

information sharing, including for the purposes of protecting

is no unified mindset on personal data that applies acrossnational security, as an important part of its culture. In

the entire world.

terms of its development strategy too, this seems to have
led to a set of values that embrace the idea of de facto

Closer to home in Japan (and while certainly not thestandards, the idea that prevalence itself is what makes amainstream world view), there is a culture of regarding

standard, something that has been a prominent factor in

information as being equivalent to value itself, a view that

the Internet as well.

comes from the Japanese concept of kotodama, the belief
that mystical powers dwell in words. So when it comes

In Europe, on the other hand, personal data was at times

to personal data, people may feel that if someone knows

something that could affect whether a person lived or died,

information about you, they actually know you in someso the ability of the subject of personal data to know andtrue sense. In older times, there was a cultural practicemanage that data is seen as a right. Further, the developmentof hiding one’s true name out of the belief that knowing

strategy is one of clarifying processes thought to have been

someone’s name would confer dominion over them. Theredeveloped as part of colonial policies, with standardizationis also a long-held idea that you should not utter unlucky

being pivotal to the global development of data protections.

thoughts (put misfortunes into words) because expressing
information can somehow lead to the events it describes

In China, which exerts a significant influence on the

coming to pass (this is a deep rabbit hole to go down, so

world in recent times, the desire to maintain the nation

I will cut the discussion short here).

is a crucial factor, and so naturally the thinking is thatjudgements about all sorts of information, including per-In any case, when it comes to personal data protection, thissonal data, should be based on national security. Hence,

Japanese style of thinking, at least, is unlikely to pass muster

China recommends its own country’s services over thoseelsewhere in terms of the mindset and meaning it implies andcreated in the US, and it imposes heavy restrictions on

the weight of the belief. Of course, there is clearly no globally

information flows between countries. And of course, theunified view, and this holds for the US mindset, the European

strategy is a nationalistic one: information flows involving

mindset, and the Chinese mindset alike. In our information

China will take place under China’s restrictions.

society, information is not just a set of symbols. There are
historical and cultural backgrounds, and therefore each culture

Of course, these are somewhat stereotypical character-has its own principles, its own ideas about the right way to

izations. I think countries’ information control strategies

handle information, so when it comes to flows of information

are carried out with an eye on any number of elements

across national borders, there is a desire to ensure that theto do with the public interest, rights, and security, rather

principles of each country or region involved are respected.

© Internet Initiative Japan Inc.

45

4.4 IIJ’s Road to BCR Approval

4.5 Looking Ahead

IIJ was an early mover on BCR approval, but the process

As discussed earlier, as the Internet spreads further, initiatives

was full of twists and turns. The impact of Brexit in particularin personal data protection will no doubt continue to affectwas significant, and the IIJ Group, which had aimed toInternet-related companies and other companies in various

obtain approval from the UKs ICO, did have to make a major

ways. This process is a necessary part of the way society

strategic change along the way.

is adapting to facets of the information age, most notably
the Internet, and even in Japan where attitudes can be

Table 1 provides a brief overview.

indifferent, these changes are something that, sooner orlater, we will be unable to ignore. Of course, even frommy own experience, I cannot declare that every company
should obtain BCR approval, but objectively speaking, there
are many companies in Japan that do need it, and they will
eventually be forced to take action of some sort.

Date

Organization

ActionJan 2016

IIJ Europe

IIJ-EU receives CEO ￼approval to start working toward GDPR compliance. Work begins in earnest.

Mar 2016

GDPR Office

Office set up by the Risk Management Office, Compliance Department, Global Business Division, and IIJ Europe.

Jun 2016

UK

Brexit prevails.

Jul 2016

GDPR Office

Decides on law firm.

Aug–Oct 2016

Risk Management Office

Creates BCR document.

Aug 2016

IIJ Europe

Starts GDPR compliance support consulting.

Oct 2016

GDPR OfficeSubmits BCRs to the Information Commissioner's Office (ICO), the UK’s personal data protection authority.

Mar 2017

UK

Prime Minister signs a letter triggering Brexit.

Aug 2017

ICO

First communication that IIJ’s BCRs are now under review. An ongoing process of revisions follows.

May 25, 2018

EU

GDPR comes into effect.

Jan 10, 2019

ICO

UK ICO’s review is completed and submitted to the co-reviewers (Germany, Netherlands).

Feb 12, 2019

EDPB

Information on competent supervisory authorities post-Brexit is published. Appears we will not make it in time for Brexit.

Mar 1, 2019

LDI-NRW

Co-reviewer comments on IIJ’s BCRs from the authority in NRW, Germany.

Mar 21, 2019

Dutch authority

Co-reviewer comments on IIJ’s BCRs from the Dutch authority.

Mar 28, 2019

GDPR Office

Presents response to comments to ICO. Brexit deadline extended to Oct 31.

May 16, 2019

ICO

Co-reviewer provides notice that the review is complete. Pre-review by the EDPB ITES (International Transfers Experts Subgroup) meeting requested.

Jun 2019

UK

Prime Minister Theresa May resigns.

Jul 2019

UK

Boris Johnson becomes Prime Minister.

Jan 31, 2020

EU/UK

Britain withdraws from the EU. The transition period runs until end-2020.

Apr 2020

EDPB

Pre-review conducted by the EDPB ITES meeting under the ICO’s guidance. Several rounds of revisions ensue.

Jun 2020

EDPB

Provides notice that ICO approvals made during the Brexit transition period are invalid.

Jul–Sep 2020

GDPR Office

The supervisory authority in NRW, Germany, where IIJ Deutschland is located, is appointed as the competent authority.

Sep 2020

GDPR OfficeFormally requests the NRW authority to act as the competent authority.

Sep–Nov 2020

ICO/LDI-NRWLDI-NRW takes over from ICO on IIJ’s BCRs.

Dec 2, 2020

LDI-NRW

LDI-NRW officially becomes IIJ’s competent authority.

Dec 31, 2020

UK

Completely withdraws from the EU.

Apr 2021

EDPB

Review at ITES meeting (review had progressed under ICO, so passage is smooth).

May 2021

GDPR Office

Creates a German-language version of the BCR document.

Jun 28, 2021

LDI-NRW

Submits BCR-C and BCR-P to the EDPB.

Jul 28, 2021

EDPB

Deliberates at a plenary meeting. All clear given.

Aug 2, 2021

EDPB

Discloses a positive official opinion on IIJ BCR-C/P.

Aug 5, 2021

LDI-NRWIIJ BCR-C/P approved.

Table 1: IIJ’s BCR Approval Process

46© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

4. Focused Research (3)The IIJ Group is also aware that it cannot rest on its laurels

work practices. At the risk of being repetitive, obtaining BCR

just because it has obtained BCR approval under the EU

approval for IIJ was never our end goal. Our bigger objective

GDPR. We realize that, at the very least, we will also need

was to use it as an opportunity to further bolster our internal

to comply with the UK’s requirements now that it is out ofcontrols around information security and privacy protection.

the EU, and that we will need to adhere with other personal

We are proud to have taken a major, if not definitive, step

data protection initiatives, including those in the US. The

toward that goal.

situation now is such that we could never keep up with allof the initiatives out there, but broad frameworks such as

I think the IIJ Group’s corporate mission can be described

APEC CBPR, at least, are something we believe we should

as the single-minded pursuit of business with the Internet

also look at complying with.

at its core. Companies that specialize exclusively in global
Internet infrastructure seem to have become rare these

IIJ is also not immune to personal information breaches anddays. So, as a company that has helped drive the spreadother incidents, so I think we could be exposed to criticism

of the Internet, I think we have a responsibility to holdalong the lines of “look who’s talking.” But given the crucial

on to our Internet-centric perspective and continue making

role of personal information protection and other aspects ofcontributions as an infrastructure company for the information

information security in our modern information society, I

society age.

think there is a clear duty to address the issues in front of
us. Organizations need to respond to the personal data

To be honest, IIJ BCR was a slog, and I honestly wouldn’t

protection and other practices found in different cultures while

recommend it to anyone. But I can say emphatically that

also enhancing their own information security capabilities.

I’m glad we did it because I believe it will be one key pillar
supporting the type of Internet we envision.

Personal information protections and other information
security measures are not something that can be thrown

Looking ahead, we will continue to strive for a safe andtogether overnight. They become meaningful only once they

secure Internet.

are established as part of organizational culture and everyday

Takamichi Miyoshi
Senior Fellow, DPO (Data Protection Officer), IIJ. Mr. Miyoshi joined Internet Initiative Planning Inc. (now Internet Initiative Japan Inc.)in April 1993. He worked on launching Internet services and the operation of service equipment/facilities. He later served in services
development and strategic planning, going on to participate in numerous study groups set up by the Ministry of Internal Affairs andCommunications and other agencies as Managing Director of IIJ. He has been in his current position since June 2015.

© Internet Initiative Japan Inc.

47

53
February 2022 Vol.
About Internet Initiative Japan Inc. (IIJ)
IIJ was established in 1992, mainly by a group of engineers who
had been involved in research and development activities related
to the Internet, under the concept of promoting the widespreaduse of the Internet in Japan.
IIJ currently operates one of the largest Internet backbones
in Japan, manages Internet infrastructures, and provides
comprehensive high-quality system environments (includingInternet access, systems integration, and outsourcing services,etc.) to high-end business users including the government andother public offices and financial institutions.
In addition, IIJ actively shares knowledge accumulated throughservice development and Internet backbone operation, andis making efforts to expand the Internet used as a social
infrastructure.­­The copyright of this document remains in Internet Initiative Japan Inc.
(“IIJ”) and the document is protected under the Copyright Law of Japanand treaty provisions. You are prohibited to reproduce, modify, or makethe public transmission of or otherwise whole or a part of this document
without IIJ’s prior written permission. Although the content of thisdocument is paid careful attention to, IIJ does not warrant the accuracy andusefulness of the information in this document.
Initiative Japan Inc. All rights reserved.
© Internet
IIJ-MKTG020-0051

Internet Initiative Japan Inc.
Address: Iidabashi Grand Bloom, 2-10-2 Fujimi, Chiyoda-ku,
Tokyo 102-0071, Japan
Email: info@iij.ad.jp URL: https://www.iij.ad.jp/en/