Internet
Infrastructure
Review
Nov.2019

Vol.

44

Periodic Observation Report

Broadband Traffic Report:
Moderate Growth in Traffic Volume
Ongoing
Focused Research

About the IIJ Public DNS Service

Internet Inf rast ruc ture Review
November 2019 Vol.44

Executive Summary ........................ ................................................ 3
1. Periodic Observation Report .......... ................................................ 4
1.1
1.2
1.3
1.4
1.5

Overview .. ................................................................................ 4
About the Data ............................................................................ 4
Users’ Daily Usage . ....................... ................................................ 5
Usage by Port ............................. ................................................ 8
Conclusion ................................ ................................................ 9

2. Focused Research ...................................................................... 10
2.1
2.2

Introduction .............................................................................. 10
What is DoT/DoH .......................... ............................................... 10

2.2.1 DNS and Privacy . ........................................................................ 10
2.2.2 DNS Transport Encryption ................................................................ 10
2.2.3 Transport Encryption and DNSSEC ........................................................ 11
2.3

IIJ Public DNS Service and DoT/DoH ...................................................... 122.3.1 The TCP barrier ........................................................................... 122.3.2 The TLS Barriers ........................... ............................................... 122.3.3 The HTTP barrier ......................................................................... 13
2.3.4 Have we overcome the barriers? .......................................................... 14
2.4

2

Public DNS and DoT/DoH ................................................................. 14© Internet Initiative Japan Inc.

Nov.2019

Vol.

44

Executive Summary

Executive Summary

In the afternoon of August 23, Amazon Web Services (AWS) experienced an approximately 6-hour outage in
the Tokyo region. The incident caused major global cloud services to go offline, and this impacted systems
and services at a great many companies, making those companies’ services unavailable to general users. While
the Internet is the ultimate distributed network, the rise of cloud computing means that data processing is becoming increasingly centralized. That an incident at a single company (AWS in this case) could have such a
far-reaching impact is a stark reminder of this centralization on the Internet. Then on August 29, a number of
ISPs reported Internet faults thought to stem from the sharp increase in traffic caused by a Microsoft Windows
Update. This was also a notable event in that the activities of a single company had a major impact on the Internet infrastructure.
The IIR introduces the wide range of technology that IIJ researches and develops, comprising periodic observation reports that provide an outline of various data IIJ obtains through the daily operation of services, as well as
focused research examining specific areas of technology.
Our periodic observation report for this issue, in Chapter 1, is our usual rundown of broadband traffic. This
report has been part of the IIR every year since 2009, and we are proud to present this valuable data tracking trends in Internet traffic over more than 10 years. This year, although we observed increases in both fixed
broadband and mobile services similar to what we saw last year, the results indicate that usage volumes at theindividual user level have not changed much over the past few years.
Our focused research report in Chapter 2 looks at DNS, one important part of the Internet’s foundations. Since
May this year, IIJ has been providing the IIJ Public DNS Service, which performs name resolution via DNS over
TLS (DoT) and DNS over HTTPS (DoH). The service is available to anyone, not just IIJ users. The report explains technical aspects of DoT/DoH and how they differ from DNSSEC and then discusses the IIJ Public DNS
Service implementation and the customizations we made.
Through activities such as these, IIJ strives to improve and develop its services on a daily basis while maintaining the stability of the Internet. We will continue to provide a variety of services and solutions designed toserve the full needs of the infrastructure that underpins companies’ business endeavors.

Junichi Shimagami
Mr. Shimagami is a Senior Executive Officer and the CTO of IIJ. His interest in the Internet led to him joining IIJ in
September 1996. After engaging in the design and construction of the A-Bone Asia region network spearheaded by IIJ,
as well as IIJ’s backbone network, he was put in charge of IIJ network services. Since 2015, he has been responsible
for network, cloud, and security technology across the board as CTO. In April 2017, he became chairman of theTelecom Services Association of Japan MVNO Council..

© Internet Initiative Japan Inc.

3

1. Periodic Observation Report

Broadband Traffic Report:
Moderate Growth in Traffic Volume Ongoing
1.1 Overview

here. As of June 2019, IPoE accounted for 19% of IN and

In this report, we analyze traffic over the broadband ac-

14% of OUT broadband traffic overall. Respectively, thesecess services operated by IIJ and present the results eachare year-on-year increases of 7 and 6 points, so use of IPoE

yearr*1*2*3*4*5*6*7*8*9*10. Here, we again report on changes in

is rising.

traffic trends over the past year, based on daily user traffic
and usage by port.

Growth in both broadband and mobile traffic has risen, withsome ups and downs, over the past two years or so. These

Figure 1 shows the overall average monthly traffic trends forfluctuations in broadband and mobile have been mostly syn-

IIJ’s fixed broadband services and mobile services. IN/OUT

chronous with each other, suggesting that the underlying

indicates the direction from the ISP perspective. IN represents

factors are the same.

uploads from users, and OUT represents user downloads.
Because we cannot disclose specific traffic numbers, weOver the past year, broadband IN traffic increased 12% andhave normalized the data, setting the latest OUT observation

broadband OUT traffic increased 19%, virtually the samein each dataset to 1.

growth rates as the year-earlier figures of 12% and 20%.
Growth in mobile traffic has slowed, IN from 69% a year ear-Since the previous edition of this report, the broadband datalier to 60% this year, and OUT from 36% to 22%. And thehave included IPv6 IPoE traffic. The thin line labeled “broad-

total volume of mobile traffic remains an order of magnitude

band-IPoE” excludes IPv6 IPoE traffic. IPv6 traffic on IIJ’s

lower than broadband traffic.

broadband services comprises both IPoE and PPPoE traffic,*11

but IPoE traffic does not pass directly through IIJ’s network

1.2 About the Dataas we use Internet Multifeed Co.’s transix service for IPoE,

As with previous reports, for broadband traffic, our analysis

and IPoE is therefore excluded from the analysis that follows

uses data sampled using Sampled NetFlow from the routers that accommodate the fiber-optic and DSL broadband

1.0

Traffic Volume

0.8

customers of our personal and enterprise broadband accessIN（broadband）
OUT（broadband）
IN（broadband-IPoE）
OUT（broadband-IPoE）
IN（mobile）
OUT（mobile）

services. For mobile traffic, we use access gateway billinginformation to determine usage volumes for personal andenterprise mobile services, and we use Sampled NetFlow
data from the routers used to accommodate these services

0.6

to determine the ports used.

0.4

Because traffic trends differ between weekdays and weekends, we analyze traffic in one-week chunks. In this report,

0.2

we look at data for the week of May 27 through June 2,
2019, and compare those data with data for the week of

0
2008

2010

2012

2014

2016

2018

Month/YearMay 28 through June 3, 2018, which we analyzed in theprevious edition of this report.

Figure 1: Monthly Broadband and Mobile Traffic over Time

4

*1

Kenjiro Cho. Broadband Traffic Report: Download Growth Slows for a Second Year Running. Vol.40. pp4-9. September 2018.

*2

Kenjiro Cho. Broadband Traffic Report: Traffic Growth Slows to a Degree. Internet Infrastructure Review. Vol.36. pp4-9. September 2017.

*3

Kenjiro Cho. Broadband Traffic Report: Traffic Growth is Accelerating. Internet Infrastructure Review. Vol.32. pp28-33. August 2016.

*4

Kenjiro Cho. Broadband Traffic Report: Comparing Broadband and Mobile Traffic. Internet Infrastructure Review. Vol.28. pp28-33. August 2015.

*5

Kenjiro Cho. Broadband Traffic Report: Traffic Volumes Rise Steadily Over the Past Year, and HTTPS Use Expands. Internet Infrastructure Review. Vol.24. pp28-33. August 2014.

*6

Kenjiro Cho. Broadband Traffic Report: The Impact of Criminalization of Illegal Downloads was Limited. Internet Infrastructure Review. Vol.20. pp32-37. August 2013.

*7

Kenjiro Cho. Broadband Traffic Report: Traffic Trends over the Past Year. Internet Infrastructure Review. Vol.16. pp33-37. August 2012.

*8

Kenjiro Cho. Broadband Traffic Report: Examining the Impact of the Earthquake on Traffic on a Macro Level. Internet Infrastructure Review. Vol.12. pp25-30. August 2011.

*9Kenjiro Cho. Broadband Traffic Report: Traffic Shifting away from P2P File Sharing to Web Services. Vol.8. pp25-30. August 2010.

*10

Kenjiro Cho. Broadband Traffic Report: Increasing Traffic for General Users. Internet Infrastructure Review. Vol.4. pp18-23. August 2009.

*11

Akimichi Ogawa. Appendix A.3 “IPv6 PPPoE and IPv6 IPoE” in Professional IPv6 (in Japanese). Lambda Note. July 2018.

© Internet Initiative Japan Inc.

Nov.2019

Vol.

44

1. Periodic Observation Report

Results are aggregated by subscription for broadband

interpretable conclusions. Note that because of the difficulty

traffic, and by phone number for mobile traffic as some sub-of distinguishing between individual and enterprise usage,

scriptions cover multiple phone numbers. The usage volume

the analysis of usage by port in the next section does in-for each broadband user was obtained by matching the IP

clude enterprise data.

address assigned to users with the IP addresses observed.
We gathered statistical information by sampling packets

Figure 2 and Figure 3 show the average daily usage dis-using NetFlow. Sampling rates were set between 1/8,192

tributions (probability density functions) for broadband andand 1/16,382, taking into account router performance andmobile users. Each compares data for 2018 and 2019 split

load. We estimated overall usage volumes by multiplying

into IN (upload) and OUT (download), with user traffic vol-observed volumes with the reciprocal of the sampling rate.

ume plotted along the X-axis and user frequency along the
Y-axis. The X-axis shows volumes between 10KB (104)

IIJ provides both fiber-optic and DSL broadband services,and 100GB (1011) using a logarithmic scale. Most users

but fiber-optic access now accounts for the vast majority offall within the 100GB (1011) range, with a few exceptions.

use. Of users observed in 2019, 98% were using fiber-optic
connections and accounted for 99% of overall broadband

The IN and OUT broadband traffic distributions are close

traffic volume.

to a log-normal distribution, which looks like a normal distribution on a semi-log plot. A linear plot would show a

1.3 Users’ Daily Usage

long-tailed distribution, with the peak close to the left and aFirst, we examine daily usage volumes for broadband andslow gradual decrease toward the right.

mobile users from several angles. Daily usage indicates theaverage daily usage calculated from a week’s worth of data

The OUT distribution is further to the right than the IN dis-for each user.

tribution, indicating that download volume is more than anorder of magnitude larger than upload volume. The peaks

Starting with this edition, we use daily usage data onlyof both the IN and OUT distributions for 2019 are slightly

on services provided to individuals. Enterprise usage var-further to the right than the peaks of the 2018 distributions,

ies widely and is readily influenced by the usage patterns

indicating that overall user traffic volumes are increasing.

of a subset of users, such that the overall distribution is

But that rightward shift in the distribution in 2019 was

clearly distorted. Individual usage, by contrast, shows asmaller than it had been in the past few years.

smooth distribution that remains stable. So we determinedthat using only the individual data to ascertain usage pat-

The peak of the OUT distribution, which appears toward theterns would yield more generally applicable and easilyright in the plot, has steadily been moving rightward over

0.8

0.8

0.4

0.2

0 4
10
（10KB）

2018（IN）
2018（OUT）
2019（IN）
2019（OUT）

0.6

Probability Density

Probability Density

0.6

Mobile（2019）2018 (IN)2018 (OUT)
2019 (IN)2019 (OUT)

0.4

0.2

5

6

10
10
（100KB） （1MB）

7

8

9

10
10
10
（10MB） （100MB） （1GB）

10

11

10
10
（10GB） （100GB）

0 4
10（10KB）

5

10（100KB）

6

10
（1MB）

7

8

10
10
（10MB） （100MB）

9

10
（1GB）

10

Users’ Daily Traffic Volume (Bytes)

Users’ Daily Traffic Volume (Bytes)

Figure 2: Daily Broadband User Traffic Volume Distribution
Comparison of 2018 and 2019

Figure 3: Daily Mobile User Traffic Volume Distribution
Comparison of 2018 and 2019

© Internet Initiative Japan Inc.

11

10
10
（10GB） （100GB）

5

the past few years, but heavy-user usage levels have notdata when out of the home/office as well as limits on mo-increased much, and as a result, the distribution is becoming

bile data. Hence, the daily average for a week’s worth ofless symmetric. The IN distribution on the left, meanwhile, isdata shows less variability between users than the data

generally symmetric and closer to a log-normal distribution.

for individual days. Plotting the distributions for individualdays in the same way results in slightly lower peaks and

The data for mobile traffic indicate that usage volumes arecorrespondingly higher tails on both sides, but the basic

significantly lower than for broadband. And limits on mo-shape and modal values of the distribution remain largely

bile data usage mean that heavy users, which fall on theunchanged. The difference in the mobile distributions versus

right-hand side of the distribution, account for only a smalllast year is also minimal.

proportion of the total, so the distribution is asymmetric.
There are also no extremely heavy users. The variability in

Table 1 shows trends in the mean and median daily trafficeach user’s daily usage volume is higher for mobile than forvalues for broadband users as well as the mode (the most fre-broadband owing to there being users who only use mobile

quent value, which represents the peak of the distribution).

Table 1: Trends in Mean and Mode of
Broadband Users’ Daily Traffic Volume
IN (MB/day)OUT (MB/day)

Year

Mean

MedianMode

MeanMedian

Mode

2007

436

5

5

718

59

56

2008

490

6

6

807

75

79

2009

561

6

6

973

91

100

2010

442

7

7

878

111

126

2011

398

9

9

931

144

200

2012

364

11

13

945

176

251

2013

320

13

16

928

208

355

2014

348

21

28

1124

311

501

Year

Mean

Median

2015

351

32

45

1399

443

708

2015

6.2

2016

361

48

63

1808

726

1000

2016

7.6

Table 2: Trends in Mean and Mode of
Mobile Users’ Daily Traffic Volume
IN (MB/day)
MeanMedian

Mode

3.2

4.5

49.2

23.5

44.7

4.1

7.1

66.5

32.7

63.1

2017

391

63

79

2285

900

1259

2017

9.3

4.9

7.9

79.9

41.2

79.4

428

66

79

2664

1083

1585

2018

10.5

5.4

8.9

83.8

44.3

79.4

2019

479

75

89

2986

1187

1995

2019

11.2

5.9

8.9

84.9

46.4

79.4

11

10

11

10

10

10

9

10

8

10

7

10

6

10

5

Mobile (2019)

Total (2019)
10

10
10
10
10
10
10

User’ s Daily Upload Volume (Bytes)

10

User’ s Daily Upload Volume (Bytes)

Mode

2018

10

9

8

7

6

5

4
4

10
(10KB)

6

OUT (MB/day)

5

10(100KB)

6

10
(1MB)

7

10
(10MB)

8

10(100MB)

9

10
(1GB)

10

10
(10GB)

11

10(100GB)

10 4

4

10
(10KB)

5

10(100KB)

6

10
(1MB)

7

10
(10MB)

8

10(100MB)

9

10
(1GB)

User’ s Daily Download Volume (Bytes)

User’ s Daily Download Volume (Bytes)

Figure 4: IN/OUT Usage for Each Broadband User

Figure 5: IN/OUT Usage for Each Mobile User

© Internet Initiative Japan Inc.

10

10
(10GB)

11

10(100GB)Nov.2019

Vol.

44

1. Periodic Observation Report

The peak was slightly off from the center of the distribution,

For broadband traffic, there was previously a clearly recog-so the distribution was adjusted to bring the mode toward

nizable cluster of heavy users spread out thinly about thethe center. Comparing the values for 2018 and 2019, theupper right of the diagonal, but this is now no longer dis-IN mode rose from 79MB to 89MB and the OUT mode rose

cernible. Variability between users in terms of usage levels

from 1,585MB to 1,995MB, translating into a growth factor

and IN/OUT ratios is wide, indicating that there is a diverseof 1.3 for both IN and OUT. Meanwhile, because the means

range of usage styles. There is almost no difference be-

are influenced by heavy users (on the right-hand side of thetween these plots and those for 2018, too.

distribution), they were significantly higher than the corresponding modes, with the IN mean being 479MB and the

For mobile traffic, the pattern of OUT being an order of magni-

OUT mean being 2,986MB in 2019. The 2018 means weretude larger also applies, but usage volumes are lower than for

428MB and 2,664MB, respectively.

broadband, and there is less variability between IN and OUT.

For mobile traffic (Table 2), the mean and modal values are

Figure 6 and Figure 7 show the complementary cumulative

close owing to the lack of heavy users. In 2019, the IN

distribution of users’ daily traffic volume. On these log-log

mode was 9MB and the OUT mode was 79MB, while theplots, the Y-axis values represent the proportion of users

means were IN 11MB and OUT 85MB. The modes for bothwith daily usage levels greater than the corresponding

IN and OUT traffic were identical to those for the previ-

X-axis values. These plots are an effective way of exam-

ous year. The modes were unchanged but the means andining the distribution of heavy users. The linear-like decline

medians increased, which reflects a slight decrease in the

toward the right-hand side of the plots indicates that the

proportion of light users, corresponding to the part of the

distributions are long-tailed and close to a power-law distri-

distribution to the left of the peak in Figure 2.

bution. Heavy users appear to be distributed statistically anddo not appear to constitute a separate, special class of user.

Figure 4 and Figure 5 plot per-user IN/OUT usage volumes
for random samples of 5,000 users. The X-axis shows

On mobile, heavy users appear to be distributed accord-

OUT (download volume) and the Y-axis shows IN (upload

ing to a power-law for OUT traffic, but the linear-like slope

volume), with both using a logarithmic scale. Users withbreaks down somewhat for IN traffic, with a large propor-

identical IN/OUT values fall on the diagonal.

tion of users uploading large volumes of data.

The cluster spread out below and parallel to the diagonal in

Traffic is heavily skewed across users, such that a smalleach of these plots represents typical users with download

proportion of users accounts for the majority of overall traf-

volumes an order of magnitude higher than upload volumes.

fic volume. For example, the top 10% of broadband users

10

0

10

In
Out

-1

In
Out

-1

10

-2

Probability Distribution

Probability Distribution

10

0

10

-3

10

-4

10

-5

-2

10

-3

10

-4

10

-5

10

10
Total (2019)

-6

10

4

10
(10KB)

5

10(100KB)

Mobile (2019)
6

10
(1MB)

7

10
(10MB)

8

10(100MB)

9

10
(1GB)

10

10
(10GB)

11

10(100GB)

12

10
(1TB)

-6

10

4

10
(10KB)

5

10(100KB)

6

10
(1MB)

7

10
(10MB)

8

10(100MB)

9

10
(1GB)

10

10
(10GB)

Users’ Daily Traffic Volume (Bytes)

Users’ Daily Traffic Volume (Bytes)

Figure 6: Complementary Cumulative Distribution of
Broadband Users’ Daily Traffic Volume

Figure 7: Complementary Cumulative Distribution of
Mobile Users’ Daily Traffic Volume© Internet Initiative Japan Inc.

11

10
(100GB)

12

10
(1TB)

7

account for 52% of total OUT and 82% of total IN traffic,

Table 3 shows the percentage breakdown of broadband

while the top 1% of users account for 17% of OUT and 58%

users’ usage by port over the past five years. In 2019, 81%

of IN traffic. As the proportion of heavy users has declinedof all traffic was over TCP connections. The proportion ofover the past few years, the skew has also decreased, albeit

traffic over port 443 (HTTPS), which was down a little last

only slightly. As for mobile, the top 10% of users account for

time, rose sharply from 41% to 52%. The proportion of traf-

43% of OUT and 47% of IN traffic, while the top 1% accountfic over port 80 (HTTP) fell from 27% to 20% here, and thefor 12% of OUT and 18% of IN traffic. The skew is less pro-figure for UDP port 443, which is used by Google’s QUIC

nounced than indicated in our reports up to last year because

protocol, fell from 10% to 8% after having risen up until the

we are now looking only at data on individuals.

previous edition of this report. These figures demonstratethat the shift from HTTP to HTTPS is ongoing, while QUIC

1.4 Usage by Port

has tapped the brakes on growth a little.

Next, we look at a breakdown of traffic and examine usage
levels by port. Recently, it has become difficult to identify

TCP dynamic port traffic, which has been on the de-applications by port number. Many P2P applications use dy-

cline, fell to 8% in 2019. The proportion accounted fornamic ports on both ends, and a large number of client/

by individual dynamic port numbers is tiny, with the mostserver applications use port 80, which is assigned to HTTP,

commonly used port 8080 only accounting for 0.5%. Port

to avoid firewalls. Hence, generally speaking, when both

1935, which is used by Flash Player and has also been inparties are using a dynamic port numbered 1024 or higher,

decline, fell to around 0.3%. Almost all other traffic here

the traffic is likely to be from a P2P application, and when

is VPN related.

one of the parties is using a well-known port lower thanap-

Table 4 shows the percentage breakdown by port for mobile

plication. In light of this, we take the lower of the source

users. The figures are close to those for broadband on theand destination port numbers when breaking down TCP andwhole, suggesting that mobile users use applications in a

UDP usage volumes by port.

manner similar to broadband users.

1024, the traffic is likely to be from a client/server

Table 3: Broadband Users’ Usage by Port

8

year

2015

2016

2017

2018

2019

protocol port

(%)

(%)

(%)

(%)

(%)

TCP

80.8

82.8

83.9

78.5

81.2

(< 1024)

63.3

69.1

72.9

68.5

73.3

Table 4: Mobile Users’ Usage by Port

year

2015

2016

2017

2018

2019

443 (https)

23.3

30.5

43.3

40.7

51.9

protocol port

(%)

(%)

(%)

(%)

(%)

80 (http)

37.9

37.1

28.4

26.5

20.4

TCP

93.8

94.4

84.4

76.6

76.9

993 (imaps)

0.1

0.1

0.2

0.2

0.3

443 (https)

37.4

43.7

53.0

52.8

55.6

22 (ssh)

0.2

0.2

0.1

0.1

0.2

80 (http)

52.5

46.8

27.0

16.7

10.3

182

0.4

0.3

0.3

0.3

0.2

31000

0.0

0.2

1.8

2.9

6.4

(>= 1024)

17.5

13.7

11.0

10.0

7.89

993 (imaps)

0.5

0.5

0.4

0.3

0.3

8080

0.3

0.2

0.3

0.3

0.5

1935 (rtmp)

0.5

0.3

0.2

0.1

0.1

1935 (rtmp)

1.8

1.5

1.1

0.7

0.3

UDP

5.2

5.0

11.4

19.4

17.3

UDP

11.4

11.1

10.5

16.4

14.1

443 (https)

1.0

1.5

7.5

10.6

8.3

443 (https)

0.9

2.4

3.8

10.0

7.8

12222

0.0

0.1

0.1

2.3

3.4

4500 (nat-t)

0.2

0.2

0.2

0.2

0.3

4500 (nat-t)

0.3

0.2

0.2

4.5

3.0

ESP

7.4

5.8

5.1

4.8

4.4

53 (dns)

0.1

0.2

0.1

0.1

0.1

IP-ENCAP

0.2

0.2

0.3

0.2

0.2

ESP

0.7

0.4

0.4

3.9

5.8

GRE

0.2

0.1

0.1

0.1

0.1

GRE

0.3

0.1

0.1

0.1

0.0

ICMP

0.0

0.0

0.0

0.0

0.0

ICMP

0.0

0.0

0.0

0.0

0.0

© Internet Initiative Japan Inc.

Nov.2019

Vol.

44

1. Periodic Observation Report

Figure 8 compares overall broadband traffic for key port

1.5 Conclusioncategories across the course of the week from which obser-

Traffic volume has been growing moderately over the pastvations were drawn in 2018 and 2019. We break the data

few years. Although I say “moderately”, it’s only moderate

into four port buckets: TCP ports 80 and 443, dynamic ports

in relation to past growth. At its annual rate of 20%, traffic(1024 and up), and UDP port 443. The data are normalized

is set to more than double over a four-year period. Bothso that peak overall traffic volume on the plot is 1. By com-

broadband and mobile traffic have been increasing, albeit

parison with 2018, the proportion of traffic over TCP portwith some ups and downs. The fact that both tend to rise

443 has risen even further whereas TCP port 80 has seen

and fall around the same time suggests that common fac-a decrease. The overall peak is between 19:00 and 23:00

tors are at play, but we have not been able to pinpoint what

hours. Traffic increases during the daytime on Saturday andthe specific factors are.

Sunday, reflecting household Internet usage times.
Both broadband and mobile usage volume by user have not
Figure 9 shows the trend for TCP ports 80 and 443 andchanged much in the past few years. No new services that

UDP port 443, which account for the bulk of mobile traf-

would drive traffic upward have appeared over that time,

fic. In the mobile space as well, TCP port 443 has seen an

and it is clear that users’ Internet usage has not changed

increase while the proportion of traffic over TCP port 80

much as a result. Video resolutions are definitely on the rise,has declined. When compared with broadband, we note that

but it looks like the accompanying rise in codec compression

mobile traffic levels remain high throughout the day, fromrates is keeping total traffic growth in check.

morning through night. The plot shows that usage times
differ from those for broadband, with three separate mobile
traffic peaks occurring on weekdays: morning commute,lunch break, and evening from 17:00 to 22:00 hours.

Figure 8: Broadband Users’ Port Usage Over a Week
2018 (top) and 2019 (bottom)

Figure 9: Mobile Users’ Port Usage Over a Week
2018 (top) and 2019 (bottom)

Kenjiro Cho
Research Director, Research Laboratory, IIJ Innovation Institute Inc.

© Internet Initiative Japan Inc.

9

2. Focused Research

About the IIJ Public DNS Service

2.1 Introduction

As it became clear that DNS was also being monitoring

IIJ released the beta version of its IIJ Public DNS Service inunder PRISM, the IETF began developing mechanisms toMay. It is a caching DNS service that only accepts DNS overensure DNS privacy, until then a perfunctory affair, through

TLS (DoT) and DNS over HTTPS (DoH), meaning that it does

its new DPRIVE (DNS PRIVate Exchange) Working Group.

not support name resolution via the usual UDP/TCP setup,

DPRIVE has published various protocol extensions/revisions

and it is available to anyone, not just IIJ users.

for DNS, including Qname Minimisation (RFC 7816) andEDNS(0) Padding Option (RFC 7830, RFC 8467), with trans-

This report explains how DoT/DoH differs from the usual

port encryption being of relatively high importance among

DNS setup and describes key considerations and future

these efforts.

challenges for IIJ in providing this service.
2.2.2 DNS Transport Encryption

2.2 What is DoT/DoH

Traditional DNS mainly uses UDP for the lower-level pro-

2.2.1 DNS and Privacy

tocol (transport), supplementing this with TCP. However,Information registered to the DNS is assumed to be publicly

plain UDP/TCP, and DNS over UDP/TCP, do not have aand widely available. For a long time, therefore, the focus of

mechanism to provide confidentiality, and communications

DNS security has been to ensure that the information is not

are easily eavesdropped since they take place in the clear.

tampered with (i.e., that its integrity is maintained), whereas

A decision was therefore made to insert an encryption layer

ensuring that the information is not intercepted (i.e., that it

between DNS and the lower layers to provide protection.

remains confidential) has not been a priority.
Various encryption layers have been proposed, with the
In 2013, however, the Snowden affair revealed the exis-

following having been standardized so far: DNS over TLS

tence of PRISM, an extensive communications monitoring(RFC 7858), DNS over DTLS (RFC 8094), and DNS overand information collection program carried out by the US

HTTPS (RFC 8484). A draft of DNS over QUIC has also been

National Security Agency (NSA). This prompted the IETF to

submitted to the IETF and is now under discussion. And if

declare that “Pervasive Monitoring Is an Attack” (RFC 7258)

HTTP/3, which is also currently under discussion, is stan-and call for protocols to be designed to withstand pervasive

dardized, DoH will automatically support HTTP/3 (Figure 1).

monitoring going forward.

DoD

DoQ

DoH

Traditional DNS
(UDP)

(TCP)

DoT

DoH

DNS

HTTP/3DTLS

HTTP/2

QUIC

TLS

UDP

TCP

IPv4/IPv6

Figure 1: DNS Transport Protocols

10© Internet Initiative Japan Inc.

Nov.2019

Vol.

44

2. Focused Research

These various encryption layers are not integrated into any one

In general, encryption guarantees both integrity andprotocol; at present, you simply select whichever suits yourconfidentiality, but when it comes to DNS transport en-users’ circumstances. But flooding the space with a bunch

cryption, encryption only happens between the user andof different protocols can have its drawbacks, so it is quite

the caching server. Since communication between theconceivable that a subset will be selected, and the rest depre-

caching server and the authoritative server use traditional

cated, at some point in the future. (At present, DNS over DTLS

DNS (not encrypted), the integrity of the information ob-

is a specification only, with no existing implementations, and it

tained by the caching server cannot be guaranteed, andseems unlikely that it will become available for use.)

its integrity cannot be guaranteed even if the information
is protected by encryption. That is, unlike encryption in

2.2.3 Transport Encryption and DNSSEC

general, the DNS transport encryption protocol only guar-

DNS already has a mechanism for verifying DNS informa-

antees confidentiality between the user and the cachingtion via digital signatures called DNSSEC. So why do weserver (Figure 2).

need a new method of transport encryption when we have
DNSSEC? And will transport encryption eventually makeDNSSEC, meanwhile, was introduced to protect against

DNSSEC unnecessary?

data forging and manipulation. It uses digital signatures toguarantee integrity, but communication itself takes place in

Before answering this question, let’s look at the scope ofthe clear and is not confidential.

transport encryption. With DNS, clients do not directly query
the server on which the master DNS information is registered

So both transport encryption and DNSSEC are mechanisms(authoritative server); instead, they query caching servers

for improving DNS security, with one focused on guarantee-

provided by ISPs and other parties. In general, the cachinging confidentiality without integrity and the other focusedservers are responsible for querying the authoritative server.

on guaranteeing integrity without confidentiality. In thatsense, they are each other’s complement, so one cannot

Transport encryption is currently only performed between

replace the other. Each protects something different, which

the user and the caching server. Communications between

means that transport encryption does not obviate the need

the caching server and the authoritative server use tradi-for DNSSEC, and vice versa.

tional DNS and are not encrypted.

Root server

Registry’s
authoritative server

Caching server

User

example.com’s
authoritative serverProtected by DoT/DoH

Traditional DNS used
Figure 2: Scope of Transport Encryption© Internet Initiative Japan Inc.

11

2.3 IIJ Public DNS Service and DoT/DoH

Although it is known that using TCP would preclude or

DoT and DoH differ from traditional DNS only in terms of trans-

greatly reduce the threat of such attack methods, the mas-

port-layer protocol. They both use the same DNS protocol assive overhead that would result from the use of TCP withtraditional DNS. Yet we still had a number of barriers to over-

DNS—because DNS, by its nature, involves the exchange ofcome to provide them as a service. Let’s look at each in turn.

a large number of small packets—has dissuaded the community from shifting to TCP as the main transport protocol.

2.3.1 The TCP barrier

Even when the Kaminsky Attack*1, which allows vastly

The major difference between DoT/DoH and traditional DNS

more efficient cache poisoning that with previously known

is that all communications take place via TCP before being

methods, was revealed in 2008, we still didn’t switch toencrypted using TLS. While traditional DNS can use either

TCP and opted instead to make do with UDP and treat the

TCP or UDP as the transport protocol, in most cases UDP issymptoms with source port randomization*2.

used, with TCP only being used in limited cases.
So there was considerable aversion to the TCP overheadWith TCP, a session must be established before communi-with DNS, but requiring TLS naturally also means requiring

cations on higher-layer protocols can start. TCP also uses

TCP. The DNS over DTLS protocol, which uses UDP, does

various techniques to ensure the reliability of transmissions,exist, but it entails a large overhead just like TCP, and more-which include checking that sent packets were in fact re-over, no one can use it because it is only a specification; no

ceived and resending them if necessary.

implementations exist.

DNS involves very little data exchange. In most cases, both

Nonetheless, we must abandon the conventional wisdom if

the query and response rarely exceed a few hundred bytes.

we are to use it as a foundation for providing a secure cach-When TCP is used here, the process of setting up and later

ing DNS service. A major factor in enabling us to provide

terminating the session accounts for vastly more of thethe IIJ Public DNS Service without restriction to anyone incommunications exchanged than the actual DNS message, re-the world is that we do not use DNS over the low-reliability

sulting in extremely poor efficiency. Possible solutions to load

UDP protocol, which dispels concern of the service being

problems include the extravagant approach of simply adding

used as a launchpad for a DNS amp attack or such like.

more servers, but nothing can really be done about increased

We should focus squarely on the benefits that using TCP

latency stemming from an increase in packet roundtrips.

actually provides.

UDP has no such mechanisms, making it fast and simple,

2.3.2 The TLS Barriers

and it is widely used in protocols that involve the exchange

DoT and DoH eschew UDP and use TCP, resulting in aof small packets, such as DNS and NTP. Its reliability, onlarge overhead, and they employ a TLS encryption layer

the other hand, is low, and many of the attacks on DNS

above the TCP layer. TLS is widely used in HTTPS andthat have so far been discovered, such as cache poison-

elsewhere, but it is certainly not lightweight, and it causes

ing and DNS amp, really stem from the use of UDP in thesignificant performance degradation with protocols thatlower layer rather than from any problems with the DNS

necessitate the high-speed exchange of small amounts ofprotocol itself.

data, such as DNS.

*1

For example, JPRS Topics & Columns, “Aratanaru DNS cache poisoning no kyoui: Kaminsky Attack no shutsugen” [New DNS cache poisoning threat: Emergenceof the Kaminsky Attack] (https://jprs.jp/related-info/guide/009.pdf, in Japanese only)*2

Randomizing the source used by clients when sending queries increases the number of factors that an attacker needs to guess when seeking to forge packets,
which reduces the probability of a successful attack.

12© Internet Initiative Japan Inc.

Nov.2019

Vol.

44

2. Focused Research

Although logic says we will take a significant performanceback to NLnet Labs, and the latest available version now in-hit compared with traditional UDP-based DNS, we needcludes our code.

tools if we are to measure how much performance actuallydegrades. TCP-based DNS has been in use for some time,

Aside from performance issues, TLS poses one more barrier,albeit only in limited cases, and tools do exist. But DNS

namely that communications are encrypted.

over TLS is completely new. And there are no satisfactory
tools for measuring performance. We needed to measure

To ensure a stable DNS service, you need a mechanism thathow much performance degrades and what the processing

allows you to collect statistics to make sure that a large num-load would be so that we could estimate just how much

ber of abnormal queries are not being sent, and that abnormal

equipment we would need to provide the service, and to do

responses are not increasing despite the queries being normal,

this we had to start by developing performance measure-so that you can investigate and take action if abnormalities

ment tools.

do arise. With traditional DNS, in most cases this sort of statistics gathering and troubleshooting was done not on the

Since TLS involves a very high processing load, options

DNS server itself but by capturing DNS packets. Since thisare available to reduce the overhead by, for example, re-process can be performed independently of the DNS server,using information from a previous connection to resume

the same method can be used regardless of what DNS servera session (TLS session resumption) and, in TLS 1.3 (theimplementation is used.

latest version), adding the application data to the initial
ClientHello/ServerHello exchange that takes place during

But with TLS, the captured packets are encrypted. Perfect

the handshake (0-RTT).

Forward Secrecy (PFS) is now commonplace, so packets cannot be decrypted even if you have the server’s private key.

But having certain functions available in the TLS protocol isThis means that the tools so far used to collect information

meaningless unless applications actually use them. Running

are no longer viable. The ability to collect statistics is needed

services in a large-scale production environment will be diffi-

for private testing of course, and it is indispensable if you plan

cult unless you make full use of these options.

to make something widely available as a service, so much sothat you may as well shelve the service without it. So we

The IIJ Public DNS Service uses Unbound, a DNS implemen-

had to rebuild the statistics collection functionality from thetation developed by NLnet Labs in the Netherlands. It is quite

ground up to enable us to gather the same sort of information

old and supported TLS before DoT became an IETF draft.

from the DNS server without relying on packet capture, andWhen we investigated Unbound’s TLS support, however,with this in place we were finally ready to launch the service.

we learned that it lacked those mechanisms for reducing the
TLS overhead; specifically, it lacked TLS session resumption

2.3.3 The HTTP barriercapabilities. And performance measurements also indicated

The barrier posed by HTTP was actually not all that high.

that performance was inadequate. Further, performance differs greatly depending on what encryption algorithm is used,

With DoH, once TLS has been applied, the DNS message

but this was hardcoded into the source. So IIJ decided tomust then be encapsulated in an HTTP message. Getting

implement the necessary features. The results were passed

the DNS server itself to speak HTTP would be quite a© Internet Initiative Japan Inc.

13

challenge, but a two-stage setup where an ordinary HTTP

users are constantly unable to use the Internet because aserver receives the queries, converts the message format,

name cannot be resolved.

and passes them to a DNS server behind it would be prettymuch just like that used by any other Web application out

Moreover, it looks like performance is even worse on

there, aside from the fact that the backend is a DNS server.

Android 10 (currently under development as of this writing)than it is on Android 9.

Clearly because we are using TCP instead of the traditional
UDP, as well TLS, the latency and other performance issues

Other than this, we have had not major problems. Latency

are unavoidable. But because we take care of the hard parts

should theoretically be worse than with traditional DNS, but

in the HTTP layer, which has a proven track record, and not

it does not appear to be a problem in practice and we havein the DNS layer, which would require us to fumble aroundnot received any complaints.

in unfamiliar territory, this should not be seen as too muchof a setback.

DoT and DoH are still new technologies and the basic specifications have only just been finalized. A lot of the peripheral

2.3.4 Have we overcome the barriers?

specifications are yet to be sorted out (for instance, DoT/As of this writing, the service has been live for about six

DoH servers presently can only be configured by hand; net-months.

work administrators are unable to distribute configurations
for automatic deployment).

We did create a bit of a stir when we published the press
release since this is the first DoT/DoH service in Japan, andGoing forward, we will continue to investigate issues with

the Android DoH client Intra*3 subsequently added an IIJ

the aim of making improvements and implement new spec-

Public DNS list option for users to select (we did not evenifications that are on the road to becoming standards in thehave to ask for the option to be added).

hopes that people will be able to use the latest technologies
with peace of mind. And we will continue to give back by

I wish we could give a glowing account of how everything wentcommunicating the insights we glean from operating ouroff without a hitch post launch, but reality does disappoint.

service to the community.

As explained, Unbound did not have TLS session resump-

2.4 Public DNS and DoT/DoH

tion, so we implemented it ourselves, but TLS session

In closing, we look at developments beyond the IIJ Public

validity appears to be extremely short on Android 9, and our

DNS Service.

session resumption implementation has proved ineffective
in many cases. When Android smartphone users visit a Web

Originally, caching DNS servers were only available to users

page and then follow a link from that page to another one,within a particular organization. They were not public in na-

often the TLS session established to resolve the previous

ture. But the realization that making them public would notdomain name has already timed out and the whole processreally be all that harmful eventually led to most of them be-

has to start again with a handshake. This handshake fre-coming available without restriction (open resolvers). Later,quently fails if the network is congested, which means that

however, the DNS amp attack was discovered and attackers*3

14Intra (https://play.google.com/store/apps/details?id=app.intra)© Internet Initiative Japan Inc.

Nov.2019

Vol.

44

2. Focused Researchbegan to use open resolvers to stage DDoS attacks. So

services selected by Firefox would automatically be usedsince around 2010, the scope of access has generally been

unless the user configures the browser to do otherwise. But

restricted to the minimum necessary.

public DNS services, because they are public, cannot resolve
namespaces such as those found on intranets. And services

Meanwhile, other services aimed at a wide range of users

like parental controls that use DNS will be ineffective if thesought to foil attacks by implementing rate limitations andbrowser selects its own DNS server instead of what thethe like, rather than address-based restrictions. The pi-

OS settings dictate. In light of these points, there is debate

oneer here is OpenDNS , but this was later followed byabout the pros and cons of imposing DoH as default.

*4

the Google Public DNS*5 service, and ever since that gained
traction, these sorts of explicitly open resolvers have gener-

Google Public DNS supports DoT/DoH, and use of these

ally been called “public DNS” services.

protocols ensures confidentiality between the user and
Google. Meanwhile, Google also supports EDNS0 Client

The DoT RFC became a standard in 2016. The public

Subnet (ECS; RFC 7871). With ECS, when a client queries a

DNS service Quad9 , launched in November 2017, andcaching server, the caching server relays information about

Cloudflare*7, launched in April 2018, supported DoT fromthe network to which the client belongs to the authorita-

the start, and Google also added support in January 2019.

tive server. This is intended to be useful in content delivery*6

traffic management. But it is important to note that tradiDoH officially became an RFC in October 2018, but im-

tional DNS is always used between caching servers that

plementations based on the draft came out ahead of that.

use ECS and the authoritative server. If the user is using

Cloudflare provided support upon launch in April 2018,

DoT/DoH, communications on the route between the user

Quad9 added support two weeks before RFC 8484 was re-and Google will not be eavesdropped, but communications

leased, and Google also later added support in June 2019.

on the route between Google and the authoritative server
can be eavesdropped because traditional DNS, which does

On the client side of things, Android has supported DoT at

not guarantee confidentiality, is used here, and this could

the OS level since August 2018, and the DoH client app

result in a breach of user privacy since the ECS information

Intra was released in October 2018. Among Web browsers,

traverses that route.

Firefox added DoH support in August 2018. In Chrome’scase, only the development version supported it as of this

There is now a definite move toward encrypting DNS trans-

writing, but the stable version may also have it by the timeport, and it is unlikely to be stopped at this point. But as

this is published.

providers of a new public DNS service, we have an important responsibility not to blindly accept every development

Mozilla has stated that it plans to make DoH the default forthat unfolds but to evaluate each one by one and determinename resolution in Firefox , meaning that the public DNS

if it really is the right way to proceed.

*8

Takanori Yamaguchi
Application Service Section, Application Support Department, IIJ. Mr. Yamaguchi works on support for DNS services etc.

*4

OpenDNS (https://www.opendns.com/)*5

Google Public DNS (https://developers.google.com/speed/public-dns/)*6

Quad9 (https://www.quad9.net/)*7

Cloudflare (https://developers.cloudflare.com/1.1.1.1/)*8

Firefox Nightly News, “What’s next in making Encrypted DNS-over-HTTPS the Default” (https://blog.mozilla.org/futurereleases/2019/09/06/whats-next-in-makingdns-over-https-the-default)© Internet Initiative Japan Inc.

15

44
November 2019 Vol.
About Internet Initiative Japan Inc. (IIJ)
IIJ was established in 1992, mainly by a group of engineers who
had been involved in research and development activities related
to the Internet, under the concept of promoting the widespreaduse of the Internet in Japan.
IIJ currently operates one of the largest Internet backbones
in Japan, manages Internet infrastructures, and provides
comprehensive high-quality system environments (includingInternet access, systems integration, and outsourcing services,etc.) to high-end business users including the government andother public offices and financial institutions.
In addition, IIJ actively shares knowledge accumulated throughservice development and Internet backbone operation, andis making efforts to expand the Internet used as a social
infrastructure.­­The copyright of this document remains in Internet Initiative Japan Inc.
(“IIJ”) and the document is protected under the Copyright Law of Japanand treaty provisions. You are prohibited to reproduce, modify, or makethe public transmission of or otherwise whole or a part of this document
without IIJ’s prior written permission. Although the content of thisdocument is paid careful attention to, IIJ does not warrant the accuracy andusefulness of the information in this document.
Initiative Japan Inc. All rights reserved.
© Internet
IIJ-MKTG020-0042

Internet Initiative Japan Inc.
Address: Iidabashi Grand Bloom, 2-10-2 Fujimi, Chiyoda-ku,
Tokyo 102-0071, Japan
Email: info@iij.ad.jp URL: https://www.iij.ad.jp/en/