TLD Anycast DNS servers to ISPs

- to Create a More Resilient DNS Environment -APRICOT 2017
Shoji Noguchi (JPRS),
Yoshibumi Suematsu (QTNet)

Copyright © 2017 Japan Registry Services Co., Ltd., and Kyushu Telecommunication Network Co., Inc.

1

Agenda1. Concept of “.jprs”
2. Background of Joint Research
3. Overview of Joint Research
4. Joint Research Report by JPRS
5. Joint Research Report by QTNet

Copyright © 2017 Japan Registry Services Co., Ltd.

2

1. Concept of “.jprs”

Copyright © 2017 Japan Registry Services Co., Ltd.

3

“.jprs” R&D Platform
Concept of “.jprs”
In order for the Internet to keep growing, as a registry
operator, we will need an environment in which tocreate innovations …
.jprs TLD can provide experimental environment for
domain names and DNS.
For demonstration of experiments– For innovative technologies and productions– Difficult to implement in a production environment

For collaborative R&D with research and business partners– Autonomous research pertaining to the Internet
Copyright © 2017 Japan Registry Services Co., Ltd.

4

2. Background of Joint Research

Copyright © 2017 Japan Registry Services Co., Ltd.

5

Japan is a Disaster-prone CountryNatural disasters in/around Japan
Earthquake, tsunami, typhoon, and volcanic eruption…

Collapsed highway
The Great Hanshin/Awaji Earthquake on Jan 17, 1995 [*1]

Tsunami Surging to town
The Great East Japan Earthquake on Mar 11, 2011 [*2]

The importance of being prepared fornatural disasters[*1] http://sciencewindow.jst.go.jp/html/sw23/sp-003[*2] http://www.bousai.go.jp/kohou/kouhoubousai/h23/63/special_01.html

Copyright © 2017 Japan Registry Services Co., Ltd.

6

Physical Geography of Japan
Characteristics of national land
Japan is an unexpectedly large land. * Geographical latitude of Japan is

not same as that of Southeast Asia.

Approx.
3,000 km

Geographical features do not always pose potential danger
to all regions of Japan but specific regions in many cases.
From <http://thetruesize.com/#/aboutModal?borders=1~!MTE3ODk5MzY.NzQyNzMzNw*MzMzOTgyNDc(MjQ5NjA3NDc~!JP*Mzg5NTA4MA.NDc5NjA5Mw(MTQw)OQ>
Copyright © 2017 Japan Registry Services Co., Ltd.

7

Logical Structure of Internet in Japan
Characteristic of Internet structureInternet resources are concentrated inTokyo and Osaka.
Internet Exchanges (IXs),
Transit connections,
Data Centers, and so on

Osaka
Tokyo
Approx.
400km

Natural disasters that occur in/around Tokyo/Osaka
can intensely affect Internet connectivity.
Copyright © 2017 Japan Registry Services Co., Ltd.

8

Goal - Enhancing the DNS Resiliency
By locations of DNS servers
Install DNS servers in several regions other than Tokyo
and Osaka.
Mitigation/distribution of DNS traffic concentration
Stabilization/reduction of RTT
Continuous provision of DNS/Internet services
In case of an outage of the DNS servers in a particular region

Copyright © 2017 Japan Registry Services Co., Ltd.

9

3. Overview of Joint Research

Copyright © 2017 Japan Registry Services Co., Ltd.

10

.jprs DNS servers
Installation sites of .jprs DNS servers
Hostname

Location

Remarks

tld1.nic.jprs

Tokyo

Osaka

tld2.nic.jprs

Osaka

tld2

tld3.nic.jprs

Worldwide

tld4.nic.jprs

Tokyo

tld5.nic.jprs

Worldwide

For R&D

Tokyo
tld1
tld4

The above locations resemble those of .jp DNS.

.jprs DNS operations.jprs DNS servers in Tokyo/Osaka are operated by JPRS.
Copyright © 2017 Japan Registry Services Co., Ltd.

11

ApproachJoint research effort with 8 domestic ISPs
Each of their service area covers designatedgeographical areas without overlapping.

How to direct DNS query to Local Node
Install tld4.nic.jprs DNS servers into
their networks as Local Node.
A) Routing configuration
BGPB) Full resolver configuration
“static-stub” zone type for BIND
– Specifying DNS servers’ IP addresses
Copyright © 2017 Japan Registry Services Co., Ltd.

Global Node (JPRS)
Local Node (ISP)
12Measurement Environment Model.jprs DNS - Global Nodes
AS18149

ISP network

ISP 1 - AS XXX1

Stub Resolver

tld1.nic.jprs

AS2914

Full Resolver

AS12041

2LD App server- Web & DNS
www.example.jprs

tld2.nic.jprs
tld3.nic.jprs
tld5.nic.jprs

Internet

Local NodeAS131905

tld4.nic.jprs

AS131905

tld4.nic.jprs

ISP 8 – AS XXX8

Collecting Data

SourceDestination

Continuous ISP’s Internet service

Stub Resolver

2LD App server – Web

wget

1

Continuous Name Resolution

Stub Resolver

2LD App server – DNS

dig

1

Reachability of .jprs DNS servers

Full Resolver

tld[1-5].nic.jp (JPRS)+
tld4.nic.jprs (ISP)

ping

1

traceroute1

tcpdumpwithout a pause

Copyright © 2017 Japan Registry Services Co., Ltd.

Tools

Interval [min.]

13

ISPs’ Measurement Environment
ISPs used routing/full resolver configuration andDNS software.
ISP

Configuration(A) Routing(B) Full resolver

DNS Software (BIND) version
Full Resolver

HOTnet(A) BGP

9.9.7

TOHKnet(A) BGP

9.9.4-RedHat-9.9.4-29.el7_2.3

HTNet(B) “static-stub”

9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6

K-OPT(A) BGP

9.9.4-RedHat-9.9.4-29.el7_2.2

Enecom(B) “static-stub”

9.9.8-P4

STNet(B) “static-stub”

9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6

QTNet(A) BGP

9.9.8-P4

OTNet(A) BGPN/A (Non-disclosure for business use)
Copyright © 2017 Japan Registry Services Co., Ltd.

Local Node9.9.8-P4

14

Evaluation Methods
 Continuous Internet service availability and reachabilityof .jprs DNSCompare traffic behavior with/without Local Node.
< Without Local Node >Global Nodes.jprs DNS are locatedonly outside ISP network.

< With Local Node >Global Nodes

Local Nodes.jprs DNS are locatedinside/outside ISP network.

Copyright © 2017 Japan Registry Services Co., Ltd.

15

Evaluation Methods
 Continuous Internet service availability and reachabilityof .jprs DNSCompare traffic behavior by turning on/off Internet
connection (hereinafter called Normal/Disaster).
< Without Local Node >Global Nodes

All .jprs DNS are unreachable
from inside ISP network.

< With Local Node >Global Nodes

Local Nodes

Only Local Node inside ISP isreachable from inside ISP network.

Copyright © 2017 Japan Registry Services Co., Ltd.

16

4. Joint Research Reportby JPRS
Copyright © 2017 Japan Registry Services Co., Ltd.

17

Enhancing the DNS Resiliency

Local Node
W/O
Normal-Disaster-WithReachability of .jprs DNS confirmed at full resolvers
Destination of .jprs DNS queries from full resolver
had inclined toward Local Node on their ISP network.
Number of DNS queries

Queries/to each .jprs DNS (IPv4+IPv6) by K-OPT
hour
120

<Normal><Disaster>80
40
06/26 6/26 6/27 6/27 6/27 6/27 6/28 6/28
13:00 19:00 1:00 7:00 13:00 19:00 1:00 7:00

GN
(tld1)

GN(tld2)

GN(tld3)

LN(tld4)

GN(tld5)

Measured using tcpdump
command at ISPs’ full resolvers.

Full resolver  .jprs DNS servers
Execute on a minute-by-minute basis
at stub resolver (dig command)* GN: Global Node, LN: Local Node

Copyright © 2017 Japan Registry Services Co., Ltd.

18

Findings at Full Resolver- Normal & W/O Local Node

Local Node
W/O

With

Normal--Disaster--Geographic distribution of DNS trafficOver concentration of DNS query to Tokyo and Osaka
DNS query by Destination and ISP at ISPs’ full resolvers
ISP

Dest.

Tokyo
tld[14]

Osaka
tld2

Worldwidetld[35]

Total

Global Node
(Tokyo, Osaka)

HOTnet@Sapporo

3,646

170

1,200

5,016

ISP

HTNet

1,476

154
(8.3)

224

(12.1)

1,854

222

1,488
(72.4)

346(16.8)

2,056

812

678

319

1,809

@Kanazawa

281km
292km
486km

828km K-OPT@OsakaTokyo Enecom@Hiroshima

401km
Osaka

(72.7)(79.6)(10.8)(3.4)(23.9)(100.0)(17.6)

QTNet@Fukuoka

2,242

3,286

2,514

8,042

Total

8,398

5,776

4,603

18,777(44.7)(30.8)(31.3)(24.5)

Copyright © 2017 Japan Registry Services Co., Ltd.

Tokyo
44.7%

(100.0)(37.5)(40.9)

Worldwide
24.5%(100.0)(44.9)(27.9)[queries (ratio)](100.0)(100.0)(100.0)

Osaka
30.8%

Measurement period:
2016/6/23 13:00 ~ 6/24 13:00
(JST)
19

Findings at Full Resolver- Disaster & With Local Node

Local Node
W/O

With

Normal--Disaster--De-concentration of DNS query to Tokyo and Osaka
DNS query by Destination and ISP at ISPs’ full resolversTokyo
tld[14]

Osaka
tld2

World
tld[35]

Local
Node

Total

HOTnet

102*

96*

162*

5,108

5,468

HTNet

36*

34*

80*

2,066

2,216

K-OPT

112*

96*

158*

2,102

2,468

Tokyo Enecom

-

-

-

1,638

1,638

QTNet

-

-

-

8,236

Total

250*

226*

400*

Global Node
(Tokyo, Osaka)
ISP

281km
292km
486km

828km

401km
Osaka

ISP

Dest.

(2.2)
(6.0)
(4.5)

(1.3)
(1.0)
(3.9)

(2.4)
(5.2)
(6.4)

(94.1)(87.8)(85.2)(100.0)(100.0)[queries (ratio)]*: packet loss Worldwide

Tokyo
2.5%*
Osaka
2.2%*

3.9%*

(100.0)8,236

19,150 20,026

Local Node
91.4%

Measurement period:
2016/6/23 13:00 ~ 6/24 13:00(JST)

DNS query addressed to other than Local Node ≠ Zero
Affected by NS selection algorithm in BIND
Copyright © 2017 Japan Registry Services Co., Ltd.

20

Local NodeFindings at Stub Resolver

W/O
Normal-Disaster-WithChanges in RTT: Normal  DisasterCompared between “(A) BGP” and “(B) static-stub.”Case “(A) BGP” - 5 ISPs selected
S.D. of RTT affected by loss was 4~10 times larger.
Setting initial timeout to 800ms from BIND 9.6.0a1.
RTT of DNS query
Avg. + S.D. [ms]HOTnet@Sapporo

0.7 + 6.7
 1.9 + 31.2

K-OPT
@Osaka7.2 + 5.4
 8.2 + 54.8

QTNet
@Fukuoka

3.7 + 17.6
 8.9 + 69.0

20

40

Copyright © 2017 Japan Registry Services Co., Ltd.

60

80

Measurement period:
2016/6/26 13:00 ~ 6/28 13:00
(JST)
21

Local NodeFindings at Stub Resolver

W/O
Normal-Disaster-WithCase “(B) static-stub” - 3 ISPs selectedThis function is provided by BIND 9.8.0 or above.
Forcing DNS queries for a zone to go to specified IP addresses

S.D. of RTT affected by loss was ~2 times larger.
S.D. of RTT(B) was 2~8 times smaller than that of (A).
RTT of DNS query
Avg. + S.D. [ms]HTNet
@Kanazawa

2.5 + 11.3
 2.6 + 14.8

Enecom@Hiroshima

2.5 + 3.4
 2.6 + 7.7

20

40

60

80

HOTnet : 31.2
K-OPT : 54.8
QTNet : 69.0
Measurement period:
2016/6/26 13:00 ~ 6/28 13:00(JST)

In a particular (B), DNS query inclines toward a destination
with a shorter RTT.
It depends on how we use either or both of (A) and (B).
Copyright © 2017 Japan Registry Services Co., Ltd.

22

Secondary Findings- Behavior of “static-stub”

Local Node
W/O
Normal-Disaster

-

With
-

RTT of Enecom was shorter than that of HTNet.
Full Resolver
DNS Software

RTT of DNS query [ms]Avg. + S.D.

Enecom

BIND 9.9.8-P4

0.9 + 0.8

HTNet

BIND 9.8.2rc1

5.1 + 4.6

5

10

15

20

Confirmed the trend of DNS queries.
Enecom - BIND 9.9.8– Sent one DNS query to all .jprs DNS servers only once a day
at 0:00 AM, and sent all the other DNS queries to Local Node.

HTNet - BIND 9.8.2– Sent DNS queries to all .jprs DNS servers approximately every 6hours, and sent all the other DNS queries to Local Node.

Assumed that the difference is a load balancing algorithm.
Copyright © 2017 Japan Registry Services Co., Ltd.

23

Continuity of Internet services

Local Node
W/O
Normal-Disaster-WithEffect of installing Local Node into ISP network
All 8 ISPs were able to continue offering their Internet
service inside their own network. 
Number of Success to access .jprs
Successes/ 2LD Web Server by K-OPT Measured using wget command at
ISPs’ stub resolvers.

hour
60
50
40
100%!
30
20
10
0
6/26 6/26 6/27 6/27 6/27 6/27 6/28 6/28 6/28
13:00 19:00 01:00 07:00 13:00 19:00 01:00 07:00 13:00<Normal><Disaster> Stub resolver  .jprs 2LD Web server Execute on a minute-by-minute basis

 All 8 ISPs could continue receiving
the results of success!

Copyright © 2017 Japan Registry Services Co., Ltd.

24

Future Works
Relating to this activity
Sharing ISPs’ .jprs DNS servers, or Local Nodes, among
themselves
Evaluation of geographical dispersal of DNSRoot, TLD, and 2nd level domain DNS servers
 Vertical integration
Full resolver and authoritative DNS
 Horizontal integration

Copyright © 2017 Japan Registry Services Co., Ltd.

25

5. Joint Research Reportby QTNet
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

26

Kyushu Telecommunication Network

AS7679

Company Name Kyushu Telecommunication Network Co., Inc. (QTNet)Telecommunications carrier in Kyushu , Japan

Services

Our coverage area:
Area: 36.750km2 (10% of Japan)
Population: 13.2Mil. (10% of Japan)

Wide-Area Ethernet
FTTH
Internet Accsess, VoIP, and TV
Kyushu is one of the 8 regions of Japan
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

27

Internet in Kyushu
 Internet connectivity in Japan concentrates in Tokyo and Osaka.
→The network of Kyushu depends on these areas.

What are problems?
 If the large disasters simultaneously
Approx.
hit in/around Tokyo and Osaka
Approx.
401km
486km
IX
 Isolated from other regions of Japan
Full Resolver(QTNet) Cannot provide our Internet services in Kyushu
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

28

Recent Natural Disasters in Kyushu
The task of immediate importance
Earthquake in Kyusyu (Kumamoto)
Date & Time（JST） Magnitude
14 April 21:26

6.5

14 April 22:07

5.8

15 April 00:03

6.4

16 April 01:25

7.3

16 April 01:45

5.9

16 April 03:55

5.8

16 April 09:48

5.4

DNS Query received at Full Resolver[qps]
requests received

M7.3
M5.8
04/15 04/15 04/15 04/15 04/15 04/15 04/15 04/15 04/16 04/16 04/16
06:00 08:20 10:40 13:00 15:20 17:40 20:00 22:20 00:40 03:00 05:20

Felt earthquakes have been approximately 4000 times since Oct. 10, 2016

In any situation, we must provide our customers with our Internet services!
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

29

Damage by Earthquake

Kumamoto Earthquakes in Apr. 14, 2016

https://ja-jp.facebook.com/kyuden.jp/posts/940170829434491
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

30

BackgroundThe advantage of using “.jprs”
 ".jprs" registry operator is same as ".jp," which is ccTLD for Japan Ratio of DNS queries by TLD in our FTTH service: “.jp” is 2nd place.
TLD ranking of request for DNS query

net
com

jp

com

jp
net
arpa
org
other

There are many DNS queries for .jp.
Many important customers have used .jp.
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

31

TLD Anycast DNS servers to QTNet

Simulating the isolation of Kyushu.

Disconnect

Process
LN

GN
1

GN
2

GN
3

GN
4

GN
5

#1
#2
#3
#4
#5

Number of DNS queries to each .jprs DNS[Queries/Hour]

80
60

#1

#2

#3

(Full resolver  .jprs DNS servers)

#4

#5

40
20
0
01/08
13:00

01/09
03:00

01/09
17:00

01/10
07:00

GN(tld1)

01/10
21:00

01/11
11:00

01/12
01:00

GN(tld2)

01/12
15:00

01/13
05:00

GN(tld3)

01/13
19:00

01/14
09:00

01/14
23:00

GN/LN(tld4)

01/15
13:00

01/16
03:00

01/16
17:00

01/17
07:00

01/17
21:00

GN(tld5)

DNS service could be provided
even under the condition that Kyushu had been isolated.
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

32

Other results

 Other interesting results by setting local node to ISP.

[ms]

Number of RTT & hop to each .jprs DNS<Normal>35.0<Disaster>

Number of DNS queries to each .jprs DNS[hop] [Queries/Hour]<Normal>30.0 70

30.0<Disaster>25.0 60

25.0

20.0

50
40

20.0
15.0
15.0
10.0

10.0

30
20

5.0 10

5.0
0.0

0.0
04/25
05:00

04/25
07:00

04/25
09:00

04/25
11:00

04/25
13:00

04/25
15:00

04/25
17:00

04/25
19:00

04/25
21:00

tld1(RTT)

tld2(RTT)

tld3(RTT)

tld4(RTT)

tld5(RTT)

tld1(hop)tld2(hop)tld3(hop)tld4(hop)tld5(hop)

Collecting Data

Source

0
04/25
05:00

04/25
07:00

04/25
09:00

04/25
11:00

04/25
13:00

tld1v4

tld2v4

tld3v4

Destination

04/25
15:00
tld4v4

04/25
17:00

04/25
19:00

04/25
21:00

tld5v4Tools

Interval [min.]RTT & hop

Stub Resolver.jprs DNS

traceroute1

DNS query

Stub Resolver.jprs DNS

dig

1

The full resolver(BIND) preferentially selects .jprs DNS with shorter RTT.
Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

33

Conclusion
Installing TLD Anycast DNS server in QTNet.
Providing DNS service could be continued under theconditions such as Kyushu is isolated.
Reducing RTT by installing .jprs local node.

Installing local node is effective both in normal
times and in large-scale disasters.
Future work
To install .jp local node in Kyushu! :-)Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.

34

Contact Us
Email:URI:

dotjprstestbed-sec@jprs.co.jp
https://nic.jprs/

Copyright © 2017 Japan Registry Services Co., Ltd.

35

APPENDICES

Copyright © 2017 Japan Registry Services Co., Ltd.

36

Sample of “static-stub” zone type
named.conf for BIND
% cat /etc/named-without-localnode.conf% cat /etc/named-with-localnode.conf(snip)(snip)

zone "jprs." {type static-stub;server-addresses {// Global Nodes (JPRS)
103.47.2.1;// tld1.nic.jprs
2001:dda::1;// tld1.nic.jprs
117.104.133.16;// tld2.nic.jprs
2001:218:3001::1;// tld2.nic.jprs
65.22.40.1;// tld3.nic.jprs
2a01:8840:1ba::1;// tld3.nic.jprs
103.198.210.1;// tld4.nic.jprs
2403:2880::1;// tld4.nic.jprs
65.22.40.129;// tld5.nic.jprs
2a01:8840:1ba::129; // tld5.nic.jprs

zone "jprs." {type static-stub;server-addresses {// Global Nodes (JPRS)
103.47.2.1;// tld1.nic.jprs
2001:dda::1;// tld1.nic.jprs
117.104.133.16;// tld2.nic.jprs
2001:218:3001::1;// tld2.nic.jprs
65.22.40.1;// tld3.nic.jprs
2a01:8840:1ba::1;// tld3.nic.jprs//
103.198.210.1;// tld4.nic.jprs//
2403:2880::1;// tld4.nic.jprs
65.22.40.129;// tld5.nic.jprs
2a01:8840:1ba::129; // tld5.nic.jprs

Rem// Local Nodes (ISP)
192.0.2.53;// tld4.nic.jprs
2001:db8::53;// tld4.nic.jprs

Add
};

};

};

};

(snip)(snip)
Copyright © 2017 Japan Registry Services Co., Ltd.

37

Local Node

Available to Local Node

W/O

With

Normal--Disaster--Mitigation of DNS traffic
De-concentration of DNS query to Tokyo and Osaka
DNS query by Destination and ISPs at ISPs’ full resolversTokyo
tld[14]

Osaka
tld2

World
tld[35]

Local
Node

Total

HOTnet

114

70

128

4,954

5,266

HTNet

128
(6.0)

(1.0)

22

112

1,880

828km K-OPT

122

176
(8.0)

(10.2)

224

1,678(76.3)(100.0)

18

11

(0.7)

13

(0.8)

1,592

1,634

Global Node
(Tokyo, Osaka)
ISP

281km
292km
486km

Tokyo

401km
Osaka

ISP

Dest.

Enecom
QTNet

Total

(2.2)

(5.5)
(1.1)

972

(1.3)

(2.4)
(5.2)

(94.1)(87.8)(97.4)(100.0)

2,142
2,200

(100.0)(15.1)

890

1,854

2,988

(50.6)(100.0)

1,696

2,067

3,071

13,484

20,318

(7.5)

(6.5)

(13.0)(73.0)

Copyright © 2017 Japan Registry Services Co., Ltd.

Tokyo
7.5%

Osaka
6.5%

(100.0)(16.4)(31.4)[queries (ratio)]

5,894(100.0)Worldwide
Local Node 13.0%

73.0%

Measurement period:
2016/6/24 13:00 ~ 6/25 13:00(JST)38