Deployment of TLD Anycast nodeto ISPs for stability and resiliency
APRICOT 2021
Takayasu Matsuura (JPRS)
Nagisa Yano (OPTAGE)

Copyright © 2021 Japan Registry Services Co., Ltd.,

1

Agenda• “.jprs” R&D Platform and Joint Research 2017
• Background of Joint Research 2021
• Overview of Joint Research• Joint Research Report by ISPs
• Consideration of LN deployment in Japan

Copyright © 2021 Japan Registry Services Co., Ltd.

2

“.jprs” R&D Platform and Joint Research 2017
 Concept of “.jprs”
In order for the Internet to keep growing, as a registry operator,
we will need an environment in which to create innovations …
.jprs TLD can provide experimental environment for domainnames and DNS

 Joint Research 2017
Japan as a disaster‐prone country, joint experiments withdomestic ISPs were conducted to verify the effectiveness ofdeploying TLD DNS Server inside the domestic ISPs against thephysical Internet divide.
“TLD Anycast DNS servers to ISPs”
https://2017.apricot.net/program/schedule/#/day/9/network‐operations‐2
Copyright © 2021 Japan Registry Services Co., Ltd.

3

Background of Joint Research 2021

The Tokyo Olympic Games are scheduled
to take place in 2021. However, it isnecessary to prepare for large‐scale cyber‐attacks that occur at every Olympic Games.
During the most recent Olympic Games, cyber‐attacks such as malware and DDoS attacks ofseveral hundred Gbps were launched against
sites related to the Games.

Need to prepare for massive DDoS
attacks

Copyright © 2021 Japan Registry Services Co., Ltd.

4

Logical Structure of Internet in Japan
Characteristic of Internet structure
Critical Internet resources are highly
Concentrated in Tokyo and Osaka.
Internet Exchanges (IXs),
Transit connections,
Data Centers, and so on
Root Servers and TLD DNS
Servers are also concentratedTokyo

Root Servers,
TLD Servers

Submarine
cableOsaka
Submarine
cable

Root Servers,
TLD Servers

When cyber attacks are targeted at facilities located in Tokyo and Osaka, ISPs in otherregions will also not able to resolve names. They will virtually lose access to local
internet resources. We call such situation “logical internet divide”.
Copyright © 2021 Japan Registry Services Co., Ltd.

5

Goal of the Project:
Enhance DNS Resiliency to logical Internet divide
 Prepare for Cyber Attacks toward 2021 Tokyo Olympic and Paralympic Games Deploy local nodes, so that DNS name resolution can continue even in the event of aDDoS attack on DNS servers in Tokyo and Osaka.
[Local Nodes Deployed]
Place Local Nodes within
Japan to reduce theimpact of logicallydivided Internet[Attack Situation]
Once TLD DNS and/or ISP
DNS stop, the Internet
will be logically divided

LN

GNGNLN

GNGN.jprs DNS
Global Node

LN.jprs DNS
Local Node
(BGP NO‐EXPORT)LN

ISP (nameresolution
not affected)

ISP (nameresolutionaffected)

Copyright © 2021 Japan Registry Services Co., Ltd.

GNCircuitCircuit(logicallydivided)

Trafficcongestion byCyber Attacks
6Overview of Joint Research Joint research effort with 3 domestic ISPs
Each of their service area covers designated geographical area
without overlap

 Way to direct DNS queries to Local Nodes
Deploy tld4.nic.jprs DNS server into
ISP networks as Local Node(BGP Anycast).

Global Node (JPRS)
GN
tld[1‐5].nic.jprs
LN Local Node (ISP)
tld4.nic.jprs (bgp no‐export)

Hokkaido

Osaka

Fukuoka

LN

Copyright © 2021 Japan Registry Services Co., Ltd.

LNWorldWideLNGNTokyoGNGNOsaka
7

Scenarios and Measurements Performed DDoS attack scenarios
 DDoS attack to DNS server without local nodes (global nodes only)
 DDoS attack to DNS server with local nodes (and global nodes)

 Measurements
 Verify continuity of name resolution from probes placed on the Internet andprobes placed inside the ISP
Description of States(1) GN only

(2) Attack to GN(3) Deploy the LN
Probesplaced onthe ISPs

GNGN

Probes placedon the Internet

GNGNLN(4) Attack to GN(5) Attack to LN

LN

LN

LNLN
GNGN

LN

DDoS Probes placed
Attack on the Internet
Copyright © 2021 Japan Registry Services Co., Ltd.

LN
GN
GN

LN

DDoS
Attack

LN
GN
GN

DDoS
Attacks

8Results of JPRS Experiment(1) GN only

(2) Attack to GN(3) Deploy the LN
Probesplaced onthe ISPs

GN

GN

GN

GN

(2)

(1)

(3)
msec

30

410

(5) Attack to LN

LN

LN

LN

LN
GN
GN

(3)

msec
20

LN

(4) Attack to GN

LN

LN
GN
GN

LN

DDoS
AttackDDoS Probes placed
Attack on the Internet

Probes placedon the Internet(1)

Description of States

(4)

LN
GN
GN

DDoS
Attacks

(5)

(3)

200
10:15 10:30 10:45 11:00 11:15 11:30 11:45 12:00 12:15 12:15 12:30 12:45 13:00 13:15 13:30 13:45 14:00 14:15 14:30 14:45 15:00 15:15 15:30 15:45 16:00 16:15 16:30 16:45 17:00
Probe on the Internet

Probe on HOTnet

Probe on OPTAGE

Probe on QTnet

Some of the OPTAGE graphs are not stable because of the experimental network.

 JPRS monitored the status of Local Node from probes set up in each region. InScenario No.4, we observed the LN servers name resolution unaffected locally.
 Scenario No.5 shows that the Local Node is able to absorb the attack.
Copyright © 2021 Japan Registry Services Co., Ltd.

9

Joint Research Reportby OPTAGE
Copyright © 2021 Japan Registry Services Co., Ltd.

10

OPTAGE Inc.
(AS17511)

Company Name

OPTAGE Inc. (former: K-opticom Corporation)
 Telecommunications carrier in Osaka , Japan

Services
 FTTH

1.6 million
Subscribers

 Internet Access VoIP TV

 MVNOFTTH Coverage Area
→ Kansai Region
Osaka
Kyoto
Hyogo

Hokkaido

Nara
Shiga
Wakayama
Hokuriku SI/ICT Solution, etc.

Tokyo

Chugoku

1.2 million
Subscribers

Chubu

KyushuOkinawa

TohokuShikoku

MVNO Coverage Area
→ All over Japan
11

Background

The advantage of using “.jprs”• ".jprs" registry operator is the same as ".jp,"
which is ccTLD for Japan.
• The number of registered domains:“.jp”is 1st place.
Number of Registered Domains (OPTAGE).JP.COM.NET.INFOMany important customers use“.jp.”12Results: ping RTT (1/2)(1) GN only

(2) Attack to GN(3) Deploy the LN(4) Attack to GN

LN

GN

msec

(1)

20

GN

(2)

GN

(1)

GN
LN

(3)

LN

LN
GN
GN

LN

(4)

GN

(5) Attack to GN, LN
LN

LN
GN
GN

LN

LN
GN
GN

(3)

(5)

LN100
10:15

10:45

11:15
11:45
Switch point

12:15

12:45

13:15
13:45
14:15
RTT_AVG_tld4_ipv4

14:45

15:15

15:45

16:15

16:45

Note: The situation (1)~(2): LN is not in service.
The situation (3) or later LN works because BGP route to the same IP as GN is advertised.

• This graph indicates ping RTT from the full resolver inside of our experimental
network to the authoritative name server.
• It shows the authoritative name server was switched from GN to LN at Switch point.
• At the situation (2) obviously it was not responding, but at the situation (4) it was.

14

Results: dig RTT (2/2)(1) GN only

(2) Attack to GN(3) Deploy the LN(4) Attack to GN

LN

GN

msec

(1)

1500

GN

(2)

GN

(1)

GN
LN

LN
GN
GN

LN

LN

(4)

(3)

(5) Attack to GN, LN
LN

LN
GN
GN

LN

(5)

GN

LN
GN
GN

(3)

LN

1000
500
0
10:15

10:45

11:15

11:45

12:15

12:45

13:15

13:45

14:15

14:45

15:15

15:45

16:15

16:45

Note: The situation (1)~(2): LN is not in service.
The situation (3) or later LN works because BGP route to the same IP as GN is advertised.

• This graph indicates dig RTT from LN, the probe set up inside of our experimental
network, to the full resolver which is also set up inside of ours.
• Domain name can be resolved by LN at the situation (4), which is the purpose ofthis experiment.
• We assume that some singular points (1500ms, but status:NOERROR) are caused bytemporary server delay due to heavy load.
13

Conclusion
General comments• Finally we found it effective to locate LN in ISP network.
• LN makes it possible to resolve domain name while GN is beingattacked.
• Also we can deal with LN-down because it occurs only whenattackerʼs source IP is ISP usersʼ.

Future tasks• The trigger and the timing to switch from GN to LN,and vice versa• The method of zone data synchronization between GN and LN• The redundancy of LN placement, which depends on the time
it takes to recover GN15

Cyber Attack Response Training In order to prepare for Cyber Attacks on LNs, exercises were conducted among TLD
operators and ISP operators to ensure that they can detect, communicate andrespond to attacks. (In general, LNs are often installed with smaller performancecompared to GNs due to lower query volume in normal times.)
Global Nodes

Local Nodes
Stub Resolver

GN

ISP 1LN

Full Resolver

GN

ISP DNSOperator

ISP 2...

LNInternet

ISP 3
DDoS
AttackGN
dnstapSharing and coordination of attack
information (Slack, Mail)

notification
Attack
Detection
SystemLN

TLD DNS Operator
Copyright © 2021 Japan Registry Services Co., Ltd.

16

Lessons Learned from Training Lessons Learned
 Sharing information among TLD Operator and ISP Operators is quite useful In practice, each ISP has its own operation body and contact point, thus
different contact paths and methods are required However, informal community to exchange information among ISPs and TLD
Operators is also quite important
Attack Detection SystemCopyright © 2021 Japan Registry Services Co., Ltd.

Sharing and collaboration of attack
information (Slack)17

Consideration of LN deployment in Japan From Disaster Perspective
Japan as an island nation, the risk ofsubmarine cables being cut by
earthquakes and other disasters exists
 LNs are preferred to be placed oneach island.

 From Cyber Attack Perspective
Since facilities are concentrated inTokyo and Osaka, cyber attacks will also
be concentrated there It is preferred to deploy LNs within
ISPs which provide local services intheir region

ISP

TLD

TLD DNS Server(Global Node)LN

LN TLD DNS Server

LN(Local Node)

LN

Attacker

TLD
LN
LN

TLD

LN

LN

LN

In general, since Japan is split in 8 regions, it is better to deploy LNs in eachregional units first. (In terms of power supply and equipment as well.
Fortunately, each region has local power provider and local ISP)
Copyright © 2021 Japan Registry Services Co., Ltd.

18

Q and A If anyone has experiences of doing this or that in the past Olympic
Games, I would like to hear about it.
 Email: dotjprstestbed‐sec@jprs.co.jp

 URL: https://tldlabs.jprs/en/
Copyright © 2021 Japan Registry Services Co., Ltd.

19