DNS Anomalies andTheir Impacts on
DNS Cache Servers
Katsuyasu TOYAMA, Keisuke ISHIBASHI, andTsuyoshi TOYONO (NTT)
Masahiro ISHINO and Chika YOSHIMURA(NTT Communications)Kazunori FUJIWARA (JPRS)
1

BackgroundIn the DNS world, these ‘players’ are related toeach other
1.
2.
3.

application or operating system in user devices
authoritative servers including root DNS servers
DNS cache servers“.”
DNS Cache servers
User.comexample.
com

Authoritative servers

2

Today’s topic
The burden of DNS Cache servers caused bymisconfigured DNS authoritative servers
Lame delegation is well known, but other factors
exist as well.

Focusing on:
1.

Virus and Worm activities
An extreme increase in queries caused DNS cache server
to become overloaded.

2.

Large RRSet and TCP filtering
Even small number of queries could cause increase inheavy load of DNS cache server.
31. Virus and Worm

4

Activities of Virus and Worm Activities of viruses and worms sometimes

cause burden on DNS cache servers.
 MyDoomAttacks SCO Web site; some subspecies attack
Microsoft Web site Antinny

Attacks ACCSJP Web site ACCSJP: Association of Copyright for Computer Software5What is Antinny?
 Antinny is a worm that infects through famous

Japanese P2P software, “Winny”.
 Some subspecies of Antinny try to send private
information of the infected user to the Web siteof ACCSJP once a month.
the first Monday of the month Apr. 5th, May 3rd, June 7th…

to expose the user as a potential criminal??
 When trying to connect to the Web site, it

resolves the FQDN “www.accsjp.or.jp”
6Worms’ impacts on networks
Target website

Access line is filled with
DDoS packets

The server isunder attack!!

Backbone is also
filled with DDoSpackets
ISP

DDoS attack

Worm-infected users

DDoS attack

Worm-infected user

7

Owner removed A RR fromauthoritative server!!
Web server

DNS authoritative server(2) Iterative queries immediately
after TTL expires

Target website(3) No such record (NXDOMAIN)

Escape fromDDoS attack

Heavy load
DDoS attack(4) “NXDOMAIN”

6 times or more
Worm-infected user

DNS cacheservers(1) Worms intensively repeat
sending queries to resolve theFQDN.
8

Effect of A RR removal
 Web site owner is happy because:

it is very easy to remove A RR
their link has become very quiet
 But… DNS cache servers are overloaded!

9

Why is DNS cache overloaded?
 ‘A RR’ was removed from authoritative server.

NXDOMAIN was returned, and its TTL was short (60sec.)
 Worms repeatedly sent queries even if they

could not resolve the name.
They never gave up…
The highest was approximately 700 queries per
second from an IP address!
 Negative cache did not seem to be effective insome Operating Systems or applications
Negative cache is disabled at any time?

10

Quick fix: Return blackhole IPaddress by each cache server
DNS Authoritative
ServerHTTP Server

No iterative query

No(Fake) zone heavy load

Target website

Escape fromDDoS attack

Blackholeforwarders(4) Discard(3) DDoS attack

DNS cacheservers(2) Blackhole IP address
Worm-infected user

(1) Worms send queries
to resolve the FQDN.
11

Increased total queries at DNS cache
Apr. 5, Monday
Antinny activated

DDoS began, thenA RR was removed

Blackhole addresswas returned
May. 3, Monday
Antinny activatedGreen: Number of total queries
Blue: Number of NXDOMAIN

12

Number of queries resolving FQDN of target
Web server (May 1st – 7th, per 5 min.)This spike indicates
extremely large
number of queries

1000000# of queries (5min)

100000

Much more queries
than usual

10000
1000
100
101
5/1

5/2

▲ 5/4
Monday May 3
5/3

5/5

5/6

5/7

5/8
13

Number of queries resolving FQDN of target
Web server (May 3rd, 01:00 – 03:00, per seconds)
Queries extremely increased only when NXDOMAIN returned
Spikes whenNXDOMAIN returned

5000# of queris (1 second)

4500
4000

350060 seconds,
Every
3000
queriesincreased

Every 600 seconds,
queries increased but
not so many

2500
2000
1500
1000

500
00:00:00
Changesin answers

1:00:00

Original A RR
with TTL 60 sec

2:00:00

3:00:00

Blackhole address
with TTL 600 sec
NXDomain with TTL 60 sec 14

Number of unique IP address(May 3rd, 01:00 – 03:00, per second) # of unique IP address increased only 2-3 times

in “NXDomain period”, while queries increased
10 times or more!
#of uniqeusers(per sec)

60
50
4030
20
1000:00:00

Changesin answers

1:00:00

Original A RR
with TTL 60 sec

2:00:00

3:00:00

Blackhole address
with TTL 600 sec
15

NXDomain with TTL 60 sec

Solution:
Return Blackhole IP Address by authoritative server
In June, ISPs collaborated to defend the attack
 Asked the administrator of the authoritative server toreturn a blackhole addressWeb server

DNS Authoritative
ServerReturn Blackhole IPaddress(2) Iterative queries

Target website (6) Discard

Escape fromDDoS attack

Noheavy load
DNS cacheservers

Blackhole(5) DDoS attack

Worm-infected user(3)Blackhole IPaddress(4) Blackhole IPaddress(1) Worms send queries
to resolve the FQDN.

16

Lessons from the attack
 Removing A RR is not a good method

Behavior of worms, resolvers at user, andauthoritative servers sometimes causes DNS cacheserver to collapse.
 Collaboration between victims and ISPs isrequired.
This time blackhole IP address worked very well.
 Generic blackhole addresses are necessary

If ISPs set the blackhole addresses to be discarded attheir routers, then an authoritative server can returnthe addresses to escape from DDoS attacks.
For instance, TEST-NET (192.0.2.0/24, RFC3330)
17

2. Large RRSet and TCP filtering

18Overload of DNS cache server
 Increasing load of DNS cache server, but..
 Number of total queries was normal
 We detected increase in the number of TCPqueries
140
120

CPU Utilization of Named(%)

100

Numbers of SYN_SENT for DNS

80
60
40
20
0
6/8 0:00

6/8 6:00

6/8 12:00

6/8 18:00

6/9 0:00

6/9 6:00

19

Why TCP queries were increased The number of truncated answers and that ofTCP packets are synchronized We guessed truncated answers caused TCPqueries.
600

600500500400400300300200200100100

06/5

6/6

6/7

6/8

Number of TCP packets sent toauthoritative servers (per 5 min.)

06/5

6/6

6/7

6/8

Number of truncated answers from20
authoritative servers (per 5 min.)Why TCP queries increased
Cache server(cont’d)User
What happened was…
query
(1) The cache server sent iterative query to
the authoritative servers.
(2) The authoritative servers answered with atruncated bit; over 512 octets answer(3) Then the cache server tried to resend
query by TCP.
The authoritative servers did not respondto TCP query.
The authoritative servers didn’t support
EDNS0
TCP packets seemed to be filtered before
authoritative server process
The cache server was waiting 75 seconds
for acknowledgement from theauthoritative servers, and pending queries
ServFail
were increased.

Authoritativeservers(1)Iterativequery(2)Answer
with TC bit(3)Resend queryby TCP
No response(3)Retry(4)Try another(UDP,TCP)

No responseTimeout⇒ the cache server overloaded (did not cache the result)

No response21Dump example% dig @yyy.yyy.yyy.yyy -x zzz.zzz.zzz.zzz(1) Iterative query

21:23:14.604332 IP xxx.xxx.xxx.xxx.51705 > yyy.yyy.yyy.yyy.53: 24775+ PTR?
zzz.zzz.zzz.zzz.in-addr.arpa. (42)TC bit(2) Answer with TC bit

21:23:14.744362 IP yyy.yyy.yyy.yyy.53 > xxx.xxx.xxx.xxx.51705: 24775| 15/0/0 PTR
redirect.***.co.in., PTR onlyoriginal.com.my., PTR redirect.***.com.cn., PTRredirect.china.***.com., PTR redirect.jp.***.com., PTR ***.co.jp., PTR redirect.***.co.jp., PTR***.ne.jp., PTR redirect.***.co.kr., PTR redirect.kt***.co.kr., PTR redirect.***.com.my., PTR
redirect.***.com.ph., PTR redirect.***.com.sg., PTR redirect.***.co.th., PTR
redirect.***.com.tw. (488)(3) Resend query by TCP (SYN packet)

21:23:14.744748 IP xxx.xxx.xxx.xxx.60035 > yyy.yyy.yyy.yyy.53: S 2600114847:2600114847(0)win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 1,nop,nop,timestamp 1455660 0>
21:23:17.736935 IP xxx.xxx.xxx.xxx.60035 > yyy.yyy.yyy.yyy.53: S 2600114847:2600114847(0)win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 1,nop,nop,timestamp 1455960 0>
21:23:20.937001 IP xxx.xxx.xxx.xxx.60035 > yyy.yyy.yyy.yyy.53: S 2600114847:2600114847(0)win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 1,nop,nop,timestamp 1456280 0>
22

Workaround1.

Asked the administrators of the authoritative
serverto change the settings to accept TCP connection, orto decrease the size of their records to fit them intoa UDP packet.
--> But the administrators will not change the settings…

2.

Denied the users sending the queryby using BIND blackhole setting
23

Another possible approach Patching BIND, not to query by TCP whentruncation occurs.
Quick hacking could reduce TCP sessions, but it may
violate RFC.
Proper modification would be: Cache the following information for the RR during the TTL

TC = 1
Does not accept TCP query Return ServFail without iterative query if above cache exists…this should be proposed to IETF
24

Lessons through this phenomenon
 Strongly recommend that administrators

check the configuration of authoritativeservers.
Answer TCP queries (mandatory)
RFC 1123
DNS servers must be able to service UDP and should be able toservice TCP queries … it should not refuse to service a TCP
query just because it would have succeeded with UDP.

and either Set the size of answer to be smaller than 512

octets, or Support EDNS0 option
25

Summary
To be ‘friendly’ to DNS cache servers,
 Administrators should check and modify
the settings of authoritative servers.
Some generic blackhole address should beused, instead of removing A RR in case ofDDoS.
Configuration should be consistent.
 Oversized RRSet with no EDNS0 and closed TCP

port is not good but often seen.
26

Contacts
 Chika YOSHIMURA (NTT Communications)

yosimura@ocn.ad.jp
 Katsuyasu TOYAMA (NTT)toyama.katsuyasu@lab.ntt.co.jp

27 Special Thanks to:

Ichiro Mizukoshi (NTT Communications)Haruhiko Ohshima (NTT Communications)
Yasuhiro Morishita (JPRS)
Hirotaka Matsuoka (NTT)28