What part of “NO” is so
hard to understand?
Geoff Huston
APNIC Labs

We were looking elsewhere…
• We were setting up a measurement experiment that was looking at
the extent of support for aggressive NSEC caching (RFC8198) in theDNS• The experiment setup involved presenting to the user a DNS namethat did not exist from a signed zone, so that we would pass an NSEC
record to a DNSSEC-aware resolver• But was was intriguing was that we were seeing many more queries
for the non-existent name than we had expectedWhat we saw:
• We used an online ad to get users to query for aunique non-existent DNS name• And then counted the number of queries we saw forthese names
uuid.example.com?
NXDOMAIN
uuid.example.com?
NXDOMAIN
uuid.example.com?
NXDOMAIN

Authoritative Server

What we saw:
• We used an online ad to get users to query for aunique non-existent DNS name• And then counted the number of queries we saw forthese names

Queried Names: 60,210,983
DNS Queries: 142,631,272• That’s an average of 2.37 queries per non-existentname!

Why so many queries?

NamesRepeat Queries

42%
58%

Expectations• If “NO means NO” then naïvely we would expect to see 1 query per
name, not 2.37 queries per name• But maybe that’s just too naïve these days…
• After all - the DNS is now SO clever that this just couldn’t be due torandom DNS insanity – could it?

Happy Eyeballs and the DNS
A ‘happy eyeballs” dual stack client will launch 2 DNS queries back-toback (roughly), for A and AAAA records of the name
• 23% of clients asked both A and AAAA records• 3% asked only AAAA records (*)• 74% asked only for A records

AAAA?
A?

Client stub
resolver* If the client asks for an AAAA record and waits for a response before asking for
the A record then the NXDOMAIN response will stop the connection process andany subsequent A query will not be performed

Factoring Happy Eyeballs• If we split out the A and AAAA queries the experiment launched
73,537,852 DNS resolution ‘events’• We saw 142,631,272 DNS queries, or an average of 1.93 queries pername
Names

That’s better, but still
unexpectedly highHappy Eyeballs

Repeat Queries

42%

48%

10%

Single vs Multiple Queries• 36,059,484 resolution ‘events’ were completed with just 1 query to
the authoritative server (49% of all resolution events)• If there were multiple queries for a name (>= 2 queries), then theaverage of the multiple queries was 2.84 queries

Distribution of Queries• Are averages misleading here?
• Is this a generic issue of re-queries across a large set of queries?
• Or a small number of queries that are the subject of a total frenzy ofre-queries?

Re-query Distribution!! Seriously????

32% of queries were parts of query sequences of 3 or more

Does UDP suck THAT much?
• Why is the total re-query rate at 51% of tests?
• Surely DNS over UDP is not THAT bad• The servers are responding to every query• The signed response is 603 bytes in size
• We are using a distributed setup of servers to localize DNS transactions• So why are the servers seeing 51% of tests generate 2 or more queries?

Re-Query Time Intervals

There are strong local peaks atregular 1 second intervals – thiswould appear to be an end host /stub resolver re-query behaviour

Re-Query Time Intervals
There are local re-query peaks here at
370 ms, 800ms, 1 sec and 1.5sec
370ms

It is likely that these time intervals
represent stub and/or recursive resolver
re-query timers

800ms
1s

1.5s

DNSSEC?
• This is a DNSSEC-signed non-existent name
• i.e. the NXDOMAIN also has an NSEC record if DO is set in the query• Is DNSSEC a factor in the excessive re-query volume?
• i.e. is the additional time to validate causing requery timers to trigger?

• We added an unsigned non-existent name to the test setRe-Queries per resolver IPaddresss
The log scale exaggerates the effect, but weobserve that a DNSSEC-signed NXDOMAIN
response generates a higher repeat query
profile from the same resolver IP addressSigned vs Unsigned
SignedUnsigned

Experiments
A/AAAA
Queries

65,686,452
81,057,694
153,697,947

69,251,349
84,979,990
122,665,888

Single Query Exps
Ratio

47,694,930
59%

60,061,746
71%

Multi-Query Exps
Re-Query Rate

33,092,764
3.19

24,918,244
2.51

Split out the ‘happy eyeballs’ factor

DNSSEC validation adds delay, and in around 12% of cases thisadditional delay causes the resolver system to re-query the nameDNSSEC!
• About 12% of cases of re-query for non-existent names appear to berelated to recursive resolver’s DNSSEC validation of NSEC records

stupid

Resolver “farms”
We also see query patterns of the form:
Resolver7x.xxx.0.178
6z.zzz.161.146
7x.xxx.0.230
6z.zzz.161.220
7x.xxx.0.188
6z.zzz.161.182
7x.xxx.0.180
6z.zzz.161.144
7x.xxx.0.226
7x.xxx.0.138
6z.zzz.161.134
6z.zzz.161.222

Query Time
0.752
0.8650.980
1.094
1.201
1.319
1.430
1.542
1.650
1.654
1.762
1.775

It appears that some resolver farms operate by farming
the query across all members of the farm. This patternseen here shows two such cases where different IP
addresses in the same subnet repeat the initial query at
approximately 100ms intervals
How common is this form of subnet-based query
repetition?

Re-query Profile
Re-queries
Same IP Address
Same Subnet

59,782,873
17,848,729
41,111,416

Wow!

Some 70% of the re-queries are from resolvers thatshare the same subnet prefix!!!
This points to some outstanding issues with resolver
farm management

It may sound odd but…
• Is NO worse than YES?
• Is NXDOMAIN part of the issue here?
• Is the re-query rate lower if the name exists in the DNS?

NXDOMAIN vs A/AAAA re-queries
NXDOMAIN Signed:
41% of experiments generate multiple queries
39% of queries are re-queries (avg of 3.19 queries per experiment)
A/AAAA Signed:
18% of experiments generate multiple queries
13% of queries are re-queries (avg of 5.81 queries per experiment)

NXDOMAIN Unsigned:
39% of experiments generate multiple queries
29% of queries are re-queries (avg of 2.51 queries per experiment)
A/AAAA Unsigned
38% of experiments generate multiple queries
36% of queries are re-queries (avg of 3.03 queries per experiment)

Fewer re-query
events but each
event has morequeries!

WTF?
• A DNSSEC-signed NXDOMAIN response generates many more requeries than a DNSSEC-signed A / AAAA responsePulling it back together• Why are there so many DNS repeat queries?
Names

Happy Eyeballs

DNSSECNXDOMAIN

Farmed Load4%13%
42%

14%

17%

10%

Other

Pulling it back together• Why are there so many DNS repeat queries?
• Happy Eyeballs

Names

Happy Eyeballs

DNSSECNXDOMAIN

Farmed Load

4%
13%
42%

14%

17%

10%

Other

Pulling it back together• Why are there so many DNS repeat queries?
• Happy Eyeballs• DNSSEC

Names

Happy Eyeballs

DNSSECNXDOMAIN

Farmed Load

4%
13%
42%

14%

17%

10%

Other

Pulling it back together• Why are there so many DNS repeat queries?
• Happy Eyeballs• DNSSEC• NXDOMAIN

Names

Happy Eyeballs

DNSSECNXDOMAIN

Farmed Load

4%
13%
42%14%17%10%

Other

Pulling it back together• Why are there so many DNS repeat queries?
• Happy Eyeballs• DNSSEC• NXDOMAIN• stupidity down on the farm

Names

Happy Eyeballs

DNSSECNXDOMAIN

Farmed Load

4%
13%
42%

14%

17%10%

Other

Pulling it back together• Why are there so many DNS repeat queries?
• Happy Eyeballs• DNSSEC• NXDOMAIN• Stupidity down on the farm
• <reasons>

Names

Happy Eyeballs

DNSSECNXDOMAIN

Farmed Load

4%
13%
42%

14%

17%10%

OtherThanks!