Dmap: Automating Domain Name Ecosystem
Measurements and Applications
Maarten Wullink, Giovane C. M. Moura,
and Cristian Hesselman
SIDN Labs
Arnhem, the Netherlands

giovane.moura@sidn.nl
giovane.moura@tudelft.nl

IEPG @ IETF102
Montreal, Canada

1/33

The situation

I Question: how to measure all these properties with current

measurement tools?
Registration* Registrant* Registrar* Registry

wikipedia.org* Provider* Auth.DNS
servers

Services
SMTP

HTTP* Hosting
Provider* Network* Contents* Emailprovider* Network* StartTLS

TLS (HTTP)

Others* Certificate* CADNS Infra.

2/33

The situation
I Answer: zmap

+ dig + masscan + ...

+ ad gustum

you can do it, but it ain’t pretty...

3/33

The situation

The problem:
1. wasted time spent on repetitive tasks
Iexecution, coordinating different tools, error handling, retries.

2. heterogeneous data formats per toolI

good luck in parsing them

3. more complexity → more errors
IAs we know, one little mistake may void entire experiments [1]4. hard to reproduce studies

4/33

Can we do better than this?

I Complexity begs for automation
I We faced the same problem at .nl

a DNS zone with 5.8m domain names
imagine periodically crawling it ...
I for multiple applications
II

We decided to build a new tool: Dmap
I Dmap: Domain name ecosystem mapper
I current tools do not fulfill our requirements
I neither current data repositories

We open it for researchers:
I https://dmap.sidnlabs.nl

5/33

Our requirements

1. Domain-centric (scan list of domains, no IP addresses)
2. Automate the execution of measurements
3. Crawl multiple protocols (DNS, HTTP, TLS, SMTP, etc)
4. Support both IPv4 and IPv6
5. Completeness: fail-safe, retry mechanism
6. Derived features (not only raw datasets)
7. SQL-based interface
8. Modular
9. Distributed
10. Open source

6/33

Dmap Architecture

I Dmapis not a stand-alone tool, like nmap, zmap...

I It is a RESTFul web service developed in Java, based onSpring Boot [2]I Can be run as a single .jar file
I Can be distributed over multiple servers for scalability
I Uses PostgreSQL to store results and metadata (not rawmeasurements)

7/33

Dmap Architecture
DomainsDBRESTful API

read

A. Data Preparationsave

1.Crawlercontroller

3. InputQueue

2.Crawler
serviceread random domain

REG,DNS, HTTP ,TLS,SMTP,SCREENSHOT

4.Crawler
Threads (n)B. Crawlers

put it back if failssave

5. Crawler Pipeline
read raw scan results

8.Persistence
Service

7.Classifiers
Threads (n)6. Raw
ResultQueue

9. Results DB
saveresearchers/analysts

C. Classifiers

Figure: Dmap Architecture

8/33

Dmap Architecture
A. Data preparation:
I Crawler service reads csv file with list of domains into

PostgreSQL
I

can be used with million domains (5.8m on .nl)

I Crawler service moves 15,000 domains at a time

Domains
DBRESTful API
1.Crawlercontroller

save

2.Crawler
serviceread

A. Data Preparation
3. InputQueue

9/33

Dmap Architecture
B. Crawlers modules:
I Crawler reads domains from Input Queue, start crawler

threads
I We wrote each crawler – based on libraries
I Each thread uses this domain in a pipelineIactual measurements of DNS, HTTP, TLS, SMTP ,Screenshot

I Users can activate/deactive crawlers per measurement
I Dmap keep state of each crawler for each domain
IIfail-safe mechanisms
in a PostgreSQL table – so you can check it laterread random domain

4.Crawler
Threads (n)B. Crawlers

put it back if fails

REG,DNS, HTTP ,TLS,
SMTP,SCREENSHOT

5. Crawler Pipelinesave

6. Raw
Result
Queue

10/33

Dmap Architecture
C. Classifiers:
read raw scan results

8.Persistence
Service

7.Classifiers
Threads (n)9. Results DB
saveresearchers/analysts

C. Classifiers

Table: Six out of thirty Dmap classifiers and features
Classifier
TLSCertServerClassParking
CMSClassBusiness
LetsEncryptFeatures
extract X.509 cert. info and type of cert
OS and Web server fingerprint
if a domain is parked and/or for sale
CMS, shopping cart and forum detection
if a domain has a registrar placeholder page
type of site (online shop)
if domain uses Let’s Encrypt Certs
11/33

Data ModelI 166 features
I in SQL format, also JSONI See https://dmap.sidnlabs.nl
I Other tables used for metadata and control
I Demo ?

Figure: Sample HTTP features for wikipedia.org
12/33

Evaluation

I Input: 5,766,118 domains from .nl DNS zone
I Date: September 3rd, 2017
I 8x 1.4GHz VM, 32 GB of RAM, 1Gbps shared line
I 6 crawlers (screenshot off), 300 crawler threads, 50 classifierthreads

13/33

Evaluation

Precision:
I We compared DNS results with OpenIntel1
I

OpenIntel Crawls only DNS records; Dmap does multi protocols

I Different vantage points, different software
I Same input: .nl domains on Sep. 3rd, 2017
I Metric : number of AAAA records (IPv6 addresses)
II1

Dmap: 913,593
OpenIntel: 913,511

http://openintel.nl

14/33

Evaluation
HTTP Crawler: we know when and why it fails, if it fails
1x107

IPv4IPv6Domains (log)

1x106100000
10000
1000
100
101OKMaUNIRTCTCOlfor nkno etwo O Err edire oo M onne LS E onte thers
me wn rk E orct E any ctio rror nt Dd U honecor
r
Rr
r
oo
Redi
RL strr
derec efuse
ErrsdorFigure: HTTP crawler result codes

15/33

Evaluation

Performance
I This is not a stateless wide scan tool
I It is way more expensive: DNS, HTTP, TLS, SMTP, over both

IPv4 and IPv6I In our setup, Dmap crawled 1 million domains a dayII

8x 1.4GHz VM, 32 GB of RAM, 1Gbps shared line
6 crawlers (screenshot off), 300 crawler threads, 50 classifierthreads

I Run it on multiple machines, designed to scale

16/33

Applications

I We present three applications for Dmap
I They are not an exhaustive list
I We are using several others internal projects
I We hope others build new applications using Dmap

17/33

Applications

Profiling Alexa 1 million domains

18/33

Applications

Profiling Alexa 1 million
I Try it ! dataset and SQL code available athttps://dmap.sidnlabs.nl
I Each result here obtained with SQL
I Hypothesis tests within seconds

query to count # IPv4 HTTP domains

select count(1) from crawl_result_http where
crawl_run=67 and ip_version=4 and crawl_status=0;19/33

Applications
1. Profiling Alexa 1 million# Domains (OK)# Unique NSes
# Unique IP# Unique ASes
# CDN Cloudflare# Domains (OK)# HTML 5
Bytes (median)
External links (median)Internal links (median)
Cookies (median)# Domains (OK)# Let’s Encrypt# Domains (OK)# Unique SMTP
# Unique IP# Unique StartTLS

DNS
IPv4
972,155
289,014210,650
18,418
117,538
HTTP
IPv4
968,338
681,757
53,889
767
1
TLS
IPv4
772,455
165,526
SMTP
IPv4
843,126
501,848286,504
302,871IPv6
153,485
26,127
19,754
3,178
115,396

IPv6/IPv4
0.16
0.09
0.09
0.17
0.98

IPv6
153,485
116,066
64,735875
1

IPv6/IPv4
0.16
0.17
1.20
1.14
1.12
1.00

IPv6
129,443
10,466

IPv6/IPv4
0.17
0.06

IPv6
190,736
24,311
10,113
8,016

IPv6/IPv4
0.23
0.05
0.04
0.03

20/33

Applications : Alexa profiling
IPv6 adoption is slightly faster on SMTP

# Domains (OK)# Unique NSes
# Unique IP# Unique ASes
# CDN Cloudflare# Domains (OK)# HTML 5
Bytes (median)
External links (median)Internal links (median)
Cookies (median)# Domains (OK)# Let’s Encrypt# Domains (OK)# Unique SMTP
# Unique IP# Unique StartTLS

DNS
IPv4
972,155
289,014210,650
18,418
117,538
HTTP
IPv4
968,338
681,757
53,889
767
1
TLS
IPv4
772,455
165,526
SMTP
IPv4843,126
501,848286,504
302,871IPv6
153,485
26,127
19,754
3,178
115,396

IPv6/IPv40.16
0.09
0.09
0.17
0.98

IPv6
153,485
116,066
64,735875
1IPv6/IPv4
0.16
0.17
1.20
1.14
1.12
1.00

IPv6
129,443
10,466

IPv6/IPv4
0.17
0.06

IPv6
190,736
24,311
10,113
8,016

IPv6/IPv4
0.23
0.05
0.04
0.03

21/33

Applications : Alexa profiling
77% support HTTPS, 1 in 5 are Let’s Encrypt# Domains (OK)# Unique NSes
# Unique IP# Unique ASes
# CDN Cloudflare# Domains (OK)# HTML 5
Bytes (median)
External links (median)Internal links (median)
Cookies (median)# Domains (OK)# Let’s Encrypt# Domains (OK)# Unique SMTP
# Unique IP# Unique StartTLS

DNS
IPv4
972,155
289,014210,650
18,418
117,538
HTTP
IPv4
968,338
681,757
53,889767
1
TLS
IPv4772,455
165,526
SMTP
IPv4
843,126
501,848286,504
302,871IPv6
153,485
26,127
19,754
3,178
115,396

IPv6/IPv4
0.16
0.09
0.09
0.17
0.98

IPv6
153,485
116,066
64,7358
75
1

IPv6/IPv4
0.16
0.17
1.20
1.14
1.12
1.00

IPv6
129,443
10,466

IPv6/IPv4
0.17
0.06

IPv6
190,736
24,311
10,113
8,016

IPv6/IPv4
0.23
0.05
0.04
0.03

22/33

Applications

Measure Web Encryption Adoption
on .nl23/33

Applications
2. Measure Web Encryption Adoption (on .nl)
I Dmap supports longitudinal studies

Table: TLS/SSL Web deployment on .nl zone over IPv4
20170903
20180201
5,766,118
5,801,191
2,595,281 (45.10%) 2,674,877 (46.10%)
Types of Certificates
Unknown
382 (0.01%)
407 (0.01%)
self-signed
883,844 (34.05%)
749,033 (28.01%)
DV1,406,072 (54.17%)
1,618,575 (60.5%)
OV
269,284 (10.37%)
272,318 (10.18%)
EV
35,699 (1.37%)
34,544 (1.29%)
Let’s Encrypt DV deployed certificates share
Let’s Encrypt (%DV)
366,623 (14.12%)
523,029 (32.31%)
Zone size
TLS/SSL

24/33

Applications: web encryption evolution on .nl
Small growth on Web encryption over 5 month period
Table: TLS/SSL Web deployment on .nl zone over IPv4
20170903
20180201
5,766,118
5,801,191
2,595,281 (45.10%)
2,674,877 (46.10%)
Types of Certificates
Unknown
382 (0.01%)
407 (0.01%)
self-signed
883,844 (34.05%)
749,033 (28.01%)
DV1,406,072 (54.17%)
1,618,575 (60.5%)
OV
269,284 (10.37%)
272,318 (10.18%)
EV
35,699 (1.37%)
34,544 (1.29%)
Let’s Encrypt DV deployed certificates share
Let’s Encrypt (%DV)
366,623 (14.12%)
523,029 (32.31%)
Zone size
TLS/SSL

25/33

Applications: web encryption evolution on .nl
Decline of self-signed, growth of DV certs
Table: TLS/SSL Web deployment on .nl zone over IPv4
20170903
20180201
5,766,118
5,801,191
2,595,281 (45.10%) 2,674,877 (46.10%)
Types of Certificates
Unknown
382 (0.01%)
407 (0.01%)
self-signed
883,844 (34.05%)
749,033 (28.01%)
1,406,072 (54.17%)
1,618,575 (60.5%)
DVOV
269,284 (10.37%)
272,318 (10.18%)
EV
35,699 (1.37%)
34,544 (1.29%)
Let’s Encrypt DV deployed certificates share
Let’s Encrypt (%DV)
366,623 (14.12%)
523,029 (32.31%)
Zone size
TLS/SSL

26/33

Applications: web encryption evolution on .nl
1/3 of .nl DV certs of domains are by Let’s Encrypt
Table: TLS/SSL Web deployment on .nl zone over IPv4
20170903
20180201
5,766,118
5,801,191
2,595,281 (45.10%) 2,674,877 (46.10%)
Types of Certificates
Unknown
382 (0.01%)
407 (0.01%)
self-signed
883,844 (34.05%)
749,033 (28.01%)
DV1,406,072 (54.17%)
1,618,575 (60.5%)
OV
269,284 (10.37%)
272,318 (10.18%)
EV
35,699 (1.37%)
34,544 (1.29%)
Let’s Encrypt DV deployed certificates share
Let’s Encrypt (%DV)
366,623 (14.12%)
523,029 (32.31%)
Zone size
TLS/SSL

27/33

Applications

CMS/Web server usage on .nl

28/33

Applications
3. CMS/Web server profiling and usage
I We wrote classifiers that identify those
I Apache/Wordpress dominates it

Table: CMS and Web Server Usage .nl zone (20170903)
CMS
Wordpress
JoomlaDrupalBlogo
Wix
Others
Total

Total (%)
848,083 (73.84%)
99,865 (8.69%)
40,798 (3.55%)
20,749 (1.80%)
17,861 (1.53%)
121,232 (10.43%)
1,148,408

Web Server
Apachenginx
MS ISS
cloud-nginx
CoyoteOthers
Total

Total (%)
3,397,930 (77.05%)
696,697 (15.79%)
249,367 (5.65%)
60,530 (1.13%)
4,237 (0.01%)905 (0.00%)
4,409,666

29/33

Summary

I Complexity is bad for measurements
I Dmap automates and reduces complexity of multi-application

measurements
I It has now 40 classifers (30 when we wrote the paper), thatadd rich set of features
I Besides, provides an SQL data model (easy to analyze)
I It’s open, scalable, and distributed
I Potential to speed up your research/PhD/project
I We developed and use it in .nl
I

Why not share it with researchers?

30/33

Future work – you can help us
Nicolaes Visscher, 1658, cartographer from Amsterdam

Antarctica was only discovered in 1820! Let’s map other protocols
https://www.newyorker.com/magazine/2018/02/12/the-white-darkness
31/33

Future work – you can help usNicolaes Visscher, 1658, cartographer from Amsterdam

Antarctica was only discovered in 1820! Let’s map other protocols
https://www.newyorker.com/magazine/2018/02/12/the-white-darkness
31/33

Questions?

Download it!
I https://dmap.sidnlabs.nl
I contact: giovane.moura@sidn.nl
I

We are hiring!
2 Future Internet Research Positions, 1 Machine Learning
Engineer
I Talk to me if you’re interestedIIhttps://sidnlabs.nl

32/33

Bibliography I[1] V. Paxson, “Strategies for Sound Internet Measurement,” inProceedings of the 4th ACM SIGCOMM Conference on
Internet Measurement, ser. IMC ’04. New York, NY, USA:ACM, 2004, pp. 263–271. [Online]. Available:
http://doi.acm.org/10.1145/1028788.1028824[2] Spring, “Spring Boot,” Feb. 2018,https://projects.spring.io/spring-boot/.

33/33