A Look at Router
Geolocation in Public andCommercial Databases
Manaf Gharaibeh1, Anant Shah1,
Bradley Huffaker2, Han Zhang1, Roya
Ensafi3, Christos Papadopoulos1
1Colorado State University

2CAIDA / UC San Diego

3University of Michigan

IIJ Presentation
Tokyo, Japan
November 21, 2017

IP geolocation•

IP geolocation
maps an IP
address to aphysical real-world
location

ACM Internet Measurement Conference 20172

The edge vs the center•

Most of the money
and commercial
interest is in theedge.
– users– content•

So geolocation
databases focus
most effort onedge
ACM Internet Measurement Conference 20173

The edge vs the center•

Many importantresearch questions
focus on the center.
– Censorship– Geographic stretch– Ownership•

How accurate are
the geolocation
databases for thecenter?
ACM Internet Measurement Conference 2017

4

Motivation•

Router geolocation is used in network research:– BGP route visualization and detection of BGP threats– Detection of routing paths that experience international detours– Studying censorship and monitoring•

Geolocation databases (geo-DBs) accuracy for
infrastructure addresses– Geo-DBs accuracy evaluation is dominated by the results overend-host addresses– Researchers are left unsure about the geo-DBs accuracy overinfrastructure addresses such as routers
ACM Internet Measurement Conference 20175

Goals•Quantify geo-DBs coverage and consistency for router
geolocation•

Quantify expected accuracy for router geolocation– Identify which geo-DBs perform better and where (regional
evaluation)

ACM Internet Measurement Conference 20176

Geo-DBs in this study
Free

Commercial

IP2Location DB11.Lite

Digital Envoy NetAcuity*

MaxMind GeoLite

MaxMind GeoIP2+•*Netacuity: CAIDA has agreement for free access•+GeoIP2: purchased access at full price

ACM Internet Measurement Conference 2017

7

Validation datasets
DatasetArk-topo-router

Source/method

CAIDA Router Topology*

IP addresses
count

1.64M
0.69M (city consitency) +Used to study

Coverage & Consistency

Ground Truth
DNS-based

RTT-proximity

CAIDA DNS Dataset*
Location hints groundtruth rules

RIPE Atlas traceroute built-in
measurement / RTT-based

11,857

4,838

Accuracy* Macroscopic Internet Topology Data Kit (ITDK)
http://www.caida.org/data/internet-topology-data-kit/+ IPs with city-level coordinates in all geo-DBs
ACM Internet Measurement Conference 2017

8

DNS-based (accuracy validation)•

Some operators encode geographic hints into some
DNS names•

Operators provided geographic heuristics for 7 domains*…<airport code>\d*.atlas.cogentco.com
be1273.ccr41.lax04.atlas.cogentco.com
be3257.ccr41.iad02.atlas.cogentco.com
te0-7-0-1.rcr21.b054208-1.lhr01.atlas.cogentco.com

Los Angeles, US
Washington, US
London, UK

Domainbelwue.
decogentco.comdigitalwest.
net

ntt.netpeak10.net

seabone.netpnap.net

IP address
count

23

6,462

29

2,331

170

1,405

1,437

*Huffaker et al., DRoP: DNS-based Router Positioning. ACM SIGCOMM Computer Communication Review 44, 3 (2014), 5–13.

ACM Internet Measurement Conference 20179

RTT-proximity (accuracy validation)•

Leverage RIPE Atlas built-in traceroute measurements
data– From May 25th, 2016
– Find all IP hops within 0.5ms threshold from monitor•

IP are then within 50 km from the probe– Associate each IP with monitor– Filter incorrectly relocated probes

0.5 ms– 4,838 addresses satisfy the RTT threshold

ACM Internet Measurement Conference 201710

Incorrectly geolocated Atlas probes
RTT-nearby probes with very different locations•

Insight: RTT-nearby probes should also be neareach other•495 RTT-proximity addresses have RTT-nearby
groups of 2 or more probes•–

Only 12 addresses (2.4%) have RTT-nearby probeswith inconsistent locations.

–

4 have prominent location inconsistencies.

–

8 have relatively small disagreements (< 128 km)

Overall, 223 probes are part of one or more RTTnearby groups–•

Within 0.5ms from bothprobes

RTT-nearby
probesOnly 5 probes (2.2%) are disqualified (along with 13
interface addresses associated with them in thedataset)Internet Measurement Conference 2017
The finalACM
RTT-proximity
dataset has 4,838 addresses

11Methodology•

40 km city radius– Distance between database coordinates for the same city•

Coverage– IP has an answer at the given level•

Consistency (geo-DB vs itself)– All the router’s IPs has the same country– All the router’s IPs are with in a city radius•

Accuracy (geo-DBs vs ground truth)– IP address has the same country as GT– IP address is with in city radius of the GT (Geoname
coordinates)
ACM Internet Measurement Conference 201712

Ark-topo-router (coverage validation)
Geo-DB

IP2LocationLite

NetAcuity

MaxMindGeoLite

MaxMind-Paid

Country

~100%

~100%

99.3%

99.3%

City

99.9%

99.9%

43%

61.6%

•

Country level– All databases provided country level geolocations for all IP•

City level– IP2Location-Lite and Netacuity provided almost 100% coverage– MaxMind-GeoLite covers 43%, paid improves to 61%* Macroscopic Internet Topology Data Kit (ITDK)
http://www.caida.org/data/internet-topology-data-kit/
ACM Internet Measurement Conference 201713

Ark-topo-router (cross consistency)•

Country-level (1.64M IPs)
40 km city limit– Pairwise > 97% for any two geo-DBs

City-level (0.69M IPs)
– The 2 MaxMind DBs disagree on
11.4% of IPs

CDF•– Different vendors disagree on at least
29% of IPs

0.0 0.2 0.4 0.6 0.8 1.0– 95.8% for all 4 geo-DBs agreements
40KM
MM_G vs MM_P
IP2L vs NetA

MM_G vs MM_P
IP2Loc vs NetA
MM_P vs NetA
IP2L vs MM_P

IP2L vs MM_P
MM_P vs NetA

10−2

10−1

100

101

102

103

104

Distance (KM)

ACM Internet Measurement Conference 201714

Quantifying Geo-DBs accuracy
Using ground truth data (DNS-based + RTT-proximity)

IP2Location-Lite and MaxMind DBs are comparable:
77.5% to 78.6% accuracy–

NetAcuity: 89.4%

1.0–

MaxMind-Paid vs. MaxMind-GeoLite:–•

30.4% for geolite•

41.3% for paid

NetAcuity highest with 73% accuracy

0.6–

NetAcuity
MM_Paid

0.4

IP2Location-Lite: lowest accuracy

IP2Loc(16538)
MM_Paid(6848)
MM_GeoLite(5037)
NetAcuity(16519)

IP2Loc

0.2

–

0.8

City-level (40 km city radius)

40 km, is a knee forboth NetAcuity a& MM

MM_GeoLite

40 km

0.0

•

Country-level

CDF•

10−3 10−2 10−1 100

101

102

103

104

Distance (km)Geolocation
error
ACM Internet Measurement Conference 201715

Geo-DBs regional accuracy
City-level breakdown by RIR
MMGeoLite

102

1.0

40 km
10−4 10−3 10−2 10−1 100 101 102 103 104

ACM Internet Measurement Conference 2017

0.4
0.2

40 km

Distance (km)Distance (km)ARIN (10588)
APNIC (931)
AFRINIC (131)LACNIC (52)
RIPENCC (4817)

LACNICRIPENCCARIN

AFRINIC

APNIC

40 km

0.0

0.2

ARIN

APNIC

MM-Paid

ARIN
AFRINIC

0.4

AFRINIC

0.6

0.8

LACNICRIPENCC

APNIC

10−4 10−3 10−2 10−1 100 101 102 103 104

104

Distance (km)ARIN (4761)
APNIC (468)
AFRINIC (58)LACNIC (38)
RIPENCC (1523)

0.6

0.8

103

1.0

101

RIPENCC

ARIN (3826)
LACNICAPNIC (381)
AFRINIC (34)LACNIC (31)
RIPENCC (765)

0.0

40 km

0.0

1.0
0.8
0.6

CDF

RIPENCC

CDF

0.8
0.6
0.4

CDF

0.2

ARIN

0.4

LACNIC and AFRINIC
under sampled

APNIC

10−3 10−2 10−1 100

0.2•

Much of NetAcuity’s
advantage comes in
ARIN

ARIN (10608)
LACNICAPNIC (928)
AFRINICAFRINIC (130)LACNIC (52)
RIPENCC (4820)

0.0•

ARIN does does
poorly across all geoDBs

CDF

•

1.0

IP2Locationlite

10−1

NetAcuity

100

101

102

103

104

Distance (km)16

Do databases take advantage oflocation hints?

0.0

ARIN_DNS

10−1

0.8
0.6

ARIN_DNS (3969)
ARIN_RTT (792)
RIPENCC_DNS (264)
RIPENCC_RTT (1259)

RIPENCC_RTT

RIPENCC_DNS

0.4
0.2

CDF

0.0

ARIN_RTT

RIPENCC_DNS

0.2

0.4

0.6

ARIN_DNS (9569)
ARIN_RTT (1019)
RIPENCC_DNS (1679)RIPENCC_RTT
RIPENCC_RTT (3138)
NetAcuity

CDF

0.8

1.0

NetAcuity results over DNS-based data
are somewhat better than its resultsover RTT-proximity data

MaxMind-Paid•

1.0

Results vs. DNS-based set and vs. RTT-bases set (city-level)

ARIN_RTT
ARIN_DNS

40 km
10−4 10−3 10−2 10−1 100 101 102 103 104

Distance (km)

40 km
100

101

102

103

104

Distance (km)ACM Internet Measurement Conference 201717

Summary country level•

Good coverage for all databases•

IP2Location-Lite and MaxMind have similar accuracy(77.5% to 78.6%)•

NetAcuity highest accuracy (89.4%)

ACM Internet Measurement Conference 201718

Summary city level•

IP2Location-Lite:–•

High coverage (99.9%), but low accuracy (36%)

MaxMind-GeoLite vs. MaxMind-Paid (what you pay for):– Large coverage increase from 43% to 61%– Moderate accuracy increase from 47% to 52%– Poor ARIN accuracy 35% and 40%•

NetAcuity:–

High coverage (99.9%) and highest accuracy (73%)–

Better ARIN accuracy (69%)

ACM Internet Measurement Conference 201719

Conclusions•

All geo-DBs have room to improve their router
geolocation accuracy at both country- and city-level•

Researchers need to be aware of the geo-DBs
inaccuracies and their impact on their research results

Our ground truth dataset is available via IMPACT:
https://www.impactcybertrust.org/dataset_view?idDataset=792

ACM Internet Measurement Conference 201720

BackupSlides2
1

Number of router Interfaces

Regional (RIR) accuracy
12,00023.0%

10,000

11.4%

19.6%

21.1%
Percent incorrect

8,000
6,000

Incorrect
22.6%

4,000
2,000
0

10.0%

29.1%

29.5%
Correct

19.8%
6.2%
0.0%

7.3%
6.1%

IP2Loc-Lite

MM-GeoLite

0.0%

7.2%
6.1%

0.0%

MM-Paid

6.4%
6.1%

0.0%

NetAquity•

AFRINIC and LACNIC are under sampled•

NetAcuity is the most accurate in all regions•

IP2Location-Lite, MaxMind DBs are comparable at country-level
ACM Internet Measurement Conference 2017

RIPENCC-Inc.
RIPENCC-Cor.
LACNIC-Inc.
LACNIC-Cor.
ARIN-Inc.
ARIN-Cor.
APNIC-Inc.
APNIC-Cor.
AFRINIC-Inc.
AFRINIC-Cor.

22

Incorrectly geolocated Atlas probes
Probes with default country-level coordinates•

Typically near the geographic center of a country•

Indicate lack of specific city-level location•

E.g., The United States: 38 00 N, 97 00 W•

Out of 1,387 probes associated with our 0.5ms thresholddata– 19 probes have default country coordinates– Associated with 109 IP addresses– All are omitted from the dataset

ACM Internet Measurement Conference 201723

How often IP addresses
move?
•

For the 11,857 DNS-based
addresses•

Between May 2016 and
September 2017•

Hostnames changed for 24% of
the addresses– Not all hostnames changes
indicate location changes– Only 30.8% have different
location (7.4% of all DNS-based
addresses
in about
16 2017months)ACM Internet Measurement
Conference

No rDNS
6.9%

Different
hostnam
e
24.0%
Same
hostname
69.1%

Geolocating
addresses
using their
domain-specificground truth
rules
Different
location
30.8%
No
geolocat
ion
1.5%

Same
location
67.7%

24

Recommendations forresearchers•

NetAcuity has the best combination of coverage, accuracy across all
regions–•

IP2Location-Lite overall accuracy is too low–••

We recommend NetAcuity to geolocate routers if geo-DBs is the only
option available

We do not recommend it

MaxMind DBs are doing bad in ARIN, good in other regions, but have very
low city-level coverage–

We do not recommend them if high city-level accuracy and coverage are
required–

We recommend the paid version over the public one (better city-level
coverage and accuracy)

All geo-DBs show less accuracy for ARIN addresses
ACM Internet Measurement Conference 201725

Incorrectly Geolocated Atlas
Probes•Second method is based on the insight:– Multiple probes near the same router should also be near eachother
– 495 RTT-proximity addresses have RTT-nearby groups of 2 ormore probes– Only 12 addresses (2.4%) have RTT-nearby probes withinconsistent locations.
– 4 have prominent location inconsistencies.
– The 8 remaining addresses have relatively small
disagreements (< 128 km)– Overall, we have 223 different probes that are part of one or
more
RTT-nearby
groupsACM Internet
Measurement Conference
2017

26

DNS-based set•

We decode location hints in routers’ hostnames– Use domain-specific rules from 7 ground truth domains*– Rules are obtained from the domains operators•

Performing reverse DNS lookups to the Ark-topo-routeraddresses– 905K addresses have hostnames (55%)– About 13.5K belong to the 7 ground truth domains– 11,857
addresses are geolocated using the groundtruth rules:
belwue. cogentco.c digitalwest.
seabone.ne

DomainIP address
count

de

om

net

23

6,462

29

ntt.netpeak10.net

2,331

170

t
1,405

pnap.net
1,437*Huffaker et al., DRoP: DNS-based Router Positioning. ACM SIGCOMM Computer Communication Review 44, 3 (2014), 5–13.

ACM Internet Measurement Conference 201727

DNS-based data correctness
Agreement with latency measurement data••

Our RTT-proximity ground truth–

109 common addresses–

105 addresses agree within 10 km and 4 addresses agree within 43 km

Using a second RTT-proximity dataset**–

A set of routers within 1ms RTT threshold from Atlas probes (collected on April
2017)

–

384 addresses are common with our DNS-based dataset–

355 addresses (92.45%) agree within 100 km (337 addresses (87.8%)
agree within 40 km)–

19 addresses are likely reassigned to hosts at different locations (asrecent rDNS records show)•–

No conflict with the DNS-based data

Remaining 10 addresses disagreements might be a result of stale hostnames,or few
incorrect
AtlasConferenceprobes2017
locations
ACM Internet
Measurement**Giotsas et al. 2016. The Remote Peering Jedi a Portal in the Remote Peering Ecosystem (RIPE 73).

28

Regional and topologicaldistribution
DNS-based and RTT-proximity sets
UniqueGroundCountrie
coordina
IP count
truth
s
tes

ARIN

DNSbased

11,857

53

238

9,588

560

0

RTTproximit
y

4,838

118

1,347

1,123

372

131

ACM Internet Measurement Conference 2017

RIPENCC

Transit
ASes

0

1,709

99.9%

52

3,160

74.5%

APNIC AFRINIC LACNIC

29

Quantifying Geo-DBs regional
accuracy
8,000

AFRINIC and LACNIC are under 6,000
sampled
4,000

Incorrect
22.6%

NetAcuity is the most accurate in 2,0000
all regions
IP2Loc-Lite
IP2Location-Lite, MaxMind DBs
are comparable at country-level

10.0%

29.1%

29.5%
Correct

19.8%
6.2%
0.0%

7.3%
6.1%
0.0%

7.2%
6.1%
0.0%

6.4%
6.1%
0.0%

MM-GeoLite

MM-Paid

NetAquity

IP2Loc-Lite

MM-GeoLite

MM-Paid

RIPENCC-Inc.
RIPENCC-Cor.
LACNIC-Inc.
LACNIC-Cor.
ARIN-Inc.
ARIN-Cor.
APNIC-Inc.
APNIC-Cor.
AFRINIC-Inc.
AFRINIC-Cor.

NetAcuity

1.0
0.8

However, geo-DBs accuracy
varies greatly from one country toanother (as the bottom graphshows)

0.6
0.4
0.2

101

114

119

119

121

126

129

131

132

199

227

278

442

495

557

561

724

0.0

852

•

Percent incorrect

1204

•

11.4%

19.6%

21.1%

10,000

8304

•

23.0%

Fraction correct•

Number of router Interfaces

12,000Country-level breakdown by RIR

US DE GB IT FR NL JP CA ES SG CH RU PL BG AU CZ SE RO UA HK

ACM Internet Measurement Conference 201730

Quantifying Geo-DBs regional
accuracy
Country-level breakdown by RIR•

NetAcuity is the mostconsistent: at least 74% in all
countries

ACM Internet Measurement Conference 2017

MM-GeoLite

MM-Paid

NetAcuity

1.0
0.8
0.6
0.4
0.2

101

114

119

119

121

126

129

131

132

199

227

278

442

495

557

561

724

852

0.0

1204

Geo-DBs accuracy varies
greatly from one country toanother

IP2Loc-Lite

8304

•

Graph shows top 20
countries in ground truth(number of addresses)

Fraction correct•

US DE GB IT FR NL JP CA ES SG CH RU PL BG AU CZ SE RO UA HK

31

Low city-level accuracy at
ARIN
MaxMind-Paid as a case study•

2,793 ARIN addresses are not in the US––•

1,955 of them (70%) are geolocated to the US•

519 of the 1,955 addresses have city-level geolocation•

504 out the 519 have disagreements > 1,000 km with ground truth

Possible fallback to registry information

3,897 ARIN addresses are located in the US with city-level
information–

2,267 (58.2%) have geolocation error > 40 km•

91% of them have block-level (/24 block or larger) locations•

Compared to 78% of the correctly geolocated addresses at city-level–

Block-level location assignments can be responsible for large
geolocation errors (previous work)

–

WeACM
didn’tinvestigate blocks co-locality in this work
Internet Measurement Conference 201732