Gott ist tot
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

1

George and I are Friends

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike2

Well, maybe more like
brothers; we fightBut we, not our masters,
are responsible
for our actions
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

3

RPKI Publication,
What are theActual Problems?
IEPG / Montréal
2018.07.15

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike4

Don’t Panic• I am an Engineer, we always think
about the problems• I am also a Researcher, we are onlyinterested in the problems• The RPKI is going fairly well
• But I want to talk about the problems• Some of these data are old
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike5

Routing Relies on It!
• If my routing relies on the RPKI, then I
care a lot about publication reliability• Of course, good relying party software
will expect failures, so this is not a killer• But when we look at current publication,much is not operational quality• This has to be fixed
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

6

What Matters• What matters is what the normal
customer sees when they install RP
software and just run it• Do not tell them to tune it. Whatdo you not understand about ‘normaluser?’2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

7

SW Installed as Shipped

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

8

Same for Dragon
Research RP
Except Eye Candy
Has Less Sugar2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

9And there is NORPKI Trust Anchor
Roll ProtocolOops!
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

10Certification Problems
For a few hours this weekend, everythingcertified below your RPKI working CA
went missing, because the EE certificate
in your working CA's manifest expired,thus the signature on the manifest wasinvalid, thus the working CA had no
verifiable children.
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike11

Three of the five RIRs have now been
through a cycle of having some accident
take their CA offline for a few days
(weekend or on that order), only to discoverthat the manifest EE certificate lifetimes
they were using was not long enough tosurvive the CA outage. This is not aboutstale manifests (thisUpdate/nextUpdate),
it's about manifest EE certificates expiring(EE certificate notAfter).
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike12

For years now I've been trying to get theattention of the RIRs on this issue, but theirimplementers keep telling me that they believethat having a relatively short manifest EE
certificate lifetime is important to protect
them from something, not really clear what
when I press them on this point, but they
don't want to change it. Last time I checked,
the combination of the three outages
mentioned above and my whining has gotten
them to push back to perhaps one week forthe manifest lifetime, which means that they
can now survive having their CA down for aweek.

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike13

Following Graphs arefrom DRL’s
Relying Party
Software Web Page
Many are old
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

14

Not BadAn ISPAn RIR2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

15

Very Bad• They do not monitor and have no real NOC• They do not work weekends• I had to write a friend in the RIR’s Engineering
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike16

An RIR, OK but Ugly

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

17

Something Rotten in AMS

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

18

Cause• This was an NFS problem (NFS is Evil!)• It went on for months• DRL logs had full detail showing “NFS”• But “Nothing Can Be Wrong at the RIR”
• Many weeks later it was fixed, but small
problems remained as they kept usingNFS
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

19

LACNIC TA Expired

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

20

Just Weird871553 -rw-r--r-x 4 rcynic rcynic 1969 Feb 17 13:26:29 2012/usr/home/rpki/rcynic/data/authenticated.2012-0711T00:00:00Z/rpki.afrinic.net/member_repository/F3634D22/92EF889
0119911E0A59EB577833A7E19/79FBE550468F11E19086CABE31FFE8A0.roa
871602 -rw-r--r-x 4 rcynic rcynic 2009 Feb 17 13:26:26 2012/usr/home/rpki/rcynic/data/authenticated.2012-0711T00:00:00Z/rpki.afrinic.net/member_repository/F3634D22/92EF889
0119911E0A59EB577833A7E19/82331D8C6C2011E0890EBAC0A0C76497.roa

And we wrote to them multiple times and received only
snarky responses

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike21

RFC 7115 Sec 3
The RPKI repository design [RFC6481]
anticipated a hierarchic organization ofrepositories, as this seriously improves
the performance of relying parties
gathering data over a non-hierarchicorganization. Publishing parties MUST
implement hierarchic directory
structures.

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike22

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

23

OOPS!
RIPE & APNIC Very Slow
Deliberately Bad Directory Structure
Violates RFC
300

250

Seconds / Connection (Average Per Session)

200

rpki.apnic.net
rpki.ripe.net
localcert.ripe.net
repository.lacnic.net
rpki.afrinic.net
rpki-pilot.arin.net
arin.rpki.net
repo0.rpki.net
rgnet.rpki.net

150

100

500
2011-10

2011-11

2011-12

2012-01

2012-022012-032012-04

2012-05

2012-06

2012-072012-08

Fetch TimeRIPE Fixed Theirs

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike24

APNIC still has not.
Is still not hierarchic
per 71152018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

25

Conservative Software
Saves Us
• Of course, good relying party software
will expect failures, so this is not a killer• DRL relying party software uses old data
if it can not fetch new• As RPKI data are fairly stable, this is OK• But one RIR had an in-addr disaster
which lasted five days!
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

26

But we have had
no major RPKI
disasters recently

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

27

So We Have toMake Some2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

28

Lame Delegation• APNIC is Publishing a Child Repo which isUnreachable• It is CNNIC• Think Great Firewall• APNIC & CNNIC are working on this issue• But been going on for many months• This is the same as DNS Lame Delegation
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike29

DNS Root Change• DNS Root Servers occasionally change IPaddress• Multiple notices go out to the world• 82.378% of relying parties ignore it• The long tail of access to the old address
goes for many years• But the DNS protocol is designed for
this; no one notices, and it all works
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

30

RPKI Trust Anchor Change• An RIR wants to change their RPKI trustanchor; and they have done this a lot• The RIR sends out an email or six• Most relying parties do not see orunderstand it• The protocol was NOT designed for this• Things break; the RIRs blame the user
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

31

Now let’s deploy amajor change to thecore of crypto
validation!
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

32

Flag Day, Eh?
# zgrep -h rrdp /var/log/apache2/access.log* \| awk '{print $1}’ | sort -u | wc -l
319# zgrep -h rpki /var/log/rsync.log* \| awk '{print $6}'| grep '^\[‘ | sort -u | wc -l
227

It’s barely deployed and there are
more then 500 out there
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike33

It is a Non-Trivial Flag
Day and the RIRs Have
Not Written theTransition RFC!
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike34

If the IANA was thesingle point of trust, as
was expected, at least
they seem to know how
to deal with rolls
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

35

The RIRs are Not
Network Operators
They’re PTTs,“There can be noproblem”
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

36

Conclusions• RPKI protocols do not have the resilienceof the DNS. Oops! Our bad.
• RIRs have publication problems repeatedly• Validation Reconsidered solves a ‘problem’
that RIRs are not actually having• And it will make things less predictableand understood• And it’s a flag day which many users are
not actually going to follow
2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike37

We Will Learn to Love
Validation Reconsidered2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

38

Stickers I Had Made
15 Years Ago

2018.07.15 IEPG RPKI Measure

Creative Commons: Attribution & Share Alike

39