Clusters of Re-Used Keys
stephen.farrell@cs.tcd.ie
Trinity College Dublin
20180710

stephen.farrell@cs.tcd.ie 1/55

The TL;DR slide●

Me: Trinity College Dublin, School of Computer Science and Statistics– Research topics: security, privacy, delay-tolerant networking

This talk: (small) country-scale network scans show that there are a lotof cryptographic host/server keys for mail, web and SSH services
being re-used on a lot of different IP addresses, which is surprising– Around 20180316, if you used TLS/SSH via a standard SSH/TLS
port with a randomly chosen mail server in Ireland, (so that host also
listens on port 25), then the probability that some other IP address in
Ireland shares a host/server key with that mail server was >=53%● Preprint: https://eprint.iacr.org/2018/299
Code: https://github.com/sftcd/surveys/
Graphs: https://down.dsg.cs.tcd.ie/runs/
This: https://down.dsg.cs.tcd.ie/misc/hark.pdf●

stephen.farrell@cs.tcd.ie 2/55

Contents
Background● Methodology● Results●– Skip to slide #25 for annotated graphs– Skip to slide #35 for numbers

Causes/Mitigations● Future Work and Conclusions●

stephen.farrell@cs.tcd.ie 3/55

Background
I’ve been wondering whether it’s possible to help
improve Internet security and privacy based on
local actions and measurements, compared toInternet-scale action and measurement● We had a workshop in Dublin in Sep 2017●

– https://responsible.ie/●

One outcome there was that local (Irish) Internet
measurement should have low-hanging fruit– So I set a student project...

stephen.farrell@cs.tcd.ie 4/55

Student Project
Develop a useful survey tool of Irish mail server deployments
censys.io collate Internet-scale surveys and make the results of
those public via their web site and an API. A quick search there
says there are 12582 email servers in Ireland, of whom 62% dosome form of STARTTLS for mail transport security. The goal here
is to use the API to develop a tool that can be run periodically, thatanalyses that data and produces a list of email addresses to which
one might send useful advice as to how they could improve theirmail transport security. For example, if the data indicate some
deployments are running a server that could easily be fixed, (sayby just updating a certificate), then crafting a mail with
deployment-specific instructions as to how to do that could be
good. Actually contacting the postmasters involved is not a part ofthis project but may be done later based on results found.
https://studentprojects.scss.tcd.ie/#sfarrel6

stephen.farrell@cs.tcd.ie 5/55

I started gathering project data…
Using censys.io python client API● Weekly cron job downloading details of mailservers (port 25 listeners) geo-located to Ireland● Two odd things, and one less surprising thingseen…
●– Unsurprising: mega-scalers cause me problems:-)– Oddity#1: changes in number of listeners...
– Oddity#2: fewer keys than expected!
●

Oddity#2 caused this work

stephen.farrell@cs.tcd.ie 6/55

Aside: Mega-scalers…
I tend to not have any “live” accounts withmega-scalers – not 100% and not due tozealotry but I do always push back on things
wanting me to create new accounts● Censys.io went commercial in Dec 2017 and sowanted their users (whether paid or freeresearchers) to use a mega-scaler’s storage●– Entirely reasonable, but not for me:-)●Result: I decided to do my own scans, which in
the end was beneficial (for me)
stephen.farrell@cs.tcd.ie 7/55

Oddity#1: port 25 listener numbers...

Didn’t investigate that jump (yet), could be due to GPDR preparation maybe?
Numbers haven’t gone back down: 20180316: 24,774 port 25 listeners.
stephen.farrell@cs.tcd.ie 8/55

Oddity#2: Fewer keys seen...
●Counting keys in run IE-20171130…
$ HostPortKeyCount.py -f all-key-fingerprints.json...lots of output…
hosts: 10657
hostsports: 25935
fps: 12889●

From the ~23.6k port 25 listeners, ~10.4k do some kind of
crypto, with a total of ~25.9k host/port combinations doing
crypto, but we only see ~12.9k different keys
– Not a huge discrepancy, but worth a look...

●Ports checked: 22, 25, 110, 143, 443, (587,) 993– 587 wasn’t part of 2017/Censys.io runs, but was added forlater/local runs

stephen.farrell@cs.tcd.ie 9/55

Clusters
If there are fewer keys than services using keys, then clearlysome keys are being re-used, e.g. same key for port 25 and587 on one host is reasonable, less so on two hosts● Examining the data, it became clear there was some structure
visible in the key re-uses…
● Definition: If the same key is seen on two IP addresses that arepart of a scan population, then those IP addresses are part ofthe same cluster●– Regardless of the port/service with which the key is used
– Single-address key re-uses do not contribute to clusters●

We see re-uses on every possible combination of port/services– Except we’ve yet to see any key shared between SSH and TLS

stephen.farrell@cs.tcd.ie 10/55

Causes...
●

There are many possible causes for key re-use on different IP
addresses (more later) but minimally, one could be mirroring theentire content/disk of a web server for redundancy– From an Internet vantage-point that looks the same as any other form of key

re-use– Note that not all re-uses are as benign as this one…
●

We can visualise that with a graph like this:– Nodes represent IP addresses– Node numbers are anonymised IP addresses– Node background colour represents the AS of theIP address– Edges represent key re-uses– Edge-colour represents the pair of ports,●

e.g. black is for 22-22 (SSH)…

stephen.farrell@cs.tcd.ie 11/55

Legend for edge-colours

stephen.farrell@cs.tcd.ie 12/55

Issues with key re-use●

Not all re-uses are “bad” but there are risks…
– Multi-homed hosts aren’t bad:-)●Leaking keys allow masquerade– With RSA key transport, it’s worse

Risk may increase faster than linearly withcluster-size● Lots of related work - see pre-print and Heniniger/
Holz etc● Clusters and scale of scans here may be new-ish●

stephen.farrell@cs.tcd.ie 13/55

“Safe” re-use
Same listener process on same IP (e.g. ports 25/587)● Multi-homed hosts with >1 IPv4 address● Mirroring, as in earlier slide● Some HSM use-cases, maybe for a top-of-rack HSM● Multiple machines or VMs where the one and only admin has
really thought it all through:-)●– Some wildcard cert use-cases can fit this scenario●

Notes:– “Real” keys may not be visible via IPv4 address scanning, but only if e.g.

a DNS name is supplied in a TLS SNI extension or HTTP Host header– In general, we can’t distinguish any of the above from less safe re-uses,from an Internet vantage point

stephen.farrell@cs.tcd.ie 14/55

Dangers of re-use (1)Leakage – most private keys are stored in files,
enabling...
● Masquerade – anyone with a re-used private key canmasquerade as other IPs in the same cluster, enabling...
●– Misdirected mail – if I have the right key, I can defeat even

MTA-STS via usual MX/BGP tricks– Credential exposure – SSH password logins, IMAP/SMTP
passwords could be captured and may reach beyond thecluster– Web origins – h2/ORIGIN frame/secondary cert proposals
could enable spoofing any web server in the cluster from anyother sharing the same key

stephen.farrell@cs.tcd.ie 15/55

Dangers of re-use (2)RSA key transport – if using old TLS ciphersuites, thenoffline decryption of sessions based on recorded ciphertext
is easy● Cross-protocol attacks may be more likely to succeed● (Not sure if significant, but) Formal analyses of protocol
security (e.g. for TLS1.3) didn’t take large-scale key re-use
into account● Data recovery – if I can recover old key values from thedisk on my new VM, then key re-use makes that vuln.
more dangerous● Laziness – re-using keys like this gives an impression ofcarelessness to the Internet●

stephen.farrell@cs.tcd.ie 16/55

Dangers of re-use (3)Risk may increase as cluster-size increases…
● Let’s assume:●– Per-host cost of a leak is ‘c’– Probability of a leak of one key is ‘p’– Cluster size is ‘n’:

Cost of leaks without re-use for ‘n’ hosts: ~c*p*n● Cost of leaks for cluster of ‘n’ hosts: ~c*p*n2● Won’t always apply, but seems correct in general thatrisk increases faster than linearly with key re-use●

stephen.farrell@cs.tcd.ie 17/55

Related Work●

By no means 1st to survey keys, e.g.
– Heninger, et al., “Mining your ps and qs: Detection ofwidespread weak keys in network devices.” 2012
– Holz, et al., “TLS in the wild: An internet-wide analysis of
tls-based protocols for electronic communication,” 2015.
– More in preprint●

This work differs a bit by:– Looking at structure (clusters)– Small-country scale scans– Slightly different background/research questions

stephen.farrell@cs.tcd.ie 18/55

Methods●

2017: Using censys.io; 2018: Ran zmap/zgrab locally– Port 25 listeners for IE, EE, FI, PT, LU, NZ, NA, UY, SI and

SG– Then scan those for ports: 22, 25, 110, 143, 443, 587, 993– Usual zgrab SSH/TLS metadata stored (loadsa json;-)●Analysis code:– Find clusters and make pretty pictures– Compare runs over time and cross-border●

Ethics: slowly scanning port 25 listeners isn’t intrusive– Leave usual breadcrumbs in case someone objects

stephen.farrell@cs.tcd.ie 19/55

Methods (1)Different methods in 2017 and 2018
● 2017: Using censys.io, selected port 25 listeners in two
similarly sized countries: IE (Ireland) and EE (Estonia),
which gives zgrab output● 2018: Locally use zmap to find port 25 listeners basedon MaxMind prefixes for IE, EE, FI, PT, LU, NZ, NA,
UY, SI and SG● For each country: use zgrab to gather banner, SSH and
TLS information from hosts for these ports:●– 22, 25, 110, 143, 443, 587, 993– Port 587 wasn’t included in censys.io 2017 scans

stephen.farrell@cs.tcd.ie 20/55

Methods (2)●Use SHA256 fingerprint of SSH/TLS server keys seenin zgrab outputs to find clusters of re-used keys
– Pretty obvious but we do see clusters “merging” during

analysis runs, so >1 iteration over the data is neededStore names from reverse-DNS, certificates andbanners, and matching A record, if present● Store usual main TLS/SSH crypto parameters●– Key type/size, cipher-suite, browser-trusted?, self-signed?,

expired?

●Generate per-cluster JSON files, graphviz dot files
and images (of graph and word-cloud of names)

stephen.farrell@cs.tcd.ie 21/55

Methods (3)●Validation: my code or zmap or zgrab could havebugs
– s/could/invariably does/ for my code at least:-)●Developed tool to validate a cluster using different
code base from scanning/analysis code:– Based on ‘ssh-keyscan’ and ‘openssl s_client’ instead ofzgrab– Likely still some shared lower level library code but
seems ok●That, and manual inspection, shows clusters arerealstephen.farrell@cs.tcd.ie 22/55

Methods (4)Port 25 listeners are mail servers, so we’re less likely to see sensitive data
● Slowly scanning for public services isn’t intrusive, so we go slow
● We published PTR, TXT RR etc. findable from scanning host source IP● We don’t publish identifying information as that could help with lateral
movement●– That could change in future, but only on a case-by-case basis, e.g. for someproduct with hard-coded keys if vendor unresponsive– Happy to take advice on that aspect as I’m a first-timer for that●

Happy to send full information to asset-holders where we have a relevant
contact– We anonymise identifying information for IP address that are linked to them, but are

not theirs, except for the AS number of the linking IP– Have tooling to extract data from scans for a specific IPv4 prefix or ASN, that does
the above anonymisation step for “outside” addresses

stephen.farrell@cs.tcd.ie 23/55

Methods (5)Compare clusters in same population (country code) over time
● Compare clusters in different populations to see what cross-borderclusters we find
● Developed code to be usable from very modest scanning machine(for zmap/zgrab stages at least)●– 0.75 GB RAM, 7.5GB disk, “25%” of an AMD Opteron 62xx class CPU,

10Mbps bandwidth– Analysis stages benefit from more RAM/disk, usually a laptop is fine
– Doesn’t need to be fast – days per run are just fine

Only IPv4 scanning, no IPv6 for now● Note that our trigger is “listens on port 25” not “does TLS on port 25”● All the code is at:●https://github.com/sftcd/surveys

stephen.farrell@cs.tcd.ie 24/55

Results...
IE-20180318 Cluster 76
3 hosts sharing ecdsa-sha2-nistp256 SSH host keys in 2 different ASes
SMTP banners not obviously related

stephen.farrell@cs.tcd.ie 25/55

Results...
IE-20180318 Cluster 8
4 hosts sharing HTTPS server keys
The certificate is browser-trusted, issued in 2017 (according to crt.sh)
The hosts each have a different SSH host key
Most clusters are small like this and fairly boring

stephen.farrell@cs.tcd.ie 26/55

Results...

IE-20180318 Cluster 462
14 hosts sharing a 1024-bit RSA key over 6 ASes
Cipher-suite uses RSA key transportThis is due to a vendor product with a “demonstration” TLS key pairSame key visible in all country scans (see cross-border later)
One host I checked here uses that “live” for it’s MX and is a destination for sensitive emails
I let the relevant CERT-like entity know about thatOn 20180607 the vendor concerned stated that they would contact affected customers andfix the issue in the product. (Yay!)

stephen.farrell@cs.tcd.ie 27/55

Results...
IE-20180318 Cluster 333
21 hosts sharing mail and web keys in various ways
One shared key has been browser-trusted since 2014 (crt.sh says)
Some commonality in banner names
This was cluster 10 in the IE-20171130 scan

stephen.farrell@cs.tcd.ie 28/55

Results...
IE-20180318 Cluster 103
25 hosts sharing SSH host keys, no TLS services are visible at these IPs
The AS here is a hyper-scaler with a local hosting presence

stephen.farrell@cs.tcd.ie 29/55

Results...
IE-20180318 Cluster 32
92 hosts sharing web and mail keys
About 50 different looking SMTP banners
Possibly a mobile web application developer doing stuff for it’s customers

stephen.farrell@cs.tcd.ie 30/55

Results...
SI-20180514 Cluster 152
46 hosts sharing web and mail keys
Many clusters (incl. this) include names of hosting tools in banners/SANs
Support fora for such tools do seem to indicate key mgmt is considered tedious/hardstephen.farrell@cs.tcd.ie 31/55

Results...
SG-20180430 Cluster 128
58 hosts sharing web and mail keys over 15 different Ases
11 TLS key pairs used for 151 host/port combinations
145 services with 1024-bit RSA, 38 with 2048 (32 SSH)
SMTP banners don’t seem related

stephen.farrell@cs.tcd.ie 32/55

Results...
FI-20180328 Cluster 835
80 hosts sharing SSH, web and mail keys;
One SSH key, seen 46 times here, is also seen 10 times in PT!
For some reason, this is my favourite image, but I have no idea what’s going on here;-)IPs seem to relate to a local ICT consultancy – sent ‘em mail on 20180708

stephen.farrell@cs.tcd.ie 33/55

Results...
Want more graphs? There’s 10,962 of those athttps://down.dsg.cs.tcd.ie/runs● Some graphs are too big to render using graphviz
tools (at least, I’ve failed to find a way to do ‘em;-)●stephen.farrell@cs.tcd.ie 34/55

Numeric results (1)Mostly self-explanatory, except:● Analysis date: when analysis finished, sometimes
after bug fixing:-)● “Out of country”: sometimes MaxMind DB is out ofdate at scan time or analysis time, or maybe BGPchanges happened● HARK % == “Hosts are re-using keys” percentage is:●#hosts-in-clusters/#hosts-doing-some-crypto– Could be a useful metric for repeated runs?
– Would like it to decrease, not necessarily to zero

stephen.farrell@cs.tcd.ie 35/55

Numeric results (2)stephen.farrell@cs.tcd.ie 36/55

Numeric results (3)stephen.farrell@cs.tcd.ie 37/55

Cluster size distribution (1)stephen.farrell@cs.tcd.ie 38/55

Cluster size distribution (2)stephen.farrell@cs.tcd.ie 39/55

Evolution over time (1)●Clusters are determined by IP addresses and keys
– Keys can be moved, deleted, generated– IP addresses appear, disappear or can be re-purposed●

We see pretty much every possible kind of
evolution from IE-2017 to IE-2018– New clusters appear, old ones disappear– New ones can be linked to old via both keys and IP

addresses, or just via keys, or just via IP addresses– More complex evolution is also possible (e.g. twoclusters turns into 3)

stephen.farrell@cs.tcd.ie 40/55

Evolution over time (2)stephen.farrell@cs.tcd.ie 41/55

Evolution over time (3)●The complex cluster changes in Ireland– Orange: 2017, Green: 2018
– Blue arrows indicate key linkage– Black arrows indicate IP address linkage

stephen.farrell@cs.tcd.ie 42/55

101 cross-border clusters (1)●As we see clusters that have addresses in
multiple ASes we may well see cross-borderlinks between clusters– And we do. In every case. In 101 different ways

involving 7,468 IP addresses.

stephen.farrell@cs.tcd.ie 43/55

101 cross-border clusters (2)stephen.farrell@cs.tcd.ie 44/55

101 cross-border clusters (3)Same as previous slide but only
showing 44 cases with >2 countries
stephen.farrell@cs.tcd.ie 45/55

101 cross-border clusters (4)●All (but one) of the cross-border “super” clusters so far are
fully connected, or, are two fully connected graphs linked
by one cluster– Example on next slide●

That might indicate that there are common causes here
– Smells like hard-coded keys?
– Or maybe wild-card certs used in many places– Analysis here is still a work-in-progress●

Note: cross-border analysis code only considers hosts
already in clusters in one country, so some links won’t beseen, e.g. if just one IP address in each country has a
copy of a re-used key

stephen.farrell@cs.tcd.ie 46/55

Cross-border example (1)SCNZ342 is a set of linked cross-border clusters – the LHS of thisincludes the IE462 cluster previously shown and due to a vendor productthat ships with a “demonstration” key pair. The RHS of this seems to becaused by the same product with a different “demonstration” key,
perhaps an earlier or later version. There are 85 hosts in total in theseclusters over 42 ASes.
On 20180607 the vendor concerned stated that they would contact
affected customers and fix the issue in the product. (again: Yay!)
stephen.farrell@cs.tcd.ie 47/55

Cross-border example (2)SCPT151 is a cross-border cluster of SSH host keys involving 21 hosts
on 3 countries over 6 ASes.
It turns out I “own” one of the IE hosts. That’s a VPS I started renting in2013. The offending ECDSA host key file is dated in 2012, a little morethan a year before I starting paying for service.
I opened a ticket with the hoster – they answered, nothing else odd seen.
I expect the ECDSA key was in the image used in 2012.
stephen.farrell@cs.tcd.ie 48/55

Cross-border example (3)
EE376

PT268

UY134
IE329

SG674

SI460

SCUY134 is a cross-border cluster of keys involving 99 hosts in 6
countries over 24 ASes.
This is (partly) caused by an open-source package intended fordevelopers that ships with a 1024-bit RSA key. The package is notintended for real deployment, just for easy development, but of course
users do what they want later;-)
The hard-coded key is visible on port 443 on 27 of the 99 IP addresses inthe super-cluster. It may be (not sure) that other OSS packages use thisone and inherit the infelicity.
I contacted the maintainers via their security notification mail address.
They answered the next day (20180614) saying that they would add keygeneration at install-time to a future version.
stephen.farrell@cs.tcd.ie 49/55

Confirmed causes●

SSH Host Key generation prior to virtualisation– Seems like a mistake that happens over and over, in various different ways

Products that ship with default or “demonstration” keys
● Large scale use of wildcard certs (is a 1991 address cluster for ~250
customers of a marketing campaign tool reasonable?)● Mega-SANs: Saw one cert with >1500 SubjectAltNames (SANs) andhad to write code to stop name analysis after 100 SANs because
they slow down the analysis● Multi-homed hosts with up to ~20 IP addresses (an offering from one
IE hoster)● The above have all been confirmed with asset-holders, the preprint
describes some more not-yet-confirmed possible causes●– Confirming more is a work-in-progress – help appreciated!

stephen.farrell@cs.tcd.ie 50/55

Multi-Homed Hosts●

Multi-homed hosts aren’t in general detectable
via IPv4 address scans, so add “noise” to our
results. That may affect about 29% of clusters.

stephen.farrell@cs.tcd.ie 51/55

TLS Versions●

TLSv1.0 is still high! Don’t have rate-of-change
info yet.

stephen.farrell@cs.tcd.ie 52/55

Mitigations
Rotate keys – for TLS, certbot/LE combination should
break up clusters in a few months if used
● Measure – check your network/customers etc to see if
keys are being re-used in ways you don’t expect● SSH clients could react to multiple copies of public
key in known-hosts● CAs could (in principle) be less tolerant of key re-use● SSH protocol could (in principle) be changed to do
key rotation by default for host keys and client keys●

stephen.farrell@cs.tcd.ie 53/55

Conclusion
Key re-use is much more widespread than (at least I)
envisaged● Rotating keys would result in improvements● It’s likely this is part of us all learning how to manage
crypto at scale● Measurement always does seem to turn up new stuff,
in this case structure, go do some!
●– Don’t assume you won’t be found, if doing something odd;-)●

If you’re a relevant asset-holder, do get in touch – I’dlove to try help you improve your network posture andlearn more about how this happens with your help

stephen.farrell@cs.tcd.ie 54/55

Thanks/Questions
Offline questions are welcome too...
– stephen.farrell@cs.tcd.ie– PGP Key ID: 0x5AB2FAF17B172BEA● Preprint: https://eprint.iacr.org/2018/299● Code: https://github.com/sftcd/surveys/● Graphs: https://down.dsg.cs.tcd.ie/runs/● This: https://down.dsg.cs.tcd.ie/misc/hark.pdf●

stephen.farrell@cs.tcd.ie 55/55