FOR A BETTER
INTERNET
2020 ANNUAL REPORT

2020 A N N U A L R E P O R T | 1

CONTENTS
A Note from our Executive Director 										3The Internet Calamity That Wasn’t											8Eliminating Barriers														11Establishing a New Root													14Serving at Scale															19A Global Community														23Organizational Updates													282020 A N N U A L R E P O R T | 2

OUR WORK &THE ROAD AHEAD
A NOTE FROM OUR EXECUTIVE DIRECTOR
ISRG’s first project, Let’s Encrypt, has been wildly successful. We’re now helping
to secure more than 225 million websites and the Web is making great progress
towards 100% HTTPS. We’ve put in a lot of hard work and dealt with some challenges
along the way, but at a high level the outlook is quite sunny. I’m incredibly proud toshare in this report some of what our organization has accomplished in 2020...

2020 A N N U A L R E P O R T | 3

While I’m deeply appreciative of being in this position today, I don’t let it distract

information they need without any entity having the ability to access any individual

me, or our fantastic Board of Directors, from thinking diligently about the risks

user’s data. Much like Let’s Encrypt, it protects people without them having to know

on the road ahead. A big part of our job is to look into the future, see threats

anything about it.

and challenges, and prepare to face them as best we can. I’m sometimes asked
what I view as the biggest threat to our organization and our ability to pursue

Despite 2020 being a year of unprecedented, global challenges, ISRG is well positioned for theour mission and my answer is simple: being taken for granted.

years ahead. Our current momentum is possible through new major in-kind donations, nearly
90% of our existing sponsors renewing their support for 2020, funding from the Ford

When digital security and privacy is your goal, ease ofuse has to be your focus. When we examine why realworld systems aren’t secure, it usually isn’t because
we don’t have the technological means to securethem. The problem is almost always that the solutions
are not easy enough to use, either for implementers or
consumers.
HTTPS has been around since the mid-90s but uptake
was abysmally slow because SSL/TLS certificatesweren’t easy to get or manage. Let’s Encrypt madeWHEN YOUR STRATEGYAS A NONPROFIT IS TOGET OUT OF THE WAY,TO OFFER SERVICES
THAT PEOPLE DON’T
NEED TO THINK ABOUT,
YOU’RE RUNNING A
REAL RISK THAT YOU’LLEVENTUALLY BE TAKENFOR GRANTED.

Foundation and the Bill & Melinda Gates Foundation, and by welcoming new majorsponsors, including AWS, Thales, and Avast.
When your strategy as a nonprofit is to get out of the way, to offer services thatpeople don’t need to think about, you’re running a real risk that you’ll eventually
be taken for granted. There is a tension between wanting your work to be invisible
and the need for recognition of its value. If people aren’t aware of how valuable ourservices are then we may not get the support we need to continue providing them.
How are we going to mitigate this risk? The most important thing we can do iscontinue to communicate effectively with people who are in a position to understandgetting and managing certificates easy and as a result HTTPS adoption rates

our work and support it. The most important things you can do as a supporter include being an

shot up. Critically, the answer wasn’t to get people to think more about theiradvocate for your company sponsoring us, making an individual donation, or going over thiscertificates—we needed to make it possible for people to spend much less timeannual report with a few people that you think should know more about us.

thinking about certificates. Ideally we’d be invisible—server software should justget and manage certificates automatically.

On behalf of the hundreds of millions of people benefiting from Let’s Encrypt around the world
and our team of sixteen dedicated to this work, thank you for your support.

Our next project after Let’s Encrypt is going live shortly: ISRG Prio Services. It’sa system for collecting digital metrics that allows organizations to collect the

202 0 A NN U A L R E P OR T | 4

JOSH AASEXECUTIVE DIRECTOR

2020 A N N U A L R E P O R T | 4

2018

2017

2020

2019

200M

WEBSITES SERVED
232,083,900

IMPACT ON THE WEB
150M

As of November 2020, Let’s Encrypt serves 232 million websites with 144
million active certificates. Since 2013, HTTPS page loads have grownfrom 25% to 84% globally; it’s close to 92% in the United States.

100M

ACT I V E C E RT I F I CAT E S
14 5,199,80 0
49%

50M

REGISTERED DOMAINS SERVED
6 7,7 9 3 , 2 4 0

2020
2019
2018
2017
2016
2015
2014
2013

P E R C E N TAG E O F W E B PAG E S LOA D E D BY F I R E F OX U S I N G H T T P S

25%

33%

39%

49%

67%

75%

79%

84%

2020 A N N U A L R E P O R T | 5

“

I N M A N Y WAY S ,ISRG has come in and made it really easy to have access to encryption,
but now that’s the expectation. People expect it to be easy and to be
there, and there will be a decreasing visibility in to how that works
because Let’s Encrypt has done such a good job of obviating the need toknow how that works.
In many ways that success is also the new challenge. I am happy to workon making sure that ISRG and Let’s Encrypt remain viable and that ISRG isready to tackle some of our next challenges.”
DAV I D N A L L E Y
ISRG BOARD OF DIRECTORSPRINCIPAL , OPEN SOURCEST R AT EGY & M A R K E T I N GAMAZON

2020 A N N U A L R E P O R T | 6

WELCOME TO THESE NEW SUPPORTERS
We’re proud to call 94 organizations sponsors and funders to ISRG and Let’s
Encrypt. These are the organizations we’ve welcomed into the fold this year.

2020 A N N U A L R E P O R T | 7

THE INTERNETCALAMITY
THAT WASN’T
A STO RY O F R E M E D I AT I O N A N DRESPONSIVENESS
That’s what WIRED called our remediation of a bug that we became awareof in early March of this year. The security implications of this bug wererelatively limited: Our certificate authority (CA) software, Boulder, checks for
CAA records at the same time it validates a subscriber’s control of a domainname. In some cases, it is supposed to do this check two times, but...

2020 A N N U A L R E P O R T | 8

we learned that it was only checking once. In theory, this could

we’re prepared to respond with speed and transparency.

mean that a subscriber could change their CA preference fromIn less than 48 hours, our users and large integrators were

including Let’s Encrypt to excluding us for issuance, but weable to replace over 1.7M certificates. That is an amazing turnaround time for such a volume and a testament to automation

WE DETERMINED THERE WERE MORE
THAN THREE MILLION NON-COMPLIANT
CERTIFICATES, ABOUT 2.6% OF OURTOTAL ACTIVE CERTIFICATES.

and coordination. Many readers of this report played a hand in
this effort and our gratitude for your work persists. After thoseinitial 48 hours, we tracked the replacement of the remaining
certificates over the following weeks until all certificates were
replaced before being revoked.

wouldn’t know that and issue a certificate anyway because weskipped the second check.

Our focus on automation allowed us, and our subscribers, tomake great progress in a short amount of time. We also learned

As soon as we discovered the bug, we turned off issuance, fixed

a lot about how we can do even better in the future. ISRG

it, and resumed our service. This remediation took the team justleadership spent much of 2020 analyzing our processes andtwo hours and 14 minutes. We determined there were more thaninfrastructure to identify opportunities for a better response thethree million non-compliant certificates, about 2.6% of our total

next time a bug emerges. We look forward to sharing the details

active certificates.

of this work in the coming months.

Staff worked tirelessly to identify and communicate with affectedsubscribers. Issues like this are never pleasant, but bugs area reality in the software world and we work hard to make sure

2020
2020AANNNNUUAALLRREEPPOORRTT||99

“THE INTRICACIES OF INTERNETI N F R A S T R U C T U R E A R E G E N E R A L LY I G N O R E D
U N T I L S O M E T H I N G G O E S T E R R I B LY W R O N G .
THIS TIME, THOUGH, IT’S USEFUL TO REFLECTO N W H AT W E N T R I G H T. F O R O N C E , T H E
S T O R Y I S T H A T N O T H I N G B R O K E .”2020 A N N U A L R E P O R T | 10

ELIMINATINGBARRIERST WO N E W P R I VACY & S E C U R I T Y I N I T I AT I V E SThis year, we’ve taken the initial steps beyond running the world’s largest certificate authority,
Let’s Encrypt, to expand upon our mission of removing barriers to secure communication overthe internet. This work has us focused on two projects: building a privacy-respecting system,
ISRG Prio Services, for the collection of application metrics, and funding the work to movecritical components of the ubiquitous software curl to memory safe code.

2020 A N N U A L R E P O R T | 11

INTRODUCING ISRG PRIO SERVICES FORP R I VACY R E S P E C T I N G M E T R I C S C O L L E C T I O NApplications such as web browsers, mobile applications, andwebsites generate metrics. Normally they would just send
all of the metrics back to the application developer, but with
Prio, applications split the metrics into two anonymized andencrypted shares and upload each share to different processorsthat do not share data with each other. This way only minimal
information about the original metrics is revealed to either
processor. Each processor then aggregates its shares into apartial sum. The partial sums can then be combined into a
final aggregation, permitting useful statistics over the wholebody of metrics while revealing minimal information about
individual users. To learn more about the foundations of Prio, we
recommend reading the Prio research paper by Henry CorriganGibbs and Dan Boneh of Stanford University.
We have been researching Prio technology for some time
because the privacy provided by this service can deliversignificant benefits to the public. Application end-users
have little control over the metrics that are collected about
their application usage and how that information is used bydevelopers. When applications use systems like ISRG’s Prio
Services, end-users won’t have to just trust that they are safe
from an attacker stealing and disclosing their information, or acompany selling their personal data, or a government collecting
their information for mass surveillance. By offering low-cost andeasy-to-use cryptographic privacy protection for user metrics,
ISRG will be taking a significant step to protect the general publicfrom privacy violations. It is our hope that privacy respecting
metrics collection will become an expectation for application
developers. We are excited to offer this service to lead the way.
ISRG will operate Prio data share processors as a service tofacilitate a subscriber’s private metrics collection system. Our
Site Reliability Engineering team maintains our open source data
share processor and operates a 24/7 oncall schedule to ensure it
functions smoothly.
Subscribers to ISRG’s Prio Services are responsible for getting asecond data share processor which implements the same protocol
as ours, as well as sharing, encrypting, and uploading metrics fromtheir applications to the data share processors and assembling thefinal aggregation.
We believe we will be the first organization to operate Prio services
in a production capacity.
We’d like to thank Dan Boneh and Henry Corrigan-Gibbs for theirincredible work developing the Prio idea and system. We’d also
like to thank the people at Mozilla Firefox who have begun toexperiment with using Prio for Firefox and sharing their experience.

2020 A N N U A L R E P O R T | 12

ADDRESSING ONE OF THE BIGGESTT H R E AT S TO I N T E R N E T S EC U R I T Y
Memory safety vulnerabilities represent one of the biggest
memory safe by default we’ll better protect the billions of
threats to Internet security. As such, we at ISRG are interested
people who rely on systems using curl.
in finding ways to make the most heavily
relied-upon software on the Internet
Users who need to continue using the
ISRG IS FUNDING
memory safe. We are working with Daniel DANIEL STENBERG,
unsafe C back-ends for whatever reason will
Stenberg, author of ubiquitous curlbe able to continue doing so by building curl
AUTHOR OF CURL
software and WolfSSL, to make criticalwith the C back-ends enabled.
SOFTWARE, TO MAKE
parts of the curl codebase memory safe.
CRITICAL PARTS OF
We’d like to thank Daniel for his willingness
THE CURL CODEBASEISRG is funding Daniel to work on adding
to be a leader on this issue. It’s not easy tosupport for Hyper as an HTTP back-end
make such significant changes to how wildly
MEMORY SAFE
for curl. Hyper is a fast and safe HTTP
successful software is built, but we’ve come
implementation written in Rust.
up with a great plan and together we’re going to make oneof the most critical pieces of networking software in the
At the same time, ISRG engineers will add support for Rustls
world significantly more secure. We think this project can
as a TLS back-end for curl. Rustls is a safe implementation
serve as a template for how we might secure more criticalof TLS, including certificate verification and the network
software, and we’re excited to learn along the way.
protocol written in Rust. It has been audited and we suggest
reading the conclusions on page 11 of the report if you want
We’d also like to thank everyone involved in creating Hyper,
to get even more excited about Rustls.
Rustls, and the libraries they depend on. In particular we’dlike to thank Sean McArthur for his work on Hyper, Joseph
At first the memory-safe HTTP and TLS backends will be optBirr-Pixton for his work on Rustls, and Brian Smith for his
in. We will work with Daniel and various partners to makework on Ring (which Rustls uses).
sure they are extensively tested, and if all goes well the plan
is for the memory safe back-ends to become the default. By
making the most frequently used networking code in curl

2020 A N N U A L R E P O R T | 13

ESTABLISHINGA NEW ROOT
MAJOR STEPS FORWARD FOR LET’S ENCRYPT
On Thursday, September 3rd, 2020, Let’s
Encrypt issued six new certificates: one root,
four intermediates, and one cross-sign. These
new certificates are part of our larger plan to
improve privacy on the web, by making ECDSA
end-entity certificates widely available, and by

making certificates smaller. Given that we issue
1.5 million certificates every day, what makesthese ones special? Why did we issue them? How
did we issue them? Let’s answer these questions,and in the process take a tour of how Certificate
Authorities think and work.

2020 A N N U A L R E P O R T | 14

THE BACKSTORY
Every publicly-trusted certificate authority (such as Let’s
Encrypt) has at least one root certificate which is incorporated
into various browser and operating system vendors’ (e.g.
Mozilla, Google) trusted root stores. This is what allows users
who receive a certificate from a website to confirm that thecertificate was issued by an organization that their browser
trusts. But root certificates, by virtue of their widespread trust
and long lives, must have their corresponding private key
carefully protected and stored offline, and therefore can’t be
used to sign things all the time. So every certificate authority(CA) also has some number of “intermediates”, certificates
which are able to issue additional certificates but are notroots, which they use for day-to-day issuance.
For the last five years, Let’s Encrypt has had one root: the ISRG
Root X1, which has a 4096-bit RSA key and is valid until 2035.
Over that same time, we’ve had four intermediates: the Let’s
Encrypt Authorities X1, X2, X3, and X4. The first two were
issued when Let’s Encrypt first began operations in 2015, andwere valid for 5 years. The latter two were issued about a year
later, in 2016, and are also valid for 5 years, expiring about
this time next year. All of these intermediates use 2048-bit
RSA keys. In addition, all of these intermediates are crosssigned by IdenTrust’s DST Root CA X3, another root certificate
controlled by a different certificate authority which is trusted
by most root stores.

responses that indicate the intermediate certificates have notbeen revoked. This is important because the only other thing
capable of signing such statements is our root itself, and as
mentioned above, the root needs to stay offline and safely
secured.

T H E N E W C E RT I F I CAT E S
For starters, we’ve issued two new 2048-bit RSA intermediates
which we’re calling R3 and R4. These are both issued by ISRG
Root X1, and have 5-year lifetimes. They are also cross-signed by
IdenTrust. They’re basically direct replacements for our current
X3 and X4, which are expiring in a year. We expect to switch our
primary issuance pipeline to use R3 later this year, which won’t
have any real effect on issuance or renewal.
The other new certificates are more interesting. First up, we
have the new ISRG Root X2, which has an ECDSA P-384 key
instead of RSA, and is valid until 2040. Issued from that, we have
two new intermediates, E1 and E2, which are both also ECDSA
and are valid for 5 years.
Notably, these ECDSA intermediates are not cross-signed by
IdenTrust’s DST Root CA X3. Instead, the ISRG Root X2 itself iscross-signed by our existing ISRG Root X1. An astute observer
might also notice that we have not issued an OCSP Signing
Certificate from ISRG Root X2.

Finally, we also have the ISRG Root OCSP X1 certificate.
This one is a little different – it doesn’t issue any certificates.
Instead, it signs Online Certificate Status Protocol (OCSP)

2020 A N N U A L R E P O R T | 15

W H Y W E I S S U E D A N EC D SA R O OT & I N T E R M E D I AT E S
There are lots of other articles you can read about thebenefits of ECDSA (smaller key sizes for the same level of
security; correspondingly faster encryption, decryption,
signing, and verification operations; and more). But for us,
the big benefit comes from their smaller certificate sizes.
Every connection to a remote domain over https:// requires
a TLS handshake. Every TLS handshake requires that theserver provide its certificate. Validating that certificate
requires a certificate chain (the list of all intermediates up
to but not including a trusted root), which is also usuallyprovided by the server. This means that every connection—
and a page covered in ads and tracking pixels might havedozens or hundreds—ends up transmitting a large amountof certificate data. And every certificate contains both its
own public key and a signature provided by its issuer.
While a 2048-bit RSA public key is about 256 bytes long, an
ECDSA P-384 public key is only about 48 bytes. Similarly,
the RSA signature will be another 256 bytes, while the
ECDSA signature will only be 96 bytes. Factoring in some
additional overhead, that’s a savings of nearly 400 bytes
per certificate. Multiply that by how many certificates arein your chain, and how many connections you get in a day,and the bandwidth savings add up fast.

202 0 A NN U A L R E P OR T | 16

These savings are a public benefit both for our subscribers –who can be sites for which bandwidth can be a meaningful
cost every month – and for end-users, who may have limitedor metered connections. Bringing privacy to the whole Web
doesn’t just mean making certificates available, it means
making them efficient, too.
As an aside: since we’re concerned about certificate sizes,
we’ve also taken a few other measures to save bytes in our
new certificates. We’ve shortened their Subject Common
Names from “Let’s Encrypt Authority X3” to just “R3”, relying
on the previously-redundant Organization Name field to
supply the words “Let’s Encrypt”. We’ve shortened their
Authority Information Access Issuer and CRL Distribution
Point URLs, and we’ve dropped their CPS and OCSP urls
entirely. All of this adds up to another approximately 120
bytes of savings without making any substantive change tothe useful information in the certificate.

2020 A N N U A L R E P O R T | 16

WHY WE CROSS-SIGNED THE ECDSA ROOT
Cross-signing is an important step, bridging the gap between whena new root certificate is issued and when that root is incorporated
into various trust stores. We know that it is going to take 5 years or
so for our new ISRG Root X2 to be widely trusted itself, so in order for
certificates issued by the E1 intermediate to be trusted, there needs
to be a cross-sign somewhere in the chain.

certificates will get faster without users or websites having tochange anything. The tradeoff though is that, as long as X2 isn’t
in trust stores, user agents will have to validate a chain with twointermediates: both E1 and X2 chaining up to the X1 root. This takes
more time during certificate validation.

We had basically two options: we could cross-sign the new ISRG Root
X2 from our existing ISRG Root X1, or we could cross-sign the new E1
and E2 intermediates from ISRG Root X1. Let’s examine the pros andcons of each.

Cross-signing the intermediates directly has the opposite tradeoff.
On the one hand, all of our chains will be the same length, with justone intermediate between the subscriber certificate and the widelytrusted ISRG Root X1. But on the other hand, when the ISRG Root X2
does become widely trusted, we’d have to go through another chain
switch in order for anyone to gain the benefits of an all-ECDSA chain.

Cross-signing the new ISRG Root X2 certificate means that, if auser has ISRG Root X2 in their trust store, then their full certificate
chain will be 100% ECDSA, giving them fast validation, as discussed
above. And over the next few years, as ISRG Root X2 is incorporated
into more and more trust stores, validation of ECDSA end-entity

In the end, we decided that providing the option of all-ECDSA chains
was more important, and so opted to go with the first option, andcross-sign the ISRG Root X2 itself.

2020 A N N U A L R E P O R T | 17

PUTTING IT ALL TOGETHER
Now that we’ve shared our new certificates look the way they do,there’s one last thing we’d like to mention: how we actually went
about issuing them.
The creation of new root and intermediate certificates is a big deal,
since their contents are so regulated and their private keys haveto be so carefully protected. So much so that the act of issuing
new ones is called a “ceremony”. Let’s Encrypt believes strongly in
automation, so we wanted our ceremony to require as little human
involvement as possible.
Over the last few months we’ve built a ceremony tool which, given
appropriate configuration, can produce all of the desired keys,
certificates, and requests for cross-signs. We also built a demo
of our ceremony, showing what our configuration files would
be, and allowing anyone to run it themselves and examine theresulting output. Our SREs put together a replica network, completewith Hardware Security Modules, and practiced the ceremony multiple
times to ensure it would work flawlessly. We shared this demo with ourtechnical advisory board, our community, and various mailing lists, andin the process received valuable feedback that actually influenced someof the decisions we’ve talked about above! Finally, on September 3rd, our
Executive Director met with SREs at a secure datacenter to execute thewhole ceremony, and record it for future audits.
And now the ceremony is complete. We’ve updated our certificates page
to include details about all of our new certificates, and are beginningthe process of requesting that our new root be incorporated into various
trust stores. We’d like to thank IdenTrust for their early and ongoingsupport of our vision to change security on the Web for the better.

2020 A N N U A L R E P O R T | 18

SERVING
AT SCALET H E N E X T G E N DATA BAS E S E RV E R S
POWERING LET’S ENCRYPT
A database is at the heart of how Let’s Encrypt manages certificate
issuance. If this database isn’t performing well enough, it can cause
API errors and timeouts for our subscribers. Database performanceis the single most critical factor in our ability to scale while meeting
service level objectives. Here’s a closer look at how our infrastructureis able to serve the world.

2020 A N N U A L R E P O R T | 19

H A R DWA R E S P EC I F I CAT I O N S
The previous generation of database hardware was powerful but it was
regularly being pushed to its limits. For the next generation, we wanted
to more than double almost every performance metric in the same 2U
form factor. In order to pull that off, we needed AMD EPYC chips and Dell’sPowerEdge R7525 was ideal. Here are the specifications:

PREVIOUS GEN

NEXT GEN

2x Intel Xeon E5-2650
Total 24 cores / 48 threads

2x AMD EPYC 7542
Total 64 cores / 128 threads

1TB 2400MT/s

2TB 3200MT/s

Our certificate authority software, Boulder, uses MySQL-style schemas andqueries to manage subscriber accounts and the entire certificate issuanceprocess. It’s designed to work with a single MySQL, MariaDB, or Percona
database. We currently use MariaDB, with the InnoDB database engine.

24x 3.8TB Samsung PM883
SATA SSD
560/540 MB/s read/write

24x 6.4TB Intel P4610
NVMe SSD
3200/3200 MB/s read/write

We run the CA against a single database in order to minimize complexity.
Minimizing complexity has been good for security, reliability, and reducing
maintenance burden. We do have a number of replicas of the databaseactive at any given time, and we direct some read operations to replica
database servers to reduce load on the primary.

W H AT E X AC T LY A R E W E D O I N G
WITH THESE SERVERS?

One consequence of this design is that our database machines need to
be pretty powerful. Eventually we may need to shard or break the single
database into multiple databases, but hardware advancements haveallowed us to avoid that so far.
2020 A N N U A L R E P O R T | 20

H A R DWA R E S P EC I F I CAT I O N S
By going with AMD EPYC, we were able to get 64 physical CPU cores while keeping clock
speeds high: 2.9GHz base with 3.4GHz boost. More importantly, EPYC provides 128 PCIe
v4.0 lanes, which allows us to put 24 NVMe drives in a single machine. NVMe is incredibly
fast (~5.7x faster than the SATA SSDs in our previous-gen database servers) because
it uses PCIe instead of SATA. However, PCIe lanes are typically very limited: modern
consumer chips typically have only 16 lanes, and Intel’s Xeon chips have 48. By providing
128 PCI lanes per chip (v4.0, no less), AMD EPYC has made it possible to pack large
numbers of NVMe drives into a single machine.

OPENZFS & NVMe
NVMe drives are becoming increasingly popular because of their incredible performance.
Up until recently, though, it was nearly impossible to get many of them in a single
machine because NVMe uses PCIe lanes. Those were very limited: Intel’s Xeon processors
come with just 48 PCIe v3 lanes, and a number of those are used up by the chipset andadd-on cards such as network adapters and GPUs. You can’t fit many NVMe drives in theremaining lanes.
AMD’s latest generation of EPYC processors come with 128 PCIe lanes - more than double
what Intel offers - and they’re PCIe v4. This is enough to pack a 2U server full of NVMe
drives (24 in our case).
Once you have a server full of NVMe drives, you have to decide how to manage them. Our
previous generation of database servers used hardware RAID in a RAID-10 configuration,
but there is no effective hardware RAID for NVMe, so we needed another solution. One
option was software RAID (Linux mdraid), but we got several recommendations for
OpenZFS and decided to give it a shot. We’ve been very happy with it.

2020 A N N U A L R E P O R T | 21

PERFORMANCE IMPACT
We’ll start by looking at our median time to process a request because it bestreflects subscribers’ experience. Before the upgrade, we turned around the
median API request in ~90 ms. The upgrade decimated that metric to ~9 ms!
M E D I A N A P I R EQ U E ST L AT E N CY
250 ms200 ms
150 ms
100 ms
50 ms
0

9/20

ms
9/7

9/8

9/9

9/ 1 0

9/11

9/12

In the week before we upgraded our primary database server, its CPU usage(from /proc/stat) averaged over 90%. The new AMD EPYC CPUs sit at about
25%. You can see in this graph where we promoted the new database server
from replica (read-only) to primary (read/write) on September 15.
DATA BAS E S E RV E R C P U U SAG E

9/13

9/14

9/15

9/16

9/17

9/18

9/19

9/20

The upgrade greatly reduced our overall database latency. The average query responsetime (from INFORMATION_SCHEMA) used to be ~0.45ms. Queries now average threetimes faster, about 0.15ms.
AV E R AG E Q U E RY R E S P O N S E T I M E
6 0.0 %
55.0 %
5 0.0 %

100.0%

4 5.0 %
4 0.0 %

8 0.0 %

3 5.0 %
3 0.0 %

6 0.0 %

2 5.0 %
2 0.0 %

4 0.0 %

15.0%
10.0%

2 0.0 %

0 5.0 %
0.0 %

0.0 %
9/7

9/8

9/9

9/10

9/11

9/12

9/13

9/14

9/15

9/16

9/17

9/18

9/19

9/20

9/7

9/8

9/9

9/10

9/11

9/12

9/13

9/14

9/15

9/16

9/17

9/18

9/20

9/21

2020 A N N U A L R E P O R T | 22

A GLOBAL
COMMUNITY
USERS AND SUPPORTERS AROUND THE GLOBEISRG is proud to serve hundreds of millions of people
around the world. And while we strive for our work
to be easy-to-use, we know that our impact is only
possible through the support we receive in time andresources from a, comparatively, small group of folks.
“Supporter” is a word we throw around often, butthat one word isn’t adequate in explaining just how

many ways people make our work possible. Fromniche technical advice on our community forum, toin-kind donations of equipment, to integrators who
manage TLS certificates for sometimes millions of
their customers, here’s a closer look at some of thepeople and organizations who are critical to ISRG
and Let’s Encrypt.

2020 A N N U A L R E P O R T | 23

A COMMUNITY LEAD BY GENEROUS EXPERTS
FOR PEOPLE AROUND THE WORLD
Since 2015, a relatively small group of leaders
have made community.letsencrypt.org arobust and vibrant home for questions, ideas,
and help for users all around the world.

This year, there have been nearly 40,000 posts
and 19,700,000 page views. These community
members are critical to our mission ofadvancing HTTPS adoption.

Over the last few years, community moderator
Tom Delmas has been outstanding in helping
contributors from around the world translate
letsencrypt.org into thirteen different languages.

Building a community that’s beneficial isa task in itself. Having it be a resource forreliable technical support is a feat! That featis achieved, every day, led by hundreds ofcommunity members who provide technicalsupport, on average, in just two hours.

With Let’s Encrypt being freely available inevery country and municipality it can be, it’s
important that our resources and information
be as widely accessible as possible.

These volunteer contributors have made ourdocumentation and support available to speakersof German, French, Spanish, Hebrew, Indonesian,
Japanese, Korean, Portuguese Russian, Serbian,
Swedish, Vietnamese, and Chinese (PRC).

Our thanks to these community forum leaders for theiroutstanding (and numerous!) contributions through 2020

Our thanks to these volunteer contributors fortranslating letsencrypt.org into thirteen languages.

_azJuergenAuer

rg305

stevenzhu

eumel8shuuji3linherest

9peppemnordhoffRiptdelmaspfontelahenrychoi7

timtorChen

bruncsakorangepizza

rmbolgergeorgeperez

brwolfgang

zaoqi

griffin

Osirissahsanu

LOICEV

goodhousekeeper

DDoSolitary

JimPaspetercooperjr

schoenyaronsfilipajdacic

stevenzhu25

jmorahanPatchesski192man

chez14labanskoller

ZeroSimple

2020 A N N U A L R E P O R T | 24

WITH US SINCE THE BEGINNING
These organizations joined us in 2015, when Let’s Encrypt was just an ambitious
idea, and gave us the launchpad to change the web. We are grateful for theirdauntless support and leadership in building a better internet.

2020 A N N U A L R E P O R T | 25

A T R U ST E D PA RT N E R F O R DAT TO
Datto is the world’s leading provider of cloud-based software and technology
solutions purpose-built for delivery by managed service providers (MSPs).
They have been a sponsor and user of Let’s Encrypt since 2017. Since thenthey’ve gone on to grow dramatically. We sat down with Christopher Hoult,Principal Software Engineer, to chat about Datto’s work with Let’s Encrypt.

smooth, from the quality of the ACME documentation to the wonderful relationshipwith our contacts at Let’s Encrypt. When it came to upgrading from ACMEv1 to v2 in thepast year, we were incredibly well looked-after by Let’s Encrypt, who ensured we wereboth aware of the upcoming changes as well as following up with us after we’d put ourchanges into production. We very much feel like we’re part of a team with Let’s Encrypt.

LE: Datto joined us in 2018 as a sponsor. What’s changed between now andthen? How has using Let’s Encrypt certificates helped you scale?

The reason we chose Let’s Encrypt for our products was because of its programmatic
approach to certificate issuance, as well as straightforward renewal process andglobal trust. Many of our customers deal with hundreds of support issues per day, andanything that slows them down as they either setup new installations or investigate
issues reduces their ability to help other clients. Let’s Encrypt is essential to us for
supplying the kind of smooth customer experience that Datto is known for as part ofour mission to keep the MSP world moving.

CH: The biggest change over the past two years has been that our BCDR
business has experienced exceptional growth, with the number of customer
devices increasing significantly each year. Let’s Encrypt has scaled alongwith us, allowing us to continue supporting our over 17,000 Managed Service
Provider (MSPs) partners to support their customers by providing easy andsecure HTTPS access to on-site devices in the field.
Security is incredibly important to us and those we serve, and so being
able to use a globally-trusted PKI provider gives us a highly cost-effective
mechanism to create the best experience for everyone without compromisingon safety, as well as building our products knowing that we’re incorporating
the firepower of the best experts in the business.
LE: What was the process like for setting up your Let’s Encrypt
implementation? What made you decide to use Let’s Encrypt?
CH: The process for setting up our Let’s Encrypt implementation was verystraightforward - even though we required the creation of a custom clientfor our solution, the entire experience of working with Let’s Encrypt was

LE: Have you found that getting a Let’s Encrypt cert is fast? Does the speed of getting acert help you and your customers?
CH: Operating at the scale we do—17,000 customers and growing, and each customer
having their own end customers making use of our solutions—it’s imperative that
their experience is fast and reliable. Let’s Encrypt’s performance at scale has been
particularly impressive, with certificates being issued within seconds using the DNS
challenge mechanism. The fact that we can cycle our certificates with confidencemeans we can continue to provide the seamless experience Datto is known for without
cutting any corners on security.
We are grateful for Datto’s longtime support, and proud to call them a partner inbuilding a better web.

2020 A N N U A L R E P O R T | 26

2020 SPONSORS & FUNDERS
Ninety-four sponsors and funders from more than twenty
countries around the world supported us in 2020.
From one-employee shops to thousand-employee
companies, we are proud to have support from theseorganizations who prioritize the importance of investingin a more secure and privacy-respecting Web.

EVEN IF AN OPEN SOURCE PROJECT IS USEDB Y 1 0 0 % O F W E B S I T E S O N T H E I N T E R N E T, T H E
H A R D E R P A R T, I R O N I C A L LY, I S M A K I N G E N O U G H
MEANING FOR 1,000 PEOPLE: BECOMING VISIBLE
A N D P R E S E N T E N O U G H T H AT T H E Y C A R E W H AT
H A P P E N S T O YO U A N D YO U R P R O J E C T, V E R S U S
A N Y O T H E R O P P O R T U N I T Y O U T T H E R E .”NADIA EGHBAL

WORKING IN PUBLIC

2020 A N N U A L R E P O R T | 27

ORGANIZATIONALUPDATESNEW FACES, FINANCIALS, AND OUR LEADERSHIP
2020 has been a year of momentum for ISRG. Let’s Encrypt is serving reliably,at scale, supported by a robust global community. Our work as a nonprofit
organization is emboldened by our longtime support from a core group offunders and inspired by new investments in our work. Here is a closer look at
new team members, our revenues in 2020 and our leadership.

2020 A N N U A L R E P O R T | 28

A D D I T I O N S TO I S R G STA F F
My name is Kiel Christofferson, I’ve been with ISRG for almost three years, andthis year I became the manager focused on helping each member of our Site
Reliability Engineering (SRE) team grow to meet their goals. We’ve been fortunate
to hire four new people, two SREs and two developers. I’m excited to share a little
bit about what I’ve learned while working with them.
Reflecting on the results of this year’s hiring efforts, I remain impressed andvery grateful that we find coworkers who are able to bring their considerable
experience, their fantastic skills (both well-established and brand-new), and their
humility, to elevate the teams they join.
Our first two additions of the year (on the same day!) were Aaron Gable joining
the Boulder Development team, and Tim Geoghegan joining the SRE Team.
Aaron Gable previously worked on Chromium developer tools and infrastructure
at Google. Right after he started, Aaron very quickly churned through code tohelp complete a tricky migration. Since then, he’s stepped in to pinch-hit as aTeam Lead and made it look easy (it’s not). My own understanding of particularcode paths has been improved by his careful explanations.
Tim Geoghegan previously worked on hardware security at Square and Apple. I’ve
seen Tim take on the giant hurdles of new projects which may have global impact
and involve new problem spaces. He summons the drive and dedication one
might need to cross the United States on a bicycle (which Tim has also done!).
Next we brought Samantha Frank to the Boulder Dev team. Samantha previously
worked on incident reduction and Change Management automation at
Amazon. Golang may be relatively new to her, but I notice her striding throughimprovements to error assertions within every single Boulder component,

improving testing ergonomics, and collaborating with SRE team members toevaluate component performance within a changing infrastructure. Samantha
is also helping to improve our hiring process.
Most recently we added Amir Omidi to our SRE team. Amir previously worked
with the Azure Data team at Microsoft. Amir has ramped-up quickly and istackling the intricacies of implementing a brand new component for his
project with confidence. His thorough research and clean code are helping tobuild a product that we can capably support long-term.
J.C. Jones will be joining our SRE team at the end of November. In many ways,
we are welcoming an old friend: J.C. was one of the original collaboratorson the Let’s Encrypt project while it was incubated within Mozilla. He hascontinued to be a valuable member of our community over the years and weare thrilled he will be contributing in a formal capacity again.
Our new team members meet and exceed the expectations of an organizationthat works hard to function transparently. They thoughtfully engage with theircoworkers and with our partners. They leverage their experience to improve
the work we do, yet remain open to learning and to the peculiarities of offering
an automated Certificate Authority. I have watched each of them dig into theunfamiliar, the compliance-restricted, the documentation-lacking, the criticalcommunications with external partners, the technical debt, and the loose
ends, all to contribute fantastically to a better internet.

KIEL CHRISTOFFERSONSITE RELIABILITY ENGINEERTEAM MANAGER

2020 A N N U A L R E P O R T | 29

2020 REVENUE BY SOURCE
2020 is on track to being one of ISRG’s most successful fundraisingyears to date. This is made possible by ninety-four sponsors, major
national grantmaking organizations, substantial equipment donations,and support from thousands of individuals around the world.
F U N D I N G S O U R C E 					 P E R C E N T O F TOTA L

Platinum Sponsorships					37%

Gold Sponsorships						3%Silver Sponsorships			 		29%

Corporate Donations

				12%Grants									15%

Individual Donations					4%202 0 A NN U A L R E P OR T | 30

2020 A N N U A L R E P O R T | 30

2020 BOARD OF DIRECTORSA A N C H A L G U P TA
INDEPENDENTCHRISTINE RUNNEGAR
INTERNET SOCIETYJOSH AASINTERNET SECURITY RESEARCH GROUPMAX HUNTERE L E C T R O N I C F R O N T I E R F O U N DAT I O NTECHNICALADVISORY
BOARD
RICH SALZ | AKAMAIJOE HILDEBRAND | MOZILLAJACOB HOFFMAN-ANDREWS | EFF

DAV I D N A L L E Y
AMAZON WEB SERVICES

PASCAL JAILLONOVH CLOUDYUETING LEE | FACEBOOKJ.C. JONES | MOZILLARUSS HOUSLEY | INDEPENDENTRYAN HURST | GOOGLESTEPHEN KENT | INDEPENDENTJENNIFER GRANICKAMERICAN CIVIL LIBERTIES UNION

J. ALEX HALDERMAN
UNIVERSIT Y OF MICHIGAN

RICHARD BARNES
CISCO

KAREN O’DONOGHUE | INTERNET SOCIETY
IVAN RISTIC | INDEPENDENT

VICKY CHIN
MOZILLA

2020 A N N U A L R E P O R T | 31

ONWARDS TO A
BETTER INTERNET
Thanks to our staff, community, users, sponsors, grantmakers,

SUPPORT OUR WORK

and individual donors, ISRG and its projects are building abetter internet for everyone, everywhere.

The mission of Internet Security Research Group (ISRG) is to reduce financial, technological, andeducational barriers to secure communication over the Internet. ISRG is a California publicbenefit corporation, recognized by the IRS as a tax-exempt organization under Section 501(c)(3).
For more on our work, visit https://abetterinternet.org

2020 A N N U A L R E P O R T | 32

OUR THANKS TO THESE INDIVIDUAL DONORS
This year we received more than 12,000 gifts from 50+ countries
around the world. Our thanks to these donors for their support.

@repkam09

Boris Pavlenko

Abdul Kalam

gitpod.io

In memory of Dusty

Karol Augustin

MarianPrachetas Prabhu

Abdessamad Razougui

Boyan Zahariev

Desia Yamese Clements

GORO

In memory of our beloved

Kerim Güney

NAKA Hirotoshi

Pranke GmbH

Aileen M. Baluyut

Brian Duncan

Lewis

hamurabi delgado

kevfrancisco.dev

Nemo

Private HouseRoiEX

AJ Jordan

Brisa

DG6JS

HARUKI SATO

Infinistats Ltd

Kevin Kaland, WizOne

ngoralski

Priyanshu Ojha

ROR Branch of B6

Alex C

Bruno

dohq

Helpful Digital

Ingrid Dominguez

Alex C.

Bruno Miyamotto

Dominic

HemantInternational Documents

Alex Qyoun-ae

Bt0dotninja

Don Hawkins

Henri Hannetel

Alexis Touet

bto

Project “Pasos de Jesús”

Hofstadter, Inc

Algolia

campoint AG

Drakkar

AlgoSecure

Carl Frederic de Celles

alpet

cat, Beans

Solutionsngsw

Project

Roger Goudarzi & Nicola
DownesInvestment LLC

Kia Lo

NICRONICSQBIST Inc.

Sam Klugherz

Kill the groove

Nikhil Karkare

Quentonian Bonaparte

SamePostRu

IONICA

Krishnan Rajendram

Nikola Jocic

Kitty-Kizzle III

Samphan Raruenrom

Hotsoft Informática

Isaiah Banks

Lluis Gras Giné

nimnaij

quickytools

Satish Yadav

Durongrit Tripak

Humu, Inc.

James Agbormbai

lmcs from Perú

Noritaka Horio

RALPH HAHN NET

Sawl Stone

Chris Grindstaff

eClinPro

Ian Norman

Jean H Neas

Lukas Gotter

OH JONG IN

Ramsés CruzSD

alpha-mouse

Chris Wilson

efeyopixel.com

Image-charts.com

Jeff Geerling

Lyuboslav Petrov

Oleksandr K

Ranking Software

Sébastien Coutu

ANN JI HAN

Christoph Noppeney

Egor Petukhovsky

In honor of our little

Jesus Gaibor

Malik W Dixon

Olivier Mondoloni

RedisGreen

SergeyAnton

Christopher

epidata.dk

Jesús M.

Manfred Riedl

Ollie Parsley

Regis LEMAIRE

Sergio IT

Anton Schegg

Ciera

Eugene Blundell

In loving memory of Boo,

Jon O’Brien

Marian Hähnlein

OneIT

Rene Gielen

Anunay Kulshrestha

Cloud9Dynamics

Ezra Wolfe

our sweet Boston terrier.

Jose Gomez

Marius Kjærstad

Oskar Holowaty

Revolution

Arkadiy Tetelman

Dan Zachareas

Fabio Neves

Joseph Catrambone

MASAHARU SAKAKIPaul Reinheimer

Ricardobasst85 on Twitter

Dana Ariya

Fernando Luccarini

Kitten Koo, Fluffy and

JP POZZI

Matt Brown

Paulius Gudonis, neqsoft

Richard Renard

Behr & Tom the Corgis

Daniel Spangenberg

Forrest Hoffman

Fritz - some amazing

Juan Vera

Matthew Bobletz

PEDIPA

ricticBen Sykes

DARWIN BAISA

Foxdebugchildhood kitties.

Julia Brunenberg

Matthieu BarallePeter Götz, Agile Coach &Rnig

BENEDICTO PEREZDavid McKendrick

Foxel

In memory of A.J. theJulien Marque

Mauricio Mercado

Bernardo Garcia

DavidH

Gabriel Bautista

Budgie Car Alarm

Kalvö Koloniby

michael barker

Peter Norell

Robert Abela from WP

bn.dyn-berlin.de

Dean Oakley

Geoffrey R.

(non-profit)

Michael DowningPhil Pennock

White Security

BobbyJo / WarpCoil

Dedicated for Dr APJ

Gerald Luke

MULLER REBEYROL

Philippe

Robin Stafford Group

Wildcat, Tarzan!

In loving memory of

In memory of
Chris Laursen

CanadaKamol Chalermviriya

Trainer

Rober Pankrath

Donors listed alphabetically and recognized exactly as submitted.
2020 A N N U A L R E P O R T | 33