Improving High–Bandwidth TLS in the FreeBSD
kernelRandall Stewart

Scott Long

Netflix Inc.
121 Albright Way
Los Gatos, CA 95032
USA
Email: rrs@netflix.com

Netflix Inc.
121 Albright Way
Los Gatos, CA 95032
USA
Email: scottl@netflix.com

Abstract—In our 2015 paper, “Optimizing TLS for High–
Bandwidth Applications in FreeBSD”, we demonstrated the costof TLS encryption on high–bandwidth video serving on Netflix’s
OpenConnect Appliance (OCA [1]) and explored methods for
reducing that cost via in–kernel encryption. The results showedthat such a concept is feasible, but only a small performancegain was achieved. In this paper we describe the improvements
made since then to reduce the cost of encryption. We also
compare the performance of several different bulk encryption
implementations.

I. I NTRODUCTION
The importance of Transport Layer Security [2] (TLS)
continues to grow as businesses and their customers cometo appreciate the value of communication privacy. Netflix
announced in 2015 that it would start on the path to encrypting
the video and audio playback sessions of its streaming service
in an attempt to help users protect their viewing privacy [3].
Enabling this comes with a significant computational costto the OpenConnect serving platform, so work continues inexploring and implementing new ways to optimize TLS bulk
encryption and thus lower capital and operational costs.
An OCA is a FreeBSD-based appliance that serves movies
and television programming to Netflix subscribers. Confidential customer data like payment information, account authentication, and search queries are exchanged via an encrypted TLS
session between the client and the various application serversthat make up the Netflix infrastructure. The audio and video
objects are statically encrypted by Digital Rights Management(DRM) that is pre-encoded into the objects prior to them being
distributed to the OCA network for serving.
The Netflix OpenConnect Appliance is a server-class computer based on an Intel 64bit Xeon CPU and running FreeBSD
10.2 and Nginx 1.9. The platform evolves yearly as commodity
components increase in capability and decrease in price; themost recent generation of the platform holds between 10TB
and 200TB of multimedia objects, and can accommodate
anywhere from 10,000 to 40,000 simultaneous long–lived TCP
sessions with customer client systems. The servers are also
designed to deliver between 10Gbps and 40Gbps of continuous
bandwidth utilization. Communication with the client is overthe HTTP protocol, making the system essentially into a large
static-content web server.

Until 2015, these audio and video objects were not encrypted on a per-session basis. Netflix is now in the processof revising and releasing its client playback software to include support for TLS playback encryption, with the goal of
updating all playback devices that are both capable of being
upgraded and have the computational capacity to support TLS
decryption. By the end of 2016 we expect the majority ofstreaming sessions to be using TLS encryption.
As the number of client sessions doing TLS grows acrossthe OpenConnect network, demand increases on the server–side to accommodate these sessions. Building on the resultsfrom our work in 2015 [4], we looked for ways to reducethe cost of encryption on our deployed fleet of hardware aswell as reduce the number of servers needed to accommodate
future growth. This investigation covered three areas: whatis the ideal cipher for bulk encryption, what is the best
implementation of the chosen cipher, and are there ways toimprove the data path to and from that cipher implementation.
II. B ULK C IPHER S ELECTION
The Cipher Block Chaining (CBC) is commonly used toimplement bulk data encryption for TLS sessions as it iswell studied and relatively easy to implement. However itcomes with a high computational cost as the plaintext data
must be processed twice, once to generate the encrypted
output, and once to generate the SHA hash that verifies theintegrity of the encrypted data. The AES-GCM cipher, basedon Galois/Counter Mode (GCM [5]), provides adequate dataprotection and does not require that the plaintext be processed
twice. It is included in TLS 1.2 and later and is available in allmodern versions of OpenSSL and its derivatives. Decryption
is also computationally cheap, and this combined with ubiquitous availability makes it attractive to both client and server
platforms.
We decided that we would transition our TLS platform toprefer GCM, and fall back to CBC only for primitive clientsthat couldn’t be upgraded to support GCM. We estimate thatonce TLS is rolled out to our entire network that only a small
percentage of playback sessions will use CBC.

III. CIPHER I MPLEMENTATION
Synthetic performance testing of the OpenCrypto Framework AES-CBC cipher showed it to be less performant
than the equivalent implementation from OpenSSL. We alsoneeded to investigate AES-GCM performance, but found that
OpenSSL 1.0.1 as of early 2015 did not have an AESNIoptimized implementation for it. Our search for alternatives
led us to BoringSSL [6], which had a well-performing AESNI
implementation of AES-GCM.
In mid–2015 we were introduced to the Intel Intelligent
Storage Acceleration Library (ISA-L [7]). It provided animplementation of AES-GCM that was hand-tuned for specificIntel model families and their instruction sets. Testing showed
it to be an improvement over the BoringSSL ciphers. Results
are included below.
One drawback to the ISA-L was that it was written in theYASM dialect, which is not directly compatible with the GCC
and LLVM toolchain assemblers in FreeBSD. That required us
to modify the kernel makefile infrastructure in a rudimentary
way to allow YASM to be called as an external assembler on
the needed source files, as so:compile-with "/usr/local/bin/yasm -g dwarf2-f elf64 ${INTELISAINCLUDES} -o ${.TARGET}$S/contrib/intel_isa/aes/${.PREFIX}.asm"

IV. DATA PATH I MPROVEMENTS
Our initial work in 2015 [4] used the AESNI implementation built into FreeBSD and the Open Crypto Framework(OCF) to perform bulk encryption. The results in our previous
paper showed a small improvement in performance, but notnearly the results we had hoped to gain. We knew of several
areas where our results could be improved including:
1) Extra copies were being made during kernel data processing due to the encrypt in-place requirement of ourAESNI implementation.
2) The nginx calls into the TLS code were not passing incorrect flags with the sendfile(2) call. This meantthat hot content was not being properly cached.
3) Many times during processing an mbuf chain waswalked to gain addresses for encryption; this constantwalking of the mbuf linked lists caused added ovecipherrhead and further polluted the CPU cache.
We decided to pass in to our new ciphers an array of pointers
to encrypt from and to, i.e. an iovec. This iovec array would
be filled in during the initial setup of the sendfile call, as eachpage was setup for I/O, thus eliminating the need for traversing
a linked list of mbufs. We also redesigned the mbuf allocation
routines to have the ability, as allocation occurred, to include
this new ”mbuf map”.
Since a large part of our data was being encrypted we also
designed a new special mbuf zone that required less overhead
during allocation. A typical one page mbuf required three
separate allocations (one for the mbuf, one for the refcountand one for the page). We redesigned this to make the page
and the mbuf an indivisible unit where FreeBSD’s UMA

would allocate a page and mbuf together during the UMA’s
initialization routine and the UMA constructor would only be
used to reset pointers within the tied entity. We also embedded
the reference count within the mbuf. This required some small
tricks with copies (we don’t actually free the original mbuf
until all copies are free) but proved quite effective at reducingmbuf overhead.
Switching to the iovec array forced us to abandon theOpenCrypto Framework API and access the cipher routines
directly. We still wanted to be able to fall back to OpenCrypto
for testing purposes, so we created a layer that abstracts thememory allocation and iovec handling for low-level cipher
access while still allowing interoperability with OpenCrypto.
The translation is transparent to the upper layers and isselectable at runtime. This work also gave us the chance tofind and fix codepaths that were making unnecessary copiesof data. We also fixed the incorrect sendfile flag usage.
V. R ESULTS
After adding all the improvements we deployed our newfirmware on three different machines. These machines werefed live traffic while gathering CPU and bandwidth measurements during busy hours. The same software was used in
all measurements the only difference being changes to theconfiguration so that the software would:
1) Disable all sendfile enhancements and use justOpenSSL, reading from the file and writing the encrypted data down the TCP connection.
2) Using the sendfile enhancement with the encryption setto use boringSSL.
3) Using the sendfile enhancement with the encryption set
to use Intel’s ISA library.
Thus each machine provided us with three sets of results. Themachine types were as follows:
1) Rev H storage (HDD) platform, CPU E5-2650Lv2 at
1.7Ghz with 20 cores (Hyperthreaded) the cpu classbeing an Ivy Bridge Xeon.
2) Rev F Flash (SSD) cache platform, CPU E5-2697v2
at 2.7Ghz with 24 cores (Hyperthreaded) the cpu classbeing an Ivy Bridge Xeon.
3) Rev N Flash (SSD) cache platform, CPU E5-2697v3
at 2.6Ghz with 28 cores (Hyperthreaded) the cpu classbeing a Haswell Xeon.
Each sets of results will be labeled Rev H, Rev F or Rev N withthe test undergone. We show approximately one hour of traffic
during a busy period. For visual clarity, the legends have been
removed from the graphs; the green x plots are bandwidth in
Gbps, and the red + plots are CPU system usage percentage.
We see in Fig 1 what happens when only OpenSSL is used.
The CPU limits we have set are the standard 80%, however thestorage cache is disk bound hitting between 60-65% CPU andtopping our performance out at about 12-12.5Gps of serving
traffic. The sendfile feature adds considerable improvement as
we see in the next two figures.
The results of BoringSSL, in kernel using sendfile, are seenin Fig 2. The CPUs tend to be used a bit more (55-70% CPUFig. 1. OCA Rev H Performance using user space OpenSSL

Fig. 4. OCA Rev F Performance using user space OpenSSL

Fig. 2. OCA Rev H Performance using in kernel BoringSSL

Fig. 5. OCA Rev F Performance using in kernel BoringSSLutilization) with overall output performance increasing to 1516Gps. This is what we expect with use of the sendfile call tohelp streamline the I/O.
For final comparison we put in-place the ISA library, again
in kernel with sendfile, results can be seen in Fig 3 andshow another improvement moving us to as much as 18G but
generally holding around 16-16.5G.
In Fig 4 we see OpenSSL this time hitting maximum CPU.
This is because SSDs have a significantly greater I/O capacity
so we no longer hit the disk limits seen in the Rev H. We

see that running with an average of 80% CPU we maintainsomewhere between 22-23Gbps. This gives us our baseline tocompare any improvements.
In Fig 5 we see the results of using the kernel encryption
with sendfile and BoringSSL, here we are able to maintainbetween 25-25.5Gbps while maintaining our goal of 80% CPU
utilization.
The ISA library shown in Fig 6 gives us a slight improvement over the previous results getting us again around 2525.5G but tending to stay towards the higher end of the range.

Fig. 3. OCA Rev H Performance using in kernel ISA

Fig. 6. OCA Rev F Performance using in kernel ISATABLE I
C IPHER C OMPARISON C HARTRevFBaseline
BoringSSL
ISA-LFig. 7. OCA Rev N Performance using user space OpenSSL

cpu%
60-65
55-70
55-70

BW
12-12.5
15-16
15-16.5

RevH
cpu%
BW80
22-23
80
25-25.5
80
25.5

RevN
cpu%
BW
70-75
29-30
53
35.5-3650.5
35.5-36a burst up to 57% at one point. The ISA library results canbe seen in Fig 9 and show similar results to what we see in
the BoringSSL case with the exception that our CPU use istending to stay towards 50.5% The results are tabulated inTable I.
VI. S UMMARY AND F UTURE D IRECTIONS

Fig. 8. OCA Rev N Performance using in kernel BoringSSL

Both the Rev F and Rev H are Ivy Bridge machines (v2CPU’s). We anticipate better performance out of a Haswellmachine (v3 CPU). Our Rev N shows promising results in thenext set of figures.
Interestingly the OpenSSL results seen in Fig 7 do not reachthe full CPU target of 80%. Checking the machine healthstatistics we found that the SSDs had reached there maximum
at around 29-30Gps that we see maintained in the graph. InFig 8 we see us reach the interfaces maximum capacity of
35.5-36Gbps, with the CPU tending to stay around 53% with

With our latest work overall performance has improved asmuch as 30%, a vast improvement from our original results.
Still left unexplored is use of offload cards to assist in ourefforts to encrypt all data. One question we have is if a card
can be used in our hardware design in such a way that it takes
less CPU and PCIe bandwidth than just running the AESNI
instructions themselves? As new generations of Intel CPU’s
become available it is possible that the cost of feeding data to
an auxiliary card and collecting the encrypted results will be
more than the actual AESNI instructions themselves.
We were also forced to set aside the OpenCrypto Framework
in order to achieve certain optimization. Similar optimizations
might be useful for other crypto consumers in the kernel, so
we will explore ways to extend the OCF API as we work tomove the code upstream to FreeBSD.
Encrypting in software still requires that data pages gettouched by the CPU, causing CPU cache pollution. The data isalmost never re-used, so the caching is wasted and needlessly
evicts other system data that could be re-used. We are currently
investigating fine-grained use of cache-control features of theIntel CPUs to limit the amount of data that is put into thelast–layer caches during encryption. We are also working with
the Intel ISA-L team to develop routines that use uncachedload/store assembly opcodes for data movement through thecipher.
VII. ACKNOWLEDGEMENTS
The authors would like to thank the Intel ISA-L team forreaching out to us to assist in achieving our performancegoals, and David Wolfskill and Warner Losh for their supportand reviews. We would also like to thank the entire Netflix
OpenConnect team for providing an outstanding environment
for this work.
R EFERENCES

Fig. 9. OCA Rev N Performance using in kernel ISA[1] Netflix, “Netflix Open Connect”, http://openconnect.itp.netflix.com/
openconnect/index.html, 2016[2] T Dierks, E Rescorla, “The Transport Layer Security (TLS) Protocol
Version 1.2”, RFC 5246, August 2008[3] M Watson, “HTTPS performance experiments for large scale content distribution”, https://lists.w3.org/Archives/Public/www-tag/2015Apr/0027.
html, April 2015

[4] R Stewart, J M Gurney, S Long, “Optimizing TLS for High–
Bandwidth Applications in FreeBSD”, https://people.freebsd.org/∼rrs/asiabsd 2015 tls.pdf, March 2015
[5] D A McGrew, J Viega, “The Galois/Counter Mode of Operation (GCM)” http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/proposedmodes/gcm/gcm-spec.pdf, 2005
[6] Google, “BoringSSL” https://boringssl.googlesource.com/boringssl/,
2016[7] Intel, “Optimizing Storage Solutions Using The Intel Intelligent
Storage Accelerations Library”, https://software.intel.com/en-us/articles/
optimizing-storage-solutions-using-the-intel-intelligent-storage-acceleration-library/,2015