Large-scale Certificate Management on Multi-tenant Web Servers
Ryosuke Matsumoto

Kenji Rikitake

Kentaro Kuribayashi

GMO Pepabo, Inc.
Email: matumotory@pepabo.com

GMO Pepabo, Inc. / KRPEO
Email: kenji.rikitake@acm.org

GMO Pepabo, Inc.
Email: antipop@pepabo.com

Abstract—In large-scale certificate management on multitenant web servers, preloading a large number of certificatesfor managing a large number of hosts under the single server
process results in increasing the required memory usage dueto the respective page table entry manipulation, which may
be poor resource efficiency and reduced capacity. To solve
this issue, we propose a method to dynamically load thecertificates bound to the hostnames found during the SSL/TLS
handshake sequences without preloading, provided the Server
Name Indication (SNI) extension is available. We implement
the function of choosing the respective certificates with thengx mruby module which extend Web server functions using
mruby with small memory footprint while maintaining theexecution speed. We also evaluated the proposed method on a
Web hosting service of authors’ place of an employer.

1. Introduction
Free Domain-validated (DV) certificates such as Let’s
Encrypt [3] are beginning to be provided, and supporting
HTTPS becomes relatively low cost. Supporting HTTPS has
become an urgent task by Web hosting companies.
In the Web hosting service based on the multi-tenantarchitecture [7], single server process group need to managea large number of hosts [16] to provide at a low price byreducing the hardware cost and operation cost by accommodating hosts with high integration. The term single server
process group means that a server process was shared bya large number of hosts, not activated a process for eachhost on the highly integrated multi-tenant architecture. The
number of server processes does not depend on the numberof hosts, but rather depends on the Web server implementation, which may invoke hundreds of server processes at thestartup.
To communicate with HTTPS, existing Web server software needs to load the secret key paired with the server
certificate for each host at the server process startup [5].
However, the highly integrated multi-tenant architecture
don’t take advantage of reducing the hardware cost andoperation cost with the existing mechanism. The reason forthis is that if hosts are accommodated in a high degree of
integration, it takes a lot of time to start the server processby reading a large number of server certificates and secretkeys, and the memory usage of the server process increases
depending on the number of hosts.

In this paper, we propose large-scale certificate management mehtod which effectively and efficiently reduces
the memory consumption of the Web server process bydynamically acquiring corresponding server certificate andsecret key data each request in a highly integrated multitenant Web server.
The proposed method does not preload a server certificate and a secret key at the Web server process startup, butrather dynamically loads the server certificate and the secret
key from a database each request based on the requested
hostname or IP address/port during SSL/TLS handshake.
We implement the new feature of ngx mruby [15]that
can handle the loading phase of certificates. ngx mruby[15] is a fast and memory-efficient Web server extension
mechanism scripting with mruby [9] for nginx [10]. Server
certificates and secret keys are stored in Redis [17], which
is a kind of KVS [4], and the certificate and secret key
corresponding to a hostname are acquired by the mruby
code.
The rest of the paper is organized as follows. First, wesummarize the tasks for constantly using HTTPS in a highly
integrated multi-tenant Web server in section 2. We describe
the architecture and implementation of the proposed method
in section 3. In section 4, we quantitatively verify the problems of the existing method and evaluate the effectiveness ofthe proposed method. In section 5, we evaluate the proposed
method in production of hosting service of our employer for
one month. Section 6 concludes the paper.

2. Related works
Web hosting service [12] is a typical application service of highly integrated multi-tenant architecture. The Web
hosting service shares server resources among multiple hosts
and provides an HTTP server function for each hostname.
In the Web hosting service, the function that is identified
by a Fully Qualified Domain Name (FQDN) and serves
the corresponding content is called a host. In this paper,
we call a multi-tenant architecture which can accommodate
tens of thousands of hosts as a highly integrated multi-tenantarchitecture [13].
The highly integrated multi-tenant architecture adopts
the virtual host method [19] which processes multiple hosts
by a single server process group. Popular Web server software such as Apache httpd [18] and nginx can handle

multiple hosts by a single server process group using thevirtual host method like VirtualHost configuration of Apache
httpd.
In the existing server certificate management of web
servers, the Web server loads the certificate associated withthe hostname into a memory at a Web server process startup[5]. The Web server reads out the certificate corresponding
to an IP address/port or a hostname from memory at each
SSL/TLS handshake and starts the HTTPS session. Thismethod can process at high speed during the SSL/TLS
handshake since the certificate is loaded in advance to thememory.
A highly integrated multi-tenant architecture needs tomake the configuration and adopts the process model independent from the number of hosts with the virtual host
method, since the architecture manages a large number ofhosts. In operation in a production environment, a single
server process may accommodate more than several tens of
thousands of hosts.
The existing method needs to load a large number ofcertificates and secret keys in the memory at the server process startup. In the system configuration of a reverse proxy
for the TLS termination, the system needs to first perform
TLS communication on the reverse proxy to the hostnamesof all the hosts accommodated in a large number of hosting
servers. The reverse proxy must manage configurations andcertificates on hundreds of thousands of hostnames. In thatcase, as the number of server certificate increase, the loading
time of configurations and certificates data at the server
process startup greatly increases. Also, the memory usageof the server process greatly increases. These increasing
resources may cause a serious problem.

3. Proposed method

certificate in the hostname, since SNI tells the unencrypted
hostname to the server during SSL/TLS handshake. SNI
is commonly used to accomodate a large number of hosts
virtually with a single server process group and a single IP
address in the highly integrated multi-tenant architecture.
We propose the method that the server certificate andsecret key of the request are dynamically loaded fromdata-store like database, file system or API, based on therequested hostname during the SSL/TLS handshake when
an HTTPS request comes to the Web server, with SNI in
the requirement (1). The dynamic certificate loading meetsthe requirement (3).
In the proposed method, the startup time of the web
server process does not depend on the number of hosts
and memory usage, and does not increase, which meets therequirement (2). The dynamic certificate loading does notneed to load a large number of server certificates beforehandat the startup. Even if the number of certificates increases,
the proposed method does not cause the problem of taking along time to reload the server process when a configuration
change occurs, since the startup speed of the server process
is not slowed down by dynamic certificate loading eachrequest. Also, adding more hosts by changing the configuration does not require the server process reloading, since
the proposed method can dynamically analyze the certificate
location from the hostname.
By aggregating data in databases and caches that cancommunicate via TCP, Our proposed method ensures availability and performance of the service system by increasing
the number of Web servers using a scale-out model as thenumber of HTTPS requests increases. Our system can easily
prepare HTTPS environment by linking the databases that
store the user data like the hostname, certificates, and secretkeys data, under TLS option contacts with customers in a
production environment.

3.1. Large-scale Certificate Management

3.2. Implementation

Our proposed method meets the following three requirements, which are essential in the highly integrated
multi-tenant architecture which requires maximization ofbalance between computer resource, performance efficiency
and efficiency of the system operation cost while solving
the problem described in section 2:

In our proposed method implementation, we used
ngx mruby which can extend nginx scripting with mruby
and process at high speed with less memory usage. Also,
the OpenSSL [11] version 1.0.2 or later have a functionthat calls back an extension function of SSL/TLS handshake
behavior such as custom loading server certificates andsecret keys during SSL/TLS handshake, SSL_CTX_set_
cert_cb(). By making this function executable fromngx mruby, the callback function using mruby during
SSL/TLS handshake on nginx [14] can be written by
ngx mruby, which enables the server administrator to easily
implement the dynamic certificate loading algorithm for thevarious systems.
Figure 1 shows an implementation example of dynamically loading server certificates and secret keys at SSL/TLS
handshake using ngx mruby. The server certificates are
stored in Key-Value Store (KVS) such as Redis for therequested hostname of each request. The certificate data andthe certificate key data methods pass data of a certificate
and a private key themselves, not a file.

1)
2)
3)

To support the Server Name Indication (SNI) extension to accommodate hosts;
To avoid loading all Web server certificates forfaster startup of the server processes; andTo ensure that the memory usage of the Web server
process is independent of the number of hosts, bydynamically loading the associated server certificates during each SSL/TLS handshake.

SNI [1] in the requirement (1) is an extended specification of SSL/TLS. Serving multiple HTTPS servers bya small number of IP address is critical to provide at alow price under the cost constraint of the highly integrated
multi-tenant architecture. SNI allows selective use the server

server {listen443 ssl;server_name_;ssl_protocols
TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers
HIGH:!aNULL:!MD5;
ssl_certificate/path/to/dummy.crt;
ssl_certificate_key /path/to/dummy.key;
mruby_ssl_handshake_handler_code ’
ssl = Nginx::SSL.new
host = ssl.servername
redis = Redis.new "127.0.0.1", 6379
ssl.certificate_data = redis["#{host}.crt"]
ssl.certificate_key_data = redis["#{host}.key"]’;}

Figure 1. KVS-based Configuration Example of Dynamic Server Certificate
Management.

TABLE 1. E XPERIMENTAL E NVIRONMENT.

CPU
MemoryServerSpecification
Intel Xeon E5-2620 v3 2.40GHz 24thread
32GBytes
NEC Express5800/R120f-2E

TABLE 2. R ESULT OF S TARTUP T IME BY E XISTING M ETHOD .
itemreal timesystem CPU usage timeuser CPU usage time
virtual memory size (VSZ)
physical memory size (RSS)

value
42.662 sec
37.280 sec
5.387 sec
3207592 KBytes
3175912 KBytes

method from the experiment. Next, we compare the existing
method (preloading) with the proposed method (dynamic
loading) that saves the server certificate and secret key data
in Redis and acquires data from the file and Redis foreach SSL/TLS handshake. Table 1 shows the experimentenvironment.

4.1. Verification of memory usage and startup timeof existing methods

Figure 2. System Example of Dynamic Server Certificate Management.

In Figure 2, a design example in operation of production
is described. The administrator saves the server certificate
and secret key data in the database. When nginx receives
an HTTPS request, it loads the server certificate and secret
key from the database via ngx mruby scripts and establishes
the SSL/TLS session. The proposed method implementation
temporarily stores cache data using KVS such as Redis
which enables high-speed access to the data to reduce theconnection cost to the database for each request.
Even if the number of servers is increased for availability
or performance, our proposed method implementation caneasily share server certificate data via TCP connection with
database or cache server. If the network latency from the
Web server to the cache server becomes a performance
problem, the in-memory cache can also be used for mitigating performance degradation. The implementation ofngx mruby version 1.16.0 has already been published as
OSS as of February 2016.

4. Evaluation and consideration in the production environment
To confirm the effectiveness of the proposed method, inthe existing problem described in section 2, we clarify the
problem of the time of startup (preloading) of the existing

In the table 1 environment, we verify the problem described in section 2 by using nginx version 1.11.13. We
generated server certificates and secret keys of the key
length of 4096 bits for one hundred thousand hosts by theopenssl command for each host configuration of nginx
and measured the memory usage and startup time of nginx
server processes.
The master process of nginx initially loads all the server
certificate data and copies the worker process for request
processing by the fork() system call after the initial processing of master process is completed. In this experimental
environment, the number of worker process is 24 processes,
which is the number of logical core of the CPU using configuration setting parameter worker_processes auto.
We measured CPU metrics using the Linux time command until all the worker processes have completed theinitialization. We also determined memory usage size ofa certain worker process, arbitrarily picked up from whole
worker processes, using the Linux ps command; we adopted
both fields on virtual memory (VSZ) and physical memory(RSS). We later discuss whole memory usage size comparison of the existing method and the proposed method at
section 4.2.
Table 2 shows the result. Loading of server certificatesdepends on the performance per core since a single masterprocess uses only one CPU at first. In the era when increasing the CPU usage efficiency by the number of cores,
shortening this processing time is difficult. There were no
notable points about the usage time of the user CPU andsystem CPU.
Memory usage size of a worker process acquired fromboth virtual memory (VSZ) and physical memory (RSS)
fields from ps command are about 3 GBytes as well.

TABLE 3. E XPERIMENTAL R ESULT OF P ROPOSED M ETHOD .

Nsc
10
100
500
1000

proposed method
dynamic loading(req/sec)
171456.60
172383.84
172714.81
171872.24

existing method
preloading(req/sec)
171914.98
172758.28
173631.06
173272.53

Physical machines in recent days often have a large sizeof physical memories; the size is usually over tens or
thousands of gigabytes. Therefore, it is no problem if aprocess occupies even 3 GBytes or so memory space.

4.2. Performance
methodevaluation

of

the

proposed

We evaluated the performance of the proposed method.
We used nginx which is linked ngx mruby module as web
server software for evaluation.
We set the nginx configuration of both the existing
method and the proposed method to different ports. The
configuration of the proposed method is a configuration
to load the server certificate and secret key with the requested hostname as the key during SSL/TLS handshake.
The configuration of the existing method is a configurationof preloading server certificates at the startup time of the
Web server process. Both configurations fixed the cipher
suites on the server side to prevent the SSL/TLS session
cache to maximize the impact during SSL/TLS handshake.
In the case of configuration to load multiple certificates,
the calculation of loading certificates amount required forthe existing method is O(1) since the method treats correspondence between hostname and certificate as a hashalgorithm in nginx. Also, the calculation of loading certificates amount required for the proposed method is O(1)since the method also acquires the certificate from the KVS
using the requested hostname as the key. From the above,
we have determined that it is necessary and sufficient for
the evaluation in this experiment to set one certificate to
be read in the configuration of the existing method and the
proposed method.
We used HTTPS benchmark software wrk [20] which
supports multi threading to compare performance. Whilechanging the number of simultaneous connections, we sent
5 million requests as the total and measured the number ofrequests per second.
We adopted TLSv1.2 of nginx configuration that enables
TLS version 1.2. We also adopted ECDHE-RSA-AES128GCM-SHA256 [6] as the cipher suites from which Mozillarecommends. The content requested uses index.html of 612
bytes enclosed with nginx by default.
Table 3 shows the result. Nsc in the table is an abbreviation for Number of simultaneous connections. In theexperimental results, we observed that there is almost no
performance difference between preloading and dynamic
loading method.

TABLE 4. P RODUCTION SERVER SPECIFICATIONS .

CPU
MemoryServerSpecifications
Intel Xeon CPU E5-2430 v2 2.50GHz 12thread
32GBytes
NEC Express5800/E120e-M

We considered that the process of dynamically loadinga certificate is almost negligible because encryption andcompound processing in SSL/TLS handshake is very large.
When the number of simultaneous connections is thousand,both the existing method and the proposed method aresomewhat degraded in performance, but this result is within
the error range since the difference is also less than 1%.
In the preloading method, there is a problem that theamount of memory usage increases as the number of hosts
increases in the highly integrated multi-tenant architecture.
On the other hand, the memory usage at startup is verysmall since the dynamic loading method does not require
the initializing process to store all SSL/TLS configuration
data such as server certificate in memory at startup.

4.3. Evaluation in production
We applied the proposed method to a hosting service
production of our employer and evaluated it on the operationin the production.
The hosting service adopted the existing method(preloading) that loads the certificate at the time of startup
using Apache httpd before applying the proposed method.
As the evaluation method, We measured the transitionof the total number of certificates the number of requests
processed per second, the CPU usage rate, and memory
usage for one month from March 4 to April 4 of 2017 whenthe existing preloading method was adopted. We compared
the transition with the same kind of measured values during
the one month from July 22 to August 22 of the same year
after applying the proposed method (dynamic loading).
We have developed a production environment based on
Figure 2. Also, the server hardware which was adopted in
the preloading method and the dynamic loading method has
the same specifications.
Table 4 shows server specifications.
Figure 3 shows the transition of the number of certificates for one month. Figure 4 indicates the transition of thenumber of requests processed per second. Figure 5 shows thetransition of CPU utilization of the server. Figure 6 shows
the transition of the memory usage of the server.
Figure 3 indicates that the number of certificates increased by about 400 in one month in the preloading methodof the existing method. In that case, as described in the 4.1
section, Figure 6 shows that the usage has increased by about
1 GBytes since the preloading method reads all certificatesat startup. On the other hand, the dynamic loading methodof the proposed method of Figure 3 shows that the numberof certificates has increased by more than 10,000 in one
month. The increase in the number of certificates was due
to a newly-provided free HTTPS certificate service of ourFigure 6. Memory usage in a month.
Figure 3. The number of certificates in a month.

Figure 4. Request per second in a month.

employer. The number of certificates of servers processing
by dynamic loading method is 10 times to 15 times that of
preloading method, and the number of requests in second ismore than six times from Figure 4.
However, as shown by Figure 5 and Figure 6, the transition of CPU usage and memory usage are less than theserver which was processing with the existing preloading
method.
Regarding the problem of increasing the memory usage
depending on the number of certificates in the preloading
method, as shown by Figure 6, the proposed method does
not significantly increase in memory usage. The memory
usage can be greatly reduced by the proposed method. Thisis because it is not necessary to read a certificate of a domain
without access by loading only the certificate corresponding
to the domain HTTP requested from clients.
It is possible to reload, graceful restart command by
nginx, the server process online without failing the request
since the time required for reloading the configuration of the
Web server process is greatly shortened. By shortening thistime, the proposed method can freely release the memory
usage, and the memory usage as a whole can be reduced.
Depending on the implementation of the web server, normally the graceful restart function to reload the configuration
without missing the request completes in seconds. In theexisting method, when the number of certificates to be loadat the time of activation increases, it takes time to reload
for several tens of seconds and even several minutes. Evenif graceful restarting, the service was stopped due to therequest timeout.

4.4. The discussion of the evaluation result

Figure 5. CPU usage in a month.

Figure 3 and Figure 6 indicate that the number ofcertificates increases by about 400 in one month in the
preloading method and the memory increases by about 1
GBytes. The breakdown of the memory increase amount percertificate is the configuration of the host, the data of thecertificate and the secret key, and the memory usage amountused when the Web server processes HTTPS requests. In
other words, when trying to process 20000 server certificatesby the preloading method, it is calculated that 50 GBytesof additional memory is required. The proposed method canprocess 20000 certificates with about 3 GBytes, so that theresource usage can be greatly improved.
When using the server equipped with 32GBytes of memory in table 4, if the number of certificates reaches 200000

or more in the future, the existing method requires memory
over 500 GBytes from the viewpoint of the memory usage
mentioned above. In other words, in the existing method,more than 15 servers with 32 Gbytes memory installed are
required.
On the other hand, in the proposed method, Figure 6
indicates that memory usage hardly depends on the numberof certificates. From the viewpoint of memory usage, thisresult shows that even if the number of certificates reaches
200000, even one server can process it. In the proposed
method, in the situation where future HTTPS communication becomes commonplace, the number of servers can begreatly reduced.
In the existing method, the time for reloading the server
process becomes longer as the number of certificates increases, so that there was a problem that the service stoppage time due to reloading becomes long at loading a new
configuration or registering a new certificate.
However, in the proposed method, the proposed method
can reload the process in a short time and shorten the service
downtime since the certificate is not loaded at server process
startup,
Also, applying a configuration of a new certificate ofother host does not need the reloading server process. The
proposed method can dynamically analyze where the certificate is located from the hostname like file path including
a hostname or database with a hostname as key. When
adding a new certificate, if the server administrator registers
certificate data in the database, the proposed method canapply HTTPS to the existing site without reloading theserver process.
Therefore, the service stoppage time can be shortened asa whole by the proposed method, and it becomes possible
to adopt a system configuration easy to operate.

5. Conclusion
As RFC adoption of HTTP/2 protocol is required on
HTTPS, the existing sites need to support HTTPS. The existing method takes time to start up, since a highly integrated
multi-tenant Web server needs to load a large number ofserver certificates at a server process startup.
The proposed method dynamically loads the server
certificate and secret key corresponding to the requested
hostname using SNI during SSL/TLS handshake and thencommunicates via HTTPS. By using the proposed method,
the server can communicate via HTTPS without loading alarge number of server certificates at startup. Moreover, thecost of dynamically loading a certificate is low comparedwith the cost of CPU usage time from the whole SSL/TLS
handshake, and the experimental results show that performance does not cause a problem in practical use.
As a result of introducing the proposed method to the
production environment of hosting service, resource usage
can be greatly reduced compared with the existing method,
and the proposed method is sufficiently effective on theoperation of the production environment.

Furthermore, even if the performance becomes insufficient due to the processing of HTTPS, the proposed method
can easily scale up servers using the scale-out model by thecentralized management of the server certificate data with areverse proxy put in front of the HTTPS servers.
We conclude that the proposed method is one promising
method of a practical system design for supporting HTTPS
of highly integrated multi-tenant architecture.

References[1]

Eastlake D, Transport Layer Security (TLS) Extensions: Extension
Definitions, RFC 6066, 2011.
[2] Ferdman M, Adileh A, Kocberber O, Volos S, Alisafaee M, Jevdjic
D, Falsafi B, Clearing the clouds: a study of emerging scale-out
workloads on modern hardware, ACM SIGPLAN Notices, Vol. 47,
No. 4, pp. 37-48, March 2012.
[3] Internet Security Research Group (ISRG), Let’s Encrypt - Free
SSL/TLS Certificates, https://letsencrypt.org/.
[4] Han J, Haihong E, Le G, Du J, Survey on NoSQL database. 2011
6th International Conference on Pervasive computing and applications(ICPCA), pp. 363-366, October 2011.
[5] Let’sEncrypt Community Support, Apache Module mod vhost alias& LE, https://community.letsencrypt.org/t/apache-module-modvhost-alias-le/9476.
[6] Mozilla Project, mozilla wiki Security/Server Side TLS, https://wiki.
mozilla.org/Security/Server Side TLS.
[7] Mietzner R, Metzger A, Leymann F, Pohl K, Variability Modelingto Support Customization and Deployment of Multi-tenant-aware
Software as a Service Applications, the 2009 ICSE Workshop on
Principles of Engineering Service Oriented Systems, pp. 18-25, May
2009.
[8] Naylor D, Finamore A, Leontiadis I, Grunenberger Y, Mellia M,
Munafò M, Steenkiste P, The cost of the S in HTTPS, the 10th ACM
International on Conference on emerging Networking Experiments
and Technologies (CoNEXT ’14), pp. 133-140, ACM, December
2014.
[9] NPO mruby forum, http://forum.mruby.org/.
[10] Nginx, Nginx, http://nginx.org/ja/.
[11] OpenSSL Software Foundation, OpenSSL, https://www.openssl.org/.
[12] Prodan R, Ostermann S, A Survey and Taxonomy of Infrastructure
as a Service and Web Hosting Cloud Providers,10th IEEE/ACM
International Conference on Grid Computing, pp. 17-25, October
2009.
[13] Ryosuke M, Studies on Highly Integrated Multi-Tenant Architecture
for Web Servers, https://dx.doi.org/10.14989/doctor.k20590, Kyoto
University, Ph.D. thesis, 2017.
[14] Ryosuke M, ngx mruby: Support ssl handshake handler and dynamic
certificate change, https://github.com/matsumotory/ngx mruby/pull/
145.
[15] Ryosuke Matsumoto, Yasuo Okabe, mod mruby: A Fast andMemory-Efficient Web Server Extension Mechanism Using Scripting
Language, IPSJ Journal, Vol.55, No.11, pp.2451-2460, Nov 2014.
[16] Ryosuke Matsumoto, Masashi Kawahara, Teruo Matsuoka, Improvement of Security and Operation Technology for a Highly Scalable
and Large-scale Shared Web Virtual Hosting System, IPSJ Journal,
Vol.54, No.3, pp.1077-1086, Mar 2013.
[17] Sanfilippo S, Noordhuis P, Redis, https://redis.io/.
[18] The Apache Software Foundation, Apache HTTP Server, http://httpd.
apache.org/.
[19] The Apache Software Foundation, Apache Virtual Host documentation, http://httpd.apache.org/docs/2.2/en/vhosts/.
[20] Will Glozer, wrk - a HTTP benchmarking tool, https://github.com/
wg/wrk.