BENCHMARK RESULTS

NGINX Proxy Running with Onload®
Sees a 280% Performance Gain

What is NGINX Proxy?
Open source NGINX is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server.
NGINX Plus is a software load balancer, web server, and content cache built on top of open source NGINX. NGINX
Plus has exclusive enterprise-grade features beyond what’s available in the open source offering, including session
persistence, configuration via API, and active health checks. Open source NGINX was used for this benchmarking.
NGINX is heavily network dependent by design, so its performance can be significantly improved through enhancements
to the underlying networking layer.

Key Observations from Performance Testing• Solarflare’s Onload delivers an average performancegain of 280% for NGINX Proxy when measuring
connections per second (CPS) across the range of
two to 40 workers.
• When processing 1KB GET requests using 40 or fewer
NGINX Proxy instances Onload on 100GbE delivers anaverage performance gain of 178%.

• When processing 10KB Get requests using 40 or fewer
NGINX Proxy instances Onload on 100GbE delivers anaverage performance gain of 108%.
• When looking at the 99th percentile tail latency
Onload averaged out at 30% better than the kernel
at 2.2ms with a peak value of 2.9ms through 4
million requests per second (RPS). It should be
noted that at 1.1 million RPS the kernel latency shot
through the roof at 70ms and kept climbing.

BENCHMARK RESULTS

Why NGINX Proxy Benefits from Kernel Bypass
Since NGINX Proxy is network intensive, every request includes network processing overhead. Whenever an application
like NGINX Proxy touches hardware, other than the CPU or memory, and in this case the network, it must make atleast one, and sometimes several calls to the operating system kernel. Each request is additional overhead that
requires both CPU cycles and processing time. Solarflare’s Onload moves the network processing required by NGINX
Proxy from the kernel into NGINX Proxy’s own application space in memory. This single modification improves NGINX
Proxy performance by 280% on average as can be seen in the graph.
Description of Test Platforms
For this testing, we used two Dell EMC PowerEdge R640 dual socket Intel Xeon systems, each with a pair of 100GbE
Solarflare X2541 cards. These cards were then connected back to back for this testing.
Both systems had two Intel Gold 6148 CPUs clocked at 2.40GHz with 20 cores per processor, 192 GB of memory.

wrk2 and NGINX
web server

NGINX proxyserver

2 x Solarflare X2541

2 x Solarflare X2541

2 x QSFP28 to QSFP28
DAC cables

Tuning Configuration
Below are the changes we made to the standard install
beyond simply leveraging Onload:• Enable clustering (multiple EF_CLUSTER_* options) sothat the adapter does the work of spreading the load
to different workers.
• Enable socket caching (EF_SOCKET_CACHE_MAX) toimprove performance of establishing both acceptedand outgoing active connections.
• Use TCP shared local ports feature (multiple EF_TCP_SHARED_LOCAL_PORTS_* options) for outgoingconnections further improving performance.

• Use Epoll mode 3 (EF_UL_EPOLL=3) which gives bestepoll performance when a large number of sockets
are in the epoll set.
• Enable event polling without interrupts within
epoll_wait calls to reduce latency and avoid context
switches (EF_POLL_USEC and multiple EF_*_SPINoptions)• Use scalable filter mode (EF_SCALABLE_FILTERS andEF_SCALABLE_FILTERS_ENABLE options) to avoid
resource constraints when using a very large numberof connections.

BENCHMARK RESULTS

Observations
NGINX Proxy relies on the operating system’scommunications stack to process network I/O requests,
but in high core count environments, this stack has
become the new bottleneck. Here are some additional
points to consider:• The connections per second shows great
improvement with Onload, peaking at animprovement of 366% over the kernel stack. Withlarge numbers of proxy workers (32 to 40) theOnload performance begins to level out.
• The requests per second also shows great
improvement with Onload, peaking at animprovement of 204% over the kernel stack. With
40 worker processes, results continue to improve,
indicating that further performance is available fromOnload.
• The throughput shows significant improvementwith Onload, peaking at an improvement of 139%
over the kernel stack. With large numbers of proxy
workers (32 to 40) the Onload performance again
begins to level out.

• The latency figures output by wrk2, show the timefrom when the packet should have been sent(according to the configured packet rate), until when
the packet was actually received. The 99th percentile
figure is reported. When the kernel stack packet rateis raised above 1.1 million requests per second, it
can no longer maintain this rate. Jitter increases, thenumber of outliers exceeds 1%, and so the reported
latency suddenly and dramatically increases. Any
further small increase in load would make the server
appear completely unresponsive to an end user. In
contrast, Onload continues to deliver low latency
with 4 million requests per second and is actuallytrending towards even lower latency. The stable andlow value for the 99th percentile of latency indicates
low jitter and predictable performance.
For More Testing Details
Check out Solarflare’s Onload NGINX Proxy Cookbook
for the exact installation and testing process.

The above benefit statements are the result of benchmarking designed to focus on the value of optimizing networking through Onload kernel
bypass. Real world use cases are not the same as benchmarks and as such the role that networking plays may vary, so your overall measurable
benefits may be different.

For more information please visit:
solarflare.com

Contact Us:
US +1 949 581 6830
UK +44 (0) 1223 477171
HK +852 2624 8868
Email: sales@solarflare.com

SF-122347-CD Issue 3
NGINX Proxy Benchmark Results 112219
Copyright © 2019 Xilinx, Inc. Onload and all other designated brands included herein are trademarks of Xilinx in the United States and othercountries. All other trademarks are property of their respective owners.