Project: Apache Mesos (Orchestration & Management / Scheduling & Orchestration)
File: 2018-10-08-mesos-1-7-0-performance-improvements.md

layout: post title: Performance Improvements in Mesos 1.7.0 published: true post_author: display_name: Benjamin Mahler gravatar: fb43656d4d45f940160c3226c53309f5 twitter: bmahler tags: Performance Scalability and performance are key features for Mesos. Some users of Mesos already run production clusters that consist of many tens of thousands of nodes. However, there remains a lot of room for improvement across a variety of areas of the system. The Mesos community has been working hard over the past few months to address several performance issues that have been affecting users. The following are some of the key performance improvements included in Mesos 1.7.0: Master `/state` endpoint:* Adopted and reduced copying for a 2.3x throughput improvement due to a ~55% decrease in latency (). Also, added parallel processing of `/state` requests to reduce master backlogging / interference under high request load (). Allocator:* in 1.7.1 (these patches did not make 1.7.0 and were backported to 1.7.x), allocation cycle time was reduced. Some benchmarks show an 80% reduction. This, together with the reduced master backlogging from `/state` improvements, substantially reduces the end-to-end offer cycling time between Mesos and schedulers. Agent `/containers` endpoint:* Fixed a performance issue that caused high latency / cpu consumption when there are many containers on the agent (). Agent container launching performance improvements*: Some initial benchmarking shows a ~2x throughput improvement for both launch and destroy operations. Before we dive into the details of these performance improvements, I would like to recognize and thank the following contributors: Alex Rukletsov and Benno Evers*: for working on the parallel serving of master `/state` and providing benchmarking data. Meng Zhu*: for authoring patches to help improve the allocator performance. Stphane Cottin and Stephan Erb*: for reporting the `/containers` endpoint performance issue and providing performance data. Jie Yu*: for working on the container launching benchmarks and performance improvements. The `/state` endpoint of the master returns the full cluster state and is frequently polled by tooling (e.g. DNS / service discovery systems, backup systems, etc). We focused on improving performance of this endpoint as it is rather expensive and is common performance pain point for users. In Mesos we perform JSON serialization by directly going from C++ objects to serialized JSON via an internal library called . This library had some performance bottlenecks, primarily in the use of `std::ostream` for serialization: See for a discussion of its performance issues with strings. See and for integer-to-string and double-to-string performance comparisons against `std::ostream`. We found that RapidJSON has a performance focused approach that addresses these issues: Like `jsonify`, it also supports directly serializing from C++ without converting through intermediate JSON objects (via a `Writer` interface). It eschews `std::ostream` (although it introduced support for it along with a ). It performs fast integer-to-string and double-to-string conversions (see performance comparison linked

After adapting `jsonify` to use RapidJSON and eliminating some additional `mesos::Resource` copying, we ran the master state query benchmark. This benchmark builds up a large simulated cluster in the master and times the end-to-end response time from a client's perspective: This is a box plot, where the box indicates the range of the 1st and 3rd quartiles, and the lines extend to the minimum and maximum values. The results above showed a reduction in the client's end-to-end time to receive the response from approximately 7 seconds down to just over 3 seconds when both rapidjson and the `mesos::Resource` copy elimination are applied. An 55% decrease in latency, which yields a 2.3x throughput improvement of state serving. In Mesos, we use an asynchronous programming model based on actors and futures (see ). Each actor in the system operates as an HTTP server in the sense that it can set up HTTP routes and respond to requests. The master actor hosts the majority of the v0 master endpoints, including `/state`. In an actor-based model, each actor has a queue of events and processes those events in serial (without parallelism). As a result, actors can be overwhelmed if there are too many expensive events in their queue. HTTP requests to the `/state` endpoint are an example of an expensive event, especially for larger clusters. If multiple clients are polling the `/state` endpoint, responsiveness of the master can degrade significantly. In order to improve the ability to serve multiple clients of `/state`, we introduced parallel serving of the `/state` endpoint via a batching technique (see ). This was possible since `/state` is read-only against the master actor, and we accomplish this by spawning other worker actors and blocking the master until they complete (see for potential library generalizations of this technique). A benchmark was implemented that polls the masters `/state` endpoint concurrently from multiple clients and measures the observed response times across 1.6.0 and 1.7.0: The benchmark demonstrates a marked improvement in the response times as the number of clients polling the /state endpoint grows. These numbers were obtained using an optimized build on a machine with 4 x 2.9Ghz CPUs, and LIBPROCESSNUMWORKER_THREADS was set to 24. A virtual cluster was created with 100 agents, 10 running and 10 completed frameworks with 10 tasks each, on each agent. Every client polls the `/state` endpoint 50 times. Small dots denote raw measured response times, big dots denote the arithmetic mean. The top and bottom 2% of raw times were removed to filter outliers. Several users reported that the masters resource allocator was taking a long time to perform allocation cycles on larger clusters (e.g. high agent / framework counts). We investigated this issue and found that the main scalability limitation was due to excessive re-computation of the DRF ordering of roles / frameworks (see and for

We ran an existing micro-benchmark of the allocator that creates clusters with a configurable number of agents and frameworks: The results show an ~80% reduction in allocation cycle time in 1.7.1 for this particular setup (all frameworks in a single role, no filtering). Since this is a substantial improvement to a long-standing pain point for large scale users, we backported the changes to 1.7.1 since they are not included in 1.7.0. Future work is underway to improve the allocation cycle performance when quota is enabled (see: ). Reported in , during the collection of container resource consumption metrics, there are many reads of `/proc/mounts` being performed. The system mount table will be large and expensive to read if there are a lot of containers running on the agent using their own root filesystems. This was only incidentally being done as a result of some cgroup related verification code performed before reading a cgroup file. Since this code was only in place to provide better error messages, it was removed. Stephen Erb provided the following graphs that show the impact of deploying the change. First, we can see the tasks (i.e. containers) per agent: The agent with the most tasks has ~150 containers, the median and average are both around 50 containers. The following graph provided by Stephan Erb shows the latency of the `/containers` endpoint before and after deploying the fix on the same cluster: Prior to the change, the agent with the worst `/containers` latency took between 5-10 seconds to respond, and the median latency across agents was around 1 second. After the change, all of the agents have sub-second `/containers` latency. From the user reports originally in , we identified that the container launching throughput would suffer from the same issue of expensive `/proc/mounts` reads shown above in the `/containers` endpoint improvements. See . To remedy this, we moved the cgroups verification code to the call-sites. Since the cgroup just needs to be verified once during the bootstrapping agent bootstrap, this optimization significantly reduces the overhead of launching and destroying containers. A preliminary benchmark shows that the container launch / destroy throughput gained a 2x throughput improvement thanks to a 50% reduction in latency. This test uses an docker image based on the host OS image of the machine its running on: In this particular benchmark (see ), a single agent is able to launch 1000 containers in about 30 seconds, destroying those 1000 containers in just over 20 seconds. This numbers were obtained on a server with 2 x Intel(R) Xeon(R) CPU E5-2658 v3. The backlog of performance worked in tracked in JIRA, see . Any ticket with the `performance` label is picked up by this JIRA board. If you are a user and would like to suggest some areas for performance improvement, please let us know by emailing `dev@apache.mesos.org` and we would be happy to help!