jolicode/elasticsearch-php-benchmark — pull request #1 by polyfractal (2013-11-22 14:09:06+00:00)
Title: Benchmark updates

This is a relatively large PR. I wanted to explain the rationale behind each change, so I did not squash my commits which will allow me to explain each one in turn. Hopefully my reasoning will be clear and not look like I'm trying to skew results unfairly :)

**Changes to the benchmark itself**
These changes should affect all clients equally. The goal behind these changes was to ensure a fair timing of the client itself, without being affected by PHP or the JVM.

- [[8b49c]](8b49c832d6cff45235b3cb4348fb525bf0753514) Change required PHP version to at least 5.5.0. It doesn't make sense to benchmark an application on anything older than 5.5. If an organization is concerned about performance they will be running the most current version of PHP. Backwards-compatibility is perfectly acceptable (all listed clients are backwards compatible to 5.3) but it would be very wrong if clients were optimizing for a 4-year-old version of PHP instead of the latest

- [[dc48d]](dc48d1f6c6d1c252209bedb06424c688f51f04ab) Caches should be explicitly cleared. Deleting the index should clear all associated caches, but being explicit doesn't hurt.

- [[7984f5]](7984f5153ed84f51a631fa3d53f3067ea7453769) This one is important: the index and caches need to be deleted before every test run. If they are not deleted, the clients earlier in the benchmark will "warm" the caches inside of elasticsearch. During this warming process they are hitting cold caches and suffering a latency penalty as elasticsearch loads the data to memory. Later clients won't hit this and thus gain a benefit. By deleting before each test run we ensure that the "slate is clean" for every client

- [[524b4]](524b471b58e4cc67e6e5ae9f63f05d1b94ba7126) Similar to the last change, the JVM itself needs to be warmed. The JVM uses Just-In-Time compilation and outputs bytecode dynamically. As the JVM runs, it analyzes "hot paths" through the code and optimizes the JIT output that is being produced. There are also other factors, such as new-eden heap sizing, which take a while to optimize. This commit adds a "warming" phase to the test by simply running the benchmarked code five times, which should be sufficient to warm up all the paths through the ES codebase. It also means that we now need to pass the ES binary to the test as a path (shown later)

- [[0fbb5]](0fbb596d3dfd0208138eae161ad26014fedc237e) Refactor the timing mechanism slightly by injecting the stopwatch object into each test method. Previously, the stopwatch was recording the timing of "extra" code that was unrelated to the test, such as creation of temp variables and result checking. This commit simply moves the timing to wrap the function call itself. It also fixes some consistency issues (temp variables vs inline declaration of arrays, return values, etc), as well as summing the individual events to arrive at a "total" instead of simply timing the whole process (which includes a lot of extra time lost to PHP in the test itself)

- [[bd03c]](bd03c1a2345da8036dfa4d2a640086cd1e964399) [[752f0]](752f01d1eb6b4d86e5dfcbd1afd0054b8c3b396b) This adds a new set of tests call "persistent". The previous benchmark only tested "transient" situations - whenever you need to quickly instantiate a client, perform a single query and then shut it down. This is a great way to profile use-cases such as autocomplete, suggestions or even simple search box. Any time where the client turning on fast is the priority. In contrast, "persistent" tests situations where a single client is spawned and performs a large number of requests, such as bulk importing or analytics. These are very different scenarios and should be tested independently.

*(In practice, "transient" times will be much decreased since an optimal system will have a bytecode cache enabled (Zend, APC) and the largest overhead in "transient" for all clients is autoloading.)*

- [[0afc3]](0afc3fbf6ebf5f7366b5f9c810dd436dc4228beb) We must wait for the cluster to become yellow before indexing documents. Due to the automated speed of the script, it is possible to begin trying to insert documents while the index is still initializing (red state), which means the docs are not inserted and benchmarking is completely wrong. It's possible this is what caused that strange spike in your Elastica Suggestion test, for example. Also reduced replicas to zero since it doesn't make sense for a single-node test.

- This is not a change, but I would recommend not using StopWatch in the future. It [rounds the microtime](https://github.com/symfony/symfony/blob/51c6d7696c2fb98f7720d842c502662e6451f925/src/Symfony/Component/Stopwatch/StopwatchEvent.php#L226) and [truncates to integer](https://github.com/symfony/symfony/blob/51c6d7696c2fb98f7720d842c502662e6451f925/src/Symfony/Component/Stopwatch/StopwatchPeriod.php#L33). I found that these rounding problems contribute 5-15ms of inaccuracy for all clients. It seems like bad practice to round/truncate timing data when you are interested in times...I'm not sure why Symfony chose to do this.

**Changes that affect just Elasticsearch-php**

- [[8c369]](8c369d58b5d223d69cab338018e2ad8cf59f6e10) This changes the Elasticsearch-PHP client to use dev-master since some changes are untagged at the moment. It then adds per-scenario configuration. In all cases, the connection class has been changed from Guzzle to an internal implementation of curl-multiselect. I've been very unhappy with Guzzle performance (largely due to autoloading and excessive classes)...it is possible that CurlMultiConnection will become the default soon. The connection pool is also changed to either StaticNoPing (for the transient cases where you optimistically assume all connections are alive) or Static (for persistent, where you can afford occasional pings). Lastly, the selector is switched to Sticky for "persistent" since that is optimal but not able to become a default (since it depends on an organizations load-balancing architecture).

**Changes to other clients**
Since all clients now receive the benchmark type (transient vs persistent), they can be optimized in a similar fashion as the elasticsearch-php client. I'm not very familiar with their code, however, so if anyone else can add per-configuration settings to them that would be great!

**Changes in Elasticsearch-php itself**
In the spirit of fairness and disclosure, there were a few changes to the client itself which were not available when you ran the benchmark. Here are the recent changes which help a bit:

- [[6ec9c]](6ec9cd5497c12dd9024e8dcd326134372aaec90d) This implements a StaticNoPingConnectionPool class. The [issue](https://github.com/elasticsearch/elasticsearch-php/issues/21) had been open for about a week and the code was done, but I did not push it before going on a trip for work. Pushed when I got home so this was unavailable to your tests. Shaves about 100ms off each test for the client (all due to 1-2ms pings which adds up over 500 iterations)

- [[0cf2c]](0cf2c938e5941a026687cda60bc6fdc76a494702) This removes some header parsing that was unnecessary, and disables SSL validation. Unsure the impact on the test, but the header parsing utilized a Guzzle class which causes two files to be autoloaded...so it probably had some impact on the transient tests.

- [[043e8]](043e8fbce79afe854b2b38347f6d92bfe45daba5) Not a performance boost, but a bug in the StickyRoundRobinSelector which I found by using your benchmark code :)

**Running the benchmark, results**
Phew! Sorry for such a long PR message. The test is now invoked by passing a path to ES itself:

./bin/run-all.sh "/usr/share/elasticsearch/bin/elasticsearch -Des.path.conf=/etc/elasticsearch/"

The benchmark will spin up elasticsearch, warm it with some queries then perform the timing. This means that ES cannot be running on your system before the benchmark.

[The results of the modified code can be found here for my system](https://gist.github.com/polyfractal/7600115).

And some graphs:

![persistent](https://f.cloud.github.com/assets/1224228/1600901/303fc204-537d-11e3-88c7-bec602d88a1c.png)
![transient](https://f.cloud.github.com/assets/1224228/1600902/304c8426-537d-11e3-802d-17c7af62e78b.png)

**Conclusion**
So that was a really long post, sorry! Because benchmarks can be a sensitive subject, and since multiple client authors are involved, I wanted to be very open and clear about each of my proposed changes. I don't want it to look like I am tweaking the test to benefit the Elasticsearch-php client. The majority of the changes benefit all clients (I saw an improvement in times across the board for all clients).

The few changes that affect Elasticsearch-php are either going to become defaults soon, or are unable to be defaults because they may not apply to your organizations architecture (StickyRoundRobin for example)

There is definitely profiling to be done on the elasticsearch-php client...for example I'm unsure why GETs are so consistently slow. These issues will be profiled and tuned for sure! But I think the changes more accurately reflect the performance of my and other clients.