Graylog2/graylog2-server — pull request #120 by ssevertson (2013-03-05 22:24:11+00:00)
Title: Performance optimizations

We've studied Graylog2-Server performance based on metrics from our production environment, and attempted to optimize the parts of the system we observed as slow.

Included in my commits:
* Replace org.json.simple with Jackson stream-based parser, resulting in significant speedups depending on message size. However, Jackson is more strict than org.json.simple (see https://groups.google.com/forum/?hl=en&fromgroups=#!topic/graylog2/8fNeWiFPauQ for further discussion).
* Improve MongoDB -> StreamImpl performance. Since these objects are only cached for 5 seconds, reducing the conversion overhead helps to smooth out frequent CPU bumps.
* StreamRuleImpl now caches pre-configured Matcher instances, instead of re-building them for each log message.
* Matchers do more work at construction, rather than for each message (parsing regular expressions, particularly).
* Introduced a Pattern/PatternFactory, which examines regular expressions, and determines if any optimizations can be applied. For example, Pattern.compile("^.*foo.bar.*$").matcher("foo-bar").matches() can be reduced to Pattern.compile("foo.bar").matcher("foo-bar").find(), which runs about 4x faster. If the pattern is determined to use no regular expression features, we can even switch to String.indexOf and String.equals, which are several orders of magnitude faster than regular expressions.
* Applied the Pattern/PatternFactory to both Streams and Blacklists.

This code is currently deployed to our production monitoring environment, and we've observed a drop from ~80% CPU usage with 120 messages/second, to less than 20% with the same rate and types of messages. Average time per message spent parsing GELF from ~4ms to ~.1 ms per message, Stream Filters from ~8ms to ~.15 ms, and Blacklists from ~4ms to .16 ms.

I've included additional tests where appropriate. Please let me know if there's anything I can do to ensure these improvements are incorporated in a future release.