Graylog2/graylog2-server — pull request #83 by mhart (2012-06-18 01:52:40+00:00)
Title: Fix nasty race condition when handling chunked GELF messages

Without this patch, I lose more than 70% of chunked GELF messages when testing on localhost. With this patch, I don't lose any.

This seems to be relatively common too:

https://groups.google.com/d/topic/graylog2/R6_idX-2HfY/discussion
https://groups.google.com/d/topic/graylog2/jkPmQp8oklE/discussion

It's pretty straightforward as to why this occurs - if `insertChunk()` is called, then there's a high chance `messageMap.get()` will return `null` if two chunks from the same message are being processed concurrently (which happens a lot with chunked messages on localhost). This then leads to a new `ChunkedGELFMessage` being created for each chunk, and one will overwrite the other in the `messageMap.put`. The surviving chunk will then be removed 5 seconds later as Graylog thinks the whole message is incomplete (the debug logs reveal this occurring).

An easy fix is just to make `insertChunk()` synchronized, but that would have more of a performance hit locking on the whole manager (or map) than if we just optimistically insert using `ConcurrentMap`'s `putIfAbsent()`, and then lock only on the message itself when inserting new chunks into it.