elasticsearch/logstash — pull request #1528 by junggun-ds (2014-07-14 02:40:30+00:00)
Title: S3 output critical race condition problem fixed

there was a critical problem in the logstash S3 output plugin that can cause data loss and data corruption of old logs; while rolling to the next file and uploading the last complete temp file into s3, output thread keeps writing to the file being uploaded. it could cause

1. having some data appended after upload completion, which can cause data loss
2. having some data appended even after the temp file has been deleted, which can cause some corrupted small sized file with the same name. If recovery flag is on, after logstash restart the small corrupted file will be uploaded into s3. This process will overwrite the original backup'ed files and cause the backup file loss (Our company has been heavily and pitifully affected by this)

We have written this patch which properly works with ruby file objects and use Mutex properly while rolling to the next temp file.