fluent/fluent-plugin-webhdfs — pull request #8 by choplin (2013-07-18 09:06:28+00:00)
Title: add an append option

The problem is that an append method of WebHDFS is not atomic. In more
detail, if a datanode which you are communicating via a append method
shutdown for some reason, the file which you have benn writing will remain on
HDFS containing incomplete contents, and there is no simple way to know how
much contents was written.

However, Fluentd will retry to write whole contents in a paticular chunk when
emit transaction fail because of a datanode failure.

Consequently, a dumplication of content will occures unexpectedly.

To avoid this problem, set an append option to false. When append is set to
false, flunet-plugin-webhdfs write each chunk to indivisual file on HDFS, and
a file name will not change through retries.

This patch also produces ${chunk_id} placeholder. This is availabale for path
option and replaced with chunk#unique_id in WebHDFSOutput#write method.