dianping/flume — pull request #2 by cltlfcjin (2013-06-13 03:46:15+00:00)
Title: We have modified flume 1.3.1 due to our bussness and have fixed several ...

...bugs we realized.

Duplicated data originate from using spolling dir source and memory
channel, and has serious impact on our applications. Based on this,
we modified the follow source codes:

modified: flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java
modified: flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
modified: flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
modified: flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java
modified: flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java
modified: flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
modified: flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
modified: flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java

We add a file open timestamp to event header to distinguish which file in
HDFS is the last one readed for same log file. In other words, a log
file in spolling direction could be reread because of a crash take plase in
an agent.

We add line number to header for every event to discard the repeating data
in a HDFS file which running by our duplicate java procedure.

Before rename a log file to a completed one, we first verify the queue
whether empty or not. Because if rename operate happen when the channel
queue is not empty and the agent is crashing at the same time, it may lead
to a duplicated data in HDFS or other sinks.