shameless steal code from yarn-app
Building a kafka meta datastructure for future use.
inherit from AbstractApplicationMaster. Changed the artifact in pom.xml
Correctly pass parameters from client to AM. Was missing a main method.
Added a bunch of log statemtns. Turns out that the kafka daemon wasn't running properly.
ScribenginAM creates its own logger.
Seems to be getting the offset from kafka.
Use simpleConsumer to consume messages from kafka.
Schedule a commit timer.
Able to append bytes to hdfs.
incoporated the appending/writing code with the reading code
Synchronize between data writing and commit
It appears that it's writing the offset to hdfs. Need to do more testing.
committing the offset.
Reading and writing the offset correctly.
Forgot to check in OffsetReader.java.
Added a new log entry format. Comes with a unit test.
CommitLog. Stole minicluster from Tuan. Still in the process of integrating everything.
Use miniCluster's FS during unit test.
Fixed: need to start the iterator from the tail end of the list.