shameless steal code from yarn-app
Building a kafka meta datastructure for future use.
inherit from AbstractApplicationMaster. Changed the artifact in pom.xml
Correctly pass parameters from client to AM. Was missing a main method.
Added a bunch of log statemtns. Turns out that the kafka daemon wasn't running properly.
ScribenginAM creates its own logger.
Seems to be getting the offset from kafka.
Use simpleConsumer to consume messages from kafka.
Schedule a commit timer.
Able to append bytes to hdfs.
incoporated the appending/writing code with the reading code
Synchronize between data writing and commit
It appears that it's writing the offset to hdfs. Need to do more testing.