Correct.  The default is to continue where you last left off.  There is also autoffset.reset=largest which lets you skip passed events that you haven't read (ie- when taking a consumer down for maintenance and bringing it back online, there could be hours worth of data queued up that you may not be interested in).

Here is some background on setting offsets using the high level consumer (which spark and most other people use):
https://groups.google.com/forum/?fromgroups=#!topic/kafka-dev/7n_l-O70mk0
http://grokbase.com/t/kafka/users/12b9bmsy7k/question-about-resetting-offsets-and-the-high-level-consumer

Long story story short, setting arbitrary offsets using the high level api isn't supported (which is why kafka had to be modified to use ZookeeperConsumerConnector in Spark).  In the second link Jun mentions that exposing that functionality would be post 0.8.


> can you give an example on how users who may have used the arbitrary initial offset can migrate their code to the new API? Is it even possible?

Since the high level consumer doesn't support setting arbitrary offsets, it shouldn't have been exposed in the first place.  Worst case, if someone still needed this functionality in spark, they should be able to copy the old code that was in spark and use that as a template.  Ie- have their spark driver connect as a consumer, call setOffets, and then close before consuming.  After that, they'd call kafkaStream and it would pick up where they told it to.

The "proper" way to allow arbitrary offsets in kafka currently is to use the lower level kafka API (called the SimpleConsumer, which is a misnomer because it is significantly more complex to use).  While the SimpleConsumer lets you manage offsets, there is a great deal that it doesn't do for you.  Integrating with zookeeper, manage saving/loading offsets, etc.. is all left up to you.


In my book the best bet is to stick with the high level consumer.  When arbitrary offsets are exposed beyond kafka 0.8 they should be able to be passed in as kafkaParams, so we won't have to worry about making any more API changes down the road.


That is probably WAY more than you ever wanted to know about Kafka =)