apache/incubator-spark — pull request #56 by jerryshao (2013-10-12 13:10:52+00:00)
Title: Upgrade Kafka 0.7.2 to Kafka 0.8.0-beta1 for Spark Streaming

This patch upgrade Kafka dependency from 0.7.2 to 0.8.0-beta1.

Kafka 0.8.0 adds many attractive features compared to 0.7.2 like replication, multiple log dirs, many users start to use Kafka 0.8.0. Most importantly it can be built against different version of Scala including Scala 2.9.2, which alleviates painful Scala compatibility between Spark and Kafka.

This patch removes locally maintained Kafka-0.7.2-spark dependency, use public Apache kafka_2.9.2-0.8.0-beta1 dependency, also update Kafka streaming API to be consistent with Kafka 0.8.0.