apache/spark — pull request #2806 by davies (2014-10-14 21:04:47+00:00)
Title: [SPARK-3916] [Streaming] discover new appended data for fileStream()

In a case that new data will be appended to existed files continuously, then fileStream() should discovery the new appended data. This patch brings this ability to fileStream.

In order to get an RDD based on partial data of file, added a private partialHadoopRDD() API.

cc @tdas