apache/spark — pull request #2765 by wangxiaojing (2014-10-11 08:57:59+00:00)
Title: [spark-3586][streaming]Support nested directories in Spark Streaming

For text files, the method streamingContext.textFileStream(dataDirectory).
The improvement of the streaming to Support subdirectories,spark streaming can monitor the subdirectories dataDirectory and process any files created in that directory.
eg:
streamingContext.textFileStream(/test).
Look at the direction contents:
/test/file1
/test/file2
/test/dr/file1
if the directory "/test/dr/" have new file "file2" ,spark streaming can process the file