apache/incubator-spark — pull request #651 by yinxusen (2014-02-25 14:55:38+00:00)
Title: [SPARK-1133] add new small files input

Hi community,

I add the PR for [SPARK-1133](https://spark-project.atlassian.net/browse/SPARK-1133), which brings a new files reader API in MLlib.

This PR contains 3 java files, 1 scala testsuite, and 1 API in `MLUtils`. The java files are intent to add a new InputFormat of HDFS, new API called `smallTextFiles` is similar with `textFiles` in `SparkContext`.

`smallTextFiles` will digest a directory of text files, and return an `RDD[(String, String)]`, the former String is the file name, while the latter one is the contents of the text file.

hadoop-test package is used for writing the testsuite `SmallTextFilesSuite`. In the testsuite, I test both HDFS input with `MiniDFSCluster` and local disk input.

Thanks @liancheng for the discussion, and the help of refactoring, and thanks @mridulm for the detailed review and suggestions.

Any advises are always welcome.