apache/spark — pull request #164 by yinxusen (2014-03-18 04:33:06+00:00)
Title: [SPARK-1133] add small files input in MLlib

I add the pull request for the JIRA issue [SPARK-1133](https://spark-project.atlassian.net/browse/SPARK-1133), which brings a new files reader API in MLlib.

The interface exposes to end-user is `smallTextFiles()` in MLUtils.scala, which is similar with `textFile()` in SparkContext.scala. The interface reads a directory of text files from HDFS or native disk, then creates an `RDD[(String, String)]`, the former string is the file name, while the latter is the file content.

This interface is really useful when you want to read a bunch of files in your application. Take Latent Dirichlet Allocation, a.k.a. LDA as an example, it is urgent to use this interface.

The typical scenario here is reading a bunch of files from native disk, then processing it, and saving as a `sequenceFile` in the end. It can also read a directory of files in HDFS, though it is not a good practice.

Similar implementation in Mahout is named `sequenceFileFromDirectory`, but it is not that easy to use. Here `newAPIHadoopFile()` is used instead of `hadoopFile()`, because of the `CombineFileInputFormat` in the new Hadoop API has the opportunity to set `isSplitable()` to false, otherwise we should use shuffle to merge file content after reading blocks from HDFS.