apache/spark — pull request #2589 by chenghao-intel (2014-09-30 03:27:58+00:00)
Title: [SPARK-3739] [SQL] Update the split num base on block size for table scanning

Source file input split is probably better based on block size of HDFS, while scanning Hive table, other than the settings of 'mapred.reducer.tasks' or 'taskScheduler.defaultParallelism'. Checkout http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-core/2.3.0-mr1-cdh5.1.0/org/apache/hadoop/mapred/FileInputFormat.java#203 for how Hadoop handles the input splits.

Currently, too many splits for small source table file, which may causes performance issue, as well as the different result of Hive in unit test.