apache/spark — pull request #3496 by chenghao-intel (2014-11-27 08:48:05+00:00)
Title: [SPARK-4636] [SQL] WIP: Cluster By & Distribute By doesn't follow the result of Hive

This is a very interesting bug.
Semantically, Cluster By & Distribute By will not cause a global ordering, as described in Hive wiki:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy

However, the partition keys are sorted in MapReduce after shuffle, so from the user point of view, the partition key itself is global ordered, and it may looks like:
http://stackoverflow.com/questions/13715044/hive-cluster-by-vs-order-by-vs-sort-by

So I change the `HashPartitioning` to `RangePartitioning` for the `Repartition` operator.

And 2 issues still need to be figured out:
[ ] Rename the `Repartition` to `SortPartition`, this probably depends on #3386 #3481
[ ] Optimization for clauses like `DISTRIBUTE BY key SORT BY key`, we can ignore the `SORT BY` in this case.
[ ] `ORDER BY` can be optimized without shuffling all of the data to a single node.