Take a look at use of PartitionPruningRDD ..
On Mar 4, 2014 9:33 AM, "LiGuoqiang" <notifications@github.com> wrote:

> In core/src/main/scala/org/apache/spark/SparkContext.scala:
>
> > @@ -847,6 +847,8 @@ class SparkContext(
> >        partitions: Seq[Int],
> >        allowLocal: Boolean,
> >        resultHandler: (Int, U) => Unit) {
> > +    val partitionRange = (0 until rdd.partitions.size)
> > +    require(partitions.forall(partitionRange.contains(_)), "partition index out of range")
>
>
>    1.
>
>    Can be changed
>
>    require(partitions.toSet.diff(rdd.partitions.map(_.index).toSet).isEmpty, "partition index out of range")
>
>     2.
>
>    Custom Partitioner causing a lot of problems
>
>    val partitioner = new Partitioner {
>      override def numPartitions: Int = 2
>      override def getPartition(key: Any): Int = key.hashCode() % 2
>     }
>
>    val pairs = sc.parallelize(Array((1, 2), (3, 4), (5, 6), (-1, 7)))
>    val shuffled = pairs.partitionBy(partitioner)
>    shuffled.count
>
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/44/files#r10242185>
> .
>