amplab-extras/SparkR-pkg — pull request #108 by concretevitamin (2014-11-15 04:01:13+00:00)
Title: Fix slow take(): deserialize only up to necessary # of elements.

JIRA: [SPARKR-139](https://sparkr.atlassian.net/browse/SPARKR-139)

Previously, before trying to take as many elements as necessary from a partition, `take()` naively deserializes all elements in that partition at once. When an RDD has a giant partition and the number of elements to take is small -- which is the majority use case of this function -- this behavior results in driver getting OOM / being stuck forever. This patch changes the behavior so that only necessary number of elements will be examined & deserialized to the R side.

Tested on a ~70M textfile dataset with 2 partitions. Before this patch, taking 1 element results in OOM/stuck behavior; with this patch the operation finishes fairly quickly.