mesos/spark — pull request #445 by JoshRosen (2013-02-04 05:23:08+00:00)
Title: Fix exit status in PySpark unit tests; fix/optimize PySpark's RDD.take()

This pull request fixes three problems that I ran into while working on the Python tutorial:

- My custom doctest `__main__` was not using the number of failures to determine the exit status, which could lead to doctest failures going undetected unless you inspected the logs.

- `PythonRDD.takePartition` crashed when consuming the output of `textFile()` because it returned the iterator passed to the `runJob` callback function rather than reading it into an array. This caused problems because the HDFS iterator was closed in an `onComplete` callback.

- PySpark's `RDD.take()` and `RDD.first()` were slow because they transferred more objects from Java to Python than were strictly necessary.