Merge branch 'master' of github.com:tbfenet/incubator-spark

PartitionPruningRDD is using index from parent

I was getting a ArrayIndexOutOfBoundsException exception after doing union on pruned RDD. The index it was using on the partition was the index in the original RDD not the new pruned RDD.

(cherry picked from commit 2fead510f74b962b293de4d724136c24a9825271)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #196 from pwendell/master

TimeTrackingOutputStream should pass on calls to close() and flush().

Without this fix you get a huge number of open files when running shuffles.

(cherry picked from commit f20093c3afa68439b1c9010de189d497df787c2a)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #193 from aoiwelle/patch-1

Fix Kryo Serializer buffer documentation inconsistency

The documentation here is inconsistent with the coded default and other documentation.

(cherry picked from commit 086b097e33a2ce622ec6352819bccc92106f43b7)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #200 from mateiz/hash-fix

AppendOnlyMap fixes

- Chose a more random reshuffling step for values returned by Object.hashCode to avoid some long chaining that was happening for consecutive integers (e.g. `sc.makeRDD(1 to 100000000, 100).map(t => (t, t)).reduceByKey(_ + _).count`)
- Some other small optimizations throughout (see commit comments)

(cherry picked from commit 718cc803f7e0600c9ab265022eb6027926a38010)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #197 from aarondav/patrick-fix

Fix 'timeWriting' stat for shuffle files

Due to concurrent git branches, changes from shuffle file consolidation patch
caused the shuffle write timing patch to no longer actually measure the time,
since it requires time be measured after the stream has been closed.

(cherry picked from commit 972171b9d93b07e8511a2da3a33f897ba033484b)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #201 from rxin/mappartitions

Use the proper partition index in mapPartitionsWIthIndex

mapPartitionsWithIndex uses TaskContext.partitionId as the partition index. TaskContext.partitionId used to be identical to the partition index in a RDD. However, pull request #186 introduced a scenario (with partition pruning) that the two can be different. This pull request uses the right partition index in all mapPartitionsWithIndex related calls.

Also removed the extra MapPartitionsWIthContextRDD and put all the mapPartitions related functionality in MapPartitionsRDD.

(cherry picked from commit 14bb465bb3d65f5b1034ada85cfcad7460034073)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #219 from sundeepn/schedulerexception

Scheduler quits when newStage fails

The current scheduler thread does not handle exceptions from newStage stage while launching new jobs. The thread fails on any exception that gets triggered at that level, leaving the cluster hanging with no schduler.

(cherry picked from commit 740922f25d5f81617fbe02c7bcd1610d6426bbef)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #181 from BlackNiuza/fix_tasks_number

correct number of tasks in ExecutorsUI

Index `a` is not `execId` here

(cherry picked from commit f568912f85f58ae152db90f199c1f3a002f270c1)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #218 from JoshRosen/spark-970-pyspark-unicode-error

Fix UnicodeEncodeError in PySpark saveAsTextFile() (SPARK-970)

This fixes [SPARK-970](https://spark-project.atlassian.net/browse/SPARK-970), an issue where PySpark's saveAsTextFile() could throw UnicodeEncodeError when called on an RDD of Unicode strings.

Please merge this into master and branch-0.8.

(cherry picked from commit 8a3475aed66617772f4e98e9f774b109756eb391)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #95 from aarondav/perftest

Minor: Put StoragePerfTester in org/apache/
(cherry picked from commit a51359c917a9ebe379b32ebc53fd093c454ea195)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #223 from rxin/transient

Mark partitioner, name, and generator field in RDD as @transient.

As part of the effort to reduce serialized task size.
(cherry picked from commit d6e5473872f405a6f4e466705e33cf893af915c1)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #227 from pwendell/master

Fix small bug in web UI and minor clean-up.

There was a bug where sorting order didn't work correctly for write time metrics.

I also cleaned up some earlier code that fixed the same issue for read and
write bytes.
(cherry picked from commit 182f9baeed8e4cc62ca14ae04413394477a7ccfb)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Change log for release 0.8.1-incubating
Merge pull request #228 from pwendell/master

Document missing configs and set shuffle consolidation to false.
(cherry picked from commit 5d460253d6080d871cb71efb112ea17be0873771)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Bumping version numbers for 0.8.1 release
Merge pull request #232 from markhamstra/FiniteWait

jobWaiter.synchronized before jobWaiter.wait

...else ``IllegalMonitorStateException`` in ``SimpleFutureAction#ready``.

(cherry picked from commit 078049877e123fe7e4c4553e36055de572cab7c4)
Signed-off-by: Reynold Xin <rxin@apache.org>