mesos/spark — Issue #110: Bad behavior when saving to S3
Author: mateiz
Created: 2012-02-10 06:36:44+00:00
State: open

Right now the output committer part of save() operations runs on the master, but unfortunately, when accessing S3 storage through Hadoop, this operation takes quite a long time. We should either run it in parallel (as a set of tasks) or not use the Hadoop IO libraries for S3.