Fix list files
Merge pull request #1 from zago/s3Client

Fix list files
Improved conf
Increase timeout
Increased another timeout to avoid problems like FetchFailure
Increase number o frameSize

Error from job

org.apache.spark.SparkException: Map output statuses were 104316083 bytes which exceeds spark.akka.frameSize (15728640 bytes).

From documentation:
Maximum message size to allow in "control plane" communication (for serialized tasks and task results), in MB. Increase this if your tasks need to send back large results to the driver (e.g. using collect() on a large dataset).

Let's try
Downgrading spark, increasing driver memory and fixing configuration
Increase frame size and create union rdd instead of folding many rdds
Fixed build
Use master instance size that is consistent with driver size
Leave more memory for OS
Downgraded hadoop version to be able to remove workarounds
Downgraded hadoop version to be able to remove workarounds
Making stuff work with the right versions
Added master ami option
Allow instance types