Updated openstack/openstack
Project: openstack/sahara  413e2ffaccaeca0dd51453338c66cc9d2172b49f

Implement EDP for a Spark standalone cluster

This change adds an EDP engine for a Spark standalone cluster.
The engine uses the spark-submit script and various linux
commands via ssh to run, monitor, and terminate Spark jobs.

Currently, the Spark engine can launch "Java" job types (this is
the same type used to submit Oozie Java action on Hadoop clusters)

A directory is created for each Spark job on the master node which
contains jar files, the script used to launch the job, the
job's stderr and stdout, and a result file containing the exit
status of spark-submit.  The directory is named after the Sahara
job and the job execution id so it is easy to locate.  Preserving
these files is a big help in debugging jobs.

A few general improvements are included:
* engine.cancel_job() may return updated job status
* engine.run_job() may return job status and fields for job_execution.extra
in addition to job id

Still to do:
* create a proper Spark job type (new CR)
* make the job dir location on the master node configurable (new CR)
* add something to clean up job directories on the master node (new CR)
* allows users to pass some general options to spark-submit itself (new CR)

Partial implements: blueprint edp-spark-standalone

Change-Id: I2c84e9cdb75e846754896d7c435e94bc6cc397ff