Run Job even if input s3 url is empty

Temporary solution. Even if input provided by user is 's3://', run
the job. Useful in case of jobs similar to TeraGen.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Integrated logging

The set_prefixed_format() function updates the formatter and sets
prefix for every logged message. This prefix, cluster_id, will act
as a filter during search/debugging.

MyStreamIO will be use to redirect stdout of fabric.run/sudo(). The
write() method has been updated to log directly using flask logger.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Setting logging handler once before app init

In previous implementation, a new handler was being created
and added to app.logger. This one reuses, the original handler.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Forwarding subprocess's stdout to logger

Using subprocess.Popen(), the output of the process is written to
logger line by line.

TODO: Currently the tab and newline character are displayed grossly
in syslog. Not sure where is the issue.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Renamed StreamIO subclass to LogStream

Renamed StreamIO subclass to LogStream and created a new method
add_logger() to identify the logger to be used.

Unable to use current_app.logger due to app_context issues.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Updated Remote.{run,sudo} to send output to logger

Remote.__init__() creates new LogStream object.

Remote.run/sudo() redirect stdout and stderr to LogStream.
Not sure if latter is necessary, since according to fabric docs,
they are send combined by default.

http://docs.fabfile.org/en/1.8/usage/interactivity.html#combining-stdout-and-stderr

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Changed print statements to logger.log

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
import current_app in log.py

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
changed port value to Int type

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Added 60 sec timeout for ssh connections

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Set fabric's conn_attempts to 3, removed timeout

Timeout signifies the time until which the fabric run/sudo operation
should return. This isn't what we need.

There is not direct option for connection timeout in fabric so set
env.connection_attempt to 3.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Add support to copy output to S3

After job completion all the content in the output_dir is
transferred to the output location(s3 bucket).

If the output and input buckets are not same, the output_bucket
is mounted and data is transferred.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Set fabric timeout to 60 and attempts to 10

EC2 instance takes ~10 mins to be in working state(ssh working).
Instead of increasing timeout, we increased the number of attempts.

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Changed Name to MultiStack

Updated the following entities to refer to multistack

* imports
* function names
* filenames
* cookbook names

Signed-off-by: Shashank Sahni <shredder12@gmail.com>
Merge pull request #5 from shredder12/master

Updated to MultiStack, support for full job execution, copy to S3, performance improvements and bug fixes