Merge pull request #500 from campanja/s3client-accept-any-boto-config

S3client accept any boto config
Make S3Target clean up when using format
Fixes wrong units in disable e-mail message

When the units for the disable parameters changed, I neglected to
update the e-mail message accordingly.
Merge pull request #503 from Houzz/disable_email_fix

Fixes wrong units in disable e-mail message
Another Hadoop native libraries warning fix.

By default 64bit Apache Hadoop binaries come with 32bit native libraries, which, unless compiled separately, cause warning header to be appended:

"OpenJDK 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'."

>>> from luigi import hdfs
>>> cmd = ['hadoop', 'fs', '-count', '/cache']
>>> stdout = hdfs.call_check(cmd)
>>> print stdout
OpenJDK 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
           1            1           50214873 /cache

This fix will get rid of this header.
Merge pull request #505 from liquid-sky/master

Another Hadoop native libraries warning fix.
Remove trailing slash when using snakebite rename

Old functionality caused odd behavior when doing:

    snakebite.rename('foo/', 'bar/')

and the bar did not exist. As result it created the 'bar' first and
then moved 'foo' in there, resulting in 'bar/foo'.

By stripping the slashes on the right, the behavior works as one would
think: renames 'foo' to 'bar'.
Merge pull request #507 from nailor/strip-trailing-slash-snakebite-rename

Remove trailing slash when using snakebite rename
Merge pull request #501 from campanja/s3target-clean-up-after-unpacking

Make S3Target clean up when using format
Add tests for the trailing slash snakebite rename case
Merge pull request #508 from nailor/add-unittest-trailing-slash

Add tests for the trailing slash snakebite rename case
Updates documentation for configuration

This updates the documentation to contain explanations of most configuration
options that can be specified in client.cfg. I tried to include as many as
possible, but I've probably missed some. I grepped the code to try to find all
of them. For completeness, these include my best understanding of how each
parameter works. Many descriptions are therefore incomplete and need to be
filled out better by people who understand them.

Some descriptions may be wrong, but only in ways that reflect the understanding
an average user would have after reading the code to figure out how to properly
configure luigi. So this shouldn't be any worse than what we already have even
where it's inaccurate.
added PySpark support
Implemented and tested PySpark support
Merge pull request #509 from Houzz/config_param_docs

Updates documentation for configuration
Takes number of workers into account in scheduler resource planning

I've noticed recently that sometimes I'll have a job with thousands of pending
tasks scheduled all using the same resource, but it's not running anything. All
of the tasks are ready and nothing else is using the resource.

It turned out that another worker had several tasks using the same resource
scheduled with higher priority. It only had one worker and wasn't going to get
to them anytime soon, but the greedy scheduling assigned all the resources to
this worker so it could sit for hours preparing to use them.

In order to alleviate this issue and allow resources to be used more freely,
this commit only allows workers to reserve resources when they are actually
about to use them.
Running hdfs tests locally with minicluster
initial RangeHourly implementation

Efficiently produces contiguous completed ranges of recurring tasks.

Already in:
* RangeHourlyBase general API
* RangeHourly subclass which automagically facilitates filesystem listing
  as the most common case. Supports tasks with multiple outputs and, as a
  convenience, WrapperTask hierarchies as well
* Convenient CLI usage

Coming soon:
* Events emitted with relevant metrics
* RangeDaily/Weekly/X-ly generalization
* More sanity checks to behave gracefully when preconditions aren't
  satisfied

Wishlist:
* More common cases facilitated, that are not like filesystem listing
Merge pull request #480 from spotify/ulzha/range

RangeHourly
Add support for S3 multipart uploads, allowing files larger than 5GB to be uploaded to S3