Updates documentation for configuration

This updates the documentation to contain explanations of most configuration
options that can be specified in client.cfg. I tried to include as many as
possible, but I've probably missed some. I grepped the code to try to find all
of them. For completeness, these include my best understanding of how each
parameter works. Many descriptions are therefore incomplete and need to be
filled out better by people who understand them.

Some descriptions may be wrong, but only in ways that reflect the understanding
an average user would have after reading the code to figure out how to properly
configure luigi. So this shouldn't be any worse than what we already have even
where it's inaccurate.
added PySpark support
Implemented and tested PySpark support
Merge pull request #509 from Houzz/config_param_docs

Updates documentation for configuration
Takes number of workers into account in scheduler resource planning

I've noticed recently that sometimes I'll have a job with thousands of pending
tasks scheduled all using the same resource, but it's not running anything. All
of the tasks are ready and nothing else is using the resource.

It turned out that another worker had several tasks using the same resource
scheduled with higher priority. It only had one worker and wasn't going to get
to them anytime soon, but the greedy scheduling assigned all the resources to
this worker so it could sit for hours preparing to use them.

In order to alleviate this issue and allow resources to be used more freely,
this commit only allows workers to reserve resources when they are actually
about to use them.
Running hdfs tests locally with minicluster
initial RangeHourly implementation

Efficiently produces contiguous completed ranges of recurring tasks.

Already in:
* RangeHourlyBase general API
* RangeHourly subclass which automagically facilitates filesystem listing
  as the most common case. Supports tasks with multiple outputs and, as a
  convenience, WrapperTask hierarchies as well
* Convenient CLI usage

Coming soon:
* Events emitted with relevant metrics
* RangeDaily/Weekly/X-ly generalization
* More sanity checks to behave gracefully when preconditions aren't
  satisfied

Wishlist:
* More common cases facilitated, that are not like filesystem listing
Merge pull request #480 from spotify/ulzha/range

RangeHourly
Add support for S3 multipart uploads, allowing files larger than 5GB to be uploaded to S3
correct a typo in the command line options documentation, expand the meaning of 'logdir'
Add 'tools' package so the package works when installed
fix indentation
Merge pull request #516 from hydrogen18/master

Add 'tools' package & Improve command line option documentation
Merge pull request #510 from anyman/master

Running hdfs tests locally with minicluster
Merge pull request #515 from ddaniels888/s3_multipart_upload

Add support for S3 multipart uploads, allowing files larger than 5GB to be uploaded to S3
Remove deprecated interface.LuigiConfigParser

It's been depreceated for > 1 year.

Instead use luigi.configuration.LuigiConfigParser
Merge pull request #523 from Tarrasch/remove_configuration_LuigiConfigParser

Remove deprecated interface.LuigiConfigParser
Remove deprecated interface.get_config()

Been >1 year. Instead, use luigi.configuration.get_config()
Merge pull request #525 from Tarrasch/remove_deprecated_get_config

Remove deprecated interface.get_config()
Lessen verbosity of tests

The problem at hand is that the travis logs get truncated to 10'000
lines.  Even though you can download the whole log file, you still get
completely log blind due to the super-verbose output.

The reason we get so much output is because the python logging module
defaults to outputting to stderr which nosetests does not capture. We
want it to capture the output from the loggers, since you only care
about the output if the test fails.

My solution is nice in the sense that it doesn't touch the code at all.
Instead, we configure the logger through the built in way of doing
configuration files in luigi.

I ping @jcrobak since he added client.cfg in `.gitignore` in commit
c642a9ec. I found this use case for commiting client.cfg sensible.

The logging config file was inspired by
https://docs.python.org/2/howto/logging.html