added PySpark support
Implemented and tested PySpark support
Takes number of workers into account in scheduler resource planning

I've noticed recently that sometimes I'll have a job with thousands of pending
tasks scheduled all using the same resource, but it's not running anything. All
of the tasks are ready and nothing else is using the resource.

It turned out that another worker had several tasks using the same resource
scheduled with higher priority. It only had one worker and wasn't going to get
to them anytime soon, but the greedy scheduling assigned all the resources to
this worker so it could sit for hours preparing to use them.

In order to alleviate this issue and allow resources to be used more freely,
this commit only allows workers to reserve resources when they are actually
about to use them.
Add support for S3 multipart uploads, allowing files larger than 5GB to be uploaded to S3
Merge pull request #515 from ddaniels888/s3_multipart_upload

Add support for S3 multipart uploads, allowing files larger than 5GB to be uploaded to S3
Remove deprecated interface.LuigiConfigParser

It's been depreceated for > 1 year.

Instead use luigi.configuration.LuigiConfigParser
Merge pull request #523 from Tarrasch/remove_configuration_LuigiConfigParser

Remove deprecated interface.LuigiConfigParser
Remove deprecated interface.get_config()

Been >1 year. Instead, use luigi.configuration.get_config()
Merge pull request #525 from Tarrasch/remove_deprecated_get_config

Remove deprecated interface.get_config()
Lessen verbosity of tests

The problem at hand is that the travis logs get truncated to 10'000
lines.  Even though you can download the whole log file, you still get
completely log blind due to the super-verbose output.

The reason we get so much output is because the python logging module
defaults to outputting to stderr which nosetests does not capture. We
want it to capture the output from the loggers, since you only care
about the output if the test fails.

My solution is nice in the sense that it doesn't touch the code at all.
Instead, we configure the logger through the built in way of doing
configuration files in luigi.

I ping @jcrobak since he added client.cfg in `.gitignore` in commit
c642a9ec. I found this use case for commiting client.cfg sensible.

The logging config file was inspired by
https://docs.python.org/2/howto/logging.html
Merge pull request #524 from Tarrasch/less_verbose_tests

Lessen verbosity of tests
Make streaming map-reduce use atomic move_dir

This is essentially using the code introduced in spotify/luigi#393,
solving the exact same issue described there.

Without this patch, the renaming is of LBYL nature, and you could end up
with moving a directory inside another instead of actually renaming a
directory.

So others don't fall into the same pitfall, I added a comment to the
rename and move function discouraging it to be used for directories.
Add missing `pass` keyword

I have no idea why this compiled before.
Update documentation for Task.complete()

The old documentation was lying I believe, there is no "False only first
time" behavior.  I even wrote this small sample program.

    import luigi.interface
    import luigi.task

    luigi.interface.setup_interface_logging()

    class MyTask(luigi.task.Task):
        def run(self):
            print 'hi'

    my_task = MyTask()

    print my_task.complete()
    my_task.run()
    print my_task.complete()

Which gave:

    $ ipython arash_test.py
    /home/arash/spotify/repos/luigis/github_luigi/luigi/task.py:471: UserWarning: Task MyTask() without outputs has no custom complete() method
      warnings.warn("Task %r without outputs has no custom complete() method" % self)
    False
    hi
    False

Also removed the 2 year old TODO. It's hard to change semantics this far
up in the project I think.
Simplify Task.complete()

Just using standard functional tools. Note that it's necessary to use
itertools to not change the semantics, in case one of the exists()
methods have side effects.
Remove ancient file test/test.py

It doesn't even contains tests. I guess this is from pre-nosetests time.

    $ LUIGI_CONFIG_PATH=test/testconfig/client.cfg HADOOP_HOME=cdh ./scripts/ci/run_tests.sh test/test.py
    Using  distribution of Hadoop from cdh

    ----------------------------------------------------------------------
    Ran 0 tests in 0.000s

    OK
Merge pull request #527 from Tarrasch/remove_ancient_test

Remove ancient file test/test.py
Merge pull request #512 from ziky90/master

Added support for PySpark
Merge pull request #526 from Tarrasch/simplify_task_complete

Simplify task complete
Fix test_getpcmd, which occasionally fails

This have been pestering me and other luigi developers a lot. For
example, when you send a correct PR, it will sometimes fail on Travis
and you have to retrigger the bulid.

For example, it happened to me recently. This is what Travis prints:

    ======================================================================
    FAIL: test_getpcmd (lock_test.TestCmd)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/travis/build/spotify/luigi/test/lock_test.py", line 32, in test_getpcmd
        "sleep 1"
    AssertionError: '[sleep]' != 'sleep 1'

    ----------------------------------------------------------------------