added PySpark support
Implemented and tested PySpark support
Remove ancient file test/test.py

It doesn't even contains tests. I guess this is from pre-nosetests time.

    $ LUIGI_CONFIG_PATH=test/testconfig/client.cfg HADOOP_HOME=cdh ./scripts/ci/run_tests.sh test/test.py
    Using  distribution of Hadoop from cdh

    ----------------------------------------------------------------------
    Ran 0 tests in 0.000s

    OK
Merge pull request #527 from Tarrasch/remove_ancient_test

Remove ancient file test/test.py
Merge pull request #512 from ziky90/master

Added support for PySpark
Add missing `pass` keyword

I have no idea why this compiled before.
Update documentation for Task.complete()

The old documentation was lying I believe, there is no "False only first
time" behavior.  I even wrote this small sample program.

    import luigi.interface
    import luigi.task

    luigi.interface.setup_interface_logging()

    class MyTask(luigi.task.Task):
        def run(self):
            print 'hi'

    my_task = MyTask()

    print my_task.complete()
    my_task.run()
    print my_task.complete()

Which gave:

    $ ipython arash_test.py
    /home/arash/spotify/repos/luigis/github_luigi/luigi/task.py:471: UserWarning: Task MyTask() without outputs has no custom complete() method
      warnings.warn("Task %r without outputs has no custom complete() method" % self)
    False
    hi
    False

Also removed the 2 year old TODO. It's hard to change semantics this far
up in the project I think.
Simplify Task.complete()

Just using standard functional tools. Note that it's necessary to use
itertools to not change the semantics, in case one of the exists()
methods have side effects.
Make mapreduce tests not print garbage

This patch removes the output looking like this when running the tests:

    reporter:counter:word,iljlk,1
    reporter:counter:word,jlk,1
    reporter:counter:word,jlk,1
    reporter:counter:word,jk,1
    reporter:counter:word,jkl,1
    reporter:counter:word,jlk,1
    reporter:counter:word,jlkj,1
    reporter:counter:word,j,1
    reporter:counter:word,ioj,1
    reporter:counter:word,ioj,1
    reporter:counter:word,kuh,1
    reporter:counter:word,kjh,1
Mute test output 'error: No task XYZ.'

You would previously see:

        test_non_existent_class (test.cmdline_test.CmdlineTest) ... nosetests: error: No task XYZ.

It's pretty annoying to ctrl+f for "error" and get false positives.
Mute testcase test_bin_luigi

Previously you saw:

    test_bin_luigi (test.cmdline_test.CmdlineTest) ... DEBUG: Checking if WriteToFile(filename=/tmp/luigi-tmp-465408515) is complete
    INFO: Scheduled WriteToFile(filename=/tmp/luigi-tmp-465408515) (PENDING)
    INFO: Done scheduling tasks
    INFO: Running Worker with 1 processes
    DEBUG: Asking scheduler for work...
    DEBUG: Pending tasks: 1
    INFO: [pid 2629] Worker Worker(salt=098531595, workers=1, host=arash-spotify-T440s, username=arash, pid=2629) running   WriteToFile(filename=/tmp/luigi-tmp-465408515)
    INFO: [pid 2629] Worker Worker(salt=098531595, workers=1, host=arash-spotify-T440s, username=arash, pid=2629) done      WriteToFile(filename=/tmp/luigi-tmp-465408515)
    DEBUG: 1 running tasks, waiting for next task to finish
    DEBUG: Asking scheduler for work...
    INFO: Done
    INFO: There are no more tasks to run at this time
    INFO: Worker Worker(salt=098531595, workers=1, host=arash-spotify-T440s, username=arash, pid=2629) was stopped. Shutting down Keep-Alive thread