Fix lock file owner issue

When user abc123 runs luigi and leaves a lock file, you will have a
state something like this

    $ ll /var/tmp/luigi/4c2ae353c4616cf210b3dc2b673d07ab.pid
    -rwxrwxrwx 1 abc123 abc123 6 Oct 14 14:21 /var/tmp/luigi/4c2ae353c4616cf210b3dc2b673d07ab.pid*

Then, when user xyz789 runs the same luigi command which has the same
hash for the lock file, the user will run on a runtime exceptin when
the code in `luigi/lock.py` is doing chmod on the file, because the
owner of the file is abc123. Since there's no way to chown the file to
yourself if you're not the owner. So we just delete the file before
doing an overwrite of the file. This is the most reliable solution I
coud come up with.

I would love to create a test case for this, but I'm not sure how to do
it. Is there any way to chown a file to some other user (anyone) that
works on all supported luigi platforms?
Adds DISABLED status for tasks in scheduler.

Tasks can get the disabled status either through scheduling or from repeated
failures. The scheduler handles disabling via repeated failures, and tasks
disabled this way are automatically re-enabled after a fixed amount of time
or when pruned. This allows tasks that are repeatedly running and blocking
progress on other tasks to be pushed aside until someone can have a look.

Tasks can be set as disabled via the disabled parameter in Task. Tasks with
this parameter set to true have will be scheduled as disabled and will not be
run. This disable can only be removed by setting disabled to false in the task
and re-scheduling it. This makes it easier to stop running broken tasks without
having to manually figure out all upstream dependents and remove those as well.
Merge pull request #482 from Houzz/disabled_tasks

Adds DISABLED status for tasks in scheduler.
Fixes scheduler bug caused by updating priority on pruned tasks

If a task is pruned in the scheduler but a task that depends on it still exists,
the priority update recursion in add_task becomes vulnerable to encountering the
pruned task before it has been added back. Even worse, if the task that depends
on the pruned task is done, the pruned task will never be rescheduled.

This adds a test to demonstrate the code flaw and fixes it by skipping over
missing tasks in priority update. The test also shows that the priority will
still be updated appropriately if the task is eventually added back.
Merge pull request #499 from Houzz/priority_update_bug_fix

Fixes scheduler bug caused by updating priority on pruned tasks
Merge pull request #497 from Tarrasch/fix_lock_file_owner

Fix lock file owner issue
Fix pep8 errors
S3Client accepts kwargs and pass them to boto

All options can be inserted in s3 section in config. All values will be
converted to int if possible to allow port etc to be specified.
Fix pep8 errors
Test s3 config from luigi configuration
Merge pull request #500 from campanja/s3client-accept-any-boto-config

S3client accept any boto config
Make S3Target clean up when using format
Fixes wrong units in disable e-mail message

When the units for the disable parameters changed, I neglected to
update the e-mail message accordingly.
Merge pull request #503 from Houzz/disable_email_fix

Fixes wrong units in disable e-mail message
Another Hadoop native libraries warning fix.

By default 64bit Apache Hadoop binaries come with 32bit native libraries, which, unless compiled separately, cause warning header to be appended:

"OpenJDK 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'."

>>> from luigi import hdfs
>>> cmd = ['hadoop', 'fs', '-count', '/cache']
>>> stdout = hdfs.call_check(cmd)
>>> print stdout
OpenJDK 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
           1            1           50214873 /cache

This fix will get rid of this header.
Merge pull request #505 from liquid-sky/master

Another Hadoop native libraries warning fix.
Remove trailing slash when using snakebite rename

Old functionality caused odd behavior when doing:

    snakebite.rename('foo/', 'bar/')

and the bar did not exist. As result it created the 'bar' first and
then moved 'foo' in there, resulting in 'bar/foo'.

By stripping the slashes on the right, the behavior works as one would
think: renames 'foo' to 'bar'.
Merge pull request #507 from nailor/strip-trailing-slash-snakebite-rename

Remove trailing slash when using snakebite rename
Merge pull request #501 from campanja/s3target-clean-up-after-unpacking

Make S3Target clean up when using format
Running hdfs tests locally with minicluster