added filter for luigi visualizer
Fixes multiple purging bug.

If two worker processes get killed at the same time, we'll add them both to the
task handling queue as failures, then remove one of them from running tasks.
After this, we'll add the other one to the queue and remove it. On the third
attempt at task handling, we'll have the second task in the queue but not in
the running dict, resulting in duplicated error messages and a KeyError when
popping from running.

Even if we cut off the purge loop after it finds its first item, this could
still happen if there is already stuff in the queue or other workers insert
into the queue during/after the loop. In order to cleanly avoid removing items
multiple times, this ignores queue items that aren't in the running dict.
Fix a markup error in luigi/task.py.
Fix another markup in the externalize() function.
Fixes a typo in a variable name in fix_time
Possible flakiness fix for test_dynamic_dependencies

test_dynamic_dependencies has been flaky lately (see #571). I can't reproduce it
locally, so I assume it has something to do with Travis running things more in
parallel than I do. test_dynamic_dependencies uses mktemp, which according to
the tempfile documentation should not be used and causes race conditions in
which another program may get access to the temp file first. Replacing this with
mkdtemp should fix the issue while also clarifying that a temporary directory is
wanted and not a temporary file. I haven't been able to verify that this will
fix the flakiness, but it should probably be done anyway. I also added cleanup.
Merge pull request #574 from Houzz/fix_time_typo

Fixes a typo in a variable name in fix_time
Merge pull request #573 from berkerpeksag/patch-1

Fix a markup error in luigi/task.py.
Merge pull request #576 from Houzz/test_dynamic_dependencies_fix

Possible flakiness fix for test_dynamic_dependencies
Merge pull request #572 from Houzz/multi_purge_bug

Fixes multiple purging bug.
Merge pull request #568 from Houzz/viz_filter_pr

Add a task filter for visualizer
Uses the right task_id for unfulfilled counts in rescheduling

When rescheduling a task, we loop over all the missing dependencies and count
how many times they've each been rescheduled. Due to bit of a typo, we actually
looped over the missing dependencies and counted the dependent task each time.
This fixes that typo and counts the correct tasks.
Merge pull request #575 from Houzz/use_task_id_in_reschedule

Uses the right task_id for unfulfilled counts in rescheduling
bypass os.getuid on Windows
Fix SIGQUIT on Windows
Merge pull request #579 from majj/master

bypass os.getuid on Windows
Groups tasks that supersede each other together in the scheduler

Many tasks supersede older versions of the same job. For example, if I do an
hourly sqoop job to copy data from a production db to a hive table, completing
the 8am copy means I don't need to run the 7am copy. If both the 7am copy and
the 8am copy are pending and ready, I should run the 8am copy and consider them
both complete.

This commit achieves these goals by allowing jobs to define two optional
properties. The first is supersedes_bucket. Tasks with the same supersedes
bucket (if not None) are considered by the scheduler to supersede each other.
The scheduler will mark all earlier versions of the task DONE when one of them
completes. It us up to the user to ensure that earlier versions of the task
become complete when a later version is run.

Earlier tasks are identified by having a lower supersedes_priority. The
scheduler will prefer to schedule tasks with higher supersedes_priority.
Supersedes priority can be any type, but must be comparable between tasks in
the same bucket.
Speeds up bucket retrieval to avoid quadratic runtime

The original implementation of supersedes buckets involved calling the linear
function get_supersedes_bucket_tasks for each item with a supersedes bucket
during get_work. This could lead to quadratic runtime if enough items are in
supersedes buckets. I saw get_work running for 6 seconds after a bug lead to
~10k pending items piling up. With multiple calls coming in simultaneously, this
was enough to exceed the timeout and prevent any worker from running jobs.

This commit solves the complexity issue by pre-computing all the priorities in
a linear scan. Each supersedes bucket is sorted by supersedes_priority and
priorities updated so that nothing has smaller priority than a preceding task.
This brings the runtimes back down to milliseconds.

To verify the fix, I added 10,000 tasks in the same bucket. On my laptop, I
killed this after about half a minute pre-fix. Post-fix it takes about 90ms.