Allows workers to keep alive only for unique pending tasks

With resource limitations, it's very easy to have a lot of workers
sitting around just waiting for resources. This is problematic for
people using locks to prevent scheduling, as it will stop new tasks from
getting scheduled. To limit this behavior, this commit adds an option to
keep alive only if there is a pending task with all requirements done
which no other worker can work on. So older workers whose tasks are all
being waited on by newer workers can just die. I require the unique
tasks to be ready because all of my workers have a unique WrapperTask at
the root, and I don't want to wait for those. Having a unique ready task
tends to mean the same thing as having a unique pending task other than
the root one for me, so this works well for my scheduling patterns.

Since this won't work well for all scheduling patterns, it has to be
activated via a config parameter, worker-require-unique-ready as well as
enabling worker-keep-alive.
Add pyparsing as deb build dependency

I verified that this works by running this on Spotify's internal build
agents. Previous builds were failing with message

    ImportError: No module named pyparsing
Merge pull request #475 from Tarrasch/add_pyparsing_dependency

Add pyparsing as build dependency
Remove SkipTest which isn't Python 2.6-compatible.
Adding python-sqlalchemy debian dependency as to not break db_task_history tests
Merge pull request #477 from freider/remove_skiptest

Remove SkipTest which isn't Python 2.6-compatible.
Add build packages as runtime dependency in deb

Packages that depend on luigi, like company internal versions of luigi,
will fail, as it's not sufficient to only have pyparsing as a build
dependency.
Merge pull request #478 from spotify/tarrasch/add-pyparsing-dependency

Add python-pyparsing as runtime dependency in deb
Merge pull request #461 from Houzz/unique_pending

Allows workers to keep alive only for unique pending tasks
Prefers tasks with more dependents in scheduling
Adds pyparsing to requirements.txt
Merge pull request #462 from Houzz/schedule_for_dependents

Prefers tasks with more dependents in scheduling
add support for spark 1.x
fix for python 2.6
include priority and resource information in visualizer
Fix deprecation warning when importing luigi.hdfs.

You will still see a deprecation warning when creating an
HdfsTarget if you haven't specified a config.
Merge pull request #485 from jcrobak/fix-luigi-hdfs-deprecation-warning

Fix deprecation warning when importing luigi.hdfs.
Merge pull request #483 from RUNDSP/spark-1x-job-pr

add support for spark 1.x
Merge pull request #484 from Houzz/viz

include priority and resource information in visualizer
This refactoring is functionally equivalent, but separates the state of the scheduler into its own class. The interface between the scheduler and the state is defined in a way so that it can be easily replaced with a SQL database or some other storage engine.
Adds sudo user to worker info when available

Many of the jobs I use have to be run as hadoop, so it's impossible to tell from
the user who actually ran the job. This adds sudo_user to the worker info so
that it's easier to tell who's running what. This is very handy when I need to
figure out why a large backfill or test job is running if it's causing a lot of
errors or hogging all the resources.