Allows workers to keep alive only for unique pending tasks

With resource limitations, it's very easy to have a lot of workers
sitting around just waiting for resources. This is problematic for
people using locks to prevent scheduling, as it will stop new tasks from
getting scheduled. To limit this behavior, this commit adds an option to
keep alive only if there is a pending task with all requirements done
which no other worker can work on. So older workers whose tasks are all
being waited on by newer workers can just die. I require the unique
tasks to be ready because all of my workers have a unique WrapperTask at
the root, and I don't want to wait for those. Having a unique ready task
tends to mean the same thing as having a unique pending task other than
the root one for me, so this works well for my scheduling patterns.

Since this won't work well for all scheduling patterns, it has to be
activated via a config parameter, worker-require-unique-ready as well as
enabling worker-keep-alive.
Merge pull request #461 from Houzz/unique_pending

Allows workers to keep alive only for unique pending tasks
Prefers tasks with more dependents in scheduling
Adds pyparsing to requirements.txt
Merge pull request #462 from Houzz/schedule_for_dependents

Prefers tasks with more dependents in scheduling
add support for spark 1.x
fix for python 2.6
include priority and resource information in visualizer
Fix deprecation warning when importing luigi.hdfs.

You will still see a deprecation warning when creating an
HdfsTarget if you haven't specified a config.
Merge pull request #485 from jcrobak/fix-luigi-hdfs-deprecation-warning

Fix deprecation warning when importing luigi.hdfs.
Merge pull request #483 from RUNDSP/spark-1x-job-pr

add support for spark 1.x
Merge pull request #484 from Houzz/viz

include priority and resource information in visualizer
This refactoring is functionally equivalent, but separates the state of the scheduler into its own class. The interface between the scheduler and the state is defined in a way so that it can be easily replaced with a SQL database or some other storage engine.
Adds sudo user to worker info when available

Many of the jobs I use have to be run as hadoop, so it's impossible to tell from
the user who actually ran the job. This adds sudo_user to the worker info so
that it's easier to tell who's running what. This is very handy when I need to
figure out why a large backfill or test job is running if it's causing a lot of
errors or hogging all the resources.
Merge pull request #487 from Houzz/sudo_user

Adds sudo user to worker info when available
Adds number of workers to worker_info
Adds a worker list view to the visualizer
Merge pull request #488 from Houzz/workers_count

Adds number of workers to worker_info
Merge pull request #446 from spotify/erikbern/scheduler-refactoring

Erikbern/scheduler refactoring
Merge pull request #489 from Houzz/worker_viz

Adds a worker list view to the visualizer