Add support for adding egg-info files to packages archive.

In order for packages that make use of entry points to work on Hadoop
node, the relevant egg-info directories also need to be copied to those
nodes. This change performs a simple check to see if a related egg-info
is present for a requested package, and adds those files as well.
Add test data for creating package archives with egg-info.
_hdfs_test: 2.6 backport for assertRegexpMatches
Fix deprecation warning

We should use raise_if_exists over fail_if_exists. I guess my clever and
tricky `deprecate_kwarg` method is a bit too meta. :)
_hdfs_test: Fix failing test case

I think this started to fail since the changes I introduced in
spotify/luigi#393.
hdfs: Add unix username to /tmp paths

The idea is that we would rather see paths like

        /tmp/<username>/blah/luigi-tempfile-1234

over just

        /tmp/blah/luigi-tempfile-1234

The (very) common issue that Spotify is experiencing is that users first
run their job as their own user to test their luigi job, then they use
the production user. The production user will then fail to run, because
the owner of the file will be the interactive user and not the
production user. It's a serious waste of time to chowning folders all
the time and encountering quite unnecessary permission errors.
Fixes resource tracking bug in scheduler
Merge pull request #458 from Houzz/update_resources

Fixes resource tracking bug in scheduler
More robust exists for hive.

Instead of trying to scrape the output for a "table not found" we can just look at the database and see if the table exists by querying for all the tables with the exact name.
Merge pull request #454 from edx/edx/brianhw/include-egginfo-in-packages

Edx/brianhw/include egginfo in packages
Merge pull request #456 from Tarrasch/fix-_hdfs_test

Fix hdfs tests
Merge pull request #455 from Tarrasch/tmp-paths-with-username

Hdfs /tmp paths with username
v1.0.19
Allows configuration of wait time for next task to finish
Merge pull request #433 from hwrdprkns/better_hive_exists

More robust exists for hive.
Merge pull request #460 from Houzz/worker_sleep

Allows configuration of wait between tasks finishing
refactor to make a reusable Register.get_task_cls
Merge pull request #463 from spotify/ulzha/range

refactor to make a reusable Register.get_task_cls
Add support for dynamic requiremnets

Tasks may now yield new requirements in run just as they are returned
in requires. If requirements are fulfilled when yielded the task will
continue, if not it will be rescheduled to be run when requirements are
finished. Note that this means that the task will be rerun from start.
Add dynamic requirement example