Add auto-docs for new luigi.contrib packages.
Fix luigi.s3 and luigi.webhdfs default imports

Don't fail when importing luigi.s3 or luigi.webhdfs if deps
aren't satisified. This improves the ability to generate api
docs for these modules when required packages aren't installed,
and it is in line with the pattern used in luigi.{hdfs,hive,postgres}
and other places.
Merge pull request #427 from jcrobak/doc-fixes

Doc fixes
Fix deprecation warnings being emitted

~arash/luigi/parameter.py:168: DeprecationWarning: Use has_value rather than has_default. The meaning of "default" has changed
  warnings.warn('Use has_value rather than has_default. The meaning of "default" has changed', DeprecationWarning)

I verified that it works for me by running a normal Spotify luigi job.
Merge pull request #431 from Tarrasch/fix_has_default_deprecation

Fix deprecation warnings being emitted
Don't raise if parent dir exists when writing hdfs files
Don't import luigi.worker from __init__.py

There's some really fishy bug going on with multiprocessing that we are investigating. It *seems* like (but still to be confirmed) that luigi.hadoop ends up importing luigi.worker which in turn imports multiprocessing. multiprocessing *seems* to have a bug causing an exception in the global scope in case the temporary directory is too long (bind() fails if the temp path is > 107 characters, this is a known UNIX issue)

Exception from a Hadoop job:

  __import__('pkg_resources').declare_namespace(__name__)
Process SyncManager-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 517, in
_run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 136, in
__init__
    self.listener = Listener(address=address, backlog=5)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 106, in
__init__
    self._listener = SocketListener(address, family, backlog)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 227, in
__init__
    self._socket.bind(address)
  File "<string>", line 1, in bind
error: AF_UNIX path too long

Even if the theory above isn't correct, this still cleans up some import dependencies, by moving Event to its own module
Merge pull request #438 from spotify/erikbern/import-worker-event

Don't import luigi.worker from __init__.py
Make TaskProcess catch all Exceptions

"except Exception" will not catch all types of exceptions.
Also moved the put on queue to a finally block in case the error
handeling crashes for some reason
Make sure variables exists when put on queue
Merge pull request #439 from tommyengstrom/no-worker-response-bugfix

Make TaskProcess catch all Exceptions
Test for #439 – worker that catches a SystemExit exception
Fix deprecation warning

     INFO:subp.spluigi:/home/arash/spotify/repos/luigis/github_luigi/luigi/parameter.py:190: DeprecationWarning: Use value rather than default. The meaning of "default" has changed
     INFO:subp.spluigi:  warnings.warn('Use value rather than default. The meaning of "default" has changed', DeprecationWarning)

I verified this by running a normal Spotify luigi job.
Merge pull request #440 from spotify/erikbern/unittest-439

Test for #439 – worker that catches a SystemExit exception
Merge pull request #441 from Tarrasch/default_deprecation_warning

Fix deprecation warning
Catch dead child processes

See discussion in #440 and #434 - this might address some of the issues

There's still some issues where events aren't triggered for dead children, but that bug has always been there, so let's fix separately
Merge pull request #437 from freider/fix_open_mkdir

Don't raise if parent dir exists when writing hdfs files
Merge pull request #442 from spotify/erikbern/dead-children-processes

Catch dead child processes
Add support for dynamic requiremnets

Tasks may now yield new requirements in run just as they are returned
in requires. If requirements are fulfilled when yielded the task will
continue, if not it will be rescheduled to be run when requirements are
finished. Note that this means that the task will be rerun from start.
Add dynamic requirement example