Don't raise if parent dir exists when writing hdfs files
Don't import luigi.worker from __init__.py

There's some really fishy bug going on with multiprocessing that we are investigating. It *seems* like (but still to be confirmed) that luigi.hadoop ends up importing luigi.worker which in turn imports multiprocessing. multiprocessing *seems* to have a bug causing an exception in the global scope in case the temporary directory is too long (bind() fails if the temp path is > 107 characters, this is a known UNIX issue)

Exception from a Hadoop job:

  __import__('pkg_resources').declare_namespace(__name__)
Process SyncManager-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 517, in
_run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 136, in
__init__
    self.listener = Listener(address=address, backlog=5)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 106, in
__init__
    self._listener = SocketListener(address, family, backlog)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 227, in
__init__
    self._socket.bind(address)
  File "<string>", line 1, in bind
error: AF_UNIX path too long

Even if the theory above isn't correct, this still cleans up some import dependencies, by moving Event to its own module
Merge pull request #438 from spotify/erikbern/import-worker-event

Don't import luigi.worker from __init__.py
Make TaskProcess catch all Exceptions

"except Exception" will not catch all types of exceptions.
Also moved the put on queue to a finally block in case the error
handeling crashes for some reason
Make sure variables exists when put on queue
Merge pull request #439 from tommyengstrom/no-worker-response-bugfix

Make TaskProcess catch all Exceptions
Test for #439 – worker that catches a SystemExit exception
Fix deprecation warning

     INFO:subp.spluigi:/home/arash/spotify/repos/luigis/github_luigi/luigi/parameter.py:190: DeprecationWarning: Use value rather than default. The meaning of "default" has changed
     INFO:subp.spluigi:  warnings.warn('Use value rather than default. The meaning of "default" has changed', DeprecationWarning)

I verified this by running a normal Spotify luigi job.
Merge pull request #440 from spotify/erikbern/unittest-439

Test for #439 – worker that catches a SystemExit exception
Merge pull request #441 from Tarrasch/default_deprecation_warning

Fix deprecation warning
Catch dead child processes

See discussion in #440 and #434 - this might address some of the issues

There's still some issues where events aren't triggered for dead children, but that bug has always been there, so let's fix separately
Merge pull request #437 from freider/fix_open_mkdir

Don't raise if parent dir exists when writing hdfs files
Merge pull request #442 from spotify/erikbern/dead-children-processes

Catch dead child processes
Updates test/requirements.txt to allow running of all unit tests
Merge pull request #447 from Houzz/update_requirements

Updates test/requirements.txt to allow running of all unit tests
Allows sending of html e-mail
Merge pull request #448 from Houzz/html_email

Allows sending of html e-mail
A better check to see if the hive table exists.

Instead of trying to scrape the output for a "table not found" we can just look at the database and see if the table exists by querying for all the tables with the exact name.