Uses mkdir function in HdfsAtomicWritePipe init
use amazon ses for emailing
seperated notify commands for ses and email
renamed notify to send_email, removed globals()
Refactored luigi.worker a bit to use multiprocessing

...instead of os.fork and os.wait

This commit features the following stuff
- multiprocessing instead of os.fork
- Added a bunch of unit tests to run with multiple workers
- Fixed MockFile to support multiple processes
- Multiple processes are now supported even with a local scheduler

Other things I had to fix that was marginally related:
- Added --no-lock to a bunch of tests
- Fixed an issue in lock_test.py that triggered an error on my machine

I have no idea if/how this will work on non-Unix platforms (eg. Windows). On thing that should work (in theory) is to replace multiprocessing.Process with threading.Thread. I doubt that the previous code was working since it was relying directly on os.fork.
Merge pull request #407 from Houzz/simplify_atomic_mkdir

Uses mkdir function in HdfsAtomicWritePipe init
Merge pull request #419 from spotify/erikbern/worker-multiprocessing

Refactored luigi.worker a bit to use multiprocessing
Merge pull request #416 from tresata/notifications-boto

Allow use of Amazon SES for notification Emailing
Adding openssh-client as a dependency to make sure ssh tests work.

On build systems without the ssh executable some ssh tests would previously fail
Merge pull request #425 from freider/freider/ssh_dependency

Adding openssh-client as a dependency to make sure ssh tests work.
Add auto-docs for new luigi.contrib packages.
Fix luigi.s3 and luigi.webhdfs default imports

Don't fail when importing luigi.s3 or luigi.webhdfs if deps
aren't satisified. This improves the ability to generate api
docs for these modules when required packages aren't installed,
and it is in line with the pattern used in luigi.{hdfs,hive,postgres}
and other places.
Merge pull request #427 from jcrobak/doc-fixes

Doc fixes
Fix deprecation warnings being emitted

~arash/luigi/parameter.py:168: DeprecationWarning: Use has_value rather than has_default. The meaning of "default" has changed
  warnings.warn('Use has_value rather than has_default. The meaning of "default" has changed', DeprecationWarning)

I verified that it works for me by running a normal Spotify luigi job.
Merge pull request #431 from Tarrasch/fix_has_default_deprecation

Fix deprecation warnings being emitted
Don't import luigi.worker from __init__.py

There's some really fishy bug going on with multiprocessing that we are investigating. It *seems* like (but still to be confirmed) that luigi.hadoop ends up importing luigi.worker which in turn imports multiprocessing. multiprocessing *seems* to have a bug causing an exception in the global scope in case the temporary directory is too long (bind() fails if the temp path is > 107 characters, this is a known UNIX issue)

Exception from a Hadoop job:

  __import__('pkg_resources').declare_namespace(__name__)
Process SyncManager-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 517, in
_run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/lib/python2.6/multiprocessing/managers.py", line 136, in
__init__
    self.listener = Listener(address=address, backlog=5)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 106, in
__init__
    self._listener = SocketListener(address, family, backlog)
  File "/usr/lib/python2.6/multiprocessing/connection.py", line 227, in
__init__
    self._socket.bind(address)
  File "<string>", line 1, in bind
error: AF_UNIX path too long

Even if the theory above isn't correct, this still cleans up some import dependencies, by moving Event to its own module
Merge pull request #438 from spotify/erikbern/import-worker-event

Don't import luigi.worker from __init__.py
Make TaskProcess catch all Exceptions

"except Exception" will not catch all types of exceptions.
Also moved the put on queue to a finally block in case the error
handeling crashes for some reason
Make sure variables exists when put on queue
Merge pull request #439 from tommyengstrom/no-worker-response-bugfix

Make TaskProcess catch all Exceptions