Optional parameter 'parents' to HdfsClient.mkdir()

This patch should not change behavior, since the optional parameters
default is to what was used previously.

The name parents was used since it is what is used by the snakebite
client.
Make HdfsTarget.move_dir() atomic again

As was said in the comments of that function, it relied on mkdir failing
if the directory already exists. This was not explicit in the code
unfortunately and the underlying behavior of mkdir() changed since
move_dir() was written.
Fix imports in _hdfs_test

There is no such thing as unittes2 right? I mean the code always threw ...
Rename fail_if_exists to raise_if_exists

I'm not sold myself that adding the decorator is the cleanest way. Some
static analyzers will go wild, but you have to live with that when
using such dynamic programming languages.
Add default / configurable rpc connect timeout

Without this timeout, it was possible for rpc connections
(for example, reporting that a task was done) to hang forever
Gracefully close worker on term.

This change (attempts to) add a signal handler to the worker allowing it
to gracefully shutdown on a TERM signal. This means that it will finish
the tasks that it has accepted but not request any more work.
Merge pull request #398 from danielhfrank/rpc-connect-timeout

Add default / configurable rpc connect timeout
Merge pull request #397 from Tarrasch/raise-if-exists

Rename fail_if_exists to raise_if_exists
Merge pull request #394 from spotify/arash/clean-hdfs-test

Fix imports in _hdfs_test
Add raise_if_exists flag to hdfs_client.mkdir()

It is *way* to complicated to actually implement it. So I just went with
assertions to make sure that the programmers who want can explicitly
state their intent.  See spotify/luigi#393 for a discussion about this.

Oh, and I also started to use it at one place to get the ball rolling.
Push down parent and raise_if_exists flags to abc

From real life experience of how hdfs.mkdir() behavior changed due to
lack of explicitness, we think it's good for all FileSystem
implementations to encourage explicitness in their implementations of
mkdir().

Discussion in spotify/luigi#393
Sync mkdir raise_if_exists=False up to Snakebite

This will *slightly* break backwards compatibility. But the behavior was
really undefined before so.
Equality operator for Task
Merge pull request #400 from Houzz/eq_op

Adds equality operator to Task
Make S3Target respect format
Merge pull request #393 from Tarrasch/movedir-atomic

Make `move_dir()` atomic again
Create CascadingClient for fallbacking clients

The intended use case is that a company (like Spotify) is going to
switch over from a slow HdfsClient to a much faster SnakebiteHdfsClient.
To avoid surprises if say the SnakebiteHdfsClient can't handle some
corner case, then a given fallback will be used.

Example instantiation:

    client = CascadingClient([SnakebiteHdfsClient(), HdfsClient()])
Merge pull request #395 from Tarrasch/chained-hdfs-client

Chainable hdfs-clients
Overhaul the hdfs client configuration

So the old syntax was something very hackish and very special-case oriented.

    [hdfs]
    use_snakebite: true

Now it becomes:

    [hdfs]
    client: snakebite

This patch also adds the configuration:

    [hdfs]
    client: snakebite_with_hadoopcli_fallback

This patch maintains backwards compatibility with old configuration.

This patch depend on spotify/luigi#395, so that must be merged before
this one.
Merge pull request #403 from Tarrasch/hdfs-client-configuration

Overhaul the hdfs client configuration