Merge pull request #392 from erikbern/worker-cleanup

Simplified some worker logic
Optional parameter 'parents' to HdfsClient.mkdir()

This patch should not change behavior, since the optional parameters
default is to what was used previously.

The name parents was used since it is what is used by the snakebite
client.
Make HdfsTarget.move_dir() atomic again

As was said in the comments of that function, it relied on mkdir failing
if the directory already exists. This was not explicit in the code
unfortunately and the underlying behavior of mkdir() changed since
move_dir() was written.
Fix imports in _hdfs_test

There is no such thing as unittes2 right? I mean the code always threw ...
Rename fail_if_exists to raise_if_exists

I'm not sold myself that adding the decorator is the cleanest way. Some
static analyzers will go wild, but you have to live with that when
using such dynamic programming languages.
Add default / configurable rpc connect timeout

Without this timeout, it was possible for rpc connections
(for example, reporting that a task was done) to hang forever
Merge pull request #398 from danielhfrank/rpc-connect-timeout

Add default / configurable rpc connect timeout
Merge pull request #397 from Tarrasch/raise-if-exists

Rename fail_if_exists to raise_if_exists
Merge pull request #394 from spotify/arash/clean-hdfs-test

Fix imports in _hdfs_test
Add raise_if_exists flag to hdfs_client.mkdir()

It is *way* to complicated to actually implement it. So I just went with
assertions to make sure that the programmers who want can explicitly
state their intent.  See spotify/luigi#393 for a discussion about this.

Oh, and I also started to use it at one place to get the ball rolling.
Push down parent and raise_if_exists flags to abc

From real life experience of how hdfs.mkdir() behavior changed due to
lack of explicitness, we think it's good for all FileSystem
implementations to encourage explicitness in their implementations of
mkdir().

Discussion in spotify/luigi#393
Sync mkdir raise_if_exists=False up to Snakebite

This will *slightly* break backwards compatibility. But the behavior was
really undefined before so.
Equality operator for Task
Merge pull request #400 from Houzz/eq_op

Adds equality operator to Task
Merge pull request #393 from Tarrasch/movedir-atomic

Make `move_dir()` atomic again
Create CascadingClient for fallbacking clients

The intended use case is that a company (like Spotify) is going to
switch over from a slow HdfsClient to a much faster SnakebiteHdfsClient.
To avoid surprises if say the SnakebiteHdfsClient can't handle some
corner case, then a given fallback will be used.

Example instantiation:

    client = CascadingClient([SnakebiteHdfsClient(), HdfsClient()])