Takes number of workers into account in scheduler resource planning

I've noticed recently that sometimes I'll have a job with thousands of pending
tasks scheduled all using the same resource, but it's not running anything. All
of the tasks are ready and nothing else is using the resource.

It turned out that another worker had several tasks using the same resource
scheduled with higher priority. It only had one worker and wasn't going to get
to them anytime soon, but the greedy scheduling assigned all the resources to
this worker so it could sit for hours preparing to use them.

In order to alleviate this issue and allow resources to be used more freely,
this commit only allows workers to reserve resources when they are actually
about to use them.
Make streaming map-reduce use atomic move_dir

This is essentially using the code introduced in spotify/luigi#393,
solving the exact same issue described there.

Without this patch, the renaming is of LBYL nature, and you could end up
with moving a directory inside another instead of actually renaming a
directory.

So others don't fall into the same pitfall, I added a comment to the
rename and move function discouraging it to be used for directories.
Fix test_getpcmd, which occasionally fails

This have been pestering me and other luigi developers a lot. For
example, when you send a correct PR, it will sometimes fail on Travis
and you have to retrigger the bulid.

For example, it happened to me recently. This is what Travis prints:

    ======================================================================
    FAIL: test_getpcmd (lock_test.TestCmd)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/travis/build/spotify/luigi/test/lock_test.py", line 32, in test_getpcmd
        "sleep 1"
    AssertionError: '[sleep]' != 'sleep 1'

    ----------------------------------------------------------------------
Fix test and unveil horrible naming convention

This patch fixes test case

    test_run_real (test.hadoop_test.HadoopJobTest)

It's very very saddening to have to commit this. But at the moment I
just want the test case to pass.

Apperently "File" uses "move", "MockFile" uses "rename" and
"HdfsTarget" uses both "move" and "rename". Yuck!
Merge pull request #529 from Tarrasch/fix_getpcmd_test

Fix test_getpcmd, which occasionally fails
emit metrics about range completeness as events

A nice source of health information for a scheduled job. See RangeEvent documentation.
Merge pull request #513 from Houzz/greedy_num_workers

Takes number of workers into account in scheduler resource planning
Merge pull request #522 from Tarrasch/mapreduce_atomic_movedir

Make streaming map-reduce use atomic move_dir
Merge pull request #518 from spotify/ulzha/range

emit metrics about range completeness as events
Refactor webhdfs.py

I refactored the whole file using the hdfs library instead of whoops because apparently whoops has some problems when uploading files.

I also added a basic WebHdfsTarget that provides basic read and write operations. The WebHdfsTarget is based on the S3Target. A simple example would be:

from luigi import webhdfs

target = webhdfs.WebHdfsTarget('TEST/file.py')

output = target.open('w')
output.write('pew pew pew\npow pow pow\n')
output.write('piw piw piw\npow pow pow')
output.close()

target = webhdfs.WebHdfsTarget('TEST/file.py')
input = target.open('r')
print input.read()
print [i for i in input]
input.close()

Output:

pew pew pew
pow pow pow
piw piw piw
pow pow pow
['pew pew pew', 'pow pow pow', 'piw piw piw', 'pow pow pow']
Adding support for basic auth for task esindex.CopyToIndex & target
esindex.ElasticsearchTarget & test coverage

Allow Elasticsearch tests with HTTP basic auth.

Added test support for HTTP basic auth and factored out ES settings.

To use a non-standard host and port, use `ESINDEX_TEST_HOST`,
`ESINDEX_TEST_PORT` environment variables to override defaults.

To test HTTP basic authentication use `ESINDEX_TEST_HTTP_AUTH`.

Example running tests against port 9201 with basic auth:

    $ ESINDEX_TEST_PORT=9201 ESINDEX_TEST_HTTP_AUTH='admin:admin' nosetests test/_esindex_test.py
Add a lock for when the process forks

Deadlocks were occuring with DNS lookups because of the following case:

"The non-obvious problem in this approach is that at the moment of the
fork(2) call some threads may be in critical sections of code, doing
non-atomic operations protected by mutexes. In the child process the
threads just disappears and left data half-modified without any
possibility to "fix" them, there is no way to say what other threads
were doing and what should be done to make the data consistent.
Moreover: state of mutexes is undefined, they might be unusable and the
only way to use them in the child is to call pthread_mutex_init() to
reset them to a usable state. It's implementation dependent how mutexes
behave after fork(2) was called. On my Linux machine locked mutexes are
locked in the child."
[1] http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

This thread-fork issue can be replicated quickly by having a background
thread call getaddrinfo() repeatedly, and then have the main thread
start a child process which itself calls getaddrinfo().
Timing-dependent, the child process may hang in a locked state. Example
script can be found here:
https://gist.github.com/andpol/ca7432afc09b1bc914f5
Merge pull request #504 from danielfrg/patch-1

Refactor webhdfs.py
Annotating tests needing the minicluster
Merge pull request #535 from anyman/master

Annotating tests needing the minicluster
Add support for writing a string directly to s3
Updating debian/control with dependencies
S3Client put_string testcase added
Merge pull request #538 from anyman/master

Updating debian/control with dependencies
Merge pull request #532 from dvxam/dvxam/es-basic-auth-support

Adding support for basic auth for task esindex.CopyToIndex && target