spotify/luigi — pull request #504 by danielfrg (2014-10-16 21:34:37+00:00)
Title: Refactor webhdfs.py

I refactored the whole file using the [hdfs](https://pypi.python.org/pypi/hdfs/0.2.11) library instead of whoops because apparently whoops has some problems when uploading files.

I also added a basic `WebHdfsTarget` that provides basic read and write operations. The `WebHdfsTarget` is based on the `S3Target`. A simple example would be:

```python
from luigi import webhdfs

target = webhdfs.WebHdfsTarget('TEST/file.py')

output = target.open('w')
output.write('pew pew pew\npow pow pow\n')
output.write('piw piw piw\npow pow pow')
output.close()

target = webhdfs.WebHdfsTarget('TEST/file.py')
input = target.open('r')
print input.read()
print [i for i in input]
input.close()
```

Output:
```
pew pew pew
pow pow pow
piw piw piw
pow pow pow
['pew pew pew', 'pow pow pow', 'piw piw piw', 'pow pow pow']
```

I haven't updated the tests because it needs a webhdfs endpoint to be available, instead maybe I could write some docs if you guys are interested in this.