Add HDFS reading
State: closed

 - Add HDFS reading using `libhdfs3`

- [x] closes #18199
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
@AbdealiJK I cherry-picked your first commit about the parquet error message (as this was an easy change) and merged that in a separate PR (https://github.com/pandas-dev/pandas/pull/18717).

Can you update the hdfs part according to the comments?
Thanks Jeff. Modifying this is on my ToDo - a bit pre-occupied at the
moment.  Will do it soon.

On Dec 11, 2017 5:19 AM, "Joris Van den Bossche" <notifications@github.com>
wrote:

> @AbdealiJK <https://github.com/abdealijk> I cherry-picked your first
> commit (as this was an easy change) and merged that in a separate PR (
> #18717 <https://github.com/pandas-dev/pandas/pull/18717>).
>
> Can you update the hdfs part according to the comments?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/18568#issuecomment-350592114>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACGUp9qUKaEqZNXVDuVXIqtlG8nO2TAPks5s_G3-gaJpZM4QvcDi>
> .
>

Hello @AbdealiJK! Thanks for updating the PR.



 - In the file [`pandas/io/common.py`](https://github.com/pandas-dev/pandas/blob/a55e848f33ec7c738a3d4fd3f48dea53e9d8bed1/pandas/io/common.py), following are the PEP8 issues :

> [Line 106:5](https://github.com/pandas-dev/pandas/blob/a55e848f33ec7c738a3d4fd3f48dea53e9d8bed1/pandas/io/common.py#L106): [E722](https://duckduckgo.com/?q=pep8%20E722) do not use bare except'

 - In the file [`pandas/io/hdfs.py`](https://github.com/pandas-dev/pandas/blob/a55e848f33ec7c738a3d4fd3f48dea53e9d8bed1/pandas/io/hdfs.py), following are the PEP8 issues :

> [Line 5:1](https://github.com/pandas-dev/pandas/blob/a55e848f33ec7c738a3d4fd3f48dea53e9d8bed1/pandas/io/hdfs.py#L5): [E722](https://duckduckgo.com/?q=pep8%20E722) do not use bare except'






For the unit test: I need some help in setting up HDFS inside Travis.
The way other orgs are doing this is by using their own dockers: [Example of hdfs3's CI](https://github.com/dask/hdfs3/blob/master/continuous_integration/Dockerfile)

Pandas does not seem to have it's own docker ... so a little confused about this.

Manually I've tested the following:
 - If hdfs3 is not installed, Throws:
```
ImportError: The hdfs3 library is required to handle hdfs files
```
 - If hdfs3 is installed but libhdfs3 is not installed, Throws:
```
ImportError: Can not find the shared library: libhdfs3.so
See installation instructions at http://hdfs3.readthedocs.io/en/latest/install.html
```
 - If hdfs3 is installed it works for the code: pd.read_csv("hdfs://localhost:9000/tmp/a.csv")
 - If hdfs3 is installed it works if export HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.0/libexec/etc/hadoop/ is done and pd.read_csv("hdfs:///tmp/a.csv") is provided


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18568?src=pr&el=h1) Report
> Merging [#18568](https://codecov.io/gh/pandas-dev/pandas/pull/18568?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c28b6249471ffddfed51d62437a65ca97766fb7f?src=pr&el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `41.02%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18568/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18568?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18568      +/-   ##
==========================================
- Coverage   91.64%   91.58%   -0.06%     
==========================================
  Files         154      155       +1     
  Lines       51428    51459      +31     
==========================================
+ Hits        47129    47131       +2     
- Misses       4299     4328      +29
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.45% <41.02%> (-0.04%)` | :arrow_down: |
| #single | `40.82% <23.07%> (-0.13%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18568?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/hdfs.py](https://codecov.io/gh/pandas-dev/pandas/pull/18568/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2hkZnMucHk=) | `0% <0%> (ø)` | |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/18568/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `88.23% <100%> (+3.23%)` | :arrow_up: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/18568/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `70.11% <70.58%> (+0.62%)` | :arrow_up: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/18568/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `58.69% <75%> (-0.08%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18568/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18568/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.68% <0%> (-0.11%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18568?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18568?src=pr&el=footer). Last update [c28b624...a55e848](https://codecov.io/gh/pandas-dev/pandas/pull/18568?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

if you can rebase. this would require a testing setup similar to this:

https://github.com/dask/dask/tree/master/continuous_integration/hdfs


closing as stale. nice idea, but we have to be able to automatically test this.