[ENH] Add read support for Google Cloud Storage
State: closed

- [x] closes #19454
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Couple of remaining issues:
- I had to do some weird logging magic to suppress the output of `gcsfs` when catching authentication exceptions
- Proper Parquet support seems to require more specialization so I didn't try
- I'm not sure how to properly test this functionality without a `pandas-test` bucket and/or a mock GCS library like `moto`

cc @martindurant who might have some thoughts on other things I am doing wrong

Labels:
name: Enhancement

Comments:
I'm also not sure what the all the various requirements files I touched are, I just mimicked the s3fs pattern; looks like Circle is unhappy with that, should I revert everything in `ci/` or is there some other step I'm missing?
Testing in gcsfs is done via the VCR package, which records all HTTP calls and their responses from the server, for playback during the running of a test. It is rather finicky/awkward to use.

Parquet is difficult because you need to supply the `open` function rather than file-like objects?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20729?src=pr&el=h1) Report
> Merging [#20729](https://codecov.io/gh/pandas-dev/pandas/pull/20729?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/1033e8b1195d4071253889ada60523832285354c?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `90%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/20729/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/20729?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #20729      +/-   ##
==========================================
- Coverage   91.91%    91.9%   -0.01%     
==========================================
  Files         153      154       +1     
  Lines       49532    49550      +18     
==========================================
+ Hits        45525    45541      +16     
- Misses       4007     4009       +2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.3% <90%> (-0.01%)` | :arrow_down: |
| #single | `41.86% <40%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/20729?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/\_print\_versions.py](https://codecov.io/gh/pandas-dev/pandas/pull/20729/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3ByaW50X3ZlcnNpb25zLnB5) | `15.71% <ø> (ø)` | :arrow_up: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/20729/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `92.47% <ø> (ø)` | :arrow_up: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/20729/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `71.2% <100%> (+0.92%)` | :arrow_up: |
| [pandas/io/gcs.py](https://codecov.io/gh/pandas-dev/pandas/pull/20729/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2djcy5weQ==) | `80% <80%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20729?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/20729?src=pr&el=footer). Last update [1033e8b...2745359](https://codecov.io/gh/pandas-dev/pandas/pull/20729?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

The issue with anonymous credentials in `gcsfs` appears to be resolved by https://github.com/dask/gcsfs/pull/100, so once that is released I think this PR should cover most use cases.

Any more thoughts re: testing? Would simply mocking out `gcsfs.GCSFileSystem.read` to return a simple CSV string be sufficient?
This was not implemented in Dask (yet), but the [MemoryFileSystem](https://github.com/dask/dask/pull/2741/files#diff-75106b81d12349bcb6c6cb9a4237b45c) might be a nice and simple mock for GCSFileSystem.
gcsfs v0.0.7 being released: https://github.com/conda-forge/gcsfs-feedstock/pull/7
Any more thoughts about testing? Would a limited set of tests based on `mock.patch`ing `GCSFileSystem.open` be sufficient for now?
I think we would need some tests, mocking would be ok
Hello @bnaul! Thanks for updating the PR.





















Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on June 25, 2018 at 23:43 Hours UTC
@TomAugspurger sorry for the delay, was away for a while; made the requested changes!
Changes look good, but the CI is failing on builds that don't have gcsfs.
Could you

1. Add a `td.skip_if_no('gcsfs')` decorator to the tests that use gcsfs
2. Add a test that is skipped if gcsfs *is* present, and checks that not
having gcsfs raises with the right error message.

On Wed, Jun 6, 2018 at 7:44 PM, Brett Naul <notifications@github.com> wrote:

> @TomAugspurger <https://github.com/TomAugspurger> sorry for the delay,
> was away for a while; made the requested changes!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20729#issuecomment-395257451>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIrs1pKdLMMEHHo_6qsHegV6FrlDyks5t6HdugaJpZM4TZS32>
> .
>

Still waiting on Appveyor but CircleCI and Travis are happy now
💚  @TomAugspurger 
I meant a fixture that provides the imported module. Like pytest imporotskip, but with the try / except for the two locations.

________________________________
From: Brett Naul <notifications@github.com>
Sent: Monday, June 25, 2018 11:40:55 AM
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] [ENH] Add read support for Google Cloud Storage (#20729)


@bnaul commented on this pull request.

________________________________

In pandas/tests/io/test_gcs.py<https://github.com/pandas-dev/pandas/pull/20729#discussion_r197863281>:

> +
+from pandas import DataFrame, read_csv
+from pandas.compat import BytesIO
+from pandas.io.common import is_gcs_url
+from pandas.util import _test_decorators as td
+
+
+def test_is_gcs_url():
+    assert is_gcs_url("gcs://pandas/somethingelse.com")
+    assert is_gcs_url("gs://pandas/somethingelse.com")
+    assert not is_gcs_url("s3://pandas/somethingelse.com")
+
+
+@td.skip_if_no('gcsfs')
+def test_read_csv_gcs():
+    try:


@TomAugspurger<https://github.com/TomAugspurger> do you mean a fixture that does use mock, but in another directory that won't get imported outside of testing? Or a fixture that doesn't use mock at all?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/pull/20729#discussion_r197863281>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIvOshqCcoEHlHlqX7eiP8SMY_LhIks5uARKXgaJpZM4TZS32>.

Made a pytest fixture; if it belongs in somewhere more generic let me know, the various testing utils seem spread out over quite a few different places and it's a bit confusing to me
Not quite what I had in mind. I'll push a change in a minute :)
Sounds good 🙃 
OK, pushed. Only real changes were

1. returning `mock` instead of `mock.patch`
2. moving to `conftest.py`, since it's a generally useful fixture
3. skipping if neither `unittest.mock` nor `mock` are available


Fixed the linting error I introduced.

May have a random failure on PY37, will investigate more later.
Oops might have overwritten that w/ the same change but I assume it's fine
@jreback looking 🍏 
Thanks @bnaul!