[WIP]REGR: Fixed reading from public S3 buckets with credentials
State: closed

Closes https://github.com/pandas-dev/pandas/issues/34626

This works in 1.0.4 I think, so no whatsnew.

I'd like to wait for https://github.com/pandas-dev/pandas/pull/34266 to get in first. I'll need to update things.

And in the future, I'd like to deprecate this behavior in favor of something like

```python
pd.read_csv("s3://path/to/key", storage_options={"anon": True})
```

It should be easy to detect the cases when we need to do that. We just don't have the `storage_options` kwarg yet.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Ahh just realised we are looking at the same thing @TomAugspurger . I think the case  #34626 is trying to address is reading without AWS Credentials right? I've put a test together for this: https://github.com/pandas-dev/pandas/pull/34877
The credentials @TomAugspurger is adding here are dummy (non-working) credentials to avoid a moto failure. But: since this is reading from an actual s3, and nothing is being mocked, moto should not be involved here?
> The credentials @TomAugspurger is adding here are dummy (non-working) credentials to avoid a moto failure. But: since this is reading from an actual s3, and nothing is being mocked, moto should not be involved here?

Agree - think it should be as simple as: https://github.com/pandas-dev/pandas/pull/34877/files#diff-681243e864617f600083a270b329d17dR36 - unless i'm missing something?
I thought the issue was using credentials that are invalid for a public bucket. I don’t know what s3fs’ default behavior is when no credentials are found. 

> On Jun 20, 2020, at 04:29, Ali McMaster <notifications@github.com> wrote:
> 
> ﻿
> The credentials @TomAugspurger is adding here are dummy (non-working) credentials to avoid a moto failure. But: since this is reading from an actual s3, and nothing is being mocked, moto should not be involved here?
> 
> Agree - think it should be as simple as: https://github.com/pandas-dev/pandas/pull/34877/files#diff-681243e864617f600083a270b329d17dR36 - unless i'm missing something?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@TomAugspurger - my understanding was:

```
import pandas as pd
res = pd.read_csv("s3://gdelt-open-data/events/1981.csv", nrows=3)
```
1) **With AWS Credentials in my Env:**
pandas 1.0.3 & 1.0.4 -> Works

2) **Without AWS Credentials in Env**
1.0.3 - Works
1.0.4 - NoCredentialsError: Unable to locate credentials

I think (2) was the regression in #34626 - hence my PR [here](https://github.com/pandas-dev/pandas/pull/34877)

But also think we should merge the case you are testing here ~ Dummy AWS Creds since maybe that tests (1) here?
Sounds good. Can you add the tests from my PR and I'll just close mine?

Regardless, I think we should wait till https://github.com/pandas-dev/pandas/pull/34266 is in.
> Sounds good. Can you add the tests from my PR and I'll just close mine?
> 
> Regardless, I think we should wait till #34266 is in.

Sure and apologies for the mix up. I've added this test to the PR here: https://github.com/pandas-dev/pandas/pull/34877

Agree re waiting for fsspec. 
Closing in favor of #34877.