Repository: pandas-dev/pandas

Issue:
Error reading parquet from s3 with s3fs >= 0.3.0
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

df = pd.read_parquet('s3://my-bucket/df.parquet')
```

Raises

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../pandas/io/parquet.py", line 294, in read_parquet
    return impl.read(path, columns=columns, **kwargs)
  File "/.../pandas/io/parquet.py", line 192, in read
    parquet_file = self.api.ParquetFile(path, open_with=s3.s3.open)
AttributeError: 'S3File' object has no attribute 's3'
```

#### Problem description

In version 0.3.0 s3fs removed the `S3File.s3` attribute. It is replaced by `S3File.fs` (which is inherited from `fsspec.AbstractBufferedFile.fs`.

Should pandas check the s3fs version and call the right attribute based on that?

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 18.6.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.17.0
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.2.1
setuptools       : 41.0.1
Cython           : None
pytest           : 4.4.1
hypothesis       : None
sphinx           : 2.1.2
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : 2.8.3 (dt dec pq3 ext lo64)
jinja2           : 2.10.1
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : 0.3.1
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : 0.3.1
scipy            : 1.3.0
sqlalchemy       : 1.3.5
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>



Discussion:
> Should pandas check the s3fs version and call the right attribute based on that?

Sure.

cc @martindurant for the (possibly unintentional) API change.
So the `open_with` in https://github.com/pandas-dev/pandas/blob/61362be9ea4d69b33ae421f1f98b8db50be611a2/pandas/io/parquet.py#L192 will need to depend on the version of s3fs.
Indeed this is an API change. However, I am surprised that anyone is opening a file and then using the FS methods of the attribute of that file - you presumably have the FS available directly anyway at this point. 

Indeed, rather than test specifically for s3 URLs, I would strongly encourage pandas to use fsspec directly, so that then you can read from any of the implementations supported by fsspec.
Perhaps there should be a function returning both the file and the filesystem, which can be used here instead of `get_filepath_or_buffer`. That would avoid `S3File.s3`/`S3File.fs`.

If that sounds like a reasonable direction I will work on a PR.
I'm not sure what's best.

On Mon, Aug 5, 2019 at 9:58 AM Chris Stadler <notifications@github.com>
wrote:

> Perhaps there should be a function returning both the file and the
> filesystem, which can be used here instead of get_filepath_or_buffer.
> That would avoid S3File.s3/S3File.fs.
>
> If that sounds like a reasonable direction I will work on a PR.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/27756?email_source=notifications&email_token=AAKAOIX27VNYLVWZZADDDFTQDA5Z3A5CNFSM4IJLDNJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3SCWKI#issuecomment-518269737>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIX6R6HBTG6K5TWDRYLQDA5Z3ANCNFSM4IJLDNJQ>
> .
>

Ran into this issue today; just made a local, hacky in-vivo fix to the API break. Happy to help in any way to fix the issue properly.

Cheers.
For the sake of compatibility, I can make S3File.s3 -> S3File.fs alias, if that makes life easier.

Resolving patch:
diff --git a/pandas/io/parquet.py b/pandas/io/parquet.py
--- a/pandas/io/parquet.py
+++ b/pandas/io/parquet.py
@@ -184,12 +184,14 @@ def write(
 
     def read(self, path, columns=None, **kwargs):
         if is_s3_url(path):
+            from pandas.io.s3 import get_file_and_filesystem
+
             # When path is s3:// an S3File is returned.
             # We need to retain the original path(str) while also
             # pass the S3File().open function to fsatparquet impl.
-            s3, _, _, should_close = get_filepath_or_buffer(path)
+            s3, filesystem = get_file_and_filesystem(path)
             try:
-                parquet_file = self.api.ParquetFile(path, open_with=s3.s3.open)
+                parquet_file = self.api.ParquetFile(path, open_with=filesystem.open)
             finally:
                 s3.close()
         else:
diff --git a/pandas/io/s3.py b/pandas/io/s3.py
--- a/pandas/io/s3.py
+++ b/pandas/io/s3.py
@@ -1,8 +1,11 @@
 """ s3 support for remote file interactivity """
+from typing import IO, Any, Optional, Tuple
 from urllib.parse import urlparse as parse_url
 
 from pandas.compat._optional import import_optional_dependency
 
+from pandas._typing import FilePathOrBuffer
+
 s3fs = import_optional_dependency(
     "s3fs", extra="The s3fs package is required to handle s3 files."
 )
@@ -14,9 +17,9 @@ def _strip_schema(url):
     return result.netloc + result.path
 
 
-def get_filepath_or_buffer(
-    filepath_or_buffer, encoding=None, compression=None, mode=None
-):
+def get_file_and_filesystem(
+    filepath_or_buffer: FilePathOrBuffer, mode: Optional[str] = None
+) -> Tuple[IO, Any]:
     from botocore.exceptions import NoCredentialsError
 
     if mode is None:
@@ -24,7 +27,7 @@ def get_filepath_or_buffer(
 
     fs = s3fs.S3FileSystem(anon=False)
     try:
-        filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
+        file = fs.open(_strip_schema(filepath_or_buffer), mode)
     except (FileNotFoundError, NoCredentialsError):
         # boto3 has troubles when trying to access a public file
         # when credentialed...
@@ -33,5 +36,15 @@ def get_filepath_or_buffer(
         # A NoCredentialsError is raised if you don't have creds
         # for that bucket.
         fs = s3fs.S3FileSystem(anon=True)
-        filepath_or_buffer = fs.open(_strip_schema(filepath_or_buffer), mode)
-    return filepath_or_buffer, None, compression, True
+        file = fs.open(_strip_schema(filepath_or_buffer), mode)
+    return file, fs
+
+
+def get_filepath_or_buffer(
+    filepath_or_buffer: FilePathOrBuffer,
+    encoding: Optional[str] = None,
+    compression: Optional[str] = None,
+    mode: Optional[str] = None,
+) -> Tuple[IO, Optional[str], Optional[str], bool]:
+    file, _fs = get_file_and_filesystem(filepath_or_buffer, mode=mode)
+    return file, None, compression, True


Test patch: