Repository: pandas-dev/pandas

Issue:
is_file_like requirements are too strict for boto3 S3 objects
#### Code Sample, a copy-pastable example if possible

```python
my_s3_object = boto3.client('s3').get_object(Bucket=<S3 bucket>, Key=<S3 path>)
pd.read_csv(my_s3_object['Body'])
```
#### Problem description

When trying to read csv files from AWS S3 directly using boto3, the returned botocore.response. StreamingBody object does not have all the neccessary methods required in is_file_like function of  pandas.core.dtypes.inference package. Although the pd.read_csv call worked flawlessly before, since the commit 20 days ago ( https://github.com/pandas-dev/pandas/commit/e4e87ec55765d31e59e97d89c71ed5a3fa2f3d38 ) our app is broken.

See botocore reference ( http://botocore.readthedocs.io/en/latest/reference/response.html ) for details on the StreamingBody class.

The issue is caused by the too strict constraint in is_file_like function ( https://github.com/pandas-dev/pandas/blob/master/pandas/core/dtypes/inference.py#L140 )

#### Expected Output

No error should be raised if read method is available on read operations.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Darwin
OS-release: 16.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.0rc1
pytest: 3.0.7
pip: 9.0.1
setuptools: 35.0.1
Cython: None
numpy: 1.12.0
scipy: 0.19.0
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
cc @gfyoung  - do we ever use anything more than `read()` with a file-like object? 

xref #15894
is_file_like is correct
read is just not enough you need the ability to seek and tell as well
steaming does not meet this


As far as I saw the read_csv uses the input as a buffer and it is only using the read and the close methods. It would be nice to have a less strict is_readonly_buffer_like helper function for read validation (and maybe an is_writeonly_buffer_like helper function for write validation). 
Given this breakage, I think we can relax the requirements in two ways:

1) The object has EITHER `read` OR `write` and have the user deal with the consequences if the wrong attribute is available (e.g. we have read-only on a file to which we want to write).  The intent of the user with the file object does not impact whether the object is file-like or not.

2) Remove checks for `seek` and `tell`, as it does not seem like we need either of them:

~~~python
class BadStringIO(StringIO):
   def tell(self):
      raise NotImplementedError("nope")

   def seek(self, pos, whence=0):
      raise NotImplementedError("nope")

>>> data = "a\n1"
>>> read_csv(BadStringIO(data), engine="c")
   a
0  1
>>>
>>> read_csv(BadStringIO(data), engine="python")
   a
0  1
~~~
@gfyoung 

I guess we could change ``is_file_like`` to be (pseudo-code) ``is_iterator & hasattr(read/write)``
that seems to pass the mock tests (which is not an iterator anyhow)
@djlancelot can you provide a test based on one of our s3 buckets that replicates this

e.g. a streaming version of this
``df = read_csv('s3://pandas-test/tips.csv')``

Resolving patch:
diff --git a/pandas/core/dtypes/inference.py b/pandas/core/dtypes/inference.py
--- a/pandas/core/dtypes/inference.py
+++ b/pandas/core/dtypes/inference.py
@@ -142,12 +142,8 @@ def is_file_like(obj):
     Check if the object is a file-like object.
 
     For objects to be considered file-like, they must
-    be an iterator AND have the following four methods:
-
-    1) read
-    2) write
-    3) seek
-    4) tell
+    be an iterator AND have either a `read` and/or `write`
+    method as an attribute.
 
     Note: file-like objects must be iterable, but
     iterable objects need not be file-like.
@@ -172,11 +168,8 @@ def is_file_like(obj):
     False
     """
 
-    file_attrs = ('read', 'write', 'seek', 'tell')
-
-    for attr in file_attrs:
-        if not hasattr(obj, attr):
-            return False
+    if not (hasattr(obj, 'read') or hasattr(obj, 'write')):
+        return False
 
     if not is_iterator(obj):
         return False


Test patch:
diff --git a/pandas/tests/dtypes/test_inference.py b/pandas/tests/dtypes/test_inference.py
--- a/pandas/tests/dtypes/test_inference.py
+++ b/pandas/tests/dtypes/test_inference.py
@@ -100,11 +100,41 @@ def test_is_dict_like():
 
 
 def test_is_file_like():
+    class MockFile(object):
+        pass
+
     is_file = inference.is_file_like
 
     data = StringIO("data")
     assert is_file(data)
 
+    # No read / write attributes
+    # No iterator attributes
+    m = MockFile()
+    assert not is_file(m)
+
+    MockFile.write = lambda self: 0
+
+    # Write attribute but not an iterator
+    m = MockFile()
+    assert not is_file(m)
+
+    MockFile.__iter__ = lambda self: self
+    MockFile.__next__ = lambda self: 0
+    MockFile.next = MockFile.__next__
+
+    # Valid write-only file
+    m = MockFile()
+    assert is_file(m)
+
+    del MockFile.write
+    MockFile.read = lambda self: 0
+
+    # Valid read-only file
+    m = MockFile()
+    assert is_file(m)
+
+    # Iterator but no read / write attributes
     data = [1, 2, 3]
     assert not is_file(data)
 
diff --git a/pandas/tests/io/parser/common.py b/pandas/tests/io/parser/common.py
--- a/pandas/tests/io/parser/common.py
+++ b/pandas/tests/io/parser/common.py
@@ -1685,6 +1685,26 @@ class InvalidBuffer(object):
         with tm.assert_raises_regex(ValueError, msg):
             self.read_csv(InvalidBuffer())
 
+        # gh-16135: we want to ensure that "tell" and "seek"
+        # aren't actually being used when we call `read_csv`
+        #
+        # Thus, while the object may look "invalid" (these
+        # methods are attributes of the `StringIO` class),
+        # it is still a valid file-object for our purposes.
+        class NoSeekTellBuffer(StringIO):
+            def tell(self):
+                raise AttributeError("No tell method")
+
+            def seek(self, pos, whence=0):
+                raise AttributeError("No seek method")
+
+        data = "a\n1"
+
+        expected = pd.DataFrame({"a": [1]})
+        result = self.read_csv(NoSeekTellBuffer(data))
+
+        tm.assert_frame_equal(result, expected)
+
         if PY3:
             from unittest import mock
 
diff --git a/pandas/tests/io/parser/test_network.py b/pandas/tests/io/parser/test_network.py
--- a/pandas/tests/io/parser/test_network.py
+++ b/pandas/tests/io/parser/test_network.py
@@ -176,3 +176,22 @@ def test_s3_fails(self):
         # It's irrelevant here that this isn't actually a table.
         with pytest.raises(IOError):
             read_csv('s3://cant_get_it/')
+
+    @tm.network
+    def boto3_client_s3(self):
+        # see gh-16135
+
+        # boto3 is a dependency of s3fs
+        import boto3
+        client = boto3.client("s3")
+
+        key = "/tips.csv"
+        bucket = "pandas-test"
+        s3_object = client.get_object(Bucket=bucket, Key=key)
+
+        result = read_csv(s3_object["Body"])
+        assert isinstance(result, DataFrame)
+        assert not result.empty
+
+        expected = read_csv(tm.get_data_path('tips.csv'))
+        tm.assert_frame_equal(result, expected)