Repository: pandas-dev/pandas (Python)

Issue:
BUG: pd.read_csv - encoding argument ignored for SpooledTemporaryFile
### 

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the [latest version](https://pandas.pydata.org/docs/whatsnew/index.html) of pandas.

- [ ] I have confirmed this bug exists on the master branch of pandas.


### Reproducible Example

```python
import pandas as pd
import tempfile
f = tempfile.SpooledTemporaryFile()
f.write('''\
beløp,valuta
10.0,BTC
5.0,ETH
'''.encode('latin-1'))
f.seek(0)
pd.read_csv(f, encoding='latin-1')
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 3: invalid start byte

# If I switch SpooledTemporaryFile with TemporaryFile, everything works
```


### Issue Description

When passing a SpooledTemporaryFile to pd.read_csv, the encoding-argument is ignored, and it seems like it tries to parse the bytes as utf-8. This will fail if the bytes contain non utf-8 characters and uses a different encoding.

In the example above the text contains a Norwegian letter in latin-1 encoding. Note that if I switch from using a SpooledTemporaryFile to a TemporaryFile, the problem goes away, and pd.read_csv uses the correct encoding.

### Expected Behavior

pd.read_csv should respect the encoding argument when receiving a SpooledTemporaryFile, and be able to correctly parse the example above without raising an exception.

### Installed Versions

<details>

INSTALLED VERSIONS
------------------
commit           : 73c68257545b5f8530b7044f56647bd2db92e2ba
python           : 3.8.12.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 20.6.0
Version          : Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : en_US.UTF-8

pandas           : 1.3.3
numpy            : 1.21.2
pytz             : 2021.3
dateutil         : 2.8.2
pip              : 21.1.3
setuptools       : 57.2.0.post0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 3.0.2
IPython          : 7.28.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.4.3
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : 1.7.1
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

</details>



Discussion:
Similar to https://github.com/pandas-dev/pandas/issues/43439

If you can open the `SpooledTemporaryFile` in text mode, it works:

```py
import tempfile

import pandas as pd

with tempfile.SpooledTemporaryFile(mode="w+t") as f:
    f.write("\nbeløp,valuta\n10.0,BTC\n5.0,ETH")
    f.seek(0)
    pd.read_csv(f, encoding="latin-1")
```
The reason we cannot open binary `tempfile.SpooledTemporaryFile` is because we need to wrap it inside `io.TextWrapperIO` which requires the attribute `readable` (and a  few more) which are not implemented by `tempfile.SpooledTemporaryFile`.

We could implement a simpler version of `io.TextWrapperIO` for just the functions we need. This would allow us to support binary `tempfile.SpooledTemporaryFile`. Our own implementation will most likely be slower than `io.TextWrapperIO`. This would affect to/read_csv/json when the user provides a binary handle and for any handle for the mentioned functions when the user requests compression. @jreback 
I think I found a simple way to support SpooledTemporaryFile (by lying that certain attributes exist and are true): https://github.com/pandas-dev/pandas/pull/44761 

@oskli



Patch:
diff --git a/doc/source/whatsnew/v1.4.0.rst b/doc/source/whatsnew/v1.4.0.rst
index 78031fd0456f7..7cf8c07683514 100644
--- a/doc/source/whatsnew/v1.4.0.rst
+++ b/doc/source/whatsnew/v1.4.0.rst
@@ -752,6 +752,7 @@ I/O
 - :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` with ``compression`` set to ``'zip'`` no longer create a zip file containing a file ending with ".zip". Instead, they try to infer the inner file name more smartly. (:issue:`39465`)
 - Bug in :func:`read_csv` when passing simultaneously a parser in ``date_parser`` and ``parse_dates=False``, the parsing was still called (:issue:`44366`)
 - Bug in :func:`read_csv` silently ignoring errors when failling to create a memory-mapped file (:issue:`44766`)
+- Bug in :func:`read_csv` when passing a ``tempfile.SpooledTemporaryFile`` opened in binary mode (:issue:`44748`)
 -
 
 Period
diff --git a/pandas/io/common.py b/pandas/io/common.py
index 22284e1225062..e12a7348b0075 100644
--- a/pandas/io/common.py
+++ b/pandas/io/common.py
@@ -6,7 +6,6 @@
 from collections import abc
 import dataclasses
 import gzip
-import io
 from io import (
     BufferedIOBase,
     BytesIO,
@@ -18,7 +17,6 @@
 import mmap
 import os
 from pathlib import Path
-import tempfile
 from typing import (
     IO,
     Any,
@@ -104,7 +102,7 @@ def close(self) -> None:
         avoid closing the potentially user-created buffer.
         """
         if self.is_wrapped:
-            assert isinstance(self.handle, (TextIOWrapper, BytesIOWrapper))
+            assert isinstance(self.handle, TextIOWrapper)
             self.handle.flush()
             self.handle.detach()
             self.created_handles.remove(self.handle)
@@ -779,20 +777,17 @@ def get_handle(
     # Convert BytesIO or file objects passed with an encoding
     is_wrapped = False
     if not is_text and ioargs.mode == "rb" and isinstance(handle, TextIOBase):
-        handle = BytesIOWrapper(
+        # not added to handles as it does not open/buffer resources
+        handle = _BytesIOWrapper(
             handle,
             encoding=ioargs.encoding,
         )
-        handles.append(handle)
-        # the (text) handle is always provided by the caller
-        # since get_handle would have opened it in binary mode
-        is_wrapped = True
     elif is_text and (compression or _is_binary_mode(handle, ioargs.mode)):
         handle = TextIOWrapper(
             # error: Argument 1 to "TextIOWrapper" has incompatible type
             # "Union[IO[bytes], IO[Any], RawIOBase, BufferedIOBase, TextIOBase, mmap]";
             # expected "IO[bytes]"
-            handle,  # type: ignore[arg-type]
+            _IOWrapper(handle),  # type: ignore[arg-type]
             encoding=ioargs.encoding,
             errors=errors,
             newline="",
@@ -935,7 +930,7 @@ def __init__(
         self.decode = decode
 
         self.attributes = {}
-        for attribute in ("seekable", "readable", "writeable"):
+        for attribute in ("seekable", "readable"):
             if not hasattr(f, attribute):
                 continue
             self.attributes[attribute] = getattr(f, attribute)()
@@ -976,11 +971,40 @@ def __next__(self) -> str:
         return newline.lstrip("\n")
 
 
-# Wrapper that wraps a StringIO buffer and reads bytes from it
-# Created for compat with pyarrow read_csv
-class BytesIOWrapper(io.BytesIO):
-    buffer: StringIO | TextIOBase | None
+class _IOWrapper:
+    # TextIOWrapper is overly strict: it request that the buffer has seekable, readable,
+    # and writable. If we have a read-only buffer, we shouldn't need writable and vice
+    # versa. Some buffers, are seek/read/writ-able but they do not have the "-able"
+    # methods, e.g., tempfile.SpooledTemporaryFile.
+    # If a buffer does not have the above "-able" methods, we simple assume they are
+    # seek/read/writ-able.
+    def __init__(self, buffer: BaseBuffer):
+        self.buffer = buffer
+
+    def __getattr__(self, name: str):
+        return getattr(self.buffer, name)
+
+    def readable(self) -> bool:
+        if hasattr(self.buffer, "readable"):
+            # error: "BaseBuffer" has no attribute "readable"
+            return self.buffer.readable()  # type: ignore[attr-defined]
+        return True
+
+    def seekable(self) -> bool:
+        if hasattr(self.buffer, "seekable"):
+            return self.buffer.seekable()
+        return True
+
+    def writable(self) -> bool:
+        if hasattr(self.buffer, "writable"):
+            # error: "BaseBuffer" has no attribute "writable"
+            return self.buffer.writable()  # type: ignore[attr-defined]
+        return True
 
+
+class _BytesIOWrapper:
+    # Wrapper that wraps a StringIO buffer and reads bytes from it
+    # Created for compat with pyarrow read_csv
     def __init__(self, buffer: StringIO | TextIOBase, encoding: str = "utf-8"):
         self.buffer = buffer
         self.encoding = encoding
@@ -1006,15 +1030,6 @@ def read(self, n: int | None = -1) -> bytes:
             self.overflow = combined_bytestring[n:]
             return to_return
 
-    def detach(self):
-        # Slightly modified from Python's TextIOWrapper detach method
-        if self.buffer is None:
-            raise ValueError("buffer is already detached")
-        self.flush()
-        buffer = self.buffer
-        self.buffer = None
-        return buffer
-
 
 def _maybe_memory_map(
     handle: str | BaseBuffer,
@@ -1042,10 +1057,15 @@ def _maybe_memory_map(
 
     # error: Argument 1 to "_MMapWrapper" has incompatible type "Union[IO[Any],
     # RawIOBase, BufferedIOBase, TextIOBase, mmap]"; expected "IO[Any]"
-    wrapped = cast(
-        BaseBuffer,
-        _MMapWrapper(handle, encoding, errors, decode),  # type: ignore[arg-type]
-    )
+    try:
+        wrapped = cast(
+            BaseBuffer,
+            _MMapWrapper(handle, encoding, errors, decode),  # type: ignore[arg-type]
+        )
+    finally:
+        for handle in reversed(handles):
+            # error: "BaseBuffer" has no attribute "close"
+            handle.close()  # type: ignore[attr-defined]
     handles.append(wrapped)
 
     return wrapped, memory_map, handles
@@ -1077,8 +1097,6 @@ def _is_binary_mode(handle: FilePath | BaseBuffer, mode: str) -> bool:
         codecs.StreamWriter,
         codecs.StreamReader,
         codecs.StreamReaderWriter,
-        # cannot be wrapped in TextIOWrapper GH43439
-        tempfile.SpooledTemporaryFile,
     )
     if issubclass(type(handle), text_classes):
         return False


Test patch:
diff --git a/pandas/tests/io/parser/test_encoding.py b/pandas/tests/io/parser/test_encoding.py
index 993f52b00334f..2b27332c7e85b 100644
--- a/pandas/tests/io/parser/test_encoding.py
+++ b/pandas/tests/io/parser/test_encoding.py
@@ -299,17 +299,16 @@ def test_readcsv_memmap_utf8(all_parsers):
     tm.assert_frame_equal(df, dfr)
 
 
-def test_not_readable(all_parsers, request):
+@pytest.mark.usefixtures("pyarrow_xfail")
+@pytest.mark.parametrize("mode", ["w+b", "w+t"])
+def test_not_readable(all_parsers, mode):
     # GH43439
     parser = all_parsers
-    if parser.engine in ("python", "pyarrow"):
-        mark = pytest.mark.xfail(
-            reason="SpooledTemporaryFile does only work with the c-engine"
-        )
-        request.node.add_marker(mark)
-
-    with tempfile.SpooledTemporaryFile() as handle:
-        handle.write(b"abcd")
+    content = b"abcd"
+    if "t" in mode:
+        content = "abcd"
+    with tempfile.SpooledTemporaryFile(mode=mode) as handle:
+        handle.write(content)
         handle.seek(0)
         df = parser.read_csv(handle)
     expected = DataFrame([], columns=["abcd"])
diff --git a/pandas/tests/io/test_common.py b/pandas/tests/io/test_common.py
index 94a1d614880e4..a00268d82a57d 100644
--- a/pandas/tests/io/test_common.py
+++ b/pandas/tests/io/test_common.py
@@ -607,4 +607,5 @@ def test_errno_attribute():
 
 def test_fail_mmap():
     with pytest.raises(UnsupportedOperation, match="fileno"):
-        icom.get_handle(BytesIO(), "rb", memory_map=True)
+        with BytesIO() as buffer:
+            icom.get_handle(buffer, "rb", memory_map=True)