Repository: pandas-dev/pandas (Python)

Issue:
BUG: read_csv raising when null bytes are in skipped rows
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [x] (optional) I have confirmed this bug exists on the master branch of pandas.

---

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example
I have a few weird csv files with null bytes in the first row. Since 1.2.0 they are raising even if I skip the first row,

```python
pd.read_csv("test.csv", engine="python", skiprows=[0])
pd.read_csv("test.csv", engine="c", skiprows=[0])
```
Could not get this to fail with an example created in code.
#### Problem description


Both engines raising 
```
Traceback (most recent call last):
  File "/home/developer/.config/JetBrains/PyCharm2020.3/scratches/scratch_4.py", line 377, in <module>
    pd.read_csv("/media/sf_Austausch/test.csv", engine="c", skiprows=[0], nrows=2)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 605, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 457, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 814, in __init__
    self._engine = self._make_engine(self.engine)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 1045, in _make_engine
    return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 1894, in __init__
    self._reader = parsers.TextReader(self.handles.handle, **kwds)
  File "pandas/_libs/parsers.pyx", line 517, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas/_libs/parsers.pyx", line 619, in pandas._libs.parsers.TextReader._get_header
  File "pandas/_libs/parsers.pyx", line 813, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 1942, in pandas._libs.parsers.raise_parser_error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 8: invalid continuation byte

Process finished with exit code 1
```

This worked on 1.1.5.

Relevant changes are:

https://github.com/pandas-dev/pandas/blob/9b16b1e1ee049b042a7d59cddd8fbd913137223f/pandas/io/common.py#L556:L557
infers encoding now, which leads to

https://github.com/pandas-dev/pandas/blob/9b16b1e1ee049b042a7d59cddd8fbd913137223f/pandas/io/common.py#L643:L651
 where errors is always strict for read_csv. On 1.1.5 in case of no encoding given, ``errors`` was set to ``replace``, which caused this to work. Was this an intended change? Labeling as Regression for now.


#### Expected Output
```
   !   xyz  0  1
0  *  2000  1  2
1  *  2001  0  0
```

cc @gfyoung @twoertwein
This was caused by #36997

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : cd0224d26c6050f0e638861b9e557086673c14c1
python           : 3.8.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.4.0-58-generic
Version          : #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.3.0.dev0+356.ge0cb09e917
numpy            : 1.19.2
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.2.3
setuptools       : 49.6.0.post20201009
Cython           : 0.29.21
pytest           : 6.1.1
hypothesis       : 5.37.1
sphinx           : 3.2.1
blosc            : None
feather          : None
xlsxwriter       : 1.3.7
lxml.etree       : 4.5.2
html5lib         : 1.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.18.1
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : 1.3.2
fsspec           : 0.8.3
fastparquet      : 0.4.1
gcsfs            : 0.7.1
matplotlib       : 3.3.2
numexpr          : 2.7.1
odfpy            : None
openpyxl         : 3.0.5
pandas_gbq       : None
pyarrow          : 2.0.0
pyxlsb           : None
s3fs             : 0.4.2
scipy            : 1.5.2
sqlalchemy       : 1.3.20
tables           : 3.6.1
tabulate         : 0.8.7
xarray           : 0.16.1
xlrd             : 1.2.0
xlwt             : 1.3.0
numba            : 0.51.2

</details>



Discussion:
> where errors is always strict for read_csv. On 1.1.5 in case of no encoding given, errors was set to replace, which caused this to work. Was this an intended change?

Thank you @phofl  for already narrowing it down :) That wasn't an intentional change.

I'm happy to make a PR to restore the old behavior. Which behavior is preferred?
I would say errors=replace when encoding is set to utf-8? Would be inline with the previous behavior 
Agreed this is a regression if that's the case, and restoring behavior is the right call.
@phofl Could you please read the beginning of the CSV in binary mode, that would be helpful to make a testcase out of it. The following does (unfortunately) not trigger this bug:

```py
from io import BytesIO
import pandas as pd

buffer = BytesIO()
buffer.write(b'\0' + '\na\n1'.encode('utf-8'))
buffer.seek(0)
pd.read_csv(buffer, skiprows=[0])
```
```
buffer = io.BytesIO()
buffer.write(b' xyz gem\xe4\xdf Tabelle 4 \r\n')
buffer.seek(0)
pd.read_csv(buffer, skiprows=[0])
```

raises
perfect, thank you :)



Patch:
diff --git a/doc/source/whatsnew/v1.2.1.rst b/doc/source/whatsnew/v1.2.1.rst
index 5695c817b5a3a..baeca87b8c4f8 100644
--- a/doc/source/whatsnew/v1.2.1.rst
+++ b/doc/source/whatsnew/v1.2.1.rst
@@ -24,6 +24,7 @@ Fixed regressions
 - Fixed regression in :func:`read_excel` with non-rawbyte file handles (:issue:`38788`)
 - Bug in :meth:`read_csv` with ``float_precision="high"`` caused segfault or wrong parsing of long exponent strings. This resulted in a regression in some cases as the default for ``float_precision`` was changed in pandas 1.2.0 (:issue:`38753`)
 - Fixed regression in :meth:`Rolling.skew` and :meth:`Rolling.kurt` modifying the object inplace (:issue:`38908`)
+- Fixed regression in :meth:`read_csv` and other read functions were the encoding error policy (``errors``) did not default to ``"replace"`` when no encoding was specified (:issue:`38989`)
 
 .. ---------------------------------------------------------------------------
 
diff --git a/pandas/io/common.py b/pandas/io/common.py
index 642684ca61480..8f04724773a8a 100644
--- a/pandas/io/common.py
+++ b/pandas/io/common.py
@@ -553,8 +553,7 @@ def get_handle(
     Returns the dataclass IOHandles
     """
     # Windows does not default to utf-8. Set to utf-8 for a consistent behavior
-    if encoding is None:
-        encoding = "utf-8"
+    encoding_passed, encoding = encoding, encoding or "utf-8"
 
     # read_csv does not know whether the buffer is opened in binary/text mode
     if _is_binary_mode(path_or_buf, mode) and "b" not in mode:
@@ -641,6 +640,9 @@ def get_handle(
         # Check whether the filename is to be opened in binary mode.
         # Binary mode does not support 'encoding' and 'newline'.
         if ioargs.encoding and "b" not in ioargs.mode:
+            if errors is None and encoding_passed is None:
+                # ignore errors when no encoding is specified
+                errors = "replace"
             # Encoding
             handle = open(
                 handle,


Test patch:
diff --git a/pandas/tests/io/test_common.py b/pandas/tests/io/test_common.py
index d445bece593d1..725c14f410357 100644
--- a/pandas/tests/io/test_common.py
+++ b/pandas/tests/io/test_common.py
@@ -419,3 +419,11 @@ def test_is_fsspec_url():
     assert not icom.is_fsspec_url("random:pandas/somethingelse.com")
     assert not icom.is_fsspec_url("/local/path")
     assert not icom.is_fsspec_url("relative/local/path")
+
+
+def test_default_errors():
+    # GH 38989
+    with tm.ensure_clean() as path:
+        file = Path(path)
+        file.write_bytes(b"\xe4\na\n1")
+        tm.assert_frame_equal(pd.read_csv(file, skiprows=[0]), pd.DataFrame({"a": [1]}))