Repository: modin-project/modin

Issue:
Encoding error : `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte`
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: ubuntu 18.04
- **Modin installed from : pip install modin[ray]
- **Modin version**: 0.6.3
- **Python version**: 3.7.3
<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
Hello, 
i'm trying to use modin to reduce the memory peak due the volum of the data, so i change the pandas with modin.pandas, i try to do a simple read of a file but encoded in 'latin-1' (french) . With pandas all goes smoothly but using modin i got an error of encoding as follow : 
`UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte`

the script used (which works fine on pandas but not in modin )  : 
`caract = pd.read_csv(path, sep="\t", encoding = "ISO-8859-1")`


ps ::  i tried other encoding and the same remark : works on pandas and not on modin (backed by ray) : ISO-8859-1, ISO-8859-9, latin-1

any solution ??

thanks 
### Source code / logs
`RayTaskError: ray_worker (pid=10815, host=ubuntu)
  File "pandas/_libs/parsers.pyx", line 1297, in pandas._libs.parsers.TextReader._string_convert
  File "pandas/_libs/parsers.pyx", line 1520, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte

During handling of the above exception, another exception occurred:

ray_worker (pid=10815, host=ubuntu)
  File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/modin/engines/ray/task_wrapper.py", line 8, in deploy_ray_func
    return func(**args)
  File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/modin/backends/pandas/parsers.py", line 69, in parse
    pandas_df = pandas.read_csv(BytesIO(to_read), **kwargs)
  File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 463, in _read
    data = parser.read(nrows)
  File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 1154, in read
    ret = self._engine.read(nrows)
  File "/home/lasngd/.conda/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py", line 2059, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 973, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 1105, in pandas._libs.parsers.TextReader._convert_column_data
  File "pandas/_libs/parsers.pyx", line 1158, in pandas._libs.parsers.TextReader._convert_tokens
  File "pandas/_libs/parsers.pyx", line 1281, in pandas._libs.parsers.TextReader._convert_with_dtype
  File "pandas/_libs/parsers.pyx", line 1297, in pandas._libs.parsers.TextReader._string_convert
  File "pandas/_libs/parsers.pyx", line 1520, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2: invalid continuation byte`
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
Hi @ghsama, thanks for posting!

Do you mind to share a couple of lines of synthetic or real data so I can reproduce this locally? Locally and in our test suite, we test with various forms of encoding, but I haven't been able to reproduce. If you can share some data that reproduces the issue for you, I will be able to diagnose and fix the issue. Thanks!
System information
OS Platform Windows 10 Home
**Modin installed from : pip install modin[dask]
Modin version: 0.6.3
Python version: 3.7.3

Hi @devin-petersohn,

I am facing a similar issue as @ghsama on windows with modin using dask engine. With vanilla pandas this works just fine:
`pd.read_csv('fires_50k.csv', encoding = "ISO-8859-1")`
However, while reading CSV with modin.pandas like this:
`mpd.read_csv('fires_50k.csv', encoding = "ISO-8859-1")`
throws this UnicodeDecodeError:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\pandas\io.py", line 97, in parser_func
    return _read(**kwargs)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\pandas\io.py", line 110, in _read
    pd_obj = BaseFactory.read_csv(**kwargs)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\data_management\factories.py", line 52, in read_csv
    return cls._determine_engine()._read_csv(**kwargs)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\data_management\factories.py", line 56, in _read_csv
    return cls.io_cls.read_csv(**kwargs)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\engines\base\io\text\csv_reader.py", line 197, in read
    row_lengths = cls.materialize(index_ids)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\engines\dask\task_wrapper.py", line 20, in materialize
    return client.gather(future)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\client.py", line 1876, in gather
    asynchronous=asynchronous,
  File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\client.py", line 771, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\utils.py", line 334, in sync
    raise exc.with_traceback(tb)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\utils.py", line 318, in f
    result[0] = yield future
  File "D:\Users\hvard\Anaconda3\lib\site-packages\tornado\gen.py", line 1133, in run
    value = future.result()
  File "D:\Users\hvard\Anaconda3\lib\site-packages\distributed\client.py", line 1732, in _gather
    raise exception.with_traceback(traceback)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\modin\backends\pandas\parsers.py", line 69, in parse
    pandas_df = pandas.read_csv(BytesIO(to_read), **kwargs)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 463, in _read
    data = parser.read(nrows)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1154, in read
    ret = self._engine.read(nrows)
  File "D:\Users\hvard\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 2059, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 973, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 1105, in pandas._libs.parsers.TextReader._convert_column_data
  File "pandas/_libs/parsers.pyx", line 1158, in pandas._libs.parsers.TextReader._convert_tokens
  File "pandas/_libs/parsers.pyx", line 1281, in pandas._libs.parsers.TextReader._convert_with_dtype
  File "pandas/_libs/parsers.pyx", line 1297, in pandas._libs.parsers.TextReader._string_convert
  File "pandas/_libs/parsers.pyx", line 1520, in pandas._libs.parsers._string_box_utf8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 8: invalid start byte

[This is the CSV](https://drive.google.com/open?id=1Kz5Su9EB8IHKg8eGAJs8dJD-olwj7NUJ) I am trying to read. This is a part(50K) of a large 1.88M rows dataset. I think you should be able to reproduce the issue with this data. Please do let me know otherwise.

Thanks!

Hi @devin-petersohn ,
here is a part of the csv file that causes the bug : [csvFile](https://drive.google.com/file/d/1eBRJ8HcwHY54fbXXDf6qVwM9c6sgFi_b/view)
Thanks
Thanks @ghsama and @hvardhan20! I was able to reproduce the issue with your files. 

In short, this line was the culprit and was excluding the csv reader from seeing the encoding: 

https://github.com/modin-project/modin/blob/master/modin/backends/pandas/parsers.py#L79

We should add more testing for a wider variety of encodings because this was missed by our tests. 

I will get this resolved, thanks so much for reporting!
MacOS Big Sur 11.2

python 3.8.6

ray = https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-1.2.0.dev0-cp38-cp38-macosx_10_13_x86_64.whl
modin from git = https://github.com/modin-project/modin

Got the same error with windows-1251 encoding. CSV file and traceback are attached.

[file.txt](https://github.com/modin-project/modin/files/5754208/file.txt)

Traceback (most recent call last):
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/bin/unity", line 5, in <module>
    app()
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/sector119/PycharmProjects/epsilon_unity/unity/lib/cmd/commands/db/commands/read.py", line 18, in read
    adapter.read(cache)
  File "/Users/sector119/PycharmProjects/epsilon/epsilon/lib/profiling.py", line 9, in timed
    output = func(*args, **kw)
  File "/Users/sector119/PycharmProjects/epsilon_unity/unity/lib/parser/adapter/plugins/base.py", line 87, in read
    self.dataframe = source_type.dataframe()
  File "/Users/sector119/PycharmProjects/epsilon_unity/unity/lib/parser/type/plugins/csv.py", line 10, in dataframe
    df = pd.read_csv(self.filepath,
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/pandas/io.py", line 115, in parser_func
    return _read(**kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/pandas/io.py", line 133, in _read
    pd_obj = EngineDispatcher.read_csv(**kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/data_management/factories/dispatcher.py", line 104, in read_csv
    return cls.__engine._read_csv(**kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/data_management/factories/factories.py", line 87, in _read_csv
    return cls.io_cls.read_csv(**kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/engines/base/io/file_dispatcher.py", line 29, in read
    query_compiler = cls._read(*args, **kwargs)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/modin/engines/base/io/text/csv_dispatcher.py", line 74, in _read
    empty_pd_df = pandas.read_csv(filepath_or_buffer, nrows=0)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 688, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 454, in _read
    parser = TextFileReader(fp_or_buf, **kwds)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 948, in __init__
    self._make_engine(self.engine)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 1180, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Users/sector119/.pyenv/versions/3.8.6/envs/unity-epsilon-3.8.6/lib/python3.8/site-packages/pandas/io/parsers.py", line 2010, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 537, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas/_libs/parsers.pyx", line 740, in pandas._libs.parsers.TextReader._get_header
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcc in position 32: invalid continuation byte

сс @anmyachev 
@sector119 thanks for posting. I was able to reproduce the error, but only with using `names` parameter in `read_csv` call.

Can you share your `read_csv` call? Maybe there are another details.
`df = pd.read_csv(
      filepath,
      encoding='windows-1251',
      names=['field', 'names', 'here'],
      sep=';',
      skiprows=0,
      na_values='\\N',
      engine='c'                         
)`

Thank You

Resolving patch:
diff --git a/modin/engines/base/io/text/csv_dispatcher.py b/modin/engines/base/io/text/csv_dispatcher.py
--- a/modin/engines/base/io/text/csv_dispatcher.py
+++ b/modin/engines/base/io/text/csv_dispatcher.py
@@ -59,6 +59,7 @@ def _read(cls, filepath_or_buffer, **kwargs):
         names = kwargs.get("names", None)
         index_col = kwargs.get("index_col", None)
         usecols = kwargs.get("usecols", None)
+        encoding = kwargs.get("encoding", None)
         if names is None:
             # For the sake of the empty df, we assume no `index_col` to get the correct
             # column names before we build the index. Because we pass `names` in, this
@@ -71,7 +72,9 @@ def _read(cls, filepath_or_buffer, **kwargs):
         elif index_col is None and not usecols:
             # When names is set to some list that is smaller than the number of columns
             # in the file, the first columns are built as a hierarchical index.
-            empty_pd_df = pandas.read_csv(filepath_or_buffer, nrows=0)
+            empty_pd_df = pandas.read_csv(
+                filepath_or_buffer, nrows=0, encoding=encoding
+            )
             num_cols = len(empty_pd_df.columns)
             if num_cols > len(names):
                 index_col = list(range(num_cols - len(names)))


Test patch:
diff --git a/modin/pandas/test/data/issue_976.csv b/modin/pandas/test/data/issue_976.csv
new file mode 100644
--- /dev/null
+++ b/modin/pandas/test/data/issue_976.csv
@@ -0,0 +1,5 @@
+1;11800000560005;11800000560005;������� ����� ����������;;-;���. ����;�. �������i���;������������� �����;����������� �������;105.6000
+1;10200007400477;10200007400477;�������� ����� ����������;;-;���. ������;����������;³����������� �����;����������� �������;696.6400
+1;11100008540930;11100008540930;���������� ������� ��������;2;9;���. ������;�.�������;����������� �����;����������� �������;124.4800
+1;12300000051493;12300000051493;���������� ����� ����������;;50;���. ����������;��.���������;���'�����-���������� �����;����������� �������;-0.4700
+1;12300000117460;12300000117460;����� ³���� ���������;;60;���. ���������;������;���'�����-���������� �����;����������� �������;221.0400
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -818,6 +818,24 @@ def test_read_csv_iteration(self, iterator):
 
         df_equals(modin_df, pd_df)
 
+    def test_read_csv_encoding_976(self):
+        file_name = "modin/pandas/test/data/issue_976.csv"
+        names = [str(i) for i in range(11)]
+
+        kwargs = {
+            "sep": ";",
+            "names": names,
+            "encoding": "windows-1251",
+        }
+        df1 = pd.read_csv(file_name, **kwargs)
+        df2 = pandas.read_csv(file_name, **kwargs)
+        # these columns contain data of various types in partitions
+        # see #1931 for details;
+        df1 = df1.drop(["4", "5"], axis=1)
+        df2 = df2.drop(["4", "5"], axis=1)
+
+        df_equals(df1, df2)
+
     # Quoting, Compression, and File Format parameters tests
     @pytest.mark.parametrize("compression", ["infer", "gzip", "bz2", "xz", "zip"])
     @pytest.mark.parametrize(