Repository: modin-project/modin

Issue:
read_csv difference to pandas
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution**: Ubuntu 18.04
- **Modin installed from (source or binary)**: pip
- **Modin version**: 0.6.3
- **Python version**: 3.7.3
- **Exact command to reproduce**: pd.read_csv(...)


<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
```python
import pandas as pd
pd.read_csv(...)
```
The above works without a problem. But if I do the same with `import modin.pandas as pd`, I get an error.

This is apparently a problem in vanilla pandas too, if you set the parsing engine to C. But the standard pandas parsing engine is python, which gives a different result from standard modin.

### Source code / logs
I get the folowing error with my code if done under `import modin.pandas as pd`

```
pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at row 3677254
```

I can't post a raw stack trace because of company policy, but if it is really needed i can clean it up and post it.


Discussion:
Hi @kaiogu, thanks for posting!

Are you able to provide the data types of the columns (column names not needed)? I suspect there's an issue with how we split up the parsing.

If I can get the `dtypes` I should be able to get a better idea. In order to fix this, I need to be able to reproduce it.
Hi @devin-petersohn ,

If I get the dtypes after loading the data in vanilla pandas, I get:
```python
>>> print(data.dtypes.unique())
[dtype('O') dtype('float64') dtype('int64')]
```

I should mention that I also pass dtype and usecols extra arguments to `load_csv`:

```python
usecols = ['str_field_1',
           'float_field_1',
           'int_field_1']

dtype = {'str_field_1':object'
         'float_field_1':float,
         'int_field_1':int}

data = pd.read_csv("path/to/my/file.csv",
                   sep=';',
                   decimal=',',
                   usecols=usecols,
                   dtype=dtype)
```

Also, I get a warning about not having `psutil` when importing `modin.pd`:
```
WARNING: Not monitoring node memory since `psutil` is not installed. Install this with `pip install psutil` (or ray[debug]) to enable debugging of memory-related crashes.
```

Hope this, helps!
Thanks, this does help. The `psutil` warning is because it is an optional dependency of Ray, so generally that can be ignored.

With your information I was able to reproduce it, I will paste the logs and attach the file for our tracking.

I see the issue, and I will dig into this. It will be fixed for the next release, which is going out in the next day or so.

```
In [3]: df = pd.read_csv("newlines.csv", dtype={"col1":object, "col2":int, "col3":int, "col4":int})
                                                                                                     
---------------------------------------------------------------------------
RayTaskError(ParserError)                 Traceback (most recent call last)
<ipython-input-3-0e32a863d64d> in <module>
----> 1 df = pd.read_csv("newlines.csv")

~/software_builds/modin/modin/pandas/io.py in parser_func(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)
     95         if not kwargs.get("sep", sep):
     96             kwargs["sep"] = "\t"
---> 97         return _read(**kwargs)
     98 
     99     return parser_func

~/software_builds/modin/modin/pandas/io.py in _read(**kwargs)
    108         kwargs: Keyword arguments in pandas.read_csv
    109     """
--> 110     pd_obj = BaseFactory.read_csv(**kwargs)
    111     # This happens when `read_csv` returns a TextFileReader object for iterating through
    112     if isinstance(pd_obj, pandas.io.parsers.TextFileReader):

~/software_builds/modin/modin/data_management/factories.py in read_csv(cls, **kwargs)
     50     @classmethod
     51     def read_csv(cls, **kwargs):
---> 52         return cls._determine_engine()._read_csv(**kwargs)
     53 
     54     @classmethod

~/software_builds/modin/modin/data_management/factories.py in _read_csv(cls, **kwargs)
     54     @classmethod
     55     def _read_csv(cls, **kwargs):
---> 56         return cls.io_cls.read_csv(**kwargs)
     57 
     58     @classmethod

~/software_builds/modin/modin/engines/base/io/text/csv_reader.py in read(cls, filepath_or_buffer, **kwargs)
    162         # or based on the column(s) that were requested.
    163         if index_col is None:
--> 164             row_lengths = cls.materialize(index_ids)
    165             new_index = pandas.RangeIndex(sum(row_lengths))
    166             # pandas has a really weird edge case here.

~/software_builds/modin/modin/engines/ray/task_wrapper.py in materialize(cls, obj_id)
     18     @classmethod
     19     def materialize(cls, obj_id):
---> 20         return ray.get(obj_id)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ray/worker.py in get(object_ids, timeout)
   1455                     worker.core_worker.dump_object_store_memory_usage()
   1456                 if isinstance(value, RayTaskError):
-> 1457                     raise value.as_instanceof_cause()
   1458                 else:
   1459                     raise value

RayTaskError(ParserError): ray::modin.engines.ray.task_wrapper.deploy_ray_func() (pid=46436, ip=192.168.42.11)
  File "python/ray/_raylet.pyx", line 636, in ray._raylet.execute_task
  File "/Users/DevinPetersohn/software_builds/modin/modin/engines/ray/task_wrapper.py", line 8, in deploy_ray_func
    return func(**args)
  File "/Users/DevinPetersohn/software_builds/modin/modin/backends/pandas/parsers.py", line 86, in parse
    pandas_df = pandas.read_csv(BytesIO(to_read), **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 457, in _read
    parser = TextFileReader(fp_or_buf, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 895, in __init__
    self._make_engine(self.engine)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 1135, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 1917, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 542, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas/_libs/parsers.pyx", line 822, in pandas._libs.parsers.TextReader._get_header
  File "pandas/_libs/parsers.pyx", line 937, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2132, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at row 0
```

[newlines.txt](https://github.com/modin-project/modin/files/4087091/newlines.txt)

Resolving patch:
diff --git a/modin/engines/base/io/text/csv_reader.py b/modin/engines/base/io/text/csv_reader.py
--- a/modin/engines/base/io/text/csv_reader.py
+++ b/modin/engines/base/io/text/csv_reader.py
@@ -79,7 +79,6 @@ def read(cls, filepath_or_buffer, **kwargs):
         column_names = empty_pd_df.columns
         skipfooter = kwargs.get("skipfooter", None)
         skiprows = kwargs.pop("skiprows", None)
-
         usecols = kwargs.get("usecols", None)
         usecols_md = _validate_usecols_arg(usecols)
         if usecols is not None and usecols_md[1] != "integer":
@@ -99,6 +98,10 @@ def read(cls, filepath_or_buffer, **kwargs):
             parse_dates=parse_dates,
             usecols=usecols,
         )
+        encoding = kwargs.get("encoding", None)
+        quotechar = kwargs.get("quotechar", '"').encode(
+            encoding if encoding is not None else "UTF-8"
+        )
         with cls.file_open(filepath_or_buffer, "rb", compression_type) as f:
             # Skip the header since we already have the header information and skip the
             # rows we are told to skip.
@@ -153,7 +156,9 @@ def read(cls, filepath_or_buffer, **kwargs):
                     "num_splits": num_splits,
                     **partition_kwargs,
                 }
-                partition_id = cls.call_deploy(f, chunk_size, num_splits + 2, args)
+                partition_id = cls.call_deploy(
+                    f, chunk_size, num_splits + 2, args, quotechar=quotechar
+                )
                 partition_ids.append(partition_id[:-2])
                 index_ids.append(partition_id[-2])
                 dtypes_ids.append(partition_id[-1])
diff --git a/modin/engines/base/io/text/text_file_reader.py b/modin/engines/base/io/text/text_file_reader.py
--- a/modin/engines/base/io/text/text_file_reader.py
+++ b/modin/engines/base/io/text/text_file_reader.py
@@ -1,14 +1,20 @@
 from modin.engines.base.io.file_reader import FileReader
-import os
+import re
 import numpy as np
 
 
 class TextFileReader(FileReader):
     @classmethod
-    def call_deploy(cls, f, chunk_size, num_return_vals, args):
+    def call_deploy(cls, f, chunk_size, num_return_vals, args, quotechar=b'"'):
         args["start"] = f.tell()
-        f.seek(chunk_size, os.SEEK_CUR)
-        f.readline()  # Read a whole number of lines
+        chunk = f.read(chunk_size)
+        line = f.readline()  # Ensure we read up to a newline
+        # We need to ensure that
+        quote_count = (
+            re.subn(quotechar, b"", chunk)[1] + re.subn(quotechar, b"", line)[1]
+        )
+        while quote_count % 2 != 0:
+            quote_count += re.subn(quotechar, b"", f.readline())[1]
         # The workers return multiple objects for each part of the file read:
         # - The first n - 2 objects are partitions of data
         # - The n - 1 object is the length of the partition or the index if


Test patch:
diff --git a/modin/pandas/test/data/newlines.csv b/modin/pandas/test/data/newlines.csv
new file mode 100644
--- /dev/null
+++ b/modin/pandas/test/data/newlines.csv
@@ -0,0 +1,22 @@
+col1,col2,col3,col4
+"This is a very long
+string with several
+newline characters
+that will probably cause some
+problem for Modin
+and I suspect that
+we
+will hopefully
+reproduce the issue",2,3,4
+"H",2,3,4
+"I",2,3,4
+"J",2,3,4
+"H",2,3,4
+"I",2,3,4
+"J",2,3,4
+"H",2,3,4
+"I",2,3,4
+"J",2,3,4
+"H",2,3,4
+"I",2,3,4
+"J",2,3,4
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -997,6 +997,12 @@ def test_from_csv_parse_dates(make_csv_file):
     assert modin_df_equals_pandas(modin_df, pandas_df)
 
 
+def test_from_csv_newlines_in_quotes():
+    pandas_df = pandas.read_csv("modin/pandas/test/data/newlines.csv")
+    modin_df = pd.read_csv("modin/pandas/test/data/newlines.csv")
+    assert modin_df_equals_pandas(modin_df, pandas_df)
+
+
 @pytest.mark.skip(reason="No clipboard on Travis")
 def test_to_clipboard():
     modin_df = create_test_ray_dataframe()