Repository: modin-project/modin

Issue:
Reading CSV with mismatched quotes causes infinite loop
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 19.04
- **Modin version** (`modin.__version__`): HEAD
- **Python version**: 3.8
- **Code we can use to reproduce**:

Try to read this file using the `pd.read_csv("test_quotes_1.txt", sep='\t')` function:

The sample file I am trying to read is: [test_quotes_1.txt](https://github.com/modin-project/modin/files/4584991/test_quotes_1.txt) which contains the following tab separated data:

```
a   b   c   d
e   f"  l   m
n   o   "p   q"
```

Trying to read the above data will cause an infinite loop.

### Describe the problem
When trying to read CSV (or any delimiter separated data) that has mismatched quotes, the `read_csv` method will cause an infinite loop.

I have provided a TSV example above but this same can be re-created using a CSV file like this one
[test_quotes_1_comma.txt](https://github.com/modin-project/modin/files/4584997/test_quotes_1_comma.txt)

### Source code / logs
The root cause of this issue seems to be the `while` loop that checks for quotes:

https://github.com/modin-project/modin/blob/555c89956b48e6b4ad50877ba4b7c7594cd592e5/modin/engines/base/io/text/text_file_reader.py#L26-L30

An odd number of quotes will mean that this while loop will never resolve and the program will just hang without any error messages. Since this is part of the `engines.base.io.text_file_reader` module, this issue exists with both Ray and Dask.

I think one naive solution might be to check for `EOF` and breaking the loop. A long message suggesting this might also be helpful.

Although this is a malformed data file, trying to read this with pandas succeeds.

I'm happy to send a PR addressing this.


Discussion:
On a slightly related note: It would be very useful if the `read_csv` function would also respect the `quoting=csv.QUOTE_NONE` attribute that is accepted by `pandas.read_csv`. Ideally, this flag should jump over the entire loop above.

In my dataset which contains malformed quotes, commenting out the quote checking loop causes the read of a ~200MB dataset to go from >1m to 5s (also ensures that data is read correctly). For now, I am using an invalid unicode character `\U0000FFFF` to achieve the same outcome as `csv.QUOTE_NONE`

```python
pd.read_csv(
        str(file_name),
        sep='\t',
        names=column_names,
        index_col=False,
        na_values='*',
        dtype='object',
        quoting=csv.QUOTE_NONE,
        quotechar=u"\U0000FFFF"
)
```
Hello @pratheekrebala, thank you for reporting this problem. Indeed there is a bug in `read_csv` implementation and your suggestion of handing `QUOTE_NONE` is correct.
@pratheekrebala We will happily accept your PR addressing this, it's a good find!

The loop exists to ensure that we don't try to partition the data in the middle of a quote. The requirement here is that the partitioned reader consumes up to a complete row of data, but long quotes can cause issues if they contain newlines.
@devin-petersohn: I'll send a PR to address the issue this week.
@pratheekrebala , Can you please answer for a question, are you planning to sent a PR addressing this issue or not? I would like to know this just for that we don't duplicate working.
Hi @YarShev, Apologies about the delay. I will work on this and circle back today. 
Hi, @pratheekrebala , it is great! We are waiting for your PR. Thanks a lot!

Resolving patch:
diff --git a/modin/engines/base/io/text/text_file_reader.py b/modin/engines/base/io/text/text_file_reader.py
--- a/modin/engines/base/io/text/text_file_reader.py
+++ b/modin/engines/base/io/text/text_file_reader.py
@@ -14,6 +14,8 @@
 from modin.engines.base.io.file_reader import FileReader
 import re
 import numpy as np
+import warnings
+import csv
 
 
 class TextFileReader(FileReader):
@@ -22,12 +24,21 @@ def call_deploy(cls, f, chunk_size, num_return_vals, args, quotechar=b'"'):
         args["start"] = f.tell()
         chunk = f.read(chunk_size)
         line = f.readline()  # Ensure we read up to a newline
-        # We need to ensure that
-        quote_count = (
-            re.subn(quotechar, b"", chunk)[1] + re.subn(quotechar, b"", line)[1]
-        )
-        while quote_count % 2 != 0:
-            quote_count += re.subn(quotechar, b"", f.readline())[1]
+        # We need to ensure that one row isn't being split across different partitions
+
+        if args.get("quoting", "") != csv.QUOTE_NONE:
+            quote_count = (
+                re.subn(quotechar, b"", chunk)[1] + re.subn(quotechar, b"", line)[1]
+            )
+            while quote_count % 2 != 0:
+                line = f.readline()
+                quote_count += re.subn(quotechar, b"", line)[1]
+                if not line:
+                    break
+
+            if quote_count % 2 != 0:
+                warnings.warn("File has mismatched quotes")
+
         # The workers return multiple objects for each part of the file read:
         # - The first n - 2 objects are partitions of data
         # - The n - 1 object is the length of the partition or the index if


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -23,6 +23,7 @@
 import os
 import shutil
 import sqlalchemy as sa
+import csv
 
 from .utils import (
     df_equals,
@@ -749,6 +750,35 @@ def test_from_csv_sep_none(make_csv_file):
     df_equals(modin_df, pandas_df)
 
 
+def test_from_csv_bad_quotes():
+    csv_bad_quotes = """1, 2, 3, 4
+one, two, three, four
+five, "six", seven, "eight
+"""
+
+    with open(TEST_CSV_FILENAME, "w") as f:
+        f.write(csv_bad_quotes)
+
+    pandas_df = pandas.read_csv(TEST_CSV_FILENAME)
+    modin_df = pd.read_csv(TEST_CSV_FILENAME)
+
+    df_equals(modin_df, pandas_df)
+
+
+def test_from_csv_quote_none():
+    csv_bad_quotes = """1, 2, 3, 4
+one, two, three, four
+five, "six", seven, "eight
+"""
+    with open(TEST_CSV_FILENAME, "w") as f:
+        f.write(csv_bad_quotes)
+
+    pandas_df = pandas.read_csv(TEST_CSV_FILENAME, quoting=csv.QUOTE_NONE)
+    modin_df = pd.read_csv(TEST_CSV_FILENAME, quoting=csv.QUOTE_NONE)
+
+    df_equals(modin_df, pandas_df)
+
+
 def test_from_csv_categories():
     pandas_df = pandas.read_csv(
         "modin/pandas/test/data/test_categories.csv",