Repository: modin-project/modin
Issue category: Feature Request

Warn user about heterogeneous data presence during `read_csv` call and propose workaround
Subpart of https://github.com/modin-project/modin/issues/3278.



Functions to edit:
modin/backends/pandas/parsers.py:PandasParser.get_dtypes

Patch:
diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst
index d0cd7d6f7e2..432cb095537 100644
--- a/docs/troubleshooting.rst
+++ b/docs/troubleshooting.rst
@@ -72,5 +72,127 @@ Restart your interpreter or notebook kernel.
 Avoid starting many Modin notebooks or interpreters in quick succession. Wait 2-3
 seconds before starting the next one.
 
+Importing heterogeneous data by ``read_csv``
+""""""""""""""""""""""""""""""""""""""""""""
+
+Since Modin ``read_csv`` imports data in parallel, it can occur that data read by
+different partitions can have different type (this happens when columns contains
+heterogeneous data, i.e. column values are of different types), which are handled
+differntly. Example of such behaviour is shown below.
+
+.. code-block:: python
+
+  import os
+  import pandas
+  import modin.pandas as pd
+  from modin.config import NPartitions
+
+  NPartitions.put(2)
+
+  test_filename = "test.csv"
+  # data with heterogeneous values in the first column
+  data = """one,2
+  3,4
+  5,6
+  7,8
+  9.0,10
+  """
+  kwargs = {
+      # names of the columns to set, if `names` parameter is set,
+      # header inffering from the first data row/rows will be disabled
+      "names": ["col1", "col2"],
+
+      # explicit setting of data type of column/columns with heterogeneous
+      # data will force partitions to read data with correct dtype
+      # "dtype": {"col1": str},
+  }
+
+
+  try :
+      with open(test_filename, "w") as f:
+          f.write(data)
+      
+      pandas_df = pandas.read_csv(test_filename, **kwargs)
+      pd_df = pd.read_csv(test_filename, **kwargs)
+      
+      print(pandas_df)
+      print(pd_df)
+  finally:
+      os.remove(test_filename)
+
+  Output:
+
+  pandas_df:
+    col1  col2
+  0  one     2
+  1    3     4
+  2    5     6
+  3    7     8
+  4  9.0    10
+
+  pd_df:
+    col1  col2
+  0  one     2
+  1    3     4
+  2    5     6
+  3  7.0     8
+  4  9.0    10
+
+
+In this case `DataFrame` read by pandas in the column ``col1`` contain only ``str`` data
+because of the first string value ("one"), that forced pandas to handle full column
+data as strings. Modin the fisrt partition (the first three rows) read data similary
+to pandas, but the second partition (the last two rows) doesn't contain any strings
+in the first column and it's data is read as floats because of the last column
+value and as a result `7` value was read as `7.0`, that differs from pandas output.
+
+The above example showed the mechanism of occurence of pandas and Modin ``read_csv``
+outputs discrepancy during heterogeneous data import. Please note, that similar
+situations can occur during different data/parameters combinations.
+
+**Solution**
+
+In the case if heterogeneous data is detected, corresponding warning will be showed in
+the user's console. Currently, the discrepancies of such type doesn't properly handled
+by Modin, and to avoid this issue, it is needed to set ``dtype`` parameter of ``read_csv``
+function manually to force correct data type definition during data import by
+partitions. Note, that to avoid excessive performance degradation, ``dtype`` value should
+be set fine-grained as it possible (specify ``dtype`` parameter only for columns with
+heterogeneous data).
+
+Setting of ``dtype`` parameter works well for most of the cases, but, unfortunetely, it is
+ineffective if data file contain column which should be interpreted as index
+(``index_col`` parameter is used) since ``dtype`` parameter is responsible only for data
+fields. For example, if in the above example, ``kwargs`` will be set in the next way:
+
+.. code-block:: python
+
+  kwargs = {
+      "names": ["col1", "col2"],
+      "dtype": {"col1": str},
+      "index_col": "col1",
+  }
+
+Resulting Modin DataFrame will contain incorrect value as in the case if ``dtype``
+is not set:
+
+.. code-block:: python
+
+  col1
+  one      2
+  3        4
+  5        6
+  7.0      8
+  9.0     10
+
+In this case data should be imported without setting of ``index_col`` parameter
+and only then index column should be set as index (by using ``DataFrame.set_index``
+funcion for example) as it is shown in the example below:
+
+.. code-block:: python
+
+  pd_df = pd.read_csv(filename, dtype=data_dtype, index_col=None)
+  pd_df = pd_df.set_index(index_col_name)
+  pd_df.index.name = None
 
 .. _issue: https://github.com/modin-project/modin/issues
diff --git a/modin/backends/pandas/parsers.py b/modin/backends/pandas/parsers.py
index 7173e810cfc..336a2f978dc 100644
--- a/modin/backends/pandas/parsers.py
+++ b/modin/backends/pandas/parsers.py
@@ -149,18 +149,36 @@ def get_dtypes(cls, dtypes_ids):
 
         Returns
         -------
-        pandas.Series
-            pandas.Series where index is columns names and values are
-            columns dtypes.
+        frame_dtypes : pandas.Series or dtype
+            Resulting dtype or pandas.Series where column names are used as
+            index and types of columns are used as values for full resulting
+            frame.
         """
+        # each element in `partitions_dtypes` is a Series, where column names are
+        # used as index and types of columns for different partitions are used as values
         partitions_dtypes = cls.materialize(dtypes_ids)
         if all([len(dtype) == 0 for dtype in partitions_dtypes]):
             return None
-        return (
-            pandas.concat(partitions_dtypes, axis=1)
-            .apply(lambda row: find_common_type_cat(row.values), axis=1)
-            .squeeze(axis=0)
-        )
+
+        combined_part_dtypes = pandas.concat(partitions_dtypes, axis=1)
+        frame_dtypes = combined_part_dtypes.iloc[:, 0]
+
+        if not combined_part_dtypes.eq(frame_dtypes, axis=0).all(axis=None):
+            ErrorMessage.missmatch_with_pandas(
+                operation="read_*",
+                message="Data types of partitions are different! "
+                "Please refer to the troubleshooting section of the Modin documentation "
+                "to fix this issue",
+            )
+
+            # concat all elements of `partitions_dtypes` and find common dtype
+            # for each of the column among all partitions
+            frame_dtypes = combined_part_dtypes.apply(
+                lambda row: find_common_type_cat(row.values),
+                axis=1,
+            ).squeeze(axis=0)
+
+        return frame_dtypes
 
     @classmethod
     def single_worker_read(cls, fname, **kwargs):