Repository: modin-project/modin

Issue:
read_csv dtype categoricals not working
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04.2
- **Modin installed from (source or binary)**: pip
- **Modin version**: 0.6.3
- **Python version**: 3.7.3
- **Exact command to reproduce**: See below

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

read_csv as below fails and gives
```
modin_result[0].dtype -> object
pandas_result[0].dtype -> CategoricalDtype(categories=['a', 'b'], ordered=False)
```
and then also calls such as `modin_result.select_dtypes(["object"])` fail

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
```python
import modin.pandas as pd
import pandas

pd.DataFrame(["a", "b"]).to_csv("tmp.csv", index = False)

modin_result = pd.read_csv("tmp.csv", dtype="category")
pandas_result = pandas.read_csv("tmp.csv", dtype="category")
assert modin_result.dtypes.equals(pandas_result.dtypes)
```


Discussion:
Hi @ecoughlan thanks for posting!

I can reproduce the issue, and what you're observing is an artifact of very small files. Try the following with only a change to the size of the data created:

```python
import modin.pandas as pd
import pandas

pd.DataFrame([["a", "b"][i % 2] for i in range(1000000)]).to_csv("tmp.csv", index=False)
modin_result = pd.read_csv("tmp.csv", dtype="category")
pandas_result = pandas.read_csv("tmp.csv", dtype="category")
assert modin_result.dtypes.equals(pandas_result.dtypes)
```

In very small files when reading a CSV, we can unintentionally create empty partitions, which shouldn't affect the system. This is definitely a bug, but it's an artifact of extremely small files and how we read those files in-parallel. The solution is probably to just set a minimum limit on the number of bytes we read per-worker. Thanks for posting this, it's an edge case we should be supporting.
Thank you for the very quick reply. Maybe I made my example a bit too minimal (My actual use-case is a ~100MB csv), so I tried and the above still fails for me with
```python
pd.DataFrame([str(i) for i in range(10_000_000)]).to_csv("tmp.csv", index=False)
```
so maybe the partitions are not evenly sharded in my case? Also, I'm not even able to import them as string and then `astype("category")`, or even `df["0"] = pd.Series(df["0"].values, dtype="category"))` so not sure how I can work around this.
Thanks @ecoughlan, I see the issue. It will take some time to get to the bottom of this one because we use pandas functionality to merge or combine the `dtypes` from the partitions. All of the partitions contain `CategoricalDtype`, when when merging them, it is classifying the merged `dtypes` as `object`. 

Resolving patch:
diff --git a/modin/backends/pandas/parsers.py b/modin/backends/pandas/parsers.py
--- a/modin/backends/pandas/parsers.py
+++ b/modin/backends/pandas/parsers.py
@@ -1,5 +1,7 @@
+import numpy as np
 import pandas
 from pandas.core.dtypes.cast import find_common_type
+from pandas.core.dtypes.concat import union_categoricals
 from pandas.io.common import _infer_compression
 from modin.engines.base.io import FileReader
 from modin.data_management.utils import split_result_of_axis_func_pandas
@@ -24,12 +26,27 @@ def _split_result_for_readers(axis, num_splits, df):  # pragma: no cover
     return splits
 
 
+def find_common_type_cat(types):
+    if all(isinstance(t, pandas.CategoricalDtype) for t in types):
+        if all(t.ordered for t in types):
+            return pandas.CategoricalDtype(
+                np.sort(np.unique([c for t in types for c in t.categories])[0]),
+                ordered=True,
+            )
+        return union_categoricals(
+            [pandas.Categorical([], dtype=t) for t in types],
+            sort_categories=all(t.ordered for t in types),
+        ).dtype
+    else:
+        return find_common_type(types)
+
+
 class PandasParser(object):
     @classmethod
     def get_dtypes(cls, dtypes_ids):
         return (
             pandas.concat(cls.materialize(dtypes_ids), axis=1)
-            .apply(lambda row: find_common_type(row.values), axis=1)
+            .apply(lambda row: find_common_type_cat(row.values), axis=1)
             .squeeze(axis=0)
         )
 
diff --git a/modin/engines/base/frame/data.py b/modin/engines/base/frame/data.py
--- a/modin/engines/base/frame/data.py
+++ b/modin/engines/base/frame/data.py
@@ -2,12 +2,12 @@
 import numpy as np
 from operator import itemgetter
 import pandas
-from pandas.core.dtypes.cast import find_common_type
 from pandas.core.index import ensure_index
 from pandas.core.dtypes.common import is_numeric_dtype
 
 from modin.backends.pandas.query_compiler import PandasQueryCompiler
 from modin.error_message import ErrorMessage
+from modin.backends.pandas.parsers import find_common_type_cat as find_common_type
 
 
 class BasePandasFrame(object):
@@ -112,7 +112,7 @@ def _compute_dtypes(self):
         """
 
         def dtype_builder(df):
-            return df.apply(lambda row: find_common_type(row.values), axis=0)
+            return df.apply(lambda col: find_common_type(col.values), axis=0)
 
         map_func = self._build_mapreduce_func(0, lambda df: df.dtypes)
         reduce_func = self._build_mapreduce_func(0, dtype_builder)
diff --git a/modin/engines/ray/pandas_on_ray/frame/data.py b/modin/engines/ray/pandas_on_ray/frame/data.py
--- a/modin/engines/ray/pandas_on_ray/frame/data.py
+++ b/modin/engines/ray/pandas_on_ray/frame/data.py
@@ -1,9 +1,9 @@
 import pandas
-from pandas.core.dtypes.cast import find_common_type
 
 from .partition_manager import PandasOnRayFrameManager
 from modin.engines.base.frame.data import BasePandasFrame
 from modin import __execution_engine__
+from modin.backends.pandas.parsers import find_common_type_cat as find_common_type
 
 if __execution_engine__ == "Ray":
     import ray


Test patch: