Repository: modin-project/modin

Issue:
convert_dtypes() not inferring data type of column as a whole
I have a Modin DataFrame with a sparsely populated string column, i.e. a column with a lot of `NaN` and some strings:
```
>>> df
        string_column
0                 NaN
1                 NaN
2                 NaN
3                 NaN
4                 NaN
...               ...
1133496           NaN
1133497           NaN
1133498           NaN
1133499           NaN
1133500           NaN
[1133501 rows x 1 columns]
```
The notna part:
```
>>> df.loc[df['string_column'].notna(),:]
      string_column
42          Reduced
43      Non-Reduced
44          Reduced
45      Non-Reduced
46          Reduced
...             ...
86803   Non-Reduced
86804       Reduced
86805   Non-Reduced
86846       Reduced
86847   Non-Reduced
[18126 rows x 1 columns]
```
The `dtype` of the string column is `object`:
```
>>> df.dtypes
string_column    object
dtype: object
```

I would like to have it converted into `string` using `convert_dtypes()` on the whole DataFrame (that contains also other columns not shown here, and I'm not using `astype(string`) because the data type is not known ex ante and, hence, should be inferred). However, performing `convert_dtypes()` does not change the `dtype`:
```
>>> df.convert_dtypes().dtypes
string_column    object
dtype: object
```

In pandas, this works just fine, i.e. the column is converted into `string`. I assume that in contrast to pandas, Modin does it partition-wise, and if a partition happens to contain only `NaN`, it can't determine the data type and hence the "lowest common denominator" data type for all partitions will be `object`. Performing it only on the notna part works as expected:
```
>>> df.loc[df['string_column'].notna(),:].convert_dtypes().dtypes
string_column    string
dtype: object
```

**Question:** is this expected behaviour or a bug?


Discussion:
Hi @jsarbach , thanks for opening the issue. I think your hypothesis is correct. Since Modin operates at the partition-layer, the nulls might be throwing off the type conversion. This is a bug since we should be matching pandas in this case. 

cc: @modin-project/modin-core 
@modin-project/modin-core the easiest way to do `convert_dtypes` as a full axis operation. Any other thoughts?
That sounds right. @noloerino I remember you implemented `convert_dtypes`. Should we change it to full axis?
From rereading the discussion in #4671 I think that should be fine, though it might require some nontrivial code changes since we just implement it with algebra map operator right now.

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -1446,7 +1446,7 @@ def isin_func(df, values):
     abs = Map.register(pandas.DataFrame.abs, dtypes="copy")
     applymap = Map.register(pandas.DataFrame.applymap)
     conj = Map.register(lambda df, *args, **kwargs: pandas.DataFrame(np.conj(df)))
-    convert_dtypes = Map.register(pandas.DataFrame.convert_dtypes)
+    convert_dtypes = Fold.register(pandas.DataFrame.convert_dtypes)
     invert = Map.register(pandas.DataFrame.__invert__, dtypes="copy")
     isna = Map.register(pandas.DataFrame.isna, dtypes=np.bool_)
     _isfinite = Map.register(


Test patch:
diff --git a/modin/pandas/test/dataframe/test_map_metadata.py b/modin/pandas/test/dataframe/test_map_metadata.py
--- a/modin/pandas/test/dataframe/test_map_metadata.py
+++ b/modin/pandas/test/dataframe/test_map_metadata.py
@@ -47,7 +47,7 @@
     create_test_dfs,
     default_to_pandas_ignore_string,
 )
-from modin.config import NPartitions
+from modin.config import NPartitions, StorageFormat
 from modin.test.test_utils import warns_that_defaulting_to_pandas
 
 NPartitions.put(4)
@@ -615,17 +615,14 @@ def test_convert_dtypes_single_partition(
     assert modin_result.dtypes.equals(pandas_result.dtypes)
 
 
-@pytest.mark.skipif(
-    get_current_execution() == "BaseOnPython",
-    reason="BaseOnPython cannot not have multiple partitions.",
-)
 def test_convert_dtypes_multiple_row_partitions():
     # Column 0 should have string dtype
     modin_part1 = pd.DataFrame(["a"]).convert_dtypes()
     # Column 0 should have an int dtype
     modin_part2 = pd.DataFrame([1]).convert_dtypes()
     modin_df = pd.concat([modin_part1, modin_part2])
-    assert modin_df._query_compiler._modin_frame._partitions.shape == (2, 1)
+    if StorageFormat.get() == "Pandas":
+        assert modin_df._query_compiler._modin_frame._partitions.shape == (2, 1)
     pandas_df = pandas.DataFrame(["a", 1], index=[0, 0])
     # The initial dataframes should be the same
     df_equals(modin_df, pandas_df)
@@ -638,6 +635,17 @@ def test_convert_dtypes_multiple_row_partitions():
     assert modin_result.dtypes.equals(pandas_result.dtypes)
 
 
+def test_convert_dtypes_5653():
+    modin_part1 = pd.DataFrame({"col1": ["a", "b", "c", "d"]})
+    modin_part2 = pd.DataFrame({"col1": [None, None, None, None]})
+    modin_df = pd.concat([modin_part1, modin_part2])
+    if StorageFormat.get() == "Pandas":
+        assert modin_df._query_compiler._modin_frame._partitions.shape == (2, 1)
+    modin_df = modin_df.convert_dtypes()
+    assert len(modin_df.dtypes) == 1
+    assert modin_df.dtypes[0] == "string"
+
+
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 @pytest.mark.parametrize("axis", axis_values, ids=axis_keys)
 @pytest.mark.parametrize("bound_type", ["list", "series"], ids=["list", "series"])