Repository: modin-project/modin
Pull request: Fixed #1412, losing dtypes when constructing DataFrame
<!--
Thank you for your contribution! 
Please review the contributing docs: https://modin.readthedocs.io/en/latest/contributing.html
if you have questions about contributing.
-->

## What do these changes do?

<!-- Please give a short brief about these changes. -->

- [x] passes `flake8 modin`
- [x] passes `black --check modin`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #1412 <!-- issue must be created for each patch -->
- [x] tests added and passing


Problem statement:
Incorrect dtype when DataFrame is distributed
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:

Windows 10

- **Modin version** (`modin.__version__`):

0.7.2+85.gb9b791f.dirty

- **Python version**:

Python 3.7.7

- **Code we can use to reproduce**:

```python
if __name__ == "__main__":
    import numpy as np
    import modin.pandas as pd

    s1 = pd.DataFrame(np.arange(1, 10000, dtype=np.float32), columns=["s1"])
    print(s1.dtypes['s1'])
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

On Modin resulting dtype is `float64` while on Pandas it is `float32`.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

Hints from issue discussion:
Interesting, it appears that it does not happen when data is loaded with `dtype` kwarg.
There is a bug in DataFrame constructor:
https://github.com/modin-project/modin/blob/4c772ab5a97276709ae9e035ead5198caaa7f2bf/modin/pandas/dataframe.py#L113-L116
That list comprehension expression will return a python `list` with no `dtype`, with elements of `data`.
Let's say our `data` is `np.ndarray` with `dtype=float32`, after above expression, we will get a list of array elements, and then, when we would like to create `pandas.DataFrame`:
https://github.com/modin-project/modin/blob/4c772ab5a97276709ae9e035ead5198caaa7f2bf/modin/pandas/dataframe.py#L124-L127
Pandas will automatically detect the `dtype` as `float64`, because it has no information about original data type.

Commit message: fix

Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

Patch to review:
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
index ad0a908b4ab..635bcae86e7 100644
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -111,9 +111,17 @@ def __init__(
                 "Distributing {} object. This may take some time.".format(type(data))
             )
             if is_list_like(data) and not is_dict_like(data):
-                data = [
+                old_dtype = getattr(data, "dtype", None)
+                values = [
                     obj._to_pandas() if isinstance(obj, Series) else obj for obj in data
                 ]
+                if isinstance(data, np.ndarray):
+                    data = np.array(values, dtype=old_dtype)
+                else:
+                    try:
+                        data = type(data)(values, dtype=old_dtype)
+                    except Exception:
+                        data = values
             elif is_dict_like(data) and not isinstance(
                 data, (pandas.Series, Series, pandas.DataFrame, DataFrame)
             ):
diff --git a/modin/pandas/test/test_dataframe.py b/modin/pandas/test/test_dataframe.py
index 038566535a5..d775afeb9ba 100644
--- a/modin/pandas/test/test_dataframe.py
+++ b/modin/pandas/test/test_dataframe.py
@@ -107,6 +107,10 @@ def eval_insert(modin_df, pandas_df, **kwargs):
     )
 
 
+def create_test_dfs(*args, **kwargs):
+    return pd.DataFrame(*args, **kwargs), pandas.DataFrame(*args, **kwargs)
+
+
 class TestDataFrameBinary:
     def inter_df_math_helper(self, modin_df, pandas_df, op):
         # Test dataframe to datframe
@@ -5418,6 +5422,22 @@ def test_constructor(self, data):
         modin_df = pd.DataFrame({k: pd.Series(v) for k, v in data.items()})
         df_equals(pandas_df, modin_df)
 
+    @pytest.mark.parametrize(
+        "data",
+        [
+            np.arange(1, 10000, dtype=np.float32),
+            [
+                pd.Series([1, 2, 3], dtype="int32"),
+                pandas.Series([4, 5, 6], dtype="int64"),
+                np.array([7, 8, 9], dtype=np.float32),
+            ],
+            pandas.Categorical([1, 2, 3, 4, 5]),
+        ],
+    )
+    def test_constructor_dtypes(self, data):
+        md_df, pd_df = create_test_dfs(data)
+        df_equals(md_df, pd_df)
+
     def test_constructor_columns_and_index(self):
         modin_df = pd.DataFrame(
             [[1, 1, 10], [2, 4, 20], [3, 7, 30]],


Code review comments:
diff_hunk: @@ -111,9 +111,17 @@ def __init__(
                 "Distributing {} object. This may take some time.".format(type(data))
             )
             if is_list_like(data) and not is_dict_like(data):
-                data = [
+                old_dtype = getattr(data, "dtype", None)
+                values = [
                     obj._to_pandas() if isinstance(obj, Series) else obj for obj in data
                 ]
+                if isinstance(data, np.ndarray):
+                    data = np.array(values, dtype=old_dtype)
+                else:
+                    try:
+                        data = type(data)(values, dtype=old_dtype)
+                    except Exception:
path: modin/pandas/dataframe.py
text: @user1:
When would an Exception get thrown here?

@user1:
Generic `Exception` catching is maybe not necessarily what we want.

@author:
An exception may be thrown if `type(data)` is a class that does not support `dtype` information, so its constructor raises exception cause it obtained an unexcepted keyword argument. For example, if `data` is `list`.

You're right that generic `Exception` is not suitable exception type, changed it to `TypeError` at the last commit