Repository: modin-project/modin
Pull request: [FIX] Fix #1683 - losing index names in pd.concat
<!--
Thank you for your contribution! 
Please review the contributing docs: https://modin.readthedocs.io/en/latest/contributing.html
if you have questions about contributing.
-->

## What do these changes do?
Fix of #1683: `pd.concat` losing indices names
<!-- Please give a short brief about these changes. -->

- [x] passes `flake8 modin`
- [x] passes `black --check modin`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #1683 <!-- issue must be created for each patch -->
- [x] tests added and passing


Problem statement:
pd.concat loses index names with MultiIndex
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10
- **Modin version** (`modin.__version__`): 0.7.3+170.g7a4b735
- **Python version**: 3.7.5
- **Code we can use to reproduce**:
```python
if __name__ == "__main__":
    import modin.pandas as pd
    import pandas

    kwargs1 = {
        "data": [[1, 2, 3], [4, 5, 6], [7, 8, 9]],
        "columns": pandas.Index(["a", "b", "c"], name="col1")
    }

    kwargs2 = {
        "data": [[1, 2, 3], [4, 5, 6], [7, 8, 9]],
        "columns": pandas.Index(["e", "f", "g"], name="col1")
    }

    md_df1, md_df2 = pd.DataFrame(**kwargs1), pd.DataFrame(**kwargs2)
    pd_df1, pd_df2 = pandas.DataFrame(**kwargs1), pandas.DataFrame(**kwargs2)
    keys = ["first", "second"]

    print("Pandas result:\n", pandas.concat([pd_df1, pd_df2], keys=keys, axis=1).columns)
    print("Modin result:\n", pd.concat([md_df1, md_df2], keys=keys, axis=1).columns)

```

Output:
```
Pandas result:
 MultiIndex([( 'first', 'a'),
            ( 'first', 'b'),
            ( 'first', 'c'),
            ('second', 'e'),
            ('second', 'f'),
            ('second', 'g')],
           names=[None, 'col1'])
Modin result:
 MultiIndex([( 'first', 'a'),
            ( 'first', 'b'),
            ( 'first', 'c'),
            ('second', 'e'),
            ('second', 'f'),
            ('second', 'g')],
           )
```
<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
Modin version of concat loses index names. That bug seems to appear only when `keys` parameter specified.

Hints from issue discussion:

Commit message: test fix

Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

Patch to review:
diff --git a/modin/pandas/concat.py b/modin/pandas/concat.py
index 0b0a98f4b92..bf226493a12 100644
--- a/modin/pandas/concat.py
+++ b/modin/pandas/concat.py
@@ -108,8 +108,15 @@ def concat(
             new_idx_labels = {
                 k: v.index if axis == 0 else v.columns for k, v in zip(keys, objs)
             }
-            tuples = [(k, o) for k, obj in new_idx_labels.items() for o in obj]
+            tuples = [
+                (k, *o) if isinstance(o, tuple) else (k, o)
+                for k, obj in new_idx_labels.items()
+                for o in obj
+            ]
             new_idx = pandas.MultiIndex.from_tuples(tuples)
+            old_name = _determine_name(objs, axis)
+            if old_name:
+                new_idx.names = [None] + old_name
     else:
         new_idx = None
     new_query_compiler = objs[0].concat(
@@ -132,3 +139,25 @@ def concat(
         else:
             result_df.columns = new_idx
     return result_df
+
+
+def _determine_name(objs: list, axis):
+    axis_getter = (lambda df: df.columns) if axis else (lambda df: df.index)
+    name_getter = (
+        (lambda df: axis_getter(df).names)
+        if isinstance(axis_getter(objs[0]), pandas.MultiIndex)
+        else (lambda df: axis_getter(df).name)
+    )
+
+    names = [name_getter(obj) for obj in objs]
+
+    if all(
+        [
+            names[i] == names[j]
+            for i in range(len(names))
+            for j in range(i + 1, len(names))
+        ]
+    ):
+        return [names[0]] if isinstance(names[0], str) else names[0]
+    else:
+        return None
diff --git a/modin/pandas/test/test_concat.py b/modin/pandas/test/test_concat.py
index 057bf1a283e..acfc8c0f5ef 100644
--- a/modin/pandas/test/test_concat.py
+++ b/modin/pandas/test/test_concat.py
@@ -45,6 +45,17 @@ def generate_dfs():
     return df, df2
 
 
+def generate_multiindex_dfs():
+    def generate_multiindex(index):
+        return pandas.MultiIndex.from_tuples(
+            [("a", x) for x in index.values], names=["name1", "name2"]
+        )
+
+    df1, df2 = generate_dfs()
+    df1.columns, df2.columns = map(generate_multiindex, [df1.columns, df2.columns])
+    return df1, df2
+
+
 def generate_none_dfs():
     df = pandas.DataFrame(
         {
@@ -207,3 +218,15 @@ def test_concat_with_empty_frame():
         pd.concat([modin_empty_df, modin_row]),
         pandas.concat([pandas_empty_df, pandas_row]),
     )
+
+
+def test_concat_multiindex():
+    pd_df1, pd_df2 = generate_multiindex_dfs()
+    md_df1, md_df2 = map(from_pandas, [pd_df1, pd_df2])
+
+    keys = ["first", "second"]
+
+    df_equals(
+        pd.concat([md_df1, md_df2], keys=keys, axis=1),
+        pandas.concat([pd_df1, pd_df2], keys=keys, axis=1),
+    )


Code review comments:
diff_hunk: @@ -132,3 +139,25 @@ def concat(
         else:
             result_df.columns = new_idx
     return result_df
+
+
+def _determine_name(objs: list, axis):
+    axis_getter = (lambda df: df.columns) if axis else (lambda df: df.index)
+    name_getter = (
+        (lambda df: axis_getter(df).names)
path: modin/pandas/concat.py
text: @user1:
```suggestion
        (lambda df: df.axes[axes].names)
```

@author:
we can't use this, because `_determine_name` can obtain a list of query compilers as `objs`, but query compiler does not have field `axes`

diff_hunk: @@ -132,3 +139,25 @@ def concat(
         else:
             result_df.columns = new_idx
     return result_df
+
+
+def _determine_name(objs: list, axis):
+    axis_getter = (lambda df: df.columns) if axis else (lambda df: df.index)
+    name_getter = (
path: modin/pandas/concat.py
text: @user1:
I prefer this to be `def` instead of setting equal to `lambda`.

@author:
fixed it

diff_hunk: @@ -132,3 +139,25 @@ def concat(
         else:
             result_df.columns = new_idx
     return result_df
+
+
+def _determine_name(objs: list, axis):
+    axis_getter = (lambda df: df.columns) if axis else (lambda df: df.index)
path: modin/pandas/concat.py
text: @user1:
You don't need this, you can use code below.

@author:
answered in a comment below

diff_hunk: @@ -132,3 +139,25 @@ def concat(
         else:
             result_df.columns = new_idx
     return result_df
+
+
+def _determine_name(objs: list, axis):
+    axis_getter = (lambda df: df.columns) if axis else (lambda df: df.index)
+    name_getter = (
+        (lambda df: axis_getter(df).names)
+        if isinstance(axis_getter(objs[0]), pandas.MultiIndex)
path: modin/pandas/concat.py
text: @user1:
Why is this `objs[0]`? Can you place a comment for why this check is necessary in this way (why not `df`)?
```suggestion
        if isinstance(objs[0].axes[axis], pandas.MultiIndex)
```

@author:
this is a useless check, removed it

diff_hunk: @@ -132,3 +139,25 @@ def concat(
         else:
             result_df.columns = new_idx
     return result_df
+
+
+def _determine_name(objs: list, axis):
+    axis_getter = (lambda df: df.columns) if axis else (lambda df: df.index)
+    name_getter = (
+        (lambda df: axis_getter(df).names)
+        if isinstance(axis_getter(objs[0]), pandas.MultiIndex)
+        else (lambda df: axis_getter(df).name)
path: modin/pandas/concat.py
text: @user1:
```suggestion
        else (lambda df: df.axes[axis].name)
```