Repository: pandas-dev/pandas
Pull request: BUG: DataFrame fail to construct when data is list and columns is nested list for MI
- [ ] closes #32173 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
can't create DataFrame with multiIndex by list of list
#### Code Sample, a copy-pastable example if possible

```python
df=DataFrame([1,2,3,4],[4,5,6,7], columns= [list('abcd'), list('cdef')])

```
#### Problem description
when creating DataFrame by list of list with multiIndex, raise ValueError. I'm new to pandas, I don't know why, but  it seems like a bug...

`d=DataFrame([[1,2,3,4],[4,5,6,7]], columns= [list('abcd'), list('cdef')])`

```
Traceback (most recent call `last):

 File "<ipython-input-70-b1022cf50f33>", line 1, in <module>
    d=DataFrame([[1,2,3,4],[4,5,6,7]], columns= [list('abcd'), list('cdef')])

  File "D:\Users\tangliu\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py", line 450, in __init__
    arrays, columns = to_arrays(data, columns, dtype=dtype)

  File "D:\Users\tangliu\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\construction.py", line 464, in to_arrays
    return _list_to_arrays(data, columns, coerce_float=coerce_float, dtype=dtype)

  File "D:\Users\tangliu\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\construction.py", line 504, in _list_to_arrays
    raise ValueError(e) from e

ValueError: 2 columns passed, passed data had 4 columns`
It's ok using numpy ndarray or something else
```
```
d=DataFrame([[1,2,3,4],[4,5,6,7]])

d

Out[72]: 
   0  1  2  3
0  1  2  3  4
1  4  5  6  7

d.columns=[list('abcd'),list('cdef')]

d
Out[74]: 
   a  b  c  d
   c  d  e  f
0  1  2  3  4
1  4  5  6  7
```
it's ok when using numpy's ndarray

```
d=DataFrame(np.random.randn(6,4), columns=[list('abcd'), list('cdef')])

d
Out[81]: 
          a         b         c         d
          c         d         e         f
0 -0.066216 -1.264157 -1.199212 -0.184394
1  0.413666 -0.310750  1.138304 -0.271516
2  0.263854  0.827858  0.652344 -0.024369
3  0.041060 -2.235779  0.707691  0.319255
4 -0.118407 -1.278394  0.398972  0.597036
5  0.048947  0.697497  2.168712  0.650022
```

Hints from issue discussion:
interesting, looks buggy!
take

Commit message: Merge remote-tracking branch 'upstream/master' into fix_issue_32173

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index a4c991dcc166c..c883c3c466b07 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -249,6 +249,8 @@ Other
   instead of ``TypeError: Can only append a Series if ignore_index=True or if the Series has a name`` (:issue:`30871`)
 - Set operations on an object-dtype :class:`Index` now always return object-dtype results (:issue:`31401`)
 - Bug in :meth:`AbstractHolidayCalendar.holidays` when no rules were defined (:issue:`31415`)
+- Bug in :class:`DataFrame` when initiating a frame with lists and assign ``columns`` with nested list for ``MultiIndex`` (:issue:`32173`)
+-
 -
 
 .. ---------------------------------------------------------------------------
diff --git a/pandas/core/internals/construction.py b/pandas/core/internals/construction.py
index 57ed2555761be..d75033de50d50 100644
--- a/pandas/core/internals/construction.py
+++ b/pandas/core/internals/construction.py
@@ -574,12 +574,30 @@ def _convert_object_array(content, columns, coerce_float=False, dtype=None):
     if columns is None:
         columns = ibase.default_index(len(content))
     else:
-        if len(columns) != len(content):  # pragma: no cover
+        is_mi_list = isinstance(columns, list) and all(
+            isinstance(col, list) for col in columns
+        )
+
+        if not is_mi_list and len(columns) != len(content):  # pragma: no cover
             # caller's responsibility to check for this...
             raise AssertionError(
                 f"{len(columns)} columns passed, passed data had "
                 f"{len(content)} columns"
             )
+        elif is_mi_list:
+
+            # check if nested list column, length of each sub-list should be equal
+            if len({len(col) for col in columns}) > 1:
+                raise ValueError(
+                    "Length of columns passed for MultiIndex columns is different"
+                )
+
+            # if columns is not empty and then length of sublist is not equal to content
+            elif columns and len(columns[0]) != len(content):
+                raise ValueError(
+                    f"{len(columns[0])} columns passed, passed data had "
+                    f"{len(content)} columns"
+                )
 
     # provide soft conversion of object dtypes
     def convert(arr):
diff --git a/pandas/tests/frame/test_constructors.py b/pandas/tests/frame/test_constructors.py
index 8c9b7cd060059..75c18ffbe127e 100644
--- a/pandas/tests/frame/test_constructors.py
+++ b/pandas/tests/frame/test_constructors.py
@@ -1062,6 +1062,32 @@ def test_constructor_list_of_lists(self):
         result = DataFrame(data)
         tm.assert_frame_equal(result, expected)
 
+    def test_constructor_list_like_data_nested_list_column(self):
+        # GH 32173
+        arrays = [list("abcd"), list("cdef")]
+        result = pd.DataFrame([[1, 2, 3, 4], [4, 5, 6, 7]], columns=arrays)
+
+        mi = MultiIndex.from_arrays(arrays)
+        expected = pd.DataFrame([[1, 2, 3, 4], [4, 5, 6, 7]], columns=mi)
+
+        tm.assert_frame_equal(result, expected)
+
+    def test_constructor_wrong_length_nested_list_column(self):
+        # GH 32173
+        arrays = [list("abc"), list("cde")]
+
+        msg = "3 columns passed, passed data had 4"
+        with pytest.raises(ValueError, match=msg):
+            DataFrame([[1, 2, 3, 4], [4, 5, 6, 7]], columns=arrays)
+
+    def test_constructor_inequal_length_nested_list_column(self):
+        # GH 32173
+        arrays = [list("abcd"), list("cde")]
+
+        msg = "Length of columns passed for MultiIndex columns is different"
+        with pytest.raises(ValueError, match=msg):
+            DataFrame([[1, 2, 3, 4], [4, 5, 6, 7]], columns=arrays)
+
     def test_constructor_sequence_like(self):
         # GH 3783
         # collections.Squence like


Code review comments:
diff_hunk: @@ -574,12 +574,30 @@ def _convert_object_array(content, columns, coerce_float=False, dtype=None):
     if columns is None:
         columns = ibase.default_index(len(content))
     else:
-        if len(columns) != len(content):  # pragma: no cover
+        is_mi_list = isinstance(columns, list) and all(
path: pandas/core/internals/construction.py
text: @user1:
can you add some comments on what this section is about (list-of-lists)

@user1:
I would move this whole check to _validate_columns, and remove the columns argument from _convert_object_array (and do it as an independent check before).

@author:
Nice idea! @user1 
moving the check out of `_convert_object_array` makes the scope of `_convert_object_array` clearer, and dealing with `columns` and `content` separately seem nicer

but if only moving the whole check part out will not be able to remove `columns` argument from `_convert_object_array`, because it also creates column names using length of contetn if `columns` is `None`. So i made a method called `_validate_or_indexify_columns` method (name could be improved if made method clearer ^^), and will do two things: 1. if `columns` is `None`, then indexify it with `default_index`, 2. otherwise, go through the checks.

What do you think of this change, does it make code clearer? @user1 any feedbacks are very welcome!