BUG: Prevent 3D-ndarray for nested tuple labels (#24687)
State: closed

Being a very rare issue encountered with nested tuples as column and index labels, here is the fix I've managed to come up with - for the time being.  While `clean_index_list()` in `pandas/_libs/lib.pyx` is responsible for returning an invalid result (a 3D ndarray where the inner dimensions should be nested tuples), debugging Cython is very challenging for me.  And yes, use of tuples in this way is extremely uncommon.  So far, the code has run successfully on two distance matrices.
<img width="1191" alt="fix" src="https://user-images.githubusercontent.com/17303881/51058729-b9af9080-15b7-11e9-9726-3bde8ca67330.png">


Labels:
name: Visualization
description: plotting

Comments:
@summonholmes : Thanks for contribution!  Given the limited use-case as you point, I wonder what the trade-off might be between degree of use vs. work to maintain.

Also, we're going to need at least one test and a `whatsnew` entry for this contribution (probably for `0.25.0` though, if it exists, otherwise, hold on this part).
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=h1) Report
> Merging [#24732](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fdc4db25a9988a7b595a3756760d05a6c177123d?src=pr&el=desc) will **decrease** coverage by `49.31%`.
> The diff coverage is `51.11%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24732/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #24732       +/-   ##
===========================================
- Coverage   92.39%   43.07%   -49.32%     
===========================================
  Files         166      166               
  Lines       52358    52362        +4     
===========================================
- Hits        48374    22555    -25819     
- Misses       3984    29807    +25823
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `43.07% <51.11%> (-0.03%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `56.47% <51.11%> (-39.83%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.35%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [124 more](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=footer). Last update [fdc4db2...cde96f8](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=h1) Report
> Merging [#24732](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/33f91d8f9f2e84f2b5f3ac3f0481b691c977c427?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24732/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24732      +/-   ##
==========================================
- Coverage   92.38%   92.38%   -0.01%     
==========================================
  Files         166      166              
  Lines       52358    52363       +5     
==========================================
+ Hits        48373    48377       +4     
- Misses       3985     3986       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <0%> (-0.01%)` | :arrow_down: |
| #single | `42.91% <0%> (-0.17%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `72.46% <0%> (-1.78%)` | :arrow_down: |
| [pandas/tseries/offsets.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvb2Zmc2V0cy5weQ==) | `96.69% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `93.87% <0%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/dtypes.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2R0eXBlcy5weQ==) | `95.6% <0%> (+0.02%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24732/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.09% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=footer). Last update [33f91d8...cec3d57](https://codecov.io/gh/pandas-dev/pandas/pull/24732?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> not really sure what this PR is trying to do

I admit that this specific use case for nested tuple index labels is very esoteric, but in the function `ensure_index`, Cython function `clean_index_list` very rarely returns an ndarray of shape (1, 2, 2), which will result in a dimensional error.  This PR is just a start, to convert the 3D ndarray back into the correct 2D ndarray.

> @summonholmes : Thanks for contribution! Given the limited use-case as you point, I wonder what the trade-off might be between degree of use vs. work to maintain.
> 
> Also, we're going to need at least one test and a `whatsnew` entry for this contribution (probably for `0.25.0` though, if it exists, otherwise, hold on this part).

Yes, there is trade off, and moving a check such as this somewhere else might be better.  Eventually, these efforts will have to make their way back to the Cython file, lib.pyx.
@summonholmes your are missing my point
pls show a reproducible example of what is the problem; what you are doing is likely an error
> @summonholmes your are missing my point
> pls show a reproducible example of what is the problem; what you are doing is likely an error

It's in the bug report that I created, but I'll post a more concise demo here:
```python
from pandas._libs import lib
from pandas import DataFrame
from seaborn import light_palette

# Using the nested tuple cluster
broken_cluster = {
    (("Turtle", "Chicken"), (("Man", "Monkey"), "Dog")): (0, 28.375, 31.875),
    "Tuna": (28.375, 0, 41),
    "Moth": (31.875, 41, 0)
}
broken_cluster = DataFrame(broken_cluster, index=broken_cluster.keys())
broken_cluster.style.background_gradient(
    cmap=light_palette("indigo", as_cmap=True))

# Without using the nested tuple cluster
working_cluster = {
    "S": (0, 28.375, 31.875),
    "Tuna": (28.375, 0, 41),
    "Moth": (31.875, 41, 0)
}
working_cluster = DataFrame(working_cluster, index=working_cluster.keys())
working_cluster.style.background_gradient(
    cmap=light_palette("indigo", as_cmap=True))
# Highlight mins

# The culprit:
lib.clean_index_list([(('Turtle', 'Chicken'), (('Man', 'Monkey'), 'Dog'))])[0]

```

Output of `lib.clean_index_list([(('Turtle', 'Chicken'), (('Man', 'Monkey'), 'Dog'))])[0]`:
```
array([[['Turtle', 'Chicken'],
        [('Man', 'Monkey'), 'Dog']]], dtype=object)
```
I welcome any feedback, and any indication of what I might be doing wrong.  I used tuples in this program for the sake of Pythonic automation, nothing else.  You might also wish to see #24688
```
In [4]: broken_cluster.index                                                                                                                                                                                                                                            
Out[4]: Index([(('Turtle', 'Chicken'), (('Man', 'Monkey'), 'Dog')), 'Tuna', 'Moth'], dtype='object')
```
this has NO support in pandas whatsoever. If you can raise an error in a performant way great would take it
> It's in the bug report that I created, but I'll post a more concise demo here:

What's the issue number?

It's not really clear to me what the changes here are doing. e.g. why check specifically against a shape of `(1, 2, 2)`, not not something more general? This feels like you're trying to use pandas for something it isn't well-suited for.
> > It's in the bug report that I created, but I'll post a more concise demo here:
> 
> What's the issue number?
> 
> It's not really clear to me what the changes here are doing. e.g. why check specifically against a shape of `(1, 2, 2)`, not not something more general? This feels like you're trying to use pandas for something it isn't well-suited for.

The original issue I was addressing with this PR was #24687, and after further testing I've determined that nested tuples can generate even more anomalous shapes than `(1, 2, 2)`.  The fix on my end is simply converting to string and recording the actual tuples somewhere else.  Yes, I would never use pandas like this on a normal day.  Therefore, I believe this PRs intention will change to prevent this use of pandas in the first place.

> ```
> In [4]: broken_cluster.index                                                                                                                                                                                                                                            
> Out[4]: Index([(('Turtle', 'Chicken'), (('Man', 'Monkey'), 'Dog')), 'Tuna', 'Moth'], dtype='object')
> ```
> this has NO support in pandas whatsoever. If you can raise an error in a performant way great would take it

Please correct me if I'm wrong, I'd be happy to work on a fix.  You're saying that an error should be raised, as soon as a tuple is assigned to a column or index label?   I'm assuming that the approach required is closely related to #24688 and #24702.
Thanks. I don't think the resolution of #24688 and #24702 is clear yet.
looks like this was completly reverted, closing.