ENH: better MultiIndex.__repr__
State: closed

closes #13480 
closes #12423
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Proposal to make a new repr for MultiIndex. Displaying MultiIndex will be based on displaying vertically stacked tuples, as discussed in #13480. This makes it easier to understand the structure of the MultiIndex.

In the proposal we get:
* item formatting according to each level's formatting rule,
* right-justification for each tuple item,
* row-wise truncation according to ``pd.options.display.max_seq_items``,
* column-wise truncation according to ``pd.options.display.width``,

A large MultiIndex example will now look like this:

```python
>>> n = 1_000_000
>>> ci = pd.CategoricalIndex(list('a' * n) + (['bcd'] * n),
...                          categories=['a', 'bcd'], ordered=True)
>>> dti =pd.date_range('2000-01-01', freq='s', periods=2 * n)
>>> mi = pd.MultiIndex.from_arrays([ci, ci.codes+9, dti, dti, dti],
...                                names = ['a', 'b', 'x', 'x2', 'x3'])
>>> mi
MultiIndex([(  'a',  9, '2000-01-01 00:00:00', '2000-01-01 00:00:00', ...),
            (  'a',  9, '2000-01-01 00:00:01', '2000-01-01 00:00:01', ...),
            (  'a',  9, '2000-01-01 00:00:02', '2000-01-01 00:00:02', ...),
            (  'a',  9, '2000-01-01 00:00:03', '2000-01-01 00:00:03', ...),
            (  'a',  9, '2000-01-01 00:00:04', '2000-01-01 00:00:04', ...),
            (  'a',  9, '2000-01-01 00:00:05', '2000-01-01 00:00:05', ...),
            (  'a',  9, '2000-01-01 00:00:06', '2000-01-01 00:00:06', ...),
            (  'a',  9, '2000-01-01 00:00:07', '2000-01-01 00:00:07', ...),
            (  'a',  9, '2000-01-01 00:00:08', '2000-01-01 00:00:08', ...),
            (  'a',  9, '2000-01-01 00:00:09', '2000-01-01 00:00:09', ...),
            ...
            ('bcd', 10, '2000-01-24 03:33:10', '2000-01-24 03:33:10', ...),
            ('bcd', 10, '2000-01-24 03:33:11', '2000-01-24 03:33:11', ...),
            ('bcd', 10, '2000-01-24 03:33:12', '2000-01-24 03:33:12', ...),
            ('bcd', 10, '2000-01-24 03:33:13', '2000-01-24 03:33:13', ...),
            ('bcd', 10, '2000-01-24 03:33:14', '2000-01-24 03:33:14', ...),
            ('bcd', 10, '2000-01-24 03:33:15', '2000-01-24 03:33:15', ...),
            ('bcd', 10, '2000-01-24 03:33:16', '2000-01-24 03:33:16', ...),
            ('bcd', 10, '2000-01-24 03:33:17', '2000-01-24 03:33:17', ...),
            ('bcd', 10, '2000-01-24 03:33:18', '2000-01-24 03:33:18', ...),
            ('bcd', 10, '2000-01-24 03:33:19', '2000-01-24 03:33:19', ...)],
           dtype='object', names=['a', 'b', 'x', 'x2', 'x3'], length=2000000)
```

For further examples, see the added tests in pandas/tests/indexes/multi/test_format.py.

Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: MultiIndex

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22511?src=pr&el=h1) Report
> Merging [#22511](https://codecov.io/gh/pandas-dev/pandas/pull/22511?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b8ad9da4213be55c587db015c640d927aeb7feaf?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22511/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22511?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22511      +/-   ##
==========================================
- Coverage   91.73%   91.73%   -0.01%     
==========================================
  Files         178      178              
  Lines       50774    50794      +20     
==========================================
+ Hits        46579    46595      +16     
- Misses       4195     4199       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.32% <100%> (ø)` | :arrow_up: |
| #single | `41.18% <46.66%> (-0.12%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22511?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/22511/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `98.92% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/22511/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.71% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/22511/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.73% <100%> (+0.06%)` | :arrow_up: |
| [pandas/io/formats/printing.py](https://codecov.io/gh/pandas-dev/pandas/pull/22511/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvcHJpbnRpbmcucHk=) | `86.72% <100%> (+1.16%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/22511/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/22511/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22511?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22511?src=pr&el=footer). Last update [b8ad9da...1d96c98](https://codecov.io/gh/pandas-dev/pandas/pull/22511?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

The failures are unrelated:

travis:
```
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated
```

circli-ci: py27_compat:
```
ImportError: libgfortran.so.1: cannot open shared object file: No such file or directory
```

Will rebase and force push to see if this is an intermittent failure.
I think I've adjusted for all the comments.
The trvis failure was a ResourceWarning, so unrelated to this PR.
@topper-123 : FYI, Anaconda has been having some bad servicing issues, so unfortunately, I don't think CI is going to be very cooperative at this point in time.
Ok, thanks for notifying me.

wrt. the PR, all comments by @jreback should have been addressed. Some further simplifications have also been done: So methods ``._format_space`` and ``_format_attrs`` have been removed and MultiIndex  now inherits those instead.
Hello @topper-123! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











* In the file [`pandas/tests/indexes/multi/test_format.py`](https://github.com/pandas-dev/pandas/blob/1d96c98c94a5f98dd44228b2770e9c64ddaabd64/pandas/tests/indexes/multi/test_format.py):

> [Line 156:80](https://github.com/pandas-dev/pandas/blob/1d96c98c94a5f98dd44228b2770e9c64ddaabd64/pandas/tests/indexes/multi/test_format.py#L156): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (95 > 79 characters)





##### Comment last updated at 2019-06-11 19:54:10 UTC
all comments addressed.
The CircleCi failure looks unrelated to this PR, else ok:

```bash
=================================== FAILURES ===================================
___________________________ test_tick_add_sub[Milli] ___________________________

cls = <class 'pandas.tseries.offsets.Milli'>

    @pytest.mark.parametrize('cls', tick_classes)
>   @example(n=2, m=3)
    @example(n=800, m=300)
    @example(n=1000, m=5)
    @given(n=st.integers(-999, 999), m=st.integers(-999, 999))
    def test_tick_add_sub(cls, n, m):
E   FailedHealthCheck: Data generation is extremely slow: Only produced 3 valid examples in 1.40 seconds (0 invalid ones and 0 exceeded maximum size). Try decreasing size of the data you're generating (with e.g.max_size or max_leaves parameters).
E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.too_slow to the suppress_health_check settings for this test.

pandas/tests/tseries/offsets/test_ticks.py:40: FailedHealthCheck
---------------------------------- Hypothesis ----------------------------------
You can add @seed(61657180377025658002391750862615993709) to this test or run pytest with --hypothesis-seed=61657180377025658002391750862615993709 to reproduce this failure.
-------------------- generated xml file: /reports/junit.xml --------------------
 1 failed, 31182 passed, 3616 skipped, 470 xfailed, 4 xpassed in 1103.21 seconds 
Exited with code 1
```

Maybe something with #22280, @Zac-HD ?
Should be fixed if you merge master.

________________________________
From: topper-123 <notifications@github.com>
Sent: Saturday, September 8, 2018 10:35:30 AM
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] ENH: better MultiIndex.__repr__ (#22511)


The CircleCi looks unrelated to this PR:

=================================== FAILURES ===================================
___________________________ test_tick_add_sub[Milli] ___________________________

cls = <class 'pandas.tseries.offsets.Milli'>

    @pytest.mark.parametrize('cls', tick_classes)
>   @example(n=2, m=3)
    @example(n=800, m=300)
    @example(n=1000, m=5)
    @given(n=st.integers(-999, 999), m=st.integers(-999, 999))
    def test_tick_add_sub(cls, n, m):
E   FailedHealthCheck: Data generation is extremely slow: Only produced 3 valid examples in 1.40 seconds (0 invalid ones and 0 exceeded maximum size). Try decreasing size of the data you're generating (with e.g.max_size or max_leaves parameters).
E   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.too_slow to the suppress_health_check settings for this test.

pandas/tests/tseries/offsets/test_ticks.py:40: FailedHealthCheck
---------------------------------- Hypothesis ----------------------------------
You can add @seed(61657180377025658002391750862615993709) to this test or run pytest with --hypothesis-seed=61657180377025658002391750862615993709 to reproduce this failure.
-------------------- generated xml file: /reports/junit.xml --------------------
 1 failed, 31182 passed, 3616 skipped, 470 xfailed, 4 xpassed in 1103.21 seconds
Exited with code 1

Maybe something with #22280<https://github.com/pandas-dev/pandas/pull/22280>, @Zac-HD<https://github.com/Zac-HD> ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/pull/22511#issuecomment-419651758>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHImYYRcuDGalWEySzFOwRv8X4vNElks5uY-PCgaJpZM4WMsIo>.

But the Hypothesis issue is unrelated to this PR?

EDIT: I could force push until CircleCi passes? This looks like something that will pass, if I push enough versions through...
I force pushed again, and as expected everything is green now.

Those Tick/hypothesis failures are unrelated to this PR, so can't do anything about them here...
@topper-123 you should merge master 

force pushing doesn’t do anything if the proper commits are not there
I did that. (pulled upstream master into my master, then rebased this branch on newest master, then force pushed)
Almost forgot to update the doc strings, but done now in newset commit.

In general, the MultiIndex repr is referenced very little in the main docs and is mostly auto-generated there, so this PR requires only some very minor changes there.

Else all green, and the PR is based on newest upstream master.

Is this ok, barring the small merge conflict that #22671 has caused? 

I'l resolve the conflict later today, so could take eventual requests at the same time.
Is this ok?
lgtm. @jorisvandenbossche @TomAugspurger 
Will take a look tomorrow
Hi, @jorisvandenbossche 

> a problem with the current proposal: it looks like valid code, but is actually not

I consider this a worthwhile tradeoff, but in the future we can maybe change the signature for ``MultiIndex``:
1. Change attribute name ``labels`` to ``codes`` (see #13443)
2. add a ``MultiIndex.from_codes`` class method 
3. Change ``MultiIndex.__init__`` signature to be similar to the current ``from_tuples``.

> About truncating the tuples itself if it would go over one line (the "column-wise truncation according to pd.options.display.width,"), I am not really sure about that. It really makes it very difficult to see what is in your MI object. We could also let the tuples overflow?

DataFrames also truncate (though in a slightly different way), so the same problem is there also? Maybe users who want to see all the values could be advised to adjust ``display.width``?  

> The ``dtype=object`` might be a bit confusing.

Yeah, I agree. I can remove it, and it can be added back again to the repr if/when a ``MultiDtype`` is added? (got no idea if that's in the cards)

> You now used options.display.max_seq_items to determine how many values to show. Which is certainly a reasonable choice (since the other indexes do the same). However, I have the feeling that this can give quite long outputs (of course only when still below the 100) compared to other index classes, because here each element is put under each other. Eg pd.MultiIndex.from_product([['a', 'abc'], range(50)]) gives a really long output

I see your point and agree it can be long. The same problem can be said about DataFrame though. The current implementation has the advantage that it follows the usual conventions from Pandas, as you mention. Do you have a proposal how it could be better/shorter and still reasonably consistent?


The CirceCi checks don't seem to run, been waiting for 12 hours+ now.

Wrt. the comments by @jorisvandenbossche , I changed where it should be changed (but ``pandas/core/indexes/base.py`` does not require changes), though some questions remain, see my answer to @jorisvandenbossche comment above.

> (but pandas/core/indexes/base.py does not require changes)

See eg 

https://github.com/pandas-dev/pandas/blob/bdb7a1603f1e0948ca0cab011987f616e7296167/pandas/core/indexes/base.py#L1336-L1347

(and there are some more in the file)
@jorisvandenbossche , sorry, I looked at ``pandas/core/base.py`` by mistake, sorry about that. Those have been corrected now also.

EDIT: thoughts on dropping dtype  from the repr? It doesn't add any information, but OTOH it would be more consistent keeping it as other index types all show dtype.
I'll add some tests on option_context. No comments came on the dtype question, so I'll just to the tests and hope this wil be ok to pull in now.

with regards to examples, there are not examples on how options_context works with DataFrame either that I can find, and this PR needs to be contained a bit. so it can be finished up.

An example with a DataFrame could perhaps be added to ``options.rst``, but that would be better in a different PR, IMO.
Hey @jreback , if you look at [test_format.py line 118](https://github.com/pandas-dev/pandas/pull/22511/files#diff-fa63f8f306e90649736615df4edd7b8dR118), where actually already is a test for option_context.

So, is this ok?
> with regards to examples, there are not examples on how options_context works with DataFrame either that I can find, and this PR needs to be contained a bit. so it can be finished up.

well, some examples would be useful here. better to do it in this PR.


> Hey @jreback , if you look at test_format.py line 118, where actually already is a test for option_context.

I want specific tests for MI repr. That is what you are changing. Try with different values for max_seq_items and see what happens.
> I consider this a worthwhile tradeoff, but in the future we can maybe change the signature for MultiIndex:
>
>    - Change attribute name labels to codes (see #13443)
>    - add a MultiIndex.from_codes class method
>    - Change MultiIndex.__init__ signature to be similar to the current from_tuples.

This hasn't been discussed yet AFAIK (the first item you list is only a name-change, not changing the structure of the constructor), so I wouldn't let the repr depend on that.

> > About truncating the tuples itself if it would go over one line (the "column-wise truncation according to pd.options.display.width,"), I am not really sure about that. It really makes it very difficult to see what is in your MI object. We could also let the tuples overflow?
>
> DataFrames also truncate (though in a slightly different way), so the same problem is there also? Maybe users who want to see all the values could be advised to adjust display.width?

Yep, that is true, and that can be annoying as well. Although for DataFrame you have other ways to inspect as well (eg `.info()`) and other options to deal with this (eg `max_columns`). Changing `display.width` would then also impact other things (but that is maybe not a problem).

But I think it would at least be useful to still see 'something' about the levels (apart from the name), such as the dtype?

> I see your point and agree it can be long. The same problem can be said about DataFrame though. The current implementation has the advantage that it follows the usual conventions from Pandas, as you mention. Do you have a proposal how it could be better/shorter and still reasonably consistent?

Yep, we have been having some discussion to shorten it for DataFrame as well (where it is already a bit shorter, i.e. 60 instead of 100). I suppose the better path forward is to keep following the `options.display.max_seq_items` option as you did, but maybe have a general discussion about the option, such as a lower default (or a variable default, like show everything up to 100, but above that only show first/last 10)

---


Trying to think out of the box, just some random ideas together here, a possible other repr (using the `wide_multi_index` from the conftest.py in this PR):

```
<pandas.MultiIndex>
Levels (5):
    * a: category (2) ['a', 'abc']
    * b: int64 (2) [9, 10]
    * dti_1: datetime64[ns] (2000) ['2000-01-01 00:00:00', ... ] 
    * dti_2: datetime64[ns] (2000) ['2000-01-01 00:00:00', ... ] 
    * dti_3: datetime64[ns] (2000) ['2000-01-01 00:00:00', ... ] 
[ ('a',    9, '2000-01-01 00:00:00', '2000-01-01 00:00:00', '2000-01-01 00:00:00'),
  ('a',    9, '2000-01-01 00:00:01', '2000-01-01 00:00:01', '2000-01-01 00:00:01'),
  ('a',    9, '2000-01-01 00:00:02', '2000-01-01 00:00:02', '2000-01-01 00:00:02'),
  ...,
  ('abc', 10, '2000-01-01 00:33:17', '2000-01-01 00:33:17', '2000-01-01 00:33:17'),
  ('abc', 10, '2000-01-01 00:33:18', '2000-01-01 00:33:18', '2000-01-01 00:33:18'),
  ('abc', 10, '2000-01-01 00:33:19', '2000-01-01 00:33:19', '2000-01-01 00:33:19')],
Lenght: 2000
``` 

(basically, not trying to look like valid code, which makes it possible to give more information about the individuals levels, together with the same kind of display of the data you did now)
> I want specific tests for MI repr. That is what you are changing. Try with different values for max_seq_items and see what happens.

That *is* what is being done in line 108-128 (first a test with an implicit max_seq_items equal to 100, then a test where is is set equal to 5:

https://github.com/pandas-dev/pandas/blob/acad97991d93da043cf6d132a8a216fdbfd0740e/pandas/tests/indexes/multi/test_format.py#L107-L128
can you rebase and any other comments @jorisvandenbossche