Repository: pandas-dev/pandas

Issue:
MultiIndex `to_string` edge case Error after 0.23.0 upgrade
#### Code example

```python
import pandas as pd
import numpy as np

index = pd.date_range('1970', '2018', freq='A')
data = np.random.randn(len(index))
columns1 = [
    ['This is a long title with > 37 chars.'],
    ['cat'],
]
columns2 = [
    ['This is a loooooonger title with > 43 chars.'],
    ['dog'],
]
df1 = pd.DataFrame(data=data, index=index, columns=columns1)
df2 = pd.DataFrame(data=data, index=index, columns=columns2)
df = pd.concat([df1, df2], axis=1)
df.head()
```
#### Output (using pandas 0.23.0)
```
>>> df.head()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/core/base.py", line 82, in __repr__
    return str(self)
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/core/base.py", line 61, in __str__
    return self.__unicode__()
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/core/frame.py", line 663, in __unicode__
    line_width=width, show_dimensions=show_dimensions)
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/core/frame.py", line 1968, in to_string
    formatter.to_string()
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/io/formats/format.py", line 648, in to_string
    strcols = self._to_str_columns()
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/io/formats/format.py", line 539, in _to_str_columns
    str_columns = self._get_formatted_column_labels(frame)
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/io/formats/format.py", line 782, in _get_formatted_column_labels
    str_columns = _sparsify(str_columns)
  File "/home/david/.virtualenvs/thegrid-py3-venv/lib/python3.5/site-packages/pandas/core/indexes/multi.py", line 2962, in _sparsify
    prev = pivoted[start]
IndexError: list index out of range
```

#### Problem description

After upgrading Pandas 0.22.0 to 0.23.0 I have experienced the above error. I have noticed that it is the length of the column values, `This is a long title with > 37 chars.` and `This is a loooooonger title with > 43 chars.`, that makes the difference. If I tweak the combined length of these to be <= 80 characters, there is no error, and output is as expected.

#### Expected Output (using pandas 0.22.0)
```
>>> df.head()
           This is a long title with > 37 chars.  \
                                             cat   
1970-12-31                             -1.448415   
1971-12-31                              0.081324   
1972-12-31                             -0.018105   
1973-12-31                              0.902790   
1974-12-31                              0.668474   

           This is a loooooonger title with > 43 chars.  
                                                    dog  
1970-12-31                                    -1.448415  
1971-12-31                                     0.081324  
1972-12-31                                    -0.018105  
1973-12-31                                     0.902790  
1974-12-31                                     0.668474
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-124-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_ZA.UTF-8
LOCALE: en_ZA.UTF-8

pandas: 0.23.0
pytest: None
pip: 10.0.1
setuptools: 32.3.1
Cython: None
numpy: 1.14.0
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: 2.5.3
xlrd: None
xlwt: None
xlsxwriter: 1.0.4
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
This doesn't raise for me (py36, and pandas master).

What is `pd.options.display.max_colwidth`, `pd.options.display.wdith`, and `pd.options.display.max_columns`?
@TomAugspurger Here my system pandas 0.23.0 output:
```
>>> import pandas as pd
>>> pd.options.display.max_colwidth
50
>>> pd.options.display.width
80
>>> pd.options.display.max_columns
0
```
0.22.0 output:
```
>>> import pandas as pd
>>> pd.options.display.max_colwidth
50
>>> pd.options.display.width
80
>>> pd.options.display.max_columns
20
```

If I do the following it works in 0.23.0!
```
pd.set_option("max_columns", 20)
```
Did the default setting change in 0.23.0?
Reading the docs show how [0.22](http://pandas.pydata.org/pandas-docs/version/0.22/options.html#available-options):
> In case python/IPython is running in a terminal this can be set to 0

has been updated in [0.23](https://pandas.pydata.org/pandas-docs/stable/options.html#available-options) to:

> In case Python/IPython is running in a terminal this is set to 0 by default.

However, when switching back to 0.22.0 and manually changing the `max_columns` option to `0` doesn't result in raising the exception.

:thinking:  So it still doesn't explain why there would be an error raised when `max_columns` is set to `0`? 
cc @cbrnr if you have any ideas.
I get an `AttributeError: module 'pandas._libs.tslibs.timezones' has no attribute 'tz_standardize'` when I test this with the latest master branch revision. Any ideas how to fix this? Using 0.23, I can reproduce the issue.
You need to recompile the extension modules. Commands for your platform should be in the contributing docs.

________________________________
From: Clemens Brunner <notifications@github.com>
Sent: Monday, May 28, 2018 1:48:25 AM
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] MultiIndex `to_string` edge case Error after 0.23.0 upgrade (#21180)


I get an AttributeError: module 'pandas._libs.tslibs.timezones' has no attribute 'tz_standardize' when I test this with the latest master branch revision. Any ideas how to fix this? Using 0.23, I can reproduce the issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/issues/21180#issuecomment-392435108>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIikYuSBQuVOuVYnZ4mSdLRkxq_pCks5t2525gaJpZM4UKJe1>.

Thanks, I forgot about that. Thankfully, it's not the [add one business](https://github.com/pandas-dev/pandas/commit/c9e8f59668b63738cccb913f837c529887097da1#diff-425b4da47d01dc33d86c5c697e196b70R629) (I get the same error when I revert this change). This will take a bit of work, since everything works in PyCharm but not in IPython (so debugging will be much slower for me since I'm not used to pdb at all)...
Apparently, setting `pd.options.display.max_columns = 0` in 0.22 also results in this error. So the issue was not introduced by my change, which merely changed the default to 0.
Hi @cbrnr 
> also results in this error.

Probably you ment to say `does not`? I do agree, merely changing the default to 0 should not result in the unexpected error.
No, I get the same error with pandas 0.22 if I first set `pd.options.display.max_columns = 0`. This means that this bug has been there for a while (I haven't tried older versions, but I suspect that they will behave similarly).
I do not get the exception on pandas 0.22.0 with 

```python
import pandas as pd
pd.options.display.max_columns = 0
import numpy as np

index = pd.date_range('1970', '2018', freq='A')
data = np.random.randn(len(index))
columns1 = [
    ['This is a long title with > 37 chars.'],
    ['cat'],
]
columns2 = [
    ['This is a loooooonger title with > 43 chars.'],
    ['dog'],
]
df1 = pd.DataFrame(data=data, index=index, columns=columns1)
df2 = pd.DataFrame(data=data, index=index, columns=columns2)
df = pd.concat([df1, df2], axis=1)
df.head()
```
Though it occurs to me that this probably depends on the width of the terminal.
not a regression, but still should fix.
@TomAugspurger I just tried again, I do get the error with 0.22. How are you running this code? If you are not in interactive mode (e.g. IPython), you need to change the last line to `print(df.head())` in order to produce the output. I'm running this in IPython on macOS in a normal terminal (not Jupyter QtConsole) with 100x35 window size.
@jreback could you please make a note when you're moving the milestone? This should be fixed for 0.23.1.
i made a note
and this does not need to block 0.23.1
it’s jot a regression 

pls don’t mark milestones unless ready to go
@jreback This *is* a regression in user experience. It may be an existing bug, but code that was working before, is failing now, because we changed the default. So we should still fix that existing bug for 0.23.1.

I cannot reproduce the error with the example in this issue, but I *do* see it with the example from https://github.com/pandas-dev/pandas/issues/21327
@jorisvandenbossche sure regressions happen, and we *should* fix them all. but unless this is fixed today, it will go in the next release.
We can also change the default of max_columns back to 20 for now if we don't find the effort to fix the bugs
Here's a failing unit test

```diff
diff --git a/pandas/tests/io/formats/test_format.py b/pandas/tests/io/formats/test_format.py
index f221df93d..52f83f093 100644
--- a/pandas/tests/io/formats/test_format.py
+++ b/pandas/tests/io/formats/test_format.py
@@ -305,6 +305,36 @@ class TestDataFrameFormatting(object):
             assert not has_truncated_repr(df)
             assert not has_expanded_repr(df)
 
+    def test_repr_multiindex(self):
+        # https://github.com/pandas-dev/pandas/issues/21180
+        from unittest import mock
+
+        def f():
+            return os.terminal_size((118, 96))
+
+        terminal_size = os.terminal_size((118, 96))
+
+        p1 = mock.patch('pandas.io.formats.console.get_terminal_size',
+                        return_value=terminal_size)
+        p2 = mock.patch('pandas.io.formats.format.get_terminal_size',
+                        return_value=terminal_size)
+        index = pd.date_range('1970', '2018', freq='A')
+        data = np.random.randn(len(index))
+        columns1 = [
+            ['This is a long title with > 37 chars.'],
+            ['cat'],
+        ]
+        columns2 = [
+            ['This is a loooooonger title with > 43 chars.'],
+            ['dog'],
+        ]
+        df1 = pd.DataFrame(data=data, index=index, columns=columns1)
+        df2 = pd.DataFrame(data=data, index=index, columns=columns2)
+        df = pd.concat([df1, df2], axis=1)
+
+        with p1, p2:
+            repr(df.head())
+
     def test_repr_max_columns_max_rows(self):
         term_width, term_height = get_terminal_size()
         if term_width < 10 or term_height < 10:
```
If we don't have a fix for this, I would consider reverting the `pandas.options.display.max_columns` back to 20, and work on fixing this and possibly turning back to 0 for 0.24.0.

Errors in the repr are really annoying, as you cannot even inspect the data properly to see what might be the reason something is not working.
I'm going to try to fix it now.
What's the expected behavior here? I can easily match the behavior of the non-MI case,

```python
In [3]: s = pd.DataFrame({"A" * 41: [1, 2], 'B' * 41: [1, 2]})

In [4]: with p1, p2:
   ...:     print(repr(s))
   ...:
  ...
0 ...
1 ...

[2 rows x 2 columns]

```

but that's not too useful...

Resolving patch:
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -636,10 +636,14 @@ def to_string(self):
                     mid = int(round(n_cols / 2.))
                     mid_ix = col_lens.index[mid]
                     col_len = col_lens[mid_ix]
-                    adj_dif -= (col_len + 1)  # adjoin adds one
+                    # adjoin adds one
+                    adj_dif -= (col_len + 1)
                     col_lens = col_lens.drop(mid_ix)
                     n_cols = len(col_lens)
-                max_cols_adj = n_cols - self.index  # subtract index column
+                # subtract index column
+                max_cols_adj = n_cols - self.index
+                # GH-21180. Ensure that we print at least two.
+                max_cols_adj = max(max_cols_adj, 2)
                 self.max_cols_adj = max_cols_adj
 
                 # Call again _chk_truncate to cut frame appropriately
@@ -778,7 +782,7 @@ def space_format(x, y):
 
             str_columns = list(zip(*[[space_format(x, y) for y in x]
                                      for x in fmt_columns]))
-            if self.sparsify:
+            if self.sparsify and len(str_columns):
                 str_columns = _sparsify(str_columns)
 
             str_columns = [list(x) for x in zip(*str_columns)]


Test patch:
diff --git a/pandas/tests/io/formats/test_format.py b/pandas/tests/io/formats/test_format.py
--- a/pandas/tests/io/formats/test_format.py
+++ b/pandas/tests/io/formats/test_format.py
@@ -305,6 +305,44 @@ def test_repr_non_interactive(self):
             assert not has_truncated_repr(df)
             assert not has_expanded_repr(df)
 
+    def test_repr_truncates_terminal_size(self):
+        # https://github.com/pandas-dev/pandas/issues/21180
+        # TODO: use mock fixutre.
+        # This is being backported, so doing it directly here.
+        try:
+            from unittest import mock
+        except ImportError:
+            mock = pytest.importorskip("mock")
+
+        terminal_size = (118, 96)
+        p1 = mock.patch('pandas.io.formats.console.get_terminal_size',
+                        return_value=terminal_size)
+        p2 = mock.patch('pandas.io.formats.format.get_terminal_size',
+                        return_value=terminal_size)
+
+        index = range(5)
+        columns = pd.MultiIndex.from_tuples([
+            ('This is a long title with > 37 chars.', 'cat'),
+            ('This is a loooooonger title with > 43 chars.', 'dog'),
+        ])
+        df = pd.DataFrame(1, index=index, columns=columns)
+
+        with p1, p2:
+            result = repr(df)
+
+        h1, h2 = result.split('\n')[:2]
+        assert 'long' in h1
+        assert 'loooooonger' in h1
+        assert 'cat' in h2
+        assert 'dog' in h2
+
+        # regular columns
+        df2 = pd.DataFrame({"A" * 41: [1, 2], 'B' * 41: [1, 2]})
+        with p1, p2:
+            result = repr(df2)
+
+        assert df2.columns[0] in result.split('\n')[0]
+
     def test_repr_max_columns_max_rows(self):
         term_width, term_height = get_terminal_size()
         if term_width < 10 or term_height < 10: