Repository: pandas-dev/pandas
Pull request: Set pd.options.display.max_columns=0 by default
Update: Remove everything related to `max_rows` and only deal with `max_columns` in this PR.

Changed `max_columns` to `0` (automatically adapt the number of displayed columns to the actual terminal width) when run in a terminal ~~and `max_rows` to `20` (because I'd like to see the "whole" data frame at a glance like in R's tibble)~~.

 - [x] closes #16579
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master -u -- "*.py" | flake8 --diff``
 - [x] whatsnew entry


Problem statement:
Can't pd.options.display.max_columns = 0 by default?
`pd.options.display.max_columns` is 20 by default, but when running in a terminal this value could be auto-detected by setting it to 0. I know that this setting can be customized, but for the average user it would be much more convenient if there was no extra step involved. Is there a way to detect if pandas is running in a terminal (where autodetection works)? If so, the value should be set to 0. Otherwise, for example in Jupyter notebooks, it could remain to be 20.

Hints from issue discussion:
Say you auto-detect (via get_terminal_size) and its 80, then what, how many columns should it be?

This is a pretty reasonable default.
Maybe I wasn't clear. If the width gets autodetected, a much more reasonable choice for `max_columns` would be `0`. That way, the printed object would fit within the terminal width.
@jreback could you reconsider this? Here's an example to show what I mean:

```
import numpy as np
import pandas as pd
pd.DataFrame(np.random.rand(5, 10))
```

In a terminal (mine has 100 columns), the default output of this dataframe looks like this:
```
          0         1         2         3         4         5         6  \
0  0.173421  0.463432  0.829303  0.557628  0.257765  0.712062  0.129705   
1  0.113790  0.310267  0.076708  0.576933  0.562503  0.623521  0.002787   
2  0.628581  0.740680  0.327314  0.129068  0.816575  0.470844  0.771247   
3  0.947981  0.984844  0.113195  0.449247  0.494137  0.349844  0.207327   
4  0.087635  0.678417  0.255774  0.924561  0.597169  0.089508  0.650698   

          7         8         9  
0  0.817547  0.337570  0.348707  
1  0.051971  0.983914  0.972759  
2  0.566701  0.627593  0.126499  
3  0.619493  0.292824  0.653050  
4  0.530746  0.773546  0.496640  
```

Now if I set `pd.options.display.max_columns = 0`, the output looks much nicer:

```
         0         1         2         3     ...           16        17        18        19
0  0.238365  0.438112  0.307173  0.226128    ...     0.547763  0.319377  0.688708  0.840808
1  0.153844  0.472930  0.128777  0.197224    ...     0.533801  0.530466  0.417650  0.174715
2  0.964627  0.626036  0.433339  0.834303    ...     0.036854  0.248769  0.932532  0.669207
3  0.375633  0.581266  0.026618  0.394931    ...     0.049323  0.195138  0.502164  0.454383
4  0.799988  0.137955  0.067402  0.012556    ...     0.587728  0.226844  0.447049  0.117987
```

So I was wondering why `pd.options.display.max_columns` can't be `0` by default?
@TomAugspurger ?
I usually set the max columns to 0 in my IPython init. I think it would be ok to change the default.

Do we still run into issues where we can't reliably detect the terminal width, or have those all been fixed?
I'll create a PR so that this stays on the radar. I'd also like to propose to change `pd.options.display.max_rows` to 20 instead of 60, because ideally I'd like to see a summary of the data frame including the column names. This is also how tibbles in R are displayed BTW.
Regarding the detection of the terminal width, I think it doesn't work in PyCharm's integrated terminal. However, the default of 80 is a reasonable value in this case.

Commit message: Move images to _static

Patch to review:
diff --git a/doc/source/options.rst b/doc/source/options.rst
index a82be4d84bf3f..48247eb48baaf 100644
--- a/doc/source/options.rst
+++ b/doc/source/options.rst
@@ -78,8 +78,8 @@ with no argument ``describe_option`` will print out the descriptions for all ava
 Getting and Setting Options
 ---------------------------
 
-As described above, :func:`~pandas.get_option` and :func:`~pandas.set_option` 
-are available from the pandas namespace.  To change an option, call 
+As described above, :func:`~pandas.get_option` and :func:`~pandas.set_option`
+are available from the pandas namespace.  To change an option, call
 ``set_option('option regex', new_value)``.
 
 .. ipython:: python
@@ -230,7 +230,7 @@ can specify the option ``df.info(null_counts=True)`` to override on showing a pa
    df.info()
    pd.reset_option('max_info_rows')
 
-``display.precision`` sets the output display precision in terms of decimal places. 
+``display.precision`` sets the output display precision in terms of decimal places.
 This is only a suggestion.
 
 .. ipython:: python
@@ -323,21 +323,21 @@ display.latex.multicolumn_format        'l'          Alignment of multicolumn la
 display.latex.multirow                  False        Combines rows when using a MultiIndex.
                                                      Centered instead of top-aligned,
                                                      separated by clines.
-display.max_columns                     20           max_rows and max_columns are used
+display.max_columns                     0 or 20      max_rows and max_columns are used
                                                      in __repr__() methods to decide if
                                                      to_string() or info() is used to
                                                      render an object to a string.  In
-                                                     case python/IPython is running in
-                                                     a terminal this can be set to 0 and
+                                                     case Python/IPython is running in
+                                                     a terminal this is set to 0 by default and
                                                      pandas will correctly auto-detect
-                                                     the width the terminal and swap to
+                                                     the width of the terminal and switch to
                                                      a smaller format in case all columns
                                                      would not fit vertically. The IPython
                                                      notebook, IPython qtconsole, or IDLE
                                                      do not run in a terminal and hence
                                                      it is not possible to do correct
-                                                     auto-detection. 'None' value means
-                                                     unlimited.
+                                                     auto-detection, in which case the default
+                                                     is set to 20. 'None' value means unlimited.
 display.max_colwidth                    50           The maximum width in characters of
                                                      a column in the repr of a pandas
                                                      data structure. When the column overflows,
@@ -402,9 +402,9 @@ display.html.table_schema               False        Whether to publish a Table
 display.html.border                     1            A ``border=value`` attribute is
                                                      inserted in the ``<table>`` tag
                                                      for the DataFrame HTML repr.
-display.html.use_mathjax                True         When True, Jupyter notebook will process 
-                                                     table contents using MathJax, rendering 
-                                                     mathematical expressions enclosed by the 
+display.html.use_mathjax                True         When True, Jupyter notebook will process
+                                                     table contents using MathJax, rendering
+                                                     mathematical expressions enclosed by the
                                                      dollar symbol.
 io.excel.xls.writer                     xlwt         The default Excel writer engine for
                                                      'xls' files.
@@ -422,7 +422,7 @@ io.hdf.dropna_table                     True         drop ALL nan rows when appe
 io.parquet.engine                       None         The engine to use as a default for
                                                      parquet reading and writing. If None
                                                      then try 'pyarrow' and 'fastparquet'
-mode.chained_assignment                 warn         Controls ``SettingWithCopyWarning``: 
+mode.chained_assignment                 warn         Controls ``SettingWithCopyWarning``:
                                                      'raise', 'warn', or None. Raise an
                                                      exception, warn, or no action if
                                                      trying to use :ref:`chained assignment <indexing.evaluation_order>`.
diff --git a/doc/source/whatsnew/v0.23.0.txt b/doc/source/whatsnew/v0.23.0.txt
index 107ce7855a00d..f8ef1ed53493a 100644
--- a/doc/source/whatsnew/v0.23.0.txt
+++ b/doc/source/whatsnew/v0.23.0.txt
@@ -658,6 +658,37 @@ Notice in the example above that the converted ``Categorical`` has retained ``or
 
 Note that the unintenional conversion of ``ordered`` discussed above did not arise in previous versions due to separate bugs that prevented ``astype`` from doing any type of category to category conversion (:issue:`10696`, :issue:`18593`).  These bugs have been fixed in this release, and motivated changing the default value of ``ordered``.
 
++.. _whatsnew_0230.api_breaking.pretty_printing:
+
+Better pretty-printing of DataFrames in a terminal
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Previously, the default value for the maximum number of columns was
+``pd.options.display.max_columns=20``. This meant that relatively wide data
+frames would not fit within the terminal width, and pandas would introduce line
+breaks to display these 20 columns. This resulted in an output that was
+relatively difficult to read:
+
+.. image:: print_df_old.png
+
+If Python runs in a terminal, the maximum number of columns is now determined
+automatically so that the printed data frame fits within the current terminal
+width (``pd.options.display.max_columns=0``) (:issue:`17023`). If Python runs
+as a Jupyter kernel (such as the Jupyter QtConsole or a Jupyter notebook, as
+well as in many IDEs), this value cannot be inferred automatically and is thus
+set to `20` as in previous versions. In a terminal, this results in a much
+nicer output:
+
+.. image:: print_df_new.png
+
+Note that if you don't like the new default, you can always set this option
+yourself. To revert to the old setting, you can run this line:
+
+.. code-block:: python
+
+  pd.options.display.max_columns = 20
+
+.. _whatsnew_0230.api:
+
 .. _whatsnew_0230.api.datetimelike:
 
 Datetimelike API Changes
diff --git a/pandas/core/config_init.py b/pandas/core/config_init.py
index 0edbf892172a9..b836a35b8cf29 100644
--- a/pandas/core/config_init.py
+++ b/pandas/core/config_init.py
@@ -13,6 +13,7 @@
 from pandas.core.config import (is_int, is_bool, is_text, is_instance_factory,
                                 is_one_of_factory, is_callable)
 from pandas.io.formats.console import detect_console_encoding
+from pandas.io.formats.terminal import is_terminal
 
 # compute
 
@@ -314,7 +315,11 @@ def table_schema_cb(key):
     cf.register_option('max_categories', 8, pc_max_categories_doc,
                        validator=is_int)
     cf.register_option('max_colwidth', 50, max_colwidth_doc, validator=is_int)
-    cf.register_option('max_columns', 20, pc_max_cols_doc,
+    if is_terminal():
+        max_cols = 0  # automatically determine optimal number of columns
+    else:
+        max_cols = 20  # cannot determine optimal number of columns
+    cf.register_option('max_columns', max_cols, pc_max_cols_doc,
                        validator=is_instance_factory([type(None), int]))
     cf.register_option('large_repr', 'truncate', pc_large_repr_doc,
                        validator=is_one_of_factory(['truncate', 'info']))
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
index 1731dbb3ac68d..eb96fb8793198 100644
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -625,7 +625,7 @@ def to_string(self):
                 max_len += size_tr_col  # Need to make space for largest row
                 # plus truncate dot col
                 dif = max_len - self.w
-                adj_dif = dif
+                adj_dif = dif + 1
                 col_lens = Series([Series(ele).apply(len).max()
                                    for ele in strcols])
                 n_cols = len(col_lens)
diff --git a/pandas/io/formats/terminal.py b/pandas/io/formats/terminal.py
index 4bcb28fa59b86..07ab445182680 100644
--- a/pandas/io/formats/terminal.py
+++ b/pandas/io/formats/terminal.py
@@ -17,7 +17,7 @@
 import sys
 import shutil
 
-__all__ = ['get_terminal_size']
+__all__ = ['get_terminal_size', 'is_terminal']
 
 
 def get_terminal_size():
@@ -48,6 +48,23 @@ def get_terminal_size():
     return tuple_xy
 
 
+def is_terminal():
+    """
+    Detect if Python is running in a terminal.
+
+    Returns True if Python is running in a terminal or False if not.
+    """
+    try:
+        ip = get_ipython()
+    except NameError:  # assume standard Python interpreter in a terminal
+        return True
+    else:
+        if hasattr(ip, 'kernel'):  # IPython as a Jupyter kernel
+            return False
+        else:  # IPython in a terminal
+            return True
+
+
 def _get_terminal_size_windows():
     res = None
     try:
diff --git a/pandas/tests/frame/test_dtypes.py b/pandas/tests/frame/test_dtypes.py
index 90daa9aa882c8..152159965036d 100644
--- a/pandas/tests/frame/test_dtypes.py
+++ b/pandas/tests/frame/test_dtypes.py
@@ -875,10 +875,11 @@ def test_astype_str(self):
                              columns=self.tzframe.columns)
         tm.assert_frame_equal(result, expected)
 
-        result = str(self.tzframe)
-        assert ('0 2013-01-01 2013-01-01 00:00:00-05:00 '
-                '2013-01-01 00:00:00+01:00') in result
-        assert ('1 2013-01-02                       '
-                'NaT                       NaT') in result
-        assert ('2 2013-01-03 2013-01-03 00:00:00-05:00 '
-                '2013-01-03 00:00:00+01:00') in result
+        with option_context('display.max_columns', 20):
+            result = str(self.tzframe)
+            assert ('0 2013-01-01 2013-01-01 00:00:00-05:00 '
+                    '2013-01-01 00:00:00+01:00') in result
+            assert ('1 2013-01-02                       '
+                    'NaT                       NaT') in result
+            assert ('2 2013-01-03 2013-01-03 00:00:00-05:00 '
+                    '2013-01-03 00:00:00+01:00') in result
diff --git a/pandas/tests/frame/test_repr_info.py b/pandas/tests/frame/test_repr_info.py
index 3e5aae10618e9..8fc6fef11798a 100644
--- a/pandas/tests/frame/test_repr_info.py
+++ b/pandas/tests/frame/test_repr_info.py
@@ -172,8 +172,8 @@ def test_repr_column_name_unicode_truncation_bug(self):
                                       'the CSV file externally. I want to Call'
                                       ' the File through the code..')})
 
-        result = repr(df)
-        assert 'StringCol' in result
+        with option_context('display.max_columns', 20):
+            assert 'StringCol' in repr(df)
 
     def test_latex_repr(self):
         result = r"""\begin{tabular}{llll}
diff --git a/pandas/tests/io/formats/test_format.py b/pandas/tests/io/formats/test_format.py
index 6c3b75cdfa6df..ab9f61cffc16b 100644
--- a/pandas/tests/io/formats/test_format.py
+++ b/pandas/tests/io/formats/test_format.py
@@ -961,7 +961,8 @@ def test_pprint_thing(self):
 
     def test_wide_repr(self):
         with option_context('mode.sim_interactive', True,
-                            'display.show_dimensions', True):
+                            'display.show_dimensions', True,
+                            'display.max_columns', 20):
             max_cols = get_option('display.max_columns')
             df = DataFrame(tm.rands_array(25, size=(10, max_cols - 1)))
             set_option('display.expand_frame_repr', False)
@@ -979,7 +980,8 @@ def test_wide_repr(self):
         reset_option('display.expand_frame_repr')
 
     def test_wide_repr_wide_columns(self):
-        with option_context('mode.sim_interactive', True):
+        with option_context('mode.sim_interactive', True,
+                            'display.max_columns', 20):
             df = DataFrame(np.random.randn(5, 3),
                            columns=['a' * 90, 'b' * 90, 'c' * 90])
             rep_str = repr(df)
@@ -987,7 +989,8 @@ def test_wide_repr_wide_columns(self):
             assert len(rep_str.splitlines()) == 20
 
     def test_wide_repr_named(self):
-        with option_context('mode.sim_interactive', True):
+        with option_context('mode.sim_interactive', True,
+                            'display.max_columns', 20):
             max_cols = get_option('display.max_columns')
             df = DataFrame(tm.rands_array(25, size=(10, max_cols - 1)))
             df.index.name = 'DataFrame Index'
@@ -1008,7 +1011,8 @@ def test_wide_repr_named(self):
         reset_option('display.expand_frame_repr')
 
     def test_wide_repr_multiindex(self):
-        with option_context('mode.sim_interactive', True):
+        with option_context('mode.sim_interactive', True,
+                            'display.max_columns', 20):
             midx = MultiIndex.from_arrays(tm.rands_array(5, size=(2, 10)))
             max_cols = get_option('display.max_columns')
             df = DataFrame(tm.rands_array(25, size=(10, max_cols - 1)),
@@ -1030,7 +1034,8 @@ def test_wide_repr_multiindex(self):
         reset_option('display.expand_frame_repr')
 
     def test_wide_repr_multiindex_cols(self):
-        with option_context('mode.sim_interactive', True):
+        with option_context('mode.sim_interactive', True,
+                            'display.max_columns', 20):
             max_cols = get_option('display.max_columns')
             midx = MultiIndex.from_arrays(tm.rands_array(5, size=(2, 10)))
             mcols = MultiIndex.from_arrays(
@@ -1044,15 +1049,16 @@ def test_wide_repr_multiindex_cols(self):
             wide_repr = repr(df)
             assert rep_str != wide_repr
 
-        with option_context('display.width', 150):
+        with option_context('display.width', 150, 'display.max_columns', 20):
             wider_repr = repr(df)
             assert len(wider_repr) < len(wide_repr)
 
         reset_option('display.expand_frame_repr')
 
     def test_wide_repr_unicode(self):
-        with option_context('mode.sim_interactive', True):
-            max_cols = get_option('display.max_columns')
+        with option_context('mode.sim_interactive', True,
+                            'display.max_columns', 20):
+            max_cols = 20
             df = DataFrame(tm.rands_array(25, size=(10, max_cols - 1)))
             set_option('display.expand_frame_repr', False)
             rep_str = repr(df)
@@ -1442,17 +1448,17 @@ def test_repr_html_mathjax(self):
             assert 'tex2jax_ignore' in df._repr_html_()
 
     def test_repr_html_wide(self):
-        max_cols = get_option('display.max_columns')
+        max_cols = 20
         df = DataFrame(tm.rands_array(25, size=(10, max_cols - 1)))
-        reg_repr = df._repr_html_()
-        assert "..." not in reg_repr
+        with option_context('display.max_rows', 60, 'display.max_columns', 20):
+            assert "..." not in df._repr_html_()
 
         wide_df = DataFrame(tm.rands_array(25, size=(10, max_cols + 1)))
-        wide_repr = wide_df._repr_html_()
-        assert "..." in wide_repr
+        with option_context('display.max_rows', 60, 'display.max_columns', 20):
+            assert "..." in wide_df._repr_html_()
 
     def test_repr_html_wide_multiindex_cols(self):
-        max_cols = get_option('display.max_columns')
+        max_cols = 20
 
         mcols = MultiIndex.from_product([np.arange(max_cols // 2),
                                          ['foo', 'bar']],
@@ -1467,8 +1473,8 @@ def test_repr_html_wide_multiindex_cols(self):
                                         names=['first', 'second'])
         df = DataFrame(tm.rands_array(25, size=(10, len(mcols))),
                        columns=mcols)
-        wide_repr = df._repr_html_()
-        assert '...' in wide_repr
+        with option_context('display.max_rows', 60, 'display.max_columns', 20):
+            assert '...' in df._repr_html_()
 
     def test_repr_html_long(self):
         with option_context('display.max_rows', 60):
@@ -1512,14 +1518,15 @@ def test_repr_html_float(self):
             assert u('2 columns') in long_repr
 
     def test_repr_html_long_multiindex(self):
-        max_rows = get_option('display.max_rows')
+        max_rows = 60
         max_L1 = max_rows // 2
 
         tuples = list(itertools.product(np.arange(max_L1), ['foo', 'bar']))
         idx = MultiIndex.from_tuples(tuples, names=['first', 'second'])
         df = DataFrame(np.random.randn(max_L1 * 2, 2), index=idx,
                        columns=['A', 'B'])
-        reg_repr = df._repr_html_()
+        with option_context('display.max_rows', 60, 'display.max_columns', 20):
+            reg_repr = df._repr_html_()
         assert '...' not in reg_repr
 
         tuples = list(itertools.product(np.arange(max_L1 + 1), ['foo', 'bar']))
@@ -1530,20 +1537,22 @@ def test_repr_html_long_multiindex(self):
         assert '...' in long_repr
 
     def test_repr_html_long_and_wide(self):
-        max_cols = get_option('display.max_columns')
-        max_rows = get_option('display.max_rows')
+        max_cols = 20
+        max_rows = 60
 
         h, w = max_rows - 1, max_cols - 1
         df = DataFrame({k: np.arange(1, 1 + h) for k in np.arange(w)})
-        assert '...' not in df._repr_html_()
+        with option_context('display.max_rows', 60, 'display.max_columns', 20):
+            assert '...' not in df._repr_html_()
 
         h, w = max_rows + 1, max_cols + 1
         df = DataFrame({k: np.arange(1, 1 + h) for k in np.arange(w)})
-        assert '...' in df._repr_html_()
+        with option_context('display.max_rows', 60, 'display.max_columns', 20):
+            assert '...' in df._repr_html_()
 
     def test_info_repr(self):
-        max_rows = get_option('display.max_rows')
-        max_cols = get_option('display.max_columns')
+        max_rows = 60
+        max_cols = 20
         # Long
         h, w = max_rows + 1, max_cols - 1
         df = DataFrame({k: np.arange(1, 1 + h) for k in np.arange(w)})
@@ -1555,7 +1564,8 @@ def test_info_repr(self):
         h, w = max_rows - 1, max_cols + 1
         df = DataFrame({k: np.arange(1, 1 + h) for k in np.arange(w)})
         assert has_horizontally_truncated_repr(df)
-        with option_context('display.large_repr', 'info'):
+        with option_context('display.large_repr', 'info',
+                            'display.max_columns', max_cols):
             assert has_info_repr(df)
 
     def test_info_repr_max_cols(self):
@@ -1575,8 +1585,8 @@ def test_info_repr_max_cols(self):
         # fmt.set_option('display.max_info_columns', 4)  # exceeded
 
     def test_info_repr_html(self):
-        max_rows = get_option('display.max_rows')
-        max_cols = get_option('display.max_columns')
+        max_rows = 60
+        max_cols = 20
         # Long
         h, w = max_rows + 1, max_cols - 1
         df = DataFrame({k: np.arange(1, 1 + h) for k in np.arange(w)})
@@ -1588,7 +1598,8 @@ def test_info_repr_html(self):
         h, w = max_rows - 1, max_cols + 1
         df = DataFrame({k: np.arange(1, 1 + h) for k in np.arange(w)})
         assert '<class' not in df._repr_html_()
-        with option_context('display.large_repr', 'info'):
+        with option_context('display.large_repr', 'info',
+                            'display.max_columns', max_cols):
             assert '&lt;class' in df._repr_html_()
 
     def test_fake_qtconsole_repr_html(self):


Code review comments:
diff_hunk: @@ -658,6 +658,37 @@ Notice in the example above that the converted ``Categorical`` has retained ``or
 
 Note that the unintenional conversion of ``ordered`` discussed above did not arise in previous versions due to separate bugs that prevented ``astype`` from doing any type of category to category conversion (:issue:`10696`, :issue:`18593`).  These bugs have been fixed in this release, and motivated changing the default value of ``ordered``.
 
++.. _whatsnew_0230.api_breaking.pretty_printing:
path: doc/source/whatsnew/v0.23.0.txt
text: @user1:
there is a '+' here at the beginning of the line that should not be there