Set pd.options.display.max_columns=0 by default
State: closed

Update: Remove everything related to `max_rows` and only deal with `max_columns` in this PR.

Changed `max_columns` to `0` (automatically adapt the number of displayed columns to the actual terminal width) when run in a terminal ~~and `max_rows` to `20` (because I'd like to see the "whole" data frame at a glance like in R's tibble)~~.

 - [x] closes #16579
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master -u -- "*.py" | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Could you provide some before / after screenshots? This will need some feedback from the wider community, since the visual display of DataFrames is an API grey-zone; prepare for bike-shedding 😄 

We still have some situations  where we can't detect the terminal width reliably. We need to make sure the output is handled as well as possible there.

I'm +1 for reducing the number of rows displayed. Typically I use 10 rows.
Here's the current output when printing a data frame with shape (5, 10) in a terminal with 100 characters width:

![before](https://user-images.githubusercontent.com/4377312/28365444-e460d4c6-6c88-11e7-8cbb-51527ac90026.png)

And here is the same data frame after the proposed change:

![after](https://user-images.githubusercontent.com/4377312/28365448-eb1f8500-6c88-11e7-9653-343914cda5bb.png)

I'm OK with this.  I do think it needs a big note in the whatsnew, with instructions on how to change back (maybe a ref to IPython config too).  Also looks like some tests that will need adjusted.
yeah changing the column default to auto-detect is fine. I personally use an event smaller default for ``max_rows``, but 20 looks fine. Pls update docs, a before/after screen shot that we can include in the what's new would be good (IOW your above ones). You will have to fix some tests.
Do you really want screenshots or could we mimic the old and new behavior with markdown? If you want screenshots let me modify them so that my username doesn't show up. Regarding the tests, I'll see what I can do (I certainly didn't expect to break so many tests just by changing one value :smile:).
@cbrnr 

the issue is that we are now auto-detecting and so the actual terminal width matters. yes certainly we can 'set' it so it works and show in mark down. I think screenshots might be more clear here though.

I see, I'll provide new screenshots then. Thanks for pointing out the issue, of course this makes a huge difference!
Also, the IPython QtConsole doesn't play nicely with `pd.options.display.column_width=0`:

![screen shot 2017-07-20 at 15 23 22](https://user-images.githubusercontent.com/4377312/28419500-62522a3e-6d5f-11e7-83d8-409b19dfbea1.png)

@takluyver - assuming this hasn't changed, but do you know offhand if it's still not possible to detect terminal size running in the qtconsole?  Found an older SO answer from you, thanks!
https://stackoverflow.com/questions/27813132/determining-terminal-width-in-ipython-qtconsole

No, sorry. It's a conceptual mismatch, not just a technical one. The kernel is producing output for (potentially) several frontends which may be receiving it at the moment, and for applications which may later display saved copies of that output. So questions about the shape of 'the' output area don't really make sense in the Jupyter protocol.

As I see it, the real issue is that the Qt console doesn't understand any structured way of representing a table. We turned off its HTML support because it's just too limited and tends to break richer HTML written for the notebook frontend. I have occasionally advocated for a 'simple HTML' repr option which the Qt console would display, but it's never been high priority.

In the long run, I think our plan is to make an HTML console and use QtWebkit to embed it in Qt applications. Then it should be able to display HTML tables.
Thanks @takluyver - so this isn't going to work until HTML tables are rendered (which would be awesome BTW). Is it possible to determine if Pandas is running in a real terminal or not? Could someone point me to the relevant code parts?
It's possible to distinguish terminal IPython from IPython as a kernel for a Jupyter frontend, something like this:

```python
try:
    ip = get_ipython()
except NameError:
    ... # Not IPython
else:
    if hasattr(ip, 'kernel'):
        ... # IPython as a Jupyter kernel
    else:
        ... # IPython terminal interface
```
@cbrnr can you rebase this and compose a note for the what's new?
Sure, but many tests need to be fixed and I don't know if I have the time to do that. I guess this change should be added to the API changes section?
@cbrnr this would need its own sub-section in API

yes would need to fix any tests.
cf #16800 #4907
Many tests rely on calling `str` on a data frame with the current default max number of columns. I'm not sure this will be easy to fix. This would be easy to fix if Pandas supported a pandasrc config file as proposed in #4907.
> Many tests rely on calling str on a data frame with the current default max number of columns. I'm not sure this will be easy to fix. This would be easy to fix if Pandas supported a pandasrc config file as proposed in #4907.

nothing to do with that issue 
all options already have defaults
for testing you need to setup the specific conditions for tests
generally using pd.option_context
OK, I've fixed almost all tests. Only 2 tests still fail, but I'm not sure if these failures are related to my changes:

- `pandas/tests/tseries/test_timezones.py:1290: AssertionError`
- `pandas/tests/scalar/test_timestamp.py:1110: AssertionError`

Here's the complete test output:

```
___________________________________ TestTimestamp.test_timestamp ___________________________________
[gw1] darwin -- Python 3.6.3 /Users/clemens/anaconda/envs/pandas_dev/bin/python
self = <pandas.tests.scalar.test_timestamp.TestTimestamp object at 0x10ac29940>

    def test_timestamp(self):
        # GH#17329
        # tz-naive --> treat it as if it were UTC for purposes of timestamp()
        ts = Timestamp.now()
        uts = ts.replace(tzinfo=utc)
        assert ts.timestamp() == uts.timestamp()
    
        tsc = Timestamp('2014-10-11 11:00:01.12345678', tz='US/Central')
        utsc = tsc.tz_convert('UTC')
        # utsc is a different representation of the same time
        assert tsc.timestamp() == utsc.timestamp()
    
        if PY3:
            # should agree with datetime.timestamp method
            dt = ts.to_pydatetime()
>           assert dt.timestamp() == ts.timestamp()
E           AssertionError: assert 1510231568.085538 == 1510235168.085538
E            +  where 1510231568.085538 = <built-in method timestamp of datetime.datetime object at 0x10c197d50>()
E            +    where <built-in method timestamp of datetime.datetime object at 0x10c197d50> = datetime.datetime(2017, 11, 9, 13, 46, 8, 85538).timestamp
E            +  and   1510235168.085538 = <built-in method timestamp of Timestamp object at 0x10c16fb10>()
E            +    where <built-in method timestamp of Timestamp object at 0x10c16fb10> = Timestamp('2017-11-09 13:46:08.085538').timestamp

pandas/tests/scalar/test_timestamp.py:1110: AssertionError
________________________________ TestTimeZones.test_replace_tzinfo _________________________________
[gw1] darwin -- Python 3.6.3 /Users/clemens/anaconda/envs/pandas_dev/bin/python
self = <pandas.tests.tseries.test_timezones.TestTimeZones object at 0x10e8f2f98>

    def test_replace_tzinfo(self):
        # GH 15683
        dt = datetime(2016, 3, 27, 1)
        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
    
        result_dt = dt.replace(tzinfo=tzinfo)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
    
        if hasattr(result_dt, 'timestamp'):  # New method in Py 3.3
            assert result_dt.timestamp() == result_pd.timestamp()
        assert result_dt == result_pd
        assert result_dt == result_pd.to_pydatetime()
    
        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
    
        if hasattr(result_dt, 'timestamp'):  # New method in Py 3.3
>           assert result_dt.timestamp() == result_pd.timestamp()
E           AssertionError: assert 1459036800.0 == 1459040400.0
E            +  where 1459036800.0 = <built-in method timestamp of datetime.datetime object at 0x10e8effd0>()
E            +    where <built-in method timestamp of datetime.datetime object at 0x10e8effd0> = datetime.datetime(2016, 3, 27, 1, 0).timestamp
E            +  and   1459040400.0 = <built-in method timestamp of Timestamp object at 0x10e8fcf48>()
E            +    where <built-in method timestamp of Timestamp object at 0x10e8fcf48> = Timestamp('2016-03-27 01:00:00').timestamp

pandas/tests/tseries/test_timezones.py:1290: AssertionError
```

Any ideas?
@cbrnr ignore those, see https://github.com/pandas-dev/pandas/issues/18037

python .timestamp() uses the local timezone to convert things, needs to be put into a consistent tz so it works for everyone.
OK cool, so let's wait if CIs come back happy (except for these 2 timezone-related ones). Could you help me with the whats_new entry (because we've agreed that this should be prominently visible)?

Also, I hope that my changes to the tests are OK, I mostly set the values for `max_columns` and `max_rows` to their old defaults 20 and 60, respectively.
whatsnew, make a new subsection. then put a screen shot of the before and one of the after. then it should read as if you are a user wanting to know whether this change will affect you (e.g. if you use ipython, the interpreter, etc).
for 0.22
A new subsection under "New features"? It's not really a new feature, but it doesn't fit into the other categories either.
under api breaking changes
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=h1) Report
> Merging [#17023](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/8dac633142daa8d5bcd0cf77ad89b97628d474eb?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `66.66%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17023/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17023      +/-   ##
==========================================
- Coverage   91.42%   91.38%   -0.05%     
==========================================
  Files         163      163              
  Lines       50068    50071       +3     
==========================================
- Hits        45776    45755      -21     
- Misses       4292     4316      +24
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.18% <66.66%> (-0.03%)` | :arrow_down: |
| #single | `40.39% <66.66%> (-0.04%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `96.09% <66.66%> (-2.26%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `63.38% <0%> (-1.82%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS5weQ==) | `92.02% <0%> (-0.02%)` | :arrow_down: |
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `96.01% <0%> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `95.72% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=footer). Last update [8dac633...85d6225](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=h1) Report
> Merging [#17023](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/6c0c27730fb5b20978824fb380449cd3acef55f7?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `73.33%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17023/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17023      +/-   ##
==========================================
+ Coverage   91.82%   91.84%   +0.01%     
==========================================
  Files         152      152              
  Lines       49235    49245      +10     
==========================================
+ Hits        45212    45230      +18     
+ Misses       4023     4015       -8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.23% <73.33%> (+0.01%)` | :arrow_up: |
| #single | `41.89% <66.66%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `98.24% <100%> (ø)` | :arrow_up: |
| [pandas/io/formats/terminal.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvdGVybWluYWwucHk=) | `20.98% <66.66%> (+4.54%)` | :arrow_up: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `99.24% <80%> (-0.76%)` | :arrow_down: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `96.19% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.73% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/indexes/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9wZXJpb2QucHk=) | `92.61% <0%> (ø)` | :arrow_up: |
| [pandas/core/strings.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3RyaW5ncy5weQ==) | `98.32% <0%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.18% <0%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/missing.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL21pc3NpbmcucHk=) | `91.07% <0%> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `95.85% <0%> (ø)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/pandas-dev/pandas/pull/17023/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=footer). Last update [6c0c277...f795914](https://codecov.io/gh/pandas-dev/pandas/pull/17023?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Only one Travis job failed - could you please restart it (it stalled)?

Also, is there any way to look at the rendered docs?
OK, I built it locally and it looks good IMO (although the images are rather large). I can post a screenshot if you want.
![pandas-doc-build-html-whatsnew html](https://user-images.githubusercontent.com/4377312/32656969-3f18945a-c614-11e7-9bba-3fc7ffcf344c.png)