HTML (and text) reprs for large dataframes.
State: closed

As discussed in #4886, the HTML representation of DataFrames currently starts off as a table, but switches to the condensed info view if the table exceeds a certain size (by default, more than 60 rows or 20 columns). I've seen this confusing users, who think that they suddenly have a completely different kind of object, and don't understand why.

With these changes, the HTML repr always displays the table, but truncates it when it exceeds a certain size. It reuses the same options, `display.max_rows` and `display.max_columns`.

Before:

![pandas_long_repr_before](https://f.cloud.github.com/assets/327925/1575857/ae103a8c-5153-11e3-8a2c-f770cc76bdc2.png)
![pandas_wide_repr_before](https://f.cloud.github.com/assets/327925/1575858/ae40b22a-5153-11e3-8c1e-3d9a632a5136.png)

After:

![pandas_long_repr_after](https://f.cloud.github.com/assets/327925/1575860/b35a2caa-5153-11e3-9887-69302703f545.png)
![pandas_wide_repr_after](https://f.cloud.github.com/assets/327925/1575861/b35c7bc2-5153-11e3-872c-6a85523f93db.png)


Labels:

Comments:
a couple of issues: edge formatting on wide tables, and the empty dataframe
repr looks off.

Also, the PR affects only the html repr so (QT)console users will see different behavior then
ipnb users. 

![html_repr_shot](https://f.cloud.github.com/assets/1820866/1586333/5ff5cc5c-5227-11e3-9a7f-b7aebcadd5db.png)

I've fixed the wide display issue - it wasn't truncating the extra row added for the row index names.

The empty dataframe repr is the same as what's displayed in my system installation of pandas - I agree that it's a bit odd, but I think that's a separate issue.

I agree, the behaviour of plain text reprs should be similar. Do you want me to tackle that in this PR, or separately?

this is related to #1889 as well

That'd be great, I think it fits here ok.

Not sure about putting this in 0.13, I've had bad luck with last minute changes to 
display code. @jreback?

I think this is fine with a couple of minor issues:

need a mention in v0.13.0 and main docs
about this default and how to use .info() to get existing summary view

the hard codes for max rows / columns should come purely from the options and not hard code the functions 
can u change that

I also mentioned it in the issue, but what do you think of showing `first 30 ... last 30` rows/cols instead of `first 60 ...`? As is done for the Series html repr.
(It's also what is proposed in #1889)

Makes sense to me, but can be done in a subsequent PR if needs be.
ipnb is solidifying it's browser<->kernel message passing, may soon be
time to revisit the grid view from (with paging) https://github.com/pydata/pandas/issues/2974, this time with 
built-in functionality rather then an in-process web server (like [Exhibitionist](https://github.com/Exhibitionist) does).

I have:
- Made the plain text reprs match the HTML reprs (truncating with `...` beyond max_rows/max_columns). To get the info view, you need to call the `info()` method.
- Removed the defaults for max_rows/max_cols from the methods to which they are passed - so calling `to_string()` or `to_html()` will by default show the entire DataFrame untruncated. Only the reprs automatically truncate.
- Removed the `max_info_rows` option - this was only used when displaying the info view for a repr.
- Documented the changes.

- Better deprecate `max_info_rows` rather then remove it, we may wish to move it's
  enforcement over to `info()`. There are examples of doing that in `config_init.py`. Actually,
  just generally deprecate rather then remove to reduce friction.
- you removed some of the ugliest code I've ever written - good omen.
  can you have a look at https://github.com/pydata/pandas/issues/1610?source=cc and
  see if that raises any issues with the changes? (regressions)

`max_info_rows` is back, deprecated.

I've had a brief look at #1610 - I don't think this should cause a regression, because `format.get_console_size()` checks whether it's in an interactive session.

I was very wrong with my initial objections, this is just great.

It's impossible to set default values for for `max_rows` and `max_columns` 
that make things look good on both ipnb and qtconsole (which I usually use),
but that's a pre-existing issue. IPython scroller for tall output seems too large
to me, as well.

Regardless - tested this and liked it, +1 to merge. 

@jreback, any more issues to address before the green button?

is it possible to have an option to do the exisiting behavior , but default to the new

maybe `display.notebook_repr_html = 'info'` ?

if its easy I would add this to provide back compat, if not then ok (w/o going back to @y-p admitted 'ugliest' code)

in the terminal, with `display.expand_frame_repr=False` and `display.width=0` so that auto-detection is used,
the truncation doesn't obey the width detection (since it depends on number of columns,
not terminal width). Not a blocker.

I think it should be easy enough to have an option to revert to the old behaviour (at least roughly - I'd rather not restore `max_info_rows` as well). I'll do one option for both the terminal and the notebook: `display.large_repr = 'truncate' | 'info'`.

Truncation to terminal width is harder, because that would have to propagate down into the actual formatting code, and no doubt deal with various corner cases.

Added the option in the form I described in my last message.

When truncating, having a footer with total row count would eliminate the need 
to use `df.info` in many cases and so reduce the impact of the change on existing users. 
(For example, after filtering a frame you're often interested in the size of the result).

Edit: as a header is probably better, since in ipnb you may be forced to scroll down manually to 
expose that part of the view.

I played around with some different options: showing it below the table looked more natural, and I opted to show it whether or not the table is truncated. The format is "61 rows × 26 columns". In the terminal, it shows up in [square brackets] to highlight that it's not part of the table.

The failing test attempts to roundtrip a dataframe to and from the clipboard. It tests various ways of doing this, but one of them (passing `excel=False` :confused:) will simply write `str(df)` to the clipboard. That would already not work for any dataframe large enough to get the info repr, but the test only uses a 5 × 3 frame.

Should we attempt to fix that, or simply remove the code path that writes `str(df)` to the clipboard.

The size issue is known: https://github.com/pydata/pandas/issues/5346?source=cc, re confusion see https://github.com/pydata/pandas/pull/5070.

`to_clipboard(excel=false)` should probably use `show_dimensions=False` and use
to_string directly to avoid truncation. 

I've made the clipboard use `to_string()` instead of `str()`, which should also fix #5346. We'll see what Travis says.

However, now I appear to have a merge conflict. What's the preferred strategy for pandas: rebase, merge into my branch, or let whoever merges the PR handle it?

you need to clear merge conflicts via rebasing 

see here: https://github.com/pydata/pandas/wiki/Using-Git

will need you to squash down before merging a well

Rebased, squashing a couple of commits where I had undone some change.

Mercilessly squashing to 1 commit will make life a easier imo... 

@jreback perhaps we should add that to wiki?

sure feel free to update/expand wiki

I don't follow why squashing the whole PR to one commit would be useful. It seems to defeat the point of a DVCS.

It looks live Travis failed to post the results, but the build for 643267c finished successfully.

in the context of this PR having a lot of commits is fine
however when this is merged in it is quite confusing because looking at a single commit doesn't tell you much
this pr is a unit and seeing a small number of commits is more useful that a large number that reflect individual development processes

better to have a series of self contained commits that make sense logically - a nice progression is a commit for the tests, docs, and code change

also, if it becomes necessary to revert then it is much easier to revert a single commit

so pls squash down to logical commits 

yup, no need for a single commit but the history is meant to be explantory of the change 
not a documentary on the development process (back-and-forth changes for example are useless).

You can always revert the merge commit itself to revert the PR though, so that's not relevent.

On the contrary, I think the history should, for the most part, record the development process. The merge commits record entire new features landing. However, it's not my project, so I'm happy to squash to logical commits if you tell me which ones you consider 'logical'.