Tidy representation when truncating dfs
State: closed

This PR closes #5603

Dataframes are now truncated centrally (similar to pd.Series). 

Before: 
![trunc_before](https://cloud.githubusercontent.com/assets/832380/2991707/9050ffe4-dc82-11e3-8a2c-083b09dae672.png)
After: 
![trunc_after](https://cloud.githubusercontent.com/assets/832380/3000645/649bf09e-dd26-11e3-889e-33a052b13554.png)

Ipython notebook: 

Simple Index: 
![notebook_trunc](https://cloud.githubusercontent.com/assets/832380/2991894/0031dd44-dc86-11e3-8d1c-8dc8086edac4.png)

MultiLevel Index: 
![notebook_multi](https://cloud.githubusercontent.com/assets/832380/2991895/05975656-dc86-11e3-886a-c811f4fc3fe5.png)


Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
can you put up a before/after in the top of the PR?

how long is the separating line? do we care? @jorisvandenbossche 

also, are we changing defaults for `max_rows` / `max_columns` ?

also going to need a new sub-section in v0.14.0 showing pictures like this (FYI use np.arange when creating the frame instead of Nan's)

include the pictures as pngs (so u can get the before after)

Should I place the pngs under pandas/doc/source and add them to the repo? 

they go in `pandas/doc/source/_static`, see http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#v0-13-0-january-3-2014 (look at the code in v0.13.0.txt), to see how to display it (and add to the repo)

Ok added a section in the docs: 

![screenshot from 2014-05-10 17 15 52](https://cloud.githubusercontent.com/assets/832380/2936326/0f111a48-d856-11e3-9aa2-826e2ca16ca1.png)

ok, that looks good

@jorisvandenbossche @TomAugspurger @jseabold @cpcloud 

?

@bjonen also, pls squash down to a smaller number of commits

You'll need to push your changes to the docs too.

I'd say change "and a ... signaled the end" to "and an ellipse (...) signaled the end"

Maybe some performance tests are needed? I can remeber there were some problems last time this was changed for 0.13?

@bjonen see here:  https://github.com/pydata/pandas/wiki/Performance-Testing

pls post anything > 1.2

Ok about the performance tests. 

What about the other ways to output a df - to_html, to_latex etc. 

I'm guessing that at least to_html is important to keep consistent with what to_string is doing. What do you guys think? 

hmm; normally to_html output the full output because its in a notebook which handles the full output via scrollbars. 

I guess down to 3 repr types now:
- full (no truncations),
- truncate (what this PR is about)
- info (summary view)

their are various options to switch from full to info now if the repr is too big, but truncate
is now much more the default

I think we might have to go over all the relevant options and put it out there on the ML
(this issue is already out there: https://github.com/pydata/pandas/issues/6547)

@jorisvandenbossche @jseabold ?

I don't use the notebook much so I don't know what the max_rows/cols defaults are, but from the code it seems that truncation also applies to html representations in the notebook. 

format.py l. 852

``` Python
        if len(self.frame) > self.max_rows:
            row = [''] + (['...'] * ncols)
            self.write_tr(row, indent, self.indent_delta, tags=None,
                          nindex_levels=1)
```

I am mostly done adapting this, except for the part that takes care of hierarchical indices. 

hmm I think the html repr needs to be consistent (as it's doing the same sort of truncation)

@TomAugspurger @jorisvandenbossche does html need to do the same? or is in normally turned off for html? (as in the notebook you have scroll bars for really large displays), or it just displays the info view?

this is super minor but how crazy would it be to use the `⋮` (`u'\u22ee'`) for the column ellipsis and `⋯` (`u'\u22ef'`) for the row ellipsis? even crazier might be to put `⋱` (`u'\u22f1'`) on the diagonal

Yes the html does the same (same rules for truncating or not, it also just follows the max_rows/max_columns options)

@cpcloud I like the idea. I'll add it once everything is working as expected. Getting closer... 

Would you be able to get the ellipse to center in the truncated area?

``` python
In [12]: df
Out[12]: 
                  0         1       ...         8         9
0         -0.047904  0.145739       ... -0.088891  1.131782
1         -2.326429  0.992864       ...  0.222462 -0.281965
.         .........  ........       ...  ........ .........
8          0.587134 -0.042532       ... -0.629570  0.690751
9         -1.339637 -0.612544       ...  0.500358 -1.195189

[10 rows x 10 columns]
```

(The formatting Github outputs is pretty close to what I actually see.)
It looks like it does center correctly in your examples from the original post.

Or maybe it's intentional? At https://github.com/pydata/pandas/pull/7086/files#diff-23878beaf55672cdc92c119f79fe492aR394 you call `rjust` instead of `center`.

Agreed about notebook and terminal (and qtconolse) all truncating in the same way.

@cpcloud I think we could do that as long as the terminal (or browser) is able to display them. Is that just check of `sys.getdefaultencoding()` or does the terminal have a different one?

Maybe that's best saved as a future enhancement.

@TomAugspurger Thanks, good catch. Changed to center. 

@cpcloud change is nice...let's make that an issue for 0.14.1 once this is out there....

can you also put in the top of the PR what the notebook looks like now? 

sorry.....I realize that I hadn't merged the other fix....rebase on master again (#7108) is now merged

lmk when you think you are ready (if you can squash great..., if not I can do it)

Ah ok, i'll squash and rebase then. 

I'm struggling a bit with running the vbench. I tried running it once and it got stuck. So if someone could run it that would be great - otherwise I'll give it a go again tonight. 

use `u(....)` instead of `u"..."` to escape unicode (this function handles the py2/3 issues

vbench see here: https://github.com/pydata/pandas/wiki/Performance-Testing

ping me when this is green....i'll squash it and take a look

Yes i ran

./test_perf.sh -b master -t HEAD

but it took a long time and finally hung up.