Add DataFrame.view() to display contents interactively.
State: closed

This is a first attempt at addressing #9179 

We add a DataFrame.view method to visualize the contents of a dataframe interactively.
The viewing itself is performed using the "tabview" module from https://github.com/firecat53/tabview

A few initial notes:
- tabview requires python3 currently, so please test using python3. There are no significant changes to make tabview work on python 2.7 as well but I'd like some feedback first.
- tabview tries to sys.exit() on quit even when loaded as a module. Also needs a fix. ipython complains, but prevents the module to exit.
- I have no idea where it would be more appropriate to put the dependency/suggestion to install tabview.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Ok, here's the revised model. From the commit:

Rename `view` to `interact` to be consistent on each object.

Implement a generic `pandas.interact` method which handles the conversion to a simpler list of lists. In turn, `interact` calls the internal `interact_list`, which currently uses tabview as the main backend.

This allows a replacement viewer to hook to the desired level without reimplementing boilerplate code. The current interact() tries to do a better job at detecting if the data has a builtin index and/or column names.

I also added some initial documentation. The master branch of https://github.com/wavexx/tabview also supports python 2.7 now, and should eventually get integrated.

None of the viewer parameters (start, fixed_header, etc) are currently supported by tabview. I'm going to add that later.

@jreback: comments on the API design?

Hello,

why naming method `interact` and not `view` as previously defined or maybe `display` ?
`view`or `display` is more understandable in my mind.
and maybe `edit`.

Kind regards

On 01/07/2015 06:21 PM, scls19fr wrote:

> Hello,
> 
> why naming method `interact` and not `view` as previously defined or maybe `display` ?
> 
> Kind regards

view() might be confused in the "table subset" sense.
Plus, Series (if I'm not mistaken) had already a view() method.
I wanted something consistent everywhere.

Ok. maybe `.display()` ? or `.disp()` was a better name as it's read only
@jreback your opinion ?

On 01/07/2015 06:28 PM, scls19fr wrote:

> Ok. maybe `display` ? or `disp` was a better name as it's read only

I can foresee the viewer being able to manipulate the data in the future
and just return the updated dataset. For instance, you can already sort
the data with tabview, but adding stuff like regular expression filters
and simple numerical operations would be quite convenient to add in the
future.

Also, interact() is pretty familiar to those that used the
"code.interact" modules.

It's a bit long to type though.

As a proof of concept, I wanted to implement a graphical viewer instead.
Here's a quick test:

https://github.com/wavexx/gtabview

It can be used with the framework as proposed in this pull request.

This is very cool, but I have a few thoughts/concerns:
1. This implementation converts the entire dataframe into Python native lists. This is OK for small data, but will be needlessly slow (and use lots of extra memory) for medium to big data. It would be a much better idea to directly reuse the numpy arrays which pandas uses to store information (`.values`)
2. Does the `interact` logic really belong in pandas? I can imagine different interactive backends using different logic for displaying the values from a dataframe depending on the needs of the backend. (For example, some backends would probably use my suggestion of using `.values` directly.)
3. Does this functionality even belong in pandas proper? For an example of an alternative approach, see [qgrid](http://nbviewer.ipython.org/github/quantopian/qgrid/blob/master/qgrid_demo.ipynb).

On 01/09/2015 07:23 PM, Stephan Hoyer wrote:

> This is very cool, but I have a few thoughts/concerns:
> 1. This implementation converts the entire dataframe into Python
>    native lists. This is OK for small data, but will be needlessly slow
>    (and use lots of extra memory) for medium to big data. It would be a
>    much better idea to directly reuse the numpy arrays which pandas uses
>    to store information (`.values`)

It's a known limitation. A viewer replacing the topmost call
(pd.interact()) could avoid the conversion if capable enough.

> 1. Does the `interact` logic really belong in pandas? I can imagine
>    different interactive backends using different logic for displaying
>    the values from a dataframe depending on the needs of the backend.
>    (For example, some backends would probably use my suggestion of using
>    `.values` directly.)

2 and 3 are legitimate questions.

The notebook interface (or qgrid - for example) could register itself as
the default handler of the interact() method by just replacing it.

The user would just know that interact() is expected to bring up an
interactive display according to his environment.

I'm also not sure if that constitutes an improvement over calling
another package explicitly.

The trouble right now is that there is no unique package that would
provide such a function on every environment, but something in the likes
of this PR could help bring us those faster, in an uniform way.

`view` is a confusing choice for naming, since it has a completely different meaning in pandas/numpy land (shallow copy of an array that's a different Python object but references the same underlying data).

@jtratner: note that in the last revision, it was changed to "interact" for this purpose.

@wavexx in response to your question on the ML about how a registration system might work:

If you're looking for an example of how a registration system _could_ work, pandas' Excel handling manages to do registration and allow swapping of parts transparently to the user (check out pandas/io/excel.py to see the majority of it in action).

Potentially the easiest way to integrate this with a setting could be to have a top level function (e.g., `interact`) that pulls from pandas options to determine how to interactively display the NDFrame.  The option could be a fully-qualified name of a function,  e.g. `pd.set_option("interact.pager", "tabview.view")` and then `pd.interact` could pull out the function from the string and then call it, all without the need for interaction from the user.

---

That said, I still don't understand what the benefit of this method is for pandas. What's the utility of defining a protocol when the user is going to have to set which environment they're in regardless? saving a `from tabview import view as interact` at the toplevel?

Maybe you could make this work elegantly with the rich HTML repr that NDFrames already have (I think it's called `__repr_html__` or something - which gets used by IPython QTConsole and IPython notebook.)? Then, you could just add something like `pd.set_option('rich_repr', True)` at the top of your IPython notebook and it'd automatically show an interactive pager when you show the repr of an NDFrame.  It might add a little more complexity under the hood, but it feels like it could be a better path forward.

Jeff, that was valuable advice. Since I never used the notebook
interfaces (just inefficient IMO) I have no idea of what was already
done in that regard.

I'll have a look.

closing as #9179 now closed