ENH (GH6568) Add option info_verbose
State: closed

This adds a info_verbose to the options. There's a small section in faq and basic introduction. Entry in v0.14 is still missing. 

Closes #6568


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design

Comments:
@jreback I know you were in favor of making it a seperate option instead of adding `'info_short'` to `display.large_repr`, but when looking at it now: if I see the option named `info_verbose` to be True or False, I expect this sets the default for the `info` method, while this actually only sets behaviour of `info` when this is used in the large dataframe repr. So I find this a bit confusing.
- what it actually means is `large_repr_info_verbose`, but this is then a bit a verbose name .. :-)
- it could also set the default for `df.info(verbose=..)` itself? (But I don't know if this is wanted)
- maybe nonetheless choose for `display.large_repr='info_short'` or `info_concise`? Or do you think this is more confusing? 

@bjonen @jreback What do you think?

this sets the default for df.info(...)

I agree it's only when the info repr is triggered in the first place

@bjonen if we change this option to really be a subset of large_repr

iow have 3 options: False, True/verbose (verbose=False), concise (verbose=False)

that would work yes?

though I think the default should be concise (so maybe have to fiddle with this a bit to be backward compat)

Thanks for your comments. 

I think it's a good idea to directly control `df.info(verbose=..)`. Then it makes sense to have a separate option `info_verbose`, as it is independent of `display.large_repr`.  

3 options for `large_repr` works too for me. Why do you prefer to leave the default for df.info unchanged? 

@jreback @jorisvandenbossche 
Which solution should we go with? 

So the options would be:
- having a `info_verbose` (True/False) option which would set the default for `df.info(verbose=..)` itself
- adding an option to `large_repr` ('truncate' (default), 'info', and newly 'info_short' or 'info_concise') that only sets the default for `info` when called in the repr of `df`

Personally I don't really have a preference (I won't use any of both options)

hmm I think I like the separate option

with a default of False (which is an API change)

Why a default of False? Is there a reason to change the behaviour of `df.info()`?

How do you think we should change the default on df.info? 
- We could have a wrapper that sets verbose and passes it to info but then we lose the ability to do 'df.info?' in ipython to get the default arguments. 
- Alternatively we could have something like below and somehow trigger `DataFrame.info = DataFrame.info_non_verbose` when the option 'info_verbose' changes. 

Any thoughts? 

```
    def info_verbose(self, verbose=True, buf=None, max_cols=None):
        self._info(verbose=verbose,buf=buf,max_cols=max_cols)

    def info_non_verbose(self, verbose=False, buf=None, max_cols=None):
        self._info(verbose=verbose,buf=buf,max_cols=max_cols)        

    def _info(self, verbose=True, buf=None, max_cols=None):
```

I am pretty sure the default was False in 0.12
I think got changed inadvertently in 0.13/0.13.1

@bjonen can u see if that was the case?

@jorisvandenbossche this doesn't need to be complicated
df.info() works as is and u can pass a parameter
all the option will do is to set the parameter 

Yes it did change in 0.13 but it seems it was intentional: https://github.com/pydata/pandas/issues/4886

"df.info() works as is and u can pass a parameter
all the option will do is to set the parameter"

If I understand correctly, this is the version currently implemented in the PR. That means the option value (True/False) will only have an impact when a `df` is printed and not when one calls df.info() because the default is hard coded. 

ok the default wasn't meant to change

change the signature to

df.info(verbose=None)

then handle a passed true/false as an override
if none then use the info_verbose option
which I think should default to False

Ok sounds good! 

Ok so 
- we leave large_repr default to 'truncate'. Then we are not in conflict with https://github.com/pydata/pandas/issues/4886
- We change `info_verbose` to default to False so that when a summary is printed it is the most concise summary. 

I'll adapt the PR. 

@jreback don't you have the `max_info_cols` option for that? To decide when the short and when the long summary is shown? Wouldn't that conflict with an option to set verbose to True or False?

```
max_info_columns is used in DataFrame.info method to decide if
        per column information will be printed.
```

But maybe we are just misinterpreting each other words, as I think the default for `verbose` in `df.info()` is already a long time `True`, see eg docs of 0.7: http://pandas.pydata.org/pandas-docs/version/0.7.0/generated/pandas.DataFrame.info.html

hmm don't know

@bjonen can u investigate this?

minor edit

can you post at the top of the issue what the 3 cases are (e.g. what the docs are going to show)
(an ipython picture, png)would be even better actually and you could include this in-line

@jorisvandenbossche ?

@jorisvandenbossche `max_info_columns` is currently doing something different. Maybe we can adjust it so that it can play the role of `info_verbose`. The current behavior doesn't safe any space. Also the default of 100 seems very high to me. 

`max_columns` doesn't seem to have an effect at all under large_repr = 'info'. 

In [12]: import pandas as pd

In [13]: df = pd.DataFrame(columns=['a','b','c'],index=pd.DatetimeIndex(start='19900101',end='20000101',freq='BM'))

In [15]: pd.options.display.large_repr = 'info'

In [16]: df
Out[16]:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 120 entries, 1990-01-31 00:00:00 to 1999-12-31 00:00:00
Freq: BM
Data columns (total 3 columns):
a    0 non-null object
b    0 non-null object
c    0 non-null object
dtypes: object(3)

In [17]: pd.options.display.max_columns = 1

In [18]: df
Out[18]:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 120 entries, 1990-01-31 00:00:00 to 1999-12-31 00:00:00
Freq: BM
Data columns (total 3 columns):
a    0 non-null object
b    0 non-null object
c    0 non-null object
dtypes: object(3)

In [19]: pd.options.display.max_info_columns = 1

In [20]: df
Out[20]:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 120 entries, 1990-01-31 00:00:00 to 1999-12-31 00:00:00
Freq: BM
Data columns (total 3 columns):
a    object
b    object
c    object
dtypes: object(3)

In [21]: df.info(False)
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 120 entries, 1990-01-31 00:00:00 to 1999-12-31 00:00:00
Freq: BM
Columns: 3 entries, a to c
dtypes: object(3)

@jreback Looking at previous commits in git: 

```
    git grep 'def info\(self, verbose=True.*\):' $(git rev-list --all) 
```

it seems that the default has been `True` up to now. 

@bjonen ok on the default then (I think I personally set it to False, but that is fine then)

you need to use a frame with > `max_info_columns` (e.g. 101) to get an effect

I reset the default below the number of columns (3) in the df (see previous post): 

pd.options.display.max_info_columns = 1

@bjonen hmm.. see if you can figure out from the tests what it is _supposed_ to do. All of the options have complex interactions. If its a 'bug' would rather fix than add a new option if we can (e.g. you are suggesting that if you have columns > `max_info_columns` then we basically switch `info_verbose` to `False` (instead of actually having an option), right?

Yes, exactly. I'll look into it and let you guys know. 

@bjonen That `max_columns` has no effect when `large_repr='info'` is normal, as this parameter defines how many columns are shown in the default display, not in the `info` display (for that is `max_info_columns`).

The strange behaviour of `max_info_cols` is a bug I think, a regression in current master. As with 0.13 I get:

```
In [1]: pd.__version__
Out[1]: '0.13.0'

In [5]: df = pd.DataFrame(np.random.randn(5,5))
In [6]: df
Out[6]:
          0         1         2         3         4
0  0.708876 -0.179273  1.367976 -0.929688 -1.138946
1  1.047154  1.049302 -0.248178 -0.957677  1.879843
2 -0.523272 -2.013742  2.064032 -1.389822  1.394960
3  0.224508  1.032544 -1.312425  0.123956  0.144831
4 -1.691660  0.952837  1.380545 -1.279794  1.026131

In [7]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 5 columns):
0    5  non-null values
1    5  non-null values
2    5  non-null values
3    5  non-null values
4    5  non-null values
dtypes: float64(5)

In [8]: pd.options.display.max_info_columns = 4

In [9]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Columns: 5 entries, 0 to 4
dtypes: float64(5)
In [10]:
```

which is much more logical and in line with the explanation (`max_info_columns is used in DataFrame.info method to decide if per column information will be printed.`)

But that seems like a seperate issue.

It seems the behavior was introduced in 0.13.1
Sticking with your example: 

In [22]: pd.**version**
Out[22]: '0.13.1'

In [27]: pd.options.display.max_info_columns = 100

In [28]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 5 columns):
0    5 non-null float64
1    5 non-null float64
2    5 non-null float64
3    5 non-null float64
4    5 non-null float64
dtypes: float64(5)
In [29]: pd.options.display.max_info_columns = 1

In [30]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 5 columns):
0    float64
1    float64
2    float64
3    float64
4    float64
dtypes: float64(5)

The explanation for `max_info_rows` reads: 
"df.info() will usually show null-counts for each column. For large frames this can be quite slow. max_info_rows and max_info_cols limit this null check only to frames with smaller dimensions then specified."

So the `max_info` options as they are implemented right now are concerned with improving display performance and not so much display style. 

ahh...I do recal this a bit, @y-p put this in IIRC

to not do the non-null check if you have a very large frame that would be displayed in a summary anyhow

but maybe it introduced a bug (as @jorisvandenbossche describes)

can you simply this (w/o creating more havoc!)

I just opened a seperate issue (#6939), as I thought this was seperate from this discussion? So can you maybe repeat that overthere? 
(But of  course, if it is not a regression, it has implications for this issue)

it was indeed a pr of @y-p : #5974

@bjonen ok....so pls change this PR to close #6939 in addition  (its the same fix)

Ok will do.